Cluster Shutdown and Startup

The following describes how to do a controlled shutdown of the whole cluster and get back to a fully running state.

Shutting down the cluster

When shutting down an N node cluster, start with a graceful shutdown of the node with the highest node number (usually number 3) and wait until the node is fully shutdown before proceeding with the next one. This ensures that the quorum is kept as long as possible and in the end node 1 is the most up to date node.

Starting a fully shutdown cluster

After a controlled shutdown as described in Shutting down the cluster, the cluster nodes should automatically become Active starting with the most up to date node after startup.

If the cluster is unable to automatically become Active, the administrator needs to manually bootrap the cluster from the node with the most up to date data set. The administrator can identify the node that had an Active database status last before the shutdown by comparing the Last Transaction ID shown under the Cluster tab in WebConf of all the nodes.

Even after a power outage that seems instantaneous, the Last Transaction ID of all nodes should be compared before selecting a node to Force into Active.

Do the following:

  1. Power up all nodes.
  2. Wait a minute after all nodes have started to see if the cluster automatically becomes Active.
  3. If manual intervention is needed, select the node with the highest Last Transaction ID and use Force into Active on this node (and only this node).
  4. Wait until all N nodes are fully started, and the database status is Active on each node.