Adding/Removing Servers

We can add Consul servers to the existing cluster but this has to be done carefully. A minimum quorum of servers (N/2)+1 must be available for processing changes. This means that if there are 3 server nodes, at least 2 must be available.

Removing servers must be done carefully too to avoid causing an availability outage. For a cluster of N servers, at least (N/2)+1 must be available for the cluster to function. See this deployment table. If you have 3 servers, and 1 of them is currently failed, removing any servers will cause the cluster to become unavailable.

For this reason, the Consul documentation recommends adding servers first when you're both adding and removing.

Adding New Servers

In the previous chapters we have setup a 3-server Consul cluster. Now we will add 2 new servers to it.

Our desired final configuration is as follows:

On the new server nodes, create the config.json file as before. Everything remains same except for the retry_join and the start_join parameters, which will refer to the 5 servers now and the bootstrap_expect option, which was set to 3.

Here the question arises as to whether we should have the bootstrap_expect 3 or bootstrap_expect 5?

If we set it to 5, should we restart agents 1, 2 and 3 with the new bootstrap_expect value? The bootstrap_expect is used only once in the history of the cluster. It is used for the initial bootstrap, for example the first time that a leader is elected. So we would start with bootstrap_expect 3, but then the new servers do not even need the bootstrap_expect flag. They are joining an existing cluster which has a leader, so they are not performing a bootstrapping of the leader.

/etc/consul.d/server/config.json will look like:

server4.example.com

root@server4:~#vim /etc/consul.d/server/config.json
{
"bind_addr": "192.168.1.14",
"datacenter": "dc1",
"encrypt": "EXz7LFN8hpQ4id8EDYiFoQ==",
"data_dir": "/var/consul",
"log_level": "INFO",
"enable_syslog": true,
"enable_debug": true,
"node_name": "ConsulServer4",
"server": true,
"leave_on_terminate": false,
"skip_leave_on_interrupt": true,
"rejoin_after_leave": true,
"retry_join": [ 
  "192.168.1.11:8301",
  "192.168.1.12:8301",
  "192.168.1.13:8301",
  "192.168.1.14:8301",
  "192.168.1.15:8301"
  ]
 }

server5.example.com

root@server5:~#vim /etc/consul.d/server/config.json
{
"bind_addr": "192.168.1.15",
"datacenter": "dc1",
"data_dir": "/var/consul",
"encrypt": "EXz7LFN8hpQ4id8EDYiFoQ==",
"log_level": "INFO",
"enable_syslog": true,
"enable_debug": true,
"node_name": "ConsulServer5",
"server": true,
"leave_on_terminate": false,
"skip_leave_on_interrupt": true,
"rejoin_after_leave": true,
"retry_join": [ 
  "192.168.1.11:8301",
  "192.168.1.12:8301",
  "192.168.1.13:8301",
  "192.168.1.14:8301",
  "192.168.1.15:8301"
  ]
 }

It is best to add servers one at a time, allowing them to catch up. This avoids the possibility of data loss in case the existing servers fail while bringing the new servers up-to-date.

Start the server4.example.com to add it to the existing cluster.

root@server4:~# consul agent -config-dir /etc/consul.d/server/
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Joining cluster...
    Join completed. Synced with 4 initial agents
==> Consul agent running!
     Node name: 'ConsulServer4'
    Datacenter: 'dc1'
        Server: true (bootstrap: false)
   Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
  Cluster Addr: 192.168.1.14 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
         Atlas: <disabled>
==> Log data will now stream in as it occurs:

[INFO] raft: Node at 192.168.2.14:8300 [Follower] entering Follower state
------------------------------
------------------------------
------------------------------
[INFO] agent: Joining cluster...
[INFO] agent: (LAN) joining: [192.168.1.11:8301 192.168.1.12:8301 192.168.1.13:8301 192.168.1.14:8301 192.168.1.15:8301]
------------------------------
[INFO] agent: Join completed. Synced with 4 initial agents
[INFO] agent: Synced service 'consul'

Now start the server5.example.com.

Removing Servers

It is better to first add the new nodes and then remove the old nodes.

Once you have verified the existing servers are healthy, and that the cluster can handle a node leaving, the actual process is simple. You simply issue a leave command to the server.

In our example there are 5 servers in the cluster which can tolerate 2 server failures of the cluster as per the formula. We will remove the 4th server server4.example.com (192.168.1.14).

server4.example.com

root@server4:~# consul leave
Graceful leave complete

The logs should show something like:

...
[INFO] consul: server starting leave
...
[INFO] consul: removing LAN server ConsulServer4 (Addr: 192.168.1.14:8300) (DC: dc1)
[INFO] raft: Removed ourself, transitioning to follower
[INFO] agent: requesting shutdown
[INFO] consul: shutting down server
[INFO] agent: shutdown complete

The leader logs will show something like:

...
[INFO] consul: removing LAN server ConsulServer4 (Addr: 192.168.1.14:8300) (DC: dc1)
[INFO] raft: Removed peer 192.168.1.14:8300, stopping replication (Index: 3742)
[INFO] consul: removed server 'ConsulServer4' as peer
[INFO] consul: member 'ConsulServer4' left, deregistering
[INFO] raft: aborting pipeline replication to peer 192.168.1.14:8300
...

At this point the node has been gracefully removed from the cluster, and will shut down.

Note that IPs of nodes which have left gracefully (i.e. removed using a command) will be deleted from the alive nodes list - the /var/consul/raft/peers.json file. The contents of the peers.json in our case will be:

root@server1:~# cat /var/consul/raft/peers.json
["192.168.1.12:8300","192.168.1.13:8300","192.168.1.11:8300","192.168.1.15:8300"]

Now we have 4 nodes in the cluster without any failure or downtime.

root@server1:~# consul members

Forced Removal

In some cases, it may not be possible to gracefully remove a server. For example, if the server hardware fails, then there is no way to issue a leave command. Instead, the cluster will detect the failure and replication will continuously retry.

If the server can be recovered, it is best to bring it back online and then make it gracefully leave the cluster.

If this is not a possibility, then the force-leave command can be used to force removal of the server. This can be done by invoking the command with the name of the failed node. At this point, the cluster leader will mark the node as having left the cluster and it will stop attempting to replicate.