Commission/decommission a node

Cloudera Manager makes it very simple to add and remove hosts in a cluster. All host management operations in Cloudera Manager are done from the Hosts screen. In this post, we will go through the steps of Commissioning and decommissioning a host in the CDH cluster.

There will always be failures in clusters, such as hardware issues or a need to upgrade nodes. This should be done in a graceful manner, without any data loss. Cloudera Manager helps us in this case, to gracefully decommission a node and also to commision it back when the maintenance work is completed.

To view the current state of all the hosts in the cluster, you can navigate to the hosts page as shown below.

Decommision a host using Cloudera Manager

The decommissioning of the host should ideally be done when the host is in maintenance mode. Putting host in maintenance mode is optional but is a recommended practice. Follow the steps outlined below to put the host in maintenance mode first and then decommission it.

1. Putting host in maintenance mode

1. Navigate to the Hosts screen from the Cloudera Manager toolbar.

2. Select the host to be decommissioned (node04) from the list of hosts and select the option “Enter Maintenace Mode” from the drop-down “Actions for selected“.

3. The alerts on the selected host and the roles on those hosts will be suppressed while they are in maintenance mode.

4. You can verify the host status from the “Maintenace Mode” section in the left sidebar.

2. Decommissioning host

1. Navigate to the Hosts screen from the Cloudera Manager toolbar.

2. Select the host to be decommissioned (node04) from the list of hosts and click the option “Hosts Decommission” from the drop-down “Actions for selected“.

3. The Decommissioning wizard will show the “Role Instances” that are available on the host to be decommissioned. As you can see, there is a role associated with the host i.e. DataNode.

4. The Cloudera Manager will go ahead and decommission the selected host and display the status.

5. You can verify the status of decommissioning from the “Commission State” section in the left sidebar.

Commissioning/Recommissioning a host using Cloudera Manager

Once you are confirmed of the host health and the host is ready to be commissioned back again, we can add the host back to the cluster to resume its job. Follow the steps outlined below to recommission a host.

2. Recommissioning the host

1. Navigate to the Hosts screen from the Cloudera Manager toolbar.

2. Select the host to be decommissioned (node04) from the list of hosts and click the option “Hosts Recommission” from the drop-down “Actions for selected“.

3. The Recommissioning wizard will show the “Role Instances” that are available on the host to be recommissioned. As you can see, there is a role associated with the host i.e. DataNode.

4. The Cloudera Manager will go ahead and recommission the selected host and display the status.

5. You may need to start any service which was previously running on the decommissioned host. In my case, I had a 2nd zookeeper service which was running on this node earlier. I will start it on the home page in Cloudera Manager.

1. Exiting the maintenance mode

1. Navigate to the Hosts screen from the Cloudera Manager toolbar.

2. Select the host to be recommissioned (node04) from the list of hosts and select the option “Exit Maintenace Mode” from the drop-down “Actions for selected“.

3. The wizard will list the roles which will exit the maintenance mode on the selected host.

4. Verify that no service is in maintenance mode or in stopped/faulted state on the home page of Cloudera Manager.