ORIENIT HADOOP-2.6.0 PRACTICAL CLUSTER SETUP SCREENSHOTS

Hadoop-2.6.0 Cluster Commissioning and Decommissioning Nodes

To add new nodes to the cluster:

1. Add the network addresses of the new nodes to the include file.

hdfs-site.xml
<property>
<name>dfs.hosts</name>
<value>/<hadoop-home>/conf/include</value>
</property>

yarn-site.xml
<property>
<name>yarn.resourcemanager.nodes.include-path</name>
<value>/<hadoop-home>/conf/include</value>
</property>

Datanodes that are permitted to connect to the NameNode are specified in a
file whose name is specified by the dfs.hosts property.

Includes file resides on the namenode’s local filesystem, and it contains a line for each Datanode, specified by network address (as reported by the Datanode; you can see what this is by looking at the NameNodes web UI).
If you need to specify multiple network addresses for a Datanode, put them on one line,  separated by whitespace. 
eg :
slave01
slave02
slave03
.....

Similarly, NodeManagers that may connect to the ResourceManager are specified in a file whose name is specified by the yarn.resourcemanager.nodes.include-path property. 

In most cases, there is one shared file, referred to as the include file, that both dfs.hosts and  yarn.resourcemanager.nodes.include-path refer to, since nodes in the cluster run both DataNode and NodeManager daemons.

2. Update the NameNode with the new set of permitted DataNodes using this

command:

% hdfs dfsadmin –refreshNodes

3. Update the ResourceManager with the new set of permitted NodeManagers using this command:

% yarn rmadmin –refreshNodes

4. Update the slaves file with the new nodes, so that they are included in future

operations performed by the Hadoop control scripts.

5. Start the new DataNodes and NodeManagers.

% hdfs dfsadmin -refreshNodes

% yarn rmadmin –refreshNodes

6. Check that the new DataNodes and NodeManagers appear in the web UI.

To remove nodes from the cluster:

1. Add the network addresses of the nodes to be decommissioned to the exclude file. Do not update the include file at this point.

hdfs-site.xml

<property>

<name>dfs.hosts.exclude</name>

<value>/<hadoop-home>/conf/exclude</value>

</property>

yarn-site.xml

<property>

<name>yarn.resourcemanager.nodes.exclude-path</name>

<value>/<hadoop-home>/conf/exclude</value>

</property>

The decommissioning process is controlled by an exclude file, which for HDFS is set by the dfs.hosts.exclude property and for YARN by the yarn.resourcemanager.nodes.exclude-path property. It is often the case that these properties refer to the same file. The exclude file lists the nodes that are not permitted to connect to the cluster.

2. Update the NameNode with the new set of permitted DataNodes, using this

command:

% hdfs dfsadmin –refreshNodes

3. Update the ResourceManager with the new set of permitted NodeManagers using this command:

% yarn rmadmin –refreshNodes

4. Go to the web UI and check whether the admin state has changed to “Decommission In Progress” for the DataNodes being decommissioned. They will start copying their blocks to other DataNodes in the cluster.

5. When all the DataNodes report their state as “Decommissioned,” all the blocks have been replicated. Shut down the decommissioned nodes.

6. Remove the nodes from the include file, and run:

% hdfs dfsadmin -refreshNodes

% yarn rmadmin –refreshNodes

Note: Refer Hadoop-1.2.1 Cluster Commissioning and Decommissioning Nodes this link