Showing posts with label Hadoop. Show all posts

Friday 18 March 2016

Hadoop-2.6.0 Cluster Commissioning and Decommissioning Nodes

To add new nodes to the cluster:

1. Add the network addresses of the new nodes to the include file.

hdfs-site.xml
<property>
<name>dfs.hosts</name>
<value>/<hadoop-home>/conf/include</value>
</property>

yarn-site.xml
<property>
<name>yarn.resourcemanager.nodes.include-path</name>
<value>/<hadoop-home>/conf/include</value>
</property>

Datanodes that are permitted to connect to the NameNode are specified in a
file whose name is specified by the dfs.hosts property.

Includes file resides on the namenode’s local filesystem, and it contains a line for each Datanode, specified by network address (as reported by the Datanode; you can see what this is by looking at the NameNodes web UI).
If you need to specify multiple network addresses for a Datanode, put them on one line, separated by whitespace.
eg :
slave01
slave02
slave03
.....

Similarly, NodeManagers that may connect to the ResourceManager are specified in a file whose name is specified by the yarn.resourcemanager.nodes.include-path property.

In most cases, there is one shared file, referred to as the include file, that both dfs.hosts and yarn.resourcemanager.nodes.include-path refer to, since nodes in the cluster run both DataNode and NodeManager daemons.

2. Update the NameNode with the new set of permitted DataNodes using this

command:

% hdfs dfsadmin –refreshNodes

3. Update the ResourceManager with the new set of permitted NodeManagers using this command:

% yarn rmadmin –refreshNodes

4. Update the slaves file with the new nodes, so that they are included in future

operations performed by the Hadoop control scripts.

5. Start the new DataNodes and NodeManagers.

% hdfs dfsadmin -refreshNodes

% yarn rmadmin –refreshNodes

6. Check that the new DataNodes and NodeManagers appear in the web UI.

To remove nodes from the cluster:

1. Add the network addresses of the nodes to be decommissioned to the exclude file. Do not update the include file at this point.

hdfs-site.xml

<name>dfs.hosts.exclude</name>

<value>/<hadoop-home>/conf/exclude</value>

</property>

yarn-site.xml

<name>yarn.resourcemanager.nodes.exclude-path</name>

<value>/<hadoop-home>/conf/exclude</value>

</property>

The decommissioning process is controlled by an exclude file, which for HDFS is set by the dfs.hosts.exclude property and for YARN by the yarn.resourcemanager.nodes.exclude-path property. It is often the case that these properties refer to the same file. The exclude file lists the nodes that are not permitted to connect to the cluster.

2. Update the NameNode with the new set of permitted DataNodes, using this

command:

% hdfs dfsadmin –refreshNodes

3. Update the ResourceManager with the new set of permitted NodeManagers using this command:

% yarn rmadmin –refreshNodes

4. Go to the web UI and check whether the admin state has changed to “Decommission In Progress” for the DataNodes being decommissioned. They will start copying their blocks to other DataNodes in the cluster.

5. When all the DataNodes report their state as “Decommissioned,” all the blocks have been replicated. Shut down the decommissioned nodes.

6. Remove the nodes from the include file, and run:

% hdfs dfsadmin -refreshNodes

% yarn rmadmin –refreshNodes

Note: Refer Hadoop-1.2.1 Cluster Commissioning and Decommissioning Nodes this link

Monday 4 January 2016

Old and new MapReduce APIs

The new API (which is also known as Context Objects) was primarily designed to make the API easier to evolve in the future and is type incompatible with the old one.

The new API came into the picture from the 1.x release series. However, it was partially supported in this series. So, the old API is recommended for 1.x series:

Feature\Release	1.x	0.23
Old MapReduce API	Yes	Deprecated
New MapReduce API	Partial	Yes
MRv1 runtime (Classic)	Yes	No
MRv2 runtime (YARN)	No	Yes

The old and new API can be compared as follows:

Old API	New API
The old API is in the org.apache.hadoop.mapred package and is still present.	The new API is in the org.apache.hadoop.mapreduce Package.
The old API used interfaces for Mapper and Reducer.	The new API uses Abstract Classes for Mapper and Reducer.
The old API used the JobConf, OutputCollector, and Reporter object to communicate with the MapReduce System.	The new API uses the context object to communicate with the MapReduce system.
In the old API, job control was done through the JobClient.	In the new API, job control is performed through the Job Class.
In the old API, job configuration was done with a JobConf Object	In the new APO, job configuration is done through the Configuration class via some of the helper methods on Job.
In the old API, both the map and reduce outputs are named part-nnnnn .	In the new API, the map outputs are named part-m-nnnnn and the reduce outputs are named part-r-nnnnn .
In the old API, the reduce() method passes values as a java.lang.Iterator .	In the new API, the . method passes values as a java.lang.Iterable .
The old API controls mappers by writing a MapRunnable, but no equivalent exists for reducers.	The new API allows both mappers and reducers to control the execution flow by overriding the run() method.

Monday 2 November 2015

Big Data Project on HDFS browse functionality

Monday 12 October 2015

What is Hadoop Rack Awareness and How to configure it in a cluster

Details about Hadoop Rack Awareness

The Hadoop HDFS and the Map/Reduce components are rack-aware.

The NameNode and the JobTracker obtains the rack id of the slaves in the cluster by invoking an API resolve in an administrator configured module. The API resolves the slave’s DNS name (also IP address) to a rack id. What module to use can be configured using the configuration item topology.node.switch.mapping.impl. The default implementation of the same runs a script/command configured using topology.script.file.name.

If topology.script.file.name is not set, the rack id /default-rack is returned for any passed IP address. The additional configuration in the Map/Reduce part is mapred.cache.task.levels which determines the number of levels (in the network topology) of caches.

So, for example, if it is the default value of 2, two levels of caches will be constructed – one for hosts (host -> task mapping) and another for racks (rack -> task mapping).

What is Rack Awareness in Hadoop

For small clusters in which all servers are connected by a single switch, there are only two levels of locality: “on-machine” and “off-machine.” When loading data from a DataNode’s local drive into HDFS, the NameNode will schedule one copy to go into the local DataNode, and will pick two other machines at random from the cluster.

For larger Hadoop installations which span multiple racks, it is important to ensure that replicas of data exist on multiple racks. This way, the loss of a switch does not render portions of the data unavailable due to all replicas being underneath it.

HDFS can be made rack-aware by the use of a script which allows the master node to map the network topology of the cluster. While alternate configuration strategies can be used, the default implementation allows you to provide an executable script which returns the “rack address” of each of a list of IP addresses.

The network topology script receives as arguments one or more IP addresses of nodes in the cluster. It returns on stdout a list of rack names, one for each input. The input and output order must be consistent.

To set the rack mapping script, specify the key topology.script.file.name in conf/hadoop-site.xml. This provides a command to run to return a rack id; it must be an executable script or program. By default, Hadoop will attempt to send a set of IP addresses to the file as several separate command line arguments. You can control the maximum acceptable number of arguments with the topology.script.number.args key.

Rack ids in Hadoop are hierarchical and look like path names. By default, every node has a rack id of /default-rack. You can set rack ids for nodes to any arbitrary path, e.g., /foo/bar-rack. Path elements further to the left are higher up the tree. Thus a reasonable structure for a large installation may be /top-switch-name/rack-name.

Hadoop rack ids are not currently expressive enough to handle an unusual routing topology such as a 3-d torus; they assume that each node is connected to a single switch which in turn has a single upstream switch. This is not usually a problem, however. Actual packet routing will be directed using the topology discovered by or set in switches and routers. The Hadoop rack ids will be used to find “near” and “far” nodes for replica placement (and in 0.17, MapReduce task placement).

The following example script performs rack identification based on IP addresses given a hierarchical IP addressing scheme enforced by the network administrator. This may work directly for simple installations; more complex network configurations may require a file- or table-based lookup process. Care should be taken in that case to keep the table up-to-date as nodes are physically relocated, etc. This script requires that the maximum number of arguments be set to 1.


#!/bin/bash
# Set rack id based on IP address.
# Assumes network administrator has complete control
# over IP addresses assigned to nodes and they are
# in the 10.x.y.z address space. Assumes that
# IP addresses are distributed hierarchically. e.g.,
# 10.1.y.z is one data center segment and 10.2.y.z is another;
# 10.1.1.z is one rack, 10.1.2.z is another rack in
# the same segment, etc.)
#
# This is invoked with an IP address as its only argument

# get IP address from the input
ipaddr=$0

# select “x.y” and convert it to “x/y”
segments=`echo $ipaddr | cut –delimiter=. –fields=2-3 –output-delimiter=/`
echo /${segments}

Configuring Rack Awareness in Hadoop

We are aware of the fact that hadoop divides the data into multiple file blocks and stores them on different machines. If Rack Awareness is not configured, there may be a possibility that hadoop will place all the copies of the block in same rack which results in loss of data when that rack fails.

Although rare, as rack failure is not as frequent as node failure, this can be avoided by explicitly configuring the Rack Awareness in conf-site.xml.

Rack awareness is configured using the property “topology.script.file.name” in the core-site.xml.

If “topology.script.file.name” is not configured, /default-rack is passed for any ip address i.e., all nodes are placed on same rack.

Configuring Rack awareness in hadoop involves two steps:

configure the “topology.script.file.name” in core-site.xml ,


<property>
<name>topology.node.switch.mapping.impl</name>
<value>org.apache.hadoop.net.ScriptBasedMapping</value>
<description> The default implementation of the DNSToSwitchMapping. It
invokes a script specified in topology.script.file.name to resolve
node names. If the value for topology.script.file.name is not set, the
default value of DEFAULT_RACK is returned for all node names.
</description>
</property>

<property>
<name>topology.script.file.name</name>
<value>core/rack-awareness.sh</value>
</property>

Implement the rack-awareness.sh scripts as desired, Sample rack-awareness scripts can be found here,

1. Topology Script file named as : rack-awareness.sh

A sample Bash shell script:

HADOOP_CONF=$HADOOP_HOME/conf 

while [ $# -gt 0 ] ; do
  nodeArg=$1
  exec< ${HADOOP_CONF}/topology.data 
  result="" 
  while read line ; do
    ar=( $line ) 
    if [ "${ar[0]}" = "$nodeArg" ] ; then
      result="${ar[1]}"
    fi
  done 
  shift 
  if [ -z "$result" ] ; then
    echo -n "/default/rack "
  else
    echo -n "$result "
  fi
done

2. Topology data file named as : topology.data

orienit.node1.com                     /dc1/rack1
orienit.node2.com                     /dc1/rack1

orienit.node3.com                     /dc1/rack1

orienit.node11                           /dc1/rack2

orienit.node12                           /dc1/rack2

orienit.node13                           /dc1/rack2

10.1.1.1                                     /dc1/rack3

10.1.1.2                                     /dc1/rack3

10.1.1.3                                     /dc1/rack3

References,

http://wiki.apache.org/hadoop/topology_rack_awareness_scripts

https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf

Saturday 10 October 2015

Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN.

The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.

The ResourceManager and per-node slave, the NodeManager (NM), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system.

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

The ResourceManager has two main components:

Scheduler
ApplicationsManager

The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees about restarting failed tasks either due to application failure or hardware failures.

The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a resource Container which incorporates elements such as memory, cpu, disk, network etc. In the first version, only memory is supported.

The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in.

The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure.

The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler.

The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.

MRV2 maintains API compatibility with previous stable release (hadoop-1.x). This means that all Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile.

Friday 9 October 2015

How to create a large file using shell script, To test with Hadoop Map Reduce processing

1. Open new Terminal.

2. Create 'myscript.sh' file using below command:

touch myscipt.sh

3. Create 'smallfile.txt' file using below command:

touch smallfile.txt

4. Open the file using below command:

gedit smallfile.txt

5. Enter some sample data into the file.

"
I am going to hyderabad
I am learning hadoop course
"

6. Save the file using command: Ctrl + S

7. Now prepare a large file using below steps.

8. Open the file using below command:

gedit myscript.sh

9. Add the below script into the file.

#!/bin/bash
for i in {1..1000}
do
cat smallfile.txt >> largefile.txt
done

10. Save the file using command: Ctrl + S

11. Provide execute permission to script file using below command:

chmod +x myscript.sh

12. Run the script file to generate a large file using below command:

./myscript.sh

13. Verify the 'largefile.txt' in current folder.

14. Run the above command multiple times to get more large file.

15. Try "hadoop fs -put " commad to place into HDFS.

16. Run Mapreduce programs using above file in step15.