Hadoop Framework:
Hadoop Definition
Hadoop is an open-source framework that allows to
store and process big data in a distributed environment across clusters of
computers using simple programming models. It is designed to scale up from
single servers to thousands of machines, each offering local computation and
storage.
Hadoop
is a distributed file system and data processing engine that is designed to
handle extremely high volumes of data in any structure.
Hadoop has two components:
Technically, Hadoop
consists of two key elements.
• The
first is the Hadoop Distributed File System (HDFS), which
permits the high-bandwidth, cluster-based storage essential for Big Data
computing.
• The
second part of Hadoop is then a data processing framework Called MapReduce.
Based on Google's search technology, this distributes or "maps"
large data sets across multiple servers.
• Each
server are then aggregated in the so-termed "Reduce" stage.
• The
focus is on supporting redundancy, distributed architectures, and parallel
processing.
Hadoop Architecture:
Hadoop framework includes following four
modules:
· Hadoop
Common: These are Java libraries and utilities required
by other Hadoop modules. These libraries provides file system and OS level
abstractions and contains the necessary Java files and scripts required to
start Hadoop.
· Hadoop
YARN: This is a framework for job scheduling and
cluster resource management.
· Hadoop
Distributed File System (HDFS™): A distributed file system that provides high-throughput
access to application data.
· Hadoop
MapReduce: This is YARN-based system for parallel processing of large data
sets.
We can use following diagram to explain four components available
in Hadoop framework.
MapReduce:
Hadoop MapReduce is
a software framework for easily writing applications which process big amounts
of data in-parallel on large clusters or thousands of nodes of commodity
hardware in a reliable manner.
Two different tasks that
Hadoop programs perform:
· The Map Task: It takes input data and converts it into a set
of data, the individual elements are divided into tuples (key/value pairs).
· The Reduce Task: This task takes the output from a map task as
input and combines those data tuples into a smaller set of tuples. The reduce
task is always performed after the map task.
Typically both input and
output are stored in a file-system. The framework takes care of scheduling
tasks, monitoring them and re-executes the failed tasks.
JobTracker & TaskTracker
·
The MapReduce framework works like master and slave.
· It consists of a single
master JobTracker and one slave TaskTracker
per cluster-node.
· The master is responsible for resource management, tracking
resource consumption/availability and scheduling the jobs component tasks on
the slaves, monitoring them and re-executing the failed tasks.
·
The slaves TaskTracker execute the tasks as directed by the master
and provide task-status information to the master periodically.
· The JobTracker is a single point of failure for the Hadoop
MapReduce service which means if JobTracker goes down, all running jobs are
halted.
Hadoop Distributed
File System (HDFS)
The Hadoop Distributed File System is based on the Google File System (GFS) and
provides a distributed file system that is designed to run on large clusters
(thousands of computers) of small
computer machines in a reliable, fault-tolerant manner.
HDFS uses a master/slave architecture where
master consists of a single NameNode that
manages the file system metadata and one or more slave DataNodes that store the actual data.
A file in an HDFS namespace is split into several
blocks and those blocks are stored in a set of DataNodes. The NameNode determines the mapping
of blocks to the DataNodes.
The DataNodes takes care of read and write
operation with the file system. They also take care of block creation, deletion
and replication based on instruction given by NameNode.
HDFS Architecture
Given below is the architecture of a Hadoop File
System.
HDFS follows the master-slave architecture and it
has the following elements.
Namenode:
The namenode is the commodity hardware that
contains the GNU/Linux operating system and the namenode software. It is a
software that can be run on commodity hardware.
The system having the namenode acts as the master
server and it does the following tasks:
- Manages the file system namespace.
- Regulates client’s access to files.
- It also executes file system
operations such as renaming, closing, and opening files and directories.
Datanode:
The datanode is a commodity hardware having the
GNU/Linux operating system and datanode software. For every node (Commodity
hardware/System) in a cluster, there will be a datanode. These nodes manage the
data storage of their system.
- Datanodes perform read-write
operations on the file systems, as per client request.
- They also perform operations such
as block creation, deletion, and replication according to the instructions
of the namenode.
Block:
The user data is stored in the files of HDFS. The
file in a file system will be divided into one or more segments and/or stored
in individual data nodes. These file segments are called as blocks.
In other words, the minimum amount of data that
HDFS can read or write is called a Block. The default block size is 64MB, but
it can be increased as per the need to change in HDFS configuration.
Features
of HDFS
- It
is suitable for the distributed storage and processing.
- Hadoop
provides a command interface to interact with HDFS.
- The
built-in servers of namenode and datanode help users to easily check the
status of cluster.
- Streaming
access to file system data.
- HDFS
provides file permissions and authentication.
Goals of HDFS:
· Fault detection and recovery : Since HDFS includes a large number of commodity
hardware, failure of components is frequent. Therefore HDFS should have
mechanisms for quick and automatic fault detection and recovery.
· Huge
datasets : HDFS should have hundreds of nodes per cluster
to manage the applications having huge datasets.
· Hardware
at data : A requested task can be done efficiently, when
the computation takes place near the data. Especially where huge datasets are
involved, it reduces the network traffic and increases the throughput.
The Algorithm
·
Generally MapReduce
paradigm is based on sending the computer to where the data resides!
·
MapReduce program
executes in three stages, namely map stage, shuffle stage, and reduce stage.
o Map stage : The map or mapper’s job
is to process the input data. Generally the input data is in the form of file
or directory and is stored in the Hadoop file system (HDFS). The input file is
passed to the mapper function line by line. The mapper processes the data and
creates several small chunks of data.
o Reduce stage : This stage is the
combination of the Shuffle stage and the Reduce stage.
The Reducer’s job is to process the data that comes from the mapper. After
processing, it produces a new set of output, which will be stored in the HDFS.
· During a MapReduce job,
Hadoop sends the Map and Reduce tasks to the appropriate servers in the
cluster.
· The framework manages
all the details of data-passing such as issuing tasks, verifying task
completion, and copying data around the cluster between the nodes.
· Most of the computing
takes place on nodes with data on local disks that reduces the network traffic.
After completion of the
given tasks, the cluster collects and reduces the data to form an appropriate
result, and sends it back to the Hadoop server.
Inputs and Outputs
The MapReduce framework operates on <key,
value> pairs, that is, the framework views the input to the job as a set of
<key, value> pairs and produces a set of <key, value> pairs as the
output of the job, conceivably of different types.
The key and the value classes should be in
serialized manner by the framework and hence, need to implement the Writable
interface. Additionally, the key classes have to implement the Writable-Comparable
interface to facilitate sorting by the framework. Input and Output types of a
MapReduce job: (Input) <k1, v1> -> map -> <k2, v2>->
reduce -> <k3, v3>(Output).
|
Input
|
Output
|
|
Map
|
<k1, v1>
|
list (<k2, v2>)
|
|
Reduce
|
<k2, list(v2)>
|
list (<k3, v3>)
|
Terminology:
· PayLoad - Applications implement the Map and the Reduce functions, and form
the core of the job.
· Mapper - Mapper maps the input key/value pairs to a set of intermediate
key/value pair.
· NamedNode - Node that manages the Hadoop Distributed File System (HDFS).
· DataNode - Node where data is presented in advance before any processing takes place.
· MasterNode - Node where JobTracker runs and which accepts job requests from clients.
· SlaveNode - Node where Map and Reduce program runs.
· JobTracker - Schedules jobs and tracks the assign jobs to Task tracker.
· Task
Tracker - Tracks the task and reports status to
JobTracker.
· Job - A program is an execution of a Mapper and Reducer across a
dataset.
· Task - An execution of a Mapper or a Reducer on a slice of data.
· Task
Attempt - A particular instance of an attempt to execute a
task on a SlaveNode.
