Hadoop Interview Questions part 1

1. What is Hadoop framework?

Ans: Hadoop is an open source framework which is written in java by apache software foundation. 

This framework is used to write software application which requires to process vast amount of data (It 

could handle multi tera bytes of data). It works in-parallel on large clusters which could have 1000 of 

computers (Nodes) on the clusters. It also process data very reliably and fault-tolerant manner. See 

the below image how does it looks.

2. On What concept the Hadoop framework works?

Ans: It works on MapReduce, and it is devised by the Google.

3. What is MapReduce?

Ans: Map reduces is an algorithm or concept to process Huge amount of data in a faster way. As per 

its name you can divide it Map and Reduce.

• The main MapReduce job usually splits the input data-set into independent chunks. (Big data sets in 

the multiple small datasets)

• Reduce Task: And the above output will be the input for the reduce tasks, produces the final result.

Your business logic would be written in the Mapped Task and Reduced Task. Typically both the input 

and the output of the job are stored in a file-system (Not database). The framework takes care of 

scheduling tasks, monitoring them and re-executes the failed tasks.

4. What is compute and Storage nodes?

Ans: Compute Node: This is the computer or machine where your actual business logic will be 

executed.

Storage Node: This is the computer or machine where your file system resides to store the processing 

data. In most of the cases compute node and storage node would be the same machine.

5. How does master slave architecture in the Hadoop?

Ans: the MapReduce framework consists of a single master Job Tracker and multiple slaves, each 

cluster-node will have one Task Tracker.The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring 

them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.

6. How does a Hadoop application look like or their basic components?

Ans: Minimally a Hadoop application would have following components.

• Input location of data

• Output location of processed data.

• A map task.

• A reduced task.

• Job configuration

The Hadoop job client then submits the job (jar/executable etc.) and configuration to the Job Tracker

which then assumes the responsibility of distributing the software/configuration to the slaves, 

scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.

7. Explain how input and output data format of the Hadoop framework?

Ans: The MapReduce framework operates exclusively on pairs, that is, the framework views the 

input to the job as a set of pairs and produces a set of pairs as the output of the job, conceivably of 

different types. See the flow mentioned below (input) -> map -> -> combine/sorting -> -> reduce -> 

(output)

8. What are the restriction to the key and value class?

Ans: The key and value classes have to be serialized by the framework. To make them serializable 

Hadoop provides a Writable interface. As you know from the java itself that the key of the Map 

should be comparable, hence the key has to implement one more interface Writable Comparable.

9. Explain the Word Count implementation via Hadoop framework?

Ans: We will count the words in all the input file flow as below

• Input

Assume there are two files each having a sentence Hello World Hello World (In file 1) Hello World 

Hello World (In file 2)

• Mapper: There would be each mapper for the a file

For the given sample input the first map output:

< Hello, 1>< World, 1>

< Hello, 1>

< World, 1>

The second map output:

< Hello, 1>

< World, 1>

< Hello, 1>

< World, 1>

• Combiner/Sorting (This is done for each individual map)

So output looks like this

The output of the first map:

< Hello, 2>

< World, 2>

The output of the second map:

< Hello, 2>

< World, 2>

• Reducer:

• Output

It sums up the above output and generates the output as below

< Hello, 4>

< World, 4>

Final output would look like

Hello 4 times

World 4 times10. Which interface needs to be implemented to create Mapper and Reducer for the Hadoop?

Ans: org.apache.hadoop.mapreduce.Mapper org.apache.hadoop.mapreduce.Reducer

11. What Mapper does?

Ans: Maps are the individual tasks that transform input records into intermediate records. The 

transformed intermediate records do not need to be of the same type as the input records. A given 

input pair may map to zero or many output pairs.

12. What is the Input Split in map reduce software?

Ans: An Input Split is a logical representation of a unit (A chunk) of input work for a map task; e.g., a 

filename and a byte range within that file to process or a row set in a text file.

13. What is the Input Format?

Ans: The Input Format is responsible for enumerate (itemize) the Input Split, and producing a Record 

Reader which will turn those logical work units into actual physical input records.

14. Where do you specify the Mapper Implementation?

Ans: Generally mapper implementation is specified in the Job itself.

15. How Mapper is instantiated in a running job?

Ans: The Mapper itself is instantiated in the running job, and will be passed a

Map Context object which it can use to configure itself

16. Which are the methods in the Mapper interface?

Ans: the Mapper contains the run () method, which call its own setup () method only once, it also call 

a map () method for each input and finally calls it cleanup () method. All above methods you can 

override in your code.

17. What happens if you don’t override the Mapper methods and keep them as it is?

Ans: If you do not override any methods (leaving even map as-is), it will act as the identity function, 

emitting each input record as a separate output.

18. What is the use of Context object?

Ans: The Context object allows the mapper to interact with the rest of the Hadoop system. It

Includes configuration data for the job, as well as interfaces which allow it to emit output.19. How can you add the arbitrary key-value pairs in your mapper?

Ans: You can set arbitrary (key, value) pairs of configuration data in your Job, e.g. with 

Job.getConfiguration ().set ("myKey", "myVal"), and then retrieve this data in your mapper with

context.getConfiguration ().get ("myKey"). This kind of functionality is typically done in the Mapper's 

setup () method.

20. How does Mapper’s run () method works?

Ans: The Mapper. Run () method then calls map (KeyInType, ValInType, Context) for each key/value 

pair in the Input Split for that task

21. Which object can be used to get the progress of a particular job?

Ans: Context

22. What is next step after Mapper or MapTask?

Ans: The output of the Mapper is sorted and Partitions will be created for the output. Number of 

partition depends on the number of reducer.

23. How can we control particular key should go in a specific reducer?

Ans: Users can control which keys (and hence records) go to which Reducer by implementing a custom 

Partitioner.

24. What is the use of Combiner?

Ans: It is an optional component or class, and can be specify via Job.setCombinerClass (Class Name), 

to perform local aggregation of the intermediate outputs, which helps to cut down the amount of 

data transferred from the Mapper to the Reducer.

25. How many maps are there in a particular Job?

Ans: the number of maps is usually driven by the total size of the inputs, that is, the total number of 

blocks of the input files.

Generally it is around 10-100 maps per-node. Task setup takes awhile, so it is best if the maps take at 

least a minute to execute.

Suppose, if you expect 10TB of input data and have a block size of 128MB, you'll end up with 82,000

maps, to control the number of block you can use the mapreduce.job.maps parameter (which only 

provides a hint to the framework). Ultimately, the number of tasks is controlled by the number of 

splits returned by the InputFormat.getSplits () method (which you can override).

26. What is the Reducer used for?Ans: Reducer reduces a set of intermediate values which share a key to a (usually smaller) set of 

values. The number of reduces for the job is set by theuser via Job.setNumReduceTasks (int).

27. Explain the core methods of the Reducer?

Ans: The API of Reducer is very similar to that of Mapper, there's a run() method that receives a 

Context containing the job's configuration as well as interfacing methods that return data from the 

reducer itself back to the framework. The run() method calls setup() once, reduce() once for each 

key associated with the reduce task, and cleanup() once at the end. Each of these methods can 

access the job's configuration data by using Context.getConfiguration ().

As in Mapper, any or all of these methods can be overridden with custom implementations. If none of 

these methods are overridden, the default reducer operation is the identity function; values are 

passed through without further processing.

The heart of Reducer is it’s reduce () method. This is called once per key; the second argument is an 

Iterable which returns all the values associated with that key.

28. What are the primary phases of the Reducer?

Ans: Shuffle, Sort and Reduce

29. Explain the shuffle?

Ans: Input to the Reducer is the sorted output of the mappers. In this phase the framework fetches 

the relevant partition of the output of all the mappers, via HTTP.

30. Explain the Reducer’s Sort phase?

Ans: The framework groups Reducer inputs by keys (since different mappers may have output the 

same key) in this stage. The shuffle and sort phases occur simultaneously; while map-outputs are 

being fetched they are merged (It is similar to merge-sort).