Hadoop Interview Questions part 1

1. What is Hadoop framework?

Ans: Hadoop is an open source framework which is written in java by apache software foundation. 

This framework is used to write software application which requires to process vast amount of data (It 

could handle multi tera bytes of data). It works in-parallel on large clusters which could have 1000 of 

computers (Nodes) on the clusters. It also process data very reliably and fault-tolerant manner. See 

the below image how does it looks.

2. On What concept the Hadoop framework works?

Ans: It works on MapReduce, and it is devised by the Google.

3. What is MapReduce?

Ans: Map reduces is an algorithm or concept to process Huge amount of data in a faster way. As per 

its name you can divide it Map and Reduce.

• The main MapReduce job usually splits the input data-set into independent chunks. (Big data sets in 

the multiple small datasets)

• Reduce Task: And the above output will be the input for the reduce tasks, produces the final result.

Your business logic would be written in the Mapped Task and Reduced Task. Typically both the input 

and the output of the job are stored in a file-system (Not database). The framework takes care of 

scheduling tasks, monitoring them and re-executes the failed tasks.

4. What is compute and Storage nodes?

Ans: Compute Node: This is the computer or machine where your actual business logic will be 

executed.

Storage Node: This is the computer or machine where your file system resides to store the processing 

data. In most of the cases compute node and storage node would be the same machine.

5. How does master slave architecture in the Hadoop?

Ans: the MapReduce framework consists of a single master Job Tracker and multiple slaves, each 

cluster-node will have one Task Tracker.The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring 

them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.

6. How does a Hadoop application look like or their basic components?

Ans: Minimally a Hadoop application would have following components.

• Input location of data

• Output location of processed data.

• A map task.

• A reduced task.

• Job configuration

The Hadoop job client then submits the job (jar/executable etc.) and configuration to the Job Tracker

which then assumes the responsibility of distributing the software/configuration to the slaves, 

scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.

7. Explain how input and output data format of the Hadoop framework?

Ans: The MapReduce framework operates exclusively on pairs, that is, the framework views the 

input to the job as a set of pairs and produces a set of pairs as the output of the job, conceivably of 

different types. See the flow mentioned below (input) -> map -> -> combine/sorting -> -> reduce -> 

(output)

8. What are the restriction to the key and value class?

Ans: The key and value classes have to be serialized by the framework. To make them serializable 

Hadoop provides a Writable interface. As you know from the java itself that the key of the Map 

should be comparable, hence the key has to implement one more interface Writable Comparable.

9. Explain the Word Count implementation via Hadoop framework?

Ans: We will count the words in all the input file flow as below

• Input

Assume there are two files each having a sentence Hello World Hello World (In file 1) Hello World 

Hello World (In file 2)

• Mapper: There would be each mapper for the a file

For the given sample input the first map output:

< Hello, 1>< World, 1>

< Hello, 1>

< World, 1>

The second map output:

< Hello, 1>

< World, 1>

< Hello, 1>

< World, 1>

• Combiner/Sorting (This is done for each individual map)

So output looks like this

The output of the first map:

< Hello, 2>

< World, 2>

The output of the second map:

< Hello, 2>

< World, 2>

• Reducer:

• Output

It sums up the above output and generates the output as below

< Hello, 4>

< World, 4>

Final output would look like

Hello 4 times

World 4 times10. Which interface needs to be implemented to create Mapper and Reducer for the Hadoop?

Ans: org.apache.hadoop.mapreduce.Mapper org.apache.hadoop.mapreduce.Reducer

11. What Mapper does?

Ans: Maps are the individual tasks that transform input records into intermediate records. The 

transformed intermediate records do not need to be of the same type as the input records. A given 

input pair may map to zero or many output pairs.

12. What is the Input Split in map reduce software?

Ans: An Input Split is a logical representation of a unit (A chunk) of input work for a map task; e.g., a 

filename and a byte range within that file to process or a row set in a text file.

13. What is the Input Format?

Ans: The Input Format is responsible for enumerate (itemize) the Input Split, and producing a Record 

Reader which will turn those logical work units into actual physical input records.

14. Where do you specify the Mapper Implementation?

Ans: Generally mapper implementation is specified in the Job itself.

15. How Mapper is instantiated in a running job?

Ans: The Mapper itself is instantiated in the running job, and will be passed a

Map Context object which it can use to configure itself

16. Which are the methods in the Mapper interface?

Ans: the Mapper contains the run () method, which call its own setup () method only once, it also call 

a map () method for each input and finally calls it cleanup () method. All above methods you can 

override in your code.

17. What happens if you don’t override the Mapper methods and keep them as it is?

Ans: If you do not override any methods (leaving even map as-is), it will act as the identity function, 

emitting each input record as a separate output.

18. What is the use of Context object?

Ans: The Context object allows the mapper to interact with the rest of the Hadoop system. It

Includes configuration data for the job, as well as interfaces which allow it to emit output.19. How can you add the arbitrary key-value pairs in your mapper?

Ans: You can set arbitrary (key, value) pairs of configuration data in your Job, e.g. with 

Job.getConfiguration ().set ("myKey", "myVal"), and then retrieve this data in your mapper with

context.getConfiguration ().get ("myKey"). This kind of functionality is typically done in the Mapper's 

setup () method.

20. How does Mapper’s run () method works?

Ans: The Mapper. Run () method then calls map (KeyInType, ValInType, Context) for each key/value 

pair in the Input Split for that task

21. Which object can be used to get the progress of a particular job?

Ans: Context

22. What is next step after Mapper or MapTask?

Ans: The output of the Mapper is sorted and Partitions will be created for the output. Number of 

partition depends on the number of reducer.

23. How can we control particular key should go in a specific reducer?

Ans: Users can control which keys (and hence records) go to which Reducer by implementing a custom 

Partitioner.

24. What is the use of Combiner?

Ans: It is an optional component or class, and can be specify via Job.setCombinerClass (Class Name), 

to perform local aggregation of the intermediate outputs, which helps to cut down the amount of 

data transferred from the Mapper to the Reducer.

25. How many maps are there in a particular Job?

Ans: the number of maps is usually driven by the total size of the inputs, that is, the total number of 

blocks of the input files.

Generally it is around 10-100 maps per-node. Task setup takes awhile, so it is best if the maps take at 

least a minute to execute.

Suppose, if you expect 10TB of input data and have a block size of 128MB, you'll end up with 82,000

maps, to control the number of block you can use the mapreduce.job.maps parameter (which only 

provides a hint to the framework). Ultimately, the number of tasks is controlled by the number of 

splits returned by the InputFormat.getSplits () method (which you can override).

26. What is the Reducer used for?Ans: Reducer reduces a set of intermediate values which share a key to a (usually smaller) set of 

values. The number of reduces for the job is set by theuser via Job.setNumReduceTasks (int).

27. Explain the core methods of the Reducer?

Ans: The API of Reducer is very similar to that of Mapper, there's a run() method that receives a 

Context containing the job's configuration as well as interfacing methods that return data from the 

reducer itself back to the framework. The run() method calls setup() once, reduce() once for each 

key associated with the reduce task, and cleanup() once at the end. Each of these methods can 

access the job's configuration data by using Context.getConfiguration ().

As in Mapper, any or all of these methods can be overridden with custom implementations. If none of 

these methods are overridden, the default reducer operation is the identity function; values are 

passed through without further processing.

The heart of Reducer is it’s reduce () method. This is called once per key; the second argument is an 

Iterable which returns all the values associated with that key.

28. What are the primary phases of the Reducer?

Ans: Shuffle, Sort and Reduce

29. Explain the shuffle?

Ans: Input to the Reducer is the sorted output of the mappers. In this phase the framework fetches 

the relevant partition of the output of all the mappers, via HTTP.

30. Explain the Reducer’s Sort phase?

Ans: The framework groups Reducer inputs by keys (since different mappers may have output the 

same key) in this stage. The shuffle and sort phases occur simultaneously; while map-outputs are 

being fetched they are merged (It is similar to merge-sort).

Searching Data with Apache Solr

Overview

In this tutorial we will walk through how to use Apache Solr with Hadoop to index and search data stored on HDFS. It’s not meant as a general introduction to Solr.

After working through this tutorial you will have Solr running on your Hortonworks Sandbox. You will also have a solrconfig and a schema which you can easily adapt to your own use cases. Also you will learn how to use Hadoop MapReduce to index files.  

Prerequisites

1. Hortonworks HDP Sandbox 2.1

2. Apache Solr 4.7.2

3. Lucidworks Job Jar

Remarks: I was using VMware’s Fusion to run Sandbox. If you choose Virtualbox things should look the same beside the fact your VM will not have it’s own IP address but rather Solr listening on 127.0.0.1. For convenience I added sandbox as a host to my /etc/hosts file on my Mac. Apache Solr 4.7.2 is the officially by Hortonworks supported version as I’m writing this (May 2014). 

Steps

Let’s get it started: Power-up the sandbox with at least 4GB main memory.

    ssh root@sandbox (Passwort: hadoop)
    ./start_ambari.sh

Open your browser and verify that all services are running. We will only need HDFS and MapReduce but “all lights green” is always good  

We start by creating a solr user and a folder where we are going to install the binaries:

    adduser solr
    passwd solr

    mkdir /opt/solr
    chown solr /opt/solr

Now copy the binaries you downloaded from the list of ingredients above from your host to the Sandbox from your Mac / Windows host:

    cd ~/Downloads
    scp solr-4.7.2.tar lucidworks-hadoop-1.2.0-0-0.tar solr@sandbox:/opt/solr

Next step is creating dummy data we will later on index in Solr and make searchable. As mentioned above this is “Hello World!” so better do not expect big data. The file we are going to index will be four line csv file. Type the following on your Sandbox command prompt:

    echo id,text >/tmp/mydata.csv; echo 1,Hello>>/tmp/mydata.csv; echo 2,HDP >>/tmp/mydata.csv; echo 3,and >>/tmp/mydata.csv; echo 4,Solr >>/tmp/mydata.csv

Then we need to prepare HDFS:

    su - hdfs
    hadoop fs -mkdir -p /user/solr/data/csv 
    hadoop fs -chown solr /user/solr
    hadoop fs -put /tmp/mydata.csv /user/solr/data/csv

Now it’s getting more interesting as we are about to install Solr:

    su - solr
    cd /opt/solr

    tar xzvf solr-4.7.2.tar 
    // untar is all we need to install Solr! We still need to integrate it into HDP though. 
    tar xvf lucidworks-hadoop-1.2.0-0-0.tar
    ln -s solr-4.7.2 solr
    ln -s lucidworks-hadoop-1.2.0-0-0 jobjar

Solr comes with a nice example which we will use as a starting point:

    cd solr
    cp -r example hdp 
    // Remove unnecessary files:
    rm -fr hdp/examle* hdp/multicore
    // Our core (basically the index) will be called hdp1 instead of collection1
    mv hdp/solr/collection1 hdp/solr/hdp1
    // Remove the existing core
    rm hdp/solr/hdp1/core.properties

Now comes the most difficult part: Making Solr storing its data on HDFS and creating a schema for our “Hello World” csv file. We need to modify two files solrconfig.xml and schema.xml

    vi hdp/solr/hdp1/conf/solrconfig.xml

Search for the tag

    <directoryFactory
    ..
    </directoryFactory>

And completely replace it with (Make sure you copy the full line. The lines may appeart truncated in the browser but when you copy/paste the full lines you’re good):

    <directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
      <str name="solr.hdfs.home">hdfs://sandbox:8020/user/solr</str>
      <bool name="solr.hdfs.blockcache.enabled">true</bool>
      <int name="solr.hdfs.blockcache.slab.count">1</int>
      <bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
      <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
      <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
      <bool name="solr.hdfs.blockcache.write.enabled">true</bool>
      <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
      <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
      <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
    </directoryFactory>

Now still in solrconfig.xml look for lockType. Change it to hdfs:

    <lockType>hdfs</lockType>

Save the file and open schema.xml

    vi hdp/solr/hdp1/conf/schema.xml

In the

<fields>

tag keep only the fields with the following names:

_version_
_root_

Leave the dynamic fields unchanged (they could be useful for your own use-cases but we will not need them in this example though). 
Add the following fields:

    <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
    <field name="text" multiValued="true" stored="true"  type="text_en" indexed="true"/>
    <field name="data_source" stored="false" type="text_en" indexed="true"/> 

The data_source field is required by the map reduce based indexing we will use later. The fields named id and name are matching the two columns in our csv file. 
Next remove all copyField tags and add:
Lets Add the id to text so we can search both

<copyField dest="text" source="id"/>

Now we need to create our core/index. Start solr and point your browser to it (http://sandbox:8983/solr):

    cd hdp
    java -jar start.jar

Click on “Core Admin” and fill in the fields as below:

If everything goes as expected you should see the following:

If something is broken (xml file non parseable, wrong folder…) you can easily start from fresh by:

    // stop or kill solr
    rm /opt/solr/solr/hdp/solr/hdp1/core.properties
    hadoop fs -rm -r /user/solr/hdp1
    // start solr again

Now choose the just created core “hdp1” from the dropdown box on the left:

Click on Query and press the blue “Execute Query” button. You will see that we still have 0 documents in our index which is no surprise as we have not indexed anything: 

So now we are going to index our big csv file 

    hadoop jar jobjar/hadoop/hadoop-lws-job-1.2.0-0-0.jar com.lucidworks.hadoop.ingest.IngestJob -Dlww.commit.on.close=true -DcsvFieldMapping=0=id,1=text -cls com.lucidworks.hadoop.ingest.CSVIngestMapper -c hdp1 -i /user/solr/data/csv/mydata.csv -of com.lucidworks.hadoop.io.LWMapRedOutputFormat -s http://localhost:8983/solr

If everything went well your output should look like:

    14/05/24 06:46:00 INFO mapreduce.Job: Job job_1400841048847_0036 completed successfully
    14/05/24 06:46:00 INFO mapreduce.Job: Counters: 32
        File System Counters
            FILE: Number of bytes read=0
            FILE: Number of bytes written=201410
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
            HDFS: Number of bytes read=287
            HDFS: Number of bytes written=0
            HDFS: Number of read operations=4
            HDFS: Number of large read operations=0
            HDFS: Number of write operations=0
        Job Counters 
            Launched map tasks=2
            Data-local map tasks=2
            Total time spent by all maps in occupied slots (ms)=16727
            Total time spent by all reduces in occupied slots (ms)=0
            Total time spent by all map tasks (ms)=16727
            Total vcore-seconds taken by all map tasks=16727
            Total megabyte-seconds taken by all map tasks=4181750
        Map-Reduce Framework
            Map input records=5
            Map output records=4
            Input split bytes=234
            Spilled Records=0
            Failed Shuffles=0
            Merged Map outputs=0
            GC time elapsed (ms)=146
            CPU time spent (ms)=3300
            Physical memory (bytes) snapshot=295854080
            Virtual memory (bytes) snapshot=1794576384
            Total committed heap usage (bytes)=269484032
        com.lucidworks.hadoop.ingest.BaseHadoopIngest$Counters
            DOCS_ADDED=4
            DOCS_CONVERT_FAILED=1
        File Input Format Counters 
            Bytes Read=53
        File Output Format Counters 
            Bytes Written=0

Go back to your browser and enter “HDP” in the field called “q” and press “Execute Query”: 

Congratulations!!! 

You installed and integrated Solr on HDP. Indexed a csv file through map reduce and successfully executed a Solr query against the index! 

Next steps are now installing Solr in SolrCloud mode on an HDP cluster, index real files and create a nice web app so that business users can easily search for information stored on Hadoop. 
I hope this was useful and you had fun!

How to Visualize Website Clickstream Data

Step 1: View and Refine the Website Clickstream Data in Hortonworks

In the “Loading Data into the Hortonworks Sandbox” tutorial, we loaded website data files into Hortonworks, and then used Hive queries to refine the data. Let’s take a closer look at that data.

Here’s a summary of the data we’re working with:

• Omniture logs* – website log files containing information such as URL, timestamp, IP address, geocoded IP address, and user ID (SWID).
• users* – CRM user data listing SWIDs (Software User IDs) along with date of birth and gender.
• products* – CMS data that maps product categories to website URLs.

Let’s start by looking at the raw Omniture website data. In the Hortonworks Sandbox, click the File Browser icon in the toolbar at the top of the page, then select the Omniture.0.tsv.gz file.

• The raw data file appears in the File Browser, and you can see that it contains information such as URL, timestamp, IP address, geocoded IP address, and user ID (SWID).
  The Omniture log dataset contains about 4 million rows of data, which represents five days of clickstream data. Often, organizations will process weeks, months, or even years of data.
  
• Now let’s look at the users table using HCatalog. Click the HCat icon in the toolbar at the top of the page, then click Browse Data in the users row.
  
• The users table appears, and you can see the SWID, birth date, and gender columns.
  
• You can also use HCatalog to view the data in the products table, which maps product categories to website URLs.
  
• In the “Loading Data into the Hortonworks Sandbox” tutorial, we used Apache Hive to join the three data sets into one master set. This is easily accomplished in Hadoop, even when working with millions or billions of rows of data.
  First, we used a Hive script to generate an “omniture” view that contained a subset of the data in the Omniture log table.
  
• To view the omniture data in Hive, click Table, then click Browse Datain the omniture row.
  
• The query results will appear, and you can see that the results include the data from the Omniture log table that were specified in the query.
  
• Next, we created a “webloganalytics” script to join the omniture website log data to the CRM data (registered users) and CMS data (products). This Hive query executed a join to create a unified dataset across our data sources.
  
• You can view the data generated by the webloganalytics script in Hive as described in the preceding steps.
  

Now that we have reviewed the refined website data, we can access the data with Excel.

---

Step 2: Access the Website Clickstream Data with Excel

In this section, we will use Excel Professional Plus 2013 to access the refined clickstream data.

• In Windows, open a new Excel workbook, then select Data > From Other Sources > From Microsoft Query.
  
• On the Choose Data Source pop-up, select the Hortonworks ODBC data source you installed previously, then click OK. The Hortonworks ODBC driver enables you to access Hortonworks data with Excel and other Business Intelligence (BI) applications that support ODBC.
  
• After the connection to the sandbox is established, the Query Wizard appears. Select the webloganalytics table in the Available tables and columns box, then click the right arrow button to add the entire webloganalytics table to the query. Click Next to continue.
  
• On the Filter Data screen, click Next to continue without filtering the data.
  
• On the Sort Order screen, click Next to continue without setting a sort order.
  
• Click Finish on the Query Wizard Finish screen to retrieve the query data from the sandbox and import it into Excel.
  
• On the Import Data dialog box, click OK to accept the default settings and import the data as a table.
  
• The imported query data appears in the Excel workbook.
  

Now that we have successfully imported Hortonworks Sandbox data into Microsoft Excel, we can use the Excel Power View feature to analyze and visualize the data.

---

Step 3: Visualize the Website Clickstream Data Using Excel Power View

Data visualization can help you optimize your website and convert more visits into sales and revenue. In this section we will:

• Analyze the clickstream data by location
• Filter the data by product category
• Graph the website user data by age and gender
• Pick a target customer segment
• Identify a few web pages with the highest bounce rates

In the Excel workbook with the imported webloganalytics data, select Insert > Power View to open a new Power View report.

• The Power View Fields area appears on the right side of the window, with the data table displayed on the left. Drag the handles or click the Pop Out icon to maximize the size of the data table.
  
• Let’s start by taking a look at the countries of origin of our website visitors. In the Power View Fields area, leave the country checkbox selected, and clear all of the other checkboxes. The data table will update to reflect the selections.
  
• On the Design tab in the top menu, click Map.
  
• The map view displays a global view of the data. Now let’s take a look at a count of IP address by state. First, drag the ip field into the SIZE box.
  
• Drag country from the Power View Fields area into the Filters area, then select the usa checkbox.
  
• Next, drag state into the LOCATIONS box. Remove the country field from the LOCATIONS box by clicking the down-arrow and then Remove Field.
  
• Use the map controls to zoom in on the United States. Move the pointer over each state to display the IP count for that state.
  
• Our dataset includes product data, so we can display the product categories viewed by website visitors in each state. To display product categories in the map by color, drag the category field into the COLOR box.
  
• The map displays the product categories by color for each state. Move the pointer over each state to display detailed category information. We can see that the largest number of page hits in Florida were for clothing, followed by shoes.
  
• Now let’s look at the clothing data by age and gender so we can optimize our content for these customers. Select Insert > Power View to open a new Power View report.
  

To set up the data, set the following fields and filters:

• In the Power View Fields area, select ip and age. All of the other fields should be unselected.
• Drag category from the Power View Fields area into the Filters area, then select the clothing checkbox.
• Drag gender from the Power View Fields area into the Filters area, then select the M (male) checkbox.

After setting these fields and filters, select Column Chart > Clustered Column in the top menu.

• To finish setting up the chart, drag age into the AXIS box. Also, remove ip from the AXIS box by clicking the down-arrow and then Remove Field. The chart shows that the majority of men shopping for clothing on our website are between the ages of 22 and 30. With this information, we can optimize our content for this MARKET segment.
  
• Let’s assume that our data includes information about website pages (URLs) with high bounce rates. A page is considered to have a high bounce rate if it is the last page a user visited before leaving the website. By filtering this URL data by our target age group, we can find out exactly which website pages we should optimize for this MARKET segment. Select Insert > Power View to open a new Power View report.
  

To set up the data, set the following fields and filters:

• Drag age from the Power View Fields area into the Filters area, then drag the sliders to set the age range from 22 to 30.
• Drag gender from the Power View Fields area into the Filters area, then select the M (male) checkbox.
• Drag country from the Power View Fields area into the Filters area, then select the usa checkbox.
• In the Power View Fields area, select url. All of the other fields should be unselected.
• In the Power View Fields area, move the pointer over url, click the down-arrow, and then select Add to Table as Count.

After setting these fields and filters, select Column Chart > Clustered Column in the top menu.

• The chart shows that we should focus on optimizing four of our website pages for the MARKET segment of men between the ages of 22 and 30. Now we can redesign these four pages and test the new designs based on our target demographic, thereby reducing the bounce rate and increasing customer retention and sales.
  
• You can use the controls in the upper left corner of the map to sort by Count of URL in ascending order.
  

Now that you have successfully analyzed and visualized Hortonworks Sandbox data with Microsoft Excel, you can see how Excel and other BI tools can be used with the Hortonworks platform to derive insights about customers from various data sources.

The data in the Hortonworks platform can be refreshed frequently and used for basket analysis, A/B testing, personalized product recommendations, and other sales optimization activities.

Hello World! – An introduction to Hadoop with Hive and Pig

Overview of Apache Hadoop and Hortonworks Data Platform

The Hortonworks Sandbox is a single node implementation of the Hortonworks Data Platform(HDP). It is packaged as a virtual machine to make evaluation and experimentation with HDP fast and easy. The tutorials and features in the Sandbox are oriented towards exploring how HDP can help you solve your business big data problems. The Sandbox tutorials will walk you through bringing some sample data into HDP and manipulate it using the tools built into HDP. The idea is to show you how you can get started and show you how to accomplish tasks in HDP. HDP is FREE to download and use in your enterprise and you can download it here: Hortonworks Data Platform
Download

The Apache Hadoop projects provide a series of tools designed to solve big data problems. The Hadoop cluster implements a parallel computing cluster using inexpensive commodity hardware. The cluster is partitioned across many servers to provide a near linear scalability. The philosophy of the cluster design is to bring the computing to the data. So each datanode will hold part of the overall data and be able to process the data that it holds. The overall framework for the processing software is called MapReduce. Here's a short video introduction to MapReduce: Introduction to MapReduce

Apache Hadoop can be useful across a range of use cases spanning virtually every vertical industry. It is becoming popular anywhere that you need to store, process, and analyze large volumes of data. Examples include digital marketing automation, fraud detection and prevention, social network and relationship analysis, predictive modeling for new drugs, retail in-store behavior analysis, and mobile device location-based marketing.

---

The Hadoop Distributed File System

In this section we are going to take a closer look at some of the components we will be using in the Sandbox tutorials. Underlying all of these components is the Hadoop Distributed File System(HDFS™). This is the foundation of the Hadoop cluster. The HDFS file system manages how the datasets are stored in the Hadoop cluster. It is responsible for distributing the data across the datanodes, managing replication for redundancy and administrative tasks like adding, removing and recovery of datanodes.

Apache Hive™

The Apache Hive project provides a data warehouse view of the data in HDFS. Using a SQL-like language Hive lets you create summarizations of your data, perform ad-hoc queries, and analysis of large datasets in the Hadoop cluster. The overall approach with Hive is to project a table structure on the dataset and then manipulate it with HiveQL. Since you are using data in HDFS your operations can be scaled across all the datanodes and you can manipulate huge datasets.

Apache HCatalog

The function of HCatalog is to hold location and metadata about the data in a Hadoop cluster. This allows scripts and MapReduce jobs to be decoupled from data location and metadata like the schema. Additionally since HCatalog supports many tools, like Hive and Pig, the location and metadata can be shared between tools. Using the open APIs of HCatalog other tools like Teradata Aster can also use the location and metadata in HCatalog. In the tutorials we will see how we can now reference data by name and we can inherit the location and metadata.

Apache Pig™

Pig is a language for expressing data analysis and infrastructure processes. Pig is translated into a series of MapReduce jobs that are run by the Hadoop cluster. Pig is extensible through user-defined functions that can be written in Java and other languages. Pig scripts provide a high level language to create the MapReduce jobs needed to process data in a Hadoop cluster.

That‘s all for now… let‘s get started with some examples of using these tools together to solve real problems!

Using HDP

Here we go! We're going to walk you through a series of step-by-step tutorials to get you up and running with the Hortonworks Data Platform(HDP).

Downloading Example Data

We'll need some example data for our lessons. For our first lesson, we'll be using stock ticker data from the New York Stock Exchange from the years 2000-2001. You can download this file here:

https://s3.amazonaws.com/hw-sandbox/tutorial1/NYSE-2000-2001.tsv.gz

The file is about 11 megabytes, and may take a few minutes to download. Fortunately, to learn 'Big Data' you don't have to use a massive dataset. You need only use tools that scale to massive datasets. Click and SAVEthis file to your computer.

Using the File Browser

You can reach the File Browser by clicking its icon:

The File Browser interface should be familiar to you as it is similar to the file manager on a Windows PC or Mac. We begin in our home directory. This is where we'll store the results of our work. File Browser also lets us upload files.

Uploading a File

To upload the example data you just downloaded,

• Select the 'Upload' button
• Select 'Files' and a pop-up window will appear.
• Click the button which says, 'Upload a file'.
• Locate the example data file you downloaded and select it.
• A progress meter will appear. The upload may take a few moments.

When it is complete you'll see this:

Now click the file name "NYSE-2000-2001.tar.gz". You'll see it, displayed in tabular form:

You can use File Browser just like your own computer's file manager. Next register the dataset with HCatalog.

Loading the sample data into HCatalog

Now that we've uploaded a file to HDFS, we will register it with HCatalog to be able to access it in both Pig and Hive.

Select the HCatalog icon in the icon bar at the top of the page:

Select "Create a new table from file" from the Actions menu on the left.

Fill in the Table Name field with 'nyse_stocks'. Then click on Choose a file button. Select the file we just uploaded 'NYSE-2000-2001.tsv.gz'.

You will now see the options for importing your file into a table. The File options should be fine. In Table preview set all text type fields to Column Type 'string' and all decimal fields (ex: 12.55) to Column Type 'float.' The one exception is 'stock_volume' field should be set as 'bigint.' When everything is complete click on the "Create Table" button at the bottom.

A Short Apache Hive Tutorial

In the previous sections you:

• Uploaded your data file into HDFS
• Used Apache HCatalog to create a table

Apache Hive™ provides a data warehouse function to the Hadoop cluster. Through the use of HiveQL you can view your data as a table and create queries like you would in a database.

To make it easy to interact with Hive we use a tool in the Hortonworks Sandbox called Beeswax. Beeswax gives us an interactive interface to Hive. We can type in queries and have Hive evaluate them for us using a series of MapReduce jobs.

Let's open Beeswax. Click on the bee icon on the top bar.

On the right hand side there is a query window and an execute button. We will be typing our queries in the query window. When you are done with a query please click on the execute button. Note: There is a limitation of one query in the composition window. You can not type multiple queries separated by semicolons.

Since we created our table in HCatalog, Hive automatically knows about it. We can see the tables that Hive knows about by clicking on the Tables tab.

In the list of the tables you will see our table, nyse_stocks. Hive inherits the schema and location information from HCatalog. This separates meta information like schema and location from the queries. If we did not have HCatalog we would have to build the table by providing location and schema information.

We can see the records by typing Select * from nyse_stocks in the Query window. Our results would be:

We can see the columns in the table by executing describe nyse_stocks

We will then get a description of the nyse table.

We can count the records with the query select count(*) from nyse_stocks. You can click on the Beeswax icon to get back to the query screen. Evaluate the expression by typing it in the query window and hitting execute.

This job takes longer and you can watch the job running in the log. When the job is complete you will see the results posted in the Results tab.

You can select specific records by using a query like select * from nyse_stocks where stock_symbol="IBM".

This will return the records with IBM.

So we have seen how we can use Apache Hive to easily query our data in HDFS using the Apache Hive query language. We took full advantage of HCatalog so we did not have to specify our schema or location of the data. Apache Hive allows people who are knowledgable in query languages like SQL to immediately become PRODUCTIVE with Apache Hadoop. Once they know the schema of the data can they quickly and easily formulate queries.

Pig Basics Tutorial

In this tutorial we create and run Pig scripts. On the left is a list of scripts that we have created. In the middle is an area for us to compose our scripts. We will also load the data from the table we have stored in HCatalog. We will then filter out the records for the stock symbol IBM. Once we have done that we will calculate the average of closing stock prices over this period.

The basic steps will be:

• Step 1: Create and name the script
• Step 2: Loading the data
• Step 3: Select all records starting with IBM
• Step 4: iterate and average
• Step 5: SAVE the script and execute it

Let's get started…

To get to the Pig interface click on the Pig icon on the icon bar at the top. This will bring up the Pig user interface. On the left is a list of your scripts and on the right is a composition box for your scripts.

A special feature of the interface is the Pig helper at the bottom. The Pig helper will provide us with templates for the statements, functions, I/O statements, HCatLoader() and Python user defined functions.

At the very bottom are status areas that will show the results of our script and log files

Step 1: Create and name the script

• Open the Pig interface by clicking the Pig icon at the top of the screen
  
• Title your script by filling in the title box
  

Step 2: Loading the data

Our first line in the script will load the table. We are going to use HCatalog because this allows us to share schema across tools and users within our Hadoop environment. HCatalog allows us to factor out schema and location information from our queries and scripts and centralize them in a common repository. Since it is in HCatalog we can use the HCatLoader() function. Pig makes it easy by allowing us to give the table a name or alias and not have to worry about allocating space and defining the structure. We just have to worry about how we are processing the table.

• On the right hand side we can start adding our code at Line 1
• We can use the Pig helper at the bottom of the screen to give us a template for the line. Click on Pig helper -> HCatalog->load template
• The entry %TABLE% is highlighted in red for us. Type the name of the table which is nyse_stocks.
• Remember to add the a = before the template. This SAVES the results into a. Note the `= has to have a space before and after it.

Our completed line of code will look like:

a = LOAD ‘default.nyse_stocks’ USING org.apache.hcatalog.pig.HCatLoader()

So now we have our table loaded into Pig and we stored it "a"

Step 3: Select all records starting with IBM

The next step is to select a subset of the records so that we just have the records for stock ticker of IBM. To do this in Pig we use the Filter operator. We tell Pig to Filter our table and keep all records where stock_symbol="IBM" and store this in b. With this one simple statement Pig will look at each record in the table and filter out all the ones that do not meet our criteria. The group statement is important because it groups the records by one or more relations. In this case we just specified all rather than specify the exact relation we need.

• We can use Pig Help again by clicking on Pig helper->Relational Operators->FILTER template
• We can replace %VAR% with "a" (hint: tab jumps you to the next field)
• Our %COND% is "stock_symbol =='IBM'` " (note: single quotes are needed around IBM and don't forget the trailing semi-colon)
• Pig helper -> Relational Operators->GROUP BY template
• The first %VAR% is "b" and the second%VAR%is "all". You will need to correct an irregularity in the Pig syntax here. Remove the "BY`" in the line of code.
• Again add the trailing semi-colon to the code.

So the final code will look like:

b = filter a by stock_symbol == 'IBM';
c = group b all;

Now we have extracted all the records with IBM as the stock_symbol.

Step 4: Iterate and Average

Now that we have the right set of records we can iterate through them and create the average. We use the "foreach" operator on the grouped data to iterate through all the records. The AVG() function creates the average of the stock_volume field. To wind it up we just print out the results which will be a single floating point number. If our results would be used for a future job we can SAVE it back into a table.

• Pig helper ->Relational Operators->FOREACH template will get us the code
• Our first %VAR% is c and the second %VAR% is "AVG(b.stock_volume);"
• We add the last line with Pig helper->I/O->DUMP template and replace %VAR% with "d".

Our last two lines of the script will look like:

d = foreach c generate AVG(b.stock_volume);
dump d;

So the variable "d" will contain the average volume of IBM stock when this
line is executed.

Step 5: Save the script and Execute it

We can save our completed script using the Save button at the bottom and then we can Execute it. This will create a MapReduce job(s) and after it runs we will get our results. At the bottom there will be a progress bar that shows the job status.

• At the bottom we click on the Save button again
• Then we click on the Execute button to run the script
• Below the Execute button is a progress bar that will show you how things are running.
• When the job completes you will see the results in the green box.
• Click on the Logs link to see what happened when your script ran. The average of stock_volume This is where you will see any error messages. The log may scroll below the edge of your window so you may have to scroll down.

Summary

Now we have a complete script that computes the average volume of IBM stock. You can download the results by clicking on the green download icon above the green box.

If you look at what our script has done, you see in Line 5 we:

• Pulled in the data from our table using HCatalog, we took advantage that HCatalog provided us with location and schema information, if that needs to change in the future we would not have to rewrite our script.
• Pig then went through all the rows in the table and discarded the ones where the stock_symbol field is not IBM
• Then an index was built for the remaining records
• The average of stock_volume was calculated on the records

We did it with 5 lines of Pig script code!