Cloudera Quick Start VM Big Data Assignment Help | Implement Big Data Map Reduce Problem

Jul 10, 20222 min read

Problem Requirement

Implementation

Data:

#!/usr/bin/env python
  
# import sys because we need to read and write data to STDIN and STDOUT
import sys
  
# reading entire line from STDIN (standard input)
for line in sys.stdin:
    # to remove leading and trailing whitespace
    line = line.strip()
    # split the line into words
    words = line.split()
      
    # we are looping over the words array and printing the word
    # with the count of 1 to the STDOUT
    # for word in words:
        # write the results to STDOUT (standard output);
        # what we output here will be the input for the
        # Reduce step, i.e. the input for reducer.py
    print ('%s\t%s' % (words[0], words[1]))

reducer.py

#!/usr/bin/env python

from operator import itemgetter

import sys

###################################

Write code here

##################################

How to run it

We make a directory in Hadoop named input in the location /user/cloudera/input as can be seen above, here we will be transferring our data.txt file

Here we transfer the data.txt in directory we just created (input directory)

In second command we check list of files in the directory /user/cloudera/input and we find that our data.txt file is successfully moved there

Using the -cat command we see the contants of data.txt files which contains each English word 3 times and its corresponding translation in 3 different languages

Now its time to run our mapper.py and reducer.py files on the input data.txt file we enter the command and wait for the completion of execution

Screenshot part where the end represents the progress has started and currently execution of mapper.py or map utility is 0% and

reducer.py or reduce is also 0%

Here at 13:49:31 we see that map utility is done 100% and at 13:52:13 we see that reduce utility is also completed 100% and therefore completing our task