EMS5730 MapReduce program

EMS5730 Spring 2024 Homework #0

Release date: Jan 10, 2024

Due date: Jan 21, 2024 (Sunday) 23:59 pm

(Note: The course add-drop period ends at 5:30 pm on Jan 22.)

No late homework will be accepted!

Every Student MUST include the following statement, together with his/her signature in the

submitted homework.

I declare that the assignment submitted on the Elearning system is

original except for source material explicitly acknowledged, and that the

same or related material has not been previously submitted for another

course. I also acknowledge that I am aware of University policy and

regulations on honesty in academic work, and of the disciplinary

guidelines and procedures applicable to breaches of such policy and

regulations, as contained in the website

Submission notice:

● Submit your homework via the elearning system

General homework policies:

A student may discuss the problems with others. However, the work a student turns in must

be created COMPLETELY by oneself ALONE. A student may not share ANY written work or

pictures, nor may one copy answers from any source other than one's own brain.

Each student MUST LIST on the homework paper the name of every person he/she has

discussed or worked with. If the answer includes content from any other source, the

student MUST STATE THE SOURCE. Failure to do so is cheating and will result in

sanctions. Copying answers from someone else is cheating even if one lists their name(s) on

the homework.

If there is information you need to solve a problem but the information is not stated in the

problem, try to find the data somewhere. If you cannot find it, state what data you need,

make a reasonable estimate of its value and justify any assumptions you make. You will be

graded not only on whether your answer is correct, but also on whether you have done an

intelligent analysis.

Q0 [10 marks]: Secure Virtual Machines Setup on the Cloud

In this task, you are required to set up virtual machines (VMs) on a cloud computing

platform. While you are free to choose any cloud platform, Google Cloud is recommended.

References [1] and [2] provide the tutorial for Google Cloud and Amazon AWS, respectively.

The default network settings in each cloud platform are insecure. Your VM can be hacked

by external users, resulting in resource overuse which may charge your credit card a

big bill of up to $5,000 USD. To protect your VMs from being hacked and prevent any

financial losses, you should set up secure network configurations for all your VMs.

In this part, you need to set up a whitelist for your VMs. You can choose one of the options

from the following choices to set up your whitelist: 1. only the IP corresponding to your

current device can access your VMs via SSH. Traffic from other sources should be blocked.

2. only users in the CUHK network can access your VMs via SSH. Traffic outside CUHK

should be blocked. You can connect to CUHK VPN to ensure you are in the CUHK network

(IP Range: 137.189.0.0/16). Reference [3] provides the CUHK VPN setup information from

ITSC.

a. [10 marks] Secure Virtual Machine Setup

Reference [4] and [5] are the user guides for the network security configuration of

AWS and Google Cloud, respectively. You can go through the document with respect

to the cloud platform you use. Then follow the listed steps to configure your VM's

network:

i. locate or create the security group/ firewall of your VM;

ii. remove all rules of inbound/ ingress and outbound/ egress, except for the

default rule(s) responsible for internal access within the cloud platform;

iii. add a new rule to the inbound/ ingress, with the SSH port(s) of VMs (default:

22. and source specified, e.g., '137.189.0.0/16' for CUHK users only;

iv. (Optional) more ports may be further permitted based on your needs (e.g.,

when completing Q1 below).

Q1 [90 marks + 20 bonus marks]: Hadoop Cluster Setup

Hadoop is an open-source software framework for distributed storage and processing. In this

problem, you are required to set up a Hadoop cluster using the VMs you instantiated in Q0.

In order to set up a Hadoop cluster with multiple virtual machines (VM), you can set up a

single-node Hadoop cluster for each VM first [6]. Then modify the configuration file in each

node to set up a Hadoop cluster with multiple nodes. References [7], [9], [10], [11] provide

the setup instructions for a Hadoop cluster. Some important notes/ tips on instantiating VMs

are given at the end of this section.

a. [20 marks] Single-node Hadoop Setup

In this part, you need to set up a single-node Hadoop cluster in a pseudo-distributed

mode and run the Terasort example on your Hadoop cluster.

i. Set up a single-node Hadoop cluster (recommended Hadoop version: 2.9.x,

all versions available in [16]). Attach the screenshot of http://localhost:50070

(or http://:50070 if opened in the browser of your local machine) to

verify that your installation is successful.

ii. After installing a single-node Hadoop cluster, you need to run the Terasort

example [8] on it. You need to record all your key steps, including your

commands and output. The following commands may be useful:

$ ./bin/hadoop jar \

./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar \

teragen 120000 terasort/input

//generate the data for sorting

$ ./bin/hadoop jar \

./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar \

terasort terasort/input terasort/output

//terasort the generated data

$ ./bin/hadoop jar \

./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar \

teravalidate terasort/output terasort/check

//validate the output is sorted

Notes: To monitor the Hadoop service via Hadoop NameNode WebUI (http://ip>:50070) on your local browser, based on steps in Q0, you may further allow traffic

from CUHK network to access port 50070 of VMs.

b. [40 marks] Multi-node Hadoop Cluster Setup

After the setup of a single-node Hadoop cluster in each VM, you can modify the

configuration files in each node to set up the multi-node Hadoop cluster.

i. Install and set up a multi-node Hadoop cluster with 4 VMs (1 Master and 3

Slaves). Use the 'jps' command to verify all the processes are running.

ii. In this part, you need to use the 'teragen' command to generate 2 different

datasets to serve as the input for the Terasort program. You should use the

following two rules to determine the size of the two datasets of your own:

■ Size of dataset 1: (Your student ID % 3 + 1) GB

■ Size of dataset 2: (Your student ID % 20 + 10) GB

Then, run the Terasort code again for these two different datasets and

compare their running time.

Hints: Keep an image for your Hadoop cluster. You would need to use the Hadoop

cluster again for subsequent homework assignments.

Notes:

1. You may need to add each VM to the whitelist of your security group/ firewall

and further allow traffic towards more ports needed by Hadoop/YARN

services (reference [17] [18]).

2. For step i, the resulting cluster should consist of 1 namenode and 4

datanodes. More precisely, 1 namenode and 1 datanode would be running on

the master machine, and each slave machine runs one datanode.

3. Please ensure that after the cluster setup, the number of "Live Nodes" shown

on Hadoop NameNode WebUI (port 50070) is 4.

c. [30 marks] Running Python Code on Hadoop

Hadoop streaming is a utility that comes with the Hadoop distribution. This utility

allows you to create and run MapReduce jobs with any executable or script as the

mapper and/or the reducer. In this part, you need to run the Python wordcount script

to handle the Shakespeare dataset [12] via Hadoop streaming.

i. Reference [13] introduces the method to run a Python wordcount script via

Hadoop streaming. You can also download the script from the reference [14].

ii. Run the Python wordcount script and record the running time. The following

command may be useful:

$ ./bin/hadoop jar \

./share/hadoop/tools/lib/hadoop-streaming-2.9.2.jar \

-file mapper.py -mapper mapper.py \

-file reducer.py -reducer reducer.py \

-input input/* \

-output output

//submit a Python program via Hadoop streaming

d. [Bonus 20 marks] Compiling the Java WordCount program for MapReduce

The Hadoop framework is written in Java. You can easily compile and submit a Java

MapReduce job. In this part, you need to compile and run your own Java wordcount

program to process the Shakespeare dataset [12].

i. In order to compile the Java MapReduce program, you may need to use

"hadoop classpath" command to fetch the list of all Hadoop jars. Or you can

simply copy all dependency jars in a directory and use them for compilation.

Reference [15] introduces the method to compile and run a Java wordcount

program in the Hadoop cluster. You can also download the Java wordcount

program from reference [14].

ii. Run the Java wordcount program and compare the running time with part c.

Part (d) is a bonus question for IERG 4300 but required for ESTR 4300.

IMPORTANT NOTES:

1. Since AWS will not provide free credits anymore, we recommend you to use Google

Cloud (which offers a 90-day, $300 free trial) for this homework.

2. If you use Putty for SSH client, please download from the website

https://www.putty.org/ and avoid using the default private key. Failure to do so will

subject your AWS account/ Hadoop cluster to hijacking.

3. Launching instances with Ubuntu (version >= 18.04 LTS) is recommended. Hadoop

version 2.9.x is recommended. Older versions of Hadoop may have vulnerabilities

that can be exploited by hackers to launch DoS attacks.

4. (AWS) For each VM, you are recommended to use the t2.large instance type with

100GB hard disk, which consists of 2 CPU cores and 8GB RAM.

5. (Google) For each VM, you are recommended to use the n2-standard-2 instance

type with 100GB hard disk, which consists of 2 CPU cores and 8GB RAM.

6. When following the given references, you may need to modify the commands

according to your own environment, e.g., file location, etc.

7. After installing a single-node Hadoop, you can save the system image and launch

multiple copies of the VM with that image. This can simplify your process of installing

the single-node Hadoop cluster on each VM.

8. Keep an image for your Hadoop cluster. You will need to use the Hadoop cluster

again for subsequent homework assignments.

9. Always refer to the logs for debugging single/multi-node Hadoop setup, which

contains more details than CLI outputs.

10. Please shut down (not to terminate) your VMs when you are not using them. This can

save you some credits and avoid being attacked when your VMs are idle.

Submission Requirements:

1. Include all the key steps/ commands, your cluster configuration details, source codes

of your programs, your compiling steps (if any), etc., together with screenshots, into a

SINGLE PDF report. Your report should also include the signed declaration (the first

page of this homework file).

2. Package all the source codes (as you included in step 1) into a zip file individually.

3. You should submit two individual files: your homework report (in PDF format) and a

zip file packaged all the codes of your homework.

4. Please submit your homework report and code zip file through the Blackboard

WX：codehelp