【6.824】分布式lab1 mapReduce

Paper main idea

复制代码
Figure 1 shows the overall flow of a MapReduce operation in our implementation. When the user program calls the MapReduce function, the following sequence of actions occurs (the numbered labels in Figure 1 correspond to the numbers in the list below): 

1. The MapReduce library in the user program first splits the input files into M pieces of typically 16 megabytes to 64 megabytes (MB) per piece (controllable by the user via an optional parameter). It then starts up many copies of the program on a cluster of machines. 

2. One of the copies of the program is special -- the master. The rest are workers that are assigned work by the master. There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task. 

3. A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory

4. Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function. The locations of these buffered pairs on the local disk are passed back to the master, who is responsible for forwarding these locations to the reduce workers. 

5. When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of the map workers. When a reduce worker has read all intermediate data, it sorts it by the intermediate keys so that all occurrences of the same key are grouped together. The sorting is needed because typically many different keys map to the same reduce task. If the amount of intermediate data is too large to fit in memory, an external sort is used. 

6. The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user's Reduce function. The output of the Reduce function is appended to a final output file for this reduce partition

7. When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user program returns back to the user code.

some ideas

Task Synchronous:

all map task should be earlier than reduce task!

some map task may wait seconds, so should first all map tasks done before reduce.

Lock:

workers run parallel, should add mutex to filelist when

Error Return:

workers may run some error, such as rpc sock connect and can not open files. When these errors happend, should return Error to master, other than just stop itself.

Heartbeat:

master should ask workers whether they are alive.

Implementation method:

  1. Set a new scheduled task and ask every 2 minutes.
  2. Every time a new worker requests a new task, add a judgment when rotating all tasks to assign tasks to the worker. If a task status is not Finish and times out, the task fails
  3. Write in the Done() method and continuously rotate to ensure that all tasks are completed. And take the opportunity to determine if there is a task timeout.

Performance comparison: 1>2>3

Judging by frequency. The execution frequency of 3 is too high, constantly executing and discussing the sequence.

status of task and worker

worker:

  • Finish: worker finish task X
  • Ready: worker ask for a new task
  • Wrong: worker meet some problems
  • Close: worker will be closed when master ask worker to close

task:

  • Run: some worker is running on this task
  • Done: this task is finished
  • (empty string): task not be allocated yet

lab test:

pass all the test!

相关推荐
B站_计算机毕业设计之家3 分钟前
Spark微博舆情分析系统 情感分析 爬虫 Hadoop和Hive 贴吧数据 双平台 讲解视频 大数据 Hadoop ✅
大数据·hadoop·爬虫·python·数据分析·1024程序员节·舆情分析
赵渝强老师1 小时前
【赵渝强老师】大数据日志采集引擎Flume
大数据·flume
TDengine (老段)2 小时前
TDengine 数据函数 ROUND 用户手册
java·大数据·数据库·物联网·时序数据库·tdengine·1024程序员节
TDengine (老段)2 小时前
TDengine 数学函数 RAND 用户手册
java·大数据·数据库·物联网·时序数据库·tdengine·涛思数据
Web3_Daisy2 小时前
冷换仓的隐性代价:从安全策略到地址信誉体系的重新思考
大数据·安全·web3·区块链·比特币·1024程序员节
WLJT1231231233 小时前
生活电器:重构家居体验的产业变革与发展探索
大数据·人工智能·科技·生活
GIS数据转换器4 小时前
城市基础设施安全运行监管平台
大数据·运维·人工智能·物联网·安全·无人机·1024程序员节
老葱头蒸鸡4 小时前
(3)Kafka生产者分区策略、ISR、ACK、一致性语义
分布式·kafka
搞科研的小刘选手4 小时前
【云计算专题会议】第二届云计算与大数据国际学术会议(ICCBD 2025)
大数据·人工智能·物联网·5g·云计算·6g·智能通信
电商软件开发 小银4 小时前
微信生态新机遇:视频号推客模式助力商家突围
大数据·人工智能·twitter·系统开发·实体店转型·数字化经济·视频号推客模式