因mapjoin加载内存溢出而导致return code 3
问题描述:
例行Hive作业报错
日志定位:
Starting to launch local task to process map join; maximum memory = 5172101120
[2023-10-16 07:56:51,530] - INFO: [HIVE] 2023-10-16 07:56:51 Processing rows: 6200000 Hashtable size: 6199999 Memory usage: 2857332576 percentage: 0.552
[2023-10-16 07:56:52,722] - INFO: [HIVE] Execution failed with exit status: 3
[2023-10-16 07:56:52,724] - INFO: [HIVE] FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
问题解决:
可以看到在执行mapjoin的时候出现了内存报错,由于最大内存值为5172101120,而进度55.2%的时候已经使用了2857332576,由于2857332576/0.552 = 5176327130 > 5172101120,所以报错return code 3。