几个hadoop中运行mapreduce的错误记录一下。

1、记录一下错误，修改配置后，重新格式化namenode后，机器上的datanode没有启动起来。

2026-10-04 21:08:56,865 WARN hdfs.DataStreamer: DataStreamer Exception

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/root/QuasiMonteCarlo_1791119335675_932816193/in/part0 could only be written to 0 of the 1 minReplication nodes. There are 0 datanode(s) running and 0 node(s) are excluded in this operation.

提醒：要删除hadoop下的tmpData目录，（名称根据实际情况变动）后重新格式化namenode。

2、mapred的一些配置

2026-10-04 21:17:47,500 INFO mapreduce.Job: Job job_1791119580394_0001 failed with state FAILED due to: Application application_1791119580394_0001 failed 2 times due to AM Container for appattempt_1791119580394_0001_000002 exited with exitCode: 1

Failing this attempt.Diagnostics: $2026-10-04 21:17:47.305$ Exception from container-launch.

Container id: container_1791119580394_0001_02_000001

Exit code: 1

$2026-10-04 21:17:47.309$ Container exited with a non-zero exit code 1. Error file: prelaunch.err.

Last 4096 bytes of prelaunch.err :

Last 4096 bytes of stderr :

错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

$2026-10-04 21:17:47.311$ Container exited with a non-zero exit code 1. Error file: prelaunch.err.

Last 4096 bytes of prelaunch.err :

Last 4096 bytes of stderr :

错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1791119580394_0001 Then click on links to logs of each attempt.

. Failing the application.

2026-10-04 21:17:47,524 INFO mapreduce.Job: Counters: 0

Job job_1791119580394_0001 failed!

解决方法：

打开mapred-site.xml文件，添加下面代码

<name>yarn.app.mapreduce.am.env</name>

<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

</property>

<name>mapreduce.map.env</name>

<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

</property>

<name>mapreduce.reduce.env</name>

<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

</property>

3、mapreduce_shuffle要配置。

2026-10-04 21:43:50,739 INFO mapreduce.Job: Task Id : attempt_1791121357322_0001_m_000000_2, Status : FAILED

Container launch failed for container_1791121357322_0001_01_000019 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

at sun.reflect.GeneratedConstructorAccessor48.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:163)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:394)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

2026-10-04 21:53:03,023 INFO mapreduce.Job: Task Id : attempt_1791121853079_0001_m_000000_2, Status : FAILED

$2026-10-04 21:53:00.316$ Container $pid=27514,containerID=container_1791121853079_0001_01_000069$ is running 437209600B beyond the 'VIRTUAL' memory limit. Current usage: 62.2 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1791121853079_0001_01_000069 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

|- 27514 27512 27514 27514 (bash) 0 0 115834880 308 /bin/bash -c /opt/module/jdk1.8/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/tmpData/nm-local-dir/usercache/root/appcache/application_1791121853079_0001/container_1791121853079_0001_01_000069/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.2.105 44737 attempt_1791121853079_0001_m_000000_2 69 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069/stderr

|- 27528 27514 27514 27514 (java) 84 4 2576232448 15611 /opt/module/jdk1.8/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/tmpData/nm-local-dir/usercache/root/appcache/application_1791121853079_0001/container_1791121853079_0001_01_000069/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1791121853079_0001/container_1791121853079_0001_01_000069 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.2.105 44737 attempt_1791121853079_0001_m_000000_2 69

$2026-10-04 21:53:00.327$ Container killed on request. Exit code is 143

$2026-10-04 21:53:00.755$ Container exited with a non-zero exit code 143.

配置

yarn-site.xml中添加下面内容：

<name>yarn.nodemanager.resource.memory-mb</name>

<discription>每个节点可用内存,单位MB</discription>

</property>

<name>yarn.scheduler.minimum-allocation-mb</name>

<discription>单个任务可申请最少内存，默认1024MB</discription>

</property>

<name>yarn.scheduler.maximum-allocation-mb</name>

<discription>单个任务可申请最大内存，默认8192MB</discription>

</property>

mapred-site.xml

<name>mapreduce.map.memory.mb</name>

<description>每个Map任务的物理内存限制</description>

</property>

<name>mapreduce.reduce.memory.mb</name>

<description>每个Reduce任务的物理内存限制</description>

</property>

<name>mapreduce.map.java.opts</name>

</property>

<name>mapreduce.reduce.java.opts</name>

</property>

<name>mapreduce.framework.name</name>

</property>

参考csdn:https://blog.csdn.net/qq_40059532/article/details/103493928/