Hadoop-HA高可用集群启动nameNode莫名挂掉，排错解决

岁月的眸2025-05-22 11:59

Hadoop-HA高可用集群启动nameNode莫名挂掉，排错解决

nameNode错误日志

复制代码 
2025-05-21 16:14:12,218 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:12,219 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:12,251 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:13,144 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 7009 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet.
2025-05-21 16:14:13,220 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:13,223 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:13,257 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:14,149 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 8014 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet.
2025-05-21 16:14:14,227 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:14,238 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:14,260 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:15,151 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 9016 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet.
2025-05-21 16:14:15,238 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:15,250 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:15,268 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:16,164 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 10029 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet.
2025-05-21 16:14:16,239 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:16,285 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:16,285 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:16,302 WARN org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input streams from QJM to [192.168.191.111:8485, 192.168.191.112:8485, 192.168.191.113:8485]. Skipping.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
192.168.191.112:8485: Call From node01/192.168.191.111 to node02:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
192.168.191.113:8485: Call From node01/192.168.191.111 to node03:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
192.168.191.111:8485: Call From node01/192.168.191.111 to node01:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
        at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:485)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:269)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1672)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1705)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:297)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:449)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:399)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:416)
        at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:482)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:412)
2025-05-21 16:14:16,367 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for standby state
2025-05-21 16:14:16,369 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem write lock held for 10180 ms via java.lang.Thread.getStackTrace(Thread.java:1559)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:273)
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:225)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1614)
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:337)
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:449)
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:399)
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:416)
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:482)
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:412)
        Number of suppressed write-lock reports: 0
        Longest write-lock held interval: 10180.0
        Total suppressed write-lock held time: 0.0
2025-05-21 16:14:16,414 WARN org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Edit log tailer interrupted
java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:469)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:399)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:416)
        at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:482)
        at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:412)
2025-05-21 16:14:16,421 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2025-05-21 16:14:16,466 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Starting recovery process for unclosed journal segments...
2025-05-21 16:14:17,717 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:17,718 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:17,722 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:18,720 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:18,723 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:18,724 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:19,753 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:19,754 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:19,754 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:20,755 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:20,772 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:20,773 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:21,756 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:21,814 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:21,901 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:22,776 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:22,958 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:22,973 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:23,789 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:23,990 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:24,001 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:24,798 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:25,030 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:25,049 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:25,814 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:26,057 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:26,086 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:26,862 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node01/192.168.191.111:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:27,103 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node02/192.168.191.112:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:27,148 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: node03/192.168.191.113:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2025-05-21 16:14:27,191 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [192.168.191.111:8485, 192.168.191.112:8485, 192.168.191.113:8485], stream=null))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
192.168.191.111:8485: Call From node01/192.168.191.111 to node01:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
192.168.191.113:8485: Call From node01/192.168.191.111 to node03:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
192.168.191.112:8485: Call From node01/192.168.191.111 to node02:8485 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286)
        at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createNewUniqueEpoch(QuorumJournalManager.java:204)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.recoverUnfinalizedSegments(QuorumJournalManager.java:443)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet$6.apply(JournalSet.java:616)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet.recoverUnfinalizedSegments(JournalSet.java:613)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.recoverUnclosedStreams(FSEditLog.java:1602)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1223)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1887)
        at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
        at org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64)
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1746)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1723)
        at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
        at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
2025-05-21 16:14:27,196 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [192.168.191.111:8485, 192.168.191.112:8485, 192.168.191.113:8485], stream=null))
2025-05-21 16:14:27,426 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node01/192.168.191.111
************************************************************/

出现报错的原因

我们在执行start-dfs.sh的时候，默认启动顺序是namenode => datanode => journalnode => zkfc，如果journalnode和namenode不在一台机器启动的话，很容易因为网络延迟问题导致NN无法连接JN，无法实现选举，最后导致刚刚启动的namenode会突然挂掉一个主的，留下一个standy的，虽然有NN启动时有重试机制等待JN的启动，但是由于重试次数限制，可能网络情况不好，导致重试次数用完了，也没有启动成功。

  • A： 此时需要手动启动主的那个namenode，避免了网络延迟等待journalnode的步骤，一旦两个namenode连入journalnode，实现了选举，则不会出现失败情况。

  • B: 先启动JournalNode然后再运行start-dfs.sh，

  • C: 把nn对jn的容错次数或时间调成更大的值，保证能够对正常的启动延迟、网络延迟能容错。

解决方法

  1. 在hdfs-site.xml中加入，nn对jn检测的重试次数，默认为10次，每次1000ms，故网络情况差需要增加，这里设置为30次
xml 复制代码 
    <property>
         <name>ipc.client.connect.max.retries</name>
         <value>30</value>
    </property>
  1. 先启动journalnode，再启动dfs

封装一个简单的集群启动脚本

  • 按照先启动zookeeper，再启动Hadoop的先后顺序，在启动启动的spark等等。

集群启动脚本

shell 复制代码 
#!/bin/bash

if [ $# -lt 1 ]
then
  echo "No Args Input..."
  exit ;
fi

# 启动Zookeeper集群
start_zookeeper() {
  echo " =================== 启动 zookeeper 集群 ==================="
  
  # 在 node01, node02, node03 上启动 zookeeper
  for node in node01 node02 node03
  do
    echo " --------------- 启动 $node 的 zookeeper ---------------"
    ssh $node "/opt/yjx/zookeeper-3.4.5/bin/zkServer.sh start"
  done

  # 检查每个节点的 zookeeper 状态
  echo " --------------- 检查 zookeeper 状态 ---------------"
  for node in node01 node02 node03
  do
    echo "检查 $node 的 zookeeper 状态"
    ssh $node "/opt/yjx/zookeeper-3.4.5/bin/zkServer.sh status"
  done
}

# 关闭Zookeeper集群
stop_zookeeper() {
  echo " =================== 关闭 zookeeper 集群 ==================="
  
  # 在 node01, node02, node03 上关闭 zookeeper
  for node in node01 node02 node03
  do
    echo " --------------- 检查 $node 上 Zookeeper 是否已经启动 ---------------"
    
    # 检查端口2181是否有进程占用
    process_check=$(ssh $node "netstat -tuln | grep 2181")
    if [ -z "$process_check" ]; then
      echo "$node 上的 Zookeeper 没有运行，跳过停止操作"
    else
      echo "$node 上的 Zookeeper 正在运行，执行停止操作"
      ssh $node "/opt/yjx/zookeeper-3.4.5/bin/zkServer.sh stop"
    fi
  done
}

case $1 in
"start")
  # 先启动 Zookeeper，再启动 Hadoop
  start_zookeeper

  echo " =================== 启动 hadoop 集群 ==================="
  
  
  #ssh node01 "/opt/yjx/hadoop-3.1.2/sbin/hadoop-daemon.sh start journalnode"
  #ssh node02 "/opt/yjx/hadoop-3.1.2/sbin/hadoop-daemon.sh start journalnode"
  #ssh node03 "/opt/yjx/hadoop-3.1.2/sbin/hadoop-daemon.sh start journalnode"
  echo " --------------- 启动 journalnode ---------------"
  ssh node01 hdfs --daemon start journalnode
  ssh node02 hdfs --daemon start journalnode
  ssh node03 hdfs --daemon start journalnode
  
  echo " --------------- 启动 hdfs ---------------"
  
  ssh node01 "/opt/yjx/hadoop-3.1.2/sbin/start-dfs.sh"
  
  echo " --------------- 启动 yarn ---------------"
  ssh node01  "/opt/yjx/hadoop-3.1.2/sbin/start-yarn.sh"
  echo " --------------- 启动 historyserver ---------------"
  ssh node01 "/opt/yjx/hadoop-3.1.2/bin/mapred --daemon start historyserver";;

"stop")
  # 先关闭 Hadoop，再关闭 Zookeeper
  echo " =================== 关闭 hadoop 集群 ==================="
  
  echo " --------------- 关闭 historyserver ---------------"
  ssh node01 "/opt/yjx/hadoop-3.1.2/bin/mapred --daemon stop historyserver"
  echo " --------------- 关闭 yarn ---------------"
  ssh node01 "/opt/yjx/hadoop-3.1.2/sbin/stop-yarn.sh"
  echo " --------------- 关闭 hdfs ---------------"
  ssh node01 "/opt/yjx/hadoop-3.1.2/sbin/stop-dfs.sh"

  # 关闭 Zookeeper 集群
  stop_zookeeper ;;

*)
  echo "Input Args Error..." ;;
esac
相关推荐
SelectDB11 分钟前
Apache Doris+ Paimon 2.0：构建 Agentic AI 数据闭环
大数据·数据库·数据分析
进击的横打28 分钟前
【人工智能】AI时代公司组织架构的重构
大数据·人工智能·重构
YangYang9YangYan38 分钟前
校招视角｜财务数字化岗位 SQL、工具、项目完整备考指南
大数据·数据库·sql
光锥智能1 小时前
小鹏第一位机器人自己走下产线
大数据·人工智能
GISMagic1 小时前
Agent学习，写在开始之前
大数据·学习·agent
Capricorn19881 小时前
知芽 Notebook Skill 引用存在性校验与单元记忆破解幽灵引用危机
大数据·论文阅读·人工智能·论文笔记
hgfsjk2 小时前
跨境电商ERP系统软件哪个好？2026年选型指南
大数据·经验分享·笔记·其他
APItesterCris3 小时前
告别人工盯品！借助 Open‑Claw 快速搭建电商商品全自动监控与数据分析系统（完整实操代码）
java·大数据·前端·数据库
BYSJMG3 小时前
计算机毕设选题推荐：基于大数据的公共交通运营数据分析与可视化，Spark+K-Means聚类城市运营模式
大数据·hadoop·信息可视化·数据分析·spark·kmeans·课程设计
顧棟3 小时前
【ES实战】 低版本Elasticsearch(不具备CCR)集群迁移
大数据·elasticsearch·搜索引擎
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02GitHub 镜像站点03如何新建文件夹？ 电脑新建文件夹的4种方法04历年考研数学一、数学二、数学三真题试卷及答案PDF052026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？06蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接07微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新08ZCode 周末送额度活动开启：3 亿 Token 免费领取09Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费10AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路