Macos系统上搭建Hadoop详细过程

一、环境检查

1、java环境检查

java -version

openjdk version "21.0.8" 2025-07-15 LTS

OpenJDK Runtime Environment Microsoft-11933201 (build 21.0.8+9-LTS)

OpenJDK 64-Bit Server VM Microsoft-11933201 (build 21.0.8+9-LTS, mixed mode, sharing)

2、ssh免密检查

Ahdim@Mac-Mini-Pro hadoop % ssh localhost

The authenticity of host 'localhost (127.0.0.1)' can't be established.

ED25519 key fingerprint is SHA256:kqg9GIrbTMPOEAy4wtxzz8aOk6y+U94zc14HaTKfip8.

This key is not known by any other names.

Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.

(Ahdim@localhost) Password:

Last login: Tue Nov  4 15:40:52 2025

如果提示如下信息：

Ahdim@Mac-Mini-Pro hadoop % ssh localhost

ssh: connect to host localhost port 22: Connection refused

按照如下步骤打开权限：

系统设置->通用->共享->远程登录->打开权限

3、创建目录（安装位置）

以下是我的安装位置，仅供参考，根据你的情况定义自己的安装位置
 mkdir -p /Users/Ahdim/data/server/hadoop
Ahdim@Mac-Mini-Pro hadoop % pwd                                     
/Users/Ahdim/data/server/hadoop

二、下载 & 解压

cd /Library/Hadoop

curl -O https://archive.apache.org/dist/hadoop/core/hadoop-3.3.6/hadoop-3.3.6.tar.gz

sudo tar -zxf hadoop-3.3.6.tar.gz

sudo mv hadoop-3.3.6 hadoop

三、环境配置

export HADOOP_HOME=/Library/Hadoop/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_HOME=$(/usr/libexec/java_home)
source ~/.zshrc

$HADOOP_HOME/etc/hadoop路径：

core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/Library/Hadoop/hadoop/tmp</value>
  </property>
</configuration>

hdfs-site.xml

<configuration>
  <property><name>dfs.replication</name><value>1</value></property>
  <property><name>dfs.namenode.http-address</name><value>localhost:9870</value></property>
</configuration>

mapred-site.xml

<configuration>
  <property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>

yarn-site.xml

<configuration>
  <property><name>yarn.resourcemanager.hostname</name><value>localhost</value></property>
  <property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
</configuration>

四、初始化 & 启动

首次格式化

hdfs namenode -format

一键启动

start-dfs.sh

start-yarn.sh

验证

jps # 应看到 NameNode/DataNode/ResourceManager/NodeManager

五、Web UI

• HDFS：http://localhost:9870

• YARN：http://localhost:8088

六、运行测试示例

echo "hello hadoop" > words.txt

hdfs dfs -mkdir -p /input

hdfs dfs -put words.txt /input

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /input /output

hdfs dfs -cat /output/part-r-00000