Deepin 图形化部署 Hadoop Single Node Cluster

升级操作系统和软件

快捷键 ctrl+alt+t 打开控制台窗口

更新 apt 源

shell 复制代码

sudo apt update

更新系统和软件

bash 复制代码

sudo apt -y dist-upgrade

升级后建议重启

开启ssh服务

打开资源管理器

进入系统盘

找到 etc 目录

在系统盘的 etc 目录上右键点击以管理员身份打开

输入密码

以管理员身份进入 etc 目录

进入 ssh 目录

在 /etc/ssh目录下编写 ssh_config

原始文件内容

bash 复制代码

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Protocol 2
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

在 /etc/ssh/ssh_config 文件中将 # Port 22 注释去掉
去掉注释后端文件内容

bash 复制代码

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
   Port 22
#   Protocol 2
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

重启 ssh 服务

bash 复制代码

sudo systemctl restart ssh

设置 ssh 服务开机启动

bash 复制代码

sudo systemctl enable ssh

上传安装包

将 jdk 和 hadoop 安装包拖入到 deepin 窗口即可
将桌面上的安装包剪切到用户主目录指定位置
jdk 保存在 /home/lhz/opt/java/jdk

hadoop 保存在 /home/lhz/opt
使用命令创建

bash 复制代码

mkdir -p /home/lhz/opt/java/jdk

解压安装包

选中 jdk 安装包右键解压到当前文件夹

选择解压缩后的jdk目录右键重命名

将 jdk 安装包重命名为 jdk-8

选中 hadoop 安装包右键解压到当前文件夹

选中 hadoop 解压后的目录右键重命名

将 hadoop 安装包重命名为 hadoop-3

配置环境变量

打开资源管理器

进入当前用户主目录

使用快捷键 ctrl+h 显示隐藏文件

编辑 .bashrc 文件

在 .bashrc 文件末尾追加以下内容

bash 复制代码

export JAVA_HOME=/home/lhz/opt/java/jdk/jdk-8

export HADOOP_HOME=/home/lhz/opt/hadoop-3

export HADOOP_INSTALL=${HADOOP_HOME}
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HDFS_NAMENODE_USER=lhz
export HDFS_DATANODE_USER=lhz
export HDFS_ZKFC_USER=lhz
export HDFS_JOURNALNODE_USER=lhz

export YARN_RESOURCEMANAGER_USER=lhz
export YARN_NODEMANAGER_USER=lhz

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

设置静态IP地址

网络图标右键

选择网络设置

点击网络详情查看当前相信网络信息

点击有线连接点击右侧由三角图表

点击 IPV4 下拉列表

选择手动

按照网络要求填入 IP地址、子网掩码、网关、DNS

修改主机名

控制台输入

bash 复制代码

sudo hostnamectl set-hostname hadoop

修改 hosts 文件

打开资源管理器

进入系统盘

找到 etc 目录

在系统盘的 etc 目录上右键点击以管理员身份打开

输入密码

以管理员身份进入 etc 目录

找到 /etc/hosts

内容修改为 192.168.171.129 hadoop

重启系统

修改 hadoop 配置文件

修改Hadoop配置文件在hadoop解压后的目录找到 etc/hadoop目录

修改如下配置文件

hadoop-env.sh

core-site.xml

hdfs-site.xml

workers

mapred-site.xml

yarn-site.xml

hadoop-env.sh 文件末尾追加

shell 复制代码

export JAVA_HOME=/home/lhz/opt/java/jdk/jdk-8
export HDFS_NAMENODE_USER=lhz
export HDFS_DATANODE_USER=lhz
export HDFS_ZKFC_USER=lhz
export HDFS_JOURNALNODE_USER=lhz

export YARN_RESOURCEMANAGER_USER=lhz
export YARN_NODEMANAGER_USER=lhz

core-site.xml

xml 复制代码

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/lhz/hadoop_data</value>
    </property>
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>lhz</value>
    </property>
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>hadoop.proxyuser.lhz.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.lhz.groups</name>
        <value>*</value>
    </property>
</configuration>

hdfs-site.xml

xml 复制代码

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

workers

shell 复制代码

hadoop

mapred-site.xml

xml 复制代码

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

yarn-site.xml

xml 复制代码

<?xml version="1.0"?>
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

配置ssh免密钥登录

创建本地秘钥并将公共秘钥写入认证文件

shell 复制代码

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
# 或者
ssh-copy-id hadoop

Hadoop初始化

shell 复制代码

# 格式化文件系统
hdfs namenode -format
# 启动 NameNode SecondaryNameNode DataNode 
start-dfs.sh
# 查看启动进程
jps
# 看到 DataNode SecondaryNameNode NameNode 三个进程代表启动成功

shell 复制代码

# 启动 ResourceManager daemon 和 NodeManager
start-yarn.sh
# 看到 DataNode NodeManager SecondaryNameNode NameNode ResourceManager 五个进程代表启动成功

重点提示：

shell 复制代码

# 关机之前 依关闭服务
stop-yarn.sh
stop-dfs.sh
# 开机后 依次开启服务
start-dfs.sh
start-yarn.sh

或者

shell 复制代码

# 关机之前关闭服务
stop-all.sh
# 开机后开启服务
start-all.sh

shell 复制代码

#jps 检查进程正常后开启胡哦关闭在再做其它操作

浏览器访问web页面

http://localhost:9870