Hadoop伪分布式安装配置

A、添加hadoop用户

1、添加用户组

root@hadoop00 \~\]# groupadd hadoop #### 2、添加用户并分配用户组 \[root@hadoop00 \~\]# useradd -g hadoop hadoop #### 3、修改hadoop用户密码 \[root@hadoop00 \~\]# passwd hadoop ### B、配置本地YUM源 #### 1、上传系统安装镜像到虚拟机服务器的/root目录 #### 2、新建挂载点目录 \[root@hadoop00 \~\]# mkdir /mnt/centos #### 3、挂载镜像 \[root@hadoop00 \~\]# mount -o loop /root/CentOS-6.8-x86_64-bin-DVD1.iso /mnt/centos #### 4、切换目录 \[root@hadoop00 \~\]# cd /etc/yum.repos.d #### 5、新建备份目录 \[root@hadoop00 \~\]# mkdir bakup #### 6、备份文件，把所有以Cent开头的文件全部备份移动到bakup目录中 \[root@hadoop00 \~\]# mv Cent\* bakup #### 7、新建YUM源文件，添加如下内容 \[root@hadoop00 \~\]# vi local.repo \[local

name=local

baseurl=file:///mnt/centos

enabled=1

gpgcheck=0

8、验证YUM源，出现如下可用包，表示本地YUM源配置成功。

root@hadoop00 \~\]# yum list  ### C、SSH无密码配置 #### 1、查看软件openssh与rsync安装状态 \[root@hadoop00 \~\]# rpm -qa\|grep openssh \[root@hadoop00 \~\]# rpm -qa\|grep rsync #### 2、安装openssh与rsync \[root@hadoop00 \~\]# yum -y install openssh \[root@hadoop00 \~\]# yum install openssh-clients.x86_64 -y \[root@hadoop00 \~\]# yum -y install rsync #### 3、切换到hadoop用户 \[root@hadoop00 \~\]# su - hadoop #### 4、生成SSH密码对 \[root@hadoop00 \~\]# ssh-keygen -t rsa -P '' -f \~/.ssh/id_rsa #### 5、将id_dsa.pub追加到授权的key文件中 \[root@hadoop00 \~\]# cat \~/.ssh/id_rsa.pub \>\> \~/.ssh/authorized_keys #### 6、设置授权key文件权限 \[root@hadoop00 \~\]# chmod 600 \~/.ssh/authorized_keys #### 7、测试ssh连接 \[root@hadoop00 \~\]# ssh hadoop00  无需输入密码，即可完成登录，表示SSH配置成功。 ### D、Hadoop安装与配置 #### 1、将Hadoop安装压缩包hadoop-2.7.3.tar.gz，上传至/root目录 #### 2、将压缩包解压至/usr目录 \[root@hadoop00 \~\]# tar zxvf /root/hadoop-2.7.3.tar.gz -C /usr/ #### 3、修改文件夹名称 \[root@hadoop00 \~\]# mv /usr/hadoop-2.7.3/ /usr/hadoop #### 4、将hadoop文件夹授权给hadoop用户 \[root@hadoop00 \~\]# chown -R hadoop:hadoop /usr/hadoop/ #### 5、设置环境变量 \[root@hadoop00 \~\]# vi /etc/profile （文档末尾追加如下内容） export HADOOP_HOME=/usr/hadoop export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib" #### 6、使环境变量生效 \[root@hadoop00 \~\]# source /etc/profile #### 7、测试环境变量设置 \[root@hadoop00 \~\]# hadoop version  ### E、配置HDFS #### 1、切换至Hadoop用户 \[root@hadoop00 \~\]# su - hadoop #### 2、修改hadoop-env.sh \[hadoop@hadoop00 \~\]$ cd /usr/hadoop/etc/hadoop/ \[hadoop@hadoop00 \~\]$ vi hadoop-env.sh （文档末尾追加如下内容） export JAVA_HOME=/usr/java/jdk1.8.0_162 #### 3、修改core-site.xml \[hadoop@hadoop00 \~\]$ vi core-site.xml (添加如下内容，注意标签对的匹配及唯一性) \ \ \fs.defaultFS\ \hdfs://hadoop00:9000\ \ \ \hadoop.tmp.dir\ \/usr/hadoop/tmp/\ \A base for other temporary directories.\ \ \ #### 4、修改hdfs-site.xml \[hadoop@hadoop00 \~\]$ vi hdfs-site.xml (添加如下内容) \ \ \dfs.replication\ \1\ \ \ #### 5、格式化hdfs \[hadoop@hadoop00 \~\]$ hdfs namenode -format 注：出现Exiting with status 0即为成功 #### 6、启动hdfs \[hadoop@hadoop00 \~\]$ start-dfs.sh (停止命令)# stop-dfs.sh 注：输出如下内容，则启动成功 15/09/21 18:09:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on \[Hadoop.Master

Hadoop.Master: starting namenode, logging to /usr/hadoop/logs/hadoop-hadoop-namenode-Hadoop.Master.out

Hadoop.Master: starting datanode, logging to /usr/hadoop/logs/hadoop-hadoop-datanode-Hadoop.Master.out

Starting secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is b5:96:b2:68:e6:63:1a:3c:7d:08:67:4b:ae:80:e2:e3.

Are you sure you want to continue connecting (yes/no)? yes

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-hadoop-secondarynamenode-Hadoop.Master.out

15/09/21 18:09:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicab

7、查看进程

hadoop@hadoop00 \~\]$ jps 注：输出类似如下内容 1763 NameNode 1881 DataNode 2146 Jps 2040 SecondaryNameNode #### 8、使用web浏览器查看Hadoop运行状态 http://你的服务器ip地址:50070/  #### 9、在HDFS上运行WordCount： ##### 1）创建HDFS用户目录 \[hadoop@hadoop00 \~\]$ hdfs dfs -mkdir /user \[hadoop@hadoop00 \~\]$ hdfs dfs -mkdir /user/hadoop #根据自己的情况调整 ##### 2）复制输入文件（要处理的文件）到HDFS上 \[hadoop@hadoop00 \~\]$ hdfs dfs -put /usr/hadoop/etc/hadoop/ /input ##### 3）查看我们复制到HDFS上的文件 \[hadoop@hadoop00 \~\]$ hdfs dfs -ls /input ##### 4）运行单词检索（grep）程序 \[hadoop@hadoop00 \~\]$ hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep /input output 'dfs\[a-z.\]+' (WordCount \[hadoop@hadoop00 \~\]$ hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input output #说明：output文件夹如已经存在则需要删除或指定其他文件夹。) ##### 5）查看运行结果 \[hadoop@hadoop00 \~\]$ hdfs dfs -cat output/\* #### 10、配置YARN ##### 1）修改mapred-site.xml \[hadoop@hadoop00 \~\]$ cd /usr/hadoop/etc/hadoop/ \[root@hadoop00 \~\]# cp mapred-site.xml.template mapred-site.xml \[hadoop@hadoop00 \~\]$ vi mapred-site.xml (添加如下内容) \ \ \mapreduce.framework.name\ \yarn\ \ \ ##### 2）修改yarn-site.xml \[hadoop@hadoop00 \~\]$ vi yarn-site.xml (添加如下内容) \ \ \yarn.nodemanager.aux-services\ \mapreduce_shuffle\ \ \ ##### 3）启动YARN \[hadoop@hadoop00 \~\]$ start-yarn.sh (停止yarn stop-yarn.sh) ##### 4）查看当前java进程 \[hadoop@hadoop00 \~\]$ jps ##### 5）(输出如下) 4918 ResourceManager 1663 NameNode 1950 SecondaryNameNode 5010 NodeManager 5218 Jps 1759 DataNode ##### 6）运行你的mapReduce程序 配置好如上配置再运行mapReduce程序时即是yarn中运行 ##### 7）使用web查看Yarn运行状态 http://你的服务器ip地址:8088/ ### F、HDFS常用命令参考 #### 1、创建HDFS文件夹 ##### 1）在根目录创建input文件夹 \[hadoop@hadoop00 \~\]$ hdfs dfs -mkdir -p /input ##### 2）在用户目录创建input文件夹 说明：如果不指定"/目录"，则默认在用户目录创建文件夹 \[hadoop@hadoop00 \~\]$ hdfs dfs -mkdir -p input (等同于 hdfs dfs -mkdir -p /user/hadoop/input) #### 2、查看HDFS文件夹 ##### 1）查看HDFS根文件夹 \[hadoop@hadoop00 \~\]$ hdfs dfs -ls / ##### 2）查看HDFS用户目录文件夹 \[hadoop@hadoop00 \~\]$ hdfs dfs -ls ##### 3）查看HDFS用户目录文件夹下input文件夹 \[hadoop@hadoop00 \~\]$ hdfs dfs -ls input (等同与 hdfs dfs -ls /user/hadoop/input) #### 3、复制文件到HDFS \[hadoop@hadoop00 \~\]$ hdfs dfs -put /usr/hadoop/etc/hadoop input #### 4、删除文件夹 \[hadoop@hadoop00 \~\]$ hdfs dfs -rm -r input