物联网架构之 Hadoop

修改/etc/hosts文件

192.168.107.197 node1

192.168.107.196 node2

192.168.107.195 node3

创建用户并加入组

groupadd hadoop

useradd -g hadoop hduser

passwd hduser

vim /etc/sudoers

hduser ALL=(ALL) ALL

安装JDK

rpm -ivh jdk-8u171-linux-x64.rpm

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_171-amd64

export CLASSPATH= J A V A H O M E / l i b : JAVA_HOME/lib: JAVAHOME/lib:CLASSPATH

export PATH= J A V A H O M E / b i n : JAVA_HOME/bin: JAVAHOME/bin:PATH

source /etc/profile

java -version

配置本机SSH免密码登录

ssh-keygen -t rsa

ssh-copy-id node1

ssh-copy-id node2

ssh-copy-id node3

hadoop完全分布式安装

cd /home/hduser

tar zxf hadoop-2.6.5.tar.gz

mv hadoop-2.6.5 hadoop

hadoop的环境变量

vim /etc/profile

#hadoop

export HADOOP_HOME=/home/hduser/hadoop

export PATH= H A D O O P H O M E / b i n : HADOOP_HOME/bin: HADOOPHOME/bin:PATH

source /etc/profile

配置Hadoop:

vim /home/hduser/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_171-amd64

vim /home/hduser/hadoop/etc/hadoop/yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_171-amd64

vim /home/hduser/hadoop/etc/hadoop/slaves

node2

node3

vim /home/hduser/hadoop/etc/hadoop/core-site.xml

fs.defaultFS

hdfs://node1:9000

hadoop.tmp.dir

file:/home/hduser/hadoop/tmp

vim /home/hduser/hadoop/etc/hadoop/hdfs-site.xml

dfs.namenode.secondary.http-address

node1:50090

dfs.namenode.name.dir

file:/home/hduser/hadoop/dfs/name

dfs.datanode.data.dir

file:/home/hduser/hadoop/dfs/data

dfs.replication

2

dfs.webhdfs.enabled

true

vim /home/hduser/hadoop/etc/hadoop/mapred-site.xml

mapreduce.framework.name

yarn

mapreduce.jobhistory.address

node1:10020

mapreduce.jobhistory.webapp.address

node1:19888

vim /home/hduser/hadoop/etc/hadoop/yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class

org.apache.hadoop.mapred.ShuffleHandler

yarn.resourcemanager.address

node1:8032

yarn.resourcemanager.scheduler.address

node1:8030

yarn.resourcemanager.resource-tracker.address

node1:8035

yarn.resourcemanager.admin.address

node1:8033

yarn.resourcemanager.webapp.address

node1:8088

scp -r /home/hduser/hadoop node2:/home/hduser

scp -r /home/hduser/hadoop node3:/home/hduser

验证安装配置:

cd /home/hduser/hadoop

bin/hdfs namenode -format

sbin/start-dfs.sh

jps

sbin/start-yarn.sh

sbin/start-all.sh

bin/hdfs dfsadmin -report

http://192.168.107.197:50070

sbin/stop-all.sh

mkdir file

cd file

echo "Hello World hi HADOOP" > file1.txt

echo "Hello hadoop hi CHINA" > file2.txt

sbin/start-all

bin/hadoop fs -mkdir /input2

bin/hadoop fs -put file* /input2

bin/hadoop fs -ls /input2

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar wordcount /input2/ /output2/wordcount1

bin/hadoop fs -cat /output2/wordcount1/*

HDFS的相关命令:

hdfs fsck / -files -blocks

sbin/start-balancer.sh

hadoop fs -mkdir /user

hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2

hadoop fs -ls /input2/file1.txt

hadoop fs -ls /input2/

hadoop fs -cat /input2/file1.txt /input2/file2.txt

文件转移

hadoop fs -put /home/hduser/file/file1.txt /input2

hadoop fs -put /home/hduser/file/file1.txt /home/hduser/file/file2.txt /input2

hadoop fs -get /input2/file1.txt $HOME/file.txt

hadoop fs -mv /input2/file1.txt /input2/file2.txt /user/hadoop/dir1

hadoop fs -cp /input2/file1.txt /input2/file2.txt /user/hadoop/dir1

hadoop fs -cp file:///file1.txt file:///file2.txt file:///tmp

hadoop fs -rm /input2/file3.txt

hadoop fs -rmr /input2#现在推荐使用 hadoop fs -rm -r /input2 命令

hadoop fs -test -e /input2/file3.txt

hadoop fs -test -z /input2/file1.txt

相关推荐
roshy1 小时前
x86、arm、rsc-v指令集架构,指令集、OS、应用3者的关系
java·arm开发·架构
科技百宝箱2 小时前
03-AI Agent全栈架构系统化落地指南
人工智能·架构
洛克大航海3 小时前
Ubuntu安装Hbase
大数据·linux·数据库·ubuntu·hbase
趣味编程1113 小时前
物联网系统三层架构解析
物联网·架构
mit6.8243 小时前
[Backstage] 认证请求的流程 | JWT令牌
架构
GIOTTO情4 小时前
媒介宣发的技术革命:Infoseek如何用AI重构企业传播全链路
大数据·人工智能·重构
ApacheSeaTunnel4 小时前
新兴数据湖仓手册·从分层架构到数据湖仓架构(2025):数据仓库分层的概念与设计
大数据·数据仓库·开源·数据湖·dataops·白鲸开源·底层技术
落雪财神意5 小时前
股指10月想法
大数据·人工智能·金融·区块链·期股
柳贯一(逆流河版)5 小时前
ElasticSearch 实战:全文检索与数据聚合分析的完整指南
大数据·elasticsearch·全文检索
白鲸开源5 小时前
最佳实践:基于Apache SeaTunnel从MySQL同步到PostgreSQL
大数据·mysql·postgresql