物联网架构之 Hadoop

修改/etc/hosts文件

192.168.107.197 node1

192.168.107.196 node2

192.168.107.195 node3

创建用户并加入组

groupadd hadoop

useradd -g hadoop hduser

passwd hduser

vim /etc/sudoers

hduser ALL=(ALL) ALL

安装JDK

rpm -ivh jdk-8u171-linux-x64.rpm

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_171-amd64

export CLASSPATH= J A V A H O M E / l i b : JAVA_HOME/lib: JAVAHOME/lib:CLASSPATH

export PATH= J A V A H O M E / b i n : JAVA_HOME/bin: JAVAHOME/bin:PATH

source /etc/profile

java -version

配置本机SSH免密码登录

ssh-keygen -t rsa

ssh-copy-id node1

ssh-copy-id node2

ssh-copy-id node3

hadoop完全分布式安装

cd /home/hduser

tar zxf hadoop-2.6.5.tar.gz

mv hadoop-2.6.5 hadoop

hadoop的环境变量

vim /etc/profile

#hadoop

export HADOOP_HOME=/home/hduser/hadoop

export PATH= H A D O O P H O M E / b i n : HADOOP_HOME/bin: HADOOPHOME/bin:PATH

source /etc/profile

配置Hadoop:

vim /home/hduser/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_171-amd64

vim /home/hduser/hadoop/etc/hadoop/yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_171-amd64

vim /home/hduser/hadoop/etc/hadoop/slaves

node2

node3

vim /home/hduser/hadoop/etc/hadoop/core-site.xml

fs.defaultFS

hdfs://node1:9000

hadoop.tmp.dir

file:/home/hduser/hadoop/tmp

vim /home/hduser/hadoop/etc/hadoop/hdfs-site.xml

dfs.namenode.secondary.http-address

node1:50090

dfs.namenode.name.dir

file:/home/hduser/hadoop/dfs/name

dfs.datanode.data.dir

file:/home/hduser/hadoop/dfs/data

dfs.replication

2

dfs.webhdfs.enabled

true

vim /home/hduser/hadoop/etc/hadoop/mapred-site.xml

mapreduce.framework.name

yarn

mapreduce.jobhistory.address

node1:10020

mapreduce.jobhistory.webapp.address

node1:19888

vim /home/hduser/hadoop/etc/hadoop/yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class

org.apache.hadoop.mapred.ShuffleHandler

yarn.resourcemanager.address

node1:8032

yarn.resourcemanager.scheduler.address

node1:8030

yarn.resourcemanager.resource-tracker.address

node1:8035

yarn.resourcemanager.admin.address

node1:8033

yarn.resourcemanager.webapp.address

node1:8088

scp -r /home/hduser/hadoop node2:/home/hduser

scp -r /home/hduser/hadoop node3:/home/hduser

验证安装配置:

cd /home/hduser/hadoop

bin/hdfs namenode -format

sbin/start-dfs.sh

jps

sbin/start-yarn.sh

sbin/start-all.sh

bin/hdfs dfsadmin -report

http://192.168.107.197:50070

sbin/stop-all.sh

mkdir file

cd file

echo "Hello World hi HADOOP" > file1.txt

echo "Hello hadoop hi CHINA" > file2.txt

sbin/start-all

bin/hadoop fs -mkdir /input2

bin/hadoop fs -put file* /input2

bin/hadoop fs -ls /input2

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar wordcount /input2/ /output2/wordcount1

bin/hadoop fs -cat /output2/wordcount1/*

HDFS的相关命令:

hdfs fsck / -files -blocks

sbin/start-balancer.sh

hadoop fs -mkdir /user

hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2

hadoop fs -ls /input2/file1.txt

hadoop fs -ls /input2/

hadoop fs -cat /input2/file1.txt /input2/file2.txt

文件转移

hadoop fs -put /home/hduser/file/file1.txt /input2

hadoop fs -put /home/hduser/file/file1.txt /home/hduser/file/file2.txt /input2

hadoop fs -get /input2/file1.txt $HOME/file.txt

hadoop fs -mv /input2/file1.txt /input2/file2.txt /user/hadoop/dir1

hadoop fs -cp /input2/file1.txt /input2/file2.txt /user/hadoop/dir1

hadoop fs -cp file:///file1.txt file:///file2.txt file:///tmp

hadoop fs -rm /input2/file3.txt

hadoop fs -rmr /input2#现在推荐使用 hadoop fs -rm -r /input2 命令

hadoop fs -test -e /input2/file3.txt

hadoop fs -test -z /input2/file1.txt

相关推荐
大大大大晴天12 小时前
每天认识一个组件:VictoriaLogs
大数据
谙弆悕博士15 小时前
多智能体架构详解:市场拍卖式架构(Market-based / 竞价协商)
开发语言·架构·去中心化·多智能体·智能体·拍卖架构
学长毕设15 小时前
基于SpringBoot 博物馆文物租赁及修复管理系统-附源码
大数据·数据库·spring boot
江屿风15 小时前
AI Agent架构全解析:从规划到记忆的智能跃迁
人工智能·笔记·架构·agent·c/s·mcp工作流
一条Go17 小时前
状态页与 SLA:对外承诺是怎么算出来的
后端·架构·github
EatFan17 小时前
AI Agent 工程化元年:从单体智能到 Agent 集群的架构演进与落地路径
人工智能·架构·ai agent·autogen·crewai·agent集群·记忆工程
泯泷17 小时前
别再“裸聊”大模型了!你家的 AI 为什么总胡说八道?根子在 RAG
人工智能·算法·架构
玖米GEO运营王总监17 小时前
成都AI数字人视频怎么批量生产?流程与合规标注
大数据·人工智能·音视频
龙腾AI白云19 小时前
数字孪生与计算机视觉:让机器看懂现场
大数据·数据库·机器学习·知识图谱·pygame