linux hadoop-3.3.6 hbase-2.5.7

楠奕2024-09-19 8:45

软件下载

hadoop

https://dlcdn.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz

可以直接下载到本地，也可以直接下载进虚拟机中

如果速度较慢，可以用；另一个

wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz

bash 复制代码 
# wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
--2024-09-18 14:09:21--  https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.15.130, 2402:f000:1:400::2
Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.15.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 730107476 (696M) [application/octet-stream]
Saving to: 'hadoop-3.3.6.tar.gz'

100%[=========================================================================================>] 730,107,476 1.76MB/s   in 6m 6s  

2024-09-18 14:15:27 (1.90 MB/s) - 'hadoop-3.3.6.tar.gz' saved [730107476/730107476]

hadoop-3.3.6

解压至安装目录

bash 复制代码 
tar -xzvf hadoop-3.3.6.tar.gz  -C ../apps/

检查/etc/hosts中将此主机的ip，hostname配置进去

修改配置文件

core-site.xml

XML 复制代码 
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://10.21.10.111:8020</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/data/aidacp/apps/hadoop-3.3.6/tmp</value>
  </property>
  <property>
    <name>hadoop.native.lib</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.http.authentication.simple.anonymous.allowed</name>
    <value>true</value>
  </property>
  <property>  
    <name>fs.hdfs.impl</name>  
    <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>  
    <description>The FileSystem for hdfs: uris.</description>  
  </property>
</configuration>

hdfs-site.xml

在hadoop的目录下，新建nn dn等文件夹

dfs.namenode.http-address不配置端口时，页面访问hadoop的overview访问不到

XML 复制代码 
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
 
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/data/aidacp/apps/hadoop-3.3.6/data/mn</value>
  </property>

  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/data/aidacp/apps/hadoop-3.3.6/data/dn</value>
  </property>

  <property>
　  <name>dfs.namenode.http-address</name>
　　<value>http://kvm-aiswdos-centos76-test-node1:9870</value>
  </property>
</configuration>

hbase-env.sh

若不是默认的端口22，需要配置进具体的端口，否则启动时，ssh登陆连接超时

bash 复制代码 
export HADOOP_SSH_OPTS="-p 22222"
export JAVA_HOME=/data/aidacp/apps/jdk8
export HADOOP_HOME=/data/aidacp/apps/hadoop-3.3.6
export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}

yarn-site.xml

XML 复制代码 
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>kvm-aiswdos-centos76-test-node1</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>0.0.0.0:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.application.classpath</name>
        <value>/data/aidacp/apps/hadoop-3.3.6/etc/hadoop:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/common/lib/*:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/common/*:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/hdfs:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/hdfs/lib/*:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/hdfs/*:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/mapreduce/*:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/yarn:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/yarn/lib/*:/data/aidacp/apps/hadoop-3.3.6/share/hadoop/yarn/*</value>
    </property>
</configuration>

mapred-site.xml

XML 复制代码 
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  
</configuration>

~/.bash_profile

vbscript 复制代码 
export HADOOP_HOME=/data/aidacp/apps/hadoop-3.3.6
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HDFS_NAMENODE_USER="aidacp"
export HDFS_DATANODE_USER="aidacp"
export HDFS_SECONDARYNAMENODE_USER="aidacp"
export YARN_RESOURCEMANAGER_UER="aidacp"
export YARN_NODEMANAGER_USER="aidacp"
PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

./hdfs namenode -format

bash 复制代码 
2024-04-15 16:52:36,695 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2024-04-15 16:52:36,695 INFO util.GSet: VM type       = 64-bit
2024-04-15 16:52:36,695 INFO util.GSet: 0.029999999329447746% max memory 981.5 MB = 301.5 KB
2024-04-15 16:52:36,695 INFO util.GSet: capacity      = 2^15 = 32768 entries
2024-04-15 16:52:36,747 INFO namenode.FSImage: Allocated new BlockPoolId: BP-296281341-10.19.83.151-1713171156732
2024-04-15 16:52:37,148 INFO common.Storage: Storage directory /data/yunwei/apps/hadoop-3.3.6/data/hdfs/nn has been successfully formatted.
2024-04-15 16:52:37,212 INFO namenode.FSImageFormatProtobuf: Saving image file /data/yunwei/apps/hadoop-3.3.6/data/hdfs/nn/current/fsimage.ckpt_0000000000000000000 using no compression
2024-04-15 16:52:37,643 INFO namenode.FSImageFormatProtobuf: Image file /data/yunwei/apps/hadoop-3.3.6/data/hdfs/nn/current/fsimage.ckpt_0000000000000000000 of size 401 bytes saved in 0 seconds .
2024-04-15 16:52:37,702 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2024-04-15 16:52:37,732 INFO namenode.FSNamesystem: Stopping services started for active state
2024-04-15 16:52:37,732 INFO namenode.FSNamesystem: Stopping services started for standby state
2024-04-15 16:52:37,737 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2024-04-15 16:52:37,738 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at host-10-19-83-151/10.19.83.151

./start-all.sh

bash 复制代码 
# ./start-all.sh 
WARNING: Attempting to start all Apache Hadoop daemons as aidacp in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [kvm-aiswdos-centos76-test-node1]
Starting datanodes
Starting secondary namenodes [kvm-aiswdos-centos76-test-node1]
Starting resourcemanager
Starting nodemanagers

./stop-all.sh

bash 复制代码 
# ./stop-all.sh 
WARNING: Stopping all Apache Hadoop daemons as aidacp in 10 seconds.
WARNING: Use CTRL-C to abort.
Stopping namenodes on [kvm-aiswdos-centos76-test-node1]
Stopping datanodes
Stopping secondary namenodes [kvm-aiswdos-centos76-test-node1]
Stopping nodemanagers
Stopping resourcemanager

查看页面

hadoop-cluster

hadoop-overview

相关推荐
watermelonoops5 分钟前
Deepin和Windows传文件（Xftp，WinSCP）
linux·ssh·deepin·winscp·xftp
疯狂飙车的蜗牛1 小时前
从零玩转CanMV-K230（4）-小核Linux驱动开发参考
linux·运维·驱动开发
远游客07133 小时前
centos stream 8下载安装遇到的坑
linux·服务器·centos
马甲是掉不了一点的<.<3 小时前
本地电脑使用命令行上传文件至远程服务器
linux·scp·cmd·远程文件上传
jingyu飞鸟3 小时前
centos-stream9系统安装docker
linux·docker·centos
超爱吃士力架4 小时前
邀请逻辑
java·linux·后端
清平乐的技术专栏5 小时前
Hive SQL 查询所有函数
hive·hadoop·sql
cominglately6 小时前
centos单机部署seata
linux·运维·centos
魏 无羡6 小时前
linux CentOS系统上卸载docker
linux·kubernetes·centos
CircleMouse6 小时前
Centos7, 使用yum工具，出现 Could not resolve host: mirrorlist.centos.org
linux·运维·服务器·centos
热门推荐
01〔AI 绘画〕Stable Diffusion 之 解决绘制多人或面部很小的人物时面部崩坏问题 篇02ARM学习（31）编译器对overlay方式的支持03玄机平台应急响应—webshell查杀04DisplayPort AltMode On Type-c协议_Ver2.0(学习笔记)05校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站06Docker 夺命连环 15 问076个主流的工业3D管道设计软件08基于大语言模型（LLM）的合成数据生成、策展和评估的综述09(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明10树莓派 5 AI 套件（Hailo-8L）使用教程