目录
[1、安装 jdk8](#1、安装 jdk8)
[(1)选择 jdk 版本](#(1)选择 jdk 版本)
[(2)下载 jdk 并解压](#(2)下载 jdk 并解压)
[(1)添加 hadoop 用户,配置免密登录](#(1)添加 hadoop 用户,配置免密登录)
[(2)下载 hadoop-3.4.0.tar.gz](#(2)下载 hadoop-3.4.0.tar.gz)
[(5)修改文件权限,并且换到 hadoop 用户](#(5)修改文件权限,并且换到 hadoop 用户)
[(7)验证Hadoop DFS](#(7)验证Hadoop DFS)
[(8)启动 dfs](#(8)启动 dfs)
[(11)启动 hdfs 告警:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform](#(11)启动 hdfs 告警:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform)
[3、安装 hbase](#3、安装 hbase)
[(1)下载 hbase-2.5.8-hadoop3-bin.tar.gz](#(1)下载 hbase-2.5.8-hadoop3-bin.tar.gz)
[(4)添加 zookeeper 目录](#(4)添加 zookeeper 目录)
[(5)修改文件权限,并且换到 hadoop 用户](#(5)修改文件权限,并且换到 hadoop 用户)
[(6)启动 hbase](#(6)启动 hbase)
[(8)命令行验证 hbase shell](#(8)命令行验证 hbase shell)
[(9)关闭 hbase](#(9)关闭 hbase)
[(10)hbase shell 报错:ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet。](#(10)hbase shell 报错:ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet。)
[4、gohbase 客户端外部连接测试](#4、gohbase 客户端外部连接测试)
[(1)首先使用 hbase shell 创建一个 emp 表,含有一个列组 a。](#(1)首先使用 hbase shell 创建一个 emp 表,含有一个列组 a。)
[(2)然后利用 gohbase 客户端向 hbase 插入一条测试数据](#(2)然后利用 gohbase 客户端向 hbase 插入一条测试数据)
[(3)连接报错:failed to dial RegionServer: dial tcp [::1]:16020: connectex: No connection could be made because the target machine actively refused it.](#(3)连接报错:failed to dial RegionServer: dial tcp [::1]:16020: connectex: No connection could be made because the target machine actively refused it.)
1、安装 jdk8
(1)选择 jdk 版本
(2)下载 jdk 并解压
bash
cd /usr/local
wget https://repo.huaweicloud.com/java/jdk/8u172-b11/jdk-8u172-linux-x64.tar.gz
tar zxf jdk-8u172-linux-x64.tar.gz
(3)配置环境变量
bash
# vim /etc/profile
export JAVA_HOME=/usr/local/jdk1.8.0_172
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
# source /etc/profile
#参考
https://blog.csdn.net/codedz/article/details/124044974
2、安装hadoop
(1)添加 hadoop 用户,配置免密登录
bash
useradd hadoop
#passwd hadoop
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
(2)下载 hadoop-3.4.0.tar.gz
bash
cd /usr/local
wget https://dlcdn.apache.org/hadoop/core/stable/hadoop-3.4.0.tar.gz
tar zxf hadoop-3.4.0.tar.gz
(3)配置环境变量
bash
# vim /etc/profile
export HADOOP_HOME=/usr/local/hadoop-3.4.0
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
# source /etc/profile
(4)修改配置文件
bash
cd /usr/local/hadoop-3.4.0/etc/hadoop/
vim hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_71
vim core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
vim hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/namenode </value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/datanode </value>
</property>
</configuration>
vim yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
(5)修改文件权限,并且换到 hadoop 用户
bash
chown -R hadoop:hadoop hadoop-3.4.0
su hadoop
(7)验证Hadoop DFS
bash
hdfs namenode -format
(8)启动 dfs
cpp
./sbin/start-dfs.sh
# 查看进程
jps
71168 SecondaryNameNode
71681 NameNode
71834 DataNode
72289 ResourceManager
72396 NodeManager
(9)验证yarn脚本
bash
./sbin/start-yarn.sh
(10)浏览器访问
bash
http://localhost:9870
http://localhost:8088
(11)启动 hdfs 告警:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform
不改也不影响,添加如下环境变量即可
bash
# vim /etc/profile
export JAVA_LIBRARY_PATH=/usr/local/hbase-2.5.8-hadoop3/lib/native
# source /etc/profile
参考
https://blog.csdn.net/weixin_45678985/article/details/120497297
3、安装 hbase
(1)下载 hbase-2.5.8-hadoop3-bin.tar.gz
bash
cd /usr/local
wget https://dlcdn.apache.org/hbase/2.5.8/hbase-2.5.8-hadoop3-bin.tar.gz
tar zxf hbase-2.5.8-hadoop3-bin.tar.gz
(2)配置环境变量
bash
export HBASE_HOME=/usr/local/hbase-2.5.8-hadoop3
export PATH=$PATH:$HBASE_HOME/bin
(3)修改配置文件
bash
cd /usr/local/hbase-2.5.8-hadoop3/conf
vim hbase-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_172
vim hbase-site.xml
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/zookeeper</value>
</property>
(4)添加 zookeeper 目录
bash
mkdir -p /home/zookeeper
chown -R hadoop:hadoop /home/zookeeper
(5)修改文件权限,并且换到 hadoop 用户
bash
chown -R hadoop:hadoop hbase-2.5.8-hadoop3
su hadoop
(6)启动 hbase
bash
./bin/start-hbase.sh
# 查看进程
jps
71168 SecondaryNameNode
71681 NameNode
71834 DataNode
72289 ResourceManager
72396 NodeManager
74089 HQuorumPeer
74186 HMaster
74271 HRegionServer
(7)浏览器访问
bash
http://localhost:16010
(8)命令行验证 hbase shell
bash
./bin/hbase shell
> list
(9)关闭 hbase
bash
./bin/stop-hbase.sh
# kill -9 ${HRegionServer-PID} ${HMaster-PID} ${HQuorumPeer-PID}
(10)hbase shell 报错:ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet。
关闭 hbase,修改配置文件,然后重启
bash
vim conf/hbase-env.sh
export HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP=true
参考
https://hadoopdoc.com/hbase/hbase-install
4、gohbase 客户端外部连接测试
(1)首先使用 hbase shell 创建一个 emp 表,含有一个列组 a。
Go
./bin/hbase shell
# 创建表
> create 'emp', 'a'
# 查看表列表
> list
# 查看表结构
> describe 'emp'
(2)然后利用 gohbase 客户端向 hbase 插入一条测试数据
put 'emp','2','a:age','20'
向 emp 表插入一条数据,rowkey='2',column_family='a',column='age',value='20'
Go
package main
import (
"context"
"github.com/tsuna/gohbase"
"github.com/tsuna/gohbase/hrpc"
"log"
)
func main() {
client := gohbase.NewClient("xxx.xxx.xxx.xxx:2181")
// Values maps a ColumnFamily -> Qualifiers -> Values.
values := map[string]map[string][]byte{"a": map[string][]byte{"age": []byte{20}}}
putRequest, err := hrpc.NewPutStr(context.Background(), "emp", "2", values)
if err != nil {
log.Fatalln("hrpc.NewPutStr error:", err)
return
}
rsp, err := client.Put(putRequest)
if err != nil {
log.Fatalln("client.Put error:", err)
return
}
rsp.String()
}
(3)连接报错:failed to dial RegionServer: dial tcp [::1]:16020: connectex: No connection could be made because the target machine actively refused it.
time="2024-06-04T10:51:31+08:00" level=info msg="added new region client" client="RegionClient{Addr: localhost:16020}"
time="2024-06-04T10:51:33+08:00" level=error msg="error occured, closing region client" client="RegionClient{Addr: localhost:16020}" err="failed to dial RegionServer: dial tcp [::1]:16020: connectex: No connection could be made because the target machine actively refused it."
time="2024-06-04T10:51:33+08:00" level=info msg="removed region client" client="RegionClient{Addr: localhost:16020}"
具体连接哪个 regionserver 是由 zk 返回的,默认返回的 regionserver 是主机名,由于没有配置主机名,返回的 regionserver 地址是一个本机地址 localhost:16020,导致外部无法连接。并且通过 netstat -nltp 查看端口监听状态可知,16020 监听在 127.0.0.1 网卡上。这两项都导致了在外部无法连接 16020 端口。
在服务器端:
- 设置服务器主机名为 node01;
- 查看主机对外ip,并修改 hosts 文件,将主机名绑定主机对外IP(注意不要绑定127.0.0.1,要绑定外部可以访问的ip);
- 修改 regionservers 配置文件,以主机名代替原来的 localhost。
bash
# 修改主机名为 node01
hostnamectl set-hostname node01
# 添加主机名到主机对外ip的映射
ip addr
vim /etc/hosts
xxx.xxx.xxx.xxx node01
# 修改 regionservers 配置文件
vim conf/regionservers
node01
在客户端,同样将 hbase 主机名绑定到 hbase 主机IP。
bash
xxx.xxx.xxx.xxx node01
参考