开源大数据集群部署（十九）Hbase部署

云掣YUNCHE2024-03-30 16:27

作者：櫰木

1 HBASE 安装部署

hbase组件 部署主机
HMaster hd1.dtstack.com,hd2.dtstack.com
HRegionServer hd3.dtstack.com,hd2.dtstack.com,hd1.dtstack.com

2 创建hbase Kerberos主体

在每台机器上进行生成

bash 复制代码 
bash /data/kerberos/getkeytabs.sh /etc/security/keytab/hbase.keytab hbase
bash /data/kerberos/getkeytabs.sh /etc/security/keytab/hbase.keytab HTTP

3 安装

在hd1.dtstack.com主机root权限下执行

  • 解压安装包
bash 复制代码 
[root@hd1.dtstack.com software]# pwd
[root@hd1.dtstack.com software]#tar -zvxf hbase-2.1.0-bin.tar.gz
  • 设置环境变量
bash 复制代码 
[root@hd3.dtstack.com software]# cat  >>/etc/profile<<EOF
export HBASE_HOME=/opt/hbase
export HBASE_CONF_DIR=/opt/hbase/conf
EOF
[root@hd3.dtstack.com software]# source /etc/profile
  • 修改配置文件

注意，在hadoop开启kerberos情况下，需要在core-site.xml添加以下参数

bash 复制代码 
cat /opt/hadoop/etc/hadoop/core-site.xml
    <property>
      <name>hadoop.rpc.protection</name>
      <value>authentication,privacy</value>
</property>
在hdfs-site.xml中修改和添加以下参数
cat /opt/hadoop/etc/hadoop/core-site.xml
   <property>
        <name>dfs.data.transfer.protection</name>
        <value>authentication,privacy</value>
    </property>
    <property>
      <name>dfs.encrypt.data.transfer.cipher.suites</name>
      <value>AES/CTR/NoPadding</value>
    </property>

如果你的Hadoop已经有HA，或者你已经有一个独立的ZooKeeper集群，那么你就需要在hbase-env.sh中把HBase自带的ZooKeeper关掉以防止端口冲突

bash 复制代码 
[root@hd1.dtstack.com conf]# cat >hbase-env.sh<<EOF
HBASE_ROOT_LOGGER=INFO,DRFA
export JAVA_HOME=/opt/java
export HBASE_MANAGES_ZK=false
#export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
export HBASE_OPTS="$HBASE_OPTS -Djava.security.auth.login.config=/opt/hbase/conf/zk-jaas-client.conf -Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=zookeeper -Dzookeeper.sasl.clientconfig=Client"
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Djava.security.auth.login.config=/opt/hbase/conf/zk-jaas.conf  -Djavax.security.auth.useSubjectCredsOnly=false -javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.3.1.jar=9530:/opt/prometheus/hbase_master.yml"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Djava.security.auth.login.config=/opt/hbase/conf/zk-jaas.conf  -Djavax.security.auth.useSubjectCredsOnly=false -javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.12.0.jar=9531:/opt/prometheus/hbase_regionserver.yml"
EOF
cat >zk-jaas-client.conf<<EOF
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
};
EOF
cat >zk-jaas.conf<<EOF
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytab/hbase.keytab"
principal="hbase/hd1.dtstack.com@DTSTACK.COM";
};
com.sun.security.jgss.krb5.initiate {
com.sun.security.auth.module.Krb5LoginModule required
renewTGT=false
doNotPrompt=true
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytab/hbase.keytab"
principal ="hbase/hd1.dtstack.com@DTSTACK.COM";
};
EOF

说明：

  • 启动参数值可根据具体情况调整

修改hbase-site.xml,内容如下：

c 复制代码 
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>
  <!--
    The following properties are set for running HBase as a single process on a
    developer workstation. With this configuration, HBase is running in
    "stand-alone" mode and without a distributed file system. In this mode, and
    without further configuration, HBase and ZooKeeper data are stored on the
    local filesystem, in a path under the value configured for `hbase.tmp.dir`.
    This value is overridden from its default value of `/tmp` because many
    systems clean `/tmp` on a regular basis. Instead, it points to a path within
    this HBase installation directory.
 
    Running against the `LocalFileSystem`, as opposed to a distributed
    filesystem, runs the risk of data integrity issues and data loss. Normally
    HBase will refuse to run in such an environment. Setting
    `hbase.unsafe.stream.capability.enforce` to `false` overrides this behavior,
    permitting operation. This configuration is for the developer workstation
    only and __should not be used in production!__
 
    See also https://hbase.apache.org/book.html#standalone_dist
  -->
  <property>
    <name>hbase.rootdir</name>
    <value>/hbase/</value>
  </property>
  <!--指定hbase集群为分布式集群-->
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>hd1.dtstack.com,hd2.dtstack.com,hd3.dtstack.com</value>
  </property>
  <property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase1</value>
  </property>
  <property>
    <name>hbase.master.info.port</name>
    <value>16010</value>
  </property>
  <property>
    <name>hbase.master.maxclockskew</name>
    <value>150000</value>
  </property>
  <property>
    <name>hbase.rpc.timeout</name>
    <value>300000</value>
  </property>
  <property>
    <name>hbase.master.logcleaner.ttl</name>
    <value>600000</value>
  </property>
  <property>
    <name>hbase.regionserver.handler.count</name>
    <value>25</value>
  </property>
  <property>
    <name>hbase.regionserver.optionalcacheflushinterval</name>
    <value>3600000</value>
  </property>
  <!--Client configurations-->
  <property>
    <name>hbase.client.write.buffer</name>
    <value>2097152</value>
  </property>
  <property>
    <name>hbase.client.pause</name>
    <value>100</value>
  </property>
  <property>
    <name>hbase.client.retries.number</name>
    <value>20</value>
  </property>
  <property>
    <name>hbase.client.keyvalue.maxsize</name>
    <value>10485760</value>
  </property>
  <property>
    <name>hbase.server.keyvalue.maxsize</name>
    <value>10485760</value>
  </property>
  <property>
    <name>hbase.client.perserver.requests.threshold</name>
    <value>2147483647</value>
  </property>
  <!--其他配置-->
  <property>
    <name>hfile.format.version</name>
    <value>3</value>
  </property>
  <property>
    <name>hbase.master.balancer.maxRitPercent</name>
    <value>1</value>
  </property>
  <property>
    <name>hbase.hregion.memstore.flush.size</name>
    <value>134217728</value>
  </property>
  <property>
    <name>hbase.regionserver.offheap.global.memstore.size</name>
    <value>0</value>
  </property>
  <property>
    <name>hbase.hregion.max.filesize</name>
    <value>10737418240</value>
  </property>
  <property>
    <name>hbase.hstore.compactionThreshold</name>
    <value>3</value>
  </property>
  <property>
    <name>hbase.hstore.compaction.kv.max</name>
    <value>10</value>
  </property>
  <property>
    <name>hfile.block.cache.size</name>
    <value>0.4</value>
  </property>
  <property>
    <name>io.storefile.bloom.block.size</name>
    <value>131072</value>
  </property>
  <property>
    <name>hbase.rpc.timeout</name>
    <value>60000</value>
  </property>
  <property>
    <name>hbase.client.operation.timeout</name>
    <value>120000</value>
  </property>
  <property>
    <name>hbase.table.max.rowsize</name>
    <value>1073741824</value>
  </property>
  <!-- 开启kerberos配置  -->
  <!--hbase security-->
  <property>
    <name>hbase.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hbase.regionserver.kerberos.principal</name>
    <value>hbase/_HOST@DTSTACK.COM</value>
  </property>
  <property>
    <name>hbase.regionserver.keytab.file</name>
    <value>/etc/security/keytab/hbase.keytab</value>
  </property>
  <property>
    <name>hbase.master.kerberos.principal</name>
    <value>hbase/_HOST@DTSTACK.COM</value>
  </property>
  <property>
    <name>hbase.master.keytab.file</name>
    <value>/etc/security/keytab/hbase.keytab</value>
  </property>
  <property>
    <name>hbase.rpc.protection</name>
    <value>authentication</value>
  </property>
</configuration>

注意：

  • zk-jaas.conf因为连接zk配置

  • 添加环境变量

bash 复制代码 
export HBASE_HOME=/opt/hbase
export HBASE_CONF_DIR=/opt/hbase/conf
  • 修改regionserver，添加具体节点
bash 复制代码 
[root@hd1.dtstack.com conf]# cat >regionservers<<EOF
hd3.dtstack.com
hd1.dtstack.com
hd2.dtstack.com
EOF
  • 在conf目录下创建backup-masters
bash 复制代码 
[root@hd1.dtstack.com conf]# cat >backup-masters<<EOF
hd2.dtstack.com
EOF
  • 将配置好的文件发送到其他机器
bash 复制代码 
[root@hd1.dtstack.com software]# scp conf/hbase-site.xml hd2:/opt/hbase/conf/
scp conf/hbase-site.xml hd3:/opt/hbase/conf/
[root@hd1.dtstack.com software]#scp conf/hbase-env.sh hd2:/opt/hbase/conf/
scp conf/hbase-env.sh hd3:/opt/hbase/conf/

4 hbase启动

bash 复制代码 
[hbase@hd1.dtstack.com ~]$ sh start-habse.sh

默认日志路径为$HBASE_HOME/logs

如果启动过程中遇到这个错误

使用

bash 复制代码 
cd $HBASE_HOME
cp lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar lib/

5 hbase页面

访问地址：http://hd1.dtstack.com:16010/master-status

6 hbase验证

进行kerberos验证

kinit hbase

使用hbase shell

bash 复制代码 
cd $HBASE_HOME

./bin/hbase shell

建表

bash 复制代码 
create 'user','base_info'

put数据

bash 复制代码 
put 'user', 'rowkey_10', 'base_info:birthday', '2014-07-10'

scan数据

bash 复制代码 
scan 'user'

更多技术信息请查看云掣官网https://yunche.pro/?t=yrgw

相关推荐
EasyGBS26 分钟前
国标GB28181公网直播EasyGBS国标GB28181软件管理解决方案
大数据·网络·音视频·媒体·视频监控·gb28181
2403_8757368732 分钟前
道品科技的水肥一体化智能灌溉：开启现代农业的创新征程
大数据·人工智能·1024程序员节
河南查新信息技术研究院32 分钟前
科技查新在医药健康领域的应用
大数据·科技·全文检索
青云交1 小时前
大数据新视界 -- 大数据大厂之 Impala 性能优化：应对海量复杂数据的挑战（上）（7/30）
大数据·性能优化·impala·数据分区·查询优化·海量复杂数据·经典案例
天冬忘忧1 小时前
Spark 程序开发与提交：本地与集群模式全解析
大数据·分布式·spark
NETFARMER运营坛1 小时前
如何优化 B2B 转化率？这些步骤你不可不知
大数据·安全·阿里云·ai·ai写作
易境通代购商城系统、集运SAAS系统1 小时前
集运企业营销获客策略：创新与传统并重，打造品牌忠诚度
大数据
LNTON羚通5 小时前
CPU算法分析LiteAIServer视频智能分析平台视频智能分析：抖动、过亮与过暗检测技术
大数据·目标检测·音视频·视频监控
vvw&5 小时前
如何使用Netdata部署高性能的服务器监控平台
linux·运维·服务器·docker·开源·服务器监控·netdata
热门推荐
01【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总02(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明03组基轨迹建模 GBTM的介绍与实现（Stata 或 R）04全面解析：构建基于深度学习的安全帽检测系统（UI界面+YOLO代码+数据集）05红米手机使用google play06【经验分享】Ubuntu22.04安装微信(linux官方版)07【AIGC】重塑未来的科技巨轮08【Java】Ruoyi-Vue-Plus 整合 文件分片上传至 minio、阿里云、七牛云等OSS-【后端篇】09RAG 实践- Ollama+RagFlow 部署本地知识库10Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO