安装部署CDH6.3.2

CodeDevMaster2023-12-07 17:47

Cloudera Manager

概述

CDH(Cloudera Distribution Including Apache Hadoop) 是一个基于 Apache Hadoop 的开源软件发行版。它是由Cloudera公司开发和维护的，为企业和组织提供了易于使用和管理的大数据处理解决方案。
CDH包含了一系列与Hadoop生态系统相关的开源软件，如Apache Hadoop、Apache Spark、Apache HBase、Apache Hive、Apache Impala、Apache Flume、Apache Sqoop等等。这些软件用于存储、处理和分析大规模数据集。
Cloudera为CDH提供了增强的集成、管理和安全功能，使得企业可以更轻松地部署和操作Hadoop群集。CDH还提供了一些工具和界面，用于监视和优化群集性能，以及支持各种数据处理工作负载。

主要功能

集群部署和配置：

Cloudera Manager简化了CDH集群的部署和配置过程。它提供了预定义的部署模板、角色分配和配置向导，使管理员可以快速配置各个组件和服务。

监控和警报：

Cloudera Manager监控整个大数据集群的运行状况和性能指标。它收集和展示有关各个节点、服务和作业的关键指标，并提供实时和历史性能图表。此外，它还能生成警报以提醒管理员关于潜在的问题或重要事件。

故障排除和诊断：

Cloudera Manager提供了故障排除和诊断工具，帮助管理员定位和解决集群中的问题。它提供了错误和异常日志的集中查看、问题诊断工具和故障排除向导，以加快故障修复的过程。

自动化任务和操作：

Cloudera Manager使管理员能够执行自动化任务和操作，如软件安装、配置更改、备份和恢复等。它提供了一致的集群操作和管理界面，使管理员可以在整个集群上进行一致的操作。

架构

Cloudera Manager的架构实现了集中式管理和监控的能力，管理员可以通过Web界面和API对整个集群进行配置、部署、监控和管理。

Cloudera Manager Server：

Cloudera Manager的核心组件，负责管理整个集群和与用户进行交互。它提供了一个Web界面和API，用于集群配置、部署、监控和管理。

Agents：

在每个集群节点上安装并在Cloudera Manager Server指导下运行的代理程序。它们与Cloudera Manager Server通信，收集节点的运行状况和性能指标，并执行命令和任务。

数据库：

使用数据库存储集群和配置元数据。可以选择使用内置的Derby数据库或外部数据库，如MySQL或PostgreSQL。

Cloudera Management Services：

执行各种管理任务的服务，包括Activity Monitor、Alert Publisher、Event Server、Reports Manager、Service Monitor和Host Monitor。

环境准备

节点情况

准备5个节点

vim /etc/hosts

bash 复制代码 
172.29.234.1 node01
172.29.234.2 node02
172.29.234.3 node03
172.29.234.4 node04
172.29.234.5 node05

SSH免密配置

配置各个节点进行免密登录

参考：SSH免密配置

安装JDK

在各个节点安装JDK

参考：安装JDK

安装MySQL

选取一台节点安装MySQL

参考：安装MySQL

创建数据库

bash 复制代码 
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CM安装部署

上传cdh安装文件

创建/opt/cloudera-manager目录，存放cdh安装文件

bash 复制代码 
tar -zxvf cm6.3.1-redhat7.tar.gz

cd cm6.3.1/RPMS/x86_64/

mkdir /opt/cloudera-manager

mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager
mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/

 cd /opt/cloudera-manager/

分发安装文件到各个节点

bash 复制代码 
[root@node01 ~]# ./sync.sh /opt/cloudera-manager/

安装cloudera-manager-daemons

各个节点执行安装cloudera-manager-daemons

bash 复制代码 
[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
--------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################      
--------- node02 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################      
--------- node03 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################     
--------- node04 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ######################################## 
--------- node05 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################

安装完毕后会多出一个/opt/cloudera目录

bash 复制代码 
[root@node01 cloudera-manager]# ls /opt/
cloudera

安装cloudera-manager-agent

各个节点安装cloudera-manager-agent

安装cloudera-manager-agent时，需要安装相关依赖，否则将出现如下异常：

bash 复制代码 
[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
--------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
error: Failed dependencies:
        bind-utils is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        libxslt is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        cyrus-sasl-plain is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        cyrus-sasl-gssapi is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        /lib/lsb/init-functions is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        httpd is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        mod_ssl is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        openssl-devel is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        python-psycopg2 is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        MySQL-python is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        libpq.so.5()(64bit) is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64

解决方案则是安装如下依赖：

bash 复制代码 
yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt

安装依赖时，发现在已安装了MySQL的节点出现如下异常：

bash 复制代码 
--> Running transaction check
---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed
---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed
Removing mariadb-libs.x86_64 1:5.5.68-1.el7 - u due to obsoletes from installed mysql-community-libs-5.7.43-1.el7.x86_64
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.68-1.el7 will be installed
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: MySQL-python-1.2.5-1.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: MySQL-python-1.2.5-1.el7.x86_64
--> Finished Dependency Resolution
Error: Package: MySQL-python-1.2.5-1.el7.x86_64 (base)
           Requires: libmysqlclient.so.18()(64bit)
Error: Package: MySQL-python-1.2.5-1.el7.x86_64 (base)
           Requires: libmysqlclient.so.18(libmysqlclient_18)(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

原因：

在mysql5.7版本中，缺失libmysqlclient.so.18()(64bit)这个依赖，因为安装MySQL时没有安装

解决方案:

安装mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm

bash 复制代码 
[root@node01 ~]# rpm -ivh mysql-lib/mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm
warning: mysql-lib/mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 3a79bd29: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-libs-compat-5.7.4################################# [100%]

使用脚本批量安装相关依赖：

bash 复制代码 
[root@node01 ~]# ./call.sh  yum -y install bind-utils libxslt psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python

使用脚本批量安装agent：

bash 复制代码 
[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm                                                                             --------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node02 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node03 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node04 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node05 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.

修改vim /etc/cloudera-scm-agent/config.ini文件，配置Agent，告知Server节点

bash 复制代码 
[General]
# Hostname of the CM server.
#server_host=localhost
server_host=node01

# Port that the CM server is listening on.
server_port=7182

分发配置到各个节点

bash 复制代码 
[root@node01 ~]# ./sync.sh /etc/cloudera-scm-agent/config.ini

安装cloudera-manager-server

只在node01节点安装cloudera-manager-server

bash 复制代码 
[root@node01 ~]# rpm -ivh /opt/cloudera-manager/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
warning: /opt/cloudera-manager/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:cloudera-manager-server-6.3.1-146################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.

上传CDH包到parcel-repo

bash 复制代码 
[root@node01 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* /opt/cloudera/parcel-repo

[root@node01 parcel-repo]# mv manifest.json /opt/cloudera/parcel-repo/

[root@node01 parcel-repo]# cd /opt/cloudera/parcel-repo

[root@node01 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

[root@node01 parcel-repo]# ls
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel  CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha  manifest.json

修改server的db.properties

vim /etc/cloudera-scm-server/db.properties

bash 复制代码 
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#

# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
#com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.host=node01:3306

# The database name
#com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.name=scm

# The database user
#com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.user=root

# The database user's password
#com.cloudera.cmf.db.password=
com.cloudera.cmf.db.password=123456

# The db setup type
# After fresh install it is set to INIT
# and will be changed post config.
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL

# com.cloudera.cmf.db.setupType=INIT
com.cloudera.cmf.db.setupType=EXTERNAL

导入数据

cm_prepare_database.sh是Cloudera Manager的脚本，用于准备数据库以供Cloudera Manage 使用。

具体执行命令如下：

bash 复制代码 
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
参数mysql：指示使用 MySQL 数据库

参数scm：是数据库名称

参数root：是数据库用户名

当导入数据时，出现如下错误：

bash 复制代码 
[root@node01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
Enter SCM password:
JAVA_HOME=/usr/local/jdk8
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/local/jdk8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
[2023-08-15 20:08:11,878] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.loadJDBCDriver(DbCommandExecutor.java) - Unable to find JDBC driver for database type: MySQL
[2023-08-15 20:08:11,881]ERROR     3[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor$DbConnectionTestException.logError(DbCommandExecutor.java) - JDBC Driver com.mysql.jdbc.Driver not found.
[2023-08-15 20:08:11,881]ERROR     3[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor$DbConnectionTestException.logError(DbCommandExecutor.java) - Exiting with exit code 3
--> Error 3, giving up (use --force if you wish to ignore the error)

解决方案：

下载mysql-connector-java-5.1.49.jar，将其复制到/usr/share/java路径下。
注意：改名为上述异常提到的名称：mysql-connector-java.jar

bash 复制代码 
mv mysql-connector-java-5.1.49.jar  mysql-connector-java.jar

mkdir /usr/share/java

mv mysql-connector-java.jar /usr/share/java/

再次执行，操作成功如下：

bash 复制代码 
[root@node01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
Enter SCM password:
JAVA_HOME=/usr/local/jdk8
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/local/jdk8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
Tue Aug 15 20:18:40 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[2023-08-15 20:18:40,951] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database.
All done, your SCM database is configured correctly!

启动server服务

启动server服务

bash 复制代码 
systemctl start cloudera-scm-server

停止server服务

bash 复制代码 
systemctl stop cloudera-scm-server

查看server服务状态

bash 复制代码 
[root@node01 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2023-08-15 20:46:56 CST; 1min 47s ago
  Process: 328 ExecStart=/opt/cloudera/cm/bin/cm-server (code=exited, status=1/FAILURE)
  Process: 325 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 328 (code=exited, status=1/FAILURE)

Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service: main process exited, code=exited, status=1/FAILURE
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service holdoff time over, scheduling restart.
Aug 15 20:46:56 node01 systemd[1]: Stopped Cloudera CM Server Service.
Aug 15 20:46:56 node01 systemd[1]: start request repeated too quickly for cloudera-scm-server.service
Aug 15 20:46:56 node01 systemd[1]: Failed to start Cloudera CM Server Service.
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.

启动失败，执行journalctl -xe命令查看具体异常信息

bash 复制代码 
[root@node01 ~]# journalctl -xe

Aug 15 20:46:56 node01 cm-server[328]: +======================================================================+
Aug 15 20:46:56 node01 cm-server[328]: |      Error: Unable to find a compatible version of Java on this host,|
Aug 15 20:46:56 node01 cm-server[328]: |             either because JAVA_HOME has not been set or because a   |
Aug 15 20:46:56 node01 cm-server[328]: |             compatible version of Java is not installed.             |
Aug 15 20:46:56 node01 cm-server[328]: +----------------------------------------------------------------------+
Aug 15 20:46:56 node01 cm-server[328]: | Please install either:                                               |
Aug 15 20:46:56 node01 cm-server[328]: | - a supported version of the Oracle JDK from the Oracle Java web     |
Aug 15 20:46:56 node01 cm-server[328]: |   site:                                                              |
Aug 15 20:46:56 node01 cm-server[328]: |   > http://www.oracle.com/technetwork/java/javase/index.html <       |
Aug 15 20:46:56 node01 cm-server[328]: | OR                                                                   |
Aug 15 20:46:56 node01 cm-server[328]: | - a supported version of the OpenJDK from your OS vendor.  Help for  |
Aug 15 20:46:56 node01 cm-server[328]: |   some OSes are available at:                                        |
Aug 15 20:46:56 node01 cm-server[328]: |   > http://openjdk.java.net/install/ <                               |
Aug 15 20:46:56 node01 cm-server[328]: |                                                                      |
Aug 15 20:46:56 node01 cm-server[328]: | Cloudera Manager requires Oracle JDK or OpenJDK 1.8 or later.        |
Aug 15 20:46:56 node01 cm-server[328]: | NOTE: Cloudera Manager will find the Oracle JDK when starting,       |
Aug 15 20:46:56 node01 cm-server[328]: |       regardless of whether you installed the JDK using a binary     |
Aug 15 20:46:56 node01 cm-server[328]: |       installer or the RPM-based installer.                          |
Aug 15 20:46:56 node01 cm-server[328]: +======================================================================+
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service: main process exited, code=exited, status=1/FAILURE
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service holdoff time over, scheduling restart.
Aug 15 20:46:56 node01 systemd[1]: Stopped Cloudera CM Server Service.

根据异常信息提示，找不到JAVA_HOME，喊安装JDK，明明已经安装及配置好了JAVA_HOME。

具体原因：Server会使用/usr/java下的jdk。因此，解决方案：

bash 复制代码 
mkdir -p /usr/java

ln -s /usr/local/jdk8/  /usr/java/default

重新启动

bash 复制代码 
[root@node01 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-08-15 21:08:04 CST; 5s ago
  Process: 8626 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 8629 (java)
   CGroup: /system.slice/cloudera-scm-server.service
           └─8629 /usr/java/default/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -serv...

Aug 15 21:08:04 node01 systemd[1]: Starting Cloudera CM Server Service...
Aug 15 21:08:04 node01 systemd[1]: Started Cloudera CM Server Service.
Aug 15 21:08:04 node01 cm-server[8629]: JAVA_HOME=/usr/java/default

查看日志

bash 复制代码 
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

当进度达到100%，出现7180端口号，就可以访问Web界面

bash 复制代码 
2023-08-15 21:09:37,671 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: SPNEGO authentication is not enabled
2023-08-15 21:09:37,704 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.
2023-08-15 21:09:38,060 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 40% complete
2023-08-15 21:09:39,197 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 50% complete
2023-08-15 21:09:40,132 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 60% complete
2023-08-15 21:09:40,791 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2023-08-15T13:09:40.791Z
2023-08-15 21:09:40,842 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2023-08-15T13:09:40.842Z
2023-08-15 21:09:40,861 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:223
2023-08-15 21:09:40,861 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2023-08-15T13:09:40.861Z
2023-08-15 21:09:40,925 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:236
2023-08-15 21:09:40,926 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2023-08-15T13:09:40.925Z
2023-08-15 21:09:41,528 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 70% complete
2023-08-15 21:09:42,344 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2023-08-15T13:09:42.344Z
2023-08-15 21:09:42,708 INFO WebServerImpl:org.eclipse.jetty.server.Server: jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_371-b11
2023-08-15 21:09:42,728 INFO WebServerImpl:org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@7b124d81{HTTP/1.1,[http/1.1]}{0.0.0.0:7180}
2023-08-15 21:09:42,728 INFO WebServerImpl:org.eclipse.jetty.server.Server: Started @97631ms
2023-08-15 21:09:42,728 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
2023-08-15 21:09:42,951 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 80% complete
2023-08-15 21:09:43,448 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 90% complete
2023-08-15 21:09:44,010 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (2082186246 bytes) in 3972 pieces (3972 expected) in 11334.4ms.
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Single-file torrent information:
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Torrent name: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Announced at: Seems to be trackerless
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created on..: Tue Aug 15 21:09:32 CST 2023
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created by..: cm-server
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Pieces......: 3972 piece(s) (524288 byte(s)/piece)
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Total size..: 2,082,186,246 byte(s)

启动agent节点

bash 复制代码 
[root@node01 ~]# ./call.sh systemctl start cloudera-scm-agent
--------- node01 ----------
--------- node02 ----------
--------- node03 ----------
--------- node04 ----------
--------- node05 ----------

停止agent节点

bash 复制代码 
./call.sh systemctl stop cloudera-scm-agent

CM的使用

访问CM

浏览器访问：http://IP:7180，使用用户名：admin 密码：admin进行登录

接收条款与选择订阅类型

接受最终用户许可条款和条件

选择使用免费许可类型

集群安装

设置集群名称 勾选所有节点主机 选择CDH版本，默认即可 等待下载安装 注意：

bash 复制代码 
在重新安装时，可能遇到主机运行状态不良的情况

原因：Agent的集群id与当前Server集群id不匹配造成的

解决方案：删除agent的cm_guid文件
bash 复制代码 
[root@node01 ~]# ./call.sh systemctl stop cloudera-scm-agent

[root@node01 ~]# find / -name cm_guid
/var/lib/cloudera-scm-agent/cm_guid

[root@node01 ~]# cat /var/lib/cloudera-scm-agent/cm_guid
4b751103-b1fc-4914-81d1-52b2ebc20913

[root@node01 ~]# ./call.sh rm -rf /var/lib/cloudera-scm-agent/cm_guid

[root@node01 ~]# ./call.sh systemctl start cloudera-scm-agent

点击检查网络性能与检查主机

第二个异常，按照提示处理即可：

bash 复制代码 
[root@node01 ~]# ./call.sh "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
[root@node01 ~]# ./call.sh "echo never > /sys/kernel/mm/transparent_hugepage/enabled"

第一个与第三个异常信息不明显，点击查看具体详细信息 大概意思是Java版本不兼容，解决办法就是更换受支持的Oracle JDK版本

bash 复制代码 
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
warning: oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:oracle-j2sdk1.8-1.8.0+update181-1################################# [100%]
[root@node01 ~]# ./call.sh /usr/java/jdk1.8.0_181-cloudera/


[root@node01 ~]# whereis java
java: /usr/share/java /usr/java/jdk1.8.0_181-cloudera/bin/java

配置环境变量：vim /etc/bashrc

bash 复制代码 
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

验证安装

bash 复制代码 
[root@node01 ~]# java -version
java version "1.8.0_371"
Java(TM) SE Runtime Environment (build 1.8.0_371-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.371-b11, mixed mode)

同步分发文件、配置

bash 复制代码 
[root@node01 ~]# ./sync.sh /usr/java/jdk1.8.0_181-cloudera/

[root@node01 ~]# ./sync.sh /etc/bashrc

[root@node01 ~]# ./call.sh source /etc/bashrc

重新启动Server与Agent

bash 复制代码 
systemctl start cloudera-scm-server

./call.sh systemctl start cloudera-scm-agent

再次查看，发现异常已消失，仅剩一个异常，按照上述第二个异常处理即可

群集设置

选择自定义安装 选择安装服务，这里选择安装HDFS、YARN、ZK等三个基础服务 分配节点

集群设置全部选默认即可 开始自动安装 等待安装后，其会自动启动进程

首页

配置NameNode HA

进入HDFS页面点击启用High Availability 分配角色 审核更改 等待启动服务

配置Yarn HA

相关推荐
Qspace丨轻空间1 小时前
气膜场馆：推动体育文化旅游创新发展的关键力量—轻空间
大数据·人工智能·安全·生活·娱乐
Elastic 中国社区官方博客2 小时前
如何将数据从 AWS S3 导入到 Elastic Cloud - 第 3 部分：Elastic S3 连接器
大数据·elasticsearch·搜索引擎·云计算·全文检索·可用性测试·aws
Aloudata3 小时前
从Apache Atlas到Aloudata BIG，数据血缘解析有何改变？
大数据·apache·数据血缘·主动元数据·数据链路
水豚AI课代表3 小时前
分析报告、调研报告、工作方案等的提示词
大数据·人工智能·学习·chatgpt·aigc
拓端研究室TRL6 小时前
【梯度提升专题】XGBoost、Adaboost、CatBoost预测合集：抗乳腺癌药物优化、信贷风控、比特币应用|附数据代码...
大数据
黄焖鸡能干四碗6 小时前
信息化运维方案，实施方案，开发方案，信息中心安全运维资料（软件资料word）
大数据·人工智能·软件需求·设计规范·规格说明书
编码小袁6 小时前
探索数据科学与大数据技术专业本科生的广阔就业前景
大数据
WeeJot嵌入式7 小时前
大数据治理：确保数据的可持续性和价值
大数据
zmd-zk8 小时前
kafka+zookeeper的搭建
大数据·分布式·zookeeper·中间件·kafka
激流丶8 小时前
【Kafka 实战】如何解决Kafka Topic数量过多带来的性能问题？
java·大数据·kafka·topic
热门推荐
01【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总02(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明03组基轨迹建模 GBTM的介绍与实现（Stata 或 R）04【AIGC】重塑未来的科技巨轮05全面解析：构建基于深度学习的安全帽检测系统（UI界面+YOLO代码+数据集）06【经验分享】Ubuntu22.04安装微信(linux官方版)07基于YOLOv10深度学习的CT扫描图像肾结石智能检测系统【python源码+Pyqt5界面+数据集+训练代码】深度学习实战、目标检测08Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO09RAG 实践- Ollama+RagFlow 部署本地知识库10【TC3xx芯片】TC3xx芯片电源管理系统PMS详解