目录
前提条件
安装好ZooKeeper集群，可参考：openEuler24.03 LTS下安装ZooKeeper集群
Kafka集群规划
| node2 | node3 | node4 |
|---|---|---|
| kafka | kafka | kafka |
安装的kafka版本为3.3.1，先在node2安装kafka，然后分发kafka安装目录到其他机器。
下载Kafka
浏览器访问以下地址下载Kafka
https://archive.apache.org/dist/kafka/3.3.1/kafka_2.12-3.3.1.tgz上传到node2 /opt/software目录
[liang@node2 conf]$ cd /opt/software/
[liang@node2 software]$ ls | grep kafka
kafka_2.12-3.3.1.tgz解压
[liang@node2 software]$ tar -zxvf kafka_2.12-3.3.1.tgz -C /opt/module/设置环境变量
[liang@node2 module]$ sudo vim /etc/profile.d/my_env.sh添加内容
bash
#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka_2.12-3.3.1
export PATH=$PATH:$KAFKA_HOME/bin让环境变量生效
[liang@node2 module]$ source /etc/profile配置Kafka
[liang@node2 module]$ cd $KAFKA_HOME/config
[liang@node2 config]$ ls
connect-console-sink.properties connect-mirror-maker.properties server.properties
connect-console-source.properties connect-standalone.properties tools-log4j.properties
connect-distributed.properties consumer.properties trogdor.conf
connect-file-sink.properties kraft zookeeper.properties
connect-file-source.properties log4j.properties
connect-log4j.properties producer.properties
[liang@node2 config]$ vim server.properties修改及添加内容如下
bash
# 修改
broker.id=0
log.dirs=/opt/module/kafka_2.12-3.3.1/datas
zookeeper.connect=node2:2181,node3:2181,node4:2181/kafka
# 添加
delete.topic.enable=true分发到其他机器
分发安装文件
分发安装文件
分发Kafka安装目录到node3、node4
xsync /opt/module/kafka_2.12-3.3.1修改broker id
分别修改node3、node4的broker id。注意：kafka集群broker id不能重复。
node3机器
[liang@node3 ~]$ vim /opt/module/kafka_2.12-3.3.1/config/server.properties将broker id设置为1
bash
broker.id=1node4机器
[liang@node4 ~]$ vim /opt/module/kafka_2.12-3.3.1/config/server.properties将broker id设置为2
bash
broker.id=2分发环境变量
sudo /home/liang/bin/xsync /etc/profile.d/my_env.sh根据提示输入node2机器root账户的登录密码
分别让node3及node4的环境变量生效
[liang@node3 ~]$ source /etc/profile
[liang@node4 ~]$ source /etc/profile启动Kafka
分别在集群各个机器启动Kafka
node2机器
Kafka需要用到ZooKeeper，如果还没有启动ZooKeeper，先启动ZooKeeper
[liang@node2 config]$ zk.sh start启动kafka
bash
[liang@node2 config]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties出现如下警告
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
...
经测试发送和接收数据，均正常，但警告信息太多，影响观察数据。
解决警告问题：
egrep 命令已过时，需要将kafka-run-class.sh中的egrep 改为grep -E
[liang@node2 config]$ cd $KAFKA_HOME/bin
[liang@node2 bin]$ vim kafka-run-class.sh输入/egrep关键字，回车光标跳到匹配的第一个关键字（全脚本也只有一个地方匹配）
将egrep 改为grep -E，修改后效果如下
分发修改文件
xsync /opt/module/kafka_2.12-3.3.1/bin重启kafka
[liang@node2 bin]$ kafka-server-stop.sh
[liang@node2 bin]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server.propertiesjps查看进程
[liang@node2 bin]$ jps
4626 Jps
4548 Kafka
3583 QuorumPeerMainnode3机器
启动kafka
[liang@node3 ~]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server.propertiesjps查看进程
[liang@node3 ~]$ jps
3766 Kafka
3244 QuorumPeerMain
3790 Jpsnode4机器
启动kafka
[liang@node4 bin]$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server.propertiesjps查看进程
[liang@node3 bin]$ jps
3764 Jps
3208 QuorumPeerMain
3724 Kafka测试Kafka
创建topic
bash
[liang@node2 bin]$ kafka-topics.sh --bootstrap-server node2:9092,node3:9092,node4:9092 --create --partitions 1 --replication-factor 1 --topic test1查看topic
bash
[liang@node2 bin]$ kafka-topics.sh --bootstrap-server node2:9092,node3:9092,node4:9092 --list生产者
执行如下命令生产消息
bash
[liang@node2 bin]$ kafka-console-producer.sh --bootstrap-server node2:9092,node3:9092,node4:9092 --topic test1消费者
启动一个新的终端，执行消费者命令
bash
[liang@node2 ~]$ kafka-console-consumer.sh --bootstrap-server node2:9092,node3:9092,node4:9092 --topic test1在生产者终端发送数据
观察消费者终端能正常接收生产者发来的数据
分别按Ctrl+c关闭生产者和消费者返回Linux命令行。
关闭Kafka
分别在集群各个机器关闭Kafka
[liang@node2 ~]$ kafka-server-stop.sh
[liang@node3 ~]$ kafka-server-stop.sh
[liang@node4 ~]$ kafka-server-stop.sh再关闭ZooKeeper集群
[liang@node2 ~]$ zk.sh stop集群启停脚本
在node2机器操作
vim ~/bin/kf.sh内容如下：
bash
#!/bin/bash
case $1 in
"start"){
for i in node2 node3 node4
do
echo " --------启动 $i Kafka-------"
ssh $i "/opt/module/kafka_2.12-3.3.1/bin/kafka-server-start.sh -daemon /opt/module/kafka_2.12-3.3.1/config/server.properties"
done
};;
"stop"){
for i in node2 node3 node4
do
echo " --------停止 $i Kafka-------"
ssh $i "/opt/module/kafka_2.12-3.3.1/bin/kafka-server-stop.sh stop"
done
};;
esac添加权限
chmod +x ~/bin/kf.sh如果需要在其他机器执行脚本，需要分发脚本到其他机器（可选）
xsync ~/bin测试
脚本启动kafka
zk.sh start && kf.sh start关闭kafka
kf.sh stop && zk.sh stop问题及解决
执行脚本时，输出Authorized users only. All activities may be monitored and reported.警告信息
---------- zookeeper node2 启动 ------------
Authorized users only. All activities may be monitored and reported.
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.7.1/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 8654.
---------- zookeeper node3 启动 ------------
Authorized users only. All activities may be monitored and reported.
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.7.1/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 5802.
---------- zookeeper node4 启动 ------------
Authorized users only. All activities may be monitored and reported.
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.7.1/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 5686.解决办法
若需彻底禁用SSH警告，可修改SSH配置文件/etc/ssh/sshd_config，注释或删除Banner /etc/issue.net行，并重启SSH服务。
分别在集群所有机器执行如下语句
sudo vim /etc/ssh/sshd_config找到并注释Banner /etc/issue.net
bash
#Banner /etc/issue.net重启SSH
sudo systemctl restart sshd修改后效果
[liang@node2 bin]$ kf.sh stop && zk.sh stop
--------停止 node2 Kafka-------
--------停止 node3 Kafka-------
--------停止 node4 Kafka-------
---------- zookeeper node2 停止 ------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.7.1/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
---------- zookeeper node3 停止 ------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.7.1/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
---------- zookeeper node4 停止 ------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.7.1/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED如有需要，可点击查看：
完成！enjoy it！