linux 虚拟机部署kafka+zookeeper单机方案

背景：

1.这里介绍了如何在Linux系统上安装和配置Apache ZooKeeper3.6.4，以及Kafka3.4.1，

2.包括创建目录、下载安装包、修改配置、添加环境变量、启动服务、编写启动和停止脚本。

1、安装zookeeper服务

注：这里使用zookeeper3.6.4版本

前提：zookeeper运行需要jdk环境,所以需要提前安装好jdk环境。

1.安装jdk环境：

注：这里采用了yum安装jdk的方式：
先查询jdk版本，通过yum命令

bash 复制代码

[root@xionghj /]# yum -y list *openjdk*
.......

注：这里选择的是

bash 复制代码

[root@xionghj /]# yum install -y java-1.8.0-openjdk-devel.x86_64
.....

查询jdk环境是否生效：

bash 复制代码

[root@xionghj /]# java -version
openjdk version "1.8.0_412"
OpenJDK Runtime Environment (build 1.8.0_412-b08)
OpenJDK 64-Bit Server VM (build 25.412-b08, mixed mode)

2.安装zookeeper服务：

创建zookeeper运行目录：

bash 复制代码

[root@xionghj /]# mkdir -p /usr/local/zookeeper
[root@xionghj /]# chmod 777 /usr/local/zookeeper/

下载zookeeper安装包：

bash 复制代码

[root@xionghj /]#  wget https://archive.apache.org/dist/zookeeper/zookeeper-3.6.4/apache-zookeeper-3.6.4-bin.tar.gz

解压安装包

bash 复制代码

[root@xionghj /]# mv apache-zookeeper-3.6.4-bin.tar.gz /usr/local/zookeeper/
[root@xionghj /]# cd /usr/local/zookeeper/

[root@xionghj zookeeper]# tar -zxvf apache-zookeeper-3.6.4-bin.tar.gz
.......

3.修改zookeeper的配置：

bash 复制代码

[root@xionghj zookeeper]# cd zookeeper-3.6.4/conf/

[root@xionghj conf]# cp zoo_sample.cfg zoo.cfg

[root@xionghj conf]# vim zoo.cfg

修改的内容如下：

bash 复制代码

dataDir=/usr/local/zookeeper/zookeeper-3.6.4/data

dateLogDir=/usr/local/zookeeper/zookeeper-3.6.4/logs

clientPort=2181

4.添加全局环境变量配置：

编辑环境变量文件

bash 复制代码

[root@xionghj conf] vim /etc/profile
#尾部添加如下内容
export ZOOKEEPER_HOME=/usr/local/zookeeper/zookeeper-3.6.4/
export PATH=$ZOOKEEPER_HOME/bin:$PATH
export PATH

#退出保存
:wq

重新执行一下

bash 复制代码

[root@xionghj conf] source /etc/profile

5.启动并测试zookeeper服务：

进入bin目录

bash 复制代码

[root@xionghj zookeeper-3.6.4]# cd bin/

启动服务

bash 复制代码

[root@xionghj bin]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/zookeeper-3.6.4/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

查看状态

bash 复制代码

[root@xionghj bin]# zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/zookeeper-3.6.4/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone

查看进程

bash 复制代码

[root@xionghj bin]# ps -ef | grep zookeeper
root      1070     1  2 14:16 pts/1    00:00:01 java -Dzookeeper.log.dir=/usr/local/zookeeper/zookeeper-3.6.4/bin/../logs -Dzookeeper.log.file=zookeeper-root-server-xionghj.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-server/target/classes:/usr/local/zookeeper/zookeeper-3.6.4/bin/../build/classes:/usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-server/target/lib/*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../build/lib/*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/zookeeper-prometheus-metrics-3.6.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/zookeeper-jute-3.6.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/zookeeper-3.6.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/snappy-java-1.1.8.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/slf4j-reload4j-1.7.36.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/slf4j-api-1.7.36.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient_servlet-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient_hotspot-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient_common-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/reload4j-1.2.24.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-native-unix-common-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-native-epoll-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-classes-epoll-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-resolver-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-handler-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-common-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-codec-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-buffer-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/metrics-core-3.2.5.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/json-simple-1.1.1.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jline-2.14.6.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-util-ajax-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-util-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-servlet-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-server-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-security-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-io-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-http-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/javax.servlet-api-3.1.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jackson-databind-2.13.4.2.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jackson-core-2.13.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jackson-annotations-2.13.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/commons-cli-1.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/audience-annotations-0.13.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-server/src/main/resources/lib/*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../conf: -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/local/zookeeper/zookeeper-3.6.4/bin/../conf/zoo.cfg

关闭服务

bash 复制代码

[root@xionghj bin]#  zkServer.sh stop

重启服务

bash 复制代码

[root@xionghj bin]#  zkServer.sh restart

2、安装kafka服务：

1.获取安装包：

需要从官方上获取，https://kafka.apache.org/downloads
选择的版本是kafka-3.4.1。

下载地址：https://archive.apache.org/dist/kafka/3.4.1/kafka_2.12-3.4.1.tgz

bash 复制代码

[root@xionghj kafka]# wget https://archive.apache.org/dist/kafka/3.4.1/kafka_2.12-3.4.1.tgz

2.创建kafka安装目录：

bash 复制代码

#用于存放kafka的解压包
[root@xionghj /]# mkdir -p /opt/kafka

#用于存放kafka的数据
[root@xionghj /]# mkdir -p /opt/kafka/kafka_data

#用于存放kafka的日志
[root@xionghj /]# mkdir -p /opt/kafka/kafka_log

3.解压并安装kafka服务：

bash 复制代码

[root@xionghj kafka]# tar -zxvf kafka_2.12-3.4.1.tgz

4.修改kafka服务配置：

bash 复制代码

[root@xionghj kafka]# cd /opt/kafka/kafka_2.12-3.4.1/config

[root@xionghj config ]# vim server.properties

#修改一下参数
broker.id=0

#端口号
port=9092   

#服务器IP地址，修改为自己的服务器IP
host.name=localhost

#日志存放路径，上面创建的目录
log.dirs=/opt/kafka/kafka_log

#zookeeper地址和端口，单机配置部署，localhost:2181
zookeeper.connect=localhost:2181

5.启动并测试kafka服务：

启动kafka服务

bash 复制代码

[root@xionghj config ]# /opt/kafka/kafka_2.12-3.4.1/bin/kafka-server-start.sh /opt/kafka/kafka_2.12-3.4.1/config/server.properties
........
[2024-09-05 15:28:13,286] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT://iZwz9ippftvoxv000lpl0tZ:9092, czxid (broker epoch): 25 (kafka.zk.KafkaZkClient)
[2024-09-05 15:28:13,403] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,424] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,462] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,478] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
[2024-09-05 15:28:13,485] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2024-09-05 15:28:13,494] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2024-09-05 15:28:13,519] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2024-09-05 15:28:13,529] INFO Feature ZK node created at path: /feature (kafka.server.FinalizedFeatureChangeListener)
[2024-09-05 15:28:13,533] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2024-09-05 15:28:13,542] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2024-09-05 15:28:13,583] INFO [MetadataCache brokerId=0] Updated cache from existing <empty> to latest FinalizedFeaturesAndEpoch(features=Map(), epoch=0). (kafka.server.metadata.ZkMetadataCache)
[2024-09-05 15:28:13,618] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,760] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2024-09-05 15:28:13,777] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Enabling request processing. (kafka.network.SocketServer)
[2024-09-05 15:28:13,818] INFO Kafka version: 3.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2024-09-05 15:28:13,818] INFO Kafka commitId: 8a516edc2755df89 (org.apache.kafka.common.utils.AppInfoParser)
[2024-09-05 15:28:13,818] INFO Kafka startTimeMs: 1725521293814 (org.apache.kafka.common.utils.AppInfoParser)
[2024-09-05 15:28:13,821] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2024-09-05 15:28:13,934] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Recorded new controller, from now on will use node iZwz9ippftvoxv000lpl0tZ:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
[2024-09-05 15:28:13,935] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorded new controller, from now on will use node iZwz9ippftvoxv000lpl0tZ:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)

测试kafka服务：

前提开启两个终端，一个生产者，一个消费者

生产者

bash 复制代码

[root@xionghj ~]# cd /opt/kafka/kafka_2.12-3.4.1/bin/
[root@xionghj bin]# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
>

这里输入消息

bash 复制代码

[root@xionghj bin]# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
>xionghj
>kafak zookeeper
>

消费者

bash 复制代码

[root@xionghj ~]# cd /opt/kafka/kafka_2.12-3.4.1/bin
[root@xionghj bin]# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
[2024-09-05 15:31:36,908] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,031] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,191] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

消费者就可以收到消息了

bash 复制代码

[root@xionghj bin]# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
[2024-09-05 15:31:36,908] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,031] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,191] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
xionghj
kafak zookeeper

3、开发zookeeper+kafka服务启动停止脚本：

启动脚本：

bash 复制代码

[root@xionghj / ]# cd /opt/kafka/kafka_2.12-3.4.1/bin

[root@xionghj bin]# vim kafka_start.sh

#内容如下：

#!/bin/bash
#
echo "启动zookeeper服务..."
sh /usr/local/zookeeper/zookeeper-3.6.4/bin/zkServer.sh start
sleep 5
echo "启动kafka服务..."
/opt/kafka/kafka_2.12-3.4.1/bin/kafka-server-start.sh /opt/kafka/kafka_2.12-3.4.1/config/server.properties &

停止脚本：

bash 复制代码

[root@xionghj / ]# cd /opt/kafka/kafka_2.12-3.4.1/bin

[root@xionghj bin]# vim kafka_start.sh

#!/bin/bash
echo "停止zookeeper服务..."
sh /usr/local/zookeeper/zookeeper-3.6.4/bin/zkServer.sh stop
sleep 5
echo "停止kafka服务..."
/opt/kafka/kafka_2.12-3.4.1/bin/kafka-server-stop.sh /opt/kafka/kafka_2.12-3.4.1/config/server.properties

设置可执行权限：

bash 复制代码

[root@xionghj bin]# chmod +x kafka_start.sh 
[root@xionghj bin]# chmod +x kafka_stop.sh

设置开机自起

bash 复制代码

[root@xionghj / ]#  vim /etc/rc.d/rc.local 
＃设置开机自动在后台运行脚本
bash /opt/kafka/kafka_2.12-3.4.1/kafka_start.sh &