Kafka 集群部署(CentOS 单机模拟版)

文章目录

0.前置说明

由于我们手里只有一台Linux机器,所以我们实现的是简单的单机模拟的集群部署,通过修改配置文件,启动 3 个 kafka 时用到 3 个不同的端口(9091,9092,9093)。

1.安装Java11

  1. 切换到你的工作目录下执行:
bash 复制代码
yum install java-11-openjdk -y
  1. 添加环境变量;
bash 复制代码
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.23.0.9-3.tl3.x86_64
export PATH=$PATH:$JAVA_HOME/bin
  1. 让你的环境变量生效;
bash 复制代码
source /etc/profile
  1. 测试是否安装成功;
bash 复制代码
java -version
openjdk version "11.0.23" 2024-04-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2) (build 11.0.23+9-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.23.0.9-2) (build 11.0.23+9-LTS, mixed mode, sharing)

2.集群部署

  1. 在你的工作目录下新建目录 cluster(集群);
bash 复制代码
mkdir cluster
  1. 进入cluster目录下,下载 kafka 安装包 kafka-3.6.1-src.tgz并解压。
bash 复制代码
rz -E # 上传本地安装包
tar -zxf kafka_2.12-3.6.1.tgz
  1. 改名为 kafka
bash 复制代码
mv kafka_2.12-3.6.1 kafka

2.1 安装ZooKeeper

  1. 修改文件夹名为zookeeper

因为 kafka 内置了 ZooKeeper 软件,所以此处将解压缩的文件作为 ZooKeeper 软件使用。

bash 复制代码
mv kafka/ zookeeper
  1. 修改config/zookeeper.properties文件;
bash 复制代码
cd zookeeper/
vim config/zookeeper.properties 
  • 修改dataDir,zookeeper 数据的存储文件。
bash 复制代码
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/root/cluster/zookeeper-data/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080

2.2 安装Kafka

  1. 将上面解压缩的文件复制一份,改名为broke-1
bash 复制代码
mv kafka_2.12-3.6.1/ broker-1
ll
bash 复制代码
total 8
drwxr-xr-x 7 root root 4096 Nov 24 17:43 broker-1
drwxr-xr-x 7 root root 4096 Nov 24 17:43 zookeeper
  1. 修改config/server.properties配置文件;
bash 复制代码
vim broker-1/config/server.properties
bash 复制代码
############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1 # kafka 节点数字标识,集群内具有唯一性
#......

############################# Socket Server Settings #############################

# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9091
#.......
############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/root/cluster/broker-data/broker-1 # 监听器 9091 为本地端口,如果冲突,请重新指定
# .......
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181 # 数据文件路径,如果不存在,会自动创建

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000 # ZooKeeper 软件连接地址,2181 为默认的ZK 端口号 /kafka 为ZK 的管理节点
  1. 同样的步骤,复制一份broker-2broker-3,并修改配置文件。
bash 复制代码
# broker-2
############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2 # kafka 节点数字标识,集群内具有唯一性
#......

############################# Socket Server Settings #############################

# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092
#.......
############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/root/cluster/broker-data/broker-2 # 监听器 9091 为本地端口,如果冲突,请重新指定
# .......
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181 # 数据文件路径,如果不存在,会自动创建

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000 # ZooKeeper 软件连接地址,2181 为默认的ZK 端口号 /kafka 为ZK 的管理节点
bash 复制代码
# broker-3
############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=3 # kafka 节点数字标识,集群内具有唯一性
#......

############################# Socket Server Settings #############################

# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9093
#.......
############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/root/cluster/broker-data/broker-3 # 监听器 9091 为本地端口,如果冲突,请重新指定
# .......
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181 # 数据文件路径,如果不存在,会自动创建

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000 # ZooKeeper 软件连接地址,2181 为默认的ZK 端口号 /kafka 为ZK 的管理节点

2.3 封装启动脚本

因为 Kafka 启动前,必须先启动 ZooKeeper,并且Kafka集群中有多个节点需要启动,所以启动过程比较繁琐,这里我们将启动的指令进行封装。

  1. zookeeper文件夹下创建zk.sh批处理文件;
bash 复制代码
cd zookeeper
vim zk.sh

## zk.sh
./bin/zookeeper-server-start.sh config/zookeeper.properties

chmod +x zk.sh
  1. broker-1broker-2broker-3文件夹下分别创建kfk.sh批处理文件;
bash 复制代码
cd broker-1
vim kfk.sh

## kfk.sh
./bin/kafka-server-start.sh config/server.properties

chmod +x kfk.sh
  1. cluster文件夹下创建cluster.sh批处理文件,用于启动 kafka 集群。
bash 复制代码
vim cluster.sh
bash 复制代码
# cluster.sh
cd zookeeper
./zk.sh
cd ../broker-1
./kfk.sh
cd ../broker-2
./kfk.sh
cd ../broker-3
./kfk.sh
bash 复制代码
chmod +x cluster.sh
  1. cluster文件夹下创建cluster-clear.sh批处理文件,用于清理和重置 kafka 数据。
bash 复制代码
vim cluster-clear.sh

# cluster-clear.sh 
rm -rf zookeeper-data
rm -rf broker-data
                      
chmod +x cluster-clear.sh 
  1. cluster目录下,运行./cluster.sh文件即可启动集群。
bash 复制代码
# 启动集群
./cluster

# 查看是否启动成功,如果新建了data文件,说明启动成功了
ll zookeeper-data/
total 4
drwxr-xr-x 3 root root 4096 May 27 10:20 zookeeper

ll broker-data/
total 12
drwxr-xr-x 2 root root 4096 May 27 10:21 broker-1
drwxr-xr-x 2 root root 4096 May 27 10:21 broker-2
drwxr-xr-x 2 root root 4096 May 27 10:21 broker-3
  1. 当我们想要关闭集群时,不仅要清理 zookeeper 和 kafka 的数据文件,还有kill -9 结束 zookeeper 进程与 kakfa 进程,这需要我们手动逐一kill
bash 复制代码
ps axj | grep zookeeper
kill -9 #zookeeper的PID

ps axj | grep kafka
kill -9 #3个kafka节点的PID
相关推荐
陌小呆^O^3 小时前
Cmakelist.txt之Liunx-rabbitmq
分布式·rabbitmq
斯普信专业组5 小时前
深度解析FastDFS:构建高效分布式文件存储的实战指南(上)
分布式·fastdfs
jikuaidi6yuan7 小时前
鸿蒙系统(HarmonyOS)分布式任务调度
分布式·华为·harmonyos
BestandW1shEs7 小时前
彻底理解消息队列的作用及如何选择
java·kafka·rabbitmq·rocketmq
天冬忘忧7 小时前
Kafka 生产者全面解析:从基础原理到高级实践
大数据·分布式·kafka
天冬忘忧8 小时前
Kafka 数据倾斜:原因、影响与解决方案
分布式·kafka
隔着天花板看星星8 小时前
Kafka-Consumer理论知识
大数据·分布式·中间件·kafka
holywangle8 小时前
解决Flink读取kafka主题数据无报错无数据打印的重大发现(问题已解决)
大数据·flink·kafka
隔着天花板看星星8 小时前
Kafka-副本分配策略
大数据·分布式·中间件·kafka