jdk+zookeeper+kafka 搭建kafka集群

环境准备

环境资源包:

jdk-8u341-linux-x64.tar.gz

kafka_2.12-2.2.0.tgz

zookeeper-3.4.14.tar.gz

server-id ip 状态
server1 10.206.120.10 leader
server2 10.206.120.2 follower
server3 10.206.120.3 follower

一、安装jdk

因为kafka需要Java环境,所以优先配置jdk环境,若已经配置了java环境,此步骤可以忽略

bash 复制代码
[root@VM-120-2-centos ~]# tar -xvf jdk-8u341-linux-x64.tar.gz 
[root@VM-120-2-centos ~]# mv jdk1.8.0_341/ /usr/local/
#在文件末尾加入以下语句
[root@VM-120-2-centos ~]# vim /etc/profile
export JAVA_HOME=/usr/local/jdk1.8.0_341
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib

[root@VM-120-2-centos ~]# source /etc/profile
[root@VM-120-2-centos ~]# java -version
java version "1.8.0_341"
Java(TM) SE Runtime Environment (build 1.8.0_341-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.341-b10, mixed mode)
至此jdk环境配置完成

二、zookeeper集群安装

bash 复制代码
[root@VM-120-2-centos ~]#  cd 
[root@VM-120-2-centos ~]# tar -xvf zookeeper-3.4.14.tar.gz 
[root@VM-120-2-centos ~]# mv zookeeper-3.4.14 /usr/local/zookeeper
[root@VM-120-2-centos ~]# cd /usr/local/zookeeper/conf/
[root@VM-120-2-centos ~]# cp zoo_sample.cfg zoo.cfg
[root@VM-120-2-centos ~]# vim zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataLogDir=/usr/local/zookeeper/logs
dataDir=/usr/local/zookeeper/data
clientPort=2181
autopurge.snapRetainCount=500
autopurge.purgeInterval=24
server.1= 10.206.120.10:2888:3888
server.2= 10.206.120.2:2888:3888
server.3= 10.206.120.2:2888:3888
[root@VM-120-2-centos ~]# mkdir /usr/local/zookeeper/data
#10.206.120.10服务器上执行
[root@VM-120-2-centos ~]# echo "1" > /usr/local/zookeeper/data/myid 

#10.206.120.2服务器上执行       
[root@VM-120-2-centos ~]# echo "2" > /usr/local/zookeeper/data/myid

#10.206.120.3服务器上执行       
[root@VM-120-2-centos ~]# echo "3" > /usr/local/zookeeper/data/myid
[root@VM-120-2-centos ~]# cd ../bin/
[root@VM-120-2-centos ~]# ./zkServer.sh start
[root@VM-120-2-centos ~]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[root@VM-120-2-centos ~]# netstat -ntlp
看看端口是否正常启动

二、kafka集群安装

bash 复制代码
[root@VM-120-2-centos ~]# cd 
[root@VM-120-2-centos ~]# tar -xvf kafka_2.12-2.2.0.tgz 
[root@VM-120-2-centos ~]# mv kafka_2.12-2.2.0 /usr/local/kafka
[root@VM-120-2-centos ~]# cd /usr/local/kafka/config/
[root@VM-120-2-centos ~]# cp server.properties server.properties.bak
#10.206.120.2服务器上将以下内容写入文件
[root@VM-120-2-centos ~]# vim server.properties
broker.id=2
listeners=PLAINTEXT://10.206.120.2:9092
advertised.listeners=PLAINTEXT://43.137.8.225:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=204857600
log.dirs=/usr/local/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=1680
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.206.120.2:2181,10.206.120.10:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=10
max.poll.interval.ms=800000
max.poll.records=50

#10.206.120.10服务器上将以下内容写入文件
[root@VM-120-2-centos ~]# vim server.properties
broker.id=1
listeners=PLAINTEXT://10.206.120.10:9092
advertised.listeners=PLAINTEXT://118.195.137.101:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=204857600
log.dirs=/usr/local/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=1680
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.206.120.2:2181,10.206.120.10:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=10
max.poll.interval.ms=800000
max.poll.records=50

#10.206.120.3服务器上将以下内容写入文件
[root@VM-120-2-centos ~]# vim server.properties
broker.id=3
listeners=PLAINTEXT://10.206.120.3:9092
advertised.listeners=PLAINTEXT://175.27.146.204:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=204857600
log.dirs=/usr/local/kafka/logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=1680
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.206.120.2:2181,10.206.120.10:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=10
max.poll.interval.ms=800000
max.poll.records=50

[root@VM-120-2-centos ~]# cd ../bin/
[root@VM-120-2-centos ~]# nohup ./kafka-server-start.sh ../config/server.properties &

#查看9092端口是否启动,启动即为正常
[root@VM-120-10-centos bin]# netstat -ntlp  
bash 复制代码
#下面所有命令在kafka的bin目录下执行(笔者是/usr/local/kafka/bin)
#创建topic话题
./kafka-topics.sh --create --bootstrap-server 10.206.120.10:2181,10.206.120.2:2181,10.206.120.3:2181 --replication-factor 3 --topic test --partitions 3
#开启一个生产者
./kafka-console-producer.sh --broker-list 10.206.120.10:2181,10.206.120.2:2181,10.206.120.3:2181 --topic test
#开启一个消费者
./kafka-console-consumer.sh --bootstrap-server 10.206.120.10:2181,10.206.120.2:2181,10.206.120.3:2181 --topic test --from-beginning
#列出所有topic
./kafka-topics.sh --zookeeper 10.206.120.10:2181,10.206.120.2:2181,10.206.120.3:2181 -list
#输出某个topic
./kafka-topics.sh --zookeeper 10.206.120.10:2181,10.206.120.2:2181,10.206.120.3:2181 --delete --topic test
#详细描述某个topic
./kafka-topics.sh --zookeeper 10.206.120.10:2181,10.206.120.2:2181,10.206.120.3:2181 --describe --topic test



至此集群搭建成功,可以正常使用了

相关推荐
吃我一窝jio24 分钟前
Java Bug复盘与性能优化实战
java·开发语言·笔记
Flynn Liang1 小时前
Java 序列化也能成为瓶颈:Jackson、Protobuf 与 Redis 序列化的生产实践
java·redis·性能优化·jackson·序列化·protobuf
Yeats_Liao1 小时前
物联网平台优化系列(三):全局懒加载吞掉 MQTT 初始化
java·物联网·struts
千寻技术帮1 小时前
基于Spring Boot+Vue的农产品销售管理系统(源码+文档+数据库,附部署教程)
java·数据库·spring boot·后端·mysql·vue
好用斋1 小时前
拾光坞G2使用新版Docker懒人包:一键安装、一键升级、一键卸载
java·docker·容器
码云社区2 小时前
Spring Boot陪诊陪护系统源码:医院目录、服务项目与城市业务配置
java·spring boot·后端·陪诊陪护·医院陪诊·医院陪护·陪护小程序
CoderYanger3 小时前
Java EE 进阶:9.MyBatis 操作数据库(进阶)(第七弹):图书管理系统(强制登录)
java·数据库·spring boot·spring·java-ee·tomcat·mybatis
学长毕设3 小时前
基于SpringBoot 餐饮餐厅点菜管理系统-附源码
java·spring boot·后端
学长毕设3 小时前
基于SpringBoot 校园招聘求职论坛交流系统-附源码
java·spring boot·后端
Highcharts.js7 小时前
Highcharts漏斗图表使用配置指南
java·开发语言·javascript·信息可视化·highcharts·漏斗图