01_kafka_环境搭建安装_topic管理

AntHub2023-09-12 22:53

文章目录

安装jdk

  • 略

配置主机名

  • hostnamectl set-hostname kafka_1

  • /etc/sysconfig/network

shell 复制代码 
HOSTNAME=kafka_1
  • /etc/hosts
shell 复制代码 
ip  kafka_1
  • ping kafka_1 测试

Zookeeper 下载与安装

  • 由于 集群中的 Leader 的监控 和 Topic 元数据 存储在 Zookeeper 中 ，所以需要安装 zk；
  • https://zookeeper.apache.org/releases.html 中选择需要的版本；
  • conf 目录 复制zoo_sample.cfg 为 zoo.cfg, 修改其中的配置 dataDir 指定自己的目录；
  • 启动zk：使用 zkServer.sh start zoo.cfg
shell 复制代码 
cd /opt/pkg/
wget --no-check-certificate  https://dlcdn.apache.org/zookeeper/zookeeper-3.8.2/apache-zookeeper-3.8.2-bin.tar.gz
tar zxf apache-zookeeper-3.8.2-bin.tar.gz
mv apache-zookeeper-3.8.2-bin ../app/
cd /opt/app/apache-zookeeper-3.8.2-bin/
cp zoo_sample.cfg zoo.cfg
./bin/zkServer.sh start zoo.cfg

---
[root@loaclhost bin]# ./zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/app/apache-zookeeper-3.8.2-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone

Kafka 下载与安装

shell 复制代码 
cd /opt/pkg/
wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz
tar zxf kafka_2.13-3.5.1.tgz
mv kafka_2.13-3.5.1 ../app/
cd /opt/app
cd kafka_2.13-3.5.1/
cd config/
cp server.properties server.properties.bak
vim server.properties
---
listeners=PLAINTEXT://kafka_1:9092
og.dirs=./kafka-logs
zookeeper.connect=kafka_1:2181
---

cd ../
./bin/kafka-server-start.sh -daemon config/server.properties
  • 配置： config/server.properties
shell 复制代码 
broker.id
listeners=PLAINTEXT://主机名:9092
log.dirs= # 日志目录
zookeeper.connect=主机名:2181
  • 启动：./bin/kafka-server-start.sh -daemon config/server.properties
  • 停止：./bin/kafka-server-stop.sh

测试

shell 复制代码 
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092 --create --topic topic_01 --partitions 3 --replication-factor 1 # 单机版副本因子为1 多了没有意义
./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092 --topic topic_01 --group g1 # 消费者 查看输出
./bin/kafka-console-producer.sh --broker-list kafka_1:9092  --topic topic_01  # 开始输入

root@loaclhost kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092 --create --topic topic_0

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should c

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('*') could collide. To avo
Created topic topic_01.

root@loaclhost kafka_2.13-3.5.1\]# ./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092 --topic topic* OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should c *** ** * ** *** root@loaclhost kafka_2.13-3.5.1\]# ./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092 --topic topic_01 --group g1 OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N *** ** * ** *** ![在这里插入图片描述](https://file.jishuzhan.net/article/1701609926813552641/f852e260d8fd4e6fb2b1360f6056ad9e.png)

集群版安装

  • 使用3台主机进行搭建，三台机器上都需要安装 jdk，Zookeeper，Kafka
  • 其中 /etc/hosts 都需要配置好三台主机的ip 主机名映射；
  • 需要额外注意的是，三台主机需要做一次时间同步
shell 复制代码 
yum -y install ntpdate
timedatectl set-timezone Asia/Shanghai
ntpdate ntp1.aliyun.com # aliyun 有ntp1~7， 或者 cn.pool.ntp.org
  • 复制文件到 另外两台主机
shell 复制代码 
[root@kafka_1 pkg]# pwd
/opt/pkg
scp kafka_2.13-3.5.1.tgz root@kafka_2:/opt/pkg 
scp kafka_2.13-3.5.1.tgz root@kafka_3:/opt/pkg
scp apache-zookeeper-3.8.2-bin.tar.gz root@kafka_3:/opt/pkg
scp apache-zookeeper-3.8.2-bin.tar.gz root@kafka_2:/opt/pkg
  • 配置 zoo.cfg:
shell 复制代码 
# 数据目录 每个机器中，此目录下 需要创建 myid文件内容 和server.id 对应 , 可以指定一个绝对目录
dataDir=/opt/app/cluster/apache-zookeeper-3.8.2/dataDir 

server.1=kafka_1:2888:3888
server.2=kafka_2:2888:3888
server.3=kafka_3:2888:3888
4lw.commands.whitelist=*

  • myid生成： echo 1 > /opt/app/cluster/apache-zookeeper-3.8.2/dataDir/myid # 1,2,3

  • 配置： config/server.properties

shell 复制代码 
broker.id=0 # 需要修改 1,2 
listeners=PLAINTEXT://kafka_1:9092 # 主机名和当前机器对应
log.dirs=/opt/app/cluster/kafka_2.13-3.5.1/log_dirs # 日志目录
zookeeper.connect=kafka_1:2181,kafka_2:2181,kafka_3:2181
  • 启动zk 和 kafka:
shell 复制代码 
cd /opt/app/cluster/apache-zookeeper-3.8.2/
./bin/zkServer.sh start zoo.cfg
cd /opt/app/cluster/kafka_2.13-3.5.1
./bin/kafka-server-start.sh -daemon config/server.properties
  • 报错：

Using config: /opt/app/cluster/apache-zookeeper-3.8.2/bin/.../conf/zoo.cfg

Client port found: 2181. Client address: localhost. Client SSL: false.

Error contacting service. It is probably not running.

  • 原因是防火墙没关：
shell 复制代码 
 systemctl status firewalld
#● firewalld.service - firewalld - dynamic firewall daemon
#   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
#   Active: active (running) since Sun 2023-08-27 19:48:04 CST; 2h 32min ago
#     Docs: man:firewalld(1)
# Main PID: 649 (firewalld)
#   CGroup: /system.slice/firewalld.service
#           └─649 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
#           
systemctl disable firewalld

测试

shell 复制代码 
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic01 --partitions 3 --replication-factor 3  # 创建 topic
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic02 --partitions 3 --replication-factor 3  # 创建 topic
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list # 查看
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --topic topic02 # 查看详细描述
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --alter --topic topic02 --partitions 4   # 修改分区数量,只能增不能减，设置小于3则报错
./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --delete --topic topic02  # 删除topic
	
---
# 消息的生产与消费/订阅
./bin/kafka-console-producer.sh --broker-list kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01 # 生产
./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01 --group g1 --property print.key=true  --property print.value=true  --property key.separator=, # 消费
---
# 查看消费组信息
./bin/kafka-consumer-groups.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list g1  # 查看
./bin/kafka-consumer-groups.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --group g1  # 查看详情

输出

shell 复制代码 
[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic01 --partitions 3 --replication-factor 3
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Created topic topic01.
[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --create --topic topic02 --partitions 3 --replication-factor 3
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Created topic topic02.
[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
topic01
topic02

[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --topic topic02
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Topic: topic02  TopicId: _s44mBQqTLaSmVWcZA2GTw PartitionCount: 3       ReplicationFactor: 3    Configs:
        Topic: topic02  Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,1
        Topic: topic02  Partition: 1    Leader: 1       Replicas: 1,2,0 Isr: 1,2
        Topic: topic02  Partition: 2    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2

[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --alter --topic topic02 --partitions 4

[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --describe --topic topic02
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Topic: topic02  TopicId: _s44mBQqTLaSmVWcZA2GTw PartitionCount: 4       ReplicationFactor: 3    Configs:
        Topic: topic02  Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,1
        Topic: topic02  Partition: 1    Leader: 1       Replicas: 1,2,0 Isr: 1,2
        Topic: topic02  Partition: 2    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2
        Topic: topic02  Partition: 3    Leader: 2       Replicas: 2,1,0 Isr: 2,1

[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
topic01
topic02
[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --delete --topic topic02
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-topics.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
topic01

---
# 生产者
[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-console-producer.sh --broker-list kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>123
>123
>key=123
>


# 消费者
[root@kafka_1 kafka_2.13-3.5.1]# ./bin/kafka-console-consumer.sh --bootstrap-server kafka_1:9092,kafka_2:9092,kafka_3:9092 --topic topic01 --group g1 --property print.key=true  --property print.value=true  --property key.separator=,
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N



null,123
null,123
null,key=123
相关推荐
小白学大数据2 小时前
Python+Selenium爬虫：豆瓣登录反反爬策略解析
分布式·爬虫·python·selenium
fjkxyl2 小时前
Kafka消息路由分区机制深度解析：架构设计与实现原理
分布式·kafka
只因只因爆2 小时前
spark数据清洗
大数据·分布式·spark
好吃的肘子4 小时前
ElasticSearch进阶
大数据·开发语言·分布式·算法·elasticsearch·kafka·jenkins
信徒_4 小时前
Kafka 中过多的 topic 导致整体上性能变慢的原因
分布式·kafka
松树戈4 小时前
本地 PC 使用Offset Explorer连接实体Ubuntu Kafka 【单机】超时问题解决
linux·ubuntu·kafka
MZWeiei6 小时前
Spark Streaming 内部运行机制详解
大数据·分布式·spark
yuanlaile15 小时前
RabbitMQ高并发秒杀、抢购系统、预约系统底层实现逻辑
分布式·rabbitmq·rabbitmq高并发·rabbitmq项目实战·rabbitmq实战教程
MYBOYER16 小时前
Kafka、RabbitMQ、RocketMQ的区别
kafka·rabbitmq·rocketmq
StarRocks_labs17 小时前
从InfluxDB到StarRocks：Grab实现Spark监控平台10倍性能提升
大数据·数据库·starrocks·分布式·spark·iris·物化视图
热门推荐
01KGG转MP3工具|非KGM文件|解密音频02YOLOv8入门 | 重要性能衡量指标、训练结果评价及分析及影响mAP的因素【发论文关注的指标】03从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑04【SpeedAI科研小助手】2分钟极速解决知网维普重复率、AIGC率过高，一键全文降！文件格式不变，公式都保留的！05Coze扣子平台完整体验和实践（附国内和国际版对比）06DeepSeek各版本说明与优缺点分析07【解决】Android Gradle Sync 报错 Could not read workspace metadata08YOLOv5改进 | 添加CA注意力机制 + 增加预测层 + 更换损失函数之GIoU09苍穹外卖面试总结10yolov10来了！用yolov10训练自己的数据集（原理、训练、部署、应用）