一、常用shell命令
bash
#1、创建topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replications 1 --topic test
#2、查看创建的topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
#3、生产者发布消息命令
（执行完此命令后在控制台输入要发送的消息，回车即可）
bin/kafka-console-producer.sh --broker-list 192.168.91.231:9092,192.168.91.231:9093,192.168.91.231:9094 --topic test
#4、消费者接受消息命令
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
#5、kafka启动
首先启动zookeeper zkServer.sh start(相当于一个server,kafka会连接这个server)
bin/kafka-server-start.sh config/server.properties # 启动kafka
#6、查看kafka节点数目
在zookeeper中查看，登录客户端bin/zkCli.sh 执行ls /brokers/ids 查看节点数目及节点ID,[0，1，2]
#7、kafka中的概念
生产者 Producer、代理Broker、消费者Consumer、主题Topic、分区 Partition、消费者组 Consumer Group
#8、查看主颗信息
bin/kafka-topics.sh --zookeeper 192.168.91.231:2181 [加其他选项]
eg:
bin/kafka-topics.sh --zookeeper 192.168.91.231:2181 --describe
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
#9、为主题创建分区
一共创建八个分区，编号分别为0~7
bin/kafka-topics.sh --zookeeper 192.168.91.231:2181 --alter -partitions 8 -topic test
#10、查看kafka进程
ps -eflgrep server.properties
ps -eflgrep server-1.properties
ps -eflgrep server-2.properties
#11、kafka宕机重启后，消息不会丢失
#12、kafka其中一个broker宕机后，对消费者和生产者影响很小(命令行下测试)
消费者会尝试连接，连接不到，返回java.net.ConnectException:Connection refused异常 生产者可能会在发送消息的时候报异常，但会很快连接到其他broker，继续正常使用
#13.查看kafka消息队列的积压情况
bin/kafka-consumer-groups.sh --zookeeper 192.168.91.231:2181 --describe --group console-consumer-37289
#14.kafka 中查看所有的group列表信息
bin/kafka-consumer-groups.sh --zookeeper 192.168.91.231:2181 --list二、python操作kafka
本地安装与启动(基于Docker)
#1、下载zookeeper镜像与kafka镜像:
docker pull registry.cn-shanghai.aliyuncs.com/egon-k8s-test/kafka-zookeeper:3.4.6
docker pull registry.cn-shanghai.aliyuncs.com/egon-k8s-test/wurstmeister-kafka:2.13-2.8.1
#2、本地启动zookeeper
docker run -d --name zookeeper -p 2181:2181 -t registry.cn-shanghai.aliyuncs.com/egon-k8s-test/kafka-zookeeper:3.4.6
#3、本地启动kafka(注意下述代码，将kafka启动在9092端口)
docker run -d --name kafka --publish 9092:9092 --link zookeeper \
--enV KAFKA ZO0KEEPER CONNECT=zookeeper:2181 \
--enV KAFKA ADVERTISED HOST NAME=192.168.71.113 \
--enV KAFKA ADVERTISED PORT=9092 \
registry.cn-shanghai.aliyuncs.com/egon-k8s-test/wurstmeister-kafka:2.13-2.8.1
上面写的localhost没有影响，查看端口如下
netstat -tuanlp | grep 9092
tcp 0 0 0.0.0.0:9092 0.0.0.0:*LISTEN 102483/docker-proxy
tcp6 00:::9092 :::* LISTEN 102487/docker-proxy
#4、进入kafka bash
docker exec it kafka bash
cd /opt/kafka/bin
#5、创建Topic，分区为2，Topic name为'kafka_demo'
kafka-topics.sh --create --zookeeper zookeeper:2181 \
--replication-factor 1 --partitions 2 --topic kafka_demo
kafka-topics.sh --create --zookeeper zookeeper:2181 \
--replication-factor 1 --partitions 2 --topic egon
数据存在哪里
root@web02 \~# docker exec -it kafka bash
bash-5.1#
bash-5.1#
bash-5.1#
bash-5.1# ls /kafka/
kafka-logs-f33383f9c414
bash-5.1#
bash-5.1#
bash-5.1#
bash-5.1# 1s /kafka/kafka-logs-f33383f9c414/
kafka_demo-0 kafka_demo-1
egon-0 egon-1
.........
bash-5.1#
bash-5.1#
bash-5.1#
bash-5.1# ls /kafka/kafka-logs-f33383f9c414/egon-0
00000000000000000000.index0000000000000000000.timeindex
00000000800000080000.1og leader-epoch-checkpoint
#6、查看当前所有topic
kafka-topics.sh --zookeeper zookeeper:2181 --list
#7、命令行操作
$docker exec -ti kafka sh
/ # cd /opt/kafka/bin
/ # kafka-console-producer.sh --bootstrap-server 192.168.71.113:9092 --topic test_topic
然后一行行输入，回车即发送一条消息
>111
>222
>333
另外一个终端
$ docker exec -ti kafka sh
/ # cd /opt/kafka/bin
/ # kafka-console-consumer.sh --bootstrap-server 192.168.71.113:9092 --topic test_topic --from-beginning可以收到消息
111
222
333
#8、安装kafka-python
pip install kafka-python
代码示例：
python
# pip3 install kafka-python # 版本是2.0.2
from kafka import KafkaProducer, KafkaConsumer
import json
import threading
import time
# Kafka broker address
bootstrap_servers = '192.168.71.113:9092'
# Topic name
topic = 'test_topic'
# Producer function
def kafka_producer():
producer = KafkaProducer(bootstrap_servers=bootstrap_servers,
value_serializer=lambda v: json.dumps(v).encode('utf-8'))
try:
for i in range(10):
message = {'message': f'Hello Kafka! Message {i}'}
producer.send(topic, value=message)
print(f"Sent: {message}")
time.sleep(1)
else:
print("发送完成")
except Exception as ex:
print(f"Exception occurred: {ex}")
finally:
producer.close()
# Consumer function
def kafka_consumer():
consumer = KafkaConsumer(topic,
bootstrap_servers=bootstrap_servers,
auto_offset_reset='earliest',
consumer_timeout_ms=5000) # 设置超时时间为1秒
try:
for message in consumer:
print(f"Received: {message.value}")
else:
print("消费完毕，等5000毫秒超时即可结束，执行finally内的代码")
except Exception as ex:
print(f"Exception occurred: {ex}")
finally:
print("消费者结束")
consumer.close()
# Create threads for producer and consumer
producer_thread = threading.Thread(target=kafka_producer)
consumer_thread = threading.Thread(target=kafka_consumer)
# Start both threads
producer_thread.start()
consumer_thread.start()
# Wait for threads to complete
producer_thread.join()
consumer_thread.join()
print("Kafka producer and consumer threads have finished.")
执行结果：
