分布式 - 消息队列Kafka：项目中kafka配置文件和命令行操作

文章目录

• • [1. kafka 配置文件](#1. kafka 配置文件)
  • • [1.1 进入kafka的安装目录](#1.1 进入kafka的安装目录)
    • [1.2 查看kafka的配置文件](#1.2 查看kafka的配置文件)
    • [1.3 启动 kafka-0 中的 kafka实例](#1.3 启动 kafka-0 中的 kafka实例)
  • [2. kafka 主题](#2. kafka 主题)
  • • [2.1 创建主题并查看详情](#2.1 创建主题并查看详情)
    • [2.2 查看当前容器中的所有主题](#2.2 查看当前容器中的所有主题)
    • [2.3 删除主题](#2.3 删除主题)
  • [3. kafka 生产者](#3. kafka 生产者)
  • [4. kafka 消费者](#4. kafka 消费者)

1. kafka 配置文件

1.1 进入kafka的安装目录

① 获取 kafka pod 资源：

XDR2.0.36.0(10.65.196.232) ~ #  kubectl get pod -n kafka
NAME                READY   STATUS    RESTARTS         AGE
kafka-0             1/1     Running   8 (3d21h ago)    133d
kafka-1             1/1     Running   3 (3d21h ago)    75d
kafka-2             1/1     Running   8 (3d21h ago)    133d
kafka-zookeeper-0   1/1     Running   4 (3d21h ago)    280d
kafka-zookeeper-1   1/1     Running   1 (3d21h ago)    75d
kafka-zookeeper-2   1/1     Running   29 (3d21h ago)   286d

② 进入 kafka-0 容器：

XDR2.0.36.0(10.65.196.232) ~ # kubectl exec -it kafka-0 -n kafka -- bash

I have no name!@kafka-0:/$ ls
bin  bitnami  boot  dev  docker-entrypoint-initdb.d  entrypoint.sh  etc  home  lib  lib64  media  mnt  opt  proc  root  run  run.sh  sbin  scripts  srv  sys  tmp  usr  var

③ 进入 kafka-0 的安装目录：

I have no name!@kafka-0:/$ cd /opt/bitnami/kafka/bin/

I have no name!@kafka-0:/opt/bitnami/kafka/bin$ ls
connect-distributed.sh        kafka-configs.sh             kafka-delete-records.sh   kafka-preferred-replica-election.sh  kafka-server-stop.sh                windows
connect-mirror-maker.sh       kafka-console-consumer.sh    kafka-dump-log.sh         kafka-producer-perf-test.sh          kafka-streams-application-reset.sh  zookeeper-security-migration.sh
connect-standalone.sh         kafka-console-producer.sh    kafka-features.sh         kafka-reassign-partitions.sh         kafka-topics.sh                     zookeeper-server-start.sh
jaas.conf                     kafka-consumer-groups.sh     kafka-leader-election.sh  kafka-replica-verification.sh        kafka-verifiable-consumer.sh        zookeeper-server-stop.sh
kafka-acls.sh                 kafka-consumer-perf-test.sh  kafka-log-dirs.sh         kafka-run-class.sh                   kafka-verifiable-producer.sh        zookeeper-shell.sh
kafka-broker-api-versions.sh  kafka-delegation-tokens.sh   kafka-mirror-maker.sh     kafka-server-start.sh                trogdor.sh

I have no name!@kafka-0:/opt/bitnami/kafka/bin$ cd ../config

I have no name!@kafka-0:/opt/bitnami/kafka/config$ ls
connect-console-sink.properties    connect-file-sink.properties    connect-mirror-maker.properties  kafka_jaas.conf      server.properties       zookeeper.properties
connect-console-source.properties  connect-file-source.properties  connect-standalone.properties    log4j.properties     tools-log4j.properties
connect-distributed.properties     connect-log4j.properties        consumer.properties              producer.properties  trogdor.conf

I have no name!@kafka-0:/opt/bitnami/kafka/config$

1.2 查看kafka的配置文件

① kafka-0 的配置文件 server.properties

broker.id=0

# 允许外部端口连接
listeners=INTERNAL://:9093,CLIENT://:9092,EXTERNAL://:9094

# 代理地址
advertised.listeners=INTERNAL://kafka-0.kafka-headless.kafka.svc.cluster.local:9093,CLIENT://kafka-0.kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://10.65.196.104:32111

# kafka日志的存储目录
log.dirs=/bitnami/kafka/data

# 分区数量
num.partitions=3

# zookeeper的连接地址
zookeeper.connect=kafka-zookeeper

② kafka-1的配置文件 server.properties

kubectl exec -it kafka-1  -n kafka bash
cd /opt/bitnami/kafka/config

broker.id=1

# 允许外部端口连接
listeners=INTERNAL://:9093,CLIENT://:9092,EXTERNAL://:9094

# 代理地址
advertised.listeners=INTERNAL://kafka-1.kafka-headless.kafka.svc.cluster.local:9093,CLIENT://kafka-1.kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://10.65.196.104:32112

# kafka日志的存储目录
log.dirs=/bitnami/kafka/data

# 分区数量
num.partitions=3

# zookeeper的连接地址
zookeeper.connect=kafka-zookeeper

③ kafka-2 的配置文件 server.properties

kubectl exec -it kafka-2  -n kafka bash
cd /opt/bitnami/kafka/config

broker.id=2

# 允许外部端口连接
listeners=INTERNAL://:9093,CLIENT://:9092,EXTERNAL://:9094

# 代理地址
advertised.listeners=INTERNAL://kafka-2.kafka-headless.kafka.svc.cluster.local:9093,CLIENT://kafka-2.kafka-headless.kafka.svc.cluster.local:9092,EXTERNAL://10.65.196.104:32113

# kafka日志的存储目录
log.dirs=/bitnami/kafka/data

# 分区数量
num.partitions=3

# zookeeper的连接地址
zookeeper.connect=kafka-zookeeper

1.3 启动 kafka-0 中的 kafka实例

bin/kafka-server-start.sh config/server.properties

2. kafka 主题

2.1 创建主题并查看详情

I have no name!@kafka-2:/opt/bitnami/kafka/config$ kafka-topics.sh --bootstrape-server kafka-1.kafka-headless.kafka.svc.cluster.local:9093 --create --partitions 3 --replication-factor 3 --topic 7-29-test-1
Exception in thread "main" joptsimple.UnrecognizedOptionException: bootstrape-server is not a recognized option
        at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
        at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
        at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
        at joptsimple.OptionParser.parse(OptionParser.java:396)
        at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:688)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

注意：这里之所以无法识别 --bootstrape-server 参数是因为kafka的版本低于2.2，应该使用 --zookeeper 参数：

I have no name!@kafka-1:/opt/bitnami/kafka$ kafka-topics.sh --zookeeper kafka-zookeeper --create --partitions 3 --replication-factor 3 --topic 7-29-test-1
Created topic 7-29-test-1.

I have no name!@kafka-1:/opt/bitnami/kafka$ kafka-topics.sh --zookeeper kafka-zookeeper --describe --topic 7-29-test-1
Topic: 7-29-test-1      PartitionCount: 3       ReplicationFactor: 3    Configs:
Topic: 7-29-test-1      Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2
Topic: 7-29-test-1      Partition: 1    Leader: 1       Replicas: 1,2,0 Isr: 1,2,0
Topic: 7-29-test-1      Partition: 2    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1
        
I have no name!@kafka-1:/opt/bitnami/kafka$ kafka-topics.sh --zookeeper kafka-zookeeper --create --partitions 3 --replication-factor 2 --topic 7-29-test-2
Created topic 7-29-test-2.

I have no name!@kafka-1:/opt/bitnami/kafka$ kafka-topics.sh --zookeeper kafka-zookeeper --describe --topic 7-29-test-2
Topic: 7-29-test-2      PartitionCount: 3       ReplicationFactor: 2    Configs:
Topic: 7-29-test-2      Partition: 0    Leader: 2       Replicas: 2,1   Isr: 2,1
Topic: 7-29-test-2      Partition: 1    Leader: 0       Replicas: 0,2   Isr: 0,2
Topic: 7-29-test-2      Partition: 2    Leader: 1       Replicas: 1,0   Isr: 1,0
        
I have no name!@kafka-1:/opt/bitnami/kafka$

其中，--zookeeper 参数用于指定 Kafka 集群中的 ZooKeeper 服务的地址。这个地址在kafka的配置文件config/server.properties 的 zookeeper.connect=kafka-zookeeper 属性中指定。

2.2 查看当前容器中的所有主题

I have no name!@kafka-1:/opt/bitnami/kafka$ kafka-topics.sh --zookeeper kafka-zookeeper --list
7-29
7-29-test
7-29-test-1
7-29-test-2

I have no name!@kafka-1:/opt/bitnami/kafka$

2.3 删除主题

I have no name!@kafka-1:/opt/bitnami/kafka$ kafka-topics.sh --zookeeper kafka-zookeeper --delete --topic 7-29-test-1
Topic 7-29-test-1 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

3. kafka 生产者

下面三种当时都可以：

I have no name!@kafka-1:/opt/bitnami/kafka/bin$ ./kafka-console-producer.sh  --producer.config ./jaas.conf --broker-list kafka.kafka.svc.cluster.local:9092 --topic  7-29-test-2
>

I have no name!@kafka-1:/opt/bitnami/kafka/bin$ ./kafka-console-producer.sh  --producer.config ./jaas.conf --broker-list kafka-1.kafka-headless.kafka.svc.cluster.local:9093 --topic  7-29-test-2
>

I have no name!@kafka-1:/opt/bitnami/kafka/bin$ ./kafka-console-producer.sh  --producer.config ./jaas.conf --broker-list kafka.kafka.svc.cluster.local:9092 --topic  7-29-test-2
>

--producer.config 参数指定的认证文件 jaas.conf ：

(1) 获取认证用户名和密码：

XDR2.0.30.0(10.65.196.232) ~ # kubectl get secret -n xdr-comm-config root-account -o jsonpath="{.data.account}" | base64 --decode | grep kafka_client
 "kafka_client_password": "UmDlbDB3UqTLDeXxZW67",
 "kafka_client_username": "user1",

(2) 进入bin目录并生成 jaas.conf 认证文件：

kubectl exec -it kafka-0 -n kafka -- bash 
cd /opt/bitnami/kafka/bin/

cat > jaas.conf << EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user1" password="UmDlbDB3UqTLDeXxZW67";
EOF

--broker-list 参数指定了Kafka生产者连接的broker列表。查看 kafka 的配置文件config/server.properties ：

(1) 在配置文件中查找advertised.listeners或listeners属性，这个属性定义了Kafka broker监听的地址和端口。

(2) 如果advertised.listeners属性存在，使用它的值作为broker列表的连接地址和端口。如果不存在，使用listeners属性的值。

4. kafka 消费者

./kafka-console-consumer.sh --consumer.config ./jaas.conf --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic persistent://10001001/default/xdr_dealstatus_sync --from-beginning --property print.timestamp=true 

--consumer.config 参数指定的认证文件 jaas.conf ：

(1) 获取认证用户名和密码：

XDR2.0.30.0(10.65.196.232) ~ # kubectl get secret -n xdr-comm-config root-account -o jsonpath="{.data.account}" | base64 --decode | grep kafka_client
 "kafka_client_password": "UmDlbDB3UqTLDeXxZW67",
 "kafka_client_username": "user1",

(2) 进入bin目录并生成 jaas.conf 认证文件：

kubectl exec -it kafka-0 -n kafka -- bash 
cd /opt/bitnami/kafka/bin/

cat > jaas.conf << EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user1" password="UmDlbDB3UqTLDeXxZW67";
EOF