监听数据库binlog日志变化，将变动实时发送到kafka

1. 技术栈

mysql 5.7.33
Starrocks 4.0.1
kafka 2.8.2
Debezium 1.9.7.Final

2. mysql

修改 my.cnf

cnf 复制代码

# 每个MySQL实例唯一
server-id = 1
# binlog文件名前缀
log_bin = mysql-bin
# binlog日志格式
binlog_format = ROW
# 捕获完整的行数据
binlog_row_image = FULL
# 记录SQL语句
binlog_rows_query_log_events = ON
# GTID
gtid_mode = ON
enforce_gtid_consistency = ON
# 日志保留策略
expire_logs_days = 1
# 单个binlog大小
max_binlog_size = 128M
# 表示多少个事务提交后刷盘一次
sync_binlog = 1

修改完记得重启下

可以创建一个专门的用户 debezium，然后赋予其全部权限

2. kafka

2.1 zookeeper

修改 zookeeper 的配置文件：

bash 复制代码

vim /opt/module/kafka-2.8.2/config/zookeeper.properties

此处只需要修改数据的存储路径即可

properties 复制代码

dataDir=/opt/module/kafka-2.8.2/zookeeper

2.2 kafka 配置文件

可以直接覆盖已有 kafka 的配置文件

bash 复制代码

vim /opt/module/kafka-2.8.2/config/server.properties

server.properties

properties 复制代码

# broker唯一id
broker.id=0

# 监听地址
listeners=PLAINTEXT://192.168.195.10:9092
# 对外advertised 地址
advertised.listeners=PLAINTEXT://192.168.195.10:9092

num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

log.dirs=/opt/module/kafka-2.8.2/kafka-logs

num.partitions=1
num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000


zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0

2.3 启动

bash 复制代码

cd /opt/module/kafka-2.8.2/
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
bin/kafka-server-start.sh -daemon config/server.properties

3. debezium

3.1 connect-standalone-debezium.properties

复制一份示例文件

bash 复制代码

cd /opt/module/kafka-2.8.2/config/
cp config/connect-standalone.properties /opt/module/kafka-2.8.2/config/connect-standalone-debezium.properties

connect-standalone-debezium.properties

properties 复制代码

# Kafka Connect配置
bootstrap.servers=192.168.195.10:9092

# key/value 序列化
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true

# 插件路径
plugin.path=/opt/module/kafka-2.8.2/libs

# 单机模式下，偏移量必须用文件存储
offset.storage.file.filename=/opt/module/kafka-2.8.2/data/debezium/connect-offsets.dat
config.storage.file.filename=/opt/module/kafka-2.8.2/data/debezium/connect-configs.dat
status.storage.file.filename=/opt/module/kafka-2.8.2/data/debezium/connect-status.dat

offset.flush.interval.ms=10000

提前创建文件

bash 复制代码

mkdir -p /opt/module/kafka-2.8.2/data/debezium
touch connect-offsets.dat
touch connect-configs.dat
touch connect-status.dat

3.2 mysql-debezium.properties

bash 复制代码

cd /opt/module/kafka-2.8.2/config/
vim mysql-debezium.properties

mysql-debezium.properties

properties 复制代码

name=mysql-connector
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=192.168.195.10
database.port=3306
database.user=debezium
database.password=password666*

# Kafka topic的前缀
database.server.name=mysql-debezium-server
# 只监控哪些数据库
database.include.list=javastu
# 指定表
table.include.list=javastu.sales
database.history.kafka.bootstrap.servers=192.168.195.10:9092
database.history.kafka.topic=schema-changes.inventory
include.schema.changes=true

# 让 Decimal 以字符串形式输出
decimal.handling.mode=precise

3.3 启动

bash 复制代码

cd /opt/module/kafka-2.8.2
nohup bin/connect-standalone.sh config/connect-standalone-debezium.properties config/mysql-debezium.properties > /opt/module/kafka-2.8.2/logs/debezium-connect-$(date +%Y%m%d).log 2>&1 &

查看所有的 topic

bash 复制代码

cd /opt/module/kafka-2.8.2
./bin/kafka-topics.sh --list --bootstrap-server localhost:9092

打开一个消费者的终端，然后修改一下监听的表里面的数据

bash 复制代码

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysql-debezium-server.javastu.sales