中间件：maxwell、canal

丁总学Java2024-09-23 22:13

文章目录

  • [1、底层原理：基于mysql的bin log日志实现的：把自己伪装成slave](#1、底层原理：基于mysql的bin log日志实现的：把自己伪装成slave)
  • [2、bin log 日志有三种模式：](#2、bin log 日志有三种模式：)
  • [3、maxwell只支持 row 模式：](#3、maxwell只支持 row 模式：)
  • 4、maxwell介绍
  • 5、maxwell入门
  • 6、拉取maxwell镜像命令如下
  • 7、配置数据库mysql
    • [7.1、在/var/lib/docker/volumes/mysql_conf/_data目录下创建 my.cnf](#7.1、在/var/lib/docker/volumes/mysql_conf/_data目录下创建 my.cnf)
    • [7.2、查看 my.cnf](#7.2、查看 my.cnf)
    • [7.3、编辑 my.cnf](#7.3、编辑 my.cnf)
    • [7.4、mysql 中创建 maxwell 用户](#7.4、mysql 中创建 maxwell 用户)
  • [8、重启 mysql](#8、重启 mysql)
  • [9、启动一个名为 zendesk/maxwell 的 Docker 容器，并配置 Maxwell 以监听 MySQL 数据库的变化并将这些变化输出到标准输出（stdout）](#9、启动一个名为 zendesk/maxwell 的 Docker 容器，并配置 Maxwell 以监听 MySQL 数据库的变化并将这些变化输出到标准输出（stdout）)
  • [10、修改 tingshu_album 数据库](#10、修改 tingshu_album 数据库)
  • 11、Maxwell正常关闭所有任务

1、底层原理：基于mysql的bin log日志实现的：把自己伪装成slave

  1. 所有的写操作到master主机，master会记录数据变化到 bin log 日志
  2. slave会通过 IO 线程 通过slave用户和master建立链接，并拉取bin log 日志的内容记录到自己的 relay log 中
  3. slave通过sql 线程 读取 relay log 中的内容进行 replay 重演重做，进而完成数据同步

2、bin log 日志有三种模式：

2.1、statement模式：

把 sql 语句记录到 bin log 日志。问题：当sql中有系统函数时，就会出现数据不一致。

2.2、row模式：

把变化后的数据记录到 bin log 日志。解决数据不一致问题，问题：批量操作时，使日志很大。

2.3、mixed模式：

智能选择适合的模式记录到 bin log 日志。当有系统函数时会自动选择row模式，当有批量操作时自动选择statement模式（推荐）。

3、maxwell只支持 row 模式：

会通过row模式获取 bin log 修改后的数据转化成 json 输出

4、maxwell介绍

maxwell的github地址：https://github.com/zendesk/maxwell


maxwells官网：https://maxwells-daemon.io/

Quick Start - Maxwell's Daemon：https://maxwells-daemon.io/quickstart/

5、maxwell入门

6、拉取maxwell镜像命令如下

cpp 复制代码 
[root@localhost docker]# docker pull zendesk/maxwell
Using default tag: latest
latest: Pulling from zendesk/maxwell
1efc276f4ff9: Pull complete 
a2f2f93da482: Pull complete 
12cca292b13c: Pull complete 
69e15dccd787: Pull complete 
79219af6aa7c: Pull complete 
f39f1bdf1c84: Pull complete 
3261018f1785: Pull complete 
4f4fb700ef54: Pull complete 
be1353da9f00: Pull complete 
627d862c87f8: Pull complete 
Digest: sha256:68d51e27b6de2315ea710c0fe88972d4bd246ffb2519c82a49aa90a980d6cf64
Status: Downloaded newer image for zendesk/maxwell:latest
docker.io/zendesk/maxwell:latest

7、配置数据库mysql

cpp 复制代码 
# /etc/my.cnf

[mysqld]
# maxwell needs binlog_format=row
binlog_format=row
server_id=1 
log-bin=master
cpp 复制代码 
[root@localhost etc]# docker inspect spzx-mysql
cpp 复制代码 
        "Mounts": [
            {
                "Type": "volume",
                "Name": "mysql_data",
                "Source": "/var/lib/docker/volumes/mysql_data/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "mysql_conf",
                "Source": "/var/lib/docker/volumes/mysql_conf/_data",
                "Destination": "/etc/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
cpp 复制代码 
/var/lib/docker/volumes/mysql_conf/_data

7.1、在/var/lib/docker/volumes/mysql_conf/_data目录下创建 my.cnf

sh 复制代码 
[root@localhost ~]# cd /var/lib/docker/volumes/mysql_conf/_data
[root@localhost _data]# ll
总用量 8
drwxrwxr-x. 2 root root   41 12月 26 2023 conf.d
-rw-rw-r--. 1 root root 1543 9月  13 17:35 my.cnf
-rw-r--r--. 1 root root 1448 9月  28 2021 my.cnf.fallback

此时发现已经有文件 my.cnf，这个文件是我在搭建mysql主从复制时创建的。

7.2、查看 my.cnf

sh 复制代码 
[root@localhost _data]# cat my.cnf
# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL  Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[mysqld]
# 服务器唯一id，默认值1
server-id=1
# # 设置日志格式，默认值ROW。row（记录行数据）  statement（记录sql）  mixed（混合模式）
binlog_format=STATEMENT
# # 二进制日志名，默认binlog
# # log-bin=binlog
log-bin=spzxbinlog
# # 设置需要复制的数据库，默认复制全部数据库
binlog-do-db=mydb2
binlog-do-db=mydb3
# # 设置不需要复制的数据库
binlog-ignore-db=mydb4
# #binlog-ignore-db=infomation_schema
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
secure-file-priv= NULL

# Custom config should go here
!includedir /etc/mysql/conf.d/

7.3、编辑 my.cnf

sh 复制代码 
[mysqld]
# 服务器唯一id，默认值1
server-id=1
# # 设置日志格式，默认值ROW。row（记录行数据）  statement（记录sql）  mixed（混合模式）
binlog_format=row
# # 二进制日志名，默认binlog
# # log-bin=binlog
log-bin=spzxbinlog
# # 设置需要复制的数据库，默认复制全部数据库
binlog-do-db=mydb2
binlog-do-db=mydb3
binlog-do-db=tingshu_album
# # 设置不需要复制的数据库
binlog-ignore-db=mydb4
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
binlog-ignore-db=information_schema
# #binlog-ignore-db=infomation_schema
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
secure-file-priv= NULL

# Custom config should go here
!includedir /etc/mysql/conf.d/

7.4、mysql 中创建 maxwell 用户

sql 复制代码 
mysql> CREATE USER 'maxwell'@'%' IDENTIFIED BY 'XXXXXX';
mysql> CREATE USER 'maxwell'@'localhost' IDENTIFIED BY 'XXXXXX';

mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'%';
mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'localhost';

mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'%';
mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'localhost';

7.4.1、在 docker 中连接 mysql

sh 复制代码 
[root@localhost _data]# docker exec -it spzx-mysql /bin/bash
root@ab66508d9441:/# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.27 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

7.4.2、创建 maxwell 用户

sql 复制代码 
mysql> CREATE USER 'maxwell'@'%' IDENTIFIED BY 'maxwell';
Query OK, 0 rows affected (0.11 sec)

7.4.3、授权用户maxwell从任何主机（%代表任何主机）连接到MySQL服务器，并对名为maxwell的数据库拥有所有权限（ALL）

sql 复制代码 
mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'%';
Query OK, 0 rows affected (0.01 sec)

7.4.4、授权用户maxwell从任何主机（%代表任何主机）连接到MySQL服务器，并赋予它三个特定的权限：SELECT、REPLICATION CLIENT和REPLICATION SLAVE

sql 复制代码 
mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'%';
Query OK, 0 rows affected (0.01 sec)

8、重启 mysql

sh 复制代码 
[root@localhost _data]# docker restart spzx-mysql 
spzx-mysql

9、启动一个名为 zendesk/maxwell 的 Docker 容器，并配置 Maxwell 以监听 MySQL 数据库的变化并将这些变化输出到标准输出（stdout）

sh 复制代码 
docker run -it --rm zendesk/maxwell bin/maxwell --user=$MYSQL_USERNAME \
    --password=$MYSQL_PASSWORD --host=$MYSQL_HOST --producer=stdout
sh 复制代码 
docker run -it --rm zendesk/maxwell bin/maxwell --user=maxwell \
    --password=maxwell --host=192.168.74.148 --port=3306 --producer=stdout
sh 复制代码 
[root@localhost _data]# docker run -it --rm zendesk/maxwell bin/maxwell --user=maxwell \
>     --password=maxwell --host=192.168.74.148 --port=3306 --producer=stdout
2024-09-19 09:08:15 INFO  Maxwell - Starting Maxwell. maxMemory: 1031798784 bufferMemoryUsage: 0.25
2024-09-19 09:08:15 INFO  SchemaStoreSchema - Creating maxwell database
2024-09-19 09:08:15 INFO  Maxwell - Maxwell v1.41.2 is booting (StdoutProducer), starting at Position[BinlogPosition[spzxbinlog.000003:156], lastHeartbeat=0]
2024-09-19 09:08:16 INFO  AbstractSchemaStore - Maxwell is capturing initial schema
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: spzxbinlog.000003:156
2024-09-19 09:08:17 INFO  BinaryLogClient - Connected to 192.168.74.148:3306 at spzxbinlog.000003/156 (sid:6379, cid:23)
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Binlog connected.
cpp 复制代码 
2024-09-19 09:08:15 INFO  SchemaStoreSchema - Creating maxwell database

10、修改 tingshu_album 数据库


sh 复制代码 
[root@localhost _data]# docker run -it --rm zendesk/maxwell bin/maxwell --user=maxwell \
>     --password=maxwell --host=192.168.74.148 --port=3306 --producer=stdout
2024-09-19 09:08:15 INFO  Maxwell - Starting Maxwell. maxMemory: 1031798784 bufferMemoryUsage: 0.25
2024-09-19 09:08:15 INFO  SchemaStoreSchema - Creating maxwell database
2024-09-19 09:08:15 INFO  Maxwell - Maxwell v1.41.2 is booting (StdoutProducer), starting at Position[BinlogPosition[spzxbinlog.000003:156], lastHeartbeat=0]
2024-09-19 09:08:16 INFO  AbstractSchemaStore - Maxwell is capturing initial schema
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: spzxbinlog.000003:156
2024-09-19 09:08:17 INFO  BinaryLogClient - Connected to 192.168.74.148:3306 at spzxbinlog.000003/156 (sid:6379, cid:23)
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Binlog connected.
{"database":"tingshu_album","table":"album_info","type":"update","ts":1726737222,"xid":1315,"commit":true,"data":{"id":1,"user_id":1,"album_title":"《夜色钢琴曲》maxwell","category3_id":1018,"album_intro":"《夜色钢琴曲》最新专辑上线啦 我的新专辑《夜色钢琴曲 最新专辑》（点击跳转）已经上线，新专辑是《夜...","cover_url":"https://imagev2.xmcdn.com/storages/b3d2-audiofreehighqps/91/8E/GMCoOSAFquG2AAU4zwEKNohZ.png","include_track_count":54,"is_finished":"0","estimated_track_count":164,"album_rich_intro":"<p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"tag\" style=\"padding:5px;margin:10px 0px;color:rgb(255, 255, 255);display:inline-block;\">《夜色钢琴曲》最新专辑上线啦</span> </strong></p><p style=\"color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:left;\" lang=\"en\" data-flag=\"normal\">我的新专辑<a href=\"https://www.ximalaya.com/yinyue/35219974/\" style=\"color:#4990E2;text-decoration:none;\"><b>《夜色钢琴曲 最新专辑》</b></a>（点击跳转）已经上线，新专辑是《夜色钢琴曲》的升级版，我精选了诸多经典原创作品与大家分享，愿未来的每一个夜晚，大家在钢琴曲的陪伴下，能够卸下身体的浮躁与焦虑，内心不再孤单与慌张。</p><span><br /></span><p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;text-align:justify;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"strong\" style=\"word-break:break-all;\">赵海洋出生于1988年3月31日，专业的钢琴、作曲、编曲、钢琴教师、作品轻盈舒畅，委婉通透，曲曲经典，让人沐在他的音乐月光下，洗涤凡尘心垢。相关曲谱高清音乐某宝搜索：8919005，微博：夜色钢琴赵海洋</span></strong></p><p data-flag=\"normal\" style=\"font-size:16px;line-height:30px;font-family:Helvetica, Arial, sans-serif;color:rgb(51, 51, 51);font-weight:normal;text-align:left;\" lang=\"en\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><img data-key=\"0\" src=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" alt=\"\" data-origin=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333.jpg\" data-large=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" data-large-width=\"750\" data-large-height=\"500\" data-preview=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_small.jpg\" data-preview-width=\"140\" data-preview-height=\"93\" /><br /><br /></strong></p>","quality_score":0.00,"pay_type":"0101","price_type":"0201","price":0.00,"discount":-1.0,"vip_discount":-1.0,"tracks_for_free":0,"seconds_for_free":0,"buy_notes":null,"selling_point":null,"is_open":"1","status":"0301","create_time":"2023-04-04 09:05:02","update_time":"2024-09-19 09:13:42","is_deleted":1},"old":{"album_title":"《夜色钢琴曲》1","update_time":"2024-04-24 11:18:48"}}
cpp 复制代码 
{
    "database": "tingshu_album",
    "table": "album_info",
    "type": "update",
    "ts": 1726737222,
    "xid": 1315,
    "commit": true,
    "data": {
        "id": 1,
        "user_id": 1,
        "album_title": "《夜色钢琴曲》maxwell",
        "category3_id": 1018,
        "album_intro": "《夜色钢琴曲》最新专辑上线啦 我的新专辑《夜色钢琴曲 最新专辑》（点击跳转）已经上线，新专辑是《夜...",
        "cover_url": "https://imagev2.xmcdn.com/storages/b3d2-audiofreehighqps/91/8E/GMCoOSAFquG2AAU4zwEKNohZ.png",
        "include_track_count": 54,
        "is_finished": "0",
        "estimated_track_count": 164,
        "album_rich_intro": "<p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"tag\" style=\"padding:5px;margin:10px 0px;color:rgb(255, 255, 255);display:inline-block;\">《夜色钢琴曲》最新专辑上线啦</span> </strong></p><p style=\"color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:left;\" lang=\"en\" data-flag=\"normal\">我的新专辑<a href=\"https://www.ximalaya.com/yinyue/35219974/\" style=\"color:#4990E2;text-decoration:none;\"><b>《夜色钢琴曲 最新专辑》</b></a>（点击跳转）已经上线，新专辑是《夜色钢琴曲》的升级版，我精选了诸多经典原创作品与大家分享，愿未来的每一个夜晚，大家在钢琴曲的陪伴下，能够卸下身体的浮躁与焦虑，内心不再孤单与慌张。</p><span><br /></span><p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;text-align:justify;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"strong\" style=\"word-break:break-all;\">赵海洋出生于1988年3月31日，专业的钢琴、作曲、编曲、钢琴教师、作品轻盈舒畅，委婉通透，曲曲经典，让人沐在他的音乐月光下，洗涤凡尘心垢。相关曲谱高清音乐某宝搜索：8919005，微博：夜色钢琴赵海洋</span></strong></p><p data-flag=\"normal\" style=\"font-size:16px;line-height:30px;font-family:Helvetica, Arial, sans-serif;color:rgb(51, 51, 51);font-weight:normal;text-align:left;\" lang=\"en\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><img data-key=\"0\" src=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" alt=\"\" data-origin=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333.jpg\" data-large=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" data-large-width=\"750\" data-large-height=\"500\" data-preview=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_small.jpg\" data-preview-width=\"140\" data-preview-height=\"93\" /><br /><br /></strong></p>",
        "quality_score": 0,
        "pay_type": "0101",
        "price_type": "0201",
        "price": 0,
        "discount": -1,
        "vip_discount": -1,
        "tracks_for_free": 0,
        "seconds_for_free": 0,
        "buy_notes": null,
        "selling_point": null,
        "is_open": "1",
        "status": "0301",
        "create_time": "2023-04-04 09:05:02",
        "update_time": "2024-09-19 09:13:42",
        "is_deleted": 1
    },
    "old": {
        "album_title": "《夜色钢琴曲》1",
        "update_time": "2024-04-24 11:18:48"
    }
}

11、Maxwell正常关闭所有任务

cpp 复制代码 
^C2024-09-19 09:31:56 INFO  MaxwellContext - Sending final heartbeat: 1726738316328
2024-09-19 09:32:01 WARN  MaxwellContext - Timed out waiting for heartbeat 1726738316328
2024-09-19 09:32:01 INFO  TaskManager - Stopping 3 tasks
2024-09-19 09:32:01 INFO  TaskManager - Stopping: com.zendesk.maxwell.schema.PositionStoreThread@7ff21fb0
2024-09-19 09:32:01 INFO  TaskManager - Stopping: com.zendesk.maxwell.bootstrap.BootstrapController@2ea6c604
2024-09-19 09:32:01 INFO  TaskManager - Stopping: com.zendesk.maxwell.replication.BinlogConnectorReplicator@24625899
2024-09-19 09:32:01 INFO  BinlogConnectorReplicator - Binlog disconnected.
2024-09-19 09:32:01 INFO  TaskManager - Stopped all tasks
相关推荐
阿昌喜欢吃黄桃17 天前
RocketMq事务消息原理
java·中间件·消息队列·rocketmq·mq
半夜修仙18 天前
延迟队列的介绍及常见问题
java·数据库·中间件·rabbitmq
手握风云-18 天前
一条消息的旅程：RabbitMQ 学习与实践（一）
中间件·rabbitmq
RH23121118 天前
2026.6.8Linux
java·数据库·中间件
理人综艺好会19 天前
双Token机制在实际项目中的应用与实践
中间件·token
番茄去哪了19 天前
神领物流面试题（一）
java·大数据·中间件
念何架构之路20 天前
消息中间件
中间件
都说名字长不会被发现20 天前
Spring Boot Starter 中间件账号密码加密方案设计与实现
java·spring boot·后端·中间件
瀚高PG实验室20 天前
java中间件无法连接数据库
java·数据库·中间件·瀚高数据库
之歆20 天前
Day11_Express 深入解析：从中间件到项目实战
中间件·express
热门推荐
012026 年 AI 编程工具终极横评：Cursor vs Claude Code vs Copilot vs Windsurf02GitHub 镜像站点03【AI】2026 年具身智能模型和世界模型总结042026 AI 编程工具终极实战指南：Cursor vs Claude Code vs Copilot，开发者该怎么选？052026年6月AI大模型全景报告：GPT-5.6、Claude Opus 4.8、Gemini 3.5，中美AI三足鼎立谁主沉浮？06Claude Code、Codex、Cursor三分天下：2026年AI编程Agent生态全景剖析072026 年 AI 大模型 & AI 编程工具实战全总结082026年6月AI行业全景：从百模大战到Agent元年，这30天发生了什么？09AI科技热点日报 | 2026年07月01日10Trae国际版与国内版深度测评：AI原生IDE的双生花