中间件：maxwell、canal

丁总学Java2024-09-23 22:13

文章目录

  • [1、底层原理：基于mysql的bin log日志实现的：把自己伪装成slave](#1、底层原理：基于mysql的bin log日志实现的：把自己伪装成slave)
  • [2、bin log 日志有三种模式：](#2、bin log 日志有三种模式：)
  • [3、maxwell只支持 row 模式：](#3、maxwell只支持 row 模式：)
  • 4、maxwell介绍
  • 5、maxwell入门
  • 6、拉取maxwell镜像命令如下
  • 7、配置数据库mysql
    • [7.1、在/var/lib/docker/volumes/mysql_conf/_data目录下创建 my.cnf](#7.1、在/var/lib/docker/volumes/mysql_conf/_data目录下创建 my.cnf)
    • [7.2、查看 my.cnf](#7.2、查看 my.cnf)
    • [7.3、编辑 my.cnf](#7.3、编辑 my.cnf)
    • [7.4、mysql 中创建 maxwell 用户](#7.4、mysql 中创建 maxwell 用户)
  • [8、重启 mysql](#8、重启 mysql)
  • [9、启动一个名为 zendesk/maxwell 的 Docker 容器，并配置 Maxwell 以监听 MySQL 数据库的变化并将这些变化输出到标准输出（stdout）](#9、启动一个名为 zendesk/maxwell 的 Docker 容器，并配置 Maxwell 以监听 MySQL 数据库的变化并将这些变化输出到标准输出（stdout）)
  • [10、修改 tingshu_album 数据库](#10、修改 tingshu_album 数据库)
  • 11、Maxwell正常关闭所有任务

1、底层原理：基于mysql的bin log日志实现的：把自己伪装成slave

  1. 所有的写操作到master主机，master会记录数据变化到 bin log 日志
  2. slave会通过 IO 线程 通过slave用户和master建立链接，并拉取bin log 日志的内容记录到自己的 relay log 中
  3. slave通过sql 线程 读取 relay log 中的内容进行 replay 重演重做，进而完成数据同步

2、bin log 日志有三种模式：

2.1、statement模式：

把 sql 语句记录到 bin log 日志。问题：当sql中有系统函数时，就会出现数据不一致。

2.2、row模式：

把变化后的数据记录到 bin log 日志。解决数据不一致问题，问题：批量操作时，使日志很大。

2.3、mixed模式：

智能选择适合的模式记录到 bin log 日志。当有系统函数时会自动选择row模式，当有批量操作时自动选择statement模式（推荐）。

3、maxwell只支持 row 模式：

会通过row模式获取 bin log 修改后的数据转化成 json 输出

4、maxwell介绍

maxwell的github地址：https://github.com/zendesk/maxwell


maxwells官网：https://maxwells-daemon.io/

Quick Start - Maxwell's Daemon：https://maxwells-daemon.io/quickstart/

5、maxwell入门

6、拉取maxwell镜像命令如下

cpp 复制代码 
[root@localhost docker]# docker pull zendesk/maxwell
Using default tag: latest
latest: Pulling from zendesk/maxwell
1efc276f4ff9: Pull complete 
a2f2f93da482: Pull complete 
12cca292b13c: Pull complete 
69e15dccd787: Pull complete 
79219af6aa7c: Pull complete 
f39f1bdf1c84: Pull complete 
3261018f1785: Pull complete 
4f4fb700ef54: Pull complete 
be1353da9f00: Pull complete 
627d862c87f8: Pull complete 
Digest: sha256:68d51e27b6de2315ea710c0fe88972d4bd246ffb2519c82a49aa90a980d6cf64
Status: Downloaded newer image for zendesk/maxwell:latest
docker.io/zendesk/maxwell:latest

7、配置数据库mysql

cpp 复制代码 
# /etc/my.cnf

[mysqld]
# maxwell needs binlog_format=row
binlog_format=row
server_id=1 
log-bin=master
cpp 复制代码 
[root@localhost etc]# docker inspect spzx-mysql
cpp 复制代码 
        "Mounts": [
            {
                "Type": "volume",
                "Name": "mysql_data",
                "Source": "/var/lib/docker/volumes/mysql_data/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "mysql_conf",
                "Source": "/var/lib/docker/volumes/mysql_conf/_data",
                "Destination": "/etc/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
cpp 复制代码 
/var/lib/docker/volumes/mysql_conf/_data

7.1、在/var/lib/docker/volumes/mysql_conf/_data目录下创建 my.cnf

sh 复制代码 
[root@localhost ~]# cd /var/lib/docker/volumes/mysql_conf/_data
[root@localhost _data]# ll
总用量 8
drwxrwxr-x. 2 root root   41 12月 26 2023 conf.d
-rw-rw-r--. 1 root root 1543 9月  13 17:35 my.cnf
-rw-r--r--. 1 root root 1448 9月  28 2021 my.cnf.fallback

此时发现已经有文件 my.cnf，这个文件是我在搭建mysql主从复制时创建的。

7.2、查看 my.cnf

sh 复制代码 
[root@localhost _data]# cat my.cnf
# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL  Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[mysqld]
# 服务器唯一id，默认值1
server-id=1
# # 设置日志格式，默认值ROW。row（记录行数据）  statement（记录sql）  mixed（混合模式）
binlog_format=STATEMENT
# # 二进制日志名，默认binlog
# # log-bin=binlog
log-bin=spzxbinlog
# # 设置需要复制的数据库，默认复制全部数据库
binlog-do-db=mydb2
binlog-do-db=mydb3
# # 设置不需要复制的数据库
binlog-ignore-db=mydb4
# #binlog-ignore-db=infomation_schema
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
secure-file-priv= NULL

# Custom config should go here
!includedir /etc/mysql/conf.d/

7.3、编辑 my.cnf

sh 复制代码 
[mysqld]
# 服务器唯一id，默认值1
server-id=1
# # 设置日志格式，默认值ROW。row（记录行数据）  statement（记录sql）  mixed（混合模式）
binlog_format=row
# # 二进制日志名，默认binlog
# # log-bin=binlog
log-bin=spzxbinlog
# # 设置需要复制的数据库，默认复制全部数据库
binlog-do-db=mydb2
binlog-do-db=mydb3
binlog-do-db=tingshu_album
# # 设置不需要复制的数据库
binlog-ignore-db=mydb4
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
binlog-ignore-db=information_schema
# #binlog-ignore-db=infomation_schema
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
secure-file-priv= NULL

# Custom config should go here
!includedir /etc/mysql/conf.d/

7.4、mysql 中创建 maxwell 用户

sql 复制代码 
mysql> CREATE USER 'maxwell'@'%' IDENTIFIED BY 'XXXXXX';
mysql> CREATE USER 'maxwell'@'localhost' IDENTIFIED BY 'XXXXXX';

mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'%';
mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'localhost';

mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'%';
mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'localhost';

7.4.1、在 docker 中连接 mysql

sh 复制代码 
[root@localhost _data]# docker exec -it spzx-mysql /bin/bash
root@ab66508d9441:/# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.27 MySQL Community Server - GPL

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

7.4.2、创建 maxwell 用户

sql 复制代码 
mysql> CREATE USER 'maxwell'@'%' IDENTIFIED BY 'maxwell';
Query OK, 0 rows affected (0.11 sec)

7.4.3、授权用户maxwell从任何主机（%代表任何主机）连接到MySQL服务器，并对名为maxwell的数据库拥有所有权限（ALL）

sql 复制代码 
mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'%';
Query OK, 0 rows affected (0.01 sec)

7.4.4、授权用户maxwell从任何主机（%代表任何主机）连接到MySQL服务器，并赋予它三个特定的权限：SELECT、REPLICATION CLIENT和REPLICATION SLAVE

sql 复制代码 
mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@'%';
Query OK, 0 rows affected (0.01 sec)

8、重启 mysql

sh 复制代码 
[root@localhost _data]# docker restart spzx-mysql 
spzx-mysql

9、启动一个名为 zendesk/maxwell 的 Docker 容器，并配置 Maxwell 以监听 MySQL 数据库的变化并将这些变化输出到标准输出（stdout）

sh 复制代码 
docker run -it --rm zendesk/maxwell bin/maxwell --user=$MYSQL_USERNAME \
    --password=$MYSQL_PASSWORD --host=$MYSQL_HOST --producer=stdout
sh 复制代码 
docker run -it --rm zendesk/maxwell bin/maxwell --user=maxwell \
    --password=maxwell --host=192.168.74.148 --port=3306 --producer=stdout
sh 复制代码 
[root@localhost _data]# docker run -it --rm zendesk/maxwell bin/maxwell --user=maxwell \
>     --password=maxwell --host=192.168.74.148 --port=3306 --producer=stdout
2024-09-19 09:08:15 INFO  Maxwell - Starting Maxwell. maxMemory: 1031798784 bufferMemoryUsage: 0.25
2024-09-19 09:08:15 INFO  SchemaStoreSchema - Creating maxwell database
2024-09-19 09:08:15 INFO  Maxwell - Maxwell v1.41.2 is booting (StdoutProducer), starting at Position[BinlogPosition[spzxbinlog.000003:156], lastHeartbeat=0]
2024-09-19 09:08:16 INFO  AbstractSchemaStore - Maxwell is capturing initial schema
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: spzxbinlog.000003:156
2024-09-19 09:08:17 INFO  BinaryLogClient - Connected to 192.168.74.148:3306 at spzxbinlog.000003/156 (sid:6379, cid:23)
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Binlog connected.
cpp 复制代码 
2024-09-19 09:08:15 INFO  SchemaStoreSchema - Creating maxwell database

10、修改 tingshu_album 数据库


sh 复制代码 
[root@localhost _data]# docker run -it --rm zendesk/maxwell bin/maxwell --user=maxwell \
>     --password=maxwell --host=192.168.74.148 --port=3306 --producer=stdout
2024-09-19 09:08:15 INFO  Maxwell - Starting Maxwell. maxMemory: 1031798784 bufferMemoryUsage: 0.25
2024-09-19 09:08:15 INFO  SchemaStoreSchema - Creating maxwell database
2024-09-19 09:08:15 INFO  Maxwell - Maxwell v1.41.2 is booting (StdoutProducer), starting at Position[BinlogPosition[spzxbinlog.000003:156], lastHeartbeat=0]
2024-09-19 09:08:16 INFO  AbstractSchemaStore - Maxwell is capturing initial schema
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: spzxbinlog.000003:156
2024-09-19 09:08:17 INFO  BinaryLogClient - Connected to 192.168.74.148:3306 at spzxbinlog.000003/156 (sid:6379, cid:23)
2024-09-19 09:08:17 INFO  BinlogConnectorReplicator - Binlog connected.
{"database":"tingshu_album","table":"album_info","type":"update","ts":1726737222,"xid":1315,"commit":true,"data":{"id":1,"user_id":1,"album_title":"《夜色钢琴曲》maxwell","category3_id":1018,"album_intro":"《夜色钢琴曲》最新专辑上线啦 我的新专辑《夜色钢琴曲 最新专辑》（点击跳转）已经上线，新专辑是《夜...","cover_url":"https://imagev2.xmcdn.com/storages/b3d2-audiofreehighqps/91/8E/GMCoOSAFquG2AAU4zwEKNohZ.png","include_track_count":54,"is_finished":"0","estimated_track_count":164,"album_rich_intro":"<p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"tag\" style=\"padding:5px;margin:10px 0px;color:rgb(255, 255, 255);display:inline-block;\">《夜色钢琴曲》最新专辑上线啦</span> </strong></p><p style=\"color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:left;\" lang=\"en\" data-flag=\"normal\">我的新专辑<a href=\"https://www.ximalaya.com/yinyue/35219974/\" style=\"color:#4990E2;text-decoration:none;\"><b>《夜色钢琴曲 最新专辑》</b></a>（点击跳转）已经上线，新专辑是《夜色钢琴曲》的升级版，我精选了诸多经典原创作品与大家分享，愿未来的每一个夜晚，大家在钢琴曲的陪伴下，能够卸下身体的浮躁与焦虑，内心不再孤单与慌张。</p><span><br /></span><p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;text-align:justify;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"strong\" style=\"word-break:break-all;\">赵海洋出生于1988年3月31日，专业的钢琴、作曲、编曲、钢琴教师、作品轻盈舒畅，委婉通透，曲曲经典，让人沐在他的音乐月光下，洗涤凡尘心垢。相关曲谱高清音乐某宝搜索：8919005，微博：夜色钢琴赵海洋</span></strong></p><p data-flag=\"normal\" style=\"font-size:16px;line-height:30px;font-family:Helvetica, Arial, sans-serif;color:rgb(51, 51, 51);font-weight:normal;text-align:left;\" lang=\"en\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><img data-key=\"0\" src=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" alt=\"\" data-origin=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333.jpg\" data-large=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" data-large-width=\"750\" data-large-height=\"500\" data-preview=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_small.jpg\" data-preview-width=\"140\" data-preview-height=\"93\" /><br /><br /></strong></p>","quality_score":0.00,"pay_type":"0101","price_type":"0201","price":0.00,"discount":-1.0,"vip_discount":-1.0,"tracks_for_free":0,"seconds_for_free":0,"buy_notes":null,"selling_point":null,"is_open":"1","status":"0301","create_time":"2023-04-04 09:05:02","update_time":"2024-09-19 09:13:42","is_deleted":1},"old":{"album_title":"《夜色钢琴曲》1","update_time":"2024-04-24 11:18:48"}}
cpp 复制代码 
{
    "database": "tingshu_album",
    "table": "album_info",
    "type": "update",
    "ts": 1726737222,
    "xid": 1315,
    "commit": true,
    "data": {
        "id": 1,
        "user_id": 1,
        "album_title": "《夜色钢琴曲》maxwell",
        "category3_id": 1018,
        "album_intro": "《夜色钢琴曲》最新专辑上线啦 我的新专辑《夜色钢琴曲 最新专辑》（点击跳转）已经上线，新专辑是《夜...",
        "cover_url": "https://imagev2.xmcdn.com/storages/b3d2-audiofreehighqps/91/8E/GMCoOSAFquG2AAU4zwEKNohZ.png",
        "include_track_count": 54,
        "is_finished": "0",
        "estimated_track_count": 164,
        "album_rich_intro": "<p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"tag\" style=\"padding:5px;margin:10px 0px;color:rgb(255, 255, 255);display:inline-block;\">《夜色钢琴曲》最新专辑上线啦</span> </strong></p><p style=\"color:#333333;font-weight:normal;font-size:16px;line-height:30px;font-family:Helvetica,Arial,sans-serif;hyphens:auto;text-align:left;\" lang=\"en\" data-flag=\"normal\">我的新专辑<a href=\"https://www.ximalaya.com/yinyue/35219974/\" style=\"color:#4990E2;text-decoration:none;\"><b>《夜色钢琴曲 最新专辑》</b></a>（点击跳转）已经上线，新专辑是《夜色钢琴曲》的升级版，我精选了诸多经典原创作品与大家分享，愿未来的每一个夜晚，大家在钢琴曲的陪伴下，能够卸下身体的浮躁与焦虑，内心不再孤单与慌张。</p><span><br /></span><p data-flag=\"normal\" style=\"line-height:30px;font-family:Helvetica, Arial, sans-serif;text-align:justify;\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><span data-flag=\"strong\" style=\"word-break:break-all;\">赵海洋出生于1988年3月31日，专业的钢琴、作曲、编曲、钢琴教师、作品轻盈舒畅，委婉通透，曲曲经典，让人沐在他的音乐月光下，洗涤凡尘心垢。相关曲谱高清音乐某宝搜索：8919005，微博：夜色钢琴赵海洋</span></strong></p><p data-flag=\"normal\" style=\"font-size:16px;line-height:30px;font-family:Helvetica, Arial, sans-serif;color:rgb(51, 51, 51);font-weight:normal;text-align:left;\" lang=\"en\"><strong style=\"color: rgb(252, 88, 50); word-break: break-all; font-family: Helvetica, Arial, sans-serif; font-weight: normal;\"><img data-key=\"0\" src=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" alt=\"\" data-origin=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333.jpg\" data-large=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_large.jpg\" data-large-width=\"750\" data-large-height=\"500\" data-preview=\"http://fdfs.xmcdn.com/group28/M05/5E/06/wKgJXFknzliyDqDtAAMkXHYILXQ333_mobile_small.jpg\" data-preview-width=\"140\" data-preview-height=\"93\" /><br /><br /></strong></p>",
        "quality_score": 0,
        "pay_type": "0101",
        "price_type": "0201",
        "price": 0,
        "discount": -1,
        "vip_discount": -1,
        "tracks_for_free": 0,
        "seconds_for_free": 0,
        "buy_notes": null,
        "selling_point": null,
        "is_open": "1",
        "status": "0301",
        "create_time": "2023-04-04 09:05:02",
        "update_time": "2024-09-19 09:13:42",
        "is_deleted": 1
    },
    "old": {
        "album_title": "《夜色钢琴曲》1",
        "update_time": "2024-04-24 11:18:48"
    }
}

11、Maxwell正常关闭所有任务

cpp 复制代码 
^C2024-09-19 09:31:56 INFO  MaxwellContext - Sending final heartbeat: 1726738316328
2024-09-19 09:32:01 WARN  MaxwellContext - Timed out waiting for heartbeat 1726738316328
2024-09-19 09:32:01 INFO  TaskManager - Stopping 3 tasks
2024-09-19 09:32:01 INFO  TaskManager - Stopping: com.zendesk.maxwell.schema.PositionStoreThread@7ff21fb0
2024-09-19 09:32:01 INFO  TaskManager - Stopping: com.zendesk.maxwell.bootstrap.BootstrapController@2ea6c604
2024-09-19 09:32:01 INFO  TaskManager - Stopping: com.zendesk.maxwell.replication.BinlogConnectorReplicator@24625899
2024-09-19 09:32:01 INFO  BinlogConnectorReplicator - Binlog disconnected.
2024-09-19 09:32:01 INFO  TaskManager - Stopped all tasks
相关推荐
谷大羽18 小时前
Kafka Stream实战教程
spring boot·后端·中间件·kafka·stream
隔着天花板看星星1 天前
Kafka-创建topic源码
大数据·分布式·中间件·kafka
get2002 天前
Gin 框架中间件详细介绍
中间件·gin
一水鉴天2 天前
智能工厂的设计软件 为了监管控一体化的全能Supervisor 的监督学习 之 序5 架构for认知系统 总述 （架构全图）
人工智能·学习·中间件·架构
谢尔登2 天前
【Next】中间件
服务器·javascript·中间件
极客先躯2 天前
高级java每日一道面试题-2024年11月09日-缓存中间件篇-Redis和Memecache有什么区别?
java·缓存·中间件·每日一道面试题·高级java·缓存中间件篇
隔着天花板看星星3 天前
Kafka-Controller角色需要做什么？
大数据·分布式·中间件·kafka
留乘船3 天前
使用gin -gorm-jwt-中间件拦截的一个小项目
开发语言·学习·中间件·golang·gin
Amd7945 天前
Nuxt.js 应用中的 vite：serverCreated 事件钩子
中间件·开发·vite·日志·nuxt·跨域·钩子
0x派大星6 天前
【Goland】——Gin 框架中间件详解：从基础到实战
开发语言·后端·中间件·golang·go·gin
热门推荐
01(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明02PyTorch机器学习实现液态神经网络03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04玄机平台应急响应—webshell查杀05Coze扣子平台完整体验和实践（附国内和国际版对比）06Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO07【目标跟踪】相机运动补偿08Unity中PICO实现 隔空取物 和 接触抓取物体09RAG 实践- Ollama+RagFlow 部署本地知识库10怎样让音频速度变慢？请跟随以下方法进行操作