【binlog2sql实践】MySQL数据库binlog日志ROW格式转换标准SQL

IT成长日记2025-03-18 22:17

在MySQL数据库中，二进制日志（binlog）是记录所有更改数据库数据的操作日志。binlog的格式主要有三种：STATEMENT、ROW和MIXED。其中，ROW格式的binlog记录了每一行数据的变更情况，虽然这种格式的日志文件较大，但它能够提供更详细的数据变更信息，特别适合用于数据恢复和审计。

但是ROW格式的binlog并不便于直接用于数据恢复。为了将ROW格式的binlog转化为可读的标准SQL语句，我们可以使用binlog2sql工具。本文将介绍如何使用binlog2sql工具来恢复ROW格式的binlog并转化为标准SQL。

1 下载源码包和依赖

1.1 下载binlog2sql

github下载地址：https://github.com/danfengcao/binlog2sql

1.2 安装pyhont环境

复制代码 
# yum安装python
yum install python -y

# 安装pip
[root@node1 tool]# curl https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 29 1863k   29  558k    0     0   9642      0  0:03:17  0:00:59  0:02:18  6736
 53 1863k   53  990k    0     0   8426      0  0:03:46  0:02:00  0:01:46  6540
 80 1863k   80 1502k    0     0   8492      0  0:03:44  0:03:01  0:00:43  6818
100 1863k  100 1863k    0     0   7858      0  0:04:02  0:04:02 --:--:-- 10194
[root@node1 tool]# python get-pip.py
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pip<21.0
  Downloading pip-20.3.4-py2.py3-none-any.whl (1.5 MB)
     |██████                          | 286 kB 9.0 kB/s eta 0:02:18
     |████████████████▍               | 778 kB 8.5 kB/s eta 0:01:28
     |███████████████████████████████▋| 1.5 MB 8.8 kB/s eta 0:00:02
     |████████████████████████████████| 1.5 MB 12 kB/s 
Collecting wheel
  Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, wheel
Successfully installed pip-20.3.4 wheel-0.37.1
[root@node1 tool]# 

# 验证python 版本
[root@node1 ~]# python --version
Python 2.7.5
[root@node1 ~]# 

# 验证pip版本
[root@node1 ~]# pip --version
pip 20.3.4 from /usr/lib/python2.7/site-packages/pip (python 2.7)
[root@node1 ~]#

1.3 安装依赖包

复制代码 
[root@node1 tool]# pip install mysql-replication==0.21
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting mysql-replication==0.21
  Downloading mysql-replication-0.21.tar.gz (38 kB)
Collecting pymysql
  Downloading PyMySQL-0.10.1-py2.py3-none-any.whl (47 kB)
     |████████████████████████████████| 47 kB 10 kB/s 
Building wheels for collected packages: mysql-replication
  Building wheel for mysql-replication (setup.py) ... done
  Created wheel for mysql-replication: filename=mysql_replication-0.21-py2-none-any.whl size=42064 sha256=2b895385488863729b55290e6f00cce24c1ac47f3fb7a4639d477fade0b8fafb
  Stored in directory: /root/.cache/pip/wheels/c2/d2/9b/d875336da30aa3d725c9acd8649f71d13099aa2cba48600271
Successfully built mysql-replication
Installing collected packages: pymysql, mysql-replication
Successfully installed mysql-replication-0.21 pymysql-0.10.1
[root@node1 tool]#

2 binlog2sql解析ROW格式binlog

2.1 基本语法

复制代码 
python binlog2sql.py \
-h<host> \
-P<port> \
-u<user> \
-p<password> \
-d<database> \
-t<table> \
--start-file=<binlog_file> \
[--start-position=<start_pos>] \
[--stop-position=<end_pos>] \
[--start-datetime=<start_time>] \
[--stop-datetime=<end_time>]

参数说明：

  • -h：MySQL主机地址（默认：127.0.0.1）
  • -P：MySQL端口号（默认：3306）
  • -u：MySQL用户名
  • -p：MySQL密码
  • -d：数据库名称（可选）
  • -t：表名称（可选）
  • --start-file：binlog 文件名（如 mysql-bin.000001）
  • --start-position：binlog的起始位置（可选）
  • --stop-position：binlog的结束位置（可选）
  • --start-datetime：起始时间（格式：YYYY-MM-DD HH:MM:SS，可选）
  • --stop-datetime：结束时间（格式：YYYY-MM-DD HH:MM:SS，可选）

2.2 基于时间范围解析binlog

复制代码 
python /root/tool/binlog2sql-master/binlog2sql/binlog2sql.py \
-h127.0.0.1 \
-P3306 \
-uroot \
-p'password' \
-dmydb \
-ttmp_user \
--start-file='mysql-bin.000010' \
--start-datetime='2025-03-17 00:00:00' \
--stop-datetime='2025-03-17 00:00:00' >/data/mysql/backup/backup.sql

2.3 基于位置范围解析binlog

复制代码 
python /root/tool/binlog2sql-master/binlog2sql/binlog2sql.py \
-h127.0.0.1 \
-P3306 \
-uroot \
-p'password' \
-dmydb \
-ttmp_user \
--start-file='mysql-bin.000005' \
--start-position=1234 \
--stop-position=5678 >/data/mysql/backup/backup.sql

3 恢复数据

复制代码 
# 生成的SQL文件后，就可以使用MySQL客户端工具（如mysql命令行工具）来执行这些SQL语句，从而恢复数据
mysql -h127.0.0.1 -uroot -p'passwd' testdb < data/mysql/backup/backup.sql

4 binlog2sql解析ROW格式binlog实战

4.1 准备工作

复制代码 
# 在mydb数据库中创建一张临时表，并在临时表插入10条数据
mysql> use mydb;
Database changed
mysql> CREATE TABLE tmp_user (
    ->     id INT AUTO_INCREMENT PRIMARY KEY,
    ->     name VARCHAR(50) NOT NULL,
    ->     age INT
    -> ) ENGINE=InnoDB;
INSERT INTO tmp_user (name, age) VALUES
('A', 25),
('B', 30),
Query OK, 0 rows affected (0.06 sec)

mysql> INSERT INTO tmp_user (name, age) VALUES
    -> ('A', 25),
    -> ('B', 30),
    -> ('C', 22),
    -> ('D', 28),
    -> ('E', 26),
    -> ('F', 35),
    -> ('G', 29),
    -> ('H', 31),
    -> ('I', 27),
    -> ('J', 24);
commit;Query OK, 10 rows affected (0.07 sec)
Records: 10  Duplicates: 0  Warnings: 0

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql>
mysql> select *from tmp_user;
+----+------+------+
| id | name | age  |
+----+------+------+
|  1 | A    |   25 |
|  2 | B    |   30 |
|  3 | C    |   22 |
|  4 | D    |   28 |
|  5 | E    |   26 |
|  6 | F    |   35 |
|  7 | G    |   29 |
|  8 | H    |   31 |
|  9 | I    |   27 |
| 10 | J    |   24 |
+----+------+------+
10 rows in set (0.00 sec)

mysql>

4.2 模拟删除表数据

复制代码 
# 模拟误操作将tmp_user表数据删除
mysql> delete from tmp_user;
Query OK, 10 rows affected (0.00 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql>

4.3 基于binlog实现数据恢复

4.3.1 查询误操作时间节点

复制代码 
# 这里我们是通过general_log进行查找误操作时间点，使用general_log的前提是开启了general_log
1. 查询是否开启general_log，如果是On则开启
mysql> SHOW VARIABLES LIKE 'general_log';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| general_log   | ON    |
+---------------+-------+
1 row in set (0.01 sec)

mysql> 

2. 查看slow_query_log文件路径
mysql> SHOW VARIABLES LIKE 'general_log_file';
+------------------+-----------------------------+
| Variable_name    | Value                       |
+------------------+-----------------------------+
| general_log_file | /data/mysql/log/general.log |
+------------------+-----------------------------+
1 row in set (0.00 sec)

mysql> 

3. 分析general_log的文件/data/mysql/log/general.log
[root@node3 ~]# grep -i "insert into" /data/mysql/log/general.log
2025-03-17T14:10:02.804862+08:00           35 Query     INSERT INTO tmp_user (name, age) VALUES
[root@node3 ~]#

4.3.2 binlog2sql解析导出binlog

复制代码 
python /root/tool/binlog2sql-master/binlog2sql/binlog2sql.py \
-h127.0.0.1 \
-P3306 \
-uroot \
-p'lahmy1c@' \
-dmydb \
-ttmp_user \
--start-file='mysql-bin.000005' \
--start-datetime='2025-03-17 14:10:02' \
--stop-datetime='2025-03-17 14:11:02' >/data/mysql/backup/backup.sql

[root@node3 ~]# python /root/tool/binlog2sql-master/binlog2sql/binlog2sql.py \
> -h127.0.0.1 \
> -P3306 \
> -uroot \
> -p'lahmy1c@' \
> -dmydb \
> -ttmp_user \
> --start-file='mysql-bin.000005' \
> --start-datetime='2025-03-17 14:10:02' \
> --stop-datetime='2025-03-17 14:11:02' >/data/mysql/backup/backup.sql
[root@node3 ~]#

4.3.3 恢复数据

复制代码 
mysql -h127.0.0.1 -uroot -p'lahmy1c@' mydb < /data/mysql/backup/backup.sql

[root@node3 ~]# mysql -h127.0.0.1 -uroot -p'lahmy1c@' mydb < /data/mysql/backup/backup.sql
mysql: [Warning] Using a password on the command line interface can be insecure.
[root@node3 ~]#

4.3.4 验证

复制代码 
mysql> select * from tmp_user;
+----+------+------+
| id | name | age  |
+----+------+------+
| 11 | A    |   25 |
| 12 | B    |   30 |
| 13 | C    |   22 |
| 14 | D    |   28 |
| 15 | E    |   26 |
| 16 | F    |   35 |
| 17 | G    |   29 |
| 18 | H    |   31 |
| 19 | I    |   27 |
| 20 | J    |   24 |
+----+------+------+
10 rows in set (0.00 sec)

mysql>

5 错误问题解决办法

5.1 mysql-replication依赖安装错误解决

复制代码 
[root@node1 tool]# pip install mysql-replication
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting mysql-replication
  Using cached mysql_replication-1.0.9.tar.gz (71 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python /usr/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpYwoAbg
       cwd: /tmp/pip-install-vRXNI3/mysql-replication
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/usr/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-TUvCpr/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 146, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-TUvCpr/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 127, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-TUvCpr/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 243, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-TUvCpr/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 8, in <module>
      from pathlib import Path
  ImportError: No module named pathlib
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python /usr/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpYwoAbg Check the logs for full command output.
[root@node1 tool]#

安装依赖包mysql-replication时报错：因为此时安装的是mysql_replication-1.0.9.tar.gz，版本为1.0.9，我们降低下版本，安装0.21，安装命令：

pip install mysql-replication==0.21

5.2 pymysql问题解决

复制代码 
[root@node1 tool]# python /root/tool/binlog2sql-master/binlog2sql/binlog2sql.py -h127.0.0.1 -uroot -p -dmysqldb -ttmp_user --start-file=mysql-bin.000005
Traceback (most recent call last):
  File "/root/tool/binlog2sql-master/binlog2sql/binlog2sql.py", line 7, in <module>
    from pymysqlreplication import BinLogStreamReader
  File "/usr/local/lib/python3.6/site-packages/mysql_replication-0.22-py3.6.egg/pymysqlreplication/__init__.py", line 23, in <module>
  File "/usr/local/lib/python3.6/site-packages/mysql_replication-0.22-py3.6.egg/pymysqlreplication/binlogstream.py", line 9, in <module>
ModuleNotFoundError: No module named 'pymysql.util'
[root@node1 tool]#

特别地，执行示例命令时报错，mysql-replication-0.21对于pymysql版本也有要求（一般是0.7到0.9），因此我们需要卸载pymysql，重新安装一个pymysql 0.9版本，操作如下：

复制代码 
# 卸载pymysql
[root@node1 tool]# pip uninstall pymysql
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Found existing installation: PyMySQL 0.10.1
Uninstalling PyMySQL-0.10.1:
  Would remove:
    /usr/lib/python2.7/site-packages/PyMySQL-0.10.1.dist-info/*
    /usr/lib/python2.7/site-packages/pymysql/*
Proceed (y/n)? ^[[A^[[A
Your response ('\x1b[a\x1b[a') was not one of the expected responses: y, n
Proceed (y/n)? y
  Successfully uninstalled PyMySQL-0.10.1
[root@node1 tool]# pip install pymysql==0.9
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pymysql==0.9
  Downloading PyMySQL-0.9.0-py2.py3-none-any.whl (84 kB)
     |████████████████████████████████| 84 kB 75 kB/s 
Requirement already satisfied: cryptography in /usr/lib64/python2.7/site-packages (from pymysql==0.9) (1.7.2)
Requirement already satisfied: idna>=2.0 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (2.4)
Requirement already satisfied: pyasn1>=0.1.8 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (0.1.9)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (1.9.0)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (0.9.8)
Requirement already satisfied: enum34 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (1.0.4)
Requirement already satisfied: ipaddress in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (1.0.16)
Requirement already satisfied: cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from cryptography->pymysql==0.9) (1.6.0)
Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography->pymysql==0.9) (2.14)
Installing collected packages: pymysql
Successfully installed pymysql-0.9.0
[root@node1 tool]# 

# 安装0.9版本的pymysql
[root@node3 ~]# pip install pymysql==0.9
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pymysql==0.9
  Downloading PyMySQL-0.9.0-py2.py3-none-any.whl (84 kB)
     |████████████████████████████████| 84 kB 604 kB/s 
Requirement already satisfied: cryptography in /usr/lib64/python2.7/site-packages (from pymysql==0.9) (1.7.2)
Requirement already satisfied: idna>=2.0 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (2.4)
Requirement already satisfied: pyasn1>=0.1.8 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (0.1.9)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (1.9.0)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (0.9.8)
Requirement already satisfied: enum34 in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (1.0.4)
Requirement already satisfied: ipaddress in /usr/lib/python2.7/site-packages (from cryptography->pymysql==0.9) (1.0.16)
Requirement already satisfied: cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from cryptography->pymysql==0.9) (1.6.0)
Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography->pymysql==0.9) (2.14)
Installing collected packages: pymysql
  Attempting uninstall: pymysql
    Found existing installation: PyMySQL 0.10.1
    Uninstalling PyMySQL-0.10.1:
      Successfully uninstalled PyMySQL-0.10.1
Successfully installed pymysql-0.9.0
[root@node3 ~]#

5.3 数据库连接问题解决

复制代码 
[root@node1 ~]# python /root/tool/binlog2sql-master/binlog2sql/binlog2sql.py -h192.168.10.30 -uroot -p -dmydb -ttmp_user --start-file=mysql-bin.000005
Password: 
Traceback (most recent call last):
  File "/root/tool/binlog2sql-master/binlog2sql/binlog2sql.py", line 149, in <module>
    back_interval=args.back_interval, only_dml=args.only_dml, sql_type=args.sql_type)
  File "/root/tool/binlog2sql-master/binlog2sql/binlog2sql.py", line 46, in __init__
    self.connection = pymysql.connect(**self.conn_setting)
  File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 94, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 327, in __init__
    self.connect()
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 598, in connect
    self._request_authentication()
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
    auth_packet = self._process_auth(plugin_name, auth_packet)
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 933, in _process_auth
    pkt = self._read_packet()
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
    packet.check_error()
  File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, u"Access denied for user 'root'@'192.168.10.30' (using password: YES)")
[root@node1 ~]#

mysql> SELECT user, host FROM mysql.user WHERE user = 'root';
+------+-----------+
| user | host      |
+------+-----------+
| root | localhost |
+------+-----------+
1 row in set (0.00 sec)

mysql> UPDATE mysql.user SET host='%' WHERE user='root';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

mysql>

6 总结

通过binlog2sql工具，我们可以轻松地将ROW格式的binlog转化为标准SQL语句，从而方便地进行数据恢复和审计。虽然ROW格式的binlog文件较大，但它提供了更详细的数据变更信息，结合binlog2sql工具，我们可以更好地利用这些信息来维护数据库的完整性和一致性。

相关推荐
独隅3 小时前
Linux 正则表达式 的简介
linux·mysql·正则表达式
爱吃山竹的大肚肚3 小时前
达梦（DM）数据库中设置表空间
java·数据库·sql·mysql·spring·spring cloud·oracle
启明真纳4 小时前
MySQL基本概念
数据库·mysql
栗子叶4 小时前
阅读MySQL实战45讲专栏总结
数据库·mysql·innodb·主从同步·数据库原理
码农阿豪4 小时前
时序数据爆发增长，企业如何破解存储与分析困局？
数据库·mysql·金仓
定偶4 小时前
用MySQL玩转数据可视化的技术
数据库·mysql·信息可视化
野犬寒鸦4 小时前
从零起步学习RabbitMQ || 第四章：RabbitMQ的延迟消息在项目中的运用及实现剖析
java·服务器·数据库·后端·mysql·rabbitmq
测试工程师成长之路4 小时前
用 MySQL 玩转数据可视化：从底层驱动到商业智能
数据库·mysql·信息可视化
亲爱的非洲野猪4 小时前
Cassandra vs MySQL/PostgreSQL/MongoDB/Redis/Elasticsearch：选择正确的数据存储架构
mysql·mongodb·postgresql
老师我太想进步了202614 小时前
cmd连接MySQL及相关查询
数据库·mysql
热门推荐
01GitHub 镜像站点022025 Telegram 最新免费社工库机器人（LetsTG可[特殊字符]）搭建指南（含 Python 脚本）03OpenCode 入门教程：介绍 · 安装 · 配置第三方 API (如 Claude)04Linux下V2Ray安装配置指南05安娜的档案(Anna’s Archive) 镜像网站/国内最新可访问入口（持续更新）06UV安装并设置国内源07Claude Code Skills 实用使用手册08AI 规范驱动开发“三剑客”深度对比：Spec-Kit、Kiro 与 OpenSpec 实战指南09网站改了域名，如何查找？10BongoCat - 跨平台键盘猫动画工具