TiCDC 同步 SQL_MODE 相关

作者: Brian 原文来源: https://tidb.net/blog/91f38d0b

问题澄清

下游 users表中username列默认为非空,所以ticdc应该会同步报错。但是为何ticdc并没有同步报错,user_id = 2同步成功,userame的数据还和上游不一样?

问题背景

集群版本:v6.5.3

工具版本:v6.5.3

上游表结构:username 列为 default null

SQL 复制代码
mysql> show create table users;
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                         |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| users | CREATE TABLE `users` (
  `user_id` int(11) NOT NULL,
  `username` varchar(50) DEFAULT NULL,
  PRIMARY KEY (`user_id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

下游表结构:username 列为 not null

SQL 复制代码
mysql> show create table users;
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                     |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| users | CREATE TABLE `users` (
  `user_id` int(11) NOT NULL,
  `username` varchar(50) NOT NULL,
  PRIMARY KEY (`user_id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

上游执行语句:

SQL 复制代码
mysql> insert into users (user_id ) values (2); 
Query OK, 1 row affected (0.00 sec)

mysql> select * from users;
+---------+----------+
| user_id | username |
+---------+----------+
|       2 | NULL     |
+---------+----------+
1 row in set (0.00 sec)

下游表数据:

SQL 复制代码
mysql> select * from users;
+---------+----------+
| user_id | username |
+---------+----------+
|       2 |          |
+---------+----------+
1 row in set (0.00 sec)

问题分析

1.dashboard--SQL statement

上游执行

SQL 复制代码
insert into users (user_id ) values (2);

查看下游实际执行的SQL为:

SQL 复制代码
INSERT INTO `work`.`users` (`user_id`, `username`) VALUES(2, NULL) 

2.sql mode查看

当前SQL mode 上下游相同,均为默认SQL MODE

SQL 复制代码
mysql> show variables like 'sql_mode';
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                     |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

SQL_MODE 中包含strict_trans_tables按理说不应该出现这种违反约束的问题

3.分析源码

ticdc同步会临时改变SQL MODE如下代码:

会将STRICT_TRANS_TABLES,STRICT_ALL_TABLES disable掉。

SQL 复制代码
func AdjustSQLModeCompatible(sqlModes string) (string, error) {
    needDisable := []string{
       "NO_ZERO_IN_DATE",
       "NO_ZERO_DATE",
       "ERROR_FOR_DIVISION_BY_ZERO",
       "NO_AUTO_CREATE_USER",
       "STRICT_TRANS_TABLES",
       "STRICT_ALL_TABLES",
    }
    needEnable := []string{
       "IGNORE_SPACE",
       "NO_AUTO_VALUE_ON_ZERO",
       "ALLOW_INVALID_DATES",
    }
    disable := strings.Join(needDisable, ",")
    enable := strings.Join(needEnable, ",")

    mode, err := tmysql.GetSQLMode(sqlModes)
    if err != nil {
       return sqlModes, err
    }
    disableMode, err2 := tmysql.GetSQLMode(disable)
    if err2 != nil {
       return sqlModes, err2
    }
    enableMode, err3 := tmysql.GetSQLMode(enable)
    if err3 != nil {
       return sqlModes, err3
    }
    // About this bit manipulation, details can be seen
    // https://github.com/pingcap/dm/pull/1869#discussion_r669771966
    mode = (mode &^ disableMode) | enableMode

    return GetSQLModeStrBySQLMode(mode), nil
}

4.tidb将严格模式禁用后

如下:

SQL 复制代码
#tidb数据库中默认的SQL mode 
mysql> show variables like '%sql_mode%';
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                     |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


#表结构中,username有约数据not null
mysql> show create table users;
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                     |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| users | CREATE TABLE `users` (
  `user_id` int(11) NOT NULL,
  `username` varchar(50) NOT NULL,
  PRIMARY KEY (`user_id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


#临时手动将默认有的STRICT_TRANS_TABLES禁用
mysql> set session sql_mode ="ONLY_FULL_GROUP_BY,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION";
Query OK, 0 rows affected (0.00 sec)


#username列插入null,成功插入伴有warning
mysql> insert into users values(4,null);
Query OK, 1 row affected, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+----------------------------------+
| Level   | Code | Message                          |
+---------+------+----------------------------------+
| Warning | 1048 | Column 'username' cannot be null |
+---------+------+----------------------------------+
1 row in set (0.00 sec)

根因总结

上下游表结构不完全不一致,上游username列default null,下游username列not null。

Ticdc v6.5.3同步是会临时将STRICT_TRANS_TABLES,STRICT_ALL_TABLES禁用,这样即使下游表列约束为not null,也可以成功插入null值,但是会伴随着对应warning

问题后续

https://github.com/pingcap/tiflow/pull/10644/files 第一次发版是 v8.0.0

上述 PR 合并之前,首先查询下游 SQL 模式,然后进行配置。该 PR 合并之后,不再查询下游,直接基于 TiDB 的默认 SQL 模式进行配置。使用配置后的 SQL 模式创建到下游系统的连接。

备注

关于此 case 的解决在此要特别感谢产研jinling 老师!

相关推荐
搬码后生仔9 分钟前
SQLite 是一个轻量级的嵌入式数据库,不需要安装服务器,直接使用文件即可。
数据库·sqlite
码农君莫笑11 分钟前
Blazor项目中使用EF读写 SQLite 数据库
linux·数据库·sqlite·c#·.netcore·人机交互·visual studio
江上挽风&sty13 分钟前
【Django篇】--动手实践Django基础知识
数据库·django·sqlite
张伯毅16 分钟前
Flink SQL 支持 kafka 开启 kerberos 权限控制.
sql·flink·kafka
向阳121816 分钟前
mybatis 动态 SQL
数据库·sql·mybatis
胡图蛋.18 分钟前
什么是事务
数据库
小黄人软件20 分钟前
20241220流水的日报 mysql的between可以用于字符串 sql 所有老日期的,保留最新日期
数据库·sql·mysql
青莳吖23 分钟前
Java通过Map实现与SQL中的group by相同的逻辑
java·开发语言·sql
张声录125 分钟前
【ETCD】【实操篇(三)】【ETCDCTL】如何向集群中写入数据
数据库·chrome·etcd
无为之士31 分钟前
Linux自动备份Mysql数据库
linux·数据库·mysql