TiCDC 同步 SQL_MODE 相关

作者: Brian 原文来源: https://tidb.net/blog/91f38d0b

问题澄清

下游 users表中username列默认为非空,所以ticdc应该会同步报错。但是为何ticdc并没有同步报错,user_id = 2同步成功,userame的数据还和上游不一样?

问题背景

集群版本:v6.5.3

工具版本:v6.5.3

上游表结构:username 列为 default null

SQL 复制代码
mysql> show create table users;
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                         |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| users | CREATE TABLE `users` (
  `user_id` int(11) NOT NULL,
  `username` varchar(50) DEFAULT NULL,
  PRIMARY KEY (`user_id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

下游表结构:username 列为 not null

SQL 复制代码
mysql> show create table users;
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                     |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| users | CREATE TABLE `users` (
  `user_id` int(11) NOT NULL,
  `username` varchar(50) NOT NULL,
  PRIMARY KEY (`user_id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

上游执行语句:

SQL 复制代码
mysql> insert into users (user_id ) values (2); 
Query OK, 1 row affected (0.00 sec)

mysql> select * from users;
+---------+----------+
| user_id | username |
+---------+----------+
|       2 | NULL     |
+---------+----------+
1 row in set (0.00 sec)

下游表数据:

SQL 复制代码
mysql> select * from users;
+---------+----------+
| user_id | username |
+---------+----------+
|       2 |          |
+---------+----------+
1 row in set (0.00 sec)

问题分析

1.dashboard--SQL statement

上游执行

SQL 复制代码
insert into users (user_id ) values (2);

查看下游实际执行的SQL为:

SQL 复制代码
INSERT INTO `work`.`users` (`user_id`, `username`) VALUES(2, NULL) 

2.sql mode查看

当前SQL mode 上下游相同,均为默认SQL MODE

SQL 复制代码
mysql> show variables like 'sql_mode';
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                     |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

SQL_MODE 中包含strict_trans_tables按理说不应该出现这种违反约束的问题

3.分析源码

ticdc同步会临时改变SQL MODE如下代码:

会将STRICT_TRANS_TABLES,STRICT_ALL_TABLES disable掉。

SQL 复制代码
func AdjustSQLModeCompatible(sqlModes string) (string, error) {
    needDisable := []string{
       "NO_ZERO_IN_DATE",
       "NO_ZERO_DATE",
       "ERROR_FOR_DIVISION_BY_ZERO",
       "NO_AUTO_CREATE_USER",
       "STRICT_TRANS_TABLES",
       "STRICT_ALL_TABLES",
    }
    needEnable := []string{
       "IGNORE_SPACE",
       "NO_AUTO_VALUE_ON_ZERO",
       "ALLOW_INVALID_DATES",
    }
    disable := strings.Join(needDisable, ",")
    enable := strings.Join(needEnable, ",")

    mode, err := tmysql.GetSQLMode(sqlModes)
    if err != nil {
       return sqlModes, err
    }
    disableMode, err2 := tmysql.GetSQLMode(disable)
    if err2 != nil {
       return sqlModes, err2
    }
    enableMode, err3 := tmysql.GetSQLMode(enable)
    if err3 != nil {
       return sqlModes, err3
    }
    // About this bit manipulation, details can be seen
    // https://github.com/pingcap/dm/pull/1869#discussion_r669771966
    mode = (mode &^ disableMode) | enableMode

    return GetSQLModeStrBySQLMode(mode), nil
}

4.tidb将严格模式禁用后

如下:

SQL 复制代码
#tidb数据库中默认的SQL mode 
mysql> show variables like '%sql_mode%';
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                     |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


#表结构中,username有约数据not null
mysql> show create table users;
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                                                     |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| users | CREATE TABLE `users` (
  `user_id` int(11) NOT NULL,
  `username` varchar(50) NOT NULL,
  PRIMARY KEY (`user_id`) /*T![clustered_index] CLUSTERED */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)


#临时手动将默认有的STRICT_TRANS_TABLES禁用
mysql> set session sql_mode ="ONLY_FULL_GROUP_BY,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION";
Query OK, 0 rows affected (0.00 sec)


#username列插入null,成功插入伴有warning
mysql> insert into users values(4,null);
Query OK, 1 row affected, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+----------------------------------+
| Level   | Code | Message                          |
+---------+------+----------------------------------+
| Warning | 1048 | Column 'username' cannot be null |
+---------+------+----------------------------------+
1 row in set (0.00 sec)

根因总结

上下游表结构不完全不一致,上游username列default null,下游username列not null。

Ticdc v6.5.3同步是会临时将STRICT_TRANS_TABLES,STRICT_ALL_TABLES禁用,这样即使下游表列约束为not null,也可以成功插入null值,但是会伴随着对应warning

问题后续

https://github.com/pingcap/tiflow/pull/10644/files 第一次发版是 v8.0.0

上述 PR 合并之前,首先查询下游 SQL 模式,然后进行配置。该 PR 合并之后,不再查询下游,直接基于 TiDB 的默认 SQL 模式进行配置。使用配置后的 SQL 模式创建到下游系统的连接。

备注

关于此 case 的解决在此要特别感谢产研jinling 老师!

相关推荐
胚芽鞘6811 小时前
关于java项目中maven的理解
java·数据库·maven
sun0077004 小时前
mysql索引底层原理
数据库·mysql
workflower7 小时前
MDSE和敏捷开发相互矛盾之处:方法论本质的冲突
数据库·软件工程·敏捷流程·极限编程
Tony小周7 小时前
实现一个点击输入框可以弹出的数字软键盘控件 qt 5.12
开发语言·数据库·qt
lifallen7 小时前
Paimon 原子提交实现
java·大数据·数据结构·数据库·后端·算法
TDengine (老段)8 小时前
TDengine 数据库建模最佳实践
大数据·数据库·物联网·时序数据库·tdengine·涛思数据
Elastic 中国社区官方博客8 小时前
Elasticsearch 字符串包含子字符串:高级查询技巧
大数据·数据库·elasticsearch·搜索引擎·全文检索·lucene
Gauss松鼠会8 小时前
GaussDB应用场景全景解析:从金融核心到物联网的分布式数据库实践
数据库·分布式·物联网·金融·database·gaussdb
守城小轩9 小时前
Chromium 136 编译指南 - Android 篇:开发工具安装(三)
android·数据库·redis
尽兴-9 小时前
如何将多个.sql文件合并成一个:Windows和Linux/Mac详细指南
linux·数据库·windows·sql·macos