Dinky MySQLCDC 整库同步到 Doris

资源:flink 1.17.0、dinky 1.0.2、doris-2.0.1-rc04

问题:Cannot deserialize value of type int from String ,detailMessage=unknowndatabases ,not a valid int value

2024-05-29 16:52:20.136 ERROR org.apache.doris.flink.table.DorisDynamicTableSink(468): Doris FE's response cannot map to schema. res: "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync" org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type int from String "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync": not a valid int value
at [Source: (String)""errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync""; line: 1, column: 1]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1991) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]

解决:Doris要预先建表,而MySQLCDC 整库到 MySQL 则会自动建表

参考dinky整库同步_dinky整库实时同步-CSDN博客

doris sink表不会在首次执行自动建表,需要手动建doris表,再运行flinkcdc

FlinkSQL:

EXECUTE CDCSOURCE demo_doris WITH (
  'connector' = 'mysql-cdc',
  'hostname' = '172.xxx',
  'port' = '3306',
  'username' = 'xxx',
  'password' = 'xxx',
  'checkpoint' = '10000',
  'scan.startup.mode' = 'initial',
  'parallelism' = '1',
  'database-name' = 'test',
  'table-name' = 'test\.student,',
  'sink.connector' = 'doris',
  'sink.fenodes' = '172.xxx:8030',
  'sink.username' = 'xxx',
  'sink.password' = 'xxx',
  'sink.doris.batch.size' = '1000',
  'sink.sink.max-retries' = '1',
  'sink.sink.db' = 'test',
  'sink.sink.enable-delete' = 'true',
  'sink.sink.properties.format' ='json',
  'sink.sink.properties.read_json_by_line' ='true',
  'sink.table.prefix' = 'test_',
  'sink.table.identifier' = '#{schemaName}.#{tableName}',
  'sink.sink.label-prefix' = '${idUtil.simpleUUID()}'
);
相关推荐
jerry6092 分钟前
NLP CH3复习
人工智能·python·自然语言处理
码农张思壮40 分钟前
使用 AI 编码将路书 GPX 文件导入高德地图生成路线的尝试
后端·python·openai
又南又难1 小时前
deepFM模型pytorch实现
人工智能·pytorch·python
bst@微胖子1 小时前
Python实现接口签名调用
android·java·python
佛曰我不想说话2 小时前
简洁安装配置在Windows环境下使用vscode开发pytorch
pytorch·python
deyong10242 小时前
vscode实用插件(持续更新)
ide·vscode·编辑器
摸鱼仙人~2 小时前
pytorch中nn.Conv2d详解及参数设置原则
人工智能·pytorch·python
绍兴贝贝3 小时前
代码随想录算法训练营第五十天|图论基础|深度优先搜索理论基础|KM98.所有可达路径|广度优先搜索理论基础
数据结构·人工智能·python·算法·力扣
羊小猪~~3 小时前
基于RNN模型的心脏病预测(tensorflow实现)
人工智能·python·rnn·深度学习·神经网络·机器学习·tensorflow
feifeikon3 小时前
TensorFlow DAY1:基础语法
人工智能·python·tensorflow