Dinky MySQLCDC 整库同步到 Doris

SunTecTec2024-06-02 17:54

资源：flink 1.17.0、dinky 1.0.2、doris-2.0.1-rc04

问题：Cannot deserialize value of type int from String ，detailMessage=unknowndatabases ，not a valid int value

复制代码

2024-05-29 16:52:20.136 ERROR org.apache.doris.flink.table.DorisDynamicTableSink(468): Doris FE's response cannot map to schema. res: "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync" org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type int from String "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync": not a valid int value
at [Source: (String)""errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync""; line: 1, column: 1]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1991) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]

解决：Doris要预先建表，而MySQLCDC 整库到 MySQL 则会自动建表

参考dinky整库同步_dinky整库实时同步-CSDN博客

doris sink表不会在首次执行自动建表，需要手动建doris表，再运行flinkcdc

FlinkSQL：

复制代码

EXECUTE CDCSOURCE demo_doris WITH (
  'connector' = 'mysql-cdc',
  'hostname' = '172.xxx',
  'port' = '3306',
  'username' = 'xxx',
  'password' = 'xxx',
  'checkpoint' = '10000',
  'scan.startup.mode' = 'initial',
  'parallelism' = '1',
  'database-name' = 'test',
  'table-name' = 'test\.student,',
  'sink.connector' = 'doris',
  'sink.fenodes' = '172.xxx:8030',
  'sink.username' = 'xxx',
  'sink.password' = 'xxx',
  'sink.doris.batch.size' = '1000',
  'sink.sink.max-retries' = '1',
  'sink.sink.db' = 'test',
  'sink.sink.enable-delete' = 'true',
  'sink.sink.properties.format' ='json',
  'sink.sink.properties.read_json_by_line' ='true',
  'sink.table.prefix' = 'test_',
  'sink.table.identifier' = '#{schemaName}.#{tableName}',
  'sink.sink.label-prefix' = '${idUtil.simpleUUID()}'
);