资源:flink 1.17.0、dinky 1.0.2、doris-2.0.1-rc04
问题:Cannot deserialize value of type int from String ,detailMessage=unknowndatabases ,not a valid int value
2024-05-29 16:52:20.136 ERROR org.apache.doris.flink.table.DorisDynamicTableSink(468): Doris FE's response cannot map to schema. res: "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync" org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type int from String "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync": not a valid int value
at [Source: (String)""errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync""; line: 1, column: 1]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1991) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]
解决:Doris要预先建表,而MySQLCDC 整库到 MySQL 则会自动建表
参考dinky整库同步_dinky整库实时同步-CSDN博客
doris sink表不会在首次执行自动建表,需要手动建doris表,再运行flinkcdc
FlinkSQL:
EXECUTE CDCSOURCE demo_doris WITH (
'connector' = 'mysql-cdc',
'hostname' = '172.xxx',
'port' = '3306',
'username' = 'xxx',
'password' = 'xxx',
'checkpoint' = '10000',
'scan.startup.mode' = 'initial',
'parallelism' = '1',
'database-name' = 'test',
'table-name' = 'test\.student,',
'sink.connector' = 'doris',
'sink.fenodes' = '172.xxx:8030',
'sink.username' = 'xxx',
'sink.password' = 'xxx',
'sink.doris.batch.size' = '1000',
'sink.sink.max-retries' = '1',
'sink.sink.db' = 'test',
'sink.sink.enable-delete' = 'true',
'sink.sink.properties.format' ='json',
'sink.sink.properties.read_json_by_line' ='true',
'sink.table.prefix' = 'test_',
'sink.table.identifier' = '#{schemaName}.#{tableName}',
'sink.sink.label-prefix' = '${idUtil.simpleUUID()}'
);