Dinky FlinkSQL Doris读取写入

Dinky运行前开启全局变量,以支持使用: 'sink.sink.label-prefix' = '${idUtil.simpleUUID()}'

Mysql同步Doris - testMysqlCdcDoris:

复制代码
EXECUTE CDCSOURCE demo_doris WITH (
  'connector' = 'mysql-cdc',
  'hostname' = '172.xxx',
  'port' = '3306',
  'username' = 'xxx',
  'password' = 'xxx',
  'checkpoint' = '10000',
  'scan.startup.mode' = 'initial',
  'parallelism' = '1',
  'database-name' = 'test',
  'table-name' = 'test\.student,',
  'sink.connector' = 'doris',
  'sink.fenodes' = '172.xxx:8130',
  'sink.username' = 'xxx',
  'sink.password' = 'xxx',
  'sink.doris.batch.size' = '1000',
  'sink.sink.max-retries' = '1',
  'sink.sink.db' = 'test',
  'sink.sink.enable-delete' = 'true',
  'sink.sink.properties.format' ='json',
  'sink.sink.properties.read_json_by_line' ='true',
  'sink.table.prefix' = 'test_',
  'sink.table.identifier' = '#{schemaName}.#{tableName}',
  'sink.sink.label-prefix' = '${idUtil.simpleUUID()}'
);

读取Doris - testDorisRead:

复制代码
CREATE TABLE flink_doris_source (
  aggregate_id int,
  replace_data string,
  max_data string,
  agg_item int,
  max_item int,
  min_item int
) 
WITH (
  'connector' = 'doris',
  'fenodes' = '172.xxx:8130',
  'table.identifier' = 'test.aggregate_table',
  'username' = 'xxx',
  'password' = 'xxx'
);

select * from flink_doris_source

Doris同步Doris - testDorisCdcDoris:

复制代码
-- doris source
CREATE TABLE flink_doris_source (
  aggregate_id int,
  replace_data string,
  max_data string,
  agg_item int,
  max_item int,
  min_item int
) 
WITH (
  'connector' = 'doris',
  'fenodes' = '172.xxx:8130',
  'table.identifier' = 'test.aggregate_table',
  'username' = 'xxx',
  'password' = 'xxx'
);

-- enable checkpoint
SET 'execution.checkpointing.interval' = '10s';

-- doris sink
CREATE TABLE flink_doris_sink (
  aggregate_id int,
  replace_data string,
  max_data string,
  agg_item int,
  max_item int,
  min_item int
    ) 
    WITH (
      'connector' = 'doris',
      'fenodes' = '172.xxx:8030',
      'table.identifier' = 'test.test_aggregate_table',
      'username' = 'xxx',
      'password' = 'xxx',
      'sink.label-prefix' = '${idUtil.simpleUUID()}'
);

-- submit insert job
INSERT INTO flink_doris_sink select aggregate_id, replace_data, max_data, agg_item, max_item, min_item from flink_doris_source

参考

Flink Doris Connector - Apache Doris

Doris + Flink + DolphinScheduler + Dinky 构建开源数据平台_dinky dolphinscheduler flink-CSDN博客

整库同步概述 | Dinky

相关推荐
Decay__2 小时前
PostgreSQL 数据库全栈运维实战系列 — 合集索引(24 篇)
运维·数据库·sql·mysql·架构
琳少_6 小时前
java程序开发笔记
java·开发语言·数据库·笔记
Pocker_Spades_A9 小时前
工业物联网时序数据库选型:一份能直接抄走的技术规格书
数据库·物联网·时序数据库
苜蓿muxu10 小时前
缓存三大问题:穿透、击穿、雪崩
数据库·redis
正在走向自律11 小时前
MySQL兼容性深度实践:建表、增删改查与迁移避坑全记录
数据库·sql·mysql·信创·数据库迁移
隔窗听雨眠12 小时前
openGauss开源五年的内核演进与生态工程
数据库·opengauss·国产数据库
Decay__12 小时前
PostgreSQL 体系架构全解 —— 进程模型、共享内存、MVCC 与 WAL 预写日志深入
运维·数据库·sql·mysql·架构
沐知全栈开发13 小时前
基于ClinicalTrials公开数据的临床试验分析系统-毕业设计
数据库·课程设计
半杯咖啡半行码14 小时前
Qt 隐式共享(Implicit Sharing)深度解析:机制、陷阱与 Qt6迁移
开发语言·数据库·qt
Nturmoils14 小时前
空值、日期、字符串:业务 SQL 里的函数先跑一遍
数据库