Flink CDC (session模式)

1、

复制代码
# Start YARN session
./bin/yarn-session.sh --detached

2、配置文件:

复制代码
rest.bind-port: {{REST_PORT}}
rest.address: {{NODE_IP}}
execution.target: yarn-session
yarn.application.id: {{YARN_APPLICATION_ID}}

3、mysql-doris.yml

复制代码
source:
 type: mysql
 hostname: localhost
 port: 3306
 username: root
 password: 123456
 tables: app_db.\.*
 server-id: 5400-5404
 server-time-zone: UTC

sink:
 type: doris
 fenodes: 127.0.0.1:8030
 username: root
 password: ""

pipeline:
 name: Sync MySQL Database to Doris
 parallelism: 2

4、运行测试:

复制代码
./bin/flink-cdc.sh mysql-to-doris.yaml

5、相关问题:

1、Caused by: java.lang.ClassCastException: cannot assign instance of com.starrocks.connector.flink.catalog.StarRocksCatalog to field org.apache.flink.cdc.connectors.starrocks.sink.StarRocksMetadataApplier.catalog of type com.starrocks.connector.flink.catalog.StarRocksCatalog in instance of org.apache.flink.cdc.connectors.starrocks.sink.StarRocksMetadataApplier

解决方案:

classloader.resolve-order: parent-first

2、

Caused by: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition

at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.initialize(ExecutionJobVertex.java:229)

at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.initializeJobVertex(DefaultExecutionGraph.java:914)

at org.apache.flink.runtime.executiongraph.ExecutionGraph.initializeJobVertex(ExecutionGraph.java:218)

at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.initializeJobVertices(DefaultExecutionGraph.java:896)

at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:852)

at org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:207)

at org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreExecutionGraph(DefaultExecutionGraphFactory.java:163)

at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:365)

at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:210)

at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:136)

at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:152)

at org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:119)

at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:371)

at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:348)

at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:123)

at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambdacreateJobMasterService0(DefaultJobMasterServiceFactory.java:95)

at org.apache.flink.util.function.FunctionUtils.lambdauncheckedSupplier4(FunctionUtils.java:112)

... 4 more

Caused by: java.lang.NoClassDefFoundError: org/apache/flink/cdc/runtime/typeutils/EventTypeInfo

at java.lang.Class.getDeclaredFields0(Native Method)

at java.lang.Class.privateGetDeclaredFields(Class.java:2583)

解决方案:

cp /BigData/run/flink-cdc/lib/flink-cdc-dist-3.1.0.jar /BigData/run/flink/lib/

3、问题3

Caused by: org.apache.flink.table.api.ValidationException: The MySQL server has a timezone offset (28800 seconds ahead of UTC) which does not match the configured timezone UTC. Specify the right server-time-zone to avoid inconsistencies for time-related fields.

解决方案1:

jdbc:mysql://hostname:port/database?serverTimezone=UTC

解决方案2:

SET table.local-time-zone = 'Asia/Shanghai'; -- 或者你所需的时区

解决方案3:

CREATE TABLE my_table (

...

) WITH (

'connector' = 'mysql-cdc',

'hostname' = 'localhost',

'port' = '3306',

...

'server-time-zone' = 'Asia/Shanghai' -- 或者你所需的时区

);

相关推荐
戚砚笙34 分钟前
Flink进阶:从“会用”到“用明白”的踩坑与实战总结
flink
小麦矩阵系统永久免费39 分钟前
短视频矩阵系统哪个好用?2025最新评测与推荐|小麦矩阵系统
大数据·人工智能·矩阵
贝多芬也爱敲代码9 小时前
如何减小ES和mysql的同步时间差
大数据·mysql·elasticsearch
异次元的星星10 小时前
智慧新零售时代:施易德系统平衡技术与人力,赋能门店运营
大数据·零售
深思慎考11 小时前
ElasticSearch与Kibana 入门指南(7.x版本)
大数据·elasticsearch·jenkins
银行数字化转型导师坚鹏11 小时前
如何设计优秀的企业微信私域运营实战培训方案
大数据·python·企业微信
悠闲蜗牛�12 小时前
人工智能时代下的全栈开发:整合AI、大数据与云原生的实践策略
大数据·人工智能·云原生
ml魔力信息13 小时前
活体检测与防伪技术的安全与隐私分析
大数据·人工智能·安全·隐私保护·生物识别·活体检测
数据要素X14 小时前
寻梦数据空间 | 架构篇:从概念到落地的技术实践与突破性创新
大数据·运维·数据仓库·微服务·数据治理·数据中台·可信数据空间
IT学长编程16 小时前
计算机毕业设计 基于EChants的海洋气象数据可视化平台设计与实现 Python 大数据毕业设计 Hadoop毕业设计选题【附源码+文档报告+安装调试】
大数据·hadoop·python·毕业设计·课程设计·毕业论文·海洋气象数据可视化平台