Flink CDC 3.0 本地调试

前言

flink cdc 本地调试官方没有特别说明，这里记录下。

调试流程

1. 拉取代码本地打包

通过 github 拉取3.0.0 以上版本，本地maven打包 mvn clean package -DskipTests

2. 配置启动参数

搜索启动类CliFrontend，修改运行配置。

需要配置环境变量FLINK_HOME， pipeline connector依赖包，flink-dist 包，以及指定配置文件，--use-mini-cluster true 代表使用 local 集群。

vm options 可以通过classpath添加环境变量, 注意本地local模式，运行，需要依赖flink环境，如果connector依赖hadoop相关依赖，还需要加hadoop相关依赖。但是提交到flink集群上运行不需要指定了，可以参考 flink-cdc.sh 提交逻辑。

本地debug vm options如下：

shell 复制代码

-classpath /Users/Projects/flink-cdc/flink-cdc-connectors/flink-cdc-dist/target/flink-cdc-dist-3.0-SNAPSHOT.jar:/Users/Projects/flink-cdc/flink-cdc-connectors/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/target/flink-cdc-pipeline-connector-mysql-3.0-SNAPSHOT.jar:/Users/Projects/flink-cdc/flink-cdc-connectors/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/target/flink-cdc-pipeline-connector-paimon-3.0-SNAPSHOT.jar:/Users/software/flink-1.18.1/lib/flink-dist-1.18.1.jar:/Users/software/flink-1.18.1/lib/flink-table-runtime-1.18.1.jar:/Users/software/flink-1.18.1/lib/flink-table-planner-loader-1.18.1.jar:/Users/software/flink-1.18.1/lib/flink-cep-1.18.1.jar:/Users/software/flink-1.18.1/lib/flink-connector-files-1.18.1.jar:/Users/software/flink-1.18.1/lib/flink-csv-1.18.1.jar:/Users/software/flink-1.18.1/lib/flink-json-1.18.1.jar:/Users/software/flink-1.18.1/lib/flink-table-api-java-uber-1.18.1.jar:/Users/software/hadoop-2.8.5/etc/hadoop:/Users/software/hadoop-2.8.5/share/hadoop/common/lib/*:/Users/software/hadoop-2.8.5/share/hadoop/common/*:/Users/software/hadoop-2.8.5/share/hadoop/hdfs:/Users/software/hadoop-2.8.5/share/hadoop/hdfs/lib/*:/Users/software/hadoop-2.8.5/share/hadoop/hdfs/*:/Users/software/hadoop-2.8.5/share/hadoop/yarn/lib/*:/Users/software/hadoop-2.8.5/share/hadoop/yarn/*:/Users/software/hadoop-2.8.5/share/hadoop/mapreduce/lib/*:/Users/software/hadoop-2.8.5/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/Users/Projects/flink-cdc/flink-cdc-connectors/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/target/flink-cdc-pipeline-connector-values-3.0-SNAPSHOT.jar --add-exports java.base/jdk.internal.misc=ALL-UNNAMED  --add-modules=jdk.unsupported --add-opens=java.base/sun.nio.ch=ALL-UNNAMED

mysql-to-paimin.yaml

yaml 复制代码

source:
  type: mysql
  hostname: localhost
  port: 3306
  username: root
  password: root
  tables: cdc.cdc_test

sink:
  type: paimon
  commit.user: test
  catalog.properties.warehouse: ./path

pipeline:
  name: mysql-sync-paimon
  parallelism: 1

本地运行，如果需要依赖 checkpoint做两阶段提交，需要做cp指定，目前 --use-mini-cluster true 模式，默认没有配置 cp。本地依赖的flink环境所配置的 config在 use-mini-cluster 模式没有使用。

本地执行debug, 可以手动加上cp相关改动