【flink-cdc】flink-cdc 3版本debug启动pipeline任务,mysql-doris

官方文档
github仓库地址
Flink cdc debug调试动态变更表结构

经过测试使用,在启动任务配置Modify classpath添加jar的方式,容易出错classNotFoundException等等。

一、build project

flink-cdc版本:3.2.1

bash 复制代码
mvn clean package "-Dmaven.test.skip=true" "-Drat.skip=true" "-Dcheckstyle.skip"
# 当然install 也可以
mvn clean install "-Dmaven.test.skip=true" "-Drat.skip=true" "-Dcheckstyle.skip"

二、create module for test

在flink-cdc项目中新建一个module flink-cdc-test

将需要的flink cdc的connector依赖添加至pom

xml 复制代码
    <properties>        
        <flink.version>1.18.1</flink.version>  
        <maven.compiler.source>8</maven.compiler.source>  
        <maven.compiler.target>8</maven.compiler.target>  
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>  
    </properties>  
  
    <dependencies>        
        <dependency>  
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-table-common</artifactId>  
            <version>${flink.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-runtime -->  
        <dependency>  
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-table-runtime</artifactId>  
            <version>${flink.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-api-java-bridge -->  
        <dependency>  
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-table-api-java-bridge</artifactId>  
            <version>${flink.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-core -->  
        <dependency>  
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-core</artifactId>  
            <version>${flink.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java -->  
        <dependency>  
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-streaming-java</artifactId>  
            <version>1.18.1</version>  
            <scope>test</scope>  
        </dependency>  
  
        <dependency>            
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-clients</artifactId>  
            <version>1.18.1</version>  
            <scope>test</scope>  
        </dependency>  
  
        <dependency>            
        <groupId>org.apache.flink</groupId>  
            <artifactId>flink-cdc-cli</artifactId>  
            <version>${project.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <dependency>            
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-cdc-pipeline-connector-mysql</artifactId>  
            <version>${project.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <dependency>            
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-cdc-pipeline-connector-kafka</artifactId>  
            <version>${project.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <dependency>            
            <groupId>org.apache.flink</groupId>  
            <artifactId>flink-cdc-pipeline-connector-doris</artifactId>  
            <version>${project.version}</version>  
            <scope>test</scope>  
        </dependency>  
  
        <!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->  
        <dependency>  
            <groupId>mysql</groupId>  
            <artifactId>mysql-connector-java</artifactId>  
            <version>8.0.27</version>  
        </dependency>  
    
    </dependencies>  

create pipeline yaml

创建flink cdc pipeline的配置文件mysql-to-doris.yaml

注意:drois的端口号不是9030,而是8030

yaml 复制代码
source:  
  type: mysql  
  name: MySQL-Source  
  hostname: 192.168.100.123 
  port: 3306  
  username: root  
  password: 123456  
  tables: test.student  
  server-id: 5401-5404  
  server-time-zone: UTC  
  
sink:  
  type: doris  
  name: Doris-Sink  
  fenodes: 192.168.101:8030
  username: root
  password: 123456  
  
  
pipeline:  
  name: MySQL to Doris Schema Evolution  
  parallelism: 1

create pipeline start entrypoint

测试类,需在src/test目录下:

java 复制代码
import org.apache.flink.cdc.cli.CliFrontend;  
import org.junit.jupiter.api.Test;  
  
import java.util.ArrayList;  
import java.util.List;  
  
public class MysqlPipelineTest {  
  
    @Test  
    public void testMysql() throws Exception {  
        List<String> args= new ArrayList<>();  
        args.add("D:\\flink-cdc-release-3.2.1\\flink-cdc-test\\pipelines\\mysql-to-doris.yaml");  
        args.add("--use-mini-cluster");  
        args.add("true");  
        CliFrontend.main(args.toArray(new String[0]));  
    }  
}

修改启动配置添加FLINK_HOME环境变量:

启动任务:

相关推荐
笨蛋少年派2 小时前
使用hdfs命令报错:Unknown command:dfs(环境变量正确)
大数据·hadoop·hdfs
麦麦大数据3 小时前
F036 vue+flask中医热性药知识图谱可视化系统vue+flask+echarts+mysql
vue.js·python·mysql·flask·可视化·中医中药
BullSmall3 小时前
一键部署MySQL
数据库·mysql
深兰科技3 小时前
东方财经报道|深兰科技落户张江,AI医疗与情感陪伴并进,拓展智能未来版图
大数据·人工智能·科技
Apache Flink3 小时前
理想汽车基于 Hologres + Flink 构建万亿级车联网信号实时分析平台
阿里云·flink·apache
格林威4 小时前
AOI在传统汽车制造领域中的应用
大数据·人工智能·数码相机·计算机视觉·ai·制造·aoi
kalvin_y_liu5 小时前
华为ACT三步走”实施路径,以推动行业智能化落地
大数据·人工智能·ai应用
程序新视界5 小时前
什么是OLTP ,MySQL是如何支持OLTP的?
数据库·后端·mysql
涛思数据(TDengine)6 小时前
杨凌美畅用 TDengine 时序数据库,支撑 500 条产线 2 年历史数据追溯
大数据·时序数据库·tdengine
熊文豪6 小时前
时序数据库选型指南:从大数据视角看高效存储与分析
大数据·数据库·时序数据库