Java应用Flink CDC监听MySQL数据变动内容输出到控制台

文章目录

maven 依赖

xml 复制代码
<properties>
        <flink.version>1.14.0</flink.version>
        <flink-cdc.version>2.3.0</flink-cdc.version>
    </properties>
    <dependencies>
        <!-- Flink dependencies -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-base</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-sql-connector-mysql-cdc</artifactId>
            <version>${flink-cdc.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.12</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_2.12</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime-web_2.12</artifactId>
            <version>${flink.version}</version>
        </dependency>
    </dependencies>

自定义数据变化处理器

java 复制代码
package org.example;

import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.sink.RichSinkFunction;

public class CustomSink extends RichSinkFunction<String> {
    @Override
    public void open(Configuration parameters) throws Exception {
        super.open(parameters);
    }

    @Override
    public void close() throws Exception {
        super.close();
    }

    @Override
    public void invoke(String value, Context context) throws Exception {
        //0P字段,该字段也有4种取值。分别是C(Create ) , U(Updlate) . D(Delete ),Read 。
        // 对于U操作,其数据部分同时包含了Before和After.
         System.out.println(">>>" + value);

    }
}
java 复制代码
package org.example;

import com.ververica.cdc.connectors.mysql.source.MySqlSource;
import com.ververica.cdc.debezium.DebeziumDeserializationSchema;
import com.ververica.cdc.debezium.JsonDebeziumDeserializationSchema;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.streaming.api.datastream.DataStreamSink;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class MysqlSourceExample {
    public static void main(String[] args) throws Exception {
        DebeziumDeserializationSchema debeziumDeserializationSchema = new JsonDebeziumDeserializationSchema();
        MySqlSource<String> source = MySqlSource.builder()
                .hostname("127.0.0.1")
                .port(3306)
                .databaseList("canal_manager")// set captured database
                .tableList("canal_manager.canal_user")// set captured table
                .startupOptions(StartupOptions.latest()) // 设置从最新的修改记录开始读取
                .username("root")
                .password("123456")
                .deserializer(debeziumDeserializationSchema) // converts SourceRecord to JSON string
                .includeSchemaChanges(true)
                .build();
        //启动一个webuI。
        Configuration configuration = new Configuration();
        configuration.setInteger(RestOptions.PORT, 8081);
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(configuration);

        //检者点间隔时间
        env .enableCheckpointing(5000);
        DataStreamSink<String> sink = env.fromSource(source, WatermarkStrategy.noWatermarks(), "MySQL Source")
                .addSink(new CustomSink());
        env.execute();

    }
}

验证

启动后web页面地址访问http://localhost:8081/,MySQL数据库canal_manager中的表canal_user数据发生修改,控制台有输出json:

相关推荐
蓝澈112128 分钟前
迪杰斯特拉算法之解决单源最短路径问题
java·数据结构
Kali_0735 分钟前
使用 Mathematical_Expression 从零开始实现数学题目的作答小游戏【可复制代码】
java·人工智能·免费
rzl021 小时前
java web5(黑马)
java·开发语言·前端
时序数据说1 小时前
为什么时序数据库IoTDB选择Java作为开发语言
java·大数据·开发语言·数据库·物联网·时序数据库·iotdb
戒不掉的伤怀1 小时前
【Navicat 连接MySQL时出现错误1251:客户端不支持服务器请求的身份验证协议;请考虑升级MySQL客户端】
服务器·数据库·mysql
guojl1 小时前
深度解读jdk8 HashMap设计与源码
java
guojl1 小时前
深度解读jdk8 ConcurrentHashMap设计与源码
java
爱上语文1 小时前
Redis基础(5):Redis的Java客户端
java·开发语言·数据库·redis·后端
A~taoker2 小时前
taoker的项目维护(ng服务器)
java·开发语言
HGW6892 小时前
基于 Elasticsearch 实现地图点聚合
java·elasticsearch·高德地图