Flink Table API/SQL 多分支sink

背景

在某个场景中,需要从Kafka中获取数据,经过转换处理后,需要同时sink到多个输出源中(kafka、mysql、hologres)等。两次调用execute, 阿里云Flink vvr引擎报错:

java 复制代码
public static void main(String[] args) {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
        StreamStatementSet streamStatementSet = tEnv.createStatementSet();

        String s = LocalDateTimeUtils.getDateTime(System.currentTimeMillis());

        DataStream<String> dataStream = env.fromElements(s, LocalDateTimeUtils.getDateTime(System.currentTimeMillis()));

        tEnv.executeSql(KAFKA_TABLE_SQL);
        tEnv.executeSql(KAFKA_TABLE_SQL_1);


        Table table = tEnv.fromDataStream(dataStream);
        table.insertInto("kafka_sink").execute();
        table.insertInto("kafka_sink_1").execute();

        streamStatementSet.execute();
    }
java 复制代码
Caused by: org.apache.flink.util.FlinkRuntimeException: Cannot have more than one execute() or executeAsync() call in a single environment.
	at org.apache.flink.client.program.StreamContextEnvironment.validateAllowedExecution(StreamContextEnvironment.java:199) ~[flink-dist-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:187) ~[flink-dist-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110) ~[?:?]
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:877) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:756) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:955) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:57) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]

解决

使用 StreamStatementSet. 具体参考官网:

https://nightlies.apache.org/flink/flink-docs-release-1.15/zh/docs/dev/table/data_stream_api/#converting-between-datastream-and-table

改良后的代码:

java 复制代码
public static void main(String[] args) {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
        StreamStatementSet streamStatementSet = tEnv.createStatementSet();

        String s = LocalDateTimeUtils.getDateTime(System.currentTimeMillis());

        DataStream<String> dataStream = env.fromElements(s, LocalDateTimeUtils.getDateTime(System.currentTimeMillis()));

        tEnv.executeSql(KAFKA_TABLE_SQL);
        tEnv.executeSql(KAFKA_TABLE_SQL_1);


        Table table = tEnv.fromDataStream(dataStream);

        streamStatementSet.addInsert("kafka_sink", table);
        streamStatementSet.addInsert("kafka_sink_1", table);

        streamStatementSet.execute();
    }
相关推荐
欢醉26 分钟前
Flink实时同步Oracle到StarRocks的几个踩坑经验分享
starrocks·oracle·flink
用户36105886261215 小时前
Flink 注册Slot源码深度剖析:从TaskManager注册到Slot状态机的完整链路
大数据·flink
阿里云大数据AI技术18 小时前
实时看、持续记、及时说,央视体育携手阿里云 Flink 打造大型赛事实时 AI 解说Agent
人工智能·flink
阿里云大数据AI技术2 天前
阿里云Flink + Fluss年度发布:Streaming Agent 与湖流一体
人工智能·flink·agent
用户3610588626124 天前
Flink Dispatcher启动流程源码深度剖析:从ClusterEntrypoint到JobMaster启动的完整链路
大数据·flink
用户3610588626125 天前
Flink ApplicationMaster启动流程源码深度剖析:从YARN提交到JobMaster启动的完整链路
大数据·flink
智码看视界5 天前
技术选型指南:Kafka Streams vs Flink:流处理框架选型实战对比与性能压测
flink·实时计算·状态管理·流处理·大数据架构·kafkastreams·流处理选型
用户3610588626126 天前
Flink 配置深度剖析:从配置加载到参数调优的完整指南
大数据·flink
Irene19917 天前
Flink 学习日志 01:前端转大数据的实战避坑指南
java·jvm·flink
Gl�ria8 天前
Flink Yarn-Session Job 相关常用脚本
flink