Flink Table API/SQL 多分支sink

背景

在某个场景中,需要从Kafka中获取数据,经过转换处理后,需要同时sink到多个输出源中(kafka、mysql、hologres)等。两次调用execute, 阿里云Flink vvr引擎报错:

java 复制代码
public static void main(String[] args) {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
        StreamStatementSet streamStatementSet = tEnv.createStatementSet();

        String s = LocalDateTimeUtils.getDateTime(System.currentTimeMillis());

        DataStream<String> dataStream = env.fromElements(s, LocalDateTimeUtils.getDateTime(System.currentTimeMillis()));

        tEnv.executeSql(KAFKA_TABLE_SQL);
        tEnv.executeSql(KAFKA_TABLE_SQL_1);


        Table table = tEnv.fromDataStream(dataStream);
        table.insertInto("kafka_sink").execute();
        table.insertInto("kafka_sink_1").execute();

        streamStatementSet.execute();
    }
java 复制代码
Caused by: org.apache.flink.util.FlinkRuntimeException: Cannot have more than one execute() or executeAsync() call in a single environment.
	at org.apache.flink.client.program.StreamContextEnvironment.validateAllowedExecution(StreamContextEnvironment.java:199) ~[flink-dist-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:187) ~[flink-dist-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:110) ~[?:?]
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:877) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:756) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:955) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]
	at org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:57) ~[flink-table-api-java-uber-1.15-vvr-6.0.7-1-SNAPSHOT.jar:1.15-vvr-6.0.7-1-SNAPSHOT]

解决

使用 StreamStatementSet. 具体参考官网:

https://nightlies.apache.org/flink/flink-docs-release-1.15/zh/docs/dev/table/data_stream_api/#converting-between-datastream-and-table

改良后的代码:

java 复制代码
public static void main(String[] args) {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
        StreamStatementSet streamStatementSet = tEnv.createStatementSet();

        String s = LocalDateTimeUtils.getDateTime(System.currentTimeMillis());

        DataStream<String> dataStream = env.fromElements(s, LocalDateTimeUtils.getDateTime(System.currentTimeMillis()));

        tEnv.executeSql(KAFKA_TABLE_SQL);
        tEnv.executeSql(KAFKA_TABLE_SQL_1);


        Table table = tEnv.fromDataStream(dataStream);

        streamStatementSet.addInsert("kafka_sink", table);
        streamStatementSet.addInsert("kafka_sink_1", table);

        streamStatementSet.execute();
    }
相关推荐
楠楠子呀18 小时前
从零构建高性能 WhatsApp 链接生成器与重定向服务:架构与路由解析
redis·python·容器·架构·flink·散列表·flume
头茬韭菜1 天前
功能点 9:Flink Connector
大数据·flink·fluss
Faith_xzc1 天前
一条 SQL 顶一条 Flink 链路?Doris Streaming Job 持续导入全景解析
大数据·数据库·sql·flink
何升曜4 天前
Flink Slot 分配机制深度解析:从申请到部署的完整链路
flink
我变秃了也没变强5 天前
standalone模式部署flink集群
flink·apache
渣渣盟5 天前
Flink 流式写入文件终极指南:从 Row Format 到 Bucket 分区的深度剖析
大数据·flink
CIO_Alliance5 天前
AI算法系列(3)| 实时数据管道:CDC+Flink实现库存异动秒级响应
大数据·人工智能·flink
渣渣盟6 天前
Flink 写入 Redis 实战:数据模型选型、连接池调优与常见坑
大数据·redis·flink
渣渣盟7 天前
Flink + Kafka 数据写入实战:从API调用到端到端一致性精讲
flink·kafka·linq
TDengine (老段)11 天前
TDengine 第三方工具 — Telegraf、Kafka Connect、Flink、Spark
大数据·数据库·物联网·flink·kafka·时序数据库·tdengine