FlinkSQL源码解析1--提交任务过程

提交入口

org.apache.flink.table.api.internal.TableEnvironmentImpl#executeSql

@Override

public TableResult executeSql(String statement) {

List<Operation> operations = getParser().parse(statement);

if (operations.size() != 1) {

throw new TableException(UNSUPPORTED_QUERY_IN_EXECUTE_SQL_MSG);

}

Operation operation = operations.get(0);

return executeInternal(operation);

}

提交过程概述

sql--》operation--》Transformation--》pipeline(StreamGraph)--》JobGraph--》deployJobCluster

具体过程

1、查询转成operation

sql-->SqlNodeList-->flinkPlanner.validate(sqlNode)-->Optional<Operation> operation = SqlNodeConverters.convertSqlNode(validated, context);

2、operation转成pipeline(StreamGraph)

org.apache.flink.table.api.internal.TableEnvironmentImpl#executeQueryOperation

executeQueryOperation--》

2.1、operation转成Transformation

List<Transformation<?>> transformations = translate(Collections.singletonList(sinkOperation));--》

org.apache.flink.table.planner.delegation.PlannerBase#translate

2.2、Transformation转成pipeline

Pipeline pipeline =

execEnv.createPipeline(

transformations, tableConfig.getConfiguration(), defaultJobName);

2.3、pipeline转成streamGraph

final StreamGraph streamGraph = executionEnvironment.generateStreamGraph(transformations); // StreamGraph是Pipeline接口的实现类

3、将StreamGraph转成JobGraph

org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#executeAsync(org.apache.flink.streaming.api.graph.StreamGraph)

JobClient jobClient = execEnv.executeAsync(pipeline);

3.1、final PipelineExecutor executor = getPipelineExecutor();

CompletableFuture<JobClient> jobClientFuture =

executor.execute(streamGraph, configuration, userClassloader);

3.2、createJobGraph#new StreamingJobGraphGenerator(

userClassLoader, streamGraph, jobID, serializationExecutor)

.createJobGraph();

4、将jobGraph提交到集群

org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor#execute

4.1、获取集群信息

final ClusterClientProvider<ClusterID> clusterClientProvider =

clusterDescriptor.deployJobCluster(

clusterSpecification, jobGraph, configAccessor.getDetachedMode

4.2、提交任务到集群

return CompletableFuture.completedFuture(

new ClusterClientJobClientAdapter<>(

clusterClientProvider, jobGraph.getJobID(), userCodeClassloader));

相关推荐
一切顺势而行13 小时前
Flink cdc 使用总结
大数据·flink
expect7g15 小时前
Flink-反压-1.基本概念
后端·flink
Flink_China15 小时前
淘天AB实验分析平台Fluss落地实践:更适合实时OLAP的消息队列
大数据·flink
万能小锦鲤2 天前
《大数据技术原理与应用》实验报告七 熟悉 Spark 初级编程实践
hive·hadoop·ubuntu·flink·spark·vmware·实验报告
bxlj_jcj2 天前
Flink Watermark原理与实战
大数据·flink
張萠飛2 天前
flink sql如何对hive string类型的时间戳进行排序
hive·sql·flink
張萠飛2 天前
flink sql读hive catalog数据,将string类型的时间戳数据排序后写入kafka,如何保障写入kafka的数据是有序的
hive·sql·flink
蜜獾云2 天前
Flink双流实时对账
大数据·flink·linq
亿牛云爬虫专家2 天前
Kafka与Flink打造流式数据采集方案:以二手房信息为例
flink·kafka·数据采集·爬虫代理·数据处理·二手房·定时抓取
一切顺势而行3 天前
flink 和 spark 架构的对比
架构·flink·spark