在上文《Flink-1.19.0源码详解5-JobGraph生成-前篇》已解析了JobGraph生成源码中递归创建算子链与JobVertex链头节点生成的内容,下面将继续解析JobGraph的JobEdge边创建、IntermediateDataSet数据集创建、算子链序列化、客户端请求提交的完整源码。
1.JobGraph生成核心方法
StreamingJobGraphGenerator.createJobGraph()方法为JobGraph生成的核心方法,包含了所有创建JobGraph的关键步骤。首先创建了算子链把可链的节点链起来并创建JobVertex链头节点,并对不可链的输出创建JobEdge连接前后JobVertex,将算子信息序列化到JobVertex中,并最终配置JobGraph的SlotSharingGroup、Checkpoint、SavepointRestore、ExecutionConfig完成JobGraph的创建。
源码图解:

StreamingJobGraphGenerator.createJobGraph()方法源码:
java
private JobGraph createJobGraph() {
//验证Checkpoint以及配置JobType、Dynamic
preValidate();
jobGraph.setJobType(streamGraph.getJobType());
jobGraph.setDynamic(streamGraph.isDynamic());
jobGraph.enableApproximateLocalRecovery(
streamGraph.getCheckpointConfig().isApproximateLocalRecoveryEnabled());
// Generate deterministic hashes for the nodes in order to identify them across
// submission iff they didn't change.
//创建用于标识各节点的哈希值
Map<Integer, byte[]> hashes =
defaultStreamGraphHasher.traverseStreamGraphAndGenerateHashes(streamGraph);
// Generate legacy version hashes for backwards compatibility
List<Map<Integer, byte[]>> legacyHashes = new ArrayList<>(legacyStreamGraphHashers.size());
for (StreamGraphHasher hasher : legacyStreamGraphHashers) {
legacyHashes.add(hasher.traverseStreamGraphAndGenerateHashes(streamGraph));
}
//把可链的节点链接起来
setChaining(hashes, legacyHashes);
if (jobGraph.isDynamic()) {
setVertexParallelismsForDynamicGraphIfNecessary();
}
// Note that we set all the non-chainable outputs configuration here because the
// "setVertexParallelismsForDynamicGraphIfNecessary" may affect the parallelism of job
// vertices and partition-reuse
final Map<Integer, Map<StreamEdge, NonChainedOutput>> opIntermediateOutputs =
new HashMap<>();
//对每个不能chain的输出进行序列化
setAllOperatorNonChainedOutputsConfigs(opIntermediateOutputs);
//遍历不可链的JobVertex生成JobEdge
setAllVertexNonChainedOutputsConfigs(opIntermediateOutputs);
//设置PhysicalEdges,将每个JobVertex的入边集合也序列化到该JobVertex的StreamConfig中
setPhysicalEdges();
markSupportingConcurrentExecutionAttempts();
validateHybridShuffleExecuteInBatchMode();
//为每个 JobVertex 设置所属的 SlotSharingGroup
setSlotSharingAndCoLocation();
setManagedMemoryFraction(
Collections.unmodifiableMap(jobVertices),
Collections.unmodifiableMap(vertexConfigs),
Collections.unmodifiableMap(chainedConfigs),
id -> streamGraph.getStreamNode(id).getManagedMemoryOperatorScopeUseCaseWeights(),
id -> streamGraph.getStreamNode(id).getManagedMemorySlotScopeUseCases());
//配置checkpoint
configureCheckpointing();
//配置savepointRestore
jobGraph.setSavepointRestoreSettings(streamGraph.getSavepointRestoreSettings());
final Map<String, DistributedCache.DistributedCacheEntry> distributedCacheEntries =
JobGraphUtils.prepareUserArtifactEntries(
streamGraph.getUserArtifacts().stream()
.collect(Collectors.toMap(e -> e.f0, e -> e.f1)),
jobGraph.getJobID());
for (Map.Entry<String, DistributedCache.DistributedCacheEntry> entry :
distributedCacheEntries.entrySet()) {
jobGraph.addUserArtifact(entry.getKey(), entry.getValue());
}
//配置ExecutionConfig
// set the ExecutionConfig last when it has been finalized
try {
jobGraph.setExecutionConfig(streamGraph.getExecutionConfig());
} catch (IOException e) {
throw new IllegalConfigurationException(
"Could not serialize the ExecutionConfig."
+ "This indicates that non-serializable types (like custom serializers) were registered");
}
jobGraph.setJobConfiguration(streamGraph.getJobConfiguration());
addVertexIndexPrefixInVertexName();
setVertexDescription();
//对每个链头节点,序列化其算子链,放到节点配置中
// Wait for the serialization of operator coordinators and stream config.
try {
FutureUtils.combineAll(
vertexConfigs.values().stream()
.map(
config ->
config.triggerSerializationAndReturnFuture(
serializationExecutor))
.collect(Collectors.toList()))
.get();
waitForSerializationFuturesAndUpdateJobVertices();
} catch (Exception e) {
throw new FlinkRuntimeException("Error in serialization.", e);
}
if (!streamGraph.getJobStatusHooks().isEmpty()) {
jobGraph.setJobStatusHooks(streamGraph.getJobStatusHooks());
}
//返回创建好的JobGraph
return jobGraph;
}
2.创建JobEdge边与IntermediateDataSet数据集
JobEdge与IntermediateDataSet的创建主要是通过遍历每个JobVertex链头节点所有不可链的StreamEdge,把每个不可链的边创建成IntermediateDataSet和JobEdge。

其中从setAllVertexNonChainedOutputsConfigs(opIntermediateOutputs)进入StreamingJobGraphGenerator.setAllVertexNonChainedOutputsConfigs()方法,开始进入JobEdge边与IntermediateDataSet数据集创建的源码。
源码图解:

StreamingJobGraphGenerator.setAllVertexNonChainedOutputsConfigs()方法首先对所有JobVertex进行遍历。
StreamingJobGraphGenerator.setAllVertexNonChainedOutputsConfigs()方法源码:
java
private void setAllVertexNonChainedOutputsConfigs(
final Map<Integer, Map<StreamEdge, NonChainedOutput>> opIntermediateOutputs) {
//对每个JobVertex遍历,生成JobEdge
jobVertices
.keySet()
.forEach(
startNodeId ->
setVertexNonChainedOutputsConfig(
startNodeId,
vertexConfigs.get(startNodeId),
chainInfos.get(startNodeId).getTransitiveOutEdges(),
opIntermediateOutputs));
}
对每个JobVertex节点,遍历其不可链的StreamEdge边,调用connect()进行边生成。
StreamingJobGraphGenerator.setVertexNonChainedOutputsConfig()方法源码:
java
private void setVertexNonChainedOutputsConfig(
Integer startNodeId,
StreamConfig config,
List<StreamEdge> transitiveOutEdges,
final Map<Integer, Map<StreamEdge, NonChainedOutput>> opIntermediateOutputs) {
//遍历该JobVertex所有不可链的StreamEdge,调用connect()进行边生成
LinkedHashSet<NonChainedOutput> transitiveOutputs = new LinkedHashSet<>();
for (StreamEdge edge : transitiveOutEdges) {
NonChainedOutput output = opIntermediateOutputs.get(edge.getSourceId()).get(edge);
transitiveOutputs.add(output);
//调用connect()
connect(startNodeId, edge, output);
}
}
StreamingJobGraphGenerator.connect()方法获取了当前JobVertex节点与下游JobVertex节点,获取节点的获取patitioner类型与shuffle模式,根据partitioner连接类型时POINTWISE还是ALL_TO_ALL具体生成JobEdge。
StreamingJobGraphGenerator.connect()方法源码:
java
private void connect(Integer headOfChain, StreamEdge edge, NonChainedOutput output) {
//在physicalEdgesInOrder中添加本edge
physicalEdgesInOrder.add(edge);
Integer downStreamVertexID = edge.getTargetId();
//找到本JobVertex的链头
JobVertex headVertex = jobVertices.get(headOfChain);
//获取本边连接的下游JobVertex
JobVertex downStreamVertex = jobVertices.get(downStreamVertexID);
//配置下游JobVertex的input加一
StreamConfig downStreamConfig = new StreamConfig(downStreamVertex.getConfiguration());
downStreamConfig.setNumberOfNetworkInputs(downStreamConfig.getNumberOfNetworkInputs() + 1);
//获取patitioner类型与shuffle模式
StreamPartitioner<?> partitioner = output.getPartitioner();
ResultPartitionType resultPartitionType = output.getPartitionType();
checkBufferTimeout(resultPartitionType, edge);
//根据partitioner连接类型具体生成JobEdge
JobEdge jobEdge;
if (partitioner.isPointwise()) {
jobEdge =
downStreamVertex.connectNewDataSetAsInput(
headVertex,
DistributionPattern.POINTWISE,
resultPartitionType,
output.getDataSetId(),
partitioner.isBroadcast());
} else {
jobEdge =
downStreamVertex.connectNewDataSetAsInput(
headVertex,
DistributionPattern.ALL_TO_ALL,
resultPartitionType,
output.getDataSetId(),
partitioner.isBroadcast());
}
//配置JobEdge
// set strategy name so that web interface can show it.
jobEdge.setShipStrategyName(partitioner.toString());
jobEdge.setForward(partitioner instanceof ForwardPartitioner);
jobEdge.setDownstreamSubtaskStateMapper(partitioner.getDownstreamSubtaskStateMapper());
jobEdge.setUpstreamSubtaskStateMapper(partitioner.getUpstreamSubtaskStateMapper());
if (LOG.isDebugEnabled()) {
LOG.debug(
"CONNECTED: {} - {} -> {}",
partitioner.getClass().getSimpleName(),
headOfChain,
downStreamVertexID);
}
}
JobVertex.connectNewDataSetAsInput()为具体创建JobEdge的方法,在方法中分别创建了IntermediateDataSet与JobEdge实例,并用JobEdge连接JobVertex与IntermediateDataSet。
JobVertex.connectNewDataSetAsInput()方法源码:
java
public JobEdge connectNewDataSetAsInput(
JobVertex input,
DistributionPattern distPattern,
ResultPartitionType partitionType,
IntermediateDataSetID intermediateDataSetId,
boolean isBroadcast) {
//生成IntermediateDataSet
IntermediateDataSet dataSet =
input.getOrCreateResultDataSet(intermediateDataSetId, partitionType);
//生成JobEdge
JobEdge edge = new JobEdge(dataSet, this, distPattern, isBroadcast);
//把JobEdge连接上JobVertex与IntermediateDataSet
this.inputs.add(edge);
dataSet.addConsumer(edge);
return edge;
}
其中IntermediateDataSet的创建在JobVertex.connectNewDataSetAsInput()方法中。
JobVertex.connectNewDataSetAsInput()方法源码:
java
public IntermediateDataSet getOrCreateResultDataSet(
IntermediateDataSetID id, ResultPartitionType partitionType) {
return this.results.computeIfAbsent(
//创建IntermediateDataSet
id, key -> new IntermediateDataSet(id, partitionType, this));
}
3.序列化算子计算逻辑、算子链、链输出、链内部边
JobGraph为Flink计算逻辑图,需要保存算子的计算信息,并从Flink客户端发送到Yarn上的Flink集群端,进行分布式执行。因此,在发送JobGraph前,需在每个JobVertex节点序列化JobVertex的计算逻辑、JobVertex的算子链、链输出和链内部边。

在StreamingJobGraphGenerator.createChain()方法下
首先在StreamingJobGraphGenerator.createChain()递归每个节点创建算子链时,会将节点的封装计算逻辑的StreamOperatorFactory和其可链边ChainedOutputs放入保存其需要序列化信息的Map:toBeSerializedConfigObjects中,并在后面对其进行序列化。
源码图解:
StreamingJobGraphGenerator.createChain()方法源码
java
private List<StreamEdge> createChain(
final Integer currentNodeId,
final int chainIndex,
final OperatorChainInfo chainInfo,
final Map<Integer, OperatorChainInfo> chainEntryPoints) {
//递归创建算子链...
//...
//方法内包含配置StreamOperatorFactory的序列化的代码
setOperatorConfig(currentNodeId, config, chainInfo.getChainedSources());
//配置不可链边的序列化
setOperatorChainedOutputsConfig(config, chainableOutputs);
//...
}
算子计算逻辑序列化
在StreamingJobGraphGenerator.setOperatorConfig()方法中执行setStreamOperatorFactory(vertex.getOperatorFactory())配置StreamOperatorFactory的序列化。
StreamingJobGraphGenerator.setOperatorConfig()方法源码:
java
private void setOperatorConfig(Integer vertexId, StreamConfig config, Map<Integer, ChainedSourceInfo> chainedSources) {
//...
//配置StreamOperatorFactory序列化
config.setStreamOperatorFactory(vertex.getOperatorFactory());
//...
}
StreamConfig.setStreamOperatorFactory()方法配置了StreamOperatorFactory的序列化,把StreamOperatorFactory放入保存序列化信息的Map:toBeSerializedConfigObjects中,并在后续进行具体的序列化操作。
StreamConfig.setStreamOperatorFactory()方法源码:
java
public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
if (factory != null) {
//配置OperatorFactory的序列化
toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
toBeSerializedConfigObjects.put(SERIALIZED_UDF_CLASS, factory.getClass());
}
}
算子可链边的序列化
在StreamingJobGraphGenerator.setOperatorConfig()方法中执行setOperatorChainedOutputsConfig(config, chainableOutputs)配置节点可链边ChainedOutputs的序列化。
StreamingJobGraphGenerator.setOperatorChainedOutputsConfig方法源码:
java
private void setOperatorChainedOutputsConfig(
StreamConfig config, List<StreamEdge> chainableOutputs) {
//为每个边创建序列化器
for (StreamEdge edge : chainableOutputs) {
if (edge.getOutputTag() != null) {
config.setTypeSerializerSideOut(
edge.getOutputTag(),
edge.getOutputTag()
.getTypeInfo()
.createSerializer(
streamGraph.getExecutionConfig().getSerializerConfig()));
}
}
//配置可链边的序列化
config.setChainedOutputs(chainableOutputs);
}
StreamConfig.setStreamOperatorFactory()方法配置了节点可链边ChainedOutputs的序列化,把可链边ChainedOutputs放入保存序列化信息的Map:toBeSerializedConfigObjects中,并在后续进行具体的序列化操作。
StreamConfig.setChainedOutputs()方法源码:
java
public void setChainedOutputs(List<StreamEdge> chainedOutputs) {
//配置可链边的序列化
toBeSerializedConfigObjects.put(CHAINED_OUTPUTS, chainedOutputs);
}
在StreamingJobGraphGenerator.createJobGraph()方法下
StreamingJobGraphGenerator的createJobGraph()方法为构造JobGraph主要方法,当在进行JobGraph创建时,会遍历JobGraph中所有节点的不可链的输出进行序列化,并把算子链的数据都序列化到链头的JobVertex节点的配置中,序列化每个JobVertex中toBeSerializedConfigObjects定义的配置项,完成JobGraph最终的序列化。
源码图解:

StreamingJobGraphGenerator.createJobGraph()方法源码:
java
private JobGraph createJobGraph() {
//...
//创建算子链与每个JobVertex链头节点
setChaining(hashes, legacyHashes);
//配置所有不可链边的序列化
setAllOperatorNonChainedOutputsConfigs(opIntermediateOutputs);
//...
//对每个节点序列化其配置与算子链,完成最终的序列化
// Wait for the serialization of operator coordinators and stream config.
try {
FutureUtils.combineAll(
vertexConfigs.values().stream()
.map(
config ->
config.triggerSerializationAndReturnFuture(
serializationExecutor))
.collect(Collectors.toList()))
.get();
waitForSerializationFuturesAndUpdateJobVertices();
} catch (Exception e) {
throw new FlinkRuntimeException("Error in serialization.", e);
}
//...
//最终把创建好的JobGraph进行返回
return jobGraph;
}
序列化不可链输出:
在StreamingJobGraphGenerator.createJobGraph()方法中执行了setAllOperatorNonChainedOutputsConfigs(opIntermediateOutputs)对JobGraph中所有不可链输出配置了序列化。StreamingJobGraphGenerator的setAllOperatorNonChainedOutputsConfigs()方法对所有不可链输出进行遍历,把不可链输出配置到其前置节点的需要序列化的配置中。
StreamingJobGraphGenerator.setAllOperatorNonChainedOutputsConfigs()方法源码:
java
private void setAllOperatorNonChainedOutputsConfigs(
final Map<Integer, Map<StreamEdge, NonChainedOutput>> opIntermediateOutputs) {
// set non chainable output config
//遍历每个不可链的输出
opNonChainableOutputsCache.forEach(
(vertexId, nonChainableOutputs) -> {
Map<StreamEdge, NonChainedOutput> outputsConsumedByEdge =
opIntermediateOutputs.computeIfAbsent(
vertexId, ignored -> new HashMap<>());
//序列化到他前一个节点的配置中
setOperatorNonChainedOutputsConfig(
vertexId,
vertexConfigs.get(vertexId),
nonChainableOutputs,
outputsConsumedByEdge);
});
}
setOperatorNonChainedOutputsConfig()方法创建了序列化器,并把不可链输出配置在前置节点的序列化配置中。
StreamingJobGraphGenerator.setOperatorNonChainedOutputsConfig()方法源码:
java
private void setOperatorNonChainedOutputsConfig(
Integer vertexId,
StreamConfig config,
List<StreamEdge> nonChainableOutputs,
Map<StreamEdge, NonChainedOutput> outputsConsumedByEdge) {
//为每个输出创建序列化器
// iterate edges, find sideOutput edges create and save serializers for each outputTag type
for (StreamEdge edge : nonChainableOutputs) {
if (edge.getOutputTag() != null) {
config.setTypeSerializerSideOut(
edge.getOutputTag(),
edge.getOutputTag()
.getTypeInfo()
.createSerializer(
streamGraph.getExecutionConfig().getSerializerConfig()));
}
}
//序列化到他前一个节点的配置中
List<NonChainedOutput> deduplicatedOutputs =
mayReuseNonChainedOutputs(vertexId, nonChainableOutputs, outputsConsumedByEdge);
config.setNumberOfOutputs(deduplicatedOutputs.size());
config.setOperatorNonChainedOutputs(deduplicatedOutputs);
}
StreamConfig.setStreamOperatorFactory()方法配置了不可链边NonChainedOutput的序列化,把NonChainedOutput放入保存序列化信息的Map:toBeSerializedConfigObjects中,并在后续进行具体的序列化操作。
StreamConfig.setOperatorNonChainedOutputs方法源码:
java
public void setOperatorNonChainedOutputs(List<NonChainedOutput> nonChainedOutputs) {
//进行序列化
toBeSerializedConfigObjects.put(OP_NONCHAINED_OUTPUTS, nonChainedOutputs);
}
序列化算子链并最终执行序列化
在StreamingJobGraphGenerator.createJobGraph()方法中,遍历了JobGraph中的每个JobVertex节点,对每个链头节点的算子链和配置信息进行序列化。
java
//对每个链头节点,序列化其算子链,放到节点配置中
FutureUtils.combineAll(
vertexConfigs.values().stream()
.map(
config ->
config.triggerSerializationAndReturnFuture(
serializationExecutor))
.collect(Collectors.toList()))
.get();
waitForSerializationFuturesAndUpdateJobVertices();
在StreamConfig的triggerSerializationAndReturnFuture()方法中,首先先调用serializeAllConfigs()方法对JobVertex节点的配置进行了序列化,再序列化了JobVertex节点的算子链。
StreamConfig.triggerSerializationAndReturnFuture()方法源码:
java
public CompletableFuture<StreamConfig> triggerSerializationAndReturnFuture(
Executor ioExecutor) {
FutureUtils.combineAll(chainedTaskFutures.values())
.thenAcceptAsync(
chainedConfigs -> {
try {
// Serialize all the objects to config.
//序列化节点的StreamConfig
serializeAllConfigs();
//将算子链配置写入对应节点
InstantiationUtil.writeObjectToConfig(
chainedConfigs.stream()
.collect(
Collectors.toMap(
StreamConfig::getVertexID,
Function.identity())),
this.config,
CHAINED_TASK_CONFIG);
serializationFuture.complete(this);
} catch (Throwable throwable) {
serializationFuture.completeExceptionally(throwable);
}
},
ioExecutor);
return serializationFuture;
}
StreamConfig.serializeAllConfigs()方法将之前需要序列化的Map:toBeSerializedConfigObjects配置逐项遍历,进行序列化。包含对上述解析的封装节点计算逻辑的StreamOperatorFactory、节点可链边ChainedOutputs、节点不可链输出NonChainedOutput进行具体的序列化操作。
StreamConfig.serializeAllConfigs()方法源码:
java
public void serializeAllConfigs() {
toBeSerializedConfigObjects.forEach(
(key, object) -> {
try {
//将需要序列化的配置进行序列化
InstantiationUtil.writeObjectToConfig(object, this.config, key);
} catch (IOException e) {
throw new StreamTaskException(
String.format("Could not serialize object for key %s.", key), e);
}
});
}
InstantiationUtil.writeObjectToConfig()方法为具体的序列化方法,将配置对象序列化成字节数组,并把序列化后的数据写入JobVertex的配置中。
InstantiationUtil.writeObjectToConfig()方法源码:
java
public static void writeObjectToConfig(Object o, Configuration config, String key)
throws IOException {
//序列化对象
byte[] bytes = serializeObject(o);
//将序列化的结果写入配置中
config.setBytes(key, bytes);
}
自此JobGraph生成已完成,完整的JobGraph结构如下:
完整JobGraph结构:
4.客户端向集群提交任务
完成JobGraph生成后,Flink CliFrontend客户端通过ClusterClient向HDFS上传JobGraph、Jar、Artifacts,向Yarn的Flink集群提交Request,完成Flink客户端调度,后续将进行Flink的集群端的调度与执行。
源码图解:

回到AbstractSessionClusterExecutor.execute()方法,当完成了JobGraph生成后,Flink创建了RestClusterClient客户端(ClusterClient的具体实现),向集群提交JobGraph。
AbstractSessionClusterExecutor.execute()方法源码:
java
public CompletableFuture<JobClient> execute(
@Nonnull final Pipeline pipeline,
@Nonnull final Configuration configuration,
@Nonnull final ClassLoader userCodeClassloader)
throws Exception {
//生成JobGraph
final JobGraph jobGraph =
PipelineExecutorUtils.getJobGraph(pipeline, configuration, userCodeClassloader);
//...
//建立ClusterClient,向集群提交JobGraph
ClusterClient<ClusterID> clusterClient = clusterClientProvider.getClusterClient();
//Client向集群提交JobGraph
clusterClient.submitJob(jobGraph)
//...
}
RestClusterClient.submitJob()方法为Flink任务提交的具体方法,创建了jobGraph文件并将其写到本地目录,将本地的jobGraph文件、jar、artifact上传到集群的HDFS文件系统,构建JobSubmit的请求体并向Flink集群发送请求。
RestClusterClient.submitJob()方法源码:
java
public CompletableFuture<JobID> submitJob(@Nonnull JobGraph jobGraph) {
//...
//创建jobGraph文件(flink-jobgraph-id.bin)
java.nio.file.Path jobGraphFile =
Files.createTempFile("flink-jobgraph-" +
jobGraph.getJobID(), ".bin");
//...
//把jobGraph文件写到本地文件目录
ObjectOutputStream objectOut = new ObjectOutputStream(Files.newOutputStream(
jobGraphFile))) {objectOut.writeObject(jobGraph);}
//...
//将本地的jobGraph文件、jar、artifact上传hdfs
filesToUpload.add( new FileUpload(jobGraphFile,
RestConstants.CONTENT_TYPE_BINARY));
filesToUpload.add(new FileUpload(Paths.get(jar.toUri()),
RestConstants.CONTENT_TYPE_JAR));
filesToUpload.add(new FileUpload(Paths.get(
artifactFilePath.getPath()),
RestConstants.CONTENT_TYPE_BINARY));
//...
//构建JobSubmit的请求体
final JobSubmitRequestBody requestBody =
new JobSubmitRequestBody(
jobGraphFile.getFileName().toString(),
jarFileNames,
artifactFileNames);
//...
//向集群发送JobSubmit请求
sendRetriableRequest(
JobSubmitHeaders.getInstance(),
EmptyMessageParameters.getInstance(),
requestAndFileUploads.f0,
requestAndFileUploads.f1,
isConnectionProblemOrServiceUnavailable(),
(receiver, error) -> {//异常处理}
);
//...
//删除本地生成的jobGraph文件
Files.delete(jobGraphFile);
}
RestClusterClient.sendRetriableRequest()为请求发送的方法,最终RestClient客户端通过向Yarn中Flink集群发送请求,完成Flink的客户端调度并开启了Flink的集群端调度。
RestClusterClient.sendRetriableRequest()方法源码:
java
private <
M extends MessageHeaders<R, P, U>,
U extends MessageParameters,
R extends RequestBody,
P extends ResponseBody>
CompletableFuture<P> sendRetriableRequest(
M messageHeaders,
U messageParameters,
R request,
Collection<FileUpload> filesToUpload,
Predicate<Throwable> retryPredicate,
BiConsumer<String, Throwable> consumer) {
//...
restClient.sendRequest(
webMonitorBaseUrl.getHost(),
webMonitorBaseUrl.getPort(),
headers,
messageParameters,
request,
filesToUpload);
}
至此,JobGraph生成及Flink客户端调度部分已完成,JobGraph生成的完整代码解析和完整的JobGraph结构如下:
完整代码解析:

完整JobGraph结构:
5.结语
本文及上篇博文《Flink-1.19.0源码详解5-JobGraph生成-前篇》已完整解析了JobGraph的生成源码,Flink客户端在JobGraph生成后,把调度请求和程序Jar包及依赖向Yarn中的Flink集群提交,完成Flink客户端调度,下篇博文将开始解析Flink集群端调度,继续解析ExecutionGraph生成与Task执行的源码。