Flink Checkpoint expired before completing解决方法

在Flink消费Kafka日志的时候出现了这样的一则报错，

JobManager报错如下：

java 复制代码

2024-03-07 15:21:12,500 [Checkpoint Timer] WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 181 for job 835243b848a64f2fae918faf23c5392c. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2438) [flink-dist-1.17-vvr-8.0.7-2-SNAPSHOT.jar:1.17-vvr-8.0.7-2-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_372]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_372]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_372]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_372]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_372]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP

TaskManager报错如下：

java 复制代码

2024-03-07 15:22:50,882 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 4673 for job 894b4dd298704c6b91bfaa2c06aa40b4. (2 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2259) [flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_102]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:186) [?:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:299) [?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147) [?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) [?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:834) [?:1.8.0_102]
2024-03-07 15:22:50,886 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold.

这个出错的原因主要是：CheckPoint超时导致的。

要解决这个问题：首先要从CheckPoint的原理说起：

JobManager在定时做CheckPoint操作的时候，CheckpointCoordinator会周期性的向所有source算子发送barrier
当Source算子接收到Barrier的时候，就会启动CheckPoint处理。有同步和异步两个过程，把当前的状态写入到持久化存储中，最后向CheckPointCoordinator报告快照制作情况，同事向下游算子广播该barrier，恢复数据处理
每个算子按照步骤3不断制作快照并且向下广播，直到最后的barrier传递到sink算子，Sink算子向CheckpointCoordinator报告快照完成，交给CheckpointCoordinator处理
当CheckpointCoordinator收到所有的算子报告之后，认为该周期的快照制作成功，否则，如果在规定的时间内没有收到算子的报告，超时了，则认为本周期快照制作失败。

现在回到我们的报错，这个明显是算子的Checkpoint时间超时了，点开CheckPoint节点一看，Checkpoint一直报错

刚好我们设置的Checkpoint超时是6秒，如下:

java 复制代码

env.getCheckpointConfig().setCheckpointTimeout(6000L)

为此本次报错主要解决方法有几种：

1.关闭Checkpoint，但是这个有风险

主要有2种方法：

第一种：直接关闭

java 复制代码

env.getCheckpointConfig().disableCheckpointing()

第二种：加大CheckPoint出错容忍告警次数

java 复制代码

env.getCheckpointConfig().setTolerableCheckpointFailureNumber(Integer.MAX_VALUE);

2.延长Checkpoint的超时时间并且把Checkpoint的间隔时间调长一点

java 复制代码

env.getCheckpointConfig().setCheckpointTimeout(60*1000L);//CK 1分钟延迟
env.enableCheckpointing(60*60*1000L); // CK间隔1小时做一次

3.这种方法如果不行，说明是CK过于复杂，需要较多的资源和时间，这个时候，可以考虑修改CK逻辑，使其尽量简便，也可以使用RockDB加快CK的速度