【flink】Checkpoint expired before completing.

使用flink同步数据出现错误Checkpoint expired before completing.

复制代码
11:32:34,455 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [Checkpoint Timer]  - Failed to trigger or complete checkpoint 4 for job 1b1d41031ea45d15bdb3324004c2d749. (2 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2143)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
11:32:34,459 INFO  org.jobslink.flink.sink.OperateMysqlDataSink                 [Source: CDC Sourceorg.jobslink.flink.TradeAndWorkTypeAndSkillsCDCJob -> (Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink jk_skills_base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink jk_trade_base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink jk_worktypes_base)) (1/1)#0]  - READ isExitSql is : [ SELECT count(1) count from jobslink_data_platform.src_skills_base where id= 1325753409319084034 ] 
11:32:34,468 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [flink-akka.actor.default-dispatcher-9]  - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold.
	at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:206)
	at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:169)
	at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:122)
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2082)
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2061)
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.access$600(CheckpointCoordinator.java:98)
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2143)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
11:32:34,470 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [flink-akka.actor.default-dispatcher-9]  - Job org.jobslink.flink.TradeAndWorkTypeAndSkillsCDCJob (1b1d41031ea45d15bdb3324004c2d749) switched from state RUNNING to RESTARTING.
11:32:34,471 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [flink-akka.actor.default-dispatcher-9]  - Source: CDC Sourceorg.jobslink.flink.TradeAndWorkTypeAndSkillsCDCJob -> (Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink base)) (1/1) (3525ceb58f2dc3264812966ec8600a19) switched from RUNNING to CANCELING.

任务超时了:

重新把任务配置参数,配置如下:

java 复制代码
//开启CK
env.getCheckpointConfig().setCheckpointTimeout(60000);
//设置定期安排检查点的时间间隔。
env.getCheckpointConfig().setCheckpointInterval(60000);
//设置可能同时进行的检查点尝试的最大次数
env.getCheckpointConfig().setMaxConcurrentCheckpoints(500);
//设置检查点尝试之间的最小暂停时间。
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);

或者修改

flink的 配置文件flink-conf.yaml

相关推荐
鸭鸭鸭进京赶烤5 小时前
大学专业科普 | 云计算、大数据
大数据·云计算
G皮T9 小时前
【Elasticsearch】自定义评分检索
大数据·elasticsearch·搜索引擎·查询·检索·自定义评分·_score
搞笑的秀儿12 小时前
信息新技术
大数据·人工智能·物联网·云计算·区块链
SelectDB12 小时前
SelectDB 在 AWS Graviton ARM 架构下相比 x86 实现 36% 性价比提升
大数据·架构·aws
二二孚日13 小时前
自用华为ICT云赛道Big Data第五章知识点-Flume海量日志聚合
大数据·华为
二二孚日15 小时前
自用华为ICT云赛道Big Data第四章知识点-Flink流批一体分布式实时处理引擎
大数据·华为
xufwind16 小时前
spark standlone 集群离线安装
大数据·分布式·spark
AI数据皮皮侠17 小时前
中国区域10m空间分辨率楼高数据集(全国/分省/分市/免费数据)
大数据·人工智能·机器学习·分类·业界资讯
昱禹17 小时前
Flutter 3.29+使用isar构建失败
大数据·flutter
DeepSeek大模型官方教程18 小时前
NLP之文本纠错开源大模型:兼看语音大模型总结
大数据·人工智能·ai·自然语言处理·大模型·产品经理·大模型学习