【flink状态管理(四)】MemoryStateBackend的实现

文章目录

在Flink中,默认的StateBackend实现为MemoryStateBackend,本文以MemoryStateBackend为例说明StateBackend的设计与实现。

本文介绍MemoryStateBackend中如下三个主要组件的创建过程:

  • HeapKeyedStateBackend
  • OperatorStateBackend
  • MemoryBackendCheckpointStorage

FsStateBackend和RocksDBStateBackend这两种状态后端存储的实现,功能和MemoryStateBackend类似,区别在于内部创建的KeyedStateBackend和CheckpointStorage。

1.基于MemoryStateBackend创建KeyedStateBackend

1.1. 状态初始化

AbstractStreamOperator.keyedStatedBackend()方法定义了创建和初始化KeyedStatedBackend的逻辑,具体如下。

java 复制代码
protected <K> AbstractKeyedStateBackend<K> keyedStateBackend(
   TypeSerializer<K> keySerializer,
   String operatorIdentifierText,
   PrioritizedOperatorSubtaskState prioritizedOperatorSubtaskStates,
   CloseableRegistry backendCloseableRegistry,
   MetricGroup metricGroup) throws Exception {
   if (keySerializer == null) {
      return null;
   }
   String logDescription = "keyed state backend for " + operatorIdentifierText;
   //1. 
   TaskInfo taskInfo = environment.getTaskInfo();
   final KeyGroupRange keyGroupRange = 
       KeyGroupRangeAssignment.computeKeyGroupRangeForOperatorIndex(
      taskInfo.getMaxNumberOfParallelSubtasks(),
      taskInfo.getNumberOfParallelSubtasks(),
      taskInfo.getIndexOfThisSubtask());
   // 确保恢复状态过程中构建的数据流被关闭
   CloseableRegistry cancelStreamRegistryForRestore = new CloseableRegistry();
   backendCloseableRegistry.registerCloseable(cancelStreamRegistryForRestore);
   // 创建BackendRestorerProcedure
   BackendRestorerProcedure<AbstractKeyedStateBackend<K>, KeyedStateHandle> 
      backendRestorer =
      new BackendRestorerProcedure<>(
         (stateHandles) -> stateBackend.createKeyedStateBackend(
            environment,
            environment.getJobID(),
            operatorIdentifierText,
            keySerializer,
            taskInfo.getMaxNumberOfParallelSubtasks(),
            keyGroupRange,
            environment.getTaskKvStateRegistry(),
            TtlTimeProvider.DEFAULT,
            metricGroup,
            stateHandles,
            cancelStreamRegistryForRestore),
         backendCloseableRegistry,
         logDescription);
   try {
      return backendRestorer.createAndRestore(
         prioritizedOperatorSubtaskStates.getPrioritizedManagedKeyedState());
   } finally {
      if (backendCloseableRegistry.unregisterCloseable(cancelStreamRegistryFor
         Restore)) {
         IOUtils.closeQuietly(cancelStreamRegistryForRestore);
      }
   }
}
  1. 获取当前Task的TaskInfo,并基于TaskInfo的参数创建KeyGroupRange,表示当前Task实例中存储的Key分组区间
  2. 创建CloseableRegistry并注册到backendCloseableRegistry中,用于确保在任务取消的情况下关闭在恢复状态过程中构造的数据流。
  3. 创建BackendRestorerProcedure,提供了stateBackend.createKeyedStateBackend()方法,也包含恢复历史状态数据的方法。
  4. 创建KeyedStateBackend,同时对状态数据进行恢复。prioritizedOperatorSubtaskStates是从TaskStateManager中根据OperatorID获取的算子历史状态,通过prioritizedOperatorSubtaskStates获取当前算子的PrioritizedManagedKeyedState,并基于这些状态数据恢复算子的状态。

1.2. 创建状态

接下来我们看MemoryStateBackend.createKeyedStateBackend()方法的具体实现。

java 复制代码
public <K> AbstractKeyedStateBackend<K> createKeyedStateBackend(
   Environment env,
   JobID jobID,
   String operatorIdentifier,
   TypeSerializer<K> keySerializer,
   int numberOfKeyGroups,
   KeyGroupRange keyGroupRange,
   TaskKvStateRegistry kvStateRegistry,
   TtlTimeProvider ttlTimeProvider,
   MetricGroup metricGroup,
   @Nonnull Collection<KeyedStateHandle> stateHandles,
   CloseableRegistry cancelStreamRegistry) throws BackendBuildingException {
   // 获取TaskStateManager实例
   TaskStateManager taskStateManager = env.getTaskStateManager();
   // 创建HeapPriorityQueueSetFactory实例
   HeapPriorityQueueSetFactory priorityQueueSetFactory =
      new HeapPriorityQueueSetFactory(keyGroupRange, numberOfKeyGroups, 128);
   // 创建HeapKeyedStateBackendBuilder实例HeapKeyedStateBackend
   return new HeapKeyedStateBackendBuilder<>(
      kvStateRegistry,
      keySerializer,
      env.getUserClassLoader(),
      numberOfKeyGroups,
      keyGroupRange,
      env.getExecutionConfig(),
      ttlTimeProvider,
      stateHandles,
      AbstractStateBackend.getCompressionDecorator(env.getExecutionConfig()),
      taskStateManager.createLocalRecoveryConfig(),
      priorityQueueSetFactory,
      isUsingAsynchronousSnapshots(),
      cancelStreamRegistry).build();
}
  1. 从environment参数中获取TaskStateManager实例
  2. 创建HeapPriorityQueueSetFactory实例,用于生成HeapPriorityQueueSet优先级队列,存储TimerHeapInternalTimer等数据。
  3. 调用HeapKeyedStateBackendBuilder.build()方法创建HeapKeyedStateBackend。

2. 基于MemoryStateBackend创建OperatorStateBackend

和创建KeyedStateBackend的过程相似,AbstractStreamOperator.operatorStateBackend()方法实现了创建OperatorStateBackend的方法。

java 复制代码
protected OperatorStateBackend operatorStateBackend(
   String operatorIdentifierText,
   PrioritizedOperatorSubtaskState prioritizedOperatorSubtaskStates,
   CloseableRegistry backendCloseableRegistry) throws Exception {
   String logDescription = "operator state backend for " + operatorIdentifierText;
   CloseableRegistry cancelStreamRegistryForRestore = new CloseableRegistry();
   backendCloseableRegistry.registerCloseable(cancelStreamRegistryForRestore);
   BackendRestorerProcedure<OperatorStateBackend, OperatorStateHandle> 
      backendRestorer =
      new BackendRestorerProcedure<>(
         (stateHandles) -> stateBackend.createOperatorStateBackend(
            environment,
            operatorIdentifierText,
            stateHandles,
            cancelStreamRegistryForRestore),
         backendCloseableRegistry,
         logDescription);
   try {
      return backendRestorer.createAndRestore(
         prioritizedOperatorSubtaskStates.getPrioritizedManagedOperatorState());
   } finally {
      if (backendCloseableRegistry.unregisterCloseable(cancelStreamRegistryFor
         Restore)) {
         IOUtils.closeQuietly(cancelStreamRegistryForRestore);
      }
   }
}
  1. 创建CloseableRegisty,确保在任务取消的情况下能够关闭在恢复状态时构造的数据流。
  2. 创建BackendRestorerProcedure,封装了stateBackend.createOperatorStateBackend()方法,并包含恢复历史状态数据的操作。
  3. 创建OperatorStateBackend,并恢复状态数据。

其中prioritizedOperatorSubtaskStates是从TaskStateManager中根据OperatorID获取的算子专有历史状态,可以通过prioritizedOperatorSubtaskStates获取当前算子中的PrioritizedManagedOperatorState,并基于这些状态数据恢复OperatorStateBackend中算子的状态。

3.基于MemoryStateBackend创建CheckpointStorage

在createCheckpointStorage()方法中,直接创建MemoryBackendCheckpointStorage实例并返回,没有涉及太多的流程

java 复制代码
public CheckpointStorage createCheckpointStorage(JobID jobId) throws IOException {
   return new MemoryBackendCheckpointStorage(jobId, getCheckpointPath(), 
      getSavepointPath(), maxStateSize);
}

参考:《Flink设计与实现:核心原理与源码解析》--张利兵

相关推荐
.生产的驴7 分钟前
SpringBoot 封装统一API返回格式对象 标准化开发 请求封装 统一格式处理
java·数据库·spring boot·后端·spring·eclipse·maven
猿周LV15 分钟前
JMeter 安装及使用 [软件测试工具]
java·测试工具·jmeter·单元测试·压力测试
晨集17 分钟前
Uni-App 多端电子合同开源项目介绍
java·spring boot·uni-app·电子合同
AnsenZhu19 分钟前
2025年Redis分片存储性能优化指南
数据库·redis·性能优化·分片
时间之城19 分钟前
笔记:记一次使用EasyExcel重写convertToExcelData方法无法读取@ExcelDictFormat注解的问题(已解决)
java·spring boot·笔记·spring·excel
椰羊~王小美26 分钟前
LeetCode -- Flora -- edit 2025-04-25
java·开发语言
凯酱34 分钟前
MyBatis-Plus分页插件的使用
java·tomcat·mybatis
oydcm36 分钟前
MySQL数据库概述
数据库·mysql
程序员总部44 分钟前
如何在IDEA中高效使用Test注解进行单元测试?
java·单元测试·intellij-idea
oioihoii1 小时前
C++23中if consteval / if not consteval (P1938R3) 详解
java·数据库·c++23