探究 | RocketMQ offset管理机制~

Code皮皮虾2023-08-29 10:17

前言

大家好哇，我是Code皮皮虾，今天跟大家一起了解下RocketMq的offset管理机制~

什么是offSet？

一个Topic会存在多个messageQueue，每个messageQueue都会被Consumer消费，那么Consumer该从哪开始消费呢？

这就涉及到offset了，每个messageQueue会记录一个offset表示当前的消费进度，Consumer直接从offset开始消费即可。

借用一下网上的图~

Broker管理Offset

持久化

broker在启动时，会开启定时任务，每隔5秒将ConsumerOffsetManager中的offsetTable持久化到consumerOffset.json文件中

java 复制代码 
public boolean initialize() throws CloneNotSupportedException {

  // .....

  this.scheduledExecutorService.scheduleAtFixedRate(() -> {
    try {
      // 持久化
      BrokerController.this.consumerOffsetManager.persist();
    } catch (Throwable e) {
      log.error("schedule persist consumerOffset error.", e);
    }
  }, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
  
  // .....
  
}

那么offsetTable是个什么东西？为什么要专门将它进行持久化呢？

java 复制代码 
public class ConsumerOffsetManager extends ConfigManager {

  protected ConcurrentMap<String/* topic@group */, ConcurrentMap<Integer, Long>> offsetTable =
    new ConcurrentHashMap<String, ConcurrentMap<Integer, Long>>(512);
  
}

进入到ConsumerOffsetManager中，我们可以看到offsetTable本质是一个map结构，key为topic@group，value为一个key是队列下标、value为消费偏移量offset的map结构

具体的持久化文件结构如下👇🏻

json 复制代码 
{
	"offsetTable":{
		"TopicTest@please_rename_unique_group_name_4":{0:4,1:4,2:4,3:4,4:4,5:2,6:2,7:1,8:3,9:1,10:0,11:2,12:2,13:3,14:3,15:3
		},
		"%RETRY%please_rename_unique_group_name_4@please_rename_unique_group_name_4":{0:0
		},
		"TopicTest-2@please_rename_unique_group_name_4":{0:25,1:25,2:25,3:25,4:25,5:25,6:25,7:25
		},
		"TopicTest-1@please_rename_unique_group_name_4":{0:12,1:12,2:12,3:12,4:13,5:13,6:13,7:13
		}
	}
}

offset文件默认存储路径 ：$user.home/store/config/consumerOffset.json

更新

上面我们了解了broker对于offsetTable的持久化，下面我们继续来了解offsetTable是如何被更新的

当消费者消费消息后，会将offset发送给broker，broker接收后会更新offsetTable，接着交由定时任务进行持久化

而broker则是通过ConsumerManageProcessor来处理更新offset请求的

java 复制代码 
public class ConsumerManageProcessor extends AsyncNettyRequestProcessor {

  @Override
  public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    switch (request.getCode()) {
      case RequestCode.GET_CONSUMER_LIST_BY_GROUP:
        return this.getConsumerListByGroup(ctx, request);
      case RequestCode.UPDATE_CONSUMER_OFFSET:
        // todo 更新consumer offset
        return this.updateConsumerOffset(ctx, request);
      case RequestCode.QUERY_CONSUMER_OFFSET:
        // todo 查询consumer消费最新offset
        return this.queryConsumerOffset(ctx, request);
      default:
        break;
    }
    return null;
  }
}

updateConsumerOffset中，从requestHeader中拿到consumerGroup、topic、queueId、offset等信息，调用ConsumerOffsetManager#commitOffset来更新offsetTable

java 复制代码 
private RemotingCommand updateConsumerOffset(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    final RemotingCommand response =
        RemotingCommand.createResponseCommand(UpdateConsumerOffsetResponseHeader.class);
    final UpdateConsumerOffsetRequestHeader requestHeader =
        (UpdateConsumerOffsetRequestHeader) request
            .decodeCommandCustomHeader(UpdateConsumerOffsetRequestHeader.class);

  // 从requestHeader中拿到consumerGroup、topic、queueId、offset
  this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), requestHeader.getConsumerGroup(),
        requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}

到commitOffset这里就比较明确了，根据topic、consumerGroup拼接好offsetTable对应的key，然后更新offset即可

java 复制代码 
public void commitOffset(final String clientHost, final String group, final String topic, final int queueId,
                         final long offset) {
  // 拼接offsetTable key: topic@group
  String key = topic + TOPIC_GROUP_SEPARATOR + group;
  this.commitOffset(clientHost, key, queueId, offset);
}

private void commitOffset(final String clientHost, final String key, final int queueId, final long offset) {
  ConcurrentMap<Integer, Long> map = this.offsetTable.get(key);
  if (null == map) {
    // 如果key对应的map不存在，则需要初始化
    map = new ConcurrentHashMap<Integer, Long>(32);
    
    // 更新map
    map.put(queueId, offset);
    this.offsetTable.put(key, map);
  } else {
    // 更新map
    Long storeOffset = map.put(queueId, offset);
    if (storeOffset != null && offset < storeOffset) {
      log.warn("[NOTIFYME]update consumer offset less than store. clientHost={}, key={}, queueId={}, requestOffset={}, storeOffset={}", clientHost, key, queueId, offset, storeOffset);
    }
  }
}

Consumer管理Offset

Broker管理offset的机制我们了解完了，接下来继续看看Consumer是如何管理offset的

offset初始化

consumer在消费前需要进行重平衡，重平衡后可能会为当前的Consumer分配新的messageQueue，那么此时对于该messageQueue来说不存在当前Consumer的消费进度，故需要进行offset初始化~

java 复制代码 
public long computePullFromWhereWithException(MessageQueue mq) throws MQClientException {
  long result = -1;
  final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
  final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
  switch (consumeFromWhere) {
    case CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST:
    case CONSUME_FROM_MIN_OFFSET:
    case CONSUME_FROM_MAX_OFFSET:
      // 从最新的偏移量开始消费
    case CONSUME_FROM_LAST_OFFSET: {
      // 从store中读取offset，即从broker的offsetTable中获取
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        // >= 0 直接返回
        result = lastOffset;
      }
      // 等于-1，代表首次启动，还没有offset
      else if (-1 == lastOffset) {
        if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
          result = 0L;
        } else {
          try {
            // 获取当前该消息队列当前最大偏移量
            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        }
      } else {
        // 小于-1，表示该消息进度文件中存储了错误的偏移，返回-1
        result = -1;
      }
      break;
    }
      // 从头开始消费
    case CONSUME_FROM_FIRST_OFFSET: {
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        result = lastOffset;
      } else if (-1 == lastOffset) {
        // 首次启动，还没有offset，从头开始消费
        result = 0L;
      } else {
        result = -1;
      }
      break;
    }
      // 从Consumer启动时间戳对应的消费进度开始消费
    case CONSUME_FROM_TIMESTAMP: {
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        result = lastOffset;
      } else if (-1 == lastOffset) {
        if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
          try {
            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        } else {
          try {
            // 获取消费者启动时的时间戳
            long timestamp = UtilAll.parseDate(this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeTimestamp(),
                                               UtilAll.YYYYMMDDHHMMSS).getTime();
            // 搜索时间戳对应的offset
            result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        }
      } else {
        result = -1;
      }
      break;
    }

    default:
      break;
  }

  return result;
}

学习上面源码，结合注释，我们可将offset的分配大致分为三种 ，CONSUME_FROM_LAST_OFFSET、CONSUME_FROM_FIRST_OFFSET、CONSUME_FROM_TIMESTAMP

三者对于消费进度存在 或者消费进度不合理的处理方式是一致的

  • 当消费进度存在时，直接返回最新的消费offset
  • 当消费进度不合理时，直接返回-1

唯一的区别就在于消费进度不存在时的处理 ，当消费进度不存在时，需要进行offset的初始化，即👇🏻的分配方式

  1. CONSUME_FROM_LAST_OFFSET: 从最新的偏移量开始消费
  2. CONSUME_FROM_FIRST_OFFSET: 从头开始消费
  3. CONSUME_FROM_TIMESTAMP: 从Consumer启动时间戳对应的消费进度开始消费

拉取消息

offset初始化完毕后就可以去拉取消息了~

java 复制代码 
public void pullMessage(final PullRequest pullRequest) {
  
  // ......
  
  
	// todo 构造拉取消息回调函数
  PullCallback pullCallback = new PullCallback() {
    @Override
    public void onSuccess(PullResult pullResult) {
      if (pullResult != null) {
        pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                                                                                     subscriptionData);
        switch (pullResult.getPullStatus()) {
          case FOUND:
            // 拉取成功
            long prevRequestOffset = pullRequest.getNextOffset();
            // todo 更新下一次拉取的offset
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());
            long pullRT = System.currentTimeMillis() - beginTimestamp;
            DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                                                                               pullRequest.getMessageQueue().getTopic(), pullRT);

            long firstMsgOffset = Long.MAX_VALUE;
            if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
              DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
            } else {
              firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

              DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                                                                  pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

              boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
              
              // todo 提交消费请求，此时Consumer才会去消费拉取到的消息
              DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                pullResult.getMsgFoundList(),
                processQueue,
                pullRequest.getMessageQueue(),
                dispatchToConsume);

              if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                                                       DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
              } else {
                // todo 再次将PullRequest放入队列中
                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
              }
            }

            if (pullResult.getNextBeginOffset() < prevRequestOffset
                || firstMsgOffset < prevRequestOffset) {
              log.warn(
                "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                pullResult.getNextBeginOffset(),
                firstMsgOffset,
                prevRequestOffset);
            }

            break;
          case NO_NEW_MSG:
          case NO_MATCHED_MSG:
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

            DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
            break;
          case OFFSET_ILLEGAL:
            log.warn("the pull request offset illegal, {} {}",
                     pullRequest.toString(), pullResult.toString());
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

            pullRequest.getProcessQueue().setDropped(true);
            DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

              @Override
              public void run() {
                try {
                  DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                                                          pullRequest.getNextOffset(), false);

                  DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                  DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                  log.warn("fix the pull request offset, {}", pullRequest);
                } catch (Throwable e) {
                  log.error("executeTaskLater Exception", e);
                }
              }
            }, 10000);
            break;
          default:
            break;
        }
      }
    }

    @Override
    public void onException(Throwable e) {
      if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
        log.warn("execute the pull request exception", e);
      }

      if (e instanceof MQBrokerException && ((MQBrokerException) e).getResponseCode() == ResponseCode.FLOW_CONTROL) {
        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_BROKER_FLOW_CONTROL);
      } else {
        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
      }
    }
  };

  try {
    // todo 拉取消息
    this.pullAPIWrapper.pullKernelImpl(
      pullRequest.getMessageQueue(),
      subExpression,
      subscriptionData.getExpressionType(),
      subscriptionData.getSubVersion(),
      pullRequest.getNextOffset(),
      this.defaultMQPushConsumer.getPullBatchSize(),
      sysFlag,
      commitOffsetValue,
      BROKER_SUSPEND_MAX_TIME_MILLIS,
      CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,
      CommunicationMode.ASYNC,
      pullCallback
    );
  } catch (Exception e) {
    log.error("pullKernelImpl exception", e);
    this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
  }
}

pullMessage中会构造拉取消息的回调函数PullCallback，当成功拉取到消息时，会submitConsumeRequest让Consumer真正执行消费。

offset的更新

拉取到消息后，会提交ConsumeRequest，此时Consumer才去消费消息，消费完毕后会处理消费结果 ，此时就会涉及到offset的更新

java 复制代码 
// ConsumeMessageConcurrentlyService#processConsumeResult
public void processConsumeResult(
  final ConsumeConcurrentlyStatus status,
  final ConsumeConcurrentlyContext context,
  final ConsumeRequest consumeRequest
) {
  int ackIndex = context.getAckIndex();

  if (consumeRequest.getMsgs().isEmpty())
    return;

  switch (status) {
    case CONSUME_SUCCESS:
      if (ackIndex >= consumeRequest.getMsgs().size()) {
        ackIndex = consumeRequest.getMsgs().size() - 1;
      }
      int ok = ackIndex + 1;
      int failed = consumeRequest.getMsgs().size() - ok;
      this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), ok);
      this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), failed);
      break;
    case RECONSUME_LATER:
      ackIndex = -1;
      this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(),
                                                         consumeRequest.getMsgs().size());
      break;
    default:
      break;
  }

  switch (this.defaultMQPushConsumer.getMessageModel()) {
    case BROADCASTING:
      for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
        MessageExt msg = consumeRequest.getMsgs().get(i);
        log.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
      }
      break;
    case CLUSTERING:
      List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
      for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
        MessageExt msg = consumeRequest.getMsgs().get(i);
        boolean result = this.sendMessageBack(msg, context);
        if (!result) {
          msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
          msgBackFailed.add(msg);
        }
      }

      if (!msgBackFailed.isEmpty()) {
        consumeRequest.getMsgs().removeAll(msgBackFailed);

        this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
      }
      break;
    default:
      break;
  }

  long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
  if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
    // todo 更新offset
    this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
  }
}
java 复制代码 
public class RemoteBrokerOffsetStore implements OffsetStore {
  
  private ConcurrentMap<MessageQueue, AtomicLong> offsetTable =
    new ConcurrentHashMap<MessageQueue, AtomicLong>();
  
  public void updateOffset(MessageQueue mq, long offset, boolean increaseOnly) {
    if (mq != null) {
      // 旧offset
      AtomicLong offsetOld = this.offsetTable.get(mq);
      if (null == offsetOld) {
        // 旧offset不存在，则设置为offset
        offsetOld = this.offsetTable.putIfAbsent(mq, new AtomicLong(offset));
      }

      if (null != offsetOld) {
        if (increaseOnly) {
          // 更新offset
          MixAll.compareAndIncreaseOnly(offsetOld, offset);
        } else {
          offsetOld.set(offset);
        }
      }
    }
  }
}

offSet同步到Broker

Consumer启动时，会开启一个定时任务，每隔5s将offset同步到broker

java 复制代码 
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        try {
            MQClientInstance.this.persistAllConsumerOffset();
        } catch (Exception e) {
            log.error("ScheduledTask persistAllConsumerOffset exception", e);
        }
    }
}, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);

总结

通过本文，我们了解了RocketMQ的offSet管理机制

  1. Consumer成功消费消息后，会将messageQueue的offset进行更新，同时Consumer客户端会开启一个定时任务，每隔5s将messageQueue的offset同步给Broker
  2. Broker接收到更新offset请求后，会更新本地维护offsetTable，通过也会开启一个定时任务，每隔5秒将ConsumerOffsetManager中的offsetTable持久化到consumerOffset.json文件中

我是 Code皮皮虾 ，会在以后的日子里跟大家一起学习，一起进步！ 觉得文章不错的话，可以在 掘金 关注我，这样就不会错过很多技术干货啦~

相关推荐
Charlie__ZS5 分钟前
若依框架去掉Redis
java·redis·mybatis
mit6.82411 分钟前
[Meetily后端框架] AI摘要结构化 | `SummaryResponse`模型 | Pydantic库 | vs marshmallow库
c++·人工智能·后端
陈随易24 分钟前
VSCode v1.102发布，AI体验大幅提升
前端·后端·程序员
生无谓37 分钟前
什么是跨域，如何处理跨域
后端
咖啡啡不加糖38 分钟前
RabbitMQ 消息队列：从入门到Spring Boot实战
java·spring boot·rabbitmq
Smilejudy39 分钟前
极具特色的位置运算
后端
码出极致40 分钟前
支付线上问题复盘的“5W”框架
后端
玩代码1 小时前
Java线程池原理概述
java·开发语言·线程池
ezl1fe1 小时前
RAG 每日一技（三）：不止文本，代码和Markdown如何优雅地分块？
后端
NE_STOP1 小时前
SpringBoot--如何给项目添加配置属性及读取属性
java
热门推荐
01使用Ruby接入实时行情API教程02KGG转MP3工具|非KGM文件|解密音频03Coze扣子平台完整体验和实践（附国内和国际版对比）04深度神经网络训练过程与常见概念05集群聊天服务器---MySQL数据库的建立06扣子（coze）实战|我用扣子搭建了一个自动分析小红薯笔记内容的AI应用|详细步骤拆解07【无标题】08Java学习第十五部分——MyBatis09DeepSeek各版本说明与优缺点分析10LOT: 通过逻辑增强大型语言模型的零样本Chain-of-Thought推理能力