探究 | RocketMQ offset管理机制~

Code皮皮虾2023-08-29 10:17

前言

大家好哇，我是Code皮皮虾，今天跟大家一起了解下RocketMq的offset管理机制~

什么是offSet？

一个Topic会存在多个messageQueue，每个messageQueue都会被Consumer消费，那么Consumer该从哪开始消费呢？

这就涉及到offset了，每个messageQueue会记录一个offset表示当前的消费进度，Consumer直接从offset开始消费即可。

借用一下网上的图~

Broker管理Offset

持久化

broker在启动时，会开启定时任务，每隔5秒将ConsumerOffsetManager中的offsetTable持久化到consumerOffset.json文件中

java 复制代码 
public boolean initialize() throws CloneNotSupportedException {

  // .....

  this.scheduledExecutorService.scheduleAtFixedRate(() -> {
    try {
      // 持久化
      BrokerController.this.consumerOffsetManager.persist();
    } catch (Throwable e) {
      log.error("schedule persist consumerOffset error.", e);
    }
  }, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
  
  // .....
  
}

那么offsetTable是个什么东西？为什么要专门将它进行持久化呢？

java 复制代码 
public class ConsumerOffsetManager extends ConfigManager {

  protected ConcurrentMap<String/* topic@group */, ConcurrentMap<Integer, Long>> offsetTable =
    new ConcurrentHashMap<String, ConcurrentMap<Integer, Long>>(512);
  
}

进入到ConsumerOffsetManager中，我们可以看到offsetTable本质是一个map结构，key为topic@group，value为一个key是队列下标、value为消费偏移量offset的map结构

具体的持久化文件结构如下👇🏻

json 复制代码 
{
	"offsetTable":{
		"TopicTest@please_rename_unique_group_name_4":{0:4,1:4,2:4,3:4,4:4,5:2,6:2,7:1,8:3,9:1,10:0,11:2,12:2,13:3,14:3,15:3
		},
		"%RETRY%please_rename_unique_group_name_4@please_rename_unique_group_name_4":{0:0
		},
		"TopicTest-2@please_rename_unique_group_name_4":{0:25,1:25,2:25,3:25,4:25,5:25,6:25,7:25
		},
		"TopicTest-1@please_rename_unique_group_name_4":{0:12,1:12,2:12,3:12,4:13,5:13,6:13,7:13
		}
	}
}

offset文件默认存储路径 ：$user.home/store/config/consumerOffset.json

更新

上面我们了解了broker对于offsetTable的持久化，下面我们继续来了解offsetTable是如何被更新的

当消费者消费消息后，会将offset发送给broker，broker接收后会更新offsetTable，接着交由定时任务进行持久化

而broker则是通过ConsumerManageProcessor来处理更新offset请求的

java 复制代码 
public class ConsumerManageProcessor extends AsyncNettyRequestProcessor {

  @Override
  public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    switch (request.getCode()) {
      case RequestCode.GET_CONSUMER_LIST_BY_GROUP:
        return this.getConsumerListByGroup(ctx, request);
      case RequestCode.UPDATE_CONSUMER_OFFSET:
        // todo 更新consumer offset
        return this.updateConsumerOffset(ctx, request);
      case RequestCode.QUERY_CONSUMER_OFFSET:
        // todo 查询consumer消费最新offset
        return this.queryConsumerOffset(ctx, request);
      default:
        break;
    }
    return null;
  }
}

updateConsumerOffset中，从requestHeader中拿到consumerGroup、topic、queueId、offset等信息，调用ConsumerOffsetManager#commitOffset来更新offsetTable

java 复制代码 
private RemotingCommand updateConsumerOffset(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    final RemotingCommand response =
        RemotingCommand.createResponseCommand(UpdateConsumerOffsetResponseHeader.class);
    final UpdateConsumerOffsetRequestHeader requestHeader =
        (UpdateConsumerOffsetRequestHeader) request
            .decodeCommandCustomHeader(UpdateConsumerOffsetRequestHeader.class);

  // 从requestHeader中拿到consumerGroup、topic、queueId、offset
  this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), requestHeader.getConsumerGroup(),
        requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}

到commitOffset这里就比较明确了，根据topic、consumerGroup拼接好offsetTable对应的key，然后更新offset即可

java 复制代码 
public void commitOffset(final String clientHost, final String group, final String topic, final int queueId,
                         final long offset) {
  // 拼接offsetTable key: topic@group
  String key = topic + TOPIC_GROUP_SEPARATOR + group;
  this.commitOffset(clientHost, key, queueId, offset);
}

private void commitOffset(final String clientHost, final String key, final int queueId, final long offset) {
  ConcurrentMap<Integer, Long> map = this.offsetTable.get(key);
  if (null == map) {
    // 如果key对应的map不存在，则需要初始化
    map = new ConcurrentHashMap<Integer, Long>(32);
    
    // 更新map
    map.put(queueId, offset);
    this.offsetTable.put(key, map);
  } else {
    // 更新map
    Long storeOffset = map.put(queueId, offset);
    if (storeOffset != null && offset < storeOffset) {
      log.warn("[NOTIFYME]update consumer offset less than store. clientHost={}, key={}, queueId={}, requestOffset={}, storeOffset={}", clientHost, key, queueId, offset, storeOffset);
    }
  }
}

Consumer管理Offset

Broker管理offset的机制我们了解完了，接下来继续看看Consumer是如何管理offset的

offset初始化

consumer在消费前需要进行重平衡，重平衡后可能会为当前的Consumer分配新的messageQueue，那么此时对于该messageQueue来说不存在当前Consumer的消费进度，故需要进行offset初始化~

java 复制代码 
public long computePullFromWhereWithException(MessageQueue mq) throws MQClientException {
  long result = -1;
  final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
  final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
  switch (consumeFromWhere) {
    case CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST:
    case CONSUME_FROM_MIN_OFFSET:
    case CONSUME_FROM_MAX_OFFSET:
      // 从最新的偏移量开始消费
    case CONSUME_FROM_LAST_OFFSET: {
      // 从store中读取offset，即从broker的offsetTable中获取
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        // >= 0 直接返回
        result = lastOffset;
      }
      // 等于-1，代表首次启动，还没有offset
      else if (-1 == lastOffset) {
        if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
          result = 0L;
        } else {
          try {
            // 获取当前该消息队列当前最大偏移量
            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        }
      } else {
        // 小于-1，表示该消息进度文件中存储了错误的偏移，返回-1
        result = -1;
      }
      break;
    }
      // 从头开始消费
    case CONSUME_FROM_FIRST_OFFSET: {
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        result = lastOffset;
      } else if (-1 == lastOffset) {
        // 首次启动，还没有offset，从头开始消费
        result = 0L;
      } else {
        result = -1;
      }
      break;
    }
      // 从Consumer启动时间戳对应的消费进度开始消费
    case CONSUME_FROM_TIMESTAMP: {
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        result = lastOffset;
      } else if (-1 == lastOffset) {
        if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
          try {
            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        } else {
          try {
            // 获取消费者启动时的时间戳
            long timestamp = UtilAll.parseDate(this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeTimestamp(),
                                               UtilAll.YYYYMMDDHHMMSS).getTime();
            // 搜索时间戳对应的offset
            result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        }
      } else {
        result = -1;
      }
      break;
    }

    default:
      break;
  }

  return result;
}

学习上面源码，结合注释，我们可将offset的分配大致分为三种 ，CONSUME_FROM_LAST_OFFSET、CONSUME_FROM_FIRST_OFFSET、CONSUME_FROM_TIMESTAMP

三者对于消费进度存在 或者消费进度不合理的处理方式是一致的

  • 当消费进度存在时，直接返回最新的消费offset
  • 当消费进度不合理时，直接返回-1

唯一的区别就在于消费进度不存在时的处理 ，当消费进度不存在时，需要进行offset的初始化，即👇🏻的分配方式

  1. CONSUME_FROM_LAST_OFFSET: 从最新的偏移量开始消费
  2. CONSUME_FROM_FIRST_OFFSET: 从头开始消费
  3. CONSUME_FROM_TIMESTAMP: 从Consumer启动时间戳对应的消费进度开始消费

拉取消息

offset初始化完毕后就可以去拉取消息了~

java 复制代码 
public void pullMessage(final PullRequest pullRequest) {
  
  // ......
  
  
	// todo 构造拉取消息回调函数
  PullCallback pullCallback = new PullCallback() {
    @Override
    public void onSuccess(PullResult pullResult) {
      if (pullResult != null) {
        pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                                                                                     subscriptionData);
        switch (pullResult.getPullStatus()) {
          case FOUND:
            // 拉取成功
            long prevRequestOffset = pullRequest.getNextOffset();
            // todo 更新下一次拉取的offset
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());
            long pullRT = System.currentTimeMillis() - beginTimestamp;
            DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                                                                               pullRequest.getMessageQueue().getTopic(), pullRT);

            long firstMsgOffset = Long.MAX_VALUE;
            if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
              DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
            } else {
              firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

              DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                                                                  pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

              boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
              
              // todo 提交消费请求，此时Consumer才会去消费拉取到的消息
              DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                pullResult.getMsgFoundList(),
                processQueue,
                pullRequest.getMessageQueue(),
                dispatchToConsume);

              if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                                                       DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
              } else {
                // todo 再次将PullRequest放入队列中
                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
              }
            }

            if (pullResult.getNextBeginOffset() < prevRequestOffset
                || firstMsgOffset < prevRequestOffset) {
              log.warn(
                "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                pullResult.getNextBeginOffset(),
                firstMsgOffset,
                prevRequestOffset);
            }

            break;
          case NO_NEW_MSG:
          case NO_MATCHED_MSG:
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

            DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
            break;
          case OFFSET_ILLEGAL:
            log.warn("the pull request offset illegal, {} {}",
                     pullRequest.toString(), pullResult.toString());
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

            pullRequest.getProcessQueue().setDropped(true);
            DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

              @Override
              public void run() {
                try {
                  DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                                                          pullRequest.getNextOffset(), false);

                  DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                  DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                  log.warn("fix the pull request offset, {}", pullRequest);
                } catch (Throwable e) {
                  log.error("executeTaskLater Exception", e);
                }
              }
            }, 10000);
            break;
          default:
            break;
        }
      }
    }

    @Override
    public void onException(Throwable e) {
      if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
        log.warn("execute the pull request exception", e);
      }

      if (e instanceof MQBrokerException && ((MQBrokerException) e).getResponseCode() == ResponseCode.FLOW_CONTROL) {
        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_BROKER_FLOW_CONTROL);
      } else {
        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
      }
    }
  };

  try {
    // todo 拉取消息
    this.pullAPIWrapper.pullKernelImpl(
      pullRequest.getMessageQueue(),
      subExpression,
      subscriptionData.getExpressionType(),
      subscriptionData.getSubVersion(),
      pullRequest.getNextOffset(),
      this.defaultMQPushConsumer.getPullBatchSize(),
      sysFlag,
      commitOffsetValue,
      BROKER_SUSPEND_MAX_TIME_MILLIS,
      CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,
      CommunicationMode.ASYNC,
      pullCallback
    );
  } catch (Exception e) {
    log.error("pullKernelImpl exception", e);
    this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
  }
}

pullMessage中会构造拉取消息的回调函数PullCallback，当成功拉取到消息时，会submitConsumeRequest让Consumer真正执行消费。

offset的更新

拉取到消息后，会提交ConsumeRequest，此时Consumer才去消费消息，消费完毕后会处理消费结果 ，此时就会涉及到offset的更新

java 复制代码 
// ConsumeMessageConcurrentlyService#processConsumeResult
public void processConsumeResult(
  final ConsumeConcurrentlyStatus status,
  final ConsumeConcurrentlyContext context,
  final ConsumeRequest consumeRequest
) {
  int ackIndex = context.getAckIndex();

  if (consumeRequest.getMsgs().isEmpty())
    return;

  switch (status) {
    case CONSUME_SUCCESS:
      if (ackIndex >= consumeRequest.getMsgs().size()) {
        ackIndex = consumeRequest.getMsgs().size() - 1;
      }
      int ok = ackIndex + 1;
      int failed = consumeRequest.getMsgs().size() - ok;
      this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), ok);
      this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), failed);
      break;
    case RECONSUME_LATER:
      ackIndex = -1;
      this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(),
                                                         consumeRequest.getMsgs().size());
      break;
    default:
      break;
  }

  switch (this.defaultMQPushConsumer.getMessageModel()) {
    case BROADCASTING:
      for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
        MessageExt msg = consumeRequest.getMsgs().get(i);
        log.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
      }
      break;
    case CLUSTERING:
      List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
      for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
        MessageExt msg = consumeRequest.getMsgs().get(i);
        boolean result = this.sendMessageBack(msg, context);
        if (!result) {
          msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
          msgBackFailed.add(msg);
        }
      }

      if (!msgBackFailed.isEmpty()) {
        consumeRequest.getMsgs().removeAll(msgBackFailed);

        this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
      }
      break;
    default:
      break;
  }

  long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
  if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
    // todo 更新offset
    this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
  }
}
java 复制代码 
public class RemoteBrokerOffsetStore implements OffsetStore {
  
  private ConcurrentMap<MessageQueue, AtomicLong> offsetTable =
    new ConcurrentHashMap<MessageQueue, AtomicLong>();
  
  public void updateOffset(MessageQueue mq, long offset, boolean increaseOnly) {
    if (mq != null) {
      // 旧offset
      AtomicLong offsetOld = this.offsetTable.get(mq);
      if (null == offsetOld) {
        // 旧offset不存在，则设置为offset
        offsetOld = this.offsetTable.putIfAbsent(mq, new AtomicLong(offset));
      }

      if (null != offsetOld) {
        if (increaseOnly) {
          // 更新offset
          MixAll.compareAndIncreaseOnly(offsetOld, offset);
        } else {
          offsetOld.set(offset);
        }
      }
    }
  }
}

offSet同步到Broker

Consumer启动时，会开启一个定时任务，每隔5s将offset同步到broker

java 复制代码 
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        try {
            MQClientInstance.this.persistAllConsumerOffset();
        } catch (Exception e) {
            log.error("ScheduledTask persistAllConsumerOffset exception", e);
        }
    }
}, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);

总结

通过本文，我们了解了RocketMQ的offSet管理机制

  1. Consumer成功消费消息后，会将messageQueue的offset进行更新，同时Consumer客户端会开启一个定时任务，每隔5s将messageQueue的offset同步给Broker
  2. Broker接收到更新offset请求后，会更新本地维护offsetTable，通过也会开启一个定时任务，每隔5秒将ConsumerOffsetManager中的offsetTable持久化到consumerOffset.json文件中

我是 Code皮皮虾 ，会在以后的日子里跟大家一起学习，一起进步！ 觉得文章不错的话，可以在 掘金 关注我，这样就不会错过很多技术干货啦~

相关推荐
Kali_075 分钟前
使用 Mathematical_Expression 从零开始实现数学题目的作答小游戏【可复制代码】
java·人工智能·免费
rzl0217 分钟前
java web5（黑马）
java·开发语言·前端
君爱学习23 分钟前
RocketMQ延迟消息是如何实现的?
后端
guojl37 分钟前
深度解读jdk8 HashMap设计与源码
java
Falling4240 分钟前
使用 CNB 构建并部署maven项目
后端
guojl42 分钟前
深度解读jdk8 ConcurrentHashMap设计与源码
java
程序员小假1 小时前
我们来讲一讲 ConcurrentHashMap
后端
爱上语文1 小时前
Redis基础（5）：Redis的Java客户端
java·开发语言·数据库·redis·后端
A~taoker1 小时前
taoker的项目维护（ng服务器）
java·开发语言
热门推荐
01Coze实战第13讲：飞书多维表格读取+豆包生图模型，轻松批量生成短剧封面02手机电脑之间快速传输图片视频文件，不压缩画质、不限制大小的方法！03Coze扣子平台完整体验和实践（附国内和国际版对比）04华为昇腾 910B 部署 DeepSeek-R1 蒸馏系列模型详细指南05免费可用！最强AI数字人对口型神器：让照片开口说话唱歌，支持多人对口型+全身动作，1分钟学会！（附保姆级教程）06扣子（coze）实战|我用扣子搭建了一个自动分析小红薯笔记内容的AI应用|详细步骤拆解072024年 最新 iPhone手机 历代机型、屏幕尺寸、纵横比、分辨率 整理08C#调用WechatOCR.exe实现本地OCR文字识别09DeepSeek各版本说明与优缺点分析10Coze平台 创建AI智能体的详细步骤指南