探究 | RocketMQ offset管理机制~

前言

大家好哇，我是Code皮皮虾，今天跟大家一起了解下RocketMq的offset管理机制~

---

什么是offSet？

一个Topic会存在多个messageQueue，每个messageQueue都会被Consumer消费，那么Consumer该从哪开始消费呢？

这就涉及到offset了，每个messageQueue会记录一个offset表示当前的消费进度，Consumer直接从offset开始消费即可。

借用一下网上的图~

---

Broker管理Offset

持久化

broker在启动时，会开启定时任务，每隔5秒将ConsumerOffsetManager中的offsetTable持久化到consumerOffset.json文件中

public boolean initialize() throws CloneNotSupportedException {

  // .....

  this.scheduledExecutorService.scheduleAtFixedRate(() -> {
    try {
      // 持久化
      BrokerController.this.consumerOffsetManager.persist();
    } catch (Throwable e) {
      log.error("schedule persist consumerOffset error.", e);
    }
  }, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
  
  // .....
  
}

那么offsetTable是个什么东西？为什么要专门将它进行持久化呢？

public class ConsumerOffsetManager extends ConfigManager {

  protected ConcurrentMap<String/* topic@group */, ConcurrentMap<Integer, Long>> offsetTable =
    new ConcurrentHashMap<String, ConcurrentMap<Integer, Long>>(512);
  
}

进入到ConsumerOffsetManager中，我们可以看到offsetTable本质是一个map结构，key为topic@group，value为一个key是队列下标、value为消费偏移量offset的map结构

具体的持久化文件结构如下👇🏻

{
	"offsetTable":{
		"TopicTest@please_rename_unique_group_name_4":{0:4,1:4,2:4,3:4,4:4,5:2,6:2,7:1,8:3,9:1,10:0,11:2,12:2,13:3,14:3,15:3
		},
		"%RETRY%please_rename_unique_group_name_4@please_rename_unique_group_name_4":{0:0
		},
		"TopicTest-2@please_rename_unique_group_name_4":{0:25,1:25,2:25,3:25,4:25,5:25,6:25,7:25
		},
		"TopicTest-1@please_rename_unique_group_name_4":{0:12,1:12,2:12,3:12,4:13,5:13,6:13,7:13
		}
	}
}

offset文件默认存储路径 ：$user.home/store/config/consumerOffset.json

---

更新

上面我们了解了broker对于offsetTable的持久化，下面我们继续来了解offsetTable是如何被更新的

当消费者消费消息后，会将offset发送给broker，broker接收后会更新offsetTable，接着交由定时任务进行持久化

而broker则是通过ConsumerManageProcessor来处理更新offset请求的

public class ConsumerManageProcessor extends AsyncNettyRequestProcessor {

  @Override
  public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    switch (request.getCode()) {
      case RequestCode.GET_CONSUMER_LIST_BY_GROUP:
        return this.getConsumerListByGroup(ctx, request);
      case RequestCode.UPDATE_CONSUMER_OFFSET:
        // todo 更新consumer offset
        return this.updateConsumerOffset(ctx, request);
      case RequestCode.QUERY_CONSUMER_OFFSET:
        // todo 查询consumer消费最新offset
        return this.queryConsumerOffset(ctx, request);
      default:
        break;
    }
    return null;
  }
}

updateConsumerOffset中，从requestHeader中拿到consumerGroup、topic、queueId、offset等信息，调用ConsumerOffsetManager#commitOffset来更新offsetTable

private RemotingCommand updateConsumerOffset(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    final RemotingCommand response =
        RemotingCommand.createResponseCommand(UpdateConsumerOffsetResponseHeader.class);
    final UpdateConsumerOffsetRequestHeader requestHeader =
        (UpdateConsumerOffsetRequestHeader) request
            .decodeCommandCustomHeader(UpdateConsumerOffsetRequestHeader.class);

  // 从requestHeader中拿到consumerGroup、topic、queueId、offset
  this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), requestHeader.getConsumerGroup(),
        requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}

到commitOffset这里就比较明确了，根据topic、consumerGroup拼接好offsetTable对应的key，然后更新offset即可

public void commitOffset(final String clientHost, final String group, final String topic, final int queueId,
                         final long offset) {
  // 拼接offsetTable key: topic@group
  String key = topic + TOPIC_GROUP_SEPARATOR + group;
  this.commitOffset(clientHost, key, queueId, offset);
}

private void commitOffset(final String clientHost, final String key, final int queueId, final long offset) {
  ConcurrentMap<Integer, Long> map = this.offsetTable.get(key);
  if (null == map) {
    // 如果key对应的map不存在，则需要初始化
    map = new ConcurrentHashMap<Integer, Long>(32);
    
    // 更新map
    map.put(queueId, offset);
    this.offsetTable.put(key, map);
  } else {
    // 更新map
    Long storeOffset = map.put(queueId, offset);
    if (storeOffset != null && offset < storeOffset) {
      log.warn("[NOTIFYME]update consumer offset less than store. clientHost={}, key={}, queueId={}, requestOffset={}, storeOffset={}", clientHost, key, queueId, offset, storeOffset);
    }
  }
}

---

Consumer管理Offset

Broker管理offset的机制我们了解完了，接下来继续看看Consumer是如何管理offset的

---

offset初始化

consumer在消费前需要进行重平衡，重平衡后可能会为当前的Consumer分配新的messageQueue，那么此时对于该messageQueue来说不存在当前Consumer的消费进度，故需要进行offset初始化~

public long computePullFromWhereWithException(MessageQueue mq) throws MQClientException {
  long result = -1;
  final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
  final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
  switch (consumeFromWhere) {
    case CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST:
    case CONSUME_FROM_MIN_OFFSET:
    case CONSUME_FROM_MAX_OFFSET:
      // 从最新的偏移量开始消费
    case CONSUME_FROM_LAST_OFFSET: {
      // 从store中读取offset，即从broker的offsetTable中获取
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        // >= 0 直接返回
        result = lastOffset;
      }
      // 等于-1，代表首次启动，还没有offset
      else if (-1 == lastOffset) {
        if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
          result = 0L;
        } else {
          try {
            // 获取当前该消息队列当前最大偏移量
            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        }
      } else {
        // 小于-1，表示该消息进度文件中存储了错误的偏移，返回-1
        result = -1;
      }
      break;
    }
      // 从头开始消费
    case CONSUME_FROM_FIRST_OFFSET: {
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        result = lastOffset;
      } else if (-1 == lastOffset) {
        // 首次启动，还没有offset，从头开始消费
        result = 0L;
      } else {
        result = -1;
      }
      break;
    }
      // 从Consumer启动时间戳对应的消费进度开始消费
    case CONSUME_FROM_TIMESTAMP: {
      long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
      if (lastOffset >= 0) {
        result = lastOffset;
      } else if (-1 == lastOffset) {
        if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
          try {
            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        } else {
          try {
            // 获取消费者启动时的时间戳
            long timestamp = UtilAll.parseDate(this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeTimestamp(),
                                               UtilAll.YYYYMMDDHHMMSS).getTime();
            // 搜索时间戳对应的offset
            result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
          } catch (MQClientException e) {
            log.warn("Compute consume offset from last offset exception, mq={}, exception={}", mq, e);
            throw e;
          }
        }
      } else {
        result = -1;
      }
      break;
    }

    default:
      break;
  }

  return result;
}

学习上面源码，结合注释，我们可将offset的分配大致分为三种 ，CONSUME_FROM_LAST_OFFSET、CONSUME_FROM_FIRST_OFFSET、CONSUME_FROM_TIMESTAMP

三者对于消费进度存在 或者消费进度不合理的处理方式是一致的

• 当消费进度存在时，直接返回最新的消费offset
• 当消费进度不合理时，直接返回-1

唯一的区别就在于消费进度不存在时的处理 ，当消费进度不存在时，需要进行offset的初始化，即👇🏻的分配方式

1. CONSUME_FROM_LAST_OFFSET: 从最新的偏移量开始消费
2. CONSUME_FROM_FIRST_OFFSET: 从头开始消费
3. CONSUME_FROM_TIMESTAMP: 从Consumer启动时间戳对应的消费进度开始消费

---

拉取消息

offset初始化完毕后就可以去拉取消息了~

public void pullMessage(final PullRequest pullRequest) {
  
  // ......
  
  
	// todo 构造拉取消息回调函数
  PullCallback pullCallback = new PullCallback() {
    @Override
    public void onSuccess(PullResult pullResult) {
      if (pullResult != null) {
        pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                                                                                     subscriptionData);
        switch (pullResult.getPullStatus()) {
          case FOUND:
            // 拉取成功
            long prevRequestOffset = pullRequest.getNextOffset();
            // todo 更新下一次拉取的offset
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());
            long pullRT = System.currentTimeMillis() - beginTimestamp;
            DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                                                                               pullRequest.getMessageQueue().getTopic(), pullRT);

            long firstMsgOffset = Long.MAX_VALUE;
            if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
              DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
            } else {
              firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

              DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                                                                  pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

              boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
              
              // todo 提交消费请求，此时Consumer才会去消费拉取到的消息
              DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                pullResult.getMsgFoundList(),
                processQueue,
                pullRequest.getMessageQueue(),
                dispatchToConsume);

              if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                                                       DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
              } else {
                // todo 再次将PullRequest放入队列中
                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
              }
            }

            if (pullResult.getNextBeginOffset() < prevRequestOffset
                || firstMsgOffset < prevRequestOffset) {
              log.warn(
                "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                pullResult.getNextBeginOffset(),
                firstMsgOffset,
                prevRequestOffset);
            }

            break;
          case NO_NEW_MSG:
          case NO_MATCHED_MSG:
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

            DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
            break;
          case OFFSET_ILLEGAL:
            log.warn("the pull request offset illegal, {} {}",
                     pullRequest.toString(), pullResult.toString());
            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

            pullRequest.getProcessQueue().setDropped(true);
            DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

              @Override
              public void run() {
                try {
                  DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                                                          pullRequest.getNextOffset(), false);

                  DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                  DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                  log.warn("fix the pull request offset, {}", pullRequest);
                } catch (Throwable e) {
                  log.error("executeTaskLater Exception", e);
                }
              }
            }, 10000);
            break;
          default:
            break;
        }
      }
    }

    @Override
    public void onException(Throwable e) {
      if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
        log.warn("execute the pull request exception", e);
      }

      if (e instanceof MQBrokerException && ((MQBrokerException) e).getResponseCode() == ResponseCode.FLOW_CONTROL) {
        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_BROKER_FLOW_CONTROL);
      } else {
        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
      }
    }
  };

  try {
    // todo 拉取消息
    this.pullAPIWrapper.pullKernelImpl(
      pullRequest.getMessageQueue(),
      subExpression,
      subscriptionData.getExpressionType(),
      subscriptionData.getSubVersion(),
      pullRequest.getNextOffset(),
      this.defaultMQPushConsumer.getPullBatchSize(),
      sysFlag,
      commitOffsetValue,
      BROKER_SUSPEND_MAX_TIME_MILLIS,
      CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,
      CommunicationMode.ASYNC,
      pullCallback
    );
  } catch (Exception e) {
    log.error("pullKernelImpl exception", e);
    this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
  }
}

pullMessage中会构造拉取消息的回调函数PullCallback，当成功拉取到消息时，会submitConsumeRequest让Consumer真正执行消费。

---

offset的更新

拉取到消息后，会提交ConsumeRequest，此时Consumer才去消费消息，消费完毕后会处理消费结果 ，此时就会涉及到offset的更新

// ConsumeMessageConcurrentlyService#processConsumeResult
public void processConsumeResult(
  final ConsumeConcurrentlyStatus status,
  final ConsumeConcurrentlyContext context,
  final ConsumeRequest consumeRequest
) {
  int ackIndex = context.getAckIndex();

  if (consumeRequest.getMsgs().isEmpty())
    return;

  switch (status) {
    case CONSUME_SUCCESS:
      if (ackIndex >= consumeRequest.getMsgs().size()) {
        ackIndex = consumeRequest.getMsgs().size() - 1;
      }
      int ok = ackIndex + 1;
      int failed = consumeRequest.getMsgs().size() - ok;
      this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), ok);
      this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), failed);
      break;
    case RECONSUME_LATER:
      ackIndex = -1;
      this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(),
                                                         consumeRequest.getMsgs().size());
      break;
    default:
      break;
  }

  switch (this.defaultMQPushConsumer.getMessageModel()) {
    case BROADCASTING:
      for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
        MessageExt msg = consumeRequest.getMsgs().get(i);
        log.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
      }
      break;
    case CLUSTERING:
      List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
      for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
        MessageExt msg = consumeRequest.getMsgs().get(i);
        boolean result = this.sendMessageBack(msg, context);
        if (!result) {
          msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
          msgBackFailed.add(msg);
        }
      }

      if (!msgBackFailed.isEmpty()) {
        consumeRequest.getMsgs().removeAll(msgBackFailed);

        this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
      }
      break;
    default:
      break;
  }

  long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
  if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
    // todo 更新offset
    this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
  }
}

public class RemoteBrokerOffsetStore implements OffsetStore {
  
  private ConcurrentMap<MessageQueue, AtomicLong> offsetTable =
    new ConcurrentHashMap<MessageQueue, AtomicLong>();
  
  public void updateOffset(MessageQueue mq, long offset, boolean increaseOnly) {
    if (mq != null) {
      // 旧offset
      AtomicLong offsetOld = this.offsetTable.get(mq);
      if (null == offsetOld) {
        // 旧offset不存在，则设置为offset
        offsetOld = this.offsetTable.putIfAbsent(mq, new AtomicLong(offset));
      }

      if (null != offsetOld) {
        if (increaseOnly) {
          // 更新offset
          MixAll.compareAndIncreaseOnly(offsetOld, offset);
        } else {
          offsetOld.set(offset);
        }
      }
    }
  }
}

---

offSet同步到Broker

Consumer启动时，会开启一个定时任务，每隔5s将offset同步到broker

this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        try {
            MQClientInstance.this.persistAllConsumerOffset();
        } catch (Exception e) {
            log.error("ScheduledTask persistAllConsumerOffset exception", e);
        }
    }
}, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);

---

总结

通过本文，我们了解了RocketMQ的offSet管理机制

1. Consumer成功消费消息后，会将messageQueue的offset进行更新，同时Consumer客户端会开启一个定时任务，每隔5s将messageQueue的offset同步给Broker
2. Broker接收到更新offset请求后，会更新本地维护offsetTable，通过也会开启一个定时任务，每隔5秒将ConsumerOffsetManager中的offsetTable持久化到consumerOffset.json文件中

我是 Code皮皮虾 ，会在以后的日子里跟大家一起学习，一起进步！ 觉得文章不错的话，可以在 掘金 关注我，这样就不会错过很多技术干货啦~