ThingsBoard 集群的核心逻辑源码分析

daidaidaiyu2026-10-07 22:43

集群的核心逻辑

ThingsBoard 微服务是Kafka + ZooKeeper + 多节点 对等集群的架构

\[N_ThingsBoard\]

\[N_ThingsBoard 集群的消息处理源码分析\]

\[N_ZooKeepper学习一则\]

集群角色划分

整体角色划分
flowchart TD subgraph ZK"ZooKeeper --- 服务注册与发现" zkNode"/thingsboard/nodes/\
├ node0..01 (tb-core)\
├ node0..02 (tb-rule-engine)\
└ node0..03 (tb-core)" end subgraph ServiceNodes"服务节点 (每个 JVM 进程)" N1"tb-core 节点 A\
ZkDiscoveryService\
HashPartitionService" N2"tb-rule-engine 节点 B\
ZkDiscoveryService\
HashPartitionService" N3"tb-core 节点 C\
ZkDiscoveryService\
HashPartitionService" end subgraph Kafka"Apache Kafka --- 消息总线" T1"tb_core\
partition-0 .. partition-9" T2"tb_rule_engine.Main\
partition-0 .. partition-9" T3"tb_rule_engine.HighPriority\
partition-0 .. partition-4" end N1 -->|"EPHEMERAL 节点注册"| zkNode N2 -->|"EPHEMERAL 节点注册"| zkNode N3 -->|"EPHEMERAL 节点注册"| zkNode zkNode -->|"PathChildrenCache 监听"| N1 zkNode -->|"PathChildrenCache 监听"| N2 zkNode -->|"PathChildrenCache 监听"| N3 N1 -->|"消费 0,2,4,6,8"| T1 N3 -->|"消费 1,3,5,7,9"| T1 N2 -->|"消费所有分区"| T2

核心的三剑客

组件 说明 关键的类
ZooKeeper 注册中心 服务注册（EPHEMERAL_SEQUENTIAL 节点）、节点存活检测、变更通知、断线自动摘除 ZkDiscoveryService
Kafka 消息队列 按 ServiceType 划分 Topic，每个 Topic 固定分区数；支持单消费者和每分区独立消费者两种模式；poll 循环驱动消息处理，commit 保证 at-least-once TbKafkaConsumerTemplate MainQueueConsumerManager QueueConsumerManager
分区计算与事件发布 一致性哈希将 Topic 分区分配给集群节点；感知集群拓扑变化后重算 myPartitions；差量对比后发布 PartitionChangeEvent 驱动重新订阅 HashPartitionService

一. ZooKeeper 注册中心

初始化客户端

org.thingsboard.server.queue.discovery.ZkDiscoveryService#initZkClient

java 复制代码 
private void initZkClient() {
	try {
		// CuratorFramework 客户端
		client = CuratorFrameworkFactory.newClient(zkUrl, zkSessionTimeout, zkConnectionTimeout, new RetryForever(zkRetryInterval));
		client.start();
		client.blockUntilConnected();
		// PathChildrenCache , 用于监听 zkNodesDir 下所有子节点的增删改事件。
		//ThingsBoard 用它来感知集群中其他服务节点的上线/下线, 实现服务发现。 后续 subscribeToEvents() 再将this（当前类）注册为事件监听器, 当子节点变化时触发回调处理集群拓扑变更。
		cache = new PathChildrenCache(client, zkNodesDir, true);
		cache.start();
		stopped = false;
		log.info("ZK client connected");
	} catch (Exception e) {
		log.error("Failed to connect to ZK: {}", e.getMessage(), e);
		CloseableUtils.closeQuietly(cache);
		CloseableUtils.closeQuietly(client);
		throw new RuntimeException(e);
	}
}

关于 zoo 的节点路径:

java 复制代码 
//在配置
zk_dir: "${ZOOKEEPER_NODES_DIR:/thingsboard}"

// init()
zkNodesDir = zkDir + "/nodes";   // → "/thingsboard/nodes/0...01"

// publishCurrentServer()
nodePath = client.create()
  .creatingParentsIfNeeded()
  .withMode(CreateMode.EPHEMERAL_SEQUENTIAL)   // 临时顺序节点
  .forPath(zkNodesDir + "/", self.toByteArray());

注册自己&订阅事件

org.thingsboard.server.queue.discovery.ZkDiscoveryService#onApplicationEvent

java 复制代码 
@AfterStartUp(order = AfterStartUp.DISCOVERY_SERVICE)
public void onApplicationEvent(ApplicationReadyEvent event) {
	if (stopped) {
		log.debug("Ignoring application ready event. Service is stopped.");
		return;
	} else {
		log.info("Received application ready event. Starting current ZK node.");
	}
	//1. 注意先订阅
	subscribeToEvents();
	if (client.getState() != CuratorFrameworkState.STARTED) {
		log.debug("Ignoring application ready event, ZK client is not started, ZK client state [{}]", client.getState());
		return;
	}
	log.info("Going to publish current server...");
	//2. 向 zookeeper 注册
	publishCurrentServer();
	log.info("Going to recalculate partitions...");
	recalculatePartitions();
	//3. 周期性 更新和检查是否注册到 zookeeper中
	zkExecutorService.scheduleAtFixedRate(this::publishCurrentServer, 1, 1, TimeUnit.MINUTES);
}


private void subscribeToEvents() { 
	//监听节点事件
    cache.getListenable().addListener(this);  
}

org.thingsboard.server.queue.discovery.ZkDiscoveryService#publishCurrentServer

java 复制代码 
@SneakyThrows
public synchronized void publishCurrentServer() {
	//自己的服务信息
	TransportProtos.ServiceInfo self = serviceInfoProvider.getServiceInfo();
	// 节点不存在则创建, 存在则更新数据
	if (currentServerExists()) {
		log.trace("[{}] Updating ZK node for current instance: {}", self.getServiceId(), nodePath);
		//更新
		client.setData().forPath(nodePath, serviceInfoProvider.generateNewServiceInfoWithCurrentSystemInfo().toByteArray());
	} else {
		try {
			log.info("[{}] Creating ZK node for current instance", self.getServiceId());
			// 创建节点(临时)
			nodePath = client.create()
					.creatingParentsIfNeeded()
					.withMode(CreateMode.EPHEMERAL_SEQUENTIAL).forPath(zkNodesDir + "/", self.toByteArray());
			log.info("[{}] Created ZK node for current instance: {}", self.getServiceId(), nodePath);
			client.getConnectionStateListenable().addListener(checkReconnect(self));
		} catch (Exception e) {
			log.error("Failed to create ZK node", e);
			throw new RuntimeException(e);
		}
	}
}

注册的服务类型?

将自己注册为什么服务类型, 取决于序列化到 zookeeper 中的 ServiceInfo 信息, 所以找ServiceInfo生成的逻辑

生成 ServiceInfo 的方法

java 复制代码 
@Override
public ServiceInfo generateNewServiceInfoWithCurrentSystemInfo() {
	ServiceInfo.Builder builder = ServiceInfo.newBuilder()
			.setServiceId(serviceId)
			//重点在这里
			.addAllServiceTypes(serviceTypes.stream().map(ServiceType::name).collect(Collectors.toList()))
			.setSystemInfo(getCurrentSystemInfoProto());
	if (CollectionsUtil.isNotEmpty(assignedTenantProfiles)) {
		builder.addAllAssignedTenantProfiles(assignedTenantProfiles.stream().map(UUID::toString).collect(Collectors.toList()));
	}
	if (edqsConfig != null) {
		builder.setLabel(edqsConfig.getLabel());
	}
	builder.setReady(ready);
	builder.addAllTaskTypes(taskTypes.stream().map(JobType::name).toList());
	return serviceInfo = builder.build();
}

再见 init 方法

org.thingsboard.server.queue.discovery.DefaultTbServiceInfoProvider#init

java 复制代码 
@PostConstruct
public void init() {
	if (StringUtils.isEmpty(serviceId)) {
		try {
			serviceId = InetAddress.getLocalHost().getHostName();
		} catch (UnknownHostException e) {
			serviceId = StringUtils.randomAlphabetic(10);
		}
	}
	log.info("Current Service ID: {}", serviceId);
	//当前节点的服务信息初始化
	serviceTypes = isMonolith() ?
			List.of(ServiceType.values()) :
			Collections.singletonList(ServiceType.of(serviceType));
	if (!serviceTypes.contains(ServiceType.TB_RULE_ENGINE) || assignedTenantProfiles == null) {
		assignedTenantProfiles = Collections.emptySet();
	}
	if (serviceTypes.contains(ServiceType.EDQS)) {
		ready = false;
		if (StringUtils.isBlank(edqsConfig.getLabel())) {
			edqsConfig.setLabel(serviceId);
		}
	}
	if (CollectionsUtil.isNotEmpty(availableTaskProcessors)) {
		taskTypes = availableTaskProcessors.stream()
				.map(TaskProcessor::getJobType)
				.toList();
	} else {
		taskTypes = Collections.emptyList();
	}

	generateNewServiceInfoWithCurrentSystemInfo();
}

得看 serviceType 的配置

java 复制代码 
// @Value("${service.type:monolith}")  ← 从配置读, 默认 monolith
private String serviceType;

// init() 中：
serviceTypes = isMonolith()
  ? List.of(ServiceType.values())                          // monolith: 全类型
  : Collections.singletonList(ServiceType.of(serviceType)); // 微服务: 单类型

//ServiceType.of() 做了一次转换：把配置字符串 tb-core → TB_CORE（替换 - 为 _ 再 toUpperCase）。

其他节点通过读这个 serviceTypesList 字段决定把哪些分区路由给谁, 就是 HashPartitionService.addNode() 里按 serviceType 分组的逻辑。

集群的所有类型

见 org.thingsboard.server.common.msg.queue.ServiceType 的枚举值

java 复制代码 
@RequiredArgsConstructor  
@Getter  
public enum ServiceType {  
  
    TB_CORE("TB Core"),  
    TB_RULE_ENGINE("TB Rule Engine"),  
    TB_TRANSPORT("TB Transport"),  
    JS_EXECUTOR("JS Executor"),  
    TB_VC_EXECUTOR("TB VC Executor"),  
    EDQS("TB Entity Data Query Service"),  
    TASK_PROCESSOR("Task Processor");  
  
    private final String label;  
  
    public static ServiceType of(String serviceType) {  
        return ServiceType.valueOf(serviceType.replace("-", "_").toUpperCase());  
    }  
  
}
枚举值 标签 职责
TB_CORE TB Core 核心服务，处理设备状态、订阅、API 等
TB_RULE_ENGINE TB Rule Engine 规则引擎，处理消息路由与规则链执行
TB_TRANSPORT TB Transport 传输层，处理 MQTT/HTTP/CoAP 等设备接入协议
JS_EXECUTOR JS Executor 执行规则链中的 JavaScript 脚本节点
TB_VC_EXECUTOR TB VC Executor 版本控制执行器，处理配置的 Git 同步
EDQS TB Entity Data Query Service 实体数据查询服务，高性能查询加速
TASK_PROCESSOR Task Processor 任务处理器，处理异步后台任务（Job）

见HashPartitionService 中:

TB_CORE、TB_RULE_ENGINE、TB_VC_EXECUTOR、EDQS、TASK_PROCESSOR 都有对应的 partitionSizesMap 和 partitionTopicsMap 条目参与分区计算；

TB_TRANSPORT 和 JS_EXECUTOR 本身不消费分区，但会被其他服务感知到用于路由决策。

集群变更时

其实就是在节点增删改时..

org.thingsboard.server.queue.discovery.ZkDiscoveryService#childEvent

java 复制代码 
/**
 * ZK 子节点变化回调（PathChildrenCacheListener 实现）。
 * 节点上线（`CHILD_ADDED`）时立即触发分区重算；
 * 节点下线（CHILD_REMOVED）时延迟重算, 留出时间窗口让服务重启后取消重算, 避免分区抖动。
 */
@Override
public void childEvent(CuratorFramework curatorFramework, PathChildrenCacheEvent pathChildrenCacheEvent) throws Exception {
	// 服务已停止或 ZK 客户端未就绪时忽略所有事件
	if (stopped) {
		log.debug("Ignoring {}. Service is stopped.", pathChildrenCacheEvent);
		return;
	}
	// ZK client 没准备好, 忽略..
	if (client.getState() != CuratorFrameworkState.STARTED) {
		log.debug("Ignoring {}, ZK client is not started, ZK client state [{}]", pathChildrenCacheEvent, client.getState());
		return;
	}
	// 事件携带的节点数据为空时忽略（初始化阶段偶发）
	ChildData data = pathChildrenCacheEvent.getData();
	if (data == null) {
		log.debug("Ignoring {} due to empty child data", pathChildrenCacheEvent);
		return;
	} else if (data.getData() == null) {
		log.debug("Ignoring {} due to empty child's data", pathChildrenCacheEvent);
		return;
	} else if (nodePath != null && nodePath.equals(data.getPath())) {
		// 事件来自当前节点自身：若被意外删除则立即重新注册, 其余事件忽略
		if (pathChildrenCacheEvent.getType() == CHILD_REMOVED) {
			log.info("ZK node for current instance is somehow deleted.");
			publishCurrentServer();
		}
		log.debug("Ignoring event about current server {}", pathChildrenCacheEvent);
		return;
	}
	// 将节点的字节数据反序列化为 ServiceInfo（Protobuf）
	TransportProtos.ServiceInfo instance;
	try {
		instance = TransportProtos.ServiceInfo.parseFrom(data.getData());
	} catch (InvalidProtocolBufferException e) {
		log.error("Failed to decode server instance for node {}", data.getPath(), e);
		throw e;
	}

	String serviceId = instance.getServiceId();
	ProtocolStringList serviceTypesList = instance.getServiceTypesList();

	log.trace("Processing [{}] event for [{}]", pathChildrenCacheEvent.getType(), serviceId);
	switch (pathChildrenCacheEvent.getType()) {
		case CHILD_ADDED:
			// 节点上线：检查 delayedTasks 中是否存在该服务的延迟重算任务
			// （服务重启时会先触发 CHILD_REMOVED 再触发 CHILD_ADDED, 若能在窗口内取消则跳过重算）
			ScheduledFuture<?> task = delayedTasks.remove(serviceId);
			if (task != null) {
				if (task.cancel(false)) {
					// 延迟任务取消成功：服务在时间窗口内完成重启, 无需重算分区
					log.info("[{}] Recalculate partitions ignored. Service was restarted in time [{}].",
							serviceId, serviceTypesList);
				} else {
					// 延迟任务已执行：重算已发生, 此处补一次确保状态最终一致
					log.debug("[{}] Going to recalculate partitions. Service was not restarted in time [{}]!",
							serviceId, serviceTypesList);
					recalculatePartitions();
				}
			} else {
				// 全新节点上线, 直接触发分区重算
				log.trace("[{}] Going to recalculate partitions due to adding new node [{}].",
						serviceId, serviceTypesList);
				recalculatePartitions();
			}
			break;
		case CHILD_REMOVED:
			// 节点下线：立即发布 OtherServiceShutdownEvent 通知其他组件
			zkExecutorService.submit(() -> applicationEventPublisher.publishEvent(new OtherServiceShutdownEvent(this, serviceId, serviceTypesList)));
			// 延迟触发分区重算, 给服务留出重启时间；若 CHILD_ADDED 在窗口内到来则取消此任务
			ScheduledFuture<?> future = zkExecutorService.schedule(() -> {
				log.debug("[{}] Going to recalculate partitions due to removed node [{}]",
						serviceId, serviceTypesList);
				ScheduledFuture<?> removedTask = delayedTasks.remove(serviceId);
				if (removedTask != null) {
					recalculatePartitions();
				}
			}, recalculateDelay, TimeUnit.MILLISECONDS);
			delayedTasks.put(serviceId, future);
			break;
		default:
			break;
	}
}
事件 处理逻辑 目的
CHILD_ADDED 立即 recalculatePartitions 新节点上线, 重新分配分区
CHILD_REMOVED 延迟 recalculateDelay ms 后重算 防抖：服务重启时先下线再上线, 若在时间窗口内重新注册则取消重算, 避免不必要的分区抖动
CHILD_REMOVED（自身节点） 重新 publishCurrentServer 如果是自己, 则是被意外删除则立即重新注册
org.thingsboard.server.queue.discovery.HashPartitionService#recalculatePartitions 是最终分区计算出口, 每次节点变化时调用没, 根据当前所有在线节点重新计算消息队列的分区分配。见: 三. HashPartitionService 队列分区计算

二. Kafka 消息队列

\[N_Kafka\]

首先集群中每个节点使用 Kafka 消息队列进行通信, 核心问题是

  1. 集群中: 每个角色都使用哪些生产者和消费者?
  2. 整体项目, 有哪些队列通道(Topic), 生产者, 消费者?

初始化 ProducerProvider (集群角色使用了哪些生产者?)

首先 TB 模块项目中使用 TbQueueProducerProvider 接口来获取生产者对象.

该接口有4个实现的 ProducerProvider 见下:

实现类 激活条件 对应微服务
TbCoreQueueProducerProvider @TbCoreComponent → service.type == monolith 或 tb-core tb-core
TbRuleEngineProducerProvider @ConditionalOnExpression → service.type == tb-rule-engine tb-rule-engine
TbTransportQueueProducerProvider @ConditionalOnExpression → service.type == tb-transport tb-transport
TbVersionControlProducerProvider @ConditionalOnExpression → service.type == tb-vc-executor tb-vc-executor

TbCoreQueueFactory 委托给 Factory实际创建 Producer

以 core 为例, 其最终创建注入的 TbCoreQueueFactory 取决于配置(集群, 单体, 单体+使用了kafka):

java 复制代码 
@Service
@TbCoreComponent
public class TbCoreQueueProducerProvider implements TbQueueProducerProvider {

    private final TbCoreQueueFactory tbQueueProvider;
    private TbQueueProducer<TbProtoQueueMsg<ToTransportMsg>> toTransport;
    private TbQueueProducer<TbProtoQueueMsg<ToRuleEngineMsg>> toRuleEngine;
    private TbQueueProducer<TbProtoQueueMsg<ToCoreMsg>> toTbCore;
    private TbQueueProducer<TbProtoQueueMsg<ToRuleEngineNotificationMsg>> toRuleEngineNotifications;
    private TbQueueProducer<TbProtoQueueMsg<ToCoreNotificationMsg>> toTbCoreNotifications;
    private TbQueueProducer<TbProtoQueueMsg<ToEdgeMsg>> toEdge;
    private TbQueueProducer<TbProtoQueueMsg<ToEdgeNotificationMsg>> toEdgeNotifications;
    private TbQueueProducer<TbProtoQueueMsg<ToEdgeEventNotificationMsg>> toEdgeEvents;
    private TbQueueProducer<TbProtoQueueMsg<ToUsageStatsServiceMsg>> toUsageStats;
    private TbQueueProducer<TbProtoQueueMsg<ToVersionControlServiceMsg>> toVersionControl;
    private TbQueueProducer<TbProtoQueueMsg<ToHousekeeperServiceMsg>> toHousekeeper;
    private TbQueueProducer<TbProtoQueueMsg<ToCalculatedFieldMsg>> toCalculatedFields;
    private TbQueueProducer<TbProtoQueueMsg<ToCalculatedFieldNotificationMsg>> toCalculatedFieldNotifications;

	  
	public TbCoreQueueProducerProvider(TbCoreQueueFactory tbQueueProvider) {  
	    this.tbQueueProvider = tbQueueProvider;  
	}  
	  
	@PostConstruct  
	public void init() {  
	    this.toTbCore = tbQueueProvider.createTbCoreMsgProducer();  
	    this.toTransport = tbQueueProvider.createTransportNotificationsMsgProducer();  
	    this.toRuleEngine = tbQueueProvider.createRuleEngineMsgProducer();  
	    this.toRuleEngineNotifications = tbQueueProvider.createRuleEngineNotificationsMsgProducer();  
	    this.toTbCoreNotifications = tbQueueProvider.createTbCoreNotificationsMsgProducer();  
	    this.toUsageStats = tbQueueProvider.createToUsageStatsServiceMsgProducer();  
	    this.toVersionControl = tbQueueProvider.createVersionControlMsgProducer();  
	    this.toHousekeeper = tbQueueProvider.createHousekeeperMsgProducer();  
	    this.toEdge = tbQueueProvider.createEdgeMsgProducer();  
	    this.toEdgeNotifications = tbQueueProvider.createEdgeNotificationsMsgProducer();  
	    this.toEdgeEvents = tbQueueProvider.createEdgeEventMsgProducer();  
	    this.toCalculatedFields = tbQueueProvider.createToCalculatedFieldMsgProducer();  
	    this.toCalculatedFieldNotifications = tbQueueProvider.createToCalculatedFieldNotificationMsgProducer();  
	}
  • 如果是集群+Kafka
    org.thingsboard.server.queue.provider.KafkaTbCoreQueueFactory#createTbCoreMsgProducer
java 复制代码 
@Override
public TbQueueProducer<TbProtoQueueMsg<ToCoreMsg>> createTbCoreMsgProducer() {
	TbKafkaProducerTemplate.TbKafkaProducerTemplateBuilder<TbProtoQueueMsg<ToCoreMsg>> requestBuilder = TbKafkaProducerTemplate.builder();
	requestBuilder.settings(kafkaSettings);
	requestBuilder.clientId("tb-core-to-core-" + serviceInfoProvider.getServiceId());
	requestBuilder.defaultTopic(topicService.buildTopicName(coreSettings.getTopic()));
	requestBuilder.admin(coreAdmin);
	return requestBuilder.build();
}
  • 如果是单体+使用了Kafka
    org.thingsboard.server.queue.provider.KafkaMonolithQueueFactory#createTbCoreMsgProducer
java 复制代码 
 @Override
public TbQueueProducer<TbProtoQueueMsg<ToCoreMsg>> createTbCoreMsgProducer() {
	TbKafkaProducerTemplate.TbKafkaProducerTemplateBuilder<TbProtoQueueMsg<ToCoreMsg>> requestBuilder = TbKafkaProducerTemplate.builder();
	requestBuilder.settings(kafkaSettings);
	requestBuilder.clientId("monolith-core-" + serviceInfoProvider.getServiceId());
	requestBuilder.defaultTopic(topicService.buildTopicName(coreSettings.getTopic()));
	requestBuilder.admin(coreAdmin);
	return requestBuilder.build();
}
  • 如果是单体+使用内存队列
    org.thingsboard.server.queue.provider.InMemoryMonolithQueueFactory#createTbCoreMsgProducer

..略了.. 不重要, 不适用生产..

单体+Kafka 的 QueueFactory

单体应用的 QueueFactory

org.thingsboard.server.queue.provider.KafkaMonolithQueueFactory

java 复制代码 
@Component
//如果队列类型配置的是 kafa, 并且还是单体应用类型; (所以哪怕是单体应用, 也可以单独把消息队列拆出来集群)
@ConditionalOnExpression("'${queue.type:null}'=='kafka' && '${service.type:null}'=='monolith'")
public class KafkaMonolithQueueFactory implements TbCoreQueueFactory, TbRuleEngineQueueFactory, TbVersionControlQueueFactory {

  .............

    private final AtomicLong consumerCount = new AtomicLong();
    private final AtomicLong edgeConsumerCount = new AtomicLong();

    public KafkaMonolithQueueFactory(TopicService topicService,
                                     TbKafkaSettings kafkaSettings,
                                     KafkaAdmin kafkaAdmin,
                                     TbServiceInfoProvider serviceInfoProvider,
                                     TbQueueCoreSettings coreSettings,
                                     TbQueueRuleEngineSettings ruleEngineSettings,
                                     TbQueueTransportApiSettings transportApiSettings,
                                     TbQueueTransportNotificationSettings transportNotificationSettings,
                                     TbQueueRemoteJsInvokeSettings jsInvokeSettings,
                                     TbQueueVersionControlSettings vcSettings,
                                     TbQueueEdgeSettings edgeSettings,
                                     TbQueueCalculatedFieldSettings calculatedFieldSettings,
                                     TbKafkaConsumerStatsService consumerStatsService,
                                     TbKafkaTopicConfigs kafkaTopicConfigs,
                                     EdqsConfig edqsConfig,
                                     TasksQueueConfig tasksQueueConfig) {
        this.topicService = topicService;
        this.kafkaSettings = kafkaSettings;
        this.kafkaAdmin = kafkaAdmin;
        this.serviceInfoProvider = serviceInfoProvider;
        this.coreSettings = coreSettings;
        this.ruleEngineSettings = ruleEngineSettings;
        this.transportApiSettings = transportApiSettings;
        this.transportNotificationSettings = transportNotificationSettings;
        this.jsInvokeSettings = jsInvokeSettings;
        this.vcSettings = vcSettings;
        this.consumerStatsService = consumerStatsService;
        this.edgeSettings = edgeSettings;
        this.calculatedFieldSettings = calculatedFieldSettings;
        this.edqsConfig = edqsConfig;
        this.tasksQueueConfig = tasksQueueConfig;

        this.coreAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getCoreConfigs());
        this.ruleEngineAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getRuleEngineConfigs());
        this.jsExecutorRequestAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getJsExecutorRequestConfigs());
        this.jsExecutorResponseAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getJsExecutorResponseConfigs());
        this.transportApiRequestAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getTransportApiRequestConfigs());
        this.transportApiResponseAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getTransportApiResponseConfigs());
        this.notificationAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getNotificationsConfigs());
        this.fwUpdatesAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getFwUpdatesConfigs());
        this.vcAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getVcConfigs());
        this.housekeeperAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getHousekeeperConfigs());
        this.housekeeperReprocessingAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getHousekeeperReprocessingConfigs());
        this.edgeAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getEdgeConfigs());
        this.edgeEventAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getEdgeEventConfigs());
        this.cfAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getCalculatedFieldConfigs());
        this.cfStateAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getCalculatedFieldStateConfigs());
        this.edqsEventsAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getEdqsEventsConfigs());
        this.edqsRequestsAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getEdqsRequestsConfigs());
        this.tasksAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getTasksConfigs());
    }

各个对象的作用

可以理解为每一个队列都有以下几个主要的对象, 以及他们的作用
flowchart LR subgraph 对象 direction TB Settings"TbQueueCoreSettings \
配置对象: 配置定义的\
topic名,业务上有几个分区.." TopicConfigs"TbKafkaTopicConfigs.coreConfigs\
Topic的配置参数,Admin使用这些参数来创建Topic,比如实际的分区数" Admin"coreAdmin : TbKafkaAdmin\
持有 coreConfigs\
负责管理/确保: topic 存在" end subgraph 执行时 Producer"TbCoreMsgProducer\
TbKafkaProducerTemplate\
持有 KafkaProducer + admin\
负责生产消息" Consumer"TbKafkaConsumerTemplate\
持有 KafkaConsumer + admin\
负责订阅消息" end TopicConfigs --> Admin Settings -->|"topic 名"| Producer Settings -->|"topic 名"| Consumer Admin -->|"createTopicIfNotExists"| Producer Admin -->|"createTopicIfNotExists"| Consumer

coreSettings

配置关于该队列的参数, topic 名称, 分区总数量

以 core 为例:

java 复制代码 
@Lazy
@Data
@Component
public class TbQueueCoreSettings {
	
	//Kafka topic 名称
    @Value("${queue.core.topic}")
    private String topic;
    
	//OTA 升级消息, 走独立的topic, 防止大量固件推送消息阻塞设备状态/遥测处理
    @Value("${queue.core.ota.topic:tb_ota_package}")
    private String otaPackageTopic;
	
	//统计各租户的 API 调用量、设备数等
    @Value("${queue.core.usage-stats-topic:tb_usage_stats}")
    private String usageStatsTopic;

	//清理任务 topic
    @Value("${queue.core.housekeeper.topic:tb_housekeeper}")
    private String housekeeperTopic;
    @Value("${queue.core.housekeeper.reprocessing-topic:tb_housekeeper.reprocessing}")
    private String housekeeperReprocessingTopic;
    
    //Kafka 分区总数量
	//这是 tb 的逻辑分区数,是HashPartitionService计算 理论上可以不一样, 但最好是一样的
    @Value("${queue.core.partitions}")
    private int partitions;
}
coreConfigs

都是字符串, 再转为 Map<String, String>方便使用, 这里配置会被 xxxAdmin 使用在 Kafka Broker 上创建（注意只有该topic不存在才会创建, 所以已有的再修改不会生效的）

kafka 的topic 参数说明

参考官方文档 - https://kafka.apache.org/43/configuration/topic-configs/

参数的说明:

https://kafka.apache.org/43/configuration/topic-configs/#topicconfigs_retention.ms

  • retention.ms = 604800000（7天）
    tb_core 消息是实时控制指令（设备属性更新、RPC 请求、会话超时等）, 不需要长期保留。7天是容灾窗口------如果 tb-core
    服务停机不超过7天重启, 未消费的消息仍可被处理；超过7天则认为消息已过期, 消费也无意义。
  • segment.bytes = 52428800（50MB）
    控制日志段滚动频率。50MB 是中等粒度：段太大导致 retention 清理粒度粗（只能整段删）, 段太小产生大量小文件。50MB 是官方推荐的中间值, 适合高吞吐控制类消息。
  • retention.bytes = 1048576000（~1GB）
    与 retention.ms 联合生效, 两者满足其一即触发清理。1GB 限制了单分区磁盘占用, 防止在消费速度跟不上时磁盘被打满。注意这是每个分区的限制, 由于 partitions=1, 整个 tb_core
    主题磁盘上限就是 1GB。
  • partitions = 1（物理分区数）

这个是Topic 真正的Kafka的物理分区数, 需要与 TbQueueCoreSettings.partitions（默认10）区分：

复制代码 
TbQueueCoreSettings.partitions = 10   → ThingsBoard 逻辑路由分区数
coreConfigs["partitions"] = 1         → Kafka Broker 物理分区数

TbKafkaTopicConfigs 存储着所有的 TopicConfigs:

org.thingsboard.server.queue.kafka.TbKafkaTopicConfigs

java 复制代码 
@Component
@TbKafkaComponent
public class TbKafkaTopicConfigs {

    public static final String NUM_PARTITIONS_SETTING = "partitions";

    @Value("${queue.kafka.topic-properties.core:}")
    private String coreProperties;
    @Value("${queue.kafka.topic-properties.rule-engine:}")
    private String ruleEngineProperties;
    @Value("${queue.kafka.topic-properties.transport-api:}")
    private String transportApiProperties;
    @Value("${queue.kafka.topic-properties.notifications:}")
    private String notificationsProperties;
    @Value("${queue.kafka.topic-properties.js-executor:}")
    private String jsExecutorProperties;
    @Value("${queue.kafka.topic-properties.ota-updates:}")
    private String fwUpdatesProperties;
    @Value("${queue.kafka.topic-properties.version-control:}")
    private String vcProperties;
    @Value("${queue.kafka.topic-properties.edge:}")
    private String edgeProperties;
    @Value("${queue.kafka.topic-properties.edge-event:}")
    private String edgeEventProperties;
    @Value("${queue.kafka.topic-properties.housekeeper:}")
    private String housekeeperProperties;
    @Value("${queue.kafka.topic-properties.housekeeper-reprocessing:}")
    private String housekeeperReprocessingProperties;
    @Value("${queue.kafka.topic-properties.calculated-field:}")
    private String calculatedFieldProperties;
    @Value("${queue.kafka.topic-properties.calculated-field-state:}")
    private String calculatedFieldStateProperties;
    @Value("${queue.kafka.topic-properties.edqs-events:}")
    private String edqsEventsProperties;
    @Value("${queue.kafka.topic-properties.edqs-requests:}")
    private String edqsRequestsProperties;
    @Value("${queue.kafka.topic-properties.edqs-state:}")
    private String edqsStateProperties;
    @Value("${queue.kafka.topic-properties.tasks:}")
    private String tasksProperties;

    @Getter
    private Map<String, String> coreConfigs;
	...
    @PostConstruct
    private void init() {
        coreConfigs = PropertyUtils.getProps(coreProperties);
        ...
    }

}
coreAdmin

org.thingsboard.server.queue.provider.KafkaMonolithQueueFactory#KafkaMonolithQueueFactory

java 复制代码 
public KafkaMonolithQueueFactory(TopicService topicService,
								 TbKafkaSettings kafkaSettings,
								 KafkaAdmin kafkaAdmin,
								 TbServiceInfoProvider serviceInfoProvider,
								 TbQueueCoreSettings coreSettings,
								 TbQueueRuleEngineSettings ruleEngineSettings,
								 TbQueueTransportApiSettings transportApiSettings,
								 TbQueueTransportNotificationSettings transportNotificationSettings,
								 TbQueueRemoteJsInvokeSettings jsInvokeSettings,
								 TbQueueVersionControlSettings vcSettings,
								 TbQueueEdgeSettings edgeSettings,
								 TbQueueCalculatedFieldSettings calculatedFieldSettings,
								 TbKafkaConsumerStatsService consumerStatsService,
								 TbKafkaTopicConfigs kafkaTopicConfigs,
								 EdqsConfig edqsConfig,
								 TasksQueueConfig tasksQueueConfig) {

	....
	this.coreAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getCoreConfigs());
	this.ruleEngineAdmin = new TbKafkaAdmin(kafkaSettings, kafkaTopicConfigs.getRuleEngineConfigs());
	
}

都是同一个 TbKafkaAdmin 对象不同实例进行管理, 但是每个实例的 kafkaSettings 和 kafkaTopicConfigs 不一样

ThingsBoard 在首次 createTopicIfNotExists() 时用 coreConfigs 创建 Topic, 此后不会再更新已存在的 Topic 配置。所以：如果先启动 TB（自动建了 Topic）, 再改 thingsboard.yml 里的参数 → Kafka 上的 Topic 不会跟着变, 配置已经脱节

TbCoreQueueProducerProvider (集群角色生产哪些消息?)

IDEA直接检索, 哪里调用该方法获取了生产者就知道了

org.thingsboard.server.queue.provider.TbCoreQueueProducerProvider#getTbCoreMsgProducer

实体消息路由到集群内部

org.thingsboard.server.service.queue.DefaultTbClusterService#pushMsgToCore(ToDeviceActorNotificationMsg, TbQueueCallback)

java 复制代码 
@Override
public void pushMsgToCore(TenantId tenantId, EntityId entityId, ToCoreMsg msg, TbQueueCallback callback) {
  // 用一致性哈希计算目标分区（哪个 Core 节点负责此 entityId）
  TopicPartitionInfo tpi = partitionService.resolve(ServiceType.TB_CORE, tenantId, entityId);
  // 发送到对应分区
  producerProvider.getTbCoreMsgProducer().send(tpi, new TbProtoQueueMsg<>(UUID.randomUUID(), msg), callback);
}

例如: 设备的 RPC 调用

java 复制代码 
    @ApiOperation(value = "Delete persistent RPC", notes = "Deletes the persistent RPC request." + TENANT_AUTHORITY_PARAGRAPH)
    @PreAuthorize("hasAnyAuthority('TENANT_ADMIN')")
    @RequestMapping(value = "/persistent/{rpcId}", method = RequestMethod.DELETE)
    @ResponseBody
    public void deleteRpc(
            @Parameter(description = RPC_ID_PARAM_DESCRIPTION, required = true)
            @PathVariable(RPC_ID) String strRpc) throws ThingsboardException {
        checkParameter("RpcId", strRpc);
        RpcId rpcId = new RpcId(UUID.fromString(strRpc));
        Rpc rpc = checkRpcId(rpcId, Operation.DELETE);

        if (rpc != null) {
            if (rpc.getStatus().isPushDeleteNotificationToCore()) {
                RemoveRpcActorMsg removeMsg = new RemoveRpcActorMsg(getTenantId(), rpc.getDeviceId(), rpc.getUuidId());
                log.trace("[{}] Forwarding msg {} to queue actor!", rpc.getDeviceId(), rpc);
                tbClusterService.pushMsgToCore(removeMsg, null);
            }

            rpcService.deleteRpc(getTenantId(), rpcId);
            rpc.setStatus(RpcStatus.DELETED);

            TbMsg msg = TbMsg.newMsg()
                    .type(TbMsgType.RPC_DELETED)
                    .originator(rpc.getDeviceId())
                    .copyMetaData(TbMsgMetaData.EMPTY)
                    .data(JacksonUtil.toString(rpc))
                    .build();
            tbClusterService.pushMsgToRuleEngine(getTenantId(), rpc.getDeviceId(), msg, null);
        }
    }
}

Transport 消息跨节点转发

org.thingsboard.server.common.transport.service.DefaultTransportService

java 复制代码 
 protected void sendToDeviceActor(SessionInfoProto sessionInfo, TransportToDeviceActorMsg actorMsg, ...) {
      // 包装成 ToCoreMsg, 字段为 toDeviceActorMsg
      ToCoreMsg toCoreMsg = ToCoreMsg.newBuilder().setToDeviceActorMsg(actorMsg).build();
      sendToCore(getTenantId(sessionInfo), getDeviceId(sessionInfo), toCoreMsg, ...);
  }

  private void sendToCore(TenantId tenantId, EntityId entityId, ToCoreMsg msg, UUID routingKey, ...) {
      // 同样用 partitionService.resolve() 计算目标分区
      TopicPartitionInfo tpi = partitionService.resolve(ServiceType.TB_CORE, tenantId, entityId);
      // routingKey = deviceId.getId(), 保证同一设备消息总写入同一分区（保证有序）
      tbCoreMsgProducer.send(tpi, new TbProtoQueueMsg<>(routingKey, msg), wrappedCallback);
  }

设备活动/上线消息

设备上线消息会经过 core queue

org.thingsboard.server.service.edge.rpc.processor.telemetry.BaseTelemetryProcessor

java 复制代码 
 @PostConstruct
public void init() {
	tbCoreMsgProducer = producerProvider.getTbCoreMsgProducer();
}

public List<ListenableFuture<Void>> processTelemetryMsg(TenantId tenantId, EntityDataProto entityData) throws Exception {
	log.trace("[{}] processTelemetryMsg [{}]", tenantId, entityData);
	
			.............
			//根据租户 设备ID 推断指定的分区
				TopicPartitionInfo tpi = partitionService.resolve(ServiceType.TB_CORE, tenantId, deviceId);
			//设备 deviceActivityMsg 消息
				tbCoreMsgProducer.send(tpi, new TbProtoQueueMsg<>(deviceId.getId(),
						TransportProtos.ToCoreMsg.newBuilder().setDeviceActivityMsg(deviceActivityMsg).build()), null);
			}
		}
		if (entityData.hasAttributeDeleteMsg()) {
			result.add(processAttributeDeleteMsg(tenantId, entityId, entityData.getAttributeDeleteMsg(), entityData.getEntityType()));
		}
	} else {
		log.warn("[{}] Skipping telemetry update msg because entity doesn't exists on edge, {}", tenantId, entityData);
	}
	return result;
}

DefaultTbCoreConsumerService (集群角色订阅消费消息)

注意: 序列化到 kafka 的消息定义, 在 \thingsboard-mater\common\proto\src\main\proto\queue.proto中

\[Protocol buffers 协议\]

org.thingsboard.server.service.queue.DefaultTbCoreConsumerService#init

java 复制代码 
@PostConstruct
public void init() {

    @PostConstruct
    public void init() {
        super.init("tb-core");
        this.deviceActivityEventsExecutor = MoreExecutors.listeningDecorator(Executors.newSingleThreadExecutor(ThingsBoardThreadFactory.forName("tb-core-device-activity-events-executor")));
        
	    // 核心, 消费ToCoreMsg, 每个分区独立一个 Consumer 线程（当consumerPerPartition=true 时, 见 MainQueueConsumerManager）
        this.mainConsumer = MainQueueConsumerManager.<TbProtoQueueMsg<ToCoreMsg>, QueueConfig>builder()
                .queueKey(new QueueKey(ServiceType.TB_CORE)) // 标识这是 tb_core topic
                .config(QueueConfig.of(consumerPerPartition, pollInterval)) // 每25ms poll一次
                .msgPackProcessor(this::processMsgs)// 回调
                .consumerCreator((config, tpi) -> queueFactory.createToCoreMsgConsumer())// 创建 KafkaConsumer
                .consumerExecutor(consumersExecutor)
                .scheduler(scheduler)
                .taskExecutor(mgmtExecutor)
                .build();
        // 统计用量 各租户的 API 调用量、设备数等
        this.usageStatsConsumer = QueueConsumerManager.<TbProtoQueueMsg<ToUsageStatsServiceMsg>>builder()
                .name("TB Usage Stats")
                .msgPackProcessor(this::processUsageStatsMsg)
                .pollInterval(pollInterval)
                .consumerCreator(queueFactory::createToUsageStatsServiceMsgConsumer)
                .consumerExecutor(consumersExecutor)
                .threadPrefix("usage-stats")
                .build();
        // 固件 OTA
        this.firmwareStatesConsumer = QueueConsumerManager.<TbProtoQueueMsg<ToOtaPackageStateServiceMsg>>builder()
                .name("TB Ota Package States")
                .msgPackProcessor(this::processFirmwareMsgs)
                .pollInterval(pollInterval)
                .consumerCreator(queueFactory::createToOtaPackageStateServiceMsgConsumer)
                .consumerExecutor(consumersExecutor)
                .threadPrefix("firmware")
                .build();
    }

MainQueueConsumerManager

类似组合模式 DefaultTbCoreConsumerService 委托给 MainQueueConsumerManager 它是真正使用线程轮询拉取消息的

它的核心作用是: 根据当前节点负责的分区集合，动态创建/销毁 Kafka Consumer 线程。
flowchart TD MCM"MainQueueConsumerManager" MCM -->|"consumerPerPartition=true"| CPP"ConsumerPerPartitionWrapper\
Map\<TopicPartitionInfo, ConsumerTask\>\
每个分区 = 1个KafkaConsumer + 1个线程" MCM -->|"consumerPerPartition=false"| SC"SingleConsumerWrapper\
全部分区 = 1个KafkaConsumer + 1个线程\
consumer.subscribe(allPartitions)"

分区更改时...

org.thingsboard.server.service.queue.DefaultTbCoreConsumerService#onTbApplicationEvent

java 复制代码 
@Override  
protected void onTbApplicationEvent(PartitionChangeEvent event) {  
    log.debug("Subscribing to partitions: {}", event.getCorePartitions());  
    //分区更改时, 停掉或者启动消费线程
    mainConsumer.update(event.getCorePartitions());  
    
    usageStatsConsumer.subscribe(event.getCorePartitions()  
            .stream()  
            .map(tpi -> tpi.withTopic(usageStatsConsumer.getConsumer().getTopic()))  
            .collect(Collectors.toSet()));  
}

PartitionChangeEvent 事件, 从哪里来? 见 HashPartitionService 计算完后发布的事件

最后 总结一下

回来解决: 整体项目, 有哪些队列通道(Topic), 生产者, 消费者?
flowchart LR subgraph 配置 YML"thingsboard.yml\
queue.\* 配置项" ENV"环境变量\
TB_QUEUE_CORE_TOPIC 等" Settings"TbQueueCoreSettings\
TbQueueRuleEngineSettings\
TbQueueTransportApiSettings" YML --> Settings ENV --> Settings end subgraph 创建工厂 Factory"KafkaMonolithQueueFactory\
KafkaTbCoreQueueFactory\
KafkaTbRuleEngineQueueFactory\
KafkaTbTransportQueueFactory\
KafkaTbVersionControlQueueFactory 再细化一点(xxxSettings,xxxConfigs,xxxAdmin)" end subgraph 生产提供者 Provider"TbCoreQueueProducerProvider\
TbRuleEngineProducerProvider\
TbTransportQueueProducerProvider\
TbVersionControlProducerProvider" end subgraph 消费服务 Consumer"DefaultTbCoreConsumerService\
DefaultTbRuleEngineConsumerService\
DefaultTbEdgeConsumerService\
DefaultTbCalculatedFieldConsumerService\
Defaul tClusterVersionControlService\
EdqsProcessor" end Settings --> Factory Factory --> Provider Factory --> Consumer

Topic Protobuf 类型 生产 消费 用途
tb_core ToCoreMsg transport / rule-engine tb-core 设备上行数据入口，分发至 DeviceActor
tb_core.notifications.{serviceId} ToCoreNotificationMsg tb-core tb-core 跨节点通知（RPC 回包、会话更新）
tb_rule_engine.{queueName} ToRuleEngineMsg tb-core tb-rule-engine 进入规则链执行
tb_rule_engine.notifications.{serviceId} ToRuleEngineNotificationMsg tb-core tb-rule-engine 规则链热更新通知
tb_transport.api.requests/responses TransportApiRequestMsg TransportApiResponseMsg transport / tb-core tb-core / transport 设备鉴权、属性获取 API 请求响应
tb_transport.notifications.{serviceId} ToTransportMsg tb-core transport 下行 RPC / 属性推送至设备
js_eval.requests/responses RemoteJsRequest RemoteJsResponse rule-engine / js-executor js-executor / rule-engine 规则链脚本执行请求与结果回传
edqs.* ToEdqsMsg FromEdqsMsg tb-core / edqs edqs / tb-core 实体查询索引同步与请求响应

三. HashPartitionService 队列分区计算

关键的属性

HashPartitionService有 一堆的容器集合字段

字段 类型 存储
myPartitions ConcurrentMap<QueueKey, List> 当前节点负责的分区编号列表, key 是队列标识, value 是分区下标
partitionTopicsMap ConcurrentMap<QueueKey, String> 每个队列对应的 Kafka topic 名
partitionSizesMap ConcurrentMap<QueueKey, Integer> 每个队列的总分区数
queueConfigs ConcurrentMap<QueueKey, QueueConfig> 规则引擎队列的详细配置（提交策略等）
tenantRoutingInfoMap ConcurrentMap<TenantId, TenantRoutingInfo> 租户的路由信息缓存（决定租户走哪个规则引擎实例）
currentOtherServices List 当前已知的其他集群节点列表
tbTransportServicesByType Map<String, List> 按传输类型分组的 Transport 节点列表
responsibleServices Map<TenantProfileId, List> 按租户 Profile 分组的责任节点, 用于隔离租户流量
hashFunction HashFunction Guava 哈希函数实例, 用于计算 entityId 落在哪个分区

QueueKey 是核心索引, 结构为 (ServiceType, tenantId, queueName), 三者组合唯一确定一条队列, 所有 Map 均以它为 key。

计算的逻辑

org.thingsboard.server.queue.discovery.HashPartitionService#recalculatePartitions

java 复制代码 
@Override
    public synchronized void recalculatePartitions(ServiceInfo currentService, List<ServiceInfo> otherServices) {
        log.info("Recalculating partitions");
        tbTransportServicesByType.clear();
        logServiceInfo(currentService);
        otherServices.forEach(this::logServiceInfo);

        // queueServicesMap: 每个队列 -> 集群中所有能处理该队列的节点列表
        // responsibleServices: 租户 Profile -> 被专属指定负责该 Profile 的规则引擎节点（租户隔离用）
        Map<QueueKey, List<ServiceInfo>> queueServicesMap = new HashMap<>();
        Map<TenantProfileId, List<ServiceInfo>> responsibleServices = new HashMap<>();
        addNode(currentService, queueServicesMap, responsibleServices);
        for (ServiceInfo other : otherServices) {
            addNode(other, queueServicesMap, responsibleServices);
        }
        // 按 serviceId 排序：所有节点独立计算时用相同顺序，保证结果一致无需协商
        queueServicesMap.values().forEach(list -> list.sort(Comparator.comparing(ServiceInfo::getServiceId)));
        responsibleServices.values().forEach(list -> list.sort(Comparator.comparing(ServiceInfo::getServiceId)));

        // newPartitions: 本次计算出的当前节点负责的分区编号列表，key=队列，value=分区下标列表
        final ConcurrentMap<QueueKey, List<Integer>> newPartitions = new ConcurrentHashMap<>();
        partitionSizesMap.forEach((queueKey, size) -> {
            for (int i = 0; i < size; i++) {
                try {
                    List<ServiceInfo> services = resolveByPartitionIdx(queueServicesMap.get(queueKey), queueKey, i, responsibleServices);
                    log.trace("Server responsible for {}[{}] - {}", queueKey, i, services);
                    if (services.contains(currentService)) {
                        newPartitions.computeIfAbsent(queueKey, key -> new ArrayList<>()).add(i);
                    }
                } catch (Exception e) {
                    log.warn("Failed to resolve server responsible for {}[{}]", queueKey, i, e);
                }
            }
        });
        this.responsibleServices = responsibleServices;

        // oldPartitions: 上一次的分区快照，用于差量对比
        final ConcurrentMap<QueueKey, List<Integer>> oldPartitions = myPartitions;
        myPartitions = newPartitions;

        // changedPartitionsMap: 需要通知的变更队列 -> 新分区集合（空集合表示移除）
        // oldPartitionsMap: 对应的旧分区集合，一并传入事件供监听方使用
        Map<QueueKey, Set<TopicPartitionInfo>> changedPartitionsMap = new HashMap<>();
        Map<QueueKey, Set<TopicPartitionInfo>> oldPartitionsMap = new HashMap<>();

        // removed: 旧分区中存在、新分区中不存在的队列（当前节点不再负责）
        Set<QueueKey> removed = new HashSet<>();
        oldPartitions.forEach((queueKey, partitions) -> {
            if (!newPartitions.containsKey(queueKey)) {
                removed.add(queueKey);
            }
        });
        // 规则引擎租户队列：partitionSizesMap 中有定义但新分区无分配，也视为移除
        if (serviceInfoProvider.isService(ServiceType.TB_RULE_ENGINE)) {
            partitionSizesMap.keySet().stream()
                    .filter(queueKey -> queueKey.getType() == ServiceType.TB_RULE_ENGINE &&
                                        !queueKey.getTenantId().isSysTenantId() &&
                                        !newPartitions.containsKey(queueKey))
                    .forEach(removed::add);
        }
        // 移除的队列写入空集合，消费者收到后停止订阅
        removed.forEach(queueKey -> {
            changedPartitionsMap.put(queueKey, Collections.emptySet());
        });

        myPartitions.forEach((queueKey, partitions) -> {
            if (!partitions.equals(oldPartitions.get(queueKey))) {
                changedPartitionsMap.put(queueKey, toTpiList(queueKey, partitions));
                oldPartitionsMap.put(queueKey, toTpiList(queueKey, oldPartitions.get(queueKey)));
            }
        });
        if (!changedPartitionsMap.isEmpty()) {
            changedPartitionsMap.entrySet().stream()
                    .collect(Collectors.groupingBy(entry -> entry.getKey().getType(), Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)))
                    .forEach((serviceType, partitionsMap) -> {
	                    // 发布 PartitionChangeEvent 事件, 当前节点负责的分区发生变化时发布, 按 ServiceType 分组
                        publishPartitionChangeEvent(serviceType, partitionsMap, oldPartitionsMap);
                    });
        }

        if (currentOtherServices == null) {
            // 首次计算，无旧快照，不发布拓扑变化事件
            currentOtherServices = new ArrayList<>(otherServices);
        } else {
            // changes: 本次与上次相比，节点列表发生变化的 QueueKey 集合
            Set<QueueKey> changes = new HashSet<>();
            Map<QueueKey, List<ServiceInfo>> currentMap = getServiceKeyListMap(currentOtherServices);
            Map<QueueKey, List<ServiceInfo>> newMap = getServiceKeyListMap(otherServices);
            currentOtherServices = otherServices;
            currentMap.forEach((key, list) -> {
                if (!list.equals(newMap.get(key))) {
                    changes.add(key);
                }
            });
            // newMap 中剩余的 key 是本次新增的队列
            currentMap.keySet().forEach(newMap::remove);
            changes.addAll(newMap.keySet());
            if (!changes.isEmpty()) {
	            // 发布 ClusterTopologyChangeEvent 事件: 集群节点组成变化时发布, 携带受影响的 `QueueKey` 集合
                applicationEventPublisher.publishEvent(new ClusterTopologyChangeEvent(this, changes));
                responsibleServices.forEach((profileId, serviceInfos) -> {
                    if (profileId != null) {
                        log.info("Servers responsible for tenant profile {}: {}", profileId, toServiceIds(serviceInfos));
                    } else {
                        log.info("Servers responsible for system queues: {}", toServiceIds(serviceInfos));
                    }
                });
            }
        }
		// 发布 ServiceListChangedEvent 事件, 每次重算后无条件发布，携带最新全量服务节点列表，不做差量判断
        applicationEventPublisher.publishEvent(new ServiceListChangedEvent(otherServices, currentService));
    }

应用事件

会发布三个应用事件

  1. PartitionChangeEvent 事件: 当前节点负责的分区发生变化时发布, 按 ServiceType 分组
  2. ClusterTopologyChangeEvent 事件: 集群节点组成变化时发布, 携带受影响的 QueueKey 集合
  3. ServiceListChangedEvent 事件: 每次重算后无条件发布，携带最新全量服务节点列表
相关推荐
我可能是个假开发2 小时前
FTP Unicode 文件名导致 `550` 的问题分析与解决方案
java·开发语言·spring
caoerzhong2 小时前
中小企业上 WMS 该先上哪几块：JeeWMS 开源 Java 仓库管理系统的分批上线清单
java·python·开源
【JAVA】玩家3 小时前
Spring核心原理全解析：从零到生产实战
java·后端·spring
Wang's Blog3 小时前
Java框架 SpringCloud 快速入门: 实现 Feign 最佳实践（抽取方式）
java·spring cloud
MandalaO_O4 小时前
IDEA 开发（快捷键 + 调试 + 序列化）
java·ide·intellij-idea
xiaoqiMikko5 小时前
JVM 线上排查实战(七):jps 看不见它、jstack 连不上它,可它明明活得好好的
java·jvm
狼爷6 小时前
Rust/Go/Java/Python/PHP 大比拼：负载下后端框架到底差多少？
java·后端·编程语言
念何架构之路6 小时前
zap扩展生态与总结
java·前端·数据库
第七页独白7 小时前
汽车零件厂如何通过 QMS 真正落地 IATF 16949——QMS软件系统：品质检验-内审稽核-8d客诉管理：全星质量管理软件系统
java·前端·数据库
热门推荐
01如何新建文件夹？ 电脑新建文件夹的4种方法02历年考研数学一、数学二、数学三真题试卷及答案PDF03国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）04GitHub 镜像站点05Telegram APK 下载安装完整指南 — 2026年最新06蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接07微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新08Windows 10/11 如何退出微软账户并改用本地账户登录（含找不到选项的解决方法）092026 爱知・名古屋亚运会怎么看？直播平台+赛程安排分享10给AI一张“业务概念地图“：本体如何从哲学走向企业智能