这里是weihubeats ,觉得文章不错可以关注公众号小奏技术,文章首发。拒绝营销号,拒绝标题党
RocketMQ 版本
- 5.1.0
背景
大促期间为了保证RocketMQ的稳定性,进行了扩容,扩容实战相关的操作博文之前也提到过
可以参考这里 RocketMQ无损扩容实战:weihubeats.blog.csdn.net/article/det...
实操
正常的操作流程是如何的呢?
- 停写
broker - 待
broker上面没有消息需要消费后,下线broker
RocketMQ停写之前分析过有两种
- 仅更新
Nameserve内存中的broker的写入权限,不实际去更新broker的写入权限 - 更新
broker的写入权限
1这种方式更适合broker升级更新,broker重启重新注册到Nameserve写入权限就会恢复
我们本次要缩容,所以直接更新broker的写入权限即可
broker停写
- cd到部署目录
sh
cd /home/ubuntu/rocketmq/distribution/target/rocketmq-5.1.0/rocketmq-5.1.0
- 执行停写
sh
sh bin/mqadmin updateBrokerConfig -b 192.168.1.2:30911 -k brokerPermission -v 4
观察集群流量
less
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD)
xiaozoujishu broker-c 0 192.168.1.11:30911 V5_1_0 0.00(0,0ms) 1.40(0,0ms)
xiaozoujishu broker-c 2 192.168.1.6:30911 V5_1_0 0.00(0,0ms) 0.00(0,0ms)
xiaozoujishu broker-d 0 192.168.1.4:30911 V5_1_0 xxxx.76(0,0ms) xxxx.33(0,0ms)
xiaozoujishu broker-d 1 192.168.1.3:30911 V5_1_0 xxxx.76(0,0ms) 0.00(0,0ms)
xiaozoujishu broker-e 0 192.168.1.8:30911 V5_1_0 xxxx.27(0,0ms) xxxx.11(0,0ms)
xiaozoujishu broker-e 2 192.168.1.2:30911 V5_1_0 xxxx.85(0,0ms) 0.00(0,0ms)
xiaozoujishu broker-f 0 192.168.1.9:30911 V5_1_0 xxxx.76(0,0ms) xxxx.33(0,0ms)
xiaozoujishu broker-f 2 192.168.1.1:30911 V5_1_0 xxxx.71(0,0ms) 0.00(0,0ms)
可以看到broker-c集群中的 broker的 InTPS写入流量已经为0了,OutTPS还有少量流量
观察消费者流量
sh
sh bin/mqadmin brokerConsumeStats -b 192.168.1.11:30911 -n 192.168.1.22:9876
- 展示
yaml
#Topic #Group #Broker Name #QID #Broker Offset #Consumer Offset #Diff #LastTime
%RETRY%GID_INFO_xiaozou GID_INFO_xiaozou broker-c 0 12614511 12614511 0 2023-12-11 21:42:17
xiaozou GID_INFO_xiaozou broker-c 0 321996960 321996960 0 2023-12-11 21:42:19
xiaozou GID_INFO_xiaozou broker-c 1 482991618 482991618 0 2023-12-11 21:42:18
xiaozou GID_INFO_xiaozou broker-c 2 483209653 483209653 0 2023-12-11 21:42:19
xiaozou GID_INFO_xiaozou broker-c 3 482440511 482440511 0 2023-12-11 21:42:17
xiaozou GID_INFO_xiaozou broker-c 4 486478174 486478174 0 2023-12-11 21:42:18
xiaozou GID_INFO_xiaozou broker-c 5 485433955 485433955 0 2023-12-11 21:42:19
xiaozou GID_INFO_xiaozou broker-c 6 483460016 483460016 0 2023-12-11 21:42:19
xiaozou GID_INFO_xiaozou broker-c 7 882383062 882383062 0 2023-12-11 21:42:17
xiaozou GID_INFO_xiaozou broker-c 8 485121973 485121973 0 2023-12-11 21:42:19
xiaozou GID_INFO_xiaozou broker-c 9 781475152 781475152 0 2023-12-11 21:42:17
xiaozou GID_INFO_xiaozou broker-c 10 585496105 585496105 0 2023-12-11 21:42:17
正常刚停写是还会有一些消息待消费的,就是diff不为0.我这里是停写很久了,所以所有消息都消费完了
我们等到所有的diff都为0,代表没有消息需要消费了。
这个时候就可以停掉broker
停止broker
待没有滞留消息后我们直接kill broker即可。这里推荐先kill掉slave broker,然后再kill slave broker 不要使用kill -9
停止完成后即可回收我们的机器了
总结
总的来说缩容还是比较简单的,就是broker停写,然后等broker上面的消息都消费完成后即可下线。
下线的时候先下线slave,然后是master。避免出现主从自动切换