Kafka Rebanlace次数过高问题

Kafka Rebanlace次数过高问题

环境:

Kafka Server 2.6.x

Kafka Client Java 2.8.2

缘起:

最近发现Kafka Rebalance次数着实有点多,一天达到了六十多次,感觉不太正常,于是查了下日志发现:

log 复制代码
Offset commit cannot be completed since 
the consumer is not part of an active group for auto partition assignment; 
it is likely that the consumer was kicked out of the group.

大意是某个kakfa client提交offset失败,因为已经在分组中下线。

为什么会下线?

我们来了解下什么情况下会掉线,常见情况如下:

1. 心跳原因:

kafka在n次心跳未收到后认为这个kafka client已经离线,于是server端会踢下线,至于n次是多少次,需要计算,有两个参数,一个是heartbeat.interval.ms,代表多久一次心跳,默认是3000ms,也就是3秒,还有一个参数是session.timeout.ms,代表保持session的超时时间,默认10000ms,也就是10秒。n = session.timeout.ms / heartbeat.interval.ms,也就是说3次之后不到第四次就会被踢下线,至于为什么不是正好3倍,官网解释是heartbeat.interval.ms的值建议小于session.timeout.ms1/3,两个参数官网解释如下:

session.timeout.ms

The timeout used to detect client failures when using Kafka's group management facility. The client sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this client from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms.

Type: int

Default: 10000 (10 seconds)
heartbeat.interval.ms

The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

Type: int

Default: 3000 (3 seconds)

以上来自Kafka官网 https://kafka.apache.org/28/documentation.html#consumerconfigs

2. 拉取间隔原因

和这个原因有关的参数是max.poll.interval.ms,这个参数的意思是两次poll()操作之间如果超过了这个值,也会被服务端踢下线,默认300000ms,也就是300秒,5分钟。

max.poll.interval.ms

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member. For consumers using a non-null group.instance.id which reach this timeout, partitions will not be immediately reassigned. Instead, the consumer will stop sending heartbeats and partitions will be reassigned after expiration of session.timeout.ms. This mirrors the behavior of a static consumer which has shutdown.

Type: int

Default: 300000 (5 minutes)

以上来自Kafka官网 https://kafka.apache.org/28/documentation.html#consumerconfigs

定位

当时做性能优化的时候,这个kafka处理逻辑统计了时间于是找到了以下日志:

log 复制代码
当前拉取了数据条数 10 耗时 411260ms thread: KafkaXxxReceiver-pool-3

处理10条数据居然用了411260ms,这是只是其中一条,通过模糊查询还找到了更多了超过300秒的数据,已经确认是这里的问题了。

优化思路

  1. 适当调大参数max.poll.interval.ms,或者调小每次拉取的消息数max.poll.records
  2. 因之前压测未出现此问题,需要进一步定位到底是哪一块用时较长,进行业务上的优化。
相关推荐
ZhongruiRao几秒前
Springboot+PostgreSQL+MybatisPlus存储JSON或List、数组(Array)数据
spring boot·postgresql·json
IT学长编程13 分钟前
计算机毕业设计 Java酷听音乐系统的设计与实现 Java实战项目 附源码+文档+视频讲解
java·spring boot·毕业设计·课程设计·毕业论文·音乐系统·计算机毕业设计选题
IT学长编程30 分钟前
计算机毕业设计 基于协同过滤算法的个性化音乐推荐系统的设计与实现 Java实战项目 附源码+文档+视频讲解
java·spring boot·毕业设计·毕业论文·协同过滤算法·计算机毕业设计选题·个性化音乐推荐系统
华农第一蒟蒻1 小时前
Java中JWT(JSON Web Token)的运用
java·前端·spring boot·json·token
计算机学姐1 小时前
基于SpringBoot+Vue的高校运动会管理系统
java·vue.js·spring boot·后端·mysql·intellij-idea·mybatis
老华带你飞1 小时前
公寓管理系统|SprinBoot+vue夕阳红公寓管理系统(源码+数据库+文档)
java·前端·javascript·数据库·vue.js·spring boot·课程设计
程序员陆通2 小时前
Spring Boot RESTful API开发教程
spring boot·后端·restful
我是浮夸3 小时前
MyBatisPlus——学习笔记
java·spring boot·mybatis
杨荧3 小时前
【JAVA开源】基于Vue和SpringBoot的水果购物网站
java·开发语言·vue.js·spring boot·spring cloud·开源
liuxin334455663 小时前
大学生就业招聘:Spring Boot系统的高效实现
spring boot·后端·mfc