记录一次线上因kafka宕机而导致java服务cpu飙升的情况

codeydj2025-01-06 11:11

问题背景

线上kafka（自搭）所在服务器，因为防火墙问题，导致线上服务无法连接，从而开始持续重试。

发消息为异步发送，且线上业务服务qps为4000

定位问题：问题定位较为简单，查看日志(本地模拟日志)

复制代码

2025-01-04T14:41:25,577 INFO  [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient: [Producer clientId=producer-1] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 8073 ms.
2025-01-04T14:41:25,578 WARN  [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater: [Producer clientId=producer-1] Bootstrap broker 192.168.8.16:9092 (id: -1 rack: null) disconnected
2025-01-04T14:41:46,696 INFO  [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient: [Producer clientId=producer-1] Node -1 disconnected.
2025-01-04T14:41:46,696 WARN  [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient: [Producer clientId=producer-1] Connection to node -1 (/192.168.8.16:9092) could not be established. Broker may not be available.

通过日志发现是kafka连接错误，且持续的重连

解决问题：解决也很简单，网络互通后，问题解决。

引发思考，为什么会导致服务器CPU飙升？

spring kafka producer的失败重试

spring kafka的失败重试为默认无限重试，每次请求都会发送消息进行kafka的重连尝试，从而最终导致jvm cpu飙升，服务请求拒绝