kombu 运行超长时间任务导致RabbitMQ消费者断开

modelmd2025-07-20 12:37

kombu 运行超长时间任务导致RabbitMQ消费者断开

在排查异步任务没有执行的问题时，发现RabbitMQ管理端消息队列消费者异常断开，页面显示 ... no consumers ... 如下图所示

文章会引入测试代码模拟超长时间的异步任务，以及分析消费者为啥断开，最后给出解决方案

测试代码

异步任务执行使用的时 kombu的示例代码，参考 https://docs.celeryq.dev/projects/kombu/en/stable/userguide/examples.html

  1. queues.py
python 复制代码 
from __future__ import annotations

from kombu import Exchange, Queue

task_exchange = Exchange('kombu_big_tasks', type='direct')
task_queues = [Queue('hipri', task_exchange, routing_key='hipri'),
               Queue('midpri', task_exchange, routing_key='midpri'),
               Queue('lopri', task_exchange, routing_key='lopri')]

queues.py 定义了3个队列，它们绑定到同一个交换机，使用与队列名相同的routing_key。通过不同的队列名区分优先级(hipri高优先级、midpri中优先级、lopri低优先级)

  1. worker.py
python 复制代码 
from __future__ import annotations

import os

os.environ['KOMBU_LOG_CONNECTION'] = 'True'
os.environ['KOMBU_LOG_CHANNEL'] = 'True'

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

from kombu.log import get_logger
from kombu.mixins import ConsumerMixin
from kombu.utils.functional import reprcall

from queues import task_queues

logger = get_logger(__name__)


class Worker(ConsumerMixin):

    def __init__(self, connection):
        self.connection = connection

    def get_consumers(self, Consumer, channel):
        return [Consumer(queues=task_queues,
                         accept=['pickle', 'json'],
                         callbacks=[self.process_task],
                         prefetch_count=1)]

    def process_task(self, body, message):
        fun = body['fun']
        args = body['args']
        kwargs = body['kwargs']
        logger.info('Got task: %s', reprcall(fun.__name__, args, kwargs))
        try:
            fun(*args, **kwargs)
        except Exception as exc:
            logger.error('task raised exception: %r', exc)
        message.ack()


if __name__ == '__main__':
    from kombu import Connection
    from kombu.utils.debug import setup_logging

    # setup root logger
    setup_logging(loglevel=logging.DEBUG)

    with Connection('amqp://guest:guest@localhost:5672//', heartbeat=60) as conn:
        try:
            worker = Worker(conn)
            worker.run()
        except KeyboardInterrupt:
            print('bye bye')

work.py 主要用来执行异步任务，作为RabbitMQ 的消费者

  1. tasks.py
python 复制代码 
from __future__ import annotations

import time


def hello_task(who='world'):
    print(f"execute hello task")
    time.sleep(360)
    print(f'Hello {who}')

tasks.py 声明需要执行的任务的方法，由work.py 消费者调度。在这里引入time模块来模拟执行超长时间任务的情况

  1. client.py
python 复制代码 
from __future__ import annotations

from kombu.pools import producers

from queues import task_exchange

priority_to_routing_key = {
    'high': 'hipri',
    'mid': 'midpri',
    'low': 'lopri',
}


def send_as_task(connection, fun, args=(), kwargs={}, priority='mid'):
    payload = {'fun': fun, 'args': args, 'kwargs': kwargs}
    routing_key = priority_to_routing_key[priority]

    with producers[connection].acquire(block=True) as producer:
        producer.publish(payload,
                         serializer='pickle',
                         compression='bzip2',
                         exchange=task_exchange,
                         declare=[task_exchange],
                         routing_key=routing_key)


if __name__ == '__main__':
    from kombu import Connection

    from tasks import hello_task

    connection = Connection('amqp://guest:guest@localhost:5672//')
    send_as_task(connection, fun=hello_task, args=('Kombu',), kwargs={},
                 priority='high')

client.py 发送消息到RabbitMQ，属于生产者。消息体里面包含方法、参数

问题分析

  1. 执行 worker.py，会发现管理端会产生1个exchange以及3个queue



    运行的日志输出
shell 复制代码 
[Kombu connection:0x1eb707e80d0] establishing connection...
2025-07-19 15:55:05 - kombu.connection - DEBUG - [Kombu connection:0x1eb707e80d0] establishing connection...
2025-07-19 15:55:05 - amqp - DEBUG - Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@Win10-2022QOGYT', 'copyright': 'Copyright (c) 2007-2025 Broadcom Inc and/or its subsidiaries', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 27.3.4.2', 'product': 'RabbitMQ', 'version': '4.1.2'}, mechanisms: [b'ANONYMOUS', b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[Kombu connection:0x1eb707e80d0] connection established: <Connection: amqp://guest:**@127.0.0.1:5672// at 0x1eb707e80d0>
2025-07-19 15:55:05 - kombu.connection - DEBUG - [Kombu connection:0x1eb707e80d0] connection established: <Connection: amqp://guest:**@127.0.0.1:5672// at 0x1eb707e80d0>
2025-07-19 15:55:05 - kombu.mixins - INFO - Connected to amqp://guest:**@127.0.0.1:5672//
[Kombu connection:0x1eb707e80d0] create channel
2025-07-19 15:55:05 - kombu.connection - DEBUG - [Kombu connection:0x1eb707e80d0] create channel
2025-07-19 15:55:05 - amqp - DEBUG - using channel_id: 1
2025-07-19 15:55:05 - amqp - DEBUG - Channel open
[Kombu channel:1] exchange_declare(exchange='kombu_big_tasks', type='direct', durable=True, auto_delete=False, arguments=None, nowait=False, passive=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] exchange_declare(exchange='kombu_big_tasks', type='direct', durable=True, auto_delete=False, arguments=None, nowait=False, passive=False)
[Kombu channel:1] prepare_queue_arguments({}, expires=None, message_ttl=None, max_length=None, max_length_bytes=None, max_priority=None)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] prepare_queue_arguments({}, expires=None, message_ttl=None, max_length=None, max_length_bytes=None, max_priority=None)
[Kombu channel:1] queue_declare(queue='hipri', passive=False, durable=True, exclusive=False, auto_delete=False, arguments={}, nowait=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] queue_declare(queue='hipri', passive=False, durable=True, exclusive=False, auto_delete=False, arguments={}, nowait=False)
[Kombu channel:1] queue_bind(queue='hipri', exchange='kombu_big_tasks', routing_key='hipri', arguments=None, nowait=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] queue_bind(queue='hipri', exchange='kombu_big_tasks', routing_key='hipri', arguments=None, nowait=False)
[Kombu channel:1] exchange_declare(exchange='kombu_big_tasks', type='direct', durable=True, auto_delete=False, arguments=None, nowait=False, passive=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] exchange_declare(exchange='kombu_big_tasks', type='direct', durable=True, auto_delete=False, arguments=None, nowait=False, passive=False)
[Kombu channel:1] prepare_queue_arguments({}, expires=None, message_ttl=None, max_length=None, max_length_bytes=None, max_priority=None)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] prepare_queue_arguments({}, expires=None, message_ttl=None, max_length=None, max_length_bytes=None, max_priority=None)
[Kombu channel:1] queue_declare(queue='midpri', passive=False, durable=True, exclusive=False, auto_delete=False, arguments={}, nowait=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] queue_declare(queue='midpri', passive=False, durable=True, exclusive=False, auto_delete=False, arguments={}, nowait=False)
[Kombu channel:1] queue_bind(queue='midpri', exchange='kombu_big_tasks', routing_key='midpri', arguments=None, nowait=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] queue_bind(queue='midpri', exchange='kombu_big_tasks', routing_key='midpri', arguments=None, nowait=False)
[Kombu channel:1] exchange_declare(exchange='kombu_big_tasks', type='direct', durable=True, auto_delete=False, arguments=None, nowait=False, passive=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] exchange_declare(exchange='kombu_big_tasks', type='direct', durable=True, auto_delete=False, arguments=None, nowait=False, passive=False)
[Kombu channel:1] prepare_queue_arguments({}, expires=None, message_ttl=None, max_length=None, max_length_bytes=None, max_priority=None)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] prepare_queue_arguments({}, expires=None, message_ttl=None, max_length=None, max_length_bytes=None, max_priority=None)
[Kombu channel:1] queue_declare(queue='lopri', passive=False, durable=True, exclusive=False, auto_delete=False, arguments={}, nowait=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] queue_declare(queue='lopri', passive=False, durable=True, exclusive=False, auto_delete=False, arguments={}, nowait=False)
[Kombu channel:1] queue_bind(queue='lopri', exchange='kombu_big_tasks', routing_key='lopri', arguments=None, nowait=False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] queue_bind(queue='lopri', exchange='kombu_big_tasks', routing_key='lopri', arguments=None, nowait=False)
[Kombu channel:1] basic_qos(0, 1, False)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] basic_qos(0, 1, False)
[Kombu channel:1] basic_consume(queue='hipri', no_ack=False, consumer_tag='None1', callback=<bound method Consumer._receive_callback of <Consumer: [<Queue hipri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> hipri bound to chan:1>, <Queue midpri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> midpri bound to chan:1>, <Queue lopri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> lopri bound to chan:1>]>>, nowait=True, arguments=None, on_cancel=None)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] basic_consume(queue='hipri', no_ack=False, consumer_tag='None1', callback=<bound method Consumer._receive_callback of <Consumer: [<Queue hipri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> hipri bound to chan:1>, <Queue midpri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> midpri bound to chan:1>, <Queue lopri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> lopri bound to chan:1>]>>, nowait=True, arguments=None, on_cancel=None)
[Kombu channel:1] basic_consume(queue='midpri', no_ack=False, consumer_tag='None2', callback=<bound method Consumer._receive_callback of <Consumer: [<Queue hipri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> hipri bound to chan:1>, <Queue midpri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> midpri bound to chan:1>, <Queue lopri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> lopri bound to chan:1>]>>, nowait=True, arguments=None, on_cancel=None)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] basic_consume(queue='midpri', no_ack=False, consumer_tag='None2', callback=<bound method Consumer._receive_callback of <Consumer: [<Queue hipri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> hipri bound to chan:1>, <Queue midpri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> midpri bound to chan:1>, <Queue lopri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> lopri bound to chan:1>]>>, nowait=True, arguments=None, on_cancel=None)
[Kombu channel:1] basic_consume(queue='lopri', no_ack=False, consumer_tag='None3', callback=<bound method Consumer._receive_callback of <Consumer: [<Queue hipri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> hipri bound to chan:1>, <Queue midpri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> midpri bound to chan:1>, <Queue lopri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> lopri bound to chan:1>]>>, nowait=False, arguments=None, on_cancel=None)
2025-07-19 15:55:05 - kombu.channel - DEBUG - [Kombu channel:1] basic_consume(queue='lopri', no_ack=False, consumer_tag='None3', callback=<bound method Consumer._receive_callback of <Consumer: [<Queue hipri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> hipri bound to chan:1>, <Queue midpri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> midpri bound to chan:1>, <Queue lopri -> <Exchange kombu_big_tasks(direct) bound to chan:1> -> lopri bound to chan:1>]>>, nowait=False, arguments=None, on_cancel=None)
2025-07-19 15:55:06 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:06 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : Prev sent/recv: None/None, now - 17/15, monotonic - 1966022.093, last_heartbeat_sent - 1966022.093, heartbeat int. - 60 for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:07 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:07 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : Prev sent/recv: 17/15, now - 17/15, monotonic - 1966023.093, last_heartbeat_sent - 1966022.093, heartbeat int. - 60 for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:08 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:08 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : Prev sent/recv: 17/15, now - 17/15, monotonic - 1966024.093, last_heartbeat_sent - 1966022.093, heartbeat int. - 60 for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:09 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:09 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : Prev sent/recv: 17/15, now - 17/15, monotonic - 1966025.093, last_heartbeat_sent - 1966022.093, heartbeat int. - 60 for connection e2f2a870a0e6451187c0fc3aef3d55bf
2025-07-19 15:55:10 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection e2f2a870a0e6451187c0fc3aef3d55bf

worker.py 开启了 debug级别的日志，方便排查问题。在没有执行超长时间任务之前，消费者发送心跳到RabbitMQ服务端，服务端感知消费者存在不会断开消费者的连接。

RabbitMQ 服务端的日志如下

shell 复制代码 
2025-07-19 15:55:05.840000+08:00 [info] <0.1487.0> accepting AMQP connection 127.0.0.1:59923 -> 127.0.0.1:5672
2025-07-19 15:55:05.844000+08:00 [info] <0.1487.0> connection 127.0.0.1:59923 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'

本地是window系统启动的RabbitMQ，服务端的默认日志路径为

C:\Users%USERNAME%\AppData\Roaming\RabbitMQ\log

  1. 执行 client.py，消息发送给服务端，worker.py 接收到消息之后，开始执行任务，任务开始执行后，心跳的运行日志没了，心跳发送被超长时间任务阻塞了，如下图所示

管理端这边也可以查询到有一个任务正在执行没有应答

等待一段时间之后，服务端没有接收到心跳，主动断开连接，服务端日志如下

shell 复制代码 
2025-07-19 15:55:05.840000+08:00 [info] <0.1487.0> accepting AMQP connection 127.0.0.1:59923 -> 127.0.0.1:5672
2025-07-19 15:55:05.844000+08:00 [info] <0.1487.0> connection 127.0.0.1:59923 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:09:23.336000+08:00 [info] <0.1982.0> accepting AMQP connection 127.0.0.1:60967 -> 127.0.0.1:5672
2025-07-19 16:09:23.341000+08:00 [info] <0.1982.0> connection 127.0.0.1:60967 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:09:23.350000+08:00 [warning] <0.1982.0> closing AMQP connection <0.1982.0> (127.0.0.1:60967 -> 127.0.0.1:5672, vhost: '/', user: 'guest', duration: '14ms'):
2025-07-19 16:09:23.350000+08:00 [warning] <0.1982.0> client unexpectedly closed TCP connection
2025-07-19 16:12:05.968000+08:00 [error] <0.1487.0> closing AMQP connection <0.1487.0> (127.0.0.1:59923 -> 127.0.0.1:5672, duration: '17M, 0s'):
2025-07-19 16:12:05.968000+08:00 [error] <0.1487.0> missed heartbeats from client, timeout: 60s

服务端日志分析如下：

2025-07-19 15:55:05.840000 worker.py 运行，消费者连接上RabbitMQ，日志标识 <0.1487.0> 通道标识 59923

2025-07-19 16:09:23.336000 client.py 运行，生产者连接上RabbitMQ，日志标识 <0.1982.0> 通道标识60967， 投递消息任务到RabbitMQ

2025-07-19 16:09:23.341000 消息发送成功后，生产者断开连接 client unexpectedly closed TCP connection 日志标识 <0.1982.0>

2025-07-19 16:12:05.968000 服务端没有收到消费者的心跳 <0.1487.0> missed heartbeats from client, timeout: 60s， 断开连接 closing AMQP connection <0.1487.0>

服务端断开连接之后，消费者还在执行任务，任务执行完毕之后，进行应答，由于连接已经断开，worker.py(消费者)会产生错误，之后就会重连，日志如下

shell 复制代码 
execute hello task
Hello Kombu
[Kombu channel:1] close()
2025-07-19 16:27:23 - kombu.channel - DEBUG - [Kombu channel:1] close()
2025-07-19 16:27:23 - amqp - DEBUG - Closed channel #1
[Kombu connection:0x1eb7089e950] closed
2025-07-19 16:27:23 - kombu.connection - DEBUG - [Kombu connection:0x1eb7089e950] closed
2025-07-19 16:27:23 - kombu.mixins - WARNING - Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\mixins.py", line 174, in run
    for _ in self.consume(limit=None, **kwargs):
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\mixins.py", line 190, in consume
    with self.consumer_context(**kwargs) as (conn, channel, consumers):
  File "E:\Anaconda\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\mixins.py", line 183, in consumer_context
    with self.Consumer() as (connection, channel, consumers):
  File "E:\Anaconda\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\mixins.py", line 232, in Consumer
    with self._consume_from(*self.get_consumers(cls, channel)) as c:
  File "E:\Anaconda\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\utils\compat.py", line 135, in nested
    reraise(exc[0], exc[1], exc[2])
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\exceptions.py", line 35, in reraise
    raise value
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\utils\compat.py", line 118, in nested
    vars.append(enter())
                ^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\messaging.py", line 464, in __enter__
    self.consume()
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\messaging.py", line 514, in consume
    self._basic_consume(T, no_ack=no_ack, nowait=False)
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\messaging.py", line 641, in _basic_consume
    queue.consume(tag, self._receive_callback,
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\entity.py", line 750, in consume
    return self.channel.basic_consume(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\utils\debug.py", line 69, in __wrapped
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\channel.py", line 1574, in basic_consume
    p = self.send_method(
        ^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\abstract_channel.py", line 79, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\abstract_channel.py", line 99, in wait
    self.connection.drain_events(timeout=timeout)
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\connection.py", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\method_framing.py", line 77, in on_frame
    callback(channel, msg.frame_method, msg.frame_args, msg)
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\connection.py", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\abstract_channel.py", line 156, in dispatch_method
    listener(*args)
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\channel.py", line 1629, in _on_basic_deliver
    fun(msg)
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\messaging.py", line 668, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\messaging.py", line 634, in receive
    [callback(body, message) for callback in callbacks]
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\messaging.py", line 634, in <listcomp>
    [callback(body, message) for callback in callbacks]
     ^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\worker.py", line 45, in process_task
    message.ack()
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\kombu\message.py", line 126, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\channel.py", line 1407, in basic_ack
    return self.send_method(
           ^^^^^^^^^^^^^^^^^
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\abstract_channel.py", line 70, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\method_framing.py", line 186, in write_frame
    write(buffer_store.view[:offset])
  File "F:\Github\python\kombu_worker\.venv\Lib\site-packages\amqp\transport.py", line 350, in write
    self._write(s)
ConnectionResetError: [WinError 10054] 远程主机强迫关闭了一个现有的连接。
[Kombu connection:0x1eb708c1c50] establishing connection...
2025-07-19 16:27:23 - kombu.connection - DEBUG - [Kombu connection:0x1eb708c1c50] establishing connection...

这个超长时间的任务会循环的重复运行。消费者运行超长时间任务，阻塞心跳 -> 服务端断开连接，任务重新回到队列 -> 消费者应答失败，重新连接服务端 -> 消费者运行超长时间任务，阻塞心跳，如图所示

服务端的日志表现如下

shell 复制代码 
2025-07-19 15:55:05.840000+08:00 [info] <0.1487.0> accepting AMQP connection 127.0.0.1:59923 -> 127.0.0.1:5672
2025-07-19 15:55:05.844000+08:00 [info] <0.1487.0> connection 127.0.0.1:59923 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:09:23.336000+08:00 [info] <0.1982.0> accepting AMQP connection 127.0.0.1:60967 -> 127.0.0.1:5672
2025-07-19 16:09:23.341000+08:00 [info] <0.1982.0> connection 127.0.0.1:60967 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:09:23.350000+08:00 [warning] <0.1982.0> closing AMQP connection <0.1982.0> (127.0.0.1:60967 -> 127.0.0.1:5672, vhost: '/', user: 'guest', duration: '14ms'):
2025-07-19 16:09:23.350000+08:00 [warning] <0.1982.0> client unexpectedly closed TCP connection
2025-07-19 16:12:05.968000+08:00 [error] <0.1487.0> closing AMQP connection <0.1487.0> (127.0.0.1:59923 -> 127.0.0.1:5672, duration: '17M, 0s'):
2025-07-19 16:12:05.968000+08:00 [error] <0.1487.0> missed heartbeats from client, timeout: 60s
2025-07-19 16:15:23.418000+08:00 [info] <0.2184.0> accepting AMQP connection 127.0.0.1:61211 -> 127.0.0.1:5672
2025-07-19 16:15:23.425000+08:00 [info] <0.2184.0> connection 127.0.0.1:61211 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:18:23.455000+08:00 [error] <0.2184.0> closing AMQP connection <0.2184.0> (127.0.0.1:61211 -> 127.0.0.1:5672, duration: '3M, 0s'):
2025-07-19 16:18:23.455000+08:00 [error] <0.2184.0> missed heartbeats from client, timeout: 60s
2025-07-19 16:21:23.448000+08:00 [info] <0.2248.0> accepting AMQP connection 127.0.0.1:61377 -> 127.0.0.1:5672
2025-07-19 16:21:23.453000+08:00 [info] <0.2248.0> connection 127.0.0.1:61377 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:24:23.484000+08:00 [error] <0.2248.0> closing AMQP connection <0.2248.0> (127.0.0.1:61377 -> 127.0.0.1:5672, duration: '3M, 0s'):
2025-07-19 16:24:23.484000+08:00 [error] <0.2248.0> missed heartbeats from client, timeout: 60s
2025-07-19 16:27:23.474000+08:00 [info] <0.2312.0> accepting AMQP connection 127.0.0.1:61524 -> 127.0.0.1:5672
2025-07-19 16:27:23.481000+08:00 [info] <0.2312.0> connection 127.0.0.1:61524 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:30:23.502000+08:00 [error] <0.2312.0> closing AMQP connection <0.2312.0> (127.0.0.1:61524 -> 127.0.0.1:5672, duration: '3M, 0s'):
2025-07-19 16:30:23.502000+08:00 [error] <0.2312.0> missed heartbeats from client, timeout: 60s
2025-07-19 16:33:23.507000+08:00 [info] <0.2489.0> accepting AMQP connection 127.0.0.1:61749 -> 127.0.0.1:5672
2025-07-19 16:33:23.512000+08:00 [info] <0.2489.0> connection 127.0.0.1:61749 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'
2025-07-19 16:36:23.543000+08:00 [error] <0.2489.0> closing AMQP connection <0.2489.0> (127.0.0.1:61749 -> 127.0.0.1:5672, duration: '3M, 0s'):
2025-07-19 16:36:23.543000+08:00 [error] <0.2489.0> missed heartbeats from client, timeout: 60s
2025-07-19 16:39:23.533000+08:00 [info] <0.2732.0> accepting AMQP connection 127.0.0.1:62055 -> 127.0.0.1:5672
2025-07-19 16:39:23.538000+08:00 [info] <0.2732.0> connection 127.0.0.1:62055 -> 127.0.0.1:5672: user 'guest' authenticated and granted access to vhost '/'

管理端可以查看到消费者断开的间隙，消费端正在运行超长时间任务，阻塞了心跳

解决方法

使用异步线程运行超长时间任务 防止心跳阻塞

修该 worker.py，增加线程池，异步执行任务

python 复制代码 
from __future__ import annotations

import os

os.environ['KOMBU_LOG_CONNECTION'] = 'True'
os.environ['KOMBU_LOG_CHANNEL'] = 'True'

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

from kombu.log import get_logger
from kombu.mixins import ConsumerMixin
from kombu.utils.functional import reprcall
from queues import task_queues
from concurrent.futures import ThreadPoolExecutor

logger = get_logger(__name__)


class Worker(ConsumerMixin):

    def __init__(self, connection):
        self.connection = connection
        self.thread_pool = ThreadPoolExecutor(max_workers=4, thread_name_prefix='queue_worker')

    def get_consumers(self, Consumer, channel):
        return [Consumer(queues=task_queues,
                         accept=['pickle', 'json'],
                         callbacks=[self.on_message],
                         prefetch_count=1)]

    def on_message(self, body, message):
        self.thread_pool.submit(self.process_task, body, message)

    def process_task(self, body, message):
        fun = body['fun']
        args = body['args']
        kwargs = body['kwargs']
        logger.info('Got task: %s', reprcall(fun.__name__, args, kwargs))
        try:
            fun(*args, **kwargs)
        except Exception as exc:
            logger.error('task raised exception: %r', exc)
        message.ack()


if __name__ == '__main__':
    from kombu import Connection
    from kombu.utils.debug import setup_logging

    # setup root logger
    setup_logging(loglevel=logging.DEBUG)

    with Connection('amqp://guest:guest@localhost:5672//', heartbeat=60) as conn:
        try:
            worker = Worker(conn)
            worker.run()
        except KeyboardInterrupt:
            print('bye bye')

在运行任务的时候，不会阻塞心跳了，worker.py日志如下

shell 复制代码 
2025-07-19 16:56:19 - kombu.channel - DEBUG - [Kombu channel:1] message_to_python(<amqp.basic_message.Message object at 0x000001B436F16C30>)
2025-07-19 16:56:19 - __main__ - INFO - Got task: hello_task('Kombu')
execute hello task
2025-07-19 16:56:20 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection 7fb334777d214ef1ac01025683428e30
2025-07-19 16:56:20 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : Prev sent/recv: None/None, now - 17/18, monotonic - 1969696.078, last_heartbeat_sent - 1969696.078, heartbeat int. - 60 for connection 7fb334777d214ef1ac01025683428e30
2025-07-19 16:56:21 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection 7fb334777d214ef1ac01025683428e30
2025-07-19 16:56:21 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : Prev sent/recv: 17/18, now - 17/18, monotonic - 1969697.078, last_heartbeat_sent - 1969696.078, heartbeat int. - 60 for connection 7fb334777d214ef1ac01025683428e30
2025-07-19 16:56:22 - amqp.connection.Connection.heartbeat_tick - DEBUG - heartbeat_tick : for connection 7fb334777d214ef1ac01025683428e30

可以看到管理端的任务运行完了

相关推荐
利刃大大9 分钟前
【RabbitMQ】Simple模式 && 工作队列 && 发布/订阅模式 && 路由模式 && 通配符模式 && RPC模式 && 发布确认机制
rpc·消息队列·rabbitmq·队列
J_liaty19 小时前
RabbitMQ面试题终极指南
开发语言·后端·面试·rabbitmq
maozexijr1 天前
RabbitMQ Exchange Headers类型存在的意义？
分布式·rabbitmq
独自破碎E1 天前
RabbitMQ的消息确认机制是怎么工作的？
分布式·rabbitmq
maozexijr1 天前
注解实现rabbitmq消费者和生产者
分布式·rabbitmq
Java 码农2 天前
RabbitMQ集群部署方案及配置指南09
分布式·rabbitmq
论迹2 天前
RabbitMQ
分布式·rabbitmq
Java 码农2 天前
RabbitMQ集群部署方案及配置指南08--电商业务延迟队列定制化方案
大数据·分布式·rabbitmq
Java 码农2 天前
Spring Boot集成RabbitMQ的各种队列使用案例
spring boot·rabbitmq·java-rabbitmq
vb2008112 天前
Ubuntu 系统下 RabbitMQ 作为 MQTT 代理的配置方案
mqtt·rabbitmq
热门推荐
01GitHub 镜像站点02Labelme从安装到标注：零基础完整指南03安娜的档案(Anna’s Archive) 镜像网站/国内最新可访问入口（持续更新）04Linux下V2Ray安装配置指南05jdk21下载、安装（Windows、Linux、macOS）06KGG转MP3工具|非KGM文件|解密音频07Claude Code 2.1.2 升级报错？别折腾了，一行命令搞定08【踩坑笔记】50系显卡适配的 PyTorch 安装09Opencode CLI 安装成功,但是启动失败102025-04-03 Latex学习1——本地配置Latex + VScode环境