Python pika消费Rabbit MQ数据，慢消费引起的connection reset问题

曾彪彪2025-02-06 13:50

问题描述

使用python pika框架，从Rabbit MQ消费数据时，遇到了connection reset的错误，错误内容如下：

python 复制代码 
Traceback (most recent call last):
  File "/app/utils/rabbit.py", line 27, in message_callback
    channel.basic_ack(delivery_tag=method.delivery_tag)
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 2130, in basic_ack
    self._flush_output()
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 1353, in _flush_output
    self._connection._flush_output(lambda: self.is_closed, *waiters)
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 523, in _flush_output
    raise self._closed_result.value.error
pika.exceptions.StreamLostError: Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/app/app.py", line 14, in main
    rabbit.start_listen()
  File "/app/utils/rabbit.py", line 52, in start_listen
    channel.start_consuming()
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 1883, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 2044, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 851, in process_data_events
    self._dispatch_channel_events()
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 567, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 1510, in _dispatch_events
    consumer_info.on_message_callback(self, evt.method,
  File "/app/utils/rabbit.py", line 34, in message_callback
    channel.basic_nack(delivery_tag=method.delivery_tag)
  File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 2151, in basic_nack
    self._impl.basic_nack(
  File "/usr/local/lib/python3.11/site-packages/pika/channel.py", line 401, in basic_nack
    self._raise_if_not_open()
  File "/usr/local/lib/python3.11/site-packages/pika/channel.py", line 1403, in _raise_if_not_open
    raise exceptions.ChannelWrongStateError('Channel is closed.')
pika.exceptions.ChannelWrongStateError: Channel is closed.

我的代码如下：

python 复制代码 
import pika
from utils.logger import logger
import traceback
import json
from utils.config import get_rabbit_config
from utils.mail import send_mail
import time
EXCHANGE = 'DEFAULT_EXCHANGE'
ROUTING_KEY = 'purchasing.system.contract'
QUEUE_NAME = 'PURCHASING_CONTRACT_CONSUMER'
class Rabbit():
    def __init__(self, callback=None) -> None:
        self.callback = callback
    def message_callback(self, channel, method, properties, body):
        try:
            logger.info(f'receive message: {body}')
            message = json.loads(body)
            if self.callback:
                result = self.callback(message)
                # time.sleep(5)
                # result = False
            else:
                logger.warn('self.callback is None')
            if result:
                channel.basic_ack(delivery_tag=method.delivery_tag)
            else:
                channel.basic_nack(delivery_tag=method.delivery_tag)
        except Exception as e:
            logger.error(traceback.format_exc())
            send_mail('purchasing-contract-consumer error',
                      traceback.format_exc())
            channel.basic_nack(delivery_tag=method.delivery_tag)
            time.sleep(3)
    def start_listen(self):
        config = get_rabbit_config()
        credentials = pika.PlainCredentials(
            username=config['username'], password=config['password'])
        parameters = pika.ConnectionParameters(
            # host=config['host'], port=config['port'], credentials=credentials, heartbeat=300, stack_timeout=300, socket_timeout=300, blocked_connection_timeout=300)
            host=config['host'], port=config['port'], credentials=credentials, heartbeat=120, socket_timeout=60)
        connection = pika.BlockingConnection(parameters=parameters)
        channel = connection.channel()
        # channel.exchange_declare(exchange=EXCHANGE,exchange_type='topic',durable=True)
        channel.queue_declare(queue=QUEUE_NAME, durable=True)
        channel.queue_bind(queue=QUEUE_NAME, exchange=EXCHANGE,
                           routing_key=ROUTING_KEY)
        channel.basic_consume(
            queue=QUEUE_NAME, on_message_callback=self.message_callback, auto_ack=False)
        logger.info('start listening...')
        channel.start_consuming()
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')

我的应用场景是：从rabbit 消费数据，并通过restful API调用web服务(self.callback是一个web API调用)，上报消费的数据。但是web服务响应非常慢，每次调用花费1min多钟，并且rabbit MQ中积累了大量消息。

原因分析

刚开始以为是rabbit MQ connection的超时问题引起的，于是设置连接超时时间为300s，代码如下：

host=config['host'], port=config['port'], credentials=credentials, heartbeat=300, stack_timeout=300, socket_timeout=300, blocked_connection_timeout=300)

connection reset问题依然存在，于是又怀疑是result = self.callback(message)调用时间过长，超过了1min，于是将其注释掉，改成

python 复制代码 
result = self.callback(message)
time.sleep(5)
result = False

错误依然存在，也排除了不是self.callback(message)长时间阻塞引起的问题。

查阅相关资料，发现是因为未设置prefetch_count，如果不设置这个参数，框架默认设置为0，意味着无限消费数据，此时，如果消费端处理消息的速度非常慢，并且Rabbit MQ中有大量消息堆积，那么socket的缓存区就会塞满，此时客户端socket就会告诉服务端socket，将滑动窗口设置为0。由于客户端socket的缓存区一直被沾满，服务端长时间无法发送数据，甚至连socket的心跳消息也无法发出，就会导致connection reset异常；之后如果客户端调用restful API完成数据上传，尝试调用channel.basic_ack(delivery_tag=method.delivery_tag)，此时，连接已经被重置，那么就抛出connection reset异常。

解决方案

在代码中增加prefetch_count，代码如下：

python 复制代码 
import pika
from utils.logger import logger
import traceback
import json
from utils.config import get_rabbit_config
from utils.mail import send_mail
import time
EXCHANGE = 'DEFAULT_EXCHANGE'
ROUTING_KEY = 'purchasing.system.contract'
QUEUE_NAME = 'PURCHASING_CONTRACT_CONSUMER'
class Rabbit():
    def __init__(self, callback=None) -> None:
        self.callback = callback
    def message_callback(self, channel, method, properties, body):
        try:
            logger.info(f'receive message: {body}')
            message = json.loads(body)
            if self.callback:
                result = self.callback(message)
                # time.sleep(5)
                # result = False
            else:
                logger.warn('self.callback is None')
            if result:
                channel.basic_ack(delivery_tag=method.delivery_tag)
            else:
                channel.basic_nack(delivery_tag=method.delivery_tag)
        except Exception as e:
            logger.error(traceback.format_exc())
            send_mail('purchasing-contract-consumer error',
                      traceback.format_exc())
            channel.basic_nack(delivery_tag=method.delivery_tag)
            time.sleep(3)
    def start_listen(self):
        config = get_rabbit_config()
        credentials = pika.PlainCredentials(
            username=config['username'], password=config['password'])
        parameters = pika.ConnectionParameters(
            # host=config['host'], port=config['port'], credentials=credentials, heartbeat=300, stack_timeout=300, socket_timeout=300, blocked_connection_timeout=300)
            host=config['host'], port=config['port'], credentials=credentials, heartbeat=120, socket_timeout=60)
        connection = pika.BlockingConnection(parameters=parameters)
        channel = connection.channel()
        channel.basic_qos(prefetch_count=1) # 限制消费端消费的数据，防止缓存区被占满。
        # channel.exchange_declare(exchange=EXCHANGE,exchange_type='topic',durable=True)
        channel.queue_declare(queue=QUEUE_NAME, durable=True)
        channel.queue_bind(queue=QUEUE_NAME, exchange=EXCHANGE,
                           routing_key=ROUTING_KEY)
        channel.basic_consume(
            queue=QUEUE_NAME, on_message_callback=self.message_callback, auto_ack=False)
        logger.info('start listening...')
        channel.start_consuming()
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')
        logger.warning('warning...')

设置完成后，可以正常消费数据，connection reset的异常消息。

所有文章在Github上同步，你也可以访问曾彪彪的个人博客点击查看

相关推荐
Eric.5651 小时前
python advance -----object-oriented
python
云天徽上1 小时前
【数据可视化-107】2025年1-7月全国出口总额Top 10省市数据分析：用Python和Pyecharts打造炫酷可视化大屏
开发语言·python·信息可视化·数据挖掘·数据分析·pyecharts
THMAIL2 小时前
机器学习从入门到精通 - 数据预处理实战秘籍：清洗、转换与特征工程入门
人工智能·python·算法·机器学习·数据挖掘·逻辑回归
@HNUSTer2 小时前
Python数据可视化科技图表绘制系列教程（六）
python·数据可视化·科技论文·专业制图·科研图表
THMAIL2 小时前
深度学习从入门到精通 - AutoML与神经网络搜索(NAS)：自动化模型设计未来
人工智能·python·深度学习·神经网络·算法·机器学习·逻辑回归
山烛3 小时前
深度学习：残差网络ResNet与迁移学习
人工智能·python·深度学习·残差网络·resnet·迁移学习
eleqi3 小时前
Python+DRVT 从外部调用 Revit：批量创建梁(2)
python·系统集成·revit·自动化生产流水线·外部访问
BYSJMG3 小时前
计算机毕设大数据方向：基于Spark+Hadoop的餐饮外卖平台数据分析系统【源码+文档+调试】
大数据·hadoop·分布式·python·spark·django·课程设计
OC溥哥9994 小时前
Flask论坛与个人中心页面开发教程完整详细版
后端·python·flask·html
Rhys..4 小时前
python + Flask模块学习 2 接收用户请求并返回json数据
python·学习·flask
热门推荐
012025 年高教社杯全国大学生数学建模竞赛C 题 NIPT 的时点选择与胎儿的异常判定 完整成品思路模型代码分享，全网首发高质量！！！022025年数学建模国赛C题超详细解题思路032025全国大学生数学建模C题保姆级思路模型（持续更新）：NIPT 的时点选择与胎儿的异常判定042025高教社杯国赛数学建模选题建议+初步分析052025国赛B题保姆级教程思路分析 碳化硅外延层厚度的确定06KGG转MP3工具|非KGM文件|解密音频07UV安装并设置国内源08不再让Windows更新！&Edge游戏助手卸载及关闭自动更新09奈飞工厂官网，国内Netflix影视在线看|中文网页电脑版入口10（E题｜AI 辅助智能体测）2025年高教杯全国大学生数学建模国赛解题思路|完整代码论文集合