Celery task 执行报错 TypeError: Object of type set is not JSON serializable 问题分析处理

情况描述:

定义了新的shared_task推送到生产环境后,发现无法执行,会报错set对象无法序列化,报错内容如下:

python 复制代码
Traceback (most recent call last):
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/__init__.py", line 412, in dumps
    **kw).encode(obj)
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type set is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/venv/lib64/python3.6/site-packages/celery/app/trace.py", line 449, in trace_task
    uuid, retval, task_request, publish_result,
  File "/tmp/venv/lib64/python3.6/site-packages/celery/backends/base.py", line 149, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/tmp/venv/lib64/python3.6/site-packages/celery/backends/base.py", line 342, in store_result
    request=request, **kwargs)
  File "/tmp/venv/lib64/python3.6/site-packages/django_celery_results/backends/database.py", line 19, in _store_result
    content_type, content_encoding, result = self.encode_content(result)
  File "/tmp/venv/lib64/python3.6/site-packages/django_celery_results/backends/database.py", line 52, in encode_content
    content_type, content_encoding, content = self._encode(data)
  File "/tmp/venv/lib64/python3.6/site-packages/celery/backends/base.py", line 297, in _encode
    return dumps(data, serializer=self.serializer)
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/tmp/venv/lib64/python3.6/site-packages/vine/five.py", line 194, in reraise
    raise value.with_traceback(tb)
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/__init__.py", line 412, in dumps
    **kw).encode(obj)
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
kombu.exceptions.EncodeError: Object of type set is not JSON serializable

初步分析

复制代码
a.考虑是task调用过程中,参数传输过程异常,导致参数无法序列化,因为set 就是{1,2} 这种格式的,命令行验证没问题
b.考虑是其他模块有异常,发现其他模块的定时任务能正常执行
c.在测试环境能完整复现,但是由于celery 的调用无法调试,无法得到有效的反馈
d.命令行直接执行task函数不会报错,所以函数逻辑应该没有问题
e.最终发现是结果集里面存在set集合,导致后续结果传输的过程中无法解析成json,至此问题确定

解决办法:

复制代码
将set转换成其他数据结构

总结:

复制代码
a.对于原先测试验证能通过,修改后无法验证通过的,要先找到变更的内容,修改后要验证
b.这次花费时间较长是因为认为原先验证过,认为后续小的修改没有影响,导致思路不对,修改代码要验证,不要过于自信
c.对于celery这种无法调试的调用过程,复现存在难度,那就需要从函数逻辑,返回结果里面找答案
d.可能会转换成json格式的数据,一定不能包含set集合
e.其他:@shared_task 注解的函数,会加载到内存里面,需要重启Celery才会重新加载
相关推荐
智搜广告33 分钟前
智搜广告:科技行业AI回答优化公司如何破局
大数据·python·elasticsearch·geo
IpdataCloud1 小时前
AI智能体调用工具怎么核验来源IP?归属地、网络类型与代理风险识别(含Python代码)
数据库·python·tcp/ip
2601_966949651 小时前
只拿到股票代码还不够:量化程序到底还需要哪些标的信息?——从数据建模开始理解股票元数据
开发语言·python·数据分析·pandas·量化交易·股票数据·quantdash
蒸鱼Yuzheng2 小时前
Python 游戏测试开发怎么准备:日志解析、接口校验、并发与可维护性
自动化测试·python·测试开发·面试题·游戏测试
Ivanqhz2 小时前
BM1688 双核架构
python·深度学习·神经网络·cnn·mlir
Java后端的Ai之路2 小时前
Python进阶探索17 - Python中的深拷贝与浅拷贝
人工智能·python·ai·浅拷贝·深拷贝
Asurplus3 小时前
【SpringBoot4】3、从SpringBoot4升级Sa-Token1.46.0报错
redis·sa-token·json·jackson·springboot4
西柚研究生1234563 小时前
论文分析17:YOLOv11_UAVNet:无人机航拍图像专用目标检测算法
人工智能·python·深度学习·算法·目标检测
沧海一笑-dj4 小时前
【Python】Python学习笔记-初识 Python
python·python安装·python解释器
打工仔折腾 AI4 小时前
FastAPI 从本机到生产服务器:Nginx+Gunicorn+Uvicorn 完整部署实录
人工智能·后端·python·nginx·fastapi·gunicorn