milvus插入数据时,明明不超长,但总是报长度错误?

在处理插入milvus数据时,设置了字段长度为512. 明明考虑了预留,插入的数据中没有这么长的,但还是会有报错 类似:MilvusException: (code=0, message=the length (564) of 78th string exceeds max length (512)

查找max(len(x) for x in temp_list)之类 都没有超过512过,也没超过256过,不知道哪里的数据有问题..

反复截段文本等测试后发现,例如用len(x)看到的字符串长度是10,但保存进milus的长度,并不是..

举例,把数据库长度设为一个小值16:

FieldSchema(name="question", dtype=DataType.VARCHAR, auto_id=False, max_length=16)

再把数据缩到只有一行 测试结果插入成功:

line contents is : 你好呀你好 and length is 5

Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 00:01\<00:00, 1.02s/it

index handle result: Status(code=0, message=)

insert result: (insert count: 1, delete count: 0, upsert count: 0, timestamp: 449735609509740549, success count: 1, err count: 0)

再增加一点文字长度 就报错了:

line contents is : 你好呀你好呀 and length is 6

Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 00:00\<00:00, 1.03it/s

index handle result: Status(code=0, message=)

2024-05-13 20:59:27,915 decorators.py:134 ERROR RPC error: batch_insert, <MilvusException: (code=0, message=the length (18) of 0th string exceeds max length (16))>, <Time:{'RPC start': '2024-05-13 20:59:27.912751', 'RPC error': '2024-05-13 20:59:27.915058'}>

Traceback (most recent call last):

File "/root/temp_dir/run_task.py", line 55, in <module>

XXX().create_insert_vector_db()

File "/root/temp_dir/app/service/vector_db/xx_pre_handle.py", line 63, in create_insert_vector_db

).get_or_create_db(fields, description, "possible_question_embeddings", entities)

File "/root/temp_dir/app/service/vector_db/milvus_db.py", line 23, in get_or_create_db

return self.create_and_insert(fields, description, index_field_name, entities)

File "/root/temp_dir/app/service/vector_db/milvus_db.py", line 28, in create_and_insert

self.insert_db(entities)

File "/root/temp_dir/app/service/vector_db/milvus_db.py", line 40, in insert_db

insert_result = self.collection.insert(entities)

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 497, in insert

res = conn.batch_insert(

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 135, in handler

raise e from e

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 131, in handler

return func(*args, **kwargs)

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 170, in handler

return func(self, *args, **kwargs)

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 110, in handler

raise e from e

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 74, in handler

return func(*args, **kwargs)

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 566, in batch_insert

raise err from err

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 560, in batch_insert

check_status(response.status)

File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/utils.py", line 54, in check_status

raise MilvusException(status.code, status.reason, status.error_code)

pymilvus.exceptions.MilvusException: <MilvusException: (code=0, message=the length (18) of 0th string exceeds max length (16))>

所以,可能是因为UTF-8或其他编码的原因,一些非ASCII字符可能被编码成多个字节 以保存进milvus。

所以,解决方案是 建表时FieldSchema中把max_length 设置为4倍或其他倍数于预期的最大长度。

相关推荐
qq_158895411113 小时前
Suno-API 是什么?官方 V6 模型与 Python / TypeScript 接入指南
python·ai音乐·suno api
学无止境小袁同学14 小时前
Google ADK 学习指南
python·llm·agent·多agent·google adk·llmagent·agent cli
郝学胜-神的一滴14 小时前
Effective Python 条款35:用类封装替代生成器异常注入
开发语言·人工智能·python·金融·pandas·scikit-learn·matplotlib
LK_0715 小时前
【力扣】142. 环形链表 II |Python
python·leetcode·链表
wuyk55515 小时前
Python办公自动化-第02讲-Excel文件认知与四大库选型
开发语言·python·excel
用户548178459095016 小时前
Java 与 Python 在大模型与智能体领域的选型
java·python
yiangL_16 小时前
1、Python 输入输出实验|print 与 input 人机交互实践
开发语言·python·人机交互
高洁0117 小时前
世界模型的因果内核:从预测表象到理解机制
人工智能·python·深度学习·transformer·迁移学习
数字融合17 小时前
透明化视频动态三维重建空间数据零核心技术
人工智能·python·virtualenv