milvus插入数据时,明明不超长,但总是报长度错误?

在处理插入milvus数据时,设置了字段长度为512. 明明考虑了预留,插入的数据中没有这么长的,但还是会有报错 类似:MilvusException: (code=0, message=the length (564) of 78th string exceeds max length (512)

查找max(len(x) for x in temp_list)之类 都没有超过512过,也没超过256过,不知道哪里的数据有问题..

反复截段文本等测试后发现,例如用len(x)看到的字符串长度是10,但保存进milus的长度,并不是..

举例,把数据库长度设为一个小值16:

FieldSchema(name="question", dtype=DataType.VARCHAR, auto_id=False, max_length=16)

再把数据缩到只有一行 测试结果插入成功:

line contents is : 你好呀你好 and length is 5

Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.02s/it]

index handle result: Status(code=0, message=)

insert result: (insert count: 1, delete count: 0, upsert count: 0, timestamp: 449735609509740549, success count: 1, err count: 0)

再增加一点文字长度 就报错了:

line contents is : 你好呀你好呀 and length is 6

Batches: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1.03it/s]

index handle result: Status(code=0, message=)

2024-05-13 20:59:27,915 decorators.py:134 ERROR\] RPC error: \[batch_insert\], \, \ Traceback (most recent call last): File "/root/temp_dir/run_task.py", line 55, in \ XXX().create_insert_vector_db() File "/root/temp_dir/app/service/vector_db/xx_pre_handle.py", line 63, in create_insert_vector_db ).get_or_create_db(fields, description, "possible_question_embeddings", entities) File "/root/temp_dir/app/service/vector_db/milvus_db.py", line 23, in get_or_create_db return self.create_and_insert(fields, description, index_field_name, entities) File "/root/temp_dir/app/service/vector_db/milvus_db.py", line 28, in create_and_insert self.insert_db(entities) File "/root/temp_dir/app/service/vector_db/milvus_db.py", line 40, in insert_db insert_result = self.collection.insert(entities) File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 497, in insert res = conn.batch_insert( File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 135, in handler raise e from e File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 131, in handler return func(\*args, \*\*kwargs) File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 170, in handler return func(self, \*args, \*\*kwargs) File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 110, in handler raise e from e File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/decorators.py", line 74, in handler return func(\*args, \*\*kwargs) File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 566, in batch_insert raise err from err File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 560, in batch_insert check_status(response.status) File "/root/tmp/venv_dir/1_text_simi/lib/python3.10/site-packages/pymilvus/client/utils.py", line 54, in check_status raise MilvusException(status.code, status.reason, status.error_code) pymilvus.exceptions.MilvusException: \ 所以,可能是因为UTF-8或其他编码的原因,一些非ASCII字符可能被编码成多个字节 以保存进milvus。 所以,解决方案是 建表时FieldSchema中把max_length 设置为4倍或其他倍数于预期的最大长度。

相关推荐
摸爬滚打李上进1 小时前
重生学AI第十六集:线性层nn.Linear
人工智能·pytorch·python·神经网络·机器学习
背太阳的牧羊人1 小时前
Neo4j 的向量搜索(Neo4jVector)和常见的向量数据库(比如 Milvus、Qdrant)之间的区别与联系
数据库·neo4j·milvus
凛铄linshuo2 小时前
爬虫简单实操2——以贴吧为例爬取“某吧”前10页的网页代码
爬虫·python·学习
牛客企业服务2 小时前
2025年AI面试推荐榜单,数字化招聘转型优选
人工智能·python·算法·面试·职场和发展·金融·求职招聘
胡斌附体2 小时前
linux测试端口是否可被外部访问
linux·运维·服务器·python·测试·端口测试·临时服务器
likeGhee3 小时前
python缓存装饰器实现方案
开发语言·python·缓存
项目題供诗3 小时前
黑马python(二十五)
开发语言·python
读书点滴3 小时前
笨方法学python -练习14
java·前端·python
笑衬人心。3 小时前
Ubuntu 22.04 修改默认 Python 版本为 Python3 笔记
笔记·python·ubuntu
蛋仔聊测试4 小时前
Playwright 中 Page 对象的常用方法详解
python