【python qdrant 向量数据库 完整示例代码】

测试一下python版本的dqrant向量数据库的效果,完整代码如下:

安装库

复制代码
!pip install qdrant-client>=1.1.1
!pip install -U sentence-transformers

导入

复制代码
from qdrant_client import models, QdrantClient
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("all-MiniLM-L6-v2", device="cuda")

准备测试数据集

复制代码
documents = [
    {
        "name": "The Time Machine",
        "description": "A man travels through time and witnesses the evolution of humanity."
        * 8,
        "author": "H.G. Wells",
        "year": 1895,
    },
    {
        "name": "Ender's Game",
        "description": "A young boy is trained to become a military leader in a war against an alien race."
        * 4,
        "author": "Orson Scott Card",
        "year": 1985,
    },
    {
        "name": "Brave New World",
        "description": "A dystopian society where people are genetically engineered and conditioned to conform to a strict social hierarchy."
        * 6,
        "author": "Aldous Huxley",
        "year": 1932,
    },
] * 50000

print(len(documents))

创建存储库

复制代码
qdrant = QdrantClient(":memory:")  # 内存中
# qdrant = QdrantClient(path='./qdrant')  # 存储到本地

在数据库中创建一个collection(类似一个存储桶)

复制代码
qdrant.recreate_collection(
    collection_name="my_books",
    vectors_config=models.VectorParams(
        size=encoder.get_sentence_embedding_dimension(),  # Vector size is defined by used model
        distance=models.Distance.COSINE,
    ),
)

对文档进行向量化

复制代码
import hashlib
from tqdm import tqdm

def sha256(text):

    hash_object = hashlib.sha256()
    hash_object.update(text.encode("utf-8"))
    hash_value = hash_object.hexdigest()
    return hash_value

records = []
bs = 256
for i in tqdm(range(0, len(documents), bs)):
    docs = documents[i : i + bs]
    vectors = encoder.encode(
        [doc["description"] for doc in docs], normalize_embeddings=True
    ).tolist()

    record = [
        models.Record(id=idx, vector=vec, payload=doc)  # sha256(doc['description'])
        for idx, vec, doc in zip(range(i, i + bs), vectors, docs)
    ]

    records.extend(record)

上传到向量数据库中指定的collection

复制代码
qdrant.upload_points(
    collection_name="my_books", points=records, batch_size=128, parallel=12
)

语义搜索

复制代码
query = "Aliens attack our planet"
hits = qdrant.search(
    collection_name="my_books",
    query_vector=encoder.encode(query).tolist(),
    limit=6,
)
for hit in hits:
    print(hit.payload, "score:", hit.score)

条件搜索

search only for books from 21st century

复制代码
hits = qdrant.search(
    collection_name="my_books",
    query_vector=encoder.encode("Tyranic society").tolist(),
    query_filter=models.Filter(
        must=[models.FieldCondition(key="year", range=models.Range(gte=1980))]
    ),
    limit=3,
)
for hit in hits:
    print(hit.payload, "score:", hit.score)

参考官方GitHub

github

colab

相关推荐
Wellijohn9 小时前
我把公司内部的 12 个 CRUD 接口交给了 AI Agent,然后省出了两周
人工智能·python·mcp
2601_9679801111 小时前
python pytest怎样测试代码
python·pytest·测试·文档·断言
伞伞悦读11 小时前
【第92期】逻辑回归的分数概率与阈值的实验与排查
人工智能·python·深度学习·机器学习
ch62311 小时前
🚀 从零构建本地文档智能审查 Agent:Vibe Coding 实战指南
python
摇滚侠12 小时前
《On Java 中文版 基础卷》阅读笔记 接口 18
java·笔记·python
Thanks_ks13 小时前
AI Agent 开发入门:用 Function Calling 让大模型学会 “动手干活”,附订单查询机器人实战
python·ai·大模型·openai·agent·通义千问·mcp
专注仿真13 小时前
智能体训练之模仿学习:GAIL 算法详解与实战
python·算法·仿真·多智能体强化学习·afsim智能体训练
databook14 小时前
处理一批 PDF,怎么算真的处理完了
后端·python
承渊政道16 小时前
【Python编程快速上手—让繁琐工作自动化】(列表:把数据排成队,从下标取值到深拷贝)
开发语言·python·pycharm·自动化·列表
大胖爱学习_MJ16 小时前
Agent 如何安全地拿数据库密码?基于 HashiCorp Vault 的动态凭证下发
经验分享·python