向量数据库Milvus部署及试用

向量数据库milvus

Milvus 是一种高性能、高扩展性的向量数据库，可在从笔记本电脑到大型分布式系统等各种环境中高效运行。它既可以开源软件的形式提供，也可以云服务的形式提供。

milvus安装

建议采用Docker进行安装，详见https://milvus.io/docs/zh/install_standalone-docker.md

Docker安装

# 下载安装脚本
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh

# 开启Docker容器
bash standalone_embed.sh start

# 关闭Docker容器
bash standalone_embed.sh stop

# 删除Milvus数据
bash standalone_embed.sh delete

# 可视化界面
sudo docker run --ipc=host -p 19532:3000 -e MILVUS_URL=10.1.3.118:19530 zilliz/attu:v2.4

attu界面

Docker Compose安装

重启的时候，容易出现问题，我感觉尽量不要用Docker Compose方式安装

向量数据库试用

from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility, Partition, MilvusClient
import time

MILVUS_HOST_LOCAL = '0.0.0.0'
MILVUS_HOST_ONLINE = '0.0.0.0'
MILVUS_PORT = 19530
MILVUS_USER = ''
MILVUS_PASSWORD = ''
MILVUS_DB_NAME = ''
VECTOR_SEARCH_TOP_K = 20
HYBRID_SEARCH = False

## 连接数据库
connections.connect(host='0.0.0.0', port=19530, user='',
                                password='', db_name='', timeout = 3)  # timeout=3 [cannot set]

## 字段定义
fields = [
    FieldSchema(name="pk", dtype=DataType.INT64, is_primary=True, auto_id=False),
    FieldSchema(name="random", dtype=DataType.DOUBLE),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=8)
]
schema = CollectionSchema(fields)

## session定义
milvus_session = Collection("hello_milvus", schema)

## 建立索引
milvus_session.create_index(field_name='embeddings', index_params = {"metric_type": "L2", "index_type": "IVF_FLAT", "params": {"nlist": 128}})

## 添加分区
milvus_session.create_partition('00')
partitions = [Partition(milvus_session, '00')]

## load内存
milvus_session.load()

## 插入数据
num = 1000000
import random
entities = [
    [i for i in range(num)],  # field pk
    [float(random.randrange(-20, -10)) for _ in range(num)],  # field random
    [[random.random() for _ in range(8)] for _ in range(num)],  # field embeddings
]
partitions[0].insert(entities)

## 等待数据插入完成
time.sleep(10)

## 检索配置
vectors_to_search = [[0.5, -0.5, 0.5, -0.5, 0.5, -0.5, 0.5, -0.5]]
search_params = {
    "metric_type": "L2",
    "params": {"nprobe": 10},
}

## 开始检索
for i in range(10000000) :
    start_time = time.time()
    result = milvus_session.search(data=vectors_to_search, partition_names=['00'], anns_field="embeddings",
                                            param={"metric_type": "L2", "params": {"nprobe": 10}}, limit=2,
                                            output_fields=['pk', 'random'], expr='', timeout=None)

    print(result, 'using time in {} s'.format(time.time() - start_time))

版本要求

Milvus Docker版本：milvusdb/milvus:v2.4.8

pymilvus版本：2.4.5