.Net使用Elastic.Clients.Elasticsearch在Elasticsearch8中实现向量存储和相似度检索

文章目录

一、测试环境

Elastic.Clients.Elasticsearch版本:8.13.0

Elasticsearch版本:8.13.0

二、代码

1、创建包含DenseVector的索引
csharp 复制代码
public static bool InitIndex()
{
    // 定义索引配置
    var faceVectorproperties = new Properties
        {
            { "Id" ,new KeywordProperty()},
            { "FileID" ,new KeywordProperty()},
            { "FileGUID" ,new KeywordProperty()},
            { "ResourceID" ,new KeywordProperty()},
            { "FileName" ,new TextProperty()},
            { "Embedding" ,new DenseVectorProperty{Dims = 3 } }
        };
    // 定义索引配置
    var indexConfig = new IndexState
    {
        Settings = new IndexSettings
        {
            NumberOfShards = 1, // 设置分片数
            NumberOfReplicas = 1 // 设置副本数
        },
        Mappings = new TypeMapping
        {
            Properties = faceVectorproperties
        }
    };
    //判断是否已经存在该索引
    var existFaceVectorIndexResponse = _client.Indices.ExistsAsync("FaceVector").Result;
    if (!existFaceVectorIndexResponse.IsValidResponse)
    {
        // 创建索引请求
        var createIndexRequest = new CreateIndexRequest("FaceVector")
        {
            Settings = indexConfig.Settings,
            Mappings = indexConfig.Mappings
        };
        var createFaceVectorIndexResponse = _client.Indices.CreateAsync(createIndexRequest).Result;
        if (createFaceVectorIndexResponse.Acknowledged)
        {
                //添加一条测试数据
                ES_FaceVector temp = new ES_FaceVector
                {
                    FileID = 0,
                    FileGUID = Guid.NewGuid(),
                    ResourceID = 0,
                    FileName = "测试",
                    Embedding = new float[] {1.2f,1.1f,1.3f }
                };
                var addDocResult = AddDoc<ES_FaceVector>(temp, ElasticIndexEnum.FaceVector);
        }
        else
        {
            return false;
        }
    }
    return true;
}
2、索引文档
csharp 复制代码
//批量索引文档
public static bool AddDocs<T>(List<T> data, string indexName) where T : class
{
    var bulkIndexResponse = _client.BulkAsync(b => b
        .Index(indexName)
        .IndexMany(data)
    ).Result;
    return bulkIndexResponse.IsValidResponse;
}
//单个索引文档
public static bool AddDoc<T>(T data, string indexName) where T : class
{
    var response = _client.IndexAsync(data, indexName).Result;
    return response.IsValidResponse;
}
3、对向量字段进行近似knn检索
csharp 复制代码
public static void SearchKnn()
{
    // 构建KNN查询
    var doubleArr = new[] { -0.04604065, 0.054946236, 0.057453074};
    var arrLen = doubleArr.Length;
    var knnQuery = new KnnQuery()
    {
        k = 2,
        NumCandidates = 1000,
        Field = "embedding",
        QueryVector= doubleArr.Select(s=>(float)s).ToArray()
    };
    // 构建Elasticsearch查询
    var searchRequest = new SearchRequest<ES_FaceVector>(ElasticIndexEnum.FaceVector)
    {
        Knn = new KnnQuery[] { knnQuery },
        MinScore = 0.90,
        SourceIncludes = new [] { "fileName", "embedding" }
    };

    var searchResponse = _client.Search<ES_FaceVector>(searchRequest);
    if (searchResponse.IsValidResponse)
    {
        foreach (var hit in searchResponse.Hits)
        {
            // 处理每个文档的结果
            var fileNameTemp = hit.Source.FileName;
            var embeddingTemp = hit.Source.Embedding;
            
        }
    }
    else
    {
        Console.WriteLine($"Error: {searchResponse.DebugInformation}");
    }
}

三、参考

.Net使用Elastic.Clients.Elasticsearch连接Elasticsearch8

https://www.elastic.co/guide/en/elasticsearch/client/net-api/8.13/connecting.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html


相关推荐
SafePloy安策1 小时前
ES信息防泄漏:策略与实践
大数据·elasticsearch·开源
涔溪1 小时前
Ecmascript(ES)标准
前端·elasticsearch·ecmascript
水中加点糖4 小时前
使用Spring AI中的RAG技术,实现私有业务领域的大模型系统
人工智能·function call·向量数据库·rag·springai·私有大模型·embedding模型
csdn5659738504 小时前
Elasticsearch 重建索引 数据迁移
elasticsearch·数据迁移·重建索引
天幕繁星4 小时前
docker desktop es windows解决vm.max_map_count [65530] is too low 问题
windows·elasticsearch·docker·docker desktop
Elastic 中国社区官方博客4 小时前
Elasticsearch 8.16:适用于生产的混合对话搜索和创新的向量数据量化,其性能优于乘积量化 (PQ)
大数据·数据库·人工智能·elasticsearch·搜索引擎·ai·全文检索
m1chiru4 小时前
Elasticsearch 实战应用:高效搜索与数据分析
elasticsearch
飞翔的佩奇4 小时前
ElasticSearch:使用dsl语句同时查询出最近2小时、最近1天、最近7天、最近30天的数量
大数据·elasticsearch·搜索引擎·dsl
Crazy Struggle10 小时前
.NET 8 强大功能 IHostedService 与 BackgroundService 实战
c#·.net·.net core
Elastic 中国社区官方博客11 小时前
Elasticsearch 和 Kibana 8.16:Kibana 获得上下文和 BBQ 速度并节省开支!
大数据·数据库·人工智能·elasticsearch·搜索引擎·ai·全文检索