快速入门:使用 Gemini Embeddings 和 Elasticsearch 进行向量搜索

Gemini 是 Google DeepMind 开发的多模态大语言模型家族,作为 LaMDA 和 PaLM 2 的后继者。由 Gemini Ultra、Gemini Pro 和 Gemini Nano 组成,于 2023 年 12 月 6 日发布,定位为 OpenAI 的竞争者 GPT-4。

本教程演示如何使用 Gemini API 创建嵌入并将其存储在 Elasticsearch 中。 Elasticsearch 将使我们能够执行向量搜索 (Knn) 来查找相似的文档。

准备

Elasticsearch 及 Kibana

如果你还没有安装好自己的 Elasticsearch 及 Kibana 的话,请参阅如下的文章来进行安装:

在安装的时候,请参照 Elastic Stack 8.x 的文章来进行安装。

Gemini 开发者 key

你可以参考文章 来申请一个免费的 key 供下面的开发。你也可以直接去地址进行申请。

设置环境变量

我们在 termnial 中打入如下的命令来设置环境变量:

ini 复制代码
1.  export ES_USER=elastic
2.  export ES_PASSWORD=-M3aD_m3MHCZNYyJi_V2
3.  export GOOGLE_API_KEY=YourGoogleAPIkey

拷贝 Elasticsearch 证书

我们把 Elasticsearch 的证书拷贝到当前的目录下:

bash 复制代码
1.  $ pwd
2.  /Users/liuxg/python/elser
3.  $ cp ~/elastic/elasticsearch-8.12.0/config/certs/http_ca.crt .

安装 Python 依赖包

css 复制代码
pip3 install -q -U google-generativeai elasticsearch

应用设计

我们在当前的工作目录下打入命令:

jupyter notebook

导入包及环境变量

javascript 复制代码
1.  import google.generativeai as genai
2.  import google.ai.generativelanguage as glm
3.  from elasticsearch import Elasticsearch, helpers
4.  from dotenv import load_dotenv
5.  import os

7.  load_dotenv()

9.  GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")
10.  ES_USER = os.getenv("ES_USER")
11.  ES_PASSWORD = os.getenv("ES_PASSWORD")
12.  elastic_index_name='gemini-demo'

连接到 Elasticsearch

ini 复制代码
1.  url = f"https://{ES_USER}:{ES_PASSWORD}@192.168.0.3:9200"

3.  es = Elasticsearch(
4.          hosts=[url], 
5.          ca_certs = "./http_ca.crt", 
6.          verify_certs = True
7.  )
8.  print(es.info())

上面显示我们的 es 连接是成功的。

删除索引

scss 复制代码
1.  if(es.indices.exists(index=elastic_index_name)):
2.      print("The index has already existed, going to remove it")
3.      es.options(ignore_status=404).indices.delete(index=elastic_index_name)

使用 Elasticsearch 索引文档

生成一个 title 为 "Beijing" 文档:

ini 复制代码
1.  genai.configure(api_key=GOOGLE_API_KEY)

3.  title = "Beijing"
4.  sample_text = ("Beijing is the capital of China and the center of Chinese politics, culture, and economy. This city has a long history with many ancient buildings and cultural heritage. Beijing is renowned as a cultural city in China, boasting numerous museums, art galleries, and historical landmarks. Additionally, as a modern metropolis, Beijing is a thriving business center with modern architecture and advanced transportation systems. It serves as the seat of the Chinese government, where significant decisions and events often take place. Overall, Beijing holds a crucial position in China, serving as both a preserver of traditional culture and a representative of modern development.")

6.  model = 'models/embedding-001'
7.  embedding = genai.embed_content(model=model,
8.                                  content=sample_text,
9.                                  task_type="retrieval_document",
10.                                  title=title)

12.  doc = {
13.      'text' : sample_text,
14.      'text_embedding' : embedding['embedding'] 
15.  }

17.  resp = es.index(index=elastic_index_name, document=doc)

19.  print(resp)

生成一个 title 为 "Shanghai" 的文档:

ini 复制代码
1.  title = "Shanghai"
2.  sample_text = ("Shanghai is one of China's largest cities and a significant hub for economy, finance, and trade. This modern city is located in the eastern part of China and serves as an international metropolis. The bustling streets, skyscrapers, and modern architecture in Shanghai showcase the city's prosperity and development. As one of China's economic engines, Shanghai is home to the headquarters of many international companies and various financial institutions. It is also a crucial trading port, connecting with destinations worldwide. Additionally, Shanghai boasts a rich cultural scene, including art galleries, theaters, and historical landmarks. In summary, Shanghai is a vibrant, modern city with international influence.")

4.  model = 'models/embedding-001'
5.  embedding = genai.embed_content(model=model,
6.                                  content=sample_text,
7.                                  task_type="retrieval_document",
8.                                  title=title)

10.  doc = {
11.      'text' : sample_text,
12.      'text_embedding' : embedding['embedding'] 
13.  }

15.  resp = es.index(index=elastic_index_name, document=doc)

17.  print(resp)

我们可以在 Kibana 中进行查看:

使用 Elasticsearch 来搜索文档

python 复制代码
1.  def search(question):
2.      print("\n\nQuestion: ", question)
3.      embedding = genai.embed_content(model=model,
4.                                      content=question,
5.                                      task_type="retrieval_query")

7.      resp = es.search(
8.      index = elastic_index_name,
9.      knn={
10.          "field": "text_embedding",
11.          "query_vector":  embedding['embedding'],
12.          "k": 10,
13.          "num_candidates": 100
14.          }
15.      )

17.      for result in resp['hits']['hits']:
18.          pretty_output = (f"\n\nID: {result['_id']}\n\nText: {result['_source']['text']}")
19.          print(pretty_output)
scss 复制代码
search("How do you describe Beijing?")
scss 复制代码
search("What is Shanghai like?")

从上面的输出中,我们可以看出来,当搜索的句子和文章更为接近时,相关的文档就会排在第一的位置。紧接着的是次之相关的文档。

scss 复制代码
search("which city is the capital of China?")
scss 复制代码
search("the economy engine in China")

最后,源码在位置可以进行下载:github.com/liu-xiao-gu...

相关推荐
Elastic 中国社区官方博客3 小时前
如何将数据从 AWS S3 导入到 Elastic Cloud - 第 3 部分:Elastic S3 连接器
大数据·elasticsearch·搜索引擎·云计算·全文检索·可用性测试·aws
掘金-我是哪吒3 小时前
微服务mysql,redis,elasticsearch, kibana,cassandra,mongodb, kafka
redis·mysql·mongodb·elasticsearch·微服务
研究是为了理解4 小时前
Git Bash 常用命令
git·elasticsearch·bash
晨欣8 小时前
Elasticsearch和Lucene之间是什么关系?(ChatGPT回答)
elasticsearch·chatgpt·lucene
筱源源14 小时前
Elasticsearch-linux环境部署
linux·elasticsearch
Elastic 中国社区官方博客1 天前
释放专利力量:Patently 如何利用向量搜索和 NLP 简化协作
大数据·数据库·人工智能·elasticsearch·搜索引擎·自然语言处理
Shenqi Lotus1 天前
ELK-ELK基本概念_ElasticSearch的配置
elk·elasticsearch
yeye198912241 天前
10-Query & Filtering 与多字符串多字段查询
elasticsearch
Narutolxy1 天前
精准优化Elasticsearch:磁盘空间管理与性能提升技巧20241106
大数据·elasticsearch·jenkins
谢小涛2 天前
ES管理工具Cerebro 0.8.5 Windows版本安装及启动
elasticsearch·es·cerebro