Elasticsearch:和 LIamaIndex 的集成

LlamaIndex 是一个数据框架,供 LLM 应用程序摄取、构建和访问私有或特定领域的数据。

LlamaIndex 是开源的,可用于构建各种应用程序。 在 GitHub 上查看该项目。

安装

在 Docker 上设置 Elasticsearch

使用以下 docker 命令启动单节点 Elasticsearch 实例。我们可以参考之前的文章 "Elasticsearch:如何在 Docker 上运行 Elasticsearch 8.x 进行本地开发"。我选择不使用安全配置。直接使用 docker compose 来启动 Elasticsearch 及 Kibana:

.env

复制代码
$ pwd
/Users/liuxg/data/docker8
$ ls -al
total 16
drwxr-xr-x    4 liuxg  staff   128 Jan 16 13:00 .
drwxr-xr-x  193 liuxg  staff  6176 Jan 12 08:31 ..
-rw-r--r--    1 liuxg  staff    21 Jan 16 13:00 .env
-rw-r--r--    1 liuxg  staff   733 Mar 14  2023 docker-compose.yml
$ cat .env
STACK_VERSION=8.11.3

docker-compose.yml

复制代码
version: "3.9"
services:
  elasticsearch:
    image: elasticsearch:${STACK_VERSION}
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
      - xpack.security.enabled=false
    volumes:
      - type: volume
        source: es_data
        target: /usr/share/elasticsearch/data
    ports:
      - target: 9200
        published: 9200
    networks:
      - elastic

  kibana:
    image: kibana:${STACK_VERSION}
    container_name: kibana
    ports:
      - target: 5601
        published: 5601
    depends_on:
      - elasticsearch
    networks:
      - elastic      

volumes:
  es_data:
    driver: local

networks:
  elastic:
    name: elastic
    driver: bridge

我们使用如下的命令来启动:

复制代码
docker-compose up

这样我们就完成了 Elasticsearch 及 Kibana 的安装了。我们的 Elasticsearch 及 Kibana 都没有安全的设置。这个在生产环境中不被推荐使用。

应用设计 - 组装管道

我们将使用 Jupyter notebook 来进行设计。我们在命令行中打入:

复制代码
jupyter notebook

安装依赖

我们使用如下的命令来安装 Python 的依赖包:

复制代码
pip3 install llama-index openai elasticsearch load_dotenv

我们接下来在当前的工作目录中创建一个叫做 .env 的文件:

.env

复制代码
OPENAI_API_KEY="YourOpenAIKey"

请在 .env 中创建如上所示的变量。你需要把自己的 openai key 写入到上面的文件里。

加载模块及读取环境变量

复制代码
import logging
import sys
import os
from dotenv import load_dotenv

load_dotenv()

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

os.environ["OPENAI_API_KEY"] = os.getenv('OPENAI_API_KEY')

import openai

连接到 Elasticsearch

ElasticsearchStore 类用于连接到 Elasticsearch 实例。 它需要以下参数:

  • index_name:Elasticsearch 索引的名称。 必需的。
  • es_client:可选。 预先存在的 Elasticsearch 客户端。
  • es_url:可选。Elasticsearch 网址。
  • es_cloud_id:可选。 Elasticsearch 云 ID。
  • es_api_key:可选。 Elasticsearch API 密钥。
  • es_user:可选。 Elasticsearch 用户名。
  • es_password:可选。 弹性搜索密码。
  • text_field:可选。 存储文本的 Elasticsearch 字段的名称。
  • vector_field:可选。 存储 Elasticsearch 字段的名称嵌入。
  • batch_size:可选。 批量索引的批量大小。 默认为 200。
  • distance_strategy:可选。 用于相似性搜索的距离策略。默认为 "COSINE"。

针对,我们的情况,我们可以使用如下的示例方法来进行本地连接:

复制代码
from llama_index.vector_stores import ElasticsearchStore

es = ElasticsearchStore(
    index_name="my_index",
    es_url="http://localhost:9200",
)

我们将在下面的代码中使用上述的方法来连接 Elasticsearch。

加载文档,使用 Elasticsearch 构建 VectorStoreIndex

复制代码
from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores import ElasticsearchStore

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

运行完上面的命令后,我们可以在当前的目录下查看:

复制代码
$ pwd
/Users/liuxg/python/elser
$ ls data/paul_graham/
paul_graham_essay.txt

我们可以看到一个叫做 pau_graham_essay.txt 的文件。它的内容如下:

复制代码
What I Worked On

February 2021

Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.

The first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines --- CPU, disk drives, printer, card reader --- sitting up on a raised floor under bright fluorescent lights.

...

The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.

I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear.
...

Toward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because they'd sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall. This was now only weeks away. My nice landlady let me leave my stuff in her attic. I had some money saved from consulting work I'd done in grad school; there was probably enough to last a year if I lived cheaply. Now all I had to do was learn Italian.
...

documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

from llama_index.storage.storage_context import StorageContext

vector_store = ElasticsearchStore(
    es_url="http://localhost:9200",
    # Or with Elastic Cloud
    # es_cloud_id="my_cloud_id",
    # es_user="elastic",
    # es_password="my_password",
    index_name="paul_graham",
)

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

基本查询

我们将向查询引擎询问有关我们刚刚索引的数据的问题。

复制代码
# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine()
response = query_engine.query("what were his investments in Y Combinator?")
print(response)

元数据过滤器

在这里,我们将使用元数据索引一些文档,以便我们可以将过滤器应用于查询引擎。

复制代码
from llama_index.schema import TextNode

nodes = [
    TextNode(
        text="The Shawshank Redemption",
        metadata={
            "author": "Stephen King",
            "theme": "Friendship",
        },
    ),
    TextNode(
        text="The Godfather",
        metadata={
            "director": "Francis Ford Coppola",
            "theme": "Mafia",
        },
    ),
    TextNode(
        text="Beautiful weather",
        metadata={
            "director": "Mark shuttle",
            "theme": "Mafia",
        },
    ),    
    TextNode(
        text="Inception",
        metadata={
            "director": "Christopher Nolan",
        },
    ),
]

# initialize the vector store
vector_store_metadata_example = ElasticsearchStore(
    index_name="movies_metadata_example",
    es_url="http://localhost:9200",
)
storage_context = StorageContext.from_defaults(
    vector_store=vector_store_metadata_example
)
index1 = VectorStoreIndex(nodes, storage_context=storage_context)


# Metadata filter
from llama_index.vector_stores.types import ExactMatchFilter, MetadataFilters

filters = MetadataFilters(
    filters=[ExactMatchFilter(key="theme", value="Mafia")]
)

retriever = index1.as_retriever(filters=filters)

retriever.retrieve("weather is so beautiful")

在上面,我们搜索的是 "weather is so beautiful",从而在两个 theme 为 Mafia 的 Texnode 里,Mark shuttle 位列第一。这个是因为 "weather is so beautiful" 更和 "Beautiful weather" 更为贴近。 如果我们使用如下的查询:

复制代码
retriever.retrieve("The godfather is a nice person")

很显然,这次我们的搜索结果的排序颠倒过来了。

自定义过滤器和覆盖查询

llama-index 目前仅支持 ExactMatchFilters。 Elasticsearch 支持多种过滤器,包括范围过滤器、地理过滤器等。 要使用这些过滤器,你可以将它们作为字典列表传递给 es_filter 参数。

复制代码
def custom_query(query, query_str):
    print("custom query", query)
    return query

query_engine = index.as_query_engine(
    vector_store_kwargs={
        "es_filter": [{"match": {"content": "growing up"}}],
        "custom_query": custom_query,
    }
)

response = query_engine.query("what were his investments in Y Combinator?")
print(response)

为了方便大家学习,我把所有的源码放到 github:https://github.com/liu-xiao-guo/semantic_search_es。其中相关的文件是:

更多阅读:使用 Elasticsearch 和 LlamaIndex 进行高级文本检索:句子窗口检索

相关推荐
ConardLi4 分钟前
MCP + 数据库,一种比 RAG 检索效果更好的新方式!
javascript·数据库·人工智能
火车叼位4 分钟前
初中生也能懂的贝叶斯定理:拆解一个改变世界的公式
人工智能·数学·算法
小白的高手之路6 分钟前
torch.nn中的非线性激活介绍合集——Pytorch中的非线性激活
人工智能·pytorch·python·深度学习·神经网络·机器学习·cnn
视觉语言导航8 分钟前
华东师范地面机器人融合空中无人机视角的具身导航!KiteRunner:语言驱动的户外环境合作式局部-全局导航策略
人工智能·深度学习·机器人·无人机·具身智能
纪元A梦8 分钟前
Redis最佳实践——用户会话管理详解
数据库·redis·缓存
逆风优雅15 分钟前
python 爬取网站图片的小demo
开发语言·python
多云的夏天15 分钟前
麒麟-QT-ODBC-达梦
linux·数据库·麒麟
是娜个二叉树!18 分钟前
听课笔记-nlp
人工智能·笔记·自然语言处理
京东云开发者19 分钟前
ClickHouse 的“独孤九剑”:极速查询的终极秘籍
数据库
码界筑梦坊26 分钟前
基于Pyhon的京东笔记本电脑数据可视化分析系统
python·信息可视化·数据分析·毕业设计·电脑·销量预测