Elasticsearch Python DSL 客户端开发

在今天的练习中，我们来展示如何使用 Python 客户端来开发 DSL 请求。

安装

我们可以参考之前的文章 "如何在 Linux，MacOS 及 Windows 上进行安装 Elasticsearch" 来安装自己的 Elasticsearch 及 Kibana。

写入数据

我们在 Kibana 中写入如下的数据：

`

1.  PUT my-index
2.  {
3.    "mappings": {
4.      "properties": {
5.        "title": {
6.          "type": "text"
7.        },
8.        "description": {
9.          "type": "text"
10.        },
11.        "category": {
12.          "type": "keyword"
13.        },
14.        "tags": {
15.          "type": "keyword"
16.        },
17.        "lines": {
18.          "type": "integer"
19.        }
20.      }
21.    }
22.  }

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

`

1.  POST my-index/_bulk
2.  {"index":{"_id":"1"}}
3.  {"title":"Python Elasticsearch Client","description":"Learn how to search Elasticsearch using Python.","category":"search","tags":["python","elasticsearch"],"lines":120}
4.  {"index":{"_id":"2"}}
5.  {"title":"Python Search Tutorial","description":"A beginner tutorial for Python search applications.","category":"search","tags":["python","search"],"lines":250}
6.  {"index":{"_id":"3"}}
7.  {"title":"Python Query Examples","description":"Examples of Python queries for Elasticsearch.","category":"search","tags":["python","query"],"lines":180}
8.  {"index":{"_id":"4"}}
9.  {"title":"Advanced Python Search","description":"Advanced techniques for building search applications with Python.","category":"search","tags":["python","search"],"lines":420}
10.  {"index":{"_id":"5"}}
11.  {"title":"Python Beta Features","description":"This document describes beta features in Python search.","category":"search","tags":["python","beta"],"lines":500}
12.  {"index":{"_id":"6"}}
13.  {"title":"Elasticsearch Search Guide","description":"A guide to Elasticsearch search and aggregations.","category":"search","tags":["elasticsearch","search"],"lines":350}
14.  {"index":{"_id":"7"}}
15.  {"title":"Python Machine Learning","description":"Using Python for machine learning and data analysis.","category":"analytics","tags":["python","machine-learning"],"lines":600}
16.  {"index":{"_id":"8"}}
17.  {"title":"Python Logging","description":"How to implement logging in Python applications.","category":"development","tags":["python","logging"],"lines":150}
18.  {"index":{"_id":"9"}}
19.  {"title":"Python Search Beta","description":"Experimental beta implementation of Python search.","category":"search","tags":["python","experimental"],"lines":700}
20.  {"index":{"_id":"10"}}
21.  {"title":"Python Search Performance","description":"Benchmarking Python search performance with Elasticsearch.","category":"search","tags":["python","performance"],"lines":550}

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

搜索

标准 Python

`

1.  import warnings
2.  import urllib3

4.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
5.  warnings.filterwarnings("ignore", category=DeprecationWarning)

7.  from elasticsearch import Elasticsearch

9.  client = Elasticsearch(
10.      "https://localhost:9200",
11.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
12.      verify_certs=False,
13.  )

15.  response = client.search(
16.      index="my-index",
17.      body={
18.        "query": {
19.          "bool": {
20.            "must": [{"match": {"title": "python"}}],
21.            "must_not": [{"match": {"description": "beta"}}],
22.            "filter": [{"term": {"category": "search"}}]
23.          }
24.        },
25.        "aggs" : {
26.          "per_tag": {
27.            "terms": {"field": "tags"},
28.            "aggs": {
29.              "max_lines": {"max": {"field": "lines"}}
30.            }
31.          }
32.        }
33.      }
34.  )

36.  for hit in response['hits']['hits']:
37.      print(hit['_score'], hit['_source']['title'])

39.  for tag in response['aggregations']['per_tag']['buckets']:
40.      print(tag['key'], tag['max_lines']['value'])

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

Async Python

`

1.  import asyncio
2.  import warnings
3.  import urllib3

5.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
6.  warnings.filterwarnings("ignore", category=DeprecationWarning)

8.  from elasticsearch import AsyncElasticsearch
9.  from elasticsearch_dsl import AsyncSearch, Q

11.  client = AsyncElasticsearch(
12.      "https://localhost:9200",
13.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
14.      verify_certs=False,
15.  )

17.  async def example():
18.      response = await client.search(
19.          index="my-index",
20.          body={
21.            "query": {
22.              "bool": {
23.                "must": [{"match": {"title": "python"}}],
24.                "must_not": [{"match": {"description": "beta"}}],
25.                "filter": [{"term": {"category": "search"}}]
26.              }
27.            },
28.            "aggs" : {
29.              "per_tag": {
30.                "terms": {"field": "tags"},
31.                "aggs": {
32.                  "max_lines": {"max": {"field": "lines"}}
33.                }
34.              }
35.            }
36.          }
37.      )

39.      for hit in response['hits']['hits']:
40.          print(hit['_score'], hit['_source']['title'])

42.      for tag in response['aggregations']['per_tag']['buckets']:
43.          print(tag['key'], tag['max_lines']['value'])

45.  asyncio.run(example())

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

这种方法的问题在于，它冗长、容易出现语法错误，例如嵌套不正确，而且难以修改（例如添加另一个过滤条件），写起来也绝对谈不上有趣。

让我们使用 DSL 模块重写这个示例：

`

1.  import warnings
2.  import urllib3

4.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
5.  warnings.filterwarnings("ignore", category=DeprecationWarning)

7.  from elasticsearch import Elasticsearch
8.  from elasticsearch.dsl import Search, query, aggs

10.  client = Elasticsearch(
11.      "https://localhost:9200",
12.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
13.      verify_certs=False,
14.  )

16.  s = Search(using=client, index="my-index") \
17.      .query(query.Match("title", "python"))   \
18.      .filter(query.Term("category", "search")) \
19.      .exclude(query.Match("description", "beta"))

21.  s.aggs.bucket('per_tag', aggs.Terms(field="tags")) \
22.      .metric('max_lines', aggs.Max(field='lines'))

24.  response = s.execute()

26.  for hit in response:
27.      print(hit.meta.score, hit.title)

29.  for tag in response.aggregations.per_tag.buckets:
30.      print(tag.key, tag.max_lines.value)

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

及

`

1.  import warnings
2.  import urllib3

4.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
5.  warnings.filterwarnings("ignore", category=DeprecationWarning)

7.  from elasticsearch import Elasticsearch
8.  from elasticsearch.dsl import Search, query, aggs

10.  client = Elasticsearch(
11.      "https://localhost:9200",
12.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
13.      verify_certs=False,
14.  )

16.  s = Search(using=client, index="my-index") \
17.      .query(query.Match("title", "python"))   \
18.      .filter(query.Term("category", "search")) \
19.      .exclude(query.Match("description", "beta"))

21.  s.aggs.bucket('per_tag', aggs.Terms(field="tags")) \
22.      .metric('max_lines', aggs.Max(field='lines'))

24.  response = s.execute()

26.  for hit in response:
27.      print(hit.meta.score, hit.title)

29.  for tag in response.aggregations.per_tag.buckets:
30.      print(tag.key, tag.max_lines.value)

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

正如你所看到的，DSL 模块处理了以下事项：

• 从类创建适当的 Query 对象

• 将多个查询组合成复合 bool 查询

• 将 term 查询放入 bool 查询的过滤上下文中

• 提供便捷的响应数据访问方式

• 不再到处出现花括号或方括号

持久化

让我们创建一个简单的 Python 类 ，用于表示博客系统中的一篇文章：

标准 Python

`

1.  import warnings
2.  import urllib3

4.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
5.  warnings.filterwarnings("ignore", category=DeprecationWarning)

7.  from datetime import datetime
8.  from elasticsearch.dsl import Document, Date, Integer, Keyword, Text, connections, mapped_field

10.  # Define a default Elasticsearch client
11.  connections.create_connection(
12.      hosts="https://localhost:9200",
13.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
14.      verify_certs=False,
15.  )

17.  class Article(Document):
18.      title: str = mapped_field(Text(analyzer='snowball', fields={'raw': Keyword()}))
19.      body: str = mapped_field(Text(analyzer='snowball'))
20.      tags: list[str] = mapped_field(Keyword())
21.      published_from: datetime
22.      lines: int

24.      class Index:
25.          name = 'blog'
26.          settings = {
27.            "number_of_shards": 2,
28.          }

30.      def save(self, **kwargs):
31.          self.lines = len(self.body.split())
32.          return super(Article, self).save(** kwargs)

34.      def is_published(self):
35.          return datetime.now() > self.published_from

37.  # create the mappings in elasticsearch
38.  Article.init()

40.  # create and save and article
41.  article = Article(meta={'id': 42}, title='Hello world!', tags=['test'])
42.  article.body = ''' looong text '''
43.  article.published_from = datetime.now()
44.  article.save()

46.  article = Article.get(id=42)
47.  print(article.is_published())

49.  # Display cluster health
50.  print(connections.get_connection().cluster.health())

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)收起代码块![](https://csdnimg.cn/release/blogv2/dist/pc/img/arrowup-line-top-White.png)

Async Python

`

1.  import asyncio
2.  import warnings
3.  import urllib3

5.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
6.  warnings.filterwarnings("ignore", category=DeprecationWarning)

8.  from datetime import datetime
9.  from elasticsearch.dsl import AsyncDocument, Date, Integer, Keyword, Text, async_connections, mapped_field

11.  # Define a default Elasticsearch client
12.  async_connections.create_connection(
13.      hosts="https://localhost:9200",
14.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
15.      verify_certs=False,
16.  )

18.  class Article(AsyncDocument):
19.      title: str = mapped_field(Text(analyzer='snowball', fields={'raw': Keyword()}))
20.      body: str = mapped_field(Text(analyzer='snowball'))
21.      tags: list[str] = mapped_field(Keyword())
22.      published_from: datetime
23.      lines: int

25.      class Index:
26.          name = 'blog'
27.          settings = {
28.            "number_of_shards": 2,
29.          }

31.      async def save(self, **kwargs):
32.          self.lines = len(self.body.split())
33.          return await super(Article, self).save(** kwargs)

35.      def is_published(self):
36.          return datetime.now() > self.published_from

38.  async def example():
39.      # create the mappings in elasticsearch
40.      await Article.init()

42.      # create and save and article
43.      article = Article(meta={'id': 42}, title='Hello world!', tags=['test'])
44.      article.body = ''' looong text '''
45.      article.published_from = datetime.now()
46.      await article.save()

48.      article = await Article.get(id=42)
49.      print(article.is_published())

51.      # Display cluster health
52.      print(await async_connections.get_connection().cluster.health())

54.  asyncio.run(example())

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)收起代码块![](https://csdnimg.cn/release/blogv2/dist/pc/img/arrowup-line-top-White.png)

在这个示例中，你可以看到：

• 提供一个默认连接

• 使用 Python 类型提示定义字段，并在必要时提供额外的 mapping 配置

• 设置索引名称

• 定义自定义方法

• 重写内置的 .save() 方法，以便接入持久化生命周期

• 从 Elasticsearch 中检索和保存对象

• 访问底层客户端，以使用其他 API

你可以在持久化章节中了解更多信息。

预构建的分面搜索

如果你已经定义了 Document，就可以创建一个分面搜索类，以简化搜索和过滤。

标准 Python

`

1.  import warnings
2.  import urllib3

4.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
5.  warnings.filterwarnings("ignore", category=DeprecationWarning)
6.  warnings.filterwarnings("ignore", message=".*verify_certs=False.*")

8.  from elasticsearch.dsl import FacetedSearch, TermsFacet, DateHistogramFacet, connections
9.  from example6 import Article

11.  connections.create_connection(
12.      hosts="https://localhost:9200",
13.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
14.      verify_certs=False,
15.  )

17.  class BlogSearch(FacetedSearch):
18.      doc_types = [Article, ]
19.      # fields that should be searched
20.      fields = ['tags', 'title', 'body']

22.      facets = {
23.          # use bucket aggregations to define facets
24.          'tags': TermsFacet(field='tags'),
25.          'publishing_frequency': DateHistogramFacet(field='published_from', calendar_interval='month')
26.      }

28.  # empty search
29.  bs = BlogSearch()
30.  response = bs.execute()

32.  for hit in response:
33.      print(hit.meta.score, getattr(hit, 'title', 'N/A'))

35.  for (tag, count, selected) in response.facets.tags:
36.      print(tag, ' (SELECTED):' if selected else ':', count)

38.  for (month, count, selected) in response.facets.publishing_frequency:
39.      print(month.strftime('%B %Y'), ' (SELECTED):' if selected else ':', count)

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

Async Python

`

1.  import asyncio
2.  import warnings
3.  import urllib3

5.  urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
6.  warnings.filterwarnings("ignore", category=DeprecationWarning)

8.  from elasticsearch.dsl import AsyncFacetedSearch, TermsFacet, DateHistogramFacet, async_connections
9.  from example5 import Article

11.  async_connections.create_connection(
12.      hosts="https://localhost:9200",
13.      api_key="d25PamhLQUJHblNmOTlpVmpydWc6NkJLNXA2TDFQUzFiRzRKZ2wxYy1ZZw==",
14.      verify_certs=False,
15.  )

17.  class BlogSearch(AsyncFacetedSearch):
18.      doc_types = [Article, ]
19.      # fields that should be searched
20.      fields = ['tags', 'title', 'body']

22.      facets = {
23.          # use bucket aggregations to define facets
24.          'tags': TermsFacet(field='tags'),
25.          'publishing_frequency': DateHistogramFacet(field='published_from', interval='month')
26.      }

28.  async def example():
29.      # empty search
30.      bs = BlogSearch()
31.      response = await bs.execute()

33.      for hit in response:
34.          print(hit.meta.score, hit.title)

36.      for (tag, count, selected) in response.facets.tags:
37.          print(tag, ' (SELECTED):' if selected else ':', count)

39.      for (month, count, selected) in response.facets.publishing_frequency:
40.          print(month.strftime('%B %Y'), ' (SELECTED):' if selected else ':', count)

42.  asyncio.run(example())

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

你可以在 faceted_search 章节中找到更多详细信息。

按查询更新

让我们继续使用博客文章这个简单示例，并假设每篇文章都有一定数量的点赞。在这个示例中，假设我们希望将所有匹配某个标签且不匹配某个描述的文章的点赞数增加 1。如果将其写成 dict，代码如下：

标准 Python

`

1.  from elasticsearch import Elasticsearch

3.  client = Elasticsearch()

5.  response = client.update_by_query(
6.      index="my-index",
7.      body={
8.        "query": {
9.          "bool": {
10.            "must": [{"match": {"tag": "python"}}],
11.            "must_not": [{"match": {"description": "beta"}}]
12.          }
13.        },
14.        "script"={
15.          "source": "ctx._source.likes++",
16.          "lang": "painless"
17.        }
18.      },
19.    )

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

Async Python

`

1.  from elasticsearch import AsyncElasticsearch

3.  client = AsyncElasticsearch()

5.  async def example():
6.      response = await client.update_by_query(
7.          index="my-index",
8.          body={
9.            "query": {
10.              "bool": {
11.                "must": [{"match": {"tag": "python"}}],
12.                "must_not": [{"match": {"description": "beta"}}]
13.              }
14.            },
15.            "script"={
16.              "source": "ctx._source.likes++",
17.              "lang": "painless"
18.            }
19.          },
20.        )

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

现在，使用 DSL，我们可以这样表达这个查询：

标准 Python

`

1.  from elasticsearch import AsyncElasticsearch
2.  from elasticsearch.dsl import AsyncSearch, AsyncUpdateByQuery
3.  from elasticsearch.dsl.query import Match

5.  client = AsyncElasticsearch()

7.  async def example():
8.      ubq = UpdateByQuery(using=client, index="my-index") \
9.            .query(Match("title", "python"))   \
10.            .exclude(Match("description", "beta")) \
11.            .script(source="ctx._source.likes++", lang="painless")

13.      response = await ubq.execute()

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

Async Python

`

1.  from elasticsearch import Elasticsearch
2.  from elasticsearch.dsl import Search, UpdateByQuery
3.  from elasticsearch.dsl.query import Match

5.  client = Elasticsearch()
6.  ubq = UpdateByQuery(using=client, index="my-index") \
7.        .query(Match("title", "python"))   \
8.        .exclude(Match("description", "beta")) \
9.        .script(source="ctx._source.likes++", lang="painless")

11.  response = ubq.execute()

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

正如你所看到的，Update By Query 对象提供了 Search 对象所带来的许多便利，同时还允许你以相同的方式，通过指定的脚本更新搜索结果。

ES|QL 查询

DSL 模块提供了与 ES|QL 查询构建器的集成，其中包含两个所有 Document 子类都可使用的方法：esql_from() 和 esql_execute()。使用上面的 Article 文档，我们可以通过以下 ES|QL 查询，搜索最多 10 篇标题中包含 "world" 的文章：

标准 Python

`

1.  from elasticsearch.esql import functions

3.  query = Article.esql_from().where(functions.match(Article.title, 'world')).limit(10)
4.  for a in Article.esql_execute(query):
5.      print(a.title)

`Lobster AI

Aysnc Python

`

1.  from elasticsearch.esql import functions

3.  async def example():
4.      query = Article.esql_from().where(functions.match(Article.title, 'world')).limit(10)
5.      async for a in Article.esql_execute(query):
6.          print(a.title)

`Lobster AI

查看 ES|QL 查询构建器章节，了解更多关于如何在 Python 中构建 ES|QL 查询的信息。

从标准客户端迁移

你不必为了获得 DSL 模块带来的好处而迁移整个应用程序。你可以逐步开始：从现有的 dict 创建一个 Search 对象，使用 API 对其进行修改，然后再将其序列化回 dict：

标准 Python

`

1.  body = {...}

3.  # Convert to Search object
4.  s = Search.from_dict(body)

6.  # Add some filters, aggregations, queries, ...
7.  s.filter(query.Term("tags", "python"))

9.  # Convert back to dict to plug back into existing code
10.  body = s.to_dict()

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)

Async Python

`

1.  body = {...}

3.  # Convert to Search object
4.  s = Search.from_dict(body)

6.  # Add some filters, aggregations, queries, ...
7.  s.filter(query.Term("tags", "python"))

9.  # Convert back to dict to plug back into existing code
10.  body = s.to_dict()

`Lobster AI![](https://csdnimg.cn/release/blogv2/dist/pc/img/runCode/icon-arrowwhite.png)