ELK 日志管理系统相关内容总结

Gss7772025-10-24 18:12

ELK 日志管理系统相关内容总结

一、日志管理背景与需求

  1. 传统日志管理痛点：日志分散在不同服务器，传统登录单台服务器查看的方式繁琐低效；rsyslog 虽能实现日志集中管理，但集中后的日志统计与检索（如按变动时间周期统计网站 PV、UV 数据，合并多 realserver 日志后统计）存在困难，wc、grep、awk 等命令在高要求场景下力不从心
  2. 核心需求：需要专业的集中式日志收集、分析、展示系统，满足日志集中管理、灵活统计（如多服务器日志合并统计）、图形化展示数据的需求

二、ELK 系统基础认知

  1. ELK 组成：由 Elasticsearch（ES）、Logstash、Kibana 三款开源软件组成，常搭配轻量级日志收集工具 Filebeat，官网下载地址为https://www.elastic.co/cn/downloads

  2. 各组件核心功能

    组件 核心功能
    Elasticsearch（ES） 分布式搜索引擎与文档数据库，提供数据存储、快速搜索与分析功能
    Logstash 日志采集与处理工具，通过 Input（导入日志源）、Filter（过滤，非必需）、Output（导出，必需）插件，采集多种格式数据（结构化、半结构化、非结构化）并输出到指定目的地（如 ES）
    Kibana 日志图形化展示工具，支持通过浏览器访问，可视化 Elasticsearch 中的数据
    Filebeat 轻量级日志收集工具，适用于资源有限的服务器，辅助 Logstash 采集日志

  3. 工作流程：Logstash/Filebeat 从应用服务器收集日志，经过滤后输出给 Elasticsearch 集群存储，用户通过 Kibana 访问 Elasticsearch 获取并查看图形化日志数据

  4. 应用场景：解决分散日志的集中管理、多维度统计分析（如网站 PV/UV 统计）、日志数据可视化展示等问题，适用于运维人员监控系统与业务状态

具体示例：

四台机器(内存建议大于1G，比如1.5G; filebeat服务器可为1G) :

静态IP(要求能上公网,最好用虚拟机的NAT网络类型上网)

主机名及主机名绑定

IP 地址 主机名 部署组件
192.168.100.10 vm1.cluster.com Kibana
192.168.100.20 vm2.cluster.com Elasticsearch
192.168.100.30 vm3.cluster.com Logstash

已关闭防火墙与selinux

设置时间同步

elasticsearch部署

第1步: 在elasticsearch服务器上(我这里为vm2),确认jdk(使用系统自带的openjdk就OK)

复制代码 
[root@vm2 ~]# rpm -qa | grep openjdk
java-1.8.0-openjdk-1.8.0.181-7.b13.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.181-7.b13.el7.x86_64
[root@vm2 ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

第2步: es的安装,配置：（注：这里是上传）

复制代码 
[root@vm2 ~]# rz -E
rz waiting to receive.
[root@vm2 ~]# rpm -ivh elasticsearch-6.5.2.rpm 
warning: elasticsearch-6.5.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...
   1:elasticsearch-0:6.5.2-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch

第3步: 单机es的配置与服务启动：

复制代码 
[root@vm2 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-cluster       //可以自定义一个集群名称,不配置的话默认会取名为elasticsearch

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0           //打开注释，并修改为监听所有

http.port: 9200                 //打开注释，监听端口9200
[root@vm2 ~]# systemctl start elasticsearch
[root@vm2 ~]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

启动有点慢和卡，稍等1分钟左右，查看到以下端口则表示启动OK

复制代码 
[root@vm2 ~]# netstat -ntlup | grep java
tcp6       0      0 :::9200                 :::*                    LISTEN      57963/java          
tcp6       0      0 :::9300                 :::*                    LISTEN      57963/java

9200则是数据传输端口

9300端口是集群通信端口（我们暂时还没有配置集群，现在是单点elasticsearch）

第4步: 查看状态

使用curl命令或浏览器访问http://192.168.100.20:9200/_cluster/health?pretty地址(IP为ES服务器IP)

复制代码 
[root@vm2 ~]# curl http://192.168.100.20:9200/_cluster/health?pretty
{
  "cluster_name" : "elk-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

elasticsearch集群部署

可以使用两台或两台以上ES做集群, 以下就是两台ES做集群的配置

首先在ES集群所有节点都安装ES

复制代码 
[root@vm1 ~]# rz -E
rz waiting to receive.
[root@vm1 ~]# rpm -ivh elasticsearch-6.5.2.rpm 
warning: elasticsearch-6.5.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...
   1:elasticsearch-0:6.5.2-1          ################################# [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch

[root@vm1 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-cluster

node.name: 192.168.100.10       //本机IP或主机名
node.master: false                //指定不为master节点

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0

http.port: 9200

discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]   //集群所有节点IP

[root@vm2 ~]# vim /etc/elasticsearch/elasticsearch.yml 
......
cluster.name: elk-cluster

node.name: 192.168.100.20       //本机IP或主机名
node.master: true                //指定为master节点

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0

http.port: 9200

discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]   //集群所有节点IP

启动或重启服务：

复制代码 
[root@vm1 ~]# systemctl restart elasticsearch
[root@vm1 ~]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

[root@vm2 ~]# systemctl restart elasticsearch

查看状态：

查看节点信息：

通过curl或浏览器访问http://192.168.100.20:9200/_cat/nodes?v(ip为ES节点IP,如果有ES集群,则为ES任意节

点IP)

复制代码 
[root@vm2 ~]# curl http://192.168.100.20:9200/_cat/nodes?v
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.100.20           26          69   0    0.06    0.06     0.05 mdi       *      192.168.100.20
192.168.100.10           25          68   0    0.00    0.03     0.05 di        -      192.168.100.10

查看索引信息：

通过curl或浏览器访问http://10.1.1.12:9200/_cat/indices?v：

复制代码 
[root@vm2 ~]# curl http://192.168.100.20:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

默认现在没有任何索引

新增索引：

复制代码 
[root@vm2 ~]# curl -X PUT http://192.168.100.20:9200/nginx_access_log
{"acknowledged":true,"shards_acknowledged":true,"index":"nginx_access_log"}

[root@vm2 ~]# curl http://192.168.100.20:9200/_cat/indices?v
health status index            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   nginx_access_log DywHuNoaRB6fEU4zmi0oIQ   5   1          0            0      2.2kb          1.1kb

green：所有的主分片和副本分片都已分配。你的集群是100%可用的

yellow：所有的主分片已经分片了，但至少还有一个副本是缺失的。不会有数据丢失，所以搜索结果依

然是完整的。不过，你的高可用性在某种程度上被弱化。如果 更多的 分片消失，你就会丢数据了。把

yellow 想象成一个需要及时调查的警告

red：至少一个主分片（以及它的全部副本）都在缺失中。这意味着你在缺少数据：搜索只能返回部分数

据，而分配到这个分片上的写入请求会返回一个异常

删除索引：

复制代码 
[root@vm2 ~]# curl -X DELETE http://192.168.100.20:9200/nginx_access_log
{"acknowledged":true}

ES查询语句（拓展了解）

ES提供一种可用于执行查询JSON式的语言，被称为Query DSL

针对elasticsearch的操作，可以分为增、删、改、查四个动作

查询匹配条件：

match_all

from,size

match

bool

range

查询应用案例：

导入数据源

使用官方提供的示例数据：

下载并导入进elasticsearch：（注：这里是直接上传）

复制代码 
[root@vm2 ~]# rz -E
rz waiting to receive.

导入进elasticsearch
[root@vm2 ~]# curl -H "Content-Type: application/json" -XPOST "192.168.100.20:9200/bank/_doc/_bulk?pretty&refresh" --data-binary "@accounts.json"

查询确认
[root@vm2 ~]# curl "192.168.100.20:9200/_cat/indices?v"
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   bank  dC8RjIl1QWeSMmpIpV-vsw   5   1       1000            0    949.4kb        474.7kb

查询bank索引的数据(使用查询字符串进行查询)

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?q=*&sort=account_number:asc&pretty"
{
  "took" : 98,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : null,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "0",
        "_score" : null,
        "_source" : {
          "account_number" : 0,
          "balance" : 16623,
          "firstname" : "Bradshaw",
          "lastname" : "Mckenzie",
          "age" : 29,
          "gender" : "F",
          "address" : "244 Columbus Place",
          "employer" : "Euron",
          "email" : "bradshawmckenzie@euron.com",
          "city" : "Hobucken",
          "state" : "CO"
        },
        "sort" : [
          0
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : null,
        "_source" : {
          "account_number" : 1,
          "balance" : 39225,
          "firstname" : "Amber",
          "lastname" : "Duke",
          "age" : 32,
          "gender" : "M",
          "address" : "880 Holmes Lane",
          "employer" : "Pyrami",
          "email" : "amberduke@pyrami.com",
          "city" : "Brogan",
          "state" : "IL"
        },
        "sort" : [
          1
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : null,
        "_source" : {
          "account_number" : 2,
          "balance" : 28838,
          "firstname" : "Roberta",
          "lastname" : "Bender",
          "age" : 22,
          "gender" : "F",
          "address" : "560 Kingsway Place",
          "employer" : "Chillium",
          "email" : "robertabender@chillium.com",
          "city" : "Bennett",
          "state" : "LA"
        },
        "sort" : [
          2
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : null,
        "_source" : {
          "account_number" : 3,
          "balance" : 44947,
          "firstname" : "Levine",
          "lastname" : "Burks",
          "age" : 26,
          "gender" : "F",
          "address" : "328 Wilson Avenue",
          "employer" : "Amtap",
          "email" : "levineburks@amtap.com",
          "city" : "Cochranville",
          "state" : "HI"
        },
        "sort" : [
          3
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : null,
        "_source" : {
          "account_number" : 4,
          "balance" : 27658,
          "firstname" : "Rodriquez",
          "lastname" : "Flores",
          "age" : 31,
          "gender" : "F",
          "address" : "986 Wyckoff Avenue",
          "employer" : "Tourmania",
          "email" : "rodriquezflores@tourmania.com",
          "city" : "Eastvale",
          "state" : "HI"
        },
        "sort" : [
          4
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "5",
        "_score" : null,
        "_source" : {
          "account_number" : 5,
          "balance" : 29342,
          "firstname" : "Leola",
          "lastname" : "Stewart",
          "age" : 30,
          "gender" : "F",
          "address" : "311 Elm Place",
          "employer" : "Diginetic",
          "email" : "leolastewart@diginetic.com",
          "city" : "Fairview",
          "state" : "NJ"
        },
        "sort" : [
          5
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "6",
        "_score" : null,
        "_source" : {
          "account_number" : 6,
          "balance" : 5686,
          "firstname" : "Hattie",
          "lastname" : "Bond",
          "age" : 36,
          "gender" : "M",
          "address" : "671 Bristol Street",
          "employer" : "Netagy",
          "email" : "hattiebond@netagy.com",
          "city" : "Dante",
          "state" : "TN"
        },
        "sort" : [
          6
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "7",
        "_score" : null,
        "_source" : {
          "account_number" : 7,
          "balance" : 39121,
          "firstname" : "Levy",
          "lastname" : "Richard",
          "age" : 22,
          "gender" : "M",
          "address" : "820 Logan Street",
          "employer" : "Teraprene",
          "email" : "levyrichard@teraprene.com",
          "city" : "Shrewsbury",
          "state" : "MO"
        },
        "sort" : [
          7
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "8",
        "_score" : null,
        "_source" : {
          "account_number" : 8,
          "balance" : 48868,
          "firstname" : "Jan",
          "lastname" : "Burns",
          "age" : 35,
          "gender" : "M",
          "address" : "699 Visitation Place",
          "employer" : "Glasstep",
          "email" : "janburns@glasstep.com",
          "city" : "Wakulla",
          "state" : "AZ"
        },
        "sort" : [
          8
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "9",
        "_score" : null,
        "_source" : {
          "account_number" : 9,
          "balance" : 24776,
          "firstname" : "Opal",
          "lastname" : "Meadows",
          "age" : 39,
          "gender" : "M",
          "address" : "963 Neptune Avenue",
          "employer" : "Cedward",
          "email" : "opalmeadows@cedward.com",
          "city" : "Olney",
          "state" : "OH"
        },
        "sort" : [
          9
        ]
      }
    ]
  }
}

说明：
默认结果为10条
_search 属于一类API，用于执行查询操作
q=* ES批量索引中的所有文档
sort=account_number:asc 表示根据account_number按升序对结果排序
pretty调整显示格式

查询bank索引的数据 (使用json格式进行查询)：

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search" -H 'Content-Type: application/json' -d'
> {
> "query": { "match_all": {} },
> "sort": [
> { "account_number": "asc" }
> ]
> }
> '      注意：最后为单引号
{"took":10,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1000,"max_score":null,"hits":[{"_index":"bank","_type":"_doc","_id":"0","_score":null,"_source":{"account_number":0,"balance":16623,"firstname":"Bradshaw","lastname":"Mckenzie","age":29,"gender":"F","address":"244 Columbus Place","employer":"Euron","email":"bradshawmckenzie@euron.com","city":"Hobucken","state":"CO"},"sort":[0]},{"_index":"bank","_type":"_doc","_id":"1","_score":null,"_source":{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"},"sort":[1]},{"_index":"bank","_type":"_doc","_id":"2","_score":null,"_source":{"account_number":2,"balance":28838,"firstname":"Roberta","lastname":"Bender","age":22,"gender":"F","address":"560 Kingsway Place","employer":"Chillium","email":"robertabender@chillium.com","city":"Bennett","state":"LA"},"sort":[2]},{"_index":"bank","_type":"_doc","_id":"3","_score":null,"_source":{"account_number":3,"balance":44947,"firstname":"Levine","lastname":"Burks","age":26,"gender":"F","address":"328 Wilson Avenue","employer":"Amtap","email":"levineburks@amtap.com","city":"Cochranville","state":"HI"},"sort":[3]},{"_index":"bank","_type":"_doc","_id":"4","_score":null,"_source":{"account_number":4,"balance":27658,"firstname":"Rodriquez","lastname":"Flores","age":31,"gender":"F","address":"986 Wyckoff Avenue","employer":"Tourmania","email":"rodriquezflores@tourmania.com","city":"Eastvale","state":"HI"},"sort":[4]},{"_index":"bank","_type":"_doc","_id":"5","_score":null,"_source":{"account_number":5,"balance":29342,"firstname":"Leola","lastname":"Stewart","age":30,"gender":"F","address":"311 Elm Place","employer":"Diginetic","email":"leolastewart@diginetic.com","city":"Fairview","state":"NJ"},"sort":[5]},{"_index":"bank","_type":"_doc","_id":"6","_score":null,"_source":{"account_number":6,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"},"sort":[6]},{"_index":"bank","_type":"_doc","_id":"7","_score":null,"_source":{"account_number":7,"balance":39121,"firstname":"Levy","lastname":"Richard","age":22,"gender":"M","address":"820 Logan Street","employer":"Teraprene","email":"levyrichard@teraprene.com","city":"Shrewsbury","state":"MO"},"sort":[7]},{"_index":"bank","_type":"_doc","_id":"8","_score":null,"_source":{"account_number":8,"balance":48868,"firstname":"Jan","lastname":"Burns","age":35,"gender":"M","address":"699 Visitation Place","employer":"Glasstep","email":"janburns@glasstep.com","city":"Wakulla","state":"AZ"},"sort":[8]},{"_index":"bank","_type":"_doc","_id":"9","_score":null,"_source":{"account_number":9,"balance":24776,"firstname":"Opal","lastname":"Meadows","age":39,"gender":"M","address":"963 Neptune Avenue","employer":"Cedward","email":"opalmeadows@cedward.com","city":"Olney","state":"OH"},"sort":[9]}]}}

问题: 怎么将上面json格式进行pretty查询?

查询匹配动作及案例：

match_all

匹配所有文档。默认查询

示例：查询所有，默认只返回10个文档

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
> {
> "query": { "match_all": {} }
> }
> '
{
  "took" : 6,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "25",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 25,
          "balance" : 40540,
          "firstname" : "Virginia",
          "lastname" : "Ayala",
          "age" : 39,
          "gender" : "F",
          "address" : "171 Putnam Avenue",
          "employer" : "Filodyne",
          "email" : "virginiaayala@filodyne.com",
          "city" : "Nicholson",
          "state" : "PA"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "44",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 44,
          "balance" : 34487,
          "firstname" : "Aurelia",
          "lastname" : "Harding",
          "age" : 37,
          "gender" : "M",
          "address" : "502 Baycliff Terrace",
          "employer" : "Orbalix",
          "email" : "aureliaharding@orbalix.com",
          "city" : "Yardville",
          "state" : "DE"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "99",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 99,
          "balance" : 47159,
          "firstname" : "Ratliff",
          "lastname" : "Heath",
          "age" : 39,
          "gender" : "F",
          "address" : "806 Rockwell Place",
          "employer" : "Zappix",
          "email" : "ratliffheath@zappix.com",
          "city" : "Shaft",
          "state" : "ND"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "119",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 119,
          "balance" : 49222,
          "firstname" : "Laverne",
          "lastname" : "Johnson",
          "age" : 28,
          "gender" : "F",
          "address" : "302 Howard Place",
          "employer" : "Senmei",
          "email" : "lavernejohnson@senmei.com",
          "city" : "Herlong",
          "state" : "DC"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "126",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 126,
          "balance" : 3607,
          "firstname" : "Effie",
          "lastname" : "Gates",
          "age" : 39,
          "gender" : "F",
          "address" : "620 National Drive",
          "employer" : "Digitalus",
          "email" : "effiegates@digitalus.com",
          "city" : "Blodgett",
          "state" : "MD"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "145",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 145,
          "balance" : 47406,
          "firstname" : "Rowena",
          "lastname" : "Wilkinson",
          "age" : 32,
          "gender" : "M",
          "address" : "891 Elton Street",
          "employer" : "Asimiline",
          "email" : "rowenawilkinson@asimiline.com",
          "city" : "Ripley",
          "state" : "NH"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "183",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 183,
          "balance" : 14223,
          "firstname" : "Hudson",
          "lastname" : "English",
          "age" : 26,
          "gender" : "F",
          "address" : "823 Herkimer Place",
          "employer" : "Xinware",
          "email" : "hudsonenglish@xinware.com",
          "city" : "Robbins",
          "state" : "ND"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "190",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 190,
          "balance" : 3150,
          "firstname" : "Blake",
          "lastname" : "Davidson",
          "age" : 30,
          "gender" : "F",
          "address" : "636 Diamond Street",
          "employer" : "Quantasis",
          "email" : "blakedavidson@quantasis.com",
          "city" : "Crumpler",
          "state" : "KY"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "208",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 208,
          "balance" : 40760,
          "firstname" : "Garcia",
          "lastname" : "Hess",
          "age" : 26,
          "gender" : "F",
          "address" : "810 Nostrand Avenue",
          "employer" : "Quiltigen",
          "email" : "garciahess@quiltigen.com",
          "city" : "Brooktrails",
          "state" : "GA"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "222",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 222,
          "balance" : 14764,
          "firstname" : "Rachelle",
          "lastname" : "Rice",
          "age" : 36,
          "gender" : "M",
          "address" : "333 Narrows Avenue",
          "employer" : "Enaut",
          "email" : "rachellerice@enaut.com",
          "city" : "Wright",
          "state" : "AZ"
        }
      }
    ]
  }
}

query告诉我们查询什么

match_all是我们查询的类型

match_all查询仅仅在指定的索引的所有文件进行搜索

from，size

除了query参数外，还可以传递其他参数影响查询结果，比如前面提到的sort，接下来使用的size：

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match_all": {} },
> "size": 1
> }
> '
{
  "took" : 6,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "25",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 25,
          "balance" : 40540,
          "firstname" : "Virginia",
          "lastname" : "Ayala",
          "age" : 39,
          "gender" : "F",
          "address" : "171 Putnam Avenue",
          "employer" : "Filodyne",
          "email" : "virginiaayala@filodyne.com",
          "city" : "Nicholson",
          "state" : "PA"
        }
      }
    ]
  }
}

查询1条数据

指定位置与查询条数

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match_all": {} },
> "from": 0, 
> "size": 2
> }
> '
{
  "took" : 6,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "25",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 25,
          "balance" : 40540,
          "firstname" : "Virginia",
          "lastname" : "Ayala",
          "age" : 39,
          "gender" : "F",
          "address" : "171 Putnam Avenue",
          "employer" : "Filodyne",
          "email" : "virginiaayala@filodyne.com",
          "city" : "Nicholson",
          "state" : "PA"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "44",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 44,
          "balance" : 34487,
          "firstname" : "Aurelia",
          "lastname" : "Harding",
          "age" : 37,
          "gender" : "M",
          "address" : "502 Baycliff Terrace",
          "employer" : "Orbalix",
          "email" : "aureliaharding@orbalix.com",
          "city" : "Yardville",
          "state" : "DE"
        }
      }
    ]
  }
}

from 0表示从第1个开始

size 指定查询的个数

示例: 查询account_number从第501条到510条的数据：

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query": { "match_all": {} },
"from": 500,
"size": 10,
"sort": [
{ "account_number": "asc" }
]
}
' 2>/dev/null | grep account_number
          "account_number" : 500,
          "account_number" : 501,
          "account_number" : 502,
          "account_number" : 503,
          "account_number" : 504,
          "account_number" : 505,
          "account_number" : 506,
          "account_number" : 507,
          "account_number" : 508,
          "account_number" : 509,

匹配查询字段

返回_source字段中的片段字段

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match_all": {} },
> "_source": ["account_number", "balance"]
> }
> '
{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "25",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 25,
          "balance" : 40540
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "44",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 44,
          "balance" : 34487
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "99",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 99,
          "balance" : 47159
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "119",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 119,
          "balance" : 49222
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "126",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 126,
          "balance" : 3607
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "145",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 145,
          "balance" : 47406
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "183",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 183,
          "balance" : 14223
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "190",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 190,
          "balance" : 3150
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "208",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 208,
          "balance" : 40760
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "222",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 222,
          "balance" : 14764
        }
      }
    ]
  }
}

match

基本搜索查询，针对特定字段或字段集合进行搜索

查询编号为20的账户

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": { "match": { "account_number": 20 } }
> }
> '
{
  "took" : 11,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "20",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 20,
          "balance" : 16418,
          "firstname" : "Elinor",
          "lastname" : "Ratliff",
          "age" : 36,
          "gender" : "M",
          "address" : "282 Kings Place",
          "employer" : "Scentric",
          "email" : "elinorratliff@scentric.com",
          "city" : "Ribera",
          "state" : "WA"
        }
      }
    ]
  }
}

返回地址中包含mill的账户

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query": { "match": { "address": "mill" } }   
}
'
{
  "took" : 11,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 4.89784,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "472",
        "_score" : 4.89784,
        "_source" : {
          "account_number" : 472,
          "balance" : 25571,
          "firstname" : "Lee",
          "lastname" : "Long",
          "age" : 32,
          "gender" : "F",
          "address" : "288 Mill Street",
          "employer" : "Comverges",
          "email" : "leelong@comverges.com",
          "city" : "Movico",
          "state" : "MT"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "136",
        "_score" : 4.8485627,
        "_source" : {
          "account_number" : 136,
          "balance" : 45801,
          "firstname" : "Winnie",
          "lastname" : "Holland",
          "age" : 38,
          "gender" : "M",
          "address" : "198 Mill Lane",
          "employer" : "Neteria",
          "email" : "winnieholland@neteria.com",
          "city" : "Urie",
          "state" : "IL"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "970",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 970,
          "balance" : 19648,
          "firstname" : "Forbes",
          "lastname" : "Wallace",
          "age" : 28,
          "gender" : "M",
          "address" : "990 Mill Road",
          "employer" : "Pheast",
          "email" : "forbeswallace@pheast.com",
          "city" : "Lopezo",
          "state" : "AK"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "345",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 345,
          "balance" : 9812,
          "firstname" : "Parker",
          "lastname" : "Hines",
          "age" : 38,
          "gender" : "M",
          "address" : "715 Mill Avenue",
          "employer" : "Baluba",
          "email" : "parkerhines@baluba.com",
          "city" : "Blackgum",
          "state" : "KY"
        }
      }
    ]
  }
}

返回地址有包含mill或lane的所有账户

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
{
"query": { "match": { "address": "mill lane" } }      //空格就是或的关系
}
'
{
  "took" : 12,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 19,
    "max_score" : 8.398771,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "136",
        "_score" : 8.398771,
        "_source" : {
          "account_number" : 136,
          "balance" : 45801,
          "firstname" : "Winnie",
          "lastname" : "Holland",
          "age" : 38,
          "gender" : "M",
          "address" : "198 Mill Lane",
          "employer" : "Neteria",
          "email" : "winnieholland@neteria.com",
          "city" : "Urie",
          "state" : "IL"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "472",
        "_score" : 4.89784,
        "_source" : {
          "account_number" : 472,
          "balance" : 25571,
          "firstname" : "Lee",
          "lastname" : "Long",
          "age" : 32,
          "gender" : "F",
          "address" : "288 Mill Street",
          "employer" : "Comverges",
          "email" : "leelong@comverges.com",
          "city" : "Movico",
          "state" : "MT"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "556",
        "_score" : 4.4402957,
        "_source" : {
          "account_number" : 556,
          "balance" : 36420,
          "firstname" : "Collier",
          "lastname" : "Odonnell",
          "age" : 35,
          "gender" : "M",
          "address" : "591 Nolans Lane",
          "employer" : "Sultraxin",
          "email" : "collierodonnell@sultraxin.com",
          "city" : "Fulford",
          "state" : "MD"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "934",
        "_score" : 4.4402957,
        "_source" : {
          "account_number" : 934,
          "balance" : 43987,
          "firstname" : "Freida",
          "lastname" : "Daniels",
          "age" : 34,
          "gender" : "M",
          "address" : "448 Cove Lane",
          "employer" : "Vurbo",
          "email" : "freidadaniels@vurbo.com",
          "city" : "Snelling",
          "state" : "NJ"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "970",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 970,
          "balance" : 19648,
          "firstname" : "Forbes",
          "lastname" : "Wallace",
          "age" : 28,
          "gender" : "M",
          "address" : "990 Mill Road",
          "employer" : "Pheast",
          "email" : "forbeswallace@pheast.com",
          "city" : "Lopezo",
          "state" : "AK"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "345",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 345,
          "balance" : 9812,
          "firstname" : "Parker",
          "lastname" : "Hines",
          "age" : 38,
          "gender" : "M",
          "address" : "715 Mill Avenue",
          "employer" : "Baluba",
          "email" : "parkerhines@baluba.com",
          "city" : "Blackgum",
          "state" : "KY"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "908",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 908,
          "balance" : 45975,
          "firstname" : "Mosley",
          "lastname" : "Holloway",
          "age" : 31,
          "gender" : "M",
          "address" : "929 Eldert Lane",
          "employer" : "Anivet",
          "email" : "mosleyholloway@anivet.com",
          "city" : "Biehle",
          "state" : "MS"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "637",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 637,
          "balance" : 3169,
          "firstname" : "Kathy",
          "lastname" : "Carter",
          "age" : 27,
          "gender" : "F",
          "address" : "410 Jamison Lane",
          "employer" : "Limage",
          "email" : "kathycarter@limage.com",
          "city" : "Ernstville",
          "state" : "WA"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "449",
        "_score" : 4.3719764,
        "_source" : {
          "account_number" : 449,
          "balance" : 41950,
          "firstname" : "Barnett",
          "lastname" : "Cantrell",
          "age" : 39,
          "gender" : "F",
          "address" : "945 Bedell Lane",
          "employer" : "Zentility",
          "email" : "barnettcantrell@zentility.com",
          "city" : "Swartzville",
          "state" : "ND"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "742",
        "_score" : 4.3719764,
        "_source" : {
          "account_number" : 742,
          "balance" : 24765,
          "firstname" : "Merle",
          "lastname" : "Wooten",
          "age" : 26,
          "gender" : "M",
          "address" : "317 Pooles Lane",
          "employer" : "Tropolis",
          "email" : "merlewooten@tropolis.com",
          "city" : "Bentley",
          "state" : "ND"
        }
      }
    ]
  }
}

bool

bool must 查询的字段必须同时存在

查询包含mill和lane的所有账户

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": {
> "bool": {
> "must": [
> { "match": { "address": "mill" } },
> { "match": { "address": "lane" } }
> ]
> }
> }
> }
> '
{
  "took" : 14,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 8.398771,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "136",
        "_score" : 8.398771,
        "_source" : {
          "account_number" : 136,
          "balance" : 45801,
          "firstname" : "Winnie",
          "lastname" : "Holland",
          "age" : 38,
          "gender" : "M",
          "address" : "198 Mill Lane",
          "employer" : "Neteria",
          "email" : "winnieholland@neteria.com",
          "city" : "Urie",
          "state" : "IL"
        }
      }
    ]
  }
}

bool should 查询的字段仅存在一即可

查询包含mill或lane的所有账户

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": {
> "bool": {
> "should": [
> { "match": { "address": "mill" } },
> { "match": { "address": "lane" } }
> ]
> }
> }
> }
> '
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 19,
    "max_score" : 8.398771,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "136",
        "_score" : 8.398771,
        "_source" : {
          "account_number" : 136,
          "balance" : 45801,
          "firstname" : "Winnie",
          "lastname" : "Holland",
          "age" : 38,
          "gender" : "M",
          "address" : "198 Mill Lane",
          "employer" : "Neteria",
          "email" : "winnieholland@neteria.com",
          "city" : "Urie",
          "state" : "IL"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "472",
        "_score" : 4.89784,
        "_source" : {
          "account_number" : 472,
          "balance" : 25571,
          "firstname" : "Lee",
          "lastname" : "Long",
          "age" : 32,
          "gender" : "F",
          "address" : "288 Mill Street",
          "employer" : "Comverges",
          "email" : "leelong@comverges.com",
          "city" : "Movico",
          "state" : "MT"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "556",
        "_score" : 4.4402957,
        "_source" : {
          "account_number" : 556,
          "balance" : 36420,
          "firstname" : "Collier",
          "lastname" : "Odonnell",
          "age" : 35,
          "gender" : "M",
          "address" : "591 Nolans Lane",
          "employer" : "Sultraxin",
          "email" : "collierodonnell@sultraxin.com",
          "city" : "Fulford",
          "state" : "MD"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "934",
        "_score" : 4.4402957,
        "_source" : {
          "account_number" : 934,
          "balance" : 43987,
          "firstname" : "Freida",
          "lastname" : "Daniels",
          "age" : 34,
          "gender" : "M",
          "address" : "448 Cove Lane",
          "employer" : "Vurbo",
          "email" : "freidadaniels@vurbo.com",
          "city" : "Snelling",
          "state" : "NJ"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "970",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 970,
          "balance" : 19648,
          "firstname" : "Forbes",
          "lastname" : "Wallace",
          "age" : 28,
          "gender" : "M",
          "address" : "990 Mill Road",
          "employer" : "Pheast",
          "email" : "forbeswallace@pheast.com",
          "city" : "Lopezo",
          "state" : "AK"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "345",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 345,
          "balance" : 9812,
          "firstname" : "Parker",
          "lastname" : "Hines",
          "age" : 38,
          "gender" : "M",
          "address" : "715 Mill Avenue",
          "employer" : "Baluba",
          "email" : "parkerhines@baluba.com",
          "city" : "Blackgum",
          "state" : "KY"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "908",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 908,
          "balance" : 45975,
          "firstname" : "Mosley",
          "lastname" : "Holloway",
          "age" : 31,
          "gender" : "M",
          "address" : "929 Eldert Lane",
          "employer" : "Anivet",
          "email" : "mosleyholloway@anivet.com",
          "city" : "Biehle",
          "state" : "MS"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "637",
        "_score" : 4.388994,
        "_source" : {
          "account_number" : 637,
          "balance" : 3169,
          "firstname" : "Kathy",
          "lastname" : "Carter",
          "age" : 27,
          "gender" : "F",
          "address" : "410 Jamison Lane",
          "employer" : "Limage",
          "email" : "kathycarter@limage.com",
          "city" : "Ernstville",
          "state" : "WA"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "449",
        "_score" : 4.3719764,
        "_source" : {
          "account_number" : 449,
          "balance" : 41950,
          "firstname" : "Barnett",
          "lastname" : "Cantrell",
          "age" : 39,
          "gender" : "F",
          "address" : "945 Bedell Lane",
          "employer" : "Zentility",
          "email" : "barnettcantrell@zentility.com",
          "city" : "Swartzville",
          "state" : "ND"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "742",
        "_score" : 4.3719764,
        "_source" : {
          "account_number" : 742,
          "balance" : 24765,
          "firstname" : "Merle",
          "lastname" : "Wooten",
          "age" : 26,
          "gender" : "M",
          "address" : "317 Pooles Lane",
          "employer" : "Tropolis",
          "email" : "merlewooten@tropolis.com",
          "city" : "Bentley",
          "state" : "ND"
        }
      }
    ]
  }
}

range

指定区间内的数字或者时间

操作符：gt大于，gte大于等于，lt小于，lte小于等于

查询余额大于或等于20000且小于等于30000的账户

复制代码 
[root@vm2 ~]# curl -X GET "192.168.100.20:9200/bank/_search?pretty" -H 'Content-Type:application/json' -d'
> {
> "query": {
> "bool": {
> "must": { "match_all": {} },
> "filter": {
> "range": {
> "balance": {
> "gte": 20000,
> "lte": 30000
> }
> }
> }
> }
> }
> }
> '
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 217,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "253",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 253,
          "balance" : 20240,
          "firstname" : "Melissa",
          "lastname" : "Gould",
          "age" : 31,
          "gender" : "M",
          "address" : "440 Fuller Place",
          "employer" : "Buzzopia",
          "email" : "melissagould@buzzopia.com",
          "city" : "Lumberton",
          "state" : "MD"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "400",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 400,
          "balance" : 20685,
          "firstname" : "Kane",
          "lastname" : "King",
          "age" : 21,
          "gender" : "F",
          "address" : "405 Cornelia Street",
          "employer" : "Tri@Tribalog",
          "email" : "kaneking@tri@tribalog.com",
          "city" : "Gulf",
          "state" : "VT"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "520",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 520,
          "balance" : 27987,
          "firstname" : "Brandy",
          "lastname" : "Calhoun",
          "age" : 32,
          "gender" : "M",
          "address" : "818 Harden Street",
          "employer" : "Maxemia",
          "email" : "brandycalhoun@maxemia.com",
          "city" : "Sidman",
          "state" : "OR"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "645",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 645,
          "balance" : 29362,
          "firstname" : "Edwina",
          "lastname" : "Hutchinson",
          "age" : 26,
          "gender" : "F",
          "address" : "892 Pacific Street",
          "employer" : "Essensia",
          "email" : "edwinahutchinson@essensia.com",
          "city" : "Dowling",
          "state" : "NE"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "734",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 734,
          "balance" : 20325,
          "firstname" : "Keri",
          "lastname" : "Kinney",
          "age" : 23,
          "gender" : "M",
          "address" : "490 Balfour Place",
          "employer" : "Retrotex",
          "email" : "kerikinney@retrotex.com",
          "city" : "Salunga",
          "state" : "PA"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "784",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 784,
          "balance" : 25291,
          "firstname" : "Mabel",
          "lastname" : "Thornton",
          "age" : 21,
          "gender" : "M",
          "address" : "124 Louisiana Avenue",
          "employer" : "Zolavo",
          "email" : "mabelthornton@zolavo.com",
          "city" : "Lynn",
          "state" : "AL"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "880",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 880,
          "balance" : 22575,
          "firstname" : "Christian",
          "lastname" : "Myers",
          "age" : 35,
          "gender" : "M",
          "address" : "737 Crown Street",
          "employer" : "Combogen",
          "email" : "christianmyers@combogen.com",
          "city" : "Abrams",
          "state" : "OK"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "14",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 14,
          "balance" : 20480,
          "firstname" : "Erma",
          "lastname" : "Kane",
          "age" : 39,
          "gender" : "F",
          "address" : "661 Vista Place",
          "employer" : "Stockpost",
          "email" : "ermakane@stockpost.com",
          "city" : "Chamizal",
          "state" : "NY"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "19",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 19,
          "balance" : 27894,
          "firstname" : "Schwartz",
          "lastname" : "Buchanan",
          "age" : 28,
          "gender" : "F",
          "address" : "449 Mersereau Court",
          "employer" : "Sybixtex",
          "email" : "schwartzbuchanan@sybixtex.com",
          "city" : "Greenwich",
          "state" : "KS"
        }
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "204",
        "_score" : 1.0,
        "_source" : {
          "account_number" : 204,
          "balance" : 27714,
          "firstname" : "Mavis",
          "lastname" : "Deleon",
          "age" : 39,
          "gender" : "F",
          "address" : "400 Waldane Court",
          "employer" : "Lotron",
          "email" : "mavisdeleon@lotron.com",
          "city" : "Stollings",
          "state" : "LA"
        }
      }
    ]
  }
}

elasticsearch-head

elasticsearch-head是集群管理、数据可视化、增删改查、查询语句可视化工具。从ES5版本后安装方式

和ES2以上的版本有很大的不同,在ES2中可以直接在bin目录下执行plugin install xxxx 来进行安装,但是

在ES5中这种安装方式变了,要想在ES5中安装Elasticsearch Head必须要安装NodeJs,然后通过NodeJS来

启动Head。

官网地址:https://github.com/mobz/elasticsearch-head

elasticsearch-head安装

下载相关软件并上传到服务器

官网有安装说明,可以通过git安装,也可以下载zip包解压安装

下载相应的软件包,并拷贝到ES集群的一个节点上(我这里拷贝到192.168.100.20这台,也就是vm2上)

安装nodejs（注：这里是上传）

复制代码 
[root@vm2 ~]# ls
accounts.json    Documents                initial-setup-ks.cfg            Pictures   Videos
anaconda-ks.cfg  Downloads                Music                           Public
Desktop          elasticsearch-6.5.2.rpm  node-v10.24.1-linux-x64.tar.xz  Templates
[root@vm2 ~]# tar xf node-v10.24.1-linux-x64.tar.xz -C /usr/local/
[root@vm2 ~]# mv /usr/local/node-v10.24.1-linux-x64/ /usr/local/nodejs/
[root@vm2 ~]# ls /usr/local/nodejs/bin/npm
/usr/local/nodejs/bin/npm         //确认有此命令
[root@vm2 ~]# ln -s /usr/local/nodejs/bin/npm /bin/npm
[root@vm2 ~]# ln -s /usr/local/nodejs/bin/node /bin/node

安装es-head

安装方法1(需要网速好):

复制代码 
[root@vm2 ~]# cd /etc/yum.repos.d/
[root@vm2 yum.repos.d]# ls
CentOS-Base.repo  CentOS-Debuginfo.repo  CentOS-Media.repo    CentOS-Vault.repo
CentOS-CR.repo    CentOS-fasttrack.repo  CentOS-Sources.repo
[root@vm2 yum.repos.d]# rm -rf *
[root@vm2 yum.repos.d]# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2523  100  2523    0     0  10638      0 --:--:-- --:--:-- --:--:-- 10600
[root@vm2 yum.repos.d]# cd
[root@vm2 ~]# yum -y install git
.....

[root@vm2 ~]# git clone https://github.com/mobz/elasticsearch-head.git
Cloning into 'elasticsearch-head'...
remote: Enumerating objects: 4377, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 4377 (delta 12), reused 34 (delta 12), pack-reused 4337 (from 1)
Receiving objects: 100% (4377/4377), 2.54 MiB | 696.00 KiB/s, done.
Resolving deltas: 100% (2429/2429), done.
[root@vm2 ~]# cd elasticsearch-head/

先使用npm安装grunt
npm(node package manager):node包管理工具,类似yum
Grunt是基于Node.js的项目构建工具
[root@vm2 elasticsearch-head]# npm install -g grunt-cli
/usr/local/nodejs/bin/grunt -> /usr/local/nodejs/lib/node_modules/grunt-cli/bin/grunt
+ grunt-cli@1.5.0
added 56 packages from 68 contributors in 19.056s

安装时间较久,还会在网上下载phantomjs包
[root@vm2 elasticsearch-head]# npm install

安装可能有很多错误,我这里出现了下面的错误(重点是注意红色的ERR!,黄色的WARN不用管)

解决方法：

复制代码 
[root@vm2 elasticsearch-head]# npm install phantomjs-prebuilt@2.1.16 --ignorescript
此命令执行后不用再返回去执行npm install了，直接开始启动

[root@vm2 elasticsearch-head]# nohup npm run start &
[5] 63180

安装方法二 : (网速特别慢导致安装时间过长的话可以尝试以下方法)

复制代码 
git clone慢的话就使用下载好的zip压缩包解压安装
[root@vm2 ~]# unzip elasticsearch-head-master.zip -d /usr/local/
[root@vm2 ~]# mv /usr/local/elasticsearch-head-master/ /usr/local/es-head/
[root@vm2 ~]# cd /usr/local/es-head/
[root@vm2 es-head]# npm install -g grunt-cli --
registry=http://registry.npm.taobao.org

[root@vm2 es-head]# npm install --registry=http://registry.npm.taobao.org
当安装出现下载phantomjs软件包特别慢的时候，可以ctrl+c取消，拷贝下载好的phantomjs包到特定位置
再重新安装
[root@vm2 es-head]# cp phantomjs-2.1.1-linux-x86_64.tar.bz2 /tmp/phantomjs/
注意：phantomjs请改成自己的绝对路径
[root@vm2 es-head]# npm install --registry=http://registry.npm.taobao.org

[root@vm2 es-head]# nohup npm run start &

注意: 运行 nohup npm run start & 必须要先cd到es-head的目录

第4步:浏览器访问

浏览器访问 http://es-head节点IP:9100 ,并在下面的地址里把localhost改为es-head节点IP(浏览器与

es-head不是同一节点就要做)

第5步: 修改ES集群配置文件,并重启服务：

复制代码 
[root@vm1 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-cluster

node.name: 192.168.100.10
node.master: false

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0

http.port: 9200

discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]

http.cors.enabled: true
http.cors.allow-origin: "*"      //加上最后这两句
~  

[root@vm2 ~]# vim /etc/elasticsearch/elasticsearch.yml 
.....
cluster.name: elk-cluster

node.name: 192.168.100.20
node.master: true

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0

http.port: 9200

discovery.zen.ping.unicast.hosts: ["192.168.100.10", "192.168.100.20"]

http.cors.enabled: true
http.cors.allow-origin: "*"     //加上最后这两句
~  

[root@vm1 ~]# systemctl restart elasticsearch

[root@vm2 ~]# systemctl restart elasticsearch

第6步: 再次连接就可以看到信息了

新建个索引试试

删除此索引

es-head查询验证：

logstash简介

logstash是一个开源的数据采集工具,通过数据源采集数据.然后进行过滤,并自定义格式输出到目的地。

数据分为:

  1. 结构化数据 如:mysql数据库里的表等
  2. 半结构化数据 如: xml,yaml,json等
  3. 非结构化数据 如:文档,图片,音频,视频等
    logstash可以采集任何格式的数据,当然我们这里主要是讨论采集系统日志,服务日志等日志类型数据

官方产品介绍:https://www.elastic.co/cn/products/logstash

input插件: 用于导入日志源 (配置必须)

https://www.elastic.co/guide/en/logstash/current/input-plugins.html

filter插件: 用于过滤(不是配置必须的)https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

output插件: 用于导出(配置必须)

https://www.elastic.co/guide/en/logstash/current/output-plugins.html

logstash部署

在logstash服务器上确认openjdk安装：

复制代码 
[root@vm3 ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

在logstash服务器上安装logstash：（注：这里是上传）：

复制代码 
[root@vm3 ~]# rz -E
rz waiting to receive.

[root@vm3 ~]# rpm -ivh logstash-6.5.2.rpm 
warning: logstash-6.5.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.5.2-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

配置logstash主配置文件：

复制代码 
[root@vm3 ~]# vim /etc/logstash/logstash.yml 
......
path.data: /var/lib/logstash

path.config: /etc/logstash/conf.d     //打开注释,并加上配置目录路径

http.host: "192.168.100.30"   //打开注释,并改为本机IP(这是用于xpack监控用,但要收费,所以在这里不配
置也可以)

path.logs: /var/log/logstash

启动测试：

复制代码 
[root@vm3 ~]# cd /usr/share/logstash/bin
使用下面的空输入和空输出启动测试一下
[root@vm3 bin]# ./logstash -e 'input {stdin {}} output {stdout {}}'
运行后，输入字符将被stdout做为标准输出内容输出
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2025-10-23 19:52:30.811 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2025-10-23 19:52:30.820 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.5.2"}
[INFO ] 2025-10-23 19:52:32.668 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2025-10-23 19:52:32.795 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x59e19f73 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2025-10-23 19:52:32.835 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2025-10-23 19:52:32.982 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
abc
{
      "@version" => "1",
       "message" => "abc",
    "@timestamp" => 2025-10-23T11:52:41.921Z,
          "host" => "vm3.example.com"
}
123
{
      "@version" => "1",
       "message" => "123",
    "@timestamp" => 2025-10-23T11:52:46.314Z,
          "host" => "vm3.example.com"
}

关闭启动

测试能启动成功后,ctrl+c取消,则关闭了

另一种验证方法：

复制代码 
[root@vm3 ~]# vim /etc/logstash/conf.d/test.conf
input {
        stdin {
        }
}
filter {
}
output {
        stdout {
                codec => rubydebug
        }
}
~ 
[root@vm3 ~]# cd /usr/share/logstash/bin
[root@vm3 bin]# pwd
/usr/share/logstash/bin
[root@vm3 bin]# ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/test.conf -t
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2025-10-23T20:15:12,379][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2025-10-23T20:15:13,314][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

--path.settings 指定logstash主配置文件目录
-f 指定片段配置文件
-t 测试配置文件是否正确
codec => rubydebug这句可写可不定,默认就是这种输出方式

[root@vm3 bin]# ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2025-10-23T20:17:04,902][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2025-10-23T20:17:04,912][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.2"}
[2025-10-23T20:17:04,926][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"236820fc-5c98-483f-8551-d4fd0e76598b", :path=>"/var/lib/logstash/uuid"}
[2025-10-23T20:17:06,416][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2025-10-23T20:17:06,478][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x16d5eb4d sleep>"}
The stdin plugin is now waiting for input:
[2025-10-23T20:17:06,503][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2025-10-23T20:17:06,659][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
haha
{
    "@timestamp" => 2025-10-23T12:18:11.474Z,
      "@version" => "1",
       "message" => "haha",
          "host" => "vm3.example.com"
}
hehe
{
    "@timestamp" => 2025-10-23T12:18:18.993Z,
      "@version" => "1",
       "message" => "hehe",
          "host" => "vm3.example.com"
}

-r参数很强大,会动态装载配置文件,也就是说启动后,可以不用重启修改配置文件

日志采集

采集messages日志

这里以/var/log/messages为例,只定义input输入和output输出,不考虑过滤

复制代码 
[root@vm3 bin]# vim /etc/logstash/conf.d/test.conf 
input {
    file {
        path => "/var/log/messages"
        start_position => "beginning"
    }
}       
output {
    elasticsearch{
        hosts => ["192.168.100.20:9200"]
        index => "test-%{+YYYY.MM.dd}"
    }           
}               
~  
[root@vm3 bin]# ./logstash --path.settings /etc/logstash/ -r -f /etc/logstash/conf.d/test.conf &
[4] 11335

后台运行如果要杀掉,请使用pkill java或ps查看PID再kill -9清除

通过浏览器访问es-head验证 ：

请自行练习验证:

1, 在logstash那台服务器上做一些操作(比如,重启下sshd服务), 让/var/log/message有新的日志信息，

然后验证es-head里的数据

结果: 会自动更新, 浏览器刷新就能在es-head上看到更新的数据

2, kill掉logstash进程(相当于关闭), 也做一些操作让/var/log/message日志有更新，然后再次启动

logstash

结果: 会自动连上es集群, es-head里也能查看到数据的更新

采集多日志源

复制代码 
[root@vm3 bin]# vim /etc/logstash/conf.d/test.conf 
input {
        file {
                path => "/var/log/messages"
                start_position => "beginning"
                type => "messages"
        }
        file {
                path => "/var/log/yum.log"
                start_position => "beginning"
                type => "yum"
        }
}
filter {
}
output {
        if [type] == "messages" {
        elasticsearch {
                hosts => ["192.168.100.20:9200","192.168.100.10:9200"]
                index => "messages-%{+YYYY-MM-dd}"
                }
        }
        if [type] == "yum" {
        elasticsearch {
                hosts => ["192.168.100.20:9200","192.168.100.10:9200"]
                index => "yum-%{+YYYY-MM-dd}"
                }
        }
}
~
相关推荐
半梦半醒*5 小时前
ELK1——elasticsearch
linux·运维·数据仓库·elasticsearch·centos
妹妹够啦5 小时前
宝塔部署-Nginx配置
运维·nginx·junit
love530love6 小时前
【笔记】解决 ComfyUI 安装节点 ComfyUI-Addoor （葵花宝典）后启动报错：No module named ‘ComfyUI-Addoor’
linux·运维·前端·人工智能·windows·笔记·python
Shelley种了一只橙子6 小时前
第一章部署 chrony服务器
运维·服务器
Full Stack Developme6 小时前
Linux 有哪些功能相似的命令
linux·运维·服务器
馨谙6 小时前
Linux 服务管理：重新加载 vs 重新启动的本质区别
linux·运维·服务器
岚天start7 小时前
Linux内核coredump分析方案
linux·运维·服务器·gdb·coredump·堆栈·内存快照
屁股割了还要学7 小时前
【Linux入门】常用工具：yum、vim
linux·运维·服务器·c语言·c++·学习·考研
云计算练习生7 小时前
linux shell编程实战 03 数组：批量处理数据
linux·运维·服务器·数组·shell编程
热门推荐
01BongoCat - 跨平台键盘猫动画工具02GitHub 镜像站点03UV安装并设置国内源04GitLab 零基础入门指南：从安装到项目管理全流程05Linux下V2Ray安装配置指南06NVIDIA显卡驱动、CUDA、cuDNN 和 TensorRT 版本匹配指南07Labelme从安装到标注：零基础完整指南082025软件测试面试八股文（含答案+文档）09在VSCode配置Java开发环境的保姆级教程（适配各类AI编程IDE）10jdk21下载、安装（Windows、Linux、macOS）