【Beats01】企业级日志分析系统ELK之Metricbeat与Heartbeat 监控

Beats 收集数据

Beats 是一个免费且开放的平台，集合了多种单一用途数据采集器。它们从成百上千或成千上万台机器和系统向 Logstash 或 Elasticsearch 发送数据。

虽然利用 logstash 就可以收集日志，功能强大，但由于 Logtash 是基于Java实现，需要在采集日志的主机上安装JAVA环境

logstash运行时最少也会需要额外的500M的以上的内存，会消耗比较多的内存和磁盘空间

可以采有基于Go开发的 Beat 工具代替 Logstash 收集日志，部署更为方便，而且只占用10M左右的内存空间及更小的磁盘空间。

利用 Metricbeat 监控性能相关指标

Metricbeat 可以收集指标数据，比如系统运行状态、CPU、内存利用率等。

生产中一般用 zabbix 等专门的监控系统实现此功能

官方配置说明

https://www.elastic.co/guide/en/beats/metricbeat/current/configuring-howtometricbeat.htmlhttps://www.elastic.co/guide/en/beats/metricbeat/current/configuring-howtometricbeat.html

下载 metricbeat 并安装

下载链接

https://www.elastic.co/cn/downloads/beats/metricbeat
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/8.x/apt/pool/main/m/metricbeat/
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/m/metricbeat/

范例:

#新版下载
[root@elk-web2 ~]#wget 
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/8.x/apt/pool/main/m/metricbeat/metricbeat-8.6.1-amd64.deb
#旧版下载
[root@elk-web2 ~]#wget 
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/m/metricbeat/metricbeat-7.6.2-amd64.deb
[root@elk-web2 ~]#dpkg -i metricbeat-7.6.2-amd64.deb

修改配置

[root@elk-web2 ~]#vim /etc/metricbeat/metricbeat.yml 
#setup.kibana:
#   host: "10.0.0.101:5601"  #指向kabana服务器地址和端口，非必须项，即使不设置Kibana也可以通过ES获取Metrics信息

#-------------------------- Elasticsearch output -----------------------------

output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.0.101:9200","10.0.0.102:9200","10.0.0.103:9200"]   #指向任意一个ELK集群节点即可


[root@kibana ~]#grep -Ev "#|^$" /etc/metricbeat/metricbeat.yml
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
  host: "localhost:5601"
output.elasticsearch:
  hosts:  ["10.0.0.181:9200","10.0.0.182:9200","10.0.0.183:9200"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

启动服务

[root@kibana ~]#systemctl enable --now metricbeat.service

[root@kibana ~]#systemctl status metricbeat.service

通过 Kibana 查看收集的性能指标

8.X版本界面

运行时间 --- invertory 库存

[Stack Management] ------- 数据视图

Discover

利用 Heartbeat 监控

heartbeat 用来定时探测服务是否正常运行。支持ICMP、TCP 和 HTTP，也支持TLS、身份验证和代理

官方heartbeat配置文档

https://www.elastic.co/guide/en/beats/heartbeat/current/configuring-howtoheartbeat.htmlhttps://www.elastic.co/guide/en/beats/heartbeat/current/configuring-howtoheartbeat.html

下载并安装

下载链接

https://www.elastic.co/cn/downloads/beats/heartbeat
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/8.x/apt/pool/main/h/heartbeat-elastic/
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/h/heartbeat-elastic/

#新版
[root@elk-web2 ~]#wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/8.x/apt/pool/main/h/heartbeat-elastic/heartbeat-8.6.1-amd64.deb
#旧版
[root@elk-web2 ~]#wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/apt/pool/main/h/heartbeat-elastic/heartbeat-7.6.2-amd64.deb
#安装
[root@elk-web2 ~]#dpkg -i heartbeat-7.6.2-amd64.deb
#准备需要监控的服务httpd
[root@elk-web2 ~]#apt -y install apache2

修改配置

官方参考

https://www.elastic.co/guide/en/beats/heartbeat/current/configuration-heartbeat-options.html

#官方示例 heartbeat.yml

heartbeat.monitors:

type: icmp

id: ping-myhost

name: My Host Ping

hosts: ["myhost"]

schedule: '*/5 * * * * * *' #相当于'@every 5s'

type: tcp

id: myhost-tcp-echo

name: My Host TCP Echo

hosts: ["myhost:777"] # default TCP Echo Protocol

check.send: "Check"

check.receive: "Check"

schedule: '@every 5s'

type: http

id: service-status

name: Service Status

service.name: my-apm-service-name

hosts: ["http://localhost:80/service/status"]

check.response.status: [200]

schedule: '@every 5s'

heartbeat.scheduler:

limit: 10

时间格式: 注意可以支持秒级精度

范例

[root@kibana ~]#vim /etc/heartbeat/heartbeat.yml
# Configure monitors inline
heartbeat.monitors:
- type: http
  enabled: true                 #修改此行false为true
  # List or urls to query
  urls: ["http://localhost:80/test.html"]  #修改此行，指向需要监控的服务器的地址和端口
  
  # Configure task schedule
  schedule: '@every 10s'
  
  # Total test connection and data exchange timeout
  timeout: 6s
  
- type: tcp
  id: myhost-tcp-echo
  name: My Host TCP Echo
  hosts: ["10.0.0.205:80"]  # default TCP Echo Protocol
  schedule: '@every 5s'
  
- type: icmp                   #添加下面5行，用于监控ICMP
  id: ping-myhost
  name: My Host Ping
  hosts: ["10.0.0.100"]
  schedule: '*/5 * * * * * *'
  
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "10.0.0.101:5601 #指向kibana服务器地址和端口
#-------------------------- Elasticsearch output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.0.0.181:9200","10.0.0.182:9200","10.0.0.183:9200"]  #修改此行，指向ELK集群服务器地址和端口

启动服务

[root@kibana ~]#systemctl enable --now heartbeat-elastic.service

head 插件查看索引

通过 Kibana 查看收集的性能指标

通过Kibana 可以看到主机的状态

新版

运行时间 --- 监测