Filebeat日志采集全面指南
📋 今日内容预告
- Filebeat采集数据的原理
- Filebeat采集nginx日志
- Filebeat采集tomcat日志
- Filebeat的模块处理
- Filebeat采集docker日志
- Filebeat采集MySQL日志
- Kibana出图分析指标
🔍 Filebeat采集数据的原理
1. 环境准备
bash
[root@elk92 ~]# cat /etc/filebeat/config/03-log-to-console.yaml
filebeat.inputs:
- type: log
paths:
- /tmp/student.json
output.console:
pretty: true
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/03-log-to-console.yaml
2. 测试数据采集
bash
# -n参数会去掉换行符,filebeat不会写入没换行符的句子
[root@elk92 ~]# echo -n www >> /tmp/student.json
[root@elk92 ~]# echo -n .oldboyedu. >> /tmp/student.json
[root@elk92 ~]# echo com >> /tmp/student.json
# 只有最后一条有换行符的数据可以被filebeat采集
总结Filebeat采集数据原理
- 按行采集:Filebeat默认按行采集数据,如果数据没有换行,则该条数据无法采集到
- 有状态服务:Filebeat属于有状态服务,可以记录上一次采集数据的位置点信息
bash
# 查看采集状态信息
[root@elk92 ~]# tail -1 /var/lib/filebeat/registry/filebeat/log.json
{"k":"filebeat::logs::native::412038-64768","v":{"FileStateOS":{"inode":412038,"device":64768},"id":"native::412038-64768","source":"/tmp/student.json","offset":80,"ttl":-1,"prev_id":"","timestamp":[2061921379726,1756343480],"type":"log","identifier_name":"native"}}
# 验证inode和offset信息
[root@elk92 ~]# ll -i /tmp/student.json
412038 -rw-r--r-- 1 root root 80 Aug 28 09:09 /tmp/student.json
温馨提示
-
从头采集:如果想要从头采集Filebeat的源数据文件,可以删除数据目录
bash[root@elk92 ~]# rm -rf /var/lib/filebeat/ [root@elk92 ~]# filebeat -e -c /etc/filebeat/config/03-log-to-console.yaml -
慎用操作:实际工作中慎用删除数据目录,因为会造成数据的重复采集!
🌐 Filebeat采集nginx日志
1. 安装nginx
bash
[root@elk92 ~]# apt -y install nginx
[root@elk92 ~]# systemctl start nginx
[root@elk92 ~]# ss -ntl | grep 80
LISTEN 0 511 0.0.0.0:80 0.0.0.0:*
LISTEN 0 511 [::]:80 [::]:*
2. 访问测试
http://10.0.0.92/
3. 查看nginx访问日志
bash
[root@elk92 ~]# tail -f /var/log/nginx/access.log
10.0.0.1 - - [28/Aug/2025:09:52:04 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"
10.0.0.1 - - [28/Aug/2025:09:52:04 +0800] "GET /favicon.ico HTTP/1.1" 404 197 "http://10.0.0.92/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"
4. 编写Filebeat采集配置
bash
[root@elk92 ~]# cat /etc/filebeat/config/04-nginx-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/access.log*
output.elasticsearch:
hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
index: oldboyedu-nginx-access
setup.ilm.enabled: false
setup.template.name: "oldboyedu-nginx"
setup.template.pattern: "oldboyedu-nginx*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
5. 启动Filebeat
bash
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/04-nginx-to-es.yaml
6. Kibana查看数据
和昨天一样,索引模式查看之后,Discover查看message信息
🐱 Filebeat采集tomcat日志
1. 下载tomcat
bash
wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.10/bin/apache-tomcat-11.0.10.tar.gz
2. 解压并配置
bash
[root@elk92 ~]# tar xf apache-tomcat-11.0.10.tar.gz -C /usr/local/
[root@elk92 ~]# cat /etc/profile.d/tomcat.sh
#!/bin/bash
export TOMCAT_HOME=/usr/local/apache-tomcat-11.0.10
export JAVA_HOME=/usr/share/elasticsearch/jdk
export PATH=$PATH:$TOMCAT_HOME/bin:$JAVA_HOME/bin
[root@elk92 ~]# source /etc/profile.d/tomcat.sh
[root@elk92 ~]# startup.sh
Using CATALINA_BASE: /usr/local/apache-tomcat-11.0.8
Using CATALINA_HOME: /usr/local/apache-tomcat-11.0.8
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.8/temp
Using JRE_HOME: /usr/share/elasticsearch/jdk
Using CLASSPATH: /usr/local/apache-tomcat-11.0.8/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.8/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Tomcat started.
[root@elk92 ~]# ss -ntl | grep 8080
LISTEN 0 100 *:8080 *:*
3. 访问测试
http://10.0.0.92:8080/
4. 编写Filebeat采集配置
bash
[root@elk92 ~]# cat /etc/filebeat/config/05-tomcat-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /usr/local/apache-tomcat-11.0.10/logs/localhost_access_log.*.txt
output.elasticsearch:
hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
index: oldboyedu-tomcat-access
setup.ilm.enabled: false
setup.template.name: "oldboyedu-tomcat"
setup.template.pattern: "oldboyedu-tomcat*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/05-tomcat-to-es.yaml
5. Kibana查看数据
略,见视频
🔄 Filebeat多实例
解决filebeat进程冲突
实例二在后面添加 --path.data 参数指定不同的数据目录
1. 启动实例一
bash
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/04-nginx-to-es.yaml
2. 启动实例二
bash
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/05-tomcat-to-es.yaml --path.data /tmp/xixi
3. 查看进程信息
bash
[root@elk92 ~]# ps -ef | grep filebeat | grep -v grep
root 115420 1738 1 10:31 pts/1 00:00:00 /usr/share/filebeat/bin/filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat -e -c /etc/filebeat/config/04-nginx-to-es.yaml
root 115427 1629 1 10:31 pts/0 00:00:00 /usr/share/filebeat/bin/filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat -e -c /etc/filebeat/config/05-tomcat-to-es.yaml --path.data /tmp/xixi
# 查看实例信息
cat /tmp/xixi/registry/filebeat/log.json
温馨提示
-
多实例部署 :如果在工作中需要启动多个Filebeat实例,则可以使用
--path.data参数指定不同的数据目录,若目录不存在则会自动创建 -
停止服务 :如果工作中Filebeat停止可以使用kill命令操作来停止服务
bash[root@elk92 ~]# ps -ef | grep filebeat | grep -v grep root 115549 115510 0 10:33 pts/0 00:00:00 /usr/share/filebeat/bin/filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat -e -c /etc/filebeat/config/04-nginx-to-es.yaml root 115561 115510 2 10:33 pts/0 00:00:00 /usr/share/filebeat/bin/filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat -e -c /etc/filebeat/config/05-tomcat-to-es.yaml --path.data /tmp/xixi [root@elk92 ~]# kill -9 115549 115561
📊 Filebeat采集多种业务日志
可以将nginx和tomcat同时启动
bash
[root@elk92 ~]# cat /etc/filebeat/config/06-multiple_input-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/access.log*
tags: "nginx"
- type: log
paths:
- /usr/local/apache-tomcat-11.0.10/logs/localhost_access_log.*.txt
tags: "tomcat"
output.elasticsearch:
hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
# 根据条件判断,将数据写入不同的索引
indices:
# 指定索引的名称
- index: "oldboyedu-contains-nginx-%{+yyyy.MM.dd}"
# 指定匹配条件
when.contains:
tags: "nginx"
- index: "oldboyedu-contains-tomcat-%{+yyyy.MM.dd}"
when.contains:
tags: "tomcat"
setup.ilm.enabled: false
setup.template.name: "oldboyedu-contains"
setup.template.pattern: "oldboyedu-contains*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
2. 启动实例
bash
[root@elk92 ~]# rm -rf /var/lib/filebeat/
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/06-multiple_input-to-es.yaml
3. Kibana出图展示
相关索引模式为: "oldboyedu-contains-*"
🐳 Filebeat采集docker日志(93节点)
1. 部署docker
bash
[root@elk93 ~]# wget http://192.168.16.253/Resources/Docker/scripts/oldboyedu-autoinstall-docker-docker-compose.tar.gz
[root@elk93 ~]# tar xf oldboyedu-autoinstall-docker-docker-compose.tar.gz
[root@elk93 ~]# ./install-docker.sh i
2. 导入镜像
bash
[root@elk93 ~]# wget http://192.168.16.253/Resources/Docker/images/Nginx/oldboyedu-nginx-1.27.4-alpine.tar.gz
[root@elk93 ~]# docker load < oldboyedu-nginx-1.27.4-alpine.tar.gz
[root@elk93 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx 1.27.4-alpine 1ff4bb4faebc 6 months ago 47.9MB
3. 运行容器
bash
[root@elk93 ~]# docker run -d --name c1 -p 81:80 nginx:1.27.4-alpine
52361d96e587b76dd0d46241ea02d37c36ee1ffbf2349dce73bcb504604d1265
[root@elk93 ~]# docker run -d --name c2 -p 82:80 nginx:1.27.4-alpine
fbc65cde1c0f73f4b76d3028fcba29e351cf15aa5c429685285d3c6b449e29f4
[root@elk93 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fbc65cde1c0f nginx:1.27.4-alpine "/docker-entrypoint...." 4 seconds ago Up 2 seconds 0.0.0.0:82->80/tcp, :::82->80/tcp c2
52361d96e587 nginx:1.27.4-alpine "/docker-entrypoint...." 9 seconds ago Up 7 seconds 0.0.0.0:81->80/tcp, :::81->80/tcp c1
4. 安装Filebeat
bash
[root@elk92 ~]# scp filebeat-7.17.29-amd64.deb 10.0.0.93:~
[root@elk93 ~]# dpkg -i filebeat-7.17.29-amd64.deb
5. 编写Filebeat配置文件
bash
[root@elk93 ~]# cat /etc/filebeat/docker-to-es.yaml
filebeat.inputs:
- type: container
paths:
- '/var/lib/docker/containers/*/*.log'
# 添加处理器
processors:
# 添加docker的元数据信息
- add_docker_metadata:
# 找本地的套接字文件
host: "unix:///var/run/docker.sock"
output.elasticsearch:
hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
index: oldboyedu-docker
setup.ilm.enabled: false
setup.template.name: "oldboyedu-docker"
setup.template.pattern: "oldboyedu-docker*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 6
index.number_of_replicas: 0
6. 启动实例
bash
[root@elk93 ~]# filebeat -e -c /etc/filebeat/docker-to-es.yaml
7. Kibana查询
略,见视频
参考指标: container.name : "c1"
🚫 Filebeat排除不必要的行 exclude_lines
1. 修改配置文件
bash
[root@elk93 ~]# cat /etc/filebeat/docker-to-es.yaml
filebeat.inputs:
- type: container
paths:
- '/var/lib/docker/containers/*/*.log'
# 排除不必要的行
exclude_lines: ['.*notice','.*entrypoint','.*listen']
# 添加处理器
processors:
# 添加docker的元数据信息
- add_docker_metadata:
# 找本地的套接字文件
host: "unix:///var/run/docker.sock"
output.elasticsearch:
hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
index: oldboyedu-docker
setup.ilm.enabled: false
setup.template.name: "oldboyedu-docker"
setup.template.pattern: "oldboyedu-docker*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 6
index.number_of_replicas: 0
2. 启动Filebeat实例
bash
# 删除原有状态文件
[root@elk93 ~]# rm -rf /var/lib/filebeat/
[root@elk93 ~]# filebeat -e -c /etc/filebeat/docker-to-es.yaml
📈 Kibana分析容器的相关指标
1. 访问测试
bash
# 在elk91上访问c1容器
[root@elk91 ~]# for i in `seq 1000`;do curl 10.0.0.93:81;sleep 0.1;done
# 在elk92上访问c2容器
[root@elk92 ~]# for i in `seq 3000`;do curl 10.0.0.93:82;sleep 0.1;done
同时elk93要启动filebeat
2. Kibana统计指标
-
统计PV访问量

-
统计容器数量

🧩 Filebeat的模块启用和禁用原理验证
1. 查看Filebeat支持的模块列表
bash
[root@elk92 ~]# filebeat modules list
Enabled:
Disabled:
activemq
apache
auditd
aws
awsfargate
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
2. 启用模块
bash
[root@elk92 ~]# filebeat modules enable mysql nginx tomcat
Enabled mysql
Enabled nginx
Enabled tomcat
[root@elk92 ~]# filebeat modules list
Enabled:
mysql
nginx
tomcat
Disabled:
activemq
apache
auditd
aws
awsfargate
azure
3. 禁用模块
bash
[root@elk92 ~]# filebeat modules disable mysql tomcat
Disabled mysql
Disabled tomcat
[root@elk92 ~]# filebeat modules list
Enabled:
nginx
Disabled:
activemq
apache
auditd
aws
温馨提示
-
模块文件管理:Filebeat模块底层启用和禁用本质上对应的是模块文件列表
bash[root@elk92 ~]# ll /etc/filebeat/modules.d/ | wc -l 74 -
查看启用的模块:搜索yml结尾的文件
bash[root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml -rw-r--r-- 1 root root 784 Jun 19 00:32 /etc/filebeat/modules.d/nginx.yml -
手动管理模块:通过修改文件后缀名来启用/禁用
bash# 启用tomcat模块 [root@elk92 ~]# mv /etc/filebeat/modules.d/tomcat.yml{.disabled,} [root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml -rw-r--r-- 1 root root 784 Jun 19 00:32 /etc/filebeat/modules.d/nginx.yml -rw-r--r-- 1 root root 623 Jun 19 00:32 /etc/filebeat/modules.d/tomcat.yml [root@elk92 ~]# filebeat modules list Enabled: nginx tomcat Disabled: activemq apache auditd aws # 禁用tomcat模块 [root@elk92 ~]# mv /etc/filebeat/modules.d/tomcat.yml{,.disabled} [root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml -rw-r--r-- 1 root root 784 Jun 19 00:32 /etc/filebeat/modules.d/nginx.yml [root@elk92 ~]# filebeat modules list Enabled: nginx Disabled: activemq apache auditd aws
🔧 Filebeat模块应用案例
1. 启用模块
bash
[root@elk92 ~]# filebeat modules enable tomcat
Enabled tomcat
[root@elk92 ~]# filebeat modules list
Enabled:
nginx
tomcat
Disabled:
activemq
apache
auditd
aws
...
2. 修改nginx模块配置
bash
[root@elk92 ~]# egrep -v "^.*#|^$" /etc/filebeat/modules.d/nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access.log*"]
error:
enabled: true
var.paths: ["/var/log/nginx/error.log*"]
ingress_controller:
enabled: false
3. 修改tomcat模块配置
bash
[root@elk92 ~]# egrep -v "^.*#|^$" /etc/filebeat/modules.d/tomcat.yml
- module: tomcat
log:
enabled: true
var.input: file
var.paths:
- /usr/local/apache-tomcat-11.0.10/logs/localhost_access_log.*.txt
4. 修改Filebeat配置文件
bash
[root@elk92 ~]# cat /etc/filebeat/config/07-modules-to-es.yaml
# 启用Filebeat模块
filebeat.config.modules:
# 指定模块的路径
path: ${path.config}/modules.d/*.yml
# 是否支持热加载配置
reload.enabled: true
output.elasticsearch:
hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
index: oldboyedu-modules-xixi
setup.ilm.enabled: false
setup.template.name: "oldboyedu-modules"
setup.template.pattern: "oldboyedu-modules*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
5. 启动Filebeat实例
bash
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/07-modules-to-es.yaml
6. Kibana出图展示
略,见视频

📊 基于Filebeat的模块分析nginx访问日志
1. 下载测试文件
bash
[root@elk92 ~]# wget http://192.168.16.253/Resources/ElasticStack/softwares/access.log
2. 修改Filebeat模块配置
bash
[root@elk92 ~]# egrep -v "^.*#|^$" /etc/filebeat/modules.d/nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/root/access.log*"]
error:
enabled: false
var.paths: ["/var/log/nginx/error.log*"]
ingress_controller:
enabled: false
3. 编写Filebeat配置文件
bash
[root@elk92 ~]# cat /etc/filebeat/config/08-modules_nginx-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/nginx.yml
reload.enabled: true
output.elasticsearch:
hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
index: oldboyedu-modules-efk-nginx
setup.ilm.enabled: false
setup.template.name: "oldboyedu-modules"
setup.template.pattern: "oldboyedu-modules*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
4. 启动Filebeat实例
bash
[root@elk92 ~]# filebeat -e -c /etc/filebeat/config/08-modules_nginx-to-es.yaml
5. Kibana出图分析指标
📍 相关分析字段
source.ip- 访问源IPuser_agent.device.name- 用户设备类型user_agent.os.name- 操作系统http.response.body.bytes- 响应体大小http.response.status_code- HTTP状态码url.path- 访问路径source.geo.location- 地理位置(用于全球用户分布图)
📊 Kibana统计图表类型

-
全球用户分布图
- 相关字段:
source.geo.location
- 相关字段:
-
带宽统计
- 相关字段:
http.response.body.bytes


- 相关字段:
-
PV统计
- 相关字段:无,选择'计数'即可,注意索引别选错了
-
IP统计
- 相关字段:
source.ip,选择唯一计数,记得勾选另存为

- 相关字段:
-
设备类型统计
- 相关字段:
user_agent.device.name

- 相关字段:
-
操作系统占比统计
- 相关字段:
user_agent.os.name

- 相关字段:
-
访问业务占比统计
- 相关字段:
url.path

- 相关字段:
-
仪表盘Dashboard制作
- 将以上图表整合到一个仪表盘



- 将以上图表整合到一个仪表盘
🚨 ElasticStack故障排查思路

第一反应都是检查配置文件
-
内存占用问题
bashps aux | grep filebeat kill -9 PID -
未清空本地缓存(经常发生)
bashrm -f /var/lib/filebeat
📝 今日内容回顾
- filebeat采集数据的原理 *****
- filebeat采集nginx,tomcat日志 *****
- Filebeat的多实例 ***
- Filebeat采集多个业务日志 **
- Filebeat采集docker日志 *****
- Filebeat模块采集 *****
- kibana出图展示 *****
- 故障排查技巧 *****
📋 今日作业
基础作业
- 完成课堂的所有练习并整理思维导图
- Filebeat采集MySQL日志
扩展作业
使用Filebeat采集Linux的系统日志,包括如下系统日志写入ES不同的索引:
/var/log/syslog/var/log/auth.log/var/log/bootstrap.log