ElK 8 收集 Nginx 日志

ygqygq22024-09-16 15:44

1. 说明

elk 版本：8.15.0

2. 启个 nginx

有 nginx 可以直接使用。我这里是在之前环境下 docker-compose.yml 中启动了个 nginx：

yaml 复制代码 
nginx:
  restart: always
  image: nginx:1.26.1
  ports:
    - "80:80"
    - "443:443"
  volumes:
    #- ./nginx/html:/usr/share/nginx/html
    - ./nginx/logs:/var/log/nginx
    #- ./nginx/certs:/etc/nginx/certs
    - ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf
    - ./nginx/conf/conf.d:/etc/nginx/conf.d

刚开始如果没有 nginx 配置，可以先不映射配置目录和文件，先启动，然后 docker cp 把配置和目录拷贝出来，再挂载进去。

配置 nginx/conf/conf.d/default.conf：

复制代码 
#
upstream kibana_servers {
    server kibana:5601 weight=1 max_fails=3 fail_timeout=30s;
}

server{
    listen 80;
    server_name localhost;
    # root /usr/local/nginx/html ;
    # rewrite ^(.*) https://$server_name$1 permanent;

    location / {
        proxy_pass http://kibana_servers/;
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_connect_timeout  30s;
        proxy_read_timeout     90s;
        proxy_send_timeout     90s;
        client_max_body_size   50m;
        client_body_buffer_size 16k;  # 默认是 8k 或 16k，可以根据需要调整
    }
}

3. filebeat 接入到 es

根据指引 filebeat 直接接入到 es。

filebeat 安装参考 《Docker compose 安装 ELK》中 "6. 安装 filebeat"。

filebeat 的目录下，设置 nginx 日志路径 modules.d/nginx.yml：

yaml 复制代码 
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/data/docker/elk/nginx/logs/access.log*"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/data/docker/elk/nginx/logs/error.log*"]

主配置 filebeat.yml：

yaml 复制代码 
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

output.elasticsearch:
  hosts: ["https://10.1.205.165:9200"]
  username: "filebeat_writer"
  password: "YOUR_PASSWORD"
  ssl:
    enabled: true
    # fingerprint=$(openssl x509 -fingerprint -sha256 -noout -in certs/ca/ca.crt | awk -F"=" '{print $2}' | sed 's/://g')
    ca_trusted_fingerprint: "33CB5A3B3ECCA59FDF7333D9XXXXXXXXFD34D5386FF9205AB8E1"
    # certs/ca 目录从 es 中拷过来
    certificate_authorities: ["certs/ca/ca.crt"]

# output.logstash:
#   hosts: ["10.1.205.165:5044", "10.1.205.166:5044"]

setup.kibana:
  host: "10.1.205.165:5601"

logging.level: warning

Dev Tools 创建角色：

复制代码 
PUT /_security/role/filebeat_writer_role
{
  "cluster": [
    "monitor"
  ],
  "indices": [
    {
      "names": [
        "filebeat*"
      ],
      "privileges": [
        "create_doc",
        "create",
        "delete",
        "index",
        "write",
        "all",
        "auto_configure",
        "manage"
      ],
      "allow_restricted_indices": false
    }
  ],
  "applications": [],
  "run_as": [],
  "metadata": {},
  "transient_metadata": {
    "enabled": true
  }
}

Dev Tools 创建用户：

复制代码 
POST /_security/user/filebeat_writer
{
  "password" : "YOUR_PASSWORD",
  "roles" : [ "filebeat_writer_role" ],
  "full_name" : "Filebeat Writer",
  "email" : "filebeat_writer@example.com",
  "metadata" : {}
}

启用模块：

bash 复制代码 
./filebeat modules enable nginx
./filebeat modules enable all-filesets

#
./filebeat setup
./filebeat -e

成功后，在 kibana 的 nginx log dashboard 中可以看到数据

4. filebeat 接入 logstash

主配置 filebeat.yml：

yaml 复制代码 
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

# output.elasticsearch:
#   hosts: ["https://10.1.205.165:9200"]
#   username: "filebeat_internal"
#   password: "YOUR_PASSWORD"
#   ssl:
#     enabled: true
#     # fingerprint=$(openssl x509 -fingerprint -sha256 -noout -in certs/ca/ca.crt | awk -F"=" '{print $2}' | sed 's/://g')
#     ca_trusted_fingerprint: "33CB5A3B3ECCA59FDF7333D9XXXXXXXXFD34D5386FF9205AB8E1"
#     # certs/ca 目录从 es 中拷过来
#     certificate_authorities: ["certs/ca/ca.crt"]

output.logstash:
  hosts: ["10.1.205.165:5044", "10.1.205.166:5044"]

setup.kibana:
  host: "10.1.205.165:5601"

logging.level: warning

有问题的配置我就不贴了，最终配置在下面，注释有说明解决问题的关键。

启动 filebeat 后，在 kibana Dev Tools 中查询：

复制代码 
GET filebeat-*/_search
{
  "size": 1,
  "sort": [
    {}
  ]
}

结果发现 ingest pipeline 处理异常，经过一段时间排查，找到了 相同问题链接

然后下面是最终的可用 Logstash Pipeline 设置 logstash/pipeline/filebeat.conf：

yaml 复制代码 
#
input {
  beats {
    port => 5044
  }
}

filter {
  if [@metadata][pipeline] {
    # 这里解决上面 `[event][original]` 已存在问题
    ruby {
      code => "event.remove('[event][original]')"
    }
  } else if [log_type] == "k8s" {
    # 将嵌套字段提升为顶层字段
    mutate {
      rename => {
        "[kubernetes][namespace]" => "kubernetes_namespace"
        "[kubernetes][node][name]" => "kubernetes_node_name"
        "[kubernetes][pod][ip]" => "kubernetes_pod_ip"
        "[kubernetes][pod][name]" => "kubernetes_pod_name"
        "[container][runtime]" => "container_runtime"
      }
    }

    # 使用 prune 过滤器保留所需字段
    prune {
      whitelist_names => ["@timestamp", "log_type", "container_runtime", "kubernetes_namespace", "kubernetes_node_name", "kubernetes_pod_ip", "kubernetes_pod_name", "message"]
    }
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]
      user => "logstash_writer"
      password => "logstash_password"
      ssl_enabled => true
      ssl_certificate_authorities => "/usr/share/logstash/certs/ca/ca.crt"
      action => "create"
      pipeline => "%{[@metadata][pipeline]}"
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      manage_template => false
    }
  } else if [log_type] == "k8s" {
    elasticsearch {
      hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]
      index => "k8s-%{+YYYY.MM.dd}"
      user => "logstash_writer"
      password => "logstash_password"
      ssl_enabled => true
      ssl_certificate_authorities => "/usr/share/logstash/certs/ca/ca.crt"
    }
  } else {
    # 默认索引
    elasticsearch {
      hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]
      index => "logstash-%{+YYYY.MM.dd}"
      user => "logstash_writer"
      password => "logstash_password"
      ssl_enabled => true
      ssl_certificate_authorities => "/usr/share/logstash/certs/ca/ca.crt"
      action => "create"
      manage_template => false
    }
  }
}

logstash 角色和用户权限同上面 filebeat 角色和用户。

5. 总结

Filebeat 收集 nginx、mysql 等日志，没有特别需求，最好直接接入 ES 中，少了一层 logstash 其实性能更好，问题也相对较少，官方文档推荐的。

参考资料：

1\] https://www.elastic.co/guide/en/logstash/8.15/use-ingest-pipelines.html \[2\] https://www.elastic.co/guide/en/elasticsearch/reference/8.15/ingest.html \[3\] https://www.elastic.co/guide/en/beats/filebeat/8.15/filebeat-installation-configuration.html \[4\] https://discuss.elastic.co/t/aws-ingest-pipeline-error-in-processor-rename-message-to-event-original/341472/8 \[5\] https://www.elastic.co/guide/en/ecs-logging/overview/master/intro.html

相关推荐
4***17545 小时前
Nginx auth_request详解
运维·nginx·xcode
꒰ঌ小武໒꒱5 小时前
RuoYi-Vue 前端环境搭建与部署完整教程
前端·javascript·vue.js·nginx
F***E2398 小时前
Nginx实现接口复制
运维·nginx·junit
可可苏饼干11 小时前
ELK（Elastic Stack）日志采集与分析
linux·运维·笔记·elk
通明湖13 小时前
如何从Ingress NGINX无缝迁移到OpenNJet KIC
nginx
大大大水蜜桃13 小时前
Nginx HTTPS服务搭建实验
运维·nginx·https
BugShare13 小时前
粗心大意必酿大祸，记录nginx配置文件的一次闹剧
运维·nginx
岁岁种桃花儿15 小时前
一文了解什么是http dns对比传统dns有哪些优势？
nginx·dns
徐同保18 小时前
Nginx不能转发带有websocket功能的项目解决方案
运维·websocket·nginx
Swizard19 小时前
Nginx 反向代理实战指南：从动态解析到 WebSocket
websocket·nginx·反代
热门推荐
01GitHub 镜像站点02【超详细教程】手把手教你从微软官网免费下载Windows 10官方原版ISO镜像（2025最新版）03UV安装并设置国内源04React CVE-2025-55182漏洞排查与修复指南05安娜的档案(Anna’s Archive) 镜像网站/国内最新可访问入口（持续更新）06BongoCat - 跨平台键盘猫动画工具07本地部署阿里最新开源的Z-Image08Linux下V2Ray安装配置指南09Labelme从安装到标注：零基础完整指南10在VSCode配置Java开发环境的保姆级教程（适配各类AI编程IDE）