ELK收集网络设备日志

分步走：

1、新建一台Linux虚拟机做为rsyslog服务器接收网络设备日志

2、在上述虚拟机上部署filebeat，将接收到的网络日志发送到logstash上

3、最后通过logstash针对性配置将日志发送给es并在kibana上显示。

1、部署rsyslog服务器

复制代码

1、新建rsyslog配置文件
[root@localhost ~]# vi /etc/rsyslog.d/10-network-output.conf
# 定义模板：按来源IP和日期分割日志文件
# 格式：/var/log/network-devices/<源IP>/<年-月-日>.log
template(name="NetworkDeviceLog" type="string"
    string="/home/network-devices/%fromhost-ip%/%$YEAR%-%$MONTH%-%$DAY%.log"
)

# 规则：将所有来自网络的日志（排除本地 localhost）写入上述模板
# 注意：这里假设网络设备日志没有特定的 facility 限制，如果有，可改为 local0.* 等
:fromhost-ip, !isequal, "127.0.0.1" ?NetworkDeviceLog

# 确保目录存在并设置权限
# mkdir -p /var/log/network-devices
# chown -R syslog:adm /var/log/network-devices  (CentOS可能是 root:root 或 syslogs:syslogs)

2、启动rsyslog服务，并检查端口514是否存在

复制代码

[root@localhost ~]# systemctl enable rsyslog
[root@localhost ~]# systemctl start rsyslog
[root@localhost ~]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2026-06-24 14:22:55 CST; 1h 45min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 28876 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─28876 /usr/sbin/rsyslogd -n

Jun 24 14:22:55 localhost.localdomain systemd[1]: Starting System Logging Service...
Jun 24 14:22:55 localhost.localdomain rsyslogd[28876]:  [origin software="rsyslogd" swVersion="8.24.0-55.el7" x-pid="28876" x-info="http://www.rsyslog.com"] start
Jun 24 14:22:55 localhost.localdomain systemd[1]: Started System Logging Service.

[root@localhost ~]# netstat -lntup |grep 514
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN      28876/rsyslogd
tcp6       0      0 :::514                  :::*                    LISTEN      28876/rsyslogd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           28876/rsyslogd
udp6       0      0 :::514                  :::*                                28876/rsyslogd

3、网络设备上设置策略将选定日志发送到rsyslog的514端口(可由网工配置)

4、检查是否有日志接收到

复制代码

[root@localhost home]# ll /home/network-devices/10.10.254.10/
total 32512
-rw-------. 1 root root 20922019 Jun 24 16:11 2026-06-24.log
有日志写入

5、在rsyslog上部署filebeat并修改配置文件

复制代码

部署过程忽略
[root@localhost home]# vim /opt/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  id: logs-app-network-devices                            #ID自定义
  enabled: true
  paths:
    - /home/network-devices/*/*.log    #日志路径自定义
  #tags: ["network-device"]
  encoding: utf-8

  fields:
    log_source: "network-devices"                  #系统-必填 区分索引
    log_source_env: "prod"              #环境
  fields_under_root: false
#  multiline:
#    pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}'
#    negate: true
#    match: after
#    max_lines: 1000
  scan_frequency: 10s
  close_inactive: 1m


output.logstash:
  hosts: ["10.10.200.35:8888"]
  compression_level: 3
  loadbalance: true
  bulk_max_size: 2048

logging.level: info
logging.to_files: true
logging.files:
  path: /opt/filebeat/logs            #目录自定义
  name: filebeat.log
  keepfiles: 7

6、转到logstash服务器，新建专门接收网络日志的conf

复制代码

[root@elk-lo-node03 pipeline]# pwd
/opt/logstash/config/pipeline
[root@elk-lo-node03 pipeline]# vim network-device-log.conf
input {
  beats {
    port => 8888   #与filebeat配置的发送端口一致
    # 可选：限制只接受来自特定 IP 的连接，增强安全性
    # host => "0.0.0.0"
  }
}

filter {
  # 1. 识别来源：根据 Filebeat 中定义的 fields.log_source 进行判断
  if [fields][log_source] == "network-devices" {

    # 2. 添加标签：便于在 Kibana 中快速筛选或创建专用视图
    mutate {
      add_tag => ["network_device_log"]
      add_field => { "[@metadata][target_index]" => "logs-app-network-devices" }
    }
  }

  # 其他来源的日志可以在这里添加 else if 分支处理
}

output {
  # 调试输出：确认数据结构和字段是否正确（生产环境稳定后可注释掉）
  stdout {
    codec => rubydebug
  }

  # 输出到 Elasticsearch
  elasticsearch {
    hosts => ["https://10.10.200.31:9200"]

    # 动态索引名称：
    # 最终生成的索引名为: logs-app-network-2026.06.24
    index => "%{[@metadata][target_index]}-%{+yyyy.MM.dd}"

    # 认证信息
    user => "elastic"
    password => "JcJv*N7rUT6fE6fik4oY"

    # SSL 配置（根据你的 ES 集群实际情况调整）
    ssl_certificate_verification => false

    # 模板管理（可选）：如果希望自动应用映射模板
    # manage_template => true
    # template_name => "network-log-template"
  }
}

7、因为pipeline包含多个conf，需要额外修改pipelines.yml注明

复制代码

[root@elk-lo-node03 config]# ll
total 48
-rw-r--r-- 1 root root  2924 Apr  1 17:49 jvm.options
-rw-r--r-- 1 root root  8680 Apr  1 17:49 log4j2.properties
-rw-r--r-- 1 root root   502 Jun  4 14:37 logstash.conf
-rw-r--r-- 1 root root   342 Apr  1 17:49 logstash-sample.conf
-rw-r--r-- 1 root root 15745 Apr  1 17:49 logstash.yml
drwxr-xr-x 2 root root    98 Jun 24 16:21 pipeline
-rw-r--r-- 1 root root   837 Jun 16 16:54 pipelines.yml
-rw-r--r-- 1 root root  1696 Apr  1 17:49 startup.options
[root@elk-lo-node03 config]# pwd
/opt/logstash/config
[root@elk-lo-node03 config]# vim pipelines.yml
- pipeline.id: beats-elk-log
  path.config: "/opt/logstash/config/pipeline/beats-elk-log.conf"
  pipeline.workers: 2
  pipeline.batch.size: 125
  pipeline.batch.delay: 50
  queue.type: persisted
  queue.max_bytes: 1gb
  queue.checkpoint.acks: 1024
  queue.drain: false

- pipeline.id: windows-winlogbeat-log
  path.config: "/opt/logstash/config/pipeline/windows-winlogbeat-log.conf"
  pipeline.workers: 2
  pipeline.batch.size: 125
  pipeline.batch.delay: 50
  queue.type: persisted
  queue.max_bytes: 1gb
  queue.checkpoint.acks: 1024
  queue.drain: false

- pipeline.id: network-device-log
  path.config: "/opt/logstash/config/pipeline/network-device-log.conf"
  pipeline.workers: 2
  pipeline.batch.size: 125
  pipeline.batch.delay: 50
  queue.type: persisted
  queue.max_bytes: 1gb
  queue.checkpoint.acks: 1024
  queue.drain: false

8、重启logstash，并新开窗口检查是否有接收到网络设备日志

复制代码

[root@elk-lo-node03 config]# systemctl restart logstash
[root@elk-lo-node03 config]# journalctl -u logstash.service  -f
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:        "message" => "Jun 24 16:23:29 10.10.254.10 5823434235018048(root) 43240501 HillstoneNetworks#Event@NET: ARP entry is created, 10.10.254.65, 8840.33e8.c0d7, trust-vr",
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:     "@timestamp" => 2026-06-24T08:16:47.271Z,
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:           "host" => {
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:         "name" => "localhost.localdomain"
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:     },
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:          "input" => {
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:         "type" => "log"
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:     },
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:            "ecs" => {
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:         "version" => "8.0.0"
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:     },
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:            "log" => {
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:         "offset" => 21896295,
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:           "file" => {
Jun 24 16:23:29 elk-lo-node03 logstash[193222]:             "path" => "/home/network-devices/10.10.254.10/2026-06-24.log"
检查看到日志被logstash成功接收到

9、kibana上检查索引管理上是否有自动创建索引(若没有，则说明需要修改logstash上的对应conf文件)

10、新建对应网络设备日志的数据视图

11、进入discover检查日志

以上，完成~！