logstash采集springboot微服务日志

前面一篇文章介绍了logstash采集nginx日志,这次介绍一下通过logstash采集springboot微服务中指定格式的日志。

需求

现有日志示例格式如下:

bash 复制代码
2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}
2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}
2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}

需要采集"logstashVO:"之后的JSON格式字符串,该功能主要通过网关服务收集用户访问行为,当然根据用户需求可以做的事情还可以很多,这里就不一一列举。

logstash配置

bash 复制代码
input {
    file {
        path => ["/data/logs/spring.log"]
        type => "test"
        start_position => "beginning"
    }
}
filter {
    if "c.test.authgateway.filter.GateWayFilter  : logstashVO:" not in [message] { 
        drop {}
    }
    ruby {
        code => "
            event.set('@timestamp', LogStash::Timestamp.at(event.get('@timestamp').time.localtime + 8*60*60))
        "
    }
    grok {
        match => { 
           "message" => "c\.test\.authgateway\.filter\.GateWayFilter  \: logstashVO\:%{GREEDYDATA:logstashVO}"
        }
    }
    json {
        source => "logstashVO"
    }
    json {
        source => "body"
        target => "body"
    }
    mutate {
        remove_field => ["@version","event","host","logstashVO"]
    }   
}

output {
    stdout {
        codec => rubydebug
    }
    kafka {
        bootstrap_servers => "10.16.39.121:9092"
        topic_id => "test2-logs"
        codec => json
        acks => "all"
        retries => 5
        batch_size => 16384
    }
}

注意这里的输入input.file.path对应的路径一定要是镜像中的路径/data/logs/,而不能为宿主机里的路径/data/public/logstash/logs

启动镜像

bash 复制代码
docker run --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
--privileged=true --network=host --restart=always --name logstash \
-v /data/public/logstash/config/:/usr/share/logstash/config/ \
-v /data/public/logstash/logs/:/data/logs/ \
-d logstash:8.4.0

检查日志采集

启动后,可以通过Offset Explorer连接Kafka检查是否正确采集到了相关的日志。

测试

bash 复制代码
echo '2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}' >> /data/public/logstash/logs/spring.log

执行一下,可以看到新日志进入到Kafka中

写入Kafka中日志格式如下:

bash 复制代码
{
  "@timestamp": "2025-07-26T17:48:58.874280406Z",
  "body": {},
  "url": "/test/findUserInfo.do",
  "fullName": "TEST",
  "log": {
    "file": {
      "path": "/data/logs/spring.log"
    }
  },
  "type": "test",
  "time": 1753067797629,
  "userId": 1088479580471164928,
  "message": "2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{\"body\":\"{}\",\"fullName\":\"TEST\",\"time\":1753067797629,\"url\":\"/test/findUserInfo.do\",\"userId\":1088479580471164928}"
}
相关推荐
ArabySide14 小时前
【Spring Boot】深入浅出Spring Boot中的控制反转与依赖注入
java·spring boot·后端
shepherd11114 小时前
破局延时任务(上):为什么选择Spring Boot + DelayQueue来自研分布式延时队列组件?
java·spring boot·后端
zl97989915 小时前
SpringBoot-数据访问之JDBC
java·spring boot
后端小张16 小时前
【JAVA 进阶】SpringBoot集成Sa-Token权限校验框架深度解析
java·spring boot·spring·架构·sa-token·springboot·权限框架
lang2015092817 小时前
Spring Boot Actuator应用信息Application Information全解析
spring boot·后端·elasticsearch
paopaokaka_luck17 小时前
基于SpringBoot+Vue的DIY手工社预约管理系统(Echarts图形化、腾讯地图API)
java·vue.js·人工智能·spring boot·后端·echarts
计算机学姐21 小时前
基于微信小程序的高校班务管理系统【2026最新】
java·vue.js·spring boot·mysql·微信小程序·小程序·mybatis
落日漫游1 天前
Nginx负载均衡:高性能流量调度指南
网络安全·微服务
摇滚侠1 天前
Spring Boot 3零基础教程,WEB 开发 Thymeleaf 核心语法 笔记39
spring boot·笔记·后端·thymeleaf
九丶弟1 天前
SpringBoot的cache使用说明
java·spring boot·spring·cache