logstash采集springboot微服务日志

前面一篇文章介绍了logstash采集nginx日志，这次介绍一下通过logstash采集springboot微服务中指定格式的日志。

需求

现有日志示例格式如下：

bash 复制代码

2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}
2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}
2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}

需要采集"logstashVO:"之后的JSON格式字符串，该功能主要通过网关服务收集用户访问行为，当然根据用户需求可以做的事情还可以很多，这里就不一一列举。

logstash配置

bash 复制代码

input {
    file {
        path => ["/data/logs/spring.log"]
        type => "test"
        start_position => "beginning"
    }
}
filter {
    if "c.test.authgateway.filter.GateWayFilter  : logstashVO:" not in [message] { 
        drop {}
    }
    ruby {
        code => "
            event.set('@timestamp', LogStash::Timestamp.at(event.get('@timestamp').time.localtime + 8*60*60))
        "
    }
    grok {
        match => { 
           "message" => "c\.test\.authgateway\.filter\.GateWayFilter  \: logstashVO\:%{GREEDYDATA:logstashVO}"
        }
    }
    json {
        source => "logstashVO"
    }
    json {
        source => "body"
        target => "body"
    }
    mutate {
        remove_field => ["@version","event","host","logstashVO"]
    }   
}

output {
    stdout {
        codec => rubydebug
    }
    kafka {
        bootstrap_servers => "10.16.39.121:9092"
        topic_id => "test2-logs"
        codec => json
        acks => "all"
        retries => 5
        batch_size => 16384
    }
}

注意这里的输入input.file.path对应的路径一定要是镜像中的路径/data/logs/，而不能为宿主机里的路径/data/public/logstash/logs

启动镜像

bash 复制代码

docker run --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 \
--privileged=true --network=host --restart=always --name logstash \
-v /data/public/logstash/config/:/usr/share/logstash/config/ \
-v /data/public/logstash/logs/:/data/logs/ \
-d logstash:8.4.0

检查日志采集

启动后，可以通过Offset Explorer连接Kafka检查是否正确采集到了相关的日志。

测试

bash 复制代码

echo '2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{"body":"{}","fullName":"TEST","time":1753067797629,"url":"/test/findUserInfo.do","userId":1088479580471164928}' >> /data/public/logstash/logs/spring.log

执行一下，可以看到新日志进入到Kafka中

写入Kafka中日志格式如下：

bash 复制代码

{
  "@timestamp": "2025-07-26T17:48:58.874280406Z",
  "body": {},
  "url": "/test/findUserInfo.do",
  "fullName": "TEST",
  "log": {
    "file": {
      "path": "/data/logs/spring.log"
    }
  },
  "type": "test",
  "time": 1753067797629,
  "userId": 1088479580471164928,
  "message": "2025-07-21 11:16:37.629  INFO 8 --- [reactor-http-nio-1] c.test.authgateway.filter.GateWayFilter  : logstashVO:{\"body\":\"{}\",\"fullName\":\"TEST\",\"time\":1753067797629,\"url\":\"/test/findUserInfo.do\",\"userId\":1088479580471164928}"
}