ELK框架Logstash配合Filebeats和kafka使用

ELK框架Logstash配合Filebeats和kafka使用

本文目录

配置文件结构

配置文件为:logstash.yml

需要自己新建conf文件,设置inputfilteroutput,文件结构如下,自带的logstash-sample.conf内容如下

input { 

}
filter {

}
output {

}

启动命令

bin/logstash -f config/logstash.conf

https://www.elastic.co/guide/en/logstash/current/input-plugins.html

input为标准输入,output为标准输出

input { 
  stdin { } 
}
output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
  stdout { }
}

input为log文件

output为标准输出

input {
  # 从文件读取日志信息
  file {
    path => "/xxx/demolog/logs/myapp-info.log"
    type => "ghn"
    start_position => "beginning"
  }

}

output {
  stdout { codec => rubydebug }
}

output为es

input {
  # 从文件读取日志信息
  file {
    path => "/xxx/demolog/log/demolog-*.log"
    type => "ghn"
    start_position => "beginning"
  }

}

output {
  # 输出到 elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "/xxx/elk/logstash-8.9.1/config/certs/http_ca.crt"
    index => "ghn-%{+YYYY.MM.dd}"
  }
  stdout {  }
}

input为tcp

配合springboot/springcloud使用

springboot配置

官方github:https://github.com/logfellow/logstash-logback-encoder
在pom.xml添加依赖

        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>7.4</version>
        </dependency>

在logback-spring.xml添加配置

    <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>127.0.0.1:4560</destination>
        <!-- encoder is required -->
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="info">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="stash" />
    </root>

logstash配置

input {
  # 从文件读取日志信息
  tcp {
    host => "0.0.0.0"
    mode => "server"
    port => 4560
    codec => json_lines
  }

}

output {
  # 输出到 elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "xxx/logstash-8.9.1/config/certs/http_ca.crt"
    index => "ghn-%{+YYYY.MM.dd}"
  }
  stdout { }
  # stdout { codec => rubydebug }
}
  • logstash终端查看

  • kibana查看

input为filebeats

filebeats配置

  • 配置文件位置:filebeat-8.9.1-darwin-aarch64/filebeat.yml,修改如下部分,指定log位置为springboot的目录

    filebeat.inputs:

    • type: filestream
      enabled: true
      paths:
      • /xxx/xxx/*.log
  • 启动

    ./filebeat -e -c filebeat.yml -d "publish"

与logstash建立了连接,启动成功

logstash配置

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "/xxxx/logstash-8.9.1/config/certs/http_ca.crt"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
  • 启动logstash

    bin/logstash -f config/logstash-filebeat.conf

  • 获取从filebeats发来的日志

kibana

  • kibana中数据视图已经能够看到

  • 查看详情

input为kafka

官方文档:https://www.elastic.co/guide/en/logstash/current/use-filebeat-modules-kafka.html

filebeats设置

output.kafka:
  hosts: ["localhost:9092"]
  topic: "filebeat"
  codec.json:
    pretty: false

./filebeat -e -c filebeat.yml -d "publish"

logstash配置

input {
  kafka {
    bootstrap_servers => ["localhost:9092"]
    topics => ["filebeat"]
    codec => json
  }

}

output {
  # 输出到 elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "/xxx/elk/logstash-8.9.1/config/certs/http_ca.crt"
    index => "ghn-%{+YYYY.MM.dd}"
  }
  stdout { }
  # stdout { codec => rubydebug }
}
  • kafka启动

  • logstash查看

  • kafka关闭

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html

相关推荐
飞火流星020271 小时前
SkyWalking集成Kafka实现日志异步采集经验总结
kafka·skywalking·skywalking日志采集·skywalking异步采集·skywalking配置·kafka数据压缩算法·kafka客户端工具
天天向上杰7 小时前
简识MQ之Kafka、ActiveMQ、RabbitMQ、RocketMQ传递机制
kafka·rabbitmq·rocketmq·activemq
刀客1238 小时前
kafka基本知识
分布式·kafka
陆鳐LuLu1 天前
日志管理利器:基于 ELK 的日志收集、存储与可视化实战
运维·elk·jenkins
程序猿熊跃晖1 天前
多环境日志管理:使用Logback与Logstash集成实现高效日志处理
spring boot·elk·logback
茶本无香1 天前
kafka+spring cloud stream 发送接收消息
spring cloud·kafka·java-zookeeper
Swift社区1 天前
【微服务优化】ELK日志聚合与查询性能提升实战指南
spring·elk·微服务·云原生·架构
xiao-xiang2 天前
kafka-保姆级配置说明(producer)
分布式·kafka
被程序耽误的胡先生2 天前
java中 kafka简单应用
java·开发语言·kafka
2501_903238652 天前
深入理解 Kafka 主题分区机制
分布式·kafka·个人开发