ELK框架Logstash配合Filebeats和kafka使用

ELK框架Logstash配合Filebeats和kafka使用

本文目录

配置文件结构

配置文件为:logstash.yml

需要自己新建conf文件,设置inputfilteroutput,文件结构如下,自带的logstash-sample.conf内容如下

input { 

}
filter {

}
output {

}

启动命令

bin/logstash -f config/logstash.conf

https://www.elastic.co/guide/en/logstash/current/input-plugins.html

input为标准输入,output为标准输出

input { 
  stdin { } 
}
output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
  stdout { }
}

input为log文件

output为标准输出

input {
  # 从文件读取日志信息
  file {
    path => "/xxx/demolog/logs/myapp-info.log"
    type => "ghn"
    start_position => "beginning"
  }

}

output {
  stdout { codec => rubydebug }
}

output为es

input {
  # 从文件读取日志信息
  file {
    path => "/xxx/demolog/log/demolog-*.log"
    type => "ghn"
    start_position => "beginning"
  }

}

output {
  # 输出到 elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "/xxx/elk/logstash-8.9.1/config/certs/http_ca.crt"
    index => "ghn-%{+YYYY.MM.dd}"
  }
  stdout {  }
}

input为tcp

配合springboot/springcloud使用

springboot配置

官方github:https://github.com/logfellow/logstash-logback-encoder
在pom.xml添加依赖

        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>7.4</version>
        </dependency>

在logback-spring.xml添加配置

    <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>127.0.0.1:4560</destination>
        <!-- encoder is required -->
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="info">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="stash" />
    </root>

logstash配置

input {
  # 从文件读取日志信息
  tcp {
    host => "0.0.0.0"
    mode => "server"
    port => 4560
    codec => json_lines
  }

}

output {
  # 输出到 elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "xxx/logstash-8.9.1/config/certs/http_ca.crt"
    index => "ghn-%{+YYYY.MM.dd}"
  }
  stdout { }
  # stdout { codec => rubydebug }
}
  • logstash终端查看

  • kibana查看

input为filebeats

filebeats配置

  • 配置文件位置:filebeat-8.9.1-darwin-aarch64/filebeat.yml,修改如下部分,指定log位置为springboot的目录

    filebeat.inputs:

    • type: filestream
      enabled: true
      paths:
      • /xxx/xxx/*.log
  • 启动

    ./filebeat -e -c filebeat.yml -d "publish"

与logstash建立了连接,启动成功

logstash配置

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "/xxxx/logstash-8.9.1/config/certs/http_ca.crt"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}
  • 启动logstash

    bin/logstash -f config/logstash-filebeat.conf

  • 获取从filebeats发来的日志

kibana

  • kibana中数据视图已经能够看到

  • 查看详情

input为kafka

官方文档:https://www.elastic.co/guide/en/logstash/current/use-filebeat-modules-kafka.html

filebeats设置

output.kafka:
  hosts: ["localhost:9092"]
  topic: "filebeat"
  codec.json:
    pretty: false

./filebeat -e -c filebeat.yml -d "publish"

logstash配置

input {
  kafka {
    bootstrap_servers => ["localhost:9092"]
    topics => ["filebeat"]
    codec => json
  }

}

output {
  # 输出到 elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"] 
    user => "elastic"
    password => "xxxxxx"
    ssl => "true"
    cacert => "/xxx/elk/logstash-8.9.1/config/certs/http_ca.crt"
    index => "ghn-%{+YYYY.MM.dd}"
  }
  stdout { }
  # stdout { codec => rubydebug }
}
  • kafka启动

  • logstash查看

  • kafka关闭

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html

相关推荐
biubiubiu07064 分钟前
Kafka消费者相关
分布式·kafka·linq
junzhen_chen8 分钟前
Kafka可视化工具EFAK(Kafka-eagle)安装部署
分布式·kafka
后季暖4 小时前
kafka stream对比flink
分布式·flink·kafka
信徒_14 小时前
kafka consumer 手动 ack
分布式·kafka·linq
小句18 小时前
Kafka 消费者组内分区分配策略 以及 管理控制台方案
分布式·kafka
陌殇殇20 小时前
003 SpringBoot集成Kafka操作
spring boot·kafka
陌殇殇20 小时前
004 Kafka异常处理
中间件·kafka·springboot
AustinCien1 天前
1.4常规es报错问题
运维·elk·elasticsearch
AustinCien1 天前
3.2实验filebeat->logstash->es
运维·elk
zhang98800002 天前
K8S能部署大数据集群吗?为什么?K8S的HPA功能可以作为大数据处理消息积压的问题(Kafka的分区)
大数据·kafka·kubernetes