2.3做logstash实验

收集apache日志输出到es

在真实服务器安装logstash,httpd

systemctl start httpd

echo 666 > /var/www/html/index.html

cat /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/httpd #系统内置变量

cd /usr/local/logstash/config/

cp logstash-sample.conf httpd-access.conf

vim httpd-access.conf

复制代码
input {
   stdin {}
}
filter {
  grok  {
    match => {
      "message" => "%{HTTPD_COMBINEDLOG}"
    }
    remove_field => ["message","auth","ident"]
  }
  date {
      match => ["timestamp","dd/MM/yyy:HH:mm:ss Z"]
  }
}

output {
   stdout {}
}

cat /var/log/httpd/access_log

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

logstash -f httpd-access.conf

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

可以看出apache的日志输出带有\,咱可以改进一下:

cd /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/

grep QS -R .

grep QUOTEDSTRING -R .

此时可以看出定义太复杂,咱自定义一个变量ALL,取代QS,双引号引起来。

vim /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/httpd

ALL .* #空行新增

HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} "%{ALL:referrer}" "%{ALL:agent}"

此时filter的相关配置已经完善!

接下来配置input引用apache日志,output输出到es。

vim /usr/local/logstash/config/httpd-access.conf

复制代码
input {
   file {
       path => ["/var/log/*.log","/var/log/message*"]
       type => "httpd_access"
       start_position => "beginning"
   }
}
filter {
  grok  {
    match => {
      "message" => "%{HTTPD_COMBINEDLOG}"
    }
    remove_field => ["message","auth","ident"]
  }
  date {
      match => ["timestamp","dd/MM/yyy:HH:mm:ss Z"]
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.148.132:9200"]
    index => "%{type}-%{+YYYY.MM.dd}"
  }
}

logstash -f /usr/local/logstash/config/httpd-access.conf

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

安装插件ElasticSearch Head可视化工具

(略)

测试:浏览器访问192.168.148.132:9100

http://localhost:9200/ > http://192.168.148.132:9200/ > 连接

收集nginx日志输出到es

方案一

nginx:

修改 nginx server 的配置文件

复制代码
    log_format  json '{'
                           '"client":"$remote_addr",'
                           '"time":"$time_local",'
                           '"verb":"$request_method",'
                           '"url":"$request_uri",'
                           '"status":"$status",'
                           '"size":$body_bytes_sent,' 
                           '"referer": "$http_referer",'
                           '"agent": "$http_user_agent"'
               '}';
   access_log  /var/log/nginx/access_json.log  json;   

logstash 配置文件**:**

复制代码
input {
   file {
      path => "/var/log/nginx/access_json.log"
      codec => "json"                           #输入预定义好的 JSON 数据, 可以省略掉 filter/grok 配置, 从而减轻logstash的负载
      start_position => "beginning"
   }
}
output {
    elasticsearch {
           hosts => ["192.168.10.11:9200"]
           index => "nginx-log-%{+YYYY.MM.dd}"
    }
}

方案二

Logstash 对nginx 标准日志的 grok 正则定义是:

MAINNGINXLOG %{COMBINEDAPACHELOG} %{QS:x_forwarded_for}

logstash:直接使用访问日志

复制代码
input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}"}    
    }    
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]    
    }    
    geoip {
        source => "clientip"    
    }
}
output {
    elasticsearch {     
        hosts => "192.168.11.10:9200" 
    }
}
相关推荐
czhc11400756639 分钟前
Linux 76 rsync
linux·运维·python
你不知道我是谁?1 小时前
负载均衡--四层、七层负载均衡的区别
运维·服务器·负载均衡
dyj0951 小时前
【Rancher Server + Kubernets】- Nginx-ingress日志持久化至宿主机
运维·nginx·rancher
码出钞能力3 小时前
linux内核模块的查看
linux·运维·服务器
星辰云-3 小时前
# Linux Centos系统硬盘分区扩容
linux·运维·centos·磁盘扩容
Hellc0073 小时前
Nginx 高级 CC 与 DDoS 防御策略指南
运维·nginx·ddos
feilieren4 小时前
Docker 安装 Elasticsearch 9
运维·elasticsearch·docker·es
小皮侠5 小时前
nginx的使用
java·运维·服务器·前端·git·nginx·github
Maki Winster5 小时前
在 Ubuntu 下配置 oh-my-posh —— 普通用户 + root 各自使用独立主题(共享可执行)
linux·运维·ubuntu
翻滚吧键盘6 小时前
debian及衍生发行版apt包管理常见操作
运维·debian