ELK之LogStash插件grok和geoip的配置使用

本文针对LogStash常用插件grok和geoip的使用进行说明:

一、使用grok输出结构化数据

编辑 first-pipeline.conf 文件,修改为如下内容:

bash 复制代码
input{
  #stdin{type => stdin}
  file {
    # 读取文件的路径
    path => ["/tmp/access.log"]
    start_position => "beginning"
  }
}

filter{
  grok{
    match => {"message" => "%{COMBINEDAPACHELOG}" }
  }

}

output{
  stdout{codec => rubydebug}
}

启动./logstash -f ../config/first-pipeline.conf后输出就为结构化的数据了:

bash 复制代码
{
        "message" => "140.77.188.102 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
       "response" => "200",
           "auth" => "-",
          "bytes" => "103",
       "referrer" => "\"-\"",
           "host" => "nb002",
       "@version" => "1",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
     "@timestamp" => 2022-06-26T00:28:24.302Z,
      "timestamp" => "25/Jun/2022:05:11:33 +0800",
          "ident" => "-",
    "httpversion" => "1.1",
           "path" => "/tmp/access.log",
       "clientip" => "140.77.188.102",
           "verb" => "GET",
        "request" => "/api/ss/api/v1/login/getBaseUrl"
}

二、使用grok对输出数据进行修改

编辑 first-pipeline.conf 文件,修改为如下内容:

bash 复制代码
input{
  #stdin{type => stdin}
  file {
    path => ["/tmp/access.log"]
    start_position => "beginning"
  }
}

filter{
  grok{
    match => {"message" => "%{COMBINEDAPACHELOG}" }
  }
  mutate{
    # 重命名字段
    rename => {"clientip" => "cip"}
  }
  mutate{
    # 移出特定字段
    remove_field => ["timestamp","agent"]
  }
}

output{
  stdout{codec => rubydebug}
}

重新启动./logstash -f ../config/first-pipeline.conf 后,往 /tmp/access.log 中新增一条数据,看输出:发现"clientip" 变成了 "cip" 和timestamp agent 字段已经没有了。NICE

bash 复制代码
{
           "verb" => "GET",
     "@timestamp" => 2022-06-26T00:48:28.224Z,
       "referrer" => "\"-\"",
           "path" => "/tmp/access.log",
           "auth" => "-",
        "message" => "140.77.188.102 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
       "@version" => "1",
          "ident" => "-",
       "response" => "200",
          "bytes" => "103",
        "request" => "/api/ss/api/v1/login/getBaseUrl",
    "httpversion" => "1.1",
           "host" => "nb002",
            "cip" => "140.77.188.102"
}

三、使用geoip过滤器插件

使用geoip过滤器插件,可以增强数据。

geoip插件可以针对IP地址进行地理位置信息来源的查找

编辑 first-pipeline.conf 文件,修改为如下内容:

bash 复制代码
input{
  #stdin{type => stdin}
  file {
    path => ["/tmp/access.log"]
    start_position => "beginning"
  }
}

filter{
  grok{
    match => {"message" => "%{COMBINEDAPACHELOG}" }
  }
  mutate{
    # 重命名字段
    rename => {"clientip" => "cip"}
  }
  mutate{
    # 移出特定字段
    remove_field => ["timestamp","agent"]
  }
  geoip{
    # 由于上面将clientip修改为了cip,故此处配置cip,如果没有rename字段则用clientip
    source => "cip"
  }
}

output{
  stdout{codec => rubydebug}
}

重新启动./logstash -f ../config/first-pipeline.conf 后,往 /tmp/access.log 中新增一条数据,看输出:发现输出结果中新增了geoip 字段,并展示了地区、国家、省份、经纬度等地理位置信息。

外国ip示例:

bash 复制代码
{
           "host" => "nb002",
           "auth" => "-",
          "bytes" => "103",
            "cip" => "140.77.188.104",
       "@version" => "1",
        "message" => "140.77.188.104 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
           "verb" => "GET",
        "request" => "/api/ss/api/v1/login/getBaseUrl",
       "referrer" => "\"-\"",
       "response" => "200",
          "ident" => "-",
           "path" => "/tmp/access.log",
     "@timestamp" => 2022-06-26T00:58:11.786Z,
          "geoip" => {
	         "country_code3" => "FR",
	             "longitude" => 4.85,
	                    "ip" => "140.77.188.104",
	        "continent_code" => "EU",
	           "region_name" => "Rhône",
	         "country_code2" => "FR",
	              "timezone" => "Europe/Paris",
	          "country_name" => "France",
	           "region_code" => "69",
	              "latitude" => 45.748,
	           "postal_code" => "69007",
	              "location" => {
	            "lat" => 45.748,
	            "lon" => 4.85
        },
             "city_name" => "Lyon"
    },
    "httpversion" => "1.1"
}

国内ip示例:

bash 复制代码
{
           "host" => "nb002",
           "auth" => "-",
          "bytes" => "103",
            "cip" => "175.30.108.241",
       "@version" => "1",
        "message" => "175.30.108.241 - - [25/Jun/2022:05:11:33 +0800] \"GET /api/ss/api/v1/login/getBaseUrl HTTP/1.1\" 200 103 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b\"",
           "verb" => "GET",
        "request" => "/api/ss/api/v1/login/getBaseUrl",
       "referrer" => "\"-\"",
       "response" => "200",
          "ident" => "-",
           "path" => "/tmp/access.log",
     "@timestamp" => 2022-06-26T01:00:11.972Z,
          "geoip" => {
         "country_code3" => "CN",
             "longitude" => 125.3247,
                    "ip" => "175.30.108.241",
        "continent_code" => "AS",
           "region_name" => "Jilin",
         "country_code2" => "CN",
              "timezone" => "Asia/Shanghai",
          "country_name" => "China",
           "region_code" => "JL",
              "latitude" => 43.88,
              "location" => {
            "lat" => 43.88,
            "lon" => 125.3247
        },
             "city_name" => "Changchun"
    },
    "httpversion" => "1.1"
}

END

相关推荐
Karoku06621 小时前
【企业级分布式系统】ELK-企业级日志分析系统
运维·数据库·redis·mysql·elk·缓存
一名技术极客1 天前
徒手从零搭建一套ELK日志平台
elk
jun7788954 天前
SpringBoot整合ELK使用详解
spring boot·后端·elk
yueqingll4 天前
023、ELK 从入门到实践
elk
abandondyy4 天前
ELK Elasticsearch 集群部署
大数据·elk·elasticsearch
小李叭叭叭4 天前
ELK8.15.4搭建开启安全认证
运维·elk·elasticsearch·kibana
光仔December11 天前
【Elasticsearch入门到落地】1、初识Elasticsearch
大数据·elk·elasticsearch·搜索引擎·lucene
太空眼睛13 天前
【EFK】Linux集群部署Elasticsearch最新版本8.x
linux·elk·elasticsearch·efk·master·8.15.3·node.roles
Shenqi Lotus16 天前
ELK-ELK基本概念_ElasticSearch的配置
elk·elasticsearch