ELK2——logstash

半梦半醒*2025-10-28 10:36

ELK2------logstash

一、logstash简介

logstash是一个开源的数据采集工具,通过数据源采集数据.然后进行过滤,并自定义格式输出到目的地。

数据分为:

  1. 结构化数据 如：mysql数据库里的表等

  2. 半结构化数据 如：xml、yaml、json等

  3. 非结构化数据 如：文档、图片、音频、视频等

logstash可以采集任何格式的数据,当然我们这里主要是讨论采集系统日志,服务日志等日志类型数据。

官方产品介绍:https://www.elastic.co/cn/products/logstash

input插件: 用于导入日志源 （配置必须 ）

https://www.elastic.co/guide/en/logstash/current/input-plugins.html

filter插件: 用于过滤（不是配置必须的 ）

https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

output插件: 用于导出（配置必须 ）

https://www.elastic.co/guide/en/logstash/current/output-plugins.html

二、logstash部署

1、检查防火墙selinux
powershell 复制代码 
[root@stw3 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@stw3 ~]# getenforce 
Disabled
[root@stw3 ~]# hostnamectl set-hostname logstash.example.com
[root@stw3 ~]# bash
[root@logstash ~]#
2、确认openjdk版本（自带版本即可）
powershell 复制代码 
[root@logstash ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
3、安装logstash（这里有压缩包直接拖进来即可）

下载：wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.2.rpm

powershell 复制代码 
[root@logstash ~]# rz -E
rz waiting to receive.
[root@logstash ~]# ls
anaconda-ks.cfg  Documents  initial-setup-ks.cfg  Music     Public     Videos
Desktop          Downloads  logstash-6.5.2.rpm    Pictures  Templates
[root@logstash ~]# rpm -ivh logstash-6.5.2.rpm 
warning: logstash-6.5.2.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.5.2-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
4、修改logstash主配置文件
powershell 复制代码 
[root@logstash ~]# vim /etc/logstash/logstash.yml
[root@logstash ~]# cat /etc/logstash/logstash.yml | grep -v '#' | grep -v '^$'
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
http.host: "192.168.100.30"
path.logs: /var/log/logstash
5、启动测试
powershell 复制代码 
[root@logstash ~]# cd /usr/share/logstash
[root@logstash logstash]# ls
bin           Gemfile       LICENSE.txt               modules     vendor
CONTRIBUTORS  Gemfile.lock  logstash-core             NOTICE.TXT  x-pack
data          lib           logstash-core-plugin-api  tools
[root@logstash logstash]# cd bin
//使用下面的空输入和空输出启动测试一下
[root@logstash bin]# ./logstash -e 'input {stdin {}} output {stdout {}}'
//运行后，输入字符将被stdout做为标准输出内容输出
回车之后输出一串代码，输入yyqx，messages中显示yyqx
关闭启动------Ctrl + C
另一种验证方法
powershell 复制代码 
[root@logstash bin]# vim /etc/logstash/conf.d/test.conf
[root@logstash bin]# cat /etc/logstash/conf.d/test.conf
input {
	stdin {
	}
}

filter {
}

output {
	stdout {
		codec => rubydebug
	}
}
[root@logstash bin]# pwd
/usr/share/logstash/bin
[root@logstash bin]# ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/test.conf -t
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2025-10-23T09:31:59,310][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2025-10-23T09:32:04,238][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
--path.settings 指定logstash主配置文件目录
-f 指定片段配置文件
-t 测试配置文件是否正确
codec => rubydebug这句可写可不定,默认就是这种输出方式
测试
powershell 复制代码 
[root@logstash bin]# ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf 
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2025-10-23T09:34:33,040][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2025-10-23T09:34:33,082][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.2"}
[2025-10-23T09:34:33,144][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"af9e2b99-05b7-45ad-8228-6eb251e21d25", :path=>"/var/lib/logstash/uuid"}
[2025-10-23T09:34:40,139][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2025-10-23T09:34:40,427][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x11e177f6 run>"}
The stdin plugin is now waiting for input:
[2025-10-23T09:34:40,673][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2025-10-23T09:34:41,143][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

{
       "message" => "",
          "host" => "logstash.example.com",
    "@timestamp" => 2025-10-23T01:34:43.433Z,
      "@version" => "1"
}
xixi
{
       "message" => "xixi",
          "host" => "logstash.example.com",
    "@timestamp" => 2025-10-23T01:34:54.171Z,
      "@version" => "1"
}
-r参数很强大,会动态装载配置文件,也就是说启动后,可以不用重启修改配置文件

三、日志采集

1、采集messages日志
以/var/log/messages为例,只定义input输入和output输出,不考虑过滤
powershell 复制代码 
[root@logstash bin]# vim /etc/logstash/conf.d/test.conf 
[root@logstash bin]# cat /etc/logstash/conf.d/test.conf 
input {
	file {
		path => "/var/log/messages"
		start_position => "beginning"
	}
}

output {
	elasticsearch {
		hosts => ["192.168.100.20"]
		index => "test-%{+YYYY.MM.dd}"
	}
}
[root@logstash bin]# pwd
/usr/share/logstash/bin
[root@logstash bin]# ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf &
[1] 58240
......
......
[2025-10-23T09:45:22,101][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
//后台运行如果要杀掉,请使用pkill java或ps查看PID再kill -9清除
message字段的信息就是logstash上传的/var/log/messages里的日志信息
在主机中使用su命令
powershell 复制代码 
[root@logstash bin]# su - stw
[stw@logstash ~]$

在logstash那台服务器上做一些操作(比如,重启下sshd服务), 让/var/log/message有新的日志信息，然后验证es-head里的数据。

结果: 会自动更新, 浏览器刷新就能在es-head上看到更新的数据。

2, kill掉logstash进程(相当于关闭), 也做一些操作让/var/log/message日志有更新，然后再次启动logstash。

结果: 会自动连上es集群, es-head里也能查看到数据的更新。

2、采集多日志源
powershell 复制代码 
[root@logstash bin]# vim /etc/logstash/conf.d/test.conf 
[root@logstash ~]# cat /etc/logstash/conf.d/test.conf 
input {
	file {
		path => "/var/log/messages"
		start_position => "beginning"
		type => "messages"
	}

	file {
		path => "/var/log/yum.log"
		start_position => "beginning"
		type => "yum"
	}
}

filter {

}

output {
	if [type] == "messages" {
	elasticsearch {
		hosts => ["192.168.100.10:9200","192.168.100.20:9200"]
		index => "messages-%{+YYYY-MM-dd}"
		}
	}

	if [type] == "yum" {
        elasticsearch {
                hosts => ["192.168.100.10:9200","192.168.100.20:9200"]
                index => "yum-%{+YYYY-MM-dd}"
                }
        }

}
[root@logstash bin]# ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf &
[1] 58795
[root@logstash bin]# jobs -l
[1]+ 58795 Running                 ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf &
此时只有messages的数据没有yum的
yum安装一个服务
powershell 复制代码 
[root@logstash bin]# cd /etc/yum.repos.d/
[root@logstash yum.repos.d]# ls
CentOS-Base.repo  CentOS-Debuginfo.repo  CentOS-Media.repo    CentOS-Vault.repo
CentOS-CR.repo    CentOS-fasttrack.repo  CentOS-Sources.repo
[root@logstash yum.repos.d]# rm -rf *
[root@logstash yum.repos.d]# vim server.repo
[root@logstash yum.repos.d]# cat server.repo
[aa]
name=aa1
baseurl=file:///mnt
enabled=1
gpgcheck=0
[root@logstash yum.repos.d]# mount /dev/cdrom /mnt
mount: /dev/sr0 is write-protected, mounting read-only
[root@logstash yum.repos.d]# yum -y install httpd
[root@logstash yum.repos.d]# cat /var/log/yum.log
Oct 23 10:38:29 Installed: apr-1.4.8-3.el7_4.1.x86_64
Oct 23 10:38:29 Installed: apr-util-1.5.2-6.el7.x86_64
Oct 23 10:38:29 Installed: httpd-tools-2.4.6-88.el7.centos.x86_64
Oct 23 10:38:29 Installed: mailcap-2.1.41-2.el7.noarch
Oct 23 10:38:30 Installed: httpd-2.4.6-88.el7.centos.x86_64
此时就有messages也有yum
相关推荐
java_logo3 小时前
Docker 部署 CentOS 全流程指南
linux·运维·人工智能·docker·容器·centos
会飞的小蛮猪3 小时前
ELK运维之路(Logstash-插件)
运维·经验分享·elk·elasticsearch·logstash
半梦半醒*3 小时前
ELK3——kibana
linux·运维·elasticsearch·centos·gitlab
CodeAmaz3 小时前
ELK（Elasticsearch + Logstash + Kibana + Filebeat）采集方案
java·elk·elasticsearch·1024程序员节
吃饭最爱3 小时前
redis的基础知识
1024程序员节
数据智能老司机3 小时前
OpenSearch 权威指南——OpenSearch 概览
elasticsearch·搜索引擎·aws
天***88963 小时前
怎么卸载或禁用 WPS office的wps看图软件？怎么关闭wps看图软件。或者怎么彻底删除wps?
运维·服务器·wps
weixiao04303 小时前
Linux网络 NAT、代理服务、内⽹穿透
1024程序员节
热门推荐
01GitHub 镜像站点02BongoCat - 跨平台键盘猫动画工具03UV安装并设置国内源04GitLab 零基础入门指南：从安装到项目管理全流程05Linux下V2Ray安装配置指南06NVIDIA显卡驱动、CUDA、cuDNN 和 TensorRT 版本匹配指南07Labelme从安装到标注：零基础完整指南08Burp与其他安全工具联动及代理设置教程09jdk21下载、安装（Windows、Linux、macOS）10Pycharm+Neo4j红楼梦人物关系图谱