监控架构介绍:
基本架构:
Prometheus 和 Zabbix 的对比:
安装和使用:
- Prometheus 采集、存储数据
- Grafana 用于图表展示
- alertmanager 用于接收 Prometheus 发送的警告信息
- node-exporter 用于收集操作系统和硬件信息的 metrics
shell
# 从创建一个专门的 prometheus 用户
useradd -M -s /usr/sbin/nologin prometheus
# 更改 prometheus 用户的文件权限
chown prometheus:prometheus -R /opt/prometheus
# Prometheus 以 prometheus 用户身份运行
sudo chown -R prometheus:prometheus /opt/prometheus/prometheus/data
# 755 权限设置允许 Prometheus 用户读写目录,但其他用户只能读和执行(进入目录)
sudo chmod -R 755 /opt/prometheus/prometheus/data
Prometheus Server 安装:
shell
# 下载 prometheus 二进制压缩包
wget https://github.com/prometheus/prometheus/releases/download/v2.53.2/prometheus-2.53.2.linux-amd64.tar.gz
# 解压
tar -zxvf prometheus-2.53.2.linux-amd64.tar.gz
mkdir /opt/prometheus -p
mv prometheus-2.53.2.linux-amd64 /opt/prometheus/prometheus
# 创建 systemd 服务
cat > /etc/systemd/system/prometheus.service << "EOF"
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus/prometheus/prometheus \
-- config.file=/opt/prometheus/prometheus/prometheus.yml \
-- storage.tsdb.path=/opt/prometheus/prometheus/data \ # 数据存储的路径
-- storage.tsdb.retention.time=60d \ # 数据存储的时长
-- wed.enable-lifecycle # 热加载配置
[Install]
WantedBy=multi-user.target
EOF
shell
# 重启 Prometheus
systemctl daemon-reload
systemctl start prometheus.service
# 设置 Prometheus 为开机自启
systemctl enable prometheus.service
# 检查 Prometheus 服务的开启状态
systemctl status prometheus.service
# 查看 Prometheus 服务的日志,进行故障排除
journalctl -u prometheus.service -f
prometheus 监控指标:http://192.168.10.6:9090/metrics
Alert Manager 安装:
shell
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
# 或者在本地下载,通过 scp 命令拷贝到远程 linux 服务端
# 解压
tar -zxvf alertmanager-0.27.0.linux-amd64.tar.gz
mkdir /opt/prometheus -p
mv alertmanager-0.27.0.linux-amd64 /opt/prometheus/alertmanager
# 更改 alertmanager 文件夹权限
chown prometheus:prometheus -R /opt/prometheus/alertmanager
# 创建 systemd 服务
cat > /etc/systemd/system/alertmanager.service << "EOF"
[Unit]
Description=Alert Manager
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/alertmanager/alertmanager \
--config.file=/opt/prometheus/alertmanager/alertmanager.yml \
--storage.path=/opt/prometheus/alertmanager/data
[Install]
WantedBy=multi-user.target
EOF
shell
# 重启 Prometheus
systemctl daemon-reload
systemctl start alertmanager.service
# 设置 Prometheus 为开机自启
systemctl enable alertmanager.service
# 检查 Prometheus 服务的开启状态
systemctl status alertmanager.service
# 查看 alertmanager 服务的日志,进行故障排除
journalctl -u alertmanager.service -f
Alertmanager 的指标:http://192.168.10.6:9093/metrics
shell
# 修改 prometheus 配置:/opt/prometheus/prometheus/prometheus.yml
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093 # 需根据实际情况填写 alertmanager 的地址
rule_files:
- "alert.yml" # 需根据实际修改文件名
# 增加触发器配置文件 /opt/prometheus/prometheus/alert.yml
cat > /opt/prometheus/prometheus/alert.yml << "EOF"
groups:
- name : Prometheus alert
rules:
# 对任何实例超过30s无法联系的情况,发送警报
- alert: 服务警告
expr: up == 0
for: 30s
labels:
severity: critical
annotations:
instance: "{{ $labels.instance }}"
description: "{{ $labels.job }} 服务已关闭"
EOF
# 检查配置
cd /opt/prometheus/prometheus
./prometheus check config prometheus.yml
重启 prometheus 或 重新加载配置文件:
shell
# 重启
systemctl restart prometheus
# 重载,需要 --web.enable-lifecycle 配置
curl -x POST http://localhost:9090/-/reload
Grafana 安装:
shell
wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.4.3.linux-amd64.tar.gz
tar -zxvf grafana-enterprise-9.4.3.linux-amd64.tar.gz
mv grafana-9.4.3 /opt/prometheus/grafana
chown prometheus:prometheus -R /opt/prometheus/grafana
cat > /etc/systemd/system/grafana-server.service << "EOF"
[Unit]
Description=Grafana Server
Documentation=http://docs.grafana.org
[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus/grafana/bin/grafana-server \
--config=/opt/prometheus/grafana/conf/defaults.ini \
--homepath=/opt/prometheus/grafana
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
# 查看 alertmanager 服务的日志,进行故障排除
journalctl -u grafana-server.service -f
Grafana 的访问地址:http://192.168.10.6:3000