Centos7 随笔记录记录 docker compose 统一管理 granfana + loki + vector 监控kafka 信息。
当然如果仅仅是想通过 Grafana 监控kafka，推荐使用 Grafana Prometheus 通过JMX监控kafka
目录
[1. 目录结构](#1. 目录结构)
[2. 前提已安装Docker-Compose](#2. 前提已安装Docker-Compose)
[3. docker-compose 自定义服务相关配置文件](#3. docker-compose 自定义服务相关配置文件)
[3.1 配置 docker-compose-grafana-loki-vector.yml 文件](#3.1 配置 docker-compose-grafana-loki-vector.yml 文件)
[3.2 创建 loki-config.yaml](#3.2 创建 loki-config.yaml)
[3.3 配置loki 服务](#3.3 配置loki 服务)
[4. docker-compose 单独启动指定服务](#4. docker-compose 单独启动指定服务)
[4.1 单独启动 loki](#4.1 单独启动 loki)
[4.1.1 启动loki](#4.1.1 启动loki)
[4.1.2 查询 log](#4.1.2 查询 log)
[4.2 单独启动vector](#4.2 单独启动vector)
[4.2.1 启动 vector](#4.2.1 启动 vector)
[4.2.2 查询vector log](#4.2.2 查询vector log)
[4.2.2.1 错误log 问题定位](#4.2.2.1 错误log 问题定位)
[4.2.2.2 解决方法](#4.2.2.2 解决方法)
[4.2.2.3 重启vector 容器，查看无error log](#4.2.2.3 重启vector 容器，查看无error log)
[4.3 单独启动grafana](#4.3 单独启动grafana)
[4.3.1 启动 grafana](#4.3.1 启动 grafana)
[4.3.2 查看 grafana log](#4.3.2 查看 grafana log)
[4.3.2.1 错误log 问题定位](#4.3.2.1 错误log 问题定位)
[4.3.2.2 解决方法](#4.3.2.2 解决方法)
[4.3.2.3 重启grafana 容器，查看无error log](#4.3.2.3 重启grafana 容器，查看无error log)
[4.3.3 访问grafana](#4.3.3 访问grafana)
[4.4 查看loki metrics数据 和loki 运行状态](#4.4 查看loki metrics数据 和loki 运行状态)
[5. docker-compose 启停所有服务](#5. docker-compose 启停所有服务)
[5.1 docker-compose 启动所有自定义容器服务](#5.1 docker-compose 启动所有自定义容器服务)
[5.2 docker-compose 停止所有容器服务](#5.2 docker-compose 停止所有容器服务)
[6. 配置grafana](#6. 配置grafana)
[6.1 添加loki 数据源](#6.1 添加loki 数据源)
[6.2 修改时区](#6.2 修改时区)
[6.3 添加query](#6.3 添加query)
[6.4 过滤展示kafka 消息](#6.4 过滤展示kafka 消息)
[6.5 优化后 dashboard, 展示kafka 指定字段](#6.5 优化后 dashboard, 展示kafka 指定字段)
[6.5.1 添加添加dashboard](#6.5.1 添加添加dashboard)
[6.5.2 优化dashboard, 显示指定字段](#6.5.2 优化dashboard, 显示指定字段)
[6.5.2.1 筛选kafka 原始消息](#6.5.2.1 筛选kafka 原始消息)
[6.5.2.2 优化表格显示指定字段](#6.5.2.2 优化表格显示指定字段)
[7. 其他异常问题](#7. 其他异常问题)
[7.1 grafana 页面有时跳转 网址 http://localhost:3000/XXXX 错误](#7.1 grafana 页面有时跳转 网址 http://localhost:3000/XXXX 错误)
[7.2 跳转错误网址原因定位](#7.2 跳转错误网址原因定位)
[7.3 解决方法](#7.3 解决方法)
[7.3.1 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址](#7.3.1 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址)
[7.3.2 为了grafana 有数据，必须修改vector.tml 配置参数](#7.3.2 为了grafana 有数据，必须修改vector.tml 配置参数)
[7.3.3 使配置生效](#7.3.3 使配置生效)
[7.4 验证 跳转网址正确](#7.4 验证 跳转网址正确)
1. 目录结构
# 1 创建 4个文件夹：grafana loki vector wal
# 2 为 wal 赋777 权限
[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector # 创建 grafana folder
├── grafana
├── loki # 创建 loki folder
│ └── loki-config.yaml
├── vector # 创建 vector folder
│ └── vector.toml
└── wal # 创建 wal folder,并赋 777 权限
8 directories, 5 files
[root@localhost grafana_loki_vector]#
# 为避免docker 中权限问题，为wal folder 赋权限
[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# chmod 777 wal
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 0
drwxr-xr-x. 4 root root 41 Dec 26 10:04 grafana
drwxr-xr-x. 2 root root 44 Dec 26 10:47 loki
drwxr-xr-x. 2 root root 33 Dec 21 15:03 vector
drwxrwxrwx. 3 root root 59 Dec 26 11:19 wal
[root@localhost grafana_loki_vector]#
2. 前提已安装Docker-Compose
如何安装 Docker-Compose，请参考 安装部署Docker-Compose
3. docker-compose 自定义服务相关配置文件
3.1 配置 docker-compose-grafana-loki-vector.yml 文件
# docker-compose 配置参考：https://github.com/grafana/loki/blob/master/production/docker-compose.yaml
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.yml
version: "3" # 这是Docker Compose的版本指定，该配置文件使用版本3的语法
# 网桥loki -> 方便相互通讯
networks: # 定义了一个名为loki的网络，该网络是为了Loki和其他服务之间相互通信而创建的网桥。
loki:
services: # 在这个部分定义了要运行的各个服务
loki: # Loki服务的配置
image: grafana/loki:latest # 使用了grafana/loki:latest镜像
container_name: grafana_loki_vector_demo_loki # 定义 容器名
restart: unless-stopped # 设置了重启策略
ports:
- "3100:3100" # 将主机的3100端口映射到Loki容器的3100端口，以便能够从主机访问Loki的API
volumes: # 将两个主机目录与容器内的目录进行了卷映射
- ./grafana_loki_vector/loki:/etc/loki
- ./grafana_loki_vector/wal:/wal
command: -config.file=/etc/loki/loki-config.yaml
networks:
- loki
# vector 用于对接kafka 消息
vector: # vector 服务的配置
# 指定使用 timberio/vector的Docker 镜像，并使用0.10.0-alpine 版本
image: timberio/vector:0.10.0-alpine
container_name: grafana_loki_vector_demo_vector # 定义 容器名
restart: unless-stopped # 指定容器在退出时重新启动，除非显式停止
ports:
- "80:80" # 容器的 80 端口映射到主机的 80 端口上
volumes:
# - ./grafana_loki_vector/vector/logs:/var/logs
# 本地主机目录./grafana_loki_vector/vector挂载到容器的/etc/vector目录上
- "./grafana_loki_vector/vector:/etc/vector"
# 这是要在容器中运行的命令，通过-c /etc/vector/vector.toml指定了配置文件的路径，并通过 -v 启用了详细日志输出
command: -c /etc/vector/vector.toml -v
networks:
- loki
# 用于UI展示
grafana:
image: grafana/grafana:latest
container_name: grafana_loki_vector_demo_grafana
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- "./grafana_loki_vector/grafana/data:/var/lib/grafana"
# - "./grafana_promtail_loki/grafana/config:/etc/grafana"
- "./grafana_loki_vector/grafana/log:/var/log/grafana"
environment:
# GF_EXPLORE_ENABLED: "true"
# 避免有时Grafana 会将跳转 URL 设置为 "localhost"
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000
networks:
- loki
=====================================================================
配置参数说明：
volumes: # 将两个主机目录与容器内的目录进行了卷映射
- ./grafana_loki_vector/loki:/etc/loki
# ./grafana_loki_vector/loki映射到了/etc/loki
- ./grafana_loki_vector/wal_tmp1:/wal
# ./grafana_loki_vector/wal_tmp1映射到了/wal
command: -config.file=/etc/loki/loki-local-config.yaml
# -config.file=/etc/loki/loki-local-config.yaml 指定了Loki容器启动时使用的配置文件路径
environment:
# 避免有时Grafana 会将跳转 URL 设置为 "localhost"
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000
3.2 创建 loki-config.yaml
创建loki-local-config.yaml 文件，配置loki 服务
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector
├── grafana
│ ├── data
│ └── log
├── loki
│ └── loki-config.yaml # 创建loki-config.yaml文件，配置loki 服务
├── vector
│ └── vector.toml
└── wal
3.3 配置loki 服务
# 这个是Loki的配置文件示例，其中包含了各个方面的配置，用于自定义和配置Loki日志系统的行为和存储。
auth_enabled: false # 设置认证功能是否启用。设为false表示禁用认证，即任何用户都可以访问Loki服务
server:
http_listen_port: 3100 # 配置Loki服务监听的HTTP端口。在这个示例中，Loki将监听3100端口
ingester: # 配置Loki的ingester（摄取器）相关参数
lifecycler:
address: 127.0.0.1
ring:
kvstore: # 使用内存作为kvstore存储方式
store: inmemory
replication_factor: 1 # 复制因子设置为 1
final_sleep: 0s
chunk_idle_period: 5m # chunk的闲置期 5m
chunk_retain_period: 30s # chunk的保留期 30s
schema_config: # 配置存储模式参数
configs:
- from: 2018-04-15 # 从特定日期开始使用的配置
store: boltdb # 数据存储方式（boltdb，文件系统）
object_store: filesystem # 数据存储方式 (文件系统)
schema: v9 # schema版本
index: # 索引相关参数
prefix: index_
period: 168h
storage_config: # 配置存储配置，指定Loki数据的持久化存储位置
boltdb:
directory: /tmp/loki/index # 索引数据将存储在/tmp/loki/index目录
filesystem:
directory: /tmp/loki/chunks # chunk数据将存储在/tmp/loki/chunks目录
# 配置限制参数，用于强制限制指标名称、拒绝旧样本数据，以及设置拒绝旧样本数据的最大时间范围
limits_config:
enforce_metric_name: false # 强制限制指标名称
reject_old_samples: true # 拒绝旧样本数据
reject_old_samples_max_age: 168h # 设置拒绝旧样本数据的最大时间范围
chunk_store_config: # 配置chunk存储参数，设置最大回溯时间范围
max_look_back_period: 0s # 设置最大回溯时间范围
table_manager: # 配置表管理器相关参数
chunk_tables_provisioning: # 设置chunk表的吞吐量
inactive_read_throughput: 0
inactive_write_throughput: 0
provisioned_read_throughput: 0
provisioned_write_throughput: 0
index_tables_provisioning: # 设置索引表的吞吐量
inactive_read_throughput: 0
inactive_write_throughput: 0
provisioned_read_throughput: 0
provisioned_write_throughput: 0
retention_deletes_enabled: false # 保留删除是否启用
retention_period: 0s # 保留期设置
# retention_deletes_enabled: true
# retention_period: 168h
4. docker-compose 单独启动指定服务
建议docker-compose 自定义多个服务时，先单独启动每个服务调试，没问题后，再统一启动
4.1 单独启动 loki
4.1.1 启动loki
# 启动指定自定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服务名> -d
# 启动loki
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up loki -d
[+] Running 7/7
✔ loki 6 layers [⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 24.0s
✔ c926b61bad3b Pull complete 12.0s
✔ 21a0947e3c21 Pull complete 6.0s
✔ 19779106cfff Pull complete 17.5s
✔ 9c6ee7a30a41 Pull complete 7.4s
✔ 2f9c53357ffd Pull complete 8.7s
✔ d8cb9832c63f Pull complete 11.4s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_loki Started 0.2s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
58528f1b4782 grafana/loki:latest "/usr/bin/loki -conf..." 9 seconds ago Up 8 seconds 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 18 hours ago Up 18 hours 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb..." 18 hours ago Up 18 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
[root@localhost grafana_loki_vector]#
4.1.2 查询 log
# 查看 loki 容器 log
# docker logs <loki服务容器名>
[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_loki
level=warn ts=2023-12-26T03:39:33.891828641Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."
''''''
''''''
''''''
level=info ts=2023-12-26T03:44:33.988929326Z caller=checkpoint.go:502 msg="atomic checkpoint finished" old=wal/checkpoint.000006.tmp new=wal/checkpoint.000006
[root@localhost grafana_loki_vector]#
4.2 单独启动vector
4.2.1 启动 vector
# 启动指定自定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服务名> -d
# 启动 vector
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up vector -d
[+] Running 6/6
✔ vector 5 layers [⣿⣿⣿⣿⣿] 0B/0B Pulled 30.6s
✔ 21c83c524219 Pull complete 18.0s
✔ 88943f7483ee Pull complete 2.9s
✔ 24da72aaf03b Pull complete 23.4s
✔ 4145aaf02e3f Pull complete 10.7s
✔ b16d307dc057 Pull complete 12.0s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_vector Started 0.3s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
776ab907da32 timberio/vector:0.10.0-alpine "/usr/local/bin/vect..." About a minute ago Up About a minute 0.0.0.0:80->80/tcp, :::80->80/tcp grafana_loki_vector_demo_vector
2a5581eea2eb grafana/loki:latest "/usr/bin/loki -conf..." 2 minutes ago Up 2 minutes 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 21 hours ago Up 21 hours 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb..." 21 hours ago Up 21 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
[root@localhost grafana_loki_vector]#
4.2.2 查询vector log
# 查看 vector 容器log
# docker logs <vector 容器名>
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:02:01.842 INFO vector: Log level "debug" is enabled.
Dec 26 06:02:01.857 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Dec 26 06:02:01.862 INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 20:01:29 +0000" arch="x86_64"
Dec 26 06:02:01.876 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:02:01.876 INFO vector::topology: Running healthchecks.
Dec 26 06:02:01.876 INFO vector::topology: Starting source "in"
Dec 26 06:02:01.876 INFO vector::topology: Starting sink "out"
Dec 26 06:02:01.876 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/ready method=GET
Dec 26 06:02:01.884 DEBUG http: vector::sinks::util::http: response. status=503 version=HTTP/1.1
Dec 26 06:02:01.885 ERROR vector::topology::builder: Healthcheck: Failed Reason: A non-successful status returned: 503 Service Unavailable # Error Log
Dec 26 06:02:05.978 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/loki/api/v1/push method=POST
Dec 26 06:02:06.060 DEBUG http: vector::sinks::util::http: response. status=204 version=HTTP/1.1
[root@localhost grafana_loki_vector]#
4.2.2.1 错误log 问题定位
# 确认 Vector 可以连接到 Loki 服务
在 Vector 配置文件中配置的 Loki 服务的地址应该是可访问的。
可以尝试使用 curl 或类似的工具连接到 Loki 服务
# curl http://loki:3100/ready
[root@localhost grafana_loki_vector]# curl http://loki:3100/ready
curl: (6) Could not resolve host: loki; Unknown error
[root@localhost grafana_loki_vector]#
总结： curl 请求无法解析 loki 主机名，并出现了 Unknown error 错误，vector 无法连接loki
4.2.2.2 解决方法
1） 检查 DNS 配置：确认您的 DNS 配置正确，可以解析主机名
# nslookup loki
[root@localhost grafana_loki_vector]# nslookup loki
Server: 114.114.114.114
Address: 114.114.114.114#53
** server can't find loki: NXDOMAIN # loki无法解析为有效的 IP 地址，返回了 NXDOMAIN 错误。
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
2）添加主机名映射：如果 DNS 解析不起作用，您可以尝试手动添加主机名到 IP 地址的映射
[root@localhost grafana_loki_vector]# cat /etc/hosts # 添加前
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat /etc/hosts # 添加后
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.247 loki
[root@localhost grafana_loki_vector]#
3）检查vector 是否可以连接loki
# curl http://loki:3100/ready
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# curl http://loki:3100/ready
ready # 可以连接loki
[root@localhost grafana_loki_vector]#
4.2.2.3 重启vector 容器，查看无error log
# 重启vector 容器 or 重新创建 vector 容器
# 无error log, vector 服务运行正常
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:20:58.569 INFO vector: Log level "debug" is enabled.
Dec 26 06:20:58.583 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Dec 26 06:20:58.588 INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 20:01:29 +0000" arch="x86_64"
Dec 26 06:20:58.602 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:20:58.602 INFO vector::topology: Running healthchecks.
Dec 26 06:20:58.602 INFO vector::topology: Starting source "in"
Dec 26 06:20:58.602 INFO vector::topology: Starting sink "out"
Dec 26 06:20:58.602 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/ready method=GET
Dec 26 06:20:58.606 DEBUG http: vector::sinks::util::http: response. status=200 version=HTTP/1.1
Dec 26 06:20:58.607 INFO vector::topology::builder: Healthcheck: Passed.
[root@localhost grafana_loki_vector]#
4.3 单独启动grafana
4.3.1 启动 grafana
# 启动指定自定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服务名> -d
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[+] Running 11/11
✔ grafana 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 106.8s
✔ 96526aa774ef Pull complete 2.3s
✔ 8107a14e7703 Pull complete 1.4s
✔ 41aaa20cd87e Pull complete 2.7s
✔ e00aef65f013 Pull complete 5.1s
✔ 257ea9b0297f Pull complete 5.5s
✔ 599ebbeda3b8 Pull complete 6.1s
✔ ca6b21901a74 Pull complete 59.8s
✔ b5fb47d1c454 Pull complete 92.5s
✔ 4fc0f14a8243 Pull complete 8.0s
✔ f638e65e9fc0 Pull complete 13.7s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_grafana Started 0.3s
[root@localhost grafana_loki_vector]#
4.3.2 查看 grafana log
# 查看 vector 容器log
# docker logs <vector 容器名>
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
GF_PATHS_DATA='/var/lib/grafana' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied
.....
....
[root@localhost grafana_loki_vector]#
4.3.2.1 错误log 问题定位
这些错误通常表示在容器中的某个目录或文件上没有写入权限。
4.3.2.2 解决方法
# 解决此问题的一种可能方法是确保您在运行容器时具有足够的权限
# 确保您以具有适当权限的用户身份运行 Docker 命令
# sudo usermod -aG docker $USER
# 更改目标目录的所有者和权限
sudo chown -R 472:472 ./grafana_loki_vector/grafana/data
sudo chmod -R 755 ./grafana_loki_vector/grafana/data
# 检查您的目标目录权限，并确保它具有足够的写入权限
[root@localhost grafana_loki_vector]# sudo chmod -R 777 ./grafana_loki_vector/grafana/data
[root@localhost grafana_loki_vector]#
4.3.2.3 重启grafana 容器，查看无error log
# 重启grafana 容器 or 重新创建 grafana 容器
# 无error log, grafana服务运行正常
# 此处每次都是：
1） 停止容器 # docker stop <container_ID or container_name>
2） 删除容器 # docker rm <container_ID or container_name>
3） 删除镜像 # docker rmi <image_ID>
4) 创建 grafana 容器
5） 查看 log
[root@localhost grafana_loki_vector]# docker stop 839b53f097ee # 停止容器
839b53f097ee
[root@localhost grafana_loki_vector]# docker rm 839b53f097ee # 删除容器
839b53f097ee
[root@localhost grafana_loki_vector]# docker images # 查询镜像ID
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 8387f19108f9 7 days ago 399MB
grafana/loki latest c2c89eb81f6e 13 days ago 67.7MB
hello-world latest 9c7a54a9a43c 7 months ago 13.3kB
wurstmeister/kafka latest a692873757c0 19 months ago 468MB
timberio/vector 0.10.0-alpine 2ebd263245dd 3 years ago 80.7MB
wurstmeister/zookeeper latest 3f43f72cb283 4 years ago 510MB
[root@localhost grafana_loki_vector]# docker rmi 8387f19108f9 # 删除镜像
Untagged: grafana/grafana:latest
Untagged: grafana/grafana@sha256:6b5b37eb35bbf30e7f64bd7f0fd41c0a5b7637f65d3bf93223b04a192b8bf3e2
Deleted: sha256:8387f19108f986254ecb7eeb1124445a9aeda16602114d98348ed5cd49151a7a
Deleted: sha256:06619c984d0071cd4b3a8fbfb377c1ca5b991497f4e1f6c2f280b9f33d402cb4
Deleted: sha256:5fe5dac0845a9eb44e2c5cfed4fb2754a4adf3224209f58c1a46cb7bd250f2e7
Deleted: sha256:91a3b38581f407bb488e03f002566138875732ba1da4a5e91c55b3dec08a2756
Deleted: sha256:18c0364cc25d61f2e9c8ba8582944e1fc6528ba44abc25ffc3e3bc5ed6a8dec9
Deleted: sha256:e335659dfae7632b652b2c796e742f11b2a9f0c0c7def8511b4df29158a50ee5
Deleted: sha256:5f7c41d4921ce413887919068764757d42744f4898c3f790953fee2e1b33b79b
Deleted: sha256:e8df864ddb32cefc11b6a549add118948327816e858570f6fcbb4b393643d196
Deleted: sha256:14b7f68d3d9094af53f80ebf90273ec864b731545874ee30cfb09cae5d097935
Deleted: sha256:baf151d12231a621fead4ca244823c83fb4ae6e01c36501e37c48a964138e724
Deleted: sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438
[root@localhost grafana_loki_vector]#
# 创建 grafana 容器
[root@localhost grafana_loki_vector]# sudo docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[+] Running 11/11
✔ grafana 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 113.6s
✔ 96526aa774ef Pull complete 12.7s
✔ 8107a14e7703 Pull complete 2.3s
✔ 41aaa20cd87e Pull complete 2.2s
✔ e00aef65f013 Pull complete 10.4s
✔ 257ea9b0297f Pull complete 5.5s
✔ 599ebbeda3b8 Pull complete 10.5s
✔ ca6b21901a74 Pull complete 83.9s
✔ b5fb47d1c454 Pull complete 97.5s
✔ 4fc0f14a8243 Pull complete 17.4s
✔ f638e65e9fc0 Pull complete 28.6s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_grafana Started 0.5s
[root@localhost grafana_loki_vector]#
# 查看 log
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
logger=settings t=2023-12-26T07:25:30.701353225Z level=info msg="Starting Grafana" version=10.2.3 commit=1e84fede543acc892d2a2515187e545eb047f237 branch=HEAD compiled=2023-12-18T15:46:07Z
logger=settings t=2023-12-26T07:25:30.701604782Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
......
......
......
logger=ngalert.scheduler t=2023-12-26T07:26:18.891340858Z level=info msg="Starting scheduler" tickInterval=10s
logger=ticker t=2023-12-26T07:26:18.891454409Z level=info msg=starting first_tick=2023-12-26T07:26:20Z
logger=http.server t=2023-12-26T07:26:18.897364437Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=plugins.update.checker t=2023-12-26T07:26:19.502508394Z level=info msg="Update check succeeded" duration=611.312832ms
logger=grafana.update.checker t=2023-12-26T07:26:20.504658267Z level=info msg="Update check succeeded" duration=1.613658029s
logger=infra.usagestats t=2023-12-26T07:26:51.904796949Z level=info msg="Usage stats are ready to report"
[root@localhost grafana_loki_vector]#
4.3.3 访问grafana
打开 http://<grafana 容器所在主机IP>:3000
默认 user/pwd: admin/admin
4.4 查看loki metrics数据 和loki 运行状态
# 查看loki metrics数据
# 访问：http://<loki 容器所在主机IP>:3100/metrics
# 查看loki运行状态
# 访问：http://<loki 容器所在主机IP>:3100/ready
到此已经调试完成，可以通过docker-compose 统一启动所有服务
5. docker-compose 启停所有服务
5.1 docker-compose 启动所有自定义容器服务
# 启动 定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[+] Running 3/3
✔ Container grafana_loki_vector_demo_loki Started 0.0s
✔ Container grafana_loki_vector_demo_vector Started 0.0s
✔ Container grafana_loki_vector_demo_grafana Started 0.0s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
252ca4e81d6f grafana/grafana:latest "/run.sh" 2 hours ago Up 2 seconds 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafana_loki_vector_demo_grafana
7068d0da2577 timberio/vector:0.10.0-alpine "/usr/local/bin/vect..." 3 hours ago Up 2 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp grafana_loki_vector_demo_vector
2a5581eea2eb grafana/loki:latest "/usr/bin/loki -conf..." 3 hours ago Up 2 seconds 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 24 hours ago Up 24 hours 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb..." 24 hours ago Up 24 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
[root@localhost grafana_loki_vector]#
5.2 docker-compose 停止所有容器服务
# docker-compose 停止所有容器服务
# docker-compose -f <docker-compose 自定义容器服务配置 yml 文件> -p <项目名> down
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[+] Running 4/4
✔ Container grafana_loki_vector_demo_vector Removed 0.4s
✔ Container grafana_loki_vector_demo_loki Removed 2.0s
✔ Container grafana_loki_vector_demo_grafana Removed 0.0s
✔ Network grafana_loki_vector_loki Removed 0.2s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 3 days ago Up 3 days 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb..." 3 days ago Up 3 days 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
b38e9b5b6a2e hello-world "/hello" 3 weeks ago Exited (0) 3 weeks ago infallible_rosalind
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 8387f19108f9 9 days ago 399MB
grafana/loki latest c2c89eb81f6e 2 weeks ago 67.7MB
hello-world latest 9c7a54a9a43c 7 months ago 13.3kB
wurstmeister/kafka latest a692873757c0 19 months ago 468MB
timberio/vector 0.10.0-alpine 2ebd263245dd 3 years ago 80.7MB
wurstmeister/zookeeper latest 3f43f72cb283 4 years ago 510MB
[root@localhost grafana_loki_vector]#
6. 配置grafana
6.1 添加loki 数据源
如果找不到data Source, 可以直接search 查找。
6.2 修改时区
6.3 添加query
click 'Home' -> Explore -> change 'loki' option
6.4 过滤展示kafka 消息
原始kafka 消息数据：
{
"currentIndex":0,
"header":{
"headerMap":{
"35":"381166"
}
},
"mapList":[
{
"10007":"1"
},
{
"37":"48",
"38":"300000",
"44":"15007900",
"48":"688021",
"1301":"1",
"11001":"国盾量子",
"11436":"20231227",
"28000":"20231227",
"28001":"1703642600915",
"28005":"2",
"28101":"893786271",
"43220":"金额:[3000000,450416980],数量:[300000,300100],价格涨跌幅:[0.02,0.19934],市场成交量占比:[0.3,0.967752],区间内最高卖出成交价: 1799.8,昨收盘价: 1500,下单数量: 300000,当前交易价格: 1500.79,挂单数量: 0,挂单金额: 0,成交数量: 100,成交金额: 179980,市场成交数量: 10100",
"64101":"2403",
"64102":"1101",
"64103":"2403-4096个账户-4096个证券",
"65098":"1",
"66002":"5",
"66003":"1",
"66004":"0",
"66005":"2",
"91001":"200000",
"93020":"2"
}
]
}
将以上kafka 原始消息数据，筛选过滤出指定字段，语句如下：
# 将以上kafka 原始消息数据，筛选过滤出指定字段，语句如下：
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"64103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``
6.5 优化后 dashboard, 展示kafka 指定字段
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]"|line_format "{{.mapList1}}"| __error__=``
6.5.1 添加添加dashboard
Clieck 'Add visualizaion' Button or click 'Add'-> Visualization
6.5.2 优化dashboard, 显示指定字段
6.5.2.1 筛选kafka 原始消息
clieck Query 输入 将 kafka 原始消息数据，筛选过滤出指定字段语句
过滤语句：
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"64103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``
注意：
key=<vector.toml 中sinks.out中的自定义的labels.key>
[root@localhost vector]# cat vector.toml
# __ __ __
# \ \ / / / /
# \ V / / /
# \_/ \/
#
# V E C T O R
# Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------
# Note: A full config spec is located at ./vector.spec.toml and examples
# in the ./examples folder.
# Vector config kafka + Loki
[ sources.in ]
type = "kafka"
bootstrap_servers = "192.168.2.247:9092" # kafka 地址
group_id = "kafka_mag_groupId1225" # "<消费组id>"
topics = ["alarm_warning","kafka_consumer"] # kafka_consumer
auto_offset_reset = "smallest" # 从最早未处理的消息开始消费
# topics = ["^(prefix1|prefix2)-.+", "topic-1", "topic-2"] # topic名字，支持正则
#[sources.in.encoding]
#codec = "json"
#timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
#json.fields = ["level", "message"]
[sinks.out]
endpoint = "http://loki:3100"
inputs = ["in"] # source.in
type = "loki"
labels.key = "mag1225" # 自定义key
# labels.key = "{{ event_field }}" # event的动态值
#[sinks.out.encoding]
#codec = "json"
# timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
# json.fields = ["level", "message"]
[root@localhost vector]#
6.5.2.2 优化表格显示指定字段
# 优化已表格显示指定字段
1. Click 'Transform data' --> Add Transformation --> Reformat -> Extract fields
2. 添加Extract files -->Source:Line -->Format:JSON --> 打开 'Replacee all fields' --> 打开'Keep time'
重名列名->调整顺序->显示隐藏指定列，配置如下：
3. click 'Add another transformation' --> 'Reorder and rename' --> 'Organize fields by name'
7. 其他异常问题
7.1 grafana 页面有时跳转 网址 http://localhost:3000/XXXX 错误
grafana 容器所在主机IP:192.168.2.247
grafana web address: http://192.168.2.247:3000
grafana web 页面有时操作跳转 网址：http://localhost:3000/XXXX,导致页面跳转错误
7.2 跳转错误网址原因定位
原因定位：
Grafana Explore 页面上的 "Add to Dashboard" 后，跳转网址为 "http://localhost:3000/..." 的问题可能是因为 Grafana 配置中使用了相对 URL 路径导致的
7.3 解决方法
7.3.1 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址
1. 修改 docker-compose-grafana-loki-vector.yml 文件，
将 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址
environment:
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000
[root@localhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.yml
# docker-compose配置参考：https://github.com/grafana/loki/blob/master/production/docker-compose.yaml
version: "3"
# 网桥loki -> 方便相互通讯
networks:
loki:
services:
# Loki日志系统
loki:
image: grafana/loki:latest
container_name: grafana_loki_vector_demo_loki
restart: unless-stopped
ports:
- "3100:3100"
volumes:
- ./grafana_loki_vector/loki:/etc/loki
- ./grafana_loki_vector/wal:/wal
command: -config.file=/etc/loki/loki-config.yaml
networks:
- loki
# 用于UI展示
grafana:
image: grafana/grafana:latest
container_name: grafana_loki_vector_demo_grafana
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- "./grafana_loki_vector/grafana/data:/var/lib/grafana"
# - "./grafana_promtail_loki/grafana/config:/etc/grafana"
- "./grafana_loki_vector/grafana/log:/var/log/grafana"
environment:
# GF_EXPLORE_ENABLED: "true"
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000 # 修改正确网址 grafana 容器所在主机
networks:
- loki
# vector 用于对接kafka 消息
vector:
image: timberio/vector:0.10.0-alpine
container_name: grafana_loki_vector_demo_vector
restart: unless-stopped
ports:
- "80:80"
volumes:
# - ./grafana_loki_vector/vector/logs:/var/logs
- "./grafana_loki_vector/vector:/etc/vector"
command: -c /etc/vector/vector.toml -v
networks:
- loki
[root@localhost grafana_loki_vector]#
7.3.2 为了grafana 有数据，必须修改vector.tml 配置参数
# 为了grafana 展示kafka 数据，必须修改vector.tml 文件中2个参数
1. group_id = "kafka_mag_groupId1228" # 任意值 从 *1225 改为 *1228
2. labels.key = "mag1228" # 自定义key 从 *1225 改为 *1228
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat grafana_loki_vector/vector/vector.toml
# __ __ __
# \ \ / / / /
# \ V / / /
# \_/ \/
#
# V E C T O R
# Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------
# Note: A full config spec is located at ./vector.spec.toml and examples
# in the ./examples folder.
# Vector config kafka + Loki
[ sources.in ]
type = "kafka"
bootstrap_servers = "192.168.2.247:9092" # kafka 地址
group_id = "kafka_mag_groupId1228" # "<消费组id>" 修改group_ID
topics = ["alarm_warning","kafka_consumer"] # kafka_consumer
auto_offset_reset = "smallest" # 从最早未处理的消息开始消费
# topics = ["^(prefix1|prefix2)-.+", "topic-1", "topic-2"] # topic名字，支持正则
#[sources.in.encoding]
#codec = "json"
#timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
#json.fields = ["level", "message"]
[sinks.out]
endpoint = "http://loki:3100"
inputs = ["in"] # source.in
type = "loki"
labels.key = "mag1228" # 自定义key
# labels.key = "{{ event_field }}" # event的动态值
#[sinks.out.encoding]
#codec = "json"
# timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
# json.fields = ["level", "message"]
[root@localhost grafana_loki_vector]#
7.3.3 使配置生效
# 使配置生效
2. docker-compose 关闭项目
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
3. docker-copose 启动项目即可。
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
# docker-compose 关闭项目
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[+] Running 4/4
✔ Container grafana_loki_vector_demo_vector Removed 0.5s
✔ Container grafana_loki_vector_demo_loki Removed 1.7s
✔ Container grafana_loki_vector_demo_grafana Removed 0.4s
✔ Network grafana_loki_vector_loki Removed 0.1s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
# docker-copose 启动项目
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[+] Running 4/4
✔ Network grafana_loki_vector_loki Created 0.1s
✔ Container grafana_loki_vector_demo_vector Started 0.1s
✔ Container grafana_loki_vector_demo_grafana Started 0.1s
✔ Container grafana_loki_vector_demo_loki Started 0.1s
[root@localhost grafana_loki_vector]#
7.4 验证 跳转网址正确
在新页面打开，跳转网址正确 http://192.168.2.247:3000/XXXXX
到此基本完成。
当然如果仅仅是想通过 Grafana 监控kafka，推荐使用 Grafana Prometheus 通过JMX监控kafka