06 | Prometheus打通FastAPI云原生数据采集

APP(FastAPI + llama.cpp)运行在 Mac 的 Docker 里,Prometheus 运行在 Docker

Desktop 的 Kubernetes 里。目标是让 Prometheus 抓取 FastAPI 的 /metrics,并能在

Prometheus 里查询 up 等时序数据。

实际架构

text 复制代码

MacBook M4
├── Docker Desktop
│   ├── llm-serving
│   │   └── FastAPI:宿主机端口 8000
│   │       └── GET /metrics
│   └── llm-serving-llama
│       └── 模型推理后端:宿主机端口 8081
│
└── Docker Desktop Kubernetes
    └── namespace: observability
        ├── kube-prometheus-stack Prometheus
        ├── Service: llm-serving-docker
        ├── Endpoints: 192.168.65.254:8000
        └── ServiceMonitor: llm-serving-docker

采集链路

text 复制代码

FastAPI /metrics
    ↓
Mac 宿主机可达地址 192.168.65.254:8000
    ↓
Kubernetes Service + Endpoints(手动定义外部目标)
    ↓
ServiceMonitor
    ↓
Prometheus Operator 生成 scrape job
    ↓
Prometheus 每 30 秒抓取一次
    ↓
PromQL 查询与后续 Grafana 展示

关键设计 :APP 不需要部署到 Kubernetes。通过没有 selector 的 Service 加上手动编写的 Endpoints,把 Kubernetes 内部的 ServiceMonitor 发现机制指向集群外的 Mac 宿主机服务。

注意:192.168.65.254 是本次环境中实际验证可达的地址,勿假设适用于其他环境。

过程概览

步骤	操作	关键内容
1	确认环境与前提	Docker Desktop、kube-prometheus-stack、`observability` 命名空间
2	在 Docker 中运行 APP	llama.cpp 映射 8081,FastAPI 映射 8000,FastAPI 通过 `host.docker.internal` 访问后端
3	确认 APP 暴露指标	宿主机 `curl /metrics`,看到 `http_requests_total` 等指标
4	从 Kubernetes Pod 测试连通性	临时 Pod 访问 `http://192.168.65.254:8000/metrics` 返回 200
5	创建 Service、Endpoints、ServiceMonitor	无 selector 的 Service + 手动 Endpoints,标签与端口名对齐
6	确认 Operator 选择并生成配置	Prometheus CR 的 selector 为 `{}`,Secret 中出现 scrape job
7	访问 Prometheus 并查询	`up{service="llm-serving-docker"}` 返回 `1`

验证结果

Pod → 宿主机:Kubernetes 临时 Pod 访问 192.168.65.254:8000/metrics,返回 HTTP/1.1 200 OK,正文 6545 字节(终端日志)。
Operator 生成的配置里出现 job serviceMonitor/observability/llm-serving-docker/0,抓取间隔 30s、超时 10s(接入说明)。
Prometheus 查询 up{service="llm-serving-docker"},instance="192.168.65.254:8000",值为 "1"(接入说明)。

环境信息(来自日志与说明)

项目	值
宿主机	MacBook M4,Docker Desktop(容器运行时 `docker://28.4.0`)
Kubernetes	Docker Desktop Kubernetes,Server `v1.32.2`(kubectl Client `v1.33.4`),节点 `docker-desktop`,INTERNAL-IP `192.168.65.3`
监控组件	kube-prometheus-stack(Helm chart `91.8.2`,app version `v0.94.1`),Prometheus 镜像 `quay.io/prometheus/prometheus:v3.15.0-distroless`
监控命名空间	`observability`
Prometheus Pod / Service	`prometheus-kube-prometheus-stack-prometheus-0` / `kube-prometheus-stack-prometheus`
Prometheus 本地端口转发	`127.0.0.1:19090`
FastAPI	宿主机端口 `8000`,指标路径 `/metrics`
llama.cpp	宿主机端口 `8081`
Kubernetes 可达地址	`192.168.65.254:8000`
ServiceMonitor 抓取间隔 / 超时	`30s` / `10s`
镜像	`harbor.local:30080/library/llm-serving:13`、`harbor.local:30080/library/llm-serving-llama:13`

一、前提条件

接入前确认:

APP 已在 Docker 中启动并映射宿主机端口 8000;
GET /metrics 返回 HTTP 200,并输出 Prometheus exposition 格式指标;
Kubernetes Pod 能访问该宿主机地址;
kube-prometheus-stack 的 Prometheus CR 能选择该 ServiceMonitor;
Service 的标签与 ServiceMonitor selector 匹配,Endpoints 的端口名与 ServiceMonitor 的 endpoints[].port 一致。

二、在 Docker 中运行 APP

2.1 清理旧容器,确认集群内没有旧工作负载

先把本机上同名的旧容器删掉(2>/dev/null || true 让"容器不存在"时也不报错),并确认 Kubernetes 的 llm-serving 命名空间里没有同名的旧工作负载,避免两套 APP 同时存在:

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % docker rm -f llm-serving llm-serving-llama 2>/dev/null || true

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n llm-serving delete deployment \
  llm-serving llm-serving-llama

kubectl -n llm-serving delete service \
  llm-serving llm-serving-llama
Error from server (NotFound): deployments.apps "llm-serving" not found
Error from server (NotFound): deployments.apps "llm-serving-llama" not found
Error from server (NotFound): services "llm-serving" not found
Error from server (NotFound): services "llm-serving-llama" not found

四条 NotFound 说明集群里的 Deployment 和 Service 已经不存在,这是期望的状态。

2.2 运行 llama.cpp 后端容器

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % docker run -d \
  --name llm-serving-llama \
  --restart unless-stopped \
  -p 8081:8080 \
  harbor.local:30080/library/llm-serving-llama:13
1c7589edb59f8f7366e28df0b4b14b4beb534a905988301cb87700a38dc15bb3

要点:

镜像要写完整的 Harbor 地址 加标签(harbor.local:30080/library/llm-serving-llama:13);只写 llm-serving-llama:latest 时,本地没有这个名字的镜像,Docker 会去 Docker Hub 找而失败。
容器内 llama-server 监听 8080,映射到宿主机 8081(宿主机 8080 留给别的用途);
--restart unless-stopped:Docker 重启或容器异常退出后自动拉起,除非手动停止。

补充 :-p 8081:8080 会把端口绑定到宿主机所有接口(日志里看到 0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp)。llama.cpp 没有设置 API Key 且 CORS 允许所有来源,如果只想本机访问,可以写成 -p 127.0.0.1:8081:8080。

查看状态和启动日志(docker ps 里还有一大批 k8s_* 容器,是 Docker Desktop Kubernetes 的组件,这里只保留表头和本文相关的一行):

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % docker ps
docker logs --tail 100 llm-serving-llama
CONTAINER ID   IMAGE                                             COMMAND                  CREATED          STATUS                    PORTS                                         NAMES
1c7589edb59f   harbor.local:30080/library/llm-serving-llama:13   "/app/llama-server -..."   55 seconds ago   Up 54 seconds (healthy)   0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp   llm-serving-llama
...
warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
warn: LLAMA_ARG_PORT environment variable is set, but will be overwritten by command line argument --port
0.00.002.903 I srv  llama_server: initializing ...
0.00.014.657 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.015.000 W srv  llama_server: security: no API key is set and CORS allows all origins (see https://github.com/ggml-org/llama.cpp/pull/25655)
0.00.017.353 I srv    load_model: loading model '/models/qwen2-0_5b-instruct-q4_k_m.gguf'
0.00.253.094 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.00.773.333 I cmn          init: llama threadpool init, n_threads = 10
0.00.936.774 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 2048, kv_unified = 'true'
0.00.938.952 I srv  llama_server: model loaded
0.00.938.962 I srv  llama_server: listening on http://0.0.0.0:8080

容器状态是 (healthy),模型不到 1 秒就加载完毕并开始监听。

2.3 验证 llama.cpp 后端

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % curl -i --max-time 10 http://127.0.0.1:8081/health
curl -i --max-time 10 http://127.0.0.1:8081/v1/models
HTTP/1.1 200 OK
Server: llama.cpp
Access-Control-Allow-Origin:
Content-Type: application/json; charset=utf-8
Content-Length: 15
Keep-Alive: timeout=5, max=100

{"status":"ok"}HTTP/1.1 200 OK
Server: llama.cpp
Access-Control-Allow-Origin:
Content-Type: application/json; charset=utf-8
Content-Length: 665
Keep-Alive: timeout=5, max=100

{"models":[{"name":"qwen2:0.5b","model":"qwen2:0.5b","modified_at":"","size":"","digest":"","type":"model","description":"","tags":[""],"capabilities":["completion"],"parameters":"","details":{"parent_model":"","format":"gguf","family":"","families":[""],"parameter_size":"","quantization_level":""}}],"object":"list","data":[{"id":"qwen2:0.5b","aliases":["qwen2:0.5b"],"tags":[],"object":"model","architecture":{"input_modalities":["text"],"output_modalities":["text"]},"created":1791812109,"owned_by":"llamacpp","meta":{"vocab_type":2,"n_vocab":151936,"n_ctx":2048,"n_ctx_train":32768,"n_embd":896,"n_params":494032768,"size":391859712,"ftype":"Q4_K - Medium"}}]}%

再做一次最小的推理请求:

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % curl -sS --max-time 60 http://127.0.0.1:8081/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen2:0.5b",
    "messages": [
      {"role": "user", "content": "Reply with OK"}
    ]
  }'
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"OK"}}],"created":1791812135,"model":"qwen2:0.5b","system_fingerprint":"b11561-e937b17a2","object":"chat.completion","usage":{"completion_tokens":2,"prompt_tokens":22,"total_tokens":24,"prompt_tokens_details":{"cached_tokens":0}},"id":"chatcmpl-Ua9zd1lGB2X1u4QIVrXOedjx2U042lh2","timings":{"cache_n":0,"prompt_n":22,"prompt_ms":662.079,"prompt_per_token_ms":30.094499999999996,"prompt_per_second":33.2286630447424,"predicted_n":2,"predicted_ms":43.598,"predicted_per_token_ms":43.598,"predicted_per_second":22.936831964769027}}%

后端返回了 "OK",usage 里有输入 22、输出 2 个 token,timings 里还有提示词和生成的耗时。

2.4 运行 FastAPI 容器

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % docker run -d \
  --name llm-serving \
  --restart unless-stopped \
  -p 8000:8000 \
  -e LLAMA_BASE_URL=http://host.docker.internal:8081/v1 \
  -e DEFAULT_MODEL=qwen2:0.5b \
  harbor.local:30080/library/llm-serving:13
6df9a2f0bea3a3751ef81431ecdca48a0e8e2e51d07cd758ceac3a288544b35b

要点:

两个容器之间通过宿主机端口互通 :这里没有建自定义网络,FastAPI 用 host.docker.internal:8081 访问宿主机上映射出来的 llama.cpp;
环境变量 LLAMA_BASE_URL 和 DEFAULT_MODEL 与之前 Kubernetes 里的 ConfigMap llm-serving-config(LLAMA_BASE_URL: http://llm-serving-llama:8080/v1、DEFAULT_MODEL: qwen2:0.5b)是同一组配置项,只是后端地址从集群内 Service 名换成了宿主机地址;
-p 8000:8000:FastAPI 映射到宿主机 8000。

补充 :要让 Kubernetes 里的 Pod 通过宿主机地址 192.168.65.254:8000 访问到它,端口需要映射到宿主机的非回环接口(这里是默认的 0.0.0.0)。如果写成 127.0.0.1:8000:8000,Pod 的访问可能到不了------这是推测,取决于 Docker Desktop 的网络实现,以第四步的实测为准。

三、确认 APP 暴露指标

3.1 在 Mac 宿主机检查 `/metrics`

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % curl -fsS http://127.0.0.1:8000/metrics \
  | grep -E 'http_requests_total|http_request_duration_seconds'
# HELP http_requests_total Total number of requests by method, status and handler.
# TYPE http_requests_total counter
http_requests_total{handler="/metrics",method="GET",status="2xx"} 4.0
http_requests_total{handler="/health",method="GET",status="5xx"} 3.0
# HELP http_request_duration_seconds Latency with only few buckets by handler. Made to be only used if aggregation by handler is important.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{handler="/metrics",le="0.1",method="GET"} 4.0
http_request_duration_seconds_bucket{handler="/metrics",le="0.5",method="GET"} 4.0
http_request_duration_seconds_bucket{handler="/metrics",le="1.0",method="GET"} 4.0
http_request_duration_seconds_bucket{handler="/metrics",le="+Inf",method="GET"} 4.0
http_request_duration_seconds_count{handler="/metrics",method="GET"} 4.0
http_request_duration_seconds_sum{handler="/metrics",method="GET"} 0.04399662488140166
http_request_duration_seconds_bucket{handler="/health",le="0.1",method="GET"} 0.0
http_request_duration_seconds_bucket{handler="/health",le="0.5",method="GET"} 1.0
http_request_duration_seconds_bucket{handler="/health",le="1.0",method="GET"} 3.0
http_request_duration_seconds_bucket{handler="/health",le="+Inf",method="GET"} 3.0
http_request_duration_seconds_count{handler="/health",method="GET"} 3.0
http_request_duration_seconds_sum{handler="/health",method="GET"} 1.9657583741936833
# HELP http_request_duration_seconds_created Latency with only few buckets by handler. Made to be only used if aggregation by handler is important.
# TYPE http_request_duration_seconds_created gauge
http_request_duration_seconds_created{handler="/metrics",method="GET"} 1.791785135285204e+09
http_request_duration_seconds_created{handler="/health",method="GET"} 1.791785154335754e+09

这说明 /metrics 能正常返回 Prometheus 格式的指标,并且 HTTP 请求计数和耗时直方图已经有真实的数据:/metrics 自己被请求了 4 次(2xx),/health 被请求过 3 次且状态是 5xx。/health 的 3 次 5xx 是这个实例此前健康检查的结果,日志里没有说明具体原因(从 Helm 渲染出的 Deployment 注释看,FastAPI 的 /health 在后端不可用时会返回 503)。

3.2 指标内容说明

下一步从 Kubernetes 里拿到的完整输出里,指标分成两部分:

来源	指标
Python 客户端默认指标	`python_gc_objects_collected_total`、`python_gc_objects_uncollectable_total`、`python_gc_collections_total`、`python_info`
HTTP 层指标	`http_requests_total`、`http_requests_created`、`http_request_size_bytes`、`http_response_size_bytes`、`http_request_duration_highr_seconds`、`http_request_duration_seconds`

补充 :这组 HTTP 指标名与 prometheus-fastapi-instrumentator 的默认指标一致(这是根据名称的推测,日志里没有直接说明用的是哪个库)。输出里没有 process_* 一类的进程指标 (如 process_resident_memory_bytes),也没有 Token、推理耗时等 LLM 业务指标;后面写 PromQL 要以实际存在的指标为准。

四、从 Kubernetes Pod 测试网络连通性

在创建 Service / ServiceMonitor 之前,先验证 Pod 到 Mac 宿主机的网络路径:

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n observability run metrics-connectivity-test \
  --rm -it \
  --restart=Never \
  --image=curlimages/curl \
  -- curl -i --max-time 5 http://192.168.65.254:8000/metrics
HTTP/1.1 200 OK
date: Mon, 12 Oct 2026 13:36:53 GMT
server: uvicorn
content-length: 6545
content-type: text/plain; version=1.0.0; charset=utf-8

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 503.0
python_gc_objects_collected_total{generation="1"} 50.0
python_gc_objects_collected_total{generation="2"} 49.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 152.0
python_gc_collections_total{generation="1"} 13.0
python_gc_collections_total{generation="2"} 1.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="0",version="3.11.0"} 1.0
# HELP http_requests_total Total number of requests by method, status and handler.
# TYPE http_requests_total counter
http_requests_total{handler="/metrics",method="GET",status="2xx"} 5.0
http_requests_total{handler="/health",method="GET",status="5xx"} 3.0
# HELP http_requests_created Total number of requests by method, status and handler.
# TYPE http_requests_created gauge
http_requests_created{handler="/metrics",method="GET",status="2xx"} 1.7917851352851572e+09
http_requests_created{handler="/health",method="GET",status="5xx"} 1.7917851543354108e+09
# HELP http_request_size_bytes Content length of incoming requests by handler. Only value of header is respected. Otherwise ignored. No percentile calculated.
# TYPE http_request_size_bytes summary
http_request_size_bytes_count{handler="/metrics"} 5.0
http_request_size_bytes_sum{handler="/metrics"} 0.0
http_request_size_bytes_count{handler="/health"} 3.0
http_request_size_bytes_sum{handler="/health"} 0.0
...
# HELP http_request_duration_highr_seconds Latency with many buckets but no API specific labels. Made for more accurate percentile calculations.
# TYPE http_request_duration_highr_seconds histogram
http_request_duration_highr_seconds_bucket{le="0.01"} 2.0
http_request_duration_highr_seconds_bucket{le="0.025"} 4.0
http_request_duration_highr_seconds_bucket{le="0.05"} 5.0
...
http_request_duration_highr_seconds_bucket{le="+Inf"} 8.0
http_request_duration_highr_seconds_count 8.0
http_request_duration_highr_seconds_sum 2.042814124142751
...
# HELP http_request_duration_seconds Latency with only few buckets by handler. Made to be only used if aggregation by handler is important.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{handler="/metrics",le="0.1",method="GET"} 5.0
...
http_request_duration_seconds_sum{handler="/health",method="GET"} 1.9657583741936833
...
pod "metrics-connectivity-test" deleted

完整输出共 6545 字节,这里省略了几段重复结构的 _created、_bucket 行。

结果判断

HTTP 200 且有指标正文(本次结果):网络路径和 APP 的指标端点都可用,可以继续配置 Kubernetes 监控资源。
Connection refused:检查 APP 容器是否运行、宿主机端口映射是否正确、服务是否监听了正确的接口。
Operation timed out:检查 Docker Desktop 网络、宿主机地址和防火墙。
404:主机和端口已连通,但这个路径可能不是 APP 的指标端点。
连到了错误的服务:检查端口映射以及目标地址是否指向正确的容器。

要点:

192.168.65.254 与 Kubernetes 节点的 INTERNAL-IP 192.168.65.3 在同一个网段,是本次环境里 Pod 能访问到的宿主机一侧地址。
同样是"访问宿主机",Docker 容器用的是 host.docker.internal,Kubernetes Pod 用的是 192.168.65.254,两者不是同一个入口。
--rm 让测试 Pod 在命令结束后自动删除(输出最后一行 pod "metrics-connectivity-test" deleted)。
这次 /metrics 的计数是 5 次(status="2xx"),比前面宿主机上的 4 次多 1,正好是这一次访问。

五、创建 Service、Endpoints 和 ServiceMonitor

5.1 资源清单

文件路径(位于 devops-platform 仓库):

text 复制代码

observability/kube-prometheus-stack/llm-serving-docker.yaml

内容:

yaml 复制代码

apiVersion: v1
kind: Service
metadata:
  name: llm-serving-docker
  namespace: observability
  labels:
    app: llm-serving-docker
spec:
  ports:
    - name: http
      port: 8000
      targetPort: 8000
      protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: llm-serving-docker
  namespace: observability
subsets:
  - addresses:
      - ip: 192.168.65.254
    ports:
      - name: http
        port: 8000
        protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-serving-docker
  namespace: observability
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - observability
  selector:
    matchLabels:
      app: llm-serving-docker
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s

5.2 为什么需要三个资源

资源	作用	本次配置中的关键字段
Service	给监控目标建立 Kubernetes Service 身份并提供端口名	`name: llm-serving-docker`、`app: llm-serving-docker`、端口名 `http`
Endpoints	把 Service 映射到实际运行在集群外的地址	`192.168.65.254:8000`
ServiceMonitor	告诉 Prometheus Operator 选哪个 Service、用哪个端口和路径抓取	selector、`port: http`、`path: /metrics`

该 Service 没有 spec.selector,这是有意的:它不通过 Pod 标签自动发现后端,而是使用手动定义的 Endpoints 指向集群外的地址。

5.3 标签和端口名必须匹配

标签:Service 上的标签要和 ServiceMonitor 的选择器对得上。

yaml 复制代码

# Service
labels:
  app: llm-serving-docker

# ServiceMonitor
selector:
  matchLabels:
    app: llm-serving-docker

端口名 :Service、Endpoints 的端口名要和 ServiceMonitor 的 endpoints[].port 一致。

yaml 复制代码

# Service 与 Endpoints
name: http

# ServiceMonitor
endpoints:
  - port: http

ServiceMonitor.spec.endpoints[].port 引用的是 Service 端口的名称 ,不是数字 8000。本次配置里 Service 和 Endpoints 的端口名都是 http。

补充 :Kubernetes 从 v1.33 起把 Endpoints 标记为弃用,推荐改用 EndpointSlice。本环境的 Server 是 v1.32.2,手写 Endpoints 仍然可用;以后升级集群版本时,这是需要关注的地方。

5.4 应用并检查

在项目根目录运行:

bash 复制代码

kubectl apply -f observability/kube-prometheus-stack/llm-serving-docker.yaml

检查资源:

bash 复制代码

kubectl -n observability get svc,endpoints,servicemonitor llm-serving-docker

进一步核对详细配置:

bash 复制代码

kubectl -n observability get svc llm-serving-docker -o yaml
kubectl -n observability get endpoints llm-serving-docker -o yaml
kubectl -n observability get servicemonitor llm-serving-docker -o yaml

应确认 Endpoints 地址是 192.168.65.254、端口是 8000,并且 Service 标签与 ServiceMonitor 的 selector 一致。

这几条命令在终端日志里没有对应的输出,请用你自己的结果替换。

六、确认 Prometheus Operator 选择并生成了抓取配置

6.1 检查 Prometheus 资源的 ServiceMonitor 选择规则

先看当前集群里已有的 ServiceMonitor 和 observability 命名空间的 Pod 状态:

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl get servicemonitor -A
kubectl get pod -n observability
helm list -n observability
NAMESPACE       NAME                                             AGE
observability   kube-prometheus-stack-alertmanager               11d
observability   kube-prometheus-stack-apiserver                  11d
observability   kube-prometheus-stack-coredns                    11d
observability   kube-prometheus-stack-grafana                    11d
observability   kube-prometheus-stack-kube-etcd                  11d
observability   kube-prometheus-stack-kube-state-metrics         11d
observability   kube-prometheus-stack-kubelet                    11d
observability   kube-prometheus-stack-operator                   11d
observability   kube-prometheus-stack-prometheus                 11d
observability   kube-prometheus-stack-prometheus-node-exporter   11d
observability   macos-node-exporter                              11d
NAME                                                       READY   STATUS             RESTARTS         AGE
alertmanager-kube-prometheus-stack-alertmanager-0          2/2     Running            1 (39h ago)      40h
alloy-logs-54f7f58f8c-r5bm7                                1/2     Running            0                40h
kube-prometheus-stack-grafana-5f7cbbb6d4-qsms2             3/3     Running            2 (39h ago)      11d
kube-prometheus-stack-kube-state-metrics-875d5cb5f-94rm2   1/1     Running            3 (19h ago)      11d
kube-prometheus-stack-operator-7b97db9d77-mzfwg            1/1     Running            2 (19h ago)      40h
kube-prometheus-stack-prometheus-node-exporter-dqcnl       0/1     CrashLoopBackOff   1599 (77s ago)   11d
log-test                                                   1/1     Running            0                11d
loki-0                                                     2/2     Running            4 (11h ago)      40h
loki-canary-4z645                                          1/1     Running            3 (19h ago)      11d
platform-dashboard-7874d869cd-blznr                        1/1     Running            0                40h
prometheus-kube-prometheus-stack-prometheus-0              2/2     Running            2 (19h ago)      40h
tempo-698b49677c-x8v57                                     1/1     Running            2 (39h ago)      9d
NAME                 	NAMESPACE    	REVISION	UPDATED                             	STATUS  	CHART                       	APP VERSION
alloy-logs           	observability	4       	2026-10-10 22:00:16.003845 +0800 CST	deployed	alloy-1.13.0                	v1.20.0
kube-prometheus-stack	observability	5       	2026-10-10 22:00:07.431417 +0800 CST	deployed	kube-prometheus-stack-91.8.2	v0.94.1
loki                 	observability	2       	2026-10-10 21:46:01.379301 +0800 CST	deployed	loki-18.15.1                	3.7.8

可以看到:Prometheus Pod(2/2 Running)和 Operator(1/1 Running)都在正常运行;此时还没有 llm-serving-docker 这个 ServiceMonitor。列表里 node-exporter 处于 CrashLoopBackOff,与这次接入无关。

再看 Prometheus CR 的选择规则:

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n observability get prometheus -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    ...
    name: kube-prometheus-stack-prometheus
    namespace: observability
    ...
  spec:
    ...
    externalUrl: http://localhost:30080/prometheus/
    ...
    image: quay.io/prometheus/prometheus:v3.15.0-distroless
    ...
    podMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    portName: http-web
    probeNamespaceSelector: {}
    probeSelector:
      matchLabels:
        release: kube-prometheus-stack
    replicas: 1
    ...
    retention: 10d
    routePrefix: /prometheus
    ruleNamespaceSelector: {}
    ruleSelector:
      matchLabels:
        release: kube-prometheus-stack
    scrapeConfigNamespaceSelector: {}
    scrapeConfigSelector:
      matchLabels:
        release: kube-prometheus-stack
    scrapeInterval: 30s
    ...
    serviceAccountName: kube-prometheus-stack-prometheus
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector: {}
    shards: 1
    ...
  status:
    availableReplicas: 1
    conditions:
    - lastTransitionTime: "2026-10-11T11:04:56Z"
      message: ""
      observedGeneration: 5
      reason: ""
      status: "True"
      type: Available
    - lastTransitionTime: "2026-09-30T11:40:34Z"
      message: ""
      observedGeneration: 5
      reason: ""
      status: "True"
      type: Reconciled
    ...

关键点:

serviceMonitorSelector: {} 和 serviceMonitorNamespaceSelector: {} :在 Prometheus Operator 里,空的标签选择器 {} 表示匹配所有标签,空的命名空间选择器 {} 表示匹配所有命名空间,所以 observability 里的 ServiceMonitor 会被选中。
因此 ServiceMonitor 上的 release: kube-prometheus-stack 标签在本环境里不是必需的 ,但并无坏处:同一份 Prometheus CR 里的 probeSelector、ruleSelector、scrapeConfigSelector 都要求这个标签,带上它更容易在其他配置下复用。
routePrefix: /prometheus 解释了为什么后面的查询 URL 都带 /prometheus 前缀;scrapeInterval: 30s 是全局默认抓取间隔。
Available 和 Reconciled 都是 True,说明 Operator 已经把这个 Prometheus 调谐完成。

如果你的环境里这些字段缺失、为 null,或者写了明确的标签条件,要根据实际资源检查选择规则,不能直接套用本次结论。

6.2 检查 Operator 生成的 Secret

Operator 生成的抓取配置存放在:

text 复制代码

Secret: prometheus-kube-prometheus-stack-prometheus
Key: prometheus.yaml.gz

执行:

bash 复制代码

kubectl -n observability get secret \
  prometheus-kube-prometheus-stack-prometheus \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' \
  | base64 -d \
  | gunzip \
  | grep -n -B 8 -A 20 'llm-serving-docker'

本次输出(来自接入说明)确认已经生成了下面这个 job:

yaml 复制代码

job_name: serviceMonitor/observability/llm-serving-docker/0
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics

同时还生成了与 Service 标签、Endpoints 端口名匹配的 relabel 规则。

Secret 里存在该 job,说明 Operator 已经生成了配置;但这并不单独证明 Prometheus 进程已经加载了这份配置,最终以后面的 target 和查询结果为准。

七、访问 Prometheus 并查询指标

7.1 启动端口转发

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n observability port-forward \
  svc/kube-prometheus-stack-prometheus 19090:9090
Forwarding from 127.0.0.1:19090 -> 9090
Forwarding from [::1]:19090 -> 9090
Handling connection for 19090
Handling connection for 19090
Handling connection for 19090
Handling connection for 19090
Handling connection for 19090
^C%

要点:

把集群里 Prometheus Service 的 9090 端口转发到本机 19090;这个终端要保持运行,另开一个终端做查询。
每次有请求经过,终端会打印一行 Handling connection for 19090;最后的 ^C 是手动结束转发。
如果之后查询时出现 curl: (7) Failed to connect to 127.0.0.1 port 19090,通常只是端口转发进程已经退出了,重新执行 kubectl port-forward 即可,并不代表 Prometheus 有问题。

7.2 查询抓取状态

bash 复制代码

curl -sSG \
  'http://127.0.0.1:19090/prometheus/api/v1/query' \
  --data-urlencode 'query=up{service="llm-serving-docker"}' \
  | python3 -m json.tool

本次成功结果的关键字段(来自接入说明):

json 复制代码

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "up",
          "endpoint": "http",
          "instance": "192.168.65.254:8000",
          "job": "llm-serving-docker",
          "namespace": "observability",
          "service": "llm-serving-docker"
        },
        "value": [
          1791815458.234,
          "1"
        ]
      }
    ]
  }
}

解读:

status: success:Prometheus API 查询成功;
instance: 192.168.65.254:8000:被抓取的目标地址;
job: llm-serving-docker:目标所属的抓取任务,job 的值取自 Service 名;
namespace: observability:相关监控资源所在的命名空间;
value 的第二个元素为 "1":最近一次抓取成功;如果是 "0",说明目标已进入采集但抓取失败;如果 result 为空,说明没找到匹配的时序数据,需要检查 target 是否被发现、标签是否正确。

7.3 查看详细 Target 状态

bash 复制代码

curl -sS \
  'http://127.0.0.1:19090/prometheus/api/v1/targets?state=any' \
  | python3 -c '
import json, sys
data = json.load(sys.stdin)["data"]
job = "serviceMonitor/observability/llm-serving-docker/0"

for target in data.get("activeTargets", []):
    if target.get("scrapePool") == job:
        print("ACTIVE TARGET")
        print("health:", target.get("health"))
        print("scrapeUrl:", target.get("scrapeUrl"))
        print("lastError:", target.get("lastError"))
        print("lastScrape:", target.get("lastScrape"))

for target in data.get("droppedTargets", []):
    if target.get("scrapePool") == job:
        print("DROPPED TARGET")
        print(json.dumps(target, ensure_ascii=False, indent=2))
'

如果有活动目标,可以看到健康状态、抓取 URL、上次错误和上次抓取时间;如果目标出现在 droppedTargets 里,说明它被 relabel 规则丢弃了,要检查标签和端口名是否对得上。

八、常用 PromQL 查询

在 Prometheus 页面(端口转发正常时是 http://127.0.0.1:19090/prometheus/graph)可以使用下面的查询。先用这条确认该 job 下究竟有哪些指标:

promql 复制代码

{job="llm-serving-docker"}

根据第三、四节里 /metrics 的实际输出,可用的查询如下:

PromQL	用途
`up{service="llm-serving-docker"}`	APP 抓取状态(`1` 成功,`0` 失败)
`http_requests_total{job="llm-serving-docker"}`	按 `handler`、`method`、`status` 分类的请求累计数
`http_request_duration_seconds_count{job="llm-serving-docker"}`	按 handler 的请求次数
`python_gc_objects_collected_total{job="llm-serving-docker"}`	Python GC 累计回收的对象数

补充(示例,未在日志中验证):下面两条是在这些指标之上常见的写法。
promql 复制代码
# 每个 handler、每个状态类别的请求速率(5 分钟窗口)
sum by (handler, status) (rate(http_requests_total{job="llm-serving-docker"}[5m]))

# 每个 handler 的 P95 延迟
histogram_quantile(0.95,
  sum by (le, handler) (rate(http_request_duration_seconds_bucket{job="llm-serving-docker"}[5m])))
注意:接入说明里曾列出 process_resident_memory_bytes、process_cpu_seconds_total 这类进程指标,但实际的 /metrics 输出里没有它们,查询会返回空,不要依赖。

九、对照:APP 部署在集群内时的 ServiceMonitor

如果 APP 运行在 Kubernetes 里,就不需要手动 Endpoints,直接让 ServiceMonitor 选中 APP 的 Service 即可。日志里记录了这种做法(命名空间 llm-serving):

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl apply -f k8s/servicemonitor.yaml
servicemonitor.monitoring.coreos.com/llm-serving created
(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n llm-serving get servicemonitor
NAME          AGE
llm-serving   8s

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n llm-serving get servicemonitor llm-serving -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"labels":{"release":"kube-prometheus-stack"},"name":"llm-serving","namespace":"llm-serving"},"spec":{"endpoints":[{"interval":"30s","path":"/metrics","port":"http"}],"namespaceSelector":{"matchNames":["llm-serving"]},"selector":{"matchLabels":{"app":"llm-serving"}}}}
  creationTimestamp: "2026-10-12T06:30:22Z"
  generation: 1
  labels:
    release: kube-prometheus-stack
  name: llm-serving
  namespace: llm-serving
  resourceVersion: "1204100"
  uid: 6d45d2d8-7548-4276-aeed-e1822046bd27
spec:
  endpoints:
  - interval: 30s
    path: /metrics
    port: http
  namespaceSelector:
    matchNames:
    - llm-serving
  selector:
    matchLabels:
      app: llm-serving

这个 ServiceMonitor 选择的是带有标签 app: llm-serving 的 Service 。但 Helm 渲染出的 llm-serving Service 当时只有 app.kubernetes.io/managed-by=Helm 一个标签:

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n llm-serving get svc llm-serving --show-labels
kubectl -n llm-serving get endpointslice \
  -l kubernetes.io/service-name=llm-serving
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE     LABELS
llm-serving   ClusterIP   10.104.107.206   <none>        8000/TCP   3d15h   app.kubernetes.io/managed-by=Helm
NAME                ADDRESSTYPE   PORTS   ENDPOINTS     AGE
llm-serving-whcxm   IPv4          8000    10.1.19.229   3d15h

标签对不上,ServiceMonitor 就选不到这个 Service。给 Service 补上 app 标签,再用标签选择器确认能选中:

bash 复制代码

(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n llm-serving label svc llm-serving \
  app=llm-serving --overwrite
service/llm-serving labeled
(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n llm-serving get svc llm-serving --show-labels
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE     LABELS
llm-serving   ClusterIP   10.104.107.206   <none>        8000/TCP   3d15h   app.kubernetes.io/managed-by=Helm,app=llm-serving
(base) igwanglyang@MacbookProdahouzicn llm-serving % kubectl -n llm-serving get svc \
  -l app=llm-serving
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
llm-serving   ClusterIP   10.104.107.206   <none>        8000/TCP   3d16h

这一节说明了和 Docker 方案相同的核心规则:ServiceMonitor 的 selector.matchLabels 匹配的是 Service 的标签(不是 Service 的 spec.selector),port 引用的是 Service 的端口名。

补充:

kubectl label 是手动打补丁,下一次 helm upgrade 渲染出的 Service 如果没有这个标签,会把它覆盖掉。长期做法是把 app: llm-serving 写进 Chart 的 Service 模板(日志里的 helm get manifest 显示,llm-serving-llama 的 Service 模板有 app 标签,而 llm-serving 的没有)。

同一份 helm get manifest 里,llm-serving 的 NetworkPolicy 只允许来自 ingress-nginx 命名空间的流量访问 8000 端口。如果集群的网络插件强制执行 NetworkPolicy,Prometheus(在 observability 命名空间)的抓取会被挡住,需要额外放行。本文的 Docker 方案不经过这条策略。

十、后续:从基础指标扩展到 LLM 可观测性

当前打通的是"抓取链路",抓到的是 FastAPI 的 HTTP 层指标和 Python 默认指标。如果要在 Grafana 里做 LLM 监控面板,需要在 FastAPI 里补充并验证业务指标。Prometheus 无法自动推导 APP 没有暴露的 Token 或推理指标。

指标类别	建议内容	设计注意事项
请求量	请求总数、按路由或状态分类的请求数	使用 Counter,标签集合应有限(已有 `http_requests_total`)
请求延迟	请求处理时长分布	使用 Histogram,便于计算分位数(已有 `http_request_duration_*`)
错误率	请求错误总数或按状态分类的请求数	结合请求总量计算比例(可由 `status="5xx"` 算出)
Token 用量	输入 Token、输出 Token	使用 Counter,以模型等有限维度分类
推理耗时	模型调用时长	使用 Histogram
模型信息	当前使用的模型、后端状态	使用低基数标签,避免把请求 ID、用户 ID 放进标签

补充 :llama.cpp 的响应里已经带有这些原始数据(usage.prompt_tokens、usage.completion_tokens,以及 timings.prompt_per_second、timings.predicted_per_second 等,见 2.3 的输出)。FastAPI 在转发请求时可以读取它们并转成 Counter / Histogram,这样就不必重复统计。

十一、故障排查速查表

现象	优先检查
宿主机 curl 失败	APP 进程、Docker 容器状态、端口映射
Kubernetes Pod curl 失败	宿主机可达 IP、Docker Desktop 网络、防火墙、端口映射
`ServiceMonitor` 不存在	YAML 是否 apply 成功、CRD 是否安装
Secret 中没有 job	Prometheus 的 ServiceMonitor selector、namespace selector、ServiceMonitor 标签、Operator 事件/日志
Secret 中有 job,但活动 targets 里没有目标	Prometheus 实际加载的配置、target API 的 `droppedTargets`、Endpoints 发现与 relabel 规则
target 显示 DOWN	`lastError`、从 Pod 访问 `/metrics` 的结果、服务端日志
本地 `19090` 连接失败	重新执行 `kubectl port-forward` 并保持运行
`up = 1`,但没有 Token 指标	检查 APP 是否真正导出了 Token 指标,而不是继续调整 ServiceMonitor

最小验证顺序

从项目根目录执行:

bash 复制代码

# 1. 确认 Kubernetes Service、Endpoints、ServiceMonitor
kubectl -n observability get svc,endpoints,servicemonitor llm-serving-docker

# 2. 从 Kubernetes Pod 访问宿主机上的 FastAPI 指标端点
kubectl -n observability run metrics-connectivity-test \
  --rm -it --restart=Never \
  --image=curlimages/curl \
  -- curl -i --max-time 5 http://192.168.65.254:8000/metrics

# 3. 在一个终端中启动 Prometheus 端口转发
kubectl -n observability port-forward \
  svc/kube-prometheus-stack-prometheus 19090:9090

# 4. 在另一个终端查询采集状态
curl -sSG \
  'http://127.0.0.1:19090/prometheus/api/v1/query' \
  --data-urlencode 'query=up{service="llm-serving-docker"}' \
  | python3 -m json.tool

验收标准:查询结果包含 instance="192.168.65.254:8000",且 up 的值为 "1"。

06 | Prometheus打通FastAPI云原生数据采集

实际架构

采集链路

过程概览

验证结果

环境信息(来自日志与说明)

一、前提条件

二、在 Docker 中运行 APP

2.1 清理旧容器,确认集群内没有旧工作负载

2.2 运行 llama.cpp 后端容器

2.3 验证 llama.cpp 后端

2.4 运行 FastAPI 容器

三、确认 APP 暴露指标

3.1 在 Mac 宿主机检查 /metrics

3.2 指标内容说明

四、从 Kubernetes Pod 测试网络连通性

结果判断

五、创建 Service、Endpoints 和 ServiceMonitor

5.1 资源清单

5.2 为什么需要三个资源

5.3 标签和端口名必须匹配

5.4 应用并检查

六、确认 Prometheus Operator 选择并生成了抓取配置

6.1 检查 Prometheus 资源的 ServiceMonitor 选择规则

6.2 检查 Operator 生成的 Secret

七、访问 Prometheus 并查询指标

7.1 启动端口转发

7.2 查询抓取状态

7.3 查看详细 Target 状态

八、常用 PromQL 查询

九、对照:APP 部署在集群内时的 ServiceMonitor

十、后续:从基础指标扩展到 LLM 可观测性

十一、故障排查速查表

最小验证顺序

3.1 在 Mac 宿主机检查 `/metrics`