Kubernetes V1.24+ & Docker运行时 grafana容器指标显示异常

文章目录

• • 故障复现
  • • [Compatibility Matrix](#Compatibility Matrix)
    • 根本原因
    • • [获取kubelet cadvisor 指标](#获取kubelet cadvisor 指标)
  • 如何解决？
  • • [创建cadvisor ServiceAccount 和 Namespace](#创建cadvisor ServiceAccount 和 Namespace)
    • [创建cadvisor ClusterRole](#创建cadvisor ClusterRole)
    • [创建cadvisor DaemonSet](#创建cadvisor DaemonSet)
    • [创建cadvisor Service 和 ServiceMonitor](#创建cadvisor Service 和 ServiceMonitor)
    • 查看cadvisor收集到的指标
  • 校验指标

参考:

https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/

在 Kubernetes v1.24 版本中，内建组件 dockershim 被移除。 默认的容器运行时从 Docker 切换到了 containerd。如果您的集群使用 Docker 作为容器运行时，并且您的应用程序依赖于 dockershim 提供的指标,

那么部署kube-prometheus-stack 监控栈时，会出现容器指标显示异常的问题。

故障复现

根据kubernetes版本和kube-prometheus的兼容矩阵安装合适版本的kube-prometheus-stack, 容器指标却采集不到。

Compatibility Matrix

The following Kubernetes versions are supported and work as we test against these versions in their respective branches. But note that other versions might work!

kube-prometheus stack	Kubernetes 1.22	Kubernetes 1.23	Kubernetes 1.24	Kubernetes 1.25	Kubernetes 1.26	Kubernetes 1.27	Kubernetes 1.28
release-0.10	✔	✔	✗	✗	x	x	x
release-0.11	✗	✔	✔	✗	x	x	x
release-0.12	✗	✗	✔	✔	x	x	x
release-0.13	✗	✗	✗	x	✔	✔	✔
main	✗	✗	✗	x	x	✔	✔

比如我的版本是kubernetes 1.26, 那么我就需要安装kube-prometheus-stack release-0.13 版本。

如果我的容器运行时是containerd, 那么正常情况，容器指标显示是没有问题的。因为kube-prometheus-stack就是根据标准kubernetes版本去测试的。

但是如果你的容器运行时用了docker, 那么就会出现容器指标显示异常的问题。

根本原因

容器的指标是通过kubelet的metrics-endpoint暴露的，路径为/metrics/cadvisor,由内置cadvisor提供。

本来默认是containerd运行时，如果你的运行时换成了docker，就会采集容器指标变成：

容器的PromSQL查询语句：

sum(container_memory_working_set_bytes{job="kubelet", metrics_path="/metrics/cadvisor", cluster="$cluster", namespace="$namespace", pod="$pod", container!="", image!=""}) by (container)

以docker运行时查询出来的容器指标，会有多个标签是空值,如下所示：

container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20aada51_3ac1_4ab7_8ec6_87b090585825.slice",image="",name="",namespace="monitoring",pod="node-exporter-v25n7"} 327.638786615 176252734624

其中的container, image标签都是空值，因此面板监控显示的容器指标就是异常的。

获取kubelet cadvisor 指标

apiVersion: v1
kind: Secret
metadata:
  name: cadvisor-reader-token
  namespace: default
  annotations:
    kubernetes.io/service-account.name: cadvisor-reader  # 关联到目标 ServiceAccount
type: kubernetes.io/service-account-token

创建一个ServiceAccount,用于kubelet认证。

# 保存为 custom-kubelet-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-kubelet-reader
rules:
- apiGroups: [""]
  resources:
  - "nodes/proxy"           # 允许访问节点代理（包含 cAdvisor 接口）
  - "nodes/metrics"         # 节点 metrics 接口
  - "nodes/stats"           # 节点统计信息（包含容器详细数据）
  - "nodes/log"             # 节点日志（可选）
  - "nodes/spec"            # 节点规格（可选）
  verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-cadvisor-reader-binding  # 与之前同名，会覆盖旧绑定
subjects:
- kind: ServiceAccount
  name: cadvisor-reader
  namespace: default
roleRef:
  kind: ClusterRole
  name: custom-kubelet-reader  # 引用自定义角色
  apiGroup: rbac.authorization.k8s.io

• get-metrics.sh

# 定义 Secret 名称（已确认存在）
SECRET_NAME=cadvisor-reader-token

# 提取 Token 并解码（base64 解码）
TOKEN=$(kubectl get secret $SECRET_NAME -n default -o jsonpath='{.data.token}' | base64 -d)

curl -k -H "Authorization: Bearer $TOKEN" https://192.168.0.4:10250/metrics/cadvisor

就可以获取到cadvisor指标了。

如何解决？

需要安装cadvisor, 来替代kubelet默认提供的cadvisor。

参考：https://github.com/google/cadvisor/tree/master/deploy/kubernetes

创建cadvisor ServiceAccount 和 Namespace

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cadvisor
  namespace: monitoring
---
apiVersion: v1
kind: Namespace
metadata:
  name: cadvisor

创建cadvisor ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: cadvisor
  name: cadvisor
rules:
- apiGroups:
  - policy
  resourceNames:
  - cadvisor
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: cadvisor
  name: cadvisor
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cadvisor
subjects:
- kind: ServiceAccount
  name: cadvisor
  namespace: monitoring

创建cadvisor DaemonSet

apiVersion: apps/v1 # for Kubernetes versions before 1.9.0 use apps/v1beta2
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      name: cadvisor
  template:
    metadata:
      labels:
        name: cadvisor
    spec:
      serviceAccountName: cadvisor
      priorityClassName: system-node-critical
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      containers:
      - name: cadvisor
        image: dockerhub.kubekey.local/cadvisor/cadvisor:v0.45.0
        args:
          - --housekeeping_interval=10s                           # kubernetes default args
          - --max_housekeeping_interval=15s
          - --event_storage_event_limit=default=0
          - --event_storage_age_limit=default=0
          - --enable_metrics=app,cpu,disk,diskIO,memory,network,process
          - --docker_only                                         # only show stats for docker containers
          - --store_container_labels=false
          - --whitelisted_container_labels=io.kubernetes.container.name, io.kubernetes.pod.name,io.kubernetes.pod.namespace


        resources:
          requests:
            memory: 400Mi
            cpu: 400m
          limits:
            memory: 2000Mi
            cpu: 800m
        volumeMounts:
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
        - name: var-run
          mountPath: /var/run
          readOnly: true
        - name: sys
          mountPath: /sys
          readOnly: true
        - name: docker
          mountPath: /var/lib/docker
          readOnly: true
        - name: disk
          mountPath: /dev/disk
          readOnly: true
        ports:
          - name: http
            containerPort: 8080
            protocol: TCP
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      volumes:
      - name: rootfs
        hostPath:
          path: /
      - name: var-run
        hostPath:
          path: /var/run
      - name: sys
        hostPath:
          path: /sys
      - name: docker
        hostPath:
          path: /var/lib/docker
      - name: disk
        hostPath:
          path: /dev/disk

创建cadvisor Service 和 ServiceMonitor

apiVersion: v1
kind: Service
metadata:
  name: cadvisor
  labels:
    app: cadvisor
  namespace: monitoring
spec:
  selector:
    name: cadvisor
  type: NodePort
  ports:
  - name: cadvisor
    port: 8080
    protocol: TCP
    targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: cadvisor
  labels:
    app: cadvisor
  namespace: monitoring
spec:
  selector:
    name: cadvisor
  type: NodePort
  ports:
  - name: cadvisor
    port: 8080
    protocol: TCP
    targetPort: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: cadvisor
  name: cadvisor
  namespace: cadvisor
spec:
  endpoints:
    - metricRelabelings:
        - action: replace
          sourceLabels:
            - container_label_io_kubernetes_pod_name
          targetLabel: pod
        - action: replace
          sourceLabels:
            - container_label_io_kubernetes_container_name
          targetLabel: container
        - action: replace
          sourceLabels:
            - container_label_io_kubernetes_pod_namespace
          targetLabel: namespace
        - action: labeldrop
          regex: container_label_io_kubernetes_pod_name
        - action: labeldrop
          regex: container_label_io_kubernetes_container_name
        - action: labeldrop
          regex: container_label_io_kubernetes_pod_namespace
        - action: replace
          regex: 'k8s_[^_]+_([^_]+)_.*'
          replacement: $1
          sourceLabels:
            - name
          targetLabel: pod
      port: cadvisor
      relabelings:
        - action: replace
          sourceLabels:
            - __meta_kubernetes_pod_node_name
          targetLabel: node
        - action: replace
          replacement: /metrics/cadvisor
          sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
        - action: replace
          replacement: kubelet
          sourceLabels:
            - job
          targetLabel: job
  namespaceSelector:
    matchNames:
      - monitoring
  selector:
    matchLabels:
      app: cadvisor

给promethues用户足够的权限获取cadvisor命名空间下的所有资源：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-all-resources-readonly
rules:
- apiGroups: ["*"]  # 匹配所有 API 组（核心组、apps、networking.k8s.io 等）
  resources: ["*"]  # 匹配所有资源类型（pods、services、endpoints、deployments 等）
  verbs: ["get", "list", "watch"]  # 仅授予只读操作权限
---
# 绑定到 prometheus-k8s ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-k8s-all-readonly
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: prometheus-all-resources-readonly
  apiGroup: rbac.authorization.k8s.io

为什么要创建一个cadvisor命名空间？

这是因为monitoring下的ServiceMonitor对象，Prometheus-Operator监控不到。

查看cadvisor收集到的指标

通过cadvisor的NodePort端口，访问cadvisor的指标。

http://<cadvisor-node-ip>:<node-port>/metrics

我们会看到如下指标：

container_network_receive_bytes_total{container_label_io_kubernetes_container_name="POD",container_label_io_kubernetes_pod_namespace="kube-system",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60538598_0b4c_49c0_9577_c6b600e287d6.slice/docker-5e54348343024541ec3f9dbe65649f77a701f7a8808c84f8fbdfeaa20f625b20.scope",image="gcr.io/pause:3.9",interface="califffbc9ee416",name="k8s_POD_nodelocaldns-8trx7_kube-system_60538598-0b4c-49c0-9577-c6b600e287d6_1"} 5.168659e+07 1762570436207

可以看到container、image、namespace字段都有了正确的默认值。

但是pod字段为空，不过我们可以通过name字段来推测它的值，它的组成模式k8s_POD_nodelocaldns-8trx7_kube-system_60538598-0b4c-49c0-9577-c6b600e287d6_1为k8s_$deployment_$pod_name_$namespace_$pod_uid。

那么此时，我们可以通过给添加metricRelabelings来提取pod字段(yaml已包含)：

    - metricRelabelings:
        - action: replace
          regex: 'k8s_[^_]+_([^_]+)_.*'
          replacement: $1
          sourceLabels:
            - name
          targetLabel: pod

观察prometheus是否采集到了cadvisor的指标：

如果已经采集到了，那么此时我们还有一件事要做，就是找到kubelet的ServiceMonitor对象，删除默认的采集规则

使用kubectl get servicemonitor -n monitoring kubelet -o yaml来获取kubelet的ServiceMonitor对象。

删除路径为/metrics/cadvisor的采集规则,也就是如下这部分。

- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      interval: 1m
      metricRelabelings:
        - action: keep
          regex: >-
            container_cpu_usage_seconds_total|container_memory_usage_bytes|container_memory_cache|container_network_.+_bytes_total|container_memory_working_set_bytes|container_cpu_cfs_.*periods_total|container_processes.*|container_threads.*
          sourceLabels:
            - __name__
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
        - action: labeldrop
          regex: (service|endpoint)
      scheme: https
      tlsConfig:
        insecureSkipVerify: true

校验指标

Okay，所有准备工作已经完成，现在我们可以进入prometheus的web界面，查看是否采集到了cadvisor的指标。

container_memory_working_set_bytes{container="nfs-client-provisioner", endpoint="cadvisor", id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3db91060_4f1e_41b0_becc_6f4a69e86319.slice/docker-f1bc1a1d88993cc65eb09ae5b2a7f47cf596c21c7f9990d6523ba1fa997c28ab.scope", image="sha256:932b0bface75b80e713245d7c2ce8c44b7e127c075bd2d27281a16677c8efef3", instance="10.233.82.36:8080", job="kubelet", metrics_path="/metrics/cadvisor", name="k8s_nfs-client-provisioner_nfs-provisioner-nfs-client-provisioner-97d76bf9-jx7h8_default_3db91060-4f1e-41b0-becc-6f4a69e86319_1", namespace="default", node="node01", pod="nfs-provisioner-nfs-client-provisioner-97d76bf9-jx7h8", service="cadvisor"}

如图所示：我们发现prometheus之前采集的指标中为空的label，比如image、container、pod、namespace等都有了正确的值。

查看面板：

CPU & 内存指标显示正常：

网络流量指标显示正常：