HAMi + prometheus-k8s + grafana实现vgpu虚拟化监控

狂奔solar2025-01-10 13:58

最近长沙跑了半个多月，跟甲方客户对了下项目指标，许久没更新

回来后继续研究如何实现 grafana实现HAMi vgpu虚拟化监控，毕竟合同里写了需要体现gpu资源限制和算力共享以及体现算力卡资源共享监控

先说下为啥要用HAMi吧， 一个重要原因是公司有人引见了这个工具的作者， 很多问题我都可以直接向作者提问

HAMi，是一个国产的GPU与国产加速卡（支持的GPU与国产加速卡型号与具体特性请查看此项目官网：https://github.com/Project-HAMi/HAMi/）虚拟化开源项目，实现以kubernetes为基础的容器场景下GPU或加速卡虚拟化。HAMi原名"k8s-vGPU-scheduler"，

最初由我司开源，现已在国内与国际上愈加流行，是管理Kubernetes中异构设备的中间件。它可以管理不同类型的异构设备（如GPU、NPU等），在Pod之间共享异构设备，根据设备的拓扑信息和调度策略做出更好的调度决策。为了阐述的简明性，本文只提供一种可行的办法，最终实现使用prometheus抓取监控指标并作为数据源、使用grafana来展示监控信息的目的。

本文假定已经部署好Kubernetes集群、HAMi。以下涉及到的相关组件都是在kubernetes集群内安装的，相关组件或软件版本信息如下：

组件或软件名称 版本 备注
kubernetes集群 v1.23.1 AMD64构架服务器环境下
HAMi 根据向开源作者提问，当前HAMi版本发行机制还不够成熟，暂以安装HAMi的scheduler.kubeScheduler.imageTag 参数值为其版本，此值要跟kubernetes版本看齐 项目地址：https://github.com/Project-HAMi/HAMi/
kube-prometheus stack prom/prometheus:v2.27.1 关于监控的安装参见实现prometheus+grafana的监控部署_prometheus grafana监控部署-CSDN博客
dcgm-exporter nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04

HAMi 的默认安装方式是通过helm，添加Helm仓库:

复制代码 
helm repo add hami-charts https://project-hami.github.io/HAMi/

检查Kubernetes版本并安装HAMi（服务器版本为1.23.1）:

复制代码 
helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag=v1.23.1 -n kube-system

验证hami安装成功

复制代码 
kubectl get pods -n kube-system

确认hami-device-plugin和hami-scheduler都处于Running状态表示安装成功。

把helm安装转为hami-install.yaml

复制代码 
helm template hami hami-charts/hami --set scheduler.kubeScheduler.imageTag=v1.23.1 -n kube-system > hami-install.yaml

该格式部署

复制代码 
---
# Source: hami/templates/device-plugin/monitorserviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hami-device-plugin
  namespace: "kube-system"
  labels:
    app.kubernetes.io/component: "hami-device-plugin"
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
---
# Source: hami/templates/scheduler/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hami-scheduler
  namespace: "kube-system"
  labels:
    app.kubernetes.io/component: "hami-scheduler"
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
---
# Source: hami/templates/device-plugin/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hami-device-plugin
  labels:
    app.kubernetes.io/component: hami-device-plugin
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
data:
  config.json: |
    {
        "nodeconfig": [
            {
                "name": "m5-cloudinfra-online02",
                "devicememoryscaling": 1.8,
                "devicesplitcount": 10,
                "migstrategy":"none",
                "filterdevices": {
                  "uuid": [],
                  "index": []
                }
            }
        ]
    }
---
# Source: hami/templates/scheduler/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hami-scheduler
  labels:
    app.kubernetes.io/component: hami-scheduler
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
data:
  config.json: |
    {
        "kind": "Policy",
        "apiVersion": "v1",
        "extenders": [
            {
                "urlPrefix": "https://127.0.0.1:443",
                "filterVerb": "filter",
                "bindVerb": "bind",
                "enableHttps": true,
                "weight": 1,
                "nodeCacheCapable": true,
                "httpTimeout": 30000000000,
                "tlsConfig": {
                    "insecure": true
                },
                "managedResources": [
                    {
                        "name": "nvidia.com/gpu",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "nvidia.com/gpumem",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "nvidia.com/gpucores",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "nvidia.com/gpumem-percentage",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "nvidia.com/priority",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "cambricon.com/vmlu",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "hygon.com/dcunum",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "hygon.com/dcumem",
                        "ignoredByScheduler": true 
                    },
                    {
                        "name": "hygon.com/dcucores",
                        "ignoredByScheduler": true
                    },
                    {
                        "name": "iluvatar.ai/vgpu",
                        "ignoredByScheduler": true
                    }
                ],
                "ignoreable": false
            }
        ]
    }
---
# Source: hami/templates/scheduler/configmapnew.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hami-scheduler-newversion
  labels:
    app.kubernetes.io/component: hami-scheduler
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
data:
  config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: false
    profiles:
    - schedulerName: hami-scheduler
    extenders:
    - urlPrefix: "https://127.0.0.1:443"
      filterVerb: filter
      bindVerb: bind
      nodeCacheCapable: true
      weight: 1
      httpTimeout: 30s
      enableHTTPS: true
      tlsConfig:
        insecure: true
      managedResources:
      - name: nvidia.com/gpu
        ignoredByScheduler: true
      - name: nvidia.com/gpumem
        ignoredByScheduler: true
      - name: nvidia.com/gpucores
        ignoredByScheduler: true
      - name: nvidia.com/gpumem-percentage
        ignoredByScheduler: true
      - name: nvidia.com/priority
        ignoredByScheduler: true
      - name: cambricon.com/vmlu
        ignoredByScheduler: true
      - name: hygon.com/dcunum
        ignoredByScheduler: true
      - name: hygon.com/dcumem
        ignoredByScheduler: true
      - name: hygon.com/dcucores
        ignoredByScheduler: true
      - name: iluvatar.ai/vgpu
        ignoredByScheduler: true
---
# Source: hami/templates/scheduler/device-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hami-scheduler-device
  labels:
    app.kubernetes.io/component: hami-scheduler
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
data:
  device-config.yaml: |-
    nvidia:
      resourceCountName: nvidia.com/gpu
      resourceMemoryName: nvidia.com/gpumem
      resourceMemoryPercentageName: nvidia.com/gpumem-percentage
      resourceCoreName: nvidia.com/gpucores
      resourcePriorityName: nvidia.com/priority
      overwriteEnv: false
      defaultMemory: 0
      defaultCores: 0
      defaultGPUNum: 1
      deviceSplitCount: 10
      deviceMemoryScaling: 1
      deviceCoreScaling: 1
    cambricon:
      resourceCountName: cambricon.com/vmlu
      resourceMemoryName: cambricon.com/mlu.smlu.vmemory
      resourceCoreName: cambricon.com/mlu.smlu.vcore
    hygon:
      resourceCountName: hygon.com/dcunum
      resourceMemoryName: hygon.com/dcumem
      resourceCoreName: hygon.com/dcucores
    metax:
      resourceCountName: "metax-tech.com/gpu"
    mthreads:
      resourceCountName: "mthreads.com/vgpu"
      resourceMemoryName: "mthreads.com/sgpu-memory"
      resourceCoreName: "mthreads.com/sgpu-core"
    iluvatar: 
      resourceCountName: iluvatar.ai/vgpu
      resourceMemoryName: iluvatar.ai/vcuda-memory
      resourceCoreName: iluvatar.ai/vcuda-core
    vnpus:
    - chipName: 910B
      commonWord: Ascend910A
      resourceName: huawei.com/Ascend910A
      resourceMemoryName: huawei.com/Ascend910A-memory
      memoryAllocatable: 32768
      memoryCapacity: 32768
      aiCore: 30
      templates:
        - name: vir02
          memory: 2184
          aiCore: 2
        - name: vir04
          memory: 4369
          aiCore: 4
        - name: vir08
          memory: 8738
          aiCore: 8
        - name: vir16
          memory: 17476
          aiCore: 16
    - chipName: 910B3
      commonWord: Ascend910B
      resourceName: huawei.com/Ascend910B
      resourceMemoryName: huawei.com/Ascend910B-memory
      memoryAllocatable: 65536
      memoryCapacity: 65536
      aiCore: 20
      aiCPU: 7
      templates:
        - name: vir05_1c_16g
          memory: 16384
          aiCore: 5
          aiCPU: 1
        - name: vir10_3c_32g
          memory: 32768
          aiCore: 10
          aiCPU: 3
    - chipName: 310P3
      commonWord: Ascend310P
      resourceName: huawei.com/Ascend310P
      resourceMemoryName: huawei.com/Ascend310P-memory
      memoryAllocatable: 21527
      memoryCapacity: 24576
      aiCore: 8
      aiCPU: 7
      templates:
        - name: vir01
          memory: 3072
          aiCore: 1
          aiCPU: 1
        - name: vir02
          memory: 6144
          aiCore: 2
          aiCPU: 2
        - name: vir04
          memory: 12288
          aiCore: 4
          aiCPU: 4
---
# Source: hami/templates/device-plugin/monitorrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name:  hami-device-plugin-monitor
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - create
      - watch
      - list
      - update
      - patch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - update
      - list
      - patch
---
# Source: hami/templates/device-plugin/monitorrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hami-device-plugin
  labels:
    app.kubernetes.io/component: "hami-device-plugin"
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  #name: cluster-admin
  name: hami-device-plugin-monitor
subjects:
  - kind: ServiceAccount
    name: hami-device-plugin
    namespace: "kube-system"
---
# Source: hami/templates/scheduler/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hami-scheduler
  labels:
    app.kubernetes.io/component: "hami-scheduler"
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: hami-scheduler
    namespace: "kube-system"
---
# Source: hami/templates/device-plugin/monitorservice.yaml
apiVersion: v1
kind: Service
metadata:
  name: hami-device-plugin-monitor
  labels:
    app.kubernetes.io/component: hami-device-plugin
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
spec:
  externalTrafficPolicy: Local
  selector:
    app.kubernetes.io/component: hami-device-plugin
  type: NodePort
  ports:
    - name: monitorport
      port: 31992
      targetPort: 9394
      nodePort: 31992
---
# Source: hami/templates/scheduler/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: hami-scheduler
  labels:
    app.kubernetes.io/component: hami-scheduler
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: NodePort
  ports:
    - name: http
      port: 443
      targetPort: 443
      nodePort: 31998
      protocol: TCP
    - name: monitor
      port: 31993
      targetPort: 9395
      nodePort: 31993
      protocol: TCP
  selector:
    app.kubernetes.io/component: hami-scheduler
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
---
# Source: hami/templates/device-plugin/daemonsetnvidia.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: hami-device-plugin
  labels:
    app.kubernetes.io/component: hami-device-plugin
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: hami-device-plugin
      app.kubernetes.io/name: hami
      app.kubernetes.io/instance: hami
  template:
    metadata:
      labels:
        app.kubernetes.io/component: hami-device-plugin
        hami.io/webhook: ignore
        app.kubernetes.io/name: hami
        app.kubernetes.io/instance: hami
    spec:
      imagePullSecrets: 
        []
      serviceAccountName: hami-device-plugin
      priorityClassName: system-node-critical
      hostPID: true
      hostNetwork: true
      containers:
        - name: device-plugin
          image: projecthami/hami:latest
          imagePullPolicy: "IfNotPresent"
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh","-c", "cp -f /k8s-vgpu/lib/nvidia/* /usr/local/vgpu/"]
          command:
            - nvidia-device-plugin
            - --config-file=/device-config.yaml
            - --mig-strategy=none
            - --disable-core-limit=false
            - -v=false
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: NVIDIA_MIG_MONITOR_DEVICES
              value: all
            - name: HOOK_PATH
              value: /usr/local
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
              add: ["SYS_ADMIN"]
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
            - name: lib
              mountPath: /usr/local/vgpu
            - name: usrbin
              mountPath: /usrbin
            - name: deviceconfig
              mountPath: /config
            - name: hosttmp
              mountPath: /tmp
            - name: device-config
              mountPath: /device-config.yaml
              subPath: device-config.yaml
        - name: vgpu-monitor
          image: projecthami/hami:latest
          imagePullPolicy: "IfNotPresent"
          command: ["vGPUmonitor"]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
              add: ["SYS_ADMIN"]
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "all"
            - name: NVIDIA_MIG_MONITOR_DEVICES
              value: all
            - name: HOOK_PATH
              value: /usr/local/vgpu              
          volumeMounts:
            - name: ctrs
              mountPath: /usr/local/vgpu/containers
            - name: dockers
              mountPath: /run/docker
            - name: containerds
              mountPath: /run/containerd
            - name: sysinfo
              mountPath: /sysinfo
            - name: hostvar
              mountPath: /hostvar
      volumes:
        - name: ctrs
          hostPath:
            path: /usr/local/vgpu/containers
        - name: hosttmp
          hostPath:
            path: /tmp
        - name: dockers
          hostPath:
            path: /run/docker
        - name: containerds
          hostPath:
            path: /run/containerd
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
        - name: lib
          hostPath:
            path: /usr/local/vgpu
        - name: usrbin
          hostPath:
            path: /usr/bin
        - name: sysinfo
          hostPath:
            path: /sys
        - name: hostvar
          hostPath:
            path: /var
        - name: deviceconfig
          configMap:
            name: hami-device-plugin
        - name: device-config
          configMap:
            name: hami-scheduler-device
      nodeSelector: 
        gpu: "on"
---
# Source: hami/templates/scheduler/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hami-scheduler
  labels:
    app.kubernetes.io/component: hami-scheduler
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: hami-scheduler
      app.kubernetes.io/name: hami
      app.kubernetes.io/instance: hami
  template:
    metadata:
      labels:
        app.kubernetes.io/component: hami-scheduler
        app.kubernetes.io/name: hami
        app.kubernetes.io/instance: hami
        hami.io/webhook: ignore
    spec:
      imagePullSecrets: 
        []
      serviceAccountName: hami-scheduler
      priorityClassName: system-node-critical
      containers:
        - name: kube-scheduler
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.31.0
          imagePullPolicy: "IfNotPresent"
          command:
            - kube-scheduler
            - --config=/config/config.yaml
            - -v=4
            - --leader-elect=true
            - --leader-elect-resource-name=hami-scheduler
            - --leader-elect-resource-namespace=kube-system
          volumeMounts:
            - name: scheduler-config
              mountPath: /config
        - name: vgpu-scheduler-extender
          image: projecthami/hami:latest
          imagePullPolicy: "IfNotPresent"
          env:
          command:
            - scheduler
            - --http_bind=0.0.0.0:443
            - --cert_file=/tls/tls.crt
            - --key_file=/tls/tls.key
            - --scheduler-name=hami-scheduler
            - --metrics-bind-address=:9395
            - --node-scheduler-policy=binpack
            - --gpu-scheduler-policy=spread
            - --device-config-file=/device-config.yaml
            - --debug
            - -v=4
          ports:
            - name: http
              containerPort: 443
              protocol: TCP
          volumeMounts:
            - name: tls-config
              mountPath: /tls
            - name: device-config
              mountPath: /device-config.yaml
              subPath: device-config.yaml
      volumes:
        - name: tls-config
          secret:
            secretName: hami-scheduler-tls
        - name: scheduler-config
          configMap:
            name: hami-scheduler-newversion
        - name: device-config
          configMap:
            name: hami-scheduler-device
---
# Source: hami/templates/scheduler/webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: hami-webhook
webhooks:
  - admissionReviewVersions:
    - v1beta1
    clientConfig:
      service:
        name: hami-scheduler
        namespace: kube-system
        path: /webhook
        port: 443
    failurePolicy: Ignore
    matchPolicy: Equivalent
    name: vgpu.hami.io
    namespaceSelector:
      matchExpressions:
      - key: hami.io/webhook
        operator: NotIn
        values:
        - ignore
    objectSelector:
      matchExpressions:
      - key: hami.io/webhook
        operator: NotIn
        values:
        - ignore
    reinvocationPolicy: Never
    rules:
      - apiGroups:
          - ""
        apiVersions:
          - v1
        operations:
          - CREATE
        resources:
          - pods
        scope: '*'
    sideEffects: None
    timeoutSeconds: 10
---
# Source: hami/templates/scheduler/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hami-admission
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
---
# Source: hami/templates/scheduler/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hami-admission
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      #- validatingwebhookconfigurations
      - mutatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: hami/templates/scheduler/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name:  hami-admission
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hami-admission
subjects:
  - kind: ServiceAccount
    name: hami-admission
    namespace: "kube-system"
---
# Source: hami/templates/scheduler/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name:  hami-admission
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: hami/templates/scheduler/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: hami-admission
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: hami-admission
subjects:
  - kind: ServiceAccount
    name: hami-admission
    namespace: "kube-system"
---
# Source: hami/templates/scheduler/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: hami-admission-create
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: hami-admission-create
      labels:
        helm.sh/chart: hami-2.4.0
        app.kubernetes.io/name: hami
        app.kubernetes.io/instance: hami
        app.kubernetes.io/version: "2.4.0"
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
        hami.io/webhook: ignore
    spec:
      imagePullSecrets: 
        []
      containers:
        - name: create
          image: liangjw/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --cert-name=tls.crt
            - --key-name=tls.key
            - --host=hami-scheduler.kube-system.svc,127.0.0.1
            - --namespace=kube-system
            - --secret-name=hami-scheduler-tls
      restartPolicy: OnFailure
      serviceAccountName: hami-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: hami/templates/scheduler/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: hami-admission-patch
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: hami-2.4.0
    app.kubernetes.io/name: hami
    app.kubernetes.io/instance: hami
    app.kubernetes.io/version: "2.4.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: hami-admission-patch
      labels:
        helm.sh/chart: hami-2.4.0
        app.kubernetes.io/name: hami
        app.kubernetes.io/instance: hami
        app.kubernetes.io/version: "2.4.0"
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
        hami.io/webhook: ignore
    spec:
      imagePullSecrets: 
        []
      containers:
        - name: patch
          image: liangjw/kube-webhook-certgen:v1.1.1
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=hami-webhook
            - --namespace=kube-system
            - --patch-validating=false
            - --secret-name=hami-scheduler-tls
      restartPolicy: OnFailure
      serviceAccountName: hami-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000

部署dcgm-exporter

复制代码 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: "dcgm-exporter"
  labels:
    app.kubernetes.io/name: "dcgm-exporter"
    app.kubernetes.io/version: "3.6.1"
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: "dcgm-exporter"
      app.kubernetes.io/version: "3.6.1"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: "dcgm-exporter"
        app.kubernetes.io/version: "3.6.1"
      name: "dcgm-exporter"
    spec:
      containers:
      - image: "nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04"
        env:
        - name: "DCGM_EXPORTER_LISTEN"
          value: ":9400"
        - name: "DCGM_EXPORTER_KUBERNETES"
          value: "true"
        name: "dcgm-exporter"
        ports:
        - name: "metrics"
          containerPort: 9400
        securityContext:
          runAsNonRoot: false
          runAsUser: 0
          capabilities:
            add: ["SYS_ADMIN"]
        volumeMounts:
        - name: "pod-gpu-resources"
          readOnly: true
          mountPath: "/var/lib/kubelet/pod-resources"
      volumes:
      - name: "pod-gpu-resources"
        hostPath:
          path: "/var/lib/kubelet/pod-resources"

---

kind: Service
apiVersion: v1
metadata:
  name: "dcgm-exporter"
  labels:
    app.kubernetes.io/name: "dcgm-exporter"
    app.kubernetes.io/version: "3.6.1"
spec:
  selector:
    app.kubernetes.io/name: "dcgm-exporter"
    app.kubernetes.io/version: "3.6.1"
  ports:
  - name: "metrics"
    port: 9400

dcgm-exporter安装成功

参考这个hami-vgpu dashboard 下载panel 的json文件

hami-vgpu-dashboard | Grafana Labs 导入后grafana中将创建一个名为"hami-vgpu-dashboard"的dashboard，但此页面中有一些Panel如vGPUCorePercentage还没有数据

ServiceMonitor 是 Prometheus Operator 中的一个自定义资源，主要用于监控 Kubernetes 中的服务。它的作用包括：

1. 自动化发现

ServiceMonitor 允许 Prometheus 自动发现和监控 Kubernetes 中的服务。通过定义 ServiceMonitor，您可以告诉 Prometheus 监控特定服务的端点。

2. 配置抓取参数

您可以在 ServiceMonitor 中设置抓取的相关参数，例如：

  • 抓取间隔：定义 Prometheus 多频繁抓取数据（如每 30 秒）。
  • 超时：定义抓取请求的超时时间。
  • 标签选择器：指定要监控的服务的标签，确保 Prometheus 仅抓取相关服务的数据。

dcgm-exporter需要配置两个service monitor

hami-device-plugin-svc-monitor.yaml

复制代码 
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hami-device-plugin-svc-monitor
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: hami-device-plugin
  namespaceSelector:
    matchNames:
      - kube-system
  endpoints:
    - path: /metrics
      port: monitorport
      interval: "15s"
      honorLabels: false
      relabelings:
        - sourceLabels: [__meta_kubernetes_endpoints_name]
          regex: hami-.*
          replacement: $1
          action: keep
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          regex: (.*)
          targetLabel: node_name
          replacement: ${1}
          action: replace
        - sourceLabels: [__meta_kubernetes_pod_host_ip]
          regex: (.*)
          targetLabel: ip
          replacement: $1
          action: replace

hami-scheduler-svc-monitor.yaml

复制代码 
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hami-scheduler-svc-monitor
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: hami-scheduler
  namespaceSelector:
    matchNames:
      - kube-system
  endpoints:
    - path: /metrics
      port: monitor
      interval: "15s"
      honorLabels: false
      relabelings:
        - sourceLabels: [__meta_kubernetes_endpoints_name]
          regex: hami-.*
          replacement: $1
          action: keep
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          regex: (.*)
          targetLabel: node_name
          replacement: ${1}
          action: replace
        - sourceLabels: [__meta_kubernetes_pod_host_ip]
          regex: (.*)
          targetLabel: ip
          replacement: $1
          action: replace

确认创建的ServiceMonitor

启动gpu pod一个测试下

复制代码 
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod-1
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.2.1
      command: ["sleep", "infinity"]
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 1000
          nvidia.com/gpucores: 10

如果看到pod一直pending 状态

检查下节点如果出现下面gpu为0的情况

需要

复制代码 
   docker：
		1:下载NVIDIA-DOCKER2安装包并安装
		2:修改/etc/docker/daemon.json文件内容加上
			{
			"default-runtime": "nvidia",
				"runtimes": {
					"nvidia": {
						"path": "/usr/bin/nvidia-container-runtime",
						"runtimeArgs": []
					}
				},
			}
	k8s:
		1:下载k8s-device-plugin 镜像
		2:编写nvidia-device-plugin.yml创建驱动pod

使用这个yml进行创建

复制代码 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      priorityClassName: "system-node-critical"
      containers:
      - image: nvidia/k8s-device-plugin:1.11
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

gpu pod启动后进入查看下， gpu内存和限制的大小相同设置成功

访问下{scheduler node ip}:31993/metrics

日志最后有两行

复制代码 
vGPUPodsDeviceAllocated{containeridx="0",deviceusedcore="40",deviceuuid="GPU-7666e9de-679b-a768-51c6-260b81cd00ec",nodename="192.168.110.126",podname="gpu-pod-1",podnamespace="default",zone="vGPU"} 1.048576e+10
vGPUPodsDeviceAllocated{containeridx="0",deviceusedcore="40",deviceuuid="GPU-7666e9de-679b-a768-51c6-260b81cd00ec",nodename="192.168.110.126",podname="gpu-pod-2",podnamespace="default",zone="vGPU"} 1.048576e+10

可以看到相同deviceuuid的gpu被不同pod共享使用

exec进入hami-device-plugin daemonset里面执行nvidia-smi -L 可以看到机器上所有显卡的信息

root@node126:/# nvidia-smi -L

GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-7666e9de-679b-a768-51c6-260b81cd00ec)

GPU 1: NVIDIA GeForce RTX 4090 (UUID: GPU-9f32af29-1a72-6e47-af2c-72b1130a176b)

root@node126:/#

之前创建的两个serviceMonitor会去请求

app.kubernetes.io/component: hami-scheduler 和app.kubernetes.io/component: hami-device-plugin 的/metrics 接口获取数据

当gpu-pod跑起来以后查看hami-vgpu-metrics-dashboard

相关推荐
企鹅侠客1 天前
Prometheus operator怎么添加targets和告警规则
运维·云原生·kubernetes·prometheus·pod
木二2 天前
附042.Kubernetes_v1.32.3生成环境高可用部署
云原生·kubernetes·prometheus·ingress·longhorn
yunson_Liu3 天前
kubernet在prometheus+alertmanager+grafana框架下新增部署loki模块
grafana·prometheus·loki
树下一少年3 天前
docker-compose部署prometheus+grafana+node_exporter
docker·json·grafana·prometheus·node_exporter
云上艺旅9 天前
K8S学习之基础四十一：Prometheus基于Pushgateway采集数据
学习·云原生·容器·kubernetes·prometheus
云上艺旅9 天前
K8S学习之基础三十八：Kube-static-metrics监控
学习·云原生·容器·kubernetes·prometheus
sj116373940312 天前
Prometheus Exporter系列-Mysql_Exporter一键部署
prometheus
云上艺旅13 天前
K8S学习之基础三十六：node-exporter部署
学习·云原生·贪心算法·kubernetes·prometheus
云上艺旅13 天前
K8S学习之基础四十：K8S配置altermanager发送告警到钉钉群
学习·云原生·容器·kubernetes·钉钉·prometheus
云上艺旅14 天前
K8S学习之基础三十五：k8s之Prometheus部署模式
学习·云原生·容器·kubernetes·prometheus
热门推荐
01我决定放弃搞 Java 了02DeepSeek各版本说明与优缺点分析03如何在WPS和Word/Excel中直接使用DeepSeek功能04本地化部署AI知识库：基于Ollama+DeepSeek+AnythingLLM保姆级教程05苍穹外卖面试总结06从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑07循环神经网络RNN（含LSTM，GRU）小综述08RAG 实践- Ollama+RagFlow 部署本地知识库09如何本地部署AI智能体平台，带你手搓一个AI Agent10Android下HWC以及drm_hwcomposer普法(上)