$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE READY STATUS
kube-system calico-kube-controllers-8f59968d4-zrsdt 1/1 Running 0 18m
kube-system calico-node-c257f 1/1 Running 0 18m
部署dcgm-exporter
$ helm repo add gpu-helm-charts https://nvidia.github.io/gpu-monitoring-tools/helm-charts
$ helm repo update
使用helm安装
$ helm install --generate-name gpu-helm-charts/dcgm-exporter
结果验证
$ helm ls
NAME NAMESPACE REVISION APP VERSION
dcgm-exporter-1-1601677302 default 1 dcgm-exporter-1.1.0 2.0.10
nvidia-device-plugin-1601662841 default 1 nvidia-device-plugin-0.7.0 0.7.0
Prometheus和Grafana服务暴露如下：
$ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default dcgm-exporter ClusterIP 10.99.34.128 9400/TCP 43
ddefault kubernetes ClusterIP 10.96.0.1 443/TCP 20m
kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 20m
kube-system kube-prometheus-stack-1602-coredns ClusterIP None 9153/TCP 2m18s
kube-system kube-prometheus-stack-1602-kube-controller-manager ClusterIP None 10252/TCP 2m18s
使用32032端口暴露的Grafana服务，访问Grafana主页。使用Prometheus chart中设置的凭证登录到仪表板：prometheus.values中的adminPassword字段。
现在要启动一个用于GPU指标的Grafana仪表板，请从Grafana仪表板（https://grafana.com/grafana/dashboards/12239）导入NVIDIA仪表板。