minikube dashboard ImagePullBackOff 失败问题的解决方法

问题

通过minikube启动k8s集群后，我们想安装启动k8s的dashboard插件，执行minikube dashboard后，一直卡住直到提示验证失败。

查看pod状态，我们看到kubernetes-dashboard下的2个pod实例的状态都是ImagePullBackOff

[dev ~]# kubectl get pod --all-namespaces
NAMESPACE              NAME                                         READY   STATUS             RESTARTS       AGE
kube-system            coredns-76fccbbb6b-wbx9v                     1/1     Running            0              110m
kube-system            etcd-minikube                                1/1     Running            0              110m
kube-system            kube-apiserver-minikube                      1/1     Running            0              110m
kube-system            kube-controller-manager-minikube             1/1     Running            0              110m
kube-system            kube-proxy-grk7h                             1/1     Running            0              110m
kube-system            kube-scheduler-minikube                      1/1     Running            0              110m
kube-system            storage-provisioner                          1/1     Running            1 (110m ago)   110m
kubernetes-dashboard   dashboard-metrics-scraper-5d59dccf9b-pzt8r   0/1     ImagePullBackOff   0              108m
kubernetes-dashboard   kubernetes-dashboard-7779f9b69b-zz2wm        0/1     ImagePullBackOff   0              108m

我们查看具体的失败原因可以看到：

• 查看 dashboard-metrics-scraper 失败原因

[dev ~]# kubectl describe pod dashboard-metrics-scraper-5d59dccf9b-pzt8r --namespace=kubernetes-dashboard
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Pulling  34m (x19 over 109m)     kubelet  Pulling image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c"
  Warning  Failed   24m (x21 over 109m)     kubelet  Failed to pull image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c": Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: server misbehaving
  Normal   BackOff  4m50s (x442 over 109m)  kubelet  Back-off pulling image "docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c"
  Warning  Failed   4m37s (x443 over 109m)  kubelet  Error: ImagePullBackOff

• 查看 kubernetes-dashboard 失败原因

[dev ~]# kubectl describe pod kubernetes-dashboard-7779f9b69b-zz2wm --namespace=kubernetes-dashboard
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Normal   Pulling  36m (x20 over 116m)   kubelet  Pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"
  Warning  Failed   31m (x21 over 116m)   kubelet  Failed to pull image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93": Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: server misbehaving
  Normal   BackOff  96s (x486 over 116m)  kubelet  Back-off pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"
  Warning  Failed   82s (x487 over 116m)  kubelet  Error: ImagePullBackOff

2个pod实例启动失败原因都是dashboard所需镜像pull失败导致。

解决办法

手动pull镜像，并设置tag

拉取失败的总共有2个镜像：

• docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c
• docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93

因为pull镜像失败的网络问题无法解决，我们只能pull国内镜像，并给镜像打上原始镜像名称的标记。

1. 进入minikube的环境

我们需要先进入minikube的环境，如果直接在host环境中拉取镜像，minikube是获取不到的。

minikube ssh

进入环境后我们可以执行 docker image ls -a查看当前环境的镜像列表，其中没有包含所缺失的镜像。

2. 拉取国内镜像，并打上原始镜像的tag

# 手动pull metrics-scraper:v1.0.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8 docker.io/kubernetesui/metrics-scraper:v1.0.8

# 手动pull dashboard:v2.7.0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0 docker.io/kubernetesui/dashboard:v2.7.0

3. 查看现有镜像

[dev ~]# minikube ssh
docker@minikube:~$ docker image ls -a
REPOSITORY                                                                    TAG        IMAGE ID       CREATED         SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.32.0    c2e17b8d0f4a   5 months ago    97MB
registry.k8s.io/kube-apiserver                                                v1.32.0    c2e17b8d0f4a   5 months ago    97MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.32.0    8cab3d2a8bd0   5 months ago    89.7MB
registry.k8s.io/kube-controller-manager                                       v1.32.0    8cab3d2a8bd0   5 months ago    89.7MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.32.0    a389e107f4ff   5 months ago    69.6MB
registry.k8s.io/kube-scheduler                                                v1.32.0    a389e107f4ff   5 months ago    69.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.32.0    040f9f8aac8c   5 months ago    94MB
registry.k8s.io/kube-proxy                                                    v1.32.0    040f9f8aac8c   5 months ago    94MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen      v1.4.4     a62eeff05ba5   7 months ago    63.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.5.16-0   a9e7e6b294ba   8 months ago    150MB
registry.k8s.io/etcd                                                          3.5.16-0   a9e7e6b294ba   8 months ago    150MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   v1.11.3    c69fa2e9cbf5   9 months ago    61.8MB
registry.k8s.io/coredns/coredns                                               v1.11.3    c69fa2e9cbf5   9 months ago    61.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.10       873ed7510279   12 months ago   736kB
registry.k8s.io/pause                                                         3.10       873ed7510279   12 months ago   736kB
kubernetesui/dashboard                                                        v2.7.0     07655ddf2eeb   2 years ago     246MB
registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard                 v2.7.0     07655ddf2eeb   2 years ago     246MB
registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper           v1.0.8     115053965e86   2 years ago     43.8MB
kubernetesui/metrics-scraper                                                  v1.0.8     115053965e86   2 years ago     43.8MB
gcr.io/k8s-minikube/storage-provisioner                                       v5         6e38f40d628d   4 years ago     31.5MB
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner       v5         6e38f40d628d   4 years ago     31.5MB

缺失的镜像列表都弄好了。

停掉当前的dashboard

这个步骤可以通过kubectl在外部删掉启动失败的2个pod，或者在minikube内部手动运行kubectl删掉都可以。

1. 在外部删掉

   kubectl delete pod dashboard-metrics-scraper-5d59dccf9b-pzt8r --namespace kubernetes-dashboard
   kubectl delete pod kubernetes-dashboard-7779f9b69b-zz2wm --namespace kubernetes-dashboard

2. 进入minikube环境，手动运行kubectl 删掉dashboard的pod

[dev ~]# minikube ssh
docker@minikube:~$ sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.32.0/kubectl delete -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
namespace "kubernetes-dashboard" deleted
clusterrole.rbac.authorization.k8s.io "kubernetes-dashboard" deleted
clusterrolebinding.rbac.authorization.k8s.io "kubernetes-dashboard" deleted
configmap "kubernetes-dashboard-settings" deleted
deployment.apps "dashboard-metrics-scraper" deleted
deployment.apps "kubernetes-dashboard" deleted
role.rbac.authorization.k8s.io "kubernetes-dashboard" deleted
rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard" deleted
serviceaccount "kubernetes-dashboard" deleted
secret "kubernetes-dashboard-certs" deleted
secret "kubernetes-dashboard-csrf" deleted
secret "kubernetes-dashboard-key-holder" deleted
service "kubernetes-dashboard" deleted
service "dashboard-metrics-scraper" deleted

编辑dashboard的deployment配置文件，修改其中的镜像地址

1. 我们需要先进入minikube环境

minikube ssh

2. 查看配置

sudo cat /etc/kubernetes/addons/dashboard-dp.yaml

配置太长了，只截取关注的这部分：

kind: Deployment
apiVersion: apps/v1
spec:
    spec:
      containers:
        - name: dashboard-metrics-scraper
          image: docker.io/kubernetesui/metrics-scraper:v1.0.8@sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c
---

kind: Deployment
apiVersion: apps/v1
spec:
    spec:
      containers:
        - name: kubernetes-dashboard
          # WARNING: This must match pkg/minikube/bootstrapper/images/images.go
          image: docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93
          ports:
            - containerPort: 9090
              protocol: TCP

可以看到，pull失败的镜像地址，就是在这个配置文件里配置，接下来，我们需要修改这个配置文件，将image地址中的sha256删掉，并添加镜像pull规则：

    spec:
      containers:
        - name: dashboard-metrics-scraper
          image: docker.io/kubernetesui/metrics-scraper:v1.0.8
          imagePullPolicy: IfNotPresent
----

    spec:
      containers:
        - name: kubernetes-dashboard
          # WARNING: This must match pkg/minikube/bootstrapper/images/images.go
          image: docker.io/kubernetesui/dashboard:v2.7.0
          imagePullPolicy: IfNotPresent

主要是2个字段：

• image 将@和后面的字符串删掉，因为我们国内的镜像这个sha256和配置中的看起来并不一样。
• imagePullPolicy 设置为 IfNotPresent，表示镜像存在就不会再pull一次。

手动启动dashboard

docker@minikube:~$ sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.32.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
namespace/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
configmap/kubernetes-dashboard-settings created
Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: non-functional in v1.27+; use the "seccompProfile" field instead
deployment.apps/dashboard-metrics-scraper created
deployment.apps/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
service/kubernetes-dashboard created
service/dashboard-metrics-scraper created

然后退出minikube环境，查看dashboard的pod运行状态：

[dev ~]# kubectl get pod -A
NAMESPACE              NAME                                          READY   STATUS              RESTARTS         AGE
kube-system            coredns-76fccbbb6b-vtdbf                      1/1     Running             5 (12m ago)      13h
kube-system            etcd-minikube                                 1/1     Running             10 (12m ago)     2d
kube-system            kube-apiserver-minikube                       1/1     Running             60 (13m ago)     2d
kube-system            kube-controller-manager-minikube              1/1     Running             34 (9m42s ago)   2d
kube-system            kube-proxy-4n788                              1/1     Running             1 (12m ago)      2d
kube-system            kube-scheduler-minikube                       1/1     Running             12 (12m ago)     2d
kubernetes-dashboard   dashboard-metrics-scraper-545b5664b6-mcrzh    1/1     Running             0                5m27s
kubernetes-dashboard   kubernetes-dashboard-7f67ccd984-ql4cn         1/1     Running             0                5m27s