Istio 自动注入 sidecar 失败导致无法访问webhook服务

张志翔的博客2023-10-27 12:29

最近工作中在部署Istio环境的过程中发现官方示例启动的pod不能访问不到Istio的webhook，这个问题也是困扰了我一天，特此记录，便于日后查阅。

我把他归类到sidecar注入失败的情况，报错如下：

1、第一种可能（我遇到的情况）

如果自动注入时，报如下错误信息：

2023-10-26T02:15:22.051580Z error installer Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded
2023-10-26T02:15:32.149109Z error installer failed to create "EnvoyFilter/istio-system/stats-filter-1.14": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded

造成上述问题的原因是 kube-apiserver 的 --enable-admission-plugins 没有配置 MutatingAdmissionWebhook,ValidatingAdmissionWebhook参数，所以解决问题的方法就是找到 kube-apiserver.yaml 配置文件，我是通过kubeadm安装的，所以路径在 /etc/kubernetes/manifests 下，把 --enable-admission-plugins 修改为 NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook 后保存文件，删掉 kube-apiserver 的pod，让他自己根据新的配置文件重新启动（注意：如果是用kubeadm安装的修改内容如果错误，可能会导致k8s集群中的kube-apiserver全部挂掉，导致无法访问集群，这时就需要从官网下载kube-apiserver二进制文件，重新拉起一个进程，再执行删除pod的操作）

注：这里我多加了一个配置 --feature-gates=ServerSideApply=false，这个配置是关闭特性门控服务端自动更新，因为我还报了另外一个错误：failed to update resource with server-side apply for obj

修改后文件内容如下：

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.61:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.61
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --feature-gates=RemoveSelfLink=false,ServerSideApply=false
    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.1.61
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.1.61
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.1.61
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

如果上述配置你已经配过了，还是出现错误： failed to create "EnvoyFilter/istio-system/stats-filter-1.16": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded，按照下述操作排除Istio网络问题（懒得翻译咯~）：

Injection works by the API server connecting to the webhook deployment (Istiod). This may cause issues if there are connectivity issues, such as firewalls, blocking this call. Depending on the Kubernetes configuration, this may required a firewall rule on port 443 or port 15017; instructions for doing so on GKE can be found here.

In order to check if the API server can access the pod, we can send a request proxied through the api server:

An example of a request that succeeds (no body found is returned from the service and indicates we do have connectivity):

$ kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4
I0618 07:39:46.663871   36880 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server rejected our request for an unknown reason",
  "reason": "BadRequest",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "no body found"
      }
    ]
  },
  "code": 400
}]
F0618 07:39:46.663940   36880 helpers.go:115] Error from server (BadRequest): the server rejected our request for an unknown reason

Similarly, we can send a request from another pod:

$ curl https://istiod.istio-system:443/inject -k
no body found

And from the istiod pod directly (note: the port here is 15017, as this is the targetPort for the Service):

$ curl https://localhost:15017/inject -k
no body found

With this information you should be able to isolate where the breakage occurs.

如果Istio网络没有问题，还是出现错误： failed to create "EnvoyFilter/istio-system/stats-filter-1.16": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded，可以不用怀疑，这一定是你的服务器资源不够用导致出现的问题，因为Docker/Containerd守护进程正在处理大量的请求或者运行多个容器，那么它的响应时间可能会延迟，导致"Context Deadline Exceeded"错误，所以请扩展你的Kubernetes集群服务器的内存/硬盘/CPU，这个问题也就解决了。

我的情况排查到最后发现是Kubernetes集群内存/硬盘/CPU资源不够导致的，多执行几遍Istio安装命令就安装成功了：

2、第二种可能

安装 Istio 时，配置了 enableNamespacesByDefault: false

sidecarInjectorWebhook:
  enabled: true
  # 变量为true，就会为所有命名空间开启自动注入功能。如果赋值为false，则只有标签为istio-injection的命名空间才会开启自动注入功能
  enableNamespacesByDefault: false
  rewriteAppHTTPProbe: false

解决方法：

# 设置标签
$ kubectl label namespace default istio-injection=enabled --overwrite

# 查看
$ kubectl get namespace -L istio-injection
NAME                   STATUS   AGE    ISTIO-INJECTION
default                Active   374d   enabled

如果要重新禁用注入istio sidecar，执行下面命令：

$ kubectl label namespace default istio-injection=disabled --overwrite

3、第三种可能

安装 Istio 时，设置 autoInject: disabled

proxy:
  includeIPRanges: 192.168.16.0/20,192.168.32.0/20
  # 是否开启自动注入功能，取值enabled则该pods只要没有被注解为sidecar.istio.io/inject: "false",就会自动注入。如果取值为disabled，则需要为pod设置注解sidecar.istio.io/inject: "true"才会进行注入
  autoInject: disabled

解决方法：

  • 第一个方法：设置 autoInject: enabled
  • 第二个方法：在 Pod 或者 Deployment 声明 sidecar.istio.io/inject: "true"
相关推荐
大G哥1 小时前
记一次K8S 环境应用nginx stable-alpine 解析内部域名失败排查思路
运维·nginx·云原生·容器·kubernetes
feng_xiaoshi1 小时前
【云原生】云原生架构的反模式
云原生·架构
程序那点事儿3 小时前
k8s 之动态创建pv失败（踩坑）
云原生·容器·kubernetes
叶北辰CHINA4 小时前
nginx反向代理，负载均衡，HTTP配置简述（说人话）
linux·运维·nginx·http·云原生·https·负载均衡
Lansonli6 小时前
云原生（四十八） | Nginx软件安装部署
nginx·云原生·ecs服务器
唐大爹15 小时前
项目实战：k8s部署考试系统
云原生·容器·kubernetes
Zl1597531597531 天前
k8s基础环境部署
云原生·容器·kubernetes
陌殇殇殇1 天前
使用GitLab CI构建持续集成案例
运维·ci/cd·云原生·容器·kubernetes·gitlab
Gogeof1 天前
云原生化 - 基础镜像（简约版）
微服务·云原生·基础镜像
Gogeof1 天前
云原生化 - 旅程（简约版）
微服务·云原生
热门推荐
01组基轨迹建模 GBTM的介绍与实现（Stata 或 R）02【经验分享】Ubuntu22.04安装微信(linux官方版)032024年高教社杯数学建模国赛C题超详细解题思路分析04分享几个惊艳的 Three.js 个人站点05安卓系列机型永久去除data分区加密 详细步骤解析06【2024数模国赛赛题思路公开】国赛B题思路丨附可运行代码丨无偿自提07CTF网络安全大赛简单的web抓包题目：HEADache08Ubuntu24.04安装中文输入法09【2024高教社杯全国大学生数学建模竞赛】B题 生产过程中的决策问题——解题思路 代码 论文10springer 在线投稿编译踩坑