Istio 自动注入 sidecar 失败导致无法访问webhook服务

最近工作中在部署Istio环境的过程中发现官方示例启动的pod不能访问不到Istio的webhook,这个问题也是困扰了我一天,特此记录,便于日后查阅。

我把他归类到sidecar注入失败的情况,报错如下:

1、第一种可能(我遇到的情况)

如果自动注入时,报如下错误信息:

复制代码
2023-10-26T02:15:22.051580Z error installer Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded
2023-10-26T02:15:32.149109Z error installer failed to create "EnvoyFilter/istio-system/stats-filter-1.14": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded

造成上述问题的原因是 kube-apiserver 的 --enable-admission-plugins 没有配置 MutatingAdmissionWebhook,ValidatingAdmissionWebhook参数,所以解决问题的方法就是找到 kube-apiserver.yaml 配置文件,我是通过kubeadm安装的,所以路径在 /etc/kubernetes/manifests 下,把 --enable-admission-plugins 修改为 NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook 后保存文件,删掉 kube-apiserver 的pod,让他自己根据新的配置文件重新启动(注意:如果是用kubeadm安装的修改内容如果错误,可能会导致k8s集群中的kube-apiserver全部挂掉,导致无法访问集群,这时就需要从官网下载kube-apiserver二进制文件,重新拉起一个进程,再执行删除pod的操作

注:这里我多加了一个配置 --feature-gates=ServerSideApply=false,这个配置是关闭特性门控服务端自动更新,因为我还报了另外一个错误:failed to update resource with server-side apply for obj

修改后文件内容如下:

复制代码
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.61:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.61
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --feature-gates=RemoveSelfLink=false,ServerSideApply=false
    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.1.61
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.1.61
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.1.61
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

如果上述配置你已经配过了,还是出现错误: failed to create "EnvoyFilter/istio-system/stats-filter-1.16": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded,按照下述操作排除Istio网络问题(懒得翻译咯~):

Injection works by the API server connecting to the webhook deployment (Istiod). This may cause issues if there are connectivity issues, such as firewalls, blocking this call. Depending on the Kubernetes configuration, this may required a firewall rule on port 443 or port 15017; instructions for doing so on GKE can be found here.

In order to check if the API server can access the pod, we can send a request proxied through the api server:

An example of a request that succeeds (no body found is returned from the service and indicates we do have connectivity):

复制代码
$ kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4
I0618 07:39:46.663871   36880 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server rejected our request for an unknown reason",
  "reason": "BadRequest",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "no body found"
      }
    ]
  },
  "code": 400
}]
F0618 07:39:46.663940   36880 helpers.go:115] Error from server (BadRequest): the server rejected our request for an unknown reason

Similarly, we can send a request from another pod:

复制代码
$ curl https://istiod.istio-system:443/inject -k
no body found

And from the istiod pod directly (note: the port here is 15017, as this is the targetPort for the Service):

复制代码
$ curl https://localhost:15017/inject -k
no body found

With this information you should be able to isolate where the breakage occurs.

如果Istio网络没有问题,还是出现错误: failed to create "EnvoyFilter/istio-system/stats-filter-1.16": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded,可以不用怀疑,这一定是你的服务器资源不够用导致出现的问题,因为Docker/Containerd守护进程正在处理大量的请求或者运行多个容器,那么它的响应时间可能会延迟,导致"Context Deadline Exceeded"错误,所以请扩展你的Kubernetes集群服务器的内存/硬盘/CPU,这个问题也就解决了。

我的情况排查到最后发现是Kubernetes集群内存/硬盘/CPU资源不够导致的,多执行几遍Istio安装命令就安装成功了:

2、第二种可能

安装 Istio 时,配置了 enableNamespacesByDefault: false

复制代码
sidecarInjectorWebhook:
  enabled: true
  # 变量为true,就会为所有命名空间开启自动注入功能。如果赋值为false,则只有标签为istio-injection的命名空间才会开启自动注入功能
  enableNamespacesByDefault: false
  rewriteAppHTTPProbe: false

解决方法:

复制代码
# 设置标签
$ kubectl label namespace default istio-injection=enabled --overwrite

# 查看
$ kubectl get namespace -L istio-injection
NAME                   STATUS   AGE    ISTIO-INJECTION
default                Active   374d   enabled

如果要重新禁用注入istio sidecar,执行下面命令:

复制代码
$ kubectl label namespace default istio-injection=disabled --overwrite

3、第三种可能

安装 Istio 时,设置 autoInject: disabled

复制代码
proxy:
  includeIPRanges: 192.168.16.0/20,192.168.32.0/20
  # 是否开启自动注入功能,取值enabled则该pods只要没有被注解为sidecar.istio.io/inject: "false",就会自动注入。如果取值为disabled,则需要为pod设置注解sidecar.istio.io/inject: "true"才会进行注入
  autoInject: disabled

解决方法:

  • 第一个方法:设置 autoInject: enabled
  • 第二个方法:在 Pod 或者 Deployment 声明 sidecar.istio.io/inject: "true"
相关推荐
阿里云云原生8 小时前
AI 时代的数据通道:云消息队列 Kafka 的演进与实践
云原生·kafka
阿里云云原生8 小时前
单提交智能评审上线!用云效精准定位复杂 MR 代码问题
云原生
阿里云云原生8 小时前
再看 AI 网关:助力 AI 应用创新的关键基础设施
云原生
阿里云云原生9 小时前
零代码改造 + 全链路追踪!Spring AI 最新可观测性详细解读
spring·云原生
霖.2410 小时前
Docker常见问题
服务器·docker·云原生·容器
荣光波比10 小时前
K8S(十七)—— Kubernetes集群可视化工具Kuboard部署与实践指南
云原生·容器·kubernetes
二宝15212 小时前
黑马商城day3-微服务01
微服务·云原生·架构
切糕师学AI13 小时前
云原生技术栈解析:宿主机、容器、Docker、Kubernetes 之间的区别于联系
docker·云原生·容器·kubernetes
云雾J视界14 小时前
Linux企业级解决方案架构:字节跳动短视频推荐系统全链路实践
linux·云原生·架构·kubernetes·音视频·glusterfs·elk stack
没有bug.的程序员19 小时前
分布式架构未来趋势:从云原生到智能边缘的演进之路
java·分布式·微服务·云原生·架构·分布式系统