k8s 1.28.2 集群部署 ingress 1.11.1 包含 admission-webhook

陈小肚2024-09-27 8:53

### 文章目录

  • [@[toc]](#文章目录 @[toc] 证书创建 部署 ingress-controller ingress 验证 创建测试 nginx pod 创建错误的 ingress 配置 创建正确的 ingress 配置)
  • [证书创建](#文章目录 @[toc] 证书创建 部署 ingress-controller ingress 验证 创建测试 nginx pod 创建错误的 ingress 配置 创建正确的 ingress 配置)
  • [部署 ingress-controller](#文章目录 @[toc] 证书创建 部署 ingress-controller ingress 验证 创建测试 nginx pod 创建错误的 ingress 配置 创建正确的 ingress 配置)
  • [ingress 验证](#文章目录 @[toc] 证书创建 部署 ingress-controller ingress 验证 创建测试 nginx pod 创建错误的 ingress 配置 创建正确的 ingress 配置)
  • [创建测试 nginx pod](#文章目录 @[toc] 证书创建 部署 ingress-controller ingress 验证 创建测试 nginx pod 创建错误的 ingress 配置 创建正确的 ingress 配置)
  • [创建错误的 ingress 配置](#文章目录 @[toc] 证书创建 部署 ingress-controller ingress 验证 创建测试 nginx pod 创建错误的 ingress 配置 创建正确的 ingress 配置)
  • [创建正确的 ingress 配置](#文章目录 @[toc] 证书创建 部署 ingress-controller ingress 验证 创建测试 nginx pod 创建错误的 ingress 配置 创建正确的 ingress 配置)
  • ingress 官方 yaml 文件：deploy.yaml
  • 基于官方 yaml 文件做了一些修改
    • 官方的 svc 是 ClusterIP 和 LoadBalancer，我这边把 LoadBalancer 改成 NodePort
    • 官方的 yaml 没有给 deployment 配置副本数，我这边改成双副本了
      • 增加了亲和性和反亲和，把 controller 绑定到固定的两个节点，方便前面加反代
    • 官方的镜像是 registry.k8s.io 仓库，国内咱们也不好拉，这边改用 k8s.m.daocloud.io 国内的 daocloud 的仓库
    • 开启 admission-webhook 需要自己生成 tls 证书，官方的模板有名字定义了，sercret 的名字是 ingress-nginx-admission
  • 开启了 admission-webhook 功能，这个功能类似于 nginx -s reload 之前去执行 nginx -t 检查配置文件是否正确，如果 admission-webhook 发现配置不正确就不会更新 ingress 的配置，就不会因为错误的配置导致 ingress reload 失败，全部崩了，影响到业务的访问

证书创建

证书这块，我用的是 cert-manager 创建的，可以看我之前的博客：cert-manager - kubernetes 集群中 TLS 证书管理工具

部署 ingress-controller

节点打标签

shell 复制代码 
k label node 192.168.22.112 ingress=true
k label node 192.168.22.113 ingress=true

通过官方 helm 做了一些修改

yaml 复制代码 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
automountServiceAccountToken: true
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: "true"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
      - namespaces
    verbs:
      - list
      - watch
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - list
      - watch
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  # Omit Ingress status permissions if `--update-status` is disabled.
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    resourceNames:
      - ingress-nginx-leader
    verbs:
      - get
      - update
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - list
      - watch
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-metrics
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: metrics
      port: 10254
      protocol: TCP
      targetPort: metrics
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
---
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      nodePort: 31080
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      nodePort: 31443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/component: controller
  replicas: 2
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        helm.sh/chart: ingress-nginx-4.11.1
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/version: "1.11.1"
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: controller
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ingress
                operator: In
                values:
                - "true"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - ingress-nginx
            topologyKey: kubernetes.io/hostname
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: k8s.m.daocloud.io/ingress-nginx/controller:v1.11.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
            - --election-id=ingress-nginx-leader
            - --controller-class=k8s.io/ingress-nginx
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            runAsNonRoot: true
            runAsUser: 101
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              drop:
              - ALL
              add:
              - NET_BIND_SERVICE
            readOnlyRootFilesystem: false
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: metrics
              containerPort: 10254
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: ingress
                operator: In
                values:
                - "true"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - ingress-nginx
            topologyKey: kubernetes.io/hostname
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
            items:
              - key: tls.crt
                path: cert
              - key: tls.key
                path: key
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: nginx
spec:
  controller: k8s.io/ingress-nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-nginx-root-cert
  namespace: ingress-nginx
spec:
  secretName: ingress-nginx-root-cert
  duration: "43800h0m0s"
  issuerRef:
    name: ingress-nginx-self-signed-issuer
  commonName: "ca.webhook.ingress-nginx"
  isCA: true
  subject:
    organizations:
      - ingress-nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
spec:
  secretName: ingress-nginx-admission
  duration: "8760h0m0s"
  issuerRef:
    name: ingress-nginx-root-issuer
  dnsNames:
    - ingress-nginx-controller-admission
    - ingress-nginx-controller-admission.ingress-nginx
    - ingress-nginx-controller-admission.ingress-nginx.svc
  subject:
    organizations:
      - ingress-nginx-admission
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ingress-nginx-self-signed-issuer
  namespace: ingress-nginx
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ingress-nginx-root-issuer
  namespace: ingress-nginx
spec:
  ca:
    secretName: ingress-nginx-root-cert
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    certmanager.k8s.io/inject-ca-from: "ingress-nginx/ingress-nginx-admission"
    cert-manager.io/inject-ca-from: "ingress-nginx/ingress-nginx-admission"
  labels:
    helm.sh/chart: ingress-nginx-4.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: "1.11.1"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        name: ingress-nginx-controller-admission
        namespace: ingress-nginx
        path: /networking/v1/ingresses

ingress 验证

创建测试 nginx pod

yaml 复制代码 
cat << EOF | kubectl apply -f -
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: nginx
  name: nginx-test
  namespace: default
spec:
  internalTrafficPolicy: Cluster
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: nginx
  name: nginx-test
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: docker.m.daocloud.io/nginx:1.26.0
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
EOF

创建错误的 ingress 配置

yaml 复制代码 
cat << EOF | kubectl apply -f -
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_pass          http://$targetbackend;
      proxy_set_header    X-Real-IP       $remote_addr;
      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  name: ingress-test
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: nginx-test
            port:
              number: 80
        path: /
        pathType: Prefix
EOF
  • 此时会出现报错，这个报错是 admission webhook 触发的，并且 ingress-controller 也不会触发重启，因为不会更新和创建 ingress 规则，所以不会影响到业务
Error from server (BadRequest): error when creating "STDIN": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
2024/09/26 07:30:35 [emerg] 166#166: no host in upstream "" in /tmp/nginx/nginx-cfg2607502217:404
nginx: [emerg] no host in upstream "" in /tmp/nginx/nginx-cfg2607502217:404
nginx: configuration file /tmp/nginx/nginx-cfg2607502217 test failed

创建正确的 ingress 配置

yaml 复制代码 
cat << EOF | kubectl apply -f -
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
  name: ingress-test
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: nginx-test
            port:
              number: 80
        path: /
        pathType: Prefix
EOF

此时会返回 created，说明 ingress 的规则配置成功了

ingress.networking.k8s.io/ingress-test created

验证 ingress

shell 复制代码 
curl 192.168.22.112:31080

正常会返回下面的内容，是 nginx 的默认页面

html 复制代码 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
相关推荐
Nue.js2 小时前
Docker中最常用的一些命令
docker·容器·eureka
basic_code4 小时前
Docker部署kafka集群
docker·容器·kafka
水宝的滚动歌词4 小时前
K8S单节点部署及集群部署
云原生·容器·kubernetes
Z y..5 小时前
记：Docker部署kafka消费者注册不到问题
docker·容器·kafka
ly49835 小时前
docker--工作目录迁移
docker·容器·eureka
yohoo菜鸟7 小时前
kubernetes简单入门实战
云原生·容器·kubernetes
狂奔solar10 小时前
分享个好玩的，在k8s上部署web版macos
前端·macos·kubernetes
mit6.82410 小时前
[Docker#9] 存储卷 | Volume、Bind、Tmpfs | -v/mount | MySQL 灾难恢复 | 问题
linux·运维·docker·容器·架构
qq_1674015110 小时前
Docker 组添加用户，设置允许普通用户操作 docker
docker·容器
Z1eaf_complete10 小时前
Docker的基础使用
运维·docker·容器·云计算
热门推荐
01玄机平台应急响应—webshell查杀02Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04智能车联网安全发展形势、挑战05文件或文件夹名称中有空格如何批量去除06组基轨迹建模 GBTM的介绍与实现（Stata 或 R）07Python数据分析案例31——中国A股的月份效应研究(方差分析，虚拟变量回归)08优化手机性能，解决卡顿问题：关闭这3个微信开关，释放内存空间09基于YOLOv10深度学习的CT扫描图像肾结石智能检测系统【python源码+Pyqt5界面+数据集+训练代码】深度学习实战、目标检测10怎样让音频速度变慢？请跟随以下方法进行操作