【K8S 运维实战】20-证书管理cert-manager

证书管理:cert-manager 与自动轮转

一句话定位:证书过期是集群的"定时炸弹"------这篇把 K8s 证书体系、cert-manager 自动轮转和过期巡检一次性配齐。

写在前面

我经历过一次惨剧:凌晨 3 点被告警叫醒,整个集群 apiserver 拒绝连接,kubelet 全报 certificate has expired or not yet valid。原因很low------kubelet 客户端证书过期了,而自动轮转没开。那一夜我手动给 200 个节点续证书续到天亮。从那以后,我对证书管理的态度就一句话:能自动化的绝不手动,能监控的绝不靠人记。这篇把证书这套体系讲透,让你的集群不再栽在过期上。

核心问题

• 证书过期导致集群挂掉怎么办? apiserver/kubelet/etcd 证书一过期,控制面直接瘫痪,kubelet 无法注册。
• 怎么避免手动续证书? kubelet 自带客户端证书轮转;cert-manager 管理 Ingress/内部服务证书。
• Let's Encrypt 怎么用? cert-manager + ACME HTTP01/DNS01 挑战,自动签发公网证书。

一、原理剖析

1.1 K8s 证书体系全景

K8s 集群里证书多到让人头大,先理清楚谁跟谁通信:

┌────────────────────────────────────────────────────────┐
│                    K8s 证书体系                        │
│                                                        │
│  ┌──────────┐    server cert    ┌──────────┐           │
│  │ kubelet  │ ◄───────────────► │ apiserver│           │
│  │          │  client cert      │          │           │
│  └──────────┘                   └────┬─────┘           │
│                                      │                 │
│  ┌──────────┐    client cert         │                 │
│  │ kubectl  │ ────────────────────► │                 │
│  │  (用户)  │                       │                 │
│  └──────────┘                       │                 │
│                                      │                 │
│  ┌──────────────────────────────┐    │                 │
│  │        apiserver              │◄───┘                 │
│  │  └─ apiserver-kubelet-client  │                     │
│  │  └─ apiserver-etcd-client     │                     │
│  └──────────────────────────────┘                     │
│                                                        │
│  ┌──────────┐    peer cert     ┌──────────┐            │
│  │  etcd    │ ◄─────────────► │  etcd    │            │
│  └──────────┘                  └──────────┘           │
└────────────────────────────────────────────────────────┘

证书分类与过期影响:

证书	路径	默认有效期	过期影响
apiserver server	apiserver.crt	1 年(kubeadm)	apiserver 不可访问
apiserver-kubelet-client	apiserver-kubelet-client.crt	1 年	apiserver 无法连 kubelet
apiserver-etcd-client	apiserver-etcd-client.crt	1 年	apiserver 无法读写 etcd
kubelet server	kubelet.crt	1 年	kubelet 10250 端口不可访问
kubelet client(轮转)	kubelet-client-current.pem	默认 1 年,轮转后自动续	kubelet 无法注册到 apiserver
etcd server/peer	server.crt/peer.crt	1 年(kubeadm)	etcd 集群通信失败
aggregation CA	front-proxy-ca.crt	10 年	API 聚合失效

最容易爆雷的是 kubelet 客户端证书,因为它直接决定节点能不能留在集群里。

1.2 kubelet 证书轮转机制

kubelet 从 1.19 起支持客户端证书自动轮转:
kube-controller-manager apiserver kubelet kube-controller-manager apiserver kubelet #mermaid-svg-dKZJwWTEB55OA7jq{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dKZJwWTEB55OA7jq .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dKZJwWTEB55OA7jq .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dKZJwWTEB55OA7jq .error-icon{fill:#552222;}#mermaid-svg-dKZJwWTEB55OA7jq .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dKZJwWTEB55OA7jq .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dKZJwWTEB55OA7jq .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dKZJwWTEB55OA7jq .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dKZJwWTEB55OA7jq .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dKZJwWTEB55OA7jq .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dKZJwWTEB55OA7jq .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dKZJwWTEB55OA7jq .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dKZJwWTEB55OA7jq .marker.cross{stroke:#333333;}#mermaid-svg-dKZJwWTEB55OA7jq svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dKZJwWTEB55OA7jq p{margin:0;}#mermaid-svg-dKZJwWTEB55OA7jq .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-dKZJwWTEB55OA7jq text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-dKZJwWTEB55OA7jq .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-dKZJwWTEB55OA7jq .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-dKZJwWTEB55OA7jq .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-dKZJwWTEB55OA7jq .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-dKZJwWTEB55OA7jq #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-dKZJwWTEB55OA7jq .sequenceNumber{fill:white;}#mermaid-svg-dKZJwWTEB55OA7jq #sequencenumber{fill:#333;}#mermaid-svg-dKZJwWTEB55OA7jq #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-dKZJwWTEB55OA7jq .messageText{fill:#333;stroke:none;}#mermaid-svg-dKZJwWTEB55OA7jq .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-dKZJwWTEB55OA7jq .labelText,#mermaid-svg-dKZJwWTEB55OA7jq .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-dKZJwWTEB55OA7jq .loopText,#mermaid-svg-dKZJwWTEB55OA7jq .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-dKZJwWTEB55OA7jq .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-dKZJwWTEB55OA7jq .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-dKZJwWTEB55OA7jq .noteText,#mermaid-svg-dKZJwWTEB55OA7jq .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-dKZJwWTEB55OA7jq .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-dKZJwWTEB55OA7jq .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-dKZJwWTEB55OA7jq .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-dKZJwWTEB55OA7jq .actorPopupMenu{position:absolute;}#mermaid-svg-dKZJwWTEB55OA7jq .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-dKZJwWTEB55OA7jq .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-dKZJwWTEB55OA7jq .actor-man circle,#mermaid-svg-dKZJwWTEB55OA7jq line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-dKZJwWTEB55OA7jq :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 监听证书剩余有效期 < 30% 启动,加载现有 client cert用现有证书注册发起 CSR(certificate signing request)CSR 进入 kubernetes.io/kube-apiserver-client-kubelet 签名器自动审批(若开启 RotateKubeletClientCertificate)签发新证书返回新证书写入 kubelet-client-current.pem,热加载

开启轮转的关键配置(kubelet config):

# /var/lib/kubelet/config.yaml
rotateCertificates: true          # 客户端证书自动轮转
serverTLSBootstrap: true           # 服务端证书 bootstrap(可选)

serverTLSBootstrap 开启后,kubelet 服务端证书也走 CSR,由 controller-manager 签发(默认还是自签)。生产建议开启,统一证书管理。

1.3 cert-manager 架构

cert-manager 是管理 Ingress/Service 证书的主力,基于 CRD:
#mermaid-svg-OXYrYsnFyIRtXjXX{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-OXYrYsnFyIRtXjXX .error-icon{fill:#552222;}#mermaid-svg-OXYrYsnFyIRtXjXX .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-OXYrYsnFyIRtXjXX .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-OXYrYsnFyIRtXjXX .marker{fill:#333333;stroke:#333333;}#mermaid-svg-OXYrYsnFyIRtXjXX .marker.cross{stroke:#333333;}#mermaid-svg-OXYrYsnFyIRtXjXX svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-OXYrYsnFyIRtXjXX p{margin:0;}#mermaid-svg-OXYrYsnFyIRtXjXX .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-OXYrYsnFyIRtXjXX .cluster-label text{fill:#333;}#mermaid-svg-OXYrYsnFyIRtXjXX .cluster-label span{color:#333;}#mermaid-svg-OXYrYsnFyIRtXjXX .cluster-label span p{background-color:transparent;}#mermaid-svg-OXYrYsnFyIRtXjXX .label text,#mermaid-svg-OXYrYsnFyIRtXjXX span{fill:#333;color:#333;}#mermaid-svg-OXYrYsnFyIRtXjXX .node rect,#mermaid-svg-OXYrYsnFyIRtXjXX .node circle,#mermaid-svg-OXYrYsnFyIRtXjXX .node ellipse,#mermaid-svg-OXYrYsnFyIRtXjXX .node polygon,#mermaid-svg-OXYrYsnFyIRtXjXX .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-OXYrYsnFyIRtXjXX .rough-node .label text,#mermaid-svg-OXYrYsnFyIRtXjXX .node .label text,#mermaid-svg-OXYrYsnFyIRtXjXX .image-shape .label,#mermaid-svg-OXYrYsnFyIRtXjXX .icon-shape .label{text-anchor:middle;}#mermaid-svg-OXYrYsnFyIRtXjXX .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-OXYrYsnFyIRtXjXX .rough-node .label,#mermaid-svg-OXYrYsnFyIRtXjXX .node .label,#mermaid-svg-OXYrYsnFyIRtXjXX .image-shape .label,#mermaid-svg-OXYrYsnFyIRtXjXX .icon-shape .label{text-align:center;}#mermaid-svg-OXYrYsnFyIRtXjXX .node.clickable{cursor:pointer;}#mermaid-svg-OXYrYsnFyIRtXjXX .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-OXYrYsnFyIRtXjXX .arrowheadPath{fill:#333333;}#mermaid-svg-OXYrYsnFyIRtXjXX .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-OXYrYsnFyIRtXjXX .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-OXYrYsnFyIRtXjXX .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OXYrYsnFyIRtXjXX .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-OXYrYsnFyIRtXjXX .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OXYrYsnFyIRtXjXX .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-OXYrYsnFyIRtXjXX .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-OXYrYsnFyIRtXjXX .cluster text{fill:#333;}#mermaid-svg-OXYrYsnFyIRtXjXX .cluster span{color:#333;}#mermaid-svg-OXYrYsnFyIRtXjXX div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-OXYrYsnFyIRtXjXX .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-OXYrYsnFyIRtXjXX rect.text{fill:none;stroke-width:0;}#mermaid-svg-OXYrYsnFyIRtXjXX .icon-shape,#mermaid-svg-OXYrYsnFyIRtXjXX .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OXYrYsnFyIRtXjXX .icon-shape p,#mermaid-svg-OXYrYsnFyIRtXjXX .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-OXYrYsnFyIRtXjXX .icon-shape .label rect,#mermaid-svg-OXYrYsnFyIRtXjXX .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OXYrYsnFyIRtXjXX .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-OXYrYsnFyIRtXjXX .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-OXYrYsnFyIRtXjXX :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 签发
生成
挂载
申请
挑战
成功后
Issuer/ClusterIssuer
Certificate
Secret TLS
Ingress/Pod
Order ACME
Challenge HTTP01/DNS01
CA:Let's Encrypt/内部

核心 CRD:

• Issuer:命名空间级签发器,只能签同命名空间证书
• ClusterIssuer:集群级签发器,可签任意命名空间证书
• Certificate:声明想要的证书(域名、有效期、目标 Secret)
• Order/Challenge:ACME 流程的中间产物,一般不用管

二、实战操作

2.1 cert-manager Helm 生产部署

# 添加 Jetstack 仓库
helm repo add jetstack https://charts.jetstack.io
helm repo update

# 安装 cert-manager(1.30 对应 cert-manager v1.15)
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.15.3 \
  --set installCRDs=true \
  --set replicaCount=3 \
  --set podDisruptionBudget.minAvailable=2 \
  --set featureGates=AdditionalCertificateOutputFormats=true \
  --set config.apiVersion=controller.config.cert-manager.io/v1alpha1 \
  --set config.kind=ControllerConfiguration \
  --set config.enableCertificateOwnerRef=true

# 验证
kubectl get pods -n cert-manager
# 期望: cert-manager / webhook / cainjector 三个 Pod Running

生产配置要点:

• replicaCount=3:cert-manager 高可用
• podDisruptionBudget:保证升级时至少 2 个可用
• enableCertificateOwnerRef:删除 Certificate 时自动清理 Secret

2.2 内部 CA 与自签证书

内部服务互调用(如 mTLS),用自建 CA 签发证书,不依赖外部。

Step 1:创建 CA 私钥和证书

# 生成 CA 私钥
openssl genrsa -out ca.key 4096

# 生成 CA 证书(10 年)
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
  -out ca.crt \
  -subj "/CN=internal-ca/O=MyOrg"

Step 2:创建 CA Secret(给 cert-manager 用)

kubectl create namespace internal
kubectl create secret tls internal-ca-secret \
  --cert=ca.crt --key=ca.key \
  -n internal

Step 3:创建 ClusterIssuer

# cluster-issuer-internal.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-secret
    # CA Secret 必须在 cert-manager 所在命名空间
---
# 或者用 SelfSigned(测试用,不适合生产)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

kubectl apply -f cluster-issuer-internal.yaml
kubectl get clusterissuer internal-ca-issuer -o wide

Step 4:申请内部证书

# certificate-internal.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: backend-tls
  namespace: internal
spec:
  secretName: backend-tls-secret
  duration: 2160h      # 90 天
  renewBefore: 360h    # 提前 15 天续
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  commonName: backend.internal.svc
  dnsNames:
    - backend
    - backend.internal
    - backend.internal.svc.cluster.local
  usages:
    - server auth
    - client auth
  # 生成 JKS/PEM 等多种格式
  additionalOutputFormats:
    - type: JKS
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048

kubectl apply -f certificate-internal.yaml
kubectl get certificate -n internal
# READY=True 表示签发成功
kubectl get secret backend-tls-secret -n internal -o yaml

2.3 Let's Encrypt ACME(HTTP01/DNS01)

公网域名证书用 Let's Encrypt,免费、自动续。两种挑战:

挑战方式	原理	适用场景	配置难度
HTTP01	在 Ingress 暴露 .well-known 验证文件	有公网 IP 的 Ingress	简单
DNS01	在 DNS 加 TXT 记录验证	通配符证书/无公网 Ingress	需 DNS API 凭证

HTTP01 ClusterIssuer:

# cluster-issuer-letsencrypt-http01.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx
            # 用指定 Ingress class 处理挑战

DNS01 ClusterIssuer(以阿里云 DNS 为例):

# cluster-issuer-letsencrypt-dns01.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-dns-account-key
    solvers:
      - dns01:
          webhook:
            groupName: acme.example.com
            solverName: alidns
            config:
              accessKeyIdRef:
                name: alidns-secret
                key: access-key-id
              accessKeySecretRef:
                name: alidns-secret
                key: access-key-secret
              region: cn-hangzhou

# 阿里云 DNS 凭证
kubectl create secret generic alidns-secret \
  --from-literal=access-key-id=LTAIxxx \
  --from-literal=access-key-secret=xxx \
  -n cert-manager

申请通配符证书:

# certificate-wildcard.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-tls
  namespace: nginx-ingress
spec:
  secretName: wildcard-example-tls
  issuerRef:
    name: letsencrypt-dns
    kind: ClusterIssuer
  commonName: "*.example.com"
  dnsNames:
    - "*.example.com"
    - "example.com"

Ingress 自动签发:

# ingress-with-tls.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: internal
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    # 指定 issuer 即可自动签发
spec:
  tls:
    - hosts:
        - app.example.com
      secretName: app-tls-secret
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-svc
                port:
                  number: 80

cert-manager 看到 Ingress 有 tls 和 issuer 注解,自动签发证书到 app-tls-secret。

2.4 kubelet 证书轮转配置

# 1. 检查 kubelet 配置
cat /var/lib/kubelet/config.yaml | grep rotate

# 期望:
# rotateCertificates: true
# serverTLSBootstrap: true

# 2. 如果没开,修改 kubelet config
kubectl -n kube-system edit configmap kubelet-config
# 加上:
# rotateCertificates: true
# serverTLSBootstrap: true

# 3. 让所有节点应用新配置
# ssh 到节点
systemctl restart kubelet

# 4. 开启 controller-manager 自动审批(生产谨慎)
# 默认 kubelet client CSR 需要手动 kubectl certificate approve
# 自动审批需配置 CSR approver(用 controller-manager 的签发器配置)
# 或者用 auto approving controller(如 kubelet-csr-approver)

自动审批方案(推荐 kubelet-csr-approver):

# 部署 kubelet-csr-approver
helm repo add postfinance https://postfinance.github.io/kubelet-csr-approver
helm install kubelet-csr-approver postfinance/kubelet-csr-approver \
  -n kube-system \
  --set providerRegex='^ip-10-0-0-.*\\.ec2\\.internal$' \
  --set bypassDnsResolution=true

2.5 证书过期巡检脚本

#!/bin/bash
# check-cert-expiry.sh - 巡检所有证书过期时间
# 使用: ./check-cert-expiry.sh [days]
THRESHOLD=${1:-30}
ALERT=0

echo "===== 1. 控制面证书(kubeadm 管理) ====="
kubeadm certs check-expiration 2>/dev/null | grep -E "CA|apiserver|kubelet|etcd|proxy|scheduler|controller"

echo ""
echo "===== 2. 各节点 kubelet 客户端证书 ====="
for node in $(kubectl get nodes -o name); do
    NODE=$(echo $node | cut -d/ -f2)
    # 通过 ssh 检查(需要 ssh 免密)
    EXPIRY=$(ssh $NODE "openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate 2>/dev/null | cut -d= -f2")
    DAYS_LEFT=$(( ($(date -d "$EXPIRY" +%s) - $(date +%s)) / 86400 ))
    if [ $DAYS_LEFT -lt $THRESHOLD ]; then
        echo "[WARNING] $NODE kubelet client cert expires in $DAYS_LEFT days"
        ALERT=1
    else
        echo "[OK] $NODE kubelet client cert: $DAYS_LEFT days left"
    fi
done

echo ""
echo "===== 3. cert-manager 证书 ====="
kubectl get certificate -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.metadata.name,READY:.status.conditions[0].type,EXPIRY:.status.notAfter 2>/dev/null

echo ""
echo "===== 4. etcd 证书 ====="
for master in master1 master2 master3; do
    ssh $master "openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate" 2>/dev/null
done

if [ $ALERT -eq 1 ]; then
    echo ""
    echo "[CRITICAL] 有证书即将过期,请立即处理!"
    exit 1
fi

# 定时巡检(每天)
# crontab -e
# 0 9 * * * /opt/check-cert-expiry.sh 30 | mail -s "K8s cert expiry report" ops@example.com

Prometheus 告警(cert-manager 自带 metrics):

# cert-manager 证书过期告警
- alert: CertManagerCertExpiringSoon
  expr: certmanager_certificate_expiration_timestamp_seconds < (time() + 86400 * 30)
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "证书 {{ $labels.name }} 将在 30 天内过期"

- alert: CertManagerCertFailed
  expr: certmanager_certificate_ready_status{condition="False"} == 1
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "证书签发失败: {{ $labels.name }}"

三、踩坑与排查

坑1:cert-manager 证书一直 NotReady

现象 :Certificate 状态 Ready=False,日志 failed to solve challenge。

原因:HTTP01 挑战要求 Ingress 有公网 IP,且 80 端口可达;内部集群没公网入口,挑战失败。

解决:内网用内部 CA(ClusterIssuer);只有公网服务用 Let's Encrypt。或者改用 DNS01。

# 查看 Challenge 详情
kubectl get challenge -n internal
kubectl describe challenge <name> -n internal
# 看是 404、超时还是 DNS 没生效

坑2:kubelet 证书过期,节点 NotReady

现象 :节点突然 NotReady,日志 certificate has expired or not yet valid。

原因 :没开 rotateCertificates,1 年到期没续。

解决(应急):

# 1. ssh 到节点
# 2. 用 kubeadm 续期
kubeadm certs renew kubelet.conf
# 3. 重启 kubelet
systemctl restart kubelet
# 4. 确认节点 Ready
kubectl get node <node>

# 长期:开启 rotateCertificates=true,见 2.4

坑3:Let's Encrypt 触发速率限制

现象 :签发证书失败,日志 too many certificates already issued。

原因:Let's Encrypt 对每个域名每周限 50 张证书,测试环境反复签发会触发。

解决:

• 测试用 --server=https://acme-staging-v02.api.letsencrypt.org/directory(staging 不限)
• 合并用通配符证书(*.example.com)替代多个单域名证书
• 善用 Certificate 的 renew,别反复删了重建

坑4:CA 证书过期,全集群瘫痪

现象:CA 过期(默认 10 年),所有证书失效,集群全挂。

原因:平时只看叶证书,忘了 CA 也有期限。

解决:提前重建 CA(复杂,需滚动替换):

# 1. 生成新 CA
openssl genrsa -out new-ca.key 4096
openssl req -x509 -new -key new-ca.key -days 3650 -out new-ca.crt -subj "/CN=kubernetes"

# 2. 用 kubeadm 重建集群证书(会自动用新 CA)
# 3. 滚动重启所有组件

# 预防:监控 CA 过期,提前 1 年规划

四、最佳实践

• kubelet 开启 rotateCertificates=true 和 serverTLSBootstrap=true
• 部署 kubelet-csr-approver 自动审批 kubelet CSR
• cert-manager 3 副本 + PDB
• 公网证书用 Let's Encrypt + DNS01(支持通配符)
• 内部服务用自建 CA(ClusterIssuer ca 类型)
• 测试环境用 ACME staging endpoint,避免速率限制
• Certificate 设置 renewBefore 提前续(默认有效期 1/3)
• 每日巡检脚本 + Prometheus 证书过期告警(30 天/7 天分级)
• 定期备份 CA 私钥(异地存储)
• CA 证书独立监控(10 年到期提前 1 年规划)
• 证书 Secret 用 RBAC 限制访问,防泄露

五、小结

证书管理是运维里"平时不起眼、出事就炸天"的环节。核心就两条主线:一是 K8s 内部证书(kubeadm 管理),靠 kubelet 自动轮转兜底;二是业务证书(Ingress/内部服务),靠 cert-manager 自动化。把自动化和监控做扎实,证书过期这种低级错误就再也不会发生。记住:能自动续的绝不手动,能监控的绝不靠人记------运维的最高境界是"忘了它的存在"。

思考题

1. kubelet 的 serverTLSBootstrap 开了之后,服务端证书由谁签发?不开会怎样?
2. Let's Encrypt 的 staging 和 production 环境证书,浏览器信任度有什么区别?
3. 内部 mTLS 场景,CA 私钥存在哪里最安全?如果泄露怎么补救?

延伸阅读

• cert-manager 官方文档:https://cert-manager.io/docs/
• K8s 证书管理:https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
• kubelet 证书轮转:https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/