prometheus+k8s+弹性伸缩，综合项目

k8s弹性伸缩综合项目

由 k8s弹性伸缩综合项目.pdf 转换（共 41 页），已按章节整理：表格/命令/重点已格式化。

第6部：kubernetes弹性伸缩

一、水平自动伸缩 HPA

1. 介绍

HPA，全称 Horizontal Pod Autoscaler，可以基于 CPU 利用率自动扩缩 ReplicationController、Deployment 和 ReplicaSet 中的 Pod 数量。除了 CPU 利用率，也可以基于其他应用程序提供的自定义度量指标来执行自动扩缩。

Pod 自动扩缩不适用于无法扩缩的对象，比如 DaemonSet。

Pod 水平自动扩缩特性由 Kubernetes API 资源和控制器实现。资源决定了控制器的行为，控制器会周期性地调整副本控制器或 Deployment 中的副本数量，以使得 Pod 的平均 CPU 利用率与用户所设定的目标值匹配。

HPA 定期检查内存和 CPU 等指标，自动调整 Deployment 中的副本数。

实际生产中，广泛使用这四类指标：

• Resource metrics：CPU、内存利用率指标
• Pod metrics：例如网络利用率和流量
• Object metrics：特定对象的指标，比如 Ingress，可以按每秒使用请求数来扩展容器
• Custom metrics：自定义监控，比如通过定义服务响应时间，当响应时间达到一定指标时自动扩容

2. metrics-server 部署

metrics-server 可以查看资源（CPU、内存、存储）使用情况。

创建 metrics-server 资源（需等待较长时间，约 40 分钟）：

[root@master ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -O metrics-server-components.yaml

[root@master ~]# sed -i 's/registry.k8s.io\/metrics-server/registry.cn-hangzhou.aliyuncs.com\/google_containers/g' metrics-server-components.yaml

[root@master ~]# vim metrics-server-components.yaml
# metrics-server v0.8.0 版本后安全要求较高，默认必须配置 --kubelet-insecure-tls，否则无法从 kubelet 拉取指标，导致探针失败。
...
containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=10250
    - --kubelet-insecure-tls     # 添加
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --kubelet-use-node-status-port
    - --metric-resolution=15s
    image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.8.0
...

[root@master ~]# kubectl apply -f metrics-server-components.yaml

删除 Pod

[root@master ~]# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
calico-kube-controllers-658d97c59c-8nnql   1/1     Running   0             16d
calico-node-5t2r8                          1/1     Running   0             3h29m
calico-node-hfkjr                          1/1     Running   0             3h29m
calico-node-s8jpz                          1/1     Running   0             3h29m
coredns-66f779496c-bcq9q                   1/1     Running   0             17d
coredns-66f779496c-wlphs                   1/1     Running   0             17d
etcd-master                                1/1     Running   2 (16d ago)   17d
kube-apiserver-master                      1/1     Running   2 (16d ago)   17d
kube-controller-manager-master             1/1     Running   2 (16d ago)   17d
kube-proxy-27r8d                           1/1     Running   2 (16d ago)   17d
kube-proxy-6wbkr                           1/1     Running   1 (16d ago)   16d
kube-proxy-92wqj                           1/1     Running   1 (16d ago)   16d
kube-scheduler-master                      1/1     Running   2 (16d ago)   17d
metrics-server-57999c5cf7-ppvqq            1/1     Running   0             176m
metrics-server-785fdd47c4-td8zt            0/1     Terminating         0      11m
metrics-server-785fdd47c4-zm4rb            0/1     Terminating         0      9m45s

# 可以把不再使用的 Pod 删除
[root@master ~]# kubectl delete pod metrics-server-785fdd47c4-td8zt -n kube-system
pod "metrics-server-785fdd47c4-td8zt" deleted

# 强制删除 Pod
[root@master ~]# kubectl delete pod metrics-server-785fdd47c4-td8zt -n kube-system --grace-period=0 --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "metrics-server-785fdd47c4-td8zt" force deleted

[root@master ~]# kubectl delete pod metrics-server-785fdd47c4-zm4rb -n kube-system --grace-period=0 --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "metrics-server-785fdd47c4-zm4rb" force deleted

查看资源创建过程：

[root@master ~]# kubectl describe pod metrics-server-57999c5cf7-ppvqq -n kube-system

使用 kubectl top 查看资源。

Pod：

[root@master ~]# kubectl top pod kube-apiserver-master -n kube-system
NAME                    CPU(cores)   MEMORY(bytes)
kube-apiserver-master   78m          384Mi

Node：

[root@master ~]# kubectl top node node1
NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node1   137m         6%     1178Mi          32%

[root@master ~]# kubectl top node node2
NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node2   100m         5%     1202Mi          32%

3. HPA 案例实战（CPU 指标）

部署应用服务

创建 nginx 的应用及服务：

[root@master hpa]# vim nginx01-svc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.26-alpine
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 200m
              memory: 100Mi
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx

应用 YAML：

[root@master hpa]# kubectl apply -f nginx01-svc.yaml

查看资源：

[root@master hpa]# kubectl get pods,svc
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-85f7bbbc89-f9ql9   1/1     Running   0          2m13s
pod/nginx-85f7bbbc89-mkhsn   1/1     Running   0          2m13s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP         5d23h
service/nginx        NodePort    10.99.85.152   <none>        80:30699/TCP   2m14s

网站访问：http://192.168.18.129:30699/

创建 HPA 对象

这是一个 HorizontalPodAutoscaler（HPA）对象的配置，它将控制 Deployment nginx 的副本数量。当 CPU 使用率超过 50% 时，HPA 将自动增加 Pod 的副本数量，最高不超过 10 个。

[root@master hpa]# vim nginx-hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
  namespace: default
spec:
  scaleTargetRef:          # 自动伸缩控制的资源
    apiVersion: apps/v1
    kind: Deployment
    name: nginx             # 一定要和 deployment 的名称一致
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization    # 目标类型的利用率
          averageUtilization: 50  # 当平均 CPU 利用率超过 50% 时，将触发自动伸缩操作

应用并查看：

[root@master hpa]# kubectl apply -f nginx-hpa.yaml
horizontalpodautoscaler.autoscaling/nginx-hpa created

[root@master hpa]# kubectl get hpa
NAME        REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   0%/50%    1         10        2          62s

执行压测

安装压力测试工具：

[root@master hpa]# yum install -y httpd-tools

进行压测，并发请求数 10000，总请求数百亿：

[root@master hpa]# ab -c 10000 -n 10000000000 http://192.168.18.129:30699/

This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.18.129 (be patient)
apr_socket_recv: Connection reset by peer (104)
Total of 93609 requests completed

观测 Pods 数量自动扩容，从原来的 2 个变成 9 个：

[root@master ~]# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-85f7bbbc89-cgdbk   1/1     Running   0          41s
nginx-85f7bbbc89-cvggh   1/1     Running   0          4m26s
nginx-85f7bbbc89-dbzdb   1/1     Running   0          41s
nginx-85f7bbbc89-f2hxg   1/1     Running   0          3m56s
nginx-85f7bbbc89-f9ql9   1/1     Running   0          44m
nginx-85f7bbbc89-jr2nl   1/1     Running   0          4m26s
nginx-85f7bbbc89-pnqzx   1/1     Running   0          41s
nginx-85f7bbbc89-r5glc   1/1     Running   0          26s
nginx-85f7bbbc89-rzw6s   1/1     Running   0          41s

资源使用降低后自动收缩，经过 5 分钟等待，Pod 数量变回 1 个：

[root@master hpa]# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-85f7bbbc89-f9ql9   1/1     Running   0          49m

---

二、通过 Prometheus 及 HPA 实现 Kubernetes 应用水平自动伸缩

1. metrics-server 部署环境

前面已经安装，直接验证，流程参照第 1 部 K8s 管理：

[root@master ~]# kubectl top pods -n kube-system
NAME                                       CPU(cores)   MEMORY(bytes)
calico-kube-controllers-658d97c59c-z9sns   2m           22Mi
calico-node-6bj6w                          65m          98Mi
calico-node-mtl8j                          60m          136Mi
calico-node-zmbmx                          70m          127Mi
coredns-66f779496c-qsc44                   3m           18Mi
coredns-66f779496c-zs95w                   3m           17Mi
etcd-master                                32m          153Mi
kube-apiserver-master                      78m          302Mi
kube-controller-manager-master             27m          55Mi
kube-proxy-2qt7j                           9m           21Mi
kube-proxy-szcm6                           1m           24Mi
kube-proxy-t2fww                           1m           22Mi
kube-scheduler-master                       5m           23Mi
metrics-server-57999c5cf7-k8thb            4m           24Mi

[root@master ~]# kubectl edit configmap kube-proxy -n kube-system
...
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: true          # 设置为 true
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
...

重启 kube-proxy：

[root@master ~]# kubectl rollout restart daemonset kube-proxy -n kube-system
daemonset.apps/kube-proxy restarted

2. MetalLB 部署

[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.2/config/manifests/metallb-native.yaml

配置二层网络：

[root@master demo]# vim ippool.yaml

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.18.240-192.168.18.250

开启二层通告：

[root@master demo]# vim l2.yaml

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system

应用前查看环境是否准备就绪：

[root@master demo]# kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-8666ddd68b-6l65c  1/1     Running   0          7m17s
speaker-4sv9l                 1/1     Running   0          7m17s
speaker-76hp5                 1/1     Running   0          7m17s
speaker-rhn46                 1/1     Running   0          7m17s

应用 YAML：

[root@master demo]# kubectl apply -f ippool.yaml -f l2.yaml
ipaddresspool.metallb.io/first-pool created
l2advertisement.metallb.io/example created

查看地址池：

[root@master demo]# kubectl get ipaddresspool -n metallb-system
NAME         AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
first-pool   true          false             ["192.168.18.240-192.168.18.250"]

3. ingress-nginx 部署

打开 GitHub 官网，搜索 ingress nginx。

版本获取：

选择 deploy：

获取 ingress-nginx 部署文件：

[root@master demo]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.13.2/deploy/static/provider/cloud/deploy.yaml

编辑 YAML 文件：

[root@master demo]# vim deploy.yaml
...
spec:
  externalTrafficPolicy: Cluster       # 347 行位置把 Local 修改为 Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
...

应用 YAML：

[root@master demo]# kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created

查看资源：

[root@master demo]# kubectl get ns
NAME                   STATUS   AGE
default                Active   6d1h
ingress-nginx          Active   55s       # 已创建
kube-node-lease        Active   6d1h
kube-public            Active   6d1h
kube-system            Active   6d1h
kubernetes-dashboard   Active   6d1h
metallb-system         Active   32m

# Pod 运行需要等 5 分钟，service 被分配 IP：192.168.18.240
[root@master demo]# kubectl get pods,svc -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-555c8f5ff6-mvnv5   1/1     Running   0          4m37s

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.102.242.209  192.168.18.240   80:31314/TCP,443:31978/TCP   4m37s
service/ingress-nginx-controller-admission   ClusterIP      10.106.224.57   <none>           443/TCP                      4m37s

4. Prometheus 部署

安装 Helm

先安装 Helm，在 GitHub 上搜索 helm。

使用迅雷下载或者 wget 下载，压缩包解压，并把命令文件放到系统命令目录：

[root@master hpa]# tar zxvf helm-v3.19.0-linux-amd64.tar.gz
linux-amd64/
linux-amd64/README.md
linux-amd64/LICENSE
linux-amd64/helm

[root@master hpa]# ls
demo  helm-v3.19.0-linux-amd64.tar.gz  linux-amd64  nginx01-svc.yaml  nginx-hpa.yaml

[root@master hpa]# cd linux-amd64/
[root@master linux-amd64]# ls
helm  LICENSE  README.md

[root@master linux-amd64]# mv helm /usr/bin/
[root@master linux-amd64]# helm version
version.BuildInfo{Version:"v3.19.0", GitCommit:"3d8990f0836691f0229297773f3524598f46bda6", GitTreeState:"clean", GoVersion:"go1.24.7"}

添加 Prometheus 仓库源

[root@master linux-amd64]# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

[root@master linux-amd64]# helm repo list
NAME                    URL
prometheus-community    https://prometheus-community.github.io/helm-charts

[root@master linux-amd64]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈

[root@master linux-amd64]# helm search repo prometheus

安装 Prometheus 全家桶

[root@master hpa]# mkdir promedir
[root@master hpa]# cd promedir/
[root@master promedir]# helm show values prometheus-community/kube-prometheus-stack --version 77.6.2 > kube-prometheus-stack.yaml
[root@master promedir]# ls
kube-prometheus-stack.yaml

Prometheus 默认只启用对 Kubernetes 集群组件监控，需要关闭，让它可以对所有应用进行服务发现：

[root@master promedir]# vim kube-prometheus-stack.yaml
...
serviceMonitorSelectorNilUsesHelmValues: false     # 4138 行

安装部署 Prometheus，命名为 kps：

[root@master promedir]# helm install kps prometheus-community/kube-prometheus-stack --version 77.6.2 -f ./kube-prometheus-stack.yaml -n monitoring --create-namespace --debug

检查创建的 Pod 资源：

[root@master promedir]# kubectl --namespace monitoring get pods -l "release=kps"
NAME                                                READY   STATUS    RESTARTS   AGE
kps-kube-prometheus-stack-operator-c9ffdc7d-8vwvz   1/1     Running   0          3m30s
kps-kube-state-metrics-bcd5b8dfc-j8wsj              1/1     Running   0          3m30s
kps-prometheus-node-exporter-98sjz                  1/1     Running   0          3m30s
kps-prometheus-node-exporter-jbqwb                  1/1     Running   0          3m30s
kps-prometheus-node-exporter-pw25m                  1/1     Running   0          3m30s

[root@master promedir]# kubectl get svc -n monitoring
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                    ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   5m3s
kps-grafana                              ClusterIP   10.105.74.172    <none>        80/TCP                       5m52s
kps-kube-prometheus-stack-alertmanager   ClusterIP   10.97.109.162    <none>        9093/TCP,8080/TCP            5m52s
kps-kube-prometheus-stack-operator       ClusterIP   10.104.170.36    <none>        443/TCP                      5m52s
kps-kube-prometheus-stack-prometheus     ClusterIP   10.97.26.101     <none>        9090/TCP,8080/TCP            5m52s
kps-kube-state-metrics                   ClusterIP   10.102.69.181    <none>        8080/TCP                     5m52s
kps-prometheus-node-exporter             ClusterIP   10.110.210.235   <none>        9100/TCP                     5m52s
prometheus-operated                      ClusterIP   None             <none>        9090/TCP                     5m2s

配置 Prometheus 与 Grafana 通过 Ingress 访问

配置 Prometheus 访问：

[root@master promedir]# vim prometheus-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-prometheus   # 自定义 ingress 名称
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
    - host: prometheus.abner.com   # 自定义域名
      http:
        paths:
          - backend:
              service:
                name: kps-kube-prometheus-stack-prometheus  # 对应创建的资源名称
                port:
                  number: 9090
            pathType: Prefix
            path: "/"

配置 Grafana 访问：

[root@master promedir]# vim grafana-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grafana    # 自定义 ingress 名称
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
    - host: grafana.abner.com  # 自定义域名
      http:
        paths:
          - backend:
              service:
                name: kps-grafana  # 对应资源名称
                port:
                  number: 80
            pathType: Prefix
            path: "/"

应用 YAML：

[root@master promedir]# kubectl apply -f prometheus-ingress.yaml -f grafana-ingress.yaml
ingress.networking.k8s.io/ingress-prometheus created
ingress.networking.k8s.io/ingress-grafana created

[root@master promedir]# kubectl get ingress -n monitoring
NAME                 CLASS   HOSTS                  ADDRESS          PORTS   AGE
ingress-grafana      nginx   grafana.abner.com      192.168.18.240   80      70s
ingress-prometheus   nginx   prometheus.abner.com   192.168.18.240   80      70s

打开 Windows 主机，修改 hosts 文件，方便直接域名解析。目录：C:\Windows\System32\drivers\etc

添加 IP-域名的映射：

打开浏览器访问 Prometheus 首页：http://prometheus.abner.com/

点击 Status → Targets 可以直接看到集群中组件状态：

打开浏览器访问 Grafana 首页：http://grafana.abner.com/

查看用户名和密码：

[root@master promedir]# kubectl get secret kps-grafana -n monitoring -o yaml
apiVersion: v1
data:
  admin-password: cHJvbS1vcGVyYXRvcg==       # 等会使用 base64 进行解析
  admin-user: YWRtaW4=                   # 等会使用 base64 进行解析
  ldap-toml: ""
kind: Secret
...

使用 base64 进行密文解析：

# 用户名
[root@master promedir]# echo -n "YWRtaW4=" | base64 --decode
admin

# 密码
[root@master promedir]# echo -n "cHJvbS1vcGVyYXRvcg==" | base64 --decode
prom-operator

登录成功：

查看对所有 Node 的数据监控：

5. 部署 Web 类应用 nginx

部署 nginx 应用

在 PyCharm 中安装 nginx 配置插件：

编写 nginx.conf：

[root@master hpa]# mkdir nginxdir
[root@master hpa]# cd nginxdir/
[root@master nginxdir]# vim nginx.conf

worker_processes 1;

events { worker_connections 1024; }

http {
    server {
        listen 80;

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }

        location /basic_status {
            stub_status;
            allow 127.0.0.1;
            deny all;
        }
    }
}

创建 nginx 配置资源：

[root@master nginxdir]# kubectl create configmap nginx-config --from-file=nginx.conf
configmap/nginx-config created

[root@master nginxdir]# kubectl get cm
NAME               DATA   AGE
kube-root-ca.crt   1      7d
nginx-config       1      38s

创建 nginx 的 YAML

[root@master nginxdir]# vim nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-with-exporter
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.26-alpine
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
              name: nginx-config-volume
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
        - name: nginx-prometheus-exporter
          image: nginx/nginx-prometheus-exporter:latest
          imagePullPolicy: IfNotPresent
          args: ["-nginx.scrape-uri=http://localhost/basic_status"]
          ports:
            - containerPort: 9113
              name: exporter-port
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
      volumes:
        - name: nginx-config-volume
          configMap:
            name: nginx-config
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx

应用 YAML：

[root@master nginxdir]# kubectl apply -f nginx.yaml
deployment.apps/nginx-with-exporter created
service/nginx created

[root@master nginxdir]# kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-with-exporter-864b945bc9-cpngt   2/2     Running   0          4s
nginx-with-exporter-864b945bc9-mmw58   2/2     Running   0          4s

在 Prometheus 中添加 nginx 监控配置

[root@master promedir]# vim kube-prometheus-stack.yaml
# 注意对齐方式
additionalScrapeConfigs:
  - job_name: 'nginx'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        action: keep
        regex: exporter-port
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

配置说明：

additionalScrapeConfigs:
  - job_name: 'nginx'              # 定义一个抓取任务，名字叫 nginx
    kubernetes_sd_configs:
      - role: pod                  # 通过 Kubernetes 的 "pod" 服务发现机制找到目标
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        action: keep
        regex: exporter-port       # 只保留 port 名称为 exporter-port 的容器端口
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace    # 把 Pod 的 namespace 写到 time series 的 "namespace" 标签上
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod          # 把 Pod 名称写到 time series 的 "pod" 标签上

工作逻辑：

1. 服务发现 ：kubernetes_sd_configs + role: pod → Prometheus 会去 K8s API 查询所有 Pod。
2. 筛选目标 ：relabel_configs 的第一个规则，从 Pod 的容器端口名字（__meta_kubernetes_pod_container_port_name）取值，只有当端口名是 exporter-port 时才保留这个 Pod 的端点；也就是说，只有标记了 containerPort.name: exporter-port 的 Pod 才会被抓取。
3. 加标签 ：第二条规则把 Pod 所在的命名空间加到 time series 标签里，叫 namespace；第三条规则把 Pod 的名字加到 time series 标签里，叫 pod。

这样做的好处是，Prometheus 在查询/监控的时候可以方便区分哪个 Pod、哪个 Namespace 的 nginx 指标。

注意：原有的 additionalScrapeConfigs: [] 功能项直接注释。

更新配置：

[root@master promedir]# helm upgrade kps prometheus-community/kube-prometheus-stack --version 77.6.2 -f ./kube-prometheus-stack.yaml -n monitoring

Release "kps" has been upgraded. Happy Helming!
NAME: kps
LAST DEPLOYED: Mon Sep 15 22:20:34 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 2
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=kps"

Get Grafana 'admin' user password by running:
  kubectl --namespace monitoring get secrets kps-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

等待 2-3 分钟刷新 Prometheus 网页，可以查看到监控 nginx 应用的状态：

在 Prometheus 中查看 nginx 的请求数据：

6. Prometheus-Adapter

如果目标是使用 HPA 根据 nginx_http_requests_total 指标进行自动扩缩容，那么安装和配置 Prometheus-Adapter 是必要的。这是因为 HPA 需要使用 Kubernetes 自定义指标 API 来访问这些指标，而 Prometheus-Adapter 正是用来提供这个桥接功能的。

Prometheus-Adapter 作为一个自定义指标 API 服务器，允许 Kubernetes 集群使用 Prometheus 作为后端来获取指标数据。这些数据可以用于 Kubernetes 中的水平自动扩缩容（HPA）或者 KEDA（基于事件驱动自动扩缩容）。

在 Kubernetes 生态系统中，Prometheus Adapter 是一个重要的组件，用于集成 Prometheus 和 Kubernetes 的自动伸缩功能。"Adapter" 在这里可以翻译为"适配器"。

适配器的角色： Prometheus Adapter 充当了 Prometheus 和 Kubernetes 之间的桥梁或适配器，它使 Kubernetes 能够使用 Prometheus 中收集的指标来做出自动伸缩（如水平 Pod 自动伸缩）的决策。

工作原理： Prometheus 本身收集和存储了大量的指标数据，但 Kubernetes 的 HPA（Horizontal Pod Autoscaler）或 VPA（Vertical Pod Autoscaler）无法直接使用这些数据。Prometheus Adapter 的作用是将 Prometheus 中的指标转换为 Kubernetes 可以理解的格式，并通过 Kubernetes 自定义指标 API 提供这些数据。这样，当 HPA 或 VPA 需要根据特定指标（如每秒 HTTP 请求的数量或特定队列的长度）来自动调整 Pod 的数量时，它们可以查询这些由 Prometheus Adapter 提供的指标。

配置和使用： 在 Prometheus Adapter 中，用户需要定义如何从 Prometheus 查询特定指标，以及如何将这些指标映射到 Kubernetes 可理解的资源上（如 Pod、服务等）。一旦配置完成并且 Prometheus Adapter 正在运行，用户可以在 Kubernetes HPA 或 VPA 配置中引用这些自定义指标，以实现基于更丰富指标集的自动伸缩策略。

实际应用： Prometheus Adapter 广泛应用于需要动态伸缩资源以适应不断变化工作负载的 Kubernetes 集群中。例如，如果某个服务在流量高峰期间需要更多的 Pod 来处理请求，Prometheus Adapter 可以提供相应的流量指标给 HPA，HPA 再根据这些指标来增加 Pod 的数量。

[root@master ~]# helm search repo prometheus-adapter
NAME                                    CHART VERSION   APP VERSION  DESCRIPTION
prometheus-community/prometheus-adapter 5.1.0           v0.12.0      A Helm chart for k8s prometheus adapter

下载适配器 YAML：

[root@master ~]# helm show values prometheus-community/prometheus-adapter --version 5.1.0 > prometheus-adapter.yaml

修改配置文件：

[root@master ~]# vim prometheus-adapter.yaml
...
prometheus:
  # Value is templated
  url: http://kps-kube-prometheus-stack-prometheus.monitoring.svc.cluster.local.   # 37 行修改
  port: 9090
  path: ""
...
rules:                         # 129 行添加
  default: true
  custom:
    - seriesQuery: 'nginx_http_requests_total{namespace!="",pod!=""}'    # 持续查询
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        as: "nginx_http_requests"
      metricsQuery: 'sum(rate(nginx_http_requests_total{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

说明： 要在 Prometheus-Adapter 的配置文件中添加 nginx_http_requests_total 指标，需要在 values.yaml 文件的 rules 部分进行配置，通常是在 custom 或 external 规则数组中添加新规则。

• seriesQuery 定义了 Prometheus Adapter 如何查找相关的 Prometheus 时间序列。它匹配所有的 nginx_http_requests_total 时间序列，并且确保包括命名空间和 Pod 名称。
• resources 部分定义了如何将 Prometheus 标签映射到 Kubernetes 资源。
• name 部分定义了如何转换指标名称。在这个示例中，将 xxx_total 转换为 xxx。
• metricsQuery 定义了实际的 Prometheus 查询，rate 函数用于计算每秒的速率。

请注意，这只是一个基本示例，具体配置可能需要根据实际情况进行调整。特别是 seriesQuery 和 metricsQuery 需要根据 Prometheus 设置和所需的监控指标进行精确配置。

作用： 把 Prometheus 中的 nginx_http_requests_total 转换成一个可供 HPA 使用的自定义指标 nginx_http_requests，表示"每个 Pod 的请求速率（QPS）"。

执行安装部署：

[root@master ~]# helm install prometheus-adapter prometheus-community/prometheus-adapter --namespace monitoring --version 5.1.0 -f ./prometheus-adapter.yaml

NAME: prometheus-adapter
LAST DEPLOYED: Tue Sep 16 20:27:26 2025
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
prometheus-adapter has been deployed.
In a few minutes you should be able to list metrics using the following command(s):
  kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

查看资源：

[root@master ~]# kubectl get pods -n monitoring
NAME                                                    READY   STATUS      RESTARTS   AGE
alertmanager-kps-kube-prometheus-stack-alertmanager-0   2/2     Running     0          24h
kps-grafana-58c8ff8f9b-fjgwq                            3/3     Running     0          24h
kps-kube-prometheus-stack-operator-c9ffdc7d-8vwvz       1/1     Running     0          24h
kps-kube-state-metrics-bcd5b8dfc-j8wsj                  1/1     Running     0          24h
kps-prometheus-node-exporter-98sjz                      1/1     Running     0          24h
kps-prometheus-node-exporter-jbqwb                      1/1     Running     0          24h
kps-prometheus-node-exporter-pw25m                      1/1     Running     0          24h
prometheus-adapter-6d86b4f47c-zgwpw                     1/1     Running     0          95s   # 成功运行
prometheus-kps-kube-prometheus-stack-prometheus-0       2/2     Running     0          24h

使用 jq 查看：

[root@master ~]# wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo
--2025-09-16 20:32:11--  https://mirrors.aliyun.com/repo/epel-7.repo
正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 180.97.209.55, 221.229.203.99, 180.97.209.56, ...
正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|180.97.209.55|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：664 [application/octet-stream]
正在保存至: "/etc/yum.repos.d/epel.repo"
100% [===============================================================================] 664 --.-K/s 用时 0s
2025-09-16 20:32:11 (56.7 MB/s) - 已保存 "/etc/yum.repos.d/epel.repo" [664/664])

[root@master ~]# yum install jq -y

[root@master ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

查看 default 命名空间 Pod 的每秒请求数：

[root@master ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/nginx_http_requests" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "nginx-with-exporter-864b945bc9-cpngt",
        "apiVersion": "/v1"
      },
      "metricName": "nginx_http_requests",
      "timestamp": "2025-09-16T13:08:05Z",
      "value": "33m",
      "selector": null
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "nginx-with-exporter-864b945bc9-mmw58",
        "apiVersion": "/v1"
      },
      "metricName": "nginx_http_requests",
      "timestamp": "2025-09-16T13:08:05Z",
      "value": "33m",
      "selector": null
    }
  ]
}

7. 创建 HPA 对象并实现自动伸缩

创建 HPA 资源

[root@master hpa]# vim hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
  namespace: default
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-with-exporter
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageValue
          averageValue: 150Mi
    - type: Pods
      pods:
        metric:
          name: nginx_http_requests
        target:
          type: AverageValue
          averageValue: 50

应用 YAML：

[root@master hpa]# kubectl apply -f hpa.yaml
horizontalpodautoscaler.autoscaling/nginx-hpa created

查看资源：

[root@master hpa]# kubectl get hpa
NAME        REFERENCE                        TARGETS                                  MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx-with-exporter   0%/70%, 11583488/150Mi + 1 more...      1         5         2          2m16s
# 等待 5 分钟后自动缩减为 1
[root@master hpa]# kubectl get hpa
NAME        REFERENCE                        TARGETS                                  MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx-with-exporter   1%/70%, 11264k/150Mi + 1 more...        1         5         1          11m

[root@master hpa]# kubectl describe hpa nginx-hpa
Name:                  nginx-hpa
Namespace:             default
Reference:             Deployment/nginx-with-exporter
Metrics:               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 70%
  resource memory on pods:                             11952128 / 150Mi
  "nginx_http_requests" on pods:                       33m / 50
Min replicas:         1
Max replicas:         5
Deployment pods:       2 current / 2 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from memory resource
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:           <none>

进行压测

安装压测工具：

[root@master hpa]# yum install httpd-tools -y

查看 svc，获取 nginx 访问地址：10.96.221.125

[root@master hpa]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        8d
nginx        NodePort    10.96.221.125   <none>        80:31405/TCP   24h

打开另一个终端持续性观测 Pod 状态：

[root@master ~]# watch kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-with-exporter-864b945bc9-cpngt   2/2     Running   0          24h

开始压测：

[root@master hpa]# ab -c 1000 -n 1000000 http://10.96.221.125/

This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.96.221.125 (be patient)

结果发现自动扩容为 5 个 Pod：

查看扩容事件：

[root@master ~]# kubectl describe hpa nginx-hpa
...
Events:
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  15m    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  Normal  SuccessfulRescale  2m46s  horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  2m31s  horizontal-pod-autoscaler  New size: 4; reason: pods metric nginx_http_requests above target
  Normal  SuccessfulRescale  2m16s  horizontal-pod-autoscaler  New size: 5; reason: pods metric nginx_http_requests above target

12 分钟后自动缩容为 1 个 Pod：

查看事件：

[root@master hpa]# kubectl describe hpa nginx-hpa
Events:
  Type    Reason             Age                  From                       Message
  ----    ------             ----                 ----                       -------
  Normal  SuccessfulRescale  14m                  horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  14m                  horizontal-pod-autoscaler  New size: 4; reason: pods metric nginx_http_requests above target
  Normal  SuccessfulRescale  14m                  horizontal-pod-autoscaler  New size: 5; reason: pods metric nginx_http_requests above target
  Normal  SuccessfulRescale  4m                   horizontal-pod-autoscaler  New size: 3; reason: All metrics below target
  Normal  SuccessfulRescale  3m45s (x2 over 26m)  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

---

三、垂直自动伸缩 VPA

VPA，全称 Vertical Pod Autoscaler，即垂直 Pod 自动扩缩容。它根据容器资源使用率自动设置 CPU 和内存的 requests，从而允许在节点上进行适当的调度，以便为每个 Pod 提供适当的资源。

它既可以缩小过度请求资源的容器，也可以根据其使用情况随时提升资源不足的容量。有些时候无法通过增加 Pod 数来扩容，比如数据库，这时候可以通过 VPA 增加 Pod 使用资源的大小，比如调整 Pod 的 CPU 和内存，进而调整 Pod 资源使用。

使用 VPA 的优缺点：

优点：

• Pod 资源用其所需，所以集群节点使用效率高；
• Pod 会被安排到具有适当可用资源的节点上；
• 不必运行基准测试任务来确定 CPU 和内存请求的合适值；
• VPA 可以随时调整 CPU 和内存请求，无需人为操作，因此可以减少维护时间。

缺点：

• 不能与 HPA（Horizontal Pod Autoscaler）一起使用。

1. metrics-server 部署

在前面已经部署，此处忽略。

[root@master hpa]# kubectl get pods -n kube-system | grep metrics-server
metrics-server-57999c5cf7-k8thb            1/1     Running   1 (7d1h ago)   7d1h

[root@master hpa]# kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master   443m         11%    2694Mi          34%
node1    288m         7%     2279Mi          29%
node2    345m         8%     2236Mi          29%

[root@master hpa]# kubectl top pods -n kube-system
NAME                                       CPU(cores)   MEMORY(bytes)
calico-kube-controllers-658d97c59c-z9sns   5m           23Mi
calico-node-c25gl                          76m          98Mi
calico-node-vp7hm                          64m          74Mi
calico-node-xf42x                          67m          109Mi
coredns-66f779496c-qsc44                   3m           21Mi
coredns-66f779496c-zs95w                   3m           21Mi
etcd-master                                39m          239Mi
kube-apiserver-master                      130m         625Mi
kube-controller-manager-master              31m          65Mi
kube-proxy-847sf                           22m          22Mi
kube-proxy-cdk5f                           20m          23Mi
kube-proxy-zlwrt                           11m          19Mi
kube-scheduler-master                       6m           24Mi
metrics-server-57999c5cf7-k8thb            7m           24Mi

2. 升级 openssl

以下流程在所有节点操作。

[root@master ~]# wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo

[root@master hpa]# yum install -y openssl-devel openssl11 openssl11-devel

[root@master hpa]# which openssl
/usr/bin/openssl
[root@master hpa]# which openssl11
/usr/bin/openssl11

把原来的删除，新的 openssl11 建立软连接：

[root@master hpa]# rm -rf `which openssl`
[root@master hpa]# ln -s /usr/bin/openssl11 /usr/bin/openssl
[root@master hpa]# ls -l /usr/bin/openssl
lrwxrwxrwx. 1 root root 18 9月  16 22:53 /usr/bin/openssl -> /usr/bin/openssl11

[root@master hpa]# openssl version
OpenSSL 1.1.1k  FIPS 25 Mar 2021

3. 部署 VPA

GitHub 搜索地址：https://github.com/search?q=autoscale\&type=repositories

克隆项目：

[root@master ~]# mkdir vpa
[root@master ~]# cd vpa/
[root@master vpa]# git clone https://github.com/kubernetes/autoscaler.git
正克隆到 'autoscaler'...
remote: Enumerating objects: 228840, done.
remote: Counting objects: 100% (1884/1884), done.
remote: Compressing objects: 100% (1264/1264), done.
remote: Total 228840 (delta 1222), reused 620 (delta 620), pack-reused 226956
(from 3)
接收对象中: 100% (228840/228840), 252.35 MiB | 5.78 MiB/s, done.
处理 delta 中: 100% (148187/148187), done.
Checking out files: 100% (8139/8139), done.

[root@master vpa]# ls
autoscaler

[root@master vpa]# cd autoscaler/vertical-pod-autoscaler/
[root@master vertical-pod-autoscaler]# ls
cloudbuild.yaml  deploy  e2e  examples  go.sum  MIGRATE.md  pkg  RELEASE.md
common           docs    enhancements  go.mod  hack  OWNERS  README.md

[root@master vertical-pod-autoscaler]# ls hack/
api-docs                   generate-api-docs.sh  tag-release.sh
vpa-down.sh
boilerplate.go.txt         generate-crd-yaml.sh  tools.go
vpa-process-yaml.sh
convert-alpha-objects.sh    generate-flags.sh     update-codegen.sh
vpa-process-yamls.sh
deploy-for-e2e-locally.sh  lib                   update-kubernetes-deps-in-e2e.sh
vpa-up.sh
deploy-for-e2e.sh          local-cluster.md      update-kubernetes-deps.sh
warn-obsolete-vpa-objects.sh
dev-deploy-locally.sh      run-e2e-locally.sh    verify-codegen.sh
e2e                        run-e2e.sh            verify-vpa-flags.sh
emit-metrics.py            run-e2e-tests.sh      vpa-apply-upgrade.sh

[root@master vertical-pod-autoscaler]# bash ./hack/vpa-up.sh   # 执行会失败，因为 git 版本过低，所以需要升级 git

升级 git：

[root@master hack]# git --version
git version 1.8.3.1         # 低于 2.23 就需要升级，否则会报错：git：'switch' 不是一个 git 命令。参见 'git --help'

# 卸载老版本 git
yum remove git -y

# 安装 git 依赖环境
yum groupinstall "Development Tools" -y
yum install gettext-devel perl-CPAN perl-devel zlib-devel curl-devel expat-devel -y

# 解压缩，并配置安装
tar zxvf git-2.28.0.tar.gz
cd git-2.28.0/
make prefix=/usr/local all
make prefix=/usr/local install

# 删除老版本 git 命令，因为安装环境依赖会把老版本 git 安装上
[root@master git-2.28.0]# rm -rf /usr/bin/git
[root@master bin]# ln -s /usr/local/bin/git /usr/bin/git
[root@master git-2.28.0]# git --version
git version 2.28.0

安装 VPA：

[root@master hack]# cd /root/vpa/autoscaler/vertical-pod-autoscaler/hack
[root@master hack]# bash vpa-up.sh
HEAD 目前位于 d4319b211 Merge pull request #8477 from laoj2/vpa-release-1.4
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrole.rbac.authorization.k8s.io/system:vpa-updater-in-place created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-updater-in-place-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictioner-binding created
serviceaccount/vpa-admission-controller created
serviceaccount/vpa-recommender created
serviceaccount/vpa-updater created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
deployment.apps/vpa-updater created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus (2 primes)
........+++++
...............+++++
e is 65537 (0x010001)
Generating RSA private key, 2048 bit long modulus (2 primes)
.+++++
................................................................................
.................+++++
e is 65537 (0x010001)
Signature ok
subject=CN = vpa-webhook.kube-system.svc
Getting CA Private Key
Uploading certs to the cluster.
secret/vpa-tls-certs created
Deleting /tmp/vpa-certs.
service/vpa-webhook created
deployment.apps/vpa-admission-controller created

查看 VPA 资源：

[root@master hack]# kubectl get pods -n kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
calico-kube-controllers-658d97c59c-z9sns    1/1     Running   0          8d
calico-node-tqhmm                           1/1     Running   0          56m
calico-node-x4pnz                           1/1     Running   0          56m
calico-node-zr2g6                           1/1     Running   0          55m
coredns-66f779496c-qsc44                    1/1     Running   0          8d
coredns-66f779496c-zs95w                    1/1     Running   0          8d
etcd-master                                 1/1     Running   0          8d
kube-apiserver-master                       1/1     Running   0          8d
kube-controller-manager-master               1/1     Running   0          8d
kube-proxy-847sf                            1/1     Running   0          2d22h
kube-proxy-cdk5f                            1/1     Running   0          2d22h
kube-proxy-zlwrt                            1/1     Running   0          2d22h
kube-scheduler-master                       1/1     Running   0          8d
metrics-server-57999c5cf7-k8thb             1/1     Running   1 (7d23h ago)   8d
vpa-admission-controller-7454695ff5-86z9l   1/1     Running   0          90s
vpa-recommender-66c7bfdbc4-6d5st             1/1     Running   0          92s
vpa-updater-bcf54db9c-cp27j                 1/1     Running   0          92s

案例：VPA 应用 updateMode: "Off"

在 VPA 中，updateMode 是一个重要的配置选项，它决定了 VPA 如何应用其提供的资源建议。根据不同的设置，VPA 可以采取不同的策略来更新 Pod 的资源配置：

• Off：VPA 不会应用任何资源推荐，只是收集和显示数据。
• Initial：VPA 只会在 Pod 创建时应用资源推荐。一旦 Pod 启动，即使后续有新的资源推荐，也不会再进行调整。
• Recreate：当 VPA 生成新的资源推荐时，它会终止当前的 Pod 并重新创建一个新的 Pod，新 Pod 将采用最新的资源推荐。这种方式会导致服务短暂中断，但能确保立即应用新的资源设置。
• Auto：这是默认模式。在这种模式下，VPA 会尝试在线调整运行中的 Pod 的资源请求和限制，而无需重启 Pod。如果无法在线调整（例如，由于内核或 Kubernetes 版本的限制），则会选择重新创建 Pod。

创建应用实例

此模式仅获取资源推荐，不更新 Pod。

[root@master vpa]# vim nginx01.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: dep-nginx01
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.26-alpine
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 100m
              memory: 250Mi
---
apiVersion: v1
kind: Service
metadata:
  name: nginx01-svc
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx

应用 YAML：

[root@master vpa]# kubectl apply -f nginx01.yaml
deployment.apps/dep-nginx01 created
service/nginx01-svc created

[root@master vpa]# kubectl get pods,svc
NAME                              READY   STATUS    RESTARTS   AGE
pod/dep-nginx01-b5984d7cc-2wxcm   1/1     Running   0          5m7s
pod/dep-nginx01-b5984d7cc-89vc9   1/1     Running   0          5m7s

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        9d
service/nginx01-svc   NodePort    10.106.168.28   <none>        80:32091/TCP   5m7s

[root@master vpa]# kubectl describe pod dep-nginx01-b5984d7cc-2wxcm
...
    Requests:
      cpu:        100m
      memory:     250Mi
...

创建 VPA 资源：

[root@master vpa]# vim nginx-vpa-off.yaml

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa-off
  namespace: default
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: dep-nginx01
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: "nginx"
        minAllowed:
          cpu: "250m"
          memory: "100Mi"
        maxAllowed:
          cpu: "2000m"
          memory: "2048Mi"

应用 YAML：

[root@master vpa]# kubectl apply -f nginx-vpa-off.yaml
verticalpodautoscaler.autoscaling.k8s.io/nginx-vpa-off created

# 查看资源，会看到推荐值：250m 131072k
[root@master vpa]# kubectl get vpa
NAME            MODE   CPU    MEM       PROVIDED   AGE
nginx-vpa-off   Off    250m   131072k   True       104s

压测后查看推荐值是否有变化：

[root@master vpa]# kubectl get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        9d
nginx01-svc   NodePort    10.106.168.28   <none>        80:32091/TCP   26m

[root@master vpa]# curl http://192.168.18.129:32091
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

# 进行压测
[root@master vpa]# ab -c 1000 -n 1000000 http://10.106.168.28/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.106.168.28 (be patient)
Completed 100000 requests
Completed 200000 requests
...

# 推荐值无变化
[root@master ~]# kubectl get vpa
NAME            MODE   CPU    MEM       PROVIDED   AGE
nginx-vpa-off   Off    250m   131072k   True       22m

案例：VPA 应用 updateMode: "Auto"

此模式当目前运行的 Pod 的资源达不到 VPA 的推荐值，就会执行 Pod 驱逐，重新部署新的足够资源的服务。

创建应用

[root@master vpa]# vim nginx02.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: dep-nginx02
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.26-alpine
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 100m
              memory: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: nginx02-svc
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx

应用 YAML：

[root@master vpa]# kubectl apply -f nginx02.yaml
deployment.apps/dep-nginx02 created
service/nginx02-svc created

[root@master vpa]# kubectl get pods,svc
NAME                               READY   STATUS    RESTARTS   AGE
pod/dep-nginx02-5bf78b9567-6tv8s   1/1     Running   0          2m4s
pod/dep-nginx02-5bf78b9567-jjnz8   1/1     Running   0          2m4s

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        9d
service/nginx02-svc   NodePort    10.109.68.155   <none>        80:31084/TCP   2m4s

创建 VPA 资源：

[root@master vpa]# vim nginx-vpa-auto.yaml

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa-auto
  namespace: default
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: dep-nginx02
  updatePolicy:
    updateMode: "Auto"          # 模式：Auto
  resourcePolicy:
    containerPolicies:
      - containerName: "nginx"
        minAllowed:
          cpu: "250m"
          memory: "100Mi"
        maxAllowed:
          cpu: "2000m"
          memory: "2048Mi"

应用 YAML：

[root@master vpa]# kubectl apply -f nginx-vpa-auto.yaml
verticalpodautoscaler.autoscaling.k8s.io/nginx-vpa-auto created

[root@master vpa]# kubectl get vpa
NAME             MODE   CPU    MEM       PROVIDED   AGE
nginx-vpa-auto   Auto   250m   131072k   True       62s

再次查看 Pod 资源，发现 Pod 被重建：

[root@master vpa]# kubectl get pods,svc
NAME                               READY   STATUS    RESTARTS   AGE
pod/dep-nginx02-5bf78b9567-gjqr8   1/1     Running   0          63s
pod/dep-nginx02-5bf78b9567-rqdkz   1/1     Running   0          2m3s

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP        9d
service/nginx02-svc   NodePort    10.109.68.155   <none>        80:31084/TCP   11m

查看 Pod 资源详细信息发现，使用了推荐值，进行了垂直扩展：

[root@master vpa]# kubectl describe pod dep-nginx02-5bf78b9567-gjqr8
...
  Requests:
    cpu:        250m
    memory:     131072k
...

[root@master vpa]# kubectl describe pod dep-nginx02-5bf78b9567-rqdkz
...
  Requests:
    cpu:        250m
    memory:     131072k
...