提示:本文原创作品,良心制作,干货为主,简洁清晰,一看就会
文章目录
- 前言
- 一、HPA基础知识
-
- [1.1 HPA是什么](#1.1 HPA是什么)
- [1.2 HPA 扩缩容依赖体系](#1.2 HPA 扩缩容依赖体系)
- [1.3 HPA工作原理](#1.3 HPA工作原理)
- 二、内置指标伸缩
-
- [2.1 部署metrics-server](#2.1 部署metrics-server)
- [2.2 CPU阈值扩缩容](#2.2 CPU阈值扩缩容)
- [2.3 内存阈值扩缩容](#2.3 内存阈值扩缩容)
- 三、自定义指标伸缩
-
- [3.1 安装Prometheus](#3.1 安装Prometheus)
- [3.2 安装Prometheus Adapter](#3.2 安装Prometheus Adapter)
- [3.3 QPS流量维度扩缩容](#3.3 QPS流量维度扩缩容)
前言
业务流量往往存在潮汐波动:白天访问峰值拉高服务器负载,夜间流量回落资源大量闲置。若依靠人工手动调整Pod副本数,不仅响应滞后、效率低下,还极易出现资源浪费或服务卡顿问题。Kubernetes 提供HPA水平Pod自动扩缩容组件,可依据监控指标动态增减业务Pod实例。本文将循序渐进讲解HPA两种主流伸缩方案:基于内置指标伸缩、基于自定义指标伸缩,梳理指标采集全链路与实操部署流程
一、HPA基础知识
1.1 HPA是什么
HPA 是 Kubernetes 的水平Pod自动扩缩容控制器,它像是一个"自动挡",根据CPU利用率、内存或自定义业务指标,自动增加或减少Pod的副本数量
核心作用
应对高流量:业务上涨时自动扩容,保证系统稳定不宕机
节省成本:业务低谷时自动缩容,释放闲置资源,降低云成本
工作逻辑 :它不直接操作容器,而是调整 Deployment 或 StatefulSet 的 replicas 字段,让Pod数量始终维持在"不多不少"的状态
注:它只能"横向"加减机器数量,不能"纵向"调整单台机器的CPU/内存大小,那是VPA干的活
yaml
## K8s集群同时提供两套 HPA API 版本
root@k8s-master1:~# kubectl api-versions | grep autoscaling
autoscaling/v1
autoscaling/v2
autoscaling/v1:老旧版本,仅支持 CPU 单一指标伸缩,现已逐步废弃;
autoscaling/v2:标准新版,支持 CPU、内存、自定义业务指标、外部指标多维度伸缩
1.2 HPA 扩缩容依赖体系
下图展示K8s的API聚合层架构:kube-aggregator作为统一网关,把基础监控指标、业务自定义指标的请求分别转发给metrics-server和custom-metrics-server,为HPA水平自动扩缩容提供监控数据支撑

核心组件总览
- kube-aggregator :API聚合层,内置在kube-apiserver里,是整个架构的入口网关,
APIService资源注册到aggregator,注册后aggregator才知道路径对应哪个后端服务 - kube-apiserver:K8s原生核心API(Pod/Deployment/Service等内置资源)
- metrics-server :标准指标服务,提供
内置资源指标(CPU、内存使用率),无持久化存储,只保留实时最近数据,需要单独安装 - custom-metrics-server :
自定义指标服务,提供业务自定义指标(QPS、并发、队列长度等),只有v2版本HPA能读取这套接口,需要单独安装 - Other-server:其他第三方扩展API服务
1.3 HPA工作原理
kube-aggregator 作用:统一路由、转发不同API路径到对应的后端服务
客户端(如kubectl、kube-controller-manager里的HPA控制器)只访问同一个apiserver地址,aggregator根据URL路径自动分流:
- 原生资源请求 → 交给内置kube-apiserver处理
/apis/metrics.k8s.io/开头的指标请求 → 转发给 metrics-server/apis/custom.metrics.k8s.io/开头的自定义指标请求 → 转发给 custom-metrics-server
内置指标伸缩:红色链路,获取CPU/内存基础指标
请求地址:
GET http://127.0.0.1:8080/apis/metrics.k8s.io/v1beta1/nodes
- HPA控制器发起请求到kube-aggregator(apiserver)
- aggregator识别路径
metrics.k8s.io,转发GET请求到 metrics-server - metrics-server收集节点/Pod的CPU、内存实时监控数据,返回给aggregator
- aggregator把结果原路返回给HPA控制器
- HPA拿到CPU利用率后,计算是否扩容/缩容
自定义指标伸缩:绿色路线,v2 HPA才支持
请求地址:
GET http://127.0.0.1:8080/apis/custom.metrics.k8s.io/v1beta1
- 客户端请求自定义业务指标(比如HTTP请求QPS、消息队列堆积量)
- aggregator识别路径
custom.metrics.k8s.io,转发给 custom-metrics-server - custom-metrics-server对接监控系统(Prometheus等),拉取业务指标返回
- HPA v2支持基于这类自定义指标做弹性伸缩(v1 HPA不支持)
二、内置指标伸缩
内置指标伸缩是K8S原生自带的HPA弹性伸缩方案,想要开启该能力,集群仅需单独部署 Metrics-Server 组件即可。Metrics-Server会定期汇总各节点kubelet采集的容器运行数据,通过K8s API聚合层对外提供标准资源指标接口
yaml
## 每隔15秒,HPA 控制器就会去 APIServer 拉取监控指标,计算当前 Pod 副本数是否需要扩容/缩容
root@k8s-master1:~# kubectl exec -n kube-system kube-controller-manager-k8s-master1 -- kube-controller-manager --help | grep horizontal-pod-autoscaler-sync-period
--horizontal-pod-autoscaler-sync-period duration The period for syncing the number of pods in horizontal pod autoscaler. (default 15s)
这张仅支持以 CPU、内存利用率 作为唯一伸缩判定依据,可查看节点与Pod资源负载
yaml
## 查看 HPA 伸缩容忍度,默认值:0.1(也就是 10%)
root@k8s-master1:~# kubectl exec -n kube-system $(kubectl get pod -n kube-system | grep kube-controller-manager | awk '{print $1}') -- kube-controller-manager --help | grep horizontal-pod-autoscaler-tolerance
--horizontal-pod-autoscaler-tolerance float The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling. (default 0.1)
HPA 伸缩容忍度是用来划定无需扩缩容的静止死区,避免指标小幅波动时,Pod 副本频繁扩容、缩容来回抖动
计算公式:
假设我们设定目标 CPU 利用率为 50%
下限 = 目标值 × (1 - 容忍度) = 50% × 0.9 = 45%
上限 = 目标值 × (1 + 容忍度) = 50% × 1.1 = 55%
当 Pod 平均 CPU 利用率处于 45% ~ 55% 区间内:HPA 判定负载波动属于正常范围,不执行任何扩缩容操作;
利用率>55%:才会触发扩容;
利用率<45%:才满足缩容前置条件
2.1 部署metrics-server
Metrics Server需要访问所有的节点以采集指标,默认为kubelet监听的10250端口
metrics-server下载地址:https://github.com/kubernetes-sigs/metrics-server/releases
可以选择适合自己k8s版本的metrics-server,我的k8s是1.28,我这里选择的metrics-server为v0.7.1

yaml
## 1. 没安装Metrics之前
root@k8s-master1:~# kubectl top node
error: Metrics API not available
## 2. 下载修改metrics-server清单文件
root@k8s-master1:~# ls
components.yaml
root@k8s-master1:~# vim components.yaml

yaml
root@k8s-master1:~# kubectl apply -f components.yaml
root@k8s-master1:~# kubectl get pod -A -o wide | grep metrics
kube-system metrics-server-75d7968c55-8jhp4 1/1 Running 0 39s 10.244.36.69 k8s-node1 <none> <none>
## 3. 查看所有Pod 的 CPU、内存占用,安装好metrics-server后才能使用这条命令
root@k8s-master1:~# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master1 308m 7% 1553Mi 30%
k8s-master2 212m 5% 1343Mi 26%
k8s-master3 234m 5% 1474Mi 26%
k8s-node1 183m 4% 1106Mi 29%
k8s-node2 106m 2% 1036Mi 27%
2.2 CPU阈值扩缩容
yaml
root@k8s-master1:~# mkdir /k8s/hpa -p
root@k8s-master1:~# cd /k8s/hpa/
root@k8s-master1:/k8s/hpa# vim hpa-cpu.yaml
# 1. 业务Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: hpav2-demo-deployment
spec:
replicas: 2
selector:
matchLabels:
app: hpav2-demo
template:
metadata:
labels:
app: hpav2-demo
spec:
containers:
- name: hpav2-demo-deployment
image: nginx:1.24
ports:
- containerPort: 80
resources:
requests:
cpu: "200m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
---
# 2. 后端Service
apiVersion: v1
kind: Service
metadata:
name: hpav2-demo-svc
labels:
app: nginx
spec:
ports:
- port: 80
name: http
selector:
app: hpav2-demo
---
# 3. HPA v2 自动扩缩容规则
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: hpav2-cpu-demo
spec:
# 绑定管控的Deployment
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hpav2-demo-deployment
minReplicas: 1 # 最少一个pod
maxReplicas: 10 # 最多10个pod
# 监控CPU使用率50%作为伸缩阈值
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
behavior:
scaleDown:
# 扩缩容等待时间,默认300s
stabilizationWindowSeconds: 10
yaml
## 刚创建完会有两个pod
root@k8s-master1:/k8s/hpa# kubectl apply -f hpa-cpu.yaml
root@k8s-master1:/k8s/hpa# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpav2-demo-deployment-59f8749ddf-6hrs5 1/1 Running 0 33s 10.244.169.139 k8s-node2 <none> <none>
hpav2-demo-deployment-59f8749ddf-ch7nb 1/1 Running 0 33s 10.244.36.73 k8s-node1 <none> <none>
## 当hpa检测到cpu占用率后,会根据cpu占用率扩缩容,目前cpu占用率接近0%,所以会缩容剩一个pod
root@k8s-master1:/k8s/hpa# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpav2-cpu-demo Deployment/hpav2-demo-deployment 0%/50% 1 10 2 34s
root@k8s-master1:/k8s/hpa# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpav2-demo-deployment-59f8749ddf-6hrs5 1/1 Running 0 37s 10.244.169.139 k8s-node2 <none> <none>
可以选一个集群内的机器做压力测试
yaml
## 压力测试
root@k8s-master1:/k8s/hpa# while true; do ab -k -c 100 -n 100000 http://10.244.169.139/; done
用kubectl get hpa观察cpu使用率超过55% pod的数量有没有增加;关闭压力测试再次查看cpu使用率降到45%一下后有没有缩容
基于cpu的内置指标伸缩演示视频
2.3 内存阈值扩缩容
yaml
root@k8s-master1:/k8s/hpa# vim hpa-mem.yaml
# 1. 业务 Deployment(使用一个占用内存的镜像)
apiVersion: apps/v1
kind: Deployment
metadata:
name: memory-demo-deployment
spec:
replicas: 2
selector:
matchLabels:
app: memory-demo
template:
metadata:
labels:
app: memory-demo
spec:
containers:
- name: memory-demo-container
image: nginx:1.24
ports:
- containerPort: 80
resources:
requests:
memory: "128Mi" # 内存请求,HPA 基于此计算利用率
limits:
memory: "256Mi"
---
# 2. Service(与之前类似)
apiVersion: v1
kind: Service
metadata:
name: memory-demo-svc
spec:
ports:
- port: 80
name: http
selector:
app: memory-demo
---
# 3. HPA v2 基于内存自动扩缩容
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: hpa-memory-demo
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: memory-demo-deployment
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80 # 内存使用率达到 80% 触发扩容
behavior:
scaleDown:
stabilizationWindowSeconds: 10 # 缩容等待 10 秒(便于观察)
yaml
root@k8s-master1:/k8s/hpa# kubectl apply -f hpa-mem.yaml
## 创建好后会有原始的2个pod
root@k8s-master1:/k8s/hpa# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
memory-demo-deployment-6894578fd9-2ggjj 1/1 Running 0 14s 10.244.36.76 k8s-node1 <none> <none>
memory-demo-deployment-6894578fd9-db5zs 1/1 Running 0 14s 10.244.169.142 k8s-node2 <none> <none>
## 当hpa获取到内存占用率,就会根据占用率来伸缩pod
root@k8s-master1:/k8s/hpa# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-memory-demo Deployment/memory-demo-deployment 4%/80% 1 10 2 34s
root@k8s-master1:/k8s/hpa# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
memory-demo-deployment-6894578fd9-2ggjj 1/1 Running 0 37s 10.244.36.76 k8s-node1 <none> <none>
进入某个pod内部用stress进行压力测试
yaml
root@k8s-master1:~# kubectl exec -it memory-demo-deployment-6894578fd9-2ggjj /bin/bash
root@memory-demo-deployment-6894578fd9-2ggjj:/# apt update
root@memory-demo-deployment-6894578fd9-2ggjj:/# apt -y install stress
root@memory-demo-deployment-6894578fd9-2ggjj:/# stress --vm 1 --vm-bytes 200M --vm-keep --timeout 100s
然后用kubectl get hpa查看内存占用率,查看占用率达到88%会不会扩容,占用率小于72%会不会缩容
基于内存的内置指标伸缩演示视频
三、自定义指标伸缩
实现基于自定义指标的 HPA 扩缩容,核心思路是为 HPA 提供除 CPU/内存以外的数据源。这需要额外安装组件来采集和转换指标
Prometheus(监控后端)
作用:从你的应用和集群中采集、存储各种自定义指标
前提:你的应用需要暴露一个 /metrics 接口,Prometheus 才能拉取到数据
Prometheus Adapter(核心适配器)
作用:Prometheus 的数据格式 K8s 无法直接识别。Adapter 充当"翻译官",将 Prometheus 中的指标转换为 K8s HPA 能识别的 custom.metrics.k8s.io API 格式
3.1 安装Prometheus
https://github.com/prometheus-community/helm-charts/releases
可以根据你的k8s集群选择适合自己的Prometheus版本

yaml
## 1. 创建monitoring命名空间
root@k8s-master1:~# kubectl create namespace monitoring
## 2. 下载解压Prometheus tar包
root@k8s-master1:~# ls
kube-prometheus-stack-87.17.0.tgz
root@k8s-master1:~# tar xf kube-prometheus-stack-87.17.0.tgz
root@k8s-master1:~# cd kube-prometheus-stack/
root@k8s-master1:~/kube-prometheus-stack# ls
Chart.lock charts Chart.yaml README.md templates values.yaml
## 3. 把Prometheus的svc改为nodeport
## 把prometheus.service.type改成NodePort
root@k8s-master1:~/kube-prometheus-stack# vim values.yaml
## 4. helm安装Prometheus
root@k8s-master1:~/kube-prometheus-stack# helm install prometheus . \
--namespace monitoring \
--create-namespace \
--set kube-state-metrics.image.registry=registry.cn-hangzhou.aliyuncs.com \
--set kube-state-metrics.image.repository=hujiaming/kube-state-metrics \
--set kube-state-metrics.image.tag=v2.13.0
root@k8s-master1:~/kube-prometheus-stack# kubectl get svc -n monitoring | grep prometheus-kube-prometheus-prometheus
prometheus-kube-prometheus-prometheus NodePort 10.96.110.232 <none> 9090:30090/TCP,8080:32551/TCP 35m
root@k8s-master1:~/kube-prometheus-stack# kubectl get pod -n monitoring
这里如果alertmanager,grafana如果起不来没关系,只要Prometheus是好的就行

浏览器访问http://192.168.13.136:30090可以看到Prometheus监控界面

3.2 安装Prometheus Adapter
https://github.com/prometheus-community/helm-charts/releases
根据自己K8S集群版本选择适合自己的Prometheus Adapter

yaml
root@k8s-master1:~# ls
prometheus-adapter-5.0.0.tgz
root@k8s-master1:~# tar xf prometheus-adapter-5.0.0.tgz
root@k8s-master1:~# cd prometheus-adapter/
root@k8s-master1:~/prometheus-adapter# ls
Chart.yaml ci README.md templates values.yaml
root@k8s-master1:~/prometheus-adapter# vim values.yaml


Prometheus Adapter 不会自动暴露所有指标,它需要你通过 rules 字段明确告诉它:"哪个 PromQL 指标,转换成什么名字,暴露给 HPA"
本次实验我以 Nginx 每秒请求量(QPS)作为伸缩依据,因此编写了专属映射规则;后续若想依托并发连接数、消息队列堆积量等其他业务指标实现弹性扩缩容,只需参照格式编写对应的规则配置即可
yaml
rules:
default: true
custom:
- seriesQuery: 'nginx_http_requests_total{namespace!="",pod!=""}'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

yaml
root@k8s-master1:~/prometheus-adapter# helm install prometheus-adapter . --namespace monitoring
root@k8s-master1:~/prometheus-adapter# kubectl get pods -n monitoring -o wide| grep prometheus-adapter
prometheus-adapter-8956df68c-tchsd 1/1 Running 0 54s 10.244.36.104 k8s-node1 <none> <none>
## 检查自定义指标 API 是否可用,返回 JSON 数据,说明 Adapter 已成功注册
root@k8s-master1:~/prometheus-adapter# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

yaml
## 查询具体的自定义指标
root@k8s-master1:~/prometheus-adapter# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/nginx_http_requests_per_second" | jq .

3.3 QPS流量维度扩缩容
这里我已经让Prometheus监控上了nginx
不知道怎么操作的同学可以看我这篇
Prometheus Operator监控K8S Nginx:https://blog.csdn.net/m0_63756214/article/details/161881066

yaml
## 1. 初始的nginx pod有两个
root@k8s-master1:/k8s/nginx# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-server-7987759647-6t562 2/2 Running 0 18s 10.244.36.106 k8s-node1 <none> <none>
nginx-server-7987759647-jr522 2/2 Running 0 18s 10.244.169.176 k8s-node2 <none> <none>
yaml
## 2. 创建hpa清单文件
root@k8s-master1:/k8s/nginx# vim nginx-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: nginx-qps-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx-server # 改为你的 Deployment 名称
minReplicas: 1 # 最少1个pod
maxReplicas: 10 # 最多10个pod
metrics:
- type: Pods
pods:
metric:
name: nginx_http_requests_per_second # 使用查到的名称
target:
# 阈值类型:按每个Pod平均值计算
type: AverageValue
# 单个Pod平均每秒请求量达到10,触发扩容;低于阈值满足条件后缩容
averageValue: "10"
behavior:
scaleDown:
stabilizationWindowSeconds: 10
root@k8s-master1:/k8s/nginx# kubectl apply -f nginx-hpa.yaml
## 3. 当前的QPS(0.319)远低于目标值10,缩容到1个pod
root@k8s-master1:/k8s/nginx# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-qps-hpa Deployment/nginx-server 319m/10 1 10 2 31s
root@k8s-master1:/k8s/nginx# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-server-7987759647-6t562 2/2 Running 0 104s 10.244.36.106 k8s-node1 <none> <none>
选择一台集群机器,然后做压力测试,kubectl get hpa查看QPS,观察pod有没有扩容;停止压力测试,观察pod有没有缩容
yaml
## 4. 压力测试
root@k8s-master1:~# while true; do ab -k -c 50 -n 5000 http://10.101.126.166/; done
基于QPS的自定义指标伸缩演示视频
至此,Kubernetes 中 HPA 水平 Pod 弹性扩缩容的两类实现方案、完整链路与实操细节已全部讲解完毕!
注:
文中若有疏漏,欢迎大家指正赐教。
本文为100%原创,转载请务必标注原创作者,尊重劳动成果。
求赞、求关注、求评论!你的支持是我更新的最大动力,评论区等你~