Kubernetes 调度器完全指南:从原理到生产实战
本文深入剖析 Kubernetes 调度器(Scheduler)的核心技术,涵盖调度器架构设计、调度流程、调度策略、节点亲和性、污点容忍、拓扑分布,以及生产环境最佳实践。掌握调度器是优化集群资源利用率和应用高可用的关键。
目录
一、调度器概述
1.1 什么是调度器?
调度器(Scheduler)是 Kubernetes 控制平面的核心组件,负责将 Pod 分配到合适的节点上运行。
#mermaid-svg-2dFNcAwg3SJQOnza{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2dFNcAwg3SJQOnza .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2dFNcAwg3SJQOnza .error-icon{fill:#552222;}#mermaid-svg-2dFNcAwg3SJQOnza .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2dFNcAwg3SJQOnza .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza .marker.cross{stroke:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2dFNcAwg3SJQOnza p{margin:0;}#mermaid-svg-2dFNcAwg3SJQOnza .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster-label text{fill:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster-label span{color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster-label span p{background-color:transparent;}#mermaid-svg-2dFNcAwg3SJQOnza .label text,#mermaid-svg-2dFNcAwg3SJQOnza span{fill:#333;color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .node rect,#mermaid-svg-2dFNcAwg3SJQOnza .node circle,#mermaid-svg-2dFNcAwg3SJQOnza .node ellipse,#mermaid-svg-2dFNcAwg3SJQOnza .node polygon,#mermaid-svg-2dFNcAwg3SJQOnza .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .rough-node .label text,#mermaid-svg-2dFNcAwg3SJQOnza .node .label text,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape .label,#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape .label{text-anchor:middle;}#mermaid-svg-2dFNcAwg3SJQOnza .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .rough-node .label,#mermaid-svg-2dFNcAwg3SJQOnza .node .label,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape .label,#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape .label{text-align:center;}#mermaid-svg-2dFNcAwg3SJQOnza .node.clickable{cursor:pointer;}#mermaid-svg-2dFNcAwg3SJQOnza .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza .arrowheadPath{fill:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2dFNcAwg3SJQOnza .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2dFNcAwg3SJQOnza .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2dFNcAwg3SJQOnza .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2dFNcAwg3SJQOnza .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2dFNcAwg3SJQOnza .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2dFNcAwg3SJQOnza .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster text{fill:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster span{color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2dFNcAwg3SJQOnza .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2dFNcAwg3SJQOnza rect.text{fill:none;stroke-width:0;}#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape p,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape .label rect,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2dFNcAwg3SJQOnza .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2dFNcAwg3SJQOnza .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2dFNcAwg3SJQOnza :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Pod 创建
调度器
过滤节点
打分排序
选择最优节点
绑定 Pod
1.2 调度器职责
| 职责 | 说明 |
|---|---|
| 资源匹配 | 确保节点有足够资源 |
| 策略应用 | 应用亲和性、污点等策略 |
| 负载均衡 | 尽量均匀分布 Pod |
| 高可用 | 避免单点故障 |
1.3 调度器类型
| 类型 | 说明 | 适用场景 |
|---|---|---|
| 默认调度器 | kube-scheduler | 大多数场景 |
| 自定义调度器 | 自定义逻辑 | 特殊需求 |
| 多调度器 | 多个调度器共存 | 混合场景 |
yaml
# 使用自定义调度器
apiVersion: v1
kind: Pod
metadata:
name: custom-scheduler-pod
spec:
schedulerName: my-scheduler # 指定调度器
containers:
- name: app
image: nginx
二、调度流程详解
2.1 两阶段调度
#mermaid-svg-Vn2L0wemE1PgICLr{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Vn2L0wemE1PgICLr .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Vn2L0wemE1PgICLr .error-icon{fill:#552222;}#mermaid-svg-Vn2L0wemE1PgICLr .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Vn2L0wemE1PgICLr .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr .marker.cross{stroke:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Vn2L0wemE1PgICLr p{margin:0;}#mermaid-svg-Vn2L0wemE1PgICLr .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster-label text{fill:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster-label span{color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster-label span p{background-color:transparent;}#mermaid-svg-Vn2L0wemE1PgICLr .label text,#mermaid-svg-Vn2L0wemE1PgICLr span{fill:#333;color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .node rect,#mermaid-svg-Vn2L0wemE1PgICLr .node circle,#mermaid-svg-Vn2L0wemE1PgICLr .node ellipse,#mermaid-svg-Vn2L0wemE1PgICLr .node polygon,#mermaid-svg-Vn2L0wemE1PgICLr .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .rough-node .label text,#mermaid-svg-Vn2L0wemE1PgICLr .node .label text,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape .label,#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape .label{text-anchor:middle;}#mermaid-svg-Vn2L0wemE1PgICLr .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .rough-node .label,#mermaid-svg-Vn2L0wemE1PgICLr .node .label,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape .label,#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape .label{text-align:center;}#mermaid-svg-Vn2L0wemE1PgICLr .node.clickable{cursor:pointer;}#mermaid-svg-Vn2L0wemE1PgICLr .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr .arrowheadPath{fill:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Vn2L0wemE1PgICLr .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Vn2L0wemE1PgICLr .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Vn2L0wemE1PgICLr .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Vn2L0wemE1PgICLr .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Vn2L0wemE1PgICLr .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Vn2L0wemE1PgICLr .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster text{fill:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster span{color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Vn2L0wemE1PgICLr .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Vn2L0wemE1PgICLr rect.text{fill:none;stroke-width:0;}#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape p,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape .label rect,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Vn2L0wemE1PgICLr .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Vn2L0wemE1PgICLr .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Vn2L0wemE1PgICLr :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 调度开始
预选阶段 Filtering
优选阶段 Scoring
选定节点 Binding
资源检查
策略检查
拓扑检查
打分排序
选择最优
2.2 预选阶段(Filtering)
检查节点是否满足 Pod 的基本要求:
| 检查项 | 说明 |
|---|---|
| 资源检查 | CPU、内存是否充足 |
| 端口检查 | hostPort 是否冲突 |
| 节点选择器 | nodeSelector 是否匹配 |
| 污点检查 | 是否有未容忍的污点 |
| 亲和性检查 | nodeAffinity 是否满足 |
2.3 优选阶段(Scoring)
对通过预选的节点进行打分:
| 评分项 | 说明 | 权重 |
|---|---|---|
| 资源均衡 | 资源使用越均衡分越高 | 高 |
| 亲和性 | 满足亲和性加分 | 中 |
| 拓扑分布 | 满足拓扑分布加分 | 中 |
| 镜像本地性 | 节点有镜像加分 | 低 |
2.4 调度结果
bash
# 查看调度结果
kubectl get pod <pod-name> -o wide
# NODE 列显示调度到的节点
# 查看调度事件
kubectl describe pod <pod-name> | grep -A10 Events
三、节点选择策略
3.1 nodeSelector(简单选择)
yaml
apiVersion: v1
kind: Pod
metadata:
name: node-selector-pod
spec:
nodeSelector: # 简单标签选择
disktype: ssd
environment: production
containers:
- name: app
image: nginx
给节点打标签:
bash
# 添加标签
kubectl label nodes <node-name> disktype=ssd
# 查看节点标签
kubectl get nodes --show-labels
# 删除标签
kubectl label nodes <node-name> disktype-
3.2 nodeName(直接指定)
yaml
apiVersion: v1
kind: Pod
metadata:
name: node-name-pod
spec:
nodeName: node-1 # 直接指定节点
containers:
- name: app
image: nginx
注意:
- 绕过调度器
- 节点不存在则 Pod 失败
- 适合调试场景
3.3 nodeAffinity(高级选择)
yaml
apiVersion: v1
kind: Pod
metadata:
name: node-affinity-pod
spec:
affinity:
nodeAffinity:
# 硬性要求(必须满足)
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: disktype
operator: In
values: [ssd, nvme]
- key: zone
operator: NotIn
values: [zone-a]
# 软性偏好(尽量满足)
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80 # 权重 1-100
preference:
matchExpressions:
- key: performance
operator: In
values: [high]
- weight: 20
preference:
matchExpressions:
- key: region
operator: In
values: [east]
containers:
- name: app
image: nginx
操作符:
| 操作符 | 说明 |
|---|---|
In |
在列表中 |
NotIn |
不在列表中 |
Exists |
键存在 |
DoesNotExist |
键不存在 |
Gt |
大于 |
Lt |
小于 |
四、亲和性与反亲和性
4.1 Pod 亲和性(Pod Affinity)
将 Pod 调度到与其他 Pod 相同的节点/区域。
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
affinity:
podAffinity:
# 硬性要求:必须与 cache Pod 在同一节点
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: cache
topologyKey: kubernetes.io/hostname
# 软性偏好:尽量与 db Pod 在同一区域
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: db
topologyKey: topology.kubernetes.io/zone
containers:
- name: web
image: nginx
4.2 Pod 反亲和性(Pod Anti-Affinity)
将 Pod 分散到不同节点/区域。
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
affinity:
podAntiAffinity:
# 硬性要求:必须分散到不同节点
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: web
topologyKey: kubernetes.io/hostname
# 软性偏好:尽量分散到不同区域
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: web
topologyKey: topology.kubernetes.io/zone
containers:
- name: web
image: nginx
4.3 topologyKey 说明
| topologyKey | 说明 |
|---|---|
kubernetes.io/hostname |
节点级别 |
topology.kubernetes.io/zone |
可用区级别 |
topology.kubernetes.io/region |
区域级别 |
五、污点与容忍
5.1 污点(Taints)
污点用于排斥 Pod 调度到特定节点。
bash
# 添加污点
kubectl taint nodes <node-name> key=value:effect
# 示例
kubectl taint nodes node-1 dedicated=production:NoSchedule
kubectl taint nodes node-2 gpu=true:NoExecute
kubectl taint nodes node-3 maintenance=true:PreferNoSchedule
# 删除污点
kubectl taint nodes <node-name> key=value:effect-
kubectl taint nodes node-1 dedicated:NoSchedule-
5.2 污点效果(Effect)
| 效果 | 说明 |
|---|---|
NoSchedule |
不调度新 Pod |
PreferNoSchedule |
尽量不调度 |
NoExecute |
不调度且驱逐现有 Pod |
5.3 容忍(Tolerations)
Pod 通过容忍来"接受"节点的污点。
yaml
apiVersion: v1
kind: Pod
metadata:
name: toleration-pod
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "production"
effect: "NoSchedule"
- key: "gpu"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 3600 # 容忍 3600 秒后驱逐
containers:
- name: app
image: nginx
5.4 典型应用场景
| 场景 | 配置 |
|---|---|
| 专用节点 | 污点 + 容忍 |
| GPU 节点 | 污点 + 容忍 |
| 维护节点 | NoExecute + tolerationSeconds |
| 隔离节点 | NoSchedule |
六、拓扑分布约束
6.1 TopologySpreadConstraints
更灵活地控制 Pod 在拓扑域中的分布。
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
spec:
replicas: 6
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
topologySpreadConstraints:
# 按可用区分布
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: web
# 按节点分布
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: web
containers:
- name: web
image: nginx
6.2 关键参数
| 参数 | 说明 |
|---|---|
maxSkew |
最大不均衡度 |
topologyKey |
拓扑域键 |
whenUnsatisfiable |
不满足时的策略 |
labelSelector |
Pod 选择器 |
6.3 whenUnsatisfiable 策略
| 策略 | 说明 |
|---|---|
DoNotSchedule |
不调度(硬性) |
ScheduleAnyway |
仍然调度(软性) |
七、实战:生产级调度配置
7.1 完整示例:高可用 Web 应用
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: production-web
namespace: production
spec:
replicas: 6
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
# 节点亲和性:优先 SSD 节点
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: disktype
operator: In
values: [ssd]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: performance
operator: In
values: [high]
# Pod 反亲和性:分散到不同节点
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: web
topologyKey: kubernetes.io/hostname
# 拓扑分布:均匀分布到可用区
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: web
# 容忍:允许调度到有特定污点的节点
tolerations:
- key: "production"
operator: "Equal"
value: "true"
effect: "NoSchedule"
containers:
- name: web
image: nginx:1.25.3
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
7.2 GPU 工作负载调度
yaml
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
# 节点选择:GPU 节点
nodeSelector:
gpu: "true"
# 容忍 GPU 污点
tolerations:
- key: "gpu"
operator: "Equal"
value: "true"
effect: "NoSchedule"
containers:
- name: cuda-app
image: cuda-app:1.0
resources:
limits:
nvidia.com/gpu: 1 # 请求 GPU 资源
7.3 有状态应用调度
yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
serviceName: mysql
replicas: 3
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
# 反亲和性:每个节点一个实例
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: mysql
topologyKey: kubernetes.io/hostname
# 节点选择:SSD 节点
nodeSelector:
disktype: ssd
containers:
- name: mysql
image: mysql:8.0
volumeMounts:
- name: data
mountPath: /var/lib/mysql
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 100Gi
八、常用命令速查
8.1 节点标签操作
bash
# 查看节点标签
kubectl get nodes --show-labels
# 添加标签
kubectl label nodes <node-name> disktype=ssd
# 删除标签
kubectl label nodes <node-name> disktype-
# 查看特定节点详情
kubectl describe node <node-name>
8.2 污点操作
bash
# 查看污点
kubectl describe node <node-name> | grep Taints
# 添加污点
kubectl taint nodes <node-name> key=value:NoSchedule
# 删除污点
kubectl taint nodes <node-name> key:NoSchedule-
# 删除所有污点
kubectl taint nodes <node-name> --all-
8.3 调度调试
bash
# 查看 Pod 调度结果
kubectl get pod <pod-name> -o wide
# 查看调度事件
kubectl describe pod <pod-name> | grep -A10 Events
# 查看未调度的 Pod
kubectl get pods --field-selector=status.phase=Pending
# 查看节点资源使用
kubectl top nodes
# 查看调度器日志
kubectl logs -n kube-system kube-scheduler-<node-name>
8.4 模拟调度
bash
# 使用 --dry-run 测试
kubectl apply -f pod.yaml --dry-run=client
# 查看调度配置
kubectl get configmap kube-scheduler -n kube-system -o yaml
九、最佳实践与避坑指南
9.1 必须遵守的原则
| 原则 | 说明 | 示例 |
|---|---|---|
| 使用反亲和性 | 提高可用性 | 分散到不同节点 |
| 配置拓扑分布 | 跨可用区分布 | topologySpreadConstraints |
| 合理使用污点 | 专用节点隔离 | GPU 节点 |
| 设置资源请求 | 确保调度成功 | requests |
| 避免 nodeName | 绕过调度器 | 调试除外 |
9.2 常见问题与解决
| 问题 | 原因 | 解决方案 |
|---|---|---|
| Pod 一直 Pending | 资源不足 | 增加节点或减少请求 |
| 调度不均衡 | 缺少拓扑约束 | 添加 topologySpreadConstraints |
| Pod 被驱逐 | 污点不匹配 | 添加容忍 |
| 亲和性冲突 | 条件矛盾 | 检查 affinity 配置 |
| 调度器性能 | 大量 Pod 调度 | 优化调度策略 |
9.3 调度策略优化
yaml
# 1. 使用软性约束(避免调度失败)
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: performance
operator: In
values: [high]
# 2. 合理设置资源请求(避免过度分配)
resources:
requests:
cpu: "100m" # 不要设置过高
memory: "128Mi"
# 3. 使用 PodDisruptionBudget(保证可用性)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: web
9.4 多租户调度
yaml
# 为不同团队配置不同的调度策略
# 团队 A:高性能节点
apiVersion: v1
kind: Namespace
metadata:
name: team-a
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-a-quota
namespace: team-a
spec:
hard:
requests.cpu: "100"
requests.memory: "200Gi"
# 使用节点池隔离
kubectl label nodes node-1 node-pool=team-a
kubectl label nodes node-2 node-pool=team-b
十、总结
10.1 核心要点回顾
- 调度器两阶段:预选过滤 + 优选打分
- 节点选择:nodeSelector 简单,nodeAffinity 灵活
- Pod 亲和性:控制 Pod 间的关系
- 污点容忍:节点专用化
- 拓扑分布:跨故障域分布
10.2 调度策略选择建议
| 场景 | 推荐策略 |
|---|---|
| 高可用 | 反亲和性 + 拓扑分布 |
| 性能优化 | 节点亲和性 + 专用节点 |
| 成本控制 | 软性约束 + 合理请求 |
| 多租户 | 节点池 + 污点隔离 |
10.3 学习路线
调度器基础 → 亲和性 → 污点容忍 → 拓扑分布
↓
自定义调度器 → 调度框架 → 性能优化
10.4 下一步学习
- 自定义调度器:实现特定调度逻辑
- 调度框架:扩展调度器插件
- Descheduler:重新平衡 Pod
- Cluster Autoscaler:自动扩缩节点
- Karpenter:新一代节点自动伸缩
附录:完整示例汇总
生产级调度完整配置
yaml
# 1. 高可用 Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: ha-web-app
namespace: production
spec:
replicas: 6
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
# 节点亲和性
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values: [linux]
# Pod 反亲和性
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: web
topologyKey: kubernetes.io/hostname
# 拓扑分布
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: web
# 容忍
tolerations:
- key: "production"
operator: "Equal"
value: "true"
effect: "NoSchedule"
containers:
- name: web
image: nginx:1.25.3
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
验证测试
bash
# 1. 创建 Deployment
kubectl apply -f scheduling-demo.yaml
# 2. 查看 Pod 分布
kubectl get pods -n production -o wide
# 3. 验证反亲和性(每个节点一个 Pod)
kubectl get pods -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}'
# 4. 验证拓扑分布
kubectl get nodes -L topology.kubernetes.io/zone
# 5. 测试调度失败场景
# 创建超过节点数的副本
kubectl scale deployment ha-web-app --replicas=20
# 观察 Pending Pod
kubectl get pods -n production --field-selector=status.phase=Pending