Kubernetes 调度器完全指南:从原理到生产实战

Kubernetes 调度器完全指南:从原理到生产实战

本文深入剖析 Kubernetes 调度器(Scheduler)的核心技术,涵盖调度器架构设计、调度流程、调度策略、节点亲和性、污点容忍、拓扑分布,以及生产环境最佳实践。掌握调度器是优化集群资源利用率和应用高可用的关键。


目录

  1. 调度器概述
  2. 调度流程详解
  3. 节点选择策略
  4. 亲和性与反亲和性
  5. 污点与容忍
  6. 拓扑分布约束
  7. 实战:生产级调度配置
  8. 常用命令速查
  9. 最佳实践与避坑指南

一、调度器概述

1.1 什么是调度器?

调度器(Scheduler)是 Kubernetes 控制平面的核心组件,负责将 Pod 分配到合适的节点上运行。
#mermaid-svg-2dFNcAwg3SJQOnza{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2dFNcAwg3SJQOnza .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2dFNcAwg3SJQOnza .error-icon{fill:#552222;}#mermaid-svg-2dFNcAwg3SJQOnza .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2dFNcAwg3SJQOnza .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2dFNcAwg3SJQOnza .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza .marker.cross{stroke:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2dFNcAwg3SJQOnza p{margin:0;}#mermaid-svg-2dFNcAwg3SJQOnza .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster-label text{fill:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster-label span{color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster-label span p{background-color:transparent;}#mermaid-svg-2dFNcAwg3SJQOnza .label text,#mermaid-svg-2dFNcAwg3SJQOnza span{fill:#333;color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .node rect,#mermaid-svg-2dFNcAwg3SJQOnza .node circle,#mermaid-svg-2dFNcAwg3SJQOnza .node ellipse,#mermaid-svg-2dFNcAwg3SJQOnza .node polygon,#mermaid-svg-2dFNcAwg3SJQOnza .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .rough-node .label text,#mermaid-svg-2dFNcAwg3SJQOnza .node .label text,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape .label,#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape .label{text-anchor:middle;}#mermaid-svg-2dFNcAwg3SJQOnza .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .rough-node .label,#mermaid-svg-2dFNcAwg3SJQOnza .node .label,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape .label,#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape .label{text-align:center;}#mermaid-svg-2dFNcAwg3SJQOnza .node.clickable{cursor:pointer;}#mermaid-svg-2dFNcAwg3SJQOnza .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza .arrowheadPath{fill:#333333;}#mermaid-svg-2dFNcAwg3SJQOnza .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2dFNcAwg3SJQOnza .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2dFNcAwg3SJQOnza .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2dFNcAwg3SJQOnza .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2dFNcAwg3SJQOnza .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2dFNcAwg3SJQOnza .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2dFNcAwg3SJQOnza .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster text{fill:#333;}#mermaid-svg-2dFNcAwg3SJQOnza .cluster span{color:#333;}#mermaid-svg-2dFNcAwg3SJQOnza div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2dFNcAwg3SJQOnza .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2dFNcAwg3SJQOnza rect.text{fill:none;stroke-width:0;}#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape p,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2dFNcAwg3SJQOnza .icon-shape .label rect,#mermaid-svg-2dFNcAwg3SJQOnza .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2dFNcAwg3SJQOnza .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2dFNcAwg3SJQOnza .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2dFNcAwg3SJQOnza :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Pod 创建
调度器
过滤节点
打分排序
选择最优节点
绑定 Pod

1.2 调度器职责

职责 说明
资源匹配 确保节点有足够资源
策略应用 应用亲和性、污点等策略
负载均衡 尽量均匀分布 Pod
高可用 避免单点故障

1.3 调度器类型

类型 说明 适用场景
默认调度器 kube-scheduler 大多数场景
自定义调度器 自定义逻辑 特殊需求
多调度器 多个调度器共存 混合场景
yaml 复制代码
# 使用自定义调度器
apiVersion: v1
kind: Pod
metadata:
  name: custom-scheduler-pod
spec:
  schedulerName: my-scheduler  # 指定调度器
  containers:
  - name: app
    image: nginx

二、调度流程详解

2.1 两阶段调度

#mermaid-svg-Vn2L0wemE1PgICLr{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Vn2L0wemE1PgICLr .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Vn2L0wemE1PgICLr .error-icon{fill:#552222;}#mermaid-svg-Vn2L0wemE1PgICLr .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Vn2L0wemE1PgICLr .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Vn2L0wemE1PgICLr .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr .marker.cross{stroke:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Vn2L0wemE1PgICLr p{margin:0;}#mermaid-svg-Vn2L0wemE1PgICLr .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster-label text{fill:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster-label span{color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster-label span p{background-color:transparent;}#mermaid-svg-Vn2L0wemE1PgICLr .label text,#mermaid-svg-Vn2L0wemE1PgICLr span{fill:#333;color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .node rect,#mermaid-svg-Vn2L0wemE1PgICLr .node circle,#mermaid-svg-Vn2L0wemE1PgICLr .node ellipse,#mermaid-svg-Vn2L0wemE1PgICLr .node polygon,#mermaid-svg-Vn2L0wemE1PgICLr .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .rough-node .label text,#mermaid-svg-Vn2L0wemE1PgICLr .node .label text,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape .label,#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape .label{text-anchor:middle;}#mermaid-svg-Vn2L0wemE1PgICLr .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .rough-node .label,#mermaid-svg-Vn2L0wemE1PgICLr .node .label,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape .label,#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape .label{text-align:center;}#mermaid-svg-Vn2L0wemE1PgICLr .node.clickable{cursor:pointer;}#mermaid-svg-Vn2L0wemE1PgICLr .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr .arrowheadPath{fill:#333333;}#mermaid-svg-Vn2L0wemE1PgICLr .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Vn2L0wemE1PgICLr .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Vn2L0wemE1PgICLr .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Vn2L0wemE1PgICLr .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Vn2L0wemE1PgICLr .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Vn2L0wemE1PgICLr .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Vn2L0wemE1PgICLr .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster text{fill:#333;}#mermaid-svg-Vn2L0wemE1PgICLr .cluster span{color:#333;}#mermaid-svg-Vn2L0wemE1PgICLr div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Vn2L0wemE1PgICLr .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Vn2L0wemE1PgICLr rect.text{fill:none;stroke-width:0;}#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape p,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Vn2L0wemE1PgICLr .icon-shape .label rect,#mermaid-svg-Vn2L0wemE1PgICLr .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Vn2L0wemE1PgICLr .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Vn2L0wemE1PgICLr .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Vn2L0wemE1PgICLr :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 调度开始
预选阶段 Filtering
优选阶段 Scoring
选定节点 Binding
资源检查
策略检查
拓扑检查
打分排序
选择最优

2.2 预选阶段(Filtering)

检查节点是否满足 Pod 的基本要求:

检查项 说明
资源检查 CPU、内存是否充足
端口检查 hostPort 是否冲突
节点选择器 nodeSelector 是否匹配
污点检查 是否有未容忍的污点
亲和性检查 nodeAffinity 是否满足

2.3 优选阶段(Scoring)

对通过预选的节点进行打分:

评分项 说明 权重
资源均衡 资源使用越均衡分越高
亲和性 满足亲和性加分
拓扑分布 满足拓扑分布加分
镜像本地性 节点有镜像加分

2.4 调度结果

bash 复制代码
# 查看调度结果
kubectl get pod <pod-name> -o wide
# NODE 列显示调度到的节点

# 查看调度事件
kubectl describe pod <pod-name> | grep -A10 Events

三、节点选择策略

3.1 nodeSelector(简单选择)

yaml 复制代码
apiVersion: v1
kind: Pod
metadata:
  name: node-selector-pod
spec:
  nodeSelector:           # 简单标签选择
    disktype: ssd
    environment: production
  containers:
  - name: app
    image: nginx

给节点打标签

bash 复制代码
# 添加标签
kubectl label nodes <node-name> disktype=ssd

# 查看节点标签
kubectl get nodes --show-labels

# 删除标签
kubectl label nodes <node-name> disktype-

3.2 nodeName(直接指定)

yaml 复制代码
apiVersion: v1
kind: Pod
metadata:
  name: node-name-pod
spec:
  nodeName: node-1        # 直接指定节点
  containers:
  - name: app
    image: nginx

注意

  • 绕过调度器
  • 节点不存在则 Pod 失败
  • 适合调试场景

3.3 nodeAffinity(高级选择)

yaml 复制代码
apiVersion: v1
kind: Pod
metadata:
  name: node-affinity-pod
spec:
  affinity:
    nodeAffinity:
      # 硬性要求(必须满足)
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values: [ssd, nvme]
          - key: zone
            operator: NotIn
            values: [zone-a]
      
      # 软性偏好(尽量满足)
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80          # 权重 1-100
        preference:
          matchExpressions:
          - key: performance
            operator: In
            values: [high]
      - weight: 20
        preference:
          matchExpressions:
          - key: region
            operator: In
            values: [east]
  containers:
  - name: app
    image: nginx

操作符

操作符 说明
In 在列表中
NotIn 不在列表中
Exists 键存在
DoesNotExist 键不存在
Gt 大于
Lt 小于

四、亲和性与反亲和性

4.1 Pod 亲和性(Pod Affinity)

将 Pod 调度到与其他 Pod 相同的节点/区域。

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAffinity:
          # 硬性要求:必须与 cache Pod 在同一节点
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: cache
            topologyKey: kubernetes.io/hostname
          
          # 软性偏好:尽量与 db Pod 在同一区域
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: db
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: web
        image: nginx

4.2 Pod 反亲和性(Pod Anti-Affinity)

将 Pod 分散到不同节点/区域。

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          # 硬性要求:必须分散到不同节点
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: web
            topologyKey: kubernetes.io/hostname
          
          # 软性偏好:尽量分散到不同区域
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: web
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: web
        image: nginx

4.3 topologyKey 说明

topologyKey 说明
kubernetes.io/hostname 节点级别
topology.kubernetes.io/zone 可用区级别
topology.kubernetes.io/region 区域级别

五、污点与容忍

5.1 污点(Taints)

污点用于排斥 Pod 调度到特定节点。

bash 复制代码
# 添加污点
kubectl taint nodes <node-name> key=value:effect

# 示例
kubectl taint nodes node-1 dedicated=production:NoSchedule
kubectl taint nodes node-2 gpu=true:NoExecute
kubectl taint nodes node-3 maintenance=true:PreferNoSchedule

# 删除污点
kubectl taint nodes <node-name> key=value:effect-
kubectl taint nodes node-1 dedicated:NoSchedule-

5.2 污点效果(Effect)

效果 说明
NoSchedule 不调度新 Pod
PreferNoSchedule 尽量不调度
NoExecute 不调度且驱逐现有 Pod

5.3 容忍(Tolerations)

Pod 通过容忍来"接受"节点的污点。

yaml 复制代码
apiVersion: v1
kind: Pod
metadata:
  name: toleration-pod
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "production"
    effect: "NoSchedule"
  - key: "gpu"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 3600    # 容忍 3600 秒后驱逐
  containers:
  - name: app
    image: nginx

5.4 典型应用场景

场景 配置
专用节点 污点 + 容忍
GPU 节点 污点 + 容忍
维护节点 NoExecute + tolerationSeconds
隔离节点 NoSchedule

六、拓扑分布约束

6.1 TopologySpreadConstraints

更灵活地控制 Pod 在拓扑域中的分布。

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
      # 按可用区分布
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web
      
      # 按节点分布
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: web
      
      containers:
      - name: web
        image: nginx

6.2 关键参数

参数 说明
maxSkew 最大不均衡度
topologyKey 拓扑域键
whenUnsatisfiable 不满足时的策略
labelSelector Pod 选择器

6.3 whenUnsatisfiable 策略

策略 说明
DoNotSchedule 不调度(硬性)
ScheduleAnyway 仍然调度(软性)

七、实战:生产级调度配置

7.1 完整示例:高可用 Web 应用

yaml 复制代码
apiVersion: apps/v1
kind: Deployment
metadata:
  name: production-web
  namespace: production
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      # 节点亲和性:优先 SSD 节点
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: disktype
                operator: In
                values: [ssd]
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: performance
                operator: In
                values: [high]
        
        # Pod 反亲和性:分散到不同节点
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: web
            topologyKey: kubernetes.io/hostname
      
      # 拓扑分布:均匀分布到可用区
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web
      
      # 容忍:允许调度到有特定污点的节点
      tolerations:
      - key: "production"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      
      containers:
      - name: web
        image: nginx:1.25.3
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

7.2 GPU 工作负载调度

yaml 复制代码
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  # 节点选择:GPU 节点
  nodeSelector:
    gpu: "true"
  
  # 容忍 GPU 污点
  tolerations:
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  
  containers:
  - name: cuda-app
    image: cuda-app:1.0
    resources:
      limits:
        nvidia.com/gpu: 1     # 请求 GPU 资源

7.3 有状态应用调度

yaml 复制代码
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      # 反亲和性:每个节点一个实例
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: mysql
            topologyKey: kubernetes.io/hostname
      
      # 节点选择:SSD 节点
      nodeSelector:
        disktype: ssd
      
      containers:
      - name: mysql
        image: mysql:8.0
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 100Gi

八、常用命令速查

8.1 节点标签操作

bash 复制代码
# 查看节点标签
kubectl get nodes --show-labels

# 添加标签
kubectl label nodes <node-name> disktype=ssd

# 删除标签
kubectl label nodes <node-name> disktype-

# 查看特定节点详情
kubectl describe node <node-name>

8.2 污点操作

bash 复制代码
# 查看污点
kubectl describe node <node-name> | grep Taints

# 添加污点
kubectl taint nodes <node-name> key=value:NoSchedule

# 删除污点
kubectl taint nodes <node-name> key:NoSchedule-

# 删除所有污点
kubectl taint nodes <node-name> --all-

8.3 调度调试

bash 复制代码
# 查看 Pod 调度结果
kubectl get pod <pod-name> -o wide

# 查看调度事件
kubectl describe pod <pod-name> | grep -A10 Events

# 查看未调度的 Pod
kubectl get pods --field-selector=status.phase=Pending

# 查看节点资源使用
kubectl top nodes

# 查看调度器日志
kubectl logs -n kube-system kube-scheduler-<node-name>

8.4 模拟调度

bash 复制代码
# 使用 --dry-run 测试
kubectl apply -f pod.yaml --dry-run=client

# 查看调度配置
kubectl get configmap kube-scheduler -n kube-system -o yaml

九、最佳实践与避坑指南

9.1 必须遵守的原则

原则 说明 示例
使用反亲和性 提高可用性 分散到不同节点
配置拓扑分布 跨可用区分布 topologySpreadConstraints
合理使用污点 专用节点隔离 GPU 节点
设置资源请求 确保调度成功 requests
避免 nodeName 绕过调度器 调试除外

9.2 常见问题与解决

问题 原因 解决方案
Pod 一直 Pending 资源不足 增加节点或减少请求
调度不均衡 缺少拓扑约束 添加 topologySpreadConstraints
Pod 被驱逐 污点不匹配 添加容忍
亲和性冲突 条件矛盾 检查 affinity 配置
调度器性能 大量 Pod 调度 优化调度策略

9.3 调度策略优化

yaml 复制代码
# 1. 使用软性约束(避免调度失败)
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference:
        matchExpressions:
        - key: performance
          operator: In
          values: [high]

# 2. 合理设置资源请求(避免过度分配)
resources:
  requests:
    cpu: "100m"    # 不要设置过高
    memory: "128Mi"

# 3. 使用 PodDisruptionBudget(保证可用性)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web

9.4 多租户调度

yaml 复制代码
# 为不同团队配置不同的调度策略
# 团队 A:高性能节点
apiVersion: v1
kind: Namespace
metadata:
  name: team-a
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "100"
    requests.memory: "200Gi"

# 使用节点池隔离
kubectl label nodes node-1 node-pool=team-a
kubectl label nodes node-2 node-pool=team-b

十、总结

10.1 核心要点回顾

  1. 调度器两阶段:预选过滤 + 优选打分
  2. 节点选择:nodeSelector 简单,nodeAffinity 灵活
  3. Pod 亲和性:控制 Pod 间的关系
  4. 污点容忍:节点专用化
  5. 拓扑分布:跨故障域分布

10.2 调度策略选择建议

场景 推荐策略
高可用 反亲和性 + 拓扑分布
性能优化 节点亲和性 + 专用节点
成本控制 软性约束 + 合理请求
多租户 节点池 + 污点隔离

10.3 学习路线

复制代码
调度器基础 → 亲和性 → 污点容忍 → 拓扑分布
    ↓
自定义调度器 → 调度框架 → 性能优化

10.4 下一步学习

  1. 自定义调度器:实现特定调度逻辑
  2. 调度框架:扩展调度器插件
  3. Descheduler:重新平衡 Pod
  4. Cluster Autoscaler:自动扩缩节点
  5. Karpenter:新一代节点自动伸缩

附录:完整示例汇总

生产级调度完整配置

yaml 复制代码
# 1. 高可用 Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ha-web-app
  namespace: production
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      # 节点亲和性
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values: [linux]
        
        # Pod 反亲和性
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: web
            topologyKey: kubernetes.io/hostname
      
      # 拓扑分布
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web
      
      # 容忍
      tolerations:
      - key: "production"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      
      containers:
      - name: web
        image: nginx:1.25.3
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

验证测试

bash 复制代码
# 1. 创建 Deployment
kubectl apply -f scheduling-demo.yaml

# 2. 查看 Pod 分布
kubectl get pods -n production -o wide

# 3. 验证反亲和性(每个节点一个 Pod)
kubectl get pods -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}'

# 4. 验证拓扑分布
kubectl get nodes -L topology.kubernetes.io/zone

# 5. 测试调度失败场景
# 创建超过节点数的副本
kubectl scale deployment ha-web-app --replicas=20
# 观察 Pending Pod
kubectl get pods -n production --field-selector=status.phase=Pending

相关推荐
styshoo1 小时前
[NVSentinel] gpu-health-monitor模块调研
人工智能·云原生·nvsentinel
流烟默4 小时前
Kubernetes Service 完全指南:从原理到生产实战
云原生·容器·kubernetes
airobotcn4 小时前
智能巡检平台容器化部署:Docker+K8s在工业边缘的实践
人工智能·docker·容器·kubernetes·机器人·自动化
重庆小透明6 小时前
深入探寻微服务【第四篇微服务监控实战】
微服务·云原生·架构
流烟默8 小时前
Kubernetes 探针完全指南:健康检查机制深度解析
云原生·容器·kubernetes
Kurisu_红莉栖8 小时前
关于docker的使用心得
运维·docker·容器
GGG76610 小时前
k8s-Pod Scheduler
云原生·容器·kubernetes
GGG76611 小时前
k8s-动态卷供应
云原生·容器·kubernetes
yunwei3711 小时前
eBPF 教程:使用 fsession 追踪慢速 vfs_read 调用
linux·云原生