第4期：YARN资源调度架构 - 多租户计算的资源分配数学原理
导言：任何不理解YARN资源调度数学原理的架构师都无法设计出公平高效的多租户平台。本期我们将深入资源调度的理论基础，从公平性度量（Max-Min Fairness）出发，阐明Capacity Scheduler和Fair Scheduler的设计原理；解析YARN的容器抽象与资源隔离机制；以及为什么在云原生时代Kubernetes正在成为YARN的替代者。
4.1 资源调度的理论基础
4.1.1 Max-Min Fairness：公平性度量的数学定义
资源调度的核心问题是如何在多个竞争任务之间公平分配资源：
Max-Min Fairness的形式化定义：
给定资源总量 R 和 n 个请求队列 Q = {q₁, q₂, ..., qₙ}
定义：分配方案 A = {a₁, a₂, ..., aₙ} 是Max-Min公平的，当且仅当：
1. 约束条件：
∀i: 0 ≤ aᵢ ≤ qᵢ (不超过请求量)
Σᵢ aᵢ ≤ R (不超过资源总量)
2. 公平性条件：
按 aᵢ 升序排列得到 b₁ ≤ b₂ ≤ ... ≤ bₙ
则不存在可行分配方案 A' 使得：
- b'₁ ≥ b₁, b'₂ ≥ b₂, ..., b'ₖ ≥ bₖ 对某个 k 成立
- 且存在 j < k 使得 b'ⱼ > bⱼ
直观理解：
- 最小的请求优先得到满足
- 任何试图增加最小值而减少最大值的方案都不可行#mermaid-svg-S0WbXwgkN1NnoDUN{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-S0WbXwgkN1NnoDUN .error-icon{fill:#552222;}#mermaid-svg-S0WbXwgkN1NnoDUN .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-S0WbXwgkN1NnoDUN .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-S0WbXwgkN1NnoDUN .marker{fill:#333333;stroke:#333333;}#mermaid-svg-S0WbXwgkN1NnoDUN .marker.cross{stroke:#333333;}#mermaid-svg-S0WbXwgkN1NnoDUN svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-S0WbXwgkN1NnoDUN p{margin:0;}#mermaid-svg-S0WbXwgkN1NnoDUN .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-S0WbXwgkN1NnoDUN .cluster-label text{fill:#333;}#mermaid-svg-S0WbXwgkN1NnoDUN .cluster-label span{color:#333;}#mermaid-svg-S0WbXwgkN1NnoDUN .cluster-label span p{background-color:transparent;}#mermaid-svg-S0WbXwgkN1NnoDUN .label text,#mermaid-svg-S0WbXwgkN1NnoDUN span{fill:#333;color:#333;}#mermaid-svg-S0WbXwgkN1NnoDUN .node rect,#mermaid-svg-S0WbXwgkN1NnoDUN .node circle,#mermaid-svg-S0WbXwgkN1NnoDUN .node ellipse,#mermaid-svg-S0WbXwgkN1NnoDUN .node polygon,#mermaid-svg-S0WbXwgkN1NnoDUN .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-S0WbXwgkN1NnoDUN .rough-node .label text,#mermaid-svg-S0WbXwgkN1NnoDUN .node .label text,#mermaid-svg-S0WbXwgkN1NnoDUN .image-shape .label,#mermaid-svg-S0WbXwgkN1NnoDUN .icon-shape .label{text-anchor:middle;}#mermaid-svg-S0WbXwgkN1NnoDUN .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-S0WbXwgkN1NnoDUN .rough-node .label,#mermaid-svg-S0WbXwgkN1NnoDUN .node .label,#mermaid-svg-S0WbXwgkN1NnoDUN .image-shape .label,#mermaid-svg-S0WbXwgkN1NnoDUN .icon-shape .label{text-align:center;}#mermaid-svg-S0WbXwgkN1NnoDUN .node.clickable{cursor:pointer;}#mermaid-svg-S0WbXwgkN1NnoDUN .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-S0WbXwgkN1NnoDUN .arrowheadPath{fill:#333333;}#mermaid-svg-S0WbXwgkN1NnoDUN .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-S0WbXwgkN1NnoDUN .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-S0WbXwgkN1NnoDUN .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-S0WbXwgkN1NnoDUN .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-S0WbXwgkN1NnoDUN .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-S0WbXwgkN1NnoDUN .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-S0WbXwgkN1NnoDUN .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-S0WbXwgkN1NnoDUN .cluster text{fill:#333;}#mermaid-svg-S0WbXwgkN1NnoDUN .cluster span{color:#333;}#mermaid-svg-S0WbXwgkN1NnoDUN div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-S0WbXwgkN1NnoDUN .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-S0WbXwgkN1NnoDUN rect.text{fill:none;stroke-width:0;}#mermaid-svg-S0WbXwgkN1NnoDUN .icon-shape,#mermaid-svg-S0WbXwgkN1NnoDUN .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-S0WbXwgkN1NnoDUN .icon-shape p,#mermaid-svg-S0WbXwgkN1NnoDUN .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-S0WbXwgkN1NnoDUN .icon-shape .label rect,#mermaid-svg-S0WbXwgkN1NnoDUN .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-S0WbXwgkN1NnoDUN .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-S0WbXwgkN1NnoDUN .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-S0WbXwgkN1NnoDUN :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Max-Min Fairness分配示例
是
否
资源R = 100 units
队列请求
Q = {40, 30, 60, 50}
第1轮分配
每队列 min(100/4, 请求) = 25
分配后: {25, 25, 25, 25}
剩余: 0
但队列1只需40, 队列3请求60
迭代优化?
回收多余分配
最终分配: {25, 25, 25, 25}
4.1.2 公平调度器的数学实现
java
/**
* Max-Min Fairness的精确实现
*/
public class MaxMinFairness {
/**
* 计算Max-Min公平分配
*
* 算法：
* 1. 按请求量升序排序
* 2. 逐个分配，每轮计算"虚拟资源配额"
* 3. 达到请求上限的队列不再参与后续分配
*/
public static double[] computeFairAllocation(
double totalResources,
double[] requests
) {
int n = requests.length;
double[] allocation = new double[n];
boolean[] filled = new boolean[n];
// 初始化
for (int i = 0; i < n; i++) {
allocation[i] = 0;
filled[i] = false;
}
double remaining = totalResources;
int unfilledCount = n;
while (unfilledCount > 0 && remaining > 0) {
// 计算当前轮次的"虚拟配额"
double quota = remaining / unfilledCount;
// 计算实际分配
for (int i = 0; i < n; i++) {
if (filled[i]) continue;
double needed = requests[i] - allocation[i];
if (needed <= quota) {
// 请求已满足
allocation[i] = requests[i];
filled[i] = true;
remaining -= needed;
unfilledCount--;
} else {
// 只分配虚拟配额
allocation[i] += quota;
remaining = 0;
break;
}
}
}
return allocation;
}
/**
* 工业场景验证
*/
public static void main(String[] args) {
// 4个租户，资源总量100
double total = 100;
double[] requests = {40, 30, 60, 50};
double[] result = computeFairAllocation(total, requests);
System.out.println("Max-Min公平分配结果:");
for (int i = 0; i < result.length; i++) {
System.out.printf("租户%d: 请求=%d, 分配=%.1f, 满足率=%.1f%%\n",
i+1, (int)requests[i], result[i],
result[i]/requests[i]*100);
}
}
}4.2 YARN资源抽象与容器模型
4.2.1 资源抽象的数学表示
YARN使用向量化的资源表示：
YARN资源向量：
R = ⟨CPU_cores, Memory_MB, GPU_units, Disk_IO, Network_BW⟩
例如：
- Map任务容器: R = ⟨2, 4096, 0, 100, 50⟩
- Reduce任务容器: R = ⟨4, 8192, 0, 200, 100⟩
- ML训练容器: R = ⟨16, 32768, 4, 500, 200⟩
资源比较运算：
R₁ ≤ R₂ 当且仅当：
∀i: R₁[i] ≤ R₂[i]
资源相加运算：
R₁ + R₂ = ⟨R₁[0]+R₂[0], R₁[1]+R₂[1], ...⟩
节点可用资源：
R_available = R_total - R_used
java
/**
* YARN资源抽象的Java实现
*/
public class Resource implements Comparable<Resource> {
private final int cpuCores;
private final long memoryMB;
private final int gpuUnits;
private final int diskIO;
private final int networkBW;
// 资源向量化比较
@Override
public int compareTo(Resource other) {
if (this.cpuCores != other.cpuCores) {
return Integer.compare(this.cpuCores, other.cpuCores);
}
if (this.memoryMB != other.memoryMB) {
return Long.compare(this.memoryMB, other.memoryMB);
}
if (this.gpuUnits != other.gpuUnits) {
return Integer.compare(this.gpuUnits, other.gpuUnits);
}
if (this.diskIO != other.diskIO) {
return Integer.compare(this.diskIO, other.diskIO);
}
return Integer.compare(this.networkBW, other.networkBW);
}
// 资源运算
public Resource add(Resource other) {
return new Resource(
this.cpuCores + other.cpuCores,
this.memoryMB + other.memoryMB,
this.gpuUnits + other.gpuUnits,
this.diskIO + other.diskIO,
this.networkBW + other.networkBW
);
}
public Resource multiply(double factor) {
return new Resource(
(int)(this.cpuCores * factor),
(long)(this.memoryMB * factor),
(int)(this.gpuUnits * factor),
(int)(this.diskIO * factor),
(int)(this.networkBW * factor)
);
}
// 工业容器规格工厂
public static class ContainerSpecs {
// 标准Map容器
public static Resource MAP = new Resource(2, 4096, 0, 100, 50);
// 标准Reduce容器
public static Resource REDUCE = new Resource(4, 8192, 0, 200, 100);
// 大数据量Reduce容器
public static Resource REDUCE_LARGE = new Resource(8, 16384, 0, 500, 200);
// ML训练容器
public static Resource ML_TRAIN = new Resource(32, 65536, 4, 1000, 500);
// ML推理容器
public static Resource ML_INFER = new Resource(8, 16384, 2, 200, 100);
}
}4.2.2 YARN架构的数学流程
ApplicationMaster NodeManager-2 NodeManager-1 ResourceManager Client ApplicationMaster NodeManager-2 NodeManager-1 ResourceManager Client #mermaid-svg-QQ1nBhayMtqKgLlI{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-QQ1nBhayMtqKgLlI .error-icon{fill:#552222;}#mermaid-svg-QQ1nBhayMtqKgLlI .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-QQ1nBhayMtqKgLlI .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-QQ1nBhayMtqKgLlI .marker{fill:#333333;stroke:#333333;}#mermaid-svg-QQ1nBhayMtqKgLlI .marker.cross{stroke:#333333;}#mermaid-svg-QQ1nBhayMtqKgLlI svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-QQ1nBhayMtqKgLlI p{margin:0;}#mermaid-svg-QQ1nBhayMtqKgLlI .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-QQ1nBhayMtqKgLlI text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-QQ1nBhayMtqKgLlI .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-QQ1nBhayMtqKgLlI .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-QQ1nBhayMtqKgLlI .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-QQ1nBhayMtqKgLlI .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-QQ1nBhayMtqKgLlI #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-QQ1nBhayMtqKgLlI .sequenceNumber{fill:white;}#mermaid-svg-QQ1nBhayMtqKgLlI #sequencenumber{fill:#333;}#mermaid-svg-QQ1nBhayMtqKgLlI #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-QQ1nBhayMtqKgLlI .messageText{fill:#333;stroke:none;}#mermaid-svg-QQ1nBhayMtqKgLlI .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-QQ1nBhayMtqKgLlI .labelText,#mermaid-svg-QQ1nBhayMtqKgLlI .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-QQ1nBhayMtqKgLlI .loopText,#mermaid-svg-QQ1nBhayMtqKgLlI .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-QQ1nBhayMtqKgLlI .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-QQ1nBhayMtqKgLlI .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-QQ1nBhayMtqKgLlI .noteText,#mermaid-svg-QQ1nBhayMtqKgLlI .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-QQ1nBhayMtqKgLlI .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-QQ1nBhayMtqKgLlI .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-QQ1nBhayMtqKgLlI .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-QQ1nBhayMtqKgLlI .actorPopupMenu{position:absolute;}#mermaid-svg-QQ1nBhayMtqKgLlI .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-QQ1nBhayMtqKgLlI .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-QQ1nBhayMtqKgLlI .actor-man circle,#mermaid-svg-QQ1nBhayMtqKgLlI line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-QQ1nBhayMtqKgLlI :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} loop 资源请求循环 提交Application 返回ApplicationID 启动AM Container AM注册 请求资源 (ResourceRequest) 分配资源 (AllocateResponse) 心跳+请求资源 分配Container 分配Container Container完成 Container完成 完成Application
4.3 Capacity Scheduler深度剖析
4.3.1 容量保证的数学模型
Capacity Scheduler的容量保证：
设队列Q的保证容量为 C(Q)，则：
1. 容量保证：
∀时刻 t: 已分配资源(t) ≥ C(Q) × 总资源 × 占用系数
2. 容量上限：
队列最大容量 Max(Q) 防止过度占用
3. 队列间公平性：
空闲容量按保证容量比例分配
数学表示：
队列Q的实际资源份额 =
min(队列请求, 保证容量 + 空闲容量 × (保证容量 / Σ保证容量))4.3.2 工业级Capacity Scheduler配置
xml
<!-- capacity-scheduler.xml 工业级配置 -->
<configuration>
<!-- 根队列定义 -->
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>production,development,test</value>
</property>
<!-- 生产队列：最高优先级 -->
<property>
<name>yarn.scheduler.capacity.root.production.capacity</name>
<value>50</value> <!-- 50% -->
</property>
<property>
<name>yarn.scheduler.capacity.root.production.maximum-capacity</name>
<value>80</value> <!-- 最多80% -->
</property>
<property>
<name>yarn.scheduler.capacity.root.production.user-limit-factor</name>
<value>2</value> <!-- 单用户最大2倍保证容量 -->
</property>
<!-- 开发队列 -->
<property>
<name>yarn.scheduler.capacity.root.development.capacity</name>
<value>30</value> <!-- 30% -->
</property>
<property>
<name>yarn.scheduler.capacity.root.development.maximum-capacity</name>
<value>60</value>
</property>
<!-- 测试队列 -->
<property>
<name>yarn.scheduler.capacity.root.test.capacity</name>
<value>20</value> <!-- 20% -->
</property>
<property>
<name>yarn.scheduler.capacity.root.test.maximum-capacity</name>
<value>40</value>
</property>
<!-- 通用配置 -->
<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>40</value> <!-- 40个调度周期延迟 -->
</property>
<property>
<name>yarn.scheduler.capacity.rack-locality-additional-delay</name>
<value>10</value>
</property>
</configuration>4.4 Kubernetes与YARN的融合
4.4.1 云原生时代的资源调度趋势
#mermaid-svg-EAOgY9TnnpEUJqKX{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-EAOgY9TnnpEUJqKX .error-icon{fill:#552222;}#mermaid-svg-EAOgY9TnnpEUJqKX .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-EAOgY9TnnpEUJqKX .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-EAOgY9TnnpEUJqKX .marker{fill:#333333;stroke:#333333;}#mermaid-svg-EAOgY9TnnpEUJqKX .marker.cross{stroke:#333333;}#mermaid-svg-EAOgY9TnnpEUJqKX svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-EAOgY9TnnpEUJqKX p{margin:0;}#mermaid-svg-EAOgY9TnnpEUJqKX .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-EAOgY9TnnpEUJqKX .cluster-label text{fill:#333;}#mermaid-svg-EAOgY9TnnpEUJqKX .cluster-label span{color:#333;}#mermaid-svg-EAOgY9TnnpEUJqKX .cluster-label span p{background-color:transparent;}#mermaid-svg-EAOgY9TnnpEUJqKX .label text,#mermaid-svg-EAOgY9TnnpEUJqKX span{fill:#333;color:#333;}#mermaid-svg-EAOgY9TnnpEUJqKX .node rect,#mermaid-svg-EAOgY9TnnpEUJqKX .node circle,#mermaid-svg-EAOgY9TnnpEUJqKX .node ellipse,#mermaid-svg-EAOgY9TnnpEUJqKX .node polygon,#mermaid-svg-EAOgY9TnnpEUJqKX .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-EAOgY9TnnpEUJqKX .rough-node .label text,#mermaid-svg-EAOgY9TnnpEUJqKX .node .label text,#mermaid-svg-EAOgY9TnnpEUJqKX .image-shape .label,#mermaid-svg-EAOgY9TnnpEUJqKX .icon-shape .label{text-anchor:middle;}#mermaid-svg-EAOgY9TnnpEUJqKX .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-EAOgY9TnnpEUJqKX .rough-node .label,#mermaid-svg-EAOgY9TnnpEUJqKX .node .label,#mermaid-svg-EAOgY9TnnpEUJqKX .image-shape .label,#mermaid-svg-EAOgY9TnnpEUJqKX .icon-shape .label{text-align:center;}#mermaid-svg-EAOgY9TnnpEUJqKX .node.clickable{cursor:pointer;}#mermaid-svg-EAOgY9TnnpEUJqKX .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-EAOgY9TnnpEUJqKX .arrowheadPath{fill:#333333;}#mermaid-svg-EAOgY9TnnpEUJqKX .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-EAOgY9TnnpEUJqKX .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-EAOgY9TnnpEUJqKX .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EAOgY9TnnpEUJqKX .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-EAOgY9TnnpEUJqKX .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EAOgY9TnnpEUJqKX .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-EAOgY9TnnpEUJqKX .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-EAOgY9TnnpEUJqKX .cluster text{fill:#333;}#mermaid-svg-EAOgY9TnnpEUJqKX .cluster span{color:#333;}#mermaid-svg-EAOgY9TnnpEUJqKX div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-EAOgY9TnnpEUJqKX .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-EAOgY9TnnpEUJqKX rect.text{fill:none;stroke-width:0;}#mermaid-svg-EAOgY9TnnpEUJqKX .icon-shape,#mermaid-svg-EAOgY9TnnpEUJqKX .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EAOgY9TnnpEUJqKX .icon-shape p,#mermaid-svg-EAOgY9TnnpEUJqKX .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-EAOgY9TnnpEUJqKX .icon-shape .label rect,#mermaid-svg-EAOgY9TnnpEUJqKX .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EAOgY9TnnpEUJqKX .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-EAOgY9TnnpEUJqKX .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-EAOgY9TnnpEUJqKX :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 统一架构
云原生架构
传统架构
未来演进
过渡期
YARN Cluster
HDFS
Kubernetes Cluster
S3/MinIO
Virtual Cluster
YARN on K8s
Unified Control Plane
4.4.2 YARN on Kubernetes配置
yaml
# kubernetes-yarn配置示例
apiVersion: v1
kind: ConfigMap
metadata:
name: yarn-site
data:
yarn-site.xml: |
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>65536</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>32</value>
</property>
</configuration>
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: yarn-nodemanager
spec:
serviceName: yarn-nm
replicas: 10
selector:
matchLabels:
app: yarn-nodemanager
template:
spec:
containers:
- name: nodemanager
image: hadoop:3.3.4
env:
- name: YARN_CONF_dir
value: /opt/hadoop/etc/hadoop
resources:
requests:
memory: "64Gi"
cpu: "32"
limits:
memory: "64Gi"
cpu: "32"
volumeMounts:
- name: yarn-conf
mountPath: /opt/hadoop/etc/hadoop
volumes:
- name: yarn-conf
configMap:
name: yarn-site4.5 本期小结
YARN资源调度是分布式计算系统的核心：
┌─────────────────────────────────────────────────────────────┐
│ YARN资源调度知识体系 │
├─────────────────────────────────────────────────────────────┤
│ 第1层：理论基础层 │
│ ├── Max-Min Fairness：最小请求优先满足 │
│ ├── 资源向量：⟨CPU, Memory, GPU, Disk, Network⟩ │
│ └── 公平性度量：不存在使最小值更大而不减少最大值的方案 │
├─────────────────────────────────────────────────────────────┤
│ 第2层：调度器层 │
│ ├── Capacity Scheduler：保证容量+上限容量 │
│ ├── Fair Scheduler：动态重分配+最小最大公平 │
│ └── 队列层级：支持多租户层级结构 │
├─────────────────────────────────────────────────────────────┤
│ 第3层：容器层 │
│ ├── Container抽象：资源向量+生命周期管理 │
│ ├── 本地性调度：Node > Rack > Off-switch │
│ └── 资源隔离：CGroups/Linux Container │
├─────────────────────────────────────────────────────────────┤
│ 第4层：演进层 │
│ ├── YARN on Kubernetes：容器化部署 │
│ ├── GPU调度：NVIDIA Device Plugin │
│ └── 云原生融合：Serverless计算 │
└─────────────────────────────────────────────────────────────┘下期预告 ：第5期：Kafka消息队列 - 工业数据流传输的可靠保证机制------深度解析Kafka的ISR机制、Exactly-Once语义、以及工业场景的高吞吐配置优化。
作者：高炉炼铁智能化技术研究者，专注钢铁冶金与人工智能 交叉领域。
👍 如果觉得有帮助，请点赞、收藏、转发！
版权归作者所有，未经许可请勿抄袭，套用，商用(或其它具有利益性行为) 。
🔔 关注专栏，不错过后续精彩内容！