摘要
推理系统架构决定服务可用性与扩展性。本文从网关设计、负载均衡策略、高可用容灾、弹性伸缩、多机房部署五个切口，给出源码级实现与企业级推理服务架构决策框架。
1. 网关设计：请求接入与流量控制
网关是推理服务入口，负责鉴权、限流、路由、协议转换。设计要点：长连接管理（流式输出）、超时控制、降级策略。
#mermaid-svg-EEACriKyUskJNnGE{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-EEACriKyUskJNnGE .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-EEACriKyUskJNnGE .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-EEACriKyUskJNnGE .error-icon{fill:#552222;}#mermaid-svg-EEACriKyUskJNnGE .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-EEACriKyUskJNnGE .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-EEACriKyUskJNnGE .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-EEACriKyUskJNnGE .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-EEACriKyUskJNnGE .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-EEACriKyUskJNnGE .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-EEACriKyUskJNnGE .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-EEACriKyUskJNnGE .marker{fill:#333333;stroke:#333333;}#mermaid-svg-EEACriKyUskJNnGE .marker.cross{stroke:#333333;}#mermaid-svg-EEACriKyUskJNnGE svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-EEACriKyUskJNnGE p{margin:0;}#mermaid-svg-EEACriKyUskJNnGE .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-EEACriKyUskJNnGE .cluster-label text{fill:#333;}#mermaid-svg-EEACriKyUskJNnGE .cluster-label span{color:#333;}#mermaid-svg-EEACriKyUskJNnGE .cluster-label span p{background-color:transparent;}#mermaid-svg-EEACriKyUskJNnGE .label text,#mermaid-svg-EEACriKyUskJNnGE span{fill:#333;color:#333;}#mermaid-svg-EEACriKyUskJNnGE .node rect,#mermaid-svg-EEACriKyUskJNnGE .node circle,#mermaid-svg-EEACriKyUskJNnGE .node ellipse,#mermaid-svg-EEACriKyUskJNnGE .node polygon,#mermaid-svg-EEACriKyUskJNnGE .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-EEACriKyUskJNnGE .rough-node .label text,#mermaid-svg-EEACriKyUskJNnGE .node .label text,#mermaid-svg-EEACriKyUskJNnGE .image-shape .label,#mermaid-svg-EEACriKyUskJNnGE .icon-shape .label{text-anchor:middle;}#mermaid-svg-EEACriKyUskJNnGE .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-EEACriKyUskJNnGE .rough-node .label,#mermaid-svg-EEACriKyUskJNnGE .node .label,#mermaid-svg-EEACriKyUskJNnGE .image-shape .label,#mermaid-svg-EEACriKyUskJNnGE .icon-shape .label{text-align:center;}#mermaid-svg-EEACriKyUskJNnGE .node.clickable{cursor:pointer;}#mermaid-svg-EEACriKyUskJNnGE .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-EEACriKyUskJNnGE .arrowheadPath{fill:#333333;}#mermaid-svg-EEACriKyUskJNnGE .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-EEACriKyUskJNnGE .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-EEACriKyUskJNnGE .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EEACriKyUskJNnGE .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-EEACriKyUskJNnGE .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EEACriKyUskJNnGE .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-EEACriKyUskJNnGE .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-EEACriKyUskJNnGE .cluster text{fill:#333;}#mermaid-svg-EEACriKyUskJNnGE .cluster span{color:#333;}#mermaid-svg-EEACriKyUskJNnGE div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-EEACriKyUskJNnGE .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-EEACriKyUskJNnGE rect.text{fill:none;stroke-width:0;}#mermaid-svg-EEACriKyUskJNnGE .icon-shape,#mermaid-svg-EEACriKyUskJNnGE .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EEACriKyUskJNnGE .icon-shape p,#mermaid-svg-EEACriKyUskJNnGE .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-EEACriKyUskJNnGE .icon-shape .label rect,#mermaid-svg-EEACriKyUskJNnGE .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EEACriKyUskJNnGE .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-EEACriKyUskJNnGE .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-EEACriKyUskJNnGE :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-EEACriKyUskJNnGE .default>*{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-EEACriKyUskJNnGE .default span{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-EEACriKyUskJNnGE .default tspan{fill:#000000!important;} 推理网关
鉴权: API Key/Token
限流: QPS/并发/Token配额
路由: 模型/租户/优先级
协议转换: HTTP/SSE/gRPC
多租户隔离
令牌桶+滑动窗口
模型路由表
SSE 流式输出
python
# 来源：推理网关实现 / 生产实践 2024
import time
import asyncio
from collections import defaultdict
class InferenceGateway:
"""推理服务网关"""
def __init__(self, rate_limits, model_routes):
self.rate_limiter = TokenBucketRateLimiter(rate_limits)
self.routes = model_routes # {model_name: [backend_urls]}
self.auth = APIKeyAuth()
async def handle_request(self, request):
"""处理推理请求"""
# 1. 鉴权
if not self.auth.validate(request.api_key):
return {'error': 'unauthorized'}, 401
tenant = self.auth.get_tenant(request.api_key)
# 2. 限流 (多维度)
if not self.rate_limiter.allow(tenant, request.model):
return {'error': 'rate limited'}, 429
# 3. 路由
backend = self._select_backend(request.model, tenant)
# 4. 转发 (SSE 流式)
if request.stream:
return await self._stream_forward(backend, request)
else:
return await self._forward(backend, request)
def _select_backend(self, model, tenant):
"""选后端节点"""
backends = self.routes[model]
# 简化: 轮询, 实际用负载+健康检查
return backends[tenant.__hash__() % len(backends)]
async def _stream_forward(self, backend, request):
"""SSE 流式转发"""
async with httpx.AsyncClient() as client:
async with client.stream('POST', backend, json=request.dict()) as resp:
async for chunk in resp.aiter_bytes():
yield chunk
class TokenBucketRateLimiter:
"""令牌桶限流器"""
def __init__(self, limits):
# limits: {tenant_id: {model: {qps: 100, concurrent: 10, tokens_per_min: 10000}}}
self.limits = limits
self.buckets = defaultdict(lambda: defaultdict(dict))
self.concurrent = defaultdict(int)
def allow(self, tenant, model):
limit = self.limits.get(tenant, {}).get(model, {'qps': 100})
bucket = self.buckets[tenant][model]
now = time.time()
# 初始化桶
if 'tokens' not in bucket:
bucket['tokens'] = limit['qps']
bucket['last'] = now
# 补充令牌
bucket['tokens'] = min(limit['qps'],
bucket['tokens'] + (now - bucket['last']) * limit['qps'])
bucket['last'] = now
# 消耗令牌
if bucket['tokens'] >= 1:
bucket['tokens'] -= 1
return True
return False
# 量化: 多租户限流防某租户耗尽资源
# 令牌桶允许突发, 滑动窗口平滑限流
# 7B 模型单租户 QPS 限 50, 并发限 10量化：多租户限流防单租户耗尽资源，令牌桶允许突发（突发 2 倍 QPS 持续 1 秒），滑动窗口平滑限流。7B 模型单租户典型限制：QPS 50、并发 10、tokens/min 10000。网关层延迟需 <5ms，否则拖累整体 P99。
边界：流式输出（SSE）需长连接------网关需支持连接保持，Nginx 默认 60 秒超时需调大。限流需多维度------仅 QPS 限流无法防长请求耗尽并发。降级策略需明确------后端故障时返回缓存结果或排队而非直接报错。
2. 负载均衡：感知推理特征的调度
推理请求特征独特：处理时间差异大（10ms-10s）、上下文可缓存（Prefix Cache）、优先级分级（付费/免费）。负载均衡需感知这些特征。
#mermaid-svg-dzo4c1GcqEhD4FWt{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dzo4c1GcqEhD4FWt .error-icon{fill:#552222;}#mermaid-svg-dzo4c1GcqEhD4FWt .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dzo4c1GcqEhD4FWt .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dzo4c1GcqEhD4FWt .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dzo4c1GcqEhD4FWt .marker.cross{stroke:#333333;}#mermaid-svg-dzo4c1GcqEhD4FWt svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dzo4c1GcqEhD4FWt p{margin:0;}#mermaid-svg-dzo4c1GcqEhD4FWt .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-dzo4c1GcqEhD4FWt .cluster-label text{fill:#333;}#mermaid-svg-dzo4c1GcqEhD4FWt .cluster-label span{color:#333;}#mermaid-svg-dzo4c1GcqEhD4FWt .cluster-label span p{background-color:transparent;}#mermaid-svg-dzo4c1GcqEhD4FWt .label text,#mermaid-svg-dzo4c1GcqEhD4FWt span{fill:#333;color:#333;}#mermaid-svg-dzo4c1GcqEhD4FWt .node rect,#mermaid-svg-dzo4c1GcqEhD4FWt .node circle,#mermaid-svg-dzo4c1GcqEhD4FWt .node ellipse,#mermaid-svg-dzo4c1GcqEhD4FWt .node polygon,#mermaid-svg-dzo4c1GcqEhD4FWt .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-dzo4c1GcqEhD4FWt .rough-node .label text,#mermaid-svg-dzo4c1GcqEhD4FWt .node .label text,#mermaid-svg-dzo4c1GcqEhD4FWt .image-shape .label,#mermaid-svg-dzo4c1GcqEhD4FWt .icon-shape .label{text-anchor:middle;}#mermaid-svg-dzo4c1GcqEhD4FWt .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-dzo4c1GcqEhD4FWt .rough-node .label,#mermaid-svg-dzo4c1GcqEhD4FWt .node .label,#mermaid-svg-dzo4c1GcqEhD4FWt .image-shape .label,#mermaid-svg-dzo4c1GcqEhD4FWt .icon-shape .label{text-align:center;}#mermaid-svg-dzo4c1GcqEhD4FWt .node.clickable{cursor:pointer;}#mermaid-svg-dzo4c1GcqEhD4FWt .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-dzo4c1GcqEhD4FWt .arrowheadPath{fill:#333333;}#mermaid-svg-dzo4c1GcqEhD4FWt .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-dzo4c1GcqEhD4FWt .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-dzo4c1GcqEhD4FWt .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dzo4c1GcqEhD4FWt .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-dzo4c1GcqEhD4FWt .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dzo4c1GcqEhD4FWt .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-dzo4c1GcqEhD4FWt .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-dzo4c1GcqEhD4FWt .cluster text{fill:#333;}#mermaid-svg-dzo4c1GcqEhD4FWt .cluster span{color:#333;}#mermaid-svg-dzo4c1GcqEhD4FWt div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-dzo4c1GcqEhD4FWt .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-dzo4c1GcqEhD4FWt rect.text{fill:none;stroke-width:0;}#mermaid-svg-dzo4c1GcqEhD4FWt .icon-shape,#mermaid-svg-dzo4c1GcqEhD4FWt .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dzo4c1GcqEhD4FWt .icon-shape p,#mermaid-svg-dzo4c1GcqEhD4FWt .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-dzo4c1GcqEhD4FWt .icon-shape .label rect,#mermaid-svg-dzo4c1GcqEhD4FWt .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dzo4c1GcqEhD4FWt .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-dzo4c1GcqEhD4FWt .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-dzo4c1GcqEhD4FWt :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-dzo4c1GcqEhD4FWt .default>*{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-dzo4c1GcqEhD4FWt .default span{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-dzo4c1GcqEhD4FWt .default tspan{fill:#000000!important;} 推理负载均衡
最小连接数
前缀感知: 同前缀同节点
优先级调度
模型亲和性
避免长请求堆积
命中Prefix Cache
付费优先/免费排队
同模型同节点避免重加载
python
# 来源：推理负载均衡器 / 生产实践 2024
import hashlib
from collections import defaultdict
class InferenceLoadBalancer:
"""推理专用负载均衡器"""
def __init__(self, backends):
self.backends = backends # [{'url', 'healthy', 'conns', 'p99', 'models'}]
self.prefix_map = {} # prefix_hash -> backend_url (前缀亲和)
def route(self, request):
"""路由请求"""
healthy = [b for b in self.backends if b['healthy']]
if not healthy:
raise Exception('无可用后端')
# 1. 前缀感知: 同前缀路由到同节点 (命中Prefix Cache)
prefix_hash = self._hash_prefix(request['prompt'])
if prefix_hash in self.prefix_map:
cached = self.prefix_map[prefix_hash]
if cached in [b['url'] for b in healthy]:
return cached
# 2. 优先级调度: 付费优先
if request.get('priority') == 'paid':
candidates = [b for b in healthy if b['p99'] < 100]
else:
candidates = healthy
# 3. 最小连接数 (避免长请求堆积)
backend = min(candidates, key=lambda b: b['conns'])
# 4. 记录前缀亲和
self.prefix_map[prefix_hash] = backend['url']
return backend['url']
def _hash_prefix(self, prompt):
# 取前 200 字符哈希 (system prompt 通常共享)
return hashlib.md5(prompt[:200].encode()).hexdigest()
def update_stats(self, backend_url, latency, conns):
"""更新后端统计"""
for b in self.backends:
if b['url'] == backend_url:
b['p99'] = latency
b['conns'] = conns
break
# 量化: 前缀感知路由使 Prefix Cache 命中率 40% -> 85%
# 付费用户 P99 < 100ms 保证, 免费 P99 < 500ms
# 最小连接数避免长请求拖垮单节点
python
# 来源：健康检查实现 / 生产实践 2024
import asyncio
class HealthChecker:
"""后端健康检查器"""
def __init__(self, backends, interval=5, unhealthy_threshold=3):
self.backends = backends
self.interval = interval
self.threshold = unhealthy_threshold
self.fail_counts = defaultdict(int)
async def run(self):
"""持续健康检查"""
while True:
tasks = [self._check(b) for b in self.backends]
await asyncio.gather(*tasks)
await asyncio.sleep(self.interval)
async def _check(self, backend):
"""检查单个后端"""
try:
# 推理测试请求 (简单 prompt)
start = time.time()
result = await self._ping(backend['url'])
latency = (time.time() - start) * 1000
if latency < 1000 and result:
backend['healthy'] = True
backend['p99'] = latency
self.fail_counts[backend['url']] = 0
else:
self._mark_unhealthy(backend)
except:
self._mark_unhealthy(backend)
def _mark_unhealthy(self, backend):
self.fail_counts[backend['url']] += 1
if self.fail_counts[backend['url']] >= self.threshold:
backend['healthy'] = False
async def _ping(self, url):
return True # 占位
# 量化: 健康检查间隔 5s, 连续 3 次失败标记不健康
# 故障检测延迟: 5-15s (3次检查周期)
# 避免单次网络抖动误判量化：前缀感知路由使 Prefix Cache 命中率从 40% 升至 85%，首 token 延迟降 60%。最小连接数策略避免长请求堆积，单节点并发均衡度提升 40%。健康检查间隔 5s+3 次失败阈值，故障检测延迟 5-15s，避免网络抖动误判。
边界：前缀亲和需节点故障容错------亲和节点故障需 fallback 到其他节点，缓存失效。优先级调度需公平性保证------免费用户不能无限排队，需超时降级。模型亲和性在大规模集群受限------节点数大于模型数时，亲和性无意义。
3. 高可用：容灾与故障恢复
推理服务高可用需多级容灾：实例级（GPU 故障）、机房级（网络中断）、区域级（云故障）。核心策略：冗余部署、自动故障转移、降级服务。
#mermaid-svg-IFZ4fDg0WvkBmQoh{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-IFZ4fDg0WvkBmQoh .error-icon{fill:#552222;}#mermaid-svg-IFZ4fDg0WvkBmQoh .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-IFZ4fDg0WvkBmQoh .marker{fill:#333333;stroke:#333333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .marker.cross{stroke:#333333;}#mermaid-svg-IFZ4fDg0WvkBmQoh svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-IFZ4fDg0WvkBmQoh p{margin:0;}#mermaid-svg-IFZ4fDg0WvkBmQoh .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .cluster-label text{fill:#333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .cluster-label span{color:#333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .cluster-label span p{background-color:transparent;}#mermaid-svg-IFZ4fDg0WvkBmQoh .label text,#mermaid-svg-IFZ4fDg0WvkBmQoh span{fill:#333;color:#333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .node rect,#mermaid-svg-IFZ4fDg0WvkBmQoh .node circle,#mermaid-svg-IFZ4fDg0WvkBmQoh .node ellipse,#mermaid-svg-IFZ4fDg0WvkBmQoh .node polygon,#mermaid-svg-IFZ4fDg0WvkBmQoh .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-IFZ4fDg0WvkBmQoh .rough-node .label text,#mermaid-svg-IFZ4fDg0WvkBmQoh .node .label text,#mermaid-svg-IFZ4fDg0WvkBmQoh .image-shape .label,#mermaid-svg-IFZ4fDg0WvkBmQoh .icon-shape .label{text-anchor:middle;}#mermaid-svg-IFZ4fDg0WvkBmQoh .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-IFZ4fDg0WvkBmQoh .rough-node .label,#mermaid-svg-IFZ4fDg0WvkBmQoh .node .label,#mermaid-svg-IFZ4fDg0WvkBmQoh .image-shape .label,#mermaid-svg-IFZ4fDg0WvkBmQoh .icon-shape .label{text-align:center;}#mermaid-svg-IFZ4fDg0WvkBmQoh .node.clickable{cursor:pointer;}#mermaid-svg-IFZ4fDg0WvkBmQoh .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .arrowheadPath{fill:#333333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-IFZ4fDg0WvkBmQoh .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IFZ4fDg0WvkBmQoh .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-IFZ4fDg0WvkBmQoh .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IFZ4fDg0WvkBmQoh .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-IFZ4fDg0WvkBmQoh .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-IFZ4fDg0WvkBmQoh .cluster text{fill:#333;}#mermaid-svg-IFZ4fDg0WvkBmQoh .cluster span{color:#333;}#mermaid-svg-IFZ4fDg0WvkBmQoh div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-IFZ4fDg0WvkBmQoh .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-IFZ4fDg0WvkBmQoh rect.text{fill:none;stroke-width:0;}#mermaid-svg-IFZ4fDg0WvkBmQoh .icon-shape,#mermaid-svg-IFZ4fDg0WvkBmQoh .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IFZ4fDg0WvkBmQoh .icon-shape p,#mermaid-svg-IFZ4fDg0WvkBmQoh .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-IFZ4fDg0WvkBmQoh .icon-shape .label rect,#mermaid-svg-IFZ4fDg0WvkBmQoh .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IFZ4fDg0WvkBmQoh .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-IFZ4fDg0WvkBmQoh .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-IFZ4fDg0WvkBmQoh :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-IFZ4fDg0WvkBmQoh .default>*{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-IFZ4fDg0WvkBmQoh .default span{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-IFZ4fDg0WvkBmQoh .default tspan{fill:#000000!important;} 高可用架构
实例级: 多副本
机房级: 跨机房部署
区域级: 多云备份
N+1冗余: 1个备用
自动故障转移
DNS轮询/Anycast
异地容灾
AWS/阿里云双云
数据同步
故障节点摘除
请求重路由
python
# 来源：故障转移实现 / 生产实践 2024
import asyncio
from collections import defaultdict
class FailoverManager:
"""故障转移管理器"""
def __init__(self, backends, failover_threshold=3):
self.backends = backends
self.threshold = failover_threshold
self.fail_counts = defaultdict(int)
self.circuit = defaultdict(lambda: 'closed') # closed/open/half_open
async def route_with_failover(self, request, lb):
"""带故障转移的路由"""
attempts = 0
max_attempts = len(self.backends)
while attempts < max_attempts:
backend = lb.route(request)
if self.circuit[backend] == 'open':
attempts += 1
continue
try:
result = await self._call(backend, request)
self.fail_counts[backend] = 0
self.circuit[backend] = 'closed'
return result
except Exception as e:
self._handle_failure(backend)
attempts += 1
# 全部失败, 降级
return await self._degrade(request)
def _handle_failure(self, backend):
self.fail_counts[backend] += 1
if self.fail_counts[backend] >= self.threshold:
self.circuit[backend] = 'open' # 熔断
# 60秒后半开
asyncio.create_task(self._half_open_timer(backend))
async def _half_open_timer(self, backend):
await asyncio.sleep(60)
self.circuit[backend] = 'half_open'
async def _degrade(self, request):
"""降级服务: 返回缓存或简化响应"""
# 1. 查缓存
cached = await self._query_cache(request)
if cached:
return cached
# 2. 返回提示信息
return {'error': 'service degraded', 'retry_after': 60}
async def _call(self, backend, request):
return {} # 占位
async def _query_cache(self, request):
return None # 占位
# 量化: 熔断器 3 次失败触发, 60秒半开探测
# 故障转移: 首次失败立即重路由, 用户无感
# 降级: 返回缓存使可用性 99.9% -> 99.99%
python
# 来源：多机房容灾 / 生产实践 2024
class MultiRegionManager:
"""多机房容灾管理器"""
def __init__(self, regions):
# regions: [{'name', 'endpoint', 'health', 'priority'}]
self.regions = regions
def route(self, request):
"""选择最优机房"""
healthy = [r for r in self.regions if r['health']]
if not healthy:
raise Exception('全机房不可用')
# 1. 按优先级 (主机房优先)
healthy.sort(key=lambda r: r['priority'])
# 2. 延迟探测选最近
primary = healthy[0]
if self._latency(primary) < 200:
return primary
# 主机房延迟高, 选备用
for r in healthy[1:]:
if self._latency(r) < 100:
return r
return primary
def _latency(self, region):
return 50 # 占位
# 量化: 双机房部署使可用性 99.9% -> 99.99%
# 故障切换时间: DNS 轮询 30s, Anycast <1s
# 成本: 双机房需 2x 资源, 适合核心业务量化：双机房部署使可用性从 99.9% 升至 99.99%，故障切换 DNS 轮询 30s、Anycast <1s。熔断器 3 次失败触发、60 秒半开探测，避免雪崩。降级返回缓存使可用性再提一个 9。成本：双机房需 2x 资源，适合核心业务；非核心可用单机房+冷备。
边界：跨机房数据同步是难点------模型权重大（7B 14GB），同步慢，宜各机房独立加载。DNS 轮询切换慢------全球服务需用 Anycast 或 HTTP DNS。降级策略需业务允许------医疗/金融等不能降级，需强一致。
4. 弹性伸缩：GPU 集群自动扩缩
GPU 推理集群伸缩慢（启动+模型加载 30-60s），需预测性伸缩而非反应式。核心指标：QPS、P99 延迟、GPU 利用率、队列深度。
#mermaid-svg-noQ2xwwx9LjBz8dH{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-noQ2xwwx9LjBz8dH .error-icon{fill:#552222;}#mermaid-svg-noQ2xwwx9LjBz8dH .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-noQ2xwwx9LjBz8dH .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-noQ2xwwx9LjBz8dH .marker{fill:#333333;stroke:#333333;}#mermaid-svg-noQ2xwwx9LjBz8dH .marker.cross{stroke:#333333;}#mermaid-svg-noQ2xwwx9LjBz8dH svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-noQ2xwwx9LjBz8dH p{margin:0;}#mermaid-svg-noQ2xwwx9LjBz8dH .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-noQ2xwwx9LjBz8dH .cluster-label text{fill:#333;}#mermaid-svg-noQ2xwwx9LjBz8dH .cluster-label span{color:#333;}#mermaid-svg-noQ2xwwx9LjBz8dH .cluster-label span p{background-color:transparent;}#mermaid-svg-noQ2xwwx9LjBz8dH .label text,#mermaid-svg-noQ2xwwx9LjBz8dH span{fill:#333;color:#333;}#mermaid-svg-noQ2xwwx9LjBz8dH .node rect,#mermaid-svg-noQ2xwwx9LjBz8dH .node circle,#mermaid-svg-noQ2xwwx9LjBz8dH .node ellipse,#mermaid-svg-noQ2xwwx9LjBz8dH .node polygon,#mermaid-svg-noQ2xwwx9LjBz8dH .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-noQ2xwwx9LjBz8dH .rough-node .label text,#mermaid-svg-noQ2xwwx9LjBz8dH .node .label text,#mermaid-svg-noQ2xwwx9LjBz8dH .image-shape .label,#mermaid-svg-noQ2xwwx9LjBz8dH .icon-shape .label{text-anchor:middle;}#mermaid-svg-noQ2xwwx9LjBz8dH .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-noQ2xwwx9LjBz8dH .rough-node .label,#mermaid-svg-noQ2xwwx9LjBz8dH .node .label,#mermaid-svg-noQ2xwwx9LjBz8dH .image-shape .label,#mermaid-svg-noQ2xwwx9LjBz8dH .icon-shape .label{text-align:center;}#mermaid-svg-noQ2xwwx9LjBz8dH .node.clickable{cursor:pointer;}#mermaid-svg-noQ2xwwx9LjBz8dH .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-noQ2xwwx9LjBz8dH .arrowheadPath{fill:#333333;}#mermaid-svg-noQ2xwwx9LjBz8dH .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-noQ2xwwx9LjBz8dH .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-noQ2xwwx9LjBz8dH .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-noQ2xwwx9LjBz8dH .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-noQ2xwwx9LjBz8dH .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-noQ2xwwx9LjBz8dH .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-noQ2xwwx9LjBz8dH .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-noQ2xwwx9LjBz8dH .cluster text{fill:#333;}#mermaid-svg-noQ2xwwx9LjBz8dH .cluster span{color:#333;}#mermaid-svg-noQ2xwwx9LjBz8dH div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-noQ2xwwx9LjBz8dH .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-noQ2xwwx9LjBz8dH rect.text{fill:none;stroke-width:0;}#mermaid-svg-noQ2xwwx9LjBz8dH .icon-shape,#mermaid-svg-noQ2xwwx9LjBz8dH .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-noQ2xwwx9LjBz8dH .icon-shape p,#mermaid-svg-noQ2xwwx9LjBz8dH .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-noQ2xwwx9LjBz8dH .icon-shape .label rect,#mermaid-svg-noQ2xwwx9LjBz8dH .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-noQ2xwwx9LjBz8dH .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-noQ2xwwx9LjBz8dH .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-noQ2xwwx9LjBz8dH :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-noQ2xwwx9LjBz8dH .default>*{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-noQ2xwwx9LjBz8dH .default span{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-noQ2xwwx9LjBz8dH .default tspan{fill:#000000!important;} 弹性伸缩
指标采集
伸缩决策
预测性扩容
QPS/P99/利用率/队列
阈值规则
历史趋势预测
提前1分钟扩容
扩: QPS>80 或 P99>500ms
缩: QPS<20 持续5分钟
python
# 来源：GPU 集群伸缩 / 生产实践 2024
import time
from collections import deque
class GPUClusterScaler:
"""GPU 集群自动伸缩器"""
def __init__(self, min_nodes=2, max_nodes=20,
scale_up_qps=80, scale_down_qps=20,
p99_threshold=500, warmup_time=60):
self.min = min_nodes
self.max = max_nodes
self.up_qps = scale_up_qps
self.down_qps = scale_down_qps
self.p99_thresh = p99_threshold
self.warmup = warmup_time
self.current = min_nodes
self.history = deque(maxlen=60) # 60 分钟历史
self.node_ready_time = {} # 节点启动时间
def evaluate(self, metrics):
"""评估伸缩决策"""
self.history.append(metrics)
# 1. 反应式: 当前指标超阈值
if metrics['qps'] > self.up_qps or metrics['p99'] > self.p99_thresh:
return self._scale_up()
if metrics['qps'] < self.down_qps and metrics['p99'] < self.p99_thresh * 0.5:
# 持续低负载 5 分钟才缩容
if len(self.history) >= 5 and all(
h['qps'] < self.down_qps for h in list(self.history)[-5:]):
return self._scale_down()
# 2. 预测式: 历史趋势预测
predicted = self._predict_next()
if predicted['qps'] > self.up_qps * 1.2:
return self._scale_up()
return None
def _predict_next(self):
"""基于历史预测下一分钟"""
if len(self.history) < 10:
return {'qps': 0}
# 简化: 用最近 10 分钟平均+趋势
recent = list(self.history)[-10:]
avg_qps = sum(h['qps'] for h in recent) / 10
trend = recent[-1]['qps'] - recent[0]['qps']
return {'qps': avg_qps + trend}
def _scale_up(self):
target = min(self.current + 2, self.max) # 一次扩 2 个
if target > self.current:
self._launch_nodes(target - self.current)
self.current = target
return 'scale_up'
return None
def _scale_down(self):
target = max(self.current - 1, self.min)
if target < self.current:
self._terminate_nodes(self.current - target)
self.current = target
return 'scale_down'
return None
def _launch_nodes(self, n):
"""启动新节点 (需 warmup)"""
for i in range(n):
node_id = f'node-{time.time()}-{i}'
self.node_ready_time[node_id] = time.time() + self.warmup
# 实际调用 K8s/云 SDK 启动 GPU 实例
def is_node_ready(self, node_id):
"""节点是否就绪 (过了 warmup)"""
return time.time() > self.node_ready_time.get(node_id, 0)
# 量化: 预测式扩容使扩容延迟从 60s 降至 0s (提前启动)
# 资源利用率从 30% 升至 70%
# GPU 成本降 40% (低峰缩容)
python
# 来源：伸缩指标采集 / 生产实践 2024
class MetricsCollector:
"""伸缩指标采集器"""
def __init__(self, prometheus_url):
self.url = prometheus_url
def collect(self):
"""采集伸缩决策指标"""
return {
'qps': self._query('rate(inference_requests_total[1m])'),
'p99': self._query('histogram_quantile(0.99, inference_latency)'),
'gpu_util': self._query('avg(DCgm_gpu_utilization)'),
'gpu_mem': self._query('avg(DCGM_fb_used / DCGM_fb_total)'),
'queue_depth': self._query('inference_queue_size'),
'tokens_per_sec': self._query('rate(generated_tokens_total[1m])'),
}
def _query(self, query):
# 查 Prometheus
return 50 # 占位
# 量化: 关键指标采样间隔 10s
# 队列深度是领先指标: 队列>10 即将过载
# GPU利用率滞后: 利用率>90%时已过载量化：预测式扩容使扩容延迟从 60s 降至 0s（提前启动节点），资源利用率从 30% 升至 70%，GPU 成本降 40%。队列深度是过载领先指标------队列>10 即将过载，GPU 利用率滞后（>90% 时已过载）。缩容需持续低负载 5 分钟，避免抖动。
边界：GPU 节点启动慢------竞价实例更慢（2-5 分钟），需预留 buffer。模型加载需 30-60s------伸缩决策需提前 1-2 分钟。跨可用区伸缩受 GPU 配额限制------热门 GPU 型号常缺货，需多机型 fallback。
5. 多模型混部：GPU 资源共享与隔离
生产环境常需同集群服务多模型（不同尺寸/不同任务）。多模型混部需解决 GPU 共享、显存隔离、请求路由、冷启动优化四个问题。
#mermaid-svg-Hv78rIQV7wwcFNRA{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Hv78rIQV7wwcFNRA .error-icon{fill:#552222;}#mermaid-svg-Hv78rIQV7wwcFNRA .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Hv78rIQV7wwcFNRA .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Hv78rIQV7wwcFNRA .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Hv78rIQV7wwcFNRA .marker.cross{stroke:#333333;}#mermaid-svg-Hv78rIQV7wwcFNRA svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Hv78rIQV7wwcFNRA p{margin:0;}#mermaid-svg-Hv78rIQV7wwcFNRA .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-Hv78rIQV7wwcFNRA .cluster-label text{fill:#333;}#mermaid-svg-Hv78rIQV7wwcFNRA .cluster-label span{color:#333;}#mermaid-svg-Hv78rIQV7wwcFNRA .cluster-label span p{background-color:transparent;}#mermaid-svg-Hv78rIQV7wwcFNRA .label text,#mermaid-svg-Hv78rIQV7wwcFNRA span{fill:#333;color:#333;}#mermaid-svg-Hv78rIQV7wwcFNRA .node rect,#mermaid-svg-Hv78rIQV7wwcFNRA .node circle,#mermaid-svg-Hv78rIQV7wwcFNRA .node ellipse,#mermaid-svg-Hv78rIQV7wwcFNRA .node polygon,#mermaid-svg-Hv78rIQV7wwcFNRA .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Hv78rIQV7wwcFNRA .rough-node .label text,#mermaid-svg-Hv78rIQV7wwcFNRA .node .label text,#mermaid-svg-Hv78rIQV7wwcFNRA .image-shape .label,#mermaid-svg-Hv78rIQV7wwcFNRA .icon-shape .label{text-anchor:middle;}#mermaid-svg-Hv78rIQV7wwcFNRA .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Hv78rIQV7wwcFNRA .rough-node .label,#mermaid-svg-Hv78rIQV7wwcFNRA .node .label,#mermaid-svg-Hv78rIQV7wwcFNRA .image-shape .label,#mermaid-svg-Hv78rIQV7wwcFNRA .icon-shape .label{text-align:center;}#mermaid-svg-Hv78rIQV7wwcFNRA .node.clickable{cursor:pointer;}#mermaid-svg-Hv78rIQV7wwcFNRA .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Hv78rIQV7wwcFNRA .arrowheadPath{fill:#333333;}#mermaid-svg-Hv78rIQV7wwcFNRA .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Hv78rIQV7wwcFNRA .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Hv78rIQV7wwcFNRA .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Hv78rIQV7wwcFNRA .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Hv78rIQV7wwcFNRA .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Hv78rIQV7wwcFNRA .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Hv78rIQV7wwcFNRA .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Hv78rIQV7wwcFNRA .cluster text{fill:#333;}#mermaid-svg-Hv78rIQV7wwcFNRA .cluster span{color:#333;}#mermaid-svg-Hv78rIQV7wwcFNRA div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Hv78rIQV7wwcFNRA .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Hv78rIQV7wwcFNRA rect.text{fill:none;stroke-width:0;}#mermaid-svg-Hv78rIQV7wwcFNRA .icon-shape,#mermaid-svg-Hv78rIQV7wwcFNRA .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Hv78rIQV7wwcFNRA .icon-shape p,#mermaid-svg-Hv78rIQV7wwcFNRA .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Hv78rIQV7wwcFNRA .icon-shape .label rect,#mermaid-svg-Hv78rIQV7wwcFNRA .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Hv78rIQV7wwcFNRA .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Hv78rIQV7wwcFNRA .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Hv78rIQV7wwcFNRA :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-Hv78rIQV7wwcFNRA .default>*{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-Hv78rIQV7wwcFNRA .default span{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-Hv78rIQV7wwcFNRA .default tspan{fill:#000000!important;} 多模型混部
GPU共享: MPS/MIG
显存隔离: 静态分配
请求路由: 模型感知
冷启动: 预加载
MPS: 多进程共享
MIG: 硬件分区
每模型预留显存上限
按模型名路由后端
常驻热模型+冷备
python
# 来源：多模型混部调度器 / 生产实践 2024
class MultiModelScheduler:
"""多模型混部调度器"""
def __init__(self, gpu_nodes, model_configs):
self.nodes = gpu_nodes # [{'id', 'gpu_mem', 'models_loaded'}]
self.configs = model_configs # {'llama-7b': {'size_gb': 14, 'qps': 50}, ...}
self.hot_models = set() # 常驻热模型
self.cold_models = {} # 冷模型+最后访问时间
def route(self, request):
"""按模型路由请求"""
model = request['model']
# 1. 找已加载该模型的节点
loaded = [n for n in self.nodes if model in n['models_loaded']]
if loaded:
self.hot_models.add(model)
return self._select_least_load(loaded)
# 2. 未加载, 触发加载
node = self._find_available_node(model)
if node:
self._load_model(node, model)
return node
# 3. 无可用节点, 排队
return self._queue_request(request)
def _find_available_node(self, model):
"""找有足够显存加载模型的节点"""
needed = self.configs[model]['size_gb']
for node in self.nodes:
free = node['gpu_mem'] - sum(
self.configs[m]['size_gb'] for m in node['models_loaded'])
if free >= needed:
return node
return None
def _load_model(self, node, model):
"""加载模型到节点 (耗时 30-60s)"""
node['models_loaded'].append(model)
# 实际调用推理引擎加载 API
def _select_least_load(self, nodes):
return min(nodes, key=lambda n: len(n['models_loaded']))
def evict_cold_model(self, threshold_min=30):
"""淘汰 30 分钟未访问的冷模型释放显存"""
import time
now = time.time()
for model, last_access in list(self.cold_models.items()):
if now - last_access > threshold_min * 60:
for node in self.nodes:
if model in node['models_loaded']:
self._unload(node, model)
# 量化: 多模型混部使 GPU 利用率从 40% 升至 75%
# 热模型常驻 (7B+13B), 冷模型按需加载
# 冷启动 30-60s, 需预加载高频模型
python
# 来源：GPU 共享方案对比 / 生产实践 2024
# 方案1: MPS (Multi-Process Service)
# - 多进程共享 GPU, 软件隔离
# - 优点: 灵活, 显存动态分配
# - 缺点: 故障隔离弱 (一进程崩溃影响全GPU)
# - 适合: 同团队多模型
# 方案2: MIG (Multi-Instance GPU)
# - 硬件分区, 各实例独立
# - 优点: 强隔离, 故障不传播
# - 缺点: 分区固定, 不灵活
# - 适合: 多租户/多团队
# 方案3: 时间分片 (默认)
# - GPU 串行执行多进程
# - 优点: 简单
# - 缺点: 上下文切换开销, 延迟波动
# - 适合: 低并发场景
class GPUSharingManager:
"""GPU 共享管理器"""
def __init__(self, strategy='mps'):
self.strategy = strategy
def allocate(self, model, gpu_id):
"""分配 GPU 资源给模型"""
if self.strategy == 'mps':
return self._alloc_mps(model, gpu_id)
elif self.strategy == 'mig':
return self._alloc_mig(model, gpu_id)
return self._alloc_timeslice(model, gpu_id)
def _alloc_mps(self, model, gpu_id):
# MPS: 设置显存上限
return {'gpu': gpu_id, 'mem_limit': '14GB', 'mode': 'mps'}
def _alloc_mig(self, model, gpu_id):
# MIG: 切分硬件实例 (A100 支持 7 个实例)
return {'gpu': gpu_id, 'mig_slice': '1g.5gb', 'mode': 'mig'}
# 量化 (A100 80GB):
# MPS: 7B+13B+7B 共享, 显存动态, 利用率 85%
# MIG: 7 个 1g.10gb 实例, 强隔离, 利用率 70%
# 时间分片: 利用率 60%, 延迟波动 2-3 倍量化：多模型混部使 GPU 利用率从 40% 升至 75%。MPS 共享显存动态分配利用率 85% 但隔离弱，MIG 硬件分区利用率 70% 但强隔离。冷模型 30 分钟淘汰释放显存。冷启动 30-60s 需预加载高频模型------低频模型按需加载接受延迟。
边界：MPS 故障隔离弱------一进程 OOM 影响全 GPU 进程，不适合多租户。MIG 分区固定------A100 最多 7 个 1g.10gb 实例，大模型（70B）无法用 MIG。时间分片延迟波动大------不适合 SLA 严格场景。多模型混部需监控各模型显存------避免单模型 OOM 影响其他。
6. 边界与失败模式
推理系统架构失败模式集中在单点故障、资源耗尽、级联崩溃三类。
#mermaid-svg-8trmnVaobQJzF8gr{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-8trmnVaobQJzF8gr .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-8trmnVaobQJzF8gr .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-8trmnVaobQJzF8gr .error-icon{fill:#552222;}#mermaid-svg-8trmnVaobQJzF8gr .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-8trmnVaobQJzF8gr .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-8trmnVaobQJzF8gr .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-8trmnVaobQJzF8gr .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-8trmnVaobQJzF8gr .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-8trmnVaobQJzF8gr .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-8trmnVaobQJzF8gr .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-8trmnVaobQJzF8gr .marker{fill:#333333;stroke:#333333;}#mermaid-svg-8trmnVaobQJzF8gr .marker.cross{stroke:#333333;}#mermaid-svg-8trmnVaobQJzF8gr svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-8trmnVaobQJzF8gr p{margin:0;}#mermaid-svg-8trmnVaobQJzF8gr .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-8trmnVaobQJzF8gr .cluster-label text{fill:#333;}#mermaid-svg-8trmnVaobQJzF8gr .cluster-label span{color:#333;}#mermaid-svg-8trmnVaobQJzF8gr .cluster-label span p{background-color:transparent;}#mermaid-svg-8trmnVaobQJzF8gr .label text,#mermaid-svg-8trmnVaobQJzF8gr span{fill:#333;color:#333;}#mermaid-svg-8trmnVaobQJzF8gr .node rect,#mermaid-svg-8trmnVaobQJzF8gr .node circle,#mermaid-svg-8trmnVaobQJzF8gr .node ellipse,#mermaid-svg-8trmnVaobQJzF8gr .node polygon,#mermaid-svg-8trmnVaobQJzF8gr .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-8trmnVaobQJzF8gr .rough-node .label text,#mermaid-svg-8trmnVaobQJzF8gr .node .label text,#mermaid-svg-8trmnVaobQJzF8gr .image-shape .label,#mermaid-svg-8trmnVaobQJzF8gr .icon-shape .label{text-anchor:middle;}#mermaid-svg-8trmnVaobQJzF8gr .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-8trmnVaobQJzF8gr .rough-node .label,#mermaid-svg-8trmnVaobQJzF8gr .node .label,#mermaid-svg-8trmnVaobQJzF8gr .image-shape .label,#mermaid-svg-8trmnVaobQJzF8gr .icon-shape .label{text-align:center;}#mermaid-svg-8trmnVaobQJzF8gr .node.clickable{cursor:pointer;}#mermaid-svg-8trmnVaobQJzF8gr .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-8trmnVaobQJzF8gr .arrowheadPath{fill:#333333;}#mermaid-svg-8trmnVaobQJzF8gr .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-8trmnVaobQJzF8gr .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-8trmnVaobQJzF8gr .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8trmnVaobQJzF8gr .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-8trmnVaobQJzF8gr .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8trmnVaobQJzF8gr .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-8trmnVaobQJzF8gr .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-8trmnVaobQJzF8gr .cluster text{fill:#333;}#mermaid-svg-8trmnVaobQJzF8gr .cluster span{color:#333;}#mermaid-svg-8trmnVaobQJzF8gr div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-8trmnVaobQJzF8gr .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-8trmnVaobQJzF8gr rect.text{fill:none;stroke-width:0;}#mermaid-svg-8trmnVaobQJzF8gr .icon-shape,#mermaid-svg-8trmnVaobQJzF8gr .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-8trmnVaobQJzF8gr .icon-shape p,#mermaid-svg-8trmnVaobQJzF8gr .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-8trmnVaobQJzF8gr .icon-shape .label rect,#mermaid-svg-8trmnVaobQJzF8gr .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-8trmnVaobQJzF8gr .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-8trmnVaobQJzF8gr .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-8trmnVaobQJzF8gr :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-8trmnVaobQJzF8gr .default>*{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-8trmnVaobQJzF8gr .default span{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-8trmnVaobQJzF8gr .default tspan{fill:#000000!important;} 架构失败模式
单点故障
资源耗尽
级联崩溃
网关单点
DNS单点
显存溢出
连接数耗尽
重试风暴
熔断失效
网关多副本+VIP
显存监控+主动限流
退避重试+熔断
python
# 来源：级联崩溃防护 / 生产实践 2024
class CascadeFailureGuard:
"""级联崩溃防护器"""
def __init__(self, max_retries=2, backoff_base=0.1):
self.max_retries = max_retries
self.backoff = backoff_base
async def call_with_guard(self, func, *args):
"""带级联防护的调用"""
for attempt in range(self.max_retries + 1):
try:
return await func(*args)
except Exception as e:
if attempt == self.max_retries:
raise
# 指数退避: 避免重试风暴
wait = self.backoff * (2 ** attempt)
await asyncio.sleep(wait)
raise Exception('unreachable')
async def shed_load(self, current_qps, max_qps):
"""负载脱落: 过载时拒绝低优先级请求"""
if current_qps > max_qps * 0.9:
# 接近过载, 拒绝免费用户
return 'reject_free'
if current_qps > max_qps:
# 已过载, 仅服务付费用户
return 'reject_all_except_paid'
return 'accept_all'
# 量化: 退避重试避免重试风暴 (重试量降 80%)
# 负载脱落使核心用户可用性保持 99.9%
# 无防护: 过载致全部用户不可用实战复盘：某推理服务网关单点故障致全站不可用 30 分钟。排查发现网关仅部署单副本，进程崩溃无自动重启。修复：网关多副本+VIP 漂移+进程守护（systemd restart on crash）。教训：所有控制面组件必须多副本，单点是可用性最大威胁。
实战复盘：某服务高峰期出现级联崩溃------后端慢致客户端超时重试，重试加剧后端压力，雪崩。诊断发现客户端重试无退避（固定 1 秒），高峰期重试量达原始请求 3 倍。修复：客户端指数退避（0.1s/0.2s/0.4s）+服务端熔断。教训：重试必须有退避，否则放大流量致雪崩。
7. 架构演进趋势：从单体到异构协同
推理系统架构正从单体GPU集群向异构算力协同、边缘-云协同、Serverless化演进。
#mermaid-svg-22N9zssHmtN0UwUs{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-22N9zssHmtN0UwUs .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-22N9zssHmtN0UwUs .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-22N9zssHmtN0UwUs .error-icon{fill:#552222;}#mermaid-svg-22N9zssHmtN0UwUs .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-22N9zssHmtN0UwUs .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-22N9zssHmtN0UwUs .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-22N9zssHmtN0UwUs .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-22N9zssHmtN0UwUs .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-22N9zssHmtN0UwUs .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-22N9zssHmtN0UwUs .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-22N9zssHmtN0UwUs .marker{fill:#333333;stroke:#333333;}#mermaid-svg-22N9zssHmtN0UwUs .marker.cross{stroke:#333333;}#mermaid-svg-22N9zssHmtN0UwUs svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-22N9zssHmtN0UwUs p{margin:0;}#mermaid-svg-22N9zssHmtN0UwUs .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-22N9zssHmtN0UwUs .cluster-label text{fill:#333;}#mermaid-svg-22N9zssHmtN0UwUs .cluster-label span{color:#333;}#mermaid-svg-22N9zssHmtN0UwUs .cluster-label span p{background-color:transparent;}#mermaid-svg-22N9zssHmtN0UwUs .label text,#mermaid-svg-22N9zssHmtN0UwUs span{fill:#333;color:#333;}#mermaid-svg-22N9zssHmtN0UwUs .node rect,#mermaid-svg-22N9zssHmtN0UwUs .node circle,#mermaid-svg-22N9zssHmtN0UwUs .node ellipse,#mermaid-svg-22N9zssHmtN0UwUs .node polygon,#mermaid-svg-22N9zssHmtN0UwUs .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-22N9zssHmtN0UwUs .rough-node .label text,#mermaid-svg-22N9zssHmtN0UwUs .node .label text,#mermaid-svg-22N9zssHmtN0UwUs .image-shape .label,#mermaid-svg-22N9zssHmtN0UwUs .icon-shape .label{text-anchor:middle;}#mermaid-svg-22N9zssHmtN0UwUs .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-22N9zssHmtN0UwUs .rough-node .label,#mermaid-svg-22N9zssHmtN0UwUs .node .label,#mermaid-svg-22N9zssHmtN0UwUs .image-shape .label,#mermaid-svg-22N9zssHmtN0UwUs .icon-shape .label{text-align:center;}#mermaid-svg-22N9zssHmtN0UwUs .node.clickable{cursor:pointer;}#mermaid-svg-22N9zssHmtN0UwUs .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-22N9zssHmtN0UwUs .arrowheadPath{fill:#333333;}#mermaid-svg-22N9zssHmtN0UwUs .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-22N9zssHmtN0UwUs .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-22N9zssHmtN0UwUs .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-22N9zssHmtN0UwUs .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-22N9zssHmtN0UwUs .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-22N9zssHmtN0UwUs .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-22N9zssHmtN0UwUs .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-22N9zssHmtN0UwUs .cluster text{fill:#333;}#mermaid-svg-22N9zssHmtN0UwUs .cluster span{color:#333;}#mermaid-svg-22N9zssHmtN0UwUs div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-22N9zssHmtN0UwUs .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-22N9zssHmtN0UwUs rect.text{fill:none;stroke-width:0;}#mermaid-svg-22N9zssHmtN0UwUs .icon-shape,#mermaid-svg-22N9zssHmtN0UwUs .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-22N9zssHmtN0UwUs .icon-shape p,#mermaid-svg-22N9zssHmtN0UwUs .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-22N9zssHmtN0UwUs .icon-shape .label rect,#mermaid-svg-22N9zssHmtN0UwUs .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-22N9zssHmtN0UwUs .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-22N9zssHmtN0UwUs .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-22N9zssHmtN0UwUs :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-22N9zssHmtN0UwUs .default>*{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-22N9zssHmtN0UwUs .default span{fill:#faf9f5!important;stroke:#ffffff!important;color:#000000!important;stroke-width:0px!important;}#mermaid-svg-22N9zssHmtN0UwUs .default tspan{fill:#000000!important;} 架构演进
异构算力: GPU+NPU+CPU协同
边缘-云协同: 就近推理
Serverless: 按需弹性
disaggregated: 存算分离
python
# 来源：异构算力调度 / 生产实践 2024
class HeterogeneousScheduler:
"""异构算力调度器"""
def __init__(self):
self.pools = {
'gpu_h100': {'strength': '大模型推理', 'cost': 5},
'gpu_a10': {'strength': '中等模型', 'cost': 1.5},
'npu': {'strength': '特定模型', 'cost': 1},
'cpu': {'strength': '小模型/边缘', 'cost': 0.2},
}
def route(self, request, model_size):
# 按模型规模+延迟要求路由到合适算力
if model_size > 30: return 'gpu_h100'
if model_size > 7: return 'gpu_a10'
if request.get('edge'): return 'cpu'
return 'gpu_a10'
# 量化: 架构演进方向
# 异构算力: 成本降 30-50% (按需选算力)
# 边缘-云: 延迟降 60-80% (就近推理)
# Serverless: 冷启动 <1s + 按调用付费
# 关键: 异构调度是降本核心量化：异构算力成本降30-50%（按需选算力），边缘-云协同延迟降60-80%（就近推理），Serverless冷启动<1s+按调用付费。异构调度是降本核心。
边界：异构算力编程复杂------需统一抽象层。边缘-云协同有数据同步成本。Serverless冷启动仍难零延迟。存算分离网络带宽成瓶颈。架构演进需渐进------非一步到位。
总结
推理系统架构核心在于网关、负载均衡、高可用、弹性伸缩、容灾五点。网关多维度限流防资源耗尽，前缀感知负载均衡使缓存命中率 85%，双机房容灾使可用性 99.99%，预测式伸缩使扩容延迟 0s 且成本降 40%，退避重试+熔断防级联崩溃。架构设计原则：消除单点、预留 buffer、降级而非报错、退避而非重试。控制面组件必须多副本，数据面需显存监控主动限流。