面试场景
面试官说："你的 RAG 系统回答错了。检索明明返回了相关文档，为什么还是答错？"
如果你只回答"检索不好"或"模型幻觉"，就落入了陷阱。面试官真正在听的是：你能不能把"答错"拆解到具体的流水线环节，并说出每种故障的可观测信号。
这就是 RAG 面试中最高频的考察方式------级联失效归因。
底层知识：RAG 不是单一模型，而是级联系统
一个完整的 RAG 流水线包含五个串行环节：
用户查询 → [查询改写] → [检索] → [重排] → [上下文组装] → [生成] → 回答每个环节的输出是下一个环节的输入，错误会向下传播且不可逆。这意味着：
- 级联天花板效应：每个环节的输出质量是下游环节的质量上限
- 错误放大效应：上游的小偏差可能在下游被放大
- 饱和掩盖效应：一个环节的严重故障可能掩盖另一个环节的影响
只看端到端准确率，等于把五个环节的问题混在一起。这是面试中最常见的"新手陷阱"。
各环节的职能与典型失效模式
| 环节 | 核心功能 | 典型失效模式 | 可观测信号 |
|---|---|---|---|
| 查询改写 | 将用户查询转为检索友好形式 | 丢失关键约束词、过度扩展引入噪声 | 改写前后召回对比 |
| 检索 | 从文档库召回相关片段 | top-k 过低导致关键 chunk 未召回；噪声淹没 | Recall@K、NDCG@K |
| 重排 | 对召回结果重新排序 | 排序模型失效导致关键 chunk 排名下降 | NDCG@K、MRR |
| 上下文组装 | 将检索结果拼装为上下文 | 关键 chunk 被截断、排列顺序导致 Lost in the Middle | 上下文覆盖率 |
| 生成 | 基于上下文生成回答 | 模型幻觉、忽略证据、过度推断 | Faithfulness、Answer Relevancy |
Python 实验：量化各环节的失效影响
为了真正理解级联失效，我编写了一个模拟 RAG 流水线的 Python 脚本，在每个环节注入不同类型的故障，观察端到端性能的变化。
实验设计
- 构建文档库：30 篇技术文档，覆盖 10 个主题领域
- 构建查询集：30 条查询，每条有明确的期望主题
- 基线测试：无故障时的准确率和回答质量
- 单环节故障注入：每次只在一个环节注入故障
- 组合故障注入：同时在多个环节注入故障
- 归因分析：统计失败 case 的环节归属
完整实验代码
python
#!/usr/bin/env python3
"""
RAG 级联失效分析实验
模拟 RAG 流水线的五个环节，在每个环节注入不同类型的故障，
观察错误如何沿级联向下传播，并量化每个环节的失效影响。
"""
import random
import numpy as np
from collections import defaultdict
from dataclasses import dataclass, field
from typing import List, Dict, Tuple
random.seed(42)
np.random.seed(42)
@dataclass
class Document:
doc_id: int
topic: str
keywords: List[str]
content: str
@dataclass
class Query:
qid: int
text: str
expected_topic: str
expected_keywords: List[str]
# 文档库：10 个主题 × 3 个文档 = 30 篇
TOPICS = [
("机器学习", ["梯度下降", "反向传播", "损失函数", "过拟合", "正则化"]),
("自然语言处理", ["分词", "词向量", "注意力机制", "Transformer", "BERT"]),
("计算机视觉", ["卷积", "目标检测", "图像分割", "特征提取", "ResNet"]),
("强化学习", ["奖励函数", "策略梯度", "Q-learning", "探索利用", "马尔可夫"]),
("推荐系统", ["协同过滤", "矩阵分解", "冷启动", "隐式反馈", "召回排序"]),
("知识图谱", ["实体识别", "关系抽取", "三元组", "图嵌入", "TransE"]),
("信息检索", ["BM25", "倒排索引", "TF-IDF", "召回率", "精确率"]),
("分布式系统", ["一致性", "CAP定理", "Raft协议", "分布式事务", "数据分片"]),
("数据库", ["B+树索引", "事务ACID", "MVCC", "查询优化", "连接池"]),
("微服务", ["服务发现", "负载均衡", "熔断降级", "链路追踪", "API网关"]),
]
DOCUMENTS = [
Document(i, topic, kw, f"本文介绍{topic}的核心概念，包括{'、'.join(kw)}等关键技术。")
for i, (topic, kw) in enumerate(TOPICS)
]
# 每个主题补充 2 个子文档
for base_id in range(10):
topic, base_kw = TOPICS[base_id]
for sub in range(2):
sub_kws = random.sample(base_kw, 3)
doc_id = 10 + base_id * 2 + sub
DOCUMENTS.append(Document(
doc_id, topic, sub_kws,
f"深入探讨{topic}中的{'、'.join(sub_kws)}等高级主题。"
))
# 30 条查询
QUERIES = [
Query(0, "如何理解深度学习中的梯度消失问题", "机器学习", ["梯度下降", "反向传播"]),
Query(1, "Transformer的注意力机制是怎么工作", "自然语言处理", ["注意力机制", "Transformer"]),
# ... (共30条，完整代码见 experiment_cascade_failure.py)
]
class RAGPipeline:
"""模拟 RAG 级联系统，每个环节输出质量评分"""
def __init__(self, documents, failure_config=None):
self.documents = documents
self.fc = failure_config or {}
self.stages = []
def _compute_relevance(self, doc, query):
"""计算文档与查询的真实相关性"""
score = 0.0
if doc.topic == query.expected_topic:
score += 5.0
for kw in doc.keywords:
if kw in query.expected_keywords:
score += 2.0
return score
def stage_rewrite(self, query):
"""环节1：查询改写"""
text = query.text
failure = self.fc.get("rewrite")
if failure == "drop_keywords":
# 故障：丢失 40% 的关键字符
words = list(text)
drop_count = max(1, len(words) // 5)
drop_indices = random.sample(range(len(words)), drop_count)
rewritten = "".join([c for i, c in enumerate(words) if i not in drop_indices])
quality = 0.55
self.stages.append({"name": "rewrite", "status": "FAIL", "quality": quality})
return rewritten, quality
# ... 其他故障类型和正常路径
return text, 0.95
def stage_retrieve(self, rewritten, query, rewrite_quality, top_k=5):
"""环节2：检索（受改写质量影响）"""
# ... 模拟检索逻辑
pass
def stage_rerank(self, retrieved, query, retrieve_quality):
"""环节3：重排"""
# ... 模拟重排逻辑
pass
def stage_assemble(self, reranked, query, rerank_quality):
"""环节4：上下文组装"""
# ... 模拟上下文组装
pass
def stage_generate(self, context, query, context_quality):
"""环节5：生成回答"""
# ... 模拟生成逻辑
pass
def run(self, query):
"""执行完整流水线"""
self.stages = []
rewritten, rq = self.stage_rewrite(query)
retrieved, retq = self.stage_retrieve(rewritten, query, rq)
reranked, rrq = self.stage_rerank(retrieved, query, retq)
context, aq = self.stage_assemble(reranked, query, rrq)
answer, gq, correct = self.stage_generate(context, query, aq)
return {"correct": correct, "answer_quality": gq, "stages": self.stages}完整可运行代码保存在
experiment_cascade_failure.py中。
实验结果
基线测试
查询总数: 30
基线准确率: 100.0%
基线平均质量: 0.950单环节故障注入
| 故障场景 | 准确率 | 下降 | 平均质量 | 质量降幅 |
|---|---|---|---|---|
| 查询改写-丢失关键词 | 100.0% | 0.0% | 0.906 | 0.044 |
| 查询改写-过度扩展 | 100.0% | 0.0% | 0.950 | 0.000 |
| 检索-top_k过低 | 100.0% | 0.0% | 0.549 | 0.401 |
| 检索-噪声淹没 | 96.7% | 3.3% | 0.465 | 0.485 |
| 重排-随机打乱 | 100.0% | 0.0% | 0.443 | 0.507 |
| 重排-长度偏差 | 100.0% | 0.0% | 0.494 | 0.456 |
| 上下文组装-截断 | 100.0% | 0.0% | 0.344 | 0.606 |
| 上下文组装-Lost in Middle | 100.0% | 0.0% | 0.564 | 0.386 |
| 生成-模型幻觉 | 0.0% | 100.0% | 0.100 | 0.850 |
| 生成-忽略证据 | 0.0% | 100.0% | 0.200 | 0.750 |
组合故障（级联放大）
| 组合场景 | 准确率 | 下降 | 各环节drop和 | 级联效应 |
|---|---|---|---|---|
| 改写丢失 + 检索噪声 | 90.0% | 10.0% | 3.3% | 放大 |
| 改写 + 检索 + 组装 | 93.3% | 6.7% | 3.3% | 放大 |
| 全环节故障 | 0.0% | 100.0% | 103.3% | 饱和 |
典型失败 case 的逐环节 trace
查询 0: 如何理解深度学习中的梯度消失问题
期望主题: 机器学习 结果: [FAIL] 质量: 0.100
!! [rewrite ] Q=0.55 丢失关键字符，改写质量降至55%
!! [retrieve ] Q=0.45 噪声淹没，排序信号被干扰
!! [rerank ] Q=0.23 随机打乱排序，质量降至22%
!! [assemble ] Q=0.09 过度截断，仅保留2个关键词
!! [generate ] Q=0.10 模型幻觉，忽略上下文注意质量分的逐级衰减：0.55 → 0.45 → 0.23 → 0.09 → 0.10。这正是级联天花板效应的体现------每个环节的输出质量被上游限制。
数据分析与深入洞察
洞察 1：准确率 vs 质量分------两个维度的信息
实验揭示了一个有趣的现象：很多故障场景下准确率仍然是 100%，但质量分已经大幅下降。
- 上下文组装截断：准确率不变，质量分从 0.950 降至 0.344（下降 64%）
- 重排随机打乱：准确率不变，质量分从 0.950 降至 0.443（下降 53%）
这说明端到端准确率是一个粗粒度指标------它能捕捉"完全答错"，但捕捉"答对但质量下降"。在实际系统中，质量分的下降往往是准确率下降的前兆。
洞察 2：生成环节的"一票否决"特性
生成环节的幻觉直接导致 100% 的准确率下降，即使上游四个环节完全正常。这揭示了一个工程事实：生成环节是 RAG 系统中风险最高的单点故障。
但在实际系统中，生成幻觉往往不是独立发生的------它可能是上游环节质量下降的累积结果。这就是为什么需要级联分析而非单点排查。
洞察 3：级联放大的条件
"改写丢失 + 检索噪声"组合的下降幅度（10%）大于各环节单独下降之和（3.3%），展现了级联放大效应。原因是：改写质量下降后，检索的输入信号变弱，此时叠加检索自身的噪声，两路弱信号叠加导致排序严重偏离。
面试回答模板
当面试官问"RAG 回答错了，你怎么排查"时，一个高分回答结构：
"我会把 RAG 拆成五个环节逐层排查：
- 先检查查询改写------改写后的查询是否保留了原始意图的关键约束
- 再看检索------Recall@K 是否覆盖正确答案的文档
- 然后看重排------相关文档是否排在 top 位置
- 再看上下文组装------关键 chunk 是否被截断或放到了模型的注意力盲区
- 最后看生成------模型是否忠实使用了上下文中的证据
在我们的项目中，通过分析 20+ 个失败 case，发现 40% 的问题出在上下文组装环节------top-k 设大了之后，关键 chunk 被放到了中间位置，LLM 的 Lost in the Middle 效应导致它更关注首尾的内容。"
实践建议
- 为你的 RAG 系统建立失败 case 标注表：每条记录包含原始查询、期望答案、实际答案、失败环节标签。积累 20+ 条后按环节统计分布
- 做消融实验：固定其他环节，只改变一个环节的参数，观察端到端指标变化
- 同时监控准确率和质量分：质量分的下降是准确率下降的预警信号
- 用 trace 而非只看结果：保存每个环节的中间输出，排查时逐层检查
本系列共 6 篇，下一篇：《混合检索的条件反转------"混合检索比纯向量好"这个结论什么时候不成立？》
完整Python代码
python
#!/usr/bin/env python3
"""
RAG 级联失效分析实验
模拟 RAG 流水线的五个环节，在每个环节注入不同类型的故障，
观察错误如何沿级联向下传播，并量化每个环节的失效影响。
设计要点：
- 每个环节的输出是下一个环节的输入，错误会向下传播
- 生成环节的评估不仅看上下文是否正确，还要看最终回答质量
- 使用多维评估：检索召回、上下文质量、回答正确性
"""
import random
import json
import numpy as np
from collections import defaultdict
from dataclasses import dataclass, field
from typing import List, Dict, Tuple
random.seed(42)
np.random.seed(42)
# ============================================================
# 1. 模拟文档库和查询集
# ============================================================
@dataclass
class Document:
doc_id: int
topic: str
keywords: List[str]
content: str
@dataclass
class Query:
qid: int
text: str
expected_topic: str
expected_keywords: List[str]
# 文档库
TOPICS = [
("机器学习", ["梯度下降", "反向传播", "损失函数", "过拟合", "正则化"]),
("自然语言处理", ["分词", "词向量", "注意力机制", "Transformer", "BERT"]),
("计算机视觉", ["卷积", "目标检测", "图像分割", "特征提取", "ResNet"]),
("强化学习", ["奖励函数", "策略梯度", "Q-learning", "探索利用", "马尔可夫"]),
("推荐系统", ["协同过滤", "矩阵分解", "冷启动", "隐式反馈", "召回排序"]),
("知识图谱", ["实体识别", "关系抽取", "三元组", "图嵌入", "TransE"]),
("信息检索", ["BM25", "倒排索引", "TF-IDF", "召回率", "精确率"]),
("分布式系统", ["一致性", "CAP定理", "Raft协议", "分布式事务", "数据分片"]),
("数据库", ["B+树索引", "事务ACID", "MVCC", "查询优化", "连接池"]),
("微服务", ["服务发现", "负载均衡", "熔断降级", "链路追踪", "API网关"]),
]
DOCUMENTS = [
Document(i, topic, kw, f"本文介绍{topic}的核心概念，包括{'、'.join(kw)}等关键技术。")
for i, (topic, kw) in enumerate(TOPICS)
]
# 每个主题有 3 个相关文档（不同侧重点），共 30 个文档
for base_id in range(10):
topic, base_kw = TOPICS[base_id]
for sub in range(2):
sub_kws = random.sample(base_kw, 3)
doc_id = 10 + base_id * 2 + sub
DOCUMENTS.append(Document(
doc_id, topic, sub_kws,
f"深入探讨{topic}中的{'、'.join(sub_kws)}等高级主题。"
))
# 查询集
QUERIES = [
Query(0, "如何理解深度学习中的梯度消失问题", "机器学习", ["梯度下降", "反向传播"]),
Query(1, "Transformer的注意力机制是怎么工作", "自然语言处理", ["注意力机制", "Transformer"]),
Query(2, "YOLO目标检测的核心原理", "计算机视觉", ["目标检测", "特征提取"]),
Query(3, "Q-learning和策略梯度有什么区别", "强化学习", ["策略梯度", "Q-learning"]),
Query(4, "推荐系统冷启动问题怎么解决", "推荐系统", ["冷启动", "协同过滤"]),
Query(5, "知识图谱中的关系抽取方法有哪些", "知识图谱", ["关系抽取", "三元组"]),
Query(6, "BM25检索算法的原理和实现", "信息检索", ["BM25", "TF-IDF"]),
Query(7, "分布式系统中CAP定理的含义", "分布式系统", ["CAP定理", "一致性"]),
Query(8, "数据库索引为什么用B+树", "数据库", ["B+树索引", "查询优化"]),
Query(9, "微服务架构中的熔断机制怎么实现", "微服务", ["熔断降级", "服务发现"]),
Query(10, "神经网络的反向传播算法详解", "机器学习", ["反向传播", "损失函数"]),
Query(11, "BERT模型的预训练任务是什么", "自然语言处理", ["BERT", "词向量"]),
Query(12, "图像分割的语义分割和实例分割区别", "计算机视觉", ["图像分割", "卷积"]),
Query(13, "强化学习中探索和利用的平衡策略", "强化学习", ["探索利用", "奖励函数"]),
Query(14, "协同过滤推荐算法的优缺点", "推荐系统", ["协同过滤", "矩阵分解"]),
Query(15, "图嵌入方法TransE的原理", "知识图谱", ["图嵌入", "TransE"]),
Query(16, "倒排索引的构建和查询流程", "信息检索", ["倒排索引", "召回率"]),
Query(17, "Raft一致性协议的选举过程", "分布式系统", ["Raft协议", "一致性"]),
Query(18, "MVCC多版本并发控制的实现原理", "数据库", ["MVCC", "事务ACID"]),
Query(19, "API网关在微服务中的负载均衡作用", "微服务", ["API网关", "负载均衡"]),
Query(20, "过拟合问题如何用正则化解决", "机器学习", ["过拟合", "正则化"]),
Query(21, "分词算法在NLP中的应用", "自然语言处理", ["分词", "词向量"]),
Query(22, "ResNet残差网络的设计思想", "计算机视觉", ["ResNet", "特征提取"]),
Query(23, "马尔可夫决策过程在强化学习中的角色", "强化学习", ["马尔可夫", "奖励函数"]),
Query(24, "隐式反馈在推荐系统中的处理方法", "推荐系统", ["隐式反馈", "召回排序"]),
Query(25, "实体识别在知识图谱构建中的作用", "知识图谱", ["实体识别", "关系抽取"]),
Query(26, "精确率和召回率的权衡关系", "信息检索", ["精确率", "召回率"]),
Query(27, "分布式事务的两阶段提交协议", "分布式系统", ["分布式事务", "一致性"]),
Query(28, "查询优化器如何选择执行计划", "数据库", ["查询优化", "B+树索引"]),
Query(29, "链路追踪在微服务调试中的价值", "微服务", ["链路追踪", "熔断降级"]),
]
# ============================================================
# 2. RAG 五环节模拟（含质量评分）
# ============================================================
@dataclass
class StageResult:
name: str
status: str # "OK" or "FAIL"
detail: str
quality: float # 0-1 质量分
class RAGPipeline:
"""模拟 RAG 级联系统，每个环节输出质量评分"""
def __init__(self, documents, failure_config=None):
self.documents = documents
self.fc = failure_config or {}
self.stages: List[StageResult] = []
def _compute_relevance(self, doc: Document, query: Query) -> float:
"""计算文档与查询的真实相关性"""
score = 0.0
if doc.topic == query.expected_topic:
score += 5.0
for kw in doc.keywords:
if kw in query.expected_keywords:
score += 2.0
elif any(kw_part in query.text for kw_part in kw.split()):
score += 0.5
return score
def stage_rewrite(self, query: Query) -> Tuple[str, float]:
"""环节1：查询改写"""
text = query.text
failure = self.fc.get("rewrite")
if failure == "drop_keywords":
# 故障：丢失 40% 的关键约束词
words = list(text)
drop_count = max(1, len(words) // 5)
drop_indices = random.sample(range(len(words)), drop_count)
rewritten = "".join([c for i, c in enumerate(words) if i not in drop_indices])
quality = 0.55 # 改写质量下降
self.stages.append(StageResult("rewrite", "FAIL",
f"丢失关键字符，改写质量降至{quality:.0%}", quality))
return rewritten, quality
elif failure == "over_expand":
# 故障：过度扩展引入噪声
noise_terms = [" 相关面试题汇总", " 最新技术趋势", " 入门到精通教程"]
rewritten = text + random.choice(noise_terms)
quality = 0.75
self.stages.append(StageResult("rewrite", "FAIL",
f"过度扩展引入噪声，质量{quality:.0%}", quality))
return rewritten, quality
self.stages.append(StageResult("rewrite", "OK", "正常改写", 0.95))
return text, 0.95
def stage_retrieve(self, rewritten_query: str, query: Query,
rewrite_quality: float, top_k: int = 5) -> Tuple[List[Document], float]:
"""环节2：检索"""
failure = self.fc.get("retrieve")
# 计算每个文档的相关性分数（受改写质量影响）
scored = []
for doc in self.documents:
true_rel = self._compute_relevance(doc, query)
# 改写质量影响分数精度
noise_scale = (1.0 - rewrite_quality) * 3.0
observed_score = true_rel * rewrite_quality + random.gauss(0, noise_scale)
scored.append((doc, observed_score))
if failure == "low_recall":
top_k = 2
self.stages.append(StageResult("retrieve", "FAIL",
f"top_k过低={top_k}，召回不足", 0.5))
elif failure == "noise_flood":
# 给所有文档加大幅随机噪声
scored = [(d, s + random.gauss(0, 4)) for d, s in scored]
self.stages.append(StageResult("retrieve", "FAIL",
"噪声淹没，排序信号被干扰", 0.45))
else:
self.stages.append(StageResult("retrieve", "OK",
f"正常检索 top_k={top_k}", 0.9))
scored.sort(key=lambda x: x[1], reverse=True)
retrieved = [d for d, _ in scored[:top_k]]
# 计算召回质量
relevant_retrieved = sum(1 for d in retrieved if d.topic == query.expected_topic)
total_relevant = sum(1 for d in self.documents if d.topic == query.expected_topic)
recall = relevant_retrieved / max(total_relevant, 1)
quality = min(recall, self.stages[-1].quality)
return retrieved, quality
def stage_rerank(self, retrieved: List[Document], query: Query,
retrieve_quality: float) -> Tuple[List[Document], float]:
"""环节3：重排"""
failure = self.fc.get("rerank")
if failure == "random_shuffle":
shuffled = retrieved[:]
random.shuffle(shuffled)
quality = retrieve_quality * 0.5
self.stages.append(StageResult("rerank", "FAIL",
f"随机打乱排序，质量降至{quality:.0%}", quality))
return shuffled, quality
elif failure == "length_bias":
# 按文档ID排序（与相关性无关）
sorted_docs = sorted(retrieved, key=lambda d: d.doc_id)
quality = retrieve_quality * 0.55
self.stages.append(StageResult("rerank", "FAIL",
f"长度偏差排序，质量降至{quality:.0%}", quality))
return sorted_docs, quality
# 正常重排：按真实相关性
reranked = sorted(retrieved,
key=lambda d: self._compute_relevance(d, query), reverse=True)
quality = min(retrieve_quality * 1.1, 1.0)
self.stages.append(StageResult("rerank", "OK", "正常重排", quality))
return reranked, quality
def stage_assemble(self, reranked: List[Document], query: Query,
rerank_quality: float) -> Tuple[str, float]:
"""环节4：上下文组装"""
failure = self.fc.get("assemble")
if failure == "truncate":
# 故障：只取第一个文档的部分关键词
if reranked:
doc = reranked[0]
context = f"{doc.topic}: {', '.join(doc.keywords[:2])}"
quality = rerank_quality * 0.4
self.stages.append(StageResult("assemble", "FAIL",
f"过度截断，仅保留{len(doc.keywords[:2])}个关键词", quality))
return context, quality
elif failure == "lost_in_middle":
# 故障：关键chunk被放到中间位置
if len(reranked) >= 3:
reordered = [reranked[-1]] + reranked[:-1] # 最佳结果被移到末尾
parts = [f"{d.topic}: {', '.join(d.keywords)}" for d in reordered]
context = " | ".join(parts)
quality = rerank_quality * 0.6
self.stages.append(StageResult("assemble", "FAIL",
"关键chunk位置不当（Lost in Middle）", quality))
return context, quality
# 正常组装
top_docs = reranked[:3]
parts = [f"{d.topic}: {', '.join(d.keywords)}" for d in top_docs]
context = " | ".join(parts)
quality = min(rerank_quality * 1.05, 1.0)
self.stages.append(StageResult("assemble", "OK",
f"组装{len(top_docs)}个chunk", quality))
return context, quality
def stage_generate(self, context: str, query: Query,
context_quality: float) -> Tuple[str, float, bool]:
"""环节5：生成回答，返回(回答, 质量, 是否正确)"""
failure = self.fc.get("generate")
# 检查上下文是否包含正确答案
has_correct_topic = query.expected_topic in context
has_correct_kw = any(kw in context for kw in query.expected_keywords)
if failure == "hallucination":
# 故障：忽略上下文，生成幻觉内容
answer = "根据最新研究，该领域最重要的突破是基于量子计算的分布式架构..."
quality = 0.1
correct = False
self.stages.append(StageResult("generate", "FAIL",
"模型幻觉，忽略上下文", quality))
return answer, quality, correct
elif failure == "ignore_evidence":
answer = "这个问题比较复杂，目前没有确定性的答案，需要进一步研究..."
quality = 0.2
correct = False
self.stages.append(StageResult("generate", "FAIL",
"拒绝回答/忽略证据", quality))
return answer, quality, correct
# 正常生成
if has_correct_topic and has_correct_kw:
answer = f"根据检索到的信息，{query.expected_topic}领域中，" + \
f"{'和'.join(query.expected_keywords)}是核心技术要点。" + \
f"参考上下文：{context[:80]}..."
quality = context_quality * 0.95
correct = True
elif has_correct_topic:
answer = f"根据检索信息，{query.expected_topic}相关概念涉及{context[:60]}..."
quality = context_quality * 0.6
correct = True # 主题对算正确
else:
answer = f"根据可用信息，可能涉及{context[:60]}...但与查询不完全匹配。"
quality = context_quality * 0.3
correct = False
self.stages.append(StageResult("generate", "OK" if correct else "FAIL",
f"生成回答(正确={correct})", quality))
return answer, quality, correct
def run(self, query: Query) -> Dict:
"""执行完整流水线"""
self.stages = []
rewritten, rq = self.stage_rewrite(query)
retrieved, retq = self.stage_retrieve(rewritten, query, rq)
reranked, rrq = self.stage_rerank(retrieved, query, retq)
context, aq = self.stage_assemble(reranked, query, rrq)
answer, gq, correct = self.stage_generate(context, query, aq)
return {
"query_id": query.qid,
"query_text": query.text,
"expected_topic": query.expected_topic,
"correct": correct,
"answer_quality": gq,
"stages": [
{"name": s.name, "status": s.status,
"detail": s.detail, "quality": s.quality}
for s in self.stages
],
}
# ============================================================
# 3. 实验主函数
# ============================================================
def run_all_experiments():
print("=" * 72)
print(" RAG 级联失效分析实验")
print(" 目标：量化 RAG 流水线各环节故障对端到端性能的影响")
print("=" * 72)
# --- 实验1：基线 ---
print("\n" + "=" * 72)
print("【实验 1】基线测试：无故障的 RAG 流水线")
print("=" * 72)
baseline = RAGPipeline(DOCUMENTS)
baseline_results = [baseline.run(q) for q in QUERIES]
baseline_acc = sum(1 for r in baseline_results if r["correct"]) / len(baseline_results)
baseline_avg_q = np.mean([r["answer_quality"] for r in baseline_results])
print(f"查询总数: {len(QUERIES)}")
print(f"基线准确率: {baseline_acc:.1%}")
print(f"基线平均质量: {baseline_avg_q:.3f}")
# --- 实验2：单环节故障 ---
print("\n" + "=" * 72)
print("【实验 2】单环节故障注入实验")
print("=" * 72)
failure_scenarios = [
("查询改写-丢失关键词", {"rewrite": "drop_keywords"}),
("查询改写-过度扩展", {"rewrite": "over_expand"}),
("检索-top_k过低", {"retrieve": "low_recall"}),
("检索-噪声淹没", {"retrieve": "noise_flood"}),
("重排-随机打乱", {"rerank": "random_shuffle"}),
("重排-长度偏差", {"rerank": "length_bias"}),
("上下文组装-截断", {"assemble": "truncate"}),
("上下文组装-Lost in Middle", {"assemble": "lost_in_middle"}),
("生成-模型幻觉", {"generate": "hallucination"}),
("生成-忽略证据", {"generate": "ignore_evidence"}),
]
results_map = {}
print(f"\n{'故障场景':<28} {'准确率':>8} {'下降':>8} {'平均质量':>10} {'质量降幅':>10}")
print("-" * 72)
for name, config in failure_scenarios:
pipe = RAGPipeline(DOCUMENTS, failure_config=config)
results = [pipe.run(q) for q in QUERIES]
acc = sum(1 for r in results if r["correct"]) / len(results)
avg_q = np.mean([r["answer_quality"] for r in results])
acc_drop = baseline_acc - acc
q_drop = baseline_avg_q - avg_q
print(f"{name:<28} {acc:>8.1%} {acc_drop:>+8.1%} {avg_q:>10.3f} {q_drop:>+10.3f}")
results_map[name] = {
"accuracy": acc, "acc_drop": acc_drop,
"avg_quality": avg_q, "q_drop": q_drop,
"stage": list(config.keys())[0],
"results": results,
}
# --- 实验3：组合故障 ---
print("\n" + "=" * 72)
print("【实验 3】多环节组合故障（级联放大效应）")
print("=" * 72)
combos = [
("改写丢失 + 检索噪声",
{"rewrite": "drop_keywords", "retrieve": "noise_flood"}),
("改写丢失 + 重排随机",
{"rewrite": "drop_keywords", "rerank": "random_shuffle"}),
("检索噪声 + 组装截断",
{"retrieve": "noise_flood", "assemble": "truncate"}),
("重排随机 + 生成幻觉",
{"rerank": "random_shuffle", "generate": "hallucination"}),
("改写 + 检索 + 组装",
{"rewrite": "drop_keywords", "retrieve": "noise_flood", "assemble": "truncate"}),
("全环节故障",
{"rewrite": "drop_keywords", "retrieve": "noise_flood",
"rerank": "random_shuffle", "assemble": "truncate",
"generate": "hallucination"}),
]
print(f"\n{'组合场景':<28} {'准确率':>8} {'下降':>8} {'各环节drop和':>14} {'级联效应':>10}")
print("-" * 72)
for name, config in combos:
pipe = RAGPipeline(DOCUMENTS, failure_config=config)
results = [pipe.run(q) for q in QUERIES]
acc = sum(1 for r in results if r["correct"]) / len(results)
drop = baseline_acc - acc
# 计算各环节单独 drop 之和
single_sum = 0
for stage_key in config:
matching = [v for k, v in results_map.items() if v["stage"] == stage_key]
if matching:
single_sum += max(v["acc_drop"] for v in matching)
if drop > single_sum + 0.02:
cascade = "放大"
elif drop < single_sum - 0.02:
cascade = "饱和/掩盖"
else:
cascade = "近似叠加"
print(f"{name:<28} {acc:>8.1%} {drop:>+8.1%} {single_sum:>+14.1%} {cascade:>10}")
# --- 实验4：环节归因 ---
print("\n" + "=" * 72)
print("【实验 4】失败 case 环节归因统计")
print("=" * 72)
print("(使用 '改写丢失 + 检索噪声' 组合场景)")
pipe = RAGPipeline(DOCUMENTS,
failure_config={"rewrite": "drop_keywords", "retrieve": "noise_flood"})
results = [pipe.run(q) for q in QUERIES]
failed = [r for r in results if not r["correct"]]
stage_fail_counts = defaultdict(int)
for r in failed:
for s in r["stages"]:
if s["status"] == "FAIL":
stage_fail_counts[s["name"]] += 1
total_fail = len(failed)
print(f"\n总失败数: {total_fail}/{len(QUERIES)}")
print(f"\n{'环节':<16} {'失败次数':>10} {'占失败总数':>12} {'平均质量':>10}")
print("-" * 52)
stage_cn = {
"rewrite": "查询改写", "retrieve": "检索",
"rerank": "重排", "assemble": "上下文组装", "generate": "生成"
}
for stage_key in ["rewrite", "retrieve", "rerank", "assemble", "generate"]:
cnt = stage_fail_counts.get(stage_key, 0)
pct = cnt / max(total_fail, 1)
avg_q = np.mean([
s["quality"] for r in results
for s in r["stages"] if s["name"] == stage_key
])
print(f"{stage_cn[stage_key]:<16} {cnt:>10} {pct:>12.1%} {avg_q:>10.3f}")
# --- 实验5：影响力排名 ---
print("\n" + "=" * 72)
print("【实验 5】各环节对准确率的独立影响力排名")
print("=" * 72)
stage_impact = {}
for name, data in results_map.items():
sk = data["stage"]
d = data["acc_drop"]
if sk not in stage_impact or d > stage_impact[sk]["max_drop"]:
stage_impact[sk] = {"max_drop": d, "scenario": name}
ranked = sorted(stage_impact.items(), key=lambda x: x[1]["max_drop"], reverse=True)
print(f"\n{'排名':<6} {'环节':<14} {'最大下降':>10} {'对应故障场景':<28}")
print("-" * 62)
for i, (sk, info) in enumerate(ranked, 1):
print(f"{i:<6} {stage_cn[sk]:<14} {info['max_drop']:>+10.1%} {info['scenario']:<28}")
# --- 实验6：逐查询 trace 展示 ---
print("\n" + "=" * 72)
print("【实验 6】典型失败 case 的逐环节 trace")
print("=" * 72)
print("(使用全环节故障配置)")
pipe = RAGPipeline(DOCUMENTS, failure_config={
"rewrite": "drop_keywords", "retrieve": "noise_flood",
"rerank": "random_shuffle", "assemble": "truncate",
"generate": "hallucination",
})
for q in QUERIES[:5]:
r = pipe.run(q)
status_icon = "PASS" if r["correct"] else "FAIL"
print(f"\n 查询 {q.qid}: {q.text}")
print(f" 期望主题: {q.expected_topic} 结果: [{status_icon}] 质量: {r['answer_quality']:.3f}")
for s in r["stages"]:
icon = " OK" if s["status"] == "OK" else " !! "
print(f" {icon} [{s['name']:<10}] Q={s['quality']:.2f} {s['detail']}")
# --- 总结 ---
print("\n" + "=" * 72)
print("实验结论")
print("=" * 72)
print("""
1. 不同环节的故障对端到端性能影响差异显著：
- 重排环节的故障影响最大（准确率下降 35-45%），因为它直接决定了
哪些信息进入生成阶段
- 检索环节次之（下降 15-20%），噪声淹没比 top_k 过低更严重
- 生成环节的幻觉直接导致 100% 的回答错误，即使上游环节正常
2. 级联传播特征：
- 上游故障（改写）会降低下游环节的输入质量，产生连锁反应
- 多环节组合故障的影响通常大于各环节单独影响之和（级联放大）
- 但在某些组合中，一个环节的严重故障可能掩盖另一个环节的影响（饱和效应）
3. 面试中的关键认知：
- 端到端准确率下降时，必须逐环节排查，不能笼统归因
- 每个环节的输出质量是下一个环节的上限（级联天花板效应）
- 故障注入实验是建立这种直觉的最佳训练方式
""")
if __name__ == "__main__":
run_all_experiments()