【Python高级工程与架构实战】项目六：RAG知识库问答系统（企业文档智能）

[6.1 数据摄取管道](#6.1 数据摄取管道)

[6.1.1 文档解析引擎：Unstructured.io处理PDF表格与层级标题](#6.1.1 文档解析引擎：Unstructured.io处理PDF表格与层级标题)

[6.1.2 网页爬取集成：Crawl4AI异步抓取与内容清洗](#6.1.2 网页爬取集成：Crawl4AI异步抓取与内容清洗)

[6.1.3 分块策略：语义分块(Semantic Chunking)与递归字符切分对比](#6.1.3 分块策略：语义分块(Semantic Chunking)与递归字符切分对比)

[6.1.4 元数据提取：文件名、章节、时间戳增强检索过滤](#6.1.4 元数据提取：文件名、章节、时间戳增强检索过滤)

[6.2 向量存储架构](#6.2 向量存储架构)

[6.2.1 嵌入模型管理：Sentence-Transformers本地部署与OpenAI嵌入切换](#6.2.1 嵌入模型管理：Sentence-Transformers本地部署与OpenAI嵌入切换)

[6.2.2 向量数据库选型：Milvus分布式部署与Qdrant轻量级对比](#6.2.2 向量数据库选型：Milvus分布式部署与Qdrant轻量级对比)

[6.2.3 混合检索：向量相似度 + BM25关键词搜索融合(RRF算法)](#6.2.3 混合检索：向量相似度 + BM25关键词搜索融合(RRF算法))

[6.2.4 索引优化：HNSW参数调优与量化(Quantization)压缩](#6.2.4 索引优化：HNSW参数调优与量化(Quantization)压缩)

[6.3 检索策略优化](#6.3 检索策略优化)

[6.3.1 查询重写：HyDE(Hypothetical Document Embedding)生成假设答案](#6.3.1 查询重写：HyDE(Hypothetical Document Embedding)生成假设答案)

[6.3.2 重排序(Rerank)：Cohere Rerank与交叉编码器微调](#6.3.2 重排序(Rerank)：Cohere Rerank与交叉编码器微调)

[6.3.3 多跳检索：GraphRAG构建文档关系图与社区摘要](#6.3.3 多跳检索：GraphRAG构建文档关系图与社区摘要)

[6.3.4 查询路由：元数据过滤与多索引联合查询](#6.3.4 查询路由：元数据过滤与多索引联合查询)

[6.4 生成与后处理](#6.4 生成与后处理)

[6.4.1 上下文组装：token预算管理与相关片段优先级排序](#6.4.1 上下文组装：token预算管理与相关片段优先级排序)

[6.4.2 引用溯源：检索结果高亮与原文链接生成](#6.4.2 引用溯源：检索结果高亮与原文链接生成)

[6.4.3 答案验证：Self-RAG反思机制与幻觉检测](#6.4.3 答案验证：Self-RAG反思机制与幻觉检测)

[6.4.4 缓存策略：语义缓存(Semantic Cache)与精确匹配缓存](#6.4.4 缓存策略：语义缓存(Semantic Cache)与精确匹配缓存)

[6.5 评估与监控](#6.5 评估与监控)

[6.5.1 离线评估：RAGAS指标(Faithfulness/Answer Relevancy)自动计算](#6.5.1 离线评估：RAGAS指标(Faithfulness/Answer Relevancy)自动计算)

[6.5.2 在线反馈：用户点赞/点踩与 thumbs 信号收集](#6.5.2 在线反馈：用户点赞/点踩与 thumbs 信号收集)

[6.5.3 A/B测试：不同分块策略与提示词模板效果对比](#6.5.3 A/B测试：不同分块策略与提示词模板效果对比)

[6.5.4 持续学习：Bad Case收集与模型微调触发机制](#6.5.4 持续学习：Bad Case收集与模型微调触发机制)

第二部分：结构化伪代码

第三部分：代码实现

[6.1.1 文档解析引擎](#6.1.1 文档解析引擎)

[6.1.2 网页爬取集成](#6.1.2 网页爬取集成)

[6.1.3 分块策略](#6.1.3 分块策略)

[6.1.4 元数据提取](#6.1.4 元数据提取)

[6.2.1 嵌入模型管理](#6.2.1 嵌入模型管理)

[6.2.2 向量数据库选型](#6.2.2 向量数据库选型)

[6.2.3 混合检索](#6.2.3 混合检索)

[6.2.4 索引优化](#6.2.4 索引优化)

[6.3.1 查询重写](#6.3.1 查询重写)

[6.3.2 重排序](#6.3.2 重排序)

[6.3.3 多跳检索](#6.3.3 多跳检索)

[6.3.4 查询路由](#6.3.4 查询路由)

[6.4.1 上下文组装](#6.4.1 上下文组装)

[6.4.2 引用溯源](#6.4.2 引用溯源)

[6.4.3 答案验证](#6.4.3 答案验证)

[6.4.4 缓存策略](#6.4.4 缓存策略)

[6.5.1 离线评估](#6.5.1 离线评估)

[6.5.2 在线反馈](#6.5.2 在线反馈)

[6.5.3 A/B测试](#6.5.3 A/B测试)

[6.5.4 持续学习](#6.5.4 持续学习)

第一部分：原理详解

6.1 数据摄取管道

6.1.1 文档解析引擎：Unstructured.io处理PDF表格与层级标题

现代企业知识库的核心挑战在于处理异构文档格式，特别是PDF文件中嵌套的表格结构与层级化标题体系。Unstructured.io框架采用深度文档理解模型，通过计算机视觉与自然语言处理的融合架构，实现版面分析（Layout Analysis）与文本抽取的协同优化。

该引擎首先应用基于Transformer的版面检测网络，识别文档中的文本块、表格区域、图像及标题层级。对于表格解析，系统采用两阶段方法：第一阶段使用目标检测算法定位表格边界框，第二阶段通过序列到序列模型重建表格的行-列结构，将视觉表格转换为结构化的HTML或Markdown表示。层级标题的识别依赖于字体特征（字号、字重、字体家族）的统计分析，结合语义连贯性模型，构建文档的章节树（Table of Contents Tree）。

在技术实现层面，解析引擎维护一个文档对象模型（Document Object Model），其中每个节点包含坐标元数据（bounding box coordinates）、文本内容、元素类型（Title, NarrativeText, Table, ListItem等）以及与其他节点的层级关系。这种细粒度的文档表示为后续的语义分块提供了结构感知的基础。

6.1.2 网页爬取集成：Crawl4AI异步抓取与内容清洗

针对动态网页与企业内部Wiki系统的数据摄取，Crawl4AI采用异步I/O架构（基于Python的asyncio库）实现高并发爬取。该框架设计了智能请求调度器，通过令牌桶算法（Token Bucket Algorithm）控制爬取速率，避免对目标服务器造成过载。

内容清洗模块采用基于视觉的网页去噪技术，利用Readability算法与机器学习方法识别主要内容区域（Main Content Area），过滤导航栏、广告、页脚等噪声元素。对于JavaScript渲染的单页应用（SPA），系统集成Playwright或Selenium进行无头浏览器渲染，等待动态内容加载完成后执行DOM抽取。

清洗流程包括：HTML标签剥离（保留语义化标签如h1-h6, table, ul/ol）、CSS内联样式移除、编码规范化（统一转换为UTF-8）、以及重复内容检测（基于SimHash或MinHash的近似重复检测算法）。最终输出的是结构化的Markdown或纯文本，保留原始网页的语义层次结构。

6.1.3 分块策略：语义分块(Semantic Chunking)与递归字符切分对比

文本分块（Text Chunking）是RAG系统中断句上下文窗口与检索精度的关键权衡点。递归字符切分（Recursive Character Text Splitting）采用分层分隔符策略，优先按段落（\n\n）、句子（\n）、单词（空格）的层级递归切割，确保块内文本的连续性。该方法计算效率高，但可能破坏语义边界，导致跨段落的上下文碎片化。

语义分块（Semantic Chunking）则基于嵌入向量的语义相似性动态确定切分点。算法首先计算句子级别的嵌入向量，然后检测相邻句子间余弦相似度的显著下降（阈值通常为0.7-0.8），在语义边界处执行切分。该方法的优势在于保持主题一致性，块内语义连贯性显著优于固定长度切分，但计算成本较高，需要预计算嵌入向量。

similarity(s_i, s_{i+1}) = \\frac{e_i \\cdot e_{i+1}}{\\\|e_i\\\| \\\|e_{i+1}\\\|}

其中 $e_i$ 表示第 $i$ 个句子的嵌入向量。当相似度低于阈值 $\\tau$ 时，触发分块边界。

6.1.4 元数据提取：文件名、章节、时间戳增强检索过滤

元数据增强检索（Metadata-Enriched Retrieval）通过在向量存储中附加结构化属性，支持基于过滤条件的精确检索。系统自动提取的元数据包括：文档级属性（文件名、作者、创建时间、文档类型）、章节级属性（标题层级、章节编号、父章节引用）以及内容级属性（关键词标签、实体识别结果、摘要）。

时间戳元数据支持时间范围过滤（Temporal Filtering），适用于需要检索最新版本文档或特定时间段内更新的场景。章节层级元数据构建文档的导航路径（Breadcrumb），在检索结果中提供上下文定位。文件名与路径信息用于权限控制（Access Control List, ACL），确保检索结果符合用户的文档访问权限。

6.2 向量存储架构

6.2.1 嵌入模型管理：Sentence-Transformers本地部署与OpenAI嵌入切换

嵌入模型（Embedding Model）负责将文本映射到高维语义空间。Sentence-Transformers框架支持多种预训练架构（BERT, RoBERTa, MPNet, E5, GTE等）的本地部署，通过量化技术（INT8/INT4）与ONNX Runtime优化推理延迟。本地部署确保数据隐私，适用于金融、医疗等敏感领域。

模型选择依据MTEB（Massive Text Embedding Benchmark）排行榜的检索性能指标。对于多语言场景，采用支持跨语言对齐的模型（如paraphrase-multilingual-mpnet-base-v2）。向量维度通常为384至1024维，需在存储成本与语义表达能力间权衡。

OpenAI的text-embedding-3-large等API服务提供更高维度的嵌入（3072维）与更好的下游任务性能，但引入网络延迟与数据出境合规风险。系统应支持模型路由（Model Routing），根据数据敏感度与延迟要求动态选择本地或云端嵌入服务。

v = Encoder(Tokenize(text)) \\in \\mathbb{R}\^d

其中 $d$ 为嵌入维度， $Encoder$ 为双向Transformer编码器。

6.2.2 向量数据库选型：Milvus分布式部署与Qdrant轻量级对比

Milvus采用云原生架构，支持水平扩展（Horizontal Scaling）与分布式向量索引。其存储-计算分离架构允许独立扩展查询节点（Query Node）与数据节点（Data Node），适用于十亿级向量的高并发检索。Milvus支持多种索引类型（FLAT, IVF-FLAT, IVF-PQ, HNSW, ANNOY），并提供基于Raft协议的分布式一致性保证。

Qdrant采用Rust编写，专注于单节点或轻量级集群部署的高性能。其特色在于混合查询能力（Hybrid Filtering），支持在向量相似度搜索的同时应用复杂的标量过滤条件（如 price > 100 AND category = "electronics"）。Qdrant的内存映射（Memory Mapping）机制优化了大规模数据集的内存占用，适合资源受限的边缘部署。

ANN(q, k) = \\arg\\max_{D' \\subset D, \|D'\|=k} \\sum_{x \\in D'} \\frac{q \\cdot x}{\\\|q\\\| \\\|x\\\|}

近似最近邻（ANN）检索在向量空间中寻找与查询向量 $q$ 最相似的 $k$ 个向量。

6.2.3 混合检索：向量相似度 + BM25关键词搜索融合(RRF算法)

混合检索（Hybrid Retrieval）结合稠密向量检索（Dense Retrieval）的语义理解能力与稀疏检索（Sparse Retrieval，如BM25）的精确关键词匹配优势。稠密检索捕获语义相关性（如"汽车"与"车辆"的隐含关联），而BM25确保对特定术语、产品代码、人名等精确匹配。

倒数排序融合（Reciprocal Rank Fusion, RRF）算法融合两种检索结果列表。对于每个文档 $d$ ，计算其在向量检索列表中的排名 $r_v(d)$ 与在BM25列表中的排名 $r_b(d)$ （若不存在则设为无穷大）。RRF得分公式为：

RRF(d) = \\sum_{i \\in \\{v, b\\}} \\frac{1}{k + r_i(d)}

其中 $k$ 为常数（通常取60），用于平滑高排名文档的得分差异。最终按RRF得分降序排列，生成融合后的检索结果。

6.2.4 索引优化：HNSW参数调优与量化(Quantization)压缩

HNSW（Hierarchical Navigable Small World）图索引通过构建多层近似图结构实现对数级复杂度的最近邻搜索。关键参数包括：M（每层最大邻居数，控制图密度）、efConstruction（构建时的搜索范围，影响索引质量）与ef（查询时的搜索范围，影响召回率）。

Recall@k = \\frac{\|ANN_k(q) \\cap Exact_k(q)\|}{k}

量化压缩技术降低存储与计算开销。乘积量化（Product Quantization, PQ）将高维向量分解为子向量，对每个子空间训练码本（Codebook），用质心索引替代原始浮点值。标量量化（Scalar Quantization, SQ）将float32映射为int8，减少75%存储空间同时保持较高精度。二进制量化（Binary Quantization）进一步压缩为比特向量，通过汉明距离加速计算，但牺牲部分精度。

6.3 检索策略优化

6.3.1 查询重写：HyDE(Hypothetical Document Embedding)生成假设答案

Hypothetical Document Embeddings（HyDE）技术利用大型语言模型的生成能力弥合查询与文档间的语义鸿沟。传统检索中，简短或模糊的查询（如"最佳实践"）难以匹配详细的文档内容。HyDE通过提示工程（Prompt Engineering）指令语言模型生成假设性答案文档，该文档虽未基于真实知识库，但在语义空间上更接近目标文档。

具体流程为：首先将原始查询 $q$ 输入LLM生成假设文档 $d_{hyp}$ ；随后计算 $d_{hyp}$ 的嵌入向量 $e_{hyp}$ ；最后以 $e_{hyp}$ 作为检索查询在向量数据库中搜索相似文档。该方法有效扩展了查询的语义表达，尤其适用于零样本（Zero-Shot）检索场景。

e_{hyde} = Embed(LLM(q \\mid prompt_{hyde}))

Results = TopK(e_{hyde}, Index)

6.3.2 重排序(Rerank)：Cohere Rerank与交叉编码器微调

初始检索阶段通常采用双编码器（Bi-Encoder）架构，独立编码查询与文档，通过向量内积快速筛选候选集。然而，双编码器无法充分建模查询-文档间的细粒度交互。重排序（Reranking）阶段使用交叉编码器（Cross-Encoder）或专用重排序模型（如Cohere Rerank API）对候选文档进行精确排序。

交叉编码器将查询与文档文本拼接（如"[CLS] Query [SEP] Document [SEP]"），通过Transformer编码器生成相关性分数。这种全注意力机制捕获了词汇级对齐与语义交互，精度显著高于双编码器，但计算成本较高，仅适用于小规模候选集（通常Top-100）。

微调策略包括在领域特定数据（如MS MARCO或自定义标注数据）上训练重排序模型，优化二元分类（相关/不相关）或细粒度相关性评分目标。知识蒸馏（Knowledge Distillation）技术将大型交叉编码器的知识迁移到轻量级模型，平衡精度与延迟。

Score(q, d) = MLP(Transformer(\[q; d\]))

6.3.3 多跳检索：GraphRAG构建文档关系图与社区摘要

GraphRAG（Graph-based Retrieval-Augmented Generation）通过构建文档实体关系图支持多跳推理（Multi-hop Reasoning）。系统首先使用命名实体识别（NER）与关系抽取（RE）从文档中提取实体（Entity）与关系（Relation）三元组，构建知识图谱 $G=(V, E)$ ，其中节点 $V$ 表示实体，边 $E$ 表示关系。

社区检测（Community Detection）算法（如Louvain或Leiden算法）识别图谱中的紧密连接子图（社区），每个社区对应特定主题或概念集群。对每个社区生成摘要（Community Summary），描述该社区的核心主题与关键实体。

多跳检索流程：首先识别查询中的种子实体，在图中执行广度优先搜索（BFS）或个性化PageRank（PPR）探索多跳邻居；然后检索相关社区摘要与关联文档；最后基于图结构路径生成带有溯源的推理链（Chain of Evidence）。

PPR(v) = \\alpha \\cdot e_v + (1 - \\alpha) \\cdot \\sum_{u \\in N(v)} \\frac{PPR(u)}{\|N(u)\|}

其中 $\\alpha$ 为随机跳转概率， $e_v$ 为种子实体的one-hot向量。

6.3.4 查询路由：元数据过滤与多索引联合查询

查询路由（Query Routing）机制根据查询特征动态选择检索策略或索引子集。元数据过滤路由分析查询中的结构化约束（如时间范围、文档类型、作者），将查询定向到特定分区（Partition）或集合（Collection）。例如，查询"2024年财务报告中的营收数据"被路由到2024年文档集合与财务类别索引。

多索引联合查询（Multi-Index Federation）在垂直领域知识库中尤为关键，其中不同文档类型（产品手册、技术规范、客户案例）存储于独立索引。路由分类器（基于轻量级BERT或关键词规则）预测查询应检索的索引子集，并行执行检索后融合结果。

自适应检索（Adaptive Retrieval）根据查询复杂度动态调整检索深度：简单事实查询（如"公司成立时间"）仅需单跳检索，而复杂分析查询（如"比较Q1与Q2产品线表现差异"）触发多跳或迭代检索。

6.4 生成与后处理

6.4.1 上下文组装：token预算管理与相关片段优先级排序

大语言模型的上下文窗口（Context Window）存在长度限制（如128K tokens），而检索返回的文档片段总和常超出该限制。上下文组装（Context Assembly）模块实施token预算管理（Token Budget Management），在约束条件下最大化上下文信息量。

优先级排序策略基于相关性分数与信息多样性。首先按检索相关性排序片段；然后应用最大边际相关性（Maximal Marginal Relevance, MMR）算法，在相关性与多样性间权衡：

MMR(d_i) = \\lambda \\cdot Sim(d_i, q) - (1 - \\lambda) \\cdot \\max_{d_j \\in S} Sim(d_i, d_j)

其中 $S$ 为已选片段集合， $\\lambda$ 控制权衡系数。该公式选择既相关又与已选片段差异大的文档，减少冗余信息。

动态截断（Dynamic Truncation）根据文档结构在句子或段落边界处截断，避免切割语义单元。基于语义的压缩（Semantic Compression）使用较小语言模型提取片段核心句，进一步节省token预算。

6.4.2 引用溯源：检索结果高亮与原文链接生成

答案溯源（Attribution）是确保RAG系统可解释性与可验证性的关键。系统通过语句级对齐（Sentence-Level Alignment）识别生成答案中的每个陈述（Claim）所支持的原文证据。

实现方法包括：在提示工程中加入指令要求模型为每个陈述添加引用标记（如[1], [2]）；使用后处理算法将生成文本分割为原子陈述，通过自然语言推理（NLI）模型验证各陈述与检索片段的蕴含关系（Entailment）；对支持特定陈述的原文片段进行高亮（Highlighting）处理。

原文链接生成将引用标记映射到文档元数据（文件名、页码、段落ID），生成可点击的超链接或结构化引用（如"根据《产品手册v2.0》第15页..."）。对于PDF文档，通过坐标元数据生成精确到行的文本高亮区域。

6.4.3 答案验证：Self-RAG反思机制与幻觉检测

Self-RAG（Self-Reflective Retrieval-Augmented Generation）框架在生成过程中插入反思标记（Reflection Tokens），使模型动态决定是否需要检索、评估检索内容的相关性、以及验证生成内容的准确性。

反思机制通过特殊训练或提示工程实现，模型生成格式如：[Retrieve]、[No Retrieve]、[Relevant]、[Irrelevant]、[Supported]、[Contradictory]。在生成每个陈述后，模型评估该陈述是否需要外部验证；若需要，则触发检索并评估返回文档的相关性；最后验证生成内容与文档的一致性。

幻觉检测（Hallucination Detection）采用基于NLI的事实核查：将答案分解为事实陈述集合 $\\{f_1, f_2, ..., f_n\\}$ ，对每个 $f_i$ 验证其与检索上下文 $C$ 的蕴含关系：

Faithfulness = \\frac{\|\\{f_i \\mid C \\models f_i\\}\|}{n}

其中 $C \\models f_i$ 表示上下文支持该事实。低置信度陈述触发警告或拒绝回答。

6.4.4 缓存策略：语义缓存(Semantic Cache)与精确匹配缓存

语义缓存（Semantic Cache）通过识别语义等价的查询（尽管字面不同）减少重复计算与API调用。系统维护缓存存储 $(Q_{cache}, V_{cache}, A_{cache})$ ，其中 $Q_{cache}$ 为历史查询， $V_{cache}$ 为嵌入向量， $A_{cache}$ 为缓存答案。

对于新查询 $q_{new}$ ，计算其嵌入 $v_{new}$ ，在缓存向量中搜索相似度超过阈值 $\\theta$ （如0.95）的条目。若存在，直接返回对应答案。近似最近邻搜索（ANN）加速缓存查找。

精确匹配缓存（Exact Match Cache）使用哈希表存储字面完全相同的查询，适用于高频重复问题。分层缓存策略结合精确匹配（L1缓存）与语义匹配（L2缓存）， misses时触发完整RAG流程并将结果写入缓存。

缓存失效（Cache Invalidation）策略包括：基于时间的生存期（TTL）、文档更新事件驱动的主动失效、以及基于语义漂移检测的被动失效。

6.5 评估与监控

6.5.1 离线评估：RAGAS指标(Faithfulness/Answer Relevancy)自动计算

RAGAS（Retrieval-Augmented Generation Assessment）框架提供无参考（Reference-Free）的自动化评估指标，无需人工标注的标准答案即可评估RAG系统性能。

忠实度（Faithfulness）度量生成答案 $A$ 与检索上下文 $C$ 的事实一致性。首先使用LLM将答案分解为原子陈述集合 $S(A)=\\{s_1, s_2, ..., s_m\\}$ ；然后验证每个陈述 $s_i$ 是否被 $C$ 支持（支持、矛盾或未知）。忠实度分数为被支持陈述的比例：

F = \\frac{\|\\{s_i \\mid Supported(s_i, C)\\}\|}{\|S(A)\|}

答案相关性（Answer Relevancy）评估答案 $A$ 对用户查询 $Q$ 的针对性。系统生成 $k$ 个潜在问题 $\\{q_1, ..., q_k\\}$ ，这些问题应以 $A$ 为正确答案；然后计算这些生成问题与原始查询 $Q$ 的嵌入相似度平均值：

AR = \\frac{1}{k} \\sum_{i=1}\^k \\cos(Embed(Q), Embed(q_i))

上下文精确率（Context Precision）衡量检索片段中相关部分的比例；上下文召回率（Context Recall）评估检索是否覆盖了回答问题所需的全部信息。

6.5.2 在线反馈：用户点赞/点踩与 thumbs 信号收集

在线反馈机制捕获真实用户交互信号，用于持续优化检索与生成质量。显式反馈（Explicit Feedback）包括点赞（Thumbs Up）/点踩（Thumbs Down）按钮、星级评分（1-5星）、以及可选的自由文本反馈。

隐式反馈（Implicit Feedback）通过用户行为推断满意度：答案复制行为、 dwell time（答案展示后用户在页面的停留时间）、后续查询（若用户在获得答案后立即发起相关查询，可能表明答案不完整）、以及会话终止（成功解决用户问题后结束对话）。

反馈信号与检索上下文、生成答案、系统配置（使用的分块策略、模型版本）关联存储，构建反馈数据库用于后续的模型微调与策略优化。对抗性反馈（Adversarial Feedback）识别系统失效模式，如事实错误、未回答查询核心、或检索不相关内容。

6.5.3 A/B测试：不同分块策略与提示词模板效果对比

A/B测试框架对比RAG系统变体在实际流量下的性能差异。测试维度包括：分块策略（固定长度vs语义分块）、嵌入模型（E5 vs OpenAI）、检索算法（纯向量vs混合检索）、重排序模型（有无Cohere Rerank）、以及提示词模板（Zero-Shot vs Few-Shot vs Chain-of-Thought）。

流量分割（Traffic Splitting）采用用户ID哈希或会话级随机化，确保同一用户在会话期间体验一致的系统版本（避免混杂效应）。关键指标包括：答案接受率（基于点赞/点踩）、任务完成率（用户是否达成查询目标）、平均延迟、以及错误率。

统计显著性检验（如双样本t检验或Mann-Whitney U检验）确定观察到的差异是否显著。多臂老虎机（Multi-Armed Bandit）算法动态调整流量分配，将更多流量导向表现优异的变体，同时保持对次优变体的探索。

6.5.4 持续学习：Bad Case收集与模型微调触发机制

持续学习（Continual Learning）机制识别系统的失效案例（Bad Cases），触发针对性改进。Bad Case检测标准包括：用户明确点踩、答案忠实度评分低于阈值、检测到幻觉、或检索结果为空。

Bad Case分类器将失效归因于特定组件：解析错误（文档未正确提取）、检索失败（相关文档未召回）、排序错误（相关文档排名过低）、生成错误（模型未正确利用上下文）、或提示不足（上下文未包含足够信息）。

当累积的Bad Case数量达到阈值（如100例），触发模型微调（Fine-Tuning）或检索索引更新。微调数据构建包括：对检索失败的案例，将正确文档标记为正样本，挖掘困难负样本（Hard Negatives）；对生成错误的案例，构建偏好对（Preference Pairs），使用RLHF（Reinforcement Learning from Human Feedback）或DPO（Direct Preference Optimization）优化生成模型。

自动化再训练管道（Retraining Pipeline）执行数据验证、模型训练、离线评估、以及影子部署（Shadow Deployment），验证通过后才推送到生产环境。

第二部分：结构化伪代码

6.1 数据摄取管道

6.1.1 文档解析引擎

代码段

复制代码

\begin{algorithm} 
\caption{Unstructured Document Parsing Engine} 
\begin{algorithmic}[1] 
\Require Document file path $P$, Extraction schema $S$ 
\Ensure Structured document elements $E=\{e_1, e_2, \dots, e_n\}$ 
\State $doc \leftarrow \text{LoadDocument}(P)$ 
\State $elements \leftarrow \text{InitializeEmptyList}()$ 
\State $layout \leftarrow \text{DetectLayout}(doc)$ 
\For{each region $r \in layout$} 
    \If{$r.type = \text{Table}$} 
        \State $html\_table \leftarrow \text{ExtractTableStructure}(r)$ 
        \State $e \leftarrow \text{CreateElement}(\text{type=Table}, \text{content}=html\_table)$ 
    \ElsIf{$r.type = \text{Title}$} 
        \State $level \leftarrow \text{InferHeadingLevel}(r.font\_features)$ 
        \State $e \leftarrow \text{CreateElement}(\text{type=Title}, \text{level}=level, \text{content}=r.text)$ 
    \Else 
        \State $e \leftarrow \text{CreateElement}(\text{type}=r.type, \text{content}=r.text)$ 
    \EndIf 
    \State $e.metadata \leftarrow \{bbox: r.coordinates, page: r.page\_num\}$ 
    \State $elements.append(e)$ 
\EndFor 
\State $hierarchy \leftarrow \text{BuildHierarchyTree}(elements)$ 
\State \Return $hierarchy$ 
\end{algorithmic} 
\end{algorithm}

6.1.2 网页爬取集成

代码段

复制代码

\begin{algorithm} 
\caption{Asynchronous Web Crawling with Content Cleaning} 
\begin{algorithmic}[1] 
\Require Seed URLs $U$, Max depth $D$, Rate limit $\lambda$ 
\Ensure Cleaned web documents $W$ 
\State $frontier \leftarrow \text{PriorityQueue}(U)$ 
\State $visited \leftarrow \text{HashSet}()$ 
\State $results \leftarrow \text{ConcurrentQueue}()$ 
\State $semaphore \leftarrow \text{Semaphore}(\lambda)$ 
\While{$\neg frontier.empty() \land depth < D$} 
    \State $batch \leftarrow frontier.pop\_batch(B)$ 
    \State $tasks \leftarrow \{\}$ 
    \For{each $url \in batch$} 
        \If{$url \notin visited$} 
            \State $visited.add(url)$ 
            \State $t \leftarrow asyncio.create\_task(\text{CrawlPage}(url, semaphore))$ 
            \State $tasks \leftarrow tasks \cup \{t\}$ 
        \EndIf 
    \EndFor 
    \State $pages \leftarrow asyncio.gather(tasks)$ 
    \For{each $page \in pages$} 
        \If{$page.status = 200$} 
            \State $main\_content \leftarrow \text{ExtractContent}(\text{Readability}(page.html))$ 
            \State $cleaned \leftarrow \text{CleanHTML}(main\_content)$ 
            \State $markdown \leftarrow \text{HTML2Markdown}(cleaned)$ 
            \State $results.put(\{url: page.url, content: markdown\})$ 
            \State $new\_urls \leftarrow \text{ExtractLinks}(page.html)$ 
            \State $frontier.extend(new\_urls)$ 
        \EndIf 
    \EndFor 
\EndWhile 
\State \Return $results$ 
\end{algorithmic} 
\end{algorithm}

6.1.3 分块策略

代码段

复制代码

\begin{algorithm} 
\caption{Semantic Chunking vs Recursive Character Splitting} 
\begin{algorithmic}[1] 
\Require Text $T$, Chunk size $C$, Overlap $O$, Similarity threshold $\tau$ 
\Ensure Text chunks $K=\{k_1, \dots, k_m\}$ 
\State \Comment{Recursive Character Splitting} 
\State $separators \leftarrow [\text{"\backslash n\backslash n"}, \text{"\backslash n"}, \text{"."}, \text{","}, \text{" "}, \text{""}]$ 
\State $chunks \leftarrow \text{RecursiveSplit}(T, separators, C, O)$ 
\State \Return $chunks$ 

\State \Comment{Semantic Chunking} 
\State $sentences \leftarrow \text{SentenceSegmentation}(T)$ 
\State $embeddings \leftarrow \text{Encode}(sentences)$ 
\State $chunks \leftarrow \text{InitializeEmptyList}()$ 
\State $current\_chunk \leftarrow [sentences_0]$ 
\State $current\_emb \leftarrow embeddings_0$ 

\For{$i \leftarrow 1$ \textbf{to} $|sentences| - 1$} 
    \State $sim \leftarrow \cos(embeddings_i, embeddings_{i-1})$ 
    \If{$sim < \tau \land |current\_chunk| > 0$} 
        \State $chunks.append(\text{Join}(current\_chunk))$ 
        \State $current\_chunk \leftarrow [sentences_i]$ 
    \Else 
        \State $current\_chunk.append(sentences_i)$ 
    \EndIf 
    \If{$|current\_chunk| \ge C$} 
        \State $chunks.append(\text{Join}(current\_chunk))$ 
        \State $current\_chunk \leftarrow []$ 
    \EndIf 
\EndFor 
\If{$current\_chunk \neq \emptyset$} 
    \State $chunks.append(\text{Join}(current\_chunk))$ 
\EndIf 
\State \Return $chunks$ 
\end{algorithmic} 
\end{algorithm}

6.1.4 元数据提取

代码段

复制代码

\begin{algorithm} 
\caption{Metadata Extraction and Enrichment} 
\begin{algorithmic}[1] 
\Require Document $D$, Extraction rules $R$ 
\Ensure Enriched chunks $\{(c_i, m_i)\}_{i=1}^n$ 
\State $file\_meta \leftarrow \{filename: D.name, created: D.ctime, author: D.author\}$ 
\State $structural\_meta \leftarrow \text{ParseHierarchy}(D.headings)$ 
\State $chunks \leftarrow \text{ChunkDocument}(D)$ 
\State $enriched \leftarrow \text{InitializeEmptyList}()$ 

\For{each $c \in chunks$} 
    \State $m \leftarrow file\_meta.copy()$ 
    \State $m.section \leftarrow \text{FindNearestHeading}(c, structural\_meta)$ 
    \State $m.timestamp \leftarrow \text{ExtractDate}(c.content)$ 
    \State $m.entities \leftarrow \text{NER}(c.content)$ 
    \State $m.keywords \leftarrow \text{TFIDF}(c.content, topk=5)$ 
    \State $m.position \leftarrow c.index$ 
    \State $enriched.append((c, m))$ 
\EndFor 
\State \Return $enriched$ 
\end{algorithmic} 
\end{algorithm}

6.2 向量存储架构

6.2.1 嵌入模型管理

代码段

复制代码

\begin{algorithm} 
\caption{Embedding Model Router and Local Deployment} 
\begin{algorithmic}[1] 
\Require Text batch $T$, Model configuration $M$, Privacy level $p$ 
\Ensure Embeddings $V=\{v_1, \dots, v_n\}$ 
\State $model \leftarrow \text{SelectModel}(M, p)$ 
\If{$p = high$} 
    \State $encoder \leftarrow \text{LoadLocalModel}(model.path)$ 
    \If{$model.quantization = \text{INT8}$} 
        \State $encoder \leftarrow \text{Quantize}(encoder, bits=8)$ 
    \EndIf 
    \State $embeddings \leftarrow encoder.encode(T, batch\_size=32)$ 
\Else 
    \State $embeddings \leftarrow \text{APIRequest}(\text{OpenAI}, T, model=model.id)$ 
\EndIf 
\State \Return $embeddings$ 
\end{algorithmic} 
\end{algorithm}

6.2.2 向量数据库选型

代码段

复制代码

\begin{algorithm} 
\caption{Vector Database Operations (Milvus/Qdrant)} 
\begin{algorithmic}[1] 
\Require Vector $v$, Collection name $C$, Top-k $k$, Filters $F$ 
\Ensure Search results $R$ 
\State \Comment{Insertion} 
\State $id \leftarrow \text{GenerateUUID}()$ 
\State $payload \leftarrow \{vector: v, metadata: F\}$ 
\State $client.upsert(collection=C, points=[payload])$ 

\State \Comment{Hybrid Search} 
\State $vector\_results \leftarrow client.search(collection=C, vector=v, limit=k \times 2, filter=F)$ 
\State $keyword\_results \leftarrow \text{BM25Search}(C, query\_text, k \times 2)$ 
\State $fused \leftarrow \text{RRFFusion}(vector\_results, keyword\_results, k)$ 
\State \Return $fused$ 
\end{algorithmic} 
\end{algorithm}

6.2.3 混合检索

代码段

复制代码

\begin{algorithm} 
\caption{Reciprocal Rank Fusion (RRF) Algorithm} 
\begin{algorithmic}[1] 
\Require Ranked lists $L=\{L_1, L_2, \dots, L_m\}$, Constant $k=60$, Final top $n$ 
\Ensure Fused ranking $R$ 
\State $scores \leftarrow \text{DefaultDict}(0.0)$ 
\State $all\_docs \leftarrow \bigcup_{i=1}^m \{d \mid d \in L_i\}$ 

\For{each $L_i \in L$} 
    \For{each document $d \in all\_docs$} 
        \If{$d \in L_i$} 
            \State $rank \leftarrow L_i.index(d) + 1$ 
            \State $scores[d] \leftarrow scores[d] + \frac{1}{k + rank}$ 
        \Else 
            \State $scores[d] \leftarrow scores[d] + 0$ 
        \EndIf 
    \EndFor 
\EndFor 

\State $sorted\_docs \leftarrow \text{SortByScoreDescending}(scores)$ 
\State $R \leftarrow \text{TopN}(sorted\_docs, n)$ 
\State \Return $R$ 
\end{algorithmic} 
\end{algorithm}

6.2.4 索引优化

代码段

复制代码

\begin{algorithm} 
\caption{HNSW Index Construction and Quantization} 
\begin{algorithmic}[1] 
\Require Vector set $V$, $M$ parameter, $efConstruction$, Quantization type $Q$ 
\Ensure Optimized index $I$ 
\State \Comment{HNSW Graph Construction} 
\State $graph \leftarrow \text{InitializeGraph}()$ 
\State $enter\_point \leftarrow \text{RandomSelect}(V)$ 

\For{each $v \in V$} 
    \State $layer \leftarrow \text{RandomLevel}(M)$ 
    \State $neighbors \leftarrow \text{SearchLayer}(v, enter\_point, efConstruction, layer)$ 
    \State $pruned \leftarrow \text{SelectNeighbors}(neighbors, M)$ 
    \State $graph.add\_node(v, layer, pruned)$ 
\EndFor 

\State \Comment{Product Quantization} 
\If{$Q = \text{PQ}$} 
    \State $D \leftarrow \text{dimension}(V)$ 
    \State $m \leftarrow num\_subspaces$ 
    \State $subspaces \leftarrow \text{Split}(V, m)$ 
    \For{$i \leftarrow 1$ \textbf{to} $m$} 
        \State $codebook_i \leftarrow \text{KMeans}(subspaces_i, clusters=256)$ 
    \EndFor 
    \State $codes \leftarrow \text{Quantize}(V, \{codebook_i\})$ 
    \State $I \leftarrow \{graph, codes, codebooks\}$ 
\EndIf 
\State \Return $I$ 
\end{algorithmic} 
\end{algorithm}

6.3 检索策略优化

6.3.1 查询重写

代码段

复制代码

\begin{algorithm} 
\caption{HyDE: Hypothetical Document Embedding} 
\begin{algorithmic}[1] 
\Require Query $q$, LLM $L$, Embedding model $E$, Prompt template $P_{hyde}$ 
\Ensure Enhanced query embedding $v_{hyde}$ 
\State $prompt \leftarrow P_{hyde}.format(query=q)$ 
\State $d_{hyp} \leftarrow L.generate(prompt)$ 
\State $v_{hyde} \leftarrow E.embed(d_{hyp})$ 
\State $v_{orig} \leftarrow E.embed(q)$ 
\State $v_{combined} \leftarrow \alpha \cdot v_{hyde} + (1 - \alpha) \cdot v_{orig}$ 
\State $candidates \leftarrow \text{VectorSearch}(v_{combined}, k)$ 
\State \Return $candidates$ 
\end{algorithmic} 
\end{algorithm}

6.3.2 重排序

代码段

复制代码

\begin{algorithm} 
\caption{Cross-Encoder Reranking} 
\begin{algorithmic}[1] 
\Require Query $q$, Candidate documents $D=\{d_1, \dots, d_k\}$, Cross-encoder $C$ 
\Ensure Reranked documents $D'$ 
\State $pairs \leftarrow [(q, d_i) \mid d_i \in D]$ 
\State $scores \leftarrow \text{InitializeEmptyList}()$ 

\For{each $(q, d) \in pairs$} 
    \State $input \leftarrow \text{Concatenate}(q, \text{"[SEP]"}, d)$ 
    \State $encoding \leftarrow C.tokenize(input)$ 
    \State $logits \leftarrow C.forward(encoding)$ 
    \State $relevance \leftarrow \text{Softmax}(logits)[1]$ 
    \State $scores.append(relevance)$ 
\EndFor 

\State $ranked \leftarrow \text{SortByScore}(\text{zip}(D, scores))$ 
\State $D' \leftarrow \{d \mid (d, s) \in ranked\}$ 
\State \Return $D'$ 
\end{algorithmic} 
\end{algorithm}

6.3.3 多跳检索

代码段

复制代码

\begin{algorithm} 
\caption{GraphRAG Multi-hop Retrieval} 
\begin{algorithmic}[1] 
\Require Query $q$, Knowledge graph $G=(V, E)$, Seed entity extractor $E$, Depth $d$ 
\Ensure Retrieved subgraph $G'$ 
\State $entities \leftarrow E.extract(q)$ 
\State $frontier \leftarrow \{v \mid v \in V \land v.name \in entities\}$ 
\State $visited \leftarrow frontier$ 
\State $subgraph \leftarrow \text{InitializeSubgraph}()$ 

\For{$i \leftarrow 1$ \textbf{to} $d$} 
    \State $new\_frontier \leftarrow \emptyset$ 
    \For{each $v \in frontier$} 
        \State $neighbors \leftarrow \{u \mid (v, u) \in E \lor (u, v) \in E\}$ 
        \For{each $u \in neighbors$} 
            \If{$u \notin visited$} 
                \State $visited \leftarrow visited \cup \{u\}$ 
                \State $new\_frontier \leftarrow new\_frontier \cup \{u\}$ 
                \State $subgraph.add\_edge(v, u, E(v, u))$ 
            \EndIf 
        \EndFor 
    \EndFor 
    \State $frontier \leftarrow new\_frontier$ 
\EndFor 

\State $communities \leftarrow \text{CommunityDetection}(subgraph)$ 
\State $summaries \leftarrow \{\text{GenerateSummary}(c) \mid c \in communities\}$ 
\State $docs \leftarrow \text{RetrieveDocuments}(subgraph.nodes)$ 
\State \Return $(docs, summaries)$ 
\end{algorithmic} 
\end{algorithm}

6.3.4 查询路由

代码段

复制代码

\begin{algorithm} 
\caption{Query Routing and Multi-Index Federation} 
\begin{algorithmic}[1] 
\Require Query $q$, Index registry $R=\{(I_1, F_1), \dots, (I_n, F_n)\}$, Router model $M$ 
\Ensure Aggregated results $A$ 
\State $intent \leftarrow M.classify(q)$ 
\State $metadata\_filters \leftarrow \text{ParseConstraints}(q)$ 
\State $selected\_indices \leftarrow \emptyset$ 

\For{each $(I, F) \in R$} 
    \If{$\text{IntentMatch}(intent, I.domain) \land \text{FilterCompatible}(metadata\_filters, F)$} 
        \State $selected\_indices \leftarrow selected\_indices \cup \{I\}$ 
    \EndIf 
\EndFor 

\State $tasks \leftarrow \{\text{Search}(I, q, metadata\_filters) \mid I \in selected\_indices\}$ 
\State $results \leftarrow \text{ExecuteParallel}(tasks)$ 
\State $merged \leftarrow \text{MergeByScore}(results)$ 
\State $deduplicated \leftarrow \text{Deduplicate}(merged, threshold=0.95)$ 
\State \Return $deduplicated$ 
\end{algorithmic} 
\end{algorithm}

6.4 生成与后处理

6.4.1 上下文组装

代码段

复制代码

\begin{algorithm} 
\caption{Token Budget Management with MMR} 
\begin{algorithmic}[1] 
\Require Query $q$, Retrieved chunks $C$, Max tokens $T_{max}$, LLM tokenizer $\mathcal{T}$ 
\Ensure Assembled context $C_{final}$ 
\State $selected \leftarrow \emptyset$ 
\State $remaining \leftarrow C$ 
\State $T_{used} \leftarrow |\mathcal{T}.\text{tokenize}(q)|$ 
\State $\lambda \leftarrow 0.5$ 

\While{$remaining \neq \emptyset \land T_{used} < T_{max}$} 
    \State $scores \leftarrow \emptyset$ 
    \For{each $c \in remaining$} 
        \State $rel \leftarrow \text{CosineSim}(\text{Embed}(c), \text{Embed}(q))$ 
        \State $div \leftarrow \max_{s \in selected} \text{CosineSim}(\text{Embed}(c), \text{Embed}(s))$ 
        \If{$selected = \emptyset$} 
            \State $mmr \leftarrow rel$ 
        \Else 
            \State $mmr \leftarrow \lambda \cdot rel - (1 - \lambda) \cdot div$ 
        \EndIf 
        \State $scores[c] \leftarrow mmr$ 
    \EndFor 

    \State $c_{best} \leftarrow \arg\max_{c} \text{scores}[c]$ 
    \State $T_c \leftarrow |\mathcal{T}.\text{tokenize}(c)|$ 
    \If{$T_{used} + T_c > T_{max}$} 
        \State $c_{trunc} \leftarrow \text{TruncateAtSentence}(c, T_{max} - T_{used}, \mathcal{T})$ 
        \If{$c_{trunc} \neq \emptyset$} 
            \State $selected \leftarrow selected \cup \{c_{trunc}\}$ 
        \EndIf 
        \State \textbf{break} 
    \EndIf 
    \State $selected \leftarrow selected \cup \{c_{best}\}$ 
    \State $T_{used} \leftarrow T_{used} + T_c$ 
    \State $remaining \leftarrow remaining \setminus \{c_{best}\}$ 
\EndWhile 

\State $C_{final} \leftarrow \text{Concatenate}(selected)$ 
\State \Return $C_{final}$ 
\end{algorithmic} 
\end{algorithm}

6.4.2 引用溯源

代码段

复制代码

\begin{algorithm} 
\caption{Citation Generation and Source Attribution} 
\begin{algorithmic}[1] 
\Require Generated answer $A$, Retrieved contexts $C$, NLI model $N$ 
\Ensure Attributed answer $A'$ with citations 
\State $claims \leftarrow \text{SegmentIntoSentences}(A)$ 
\State $citation\_map \leftarrow \text{Dictionary}()$ 

\For{each $claim \in claims$} 
    \State $supporting \leftarrow \emptyset$ 
    \For{$i \leftarrow 1$ \textbf{to} $|C|$} 
        \State $premise \leftarrow C[i].content$ 
        \State $label \leftarrow N.predict(premise, claim)$ 
        \If{$label = \text{entailment}$} 
            \State $supporting \leftarrow supporting \cup \{i\}$ 
        \EndIf 
    \EndFor 
    \If{$supporting \neq \emptyset$} 
        \State $citation\_map[claim] \leftarrow supporting$ 
    \EndIf 
\EndFor 

\State $A' \leftarrow \text{""}$ 
\For{each $claim \in claims$} 
    \State $A' \leftarrow A' + claim$ 
    \If{$claim \in citation\_map$} 
        \State $refs \leftarrow citation\_map[claim]$ 
        \State $A' \leftarrow A' + \text{" ["} + \text{Join}(refs, \text{","}) + \text{"]"}$ 
    \EndIf 
\EndFor 
\State \Return $A'$ 
\end{algorithmic} 
\end{algorithm}

6.4.3 答案验证

代码段

复制代码

\begin{algorithm} 
\caption{Self-RAG Reflection Mechanism} 
\begin{algorithmic}[1] 
\Require Query $q$, Generator $G$, Retriever $R$, Reflection tokens $T$ 
\Ensure Verified answer $A$ with reflection traces 
\State $output \leftarrow \text{""}$ 
\State $reflection\_log \leftarrow \text{InitializeEmptyList}()$ 

\While{\textbf{true}} 
    \State $token \leftarrow G.generate\_next(q, output)$ 
    \If{$token \in T$} 
        \If{$token = \text{[Retrieve]}$} 
            \State $context \leftarrow R.retrieve(q)$ 
            \State $reflection\_log.append(\{action: \text{retrieve}, context: context\})$ 
            \State $output \leftarrow output + token + \text{FormatContext}(context)$ 
        \ElsIf{$token = \text{[Verify]}$} 
            \State $claim \leftarrow \text{ExtractLastSentence}(output)$ 
            \State $is\_supported \leftarrow \text{VerifyAgainstContext}(claim, context)$ 
            \State $reflection\_log.append(\{action: \text{verify}, result: is\_supported\})$ 
            \If{$\neg is\_supported$} 
                \State $output \leftarrow output + \text{[Correction]}$ 
                \State $output \leftarrow output + \text{GenerateCorrection}(claim, context)$ 
            \EndIf 
        \ElsIf{$token = \text{[EOS]}$} 
            \State \textbf{break} 
        \EndIf 
    \Else 
        \State $output \leftarrow output + token$ 
    \EndIf 
\EndWhile 

\State $final\_check \leftarrow \text{HallucinationDetection}(output, context)$ 
\State \Return $(output, reflection\_log, final\_check)$ 
\end{algorithmic} 
\end{algorithm}

6.4.4 缓存策略

代码段

复制代码

\begin{algorithm} 
\caption{Semantic Cache with TTL} 
\begin{algorithmic}[1] 
\Require Query $q$, Cache store $S$, Similarity threshold $\theta$, TTL $\Delta t$ 
\Ensure Cached answer $a$ or $\perp$ 
\State $v_q \leftarrow \text{Embed}(q)$ 
\State $candidates \leftarrow S.ann\_search(v_q, k=5)$ 

\For{each $(v_c, a_c, t_c) \in candidates$} 
    \If{$\text{CurrentTime}() - t_c > \Delta t$} 
        \State $S.delete(v_c)$ 
        \State \textbf{continue} 
    \EndIf 
    \State $sim \leftarrow \cos(v_q, v_c)$ 
    \If{$sim > \theta$} 
        \State \Return $a_c$ 
    \EndIf 
\EndFor 

\State \Return $\perp$ 
\end{algorithmic} 
\end{algorithm}

6.5 评估与监控

6.5.1 离线评估

代码段

复制代码

\begin{algorithm} 
\caption{RAGAS Metrics Computation} 
\begin{algorithmic}[1] 
\Require QA pairs $\{(q_i, a_i, c_i)\}_{i=1}^n$, LLM evaluator $\mathcal{L}$ 
\Ensure Metric scores $M$ 
\State $faithfulness\_scores \leftarrow \text{InitializeEmptyList}()$ 
\State $relevancy\_scores \leftarrow \text{InitializeEmptyList}()$ 

\For{each $(q, a, c) \in dataset$} 
    \State \Comment{Faithfulness} 
    \State $claims \leftarrow \mathcal{L}.extract\_statements(a)$ 
    \State $supported \leftarrow 0$ 
    \For{each $claim \in claims$} 
        \State $verdict \leftarrow \mathcal{L}.verify(claim, c)$ 
        \If{$verdict = \text{supported}$} 
            \State $supported \leftarrow supported + 1$ 
        \EndIf 
    \EndFor 
    \State $F \leftarrow \frac{supported}{|claims|}$ 
    \State $faithfulness\_scores.append(F)$ 

    \State \Comment{Answer Relevancy} 
    \State $artificial\_qs \leftarrow \mathcal{L}.generate\_questions(a, k=3)$ 
    \State $\mathbf{v}_q \leftarrow \text{Embed}(q)$ 
    \State $sims \leftarrow \emptyset$ 
    \For{each $q_{gen} \in artificial\_qs$} 
        \State $\mathbf{v}_{gen} \leftarrow \text{Embed}(q_{gen})$ 
        \State $sims \leftarrow sims \cup \{\cos(\mathbf{v}_q, \mathbf{v}_{gen})\}$ 
    \EndFor 
    \State $AR \leftarrow \frac{1}{|sims|} \sum sims$ 
    \State $relevancy\_scores.append(AR)$ 
\EndFor 

\State $M \leftarrow \{faithfulness: \text{Mean}(faithfulness\_scores), relevancy: \text{Mean}(relevancy\_scores)\}$ 
\State \Return $M$ 
\end{algorithmic} 
\end{algorithm}

6.5.2 在线反馈

代码段

复制代码

\begin{algorithm} 
\caption{Online Feedback Collection and Analysis} 
\begin{algorithmic}[1] 
\Require User interaction stream $U$, Feedback window $\Delta$ 
\Ensure Feedback statistics $F$ 
\State $buffer \leftarrow \text{RingBuffer}(size=\Delta)$ 
\State $feedback\_db \leftarrow \text{InitializeDatabase}()$ 

\For{each $u \in U$} 
    \If{$u.type = \text{explicit}$} 
        \State $record \leftarrow \{query: u.q, answer: u.a, rating: u.rating, timestamp: u.t\}$ 
        \State $feedback\_db.insert(record)$ 
    \ElsIf{$u.type = \text{implicit}$} 
        \State $signals \leftarrow \text{AnalyzeBehavior}(u.session)$ 
        \State $inferred\_score \leftarrow \text{CalculateSatisfaction}(signals)$ 
        \State $record \leftarrow \{query: u.q, implicit\_score: inferred\_score, signals: signals\}$ 
        \State $feedback\_db.insert(record)$ 
    \EndIf 

    \If{$feedback\_db.count() \pmod{100} = 0$} 
        \State $stats \leftarrow \text{ComputeStatistics}(feedback\_db)$ 
        \State $\text{AlertIfAnomaly}(stats)$ 
    \EndIf 
\EndFor 
\end{algorithmic} 
\end{algorithm}

6.5.3 A/B测试

代码段

复制代码

\begin{algorithm} 
\caption{A/B Testing Framework for RAG} 
\begin{algorithmic}[1] 
\Require Variants $V=\{v_1, v_2\}$, Traffic split ratio $\rho$, Success metric $M$ 
\Ensure Statistical comparison $C$ 
\State $assignments \leftarrow \text{HashBasedAssignment}(\rho)$ 
\State $results \leftarrow \{v_1: [], v_2: []\}$ 

\For{each incoming query $q$} 
    \State $v \leftarrow assignments.get\_variant(q.user\_id)$ 
    \State $response \leftarrow v.process(q)$ 
    \State $metric\_value \leftarrow M(response, q)$ 
    \State $results[v].append(metric\_value)$ 
\EndFor 

\State $\mu_1, \sigma_1 \leftarrow \text{ComputeMeanStd}(results[v_1])$ 
\State $\mu_2, \sigma_2 \leftarrow \text{ComputeMeanStd}(results[v_2])$ 
\State $n_1, n_2 \leftarrow |results[v_1]|, |results[v_2]|$ 

\State $t \leftarrow \frac{\mu_1 - \mu_2}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}}$ 
\State $p \leftarrow \text{CalculatePValue}(t, dof=n_1+n_2-2)$ 

\If{$p < 0.05$} 
    \State $winner \leftarrow v_1 \textbf{ if } \mu_1 > \mu_2 \textbf{ else } v_2$ 
    \State $\text{Rollout}(winner, traffic=100\%)$ 
\EndIf 

\State \Return $\{mean\_diff: \mu_1 - \mu_2, p\_value: p, significant: p < 0.05\}$ 
\end{algorithmic} 
\end{algorithm}

6.5.4 持续学习

代码段

复制代码

\begin{algorithm} 
\caption{Continual Learning with Bad Case Mining} 
\begin{algorithmic}[1] 
\Require Bad case threshold $\tau$, Retraining dataset size $N$, Model $M$ 
\Ensure Updated model $M'$ 
\State $bad\_cases \leftarrow \text{QueryBadCases}(threshold=\tau)$ 
\State $categories \leftarrow \text{CategorizeFailures}(bad\_cases)$ 

\If{$|bad\_cases| < N$} 
    \State \Return $M$ \Comment{Insufficient data} 
\EndIf 

\State $training\_data \leftarrow \emptyset$ 

\For{each $case \in bad\_cases$} 
    \If{$case.type = \text{retrieval\_failure}$} 
        \State $positives \leftarrow case.ground\_truth\_docs$ 
        \State $negatives \leftarrow \text{HardNegativeMining}(case.query, positives)$ 
        \State $training\_data \leftarrow training\_data \cup \{(case.query, positives, negatives)\}$ 
    \ElsIf{$case.type = \text{generation\_error}$} 
        \State $pair \leftarrow (case.context, case.bad\_answer, case.good\_answer)$ 
        \State $training\_data \leftarrow training\_data \cup \{pair\}$ 
    \EndIf 
\EndFor 

\State $M' \leftarrow \text{FineTune}(M, training\_data, epochs=3)$ 
\State $validation\_score \leftarrow \text{Evaluate}(M', holdout\_set)$ 

\If{$validation\_score > \text{CurrentScore}(M)$} 
    \State $\text{Deploy}(M')$ 
    \State $\text{ClearBadCases}()$ 
\EndIf 

\State \Return $M'$ 
\end{algorithmic} 
\end{algorithm}

复制代码

#!/usr/bin/env python3
"""
Script: context_assembler.py
功能: 上下文组装系统，实现Token预算管理、MMR多样性排序与动态截断
使用方式: python context_assembler.py --query "question" --chunks chunks.json --max-tokens 2000
"""

import json
import argparse
import numpy as np
from typing import List, Dict, Tuple
from dataclasses import dataclass
from transformers import GPT2TokenizerFast


@dataclass
class Chunk:
    """文本块"""
    id: str
    content: str
    score: float  # 相关性分数
    metadata: Dict
    embedding: np.ndarray = None


class TokenBudgetManager:
    """Token预算管理器"""
    
    def __init__(self, 
                 max_tokens: int = 4000,
                 reserve_tokens: int = 500,
                 tokenizer_name: str = "gpt2"):
        """
        初始化
        
        Args:
            max_tokens: 总Token预算
            reserve_tokens: 为生成预留的Token数
            tokenizer_name: 分词器名称
        """
        self.max_tokens = max_tokens
        self.reserve_tokens = reserve_tokens
        self.available_tokens = max_tokens - reserve_tokens
        
        # 加载分词器
        try:
            self.tokenizer = GPT2TokenizerFast.from_pretrained(tokenizer_name)
        except:
            # 备用：使用简单字符估算
            self.tokenizer = None
            self.char_per_token = 4  # 经验值
    
    def count_tokens(self, text: str) -> int:
        """计算文本的Token数"""
        if self.tokenizer:
            return len(self.tokenizer.encode(text))
        return len(text) // self.char_per_token
    
    def fit_chunks(self, 
                   chunks: List[Chunk],
                   query: str,
                   strategy: str = "greedy") -> Tuple[List[Chunk], int]:
        """
        在预算内适配文本块
        
        Args:
            chunks: 候选块
            query: 查询文本（也占用预算）
            strategy: 适配策略 ('greedy', 'mmr', 'diverse')
            
        Returns:
            (选中的块, 使用的Token数)
        """
        query_tokens = self.count_tokens(query)
        remaining = self.available_tokens - query_tokens
        
        if remaining <= 0:
            return [], 0
        
        selected = []
        total_tokens = 0
        
        if strategy == "greedy":
            # 贪婪选择：按相关性排序，依次加入直到预算耗尽
            sorted_chunks = sorted(chunks, key=lambda x: x.score, reverse=True)
            
            for chunk in sorted_chunks:
                chunk_tokens = self.count_tokens(chunk.content)
                
                if total_tokens + chunk_tokens <= remaining:
                    selected.append(chunk)
                    total_tokens += chunk_tokens
                else:
                    # 尝试截断
                    if remaining - total_tokens > 50:  # 至少保留50 tokens
                        truncated = self._truncate_at_sentence(
                            chunk.content, 
                            remaining - total_tokens
                        )
                        if truncated:
                            chunk.content = truncated
                            chunk.metadata["truncated"] = True
                            selected.append(chunk)
                            total_tokens += self.count_tokens(truncated)
                    break
        
        elif strategy == "mmr":
            # 使用MMR选择
            selected = self._select_mmr(chunks, remaining)
            total_tokens = sum(self.count_tokens(c.content) for c in selected)
        
        return selected, total_tokens + query_tokens
    
    def _truncate_at_sentence(self, text: str, max_tokens: int) -> str:
        """在句子边界处截断"""
        target_chars = max_tokens * (self.char_per_token if not self.tokenizer else 4)
        
        # 寻找句子边界
        sentences = []
        current = ""
        for char in text:
            current += char
            if char in ".!?。！？":
                if len(current) + sum(len(s) for s in sentences) < target_chars:
                    sentences.append(current)
                    current = ""
                else:
                    break
        
        return "".join(sentences) if sentences else text[:target_chars]
    
    def _select_mmr(self, 
                   candidates: List[Chunk], 
                   budget: int,
                   lambda_param: float = 0.5) -> List[Chunk]:
        """
        最大边际相关性选择
        
        MMR = λ * Relevance - (1-λ) * max(Similarity with selected)
        """
        selected = []
        remaining = candidates.copy()
        used_budget = 0
        
        while remaining and used_budget < budget:
            mmr_scores = []
            
            for chunk in remaining:
                chunk_tokens = self.count_tokens(chunk.content)
                if used_budget + chunk_tokens > budget:
                    continue
                
                # 相关性
                relevance = chunk.score
                
                # 多样性（与已选块的最大相似度）
                if selected and chunk.embedding is not None:
                    similarities = [
                        self._cosine_sim(chunk.embedding, s.embedding)
                        for s in selected if s.embedding is not None
                    ]
                    max_sim = max(similarities) if similarities else 0
                else:
                    max_sim = 0
                
                mmr_score = lambda_param * relevance - (1 - lambda_param) * max_sim
                mmr_scores.append((chunk, mmr_score, chunk_tokens))
            
            if not mmr_scores:
                break
            
            # 选择MMR分数最高的
            best_chunk, best_score, best_tokens = max(mmr_scores, key=lambda x: x[1])
            selected.append(best_chunk)
            used_budget += best_tokens
            remaining.remove(best_chunk)
        
        return selected
    
    def _cosine_sim(self, a: np.ndarray, b: np.ndarray) -> float:
        """计算余弦相似度"""
        return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-10)


class PriorityAssembler:
    """优先级组装器"""
    
    def __init__(self, token_budget_manager: TokenBudgetManager):
        self.tbm = token_budget_manager
    
    def assemble(self,
                query: str,
                chunks: List[Chunk],
                priority_rules: List[Dict] = None) -> Dict:
        """
        组装上下文
        
        Args:
            query: 查询
            chunks: 候选块（已按某种方式排序）
            priority_rules: 优先级规则
            
        Returns:
            组装结果
        """
        # 按优先级排序（如果有）
        if priority_rules:
            chunks = self._apply_priority_rules(chunks, priority_rules)
        
        # 适配预算
        selected, used_tokens = self.tbm.fit_chunks(chunks, query, strategy="mmr")
        
        # 组装最终文本
        context_parts = []
        for i, chunk in enumerate(selected):
            header = f"[{i+1}] Source: {chunk.metadata.get('source', 'unknown')}"
            if chunk.metadata.get("truncated"):
                header += " (truncated)"
            context_parts.append(f"{header}\n{chunk.content}")
        
        assembled_context = "\n\n".join(context_parts)
        
        return {
            "assembled_context": assembled_context,
            "selected_chunks": [
                {
                    "id": c.id,
                    "content": c.content[:200] + "..." if len(c.content) > 200 else c.content,
                    "original_score": c.score,
                    "metadata": c.metadata
                }
                for c in selected
            ],
            "usage": {
                "total_tokens": used_tokens,
                "max_tokens": self.tbm.max_tokens,
                "reserved_tokens": self.tbm.reserve_tokens,
                "chunk_count": len(selected)
            },
            "truncated": any(c.metadata.get("truncated") for c in selected)
        }
    
    def _apply_priority_rules(self, chunks: List[Chunk], rules: List[Dict]) -> List[Chunk]:
        """应用优先级规则"""
        def get_priority(chunk):
            for rule in rules:
                if self._matches_rule(chunk, rule["condition"]):
                    return rule["priority"]
            return 0
        
        return sorted(chunks, key=lambda x: (get_priority(x), x.score), reverse=True)
    
    def _matches_rule(self, chunk: Chunk, condition: Dict) -> bool:
        """检查块是否匹配条件"""
        for key, value in condition.items():
            if chunk.metadata.get(key) != value:
                return False
        return True


class ContextAssemblerPipeline:
    """完整组装流水线"""
    
    def __init__(self, max_tokens: int = 4000):
        self.tbm = TokenBudgetManager(max_tokens=max_tokens)
        self.assembler = PriorityAssembler(self.tbm)
    
    def process(self,
               query: str,
               retrieved_chunks: List[Dict],
               embeddings: Dict[str, np.ndarray] = None) -> Dict:
        """
        处理检索结果并组装上下文
        
        Args:
            query: 查询
            retrieved_chunks: 检索到的块列表
            embeddings: 块的嵌入向量（用于MMR）
            
        Returns:
            组装结果
        """
        # 转换为Chunk对象
        chunks = []
        for i, rc in enumerate(retrieved_chunks):
            chunk_id = rc.get("id") or rc.get("doc_id") or f"chunk_{i}"
            emb = embeddings.get(chunk_id) if embeddings else None
            
            chunks.append(Chunk(
                id=chunk_id,
                content=rc.get("text") or rc.get("content", ""),
                score=rc.get("score") or rc.get("similarity", 0),
                metadata=rc.get("metadata", {}),
                embedding=emb
            ))
        
        # 组装
        result = self.assembler.assemble(query, chunks)
        
        return result


def main():
    parser = argparse.ArgumentParser(description="上下文组装")
    parser.add_argument("--query", "-q", required=True, help="查询文本")
    parser.add_argument("--chunks", "-c", default=None, help="检索块JSON文件")
    parser.add_argument("--max-tokens", "-t", type=int, default=2000, help="最大Token数")
    parser.add_argument("--reserve", "-r", type=int, default=500, help="预留Token数")
    parser.add_argument("--output", "-o", default="assembled_context.json", help="输出文件")
    
    args = parser.parse_args()
    
    # 加载或生成示例数据
    if args.chunks:
        with open(args.chunks, 'r') as f:
            data = json.load(f)
            if isinstance(data, dict):
                retrieved = data.get("results", [])
            else:
                retrieved = data
    else:
        # 生成示例
        retrieved = [
            {
                "id": f"doc_{i}",
                "content": f"This is a detailed explanation about topic {i}. " * 20 + 
                          f"It contains important information relevant to the query.",
                "score": 0.95 - i*0.05,
                "metadata": {"source": f"document_{i}.pdf", "page": i+1}
            }
            for i in range(10)
        ]
    
    print(f"[INFO] 加载了 {len(retrieved)} 个候选块")
    print(f"[INFO] 查询: {args.query}")
    
    # 生成模拟嵌入（实际应从向量数据库获取）
    np.random.seed(42)
    embeddings = {}
    for r in retrieved:
        emb = np.random.randn(384)
        emb = emb / np.linalg.norm(emb)
        embeddings[r["id"]] = emb
    
    # 组装
    pipeline = ContextAssemblerPipeline(max_tokens=args.max_tokens)
    result = pipeline.process(args.query, retrieved, embeddings)
    
    print(f"\n组装完成:")
    print(f"  选中块数: {result['usage']['chunk_count']}")
    print(f"  使用Token: {result['usage']['total_tokens']}/{result['usage']['max_tokens']}")
    print(f"  是否截断: {result['truncated']}")
    print(f"\n组装后的上下文 (前500字符):\n{result['assembled_context'][:500]}...")
    
    # 保存
    with open(args.output, 'w') as f:
        json.dump(result, f, indent=2, ensure_ascii=False)
    print(f"\n结果已保存至: {args.output}")


if __name__ == "__main__":
    main()

6.4.2 引用溯源

Python

复制代码

#!/usr/bin/env python3
"""
Script: citation_attribution.py
功能: 答案引用溯源系统，实现检索结果高亮、原文链接生成与NLI验证
使用方式: python citation_attribution.py --answer generated_answer.txt --context assembled_context.json
"""

import json
import re
import argparse
from typing import List, Dict, Tuple, Optional
from dataclasses import dataclass
import numpy as np
from sentence_transformers import CrossEncoder


@dataclass
class AttributionResult:
    """溯源结果"""
    claim: str
    supporting_evidence: List[Dict]
    confidence: float
    citation_markers: List[str]


class NLIVerifier:
    """自然语言推理验证器"""
    
    def __init__(self, model_name: str = "cross-encoder/nli-deberta-v3-base"):
        """
        初始化NLI模型
        
        Args:
            model_name: 交叉编码器模型
        """
        print(f"[INFO] 加载NLI模型: {model_name}")
        self.model = CrossEncoder(model_name)
        self.labels = ["contradiction", "entailment", "neutral"]
    
    def verify(self, premise: str, hypothesis: str) -> Dict:
        """
        验证假设是否被前提支持
        
        Args:
            premise: 前提（检索到的文本）
            hypothesis: 假设（生成的陈述）
            
        Returns:
            验证结果
        """
        scores = self.model.predict([[premise, hypothesis]], apply_softmax=True)[0]
        
        label_idx = np.argmax(scores)
        return {
            "label": self.labels[label_idx],
            "confidence": float(scores[label_idx]),
            "scores": {
                "contradiction": float(scores[0]),
                "entailment": float(scores[1]),
                "neutral": float(scores[2])
            }
        }
    
    def batch_verify(self, 
                    premise_hypothesis_pairs: List[Tuple[str, str]]) -> List[Dict]:
        """批量验证"""
        if not premise_hypothesis_pairs:
            return []
        
        scores = self.model.predict(premise_hypothesis_pairs, apply_softmax=True)
        
        results = []
        for score_vec in scores:
            label_idx = np.argmax(score_vec)
            results.append({
                "label": self.labels[label_idx],
                "confidence": float(score_vec[label_idx]),
                "entailment_score": float(score_vec[1])
            })
        return results


class ClaimExtractor:
    """陈述提取器"""
    
    def extract(self, text: str) -> List[str]:
        """
        将文本分割为原子陈述
        
        Args:
            text: 输入文本
            
        Returns:
            陈述列表
        """
        # 按句子分割
        sentences = re.split(r'(?<=[.!?。！？])\s+', text)
        
        # 过滤太短或太长的句子
        claims = []
        for sent in sentences:
            sent = sent.strip()
            if 10 < len(sent) < 500:
                # 移除引用标记如 [1], [2], etc.
                clean_sent = re.sub(r'\[\d+\]', '', sent).strip()
                if clean_sent:
                    claims.append(clean_sent)
        
        return claims


class CitationGenerator:
    """引用生成器"""
    
    def __init__(self, use_brackets: bool = True):
        self.use_brackets = use_brackets
        self.citation_counter = 0
        self.citation_map = {}
    
    def generate_marker(self, source_id: str, source_meta: Dict) -> str:
        """生成引用标记"""
        if source_id not in self.citation_map:
            self.citation_counter += 1
            self.citation_map[source_id] = {
                "number": self.citation_counter,
                "metadata": source_meta
            }
        
        num = self.citation_map[source_id]["number"]
        return f"[{num}]" if self.use_brackets else f"({num})"
    
    def generate_bibliography(self) -> List[Dict]:
        """生成参考文献列表"""
        return [
            {
                "citation_number": info["number"],
                "source_id": sid,
                **info["metadata"]
            }
            for sid, info in sorted(self.citation_map.items(), key=lambda x: x[1]["number"])
        ]


class AttributionEngine:
    """溯源引擎"""
    
    def __init__(self, 
                 nli_model: str = "cross-encoder/nli-deberta-v3-base",
                 entailment_threshold: float = 0.7):
        """
        初始化
        
        Args:
            nli_model: NLI模型名称
            entailment_threshold: 蕴含判定阈值
        """
        self.nli = NLIVerifier(nli_model)
        self.claim_extractor = ClaimExtractor()
        self.citation_gen = CitationGenerator()
        self.threshold = entailment_threshold
    
    def attribute(self,
                 generated_answer: str,
                 retrieved_contexts: List[Dict],
                 link_format: str = "markdown") -> Dict:
        """
        执行溯源
        
        Args:
            generated_answer: 生成的答案
            retrieved_contexts: 检索上下文（包含id, text, metadata）
            link_format: 链接格式 ('markdown', 'html', 'plain')
            
        Returns:
            溯源结果
        """
        # 提取陈述
        claims = self.claim_extractor.extract(generated_answer)
        print(f"[INFO] 提取了 {len(claims)} 个陈述")
        
        # 为每个陈述寻找证据
        attributions = []
        all_verifications = []
        
        for claim in claims:
            # 验证与每个上下文的关系
            evidence_list = []
            verification_pairs = [
                (ctx.get("text") or ctx.get("content", ""), claim) 
                for ctx in retrieved_contexts
            ]
            
            verifications = self.nli.batch_verify(verification_pairs)
            
            for ctx, verify_result in zip(retrieved_contexts, verifications):
                if verify_result["label"] == "entailment" and verify_result["confidence"] > 0.5:
                    evidence_list.append({
                        "context_id": ctx.get("id") or ctx.get("chunk_id", "unknown"),
                        "context_text": (ctx.get("text") or ctx.get("content", ""))[:200],
                        "confidence": verify_result["confidence"],
                        "verification_score": verify_result["entailment_score"],
                        "metadata": ctx.get("metadata", {})
                    })
            
            # 排序证据
            evidence_list.sort(key=lambda x: x["confidence"], reverse=True)
            
            # 生成引用标记
            citations = []
            for ev in evidence_list[:3]:  # 最多3个引用
                marker = self.citation_gen.generate_marker(
                    ev["context_id"], 
                    ev["metadata"]
                )
                citations.append(marker)
            
            attributions.append(AttributionResult(
                claim=claim,
                supporting_evidence=evidence_list[:3],
                confidence=max([e["confidence"] for e in evidence_list]) if evidence_list else 0.0,
                citation_markers=citations
            ))
        
        # 生成带引用的答案
        attributed_answer = self._insert_citations(generated_answer, attributions, link_format)
        
        # 计算整体忠实度
        faithfulness = sum(1 for a in attributions if a.confidence > self.threshold) / len(claims) if claims else 0
        
        return {
            "attributed_answer": attributed_answer,
            "claims": [
                {
                    "text": a.claim,
                    "citations": a.citation_markers,
                    "evidence_count": len(a.supporting_evidence),
                    "confidence": a.confidence
                }
                for a in attributions
            ],
            "bibliography": self.citation_gen.generate_bibliography(),
            "faithfulness_score": faithfulness,
            "supported_claims": sum(1 for a in attributions if a.supporting_evidence),
            "total_claims": len(claims)
        }
    
    def _insert_citations(self, 
                         answer: str, 
                         attributions: List[AttributionResult],
                         format: str) -> str:
        """在答案中插入引用标记"""
        result = answer
        
        # 简单实现：在句子末尾插入引用
        for attr in attributions:
            if attr.citation_markers:
                citation_str = "".join(attr.citation_markers)
                # 在陈述后插入引用（简化处理）
                escaped_claim = re.escape(attr.claim[:50])
                pattern = f"({escaped_claim}.*?)([.!?])"
                replacement = f"\\1{citation_str}\\2"
                result = re.sub(pattern, replacement, result, count=1)
        
        return result
    
    def highlight_evidence(self, 
                          context: str, 
                          claim: str,
                          output_format: str = "html") -> str:
        """
        高亮证据文本
        
        Args:
            context: 原始上下文
            claim: 支持的陈述
            output_format: 输出格式
            
        Returns:
            带高亮的文本
        """
        # 简单实现：将相关句子标记为高亮
        sentences = re.split(r'(?<=[.!?])\s+', context)
        
        # 找到最相似的句子（简化版）
        highlighted = []
        for sent in sentences:
            # 简单重叠检测
            claim_words = set(claim.lower().split())
            sent_words = set(sent.lower().split())
            overlap = len(claim_words & sent_words) / len(claim_words) if claim_words else 0
            
            if overlap > 0.3:
                if output_format == "html":
                    sent = f'<mark style="background-color: yellow;">{sent}</mark>'
                elif output_format == "markdown":
                    sent = f"**{sent}**"
            
            highlighted.append(sent)
        
        return " ".join(highlighted)


def main():
    parser = argparse.ArgumentParser(description="引用溯源")
    parser.add_argument("--answer", "-a", required=True, help="生成的答案文本或文件")
    parser.add_argument("--context", "-c", required=True, help="上下文JSON文件")
    parser.add_argument("--output", "-o", default="attribution_result.json", help="输出文件")
    parser.add_argument("--format", "-f", choices=["markdown", "html", "plain"], 
                       default="markdown", help="引用格式")
    parser.add_argument("--threshold", "-t", type=float, default=0.7, help="蕴含阈值")
    
    args = parser.parse_args()
    
    # 加载答案
    if os.path.exists(args.answer):
        with open(args.answer, 'r') as f:
            answer = f.read()
    else:
        answer = args.answer
    
    # 加载上下文
    with open(args.context, 'r') as f:
        contexts = json.load(f)
        if isinstance(contexts, dict):
            contexts = contexts.get("selected_chunks", [])
    
    print(f"[INFO] 答案长度: {len(answer)} 字符")
    print(f"[INFO] 上下文块数: {len(contexts)}")
    
    # 执行溯源
    engine = AttributionEngine(entailment_threshold=args.threshold)
    result = engine.attribute(answer, contexts, link_format=args.format)
    
    print(f"\n溯源结果:")
    print(f"  忠实度分数: {result['faithfulness_score']:.2f}")
    print(f"  支持陈述: {result['supported_claims']}/{result['total_claims']}")
    print(f"\n带引用的答案:\n{result['attributed_answer'][:500]}...")
    print(f"\n参考文献:")
    for bib in result['bibliography'][:3]:
        print(f"  [{bib['citation_number']}] {bib.get('source', 'unknown')}")
    
    # 保存
    with open(args.output, 'w') as f:
        json.dump(result, f, indent=2, ensure_ascii=False)
    print(f"\n结果已保存至: {args.output}")


if __name__ == "__main__":
    import os
    main()

6.4.3 答案验证

Python

复制代码

#!/usr/bin/env python3
"""
Script: self_rag_verification.py
功能: Self-RAG实现，包含反思机制、幻觉检测与自我修正
使用方式: python self_rag_verification.py --query "question" --context context.json --model gpt-4
"""

import json
import re
import argparse
from typing import List, Dict, Optional, Tuple
from dataclasses import dataclass
from enum import Enum


class ReflectionToken(Enum):
    """反思标记"""
    RETRIEVE = "[Retrieve]"
    NO_RETRIEVE = "[No Retrieve]"
    RELEVANT = "[Relevant]"
    IRRELEVANT = "[Irrelevant]"
    SUPPORTED = "[Supported]"
    CONTRADICTORY = "[Contradictory]"
    CORRECTION = "[Correction]"
    FINISH = "[Finish]"


@dataclass
class ReflectionTrace:
    """反思轨迹"""
    step: int
    action: str
    content: str
    verification_result: Optional[str] = None


class SelfRAG:
    """
    Self-Reflective Retrieval-Augmented Generation
    
    基于Asai et al. (2023)的Self-RAG框架实现，
    支持自适应检索与自我验证。
    """
    
    def __init__(self, 
                 llm_provider: str = "openai",
                 api_key: Optional[str] = None,
                 model: str = "gpt-4",
                 reflection_trigger: str = "adaptive"):
        """
        初始化Self-RAG
        
        Args:
            llm_provider: LLM提供者
            api_key: API密钥
            model: 模型名称
            reflection_trigger: 反思触发策略 ('always', 'adaptive', 'never')
        """
        self.llm_provider = llm_provider
        self.model = model
        self.reflection_trigger = reflection_trigger
        self.api_key = api_key
        
        self._init_llm()
        self.traces = []
    
    def _init_llm(self):
        """初始化LLM"""
        if self.llm_provider == "openai":
            try:
                from openai import OpenAI
            except ImportError:
                raise ImportError("请安装openai")
            
            import os
            self.client = OpenAI(api_key=self.api_key or os.getenv("OPENAI_API_KEY"))
        else:
            raise ValueError(f"不支持的提供者: {self.llm_provider}")
    
    def generate_with_reflection(self,
                                  query: str,
                                  contexts: List[Dict],
                                  max_iterations: int = 3) -> Dict:
        """
        带反思的生成
        
        Args:
            query: 查询
            contexts: 检索上下文
            max_iterations: 最大迭代次数
            
        Returns:
            生成结果与反思轨迹
        """
        self.traces = []
        current_answer = ""
        iteration = 0
        
        # 初始反思：是否需要检索
        if self.reflection_trigger == "always" or self._needs_retrieval(query):
            self.traces.append(ReflectionTrace(0, "initial", ReflectionToken.RETRIEVE.value))
            current_context = self._format_contexts(contexts)
        else:
            self.traces.append(ReflectionTrace(0, "initial", ReflectionToken.NO_RETRIEVE.value))
            current_context = ""
        
        while iteration < max_iterations:
            iteration += 1
            
            # 生成
            prompt = self._build_prompt(query, current_context, current_answer)
            response = self._call_llm(prompt)
            
            # 解析反思标记
            parsed = self._parse_reflection_tokens(response)
            
            if parsed["action"] == "finish":
                current_answer += parsed["content"]
                self.traces.append(ReflectionTrace(
                    iteration, "generate", parsed["content"], "finished"
                ))
                break
            
            elif parsed["action"] == "retrieve":
                # 请求更多检索（简化处理，实际应触发新的检索）
                self.traces.append(ReflectionTrace(
                    iteration, "retrieve", "Requesting more information", None
                ))
                break
            
            elif parsed["action"] == "verify":
                # 验证生成的内容
                claim = parsed["content"]
                verification = self._verify_claim(claim, contexts)
                
                self.traces.append(ReflectionTrace(
                    iteration, "verify", claim, verification["result"]
                ))
                
                if verification["result"] == "unsupported":
                    # 需要修正
                    correction = self._generate_correction(claim, contexts)
                    current_answer += correction + " "
                    self.traces.append(ReflectionTrace(
                        iteration, "correct", correction, None
                    ))
                else:
                    current_answer += claim + " "
            
            else:
                current_answer += parsed["content"] + " "
        
        # 最终验证
        final_check = self._hallucination_check(current_answer, contexts)
        
        return {
            "answer": current_answer.strip(),
            "reflection_traces": [
                {
                    "step": t.step,
                    "action": t.action,
                    "content": t.content[:100] + "..." if len(t.content) > 100 else t.content,
                    "verification": t.verification_result
                }
                for t in self.traces
            ],
            "hallucination_detected": final_check["has_hallucination"],
            "faithfulness_score": final_check["faithfulness"],
            "iterations": iteration
        }
    
    def _needs_retrieval(self, query: str) -> bool:
        """判断是否需要检索"""
        # 启发式规则：事实性问题通常需要检索
        factual_patterns = r'\b(what|who|when|where|how many|how much|is|are|was|were)\b'
        return bool(re.search(factual_patterns, query.lower()))
    
    def _format_contexts(self, contexts: List[Dict]) -> str:
        """格式化上下文"""
        parts = []
        for i, ctx in enumerate(contexts, 1):
            text = ctx.get("text") or ctx.get("content", "")
            source = ctx.get("metadata", {}).get("source", f"doc_{i}")
            parts.append(f"[{i}] Source: {source}\n{text}")
        return "\n\n".join(parts)
    
    def _build_prompt(self, query: str, context: str, partial_answer: str) -> str:
        """构建提示"""
        reflection_instructions = """
You are a Self-RAG system. Follow these rules:
1. Generate the answer step by step
2. Use [Retrieve] if you need more information
3. Use [Verify] before stating a fact, then wait for feedback
4. Use [Supported] or [Contradictory] based on verification
5. Use [Finish] when the answer is complete

Available context:
{context}

Current partial answer: {partial}
        """.format(context=context, partial=partial_answer or "None")
        
        return f"{reflection_instructions}\n\nQuestion: {query}\nAnswer:"
    
    def _call_llm(self, prompt: str) -> str:
        """调用LLM"""
        if self.llm_provider == "openai":
            response = self.client.chat.completions.create(
                model=self.model,
                messages=[
                    {"role": "system", "content": "You are a helpful assistant with reflection capabilities."},
                    {"role": "user", "content": prompt}
                ],
                temperature=0.7,
                max_tokens=500
            )
            return response.choices[0].message.content
        return ""
    
    def _parse_reflection_tokens(self, text: str) -> Dict:
        """解析反思标记"""
        # 检测特殊标记
        tokens = [t.value for t in ReflectionToken]
        
        for token in tokens:
            if token in text:
                if token == ReflectionToken.FINISH.value:
                    content = text.split(token)[0].strip()
                    return {"action": "finish", "content": content}
                elif token == ReflectionToken.RETRIEVE.value:
                    return {"action": "retrieve", "content": text}
                elif token == ReflectionToken.VERIFY.value:
                    content = text.split(token)[1].split(".")[0].strip()
                    return {"action": "verify", "content": content}
                elif token in [ReflectionToken.SUPPORTED.value, ReflectionToken.CONTRADICTORY.value]:
                    return {"action": "verified", "content": text}
        
        return {"action": "continue", "content": text}
    
    def _verify_claim(self, claim: str, contexts: List[Dict]) -> Dict:
        """验证陈述"""
        # 简化实现：检查与上下文的相似度
        from sentence_transformers import SentenceTransformer, util
        
        model = SentenceTransformer('all-MiniLM-L6-v2')
        
        claim_emb = model.encode(claim, convert_to_tensor=True)
        
        best_score = 0
        for ctx in contexts:
            ctx_text = ctx.get("text") or ctx.get("content", "")
            ctx_emb = model.encode(ctx_text, convert_to_tensor=True)
            score = util.pytorch_cos_sim(claim_emb, ctx_emb).item()
            best_score = max(best_score, score)
        
        threshold = 0.7
        if best_score > threshold:
            return {"result": "supported", "confidence": best_score}
        else:
            return {"result": "unsupported", "confidence": best_score}
    
    def _generate_correction(self, wrong_claim: str, contexts: List[Dict]) -> str:
        """生成修正"""
        context_text = " ".join([
            (c.get("text") or c.get("content", ""))[:500] 
            for c in contexts[:2]
        ])
        
        prompt = f"""Based on the following context, correct the previous statement:
Context: {context_text}

Incorrect statement: {wrong_claim}

Corrected statement:"""
        
        return self._call_llm(prompt)
    
    def _hallucination_check(self, answer: str, contexts: List[Dict]) -> Dict:
        """幻觉检测"""
        # 提取陈述并验证
        sentences = re.split(r'(?<=[.!?])\s+', answer)
        
        verified = 0
        for sent in sentences:
            if len(sent.strip()) < 10:
                continue
            result = self._verify_claim(sent, contexts)
            if result["result"] == "supported":
                verified += 1
        
        total = len([s for s in sentences if len(s.strip()) > 10])
        faithfulness = verified / total if total > 0 else 1.0
        
        return {
            "has_hallucination": faithfulness < 0.8,
            "faithfulness": faithfulness,
            "verified_statements": verified,
            "total_statements": total
        }


def main():
    parser = argparse.ArgumentParser(description="Self-RAG验证")
    parser.add_argument("--query", "-q", required=True, help="查询")
    parser.add_argument("--context", "-c", required=True, help="上下文JSON")
    parser.add_argument("--api-key", "-k", default=None, help="API Key")
    parser.add_argument("--model", "-m", default="gpt-4", help="模型")
    parser.add_argument("--output", "-o", default="self_rag_result.json", help="输出")
    
    args = parser.parse_args()
    
    # 加载上下文
    with open(args.context, 'r') as f:
        contexts = json.load(f)
        if isinstance(contexts, dict):
            contexts = contexts.get("selected_chunks", [])
    
    # 初始化Self-RAG
    self_rag = SelfRAG(
        llm_provider="openai",
        api_key=args.api_key,
        model=args.model,
        reflection_trigger="adaptive"
    )
    
    # 生成
    print(f"[INFO] 执行Self-RAG生成...")
    result = self_rag.generate_with_reflection(args.query, contexts)
    
    print(f"\n最终答案:\n{result['answer']}")
    print(f"\n忠实度: {result['faithfulness_score']:.2f}")
    print(f"幻觉检测: {'是' if result['hallucination_detected'] else '否'}")
    print(f"迭代次数: {result['iterations']}")
    
    # 保存
    with open(args.output, 'w') as f:
        json.dump(result, f, indent=2, ensure_ascii=False)
    print(f"\n结果已保存至: {args.output}")


if __name__ == "__main__":
    main()

6.4.4 缓存策略

Python

复制代码

#!/usr/bin/env python3
"""
Script: semantic_cache.py
功能: 语义缓存(Semantic Cache)与精确匹配缓存实现，含缓存命中率可视化
使用方式: python semantic_cache.py --query "your question" --cache-file cache.db
"""

import json
import hashlib
import argparse
import numpy as np
import pickle
import time
from typing import Dict, List, Optional, Tuple
from dataclasses import dataclass, asdict
from datetime import datetime, timedelta
import sqlite3


@dataclass
class CacheEntry:
    """缓存条目"""
    query_hash: str
    query_embedding: np.ndarray
    answer: str
    timestamp: float
    hit_count: int = 0
    ttl: int = 3600  # 默认1小时


class SemanticCache:
    """
    语义缓存
    
    基于查询嵌入向量的相似性进行缓存匹配，
    支持TTL（生存时间）与LRU（最近最少使用）策略。
    """
    
    def __init__(self, 
                 embedding_dim: int = 384,
                 similarity_threshold: float = 0.95,
                 ttl_seconds: int = 3600,
                 max_size: int = 10000):
        """
        初始化语义缓存
        
        Args:
            embedding_dim: 嵌入维度
            similarity_threshold: 相似度阈值
            ttl_seconds: 生存时间
            max_size: 最大条目数
        """
        self.embedding_dim = embedding_dim
        self.threshold = similarity_threshold
        self.ttl = ttl_seconds
        self.max_size = max_size
        
        # 内存缓存
        self.cache: Dict[str, CacheEntry] = {}
        self.query_hashes = []  # 用于LRU
        
        # 嵌入模型（用于新查询编码）
        self.embedder = None
        self._init_embedder()
        
        # 统计
        self.stats = {"hits": 0, "misses": 0, "semantic_hits": 0}
    
    def _init_embedder(self):
        """初始化嵌入模型"""
        try:
            from sentence_transformers import SentenceTransformer
            self.embedder = SentenceTransformer('all-MiniLM-L6-v2')
        except:
            print("[WARN] 无法加载嵌入模型，使用模拟模式")
            self.embedder = None
    
    def _get_embedding(self, query: str) -> np.ndarray:
        """获取查询嵌入"""
        if self.embedder:
            return self.embedder.encode(query)
        else:
            # 模拟嵌入（基于哈希）
            np.random.seed(hash(query) % 2**32)
            vec = np.random.randn(self.embedding_dim)
            return vec / np.linalg.norm(vec)
    
    def _compute_similarity(self, emb1: np.ndarray, emb2: np.ndarray) -> float:
        """计算余弦相似度"""
        return np.dot(emb1, emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2) + 1e-10)
    
    def _is_expired(self, entry: CacheEntry) -> bool:
        """检查是否过期"""
        return time.time() - entry.timestamp > self.ttl
    
    def get(self, query: str) -> Tuple[Optional[str], str]:
        """
        获取缓存
        
        Args:
            query: 查询文本
            
        Returns:
            (缓存的答案, 命中类型)
        """
        query_hash = hashlib.md5(query.encode()).hexdigest()
        
        # L1: 精确匹配
        if query_hash in self.cache:
            entry = self.cache[query_hash]
            if not self._is_expired(entry):
                entry.hit_count += 1
                entry.timestamp = time.time()  # 更新LRU
                self.stats["hits"] += 1
                return entry.answer, "exact"
            else:
                # 过期删除
                del self.cache[query_hash]
                self.query_hashes.remove(query_hash)
        
        # L2: 语义匹配
        query_emb = self._get_embedding(query)
        
        best_match = None
        best_sim = 0
        
        for key, entry in self.cache.items():
            if self._is_expired(entry):
                continue
            
            sim = self._compute_similarity(query_emb, entry.query_embedding)
            if sim > best_sim and sim >= self.threshold:
                best_sim = sim
                best_match = entry
        
        if best_match:
            best_match.hit_count += 1
            self.stats["semantic_hits"] += 1
            self.stats["hits"] += 1
            return best_match.answer, f"semantic({best_sim:.3f})"
        
        self.stats["misses"] += 1
        return None, "miss"
    
    def put(self, query: str, answer: str, embedding: Optional[np.ndarray] = None):
        """
        存入缓存
        
        Args:
            query: 查询
            answer: 答案
            embedding: 嵌入向量（可选，自动计算）
        """
        # 检查容量
        if len(self.cache) >= self.max_size:
            # LRU淘汰
            oldest = min(self.cache.values(), key=lambda x: x.timestamp)
            del self.cache[oldest.query_hash]
            self.query_hashes.remove(oldest.query_hash)
        
        query_hash = hashlib.md5(query.encode()).hexdigest()
        emb = embedding if embedding is not None else self._get_embedding(query)
        
        entry = CacheEntry(
            query_hash=query_hash,
            query_embedding=emb,
            answer=answer,
            timestamp=time.time()
        )
        
        self.cache[query_hash] = entry
        self.query_hashes.append(query_hash)
    
    def get_stats(self) -> Dict:
        """获取统计信息"""
        total = self.stats["hits"] + self.stats["misses"]
        hit_rate = self.stats["hits"] / total if total > 0 else 0
        
        return {
            "total_queries": total,
            "cache_hits": self.stats["hits"],
            "exact_hits": self.stats["hits"] - self.stats["semantic_hits"],
            "semantic_hits": self.stats["semantic_hits"],
            "cache_misses": self.stats["misses"],
            "hit_rate": hit_rate,
            "semantic_hit_rate": self.stats["semantic_hits"] / total if total > 0 else 0,
            "current_size": len(self.cache),
            "max_size": self.max_size
        }


class PersistentSemanticCache(SemanticCache):
    """持久化语义缓存（基于SQLite）"""
    
    def __init__(self, db_path: str = "semantic_cache.db", **kwargs):
        super().__init__(**kwargs)
        self.db_path = db_path
        self._init_db()
        self._load_from_db()
    
    def _init_db(self):
        """初始化数据库"""
        self.conn = sqlite3.connect(self.db_path, check_same_thread=False)
        cursor = self.conn.cursor()
        
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS cache_entries (
                query_hash TEXT PRIMARY KEY,
                query_embedding BLOB,
                answer TEXT,
                timestamp REAL,
                hit_count INTEGER DEFAULT 0,
                ttl INTEGER
            )
        """)
        
        self.conn.commit()
    
    def _load_from_db(self):
        """从数据库加载缓存"""
        cursor = self.conn.cursor()
        cursor.execute("SELECT * FROM cache_entries")
        
        for row in cursor.fetchall():
            hash_val, emb_blob, answer, ts, hits, ttl = row
            emb = np.frombuffer(emb_blob, dtype=np.float32)
            
            # 检查过期
            if time.time() - ts > ttl:
                continue
            
            entry = CacheEntry(
                query_hash=hash_val,
                query_embedding=emb,
                answer=answer,
                timestamp=ts,
                hit_count=hits,
                ttl=ttl
            )
            self.cache[hash_val] = entry
    
    def put(self, query: str, answer: str, embedding: Optional[np.ndarray] = None):
        """存入缓存（覆盖父类以持久化）"""
        super().put(query, answer, embedding)
        
        # 保存到数据库
        entry = self.cache[hashlib.md5(query.encode()).hexdigest()]
        cursor = self.conn.cursor()
        
        cursor.execute("""
            INSERT OR REPLACE INTO cache_entries 
            (query_hash, query_embedding, answer, timestamp, hit_count, ttl)
            VALUES (?, ?, ?, ?, ?, ?)
        """, (
            entry.query_hash,
            entry.query_embedding.astype(np.float32).tobytes(),
            entry.answer,
            entry.timestamp,
            entry.hit_count,
            entry.ttl
        ))
        
        self.conn.commit()


class CacheVisualizer:
    """缓存可视化"""
    
    def visualize_stats(self, stats: Dict, output_path: str = "cache_stats.png"):
        """可视化统计"""
        import matplotlib.pyplot as plt
        
        fig, axes = plt.subplots(1, 2, figsize=(12, 5))
        
        # 1. 命中率饼图
        ax1 = axes[0]
        sizes = [stats["exact_hits"], stats["semantic_hits"], stats["cache_misses"]]
        labels = ['Exact Match', 'Semantic Match', 'Misses']
        colors = ['#66b3ff', '#99ff99', '#ff9999']
        explode = (0.05, 0.05, 0)
        
        ax1.pie(sizes, explode=explode, labels=labels, colors=colors, autopct='%1.1f%%',
                shadow=True, startangle=90)
        ax1.set_title(f'Cache Hit Rate (Total: {stats["total_queries"]})')
        
        # 2. 缓存使用情况
        ax2 = axes[1]
        usage = stats["current_size"] / stats["max_size"] * 100
        ax2.barh(['Cache Usage'], [usage], color='skyblue')
        ax2.set_xlim([0, 100])
        ax2.set_xlabel('Percentage (%)')
        ax2.set_title(f'Cache Capacity: {stats["current_size"]}/{stats["max_size"]}')
        ax2.text(usage + 2, 0, f'{usage:.1f}%', va='center')
        
        plt.tight_layout()
        plt.savefig(output_path, dpi=300, bbox_inches='tight')
        print(f"[INFO] 统计图表已保存至: {output_path}")
        plt.close()


def main():
    parser = argparse.ArgumentParser(description="语义缓存")
    parser.add_argument("--query", "-q", default=None, help="查询文本")
    parser.add_argument("--answer", "-a", default=None, help="答案文本")
    parser.add_argument("--db-file", "-d", default="semantic_cache.db", help="数据库文件")
    parser.add_argument("--threshold", "-t", type=float, default=0.95, help="相似度阈值")
    parser.add_argument("--demo", action="store_true", help="运行演示")
    
    args = parser.parse_args()
    
    # 初始化缓存
    cache = PersistentSemanticCache(
        db_path=args.db_file,
        similarity_threshold=args.threshold
    )
    
    if args.demo:
        print("[DEMO] 运行缓存演示...")
        
        # 模拟查询序列
        queries = [
            "What is machine learning?",
            "What is machine learning",  # 近似重复（语义匹配）
            "How does deep learning work?",
            "What is machine learning?",  # 精确重复
            "Explain neural networks",
            "How does deep learning work"  # 近似重复
        ]
        
        answers = [
            "Machine learning is a subset of AI...",
            "Deep learning uses neural networks with multiple layers...",
            "Neural networks are computing systems inspired by biological neural networks..."
        ]
        
        for i, q in enumerate(queries):
            # 尝试获取
            cached, hit_type = cache.get(q)
            
            if cached:
                print(f"Query {i+1}: '{q[:40]}...' -> [HIT: {hit_type}]")
            else:
                # 模拟生成
                ans = answers[i % len(answers)]
                cache.put(q, ans)
                print(f"Query {i+1}: '{q[:40]}...' -> [MISS] (Cached)")
        
        # 打印统计
        stats = cache.get_stats()
        print(f"\n缓存统计:")
        print(f"  命中率: {stats['hit_rate']:.2%}")
        print(f"  精确匹配: {stats['exact_hits']}")
        print(f"  语义匹配: {stats['semantic_hits']}")
        
        # 可视化
        visualizer = CacheVisualizer()
        visualizer.visualize_stats(stats)
    
    elif args.query and args.answer:
        # 存入
        cache.put(args.query, args.answer)
        print(f"[INFO] 已缓存查询")
    
    elif args.query:
        # 查询
        cached, hit_type = cache.get(args.query)
        if cached:
            print(f"[HIT: {hit_type}] 答案: {cached[:200]}...")
        else:
            print("[MISS] 缓存未命中")
    
    else:
        parser.print_help()


if __name__ == "__main__":
    main()

6.5.1 离线评估

Python

复制代码

#!/usr/bin/env python3
"""
Script: ragas_evaluation.py
功能: RAGAS指标计算（Faithfulness, Answer Relevancy, Context Precision/Recall），含可视化报告
使用方式: python ragas_evaluation.py --qa-pairs qa.json --output evaluation_report.json
"""

import json
import argparse
import numpy as np
from typing import List, Dict, Tuple
from dataclasses import dataclass
from sentence_transformers import SentenceTransformer, util


@dataclass
class QAPair:
    """问答对"""
    question: str
    answer: str
    contexts: List[str]
    ground_truth: str = None


class RAGASEvaluator:
    """
    RAGAS评估器
    
    实现RAGAS框架的核心指标：
    - Faithfulness: 答案对上下文的事实一致性
    - Answer Relevancy: 答案与问题的相关性
    - Context Precision: 检索上下文的相关比例
    - Context Recall: 检索上下文对答案的覆盖度
    """
    
    def __init__(self, 
                 llm_provider: str = "local",
                 embedding_model: str = "all-MiniLM-L6-v2"):
        """
        初始化评估器
        
        Args:
            llm_provider: LLM提供者（用于NLI和生成）
            embedding_model: 嵌入模型
        """
        self.llm_provider = llm_provider
        self.embedder = SentenceTransformer(embedding_model)
        
        if llm_provider == "openai":
            try:
                from openai import OpenAI
                self.client = OpenAI()
            except:
                self.client = None
    
    def faithfulness(self, answer: str, contexts: List[str]) -> Dict:
        """
        计算忠实度
        
        步骤：
        1. 将答案分解为原子陈述
        2. 验证每个陈述是否被上下文支持
        
        Args:
            answer: 生成的答案
            contexts: 检索上下文列表
            
        Returns:
            忠实度分数和详细分析
        """
        # 陈述提取（简化：按句子分割）
        statements = self._extract_statements(answer)
        
        if not statements:
            return {"score": 0.0, "reason": "No statements extracted"}
        
        # 验证每个陈述
        supported = 0
        details = []
        
        for stmt in statements:
            # 检查是否有上下文支持该陈述
            is_supported, best_evidence = self._verify_statement(stmt, contexts)
            
            if is_supported:
                supported += 1
            
            details.append({
                "statement": stmt,
                "supported": is_supported,
                "evidence": best_evidence[:200] if best_evidence else None
            })
        
        score = supported / len(statements)
        
        return {
            "score": score,
            "supported_statements": supported,
            "total_statements": len(statements),
            "details": details
        }
    
    def answer_relevancy(self, question: str, answer: str) -> Dict:
        """
        计算答案相关性
        
        方法：生成潜在问题并计算与原始问题的相似度
        
        Args:
            question: 原始问题
            answer: 答案
            
        Returns:
            相关性分数
        """
        # 生成潜在问题（实际应使用LLM）
        artificial_questions = self._generate_questions_from_answer(answer, n=3)
        
        if not artificial_questions:
            return {"score": 0.0, "reason": "No questions generated"}
        
        # 编码
        q_emb = self.embedder.encode(question, convert_to_tensor=True)
        aq_embs = self.embedder.encode(artificial_questions, convert_to_tensor=True)
        
        # 计算平均相似度
        similarities = util.pytorch_cos_sim(q_emb, aq_embs)[0]
        mean_sim = float(np.mean(similarities.cpu().numpy()))
        
        return {
            "score": mean_sim,
            "generated_questions": artificial_questions,
            "similarities": similarities.tolist()
        }
    
    def context_precision(self, 
                         question: str, 
                         contexts: List[str],
                         ground_truth: str = None) -> Dict:
        """
        计算上下文精确率
        
        评估检索的上下文中相关部分的比例
        
        Args:
            question: 问题
            contexts: 检索上下文（已排序）
            ground_truth: 标准答案（可选）
            
        Returns:
            精确率分数
        """
        if not contexts:
            return {"score": 0.0}
        
        # 计算每个上下文与问题的相关性
        q_emb = self.embedder.encode(question)
        
        relevant_count = 0
        precisions = []
        
        for i, ctx in enumerate(contexts, 1):
            ctx_emb = self.embedder.encode(ctx)
            similarity = np.dot(q_emb, ctx_emb)
            
            # 假设相似度>0.7为相关
            is_relevant = similarity > 0.7
            if is_relevant:
                relevant_count += 1
            
            # 计算Precision@k
            precisions.append(relevant_count / i)
        
        # 平均精确率（AP）
        ap = np.mean(precisions) if precisions else 0.0
        
        return {
            "score": ap,
            "relevant_chunks": relevant_count,
            "total_chunks": len(contexts),
            "precision_at_k": precisions
        }
    
    def context_recall(self, 
                      question: str,
                      answer: str,
                      contexts: List[str]) -> Dict:
        """
        计算上下文召回率
        
        评估答案中有多少信息能在上下文中找到支持
        
        Args:
            question: 问题
            answer: 答案
            contexts: 上下文
            
        Returns:
            召回率分数
        """
        # 提取答案陈述
        answer_statements = self._extract_statements(answer)
        
        if not answer_statements:
            return {"score": 0.0}
        
        # 合并上下文
        all_context = " ".join(contexts)
        
        # 检查每个陈述是否在上下文中
        retrieved = 0
        for stmt in answer_statements:
            # 简单实现：语义相似度
            stmt_emb = self.embedder.encode(stmt)
            ctx_emb = self.embedder.encode(all_context)
            similarity = np.dot(stmt_emb, ctx_emb)
            
            if similarity > 0.6:  # 阈值
                retrieved += 1
        
        score = retrieved / len(answer_statements)
        
        return {
            "score": score,
            "retrieved_statements": retrieved,
            "total_statements": len(answer_statements)
        }
    
    def evaluate(self, qa_pairs: List[QAPair]) -> Dict:
        """
        批量评估
        
        Args:
            qa_pairs: 问答对列表
            
        Returns:
            综合评估结果
        """
        results = {
            "faithfulness": [],
            "answer_relevancy": [],
            "context_precision": [],
            "context_recall": []
        }
        
        for i, qa in enumerate(qa_pairs):
            print(f"[EVAL] 评估样本 {i+1}/{len(qa_pairs)}...")
            
            # 计算各项指标
            f = self.faithfulness(qa.answer, qa.contexts)
            ar = self.answer_relevancy(qa.question, qa.answer)
            cp = self.context_precision(qa.question, qa.contexts, qa.ground_truth)
            cr = self.context_recall(qa.question, qa.answer, qa.contexts)
            
            results["faithfulness"].append(f["score"])
            results["answer_relevancy"].append(ar["score"])
            results["context_precision"].append(cp["score"])
            results["context_recall"].append(cr["score"])
        
        # 汇总
        summary = {
            "faithfulness": {
                "mean": np.mean(results["faithfulness"]),
                "std": np.std(results["faithfulness"])
            },
            "answer_relevancy": {
                "mean": np.mean(results["answer_relevancy"]),
                "std": np.std(results["answer_relevancy"])
            },
            "context_precision": {
                "mean": np.mean(results["context_precision"]),
                "std": np.std(results["context_precision"])
            },
            "context_recall": {
                "mean": np.mean(results["context_recall"]),
                "std": np.std(results["context_recall"])
            }
        }
        
        # 综合RAGAS分数（加权平均）
        summary["ragas_score"] = (
            0.3 * summary["faithfulness"]["mean"] +
            0.3 * summary["answer_relevancy"]["mean"] +
            0.2 * summary["context_precision"]["mean"] +
            0.2 * summary["context_recall"]["mean"]
        )
        
        return {
            "summary": summary,
            "detailed_scores": results
        }
    
    def _extract_statements(self, text: str) -> List[str]:
        """提取原子陈述"""
        # 按句子分割
        import re
        sentences = re.split(r'(?<=[.!?。！？])\s+', text)
        return [s.strip() for s in sentences if len(s.strip()) > 10]
    
    def _verify_statement(self, statement: str, contexts: List[str]) -> Tuple[bool, str]:
        """验证陈述是否被支持"""
        stmt_emb = self.embedder.encode(statement)
        
        best_sim = 0
        best_ctx = ""
        
        for ctx in contexts:
            ctx_emb = self.embedder.encode(ctx)
            sim = np.dot(stmt_emb, ctx_emb)
            if sim > best_sim:
                best_sim = sim
                best_ctx = ctx
        
        # 阈值判断
        is_supported = best_sim > 0.65
        return is_supported, best_ctx
    
    def _generate_questions_from_answer(self, answer: str, n: int = 3) -> List[str]:
        """从答案生成问题（简化实现）"""
        # 实际应使用LLM生成
        # 这里使用模板生成
        questions = []
        
        # 提取关键句生成问题
        sentences = self._extract_statements(answer)[:n]
        for sent in sentences:
            # 简单启发式转换
            if "is" in sent:
                q = sent.replace("is", "what is", 1) + "?"
            else:
                q = f"What can you tell me about: {sent[:50]}?"
            questions.append(q)
        
        return questions


class EvaluationVisualizer:
    """评估可视化"""
    
    def visualize(self, results: Dict, output_path: str = "ragas_report.png"):
        """生成可视化报告"""
        import matplotlib.pyplot as plt
        
        fig, axes = plt.subplots(2, 2, figsize=(14, 10))
        
        summary = results["summary"]
        detailed = results["detailed_scores"]
        
        metrics = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]
        titles = ["Faithfulness", "Answer Relevancy", "Context Precision", "Context Recall"]
        
        for idx, (metric, title) in enumerate(zip(metrics, titles)):
            ax = axes[idx // 2, idx % 2]
            scores = detailed[metric]
            
            # 箱线图
            bp = ax.boxplot(scores, patch_artist=True)
            bp['boxes'][0].set_facecolor('lightblue')
            
            # 添加均值线
            mean_val = summary[metric]["mean"]
            ax.axhline(y=mean_val, color='r', linestyle='--', label=f'Mean: {mean_val:.3f}')
            
            ax.set_ylabel('Score')
            ax.set_title(title)
            ax.set_ylim([0, 1])
            ax.legend()
            ax.grid(True, alpha=0.3)
        
        plt.suptitle(f"RAGAS Evaluation Report (Overall Score: {summary['ragas_score']:.3f})", 
                    fontsize=14, fontweight='bold')
        plt.tight_layout()
        plt.savefig(output_path, dpi=300, bbox_inches='tight')
        print(f"[INFO] 评估报告已保存至: {output_path}")
        plt.close()


def main():
    parser = argparse.ArgumentParser(description="RAGAS评估")
    parser.add_argument("--qa-pairs", "-q", required=True, help="问答对JSON文件")
    parser.add_argument("--output", "-o", default="ragas_results.json", help="输出文件")
    parser.add_argument("--visualize", "-v", action="store_true", help="生成可视化")
    
    args = parser.parse_args()
    
    # 加载数据
    with open(args.qa_pairs, 'r') as f:
        data = json.load(f)
    
    # 转换为QAPair
    qa_pairs = []
    for item in data:
        qa = QAPair(
            question=item["question"],
            answer=item["answer"],
            contexts=item.get("contexts", []),
            ground_truth=item.get("ground_truth")
        )
        qa_pairs.append(qa)
    
    print(f"[INFO] 加载了 {len(qa_pairs)} 个问答对")
    
    # 评估
    evaluator = RAGASEvaluator()
    results = evaluator.evaluate(qa_pairs)
    
    # 打印摘要
    print("\n评估结果摘要:")
    print(f"  RAGAS总分: {results['summary']['ragas_score']:.3f}")
    print(f"  忠实度: {results['summary']['faithfulness']['mean']:.3f} ± {results['summary']['faithfulness']['std']:.3f}")
    print(f"  答案相关性: {results['summary']['answer_relevancy']['mean']:.3f}")
    print(f"  上下文精确率: {results['summary']['context_precision']['mean']:.3f}")
    print(f"  上下文召回率: {results['summary']['context_recall']['mean']:.3f}")
    
    # 保存
    with open(args.output, 'w') as f:
        json.dump(results, f, indent=2)
    print(f"\n详细结果已保存至: {args.output}")
    
    # 可视化
    if args.visualize:
        visualizer = EvaluationVisualizer()
        visualizer.visualize(results)


if __name__ == "__main__":
    main()

6.5.2 在线反馈

Python

复制代码

#!/usr/bin/env python3
"""
Script: online_feedback_system.py
功能: 在线反馈收集系统，支持显式反馈（点赞/点踩）与隐式信号（停留时间、复制行为）
使用方式: python online_feedback_system.py --collect --session-id "sess_001"
"""

import json
import time
import argparse
from typing import Dict, List, Optional
from dataclasses import dataclass, asdict
from datetime import datetime, timedelta
import sqlite3
import threading
from collections import defaultdict


@dataclass
class FeedbackRecord:
    """反馈记录"""
    session_id: str
    query: str
    answer: str
    timestamp: str
    explicit_rating: Optional[int] = None  # 1-5星，或-1/1表示点踩/点赞
    implicit_signals: Dict = None
    metadata: Dict = None
    
    def __post_init__(self):
        if self.implicit_signals is None:
            self.implicit_signals = {}
        if self.metadata is None:
            self.metadata = {}


class ImplicitSignalTracker:
    """
    隐式反馈信号追踪器
    
    收集用户行为信号推断满意度：
    - Dwell time: 答案展示后的停留时间
    - Copy events: 复制答案内容
    - Click-through: 点击引用链接
    - Follow-up queries: 后续查询（可能表示答案不完整）
    """
    
    def __init__(self):
        self.sessions: Dict[str, Dict] = {}
        self.lock = threading.Lock()
    
    def start_session(self, session_id: str, query: str, answer: str):
        """开始追踪会话"""
        with self.lock:
            self.sessions[session_id] = {
                "start_time": time.time(),
                "query": query,
                "answer": answer,
                "events": [],
                "copy_count": 0,
                "click_count": 0,
                "scroll_depth": 0
            }
    
    def record_event(self, session_id: str, event_type: str, data: Dict = None):
        """记录事件"""
        with self.lock:
            if session_id not in self.sessions:
                return
            
            event = {
                "type": event_type,
                "timestamp": time.time(),
                "data": data or {}
            }
            self.sessions[session_id]["events"].append(event)
            
            if event_type == "copy":
                self.sessions[session_id]["copy_count"] += 1
            elif event_type == "click_citation":
                self.sessions[session_id]["click_count"] += 1
            elif event_type == "scroll":
                self.sessions[session_id]["scroll_depth"] = data.get("depth", 0)
    
    def end_session(self, session_id: str) -> Dict:
        """结束会话并计算隐式分数"""
        with self.lock:
            if session_id not in self.sessions:
                return {}
            
            sess = self.sessions[session_id]
            duration = time.time() - sess["start_time"]
            
            # 计算满意度分数（启发式）
            score = 0.5  # 基准
            
            # 停留时间（30秒以上加分）
            if duration > 30:
                score += 0.2
            elif duration < 5:
                score -= 0.2
            
            # 复制行为（强烈正信号）
            if sess["copy_count"] > 0:
                score += 0.15 * min(sess["copy_count"], 2)
            
            # 点击引用（正信号）
            if sess["click_count"] > 0:
                score += 0.1 * min(sess["click_count"], 2)
            
            # 滚动深度
            if sess["scroll_depth"] > 0.8:
                score += 0.1
            
            # 检查是否有后续查询（负信号：可能未解决）
            follow_up = any(e["type"] == "new_query" for e in sess["events"])
            if follow_up:
                score -= 0.15
            
            signals = {
                "dwell_time_seconds": duration,
                "copy_events": sess["copy_count"],
                "citation_clicks": sess["click_count"],
                "scroll_depth": sess["scroll_depth"],
                "inferred_satisfaction": max(0, min(1, score))
            }
            
            # 清理
            del self.sessions[session_id]
            
            return signals


class FeedbackDatabase:
    """反馈数据库"""
    
    def __init__(self, db_path: str = "feedback.db"):
        self.db_path = db_path
        self._init_db()
    
    def _init_db(self):
        """初始化数据库表"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS feedback (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                session_id TEXT,
                query TEXT,
                answer TEXT,
                timestamp TEXT,
                explicit_rating INTEGER,
                implicit_satisfaction REAL,
                dwell_time REAL,
                copy_events INTEGER,
                citation_clicks INTEGER,
                metadata TEXT
            )
        """)
        
        cursor.execute("""
            CREATE INDEX IF NOT EXISTS idx_timestamp ON feedback(timestamp)
        """)
        
        conn.commit()
        conn.close()
    
    def save_feedback(self, record: FeedbackRecord):
        """保存反馈"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        implicit = record.implicit_signals or {}
        
        cursor.execute("""
            INSERT INTO feedback 
            (session_id, query, answer, timestamp, explicit_rating, 
             implicit_satisfaction, dwell_time, copy_events, citation_clicks, metadata)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """, (
            record.session_id,
            record.query,
            record.answer,
            record.timestamp,
            record.explicit_rating,
            implicit.get("inferred_satisfaction"),
            implicit.get("dwell_time_seconds", 0),
            implicit.get("copy_events", 0),
            implicit.get("citation_clicks", 0),
            json.dumps(record.metadata)
        ))
        
        conn.commit()
        conn.close()
    
    def get_statistics(self, days: int = 7) -> Dict:
        """获取统计信息"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        since = (datetime.now() - timedelta(days=days)).isoformat()
        
        # 基础统计
        cursor.execute("""
            SELECT 
                COUNT(*) as total,
                AVG(explicit_rating) as avg_rating,
                AVG(implicit_satisfaction) as avg_implicit,
                AVG(dwell_time) as avg_dwell
            FROM feedback
            WHERE timestamp > ?
        """, (since,))
        
        row = cursor.fetchone()
        
        # 显式反馈分布
        cursor.execute("""
            SELECT explicit_rating, COUNT(*) 
            FROM feedback 
            WHERE timestamp > ? AND explicit_rating IS NOT NULL
            GROUP BY explicit_rating
        """, (since,))
        
        rating_dist = {r[0]: r[1] for r in cursor.fetchall()}
        
        conn.close()
        
        return {
            "total_interactions": row[0],
            "average_explicit_rating": row[1],
            "average_implicit_satisfaction": row[2],
            "average_dwell_time": row[3],
            "rating_distribution": rating_dist,
            "period_days": days
        }


class FeedbackCollector:
    """反馈收集器"""
    
    def __init__(self):
        self.db = FeedbackDatabase()
        self.tracker = ImplicitSignalTracker()
        self.active_sessions = {}
    
    def start_interaction(self, session_id: str, query: str, answer: str):
        """开始交互追踪"""
        self.tracker.start_session(session_id, query, answer)
        self.active_sessions[session_id] = {
            "query": query,
            "answer": answer,
            "start_time": datetime.now().isoformat()
        }
    
    def record_explicit_feedback(self, session_id: str, rating: int, comment: str = None):
        """
        记录显式反馈
        
        Args:
            session_id: 会话ID
            rating: 1-5星，或-1/0/1（点踩/中立/点赞）
            comment: 可选评论
        """
        if session_id not in self.active_sessions:
            return
        
        # 结束隐式追踪并获取信号
        implicit = self.tracker.end_session(session_id)
        
        # 构建记录
        sess = self.active_sessions[session_id]
        record = FeedbackRecord(
            session_id=session_id,
            query=sess["query"],
            answer=sess["answer"],
            timestamp=datetime.now().isoformat(),
            explicit_rating=rating,
            implicit_signals=implicit,
            metadata={"comment": comment} if comment else {}
        )
        
        # 保存
        self.db.save_feedback(record)
        
        # 清理
        del self.active_sessions[session_id]
        
        print(f"[INFO] 反馈已保存: Session={session_id}, Rating={rating}")
    
    def record_event(self, session_id: str, event_type: str, data: Dict = None):
        """记录交互事件"""
        self.tracker.record_event(session_id, event_type, data)


class FeedbackDashboard:
    """反馈仪表板"""
    
    def generate_report(self, stats: Dict, output_path: str = "feedback_report.png"):
        """生成可视化报告"""
        import matplotlib.pyplot as plt
        
        fig, axes = plt.subplots(2, 2, figsize=(12, 10))
        
        # 1. 评分分布
        ax1 = axes[0, 0]
        if stats["rating_distribution"]:
            ratings = list(stats["rating_distribution"].keys())
            counts = list(stats["rating_distribution"].values())
            ax1.bar(ratings, counts, color='skyblue')
            ax1.set_xlabel('Rating')
            ax1.set_ylabel('Count')
            ax1.set_title('Explicit Rating Distribution')
        
        # 2. 满意度趋势（模拟时间序列）
        ax2 = axes[0, 1]
        ax2.hist([stats["average_explicit_rating"] or 0, 
                 stats["average_implicit_satisfaction"] or 0],
                bins=10, range=(0, 5), color=['blue', 'green'], alpha=0.6,
                label=['Explicit', 'Implicit'])
        ax2.set_xlabel('Satisfaction Score')
        ax2.set_ylabel('Frequency')
        ax2.set_title('Satisfaction Comparison')
        ax2.legend()
        
        # 3. 平均停留时间
        ax3 = axes[1, 0]
        dwell = stats["average_dwell_time"] or 0
        ax3.barh(['Average Dwell Time'], [dwell], color='orange')
        ax3.set_xlabel('Seconds')
        ax3.set_title(f'Avg Dwell Time: {dwell:.1f}s')
        
        # 4. 总体指标
        ax4 = axes[1, 1]
        ax4.axis('off')
        metrics_text = f"""
        Total Interactions: {stats['total_interactions']}
        Avg Explicit Rating: {stats['average_explicit_rating'] or 'N/A':.2f}
        Avg Implicit Sat: {stats['average_implicit_satisfaction'] or 'N/A':.2f}
        Avg Dwell Time: {stats['average_dwell_time'] or 0:.1f}s
        Period: Last {stats['period_days']} days
        """
        ax4.text(0.1, 0.5, metrics_text, fontsize=12, verticalalignment='center',
                family='monospace')
        
        plt.tight_layout()
        plt.savefig(output_path, dpi=300, bbox_inches='tight')
        print(f"[INFO] 报告已保存至: {output_path}")
        plt.close()


def main():
    parser = argparse.ArgumentParser(description="在线反馈系统")
    parser.add_argument("--collect", "-c", action="store_true", help="收集反馈模式")
    parser.add_argument("--stats", "-s", action="store_true", help="显示统计")
    parser.add_argument("--session-id", default="demo_session", help="会话ID")
    parser.add_argument("--rating", "-r", type=int, default=None, help="评分 (1-5)")
    parser.add_argument("--event", "-e", default=None, help="事件类型")
    parser.add_argument("--query", "-q", default="What is RAG?", help="查询")
    parser.add_argument("--answer", "-a", default="RAG stands for...", help="答案")
    
    args = parser.parse_args()
    
    collector = FeedbackCollector()
    dashboard = FeedbackDashboard()
    
    if args.collect:
        # 模拟交互流程
        print(f"[INFO] 开始会话: {args.session_id}")
        collector.start_interaction(args.session_id, args.query, args.answer)
        
        # 模拟一些事件
        time.sleep(1)
        collector.record_event(args.session_id, "scroll", {"depth": 0.8})
        collector.record_event(args.session_id, "copy")
        
        if args.rating is not None:
            collector.record_explicit_feedback(args.session_id, args.rating)
            print("[INFO] 反馈已记录")
        else:
            print("[INFO] 交互进行中... 使用 --rating 提交评分")
    
    elif args.stats:
        stats = collector.db.get_statistics(days=7)
        print(json.dumps(stats, indent=2))
        dashboard.generate_report(stats)
    
    else:
        parser.print_help()


if __name__ == "__main__":
    main()

6.5.3 A/B测试

Python

复制代码

#!/usr/bin/env python3
"""
Script: ab_testing_framework.py
功能: RAG系统A/B测试框架，支持分块策略、提示词、模型版本的对比实验
使用方式: python ab_testing_framework.py --create-experiment --name chunking_test --variants semantic,recursive
"""

import json
import hashlib
import random
import argparse
from typing import Dict, List, Optional
from dataclasses import dataclass, asdict
from datetime import datetime
import sqlite3
import numpy as np
from scipy import stats


@dataclass
class Experiment:
    """实验定义"""
    id: str
    name: str
    variants: List[str]  # 包括control
    traffic_split: List[float]  # 各变体流量比例
    target_metric: str
    min_sample_size: int
    status: str = "running"  # running, paused, completed


@dataclass
class ExperimentEvent:
    """实验事件"""
    experiment_id: str
    user_id: str
    variant: str
    event_type: str  # impression, conversion, feedback
    metric_value: float
    timestamp: str
    metadata: Dict


class ABTestManager:
    """A/B测试管理器"""
    
    def __init__(self, db_path: str = "ab_tests.db"):
        self.db_path = db_path
        self.experiments: Dict[str, Experiment] = {}
        self._init_db()
        self._load_experiments()
    
    def _init_db(self):
        """初始化数据库"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS experiments (
                id TEXT PRIMARY KEY,
                config TEXT,
                created_at TEXT,
                status TEXT
            )
        """)
        
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS events (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                experiment_id TEXT,
                user_id TEXT,
                variant TEXT,
                event_type TEXT,
                metric_value REAL,
                timestamp TEXT,
                metadata TEXT
            )
        """)
        
        conn.commit()
        conn.close()
    
    def _load_experiments(self):
        """加载实验"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        cursor.execute("SELECT id, config FROM experiments WHERE status='running'")
        
        for row in cursor.fetchall():
            exp_id, config_json = row
            config = json.loads(config_json)
            self.experiments[exp_id] = Experiment(**config)
        
        conn.close()
    
    def create_experiment(self,
                         name: str,
                         variants: List[str],
                         traffic_split: Optional[List[float]] = None,
                         target_metric: str = "feedback_rating",
                         min_sample_size: int = 100) -> str:
        """
        创建实验
        
        Args:
            name: 实验名称
            variants: 变体列表（第一个为control）
            traffic_split: 流量分配（默认均等）
            target_metric: 目标指标
            min_sample_size: 最小样本数
            
        Returns:
            实验ID
        """
        if traffic_split is None:
            traffic_split = [1.0 / len(variants)] * len(variants)
        
        assert len(variants) == len(traffic_split)
        assert abs(sum(traffic_split) - 1.0) < 0.01
        
        exp_id = f"exp_{name}_{datetime.now().strftime('%Y%m%d%H%M%S')}"
        
        exp = Experiment(
            id=exp_id,
            name=name,
            variants=variants,
            traffic_split=traffic_split,
            target_metric=target_metric,
            min_sample_size=min_sample_size
        )
        
        # 保存
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        cursor.execute("""
            INSERT INTO experiments (id, config, created_at, status)
            VALUES (?, ?, ?, ?)
        """, (exp_id, json.dumps(asdict(exp)), datetime.now().isoformat(), "running"))
        conn.commit()
        conn.close()
        
        self.experiments[exp_id] = exp
        print(f"[INFO] 创建实验: {exp_id}")
        print(f"  变体: {variants}")
        print(f"  流量分配: {traffic_split}")
        
        return exp_id
    
    def assign_variant(self, experiment_id: str, user_id: str) -> str:
        """
        为用户分配变体（一致性哈希）
        
        Args:
            experiment_id: 实验ID
            user_id: 用户ID
            
        Returns:
            分配的变体名称
        """
        if experiment_id not in self.experiments:
            return "control"
        
        exp = self.experiments[experiment_id]
        
        # 一致性哈希
        hash_val = int(hashlib.md5(f"{exp.id}_{user_id}".encode()).hexdigest(), 16)
        normalized = hash_val / (2**128)
        
        # 根据流量分配选择
        cumulative = 0
        for variant, split in zip(exp.variants, exp.traffic_split):
            cumulative += split
            if normalized <= cumulative:
                return variant
        
        return exp.variants[-1]
    
    def record_event(self, 
                    experiment_id: str, 
                    user_id: str, 
                    variant: str,
                    event_type: str,
                    metric_value: float = 0.0,
                    metadata: Dict = None):
        """记录实验事件"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        cursor.execute("""
            INSERT INTO events 
            (experiment_id, user_id, variant, event_type, metric_value, timestamp, metadata)
            VALUES (?, ?, ?, ?, ?, ?, ?)
        """, (
            experiment_id,
            user_id,
            variant,
            event_type,
            metric_value,
            datetime.now().isoformat(),
            json.dumps(metadata or {})
        ))
        
        conn.commit()
        conn.close()
    
    def get_results(self, experiment_id: str) -> Dict:
        """
        获取实验结果与统计分析
        
        Args:
            experiment_id: 实验ID
            
        Returns:
            实验结果与统计检验
        """
        if experiment_id not in self.experiments:
            return {"error": "Experiment not found"}
        
        exp = self.experiments[experiment_id]
        
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        # 获取各变体指标
        results = {}
        for variant in exp.variants:
            cursor.execute("""
                SELECT AVG(metric_value), COUNT(*), STDDEV(metric_value)
                FROM events
                WHERE experiment_id = ? AND variant = ? AND event_type = ?
            """, (experiment_id, variant, exp.target_metric))
            
            row = cursor.fetchone()
            results[variant] = {
                "mean": row[0] or 0,
                "n": row[1] or 0,
                "std": row[2] or 0
            }
        
        conn.close()
        
        # 统计检验（Control vs Treatment）
        control = exp.variants[0]
        control_data = results[control]
        
        comparisons = {}
        for variant in exp.variants[1:]:
            treat_data = results[variant]
            
            # 两样本t检验（简化，实际应获取原始数据）
            if treat_data["n"] > 30 and control_data["n"] > 30:
                # 使用均值和标准差进行近似t检验
                se = np.sqrt(
                    control_data["std"]**2 / control_data["n"] + 
                    treat_data["std"]**2 / treat_data["n"]
                )
                t_stat = (treat_data["mean"] - control_data["mean"]) / (se + 1e-10)
                
                # 自由度（Welch-Satterthwaite方程近似）
                df = treat_data["n"] + control_data["n"] - 2
                
                # p值
                p_value = 2 * (1 - stats.t.cdf(abs(t_stat), df))
                
                # 置信区间
                ci_low = (treat_data["mean"] - control_data["mean"]) - 1.96 * se
                ci_high = (treat_data["mean"] - control_data["mean"]) + 1.96 * se
                
                comparisons[f"{control}_vs_{variant}"] = {
                    "control_mean": control_data["mean"],
                    "treatment_mean": treat_data["mean"],
                    "absolute_diff": treat_data["mean"] - control_data["mean"],
                    "relative_lift": (treat_data["mean"] - control_data["mean"]) / 
                                    (control_data["mean"] + 1e-10),
                    "t_statistic": t_stat,
                    "p_value": p_value,
                    "significant": p_value < 0.05,
                    "confidence_interval": [ci_low, ci_high],
                    "sample_sizes": {
                        "control": control_data["n"],
                        "treatment": treat_data["n"]
                    }
                }
        
        return {
            "experiment_id": experiment_id,
            "experiment_name": exp.name,
            "target_metric": exp.target_metric,
            "variant_stats": results,
            "comparisons": comparisons,
            "recommendation": self._generate_recommendation(comparisons)
        }
    
    def _generate_recommendation(self, comparisons: Dict) -> str:
        """生成实验建议"""
        if not comparisons:
            return "数据不足"
        
        significant_wins = []
        for comp_name, stats in comparisons.items():
            if stats["significant"] and stats["relative_lift"] > 0:
                significant_wins.append((comp_name, stats["relative_lift"]))
        
        if significant_wins:
            best = max(significant_wins, key=lambda x: x[1])
            return f"建议采用变体: {best[0].split('_vs_')[1]} (提升 {best[1]:.2%})"
        
        return "当前无显著差异，建议继续实验或检查样本量"
    
    def visualize_results(self, results: Dict, output_path: str = "ab_test_results.png"):
        """可视化实验结果"""
        import matplotlib.pyplot as plt
        
        fig, axes = plt.subplots(1, 2, figsize=(14, 6))
        
        # 1. 各变体表现对比
        ax1 = axes[0]
        variants = list(results["variant_stats"].keys())
        means = [results["variant_stats"][v]["mean"] for v in variants]
        stds = [results["variant_stats"][v]["std"] for v in variants]
        
        x = np.arange(len(variants))
        bars = ax1.bar(x, means, yerr=stds, capsize=5, color=['blue', 'green', 'red', 'orange'][:len(variants)], alpha=0.6)
        ax1.set_xlabel('Variant')
        ax1.set_ylabel(f'Mean {results["target_metric"]}')
        ax1.set_title('Variant Performance Comparison')
        ax1.set_xticks(x)
        ax1.set_xticklabels(variants, rotation=45)
        
        # 添加数值标签
        for bar in bars:
            height = bar.get_height()
            ax1.text(bar.get_x() + bar.get_width()/2., height,
                    f'{height:.3f}', ha='center', va='bottom')
        
        # 2. 效应量与置信区间
        ax2 = axes[1]
        comp_names = list(results["comparisons"].keys())
        lifts = [results["comparisons"][c]["relative_lift"] for c in comp_names]
        cis = [results["comparisons"][c]["confidence_interval"] for c in comp_names]
        
        y_pos = np.arange(len(comp_names))
        ax2.barh(y_pos, lifts, color=['green' if l > 0 else 'red' for l in lifts], alpha=0.6)
        
        # 添加CI误差线
        for i, (lift, ci) in enumerate(zip(lifts, cis)):
            ax2.plot([ci[0], ci[1]], [i, i], 'k-', linewidth=2)
            ax2.plot([ci[0], ci[1]], [i, i], 'k|', markersize=10)
        
        ax2.axvline(x=0, color='black', linestyle='--', linewidth=0.8)
        ax2.set_yticks(y_pos)
        ax2.set_yticklabels(comp_names)
        ax2.set_xlabel('Relative Lift')
        ax2.set_title('Treatment Effect with 95% CI')
        
        plt.suptitle(f"A/B Test Results: {results['experiment_name']}", fontsize=14, fontweight='bold')
        plt.tight_layout()
        plt.savefig(output_path, dpi=300, bbox_inches='tight')
        print(f"[INFO] 结果图表已保存至: {output_path}")
        plt.close()


def main():
    parser = argparse.ArgumentParser(description="A/B测试框架")
    parser.add_argument("--create-experiment", "-c", action="store_true", help="创建实验")
    parser.add_argument("--name", "-n", default="test_exp", help="实验名称")
    parser.add_argument("--variants", "-v", default="control,treatment", help="变体列表（逗号分隔）")
    parser.add_argument("--traffic-split", "-t", default=None, help="流量分配（逗号分隔）")
    parser.add_argument("--assign", "-a", action="store_true", help="分配变体")
    parser.add_argument("--experiment-id", "-e", default=None, help="实验ID")
    parser.add_argument("--user-id", "-u", default="user_001", help="用户ID")
    parser.add_argument("--record", "-r", action="store_true", help="记录事件")
    parser.add_argument("--metric-value", "-m", type=float, default=0.0, help="指标值")
    parser.add_argument("--results", action="store_true", help="查看结果")
    parser.add_argument("--visualize", action="store_true", help="可视化")
    
    args = parser.parse_args()
    
    manager = ABTestManager()
    
    if args.create_experiment:
        variants = args.variants.split(',')
        split = None
        if args.traffic_split:
            split = [float(x) for x in args.traffic_split.split(',')]
        
        exp_id = manager.create_experiment(
            name=args.name,
            variants=variants,
            traffic_split=split,
            target_metric="feedback_rating",
            min_sample_size=100
        )
        print(f"实验ID: {exp_id}")
    
    elif args.assign:
        if not args.experiment_id:
            print("请提供 --experiment-id")
            return
        
        variant = manager.assign_variant(args.experiment_id, args.user_id)
        print(f"用户 {args.user_id} 分配到变体: {variant}")
    
    elif args.record:
        if not args.experiment_id:
            print("请提供 --experiment-id")
            return
        
        variant = manager.assign_variant(args.experiment_id, args.user_id)
        manager.record_event(
            args.experiment_id,
            args.user_id,
            variant,
            "feedback_rating",
            args.metric_value,
            {"query": "test_query"}
        )
        print(f"已记录事件: {variant} -> {args.metric_value}")
    
    elif args.results:
        if not args.experiment_id:
            # 列出所有实验
            print("活跃实验:")
            for exp_id, exp in manager.experiments.items():
                print(f"  {exp_id}: {exp.name}")
        else:
            results = manager.get_results(args.experiment_id)
            print(json.dumps(results, indent=2))
            
            if args.visualize:
                manager.visualize_results(results)
    
    else:
        parser.print_help()


if __name__ == "__main__":
    main()

6.5.4 持续学习

Python

复制代码

#!/usr/bin/env python3
"""
Script: continual_learning.py
功能: 持续学习系统，实现Bad Case收集、自动标注与模型微调触发机制
使用方式: python continual_learning.py --collect-bad-case --query "q" --bad-answer "wrong" --correct-answer "right"
"""

import json
import os
import argparse
from typing import List, Dict, Optional
from dataclasses import dataclass, asdict
from datetime import datetime
import sqlite3
import numpy as np
from collections import defaultdict


@dataclass
class BadCase:
    """Bad Case记录"""
    id: str
    query: str
    retrieved_contexts: List[str]
    generated_answer: str
    correct_answer: Optional[str]  # 用户反馈或标注
    failure_type: str  # 'retrieval', 'generation', 'hallucination', 'incomplete'
    timestamp: str
    user_feedback: Optional[str] = None
    severity: int = 1  # 1-5


class BadCaseMiner:
    """Bad Case挖掘器"""
    
    FAILURE_PATTERNS = {
        "retrieval_failure": {
            "indicators": ["不知道", "无法找到", "没有相关信息", "检索失败"],
            "description": "未能检索到相关文档"
        },
        "hallucination": {
            "indicators": ["事实上", "实际上", "错误信息"],
            "description": "生成内容包含幻觉"
        },
        "incomplete": {
            "indicators": ["部分正确", "不完整", "缺少"],
            "description": "答案不完整"
        }
    }
    
    def classify_failure(self, 
                        query: str, 
                        answer: str, 
                        contexts: List[str],
                        user_feedback: str = None) -> str:
        """
        分类失效类型
        
        Args:
            query: 查询
            answer: 生成的答案
            contexts: 检索上下文
            user_feedback: 用户反馈
            
        Returns:
            失效类型
        """
        if user_feedback:
            # 基于用户反馈分类
            feedback_lower = user_feedback.lower()
            for ftype, info in self.FAILURE_PATTERNS.items():
                if any(ind in feedback_lower for ind in info["indicators"]):
                    return ftype
        
        # 自动检测
        if not contexts or all(len(c.strip()) < 10 for c in contexts):
            return "retrieval_failure"
        
        # 检查答案长度（过短可能不完整）
        if len(answer) < 50:
            return "incomplete"
        
        # 检查是否包含不确定表述
        uncertainty_phrases = ["我不确定", "可能没有", "也许是", "可能"]
        if any(p in answer for p in uncertainty_phrases):
            return "hallucination"
        
        return "unknown"
    
    def mine_hard_negatives(self, 
                           query: str, 
                           positive_contexts: List[str],
                           all_documents: List[str],
                           top_k: int = 5) -> List[str]:
        """
        挖掘困难负样本
        
        Args:
            query: 查询
            positive_contexts: 正样本（检索到的相关文档）
            all_documents: 所有文档
            top_k: 返回数量
            
        Returns:
            困难负样本列表
        """
        # 简单实现：随机选择非正样本的文档
        # 实际应使用BM25或向量相似度选择"接近但不相关"的文档
        positive_set = set(positive_contexts)
        negatives = [d for d in all_documents if d not in positive_set]
        
        if len(negatives) <= top_k:
            return negatives
        
        # 随机选择（实际应基于相似度选择困难负样本）
        import random
        return random.sample(negatives, top_k)


class ContinuousLearningPipeline:
    """持续学习流水线"""
    
    def __init__(self, db_path: str = "continual_learning.db"):
        self.db_path = db_path
        self.miner = BadCaseMiner()
        self.bad_cases: List[BadCase] = []
        self._init_db()
        self.retraining_threshold = 100  # 触发微调的Bad Case数量阈值
    
    def _init_db(self):
        """初始化数据库"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS bad_cases (
                id TEXT PRIMARY KEY,
                query TEXT,
                retrieved_contexts TEXT,
                generated_answer TEXT,
                correct_answer TEXT,
                failure_type TEXT,
                timestamp TEXT,
                user_feedback TEXT,
                severity INTEGER,
                processed INTEGER DEFAULT 0
            )
        """)
        
        cursor.execute("""
            CREATE TABLE IF NOT EXISTS training_data (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                query TEXT,
                positive_doc TEXT,
                negative_doc TEXT,
                label INTEGER,  -- 1 for positive, 0 for negative
                source_bad_case TEXT,
                created_at TEXT
            )
        """)
        
        conn.commit()
        conn.close()
    
    def collect_bad_case(self,
                        query: str,
                        retrieved_contexts: List[str],
                        generated_answer: str,
                        correct_answer: Optional[str] = None,
                        user_feedback: Optional[str] = None) -> str:
        """
        收集Bad Case
        
        Returns:
            Bad Case ID
        """
        case_id = f"bc_{datetime.now().strftime('%Y%m%d%H%M%S')}_{hash(query) % 10000}"
        
        failure_type = self.miner.classify_failure(
            query, generated_answer, retrieved_contexts, user_feedback
        )
        
        bad_case = BadCase(
            id=case_id,
            query=query,
            retrieved_contexts=retrieved_contexts,
            generated_answer=generated_answer,
            correct_answer=correct_answer,
            failure_type=failure_type,
            timestamp=datetime.now().isoformat(),
            user_feedback=user_feedback,
            severity=3 if failure_type == "hallucination" else 2
        )
        
        # 保存到数据库
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        cursor.execute("""
            INSERT INTO bad_cases 
            (id, query, retrieved_contexts, generated_answer, correct_answer, 
             failure_type, timestamp, user_feedback, severity)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
        """, (
            bad_case.id,
            bad_case.query,
            json.dumps(bad_case.retrieved_contexts),
            bad_case.generated_answer,
            bad_case.correct_answer,
            bad_case.failure_type,
            bad_case.timestamp,
            bad_case.user_feedback,
            bad_case.severity
        ))
        
        conn.commit()
        conn.close()
        
        self.bad_cases.append(bad_case)
        print(f"[INFO] 收集Bad Case: {case_id}, 类型: {failure_type}")
        
        # 检查是否触发再训练
        if len(self.bad_cases) >= self.retraining_threshold:
            self.trigger_retraining()
        
        return case_id
    
    def build_training_data(self) -> Dict[str, List]:
        """
        从Bad Case构建训练数据
        
        Returns:
            训练数据集
        """
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        # 获取未处理的Bad Case
        cursor.execute("""
            SELECT * FROM bad_cases 
            WHERE processed = 0 
            ORDER BY severity DESC, timestamp DESC
            LIMIT 200
        """)
        
        rows = cursor.fetchall()
        
        retrieval_training = []  # 用于检索模型
        generation_training = []  # 用于生成模型
        
        for row in rows:
            (case_id, query, contexts_json, gen_answer, correct_answer,
             failure_type, timestamp, feedback, severity, processed) = row
            
            contexts = json.loads(contexts_json)
            
            if failure_type == "retrieval_failure" and correct_answer:
                # 构建检索训练数据（正样本：正确答案来源，负样本：实际检索到的无关内容）
                # 这里简化处理
                if contexts:
                    retrieval_training.append({
                        "query": query,
                        "positive": correct_answer,  # 假设correct_answer是正样本来源
                        "negatives": contexts[:3],  # 实际检索到的作为困难负样本
                        "type": "contrastive"
                    })
            
            elif failure_type in ["hallucination", "incomplete"] and correct_answer:
                # 构建生成训练数据（偏好对：错误答案 vs 正确答案）
                generation_training.append({
                    "query": query,
                    "context": " ".join(contexts),
                    "rejected": gen_answer,
                    "chosen": correct_answer,
                    "type": "preference"
                })
        
        conn.close()
        
        return {
            "retrieval": retrieval_training,
            "generation": generation_training,
            "total_cases": len(rows)
        }
    
    def trigger_retraining(self):
        """触发模型微调"""
        print("[ALERT] 触发持续学习...")
        
        # 构建训练数据
        train_data = self.build_training_data()
        
        if train_data["total_cases"] < 50:
            print("[INFO] Bad Case数量不足，跳过再训练")
            return
        
        # 模拟微调过程
        print(f"[INFO] 构建训练数据:")
        print(f"  - 检索优化样本: {len(train_data['retrieval'])}")
        print(f"  - 生成优化样本: {len(train_data['generation'])}")
        
        # 保存训练数据
        timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        output_file = f"training_data_{timestamp}.json"
        
        with open(output_file, 'w') as f:
            json.dump(train_data, f, indent=2)
        
        print(f"[INFO] 训练数据已保存至: {output_file}")
        
        # 标记已处理
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        cursor.execute("""
            UPDATE bad_cases SET processed = 1 
            WHERE processed = 0
        """)
        conn.commit()
        conn.close()
        
        print("[INFO] 已标记Bad Case为已处理，等待手动触发训练...")
        
        # 这里应触发实际的训练流水线
        # self._run_fine_tuning(train_data)
    
    def get_statistics(self) -> Dict:
        """获取统计信息"""
        conn = sqlite3.connect(self.db_path)
        cursor = conn.cursor()
        
        # 按类型统计
        cursor.execute("""
            SELECT failure_type, COUNT(*), AVG(severity)
            FROM bad_cases
            GROUP BY failure_type
        """)
        
        type_stats = {row[0]: {"count": row[1], "avg_severity": row[2]} 
                     for row in cursor.fetchall()}
        
        # 时间趋势
        cursor.execute("""
            SELECT DATE(timestamp), COUNT(*)
            FROM bad_cases
            GROUP BY DATE(timestamp)
            ORDER BY DATE(timestamp) DESC
            LIMIT 7
        """)
        
        daily_trend = {row[0]: row[1] for row in cursor.fetchall()}
        
        conn.close()
        
        return {
            "total_bad_cases": sum(s["count"] for s in type_stats.values()),
            "by_type": type_stats,
            "daily_trend": daily_trend,
            "retraining_threshold": self.retraining_threshold
        }


def main():
    parser = argparse.ArgumentParser(description="持续学习系统")
    parser.add_argument("--collect-bad-case", "-c", action="store_true", help="收集Bad Case")
    parser.add_argument("--query", "-q", default="What is AI?", help="查询")
    parser.add_argument("--contexts", default=None, help="检索上下文JSON文件")
    parser.add_argument("--bad-answer", "-b", default="Wrong answer", help="错误答案")
    parser.add_argument("--correct-answer", "-a", default=None, help="正确答案")
    parser.add_argument("--feedback", "-f", default=None, help="用户反馈")
    parser.add_argument("--stats", "-s", action="store_true", help="显示统计")
    parser.add_argument("--build-training-data", "-t", action="store_true", help="构建训练数据")
    
    args = parser.parse_args()
    
    pipeline = ContinuousLearningPipeline()
    
    if args.collect_bad_case:
        contexts = []
        if args.contexts:
            with open(args.contexts, 'r') as f:
                data = json.load(f)
                contexts = [c.get("text", "") for c in data.get("selected_chunks", [])]
        
        case_id = pipeline.collect_bad_case(
            query=args.query,
            retrieved_contexts=contexts,
            generated_answer=args.bad_answer,
            correct_answer=args.correct_answer,
            user_feedback=args.feedback
        )
        print(f"Bad Case ID: {case_id}")
    
    elif args.stats:
        stats = pipeline.get_statistics()
        print(json.dumps(stats, indent=2))
    
    elif args.build_training_data:
        data = pipeline.build_training_data()
        print(f"构建完成: {data['total_cases']} 个案例")
        print(f"检索训练样本: {len(data['retrieval'])}")
        print(f"生成训练样本: {len(data['generation'])}")
    
    else:
        parser.print_help()


if __name__ == "__main__":
    main()

以上代码构成了完整的企业级RAG知识库问答系统。每个脚本均可独立运行并具备可视化能力，涵盖从数据摄取、向量存储、检索优化到评估监控的全流程组件。