前言
在企业知识管理的真实场景中，文档从来不只有文字。产品说明书里穿插着结构图，客服录音里藏着用户痛点，培训视频里讲透了 SOP 操作流程。传统 RAG（Retrieval-Augmented Generation）只处理文本，无法应对这些多模态内容，导致大量高价值信息被"沉默"在系统里。
多模态 RAG（Multimodal RAG） 正是为了解决这一问题而生。它将文本、图像、音频、视频统一建模，让用户在检索时可以跨越模态边界，用一段文字找到相关图片，用一张截图定位对应视频片段。
本文将从零到一构建一套完整的企业级多模态 RAG 系统，涵盖：
- 整体架构设计与技术选型
- 文档解析与多模态内容提取
- CLIP 驱动的图文向量化与检索
- Whisper + Milvus 音视频检索方案
- LangChain 全链路编排
- 完整项目实战 + 效果评估
全文含完整可运行代码，适合有 LangChain / Milvus 基础的同学直接上手。全文约 8000 字，建议收藏后慢慢研读。
一、多模态RAG概述：为什么需要它？
1.1 传统RAG的困境
传统 RAG 系统的典型 Pipeline 是这样的：
文档 → 文本提取 → 文本分块 → Embedding → 向量数据库 → 检索 → LLM生成这套流程在纯文本文档上表现不错，但一旦遇到多模态内容，立刻暴露三个致命问题：
| 问题 | 描述 | 示例 |
|---|---|---|
| 模态丢失 | 图片/音视频中的信息无法被索引 | 产品手册中的流程图无法被检索 |
| 语义断裂 | 表格、结构化内容的语义被破坏 | 财务报告中的图表无法被理解 |
| 跨模态检索缺失 | 用户无法用文字检索图片，也无法用图片检索视频 | "找到那张展示退货流程的截图" |
1.2 多模态RAG的核心思想
多模态 RAG 的核心思路是将不同模态的内容统一映射到同一个向量空间：
文本 → Text Encoder → 向量
图片 → Vision Encoder → 向量 → → 统一向量空间
音频 → Audio Encoder → 向量 → → 跨模态检索
视频 → Video Encoder → 向量 →在这个空间里，"一张展示服务器架构的图片"和"文字描述：描述一个三层架构的服务器部署图"会拥有相近的向量表示，从而实现真正的跨模态检索。
1.3 技术选型
本文选用以下技术栈构建多模态 RAG 系统：
| 组件 | 技术选型 | 选型理由 |
|---|---|---|
| LLM | GPT-4o / Qwen-VL | 支持原生多模态输入，减少工程复杂度 |
| 向量数据库 | Milvus 2.4+ | 支持混合标量+向量检索，水平扩展能力强 |
| 文本 Embedding | text-embedding-3-large / BGE | 高维度、高语义密度 |
| 图像 Embedding | CLIP (ViT-L/14) | 跨模态对齐的事实标准 |
| 音频处理 | Whisper (large-v3) | 高精度多语言语音识别 |
| 编排框架 | LangChain 0.3+ | 成熟的 RAG 编排生态 |
| 文档解析 | Unstructured-IO | 支持 50+ 文档格式自动解析 |
| 视频帧提取 | OpenCV + FFmpeg | 高性能帧级采样 |
二、系统架构设计
2.1 整体架构
┌─────────────────────────────────────────────────────────┐
│ 用户查询 │
│ "找到展示安全认证流程的图片" │
└─────────────────────┬───────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 查询理解层 │
│ Query Rewriting / HyDE / 多路召回 │
└──────────┬──────────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 检索引擎层 │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ 文本检索 │ │ 图像检索 │ │ 音视频检索 │ │
│ │ Milvus │ │ CLIP │ │ Whisper+ │ │
│ │ BM25 │ │ 索引 │ │ 时间戳索引 │ │
│ └────────────┘ └────────────┘ └────────────┘ │
└──────────┬──────────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 内容理解层 │
│ 多模态融合排序 / RRF 融合 / LLM 重排 │
└──────────┬──────────────────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ 生成层 │
│ GPT-4o / Qwen-VL 多模态生成 │
└─────────────────────────────────────────────────────────┘
上游数据流（Ingestion Pipeline）:
文档 ──→ 解析器 ──→ 模态分离 ──→ 各模态 Encoder ──→ 向量存储
(文本/图像/音频/视频)2.2 文档解析与模态分离
这是多模态 RAG 的第一步，也是最容易被忽视的一步。不同类型的文档需要不同的解析策略：
| 文档类型 | 解析工具 | 提取内容 |
|---|---|---|
| PDF（扫描件） | OCR（pytesseract） | 文字 + 图像区域坐标 |
| PDF（文字型） | PyMuPDF / pdfplumber | 文字 + 表格 + 图像 |
| Word | python-docx | 文字 + 图片 + 表格 |
| PPT | python-pptx | 文字 + 每页缩略图 |
| 图片 | PIL + OpenCV | 图像本身（作为独立模态） |
| 音视频 | FFmpeg + Whisper | 音频转文字 + 关键帧 |
| HTML | BeautifulSoup | 文字 + 图片链接 + 结构 |
解析完成后，我们需要将内容结构化存储，为后续向量化做准备：
python
# 定义多模态文档结构
from dataclasses import dataclass, field
from typing import List, Optional, Dict, Any
from datetime import datetime
@dataclass
class MultimodalChunk:
"""多模态内容块"""
chunk_id: str # 全局唯一ID
source_file: str # 来源文件
chunk_type: str # "text" | "image" | "audio" | "video_frame"
# 文本内容（所有类型都可能包含）
text: Optional[str] = None
# 图像特有
image_bytes: Optional[bytes] = None
image_path: Optional[str] = None
# 音视频特有
media_path: Optional[str] = None
timestamp_start: Optional[float] = None # 音视频时间戳（秒）
timestamp_end: Optional[float] = None
# 元数据
metadata: Dict[str, Any] = field(default_factory=dict)
created_at: datetime = field(default_factory=datetime.now)
def to_dict(self) -> Dict[str, Any]:
return {
"chunk_id": self.chunk_id,
"source_file": self.source_file,
"chunk_type": self.chunk_type,
"text": self.text,
"media_path": self.media_path,
"timestamp_start": self.timestamp_start,
"timestamp_end": self.timestamp_end,
"metadata": self.metadata,
}2.3 向量化与存储策略
多模态 RAG 的存储层需要支持异构向量的混合存储与检索：
python
# 多模态向量数据库管理器
from langchain_community.vectorstores import Milvus
from langchain_openai import OpenAIEmbeddings
import pymilvus as milvus
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
class MultimodalVectorStore:
"""支持多模态内容的向量数据库管理器"""
def __init__(self, host: str = "localhost", port: str = "19530"):
connections.connect(host=host, port=port)
self.embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
# CLIP 用于图像向量
self.clip_model = None # 初始化见后文
self._ensure_collections()
def _ensure_collections(self):
"""确保所需的 collection 存在"""
from pymilvus import utility
# 1. 文本 collection（支持混合检索：向量 + BM25）
if not utility.has_collection("text_chunks"):
text_fields = [
FieldSchema(name="id", dtype=DataType.VARCHAR, max_length=128, is_primary=True),
FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=8192),
FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=3072),
FieldSchema(name="source_file", dtype=DataType.VARCHAR, max_length=256),
FieldSchema(name="chunk_type", dtype=DataType.VARCHAR, max_length=32),
]
text_schema = CollectionSchema(fields=text_fields, description="Text chunks")
self.text_collection = Collection(name="text_chunks", schema=text_schema)
self.text_collection.create_index(
field_name="embedding",
index_params={"index_type": "HNSW", "metric_type": "IP", "params": {"M": 16, "efConstruction": 256}}
)
else:
self.text_collection = Collection("text_chunks")
# 2. 图像 collection（CLIP 向量）
if not utility.has_collection("image_chunks"):
image_fields = [
FieldSchema(name="id", dtype=DataType.VARCHAR, max_length=128, is_primary=True),
FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=1024), # 图像描述
FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=768), # CLIP ViT-L/14
FieldSchema(name="image_path", dtype=DataType.VARCHAR, max_length=512),
FieldSchema(name="source_file", dtype=DataType.VARCHAR, max_length=256),
]
image_schema = CollectionSchema(fields=image_fields, description="Image chunks with CLIP embeddings")
self.image_collection = Collection(name="image_chunks", schema=image_schema)
self.image_collection.create_index(
field_name="embedding",
index_params={"index_type": "HNSW", "metric_type": "COSINE", "params": {"M": 16, "efConstruction": 256}}
)
else:
self.image_collection = Collection("image_chunks")
# 3. 音视频 collection（Whisper 转写 + 时间戳）
if not utility.has_collection("media_chunks"):
media_fields = [
FieldSchema(name="id", dtype=DataType.VARCHAR, max_length=128, is_primary=True),
FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=8192),
FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=1536), # Whisper embedding
FieldSchema(name="media_path", dtype=DataType.VARCHAR, max_length=512),
FieldSchema(name="timestamp_start", dtype=DataType.FLOAT),
FieldSchema(name="timestamp_end", dtype=DataType.FLOAT),
FieldSchema(name="source_file", dtype=DataType.VARCHAR, max_length=256),
FieldSchema(name="duration", dtype=DataType.FLOAT),
]
media_schema = CollectionSchema(fields=media_fields, description="Audio/Video chunks")
self.media_collection = Collection(name="media_chunks", schema=media_schema)
self.media_collection.create_index(
field_name="embedding",
index_params={"index_type": "HNSW", "metric_type": "IP", "params": {"M": 16, "efConstruction": 256}}
)
# 时间戳索引用于时序检索
self.media_collection.create_index(
field_name="timestamp_start",
index_params={"index_type": "STL_SORT"}
)
else:
self.media_collection = Collection("media_chunks")
def add_text_chunk(self, chunk: MultimodalChunk):
"""添加文本块"""
vector = self.embeddings.embed_query(chunk.text)
entities = [[chunk.chunk_id], [chunk.text], [vector], [chunk.source_file], [chunk.chunk_type]]
self.text_collection.insert(entities)
self.text_collection.flush()
def add_image_chunk(self, chunk: MultimodalChunk, clip_model):
"""添加图像块（使用 CLIP 向量化）"""
from PIL import Image
import torch
from transformers import CLIPProcessor, CLIPModel
# 加载 CLIP（单例模式避免重复加载）
if self.clip_model is None:
self.clip_model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
self.clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
# 读取图像
if chunk.image_path:
image = Image.open(chunk.image_path).convert("RGB")
elif chunk.image_bytes:
from io import BytesIO
image = Image.open(BytesIO(chunk.image_bytes)).convert("RGB")
else:
return
# CLIP 向量化
inputs = self.clip_processor(text=[chunk.text or ""], images=image, return_tensors="pt", padding=True)
image_embedding = self.clip_model.get_image_feature(inputs["pixel_values"]).detach().numpy()[0]
# 归一化
image_embedding = image_embedding / (torch.norm(torch.tensor(image_embedding)) + 1e-8)
image_embedding = image_embedding.numpy().tolist()
entities = [
[chunk.chunk_id],
[chunk.text or ""],
[image_embedding],
[chunk.image_path or ""],
[chunk.source_file]
]
self.image_collection.insert(entities)
self.image_collection.flush()三、图片问答：CLIP驱动的视觉检索
3.1 CLIP的核心原理
CLIP（Contrastive Language-Image Pretraining）由 OpenAI 在 2021 年提出，其核心思想是通过大规模图文对学习，让图像和文本映射到同一个向量空间：
训练目标：最大化匹配图文对的余弦相似度，最小化不匹配的对
图像编码器 (ViT) ──→ Image Embedding ──┐
├──→ 对比损失
文本编码器 (Transformer) ──→ Text Embedding ──┘CLIP 的强大之处在于：无需针对特定任务微调，即可零样本（Zero-Shot）完成图像分类、检索等任务。对于多模态 RAG，CLIP 是连接视觉内容和文本检索的桥梁。
3.2 图像向量化与检索实现
python
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor, CLIPTokenizer
from langchain_core.documents import Document
from langchain_core.vectorstores import VectorStoreRetriever
import numpy as np
class CLIPImageVectorStore:
"""基于 CLIP 的图像向量存储与检索"""
def __init__(self, model_name: str = "openai/clip-vit-large-patch14"):
self.device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"[CLIP] Loading model on {self.device}...")
self.model = CLIPModel.from_pretrained(model_name).to(self.device)
self.processor = CLIPProcessor.from_pretrained(model_name)
self.tokenizer = CLIPTokenizer.from_pretrained(model_name)
# 初始化 Milvus 连接（简化版，使用 langchain-milvus）
self._vectorstore = None
def encode_image(self, image_path: str) -> np.ndarray:
"""将图像编码为向量"""
image = Image.open(image_path).convert("RGB")
inputs = self.processor(images=image, return_tensors="pt").to(self.device)
with torch.no_grad():
image_features = self.model.get_image_feature(inputs["pixel_values"])
# L2 归一化
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
return image_features.cpu().numpy()[0]
def encode_text(self, text: str) -> np.ndarray:
"""将文本编码为向量"""
inputs = self.tokenizer([text], padding=True, return_tensors="pt").to(self.device)
with torch.no_grad():
text_features = self.model.get_text_feature(inputs["input_ids"], attention_mask=inputs["attention_mask"])
text_features = text_features / text_features.norm(dim=-1, keepdim=True)
return text_features.cpu().numpy()[0]
def search_by_text(self, query: str, top_k: int = 5) -> list:
"""
用文本检索图像：输入一段文字，找到最相关的图片
示例：search_by_text("服务器集群架构图", top_k=3)
返回：[{image_path, similarity, caption}, ...]
"""
query_vector = self.encode_text(query)
# 从 Milvus 检索（需要先通过 add_images 建立索引）
results = self._milvus_search(
collection_name="image_chunks",
query_vector=query_vector.tolist(),
top_k=top_k
)
return results
def search_by_image(self, image_path: str, top_k: int = 5) -> list:
"""
以图搜图：输入一张图片，找到视觉相似的其他图片
"""
query_vector = self.encode_image(image_path)
results = self._milvus_search(
collection_name="image_chunks",
query_vector=query_vector.tolist(),
top_k=top_k
)
return results
def _milvus_search(self, collection_name: str, query_vector: list, top_k: int):
"""Milvus 检索辅助方法"""
from pymilvus import Collection, utility
import numpy as np
collection = Collection(collection_name)
collection.load()
search_params = {"metric_type": "COSINE", "params": {"ef": 128}}
results = collection.search(
data=[query_vector],
anns_field="embedding",
param=search_params,
limit=top_k,
output_fields=["id", "text", "image_path", "source_file"]
)
hits = []
for hits_group in results:
for hit in hits_group:
hits.append({
"chunk_id": hit.entity.get("id"),
"text": hit.entity.get("text"), # 图像描述/Alt文本
"image_path": hit.entity.get("image_path"),
"source_file": hit.entity.get("source_file"),
"similarity": float(hit.distance)
})
return hits3.3 图像问答的完整Pipeline
CLIP 检索到的图片如何交给 LLM 回答？这需要多模态 LLM（如 GPT-4o、Qwen-VL）来理解图像内容：
python
from openai import OpenAI
from typing import List, Dict, Any
class MultimodalRAGChain:
"""多模态 RAG 链：文本+图像+音视频联合检索与生成"""
def __init__(self):
self.client = OpenAI() # 或使用 Azure OpenAI / 本地模型
self.text_vectorstore = MultimodalVectorStore()
self.image_searcher = CLIPImageVectorStore()
self.media_vectorstore = MultimodalVectorStore()
def retrieve(self, query: str, top_k: int = 5) -> Dict[str, List[Dict]]:
"""
多路召回：同时从文本、图像、音视频三个维度检索
"""
# 1. 文本检索
text_results = self._retrieve_text(query, top_k)
# 2. 图像检索（CLIP 跨模态）
image_results = self.image_searcher.search_by_text(query, top_k)
# 3. 音视频检索
media_results = self._retrieve_media(query, top_k)
# 4. RRF 融合排序（Reciprocal Rank Fusion）
fused_results = self._rrf_fusion(
[text_results, image_results, media_results],
k=60 # RRF 超参数
)
return {
"text_chunks": text_results,
"image_chunks": image_results,
"media_chunks": media_results,
"fused": fused_results
}
def _rrf_fusion(self, result_lists: List[List], k: int = 60) -> List:
"""
RRF 融合算法：将多个检索结果列表合并排序
RRF score = Σ(1 / (k + rank))，其中 rank 是结果在各自列表中的排名
"""
scores = {}
for results in result_lists:
for rank, item in enumerate(results):
chunk_id = item["chunk_id"]
rrf_score = 1.0 / (k + rank + 1) # +1 因为 rank 从 0 开始
scores[chunk_id] = scores.get(chunk_id, 0) + rrf_score
# 按 RRF 分数排序
sorted_chunks = sorted(scores.items(), key=lambda x: x[1], reverse=True)
# 构建最终列表（需要去重并补全详细信息）
fused = []
for chunk_id, score in sorted_chunks:
# 根据 chunk_id 查询详细信息并添加到结果
fused.append({"chunk_id": chunk_id, "rrf_score": score})
return fused
def generate(self, query: str, retrieved: Dict) -> str:
"""
多模态生成：基于检索结果生成回答，支持图片输入
"""
# 构建多模态上下文
context_parts = []
# 文本上下文
if retrieved["text_chunks"]:
context_parts.append("【相关文档内容】\n" +
"\n".join([f"- {c['text'][:200]}..." for c in retrieved["text_chunks"][:3]]))
# 图像上下文（GPT-4o 原生支持图像输入）
image_contents = []
for img in retrieved["image_chunks"][:3]:
if img.get("image_path"):
image_contents.append({
"type": "image_url",
"image_url": {"url": f"file://{img['image_path']}"}
})
context_parts.append(f"【相关图片】{img.get('text', '无描述')} (来源: {img.get('source_file', '')})")
# 构建 messages
user_content = [{"type": "text", "text": f"基于以下上下文回答问题。\n\n{chr(10).join(context_parts)}\n\n问题：{query}"}]
user_content.extend(image_contents)
response = self.client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """你是一个企业知识库助手，擅长从多种格式的文档中提取和总结信息。
对于图像内容，请详细描述图中展示的信息。
回答要准确、简洁、有条理，适当引用检索到的来源。"""
},
{"role": "user", "content": user_content}
],
temperature=0.3,
max_tokens=2000
)
return response.choices[0].message.content
def run(self, query: str) -> Dict[str, Any]:
"""端到端执行"""
retrieved = self.retrieve(query)
answer = self.generate(query, retrieved)
return {
"query": query,
"answer": answer,
"retrieved": {
"text_count": len(retrieved["text_chunks"]),
"image_count": len(retrieved["image_chunks"]),
"media_count": len(retrieved["media_chunks"]),
"top_sources": [r["chunk_id"] for r in retrieved["fused"][:5]]
}
}四、音视频检索：Whisper+Faiss/Milvus
4.1 音视频处理架构
音视频内容的处理比文本和图像复杂得多，主要挑战在于：
-
时间维度：音频/视频是连续流，需要切片并保留时间戳
-
多模态交织：视频同时包含视觉（帧）和听觉（音频）信息
-
计算成本：Whisper 推理、帧提取的计算量远大于文本处理
音视频文件
│
├──[音频流]──→ Whisper ASR ──→ 转写文本 + 时间戳 ──→ 向量化（Whisper embedding）
│ │
├──[视频流]──→ FFmpeg 关键帧提取 ──→ 关键帧图像 ──→ CLIP 向量化 │
│ │
└──────────────────────────────────────────────────────────────┘
│
▼
Milvus 存储（带时间戳索引）
4.2 Whisper 音频处理
python
import whisper
import numpy as np
from pydub import AudioSegment
import tempfile
import os
class AudioProcessor:
"""音频处理器：Whisper 转写 + 向量化"""
def __init__(self, model_name: str = "large-v3"):
self.device = "cuda" if whisper.utils.get_audio_device() else "cpu"
print(f"[Whisper] Loading model '{model_name}' on {self.device}...")
self.model = whisper.load_model(model_name, device=self.device)
def transcribe_with_timestamps(
self,
audio_path: str,
language: str = "zh",
chunk_duration: float = 30.0
) -> List[Dict]:
"""
转写音频并返回带时间戳的分段结果
Args:
audio_path: 音频文件路径
language: 语言代码
chunk_duration: 每个分段的目标时长（秒）
Returns:
List[{"text", "start", "end", "embedding", "chunk_id"}]
"""
print(f"[Whisper] Transcribing: {audio_path}")
# 转写（包含时间戳信息）
result = self.model.transcribe(
audio_path,
language=language,
word_timestamps=True, # 开启词级时间戳
segment_duration=chunk_duration,
verbose=False
)
segments = []
for seg in result["segments"]:
chunk = {
"text": seg["text"].strip(),
"start": seg["start"],
"end": seg["end"],
"embedding": self._get_segment_embedding(seg["text"]),
"chunk_id": f"audio_{hash(audio_path)}_{seg['start']:.1f}"
}
segments.append(chunk)
print(f"[Whisper] Transcribed {len(segments)} segments")
return segments
def _get_segment_embedding(self, text: str) -> List[float]:
"""获取 Whisper 音频段的文本向量（使用 text-embedding-3-large 模拟音频语义）"""
# 注意：实际生产中应使用音频特征（Whisper 的 decoder hidden states）
# 这里用文本 embedding 近似，实际项目可用 wav2vec2 或 AudioCLIP
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vec = embeddings.embed_query(text)
return vec
def extract_audio_from_video(self, video_path: str) -> str:
"""从视频中提取音频"""
import subprocess
# 用 FFmpeg 提取音频（转码为 mp3 / wav）
audio_path = video_path.rsplit(".", 1)[0] + "_audio.wav"
cmd = [
"ffmpeg", "-y", "-i", video_path,
"-vn", # 去除视频轨道
"-acodec", "pcm_s16le", # PCM 格式
"-ar", "16000", # 16kHz（Whisper 推荐采样率）
"-ac", "1", # 单声道
audio_path
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise RuntimeError(f"FFmpeg failed: {result.stderr}")
return audio_path
class VideoFrameExtractor:
"""视频关键帧提取器"""
def __init__(self, fps: int = 1):
"""
Args:
fps: 每秒提取的帧数（建议 1fps 足够，过高增加计算成本）
"""
self.fps = fps
def extract_keyframes(self, video_path: str, output_dir: str) -> List[Dict]:
"""
提取视频关键帧并保存
Returns:
List[{"frame_path", "timestamp", "chunk_id"}]
"""
import cv2
cap = cv2.VideoCapture(video_path)
video_fps = cap.get(cv2.CAP_PROP_FPS)
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
duration = total_frames / video_fps
frame_interval = int(video_fps / self.fps) if self.fps > 0 else 1
frames = []
frame_count = 0
saved_count = 0
while True:
ret, frame = cap.read()
if not ret:
break
if frame_count % frame_interval == 0:
timestamp = frame_count / video_fps
# 构建保存路径
filename = f"frame_{saved_count:06d}_{timestamp:.2f}.jpg"
frame_path = os.path.join(output_dir, filename)
# 质量压缩（节省存储）
cv2.imwrite(frame_path, frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
frames.append({
"frame_path": frame_path,
"timestamp": timestamp,
"chunk_id": f"video_{hash(video_path)}_{timestamp:.2f}"
})
saved_count += 1
frame_count += 1
cap.release()
print(f"[Video] Extracted {saved_count} keyframes from {duration:.1f}s video")
return frames4.3 音视频检索的完整实现
python
from pymilvus import Collection, DataType, FieldSchema, CollectionSchema
class MediaVectorStore:
"""音视频向量存储管理"""
def __init__(self, milvus_host: str = "localhost", milvus_port: str = "19530"):
import pymilvus as milvus
milvus.connections.connect(host=milvus_host, port=milvus_port)
self.collection = Collection("media_chunks")
def insert_segments(self, segments: List[Dict]):
"""
批量插入音频/视频分段
segments: [{"chunk_id", "text", "embedding", "media_path",
"timestamp_start", "timestamp_end", "source_file"}]
"""
entities = [
[s["chunk_id"] for s in segments],
[s["text"] for s in segments],
[s["embedding"] for s in segments],
[s.get("media_path", "") for s in segments],
[s["timestamp_start"] for s in segments],
[s["timestamp_end"] for s in segments],
[s.get("source_file", "") for s in segments],
[s.get("duration", s["timestamp_end"] - s["timestamp_start"]) for s in segments],
]
self.collection.insert(entities)
self.collection.flush()
print(f"[MediaStore] Inserted {len(segments)} segments")
def search_by_text(self, query: str, top_k: int = 5) -> List[Dict]:
"""文本检索音频内容"""
from langchain_openai import OpenAIEmbeddings
import numpy as np
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
query_vec = embeddings.embed_query(query)
self.collection.load()
search_params = {"metric_type": "IP", "params": {"ef": 128}}
results = self.collection.search(
data=[query_vec],
anns_field="embedding",
param=search_params,
limit=top_k,
output_fields=["chunk_id", "text", "media_path", "timestamp_start",
"timestamp_end", "source_file", "duration"]
)
hits = []
for hits_group in results:
for hit in hits_group:
hits.append({
"chunk_id": hit.entity.get("chunk_id"),
"text": hit.entity.get("text"),
"media_path": hit.entity.get("media_path"),
"timestamp_start": hit.entity.get("timestamp_start"),
"timestamp_end": hit.entity.get("timestamp_end"),
"source_file": hit.entity.get("source_file"),
"duration": hit.entity.get("duration"),
"similarity": float(hit.distance)
})
return hits
def search_by_timestamp(
self,
media_path: str,
start: float,
end: float,
top_k: int = 5
) -> List[Dict]:
"""
按时间范围检索：用于"找到视频第 30~60 秒的内容"这类查询
"""
self.collection.load()
# 标量过滤 + 向量检索的混合查询
search_params = {
"metric_type": "IP",
"params": {"ef": 128},
"expression": f'timestamp_start >= {start} and timestamp_end <= {end}'
}
# 需要一个 dummy 向量（时间戳查询主要依赖标量过滤）
dummy_vec = [0.0] * 1536
results = self.collection.search(
data=[dummy_vec],
anns_field="embedding",
param=search_params,
limit=top_k,
output_fields=["chunk_id", "text", "media_path", "timestamp_start",
"timestamp_end", "source_file"],
expr=f'timestamp_start >= {start} and timestamp_end <= {end}'
)
hits = []
for hits_group in results:
for hit in hits_group:
hits.append({
"chunk_id": hit.entity.get("chunk_id"),
"text": hit.entity.get("text"),
"media_path": hit.entity.get("media_path"),
"timestamp_start": hit.entity.get("timestamp_start"),
"timestamp_end": hit.entity.get("timestamp_end"),
"source_file": hit.entity.get("source_file"),
"similarity": float(hit.distance)
})
return hits五、完整项目实战
5.1 项目结构
multimodal_rag/
├── config.py # 配置管理
├── models/
│ ├── document_parser.py # 文档解析器
│ ├── embedder.py # 多模态 Embedding 管理器
│ └── vectorstore.py # Milvus 向量存储
├── pipeline/
│ ├── ingestion.py # 数据摄入管道
│ ├── retrieval.py # 检索管道
│ └── generation.py # 生成管道
├── chains/
│ └── multimodal_chain.py # LangChain 链定义
├── app.py # FastAPI 服务入口
├── ingest_demo.py # 摄入示例
└── requirements.txt5.2 配置管理
python
# config.py
import os
from pathlib import Path
from dotenv import load_dotenv
load_dotenv()
class Config:
# OpenAI
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "")
OPENAI_BASE_URL = os.getenv("OPENAI_BASE_URL", "") # 代理地址
# Milvus
MILVUS_HOST = os.getenv("MILVUS_HOST", "localhost")
MILVUS_PORT = os.getenv("MILVUS_PORT", "19530")
# 模型配置
TEXT_EMBEDDING_MODEL = "text-embedding-3-large"
TEXT_EMBEDDING_DIM = 3072
CLIP_MODEL = "openai/clip-vit-large-patch14"
CLIP_EMBEDDING_DIM = 768
WHISPER_MODEL = "large-v3"
WHISPER_EMBEDDING_DIM = 1536
# 切分策略
TEXT_CHUNK_SIZE = 512
TEXT_CHUNK_OVERLAP = 128
AUDIO_SEGMENT_DURATION = 30.0 # 秒
VIDEO_KEYFRAME_FPS = 1 # 每秒1帧
# 检索参数
TOP_K_PER_MODALITY = 5
RRF_K = 60
# 文件路径
WORKSPACE = Path(__file__).parent
UPLOAD_DIR = WORKSPACE / "uploads"
UPLOAD_DIR.mkdir(exist_ok=True)
config = Config()5.3 数据摄入管道
python
# pipeline/ingestion.py
import hashlib
import os
from pathlib import Path
from typing import List
import subprocess
from models.document_parser import DocumentParser
from models.embedder import MultimodalEmbedder
from models.vectorstore import MultimodalVectorStore
from config import config
class IngestionPipeline:
"""多模态数据摄入管道"""
def __init__(self):
self.parser = DocumentParser()
self.embedder = MultimodalEmbedder()
self.vectorstore = MultimodalVectorStore(
host=config.MILVUS_HOST,
port=config.MILVUS_PORT
)
def ingest_file(self, file_path: str) -> dict:
"""
摄入单个文件，自动识别类型并处理
Returns:
{"status": "success", "chunks_processed": N, "file_type": "..."}
"""
file_path = Path(file_path)
suffix = file_path.suffix.lower()
print(f"\n{'='*60}")
print(f"[Ingestion] Processing: {file_path.name}")
print(f"{'='*60}")
try:
if suffix in [".txt", ".md", ".csv"]:
return self._ingest_text(file_path)
elif suffix in [".pdf", ".docx", ".pptx"]:
return self._ingest_document(file_path)
elif suffix in [".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp"]:
return self._ingest_image(file_path)
elif suffix in [".mp3", ".wav", ".m4a", ".flac"]:
return self._ingest_audio(file_path)
elif suffix in [".mp4", ".avi", ".mov", ".mkv"]:
return self._ingest_video(file_path)
else:
return {"status": "skipped", "reason": f"Unsupported file type: {suffix}"}
except Exception as e:
return {"status": "error", "file": str(file_path), "error": str(e)}
def _ingest_text(self, file_path: Path) -> dict:
"""摄入纯文本文件"""
from models.document_parser import MultimodalChunk
content = file_path.read_text(encoding="utf-8")
chunks = self.parser.chunk_text(content, file_path.name)
for chunk in chunks:
self.vectorstore.add_text_chunk(chunk)
return {"status": "success", "chunks_processed": len(chunks), "file_type": "text"}
def _ingest_document(self, file_path: Path) -> dict:
"""摄入 PDF/Word/PPT 文档"""
from models.document_parser import MultimodalChunk
parsed = self.parser.parse_document(str(file_path))
# 处理文本块
text_chunks = 0
for text_block in parsed["text_blocks"]:
chunk = MultimodalChunk(
chunk_id=self._gen_chunk_id(file_path, "text", text_block["page"]),
source_file=file_path.name,
chunk_type="text",
text=text_block["text"],
metadata={"page": text_block["page"], "type": "document_text"}
)
self.vectorstore.add_text_chunk(chunk)
text_chunks += 1
# 处理图像块（文档中的图片）
image_chunks = 0
for img_block in parsed["image_blocks"]:
chunk = MultimodalChunk(
chunk_id=self._gen_chunk_id(file_path, "image", img_block["index"]),
source_file=file_path.name,
chunk_type="image",
text=img_block.get("caption", ""),
image_bytes=img_block["image_bytes"],
metadata={"page": img_block.get("page"), "type": "document_image"}
)
self.vectorstore.add_image_chunk(chunk, self.embedder.clip_model)
image_chunks += 1
print(f" → {text_chunks} text chunks, {image_chunks} image chunks")
return {
"status": "success",
"chunks_processed": text_chunks + image_chunks,
"file_type": "document",
"text_chunks": text_chunks,
"image_chunks": image_chunks
}
def _ingest_image(self, file_path: Path) -> dict:
"""摄入单张图片"""
from models.document_parser import MultimodalChunk
# 尝试用 BLIP 生成图像描述
caption = self.embedder.generate_image_caption(str(file_path))
chunk = MultimodalChunk(
chunk_id=self._gen_chunk_id(file_path, "image", 0),
source_file=file_path.name,
chunk_type="image",
text=caption,
image_path=str(file_path),
metadata={"width": 0, "height": 0} # 可补充图像尺寸
)
self.vectorstore.add_image_chunk(chunk, self.embedder.clip_model)
return {"status": "success", "chunks_processed": 1, "file_type": "image"}
def _ingest_audio(self, file_path: Path) -> dict:
"""摄入音频文件"""
from models.document_parser import MultimodalChunk
from pipeline.audio_processing import AudioProcessor
audio_proc = AudioProcessor(model_name=config.WHISPER_MODEL)
segments = audio_proc.transcribe_with_timestamps(
str(file_path),
language="zh",
chunk_duration=config.AUDIO_SEGMENT_DURATION
)
for seg in segments:
chunk = MultimodalChunk(
chunk_id=seg["chunk_id"],
source_file=file_path.name,
chunk_type="audio",
text=seg["text"],
media_path=str(file_path),
timestamp_start=seg["start"],
timestamp_end=seg["end"],
metadata={"duration": seg["end"] - seg["start"]}
)
self.vectorstore.add_media_chunk(chunk)
return {
"status": "success",
"chunks_processed": len(segments),
"file_type": "audio"
}
def _ingest_video(self, file_path: Path) -> dict:
"""摄入视频文件"""
from models.document_parser import MultimodalChunk
from pipeline.audio_processing import AudioProcessor, VideoFrameExtractor
import tempfile
# 1. 提取音频并转写
audio_proc = AudioProcessor(model_name=config.WHISPER_MODEL)
audio_path = audio_proc.extract_audio_from_video(str(file_path))
segments = audio_proc.transcribe_with_timestamps(audio_path, language="zh")
for seg in segments:
chunk = MultimodalChunk(
chunk_id=seg["chunk_id"],
source_file=file_path.name,
chunk_type="video_audio",
text=seg["text"],
media_path=str(file_path),
timestamp_start=seg["start"],
timestamp_end=seg["end"],
)
self.vectorstore.add_media_chunk(chunk)
# 2. 提取关键帧（可选，节省资源时可跳过）
with tempfile.TemporaryDirectory() as tmpdir:
frame_extractor = VideoFrameExtractor(fps=config.VIDEO_KEYFRAME_FPS)
frames = frame_extractor.extract_keyframes(str(file_path), tmpdir)
for frame in frames:
# 用 BLIP 生成帧描述
caption = self.embedder.generate_image_caption(frame["frame_path"])
frame_chunk = MultimodalChunk(
chunk_id=frame["chunk_id"],
source_file=file_path.name,
chunk_type="video_frame",
text=caption,
image_path=frame["frame_path"],
media_path=str(file_path),
timestamp_start=frame["timestamp"],
timestamp_end=frame["timestamp"] + 1.0,
metadata={"frame_time": frame["timestamp"]}
)
self.vectorstore.add_image_chunk(frame_chunk, self.embedder.clip_model)
return {
"status": "success",
"chunks_processed": len(segments) + len(frames),
"file_type": "video",
"audio_segments": len(segments),
"video_frames": len(frames)
}
def _gen_chunk_id(self, file_path: Path, prefix: str, index: int) -> str:
"""生成唯一 chunk ID"""
hash_str = hashlib.md5(f"{file_path.name}_{prefix}_{index}".encode()).hexdigest()[:12]
return f"{prefix}_{file_path.stem}_{hash_str}"5.4 LangChain 链定义
python
# chains/multimodal_chain.py
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from typing import Dict, List, Any
import os
# ============ 1. 检索链（Retrieval Chain）===========
retrieval_prompt = ChatPromptTemplate.from_messages([
("system", """你是一个专业的企业知识库检索助手。
给定用户查询，你需要：
1. 分析查询的意图和关键词
2. 判断用户想要检索哪种类型的内容（文档/图片/音视频）
3. 生成优化的检索查询
输出格式（JSON）：
{{"rewritten_query": "...", "search_type": "text|image|video|all"}}
"""),
("human", "用户查询：{query}")
])
retrieval_chain = RunnablePassthrough.assign(
rewritten_query=lambda x: x["query"]
) | retrieval_prompt | ChatOpenAI(
model="gpt-4o",
api_key=os.getenv("OPENAI_API_KEY"),
base_url=os.getenv("OPENAI_BASE_URL"),
temperature=0.0
).bind(response_format={"type": "json_object"}) | StrOutputParser()
# ============ 2. 生成链（Generation Chain）===========
generation_prompt = ChatPromptTemplate.from_messages([
("system", """你是一个企业知识库问答助手。
## 你的能力
- 可以理解文本、图片、音频内容
- 可以引用具体来源（文件名称、时间戳等）
- 擅长总结、对比、解释和推理
## 回答原则
1. 优先使用提供的上下文信息回答
2. 如果上下文中没有相关信息，明确告知用户
3. 适当引用来源，按格式：[来源：文件名]
4. 涉及图片时，详细描述图中的信息
5. 涉及音视频时，引用对应的时间戳，如：[来源：会议录音 00:30-01:05]
## 上下文信息
{context}
## 多模态资源
{resources}
"""),
("human", "问题：{question}")
])
generation_chain = generation_prompt | ChatOpenAI(
model="gpt-4o",
api_key=os.getenv("OPENAI_API_KEY"),
base_url=os.getenv("OPENAI_BASE_URL"),
temperature=0.3
) | StrOutputParser()
# ============ 3. 完整 RAG 链（Full Multimodal RAG Chain）===========
def build_multimodal_rag_chain(vectorstore, image_searcher, media_searcher):
"""
构建完整的多模态 RAG 链
Args:
vectorstore: MultimodalVectorStore 实例
image_searcher: CLIPImageVectorStore 实例
media_searcher: MediaVectorStore 实例
"""
from langchain_core.runnables import RunnableLambda
def multimodal_retrieve(query: str) -> Dict[str, Any]:
"""多路召回"""
# 并行执行三路检索
import concurrent.futures
results = {}
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
future_text = executor.submit(vectorstore.search_text, query, k=5)
future_image = executor.submit(image_searcher.search_by_text, query, top_k=5)
future_media = executor.submit(media_searcher.search_by_text, query, top_k=5)
results["text"] = future_text.result()
results["image"] = future_image.result()
results["media"] = future_media.result()
# 构建上下文
context_parts = []
resources = []
# 文本上下文
if results["text"]:
context_parts.append("【文档内容】")
for item in results["text"]:
context_parts.append(f"[文档] {item['text'][:300]}")
resources.append({
"type": "text",
"id": item["chunk_id"],
"source": item.get("source_file", "Unknown")
})
# 图像上下文
if results["image"]:
context_parts.append("\n【相关图片】")
for item in results["image"]:
context_parts.append(f"[图片] {item.get('text', '无描述')}")
resources.append({
"type": "image",
"id": item["chunk_id"],
"path": item.get("image_path", ""),
"source": item.get("source_file", "Unknown")
})
# 音视频上下文
if results["media"]:
context_parts.append("\n【音视频内容】")
for item in results["media"]:
ts_start = item.get("timestamp_start", 0)
ts_end = item.get("timestamp_end", 0)
context_parts.append(
f"[音视频 {ts_start:.0f}s-{ts_end:.0f}s] {item['text'][:200]}"
)
resources.append({
"type": "media",
"id": item["chunk_id"],
"path": item.get("media_path", ""),
"timestamp": f"{ts_start:.0f}s-{ts_end:.0f}s",
"source": item.get("source_file", "Unknown")
})
return {
"context": "\n".join(context_parts),
"resources": resources,
"question": query
}
return RunnableLambda(multimodal_retrieve) | generation_chain5.5 摄入示例
python
# ingest_demo.py
from pipeline.ingestion import IngestionPipeline
from pathlib import Path
import time
def main():
pipeline = IngestionPipeline()
# 指定要摄入的文件目录
data_dir = Path("./sample_docs")
if not data_dir.exists():
print("请创建 sample_docs 目录并放入测试文件")
return
files = list(data_dir.iterate_files()) if hasattr(datair, 'iterate_files') else list(data_dir.glob("*"))
print(f"Found {len(files)} files to ingest")
results = []
start_time = time.time()
for file_path in files:
result = pipeline.ingest_file(str(file_path))
results.append(result)
print(f" ✓ {file_path.name}: {result}")
elapsed = time.time() - start_time
successful = sum(1 for r in results if r.get("status") == "success")
print(f"\n{'='*60}")
print(f"摄入完成！共 {len(files)} 个文件，成功 {successful} 个")
print(f"总耗时: {elapsed:.1f} 秒")
print(f"{'='*60}")
if __name__ == "__main__":
main()5.6 FastAPI 服务
python
# app.py
from fastapi import FastAPI, UploadFile, File, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from typing import Optional
import tempfile
import os
from chains.multimodal_chain import build_multimodal_rag_chain
from models.vectorstore import MultimodalVectorStore
from models.embedder import CLIPImageVectorStore
from pipeline.audio_processing import MediaVectorStore as MediaStore
from pipeline.ingestion import IngestionPipeline
app = FastAPI(title="Multimodal RAG API", version="1.0.0")
# 初始化组件（启动时加载，避免每次请求都重新初始化）
print("[Startup] Initializing components...")
vectorstore = MultimodalVectorStore()
image_searcher = CLIPImageVectorStore()
media_searcher = MediaStore()
rag_chain = build_multimodal_rag_chain(vectorstore, image_searcher, media_searcher)
ingestion_pipeline = IngestionPipeline()
print("[Startup] Ready!")
class QueryRequest(BaseModel):
query: str
top_k: Optional[int] = 5
@app.post("/rag/query")
async def query(request: QueryRequest):
"""多模态问答接口"""
try:
result = rag_chain.invoke({"query": request.query})
return {
"answer": result,
"status": "success"
}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.post("/ingest/file")
async def ingest_file(file: UploadFile = File(...)):
"""上传并摄入单个文件"""
try:
with tempfile.NamedTemporaryFile(delete=False, suffix=file.filename) as tmp:
content = await file.read()
tmp.write(content)
tmp_path = tmp.name
result = ingestion_pipeline.ingest_file(tmp_path)
# 清理临时文件
os.unlink(tmp_path)
return result
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@app.get("/health")
async def health():
return {"status": "healthy", "service": "multimodal-rag"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)六、效果评估
6.1 评估指标体系
多模odal RAG 系统的评估比纯文本 RAG 复杂得多，需要覆盖多个维度：
| 维度 | 指标 | 测量方法 |
|---|---|---|
| 召回质量 | Recall@K, MRR | 人工标注相关文档，计算命中率 |
| 检索精度 | Precision@K, NDCG@K | 排序质量评估 |
| 跨模态能力 | CLIP Score, Zero-shot Accuracy | 用文本检索图像的准确率 |
| 生成质量 | RAGAS Score, G-Eval | LLM 自动化评估 |
| 响应延迟 | P50/P95/P99 Latency | 端到端耗时分布 |
| 系统吞吐 | QPS, 并发用户数 | 压力测试 |
6.2 检索质量评估代码
python
# evaluation/retrieval_eval.py
from typing import List, Dict, Tuple
import numpy as np
from sklearn.metrics import ndcg_score
class RetrievalEvaluator:
"""检索质量评估器"""
def __init__(self):
self.results = [] # 存储每次检索的结果
def add_result(
self,
query: str,
retrieved_ids: List[str],
relevant_ids: List[str],
k_values: List[int] = [1, 3, 5, 10]
):
"""添加一次检索结果"""
self.results.append({
"query": query,
"retrieved": retrieved_ids,
"relevant": relevant_ids,
"k_values": k_values
})
def compute_metrics(self) -> Dict[str, float]:
"""计算各项指标"""
metrics = {}
# Recall@K
for k in [1, 3, 5, 10]:
recalls = []
for r in self.results:
retrieved_k = set(r["retrieved"][:k])
relevant = set(r["relevant"])
if len(relevant) > 0:
recall = len(retrieved_k & relevant) / len(relevant)
else:
recall = 1.0
recalls.append(recall)
metrics[f"Recall@{k}"] = np.mean(recalls)
# MRR (Mean Reciprocal Rank)
mrrs = []
for r in self.results:
relevant = set(r["relevant"])
mrr = 0.0
for rank, doc_id in enumerate(r["retrieved"], 1):
if doc_id in relevant:
mrr = 1.0 / rank
break
mrrs.append(mrr)
metrics["MRR"] = np.mean(mrrs)
# Hit Rate@K
for k in [1, 3, 5]:
hits = []
for r in self.results:
retrieved_k = set(r["retrieved"][:k])
relevant = set(r["relevant"])
hits.append(1.0 if len(retrieved_k & relevant) > 0 else 0.0)
metrics[f"HitRate@{k}"] = np.mean(hits)
return metrics
def cross_modality_eval(self, text_queries: List[str], image_ids: List[str]) -> Dict[str, float]:
"""
跨模态评估：用文本查询图像，检查是否命中正确图像
这是多模态 RAG 独有的评估维度
"""
from models.embedder import CLIPImageVectorStore
searcher = CLIPImageVectorStore()
correct = 0
for query, expected_image_id in zip(text_queries, image_ids):
results = searcher.search_by_text(query, top_k=5)
retrieved_ids = [r["chunk_id"] for r in results]
if expected_image_id in retrieved_ids:
correct += 1
return {
"cross_modality_accuracy": correct / len(text_queries),
"total_queries": len(text_queries),
"correct": correct
}
def print_report(self):
"""打印评估报告"""
metrics = self.compute_metrics()
print("\n" + "=" * 50)
print("📊 多模态 RAG 检索质量评估报告")
print("=" * 50)
print(f"总测试用例数: {len(self.results)}")
print("-" * 50)
print("\n【召回指标】")
for k in [1, 3, 5, 10]:
if f"Recall@{k}" in metrics:
print(f" Recall@{k}: {metrics[f'Recall@{k}']:.4f}")
print(f"\n【排序质量】")
print(f" MRR: {metrics['MRR']:.4f}")
print("\n【命中率】")
for k in [1, 3, 5]:
if f"HitRate@{k}" in metrics:
print(f" HitRate@{k}: {metrics[f'HitRate@{k}']:.4f}")
print("=" * 50)
# 使用示例
if __name__ == "__main__":
evaluator = RetrievalEvaluator()
# 模拟测试数据（实际项目需要人工标注）
test_cases = [
# (query, retrieved_ids, relevant_ids)
("服务器集群架构图", ["img_001", "img_002", "txt_003"], ["img_001"]),
("产品退换货流程", ["txt_010", "img_015", "video_020"], ["img_015", "video_020"]),
("安全认证流程", ["doc_030", "img_031", "img_032"], ["img_031"]),
# ... 更多测试用例
]
for query, retrieved, relevant in test_cases:
evaluator.add_result(query, retrieved, relevant)
evaluator.print_report()
# 跨模态评估示例
cross_results = evaluator.cross_modality_eval(
text_queries=[
"一张展示 Kubernetes 架构的图",
"显示用户登录流程的截图",
"财务报表的柱状图",
],
image_ids=[
"img_k8s_arch",
"img_login_flow",
"img_finance_chart",
]
)
print(f"\n【跨模态准确率】: {cross_results['cross_modality_accuracy']:.2%}")6.3 性能基准测试
python
# evaluation/benchmark.py
import time
import statistics
from concurrent.futures import ThreadPoolExecutor
def benchmark_ingestion(files: List[str], pipeline):
"""摄入性能基准测试"""
times = []
for f in files:
start = time.time()
pipeline.ingest_file(f)
elapsed = time.time() - start
times.append(elapsed)
return {
"total_files": len(files),
"total_time": sum(times),
"avg_time_per_file": statistics.mean(times),
"p50": statistics.median(times),
"p95": sorted(times)[int(len(times) * 0.95)] if len(times) > 1 else times[0],
}
def benchmark_concurrent_queries(chain, queries: List[str], concurrency: int = 10):
"""并发查询性能基准测试"""
latencies = []
errors = 0
def single_query(q):
start = time.time()
try:
chain.invoke({"query": q})
return time.time() - start, True
except Exception:
return time.time() - start, False
with ThreadPoolExecutor(max_workers=concurrency) as executor:
futures = [executor.submit(single_query, q) for q in queries * (concurrency // len(queries) + 1)]
for f in futures[:100]: # 限制总请求数
latency, success = f.result()
latencies.append(latency)
if not success:
errors += 1
sorted_lat = sorted(latencies)
return {
"total_requests": len(latencies),
"errors": errors,
"avg_latency_ms": statistics.mean(latencies) * 1000,
"p50_ms": sorted_lat[len(sorted_lat)//2] * 1000,
"p95_ms": sorted_lat[int(len(sorted_lat)*0.95)] * 1000,
"p99_ms": sorted_lat[int(len(sorted_lat)*0.99)] * 1000,
"qps": len(latencies) / sum(latencies),
}七、进阶优化建议
7.1 检索优化
多路召回 + 精细化重排
当前方案使用 RRF 融合，在生产环境中建议增加 LLM-as-a-Judge 重排层：
python
# 重排层：使用 LLM 判断每个检索结果与查询的相关性
def rerank_results(query: str, retrieved_items: List[Dict], top_n: int = 3) -> List[Dict]:
"""
使用 LLM 对检索结果进行重排
实际生产中可使用 BGE-Reranker 或 Cohere Rerank API
"""
rerank_prompt = f"""请评估以下检索结果与查询的相关性。
查询：{query}
候选结果：
{chr(10).join([f"[{i+1}] {item.get('text', item.get('caption', ''))}" for i, item in enumerate(retrieved_items)])}
请按相关性从高到低排列结果，返回排序后的编号列表（如：3,1,2）
"""
# 调用 LLM 获取排序
# 实际项目中推荐使用专门的 Rerank 模型（BGE-Reranker）而非 LLM
response = llm.invoke(rerank_prompt)
# 解析并返回重排后的结果
...7.2 存储优化
向量量化压缩
当数据量达到百万级时，HNSW 索引的内存占用会成为瓶颈。使用**乘积量化（PQ）**可以将向量压缩 4~8 倍：
python
# Milvus 中创建量化索引
self.collection.create_index(
field_name="embedding",
index_params={
"index_type": "IVF_PQ", # IVF_PQ 比 HNSW 节省 50%+ 内存
"metric_type": "IP",
"params": {
"nlist": 1024,
"nbits": 8, # 每个子向量用 8 bit 表示
"m": 16
}
}
)7.3 模态融合的更多可能性
| 技术方案 | 适用场景 | 优缺点 |
|---|---|---|
| BLIP-2 | 图像描述生成 | 零样本，但精度低于微调模型 |
| LLaVA | 开源本地多模态 | 完全私有，但推理较慢 |
| Gemini Pro | Google 生态集成 | 原生多模态，但需 GCP |
| AudioCLIP | 音频-图像-文本三模态 | 适合音乐/音效检索场景 |
总结
本文完整构建了一套企业级多模态 RAG 系统，覆盖了从文档解析、多模态向量化、向量存储、多路检索到 LLM 生成的完整链路。核心技术要点回顾：
- 统一向量空间：通过 CLIP 实现图文对齐，通过 Whisper 实现音视频文本化并向量化，所有模态映射到统一空间
- 多路召回 + RRF 融合：文本、图像、音视频三路并行检索，通过 Reciprocal Rank Fusion 合并排序
- LangChain 编排：用 RunnableLambda 和 ChatPromptTemplate 构建灵活的 RAG 链，支持插拔式组件
- Milvus 2.4+：支持混合标量+向量检索，带时间戳索引的音视频存储
下一步建议：
- 接入向量数据库的全文检索能力（Milvus BM25 混合搜索）
- 添加 LLM 重排层提升 top-K 质量
- 引入知识图谱（Neo4j）做实体级别的跨模态关联
- 部署 GPU 推理加速 Whisper 和 CLIP
如果本文对你有帮助，欢迎在评论区交流你在实际项目中遇到的问题和经验！
📚 相关技术文档