MLE-VLM

VLM（Vision-Language Model）是多模态 AI 的重要方向。这组题既涉及视觉与语言之间的表示学习，也涉及大模型架构、微调、可靠性评估和实际部署。CLIP、LLaVA 和 MiniGPT-4，主要是 Vision + Language。如果进一步加入 Speech、Audio 和 Video，就进入了更广泛的 Multimodal LLM / Omni-Modal Model（全模态模型） 领域。

1. Representation Alignment：图像、语音、声音和文本如何映射到可以共同理解的表示空间？

2. Temporal Synchronization：视频画面和声音如何按照真实时间关系对齐？

3. Real-time Generation：模型如何一边听、一边理解，并以低延迟的方式生成自然语音？

Q1. 多模态大模型的核心挑战是什么？如何实现视觉与语言的有效对齐和融合？

中文面试回答

多模态大模型的核心挑战是如何将具有不同统计特性和表示结构的视觉与语言信息映射到能够交互的表示空间，并实现有效的信息融合。图像通常具有二维空间结构，包含大量连续的视觉特征，而文本是离散的 Token 序列，具有语法和语义结构。因此，仅仅将图像 Embedding 和文本 Embedding 拼接起来，并不能保证模型真正理解两种模态之间的关系。我会将核心挑战分为三个方面：第一是 Representation Alignment，通过 Contrastive Learning 或可训练的 Projection Layer 建立跨模态语义关联；第二是 Cross-modal Fusion，使用 Cross-Attention、Visual Token Injection 或共享 Transformer，让语言模型能够利用视觉证据进行推理；第三是 Fine-grained Grounding，确保模型不仅知道图像整体表达什么，还能够正确识别对象、位置、属性和空间关系。此外，训练数据的图文匹配质量、不同模态的信息密度、视觉 Token 的计算成本，以及视觉幻觉问题都会影响系统性能。因此，我会分别评估 Alignment、Grounding、Reasoning 和 Hallucination，而不是只使用总体 VQA Accuracy 衡量模型能力。

English Interview Answer

The main challenge in multimodal models is aligning and integrating information from modalities with fundamentally different representations. Images contain continuous spatial information, while language consists of discrete tokens with semantic and syntactic structure.

I would divide the problem into three components. First, representation alignment establishes semantic correspondence between visual and textual features, often through contrastive learning or trainable projections. Second, cross-modal fusion allows the language model to incorporate visual evidence through mechanisms such as cross-attention or visual token injection. Third, fine-grained grounding ensures that the model can connect language to specific objects, regions, attributes, and relationships in an image.

Additional challenges include noisy image-text pairs, computationally expensive visual tokens, and visual hallucinations.

I would evaluate these components separately using retrieval, grounding, visual question answering, and hallucination benchmarks rather than relying on a single overall accuracy metric.

Follow-up：Does alignment automatically guarantee grounding?

中文：不保证。CLIP 式对比学习主要对齐整体图像和文本表示，模型可能理解图像总体语义，却无法准确定位具体对象。细粒度 Grounding 通常需要区域级监督、空间表示和专门的评估。

English:

No. Global alignment establishes semantic similarity between images and text, but it does not guarantee accurate object localization or spatial reasoning. Fine-grained grounding often requires region-level supervision and dedicated evaluation.

Q2. 请解释 CLIP 的工作原理。它如何通过对比学习连接图像和文本？

中文面试回答

CLIP（Contrastive Language--Image Pre-training）采用 Dual-Encoder 架构，包含独立的 Image Encoder 和 Text Encoder，分别将图像和文本映射到共享的 Embedding Space。它的核心训练方法是 Contrastive Learning：给定一个 Batch 中的 \(N\) 对图像与文本，模型计算所有图像和文本之间的相似度，形成一个 \(N\times N\) 的 Similarity Matrix。正确匹配的图文对位于对角线，其他组合通常作为负样本。通过双向的 Contrastive Cross-Entropy Loss，模型学习提高匹配图文的相似度，同时降低不匹配图文的相似度。CLIP 的优势是可以利用大量自然语言描述作为监督信号，而不需要为每个分类任务重新标注固定类别，并且可以通过比较图像与文本类别描述实现 Zero-shot Classification。但 CLIP 的原始结构主要学习全局表示，并不直接生成长文本，因此在精细的空间推理、计数和区域级 Grounding 方面可能存在局限。

arxiv.org

+1

English Interview Answer

CLIP is a dual-encoder vision-language model trained with contrastive learning.

It uses an image encoder and a text encoder to map images and text descriptions into a shared embedding space. During training, a batch of N matching image-text pairs produces an N-by-N similarity matrix.

The positive pairs lie along the diagonal, while the remaining combinations are treated as negatives. CLIP optimizes a symmetric contrastive loss that increases the similarity of matching pairs and decreases the similarity of mismatched pairs.

One major advantage is zero-shot transfer. Instead of training a new classifier for each task, we can encode candidate class descriptions and compare them with an image embedding.

However, CLIP primarily learns global image-text alignment, so it is not inherently a generative dialogue model or a precise object-grounding model.

Follow-up：What are false negatives in CLIP?

中文：Batch 内非配对图文通常被当作负样本，但其中可能存在语义上也正确的描述。例如两张不同的猫照片都可以与"一只猫坐在沙发上"匹配。这种 False Negative 可能干扰表示学习，因此数据去重、多正样本设计和负样本策略很重要。

English:

False negatives occur when image-text pairs treated as mismatches are actually semantically compatible. This can introduce misleading gradients during contrastive training. Better data curation and multi-positive contrastive objectives can help address this problem.

Q7. 请比较不同 VLM 架构范式：Dual-Encoder、Token Fusion、Cross-Attention 和共享编码器

中文面试回答

VLM 的架构可以根据跨模态信息交互方式划分为几类。第一类是 Dual-Encoder，以 CLIP 为代表，图像和文本分别编码，并通过共享 Embedding Space 进行语义比较，优点是可以独立预计算 Embeddings，特别适合跨模态检索和 Zero-shot Classification，但缺少细粒度的动态交互。第二类是 Visual Token Fusion，以 LLaVA 为代表，Vision Encoder 提取特征后通过 Projector 转换成 Visual Tokens，再与文本 Token 一起送入 LLM，架构简单且方便复用预训练模型，但高分辨率视觉 Token 可能增加 Attention 和上下文开销。第三类是 Cross-Attention Fusion，例如 Flamingo，将视觉特征作为 Key 和 Value，让语言隐藏状态通过 Cross-Attention 有选择地读取视觉信息，能够实现更显式的跨模态交互，但会增加专门的融合层和实现复杂度。第四类是 Unified Multimodal Transformer，让多种模态在共享的 Transformer 中联合处理，具有更紧密的表示交互潜力，但需要更复杂的 Tokenization、数据平衡和训练设计。不存在对所有任务都最优的架构：检索重视 Embedding 效率，开放式 VQA 重视跨模态推理，而实时场景还要考虑视觉 Token 和延迟成本。BLIP-2 和 Flamingo 为不同的高效连接与跨模态融合方式提供了代表性的研究案例。

proceedings.mlr.press

+1

English Interview Answer

I would compare VLM architectures based on how they represent and exchange information across modalities.

Dual-encoder models, such as CLIP, encode images and text separately into a shared embedding space. They are efficient for retrieval and zero-shot classification because embeddings can be computed independently.

Token-fusion models, such as LLaVA, project visual features into the language model's embedding space and process visual and textual tokens together. This approach is simple and effective for generative multimodal tasks.

Cross-attention architectures, such as Flamingo, allow language representations to attend explicitly to visual features. They provide flexible cross-modal interaction but introduce additional fusion components.

Unified multimodal Transformers process multiple modalities through a shared architecture, potentially enabling deeper integration but requiring more complex training and tokenization.

I would choose among these architectures based on the task, grounding requirements, latency constraints, and available training resources.

架构比较

架构	代表	优势	主要限制
Dual-Encoder	CLIP	检索高效、Embedding 可预计算	交互弱，通常不直接生成回答
Token Fusion	LLaVA	结构简单，复用 LLM	视觉 Token 占用上下文
Cross-Attention	Flamingo	显式融合不同模态	新增 Fusion Layers
Query-based Connector	BLIP-2	可压缩视觉信息、冻结大模型	Query Bottleneck 可能丢失细节
Unified Transformer	单流多模态架构	深度联合建模	训练与资源成本较高

Follow-up：When would you prefer CLIP over LLaVA?

中文：如果任务是大规模图像文本检索、相似度排序或 Zero-shot Classification，我会优先考虑 CLIP，因为图文 Embeddings 可以单独计算和建立向量索引。如果任务需要根据图片进行多轮对话、生成解释或执行复杂视觉指令，LLaVA 类生成式 VLM 更合适。

English:

I would prefer CLIP for large-scale image-text retrieval or zero-shot classification because image and text embeddings can be computed independently and indexed efficiently. I would prefer a generative VLM such as LLaVA when the task requires conversational responses, detailed explanations, or multimodal reasoning.

Q4. 什么是 Visual Instruction Tuning？为什么它如此重要？

中文面试回答

Visual Instruction Tuning（视觉指令微调）是在图像或其他视觉输入与自然语言指令共同构成的数据上，对多模态模型进行指令遵循训练，使模型不仅能够识别图像内容，还能够根据用户的问题选择合适的信息、组织回答并进行多轮对话。它和最初的 Vision-Language Alignment 不同：Alignment 主要解决视觉表示能否被语言模型有效使用，而 Instruction Tuning 进一步学习"面对不同视觉任务应该如何回答"。例如，同一张图片可能对应"描述场景""数出人物""比较两个对象""解释图表"等不同指令，模型必须根据任务需求动态关注不同信息。训练时通常采用 Conditional Next-Token Prediction，并只对目标回答 Token 计算 Loss。视觉指令微调对于提升 VQA、图像对话和复杂指令遵循非常重要，但它无法自动修复 Vision Encoder 丢失的细节，也无法保证消除幻觉。因此需要同时控制数据质量、Grounding 和评估方案。原始 LLaVA 是视觉指令微调的重要代表工作。

proceedings.neurips.cc

English Interview Answer

Visual instruction tuning trains a vision-language model to follow natural-language instructions conditioned on visual inputs.

It goes beyond basic image-text alignment. Alignment teaches the model how to represent visual information in a form that the language model can use, while instruction tuning teaches the model how to respond to different visual tasks.

For example, the same image may support image description, object counting, spatial reasoning, or visual question answering, depending on the instruction.

Training usually uses an autoregressive language-modeling objective conditioned on both the visual representation and the instruction, with the loss applied to the target response tokens.

This stage is critical for conversational usability, but it does not automatically guarantee factual grounding. I would also evaluate visual accuracy, hallucination, and robustness to different instruction formulations.

核心公式

\ \\boxed{ \\mathcal L_{\\mathrm{VIT}} = -\\sum_{t\\in\\mathrm{answer}} \\log p_\\theta( y_t\\mid I,q,y_{\

其中：

• \(I\)：Image。

• \(q\)：Text Instruction。

• \(y_t\)：目标回答 Token。

Follow-up：What happens if visual instruction data contains hallucinated descriptions?

中文：模型可能学会在缺乏视觉证据时生成听起来合理但并不真实的描述。可以通过人工审核、自动一致性检查、困难负样本、拒答和不确定性标注降低风险。

English:

Noisy or hallucinated instruction data can teach the model to produce plausible but visually unsupported answers. I would improve data quality through validation, introduce hard negative examples, and evaluate whether the model appropriately expresses uncertainty when visual evidence is insufficient.

Q5. 视频 VLM 相比静态图片，需要额外解决什么问题？

中文面试回答

视频 VLM 相比图像 VLM，最大的新增挑战是 Temporal Modeling，即模型不仅需要理解每一帧中有哪些对象，还需要理解对象和事件如何随时间变化。静态图像主要包含空间信息，而视频同时包含空间、时间和可能的音频信息。模型需要处理运动方向、动作持续时间、事件发生顺序、跨帧对象一致性和因果先后关系。例如，"一个人拿起杯子再放下"和"先放下再拿起杯子"可能包含相似的静态画面，却具有不同的时序含义。常见方法包括 Frame Sampling、Temporal Positional Encoding、Spatiotemporal Attention、Temporal Transformer、Video Token Compression 以及分层时序建模。主要工程难题是视觉 Token 数量随帧数增长，导致 Attention 计算、KV Cache 和推理延迟显著增加，同时稀疏采样可能错过短暂的重要事件。因此我会同时评估 Temporal Order Understanding、Action Recognition、Event Localization、跨帧 Grounding 以及不同 Frame Sampling Rate 下的性能。视频语言模型的研究也强调统一图像与视频表示以及长短期时序依赖的建模。

ACL Anthology

+1

English Interview Answer

Compared with static images, video introduces an additional temporal dimension.

A video-language model must understand not only the objects and relationships within individual frames, but also how objects move, how events evolve, and in what order actions occur.

Common approaches include frame sampling , temporal positional embeddings, spatiotemporal attention, and temporal Transformers. More efficient systems may compress video tokens or use hierarchical temporal representations.

The main challenge is balancing temporal coverage with computational cost. Processing more frames improves coverage but increases visual token count, memory requirements, and inference latency. Sampling too sparsely, on the other hand, may miss important short-duration events.

I would evaluate temporal ordering, action recognition, event localization, and c onsistency across frames, rather than treating video understanding as independent image captioning.

Follow-up：Why is sampling one frame per second sometimes insufficient?

中文：因为高速或短暂事件可能发生在两个采样点之间，模型可能错过关键动作。可以使用运动感知采样、关键帧检测或事件触发采样，将有限的计算资源分配给更有信息量的时间片段。

English:

A fixed sampling rate may miss short or fast events between sampled frames. Motion-aware sampling, keyframe selection, or event-driven sampling can improve temporal coverage without processing every frame.

Q6. 什么是 Grounding？如何评估 VLM 能否正确将文本对应到图像区域？

中文面试回答

Grounding 是指将语言中的实体、属性或关系与图像中的具体视觉证据建立对应关系。与整体 Image-Text Alignment 不同，Grounding 更强调细粒度的空间定位。例如输入"找到穿红色衣服、站在汽车旁边的人"，模型需要理解多个属性和关系，并正确输出该目标的 Bounding Box、Segmentation Mask 或其他空间表示。Grounding 可以通过视觉区域特征与文本表示之间的 Cross-Attention、Region Proposal、Detection Head 或专门的 Grounding Architecture 实现。评估时，我会根据任务输出选择指标：Bounding Box Grounding 常使用 IoU 和 Localization Accuracy，例如 IoU≥0.5；Segmentation Grounding 使用 Mask IoU；Phrase Grounding 还需要考虑多个实体、属性和关系的匹配准确率。常见 Benchmark 包括 RefCOCO、RefCOCO+ 和 RefCOCOg，但这些数据集也存在标注噪声和覆盖范围的限制，因此还需要针对复杂语言和小目标场景进行更严格的评估。

arxiv.org

English Interview Answer

Grounding refers to connecting linguistic expressions with specific visual evidence, such as objects, regions, attributes, or spatial relationships in an image.

Unlike global image-text alignment, grounding requires fine-grained correspondence between language and visual locations.

For example, given the instruction "Locate the person wearing a red shirt next to the car," the model must identify the correct object using both visual attributes and spatial relationships.

Depending on the architecture, the output may be a bounding box, segmentation mask, or another region representation.

I would evaluate bounding-box grounding using intersection over union and localization accuracy, and segmentation grounding using mask IoU. I would also test small objects, multiple similar objects, complex referring expressions, and robustness to language variations.

Follow-up：Does high grounding IoU guarantee correct reasoning?

中文：不保证。模型可能正确定位实体，但仍错误理解实体之间的关系、动作或原因。例如正确框出两个人，却错误判断谁先拿起物品。因此 Grounding 和 Visual Reasoning 应当独立评估。

English:

No. High localization accuracy does not guarantee correct reasoning. A model may identify the relevant objects but still misunderstand their relationships, actions, or temporal order. I would evaluate localization and reasoning separately.

Q7. 请比较不同 VLM 架构范式：Dual-Encoder、Token Fusion、Cross-Attention 和共享编码器

中文面试回答

VLM 的架构可以根据跨模态信息交互方式划分为几类。第一类是 Dual-Encoder，以 CLIP 为代表，图像和文本分别编码，并通过共享 Embedding Space 进行语义比较，优点是可以独立预计算 Embeddings，特别适合跨模态检索和 Zero-shot Classification，但缺少细粒度的动态交互。第二类是 Visual Token Fusion，以 LLaVA 为代表，Vision Encoder 提取特征后通过 Projector 转换成 Visual Tokens，再与文本 Token 一起送入 LLM，架构简单且方便复用预训练模型，但高分辨率视觉 Token 可能增加 Attention 和上下文开销。第三类是 Cross-Attention Fusion，例如 Flamingo，将视觉特征作为 Key 和 Value，让语言隐藏状态通过 Cross-Attention 有选择地读取视觉信息，能够实现更显式的跨模态交互，但会增加专门的融合层和实现复杂度。第四类是 Unified Multimodal Transformer，让多种模态在共享的 Transformer 中联合处理，具有更紧密的表示交互潜力，但需要更复杂的 Tokenization、数据平衡和训练设计。不存在对所有任务都最优的架构：检索重视 Embedding 效率，开放式 VQA 重视跨模态推理，而实时场景还要考虑视觉 Token 和延迟成本。BLIP-2 和 Flamingo 为不同的高效连接与跨模态融合方式提供了代表性的研究案例。

proceedings.mlr.press

+1

English Interview Answer

I would compare VLM architectures based on how they represent and exchange information across modalities.

Dual-encoder models, such as CLIP, encode images and text separately into a shared embedding space. They are efficient for retrieval and zero-shot classification because embeddings can be computed independently.

Token-fusion models, such as LLaVA, project visual features into the language model's embedding space and process visual and textual tokens together. This approach is simple and effective for generative multimodal tasks.

Cross-attention architectures, such as Flamingo, allow language representations to attend explicitly to visual features. They provide flexible cross-modal interaction but introduce additional fusion components.

Unified multimodal Transformers process multiple modalities through a shared architecture, potentially enabling deeper integration but requiring more complex training and tokenization.

I would choose among these architectures based on the task, grounding requirements, latency constraints, and available training resources.

架构比较

架构	代表	优势	主要限制
Dual-Encoder	CLIP	检索高效、Embedding 可预计算	交互弱，通常不直接生成回答
Token Fusion	LLaVA	结构简单，复用 LLM	视觉 Token 占用上下文
Cross-Attention	Flamingo	显式融合不同模态	新增 Fusion Layers
Query-based Connector	BLIP-2	可压缩视觉信息、冻结大模型	Query Bottleneck 可能丢失细节
Unified Transformer	单流多模态架构	深度联合建模	训练与资源成本较高

Follow-up：When would you prefer CLIP over LLaVA?

中文：如果任务是大规模图像文本检索、相似度排序或 Zero-shot Classification，我会优先考虑 CLIP，因为图文 Embeddings 可以单独计算和建立向量索引。如果任务需要根据图片进行多轮对话、生成解释或执行复杂视觉指令，LLaVA 类生成式 VLM 更合适。

English:

I would prefer CLIP for large-scale image-text retrieval or zero-shot classification because image and text embeddings can be computed independently and indexed efficiently. I would prefer a generative VLM such as LLaVA when the task requires conversational responses, detailed explanations, or multimodal reasoning.

Q8. VLM 如何处理高分辨率图像？有哪些计算和架构挑战？

中文面试回答

高分辨率图像对 VLM 的主要挑战是视觉细节保留与计算成本之间的权衡。对于基于 Vision Transformer 的编码器，图像通常被划分为固定大小的 Patch，因此图像分辨率越高，Visual Token 数量越多。例如 Patch Size 为 16 时，将图像从 \(224\times224\) 提高到 \(448\times448\)，Visual Token 数量会从 196 增加到 784，也就是增长四倍，而 Full Self-Attention 的二次计算项理论上会增长约 16 倍。简单缩小图片虽然能降低成本，但可能丢失 OCR、小目标或精细图表中的重要信息。常见解决方案包括 Dynamic Resolution、Image Tiling、Multi-scale Features、Region Cropping 和 Visual Token Compression。一种实用策略是同时保留全局低分辨率视图与局部高分辨率 Crop，使模型既理解整体上下文，又能够读取局部细节。工程上需要评估 Token Budget、Latency、Memory 和细粒度视觉性能，特别注意视觉压缩对小对象识别与文字读取造成的损失。

English Interview Answer

The main challenge of high-resolution vision-language modeling is balancing fine-grained visual information against computational cost.

In a Vision Transformer, images are typically divided into patches. Increasing image resolution significantly increases the number of visual tokens, which affects attention computation, memory usage, and multimodal context length.

Simply resizing an image to a smaller resolution can reduce cost, but it may remove important details such as small objects, text, or numerical information in charts.

Common approaches include dynamic resolution, multi-scale representations, image tiling, region cropping, and visual token compression.

I would often consider a combination of global low-resolution context and high-resolution local crops, followed by efficient token aggregation.

The final design should be evaluated on small-object recognition, OCR, grounding accuracy, latency, and memory consumption.

Follow-up：How do you compress visual tokens without losing important information?

中文：我会使用可训练的 Resampler、Q-Former、Token Pruning 或基于注意力的重要性选择方法，并采用任务敏感的评估。不能只看 Token Compression Ratio，还需要比较 OCR、小目标识别、Spatial Grounding 和 Numerical Reasoning 的性能下降，因为这些细节最容易在压缩过程中丢失。

English:

I would consider learned resamplers, query-based compression, and token selection methods. However, compression should be evaluated against task quality, especially OCR, small-object recognition, and spatial grounding. A high compression ratio is not useful if it removes critical visual evidence.

Q9. VLM Hallucination 与纯文本 LLM Hallucination 有什么不同？

中文面试回答

VLM 的 Hallucination 与文本 LLM 的共同点是模型生成了缺乏充分证据支持的内容，但 VLM 额外涉及视觉证据与生成文本之间的不一致。纯文本 LLM 可能虚构事实、来源或推理依据；VLM 除了这些问题，还可能产生 Object Hallucination、Attribute Hallucination、Relationship Hallucination 和 Temporal Hallucination。例如图片中只有两个人，模型却描述有三个人；图片中的汽车是蓝色，模型却说是红色；视频中事件 A 发生在 B 之前，模型却颠倒顺序。这类错误可能来源于语言先验压过视觉证据、训练数据中的对象共现偏差、视觉特征丢失或跨模态对齐不足。评估时可以使用 POPE 测试 Object Hallucination，结合 CHAIR、Grounding Accuracy 和人工证据核验评价更广泛的视觉错误。缓解方法包括更高质量的 Grounded Training Data、区域级证据验证、Hard Negative Examples、合理的不确定性表达以及多模态一致性检查。但语言模型生成的自信程度不能直接视为视觉正确性的证明。POPE 的研究表明，常见对象共现和视觉指令分布都可能影响模型的对象幻觉行为。

ACL Anthology

English Interview Answer

Hallucination in vision-language models refers to generating content that is not supported by the visual input.

While text-only LLMs may invent facts or explanations, VLMs introduce additional failure modes such as object hallucination, incorrect attributes, spatial relationship errors, and temporal inconsistencies.

For example, a model may describe a red car when the image contains a blue car, or report objects that are not present at all.

These errors can arise when language priors dominate visual evidence, when important visual information is lost during encoding, or when cross-modal alignment is insufficient.

I would evaluate hallucination using object-level benchmarks such as POPE, grounding metrics, and evidence-based checks. To improve reliability, I would combine higher-quality visual instruction data, hard negative examples, uncertainty handling, and independent visual verification.

Q10. 除了 Image Captioning 和 VQA，VLM 还有哪些重要应用？

中文面试回答

除了图片描述和视觉问答，我认为 VLM 的重要应用可以分为视觉信息提取、空间理解、视频推理、具身智能和专业领域多模态分析。首先，在 Document Intelligence 中，VLM 可以结合 OCR、Layout Understanding 和语言推理，从表格、合同和财务报告中提取结构化信息，并验证不同来源的数据一致性。其次，在 Visual Grounding 和 Robotic Perception 中，模型可以根据语言指令定位物体，为机器人或交互系统提供视觉理解能力。第三，在视频领域，VLM 可以用于动作识别、事件检测、视频摘要和时序问题回答。第四，在医疗影像、遥感和工业质量检测中，视觉语言模型可以帮助分析复杂视觉信息，但高风险场景必须配合专业验证与人工监督。对于金融行业，我认为特别有价值的是 Financial Document Understanding，例如将财报中的图表、表格和文本联合建模，执行图表问答、数值一致性检查和证据追踪。这类应用的重点不是生成流畅的文字，而是提高多模态 Grounding、Numerical Reliability 和 Auditability。

English Interview Answer

Beyond image captioning and visual question answering, I see several promising applications for vision-language models.

One important area is document intelligence , where VLMs can combine OCR , layout understanding, and language reasoning to extract structured information from complex documents.

Another area is visual grounding and embodied intelligence, where language instructions must be linked to objects and spatial relationships.

Video understanding is also important for event detection, temporal reasoning, and summarization.

In specialized domains, VLMs can support medical imaging , remote sensing, and industrial inspection, although high-stakes applications require additional validation.

For financial services, I find multimodal financial document analysis especially interesting. A VLM could jointly interpret tables, charts, andtext in financial reports while supporting evidence tracing and numerical consistency checks.

In these applications, I would prioritize grounding accuracy, reliability, and auditability over fluent text generation alone.

omini

还需要先区分 Speech 和 Audio：Speech 主要指人类语音，包含语言内容、说话方式和说话人信息；Audio 的范围更广，还包括音乐、环境音、机械声和其他非语言声音。

还需要先区分 Speech 和 Audio：Speech 主要指人类语音，包含语言内容、说话方式和说话人信息；Audio 的范围更广，还包括音乐、环境音、机械声和其他非语言声音。

Q1. How would you extend a VLM to understand speech and audio?

中文面试回答

我会首先区分模型需要处理的是语音转写、非语言声音理解，还是端到端语音交互。对于现有 VLM，最直接的方法是增加预训练 Audio Encoder，将声音波形或 Log-Mel Spectrogram 转换为 Audio Embeddings，再通过 Projector 或 Cross-Attention 与 Visual Tokens、Text Tokens 对齐。如果任务是视频语音理解，还需要使用时间戳和 Temporal Positional Encoding 建立音视频同步。训练上，我会先冻结主要预训练模块进行 Alignment Training，再使用包含音频、视觉和文本的指令数据进行 Multimodal Instruction Tuning。最后分别评估 ASR、Audio Understanding、Visual Grounding、Audio-Visual Reasoning 和 Hallucination，以确定新增音频模态是否真正提供了互补信息。

English Interview Answer

I would first distinguish between speech recognition, general audio understanding, and real-time spoken interaction, because they require different capabilities.

To extend an existing VLM, I would introduce a pretrained audio encoder that converts waveforms or spectrograms into audio representations. I would then use a projection module or cross-attention to align these representations with visual and textual features.

For video applications, temporal synchronization is essential because audio segments and video frames must refer to the same events.

I would start with frozen pretrained encoders and train the alignment modules, followed by multimodal instruction tuning.

Finally, I would evaluate speech recognition, audio understanding, visual grounding, audio-visual reasoning, and hallucination separately to verify whether the additional modality provides meaningful value.

Q2. Why not simply convert all speech to text before feeding it into a VLM?

中文面试回答

将语音先转换成文本是一种非常合理的工程 Baseline，因为它模块化、便于调试，并且能够复用现有 LLM。但它的主要局限是 ASR 会丢失无法用纯文本完整表达的信息，例如语调、说话人差异、背景声音和部分非语言表达。对于只需要识别用户所说内容的系统，ASR Pipeline 可能已经足够；但对于需要结合视频判断谁在说话、识别环境事件或进行实时打断恢复的任务，保留原始音频表示可能更有效。因此我会根据任务比较 Cascaded Pipeline 与 Native Audio Model，而不会默认复杂模型一定更好。

English Interview Answer

Converting speech to text first is a strong engineering baseline because it is modular, easy to debug, and compatible with existing LLMs.

However, transcription can discard information such as prosody, speaker identity cues, background sounds, and nonverbal vocalizations.

For tasks that only require understanding spoken words, an ASR-based pipeline may be sufficient. For audio-visual grounding, real-time dialogue, or understanding nonverbal events, a model that directly processes audio representations can preserve useful information.

I would compare the two approaches using task accuracy, latency, robustness, and system complexity.

Q3. What is the difference between audio-text alignment and audio-visual temporal alignment?

中文面试回答

Audio-Text Alignment 主要解决声音与语义描述之间是否匹配的问题，例如把狗叫声与文本"a dog barking"映射到相近的表示空间。Audio-Visual Temporal Alignment 则需要进一步确定声音和图像中的事件是否发生在相同时间。例如视频中两个人轮流讲话，系统需要把各个声音片段与正确的说话人和视频帧对应。因此前者通常强调语义相似度，可以通过对比学习实现；后者还要求时间戳、时序表示和跨模态事件对应关系，可能需要 Temporal Attention、同步监督和时间偏移鲁棒性测试。

English Interview Answer

Audio-text alignment focuses on semantic correspondence between sounds and their language descriptions. For example, a recording of a dog barking should be close to the textual description of that sound in a shared embedding space.

Audio-visual temporal alignment additionally requires synchronizing events across time. For example, a model must associate the correct speech segment with the person speaking in the corresponding video frames.

Contrastive learning can support semantic alignment, while temporal alignment often requires timestamps, temporal representations, and synchronization-aware training.

I would evaluate both semantic correctness and sensitivity to temporal misalignment.

Q4. How would you evaluate an Omni-Modal Model?

中文面试回答

我会建立分层评估体系，而不是只使用一个综合 Benchmark。首先测量单模态能力，包括 ASR 的 Word Error Rate、视觉问答准确率和非语言音频识别性能。其次评估跨模态推理，包括 Audio-Visual Question Answering、Speaker Grounding、Temporal Event Alignment 和 Cross-modal Consistency。对于生成式语音系统，还需要测量语音自然度、内容正确率、实时延迟、打断恢复和 Turn-taking。最后开展模态消融实验：分别移除 Audio、Video 或 Text，观察任务性能变化，判断系统是否真正利用了互补信息，而不是只依赖语言先验。对于生产部署，还要关注隐私、稳健性和计算成本。

English Interview Answer

I would evaluate an omni-modal model at multiple levels.

First, I would measure single-modality performance, including speech recognition, visual question answering, and general audio understanding.

Second, I would test cross-modal tasks such as audio-visual question answering, speaker grounding, temporal synchronization, and cross-modal consistency.

Third, for spoken interaction, I would measure response quality, speech naturalness, first-response latency, interruption handling, and turn-taking accuracy.

I would also perform modality ablation experiments by removing audio or video inputs to determine whether the model actually benefits from multimodal information.

My objective would be to measure not only overall task success, but also reliability, efficiency, and the marginal contribution of each modality.

多模态 Agent 不仅需要处理更多输入，还要决定什么时候值得使用某一种模态，以及新增模态究竟提供了多少可验证的信息价值。

例如实时会议 Agent：

• Text 提供转写内容。

• Audio 提供语调、说话状态和背景声。

• Video 提供画面、人物活动和场景证据。

• Agent State 维护当前任务与历史。

• Verifier 检查结论是否有对应的多模态证据。

Generative Model

第一部分是生成模型基础： Diffusion、VAE、GAN、Autoregressive Models、概率建模、训练目标与采样方法。

将 Portfolio Analysis 与 Generative AI 结合，帮助顾问将复杂投资组合数据转化为可理解的分析与沟通内容。

可能考查： Conditional Generation、数值准确性、Grounding、Hallucination、模型选择。

根据客户金融知识水平、投资阶段和沟通语气生成可定制的市场文章摘要，并提供所需披露信息。

Financial Deep Research Multi-Agent、SLM、Causal AI、Policy-aware Conversational AI 等。

可能考查： Generative Agents、Research Design、Causal Validation、Evaluation。

可能考查： Controllable Generation、RAG、Faithfulness、Policy-constrained Generation。

自动生成对抗测试问题与安全回答，用于金融 AI 安全评估和模型适配；公开案例中生成了 6,360 条金融建议相关合成问题。

可能考查： Synthetic Data、Data Quality、SFT、Safety Evaluation、Distribution Shift。

Q1. What is a Diffusion Model, and how does it generate new data?

中文面试回答

Diffusion Model 是一种通过学习逆向去噪过程进行数据生成的概率模型。其核心思想是首先定义一个固定的 Forward Diffusion Process，逐步向真实数据加入高斯噪声，使数据分布最终接近一个简单的先验分布，例如标准正态分布。然后训练一个神经网络学习 Reverse Denoising Process，即如何从带噪样本恢复更干净的样本。在经典 DDPM 中，模型通常预测当前数据中加入的噪声，并通过均方误差进行训练。推理阶段则从随机噪声开始，逐步执行逆向去噪，最终生成新的数据样本。Diffusion 的优势是训练相对稳定、能够学习复杂且多模态的数据分布，并且可以结合条件输入生成符合指定要求的样本；局限是传统 Diffusion 需要多个采样步骤，因此生成速度可能较慢。在金融领域，可以将 Diffusion 用于合成收益率序列、市场风险场景或其他复杂金融数据的研究，但必须验证生成数据是否保留真实市场的重要统计特性。

English Interview Answer

A diffusion model is a probabilistic generative model that learns to generate data by reversing a noise corruption process.

During the forward diffusion process, Gaussian noise is gradually added to real data until the distribution approaches a simple prior, typically a standard normal distribution.

A neural network is then trained to approximate the reverse denoising process, often by predicting the noise added at a particular timestep.

During inference, the model starts from random noise and iteratively denoises it to generate a new sample.

Diffusion models are attractive because they can model complex data distributions and support conditional generation. However, iterative sampling can be computationally expensive.

In financial applications, I would consider diffusion for synthetic time-series generation or scenario analysis, but I would validate distributional properties, temporal dependencies, and tail-risk behavior before using generated data in downstream models.

Q2. Why can DDPM training be simplified to noise prediction with MSE?

中文面试回答

DDPM 的训练可以从 Variational Lower Bound（ELBO）推导。Forward Process 使用预先定义的高斯噪声转移，使带噪样本和真实数据之间的条件分布具有可解析形式。训练时需要学习 Reverse Process，使模型预测的反向分布接近真实的反向后验。在固定方差和适当的均值参数化条件下，ELBO 中与模型相关的部分可以转化为模型预测噪声与真实噪声之间的加权平方误差。DDPM 进一步采用简化的 Noise Prediction Objective，在实践中使用均方误差训练。这种形式的好处是可以直接从任意时间步采样带噪数据，而不必逐步执行前向扩散，训练计算效率较高。但需要注意，简化 MSE 并不等于完整 ELBO 在所有设置下都严格相同；如果采用不同的方差学习、Loss Weighting 或 Prediction Parameterization，目标函数也会改变。

English Interview Answer

The DDPM training objective originates from maximizing a variational lower bound on the data likelihood.

Because the forward noising process is Gaussian, several conditional distributions can be derived analytically.

The reverse model is trained to approximate the corresponding posterior transitions.

Under a fixed-variance parameterization, the relevant KL divergence terms reduce to weighted squared errors between the true noise and the model's predicted noise.

DDPM uses a simplified noise-prediction objective based on mean squared error, which is convenient and empirically effective.

An important distinction is that the simplified MSE objective is not identical to the full ELBO under every parameterization or weighting scheme.

Q3. What is the difference between DDPM and DDIM?

中文面试回答

DDPM 和 DDIM 的主要区别在于生成采样路径以及是否可以通过更少的时间步完成生成。DDPM 通常采用随机的 Reverse Diffusion Markov Chain，从噪声逐步采样到数据，每一步都包含对应的随机性。DDIM 则构造与 DDPM 具有相同训练边缘分布的非马尔可夫过程，并使用不同的采样更新公式，使模型可以跨越部分时间步进行生成。当 DDIM 的随机性参数设为零时，采样过程可以是确定性的。因此，DDIM 通常能够在显著减少采样步骤的情况下生成质量较好的样本，而不需要重新训练原始 Noise Prediction Model。但减少采样步数可能影响生成质量和多样性，实际收益需要根据模型、数据和评价指标验证。

English Interview Answer

DDPM and DDIM differ mainly in their reverse sampling processes.

Standard DDPM sampling follows a stochastic reverse Markov chain, progressively denoising the sample over many timesteps.

DDIM introduces a non-Markovian formulation that is compatible with the same training objective and supports accelerated sampling over fewer timesteps.

When the DDIM stochasticity parameter is set to zero, sampling becomes deterministic for a fixed initial noise sample.

The main advantage of DDIM is faster generation without necessarily retraining the model.

However, fewer sampling steps can introduce quality or diversity trade-offs, so I would compare the methods under equal generation budgets and downstream quality metrics.

Q4. Compare Diffusion Models, GANs, VAEs, and Autoregressive Models.

中文面试回答

这几类生成模型都试图学习数据分布，但其建模方法不同。Autoregressive Models 使用概率链式法则，将联合分布分解为一系列条件概率，典型 LLM 就通过 Next-Token Prediction 顺序生成文本，优势是训练目标清晰且适用于离散 Token，但顺序生成可能带来延迟。VAE 使用 Encoder 和 Decoder 建立潜变量生成模型，通过重构项和 KL Regularization 优化 ELBO，优势是潜在表示有明确的概率建模基础，但可能遇到 Posterior Collapse 等问题。GAN 使用 Generator 和 Discriminator 进行对抗训练，生成速度通常较快，但训练可能不稳定，并出现 Mode Collapse。Diffusion 则通过学习反向去噪过程生成数据，通常具有较好的模式覆盖和稳定的训练目标，但多步采样可能昂贵。对于金融应用，我不会预先假设 Diffusion 一定优于其他模型，而是根据生成数据的统计特性、极端事件覆盖、计算成本及下游任务表现选择方法。

English Interview Answer

These generative models learn data distributions through different mechanisms.

Autoregressive models factorize the joint distribution into conditional probabilities and generate outputs sequentially, making them especially suitable for discrete language modeling.

VAEs introduce latent variables and optimize a variational objective combining reconstruction and regularization.

GANs train a generator and discriminator adversarially, often enabling fast generation but introducing potential training instability and mode collapse.

Diffusion models learn a reverse denoising process and can capture complex distributions, although iterative sampling can be expensive.

For financial modeling, I would not select a method based solely on its popularity. I would compare distributional fidelity, tail behavior, temporal structure, diversity, and downstream utility using consistent baselines.

方法	训练目标	生成方式	主要 Trade-off
Autoregressive	Maximum Likelihood / Cross-Entropy	顺序生成	序列依赖
VAE	ELBO	Latent Sampling + Decoder	重构质量、潜变量使用
GAN	Adversarial Objective	Generator Forward Pass	Mode Collapse、训练稳定性
Diffusion	Denoising / Variational Objective	多步去噪	Sampling Cost

5. How would you use Diffusion Models to generate synthetic financial time series?

这是我认为最值得准备的 Diffusion 应用题。

中文面试回答

如果要使用 Diffusion Model 生成金融时间序列，我会首先明确目标是生成金融市场数据用于压力测试、模型训练，还是评估风险模型，而不是直接假设合成序列能够预测未来市场。输入可以是股票收益率、成交量、利率变化或多个资产的联合时间序列。相比直接生成原始价格，我通常会优先考虑经过适当变换的 Returns，因为价格序列可能具有非平稳趋势。架构上可以使用 1D U-Net 或 Temporal Transformer 作为 Denoising Network，并通过历史市场状态、波动率状态等条件变量实现 Conditional Generation。训练时在真实时间序列上加入不同程度的高斯噪声，使模型学习恢复数据的统计结构。评估阶段，我会重点检查合成数据是否保留真实市场的 Heavy Tails、Volatility Clustering、Autocorrelation 和 Cross-asset Correlations，并进一步使用 VaR、Expected Shortfall 和风险模型回测检验其下游价值。最重要的是严格使用时间切分，避免未来信息泄漏，并检验模型在 Regime Shift 和极端市场环境下是否仍然可靠。

English Interview Answer

I would first define the intended use of synthetic financial time series, such as stress testing, data augmentation, or risk-model evaluation.

I would generally consider modeling appropriately transformed returns rather than raw prices, because financial prices often exhibit nonstationary behavior.

For the architecture, I could use a temporal diffusion model with a one-dimensional U-Net or Transformer-based denoising network. The model could be conditioned on historical observations, volatility regimes, or other information available at the generation time.

During evaluation, I would focus on financial stylized facts, including heavy-tailed returns, volatility clustering, autocorrelation, and cross-asset dependence.

I would also evaluate downstream risk metrics such as Value at Risk and Expected Shortfall, using strict temporal splits to avoid look-ahead leakage.

Importantly, generating realistic financial scenarios is not the same as accurately forecasting future prices. I would validate the model against both statistical and task-specific baselines.

2024--2025 年已有研究将 DDPM 用于生成金融时间序列，并专门比较了这些 Stylized Facts。2026 年的金融 Diffusion 综述也覆盖 Time Series、Limit Order Books 和结构化金融数据。

Q6. How would you evaluate whether synthetic financial data is realistic?

中文面试回答

我不会只通过可视化比较真实和合成时间序列，因为看起来相似并不代表具有相同的风险分布。首先，我会评估 Marginal Distribution，包括均值、方差、分位数、偏度、峰度以及 Wasserstein Distance。其次，我会检查 Temporal Dynamics，例如 Return Autocorrelation、Squared-return Autocorrelation 和 Volatility Clustering。第三，对于多资产数据，需要检查 Correlation Matrix、Tail Dependence 和极端市场环境中的联合变化。第四，我会使用 Downstream Utility Evaluation，例如用合成数据增强风险模型，再在严格保留的真实数据上评估结果。最后，应该检查 Privacy Leakage 和 Memorization，以排除模型只是复制训练序列。只有当生成数据同时满足统计真实性、任务有效性和必要的隐私要求时，才具有实际价值。

English Interview Answer

I would evaluate synthetic financial data along several dimensions rather than relying on visual similarity alone.

First, I would compare marginal distributions using moments, quantiles, and distributional distances such as Wasserstein distance.

Second, I would evaluate temporal dynamics, including return autocorrelation, volatility clustering, and autocorrelation of squared returns.

Third, for multivariate data, I would measure cross-asset correlations and tail dependence.

Fourth, I would evaluate downstream utility by training or testing risk models using synthetic data and assessing performance on held-out real market data.

Finally, I would test for memorization and privacy leakage.

Realistic-looking data is not sufficient. The generated samples need to preserve the statistical and decision-relevant properties of the target application.

评估层次	Metrics	检测的问题
Marginal Distribution	Wasserstein、Quantiles、Kurtosis	收益率分布是否真实
Temporal Dynamics	ACF、Squared-return ACF	时间依赖、波动聚集
Cross-asset Structure	Correlation、Tail Dependence	是否保留资产依赖
Risk Metrics	VaR、Expected Shortfall	是否保留尾部风险
Downstream Utility	Held-out Task Performance	是否真正有用
Privacy	Nearest-neighbor / Memorization Tests	是否复制训练样本

这里要注意 Expected Shortfall 与 VaR 的区别：VaR 描述给定置信水平下的损失分位数，而 Expected Shortfall 关注超过该分位数后的平均损失。在风险场景生成中，后者通常能提供更多关于尾部严重程度的信息。

Q8. How would you prevent hallucinations in a financial Generative AI system?

中文面试回答

对于金融 Generative AI，我会采用多层防护，而不是只依赖 Prompt Engineering。首先，将需要事实支持的内容连接到经过权限控制的可信数据源，例如已验证的投资组合数据、产品资料和适用的政策文档。其次，涉及财务计算、利率、百分比变化和风险指标时，应使用确定性工具而不是让 LLM 自由推算。第三，在生成结果后执行独立验证，包括 Source Attribution、Numerical Consistency、Policy Compliance 和 Uncertainty Handling。对于证据不充分或可能产生重大影响的情况，系统应该请求更多信息或进行人工升级。最后，必须建立包含过时信息、矛盾证据、错误数字、虚构来源和越权指令的 Regression Benchmark，并在生产环境中持续监控。这样才能将 Financial GenAI 从"生成合理文本"提升为"生成可验证的金融内容"。

English Interview Answer

For financial Generative AI, I would use multiple layers of validation rather than relying on prompting alone.

First, I would ground factual claims in trusted, access-controlled data sources.

Second, I would delegate numerical calculations to deterministic financial tools instead of relying on free-form LLM generation.

Third, I would independently verify source attribution, numerical consistency, policy compliance, and whether the available evidence is sufficient.

For high-risk or uncertain cases, I would introduce human escalation.

Finally, I would maintain regression benchmarks covering outdated information, conflicting evidence, fabricated citations, and numerical errors.

My goal would be to make financial AI outputs independently verifiable, auditable, and reliable rather than merely fluent.

v

Q9. How would you generate synthetic training data for financial AI?

中文面试回答

我会将合成数据生成设计为一个可验证的 Data Generation Pipeline，而不是单纯要求 LLM 生成大量问答。首先定义真实任务与风险分类，例如金融产品信息问答、客户沟通合规、复杂意图识别或对抗性输入；然后利用 LLM 生成具有不同表达方式、难度和边界条件的候选样本。接下来使用独立的 Policy Validator、规则检查和必要的人工审核验证标签与期望回答，去除重复样本，并控制生成数据与真实业务数据之间的分布差异。训练之后必须在独立的真实测试集上评估，而不能只报告合成测试集的 Accuracy。特别是在 Financial Safety 场景中，我会关注违反政策的比例、错误拒绝率以及是否对新的攻击方式具有泛化能力。

English Interview Answer

I would treat synthetic data generation as a controlled and validated data pipeline.

First, I would define the target tasks, risk categories, and coverage requirements. Then I would use an LLM to generate diverse candidate examples, including difficult and adversarial cases.

I would validate the generated samples using independent policy checks, quality filters, and human review when appropriate.

After deduplication and distribution analysis, I would use the validated dataset for post-training or evaluation.

Most importantly, I would measure performance on independent real-world test data to determine whether synthetic training improves actual generalization rather than just performance on similar generated examples.

Q10. How would you design a Generative AI Research Agent for financial documents?

中文面试回答

我会采用 Evidence-grounded Agent Architecture。首先对用户请求进行意图识别和权限检查，然后通过 Hybrid Retrieval 获取相关的财报、研究材料或政策文档，并保留文档版本、发布日期和引用来源。Agent Orchestrator 根据任务决定是否需要执行多文档检索、数值计算或进一步的研究步骤。涉及财务指标时使用结构化工具进行计算；生成最终报告前，由独立验证模块检查事实依据、引用准确性和数值一致性。如果缺乏证据，模型应该明确说明不确定性，而不是生成确定性的结论。评估时，我会分开测量 Retrieval Recall、Citation Faithfulness、Numerical Accuracy、Task Success、Policy Violation 和每次成功任务的成本。对于长程任务，还需要记录执行轨迹，以便定位错误是来自检索、推理、工具执行还是最终生成。

English Interview Answer

I would design a financial research agent around evidence-grounded generation and independent verification.

The system would first validate the user's access permissions and retrieve relevant financial documents using hybrid retrieval.

An orchestrator would then decide whether additional retrieval, numerical analysis, or specialized tools are required.

I would use deterministic tools for financial calculations and keep explicit references to the supporting documents.

Before generating the final report, a verification component would check factual grounding, citation accuracy, numerical consistency, and policy requirements.

I would evaluate the system at multiple levels, including retrieval recall, task success, numerical accuracy, verification failures, and cost per successful task.

Q11. What is Conditional Diffusion, and how would you control generated market scenarios?

中文面试回答

Conditional Diffusion 是在扩散模型的逆向去噪过程中加入额外条件，使生成数据满足指定的上下文或场景要求。在金融时间序列中，条件变量可以包括历史波动率、收益率状态、资产类别或特定市场 Regime。模型不再只学习无条件分布 \(p_\theta(x)\)，而是学习 \(p_\theta(x\mid c)\)，其中 \(c\) 表示条件信息。训练时，将条件嵌入输入 Denoising Network；推理时，给定不同条件生成多条可能的样本路径。也可以研究 Classifier-Free Guidance 等方法增强条件控制，但过强的 Guidance 可能降低多样性或扭曲生成分布。对于金融应用，必须检查条件是否在预测时可获得，并验证生成结果确实符合指定条件，而不是只在视觉上相似。

English Interview Answer

Conditional diffusion extends the denoising process by incorporating contextual information into the generative model.

Instead of modeling only the unconditional data distribution, the model learns to generate samples conditioned on variables such as historical volatility or market regimes.

Conditioning information can be injected into the denoising network through embeddings, feature modulation, or attention mechanisms.

During inference, different conditions can be used to generate multiple plausible scenarios.

However, I would carefully evaluate conditional consistency, diversity, and distributional fidelity, because stronger conditioning does not necessarily produce more realistic financial data.

第二部分是金融 AI 应用： 生成金融研究内容、合成金融数据、市场风险场景、LLM Safety、金融 Agent 以及模型可靠性。

Q12. Can Diffusion Models reliably generate extreme financial market events?

中文面试回答

不能直接保证。金融极端事件通常属于数据分布的尾部区域，在训练集中出现频率很低。即使 Diffusion 能够很好地拟合普通市场状态，也可能低估极端损失、极端波动或资产之间的尾部相关性。因此，我会将普通市场数据生成与 Tail-risk Scenario Generation 分开评估，使用 Tail Quantiles、Expected Shortfall、Extreme Value Diagnostics 和 Regime-specific Backtesting 等方法检查生成质量。如果目标是压力测试，还可以结合条件生成、历史压力情景和传统金融随机过程 Baseline。但对非常罕见甚至训练数据中从未出现过的事件，不能声称生成模型已经学会了其真实概率分布。

English Interview Answer

Not necessarily. Extreme market events are rare and often poorly represented in the training distribution.

A diffusion model may reproduce normal market conditions while underestimating extreme losses, volatility spikes, or tail dependence across assets.

I would evaluate tail quantiles, Expected Shortfall, and regime-specific behavior separately from overall distributional similarity.

I would also compare the generated scenarios against historical stress events and established financial simulation baselines.

The goal would be to assess whether the model provides useful stress scenarios, without assuming that it can accurately estimate the probability of unprecedented events.

Q13. What are the privacy risks of synthetic financial data?

中文面试回答

合成数据不一定天然保护隐私，因为生成模型可能记住训练样本，尤其是训练数据中罕见且可识别的记录。即使生成结果不是原始数据的完全复制，也可能通过组合多个属性泄露敏感信息。因此，我会评估 Membership Inference Risk、Nearest-neighbor Similarity 和异常样本复现风险，并根据业务要求采取数据最小化、权限控制和隐私保护训练方法。对于敏感客户数据，还需要评估 Differential Privacy 是否适用，同时明确隐私保护强度与模型效用之间的 Trade-off。不能因为数据被称为 Synthetic，就自动认为它可以不受限制地共享。

English Interview Answer

Synthetic data does not automatically guarantee privacy.

Generative models can memorize rare or sensitive training examples, and generated records may reveal information about individuals even when they are not exact copies.

I would evaluate memorization, membership inference risks, and similarity to sensitive training records.

Depending on the requirements, I would consider data minimization, access controls, and privacy-preserving training methods such as differential privacy.

I would also assess the utility-privacy trade-off rather than assuming synthetic data is inherently safe to share.

Q14. How would you evaluate whether a financial diffusion model generalizes?

中文面试回答

我会采用严格的 Time-based Split 和 Walk-forward Evaluation，而不是随机划分相邻金融时间窗口，因为随机拆分可能让训练集和测试集共享高度相关甚至重叠的市场信息。训练、验证和测试应该覆盖不同时间段，并且对 Preprocessing、Normalization 和 Conditional Features 执行同样的时间隔离。评估还要包含不同 Market Regimes，例如高波动和低波动阶段，并比较生成样本的统计真实性、下游风险模型表现和分布变化鲁棒性。如果模型只在与训练数据相似的市场状态下表现良好，而在市场结构发生变化时退化，就说明其泛化存在局限。

English Interview Answer

I would use strict temporal splits and walk-forward evaluation rather than randomly splitting financial time-series windows.

Random splits can introduce information leakage because adjacent observations may be highly correlated or overlapping.

I would evaluate the model across different market regimes, including high-volatility and low-volatility periods.

I would compare distributional fidelity, downstream risk-model performance, and robustness to distribution shifts.

I would also ensure that normalization and conditional features use only information available at the relevant historical time.

Q15. Diffusion sampling is slow. How would you reduce inference latency?

中文面试回答

传统 Diffusion 的主要推理成本来自多步执行 Denoising Network，因此我会首先分析采样步骤数量、每步计算成本和实际生成质量之间的关系。最直接的方法是使用 DDIM 或其他高效 Sampler 减少采样步骤，并通过质量指标确定可以接受的最小步数。进一步可以考虑 Distillation，将多步去噪过程压缩为较少步骤，或者使用 Latent Diffusion 在更低维度空间中生成数据。对于金融时间序列，还可以采用较轻量的 1D Network 或 Temporal Transformer，并通过 Batch Inference 提高吞吐。必须注意，减少生成时间不能以丢失 Tail-risk 和 Cross-asset Dependence 为代价，因此需要建立 Quality-versus-Latency Curve，而不是只比较每条样本的生成时间。

English Interview Answer

Diffusion inference is expensive because it repeatedly evaluates the denoising network.

I would first profile the sampling process and evaluate the relationship between the number of denoising steps, latency, and output quality.

Practical options include accelerated samplers such as DDIM, improved timestep schedules, and distillation into models requiring fewer steps.

Latent diffusion and more efficient denoising architectures may also reduce computation.

For financial data, I would ensure that faster sampling preserves the properties that matter for the downstream task, especially tail behavior and cross-asset dependence.

I would select the approach using quality-versus-latency and cost-versus-utility comparisons.

Q16. How would you design fair baselines and ablation experiments for financial Diffusion?

中文面试回答

我会先建立简单且强的 Baseline，包括 Historical Bootstrap、合适的 GARCH 类模型，以及 GAN、VAE 或其他条件生成模型。不同模型应使用相同的训练时间范围、条件信息、测试集与主要任务评价指标，并报告模型容量、训练成本和采样成本。对于 Diffusion，我会通过 Ablation 分别移除 Conditional Features、Temporal Architecture Components 或特殊 Loss Weighting，判断真正带来改善的是哪一个模块。评估时不能只选择对 Diffusion 有利的视觉指标，而应包含真实金融任务所关注的 Distribution Distance、Volatility Clustering、Tail Risk 和 Out-of-sample Utility，同时报告多随机种子结果和不确定性范围。已有金融生成模型比较研究发现，传统 Historical Simulation、GARCH 和部分 GAN 方法仍可能表现很有竞争力，因此公平 Baseline 特别重要。

arxiv.org

English Interview Answer

I would establish strong baselines before evaluating a new financial diffusion model.

These should include historical resampling methods, suitable statistical models such as GARCH, and alternative generative architectures including GANs and VAEs.

I would keep training periods, conditioning information, evaluation datasets, and downstream tasks consistent across methods.

For ablation studies, I would remove conditioning variables, architectural components, or specialized loss terms to identify which elements drive the improvements.

I would report statistical fidelity, tail-risk behavior, computational cost, and downstream utility across multiple seeds.

A more complex generative model should only be preferred if it delivers measurable value over simpler, well-established baselines.

Case Study: Design a trustworthy Generative AI system for financial research and portfolio analysis.

面试官可能先问：

We want to build a system that summarizes financial research, explains portfolio risks, and generates alternative market scenarios. How would you design it, and how would you evaluate it?

建议拆成四个模块。

1. Knowledge & RAG

Financial documents、market data、policy documents

2. Financial Analytics

Validated portfolio calculations、risk statistics

3. Generative Models

LLM explanations、optional conditional scenario generation

4. Independent Verification

Evidence、numerical checks、policy gates、audit trail

Advisor-facing Analysis

Grounded · Numerically consistent · Auditable · Human oversight

这是适合讨论的候选系统设计，不是 Vanguard 内部架构披露。

综合题：中文面试回答

我会采用 Modular Generative AI Architecture，而不是让一个模型同时承担金融计算、市场情景生成和自然语言解释。系统首先通过权限控制和可信数据检索获取财报、投资组合与政策信息，再使用确定性金融工具计算风险指标。如果业务需要研究不同市场条件下的可能结果，可以增加经过验证的 Conditional Generative Model，例如 Diffusion，用于生成合成市场情景。但这些生成结果应该被视为情景研究工具，而不是对未来市场的确定性预测。LLM 负责将已验证的计算结果和证据组织成自然语言，并根据用户需求控制解释的详细程度。最终通过独立验证检查引用、数值、政策约束和不确定性，同时保留审计记录。在评估上，我会分别报告 Financial Accuracy、Faithfulness、Tail-risk Fidelity、Policy Compliance、Human Review Outcomes、Latency 和 Cost per Successful Task。这种架构能够更清楚地区分各模块的责任，也便于定位错误和开展消融实验。

English Interview Answer

I would design the system as a modular Generative AI architecture rather than assigning every responsibility to a single language model.

First, I would retrieve trusted financial information under appropriate access controls and use validated analytics tools for portfolio and risk calculations.

If the use case requires exploring possible market scenarios, I would consider a separately validated conditional generative model, such as diffusion, for synthetic scenario generation. These scenarios would support risk analysis rather than serve as guaranteed market forecasts.

The LLM would then generate user-friendly explanations grounded in the verified calculations and supporting evidence.

Before delivering the output, an independent verification layer would check numerical consistency, citation accuracy, policy compliance, and whether uncertainty is communicated appropriately.

I would evaluate the system using financial accuracy, grounding quality, tail-risk fidelity, policy violations, latency, and cost per successful task.

My main design principle is to separate probabilistic generation, deterministic financial computation, and independent verification so the system remains reliable and auditable.

Responsible AI 基础问题

Q1. What is Responsible AI, and why is it particularly important in financial services?

中文面试回答

Responsible AI 是指在 AI 系统从设计、训练、评估到部署和监控的整个生命周期中，确保其具有可靠性、公平性、隐私保护、透明性、安全性和可问责性。在金融行业，这些要求尤其重要，因为模型输出可能影响投资决策、资产配置、退休规划以及客户对金融风险的理解。对于生成式 AI，传统的准确率已经不足以衡量系统风险，因为即使生成文本流畅，也可能包含错误数字、缺失重要风险披露、使用过期信息或给出不适当的投资建议。因此我会采用 Risk-based Evaluation，根据不同业务场景定义不同的可接受风险水平。例如内部研究摘要与直接影响客户投资决策的系统，需要不同的验证和审批机制。我的核心原则是将模型能力评估与风险治理结合，使 AI 的输出具有可验证的证据、明确的责任边界和必要的人类监督。

English Interview Answer

Responsible AI means designing, developing, evaluating, and deploying AI systems in ways that support reliability, fairness, privacy, transparency, safety, and accountability.

It is particularly important in financial services because AI-generated information can influence investment decisions, retirement planning, and clients' understanding of financial risks.

For Generative AI, traditional accuracy metrics are not sufficient. A response may sound convincing while containing incorrect numbers, outdated information, missing disclosures, or inappropriate recommendations.

I would therefore adopt a risk-based approach, defining evaluation criteria and safeguards according to the potential impact of each application.

My goal would be to build AI systems whose outputs are evidence-grounded, independently verifiable, auditable, and subject to appropriate human oversight.

Q2. How do Explainability, Reliability, Fairness, and Auditability differ?

中文面试回答

这几个概念相关，但解决的问题不同。Explainability 关注模型为什么产生某个预测，例如通过 SHAP 分析传统风险模型中不同特征的贡献；Reliability 关注模型在真实数据、分布变化和困难场景下能否稳定地给出正确结果；Fairness 关注模型是否对不同群体产生不合理的差别影响；Auditability 则关注系统能否在事后重建其使用的数据、模型版本、工具调用、验证过程和最终决策。对于金融 LLM，我不会仅依靠 SHAP 或 Attention Visualization 解释整个决策过程，而会结合 Retrieval Provenance、Numerical Verification、Counterfactual Testing 和 Execution Trace。模型能解释预测，不代表预测正确；预测正确，也不代表整个决策过程符合权限或业务要求。

English Interview Answer

Explainability, reliability, fairness, and auditability address different dimensions of AI trustworthiness.

Explainability helps us understand why a model produced a particular prediction. Reliability measures whether the system performs consistently across realistic and challenging conditions. Fairness examines whether the system produces unjustified disparities across relevant populations. Auditability ensures that decisions and actions can be reconstructed and independently reviewed.

For example, SHAP may explain feature contributions in a financial risk model, but it cannot by itself establish that the prediction is correct or compliant.

For an LLM-based financial agent, I would combine evidence provenance, numerical verification, counterfactual testing, and structured execution traces.

Responsible AI requires these mechanisms to work together rather than treating interpretability as a substitute for validation.

一个适合面试记忆的框架

维度	回答的问题	常见方法
Explainability	Why did the model predict this?	SHAP、Attribution、Evidence Analysis
Reliability	Is the result dependable?	Robustness、Calibration、Regression Tests
Fairness	Are outcomes unjustifiably different?	Group Metrics、Subgroup Testing
Privacy	Is sensitive information protected?	Access Control、Privacy Testing
Auditability	Can we reconstruct the process?	Versioning、Logs、Provenance
Accountability	Who approves and owns the decision?	Governance、Human Review、Escalation

在金融 AI 应用中，Verification（验证）是最核心的要求之一。 尤其对于 LLM、Generative AI 和 Agentic AI，模型能够生成合理的答案并不代表答案正确，更不代表可以安全地用于金融决策。

但需要强调：Verification 不只是验证最终答案，而是验证整个 AI 决策和执行过程。

我建议从五个层次理解：

验证层次	核心问题	金融场景举例
1. Data Verification	输入数据是否准确、及时、可信？	财报版本、市场数据、客户投资组合
2. Model Verification	模型是否具备足够的预测或生成能力？	Accuracy、Calibration、Robustness
3. Reasoning Verification	推理结论是否有充分依据？	引用是否支持投资风险分析
4. Execution Verification	工具调用和计算是否正确、合规？	投资组合收益率、风险计算、Agent 权限
5. Outcome Verification	最终输出是否可靠且符合业务要求？	财务数字、风险披露、合规性

更重要的是：不同生成模型需要不同验证方法

LLM / Financial Research Agent

重点验证 Factual Grounding、Numerical Accuracy、Citation Faithfulness、Policy Compliance 和工具执行轨迹。

Diffusion / Financial Scenario Generation

重点验证 Distribution Fidelity、Tail Risk、Volatility Clustering、Cross-asset Dependence 和压力测试有效性。

Traditional Financial ML

重点验证 Out-of-sample Performance、Calibration、Fairness、Robustness 和 Data Drift。

因此，不能使用同一种 Accuracy Metric 衡量所有金融 AI 系统。

LM Fine-tuning、Autonomous Agents、Knowledge Graphs 面试准备

这三个领域可以构成一个完整的金融 AI 研究系统：

• LLM Fine-tuning Paradigms：如何调整模型能力、行为和领域适应性。

• Autonomous Agents：如何让模型规划任务、调用工具、执行操作并验证结果。

• Knowledge Graphs：如何用结构化实体和关系支持事实检索、多跳推理及证据追踪。

Q1. What are the major LLM fine-tuning paradigms?

中文面试回答

LLM Fine-tuning 可以从训练目标与参数更新范围两个维度分类。按照训练目标，主要包括 Continued Pretraining、Supervised Fine-tuning（SFT）、Preference Optimization 和 Reinforcement Learning。Continued Pretraining 通常使用领域文本继续进行语言模型预训练，以适配金融等专业语料；SFT 利用高质量的指令与答案对训练模型的任务行为；DPO 等 Preference Optimization 方法利用回答偏好数据调整模型行为；RLHF 或 GRPO 等强化学习方法则根据奖励信号优化策略。按照参数更新范围，可以分为 Full Fine-tuning 和 Parameter-efficient Fine-tuning（PEFT），后者包括 LoRA、QLoRA、Adapters 和 Prompt Tuning。它们不是互斥类别，例如可以使用 LoRA 进行 SFT 或 DPO。对于金融 AI，我会首先诊断问题是缺乏领域知识、无法遵循任务格式，还是安全行为不稳定，再选择相应的方法，而不是默认必须微调所有参数。

English Interview Answer

I classify LLM fine-tuning methods along two dimensions: the training objective and the number of parameters being updated.

In terms of objectives, continued pretraining adapts the model to a domain distribution, supervised fine-tuning teaches desired task behavior, preference optimization methods such as DPO improve response preferences, and reinforcement learning optimizes behavior using reward signals.

In terms of parameter updates, we can use full fine-tuning or parameter-efficient methods such as LoRA, QLoRA, adapters, and prompt tuning.

These dimensions are not mutually exclusive. For example, LoRA can be used for supervised fine-tuning or preference optimization.

For financial applications, I would first identify the actual failure mode, such as missing domain knowledge, poor instruction following, or compliance failures, before selecting the adaptation strategy.

方法总览

方法	主要目的	所需数据	关键限制
Continued Pretraining	适应领域语料分布	大量领域文本	成本、知识遗忘
SFT	学会任务格式与行为	Instruction--Response	数据覆盖与质量
DPO	优化回答偏好	Chosen / Rejected Pairs	偏好数据质量
RLHF / GRPO	优化可定义的奖励目标	Prompt、Responses、Rewards	Reward Hacking、训练稳定性
LoRA / QLoRA	减少训练成本	可与上述目标结合	低秩容量、量化误差

Q2. How do LoRA and QLoRA work? When would you choose them?

中文面试回答

LoRA 的核心思想是冻结预训练权重，将权重更新限制在两个低秩矩阵的乘积中。对于权重矩阵 \(W\)，不是直接训练完整的 \(\Delta W\)，而是令 \(\Delta W=BA\)，其中秩 \(r\) 远小于输入输出维度，从而显著减少可训练参数量和 Optimizer States。QLoRA 则进一步将冻结的基础模型权重量化到通常为 4-bit 的表示，同时通过 LoRA Adapters 学习参数更新，因此能够进一步降低基础模型权重的存储成本。两者适合计算资源有限、需要针对不同业务维护多个 Adapter 的场景。局限是低秩更新不一定具有与 Full Fine-tuning 相同的适应容量，并且 QLoRA 需要考虑量化误差以及训练激活显存。我会根据任务难度、数据规模、GPU Memory 和 Quality Regression 来决定使用 LoRA、QLoRA 还是 Full Fine-tuning。Hugging Face PEFT 的现有实现也支持这些训练方式。

Hugging Face

+1

English Interview Answer

LoRA reduces fine-tuning cost by freezing the pretrained weight matrix and representing its update as the product of two low-rank matrices.

Instead of updating all parameters, we train a much smaller set of adapter parameters, which significantly reduces optimizer memory and makes it easier to maintain multiple task-specific adaptations.

QLoRA extends this approach by storing the frozen base model in a quantized representation, typically four bits, while training LoRA adapters.

I would consider these approaches when GPU memory is limited or multiple domain-specific adaptations are needed.

However, I would still evaluate model quality, rank sensitivity, quantization effects, and whether the adaptation capacity is sufficient for the target task.

Q3. When should we use RAG instead of fine-tuning?

中文面试回答

RAG 和 Fine-tuning 解决的问题不同。RAG 主要解决模型在推理时如何访问外部知识，特别适合不断变化、需要追踪来源或具有严格访问权限的金融数据，例如最新产品政策、研究文档和客户投资组合信息。Fine-tuning 则主要调整模型的参数化行为和能力，例如特定格式的金融分析报告生成、稳定的任务流程、领域术语使用或安全行为。将最新财报事实直接通过 Fine-tuning 写入参数通常不如使用有版本控制的检索系统灵活，因为参数更新需要训练且难以精确撤销。对于 Vanguard 类金融研究系统，我会优先建立 RAG + Tools Baseline；只有当模型在获得正确证据后仍反复出现任务行为或推理错误时，才考虑使用 SFT 或其他 Post-training 方法。两者也可以结合，例如通过 Fine-tuning 改善模型使用检索证据的能力。

English Interview Answer

RAG and fine-tuning address different problems.

RAG provides access to external knowledge at inference time, making it particularly suitable for frequently updated financial information, source attribution, and access-controlled documents.

Fine-tuning changes model behavior or capabilities, such as instruction following, structured output generation, or specialized task performance.

I would not typically use fine-tuning as the primary mechanism for storing frequently changing financial facts, because parameter updates are difficult to maintain and audit.

For a financial research assistant, I would first establish a RAG and tool-based baseline. If the model still struggles to use the retrieved evidence or follow specialized workflows, I would consider fine-tuning.

In practice, the two approaches can be complementary.

Q4. How would you fine-tune an LLM for financial compliance?

中文面试回答

我会首先将金融合规要求转化为明确、可测试的行为规范，然后构建覆盖允许请求、受限请求、模糊请求和对抗性输入的数据集。模型训练方面，可以从 SFT 开始，使用经过政策审核的示例教授正确的回答方式；对于复杂边界案例，可以进一步使用 Preference Optimization，让模型在多个候选回答之间学习更符合政策的行为。同时必须保持独立的 Test Set，衡量 Policy Violation、False Refusal、Numerical Errors 和对未见过问题的泛化能力。由于金融政策可能随时间变化，不能仅依赖参数中记忆的规则，还应该在推理阶段提供当前有效的政策文本，并使用独立的 Policy Validator。Vanguard 已公开的研究正包含自动 Red Teaming 和 Post-training Dataset Generation。

University of Toronto MScAC

English Interview Answer

I would begin by translating financial compliance requirements into explicit, testable behavioral specifications.

I would build a dataset covering allowed, restricted, ambiguous, and adversarial requests.

Supervised fine-tuning would establish the desired response patterns, while preference optimization could help with difficult policy trade-offs.

I would then evaluate policy violations, unnecessary refusals, numerical errors, and generalization to unseen cases.

Because financial policies can change, I would avoid relying exclusively on the model's internal memory. I would provide version-controlled policy information at inference time and apply independent compliance checks.

My objective would be to improve model behavior while preserving auditability and the ability to update policies without retraining the entire model.

Part II. Autonomous Agents

Q5. What is an Autonomous AI Agent? How is it different from a traditional LLM application?

中文面试回答

Autonomous Agent 是一个能够根据目标和环境状态，动态决定下一步行动、调用工具、观察结果并持续调整执行过程的系统。普通 LLM 应用通常执行单次 Prompt-to-Response，而 Agent 可以形成多轮 Decision--Action--Observation Loop，例如先检索数据，再调用计算工具，随后验证结果，并根据反馈继续执行。Agent 的自主性并不意味着它可以不受约束地执行所有操作；在金融场景中，权限、政策约束和高风险操作审批必须由系统强制执行。Agent 的主要优势是能够处理需要多个步骤、工具和外部信息的复杂任务；主要局限则是执行成本较高、错误可能跨步骤传播，以及动态行为更难测试和复现。因此我会用任务成功率、工具执行正确率、错误恢复能力和单位成功任务成本评价 Agent，而不是只看最终回答是否流畅。

English Interview Answer

An autonomous AI agent is a system that can plan actions, interact with tools or external environments, observe results, and adapt its behavior toward a goal.

Unlike a traditional prompt-response application, an agent operates through multiple decision and execution steps.

For example, a financial research agent may retrieve documents, perform calculations, verify evidence, and revise its analysis based on intermediate results.

The benefit is greater flexibility in complex, tool-dependent workflows.

However, autonomy also introduces risks such as error propagation, increased latency, unpredictable execution paths, and unauthorized actions.

For financial applications, I would enforce explicit authorization and verification boundaries, and evaluate both task outcomes and execution traces.

Q6. How would you design an autonomous financial research agent?

中文面试回答

我会将金融研究 Agent 设计成具有明确状态管理和验证机制的 Workflow。系统首先识别用户意图、检查权限并生成研究计划，然后根据任务检索经过授权的财报、政策和市场资料。对于需要计算的财务指标，Agent 应调用确定性工具，而不是通过语言生成估算。任务执行过程使用结构化状态记录当前目标、已有证据、未完成步骤和错误信息。执行结束后，独立验证模块检查数值、来源和结论是否一致；若证据不足，则进行有限次数的额外检索、修正或人工升级。对于高影响操作，必须在执行前设置 Approval Gate。评估包括 Retrieval Recall、Numerical Accuracy、Citation Faithfulness、Task Success、Policy Violations 和 Cost per Verified Task。

English Interview Answer

I would design a financial research agent as a stateful, evidence-grounded workflow.

The system would begin with intent recognition and authorization, followed by planning and retrieval from trusted financial sources.

It would delegate numerical calculations to validated tools and maintain structured state containing objectives, supporting evidence, completed steps, and unresolved issues.

An independent verification stage would check numerical consistency, source attribution, and policy compliance.

When verification fails, the workflow could perform bounded retrieval or replanning, or escalate to human review.

I would enforce approval gates before high-impact actions and evaluate both final task success and intermediate execution quality.

Q7. ReAct vs Plan-and-Execute vs Multi-Agent: which architecture would you choose?

中文面试回答

ReAct 将推理与工具行动交替执行，适合任务结构不固定、需要根据最新观察动态决定下一步的场景，但可能出现循环调用、计划漂移和成本累积。Plan-and-Execute 首先生成任务计划，再依次执行各步骤，更适合具有清晰依赖关系的复杂任务，但初始计划可能因环境变化而失效。Multi-Agent 则通过多个具有不同职责的 Agent 协作完成任务，例如 Researcher、Analyst 和 Verifier，但增加角色并不必然提高可靠性，反而可能引入通信开销和错误传播。我的选择原则是从最简单且可以验证的 Workflow 开始：如果单 Agent + Tools 已满足目标，就不需要多个 Agent；当任务包含明确的专业分工或独立验证要求，并且消融实验显示具有实际收益时，再引入 Multi-Agent 架构。

English Interview Answer

ReAct, Plan-and-Execute, and multi-agent systems offer different trade-offs.

ReAct interleaves reasoning and tool use, making it flexible for tasks where the next action depends on observations, but it can suffer from loops and uncontrolled execution costs.

Plan-and-Execute separates planning from execution, providing a clearer structure for complex tasks, although plans may become outdated.

Multi-agent architectures introduce specialized roles and coordination, potentially improving task decomposition and verification, but also increasing communication costs and failure modes.

I would start with the simplest workflow that meets the requirements and introduce additional agents only when controlled evaluations demonstrate measurable benefits.

Q8. How do you prevent failures from propagating in long-horizon agent workflows?

中文面试回答

在长程 Agent Workflow 中，我会将错误控制设计在每个关键状态转移上，而不是只进行最终答案验证。首先，将任务分成具有明确输入、输出和成功条件的步骤；其次，对工具调用结果使用 Schema Validation、单元测试或领域规则检查，阻止未经验证的中间结果直接进入后续步骤。对于跨步骤依赖，应该保留 Evidence Provenance 和 State Checkpoints，使系统能够识别错误来源并从安全状态恢复。如果验证失败，Agent 不应无限重试，而应按照有界的 Recovery Policy 进行重试、重规划或人工升级。最后，使用 Trace-based Evaluation、Failure Injection 和 Regression Testing 评估错误传播与恢复能力。LangGraph 的持久化与 Human-in-the-loop 功能就是这一类工程机制的实例。

LangChain Reference

+1

English Interview Answer

I would control failure propagation at intermediate workflow boundaries rather than relying only on final-answer verification.

Each step should have explicit inputs, outputs, and success criteria. Tool results should pass schema validation, deterministic checks, or domain-specific tests before being used downstream.

I would maintain evidence provenance and state checkpoints so the system can identify where errors originated and recover from verified states.

Recovery should be bounded, with explicit retry limits, replanning conditions, and human escalation when necessary.

Finally, I would use trace-based evaluation, failure injection, and regression testing to measure error propagation and recovery performance.

Part III. Knowledge Graphs & GraphRAG

Q9. What is a Knowledge Graph, and how does it differ from a Vector Database?

中文面试回答

Knowledge Graph 是通过实体（Entities）、关系（Relations）以及属性（Attributes）表示领域知识的结构化数据模型。典型的知识可以表示为三元组，例如（Company A, issued, Bond B）或（Bond B, hasMaturity, 2030）。Knowledge Graph 的核心优势是能够显式表达实体之间的联系、关系类型和多跳路径，因此适合实体关系检索、多跳推理和事实依赖分析。Vector Database 则存储文本或其他对象的 Embeddings，主要通过向量相似度实现语义检索。两者并不互相替代：Vector Retrieval 善于找到语义相关的文本，Knowledge Graph 善于沿结构化关系组织和约束检索。在金融研究系统中，我会采用 Hybrid Retrieval，将 Embedding Similarity、Keyword Search 和 Graph Traversal 结合，同时保留原始文档作为证据，防止图谱中的错误实体或关系被误认为已经验证的事实。

English Interview Answer

A knowledge graph represents domain information through entities, typed relationships, and attributes.

For example, it can explicitly encode relationships between companies, securities, issuers, and financial instruments.

Its main advantage is supporting structured relationship queries and multi-hop traversal.

A vector database, by contrast, represents content using embeddings and retrieves semantically similar items.

These technologies are complementary rather than interchangeable. Vector search is useful for identifying relevant text, while graph traversal makes relationships and dependencies explicit.

For financial research, I would combine semantic retrieval, keyword search, and knowledge graph traversal, while preserving links to original source documents for verification.

Q10. What is GraphRAG, and why might it outperform traditional RAG in multi-hop reasoning?

中文面试回答

GraphRAG 将 Graph-based Retrieval 与 Retrieval-Augmented Generation 结合，使模型不仅依靠语义相似度检索文档，还能利用实体及其关系识别多跳证据路径。传统 RAG 通常先将文档分块，再通过 Embedding Similarity 检索相关文本。它适合局部事实问答，但对于跨多个文档的关系型问题，相关证据可能分散在不同 Chunk 中。GraphRAG 可以先识别问题中的实体，再通过图谱关系扩展候选节点和文档，例如沿着 Company → Subsidiary → Security → Exposure 的路径寻找证据。但 GraphRAG 不一定总是优于普通 RAG，因为知识图谱建设、实体消歧、关系抽取和图检索都可能增加成本并引入错误。我会根据问题类型分别比较 Vector RAG、Hybrid RAG 和 GraphRAG，而不是假设结构越复杂效果越好。GraphRAG 的相关研究也特别关注构图成本、检索质量和多跳推理。

Microsoft Research

+2

English Interview Answer

GraphRAG combines retrieval-augmented generation with graph-structured knowledge.

Traditional RAG typically retrieves relevant text chunks through semantic similarity, which works well for many factual questions but may struggle when evidence is distributed across multiple documents.

GraphRAG represents entities and relationships explicitly, allowing retrieval to follow multi-hop paths and assemble connected evidence.

For example, a financial research question might require tracing relationships between a parent company, subsidiaries, securities, and portfolio exposure.

However, GraphRAG introduces additional construction, entity-resolution, and retrieval costs, and incorrect graph edges can propagate errors.

I would compare graph retrieval with vector and hybrid baselines, especially on tasks requiring multi-hop relational reasoning.

Q11. How would you construct a reliable financial Knowledge Graph from documents?

中文面试回答

我会将金融知识图谱建设分为信息抽取、实体规范化、关系验证、时间版本管理和质量评估几个阶段。首先从财报、公告和其他可信文档中抽取公司、证券、产品、日期及关系；然后通过 Entity Resolution 将不同名称映射到规范实体，同时避免错误合并。接下来，关系应保留对应的 Source ID、时间范围和置信度，尤其是公司持股、产品条款和政策关系可能随时间变化。对于高风险关系，可以使用规则、结构化数据库或人工抽样进行验证。评估不应只报告抽取 Accuracy，还要分别测量 Entity Precision / Recall、Relation Precision / Recall、Temporal Correctness，以及下游 Question Answering 的效果。金融 Knowledge Graph 需要支持错误纠正和版本更新，否则旧关系可能造成错误的金融解释。

English Interview Answer

I would build a financial knowledge graph through extraction, entity resolution, relation validation, and temporal versioning.

First, I would extract entities and relationships from trusted financial documents.

Next, entity resolution would map aliases and alternative names to canonical identifiers, while avoiding incorrect entity merges.

I would preserve source provenance, timestamps, and validity intervals for relationships, because financial information changes over time.

High-impact relations should be validated against structured records or reviewed samples.

I would evaluate entity and relation precision and recall, temporal correctness, and downstream question-answering accuracy.

The key requirement is that graph facts remain traceable and updateable rather than becoming an unverified static knowledge store.

Q12. How can Knowledge Graphs improve LLM reasoning without guaranteeing factual correctness?

中文面试回答

Knowledge Graph 可以通过显式实体关系和多跳路径减少 LLM 完全依赖参数记忆的问题。例如，当模型需要回答某公司通过哪些子公司持有特定资产时，可以先通过图查询找到候选关系链，再检索原始文件验证每一条边，最后根据证据生成解释。这使推理过程更加可追踪，但不能保证结论正确，因为图谱本身可能存在实体对齐错误、缺失关系、过期资料或抽取错误。因此我会区分 Graph Retrieval Accuracy 与 Reasoning Accuracy，并要求关键结论能够回溯到来源文档。对于你研究的 Graph-grounded Multi-turn Reasoning，还可以根据证据缺口动态决定是否继续检索，而不是无条件增加图谱遍历深度。

English Interview Answer

Knowledge graphs can improve LLM reasoning by making entities and relationships explicit.

Instead of relying solely on parametric memory, the model can retrieve a connected path of relevant facts and use that structure to support multi-hop reasoning.

However, graph structure does not guarantee factual correctness. Entity resolution errors, missing edges, outdated relationships, or incorrect extraction can still produce misleading results.

I would therefore verify critical graph edges against their source documents and evaluate retrieval accuracy separately from reasoning accuracy.

For multi-turn reasoning, I would also use evidence gaps and uncertainty estimates to guide additional retrieval rather than expanding the graph indiscriminately.

Part IV. 综合面试题：这三个领域如何结合？

Q13. How would you combine Fine-tuning, Autonomous Agents, and Knowledge Graphs in a financial AI system?

中文面试回答

我会让三种技术分别承担不同职责。Fine-tuning 负责改善模型在金融领域的任务行为，例如结构化分析、指令遵循或合规回答；Knowledge Graph 负责组织金融实体及关系，支持可追溯的多跳证据检索；Autonomous Agent 则负责根据任务状态动态规划、调用检索与计算工具，以及执行验证。一个具体流程是：用户提出复杂的金融研究问题，Agent 首先识别目标并检查权限，然后使用 Hybrid Retrieval 和 Knowledge Graph 查找相关证据；对于财务计算，调用确定性工具完成；LLM 根据检索与计算结果形成分析，由独立 Verifier 检查事实支持、数值准确性和政策要求。如果系统频繁出现稳定的任务行为错误，可以考虑使用 SFT 或 LoRA 改进模型，但如果只是知识更新问题，则应优先更新检索源和图谱。评估时，我会通过消融实验分别测量 Fine-tuning、GraphRAG 和 Agent Workflow 的边际价值，避免增加系统复杂度却没有实际收益。

English Interview Answer

I would assign distinct responsibilities to fine-tuning, knowledge graphs, and autonomous agents.

Fine-tuning would improve domain-specific model behavior, such as structured financial analysis or compliance-aware responses.

Knowledge graphs would organize financial entities and relationships, supporting traceable multi-hop retrieval.

The autonomous agent would coordinate planning, retrieval, tool execution, and verification.

For a complex financial research question, the agent could retrieve connected evidence through GraphRAG, perform calculations using validated financial tools, and generate a grounded explanation.

An independent verifier would then check factual support, numerical consistency, and policy requirements.

I would use ablation studies to measure the incremental value of each component, ensuring that architectural complexity translates into measurable improvements in reliability and task performance.

如何证明一个 LLM 或 Autonomous Agent 不仅能够生成合理答案，而且能够稳定、可靠、低成本地完成任务，并且在发生错误时可以检测和恢复？

Q1. 为什么 BLEU、ROUGE 不适合充分评估现代 LLM 的生成质量？

中文面试回答

BLEU 和 ROUGE 是传统 NLP 中常见的自动评估指标，主要用于机器翻译和文本摘要。BLEU 主要根据生成文本与参考答案之间的 N-gram Precision 衡量匹配程度，而 ROUGE 常通过 N-gram Recall 或最长公共子序列衡量内容覆盖。它们的问题是过度依赖词汇或表面形式重合，无法可靠地识别语义等价、事实正确性、推理质量和指令遵循。例如，同一个金融问题可能存在多种表达不同但完全正确的答案，而一个与参考答案词汇高度重合的回答也可能包含错误数字或错误推理。因此，现代 LLM 评估需要结合 Semantic Similarity、Factuality、Task-specific Verification、LLM-as-a-Judge 和人工评价。对于金融领域，我会优先验证事实证据、数值正确性和风险披露，而不会只使用 BLEU 或 ROUGE 评价模型输出。

English Interview Answer

BLEU and ROUGE are reference-based metrics originally developed for tasks such as machine translation and summarization.

They primarily measure lexical overlap between generated outputs and reference texts.

Their main limitation is that lexical similarity does not necessarily reflect semantic correctness, factual accuracy, reasoning quality, or instruction following.

Two responses can be semantically equivalent while using very different wording. Conversely, a response may have high lexical overlap with a reference while containing incorrect financial figures.

For modern LLM evaluation, I would combine semantic metrics, task-specific correctness checks, evidence verification, calibrated LLM judges, and human evaluation.

In financial applications, factual grounding and numerical correctness are substantially more important than surface-level text similarity.

Follow-up：Can a model achieve a high ROUGE score while hallucinating?

中文：可以。例如原文中的财务数字为 15%，生成摘要却写成 50%，其他句子都与参考答案高度相似，ROUGE 仍然可能很高。因此数值正确性和事实一致性必须独立评估。

English:

Yes. A summary can overlap heavily with the reference while changing a critical numerical value or making an unsupported claim. I would therefore evaluate numerical and factual consistency separately from lexical overlap.

Q3. 什么是 LLM-as-a-Judge？它有哪些偏见？

中文面试回答

LLM-as-a-Judge 是使用一个语言模型作为评估器，根据预先设计的 Rubric，对另一个模型的输出进行质量评分、成对比较或错误分类。它的主要优势是成本通常低于大规模人工评估，并且可以灵活检查 Helpfulness、Instruction Following、Coherence 和部分 Faithfulness 属性。常见的评估方式包括 Pointwise Scoring、Pairwise Comparison 和 Reference-guided Evaluation。但它存在 Position Bias、Verbosity Bias、Style Bias 和 Self-preference Bias，例如评估模型可能更偏爱放在第一个位置、表达更长或语言风格更熟悉的答案。因此，我会使用盲化的回答顺序、双向交换位置、明确评分 Rubric、多次评估，以及具有独立人工标注的 Calibration Set 来检查评审一致性。对于金融领域的数字、事实和合规性，应该优先使用确定性验证或专业审查，而不是单独依赖 LLM Judge。MT-Bench 的相关研究明确报告了这些评估偏差。

arxiv.org

English Interview Answer

LLM-as-a-Judge uses a language model to evaluate another model's outputs according to explicit criteria.

Common approaches include pointwise scoring, pairwise comparison, and reference-guided evaluation.

Its main advantage is scalability. It can evaluate open-ended responses more efficiently than large-scale human review.

However, LLM judges can exhibit position bias, verbosity bias, style bias, and self-preference bias. They can also make reasoning or factual mistakes.

To improve reliability, I would use explicit rubrics, blinded answer presentation, position swapping, repeated evaluations, and calibration against human-labeled examples.

For financial applications, I would prioritize deterministic numerical checks and source-grounded verification whenever possible, rather than using an LLM judge as the sole authority.

Q5. 为什么 Agent Evaluation 比基础 LLM Evaluation 更复杂？

中文面试回答

基础 LLM Evaluation 通常可以把模型看成从输入 Prompt 到输出 Response 的映射，但 Agent 是一个与外部环境持续交互的动态决策系统。Agent 在任务执行过程中需要选择工具、更新状态、处理异常并可能重新规划，因此最终结果取决于模型、工具、环境、上下文、执行顺序和中间状态。即使最终答案相同，两个 Agent 也可能通过完全不同的路径完成任务，其中一个可能执行了多余操作、访问了不必要的数据，甚至违反了权限约束。另外，Agent 的长程错误具有传播和累积效应，单次成功不能证明系统具有稳定可靠性。因此 Agent Evaluation 应该同时测量 Outcome Quality、Process Correctness、Tool-use Accuracy、Efficiency、Robustness、Recovery、Policy Compliance 和 Reproducibility，并在可复现的执行环境中重复测试。

English Interview Answer

Evaluating an agent is more complex than evaluating a standalone LLM because an agent is a dynamic decision-making system interacting with external environments.

Its performance depends not only on the model's responses but also on planning, tool selection, state management, external observations, and recovery from failures.

Two agents may reach the same final answer through very different execution paths, with different costs, risks, or policy violations.

Long-horizon tasks also introduce error propagation, compounding uncertainty, and nondeterministic execution outcomes.

Therefore, I would evaluate agents across outcome quality, process correctness, tool use, efficiency, robustness, failure recovery, policy compliance, and reproducibility.

I would also repeat tasks under controlled environment conditions to estimate execution reliability.

Follow-up：Can an agent pass the final task while violating process constraints?

中文：可以。例如 Agent 最终正确完成报告，但中途访问了未授权文件，或者通过没有经过验证的工具修改了系统状态。因此需要分别设计 Outcome-based Evaluator 和 Process-based Evaluator，并将严重权限违规作为硬性失败条件。

English:

Yes. An agent may produce the correct final result while violating authorization rules or executing unsafe intermediate actions. I would therefore evaluate both the final outcome and the execution trace, with hard constraints for critical policy or authorization violations.

Q6. 哪些 Benchmark 专门用于 Agent Evaluation？

中文面试回答

Agent Benchmark 与传统 LLM Benchmark 最大的区别是，它们通常包含可交互的环境、工具接口、初始状态和任务成功条件，而不是只有 Question--Answer Pairs。AgentBench 通过多个交互环境测试 Agent 的决策与执行能力；GAIA 侧重真实世界中的多步推理、网页检索和工具使用；WebArena 使用可运行的网站环境评估网页操作任务；OSWorld 则在真实桌面和应用环境中评估 Computer-use Agents。对于 Coding Agents，SWE-bench 通过真实软件仓库中的问题和测试来衡量代码修复能力。此外，τ-bench 关注 Agent 与模拟用户、业务工具及政策规则的交互。设计 Agent Benchmark 时，我会特别关注环境是否可复现、评分是否基于真实执行结果，以及 Benchmark 是否同时覆盖成功案例和错误恢复。

English Interview Answer

Agent benchmarks differ from traditional LLM benchmarks because they typically require interaction with tools, environments, and changing states.

AgentBench evaluates decision-making across multiple interactive environments. GAIA emphasizes real-world reasoning, browsing, and tool use. WebArena evaluates agents performing tasks on functional websites, while OSWorld focuses on computer-use tasks in desktop environments.

SWE-bench evaluates software engineering agents through real repository issues and executable tests. Tau-bench focuses on interactions involving simulated users, tools, and domain-specific policies.

I would select benchmarks according to the capabilities required by the application and carefully assess environment reproducibility, execution-based scoring, and coverage of relevant failure modes.

Q7. 除了最终 Task Success，还应该评估哪些过程指标？

中文面试回答

我会将 Agent Evaluation 分成 Outcome、Process、Efficiency、Reliability 和 Safety 五个层次。Outcome 关注最终任务是否完成；Process 关注每个动作是否符合任务要求、工具参数是否正确、证据是否充分，以及是否进行了必要的验证；Efficiency 关注 Token Consumption、Tool Calls、Latency 和执行成本；Reliability 关注重复运行的一致性、异常处理、Retry Success 和错误恢复；Safety 则关注权限违规、敏感信息访问以及不必要的副作用。对于长程 Coding Agent 或 Financial Research Agent，我会特别记录首次错误发生的位置、错误是否向下游传播，以及 Verifier 能否在结果交付之前发现错误。另外，我会通过 Ablation Study 判断增加 Planner、Reviewer 或 Verifier 是否真正提高了单位成本下的成功率，而不是简单假设更多 Agent 一定更好。

English Interview Answer

I would evaluate agents across five dimensions: outcome, process quality, efficiency, reliability, and safety.

Outcome metrics measure whether the task was completed. Process metrics evaluate the correctness of tool calls, evidence use, intermediate actions, and verification steps.

Efficiency metrics capture latency, token usage, tool-call counts, and execution cost.

Reliability metrics include repeated-run consistency, failure recovery, and robustness to unexpected observations.

Safety metrics cover policy violations, unauthorized actions, and unintended side effects.

For long-horizon workflows, I would also identify where failures originate, whether they propagate downstream, and whether independent verification catches them before delivery.

高频过程指标

类别	Metric	解释
Outcome	Task Success Rate	最终成功比例
Process	Tool Call Validity	API 名称及参数是否正确
Process	Evidence Coverage	必要结论是否有证据
Process	Verification Recall	验证器发现真实错误的能力
Efficiency	Cost per Successful Task	单次成功的平均成本
Efficiency	p50 / p95 Latency	常规与尾部延迟
Reliability	Repeated-run Success	同任务多次运行表现
Reliability	Recovery Rate	失败后成功恢复的比例
Safety	Unauthorized Action Rate	越权执行比例

特别注意：Verification Recall 高并不代表 Verifier 整体好，因为一个永远拒绝所有输出的 Verifier 可以发现全部真实错误，但会造成很高的 False Rejection Rate。

因此还要报告 Verification Precision、False Acceptance 和 False Rejection。

Follow-up：What if a verifier improves quality but doubles the cost?

中文：我会评估 Verifier 的边际收益，而不是只看正确率。例如比较加入 Verifier 前后的 False Acceptance、Task Success、Latency 和 Cost per Verified Success。如果验证只改善了低风险任务的一小部分问题，却显著增加成本，可以采用 Risk-based Verification，只对高风险或证据不足的步骤启动更昂贵的验证。

English:

I would evaluate the marginal benefit of verification using improvements in false acceptance, task success, and cost per verified result. If verification is expensive, I would consider risk-based verification, applying more intensive checks to high-impact or uncertain steps rather than treating every action identically.

Q8. 什么是 Red Teaming？它在发现 LLM 和 Agent 安全漏洞方面有什么作用？

中文面试回答

Red Teaming 是通过主动设计困难、对抗性或异常输入，系统地发现 AI 模型与应用中的安全弱点和行为缺陷。对于 LLM，重点可能包括 Prompt Injection、错误或误导性信息、隐私泄露、偏见和违反安全策略的回答；对于 Autonomous Agent，还需要检查工具调用权限、跨步骤错误传播、不可信工具输出以及未经授权的环境操作。我会首先建立 Risk Taxonomy，然后构造正常、边界和对抗性测试案例，使用自动生成方法扩大覆盖范围，并由独立规则、验证器和必要的人工审查判断是否真的发生违规。确认的失败案例会进入 Regression Dataset，用于模型修正和后续版本验证。对金融 AI 来说，Red Teaming 的目标不只是发现模型是否会说错话，更重要的是发现错误是否可能导致不适当的金融解释、数据泄露或未经授权的系统行为。

English Interview Answer

Red teaming is a systematic process for identifying weaknesses in AI systems by testing them with adversarial, unusual, or high-risk scenarios.

For LLMs, common risk categories include misleading outputs, prompt injection, privacy leakage, biased behavior, and policy violations.

For autonomous agents, I would additionally evaluate unauthorized tool execution, unsafe state changes, and failure propagation across workflow steps.

I would define a clear risk taxonomy, generate diverse test cases, evaluate outcomes using independent checks, and use human review for ambiguous findings.

Confirmed failures would become regression tests for future model versions.

In financial applications, the objective is to identify harmful system behavior before it affects customers or critical business processes.

Q10. 如何持续监控线上 LLM 或 Agent，发现性能衰退和行为漂移？

中文面试回答

我会建立 Offline Evaluation、Online Monitoring 和 Incident-driven Evaluation 相结合的持续评估体系。在上线之前，使用固定 Regression Suite 测试模型的任务能力、事实性、数值正确性、安全性和工具执行行为；上线之后，通过结构化 Traces 记录模型版本、Prompt 版本、检索结果、工具调用、验证结果以及必要的延迟和成本指标，同时对敏感数据进行最小化采集与访问控制。监控中需要区分 Data Drift、Concept Drift 和 Behavioral Drift。例如用户请求分布变化属于输入层面的漂移，而模型或 Prompt 更新后出现更高的错误拒绝率则属于行为变化。对于 Agent，我还会观察 Tool Failure、Retry Loops、Recovery Rate、Verification False Acceptance 和 Long-horizon Task Success。发现异常后，通过 Canary Deployment、Shadow Evaluation、Regression Testing 和必要的 Rollback 控制风险，并将确认的失败案例加入长期测试集。

English Interview Answer

I would establish a continuous evaluation system combining offline benchmarks, online monitoring, and incident-driven testing.

Before deployment, I would validate model capability, factual accuracy, numerical correctness, safety, and tool-use behavior using a versioned regression suite.

In production, I would collect structured traces covering model versions, prompts, retrieval outcomes, tool calls, verification results, latency, and cost, subject to appropriate privacy controls.

I would monitor data drift, concept drift, and behavioral drift separately because they have different root causes.

For autonomous agents, I would additionally track tool failures, retry loops, recovery rates, verification failures, and long-horizon task success.

When degradation is detected, I would investigate the affected component, use controlled canary or shadow evaluations, and roll back changes when necessary.

Confirmed failures would become new regression tests.

Follow-up：How would you identify whether a regression comes from the model, retrieval, tools, or orchestration?

中文：我会利用版本化 Trace 和 Component-level Evaluation 进行定位。首先通过 Replay 将相同请求、检索证据及工具响应固定，判断新旧模型是否表现不同；然后分别替换 Retriever、Prompt、Tool 或 Agent Planner，开展受控消融。如果只在更新 Retriever 后出现证据缺失，则优先定位检索层；如果证据正确但最终答案错误，则需要进一步检查 Reasoning 或 Generation。所有 Replay 必须处理数据版本、环境状态和不可复现的外部依赖。

English:

I would use versioned execution traces, controlled replay, and component-level ablations. By holding inputs, retrieved evidence, and tool responses constant, I could compare model versions. I would then change one component at a time---retrieval, prompting, tools, or orchestration---to isolate the regression. For stateful agents, I would also control environment versions and external dependencies to make the comparison reproducible.

综合题：Design an Evaluation Framework for a Financial Research Agent

这道题可以把上面十个知识点整合成一个完整的 Senior Scientist 回答。

中文面试回答

对于 Financial Research Agent，我会采用多层、证据驱动的评估体系。首先建立任务级 Benchmark，覆盖单文档问答、多文档综合、多跳关系推理、财务计算和政策边界。其次对 Agent Execution Trace 进行分层评价，检查 Retrieval Coverage、Tool-call Correctness、State Transitions、Evidence Provenance 和 Verification Results。第三，在最终结果层面评估 Task Success、Numerical Accuracy、Citation Faithfulness 和 Policy Compliance。对于长程任务，我会引入 Failure Injection，测试工具失败、信息缺失和状态不一致时的恢复能力。最后，通过重复运行评估稳定性，通过 Ablation 比较单 Agent、Multi-Agent、GraphRAG 和不同验证机制的边际贡献，并报告成功率、风险以及每次已验证成功任务的成本。上线后将确认的失败案例持续纳入 Regression Suite，形成完整的 Evaluation Lifecycle。

English Interview Answer

I would design a hierarchical, evidence-driven evaluation framework for a financial research agent.

First, I would create a task-level benchmark covering document question answering, multi-document synthesis, relational reasoning, financial calculations, and policy-sensitive scenarios.

Second, I would evaluate execution traces, including retrieval coverage, tool-call correctness, state transitions, evidence provenance, and verification outcomes.

Third, I would measure final task success, numerical accuracy, citation faithfulness, and policy compliance.

For long-horizon workflows, I would introduce failure injection to test recovery from missing information, tool failures, and inconsistent states.

I would also repeat evaluations to measure reliability and conduct controlled ablations to understand the contributions of specialized agents, GraphRAG, and verification components.

Finally, I would monitor cost per verified successful task and continuously expand the regression benchmark with confirmed production failures.

Long-horizon Agent Evaluation 最核心的研究问题之一：如何自动识别 Agent 的错误自信（False Confidence），尤其是在 Agent 根本不知道自己缺少什么知识的情况下。

我们不应该只设计一个预测 Agent 是否正确的 Confidence Score，而应该设计一套能够发现证据不足、识别潜在错误，并决定是否允许 Agent 继续执行的验证机制。

因为 Agent 可能在缺乏知识的情况下，仍然产生高度一致、逻辑流畅的计划。即使让它进行 Self-Reflection，也可能无法发现问题。

中文面试回答

我认为传统 Agent Evaluation 的核心局限是过度关注最终 Task Success，而忽视 Agent 在长程执行过程中是否具有足够证据支持自己的决策。尤其当 Agent 缺乏某项关键知识时，它可能并不知道自己存在知识缺口，却仍然自信地执行后续操作。因此，我希望研究 Evidence-grounded Epistemic Uncertainty Detection，不仅评估模型报告的 Confidence，还通过任务约束、依赖关系、证据覆盖率、隐含前提验证和工具执行结果，识别 Agent 何时不应该继续执行。评估上，我会重点测量 False Confidence Detection Recall、False Alarm Rate、Detection Lead Time、Error Propagation 和 Cost per Prevented Failure。进一步通过 Controlled Failure Injection 和 Ablation Study，验证风险驱动的检索、测试或人工干预是否真正提高长程任务的可靠性。对于金融 AI，这种机制尤其重要，因为一个未被发现的错误前提可能影响多个下游分析步骤，最终产生看似合理但缺乏证据支持的结论。

English Interview Answer

I believe one of the fundamental limitations of current agent evaluation is that it focuses heavily on final task success, while paying insufficient attention to whether an agent has enough evidence to justify its intermediate decisions.

In long-horizon workflows, an agent may be unaware of missing knowledge and continue confidently based on unsupported assumptions.

My research interest is in evidence-grounded epistemic uncertainty detection. Instead of relying only on self-reported confidence, I would use task requirements, dependency structures, evidence coverage, assumption validation, and tool execution results to identify unjustified continuation.

I would evaluate the framework using false-confidence detection recall, false-alarm rates, detection lead time, downstream error propagation, and the cost of preventing failures.

The key research question is whether targeted verification can detect previously unrecognized evidence gaps before they cause irreversible downstream errors.

This is particularly relevant to financial AI, where an unsupported assumption can propagate across multiple analysis steps and lead to a convincing but unreliable conclusion.