后端第十三课：大模型评测（Evals）从 0 到 1——看完面试能讲明白

👋 我是 Yulia，一线 AI 应用开发工程师：4 年工程经验，在职做大模型应用落地。

只写一件事------把最新 AI 技术讲成人话 + 给你能直接上手用的代码 （真实可验证，不搬运不注水）。

有用就点个关注，持续更新大模型应用的前沿实战。
📚 本文是《后端开发从入门到靠谱》系列第 13 篇。前面做了 RAG、Agent，但有个灵魂问题一直没解决：你怎么知道你的 Agent 改完是变好了还是变差了？ 这一课讲大模型评测（Evals）------面试高频考点，也是从"会写"到"靠谱"的分水岭。本文参考 Anthropic 工程团队的《Demystifying evals for AI agents》整理。

你改了个 prompt，兴冲冲上线，结果用户反馈"怎么变笨了"------但你根本不知道是哪里坏了 ，只能猜。这就是没有评测的团队日常：瞎飞（flying blind）。

评测（Eval） 就是来治这个的。这篇从"是什么"讲到"怎么从 0 建一套"，重点是让你看完能说、能举例子、面试能侃侃而谈。

读完你会拿到：

一句话理解评测：给 AI"出卷子 + 批卷子"
8 个核心概念（task / grader / transcript / outcome...面试必考，用例子记住）
三类评分器（代码 / 模型 / 人工）的优缺点与选择
能力评测 vs 回归评测（面试爱问的区别）
非确定性的两个指标：pass@k 和 pass^k（面试加分点）
从 0 到 1 的 8 步路线（完整的步骤流程）
评测的闭环（怎么形成持续改进的正循环）
一页纸"面试速记"（3 分钟把 Evals 讲清楚）

先看全景图，评测的闭环长这样：
#mermaid-svg-h2Gjl6pfpu5IXle6{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-h2Gjl6pfpu5IXle6 .error-icon{fill:#552222;}#mermaid-svg-h2Gjl6pfpu5IXle6 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-h2Gjl6pfpu5IXle6 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .marker.cross{stroke:#333333;}#mermaid-svg-h2Gjl6pfpu5IXle6 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-h2Gjl6pfpu5IXle6 p{margin:0;}#mermaid-svg-h2Gjl6pfpu5IXle6 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cluster-label text{fill:#333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cluster-label span{color:#333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cluster-label span p{background-color:transparent;}#mermaid-svg-h2Gjl6pfpu5IXle6 .label text,#mermaid-svg-h2Gjl6pfpu5IXle6 span{fill:#333;color:#333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .node rect,#mermaid-svg-h2Gjl6pfpu5IXle6 .node circle,#mermaid-svg-h2Gjl6pfpu5IXle6 .node ellipse,#mermaid-svg-h2Gjl6pfpu5IXle6 .node polygon,#mermaid-svg-h2Gjl6pfpu5IXle6 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-h2Gjl6pfpu5IXle6 .rough-node .label text,#mermaid-svg-h2Gjl6pfpu5IXle6 .node .label text,#mermaid-svg-h2Gjl6pfpu5IXle6 .image-shape .label,#mermaid-svg-h2Gjl6pfpu5IXle6 .icon-shape .label{text-anchor:middle;}#mermaid-svg-h2Gjl6pfpu5IXle6 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-h2Gjl6pfpu5IXle6 .rough-node .label,#mermaid-svg-h2Gjl6pfpu5IXle6 .node .label,#mermaid-svg-h2Gjl6pfpu5IXle6 .image-shape .label,#mermaid-svg-h2Gjl6pfpu5IXle6 .icon-shape .label{text-align:center;}#mermaid-svg-h2Gjl6pfpu5IXle6 .node.clickable{cursor:pointer;}#mermaid-svg-h2Gjl6pfpu5IXle6 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .arrowheadPath{fill:#333333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-h2Gjl6pfpu5IXle6 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-h2Gjl6pfpu5IXle6 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-h2Gjl6pfpu5IXle6 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-h2Gjl6pfpu5IXle6 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-h2Gjl6pfpu5IXle6 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cluster text{fill:#333;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cluster span{color:#333;}#mermaid-svg-h2Gjl6pfpu5IXle6 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-h2Gjl6pfpu5IXle6 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-h2Gjl6pfpu5IXle6 rect.text{fill:none;stroke-width:0;}#mermaid-svg-h2Gjl6pfpu5IXle6 .icon-shape,#mermaid-svg-h2Gjl6pfpu5IXle6 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-h2Gjl6pfpu5IXle6 .icon-shape p,#mermaid-svg-h2Gjl6pfpu5IXle6 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-h2Gjl6pfpu5IXle6 .icon-shape .label rect,#mermaid-svg-h2Gjl6pfpu5IXle6 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-h2Gjl6pfpu5IXle6 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-h2Gjl6pfpu5IXle6 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-h2Gjl6pfpu5IXle6 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-h2Gjl6pfpu5IXle6 .blue>*{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .blue span{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .blue tspan{fill:#fff!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .orange>*{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .orange span{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .orange tspan{fill:#fff!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .green>*{fill:#59A14F!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .green span{fill:#59A14F!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .green tspan{fill:#fff!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .red>*{fill:#57A0E1!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .red span{fill:#57A0E1!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .red tspan{fill:#fff!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cyan>*{fill:#76B7B2!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cyan span{fill:#76B7B2!important;color:#fff!important;stroke:none!important;}#mermaid-svg-h2Gjl6pfpu5IXle6 .cyan tspan{fill:#fff!important;} 改进 prompt 代码
防回退
定义成功标准
建评测任务
跑评测
读轨迹分析
发现问题
持续跑回归

一、评测是什么？一句话 + 一个类比

评测（Evaluation，简称 Eval）就是：给 AI 一个输入，再用一套评分逻辑去衡量它的输出成不成功。 换句话说------给 AI 出卷子，再批卷子。
#mermaid-svg-dVl3Ui5J15Sl29Ts{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dVl3Ui5J15Sl29Ts .error-icon{fill:#552222;}#mermaid-svg-dVl3Ui5J15Sl29Ts .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dVl3Ui5J15Sl29Ts .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .marker.cross{stroke:#333333;}#mermaid-svg-dVl3Ui5J15Sl29Ts svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dVl3Ui5J15Sl29Ts p{margin:0;}#mermaid-svg-dVl3Ui5J15Sl29Ts .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .cluster-label text{fill:#333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .cluster-label span{color:#333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .cluster-label span p{background-color:transparent;}#mermaid-svg-dVl3Ui5J15Sl29Ts .label text,#mermaid-svg-dVl3Ui5J15Sl29Ts span{fill:#333;color:#333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .node rect,#mermaid-svg-dVl3Ui5J15Sl29Ts .node circle,#mermaid-svg-dVl3Ui5J15Sl29Ts .node ellipse,#mermaid-svg-dVl3Ui5J15Sl29Ts .node polygon,#mermaid-svg-dVl3Ui5J15Sl29Ts .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-dVl3Ui5J15Sl29Ts .rough-node .label text,#mermaid-svg-dVl3Ui5J15Sl29Ts .node .label text,#mermaid-svg-dVl3Ui5J15Sl29Ts .image-shape .label,#mermaid-svg-dVl3Ui5J15Sl29Ts .icon-shape .label{text-anchor:middle;}#mermaid-svg-dVl3Ui5J15Sl29Ts .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-dVl3Ui5J15Sl29Ts .rough-node .label,#mermaid-svg-dVl3Ui5J15Sl29Ts .node .label,#mermaid-svg-dVl3Ui5J15Sl29Ts .image-shape .label,#mermaid-svg-dVl3Ui5J15Sl29Ts .icon-shape .label{text-align:center;}#mermaid-svg-dVl3Ui5J15Sl29Ts .node.clickable{cursor:pointer;}#mermaid-svg-dVl3Ui5J15Sl29Ts .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .arrowheadPath{fill:#333333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-dVl3Ui5J15Sl29Ts .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dVl3Ui5J15Sl29Ts .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-dVl3Ui5J15Sl29Ts .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dVl3Ui5J15Sl29Ts .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-dVl3Ui5J15Sl29Ts .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-dVl3Ui5J15Sl29Ts .cluster text{fill:#333;}#mermaid-svg-dVl3Ui5J15Sl29Ts .cluster span{color:#333;}#mermaid-svg-dVl3Ui5J15Sl29Ts div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-dVl3Ui5J15Sl29Ts .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-dVl3Ui5J15Sl29Ts rect.text{fill:none;stroke-width:0;}#mermaid-svg-dVl3Ui5J15Sl29Ts .icon-shape,#mermaid-svg-dVl3Ui5J15Sl29Ts .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dVl3Ui5J15Sl29Ts .icon-shape p,#mermaid-svg-dVl3Ui5J15Sl29Ts .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-dVl3Ui5J15Sl29Ts .icon-shape .label rect,#mermaid-svg-dVl3Ui5J15Sl29Ts .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dVl3Ui5J15Sl29Ts .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-dVl3Ui5J15Sl29Ts .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-dVl3Ui5J15Sl29Ts :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-dVl3Ui5J15Sl29Ts .blue>*{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .blue span{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .blue tspan{fill:#fff!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .orange>*{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .orange span{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .orange tspan{fill:#fff!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .green>*{fill:#59A14F!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .green span{fill:#59A14F!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dVl3Ui5J15Sl29Ts .green tspan{fill:#fff!important;} 输入任务
AI Agent
输出
评分器
成功或失败

早期的 LLM 评测是单轮的（一个 prompt → 一个回答 → 打分），很简单。但现在做 Agent 评测就难多了------因为 Agent 会多轮操作、调用工具、改变环境，错误还会累积放大。

关键认知：让 Agent 有用的能力（自主、智能、灵活），恰恰也是让 Agent 难评测的能力。 这是理解后面所有内容的总纲。

二、8 个核心概念：面试必考，用例子记住

Anthropic 在评测 Agent 时用了 8 个精确定义，面试问"你懂评测吗"，把这 8 个词讲清楚就是专业的表现。

概念	中文	一句话解释
task	任务/测试用例	单个测试，有明确输入和成功标准
trial	试次	对同一个任务的一次尝试（模型输出有随机性，要跑多次）
grader	评分器	给 Agent 表现打分的逻辑，可含多个断言
transcript	轨迹/记录	一次试次的完整记录（输出、工具调用、推理、中间结果）
outcome	结果	环境在试次结束时的最终状态
evaluation harness	评测基础设施	端到端跑评测、记录、打分的框架
agent harness	Agent 脚手架	让模型变成 Agent 的系统（评测的是"脚手架+模型"一起）
evaluation suite	评测集	一组任务，测某个特定能力

最关键的一对概念是 transcript vs outcome，用一个例子秒懂：
#mermaid-svg-EruqppGuEVbd2362{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-EruqppGuEVbd2362 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-EruqppGuEVbd2362 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-EruqppGuEVbd2362 .error-icon{fill:#552222;}#mermaid-svg-EruqppGuEVbd2362 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-EruqppGuEVbd2362 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-EruqppGuEVbd2362 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-EruqppGuEVbd2362 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-EruqppGuEVbd2362 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-EruqppGuEVbd2362 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-EruqppGuEVbd2362 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-EruqppGuEVbd2362 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-EruqppGuEVbd2362 .marker.cross{stroke:#333333;}#mermaid-svg-EruqppGuEVbd2362 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-EruqppGuEVbd2362 p{margin:0;}#mermaid-svg-EruqppGuEVbd2362 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-EruqppGuEVbd2362 .cluster-label text{fill:#333;}#mermaid-svg-EruqppGuEVbd2362 .cluster-label span{color:#333;}#mermaid-svg-EruqppGuEVbd2362 .cluster-label span p{background-color:transparent;}#mermaid-svg-EruqppGuEVbd2362 .label text,#mermaid-svg-EruqppGuEVbd2362 span{fill:#333;color:#333;}#mermaid-svg-EruqppGuEVbd2362 .node rect,#mermaid-svg-EruqppGuEVbd2362 .node circle,#mermaid-svg-EruqppGuEVbd2362 .node ellipse,#mermaid-svg-EruqppGuEVbd2362 .node polygon,#mermaid-svg-EruqppGuEVbd2362 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-EruqppGuEVbd2362 .rough-node .label text,#mermaid-svg-EruqppGuEVbd2362 .node .label text,#mermaid-svg-EruqppGuEVbd2362 .image-shape .label,#mermaid-svg-EruqppGuEVbd2362 .icon-shape .label{text-anchor:middle;}#mermaid-svg-EruqppGuEVbd2362 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-EruqppGuEVbd2362 .rough-node .label,#mermaid-svg-EruqppGuEVbd2362 .node .label,#mermaid-svg-EruqppGuEVbd2362 .image-shape .label,#mermaid-svg-EruqppGuEVbd2362 .icon-shape .label{text-align:center;}#mermaid-svg-EruqppGuEVbd2362 .node.clickable{cursor:pointer;}#mermaid-svg-EruqppGuEVbd2362 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-EruqppGuEVbd2362 .arrowheadPath{fill:#333333;}#mermaid-svg-EruqppGuEVbd2362 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-EruqppGuEVbd2362 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-EruqppGuEVbd2362 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EruqppGuEVbd2362 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-EruqppGuEVbd2362 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EruqppGuEVbd2362 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-EruqppGuEVbd2362 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-EruqppGuEVbd2362 .cluster text{fill:#333;}#mermaid-svg-EruqppGuEVbd2362 .cluster span{color:#333;}#mermaid-svg-EruqppGuEVbd2362 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-EruqppGuEVbd2362 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-EruqppGuEVbd2362 rect.text{fill:none;stroke-width:0;}#mermaid-svg-EruqppGuEVbd2362 .icon-shape,#mermaid-svg-EruqppGuEVbd2362 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EruqppGuEVbd2362 .icon-shape p,#mermaid-svg-EruqppGuEVbd2362 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-EruqppGuEVbd2362 .icon-shape .label rect,#mermaid-svg-EruqppGuEVbd2362 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EruqppGuEVbd2362 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-EruqppGuEVbd2362 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-EruqppGuEVbd2362 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-EruqppGuEVbd2362 .blue>*{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-EruqppGuEVbd2362 .blue span{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-EruqppGuEVbd2362 .blue tspan{fill:#fff!important;}#mermaid-svg-EruqppGuEVbd2362 .orange>*{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-EruqppGuEVbd2362 .orange span{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-EruqppGuEVbd2362 .orange tspan{fill:#fff!important;}#mermaid-svg-EruqppGuEVbd2362 .red>*{fill:#E15759!important;color:#fff!important;stroke:none!important;}#mermaid-svg-EruqppGuEVbd2362 .red span{fill:#E15759!important;color:#fff!important;stroke:none!important;}#mermaid-svg-EruqppGuEVbd2362 .red tspan{fill:#fff!important;} transcript 嘴上说
已订好机票
outcome 实际状态
数据库里真有预订吗

一个订机票的 Agent，transcript 里它可能最后说"您的机票已订好"，但 outcome 要看数据库里是不是真的生成了预订记录 。嘴上说完成 ≠ 真的完成------这是 Agent 评测和普通"看回答像不像"最大的区别。

三、为什么必须做评测？（面试讲"价值"）

很多人觉得评测是"额外的负担"，拖慢上线。但 Anthropic 的观察是：原型阶段靠手动测试 + 直觉能走很远，一旦上线规模化，没评测就会崩。

崩的临界点通常是：用户反馈"改了之后变差了"，团队没有方法验证，只能猜。没有评测，你就无法：① 区分"真回退"和"噪声"；② 上线前自动测几百个场景；③ 量化"到底有没有改进"。

评测还有三个常被忽视的价值：

逼你定义"成功"：两个工程师看同一份需求，对边界情况的理解可能完全不同，评测集能消除这种歧义；
加速换新模型：新模型出来，有评测的团队几天就能评估、调 prompt、升级；没评测的要测好几周；
白送基线和回归测试：有了评测，延迟、token 用量、单任务成本、错误率都能在一组固定任务上持续追踪。

四、三类评分器（grader）：怎么给 Agent 打分

评测设计最核心的一步是选对评分器。三类：
#mermaid-svg-fEx7ZwhOY6XhCPQg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-fEx7ZwhOY6XhCPQg .error-icon{fill:#552222;}#mermaid-svg-fEx7ZwhOY6XhCPQg .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-fEx7ZwhOY6XhCPQg .marker{fill:#333333;stroke:#333333;}#mermaid-svg-fEx7ZwhOY6XhCPQg .marker.cross{stroke:#333333;}#mermaid-svg-fEx7ZwhOY6XhCPQg svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-fEx7ZwhOY6XhCPQg p{margin:0;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge{stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section--1 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section--1 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section--1 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section--1 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section--1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section--1 text{fill:#ffffff;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon--1{font-size:40px;color:#ffffff;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge--1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth--1{stroke-width:17;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section--1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-0 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-0 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-0 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-0 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-0 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-0{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-0{stroke-width:14;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-1 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-1 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-1 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-1 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-1 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-1{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-1{stroke-width:11;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 text{fill:#ffffff;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-2{stroke-width:8;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-3 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-3 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-3 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-3 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-3 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-3{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-3{stroke-width:5;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-4 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-4 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-4 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-4 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-4 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-4{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-4{stroke-width:2;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-5 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-5 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-5 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-5 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-5 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-5{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-5{stroke-width:-1;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-6 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-6 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-6 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-6 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-6 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-6{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-6{stroke-width:-4;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-7 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-7 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-7 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-7 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-7 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-7{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-7{stroke-width:-7;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-8 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-8 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-8 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-8 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-8 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-8{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-8{stroke-width:-10;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-9 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-9 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-9 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-9 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-9 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-9{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-9{stroke-width:-13;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-10 rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-10 path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-10 circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-10 polygon,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-10 text{fill:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .node-icon-10{font-size:40px;color:black;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge-depth-10{stroke-width:-16;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:lightgray;}#mermaid-svg-fEx7ZwhOY6XhCPQg .disabled text{fill:#efefef;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-root rect,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-root path,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-root circle,#mermaid-svg-fEx7ZwhOY6XhCPQg .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-root text{fill:#ffffff;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-root span{color:#ffffff;}#mermaid-svg-fEx7ZwhOY6XhCPQg .section-2 span{color:#ffffff;}#mermaid-svg-fEx7ZwhOY6XhCPQg .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-fEx7ZwhOY6XhCPQg .edge{fill:none;}#mermaid-svg-fEx7ZwhOY6XhCPQg .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-fEx7ZwhOY6XhCPQg :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 三类评分器
代码评分
快便宜客观
字符串单测
模型评分
灵活有 nuance
需人类校准
人工评分
金标准
贵慢

类型	方法举例	强项	弱点
代码评分	字符串匹配、单元测试、静态分析、状态验证	快、便宜、客观、可复现	对"合理但不符合预期格式"的变体太脆弱
模型评分（LLM-as-judge）	评分表（rubric）、自然语言断言、成对比较、多法官共识	灵活、可扩展、有 nuance	非确定、贵、需要人工校准
人工评分	专家评审、众包、抽样	金标准、最贴合用户判断	贵、慢、难规模化

选择原则 ：能用代码评分（确定性）就用代码；需要灵活性才用模型评分；人工评分主要用于校准模型评分器。

一个经典反面教材：Anthropic 的 Opus 4.5 在 CORE-Bench 上最初只得了 42%，后来发现是评分太死板 ------它把"96.12"判成错，只因为预期答案是"96.124991..."。修掉这些评分 bug 后，分数跳到 95%。评分器本身可能就是坑。

五、能力评测 vs 回归评测：面试爱问的区别

能力评测（Capability eval） ：问"这个 Agent 能做多好 ？"应该从低通过率起步，专挑它现在还搞不定的任务，给团队一个爬坡目标。
回归评测（Regression eval） ：问"它还做得到以前会做的事吗 ？"应该接近 100% 通过率，防止改坏东西。

#mermaid-svg-ehdicEiXLo5QFwHt{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ehdicEiXLo5QFwHt .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ehdicEiXLo5QFwHt .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ehdicEiXLo5QFwHt .error-icon{fill:#552222;}#mermaid-svg-ehdicEiXLo5QFwHt .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ehdicEiXLo5QFwHt .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ehdicEiXLo5QFwHt .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ehdicEiXLo5QFwHt .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ehdicEiXLo5QFwHt .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ehdicEiXLo5QFwHt .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ehdicEiXLo5QFwHt .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ehdicEiXLo5QFwHt .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ehdicEiXLo5QFwHt .marker.cross{stroke:#333333;}#mermaid-svg-ehdicEiXLo5QFwHt svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ehdicEiXLo5QFwHt p{margin:0;}#mermaid-svg-ehdicEiXLo5QFwHt .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ehdicEiXLo5QFwHt .cluster-label text{fill:#333;}#mermaid-svg-ehdicEiXLo5QFwHt .cluster-label span{color:#333;}#mermaid-svg-ehdicEiXLo5QFwHt .cluster-label span p{background-color:transparent;}#mermaid-svg-ehdicEiXLo5QFwHt .label text,#mermaid-svg-ehdicEiXLo5QFwHt span{fill:#333;color:#333;}#mermaid-svg-ehdicEiXLo5QFwHt .node rect,#mermaid-svg-ehdicEiXLo5QFwHt .node circle,#mermaid-svg-ehdicEiXLo5QFwHt .node ellipse,#mermaid-svg-ehdicEiXLo5QFwHt .node polygon,#mermaid-svg-ehdicEiXLo5QFwHt .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ehdicEiXLo5QFwHt .rough-node .label text,#mermaid-svg-ehdicEiXLo5QFwHt .node .label text,#mermaid-svg-ehdicEiXLo5QFwHt .image-shape .label,#mermaid-svg-ehdicEiXLo5QFwHt .icon-shape .label{text-anchor:middle;}#mermaid-svg-ehdicEiXLo5QFwHt .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ehdicEiXLo5QFwHt .rough-node .label,#mermaid-svg-ehdicEiXLo5QFwHt .node .label,#mermaid-svg-ehdicEiXLo5QFwHt .image-shape .label,#mermaid-svg-ehdicEiXLo5QFwHt .icon-shape .label{text-align:center;}#mermaid-svg-ehdicEiXLo5QFwHt .node.clickable{cursor:pointer;}#mermaid-svg-ehdicEiXLo5QFwHt .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ehdicEiXLo5QFwHt .arrowheadPath{fill:#333333;}#mermaid-svg-ehdicEiXLo5QFwHt .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ehdicEiXLo5QFwHt .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ehdicEiXLo5QFwHt .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ehdicEiXLo5QFwHt .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ehdicEiXLo5QFwHt .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ehdicEiXLo5QFwHt .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ehdicEiXLo5QFwHt .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ehdicEiXLo5QFwHt .cluster text{fill:#333;}#mermaid-svg-ehdicEiXLo5QFwHt .cluster span{color:#333;}#mermaid-svg-ehdicEiXLo5QFwHt div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ehdicEiXLo5QFwHt .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ehdicEiXLo5QFwHt rect.text{fill:none;stroke-width:0;}#mermaid-svg-ehdicEiXLo5QFwHt .icon-shape,#mermaid-svg-ehdicEiXLo5QFwHt .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ehdicEiXLo5QFwHt .icon-shape p,#mermaid-svg-ehdicEiXLo5QFwHt .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ehdicEiXLo5QFwHt .icon-shape .label rect,#mermaid-svg-ehdicEiXLo5QFwHt .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ehdicEiXLo5QFwHt .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ehdicEiXLo5QFwHt .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ehdicEiXLo5QFwHt :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-ehdicEiXLo5QFwHt .blue>*{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-ehdicEiXLo5QFwHt .blue span{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-ehdicEiXLo5QFwHt .blue tspan{fill:#fff!important;}#mermaid-svg-ehdicEiXLo5QFwHt .orange>*{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-ehdicEiXLo5QFwHt .orange span{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-ehdicEiXLo5QFwHt .orange tspan{fill:#fff!important;} 低通过率起步
近百分百
能力评测
能做多好
回归评测
还做得到吗

两者配合：当你爬能力评测时，也要跑回归评测，确保"变强"的同时没有"变坏"。能力评测的通过率高到接近满分后，可以"毕业"成为回归套件------从"能不能做到"变成"能不能稳定做到"。

六、非确定性：pass@k 和 pass^k（面试加分点）

Agent 每次跑结果都不一样（同一个任务这次过、下次挂）。所以要跑多次，并用两个指标衡量：

pass@k ：k 次尝试里至少一次成功 的概率。k 越大越高（机会越多）。pass@1=50% 意思是"一半任务第一次就成功"。
pass^k ：k 次全部成功 的概率。k 越大越低（要求越严）。比如单次成功率 75%，跑 3 次全过概率是 0.75³ ≈ 42%。

#mermaid-svg-dN1R4pfHFSddNG88{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-dN1R4pfHFSddNG88 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-dN1R4pfHFSddNG88 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-dN1R4pfHFSddNG88 .error-icon{fill:#552222;}#mermaid-svg-dN1R4pfHFSddNG88 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-dN1R4pfHFSddNG88 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-dN1R4pfHFSddNG88 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-dN1R4pfHFSddNG88 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-dN1R4pfHFSddNG88 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-dN1R4pfHFSddNG88 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-dN1R4pfHFSddNG88 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-dN1R4pfHFSddNG88 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-dN1R4pfHFSddNG88 .marker.cross{stroke:#333333;}#mermaid-svg-dN1R4pfHFSddNG88 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-dN1R4pfHFSddNG88 p{margin:0;}#mermaid-svg-dN1R4pfHFSddNG88 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-dN1R4pfHFSddNG88 .cluster-label text{fill:#333;}#mermaid-svg-dN1R4pfHFSddNG88 .cluster-label span{color:#333;}#mermaid-svg-dN1R4pfHFSddNG88 .cluster-label span p{background-color:transparent;}#mermaid-svg-dN1R4pfHFSddNG88 .label text,#mermaid-svg-dN1R4pfHFSddNG88 span{fill:#333;color:#333;}#mermaid-svg-dN1R4pfHFSddNG88 .node rect,#mermaid-svg-dN1R4pfHFSddNG88 .node circle,#mermaid-svg-dN1R4pfHFSddNG88 .node ellipse,#mermaid-svg-dN1R4pfHFSddNG88 .node polygon,#mermaid-svg-dN1R4pfHFSddNG88 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-dN1R4pfHFSddNG88 .rough-node .label text,#mermaid-svg-dN1R4pfHFSddNG88 .node .label text,#mermaid-svg-dN1R4pfHFSddNG88 .image-shape .label,#mermaid-svg-dN1R4pfHFSddNG88 .icon-shape .label{text-anchor:middle;}#mermaid-svg-dN1R4pfHFSddNG88 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-dN1R4pfHFSddNG88 .rough-node .label,#mermaid-svg-dN1R4pfHFSddNG88 .node .label,#mermaid-svg-dN1R4pfHFSddNG88 .image-shape .label,#mermaid-svg-dN1R4pfHFSddNG88 .icon-shape .label{text-align:center;}#mermaid-svg-dN1R4pfHFSddNG88 .node.clickable{cursor:pointer;}#mermaid-svg-dN1R4pfHFSddNG88 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-dN1R4pfHFSddNG88 .arrowheadPath{fill:#333333;}#mermaid-svg-dN1R4pfHFSddNG88 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-dN1R4pfHFSddNG88 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-dN1R4pfHFSddNG88 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dN1R4pfHFSddNG88 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-dN1R4pfHFSddNG88 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dN1R4pfHFSddNG88 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-dN1R4pfHFSddNG88 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-dN1R4pfHFSddNG88 .cluster text{fill:#333;}#mermaid-svg-dN1R4pfHFSddNG88 .cluster span{color:#333;}#mermaid-svg-dN1R4pfHFSddNG88 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-dN1R4pfHFSddNG88 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-dN1R4pfHFSddNG88 rect.text{fill:none;stroke-width:0;}#mermaid-svg-dN1R4pfHFSddNG88 .icon-shape,#mermaid-svg-dN1R4pfHFSddNG88 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-dN1R4pfHFSddNG88 .icon-shape p,#mermaid-svg-dN1R4pfHFSddNG88 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-dN1R4pfHFSddNG88 .icon-shape .label rect,#mermaid-svg-dN1R4pfHFSddNG88 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-dN1R4pfHFSddNG88 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-dN1R4pfHFSddNG88 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-dN1R4pfHFSddNG88 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-dN1R4pfHFSddNG88 .blue>*{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dN1R4pfHFSddNG88 .blue span{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dN1R4pfHFSddNG88 .blue tspan{fill:#fff!important;}#mermaid-svg-dN1R4pfHFSddNG88 .orange>*{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dN1R4pfHFSddNG88 .orange span{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-dN1R4pfHFSddNG88 .orange tspan{fill:#fff!important;} pass at k
至少一次成功越跑越高
pass 幂 k
全部成功越跑越低

怎么选 ：给"只要有一次成功就行"的工具（比如写代码，找到解法就行）用 pass@k；给"每次都得不翻车"的客户侧 Agent 用 pass^k。面试能讲清这两个，是明显的加分点。

七、从 0 到 1 的 8 步路线（核心：步骤 + 流程）

这是 Anthropic 给的实战路线，也是面试能"从头到尾讲一遍"的骨架：
#mermaid-svg-XXatuPqWOy9kncv9{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-XXatuPqWOy9kncv9 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-XXatuPqWOy9kncv9 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-XXatuPqWOy9kncv9 .error-icon{fill:#552222;}#mermaid-svg-XXatuPqWOy9kncv9 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-XXatuPqWOy9kncv9 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-XXatuPqWOy9kncv9 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-XXatuPqWOy9kncv9 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-XXatuPqWOy9kncv9 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-XXatuPqWOy9kncv9 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-XXatuPqWOy9kncv9 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-XXatuPqWOy9kncv9 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-XXatuPqWOy9kncv9 .marker.cross{stroke:#333333;}#mermaid-svg-XXatuPqWOy9kncv9 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-XXatuPqWOy9kncv9 p{margin:0;}#mermaid-svg-XXatuPqWOy9kncv9 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-XXatuPqWOy9kncv9 .cluster-label text{fill:#333;}#mermaid-svg-XXatuPqWOy9kncv9 .cluster-label span{color:#333;}#mermaid-svg-XXatuPqWOy9kncv9 .cluster-label span p{background-color:transparent;}#mermaid-svg-XXatuPqWOy9kncv9 .label text,#mermaid-svg-XXatuPqWOy9kncv9 span{fill:#333;color:#333;}#mermaid-svg-XXatuPqWOy9kncv9 .node rect,#mermaid-svg-XXatuPqWOy9kncv9 .node circle,#mermaid-svg-XXatuPqWOy9kncv9 .node ellipse,#mermaid-svg-XXatuPqWOy9kncv9 .node polygon,#mermaid-svg-XXatuPqWOy9kncv9 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-XXatuPqWOy9kncv9 .rough-node .label text,#mermaid-svg-XXatuPqWOy9kncv9 .node .label text,#mermaid-svg-XXatuPqWOy9kncv9 .image-shape .label,#mermaid-svg-XXatuPqWOy9kncv9 .icon-shape .label{text-anchor:middle;}#mermaid-svg-XXatuPqWOy9kncv9 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-XXatuPqWOy9kncv9 .rough-node .label,#mermaid-svg-XXatuPqWOy9kncv9 .node .label,#mermaid-svg-XXatuPqWOy9kncv9 .image-shape .label,#mermaid-svg-XXatuPqWOy9kncv9 .icon-shape .label{text-align:center;}#mermaid-svg-XXatuPqWOy9kncv9 .node.clickable{cursor:pointer;}#mermaid-svg-XXatuPqWOy9kncv9 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-XXatuPqWOy9kncv9 .arrowheadPath{fill:#333333;}#mermaid-svg-XXatuPqWOy9kncv9 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-XXatuPqWOy9kncv9 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-XXatuPqWOy9kncv9 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-XXatuPqWOy9kncv9 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-XXatuPqWOy9kncv9 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-XXatuPqWOy9kncv9 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-XXatuPqWOy9kncv9 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-XXatuPqWOy9kncv9 .cluster text{fill:#333;}#mermaid-svg-XXatuPqWOy9kncv9 .cluster span{color:#333;}#mermaid-svg-XXatuPqWOy9kncv9 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-XXatuPqWOy9kncv9 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-XXatuPqWOy9kncv9 rect.text{fill:none;stroke-width:0;}#mermaid-svg-XXatuPqWOy9kncv9 .icon-shape,#mermaid-svg-XXatuPqWOy9kncv9 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-XXatuPqWOy9kncv9 .icon-shape p,#mermaid-svg-XXatuPqWOy9kncv9 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-XXatuPqWOy9kncv9 .icon-shape .label rect,#mermaid-svg-XXatuPqWOy9kncv9 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-XXatuPqWOy9kncv9 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-XXatuPqWOy9kncv9 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-XXatuPqWOy9kncv9 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-XXatuPqWOy9kncv9 .blue>*{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-XXatuPqWOy9kncv9 .blue span{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-XXatuPqWOy9kncv9 .blue tspan{fill:#fff!important;}#mermaid-svg-XXatuPqWOy9kncv9 .orange>*{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-XXatuPqWOy9kncv9 .orange span{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-XXatuPqWOy9kncv9 .orange tspan{fill:#fff!important;} 0 尽早开始
1 用手动测试
2 无歧义任务
3 平衡问题集
4 稳定环境
5 精心设计评分
6 读轨迹
7 监控饱和
8 长期维护

Step 0：尽早开始。 别以为要几百个任务才配做评测------20~50 个从真实失败里提炼的简单任务就是很好的起点。越晚建，越难建（因为要从一个活系统里反推成功标准）。

Step 1：从你已经在手动测的东西开始。 把每次发版前手动验证的行为、bug 追踪器里的用户报错，转成测试用例。用户真实踩的坑 = 最好的评测素材。

Step 2：写无歧义的任务 + 参考解。 一个好任务的标准是：两个领域专家能独立得出相同的"过/不过"判定 。给每个任务写一个"参考解"（已知能通过所有评分的标准答案），证明任务可解、评分器配置正确。如果模型 0% 通过，多半是任务写坏了，不是模型不行。

Step 3：建平衡的问题集。 正反都要测，否则会"单向优化"。比如只测"该搜索时有没有搜索"，最后会得到一个"啥都去搜"的 Agent；还得测"不该搜索时有没有乱搜"。

Step 4：建稳定的评测环境。 每次试次从干净环境开始，避免共享状态（残留文件、缓存、上次的 git 历史）。共享状态会让失败"相关"（一个环境问题拖垮一批），评测结果就不准。

Step 5：精心设计评分器。 三个要点：① 评结果，别评路径 ------Agent 常能走出一条你没预料到的正确路径，死抠"必须按这个顺序调工具"会把好答案误判成错；② 多组件任务给部分分 （识别了问题但没退款的客服，比啥也没做的强）；③ LLM-as-judge 要给个"不知道"的出口防幻觉，并和人类专家校准。

Step 6：读轨迹（transcript）。 分数不涨时，读轨迹才知道是"Agent 真错了"还是"你的评分器冤枉了它"。失败的判定要"看起来公平"------能说清 Agent 到底错在哪。

Step 7：监控能力评测饱和。 一个评测考到 100% 就只测回归、不给改进信号了。分数不涨 ≠ 没进步------可能是评测饱和了，该换更难的题。

Step 8：长期维护。 评测集是"活物"，要有专人/团队负责基础设施，让领域专家和产品团队贡献任务。把维护评测当成维护单元测试一样日常。

八、评测的闭环：怎么形成持续改进的正循环

把 8 步串起来，就是一个闭环（这也是面试里最能体现"系统性思维"的地方）：
#mermaid-svg-zvyVfIXKtxhu8Hkl{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-zvyVfIXKtxhu8Hkl .error-icon{fill:#552222;}#mermaid-svg-zvyVfIXKtxhu8Hkl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-zvyVfIXKtxhu8Hkl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .marker.cross{stroke:#333333;}#mermaid-svg-zvyVfIXKtxhu8Hkl svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-zvyVfIXKtxhu8Hkl p{margin:0;}#mermaid-svg-zvyVfIXKtxhu8Hkl .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .cluster-label text{fill:#333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .cluster-label span{color:#333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .cluster-label span p{background-color:transparent;}#mermaid-svg-zvyVfIXKtxhu8Hkl .label text,#mermaid-svg-zvyVfIXKtxhu8Hkl span{fill:#333;color:#333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .node rect,#mermaid-svg-zvyVfIXKtxhu8Hkl .node circle,#mermaid-svg-zvyVfIXKtxhu8Hkl .node ellipse,#mermaid-svg-zvyVfIXKtxhu8Hkl .node polygon,#mermaid-svg-zvyVfIXKtxhu8Hkl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-zvyVfIXKtxhu8Hkl .rough-node .label text,#mermaid-svg-zvyVfIXKtxhu8Hkl .node .label text,#mermaid-svg-zvyVfIXKtxhu8Hkl .image-shape .label,#mermaid-svg-zvyVfIXKtxhu8Hkl .icon-shape .label{text-anchor:middle;}#mermaid-svg-zvyVfIXKtxhu8Hkl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-zvyVfIXKtxhu8Hkl .rough-node .label,#mermaid-svg-zvyVfIXKtxhu8Hkl .node .label,#mermaid-svg-zvyVfIXKtxhu8Hkl .image-shape .label,#mermaid-svg-zvyVfIXKtxhu8Hkl .icon-shape .label{text-align:center;}#mermaid-svg-zvyVfIXKtxhu8Hkl .node.clickable{cursor:pointer;}#mermaid-svg-zvyVfIXKtxhu8Hkl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .arrowheadPath{fill:#333333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-zvyVfIXKtxhu8Hkl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-zvyVfIXKtxhu8Hkl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-zvyVfIXKtxhu8Hkl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-zvyVfIXKtxhu8Hkl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-zvyVfIXKtxhu8Hkl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-zvyVfIXKtxhu8Hkl .cluster text{fill:#333;}#mermaid-svg-zvyVfIXKtxhu8Hkl .cluster span{color:#333;}#mermaid-svg-zvyVfIXKtxhu8Hkl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-zvyVfIXKtxhu8Hkl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-zvyVfIXKtxhu8Hkl rect.text{fill:none;stroke-width:0;}#mermaid-svg-zvyVfIXKtxhu8Hkl .icon-shape,#mermaid-svg-zvyVfIXKtxhu8Hkl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-zvyVfIXKtxhu8Hkl .icon-shape p,#mermaid-svg-zvyVfIXKtxhu8Hkl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-zvyVfIXKtxhu8Hkl .icon-shape .label rect,#mermaid-svg-zvyVfIXKtxhu8Hkl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-zvyVfIXKtxhu8Hkl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-zvyVfIXKtxhu8Hkl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-zvyVfIXKtxhu8Hkl :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}#mermaid-svg-zvyVfIXKtxhu8Hkl .blue>*{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .blue span{fill:#4E79A7!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .blue tspan{fill:#fff!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .orange>*{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .orange span{fill:#F28E2B!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .orange tspan{fill:#fff!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .green>*{fill:#59A14F!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .green span{fill:#59A14F!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .green tspan{fill:#fff!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .red>*{fill:#57A0E1!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .red span{fill:#57A0E1!important;color:#fff!important;stroke:none!important;}#mermaid-svg-zvyVfIXKtxhu8Hkl .red tspan{fill:#fff!important;} 定义成功建任务
跑评测拿基线
读轨迹定位失败
改进 prompt 代码
回归防回退
能力饱和换更难

闭环的关键 ：评测不是"做完一次就完了"，而是定义成功 → 建任务 → 跑 → 读轨迹 → 改进 → 再跑 的持续循环。这就是 Anthropic 推荐的 eval-driven development（评测驱动开发） ：先用评测定义"想要的能力"，再迭代直到 Agent 做好------而不是先写功能、再事后补评测。

九、评测不是全部：完整观测体系

面试常被追问："光靠评测够吗？"答案是不够。自动评测只是理解 Agent 表现的一种方法，完整图景还包括：

方法	作用
自动评测	不碰真实用户，可每次提交都跑，可复现
生产监控	线上真实指标（延迟、错误率、成本）
用户反馈	真实用户的主观感受
A/B 测试	新旧版本对照
人工 transcript 审查	定期抽读，发现评测没覆盖的问题

自动评测 + 生产监控 + 用户反馈，三者一起，才构成对 Agent 的完整理解。这句话在面试里说出来，是成熟的体现。

十、面试速记：3 分钟把 Evals 讲清楚

如果面试官问"你们怎么做 Agent 评测"，用这个框架答：

先讲价值：没有评测 = 瞎飞，改完不知道好坏，只能等用户骂。
再讲概念：task（任务）、grader（评分器）、transcript（轨迹）vs outcome（结果）------重点强调"看结果不看嘴说"。
讲三类评分器：代码（客观快）、模型（灵活需校准）、人工（金标准贵慢），按需组合。
讲两种评测：能力评测（低通过率起步、爬坡）vs 回归评测（近 100%、防回退）。
讲非确定性：pass@k（至少一次成功）vs pass^k（全部成功）。
讲闭环：定义成功 → 建任务 → 跑 → 读轨迹 → 改进 → 再跑，eval-driven development。
收尾讲边界：自动评测只是其一，还要配生产监控、用户反馈、A/B。

再备一个具体例子（面试最有说服力）：Opus 4.5 在 CORE-Bench 上 42% → 修评分 bug 后 95%，说明"评分器本身也可能是问题来源"------这个例子能证明你真的理解评测的坑。

总结：一张图带走

这课的核心，一句话串起来：

评测 = 给 AI 出卷子 + 批卷子，是"改完变好还是变坏"的唯一可靠答案。
8 个概念：task / trial / grader / transcript / outcome / harness / suite，重点是 transcript（嘴说）vs outcome（实际状态）。
三类评分器：代码（客观快）、模型（灵活需校准）、人工（金标准贵），按需组合。
两种评测：能力（低通过率爬坡）vs 回归（近 100% 防回退）。
非确定性：pass@k（至少一次成功）vs pass^k（全部成功）。
8 步路线 + 闭环 ：尽早开始 → 无歧义任务 → 平衡问题集 → 稳定环境 → 精心评分 → 读轨迹 → 监控饱和 → 长期维护，形成 eval-driven
development 的正循环。

三个立即可用的行动：

从你手头的 Agent/RAG 项目里，挑 10 个真实失败案例，写成 task（问题 + 成功标准）；
给其中一个 task 写个"参考解"，验证你的评分器配置对不对；
跑两遍同一个任务，观察结果是否一致，感受非确定性，然后决定你的场景该用 pass@k 还是 pass^k。

如果这篇对你有帮助，点个赞、收藏一下，面试前翻出来过一遍"面试速记"那节。

关注我 ，我会持续更新 AI 应用开发的前沿实战，下一篇可以讲评测的具体落地：怎么搭一个能跑、能打分的 eval harness。

你在做 Agent 时，是不是也经历过"改完不知道好坏、只能等用户骂"的瞎飞时刻？评论区聊聊你踩过的评测坑，我挨个回。

（本文核心框架与案例参考 Anthropic 工程团队《Demystifying evals for AI agents》，结合个人实战整理，具体指标与工具随版本变化，评论区提醒我。）