Spring AI Framework (六：「任务编排 + 成本仪表盘 + 轨迹评估」的 Agent 平台）

陈鋆2026-09-17 9:35

关键词：Spring Boot、Spring AI、Agent 编排、SSE、Replan、可观测性、LLM 成本、轨迹评

摘要

大模型应用从「一问一答」走向「自主规划、多步执行」后，工程上真正难的不是调用 LLM，而是**编排（规划/调度/重规划）、可观测（成本/步数/Token）、可评估（回归测试与打分）**这三件事。

本文基于一个真实落地的 Spring Boot + Spring AI 项目，围绕三块能力展开：

  1. Agent 任务编排：同步/异步双通道、SSE 实时轨迹、依赖调度与并行、失败重规划、人工审批挂起与恢复；
  2. Agent 成本仪表盘：从落库轨迹聚合成功率、平均步数、Token 与估算成本，并给出状态分布、步骤类型分布；
  3. Agent 轨迹评估：测试集 / 用例管理 + 批量运行 + 关键词重叠 & 轨迹成功率双维度打分。

一、整体架构设计

1.1 分层架构图

#mermaid-svg-OzAcSoNz9pP5Buoz{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-OzAcSoNz9pP5Buoz .error-icon{fill:#552222;}#mermaid-svg-OzAcSoNz9pP5Buoz .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-OzAcSoNz9pP5Buoz .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-OzAcSoNz9pP5Buoz .marker{fill:#333333;stroke:#333333;}#mermaid-svg-OzAcSoNz9pP5Buoz .marker.cross{stroke:#333333;}#mermaid-svg-OzAcSoNz9pP5Buoz svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-OzAcSoNz9pP5Buoz p{margin:0;}#mermaid-svg-OzAcSoNz9pP5Buoz .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-OzAcSoNz9pP5Buoz .cluster-label text{fill:#333;}#mermaid-svg-OzAcSoNz9pP5Buoz .cluster-label span{color:#333;}#mermaid-svg-OzAcSoNz9pP5Buoz .cluster-label span p{background-color:transparent;}#mermaid-svg-OzAcSoNz9pP5Buoz .label text,#mermaid-svg-OzAcSoNz9pP5Buoz span{fill:#333;color:#333;}#mermaid-svg-OzAcSoNz9pP5Buoz .node rect,#mermaid-svg-OzAcSoNz9pP5Buoz .node circle,#mermaid-svg-OzAcSoNz9pP5Buoz .node ellipse,#mermaid-svg-OzAcSoNz9pP5Buoz .node polygon,#mermaid-svg-OzAcSoNz9pP5Buoz .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-OzAcSoNz9pP5Buoz .rough-node .label text,#mermaid-svg-OzAcSoNz9pP5Buoz .node .label text,#mermaid-svg-OzAcSoNz9pP5Buoz .image-shape .label,#mermaid-svg-OzAcSoNz9pP5Buoz .icon-shape .label{text-anchor:middle;}#mermaid-svg-OzAcSoNz9pP5Buoz .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-OzAcSoNz9pP5Buoz .rough-node .label,#mermaid-svg-OzAcSoNz9pP5Buoz .node .label,#mermaid-svg-OzAcSoNz9pP5Buoz .image-shape .label,#mermaid-svg-OzAcSoNz9pP5Buoz .icon-shape .label{text-align:center;}#mermaid-svg-OzAcSoNz9pP5Buoz .node.clickable{cursor:pointer;}#mermaid-svg-OzAcSoNz9pP5Buoz .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-OzAcSoNz9pP5Buoz .arrowheadPath{fill:#333333;}#mermaid-svg-OzAcSoNz9pP5Buoz .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-OzAcSoNz9pP5Buoz .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-OzAcSoNz9pP5Buoz .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OzAcSoNz9pP5Buoz .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-OzAcSoNz9pP5Buoz .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OzAcSoNz9pP5Buoz .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-OzAcSoNz9pP5Buoz .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-OzAcSoNz9pP5Buoz .cluster text{fill:#333;}#mermaid-svg-OzAcSoNz9pP5Buoz .cluster span{color:#333;}#mermaid-svg-OzAcSoNz9pP5Buoz div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-OzAcSoNz9pP5Buoz .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-OzAcSoNz9pP5Buoz rect.text{fill:none;stroke-width:0;}#mermaid-svg-OzAcSoNz9pP5Buoz .icon-shape,#mermaid-svg-OzAcSoNz9pP5Buoz .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-OzAcSoNz9pP5Buoz .icon-shape p,#mermaid-svg-OzAcSoNz9pP5Buoz .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-OzAcSoNz9pP5Buoz .icon-shape .label rect,#mermaid-svg-OzAcSoNz9pP5Buoz .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-OzAcSoNz9pP5Buoz .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-OzAcSoNz9pP5Buoz .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-OzAcSoNz9pP5Buoz :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 存储与外部
可观测与评估
编排核心（Service）
接口层（Controller）
前端 PortalUI（Vue3 + Element Plus）
Agent 任务
Agent 仪表盘
Agent 轨迹评估
AgentController

/api/agent
AgentEvalController

/api/agent/eval
AgentOrchestratorService

规划→调度→执行→重规划→汇总
AgentPlanner
AgentStepScheduler
AgentStepExecutor

TOOL/RAG/LLM/SPECIALIST/APPROVAL
AgentDecisionEngine
AgentDashboardService
AgentEvalPipelineService
AgentEvalTestSetService
MySQL

agent_task / agent_task_step

agent_eval_*
LLM Provider

(DeepSeek/OpenAI/DashScope...)
RAG 知识库

pgvector

1.2 核心模块职责

模块 类 职责
编排器 AgentOrchestratorService 串联规划→审批→执行→重规划→汇总，管理异步线程与 SSE
规划器 AgentPlanner 将 goal 拆解为带依赖的步骤计划，失败时 replan
调度器 AgentStepScheduler 从 pending 中挑出依赖就绪步骤，识别永久阻塞步
执行器 AgentStepExecutor 按类型分发执行并累计 Token/成本，终态做 aggregate
决策引擎 AgentDecisionEngine 依据步骤结果决定 CONTINUE / REPLAN / STOP
仪表盘 AgentDashboardService 聚合近期任务成本、步数、Token、分布
评估管线 AgentEvalPipelineService 逐用例跑 Agent → 打分 → 落库 Run/RunCase
测试集服务 AgentEvalTestSetService 测试集与用例 CRUD

1.3 技术栈

  • Spring Boot 3.x + Spring AI：LLM 接入与 Chat 抽象
  • MyBatis-Plus + MySQL：任务轨迹、评估数据落库
  • Reactor（Flux）+ SSE：实时事件流推送
  • pgvector：RAG 向量检索
  • Vue 3 + Element Plus + Vite：前端门户

1.4 项目源代码

源代码

二、Agent 任务编排（同步 / 异步）

【配图 1】Agent 任务页

页面展示：左侧「目标输入 + 知识库 + LLM 提供商 + 模型 + 最大步数 + 并行执行/计划审批开关 + 异步提交/同步运行」；中间「近期任务列表」；右侧「最终回答 + 计划（步骤图）+ 步骤结果 + 事件流」。右上角为任务状态标签（如 SUCCEEDED）。

2.1 核心业务代码：统一入口

AgentController 提供同步、异步、SSE、审批恢复四类入口，路径前缀 /api/agent，由 app.agent.enabled 控制装配：

java 复制代码 
@Slf4j
@RestController
@RequestMapping("/api/agent")
@ConditionalOnProperty(prefix = "app.agent", name = "enabled", havingValue = "true", matchIfMissing = true)
public class AgentController {

    private static final MediaType SSE_UTF8 =
            new MediaType("text", "event-stream", StandardCharsets.UTF_8);

    private final AgentOrchestratorService orchestratorService;
    private final SpecialistAgentRegistry specialistRegistry;
    private final AgentDashboardService dashboardService;

    /** 同步执行：阻塞直到完成，适合短任务 / 调试 */
    @PostMapping("/run")
    public AgentRunVo run(@RequestBody AgentRunRequest request) {
        return orchestratorService.run(request);
    }

    /** 异步提交：立即返回 taskId，后台继续执行 */
    @PostMapping("/run/async")
    public AgentAsyncSubmitVo runAsync(@RequestBody AgentRunRequest request) {
        return orchestratorService.submitAsync(request);
    }

    /** 订阅步骤事件流（SSE）：event 名 = 事件类型，data = 完整事件体 */
    @GetMapping(value = "/tasks/{taskId}/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public ResponseEntity<Flux<ServerSentEvent<AgentEvent>>> stream(@PathVariable("taskId") String taskId) {
        Flux<ServerSentEvent<AgentEvent>> flux = orchestratorService.stream(taskId)
                .map(event -> ServerSentEvent.<AgentEvent>builder()
                        .event(event.getType() == null ? "message" : event.getType().name())
                        .data(event)
                        .build());
        // 显式 charset=UTF-8，避免中文事件乱码
        return ResponseEntity.ok().contentType(SSE_UTF8).body(flux);
    }

    /** 人工审批后继续 / 拒绝 WAITING_HUMAN 的任务 */
    @PostMapping("/tasks/{taskId}/resume")
    public AgentRunVo resume(@PathVariable("taskId") String taskId,
                             @RequestBody(required = false) AgentResumeRequest request) {
        return orchestratorService.resume(taskId, request);
    }
}

请求体 AgentRunRequest 承载全部运行参数：

java 复制代码 
@Data
public class AgentRunRequest {
    private String goal;                 // 目标（必填）
    private String provider;             // 提供商，如 deepseek / openai / dashscope
    private String model;                // 模型名，如 deepseek-chat
    private String conversationId;       // 会话 ID（预留）
    private Integer maxSteps;            // 覆盖默认最大步数
    private Long knowledgeBaseId;        // 知识库 ID（启用 RAG 时传入）
    private Integer topK;                // RAG TopK
    private Double similarityThreshold;  // RAG 相似度阈值
    private Boolean requirePlanApproval; // 规划后等待人工审批（建议配异步）
    private Boolean parallelSteps;       // 无依赖步骤是否并行
}

2.2 编排主流程：规划 →（审批）→ 执行 → 汇总

java 复制代码 
public AgentRunVo run(AgentRunRequest request) {
    // asyncMode=false：同步路径，不允许计划审批挂起
    AgentRuntimeState state = createState(request, false);
    AgentRuntimeState.put(state.getContext().getTaskId(), state);
    try {
        runPipeline(state, false); // 核心流水线
        return buildVo(state);
    } finally {
        // 同步任务结束后立即释放内存态，避免泄漏
        AgentRuntimeState.remove(state.getContext().getTaskId());
        eventBus.remove(state.getContext().getTaskId());
    }
}

/** 核心流水线：规划 →（可选审批）→ 执行循环 → 终态汇总 */
private void runPipeline(AgentRuntimeState state, boolean allowApprovalWait) {
    AgentRunContext context = state.getContext();
    String taskId = context.getTaskId();
    try {
        // 1) LLM/规则规划出步骤图
        AgentPlan plan = planner.plan(context);
        state.setCurrentPlan(plan);
        state.setPending(new ArrayList<>(plan.getSteps()));
        taskStore.createRunning(context, plan);           // 落库 RUNNING + 计划
        eventBus.publish(AgentEvent.builder()             // 推送 PLAN_READY
                .type(AgentEventType.PLAN_READY)
                .taskId(taskId)
                .taskStatus(AgentTaskStatus.RUNNING)
                .plan(toPlanVo(plan))
                .message("计划已生成，共 " + plan.getSteps().size() + " 步")
                .build());

        // 2) 异步 + 要求审批：执行前挂起等待人工确认计划
        if (allowApprovalWait && context.isRequirePlanApproval()) {
            waitForHuman(state, "计划待人工审批后继续执行", null);
        }

        // 3) 按依赖调度逐步/批量执行，失败可重规划
        executeLoop(state);

        // 4) 汇总答案、写 metrics、推送 TASK_FINISHED
        finalizeTask(state);
    } catch (CancellationException ex) {      // 人工拒绝 / 审批超时 → CANCELLED
        state.setStatus(AgentTaskStatus.CANCELLED);
        taskStore.finish(taskId, AgentTaskStatus.CANCELLED, "任务已取消：" + ex.getMessage(),
                Map.of("cancelled", true), state.getReplanCount());
    } catch (Exception ex) {                  // 未捕获异常 → FAILED
        state.setStatus(AgentTaskStatus.FAILED);
        taskStore.finish(taskId, AgentTaskStatus.FAILED, ex.getMessage(),
                Map.of("error", String.valueOf(ex.getMessage())), state.getReplanCount());
    }
}

2.3 流程设计图（时序）

EventBus(SSE) MySQL DecisionEngine Executor Scheduler Planner Orchestrator AgentController 前端 EventBus(SSE) MySQL DecisionEngine Executor Scheduler Planner Orchestrator AgentController 前端 #mermaid-svg-5OQyje3vBHxbAVle{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-5OQyje3vBHxbAVle .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-5OQyje3vBHxbAVle .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-5OQyje3vBHxbAVle .error-icon{fill:#552222;}#mermaid-svg-5OQyje3vBHxbAVle .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-5OQyje3vBHxbAVle .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-5OQyje3vBHxbAVle .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-5OQyje3vBHxbAVle .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-5OQyje3vBHxbAVle .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-5OQyje3vBHxbAVle .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-5OQyje3vBHxbAVle .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-5OQyje3vBHxbAVle .marker{fill:#333333;stroke:#333333;}#mermaid-svg-5OQyje3vBHxbAVle .marker.cross{stroke:#333333;}#mermaid-svg-5OQyje3vBHxbAVle svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-5OQyje3vBHxbAVle p{margin:0;}#mermaid-svg-5OQyje3vBHxbAVle .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-5OQyje3vBHxbAVle text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-5OQyje3vBHxbAVle .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-5OQyje3vBHxbAVle .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-5OQyje3vBHxbAVle .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-5OQyje3vBHxbAVle .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-5OQyje3vBHxbAVle #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-5OQyje3vBHxbAVle .sequenceNumber{fill:white;}#mermaid-svg-5OQyje3vBHxbAVle #sequencenumber{fill:#333;}#mermaid-svg-5OQyje3vBHxbAVle #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-5OQyje3vBHxbAVle .messageText{fill:#333;stroke:none;}#mermaid-svg-5OQyje3vBHxbAVle .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-5OQyje3vBHxbAVle .labelText,#mermaid-svg-5OQyje3vBHxbAVle .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-5OQyje3vBHxbAVle .loopText,#mermaid-svg-5OQyje3vBHxbAVle .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-5OQyje3vBHxbAVle .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-5OQyje3vBHxbAVle .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-5OQyje3vBHxbAVle .noteText,#mermaid-svg-5OQyje3vBHxbAVle .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-5OQyje3vBHxbAVle .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-5OQyje3vBHxbAVle .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-5OQyje3vBHxbAVle .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-5OQyje3vBHxbAVle .actorPopupMenu{position:absolute;}#mermaid-svg-5OQyje3vBHxbAVle .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-5OQyje3vBHxbAVle .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-5OQyje3vBHxbAVle .actor-man circle,#mermaid-svg-5OQyje3vBHxbAVle line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-5OQyje3vBHxbAVle :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} alt 需要重规划 loop 直到 pending 为空或达 maxSteps POST /api/agent/run/async 1 submitAsync(request) 2 taskId + streamUrl（立即返回） 3 EventSource 订阅 /tasks/{id}/stream 4 createRunning(taskId, plan) 5 plan(context) 6 AgentPlan(步骤图) 7 PLAN_READY 8 nextReady(pending, results) 9 ready 批次 10 STEP_STARTED 11 executeBatch(context, batch) 12 List<AgentStepResult> 13 saveStep(...) 14 STEP_FINISHED 15 onStepResult(result, replanCount) 16 REPLAN 17 replan(...) 18 REPLAN 19 aggregate(context, results) 20 最终回答 21 finish(status, answer, metrics) 22 TASK_FINISHED 23 最终回答 + 轨迹 24

2.4 人工审批：挂起与恢复

审批通过内存门闩 CompletableFuture 实现挂起/唤醒，最长等待 2 小时：

java 复制代码 
/** 进入人工审批挂起：写 WAITING_HUMAN、发事件，阻塞等待 resume 完成门闩 */
private void waitForHuman(AgentRuntimeState state, String reason, String stepId) throws Exception {
    CompletableFuture<AgentResumeRequest> gate = new CompletableFuture<>();
    state.setApprovalGate(gate);
    state.setStatus(AgentTaskStatus.WAITING_HUMAN);
    taskStore.updateStatus(state.getContext().getTaskId(), AgentTaskStatus.WAITING_HUMAN);
    eventBus.publish(AgentEvent.builder()
            .type(AgentEventType.WAITING_HUMAN)
            .taskId(state.getContext().getTaskId())
            .taskStatus(AgentTaskStatus.WAITING_HUMAN)
            .stepId(stepId)
            .message(reason)
            .build());

    AgentResumeRequest signal;
    try {
        signal = gate.get(2, TimeUnit.HOURS);      // 阻塞等待人工信号
    } catch (TimeoutException ex) {
        throw new CancellationException("人工审批超时");
    }
    if (signal == null || !signal.isApproved()) {  // 拒绝 → 取消整条流水线
        throw new CancellationException("人工拒绝继续执行");
    }
    state.setStatus(AgentTaskStatus.RUNNING);       // 批准 → 恢复 RUNNING
    state.setApprovalGate(null);
    taskStore.updateStatus(state.getContext().getTaskId(), AgentTaskStatus.RUNNING);
}

/** resume：唤醒被阻塞的流水线线程 */
public AgentRunVo resume(String taskId, AgentResumeRequest resumeRequest) {
    AgentRuntimeState state = AgentRuntimeState.get(taskId.trim());
    if (state == null || state.getStatus() != AgentTaskStatus.WAITING_HUMAN) {
        throw new IllegalStateException("任务不在等待审批状态: " + taskId);
    }
    CompletableFuture<AgentResumeRequest> gate = state.getApprovalGate();
    gate.complete(resumeRequest == null ? new AgentResumeRequest() : resumeRequest);
    return getTask(taskId).orElseGet(() -> AgentRunVo.builder()
            .taskId(taskId).status(AgentTaskStatus.WAITING_HUMAN).build());
}

2.5 SSE 事件类型

事件 含义 关键 payload
TASK_STARTED 任务提交 / 审批后恢复 taskId、status
PLAN_READY 首次规划完成 plan
STEP_STARTED 一批就绪步骤开始执行 readyStepIds
STEP_FINISHED 单步结束（成功/失败/跳过） step
REPLAN 步骤失败触发重规划 plan、replanCount
WAITING_HUMAN 进入人工审批挂起 stepId、message
TASK_FINISHED 到达终态 answer、replanCount
ERROR 流水线异常 message

三、执行循环、并行调度与重规划

【配图 1 局部】计划 / 步骤结果 / 事件流

截图右侧的「计划」展示步骤图（s1: TOOL → s2: LLM），「步骤结果」逐条展示 SUCCEEDED，「事件流」实时滚动 STEP_STARTED / STEP_FINISHED / PLAN_READY。

3.1 核心业务代码：执行循环

java 复制代码 
/** 执行循环：跳过永久阻塞步 → 取就绪批 → 执行 → 审批挂起/落库 → 按需重规划 */
private void executeLoop(AgentRuntimeState state) throws Exception {
    AgentRunContext context = state.getContext();
    List<AgentPlanStep> pending = state.getPending();
    List<AgentStepResult> results = state.getResults();

    while (!pending.isEmpty() && results.size() < context.getMaxSteps()) {
        // 依赖已失败且无法满足 → 标记 SKIPPED，避免死等
        for (AgentPlanStep step : stepScheduler.blockedForever(pending, results)) {
            pending.remove(step);
            AgentStepResult skipped = AgentStepResult.builder()
                    .id(step.getId()).type(step.getType())
                    .status(AgentStepStatus.SKIPPED)
                    .error("依赖步骤失败，已跳过").durationMs(0).build();
            results.add(skipped);
            taskStore.saveStep(context.getTaskId(), step, skipped, state.getStepOrder() + 1);
            emitStepFinished(context.getTaskId(), skipped);
        }

        // 选出依赖全部成功的就绪步骤
        List<AgentPlanStep> ready = stepScheduler.nextReady(pending, results);
        if (ready.isEmpty()) break; // 仍有 pending 却无就绪 → 依赖环，强制退出

        // 开启并行且就绪数 > 1 → 整批执行，否则只跑第一个
        List<AgentPlanStep> batch =
                context.isParallelSteps() && ready.size() > 1 ? ready : List.of(ready.get(0));
        pending.removeAll(batch);

        eventBus.publish(AgentEvent.builder()
                .type(AgentEventType.STEP_STARTED)
                .taskId(context.getTaskId())
                .readyStepIds(batch.stream().map(AgentPlanStep::getId).toList())
                .message("开始执行 " + batch.size() + " 个步骤").build());

        List<AgentStepResult> batchResults = executeBatch(context, batch, results);

        boolean needReplan = false;
        AgentStepResult failedForReplan = null;
        for (int i = 0; i < batch.size(); i++) {
            AgentPlanStep step = batch.get(i);
            AgentStepResult result = batchResults.get(i);

            // APPROVAL 步骤先以 WAITING 占位，再进入人工门闩；批准后改写为成功
            if (step.getType() == AgentStepType.APPROVAL
                    && result.getStatus() == AgentStepStatus.SUCCEEDED
                    && "WAITING".equals(result.getOutput())) {
                waitForHuman(state, step.getInstruction(), step.getId());
                result = AgentStepResult.builder()
                        .id(step.getId()).type(AgentStepType.APPROVAL)
                        .status(AgentStepStatus.SUCCEEDED)
                        .output("人工已批准").durationMs(result.getDurationMs()).build();
            }

            results.add(result);
            taskStore.saveStep(context.getTaskId(), step, result, state.getStepOrder() + 1);
            emitStepFinished(context.getTaskId(), result);

            // 决策引擎：失败且未超重规划次数时触发 REPLAN
            AgentDecision decision = decisionEngine.onStepResult(result, state.getReplanCount());
            if (decision == AgentDecision.REPLAN) { needReplan = true; failedForReplan = result; }
        }

        if (needReplan && failedForReplan != null) {
            state.setReplanCount(state.getReplanCount() + 1);
            int remainingBudget = context.getMaxSteps() - results.size();
            if (remainingBudget <= 0) break; // 步数预算耗尽，不再重规划

            AgentPlan replanned = planner.replan(context, results, failedForReplan, remainingBudget);
            AgentPlan merged = mergePlan(context.getGoal(), results, replanned);
            state.setCurrentPlan(merged);
            pending.clear();
            pending.addAll(replanned.getSteps());
            taskStore.updatePlan(context.getTaskId(), merged, state.getReplanCount());
            eventBus.publish(AgentEvent.builder()
                    .type(AgentEventType.REPLAN)
                    .taskId(context.getTaskId())
                    .replanCount(state.getReplanCount())
                    .plan(toPlanVo(merged))
                    .message("已重规划，剩余 " + pending.size() + " 步").build());
        }
    }
    state.setPending(pending);
}

3.2 核心业务代码：批量执行与步骤分发

java 复制代码 
/** 执行一批就绪步骤：单步同步；多步并行（单步超时 10 分钟） */
private List<AgentStepResult> executeBatch(
        AgentRunContext context, List<AgentPlanStep> batch, List<AgentStepResult> priorResults) {
    if (batch.size() == 1) {   // 单步快路径：避免无必要的线程切换
        AgentPlanStep step = batch.get(0);
        if (step.getType() == AgentStepType.APPROVAL) {
            return List.of(AgentStepResult.builder()
                    .id(step.getId()).type(AgentStepType.APPROVAL)
                    .status(AgentStepStatus.SUCCEEDED).output("WAITING").durationMs(0).build());
        }
        return List.of(stepExecutor.execute(context, step, priorResults));
    }

    // 并行：快照 prior，防止各线程看到不一致的 results 视图
    List<AgentStepResult> priorSnapshot = List.copyOf(priorResults);
    List<CompletableFuture<AgentStepResult>> futures = new ArrayList<>();
    for (AgentPlanStep step : batch) {
        if (step.getType() == AgentStepType.APPROVAL) {
            futures.add(CompletableFuture.completedFuture(/* WAITING 占位 */));
        } else {
            futures.add(CompletableFuture.supplyAsync(
                    () -> stepExecutor.execute(context, step, priorSnapshot), agentParallelExecutor));
        }
    }
    List<AgentStepResult> out = new ArrayList<>(futures.size());
    for (int i = 0; i < futures.size(); i++) {
        AgentPlanStep step = batch.get(i);
        try {
            out.add(futures.get(i).get(10, TimeUnit.MINUTES));
        } catch (Exception ex) {   // 超时/异常 → FAILED，保证 batch 与 out 对齐
            out.add(AgentStepResult.builder()
                    .id(step.getId()).type(step.getType())
                    .status(AgentStepStatus.FAILED)
                    .error(ex.getMessage()).durationMs(0).build());
        }
    }
    return out;
}

单步执行器按类型分发，并统一做异常兜底：

java 复制代码 
public AgentStepResult execute(AgentRunContext context, AgentPlanStep step,
                               List<AgentStepResult> priorResults) {
    long started = System.currentTimeMillis();
    try {
        return switch (step.getType()) {
            case RAG        -> success(context, step, null, executeRag(context, step), started, null);
            case TOOL       -> executeLlmStep(context, step, priorResults, HarnessProfile.AGENT_TOOL, started);
            case LLM        -> executeLlmStep(context, step, priorResults, HarnessProfile.AGENT_STEP, started);
            case SPECIALIST -> executeSpecialist(context, step, priorResults, started);
            case APPROVAL   -> throw new IllegalStateException("APPROVAL 步骤应由编排层处理");
        };
    } catch (Exception ex) {
        return AgentStepResult.builder()
                .id(step.getId()).type(step.getType())
                .status(AgentStepStatus.FAILED)
                .error(ex.getMessage())
                .durationMs(System.currentTimeMillis() - started).build();
    }
}

四、Agent 成本仪表盘

【配图 2】Agent 仪表盘

页面展示四张指标卡（采样任务、成功/失败/待审批、平均步数/Token、估算总成本 USD）＋ 状态分布、步骤类型分布、近期任务表格（Task / 状态 / 目标 / 步数 / Tokens / 成本 USD / 耗时 ms / 创建时间）。

4.1 核心业务代码：窗口聚合

仪表盘直接从落库的 agent_task / agent_task_step 读取近期样本做聚合，limit 被钳制在 1~200：

java 复制代码 
public AgentDashboardSummaryVo summary(int limit) {
    int resolved = Math.min(Math.max(limit, 1), 200);   // 防止 limit 过大拖垮 DB
    List<AgentTaskEntity> tasks = taskMapper.selectList(
            new LambdaQueryWrapper<AgentTaskEntity>()
                    .orderByDesc(AgentTaskEntity::getCreatedAt)
                    .last("LIMIT " + resolved));

    long totalTasks = tasks.size(), succeeded = 0, failed = 0, waiting = 0;
    double costSum = 0; long tokenSum = 0, stepSum = 0, elapsedSum = 0;
    Map<String, Long> statusDist = new LinkedHashMap<>();   // 保持首次出现顺序
    Map<String, Long> stepTypeDist = new HashMap<>();
    List<AgentDashboardTaskVo> recent = new ArrayList<>();

    for (AgentTaskEntity task : tasks) {
        statusDist.merge(task.getStatus() == null ? "UNKNOWN" : task.getStatus(), 1L, Long::sum);
        if ("SUCCEEDED".equals(task.getStatus())) succeeded++;
        else if ("FAILED".equals(task.getStatus()) || "CANCELLED".equals(task.getStatus())) failed++;
        else if ("WAITING_HUMAN".equals(task.getStatus())) waiting++;

        // metrics 在 finalizeTask 时写入；解析失败按 0 处理
        Map<String, Object> metrics = parseMetrics(task.getMetricsJson());
        double cost = asDouble(metrics.get("estimatedCostUsd"));
        long tokens = asLong(metrics.get("totalTokens"));
        long steps  = asLong(metrics.get("totalSteps"));
        long elapsed = asLong(metrics.get("elapsedMs"));
        costSum += cost; tokenSum += tokens; stepSum += steps; elapsedSum += elapsed;

        recent.add(AgentDashboardTaskVo.builder()
                .taskId(task.getTaskId()).goal(task.getGoal()).status(task.getStatus())
                .provider(task.getProvider()).model(task.getModel())
                .totalSteps((int) steps).totalTokens(tokens)
                .estimatedCostUsd(cost).elapsedMs(elapsed)
                .replanCount(task.getReplanCount())
                .createdAt(task.getCreatedAt() == null ? null : task.getCreatedAt().toString())
                .build());
    }

    // 步骤样本：约为任务数×8，且不超过 1000，用于类型分布
    List<AgentTaskStepEntity> steps = stepMapper.selectList(
            new LambdaQueryWrapper<AgentTaskStepEntity>()
                    .orderByDesc(AgentTaskStepEntity::getId)
                    .last("LIMIT " + Math.min(resolved * 8, 1000)));
    for (AgentTaskStepEntity step : steps) {
        stepTypeDist.merge(step.getStepType() == null ? "UNKNOWN" : step.getStepType(), 1L, Long::sum);
    }

    return AgentDashboardSummaryVo.builder()
            .sampleSize(totalTasks).totalTasks(totalTasks)
            .succeededTasks(succeeded).failedTasks(failed).waitingHumanTasks(waiting)
            .avgSteps(totalTasks == 0 ? 0 : (double) stepSum / totalTasks)
            .avgTokens(totalTasks == 0 ? 0 : (double) tokenSum / totalTasks)
            .avgCostUsd(totalTasks == 0 ? 0 : costSum / totalTasks)
            .totalEstimatedCostUsd(round6(costSum))
            .totalTokens(tokenSum)
            .avgElapsedMs(totalTasks == 0 ? 0 : (double) elapsedSum / totalTasks)
            .statusDistribution(statusDist)
            .stepTypeDistribution(stepTypeDist)
            .recentTasks(recent)
            .build();
}

/** 解析 metrics_json；空或非法 JSON 返回空 Map，避免打断整页汇总 */
private Map<String, Object> parseMetrics(String json) {
    if (json == null || json.isBlank()) return Map.of();
    try {
        return objectMapper.readValue(json, Map.class);
    } catch (Exception ex) {
        return Map.of();
    }
}

4.2 指标口径说明

指标 来源 说明
succeededTasks agent_task.status SUCCEEDED
failedTasks agent_task.status FAILED 与 CANCELLED 合并计入
waitingHumanTasks agent_task.status WAITING_HUMAN
avgSteps metrics.totalSteps 步数合计 / 任务数
avgTokens metrics.totalTokens Token 合计 / 任务数
totalEstimatedCostUsd metrics.estimatedCostUsd 保留 6 位小数
statusDistribution agent_task.status 状态 → 次数（含 UNKNOWN）
stepTypeDistribution agent_task_step.step_type 独立步骤窗口，最多 1000 条

五、Agent 轨迹评估

评估模块提供「测试集 / 用例 / 运行评估 / 历史 Run」四步闭环，路径前缀 /api/agent/eval，由 app.agent.eval.enabled 控制装配。

5.1 测试集管理

【配图 3】测试集 Tab

表格列：ID / 名称 / 描述 / 知识库 / 用例数 / 操作；右上角「+ 新建测试集」。示例：智慧城市（绑定 #1，3 个用例）、LLM测试集（2 个用例）。

java 复制代码 
@RestController
@RequestMapping("/api/agent/eval")
@ConditionalOnProperty(prefix = "app.agent.eval", name = "enabled", havingValue = "true", matchIfMissing = true)
public class AgentEvalController {

    private final AgentEvalTestSetService testSetService;
    private final AgentEvalPipelineService pipelineService;

    /** 创建评估测试集（可绑定知识库） */
    @PostMapping("/test-sets")
    public AgentEvalTestSetVo createTestSet(@RequestBody AgentEvalTestSetCreateRequest request) {
        return testSetService.create(request);
    }

    /** 列出全部测试集 */
    @GetMapping("/test-sets")
    public Map<String, Object> listTestSets() {
        List<AgentEvalTestSetVo> list = testSetService.list();
        return Map.of("testSets", list, "total", list.size());
    }
}
java 复制代码 
@Transactional
public AgentEvalTestSetVo create(AgentEvalTestSetCreateRequest request) {
    if (request.getName() == null || request.getName().isBlank()) {
        throw new IllegalArgumentException("name 不能为空");
    }
    LocalDateTime now = LocalDateTime.now();
    AgentEvalTestSetEntity entity = new AgentEvalTestSetEntity();
    entity.setName(request.getName().trim());
    entity.setDescription(request.getDescription());
    entity.setKnowledgeBaseId(request.getKnowledgeBaseId());  // 绑定知识库
    entity.setEnabled(true);
    entity.setCaseCount(0);
    entity.setCreatedAt(now);
    entity.setUpdatedAt(now);
    testSetMapper.insert(entity);
    return toSetVo(entity);
}

5.2 用例管理

LLM测试集 2 条

【配图 4】用例管理 Tab

展示「LLM测试集 (#1)（2 条）」用例表：目标 Goal / 期望答案 / 标签 / 操作（编辑、删除），右上角「+ 添加用例」。

java 复制代码 
/** 新增评估用例：sortOrder 缺省时自动追加到末尾 */
@Transactional
public AgentEvalTestCaseVo addCase(Long testSetId, AgentEvalTestCaseCreateRequest request) {
    requireTestSet(testSetId);
    if (request.getGoal() == null || request.getGoal().isBlank()) {
        throw new IllegalArgumentException("goal 不能为空");
    }
    AgentEvalTestCaseEntity entity = new AgentEvalTestCaseEntity();
    entity.setTestSetId(testSetId);
    entity.setGoal(request.getGoal().trim());
    entity.setExpectedAnswer(request.getExpectedAnswer());
    entity.setTags(request.getTags());
    if (request.getSortOrder() != null) {
        entity.setSortOrder(request.getSortOrder());
    } else {
        Long count = testCaseMapper.selectCount(new LambdaQueryWrapper<AgentEvalTestCaseEntity>()
                .eq(AgentEvalTestCaseEntity::getTestSetId, testSetId));
        entity.setSortOrder(count == null ? 1 : count.intValue() + 1);
    }
    entity.setEnabled(true);
    entity.setCreatedAt(LocalDateTime.now());
    entity.setUpdatedAt(LocalDateTime.now());
    testCaseMapper.insert(entity);
    refreshCaseCount(testSetId);   // 同步测试集用例数
    return toCaseVo(entity);
}

5.3 运行评估

【配图 5】运行评估 Tab

左侧配置：测试集、提供商、模型、最大步数、并行、评估维度（关键词重叠 / 轨迹成功率）、关键词阈值；右侧 Run 汇总卡（Run ID / 状态 / 通过率 / 均关键词分 / 均轨迹分 / 均成本 USD / 均步数）＋ 各用例结果表（PASS / Goal / 关键词 / 轨迹 / 步数）。

评估维度与打分公式：

  • 关键词重叠分 keywordScore：RagEvalKeywordScorer.score(expectedAnswer, actualAnswer)，衡量答案与期望答案的关键词覆盖度；
  • 轨迹成功率 trajectoryScore：succeededSteps / totalSteps，衡量执行路径的稳健性；
  • 用例通过判定 ：关键词分 ≥ 阈值 且 轨迹分 ≥ 0.5 且 任务状态非 FAILED。
java 复制代码 
@Transactional
public AgentEvalRunVo run(AgentEvalRunRequest request) {
    AgentEvalTestSetEntity testSet = testSetService.requireTestSet(request.getTestSetId());
    List<AgentEvalTestCaseEntity> cases = testSetService.listEnabledCases(testSet.getId());
    if (cases.isEmpty()) throw new IllegalArgumentException("测试集没有启用的用例: " + testSet.getId());

    AgentProperties.Eval evalCfg = agentProperties.getEval();
    boolean enableKeyword = request.getEnableKeywordOverlap() != null
            ? request.getEnableKeywordOverlap() : evalCfg.isEnableKeywordOverlap();
    boolean enableTrajectory = request.getEnableTrajectoryScore() != null
            ? request.getEnableTrajectoryScore() : evalCfg.isEnableTrajectoryScore();
    double keywordThreshold = request.getKeywordOverlapThreshold() != null
            ? request.getKeywordOverlapThreshold() : evalCfg.getKeywordOverlapThreshold();

    AgentEvalRunEntity run = new AgentEvalRunEntity();
    run.setTestSetId(testSet.getId());
    run.setStatus("RUNNING");
    run.setTotalCases(cases.size());
    run.setPassedCases(0);
    run.setStartedAt(LocalDateTime.now());
    runMapper.insert(run);

    int passed = 0; double keywordSum = 0, trajSum = 0, costSum = 0, stepsSum = 0;
    int keywordCount = 0, trajCount = 0;
    List<AgentEvalRunCaseVo> caseVos = new ArrayList<>();

    for (AgentEvalTestCaseEntity testCase : cases) {
        AgentEvalRunCaseEntity caseEntity = evaluateOne(run.getId(), testSet, testCase, request,
                enableKeyword, enableTrajectory, keywordThreshold);
        runCaseMapper.insert(caseEntity);
        if ("PASS".equals(caseEntity.getStatus())) passed++;
        if (caseEntity.getKeywordScore() != null) { keywordSum += caseEntity.getKeywordScore(); keywordCount++; }
        if (caseEntity.getTrajectoryScore() != null) { trajSum += caseEntity.getTrajectoryScore(); trajCount++; }
        if (caseEntity.getEstimatedCostUsd() != null) costSum += caseEntity.getEstimatedCostUsd();
        if (caseEntity.getTotalSteps() != null) stepsSum += caseEntity.getTotalSteps();
        caseVos.add(toCaseVo(caseEntity));
    }

    run.setPassedCases(passed);
    run.setAvgKeywordScore(keywordCount == 0 ? null : keywordSum / keywordCount);
    run.setAvgTrajectoryScore(trajCount == 0 ? null : trajSum / trajCount);
    run.setAvgCostUsd(cases.isEmpty() ? null : costSum / cases.size());
    run.setAvgSteps(cases.isEmpty() ? null : stepsSum / cases.size());
    run.setStatus("COMPLETED");
    run.setFinishedAt(LocalDateTime.now());
    runMapper.updateById(run);
    return toRunVo(run, caseVos);
}

单用例评估：调用 Agent 后提取指标并双维度打分：

java 复制代码 
private AgentEvalRunCaseEntity evaluateOne(Long runId, AgentEvalTestSetEntity testSet,
        AgentEvalTestCaseEntity testCase, AgentEvalRunRequest request,
        boolean enableKeyword, boolean enableTrajectory, double keywordThreshold) {
    AgentEvalRunCaseEntity entity = new AgentEvalRunCaseEntity();
    entity.setRunId(runId);
    entity.setTestCaseId(testCase.getId());
    entity.setGoal(testCase.getGoal());
    entity.setExpectedAnswer(testCase.getExpectedAnswer());
    long started = System.currentTimeMillis();
    try {
        // 用例直接复用编排器：同步跑一次 Agent
        AgentRunRequest runRequest = new AgentRunRequest();
        runRequest.setGoal(testCase.getGoal());
        runRequest.setProvider(request.getProvider());
        runRequest.setModel(request.getModel());
        runRequest.setMaxSteps(request.getMaxSteps());
        runRequest.setParallelSteps(request.getParallelSteps());
        runRequest.setKnowledgeBaseId(testSet.getKnowledgeBaseId());
        runRequest.setRequirePlanApproval(false);

        AgentRunVo result = orchestratorService.run(runRequest);
        entity.setAgentTaskId(result.getTaskId());
        entity.setActualAnswer(result.getAnswer());
        entity.setLatencyMs(System.currentTimeMillis() - started);

        Map<String, Object> metrics = result.getMetrics() == null ? Map.of() : result.getMetrics();
        entity.setTotalSteps(asInt(metrics.get("totalSteps")));
        entity.setSucceededSteps(asInt(metrics.get("succeededSteps")));
        entity.setEstimatedCostUsd(asDouble(metrics.get("estimatedCostUsd")));
        entity.setTotalTokens(asLong(metrics.get("totalTokens")));

        // 维度 1：关键词重叠
        Double keywordScore = enableKeyword
                ? keywordScorer.score(testCase.getExpectedAnswer(), result.getAnswer()) : null;
        entity.setKeywordScore(keywordScore);

        // 维度 2：轨迹成功率 = 成功步数 / 总步数
        Double trajectoryScore = null;
        if (enableTrajectory && entity.getTotalSteps() != null && entity.getTotalSteps() > 0) {
            int succeeded = entity.getSucceededSteps() == null ? 0 : entity.getSucceededSteps();
            trajectoryScore = (double) succeeded / entity.getTotalSteps();
        }
        entity.setTrajectoryScore(trajectoryScore);

        // 通过判定
        boolean pass = true;
        if (enableKeyword && keywordScore != null) pass = pass && keywordScorer.passes(keywordScore, keywordThreshold);
        if (enableTrajectory && trajectoryScore != null) pass = pass && trajectoryScore >= 0.5;
        if ("FAILED".equals(String.valueOf(result.getStatus()))) pass = false;
        entity.setStatus(pass ? "PASS" : "FAIL");
    } catch (Exception ex) {
        entity.setStatus("ERROR");
        entity.setErrorMessage(ex.getMessage());
        entity.setLatencyMs(System.currentTimeMillis() - started);
    }
    return entity;
}

5.4 评估运行流程设计图

#mermaid-svg-M8zDE2f0h9avUx8x{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-M8zDE2f0h9avUx8x .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-M8zDE2f0h9avUx8x .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-M8zDE2f0h9avUx8x .error-icon{fill:#552222;}#mermaid-svg-M8zDE2f0h9avUx8x .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-M8zDE2f0h9avUx8x .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-M8zDE2f0h9avUx8x .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-M8zDE2f0h9avUx8x .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-M8zDE2f0h9avUx8x .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-M8zDE2f0h9avUx8x .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-M8zDE2f0h9avUx8x .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-M8zDE2f0h9avUx8x .marker{fill:#333333;stroke:#333333;}#mermaid-svg-M8zDE2f0h9avUx8x .marker.cross{stroke:#333333;}#mermaid-svg-M8zDE2f0h9avUx8x svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-M8zDE2f0h9avUx8x p{margin:0;}#mermaid-svg-M8zDE2f0h9avUx8x .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-M8zDE2f0h9avUx8x .cluster-label text{fill:#333;}#mermaid-svg-M8zDE2f0h9avUx8x .cluster-label span{color:#333;}#mermaid-svg-M8zDE2f0h9avUx8x .cluster-label span p{background-color:transparent;}#mermaid-svg-M8zDE2f0h9avUx8x .label text,#mermaid-svg-M8zDE2f0h9avUx8x span{fill:#333;color:#333;}#mermaid-svg-M8zDE2f0h9avUx8x .node rect,#mermaid-svg-M8zDE2f0h9avUx8x .node circle,#mermaid-svg-M8zDE2f0h9avUx8x .node ellipse,#mermaid-svg-M8zDE2f0h9avUx8x .node polygon,#mermaid-svg-M8zDE2f0h9avUx8x .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-M8zDE2f0h9avUx8x .rough-node .label text,#mermaid-svg-M8zDE2f0h9avUx8x .node .label text,#mermaid-svg-M8zDE2f0h9avUx8x .image-shape .label,#mermaid-svg-M8zDE2f0h9avUx8x .icon-shape .label{text-anchor:middle;}#mermaid-svg-M8zDE2f0h9avUx8x .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-M8zDE2f0h9avUx8x .rough-node .label,#mermaid-svg-M8zDE2f0h9avUx8x .node .label,#mermaid-svg-M8zDE2f0h9avUx8x .image-shape .label,#mermaid-svg-M8zDE2f0h9avUx8x .icon-shape .label{text-align:center;}#mermaid-svg-M8zDE2f0h9avUx8x .node.clickable{cursor:pointer;}#mermaid-svg-M8zDE2f0h9avUx8x .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-M8zDE2f0h9avUx8x .arrowheadPath{fill:#333333;}#mermaid-svg-M8zDE2f0h9avUx8x .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-M8zDE2f0h9avUx8x .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-M8zDE2f0h9avUx8x .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-M8zDE2f0h9avUx8x .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-M8zDE2f0h9avUx8x .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-M8zDE2f0h9avUx8x .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-M8zDE2f0h9avUx8x .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-M8zDE2f0h9avUx8x .cluster text{fill:#333;}#mermaid-svg-M8zDE2f0h9avUx8x .cluster span{color:#333;}#mermaid-svg-M8zDE2f0h9avUx8x div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-M8zDE2f0h9avUx8x .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-M8zDE2f0h9avUx8x rect.text{fill:none;stroke-width:0;}#mermaid-svg-M8zDE2f0h9avUx8x .icon-shape,#mermaid-svg-M8zDE2f0h9avUx8x .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-M8zDE2f0h9avUx8x .icon-shape p,#mermaid-svg-M8zDE2f0h9avUx8x .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-M8zDE2f0h9avUx8x .icon-shape .label rect,#mermaid-svg-M8zDE2f0h9avUx8x .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-M8zDE2f0h9avUx8x .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-M8zDE2f0h9avUx8x .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-M8zDE2f0h9avUx8x :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 是
否
keyword≥阈值 且 traj≥0.5 且 非FAILED
否则
是
否
POST /api/agent/eval/runs (testSetId, provider, model, 维度开关, 阈值)
requireTestSet + listEnabledCases
用例为空?
抛错：测试集没有启用的用例
insert AgentEvalRun(status=RUNNING)
遍历每个用例
orchestratorService.run(goal, kbId)
读取 metrics：steps / succeeded / cost / tokens
keywordScore = 期望 vs 实际 关键词重叠
trajectoryScore = succeededSteps / totalSteps
通过判定
PASS
FAIL / ERROR
insert RunCase + 累加统计
还有用例?
更新 Run：passRate / 均分 / 均成本 / 状态 COMPLETED
返回 Run 汇总 VO

5.5 历史 Run

【配图 6】历史 Run Tab

表格列：ID / 测试集 / 状态 / 通过率 / 均成本 / 开始时间；支持「刷新」。示例多次 Run 均为 COMPLETED、通过率 100.0%。

java 复制代码 
/** 列出评估 Run；可按测试集过滤 */
@GetMapping("/runs")
public Map<String, Object> listRuns(@RequestParam(value = "testSetId", required = false) Long testSetId) {
    List<AgentEvalRunVo> list = pipelineService.listRuns(testSetId);
    return Map.of("runs", list, "total", list.size());
}

/** 查询单次 Run 详情（含各用例结果） */
@GetMapping("/runs/{id}")
public AgentEvalRunVo getRun(@PathVariable("id") Long id) {
    return pipelineService.getRun(id);
}

5.6 评估全链路数据模型

#mermaid-svg-ouope1Prvha6zOoT{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ouope1Prvha6zOoT .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ouope1Prvha6zOoT .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ouope1Prvha6zOoT .error-icon{fill:#552222;}#mermaid-svg-ouope1Prvha6zOoT .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ouope1Prvha6zOoT .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ouope1Prvha6zOoT .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ouope1Prvha6zOoT .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ouope1Prvha6zOoT .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ouope1Prvha6zOoT .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ouope1Prvha6zOoT .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ouope1Prvha6zOoT .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ouope1Prvha6zOoT .marker.cross{stroke:#333333;}#mermaid-svg-ouope1Prvha6zOoT svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ouope1Prvha6zOoT p{margin:0;}#mermaid-svg-ouope1Prvha6zOoT .entityBox{fill:#ECECFF;stroke:#9370DB;}#mermaid-svg-ouope1Prvha6zOoT .relationshipLabelBox{fill:hsl(80, 100%, 96.2745098039%);opacity:0.7;background-color:hsl(80, 100%, 96.2745098039%);}#mermaid-svg-ouope1Prvha6zOoT .relationshipLabelBox rect{opacity:0.5;}#mermaid-svg-ouope1Prvha6zOoT .labelBkg{background-color:rgba(248.6666666666, 255, 235.9999999999, 0.5);}#mermaid-svg-ouope1Prvha6zOoT .edgeLabel .label{fill:#9370DB;font-size:14px;}#mermaid-svg-ouope1Prvha6zOoT .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ouope1Prvha6zOoT .edge-pattern-dashed{stroke-dasharray:8,8;}#mermaid-svg-ouope1Prvha6zOoT .node rect,#mermaid-svg-ouope1Prvha6zOoT .node circle,#mermaid-svg-ouope1Prvha6zOoT .node ellipse,#mermaid-svg-ouope1Prvha6zOoT .node polygon{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ouope1Prvha6zOoT .relationshipLine{stroke:#333333;stroke-width:1;fill:none;}#mermaid-svg-ouope1Prvha6zOoT .marker{fill:none!important;stroke:#333333!important;stroke-width:1;}#mermaid-svg-ouope1Prvha6zOoT .edgeLabel{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ouope1Prvha6zOoT .edgeLabel .label rect{fill:rgba(232,232,232, 0.8);}#mermaid-svg-ouope1Prvha6zOoT .edgeLabel .label text{fill:#333;}#mermaid-svg-ouope1Prvha6zOoT :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 包含
发起
逐用例结果
复用编排器执行
AGENT_EVAL_TEST_SET
long
id
PK
string
name
long
knowledge_base_id
int
case_count
AGENT_EVAL_TEST_CASE
long
id
PK
long
test_set_id
FK
string
goal
string
expected_answer
string
tags
AGENT_EVAL_RUN
long
id
PK
long
test_set_id
FK
string
status
int
total_cases
int
passed_cases
double
avg_keyword_score
double
avg_trajectory_score
double
avg_cost_usd
AGENT_EVAL_RUN_CASE
long
id
PK
long
run_id
FK
string
agent_task_id
string
status
double
keyword_score
double
trajectory_score
int
total_steps
int
succeeded_steps
AGENT_TASK

六、REST 接口清单

6.1 Agent 编排 /api/agent

方法 路径 说明
POST /run 同步执行，阻塞直至完成
POST /run/async 异步提交，立即返回 taskId
GET /tasks?limit= 近期任务列表
GET /tasks/{taskId} 单任务完整轨迹
GET /tasks/{taskId}/stream SSE 步骤事件流（UTF-8）
POST /tasks/{taskId}/resume 人工审批后继续/拒绝
GET /specialists 内置专家角色列表
GET /dashboard/summary?limit= 成本/步数/Token 汇总

6.2 Agent 评估 /api/agent/eval

方法 路径 说明
POST /test-sets 创建测试集
GET /test-sets 测试集列表
POST /test-sets/{id}/cases 新增用例
GET /test-sets/{id}/cases 用例列表
PUT /test-sets/{testSetId}/cases/{caseId} 更新用例
DELETE /test-sets/{testSetId}/cases/{caseId} 删除用例
POST /runs 发起评估运行
GET /runs?testSetId= 评估 Run 列表
GET /runs/{id} 评估 Run 详情

七、关键配置项

yaml 复制代码 
app:
  agent:
    enabled: true
    max-steps: 6            # 单任务最大步骤数
    parallel-steps: true    # 无依赖步骤并行
    skills:                 # 技能开关
      tool: true
      rag: true
      llm: true
      specialist: true
      qa: true
      flow: true
      data-query: true
    eval:
      enabled: true
      enable-keyword-overlap: true     # 关键词重叠维度
      enable-trajectory-score: true    # 轨迹成功率维度
      keyword-overlap-threshold: 0.3   # 关键词通过阈值

八、总结与展望

8.1 本文要点回顾

  1. 编排 ：以 AgentOrchestratorService 为核心，规划 → 依赖调度（可并行）→ 执行 → 失败重规划 → 汇总 形成闭环；同步/异步双通道满足调试与长任务需求，SSE 让执行过程「可见」。
  2. 可观测 ：任务终态写入结构化 metrics（步数、Token、成本、耗时、重规划次数...），仪表盘按窗口聚合，指标口径清晰可查。
  3. 可评估：测试集 / 用例 / 运行 / 历史 Run 四步闭环，用「关键词重叠 + 轨迹成功率」双维度对 Agent 做回归验证，让效果优化可量化。

如果本文对你有帮助，欢迎点赞收藏；评论区可以交流你在 Agent 编排与评估中的落地经验。

版权声明：本文为原创技术文章，代码基于 Spring Boot + Spring AI 真实工程实践整理，转载请注明出处。

相关推荐
youcans_1 小时前
【嵌入式软件AI编程】12. Claude Code的基本操作
stm32·单片机·ai编程·嵌入式软件·claude code
FII工业富联科技服务1 小时前
从台积电微通道散热看 AI 热管理：热量究竟如何从芯片走向机房？
人工智能·ai
Xuantong_901 小时前
WAIC2026 深度观察：玄同科技 × 润建股份，共建中国东盟 AI 跨境 Token 产业生态，开启数字出海新范式
人工智能·ai·智能体
GISMagic2 小时前
3.从零制作 Agent 管理工具：LangChain + Node + Vue 实现多 Agent 统一调度
ai·agent
俊哥V2 小时前
AI一周事件 · 2026-09-09 至 2026-09-15
人工智能·ai
这就是佬们吗2 小时前
Function Calling 还是 MCP？先分清它们根本不是同一层的东西
人工智能·prompt·ai编程
AINative软件工程2 小时前
LLM 应用的 Graceful Shutdown 工程实践：正在进行的 AI 请求如何安全停止
云原生·ai编程
HRaitest2 小时前
AI招聘系统架构深度拆解：传统外挂式AI vs AI原生基座的本质差异与潜能边界
人工智能·ai·系统架构·视觉检测·求职招聘
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02数学公理体系大全:第六章 选择公理的等价形式及证明03历年考研数学一、数学二、数学三真题试卷及答案PDF04GitHub 镜像站点05微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新06蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接07人工智能（AI）发展史：详细里程碑082026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？09ZCode 周末送额度活动开启：3 亿 Token 免费领取10【一等奖版】2026 国赛 A题 药材的烘干问题