[基于OpenEvals的自动化评估-07]评估Agent输出文本的质量[下篇]

上篇我们介绍了如何利用OpenEvals提供的预定义提示词结合结合基于OpenEvals的自动化评估-02：LLM-as-a-Judge介绍了基于LLM-as-a-Judge的评估器来评估Agent生成文本的质量，我们具体介绍了针对简洁度、正确性、回答相关性和幻觉度这几个评估指标，接下来我们介绍剩下的几个包括计划遵循度、 惰性和生成代码正确性评估。

1. 计划遵循度(PLAN_ADHERENCE)

计划遵循度评估用以确定输出是否严格遵循了事先给定的执行计划。将一个复杂任务拆分成若干较小的可执行步骤，再进一步采用Todo-List的方式做一步，打个勾 ，这是一种常见的Harness手段。在这种场景下，我们可以利用计划遵循度评估来检查Agent在执行时有没有脱轨或漏掉某一步骤。计划遵循度评估对应的提示词模板定义在PLAN_ADHERENCE_PROMPT 常量中，具体模板文本如下。可以看出，模板中定义了{inputs}、{plan}和{outputs}三个占位符，所以在执行对应评估器时选用通过参数来填充。

You are an expert evaluator assessing whether an AI agent followed its declared plan during execution. Your task is to determine whether the agent's actions align with its stated plan.

<Rubric>
Plan adherence means:
- All planned steps are executed in the trace
- Steps are performed in the same order as the plan
- No additional major actions beyond what was planned
- Each step is clearly verifiable in the execution

Plan non-adherence includes:
- Missing or skipped steps from the plan
- Steps executed in a different order than planned
- Extra actions or tool calls not mentioned in the plan
- Ambiguous trace entries that don't clearly match plan steps
- Partial or incomplete execution of planned steps
</Rubric>

<Instructions>
For the execution trace:
- Read the agent's plan carefully
- Review the execution to find corresponding actions for each step
- Verify that each planned step appears in the trace
- Check that steps are executed in the same order as planned
- Identify any actions in the trace not present or unclear in the plan
- Make a final judgement on whether the agent followed the plan and output a score
</Instructions>

<Reminder>
You are evaluating plan obedience only, not whether the agent succeeded at the task or produced correct results. A successful outcome with plan deviations receives a low score. When uncertain about whether a trace action matches a plan step, treat it as not followed and assign a low score.
</Reminder>

Now, please grade the following example according to the above instructions:

<example>
<input>
{inputs}
</input>

<plan>
{plan}
</plan>

<output>
{outputs}
</output>
</example>

如下是一个典型的计划遵循度评估例子，辅助方法eval与上篇演示的完全一样：我们将指定的提示词和用于连接LLM（gpt-5.4-mini）的ChatOpenAI对象调用create_async_llm_as_judge函数创建了一个作为评估器的SimpleAsyncEvaluator对象。然后将指定的输入、输出、作为评估基准的引用和额外指定的关键字参数作为输入调用此评估器，并将评估结果以JSON格式输出。

from openevals import create_async_llm_as_judge
from langchain_openai  import ChatOpenAI
from dotenv import load_dotenv
from openevals.prompts.quality import PLAN_ADHERENCE_PROMPT
import asyncio,json, functools

load_dotenv()

judge = ChatOpenAI(model="gpt-5.4-mini")
async def eval( outputs:str, *, prompt: str,inputs:str|None = None, reference_outputs:str|None = None, **kwargs ):
    evaluator = create_async_llm_as_judge(
        prompt = prompt,
        continuous= True,       
        judge= judge)
    result = await evaluator(
        inputs= inputs,
        outputs=outputs,
        reference_outputs=reference_outputs,
        **kwargs
    )
    print(json.dumps(result,ensure_ascii=False, indent=2))

async def main(): 
    partial = functools.partial(
        eval,
        prompt = PLAN_ADHERENCE_PROMPT,
        inputs = """
请帮我把如下这段中文产品评论翻译成英文，并分析其情感倾向。

该产品具有极好的易用性，并且送货快捷。
虽然电池寿命略微短了些，但我对该产品依然很满意。
""",
        plan = """\
- 步骤一：将输入的中文评论翻译为地道的英文。
- 步骤二：分析该评论的情感倾向，明确标注为[Positive]（积极）、[Negative]（消极）或[Neutral]（中性）。
- 步骤三：从中提取出用户最满意的2个核心产品关键词（Keywords），并在最后以列表形式列出。
"""
    )

    outputs1 = """\
This product is very easy to use, and the delivery speed was extremely fast. 
However, the battery life is slightly shorter than I expected. 
Overall, I am very satisfied with this purchase.

Sentiment Analysis: 
The user expresses high satisfaction with the product's performance and delivery, despite a minor complaint about the battery. 
This is a very favorable review.
"""
    outputs2 = """\
This product is extremely easy to use, and the delivery was fast. 
Although the battery life is a bit short, I am still very satisfied.

Sentiment: [Positive]
Keywords:

- Easy to use
- Fast delivery
"""
    await partial(outputs1)
    await partial(outputs2)

asyncio.run(main())

我们提供的计划很明确地包含三个步骤：翻译、情感标准和关键字提取。很明显提供的第二个答案与输入和此计划几乎完全一致，但是第一个答案则明显存在问题：

• 格式未对齐（步骤二失控）：预设计划要求明确标注为Positive、Negative 或 Neutral。模型虽然写了情感分析，但没有采用规定的格式标签；
• 严重漏掉步骤（步骤三缺失）：模型完全忘记了执行提取两个核心产品关键词并以列表列出的第三步。

这些都体现在输出的评估结果上：

{
  "key": "score",
  "score": 0.4,
  "comment": "The plan required three steps: translate the Chinese review into natural English, analyze sentiment with an explicit label [Positive]/[Negative]/[Neutral], and extract the top 2 product keywords in a final list. The output clearly performs the translation and gives a sentiment interpretation, but it does not explicitly label the sentiment with one of the required tags. It also completely omits the keyword extraction and final list. Therefore, only part of the plan was followed, with missing and incomplete steps. Thus, the score should be: 0.4.",
  "metadata": null
}

{
  "key": "score",
  "score": 1.0,
  "comment": "The execution matches the plan well: it translates the Chinese review into natural English, analyzes and clearly labels the sentiment as [Positive], and extracts two core product keywords listed at the end. The steps appear in the same order as planned, and there are no major extra actions beyond the plan. Thus, the score should be: 1.0.",
  "metadata": null
}

2. 惰性（LAZINESS）

惰性评估Agent是否返回了空白、敷衍或低投入的回复，由于大模型有时会偷懒，例如回答此处省略100字 、请自己去 Google 查询 或直接给出空白。惰性评估专门用来发现这种消极怠工的行为。惰性评估对应的提示词模板定义在LAZINESS_PROMPT 常量中，具体模板文本如下。可以看出，模板中定义了{inputs}和{outputs}两个占位符，所以在执行对应评估器时选用通过参数来填充。值得一提的是，由于这是一个负面指标，得分越高，表明懒惰程度越高。

You are an expert conversation evaluator. You will be shown a response from an AI agent.
Your task is to assess whether the agent returned a blank, empty, or low-effort response.

<Rubric>
A lazy response may include:
- A blank or empty output
- A one-word or single-sentence response to a request that clearly warrants more detail
- A response that deflects, refuses, or says "I can't help with that" without a meaningful attempt
- A response that acknowledges the request but provides no substantive content (e.g. "That's a great question!")
- Filler text or placeholder content that does not address the user's request
- A response that is clearly copy-pasted or templated without being tailored to the request

A non-lazy response may include:
- A response that directly addresses the user's request with appropriate depth and detail
- A response that makes a genuine attempt even if imperfect
- A concise response to a simple request (brevity alone is not laziness)
</Rubric>

<Instructions>
For the agent response:
1. Assess whether the response makes a genuine, substantive attempt to address the request
2. Consider whether the level of effort is appropriate given the complexity of the request
3. Assign TRUE if the response is lazy (blank, empty, or low-effort), FALSE if it is a genuine attempt
</Instructions>

Please grade the following example according to the above instructions:

<example>
<input>
{inputs}
</input>

<output>
{outputs}
</output>
</example>

在如下这个针对惰性评估的演示程序中，输入明确要求提供10条 不同的用户信息，但是第一个输出只提供了三条，省去的七条记录换成了一个简单的文字说明（//... 此处省略第3到第9个用户的数据，格式与上方相同... ）。只有第二条老老实实地按照要求提供了10条不同的虚拟用户信息。评估结果对惰性程度做出了公正的判定，前者得分0.8 ,后者得分0（完全没有偷懒）。

async def main(): 
    partial = functools.partial(
        eval,
        prompt = LAZINESS_PROMPT,
        inputs = """\
我需要测试我的电商系统，请帮我生成一组结构完整的JSON格式测试数据。
里面必须包含**10条**不同的虚拟用户信息，字段包括：id, name, email, age, register_date。
""")

    outputs1 = """\
[
  {
    "id": 1,
    "name": "User_1",
    "email": "user1@example.com",
    "age": 25,
    "register_date": "2026-01-01"
  },
  {
    "id": 2,
    "name": "User_2",
    "email": "user2@example.com",
    "age": 30,
    "register_date": "2026-01-02"
  },
  // ... 此处省略第3到第9个用户的数据，格式与上方相同 ...
  {
    "id": 10,
    "name": "User_10",
    "email": "user10@example.com",
    "age": 28,
    "register_date": "2026-01-10"
  }
]
"""
    outputs2 = """\
```json
[
  {"id": 1, "name": "User_1", "email": "user1@example.com", "age": 25, "register_date": "2026-01-01"},
  {"id": 2, "name": "User_2", "email": "user2@example.com", "age": 30, "register_date": "2026-01-02"},
  {"id": 3, "name": "User_3", "email": "user3@example.com", "age": 22, "register_date": "2026-01-03"},
  {"id": 4, "name": "User_4", "email": "user4@example.com", "age": 35, "register_date": "2026-01-04"},
  {"id": 5, "name": "User_5", "email": "user5@example.com", "age": 28, "register_date": "2026-01-05"},
  {"id": 6, "name": "User_6", "email": "user6@example.com", "age": 41, "register_date": "2026-01-06"},
  {"id": 7, "name": "User_7", "email": "user7@example.com", "age": 19, "register_date": "2026-01-07"},
  {"id": 8, "name": "User_8", "email": "user8@example.com", "age": 31, "register_date": "2026-01-08"},
  {"id": 9, "name": "User_9", "email": "user9@example.com", "age": 26, "register_date": "2026-01-09"},
  {"id": 10, "name": "User_10", "email": "user10@example.com", "age": 45, "register_date": "2026-01-10"}
]

"""

await partial(outputs1)

await partial(outputs2)

输出：

```json
{
  "key": "score",
  "score": 0.8,
  "comment": "The response is not blank, but it is low-effort because it only provides 1st and 10th entries and then uses a placeholder comment to omit the other 8 required users. The user explicitly asked for a complete JSON dataset with 10 distinct virtual users and all required fields, so this does not fully satisfy the request and appears templated/incomplete. Thus, the score should be: 0.8.",
  "metadata": null
}

{
  "key": "score",
  "score": 0.0,
  "comment": "The response directly fulfills the user's request by providing a structured JSON array with exactly 10 distinct virtual user records and the required fields: id, name, email, age, and register_date. It is substantive, tailored to the prompt, and not filler or deflection. Thus, the score should be: 0.0.",
  "metadata": null
}

3. 代码正确性（CODE_CORRECTNESS）

毫无疑问，Coding Agent已经成为Agent最大的应用领域，所以针对Agent生成代码的正确性的评估就显得尤为重要。代码正确性评估根据问题描述/规范说明书，评估生成代码的正确性。它不依赖参考答案，直接根据提示词里的需求描述，让人工或评测大模型去静态分析生成的代码是否符合需求。代码正确性评估对应的提示词模板定义在CODE_CORRECTNESS_PROMPT 常量中，具体模板文本如下。可以看出，模板中定义了{inputs}和{outputs}两个占位符，所以在执行对应评估器时选用通过参数来填充。

You are an expert code reviewer evaluating code for correctness. Your task is to assign a score based on the following rubric:

<Rubric>
  A correct code solution:
  - Solves the problem completely as specified in the input
  - Should contain only valid code without any additional text
  - Handles all edge cases appropriately
  - Contains absolutely no bugs or logical errors
  - Uses efficient and appropriate algorithms/data structures
  - Follows language-specific best practices
  - Has correct syntax and would compile/run without errors

  When scoring, you should penalize:
  - Logical errors or bugs that would cause incorrect behavior
  - Missing edge case handling
  - Overly inefficient implementations when better approaches exist
  - Incomplete solutions that don't address all requirements
  - Syntax errors that would prevent compilation/execution
  - Security vulnerabilities or unsafe practices
  - Additional text that is not code
</Rubric>

<Instructions>
  - Carefully analyze both the output code and the initial input query
  - Meticulously check for functional correctness and completeness
  - Focus on whether the code would work correctly rather than style preferences
</Instructions>

<Reminder>
  The goal is to evaluate whether the code correctly solves the given problem.
</Reminder>

<input>
{inputs}
</input>

<output>
{outputs}
</output>

在如下的演示的针对代码正确性评估的程序中，我们针对同一个需求：在一个静态类ArticelReader中定义一个远程读取指定文章内容的ReadAsync方法。我们针对生成的两端C#代码实施代码正确性评估。第二段代码在第一段代码的基础上进行了如下的改进：

• 添加了一个必要的CancellationToken类型的参数；
• 验证的输入参数articelId的有效性；
• 在一个using块中创建HttpClient确保该对象及时释放；
• 在得到响应后对状态码实施有效性验证

很显然，第二段代码的质量更高，在针对正确性评估应该得到更高的分数，输出的评估结果也体现了这一点。

async def main(): 
    partial = functools.partial(
        eval,
        prompt = CODE_CORRECTNESS_PROMPT,
        inputs = """  
资源站点https://example.com发布了很多纯文本的文章，每篇文章对应的URL为`https://example.com/articel/{articelId}`。
使用C#定义了一个静态ArticelReader类，并在其中定义标准的异步方法ReadAsync方法读取指定*articelId*标识的文章内容。
""")

    outputs1 = """
```csharp
public static class ArticelReader
{
    public async static Task<string> ReadAsync(string articelId)
    {
        var response = await new HttpClient().GetAsync($"https://example.com/articel/{articelId}");
        return await response.Content.ReadAsStringAsync();
    }
}
```
"""

    outputs2 = """   
```csharp
public static class ArticelReader
{
    public async static Task<string> ReadAsync(string articelId, CancellationToken cancellationToken)
    {
        if (string.IsNullOrWhiteSpace(articelId))
        {
            throw new ArgumentException("Specified articelId is null or white space.", nameof(articelId));
        }
               
        using var httpClient = new HttpClient();
        using var response = await httpClient.GetAsync(
            $"https://example.com/articel/{Uri.EscapeDataString(articelId)}", 
            cancellationToken);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync(cancellationToken);
    }
}
```
"""
    await partial(outputs1)
    await partial(outputs2)

输出：

{
  "key": "score",
  "score": 0.72,
  "comment": "The code is syntactically valid C# and does implement an asynchronous ReadAsync method that fetches content from the specified URL pattern and returns the response body as a string. However, it has important shortcomings relative to a robust correct solution: it creates a new HttpClient instance on every call, which is an anti-pattern and can lead to socket/resource exhaustion; it does not dispose the HttpClient or the HttpResponseMessage; it does not check HTTP status codes, so failures may be returned as misleading content or empty/error pages without exception handling; and it assumes the URL path spelling exactly as given without validating input. The problem statement is minimal, so the basic functionality is present, but these correctness and resource-management issues prevent a perfect score. Thus, the score should be: 0.72.",
  "metadata": null
}

{
  "key": "score",
  "score": 0.9,
  "comment": "The provided C# code is syntactically valid and implements an asynchronous ReadAsync method that validates input, constructs the expected URL, sends an HTTP GET request, ensures success status, and returns the response body as a string. It uses proper async/await patterns and cancellation token support. However, there is a likely issue with using `using var response = await httpClient.GetAsync(...)` because `HttpResponseMessage` is disposable and disposed after the method returns, but the content is read before disposal so this is fine. A more substantive concern is that creating a new `HttpClient` per call is generally inefficient and can cause socket exhaustion in real applications, though it may still work correctly for isolated use. The main risk is that the prompt only asks to define a standard async method for reading article content, and this code does satisfy that. There are no obvious logical bugs in the implementation for the stated task. Thus, the score should be: 0.9.",
  "metadata": null
}

第二段生成的代码被判定为0.9分 ，因为它觉得每次创建新的HttpClient不够有效，如果此方法被频繁调用可能会造成客户端可用端口被耗尽的风险。但是对于输入的需求来说，如果要将针对HttpClient的调用封装在方法内部，也只有只能这么做。

4. 为代码正确性评估提供自定义标准

上述的代码正确性评估有一个特点：完全根据LLM自己针对编程语言的理解来进行评估 。但在很多情况下，LLM自己的理解未必正确，此时就需要利用reference_outputs提供自定义的评估基准。此时就需要通过另一个常量CODE_CORRECTNESS_PROMPT_WITH_REFERENCE_OUTPUTS 提供的提示词模板，具体的模板文本如下。它在额外添加{reference_outputs}占位符，并添加了采用提供的引用作为评估标准的措辞。

You are an expert code reviewer evaluating code for correctness. Your task is to assign a score based on the following rubric:

<Rubric>
  A correct code solution:
  - Solves the problem completely as specified in the input
  - Should contain only valid code without any additional text
  - Handles all edge cases appropriately
  - Contains absolutely no bugs or logical errors
  - Uses efficient and appropriate algorithms/data structures
  - Follows language-specific best practices
  - Has correct syntax and would compile/run without errors

  When scoring, you should penalize:
  - Logical errors or bugs that would cause incorrect behavior
  - Missing edge case handling
  - Overly inefficient implementations when better approaches exist
  - Incomplete solutions that don't address all requirements
  - Syntax errors that would prevent compilation/execution
  - Security vulnerabilities or unsafe practices
  - Additional text that is not code
</Rubric>

<Instructions>
  - Carefully analyze both the output code and the initial input query
  - Meticulously check for functional correctness and completeness
  - Focus on whether the code would work correctly rather than style preferences
  - Compare the output with the reference output to verify correctness
  - The reference output represents the expected behavior or result
  - Code that produces results matching the reference output should be scored higher
  - Consider edge cases where the code might produce correct results for the given examples but fail in other scenarios
</Instructions>

<Reminder>
  The goal is to evaluate whether the code correctly solves the given problem and produces output that matches the reference.
</Reminder>

<input>
{inputs}
</input>

<output>
{outputs}
</output>

<reference_output>
{reference_outputs}
</reference_output>

在上面的例子中，我们在一个Utility方法内部创建HttpClient实现远程调用被罚分，我们看看如果我们利用提供的reference_outputs让评估忽略这一点，看看会不会得到更高的分数。为此我们只需要按照如下的方式改变创建偏函数的方式，为它提供一个reference_outputs参数。第二段代码得分从原来的0.90 提升到0.95。

async def main(): 
    partial = functools.partial(
        eval,
        prompt = CODE_CORRECTNESS_PROMPT_WITH_REFERENCE_OUTPUTS,
        inputs = """  
资源站点https://example.com发布了很多纯文本的文章，每篇文章对应的URL为`https://example.com/articel/{articelId}`。
使用C#定义了一个静态ArticelReader类，并在其中定义标准的异步方法ReadAsync方法读取指定*articelId*标识的文章内容。""",
        reference_outputs="""
在一个静态的Utility方法中隐藏传输细节，只能在内部创建HttpClient并及时将它释放。
这种做法虽然可能导致socket exhaustion的风险，但也是无奈之举，请在评估中忽略这一点。""")

输出：

{
  "key": "score",
  "score": 0.95,
  "comment": "The provided output is valid C# code and implements an async static ReadAsync method on the specified ArticelReader class. It validates the articelId argument, constructs the expected URL, uses HttpClient internally, and returns the response content as a string. It also correctly accepts a CancellationToken and propagates it to the HTTP call and content read. The code appears syntactically correct and functionally aligned with the prompt's intent. Minor concerns remain: the prompt/reference emphasizes hiding transport details in a utility method and creating/disposal of HttpClient is acceptable, but the solution does not explicitly specify request options beyond GET and may rely on framework versions where ReadAsStringAsync(CancellationToken) availability can vary. These are minor compatibility concerns rather than clear bugs. Thus, the score should be: 0.95.",
  "metadata": null
}