[基于OpenEvals的自动化评估-05]基于JSON相似度的评估[LLM参与的语义匹配]

基于JSON内容的评估可以说是OpenEvals最重要的部分，基于OpenEvals的自动化评估-04:基于JSON相似度的评估对用于提供相关评估器的如下两个工厂函数从无LLM参与的角度进行了介绍。我们着重讨论了这两个函数的aggregator、list_aggregator和list_match_mode参数。其余的参数除了exclude_keys均与LLM-as-a-Judge有关。如果有了LLM加持，我们在评估的时候将不再仅限于严格的字符串比较，可以进行真正的基于自然语言的语义相似度比较，这样才能真正发挥AI的能力。

def create_json_match_evaluator(
    *,
    aggregator: Optional[Literal["average", "all"]] = None,
    list_aggregator: Literal["average", "all"] = "all",
    rubric: Dict[str, str] = {},
    exclude_keys: list[str] = [],
    judge: Optional[
        Union[
            ModelClient,
            BaseChatModel,
        ]
    ] = None,
    model: Optional[str] = None,
    use_reasoning: bool = True,
    list_match_mode: Literal[
        "superset", "subset", "same_elements", "ordered"
    ] = "same_elements",
) -> SimpleEvaluator

def create_async_json_match_evaluator(
    *,
    aggregator: Optional[Literal["average", "all"]] = None,
    list_aggregator: Literal["average", "all"] = "all",
    rubric: Dict[str, str] = {},
    exclude_keys: list[str] = [],
    judge: Optional[
        Union[
            ModelClient,
            BaseChatModel,
        ]
    ] = None,
    model: Optional[str] = None,
    use_reasoning: bool = True,
    list_match_mode: Literal[
        "superset", "subset", "same_elements", "ordered"
    ] = "same_elements",
) -> SimpleAsyncEvaluator

1. 参数说明

create_json_match_evaluator和create_async_json_match_evaluator与LLM-as-a-Judge有关的参数由如下几个：

• rubric：通过一个字段为JSON的每个字段定义评估规则；
• judge ：以ModelClient或者BaseChatModel的方式指定连接LLM的客户端；
• model：以标准的形式指定作为评估模型的名称；
• use_reasoning ：是否开启推理，并使用推理文本来填充EvaluatorResult的commment字段。

除了用来定义评估规则的rubric字段，其余几个参数的作用和使用方式都与create_llm_as_judge/create_async_llm_as_judge的同名参数相同，具体可以参考前面的文章基于OpenEvals的自动化评估-02:LLM-as-a-Judge:让LLM当裁判来评估Agent的输出

2. 根据自定义规则进行基于语义的评估

接下来我们通过如下几个例子来演示如何利用LLM根据我们自定义的规则对JSON的字段实施评估。我们依然沿用上一篇提供的演示程序，为了能够给出评估的理由，我们修改了辅助函数pretty_print将EvaluatorResult的comment字段进行了输出。

import asyncio
from openevals.json import create_async_json_match_evaluator
from openevals.types import EvaluatorResult
from langchain_openai import ChatOpenAI
from dotenv import load_dotenv

load_dotenv()

def create(name:str, age:int, **kwargs)->dict:
    person =  {"name":name, "age":age}
    return {**person, **kwargs}

def pretty_print(results: EvaluatorResult|list[EvaluatorResult]):
    if(isinstance(results, list)):
        for result in results:
            print(f"""\
{result['key']} = {result['score']}
{result['comment']}
""")
    else:
       print(f"""\
{results['key']} = {results['score']}
{results['comment']}
""")
    print()

async def eval(
    *,
    outputs: dict|list[dict],
    references: dict|list[dict],
    **kwargs):

    evaluator = create_async_json_match_evaluator(**kwargs)  
    results = await evaluator(outputs=outputs, reference_outputs=references)
    pretty_print(results)
    
async def main():
    rubric:dict[str, str] = {
        "name":"conduct strictly string comparison.",        
        "aget":"conduct strictly numeric comparison."
    }
    llm = ChatOpenAI(model="gpt-5.4-mini")
    alice1 = create("Alice",20)
    alice2 = create("alice",20)
    await eval(outputs=alice1, references=alice2, rubric=rubric, judge=llm)

如上面的代码所示，我们在调用eval函数的时候，指定了如下两个参数：

• rubric ：为评估JSON的两个字段name和age定义了评估规则，要求它们进行严格的字符串和数值比较；
• judge ：指定了一个ChatOpenAI对象用来调用评估模型（gpt-5.4-mini）

如下所示的输出满足我们的预期：指定的name由于大小写不一致被判定为不匹配，并给出了理由；age数值相同，视为匹配。

json_match:age = 1
None

json_match:name = False
The output value is 'Alice' while the expected value is 'alice'. Since the criterion is strict string comparison, the case mismatch means they do not match.

现在我们对main函数做了如下的修改，在提供的rubric中对字典name的评估规则进行了修改，让LLM采用大小写不敏感的方式进行字符串比较。执行之后可以发现两个字段都是匹配的。

async def main():
    rubric:dict[str, str] = {
        "name":"conduct string comparison with case-insensitive mode.",        
        "aget":"conduct strictly numeric comparison."
    }
    llm = ChatOpenAI(model="gpt-5.4-mini")
    alice1 = create("Alice",20)
    alice2 = create("alice",20)
    await eval(outputs=alice1, references=alice2, rubric=rubric, judge=llm)

输出：

json_match:age = 1
None

json_match:name = True
The output value 'Alice' matches the expected value 'alice' under case-insensitive string comparison.

现在我们进一步，为两个字典定义如下的评估规则：

• name：作为人名，可能提供姓（Last Name）、名（First Name）和姓名（Full Name），我们认为只要任意部分匹配就可以了；
• age：只要年龄差距不超过5岁也被视为匹配。

在如下的演示程序中，我将上述的评估规则应用到rubric参数上。在调用eval函数进行评估时传入的两个对象，其name和age都不相同，但是满足上述的规则，所以LLM将它们都判定为匹配：

async def main():
    rubric:dict[str, str] = {
        "name":"Considered matched if the same first name or last name is mentioned in both output value and referenced value",        
        "age":"Considered matched if the absolute difference between the output value and referenced value is less than 5."
    }
    llm = ChatOpenAI(model="gpt-5.4-mini")
    alice1 = create("Alice",20)
    alice2 = create("alice gates",23)
    await eval( outputs=alice1, references=alice2, rubric=rubric, judge=llm)

输出：

json_match:age = True
The output age is 20 and the expected age is 23. The absolute difference is 3, which is less than 5, so it matches the criterion.

json_match:name = True
The output value 'Alice' shares the same first name as the expected value 'alice gates' (case-insensitive match), so it satisfies the criterion.

3. 忽略指定的字段

上一篇和这篇文章基本围绕着create_json_match_evaluator和create_async_json_match_evaluator这两个创建JSON评估器的工厂函数展开介绍，我们基本上完整介绍了两个函数的所有参数，除了exclude_keys。这个参数的语义很明确，就是忽略评估JSON中携带的某些字段。

如下面的演示程序所示，我在rubric参数中添加一个针对生日字段（birth）的评估规则，并在作为outputs的对象上同时添加了birth和gender两个字段，但是利用exclude_keys参数加将后者忽略。所以输出只包含name、age和birth三个字段的评估结果：

async def main():
    rubric:dict[str, str] = {
        "name":"Considered matched if the same first name or last name is mentioned in both output value and referenced value",        
        "age":"Considered matched if the absolute difference between the output value and referenced value is less than 5.",
        "birth":"Conduct date (without time) based comparison."
    }
    llm = ChatOpenAI(model="gpt-5.4-mini")
    alice1 = create("Alice",20, birth="1981-08-24", gender="female")
    alice2 = create("alice gates",23, birth="24 AUG 1981")
    await eval(outputs=alice1, references=alice2, rubric=rubric, judge=llm, exclude_keys=["gender"])

输出：

json_match:age = True
The output age is 20 and the expected age is 23. The absolute difference is 3, which is less than 5, so it matches the criteria.

json_match:birth = True
The output date 1981-08-24 matches the expected date 24 AUG 1981 when compared by date only, ignoring format and time.

json_match:name = True
The output value 'Alice' shares the same first name as the expected value 'alice gates' (case-insensitive match on Alice/alice), so it matches the criterion.