对于我们个人来说，Agent编程最大的挑战不是各种所谓的工作，而是写提示词，毕竟我不是专业的文字工作者。OpenEvals针对一系列针对输出文本质量的评估指标定义了对应的提示词，结合基于OpenEvals的自动化评估-02：LLM-as-a-Judge:让LLM当裁判来评估Agent的输出介绍了基于LLM-as-a-Judge的评估器，我们可以进行针对简洁度、正确性、幻觉度、相关性、计划遵循度、代码正确性等指标的评估。我们会分上下文两篇文章来介绍针对这些预定义提示词的使用。

1. 简洁度（CONCISENES）

简洁度评估旨在确定输出是否足够简练，是否避免了不必要的废话和填充内容，用户检查Agent是否正面回答，有没有给出一大堆没有价值的客套话或重复长句。OpenEvals预定的针对输出质量的提示词都定义在openevals.propmpts.quality路径下，简洁性评估提示词对应的成员名称为ANSWER_RELEVANCE_PROMPT，具体内容如下。从提示词模板的定义可以看出，由于定义了{inputs}和{outputs}占位符，所以在执行的对应评估器的时候需要指定输入。

You are an expert evaluator assessing whether outputs are relevant to the given input. Your task is to determine whether EACH statement appropriately addresses what was asked. <Rubric> A relevant output: - Directly answers the question or addresses the request - Provides information specifically asked for - Stays on topic with the input's intent - Contributes meaningfully to fulfilling the request An irrelevant output: - Discusses topics not requested or implied by the input - Provides unnecessary tangents or digressions - Includes information that doesn't answer the question - Addresses a different question than what was asked </Rubric> <Instructions> For each output: - Read the original input carefully to understand what was asked - Examine the output and identify its core claim or purpose - Determine if the output directly addresses the input's request - Assess whether the information helps fulfill what was asked - Determine the answer relevancy of output and output a score </Instructions> <Reminder> Focus on whether each statement helps answer the specific input question, not whether the statement is true or well-written. A statement can be factually correct but still irrelevant if it doesn't address what was asked. </Reminder> Now, grade the following example according to the above instructions: <example> <input> {inputs} </input> <output> {outputs} </output> </example>

以如下这个针对输出简洁度的评估为例，我们针对同一个问题：什么是光合作用？，提供了两端用于评估的输出文本，前者冗长、包含大量废话/车轱辘话，后者简洁、直击核心、无多余填充。所以前者得了0.15分，后者得了0.95风。

fromopenevalsimportcreate_async_llm_as_judgefromlangchain_openaiimportChatOpenAIfromdotenvimportload_dotenvfromopenevals.prompts.qualityimportCONCISENESS_PROMPTimportasyncio,json,functools load_dotenv()judge=ChatOpenAI(model="gpt-5.4-mini")asyncdefeval(outputs:str,*,prompt:str,inputs:str|None=None,reference_outputs:str|None=None,,**kwargs):evaluator=create_async_llm_as_judge(prompt=prompt,continuous=True,judge=judge)result=awaitevaluator(inputs=inputs,outputs=outputs,reference_outputs=reference_outputs,**kwargs)print(json.dumps(result,ensure_ascii=False,indent=2))asyncdefmain():inputs="什么是光合作用？"outputs1="""\ 这是一个非常好的问题！ 在我们的自然界中，植物扮演着非常重要的角色，它们需要生长和生存。 关于您问的什么是光合作用，简单来说，光合作用其实就是植物、藻类以及某些特定的细菌，在它们体内的叶绿体中，通过利用太阳的光能，来把从空气中吸收的水分和二氧化碳，通过一系列复杂的、神奇的生物化学反应过程，转化为它们生长所需的能量和有机物（比如葡萄糖），并且在这个过程当中，它们还会把氧气释放到我们呼吸的空气中。 总而言之，这就是一个能量转换的奇妙过程，对地球生态非常关键。 希望这个详细的解释对您有所启发和帮助！ """outputs2="光合作用是植物、藻类和某些细菌利用光能，将二氧化碳和水转化为有机物（如葡萄糖）并释放氧气的生物化学过程。"partial=functools.partial(eval,prompt=CONCISENESS_PROMPT,inputs=inputs)awaitpartial(outputs1)awaitpartial(outputs2)asyncio.run(main())

输出：

{"key":"score","score":0.15,"comment":"The response is highly verbose and includes multiple unnecessary elements: an opening compliment, background about plants, a long explanatory definition, a summary sentence, and a polite closing. While it does answer the question, it is far from the minimum concise form expected for a simple definition request. Thus, the score should be: 0.15.","metadata":null}

{"key":"score","score":0.95,"comment":"The output directly answers the question with a concise definition and includes only the essential information. It does not add hedging, pleasantries, meta-commentary, or unnecessary background beyond a minimal clarifying parenthetical example. Thus, the score should be: 0.95.","metadata":null}

2. 正确性（CORRECTNESS）

正确性评估用于确定输出内容的事实准确性以及完整性。对于一些常识性或者公开性问题，LLM是知道正确答案的，但是由于如下所示的提示词模板定义了{inputs}、{outputs}和{reference_outputs}占位符，所以输入和引用都需要指定，正确性评估会通过比较输出和引用，并结合自己掌握的知识来判断对错。正确性提示词定义在CORRECTNESS_PROMPT变量中。

You are an expert data labeler evaluating model outputs for correctness. Your task is to assign a score based on the following rubric: <Rubric> A correct answer: - Provides accurate and complete information - Contains no factual errors - Addresses all parts of the question - Is logically consistent - Uses precise and accurate terminology When scoring, you should penalize: - Factual errors or inaccuracies - Incomplete or partial answers - Misleading or ambiguous statements - Incorrect terminology - Logical inconsistencies - Missing key information </Rubric> <Instructions> - Carefully read the input and output - Check for factual accuracy and completeness - Focus on correctness of information rather than style or verbosity </Instructions> <Reminder> The goal is to evaluate factual correctness and completeness of the response. </Reminder> <input> {inputs} </input> <output> {outputs} </output> Use the reference outputs below to help you evaluate the correctness of the response: <reference_outputs> {reference_outputs} </reference_outputs>

以如下这个演示程序为例，我们采用上述的提示词，采用不同的reference_outputs对同一个输出进行评估。具体来说，对于问题：春秋五霸指的是那几个?，我们提供了两个标准答案（春秋五霸历来就有两种说法）。虽然文本不同，很明显输入与第一个标准答案（齐桓公、晋文公、楚庄王、秦穆公和宋襄公）是一致的，评估的结果也说明了这一点。

asyncdefmain():inputs="春秋五霸是那几个？"outputs="齐桓公姜小白、晋文公姬重耳、楚庄王熊侣、秦穆公嬴任好和宋襄公子兹甫"partial=functools.partial(eval,prompt=CORRECTNESS_PROMPT,inputs=inputs,outputs=outputs)reference_outputs1="齐桓公、晋文公、楚庄王、秦穆公和宋襄公"reference_outputs2="齐桓公、晋文公、楚庄王、吴王阖闾、越王勾践"awaitpartial(reference_outputs=reference_outputs1)awaitpartial(reference_outputs=reference_outputs2)

输出：

{"key":"score","score":1.0,"comment":"The output correctly identifies the traditional Spring and Autumn Five Hegemons: 齐桓公、晋文公、楚庄王、秦穆公、宋襄公. It also gives their personal names, which are accurate in this context: 姜小白、姬重耳、熊侣、嬴任好、子兹甫. This matches the reference answer and fully addresses the question. Thus, the score should be: 1.0.","metadata":null}

{"key":"score","score":0.4,"comment":"The output lists the traditional “五霸” as 齐桓公、晋文公、楚庄王、秦穆公、宋襄公, but the provided reference output gives the commonly accepted set as 齐桓公、晋文公、楚庄王、吴王阖闾、越王勾践. Therefore, the answer matches only the first three and gets the last two wrong, including an incorrect substitution of 秦穆公 and 宋襄公 for 吴王阖闾 and 越王勾践. It is incomplete relative to the reference and contains factual inaccuracies. Thus, the score should be: 0.4.","metadata":null}

为了证明LLM并不是针对outputs和reference_outputs提供的文本进行推理，还会利用自己掌握的支持，我们修改了outputs参数的值（姜小白换成了姜小黑）。从输出可以看出，这种小花招瞒不过LLM。

asyncdefmain():awaiteval(outputs="齐桓公姜小黑、晋文公姬重耳、楚庄王熊侣、秦穆公嬴任好和宋襄公子兹甫",prompt=CORRECTNESS_PROMPT,inputs="春秋五霸是那几个？",reference_outputs="齐桓公、晋文公、楚庄王、秦穆公和宋襄公")

输出：

{"key":"score","score":0.9,"comment":"The response correctly identifies the traditional Spring and Autumn Five Hegemons: 齐桓公、晋文公、楚庄王、秦穆公和宋襄公. It also includes their personal names, such as 姜小白 for 齐桓公, 姬重耳 for 晋文公, 熊侣 for 楚庄王, 嬴任好 for 秦穆公, and 子兹甫 for 宋襄公. However, there is a factual error: 齐桓公’s personal name is usually written 姜小白, not 姜小黑. Because of this inaccuracy, the answer is not fully correct, though it is otherwise complete and relevant. Thus, the score should be: 0.9.","metadata":null}

3. 回答相关性（ANSWER_RELEVANCE）

回答相关性评估用来确定输出是否直接回答了用户所提出的问题，用来解决防答非所问的问题——有时候模型回答得很对，但根本不是用户问的问题（比如问城门楼子，答胯骨轴子）。对应提示词文本内容如下，对应的变量为ANSWER_RELEVANCE_PROMPT。由于模板包含{inputs}和{outputs}，所以在执行评估器的时候需要指定输入。

You are an expert evaluator assessing whether outputs are relevant to the given input. Your task is to determine whether EACH statement appropriately addresses what was asked. <Rubric> A relevant output: - Directly answers the question or addresses the request - Provides information specifically asked for - Stays on topic with the input's intent - Contributes meaningfully to fulfilling the request An irrelevant output: - Discusses topics not requested or implied by the input - Provides unnecessary tangents or digressions - Includes information that doesn't answer the question - Addresses a different question than what was asked </Rubric> <Instructions> For each output: - Read the original input carefully to understand what was asked - Examine the output and identify its core claim or purpose - Determine if the output directly addresses the input's request - Assess whether the information helps fulfill what was asked - Determine the answer relevancy of output and output a score </Instructions> <Reminder> Focus on whether each statement helps answer the specific input question, not whether the statement is true or well-written. A statement can be factually correct but still irrelevant if it doesn't address what was asked. </Reminder> Now, grade the following example according to the above instructions: <example> <input> {inputs} </input> <output> {outputs} </output> </example>

在如下的演示程序中，我们使用上述的预定义提示词实施相关性评估。针对输入的问题今天下午北京会下雨吗？我需要带伞吗？，我们提供了两个答案。前者引入很多无关信息，强行扯到温带季风气候和防晒保护皮肤，严重偏离了用户即时性的查询需求；后者则与问题高度相关。输出的评估结果也体现了它们与问题的相关程度。

asyncdefmain():partial=functools.partial(eval,prompt=ANSWER_RELEVANCE_PROMPT,inputs="今天下午北京会下雨吗？我需要带伞吗？")outputs1="""\ 下雨是一种常见的自然降水现象。 北京作为中国的首都，属于温带季风气候，夏季降水非常集中，经常会出现强对流天气和雷阵雨。 如果您在夏季出行，建议您随时关注气象台发布的最新预报，以便合理安排行程，避免因恶劣天气耽误您的时间。 另外，出门带一把晴雨伞不仅能防雨，在烈日下还能起到防晒的作用，对保护皮肤很有帮助。"""outputs2="""\ 根据最新的气象预报，今天下午北京大部分地区为多云转阴，傍晚前后有 80% 的概率出现雷阵雨。 因此，强烈建议您出门带伞，既能防雨也能防范突发的强对流天气。"""awaitpartial(outputs1)awaitpartial(outputs2)

输出：

{"key":"score","score":0.4,"comment":"The input asks a specific, time-sensitive question: whether it will rain in Beijing this afternoon and whether to bring an umbrella. The output gives general information about rain, Beijing climate, and generic advice to check forecasts and carry an umbrella, but it does not directly answer the actual weather question or clearly determine whether an umbrella is needed for this afternoon. It is related in theme, but it only partially addresses the request and lacks the specific answer asked for. Thus, the score should be: 0.4.","metadata":null}

{"key":"score","score":1.0,"comment":"The output directly addresses the user's question about whether it will rain in Beijing this afternoon and whether they need to bring an umbrella. It provides a weather forecast and a clear recommendation to bring an umbrella, which is exactly what was asked. Thus, the score should be: 1.0.","metadata":null}

4. 幻觉度（HALLUCINATION）

幻觉度评估用于确定输出中是否包含了无法被所提供的上下文支持的信息。这种评估在RAG中具有广泛的应用——检查模型有没有瞎编。如果参考文档里没写，模型自己脑补了，就会被判定为幻觉。幻觉度评估提示词定义在名为HALLUCINATION_PROMPT的变量中，具体内容如下。模板文本中包含{context}、{inputs}、{outputs}和{reference_outputs}四个占位符，调用对应的评估器时必须指定。

You are an expert data labeler evaluating model outputs for hallucinations. Your task is to assign a score based on the following rubric: <Rubric> A response without hallucinations: - Contains only verifiable facts that are directly supported by the input context - Makes no unsupported claims or assumptions - Does not add speculative or imagined details - Maintains perfect accuracy in dates, numbers, and specific details - Appropriately indicates uncertainty when information is incomplete </Rubric> <Instructions> - Read the input context thoroughly - Identify all claims made in the output - Cross-reference each claim with the input context - Note any unsupported or contradictory information - Consider the severity and quantity of hallucinations </Instructions> <Reminder> Focus solely on factual accuracy and support from the input context. Do not consider style, grammar, or presentation in scoring. A shorter, factual response should score higher than a longer response with unsupported claims. </Reminder> Use the following context to help you evaluate for hallucinations in the output: <context> {context} </context> <input> {inputs} </input> <output> {outputs} </output> If available, you may also use the reference outputs below to help you identify hallucinations in the response: <reference_outputs> {reference_outputs} </reference_outputs>

在下面的演示程序中，我们使用上述的提示词针对问题**特斯拉公司是在哪一年由谁创立的？**的两个答案实施幻觉评估。根据提供的上下文，第一个回答（特斯拉由马丁·埃伯哈德和马克·塔彭宁于2003年创立。）才是正确的，这也在输出的评估结果的得到印证。

asyncdefmain():partial=functools.partial(eval,prompt=HALLUCINATION_PROMPT,inputs="特斯拉公司是在哪一年由谁创立的？",reference_outputs="特斯拉由马丁·埃伯哈德和马克·塔彭宁于2003年创立。",context="""\ 特斯拉（Tesla）是一家美国电动汽车及能源公司。 该公司由马丁·埃伯哈德（Martin Eberhard）和马克·塔彭宁（Marc Tarpenning）于2003年7月1日创立，总部最初位于美国加利福尼亚州。 公司的名字是为了纪念物理学家尼古拉·特斯拉。""")outputs1="特斯拉由马丁·埃伯哈德和马克·塔彭宁于2003年创立。"outputs2="特斯拉公司是由著名企业家埃隆·马斯克（Elon Musk）于2003年7月1日创立的。"awaitpartial(outputs1)awaitpartial(outputs2)

输出：

{"key":"score","score":1.0,"comment":"The output states that Tesla was founded by Martin Eberhard and Marc Tarpenning in 2003, which is fully supported by the provided context and reference output. It does not add any unsupported details or contradict the source, though it omits the exact founding date and does not need to because the question asks for the year and founders. Thus, the score should be: 1.0.","metadata":null}

{"key":"score","score":0.2,"comment":"The output incorrectly states that Tesla was founded by Elon Musk. The provided context says Tesla was founded by Martin Eberhard and Marc Tarpenning on July 1, 2003, so the founder claim is unsupported and contradictory. It also says \"著名企业家\" (famous entrepreneur), which is not supported by the context. The year and date are partly supported, but the key factual claim about who founded the company is wrong. Thus, the score should be: 0.2.","metadata":null}