如何定义自定义评估器

核心概念

• 评估者

自定义评估器只是接收数据集示例和应用程序输出结果并返回一个或多个指标的函数。 这些函数可以直接传递给 evaluate() / aevaluate()。

基础示例

Python

需要 langsmith>=0.2.0

from langsmith import evaluate

def correct(outputs: dict, reference_outputs: dict) -> bool:
"""Check if the answer exactly matches the expected answer."""
    return outputs["answer"] == reference_outputs["answer"]

def dummy_app(inputs: dict) -> dict:
    return {"answer": "hmm i'm not sure", "reasoning": "i didn't understand the question"}

results = evaluate(
    dummy_app,
    data="dataset_name",
    evaluators=[correct]
)

TypeScript

需要 langsmith>=0.2.9

import type { EvaluationResult } from "langsmith/evaluation";

const correct = async ({ outputs, referenceOutputs }: {
  outputs: Record<string, any>;
  referenceOutputs?: Record<string, any>;
}): Promise<EvaluationResult> => {
  const score = outputs?.answer === referenceOutputs?.answer;
  return { key: "correct", score };
}

评估器参数

自定义评估函数必须具有特定的参数名称。它们可以接受以下参数中的任意子集：

• run: Run：应用程序在给定示例上生成的完整 运行 对象。
• example: Example：完整的数据集 示例，包括示例输入、输出（如果可用）和元数据（如果可用）。
• inputs: dict：数据集中单个示例对应的输入字典。
• outputs: dict：应用程序在给定的 inputs 上生成的输出字典。
• reference_outputs/referenceOutputs: dict: 如果可用，与示例关联的参考输出字典。

对于大多数使用场景，您只需使用 inputs、outputs 和 reference_outputs 即可。run 与 example 仅在您需要额外的跟踪信息或示例元数据（而非应用程序实际输入和输出）时才有用。

在使用 JavaScript/TypeScript 时，所有这些参数都应作为单个对象参数的一部分传入。

评估器输出

自定义评估器应返回以下类型之一：

Python 与 JavaScript/TypeScript

• dict：形如 {"score" | "value": ..., "key": ...} 的字典允许您自定义指标类型（“score”表示数值型，“value”表示分类型）及指标名称。例如，当您希望将一个整数作为分类指标记录时，此功能非常有用。

仅支持 Python

• int | float | bool：这被解释为一种可取平均值、可排序等的连续型指标。函数名称将用作该指标的名称。
• str: 这被解释为一个分类指标。函数名称用作指标的名称。
• list[dict]：使用单个函数返回多个指标。

更多示例

Python

需要 langsmith>=0.2.0

from langsmith import evaluate, wrappers
from langsmith.schemas import Run, Example
from openai import AsyncOpenAI
# Assumes you've installed pydantic.
from pydantic import BaseModel

# We can still pass in Run and Example objects if we'd like
def correct_old_signature(run: Run, example: Example) -> dict:
    """Check if the answer exactly matches the expected answer."""
    return {"key": "correct", "score": run.outputs["answer"] == example.outputs["answer"]}

# Just evaluate actual outputs
def concision(outputs: dict) -> int:
    """Score how concise the answer is. 1 is the most concise, 5 is the least concise."""
    return min(len(outputs["answer"]) // 1000, 4) + 1

# Use an LLM-as-a-judge
oai_client = wrappers.wrap_openai(AsyncOpenAI())

async def valid_reasoning(inputs: dict, outputs: dict) -> bool:
    """Use an LLM to judge if the reasoning and the answer are consistent."""

    instructions = """\

Given the following question, answer, and reasoning, determine if the reasoning for the \
answer is logically valid and consistent with question and the answer."""

    class Response(BaseModel):
        reasoning_is_valid: bool

    msg = f"Question: {inputs['question']}\nAnswer: {outputs['answer']}\nReasoning: {outputs['reasoning']}"
    response = await oai_client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": instructions,}, {"role": "user", "content": msg}],
        response_format=Response
    )
    return response.choices[0].message.parsed.reasoning_is_valid

def dummy_app(inputs: dict) -> dict:
    return {"answer": "hmm i'm not sure", "reasoning": "i didn't understand the question"}

results = evaluate(
    dummy_app,
    data="dataset_name",
    evaluators=[correct_old_signature, concision, valid_reasoning]
)

TypeScript

import { Client } from "langsmith";
import { evaluate } from "langsmith/evaluation";
import { Run, Example } from "langsmith/schemas";
import OpenAI from "openai";

// Type definitions
interface AppInputs {
    question: string;
}

interface AppOutputs {
    answer: string;
    reasoning: string;
}

interface Response {
    reasoning_is_valid: boolean;
}

// Old signature evaluator
function correctOldSignature(run: Run, example: Example) {
    return {
        key: "correct",
        score: run.outputs?.["answer"] === example.outputs?.["answer"],
    };
}

// Output-only evaluator
function concision({ outputs }: { outputs: AppOutputs }) {
    return {
        key: "concision",
        score: Math.min(Math.floor(outputs.answer.length / 1000), 4) + 1,
    };
}

// LLM-as-judge evaluator
const openai = new OpenAI();

async function validReasoning({
    inputs,
    outputs
}: {
    inputs: AppInputs;
    outputs: AppOutputs;
}) {
    const instructions = `      Given the following question, answer, and reasoning, determine if the reasoning for the       answer is logically valid and consistent with question and the answer.`;

    const msg = `Question: ${inputs.question}
Answer: ${outputs.answer}\nReasoning: ${outputs.reasoning}`;

    const response = await openai.chat.completions.create({
        model: "gpt-4",
        messages: [
            { role: "system", content: instructions },
            { role: "user", content: msg }
        ],
        response_format: { type: "json_object" },
        functions: [{
        name: "parse_response",
        parameters: {
            type: "object",
            properties: {
            reasoning_is_valid: {
                type: "boolean",
                description: "Whether the reasoning is valid"
            }
            },
            required: ["reasoning_is_valid"]
        }
        }]
    });

    const parsed = JSON.parse(response.choices[0].message.content ?? "{}") as Response;

    return {
        key: "valid_reasoning",
        score: parsed.reasoning_is_valid ? 1 : 0
    };
}

// Example application
function dummyApp(inputs: AppInputs): AppOutputs {
    return {
        answer: "hmm i'm not sure",
        reasoning: "i didn't understand the question"
    };
}

const results = await evaluate(dummyApp, {
        data: "dataset_name",
        evaluators: [correctOldSignature, concision, validReasoning],
        client: new Client()
});

相关

• 评估聚合实验结果：定义摘要评估器，用于为整个实验计算指标。
• 运行一项评估以比较两个实验：定义成对评估器，该评估器通过相互比较两个（或更多）实验来计算各项指标。