Direct Scoring Evaluation-迪斯科星球

Direct Scoring Evaluation

【免费下载链接】Agent-Skills-for-Context-EngineeringA comprehensive collection of Agent Skills for context engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, or debugging agent systems that require effective context management.项目地址: https://gitcode.com/GitHub_Trending/ag/Agent-Skills-for-Context-Engineering

You are an expert evaluator assessing the quality of an AI-generated response.

Your Task

Evaluate the response below against the specified criteria. For each criterion:

First, identify specific evidence from the response
Then, determine the appropriate score based on the rubric
Finally, provide actionable feedback

Important Guidelines

Be objective and consistent
Base scores on explicit evidence, not assumptions
Consider the original task requirements
Avoid length bias - a shorter, better answer outperforms a longer, weaker one
When uncertain between two scores, explain your reasoning then choose

Original Prompt/Task

Additional Context

Response to Evaluate

Evaluation Criteria

{{name}} (Weight: {{weight}})

{{#if rubric}}Rubric:{{#each rubric}}

{{score}}: {{description}} {{/each}} {{/if}} {{/each}}

Your Evaluation

For each criterion, provide:

Evidence: Specific quotes or observations from the response
Score: Your score according to the rubric
Justification: Why this score is appropriate
Improvement: Specific suggestion for improvement

Then provide:

Overall Assessment: Summary of quality
Key Strengths: What the response does well
Key Weaknesses: What needs improvement
Priority Improvements: Most impactful changes

Format your response as structured JSON: { "scores": [ { "criterion": "{{name}}", "evidence": ["quote1", "quote2"], "score": {{score}}, "maxScore": {{maxScore}}, "justification": "...", "improvement": "..." } ], "overallScore": {{score}}, "summary": { "assessment": "...", "strengths": ["...", "..."], "weaknesses": ["...", "..."], "priorities": ["...", "..."] } }

### 模板的三个关键设计 1. **证据先行（Evidence First）**：模板要求裁判在打分前先引用响应中的具体片段作为证据（`Evidence: Specific quotes or observations`），并在 JSON 输出中以 `evidence` 数组承载。这与 [direct-score.ts](https://link.gitcode.com/i/24346f3ac2ef304a6b33d1240b62139c) 的输出 schema 中的 `evidence: z.array(z.string())` 一一对应。强制证据先行能显著抑制"凭感觉打分"，让分数可追溯。 2. **显式偏差约束**：`Avoid length bias`、`Base scores on explicit evidence, not assumptions` 这两条直接回应了 [llm-evaluator.md](https://link.gitcode.com/i/7642e2c4b50f8e3f19db33af8906afb3) 中记录的冗长偏差与假设推断问题。此外，`When uncertain between two scores, explain your reasoning then choose` 对应仓库建议的 "Justification First" 实现策略——先解释再给分，降低随意性。 3. **结构化 JSON 输出**：强制裁判以固定 JSON 结构返回 `scores[]`、`overallScore` 与 `summary{assessment, strengths, weaknesses, priorities}`。这使得下游代码可以直接 `JSON.parse` 结果并进行加权计算（见第四节），也天然适合接入 Agent 工具调用链路。 ## 三、变量插槽体系与评分标准（Rubric）定义 模板通过 Handlebars 风格的 `{{变量}}` 插槽完成运行时填充，官方变量清单如下： | 变量 | 说明 | 是否必填 | |------|------|----------| | original_prompt | 生成该响应的原始提示词/任务 | 是 | | context | 额外上下文（RAG 检索文档、对话历史等） | 否 | | response | 被评测的响应文本 | 是 | | criteria | 评测标准数组 | 是 | | criteria.name | 标准名称（如 Accuracy） | 是 | | criteria.weight | 标准权重 | 是 | | criteria.description | 该标准衡量的具体内容 | 是 | | criteria.rubric | 各分值档位的文字描述 | 否 | ### criteria 与 rubric 的构造示例 模板自带的完整示例输入（对应"向高中生解释量子纠缠"场景）： ```json { "original_prompt": "Explain quantum entanglement to a high school student", "response": "Quantum entanglement is like having two magic coins...", "criteria": [ { "name": "Accuracy", "weight": 0.4, "description": "Scientific correctness of the explanation", "rubric": [ { "score": 1, "description": "Fundamentally incorrect" }, { "score": 3, "description": "Mostly correct with some errors" }, { "score": 5, "description": "Completely accurate" } ] }, { "name": "Accessibility", "weight": 0.3, "description": "Understandable for a high school student" }, { "name": "Engagement", "weight": 0.3, "description": "Interesting and memorable" } ] }

注意Accessibility与Engagement未提供rubric，此时裁判将基于通用评分标准打分。若提供了rubric，模板会逐档位渲染- **1**: Fundamentally incorrect这样的描述，供裁判严格对齐。

评分标尺（Scale）约束

仓库在源码层面对评分标尺做了枚举约束。在 direct-score.ts 中，RubricSchema定义了scale只能是'1-3' | '1-5' | '1-10'，默认1-5：

const RubricSchema = z.object({ scale: z.enum(['1-3', '1-5', '1-10']).default('1-5'), levelDescriptions: z.record(z.string(), z.string()).optional() });

maxScore由标尺右端解析而来：const maxScore = parseInt(scale.split('-')[1])（direct-score.ts），随后被写入每个 criterion 的输出中。同时，CriterionSchema对weight做了z.number().min(0).max(1).default(1)的约束，保证权重落在合法区间。完整输入 schema 还要求criteria数组至少包含 1 项（z.array(CriterionSchema).min(1)），从类型层面杜绝空标准评测。

四、源码级落地：从提示词到可运行的 Direct Score 工具

提示词模板只是"配方"，真正让它跑起来的是仓库中基于 Vercel AI SDK 与 Zod 实现的工具链。核心实现在 direct-score.ts。

4.1 工具定义与输入输出契约

export const DirectScoreInputSchema = z.object({ response: z.string().describe('The LLM response to evaluate'), prompt: z.string().describe('The original prompt that generated the response'), context: z.string().optional().describe('Additional context'), criteria: z.array(CriterionSchema).min(1).describe('Evaluation criteria'), rubric: RubricSchema.optional() });

输出契约DirectScoreOutputSchema与提示词模板要求的 JSON 完全对齐，并在其之上补充了两个关键字段：

weightedScore：按权重加权的总分；
metadata：包含evaluationTimeMs、model、criteriaCount，用于评测链路的观测与审计。

4.2 执行流程：内置裁判系统提示词

executeDirectScore是核心函数（direct-score.ts），它将模板的核心思想内嵌为一段精炼的 system prompt：

const systemPrompt = `You are an expert evaluator. Assess the response against each criterion. For each criterion: 1. Find specific evidence in the response 2. Score according to the rubric (1-${maxScore} scale) 3. Justify your score 4. Suggest one improvement Be objective and consistent. Base scores on explicit evidence.`;

随后把原始 prompt、可选 context、被评测响应、criteria（含权重与描述）以及可选的 rubric 档位描述拼装成 user prompt，调用generateText（模型来自环境配置，temperature: 0.3，低温度保证评测稳定性）。最后JSON.parse(result.text)解析裁判输出，并进入汇总计算。

4.3 加权总分与整体分计算逻辑

这是评测结果能否被量化的关键一环（direct-score.ts）：

const totalWeight = input.criteria.reduce((sum, c) => sum + c.weight, 0); const weightedSum = parsed.scores.reduce((sum, s) => { const criterion = input.criteria.find(c => c.name === s.criterion); return sum + (s.score * (criterion?.weight || 1)); }, 0); const overallScore = parsed.scores.reduce((sum, s) => sum + s.score, 0) / parsed.scores.length; const weightedScore = weightedSum / totalWeight;

overallScore：所有维度分数的算术平均；
weightedScore：各维度分数乘以其权重后的加权和再除以总权重，即Σ(score × weight) / Σweight。

由于模板中weight总和通常为 1（如 0.4+0.3+0.3），weightedScore此时就是加权平均。仓库还要求权重必须通过权重之和归一化，即便权重之和不为 1 也能得到 0-1 之间的规范化结果。最终两个分数都四舍五入到两位小数返回。

实现提示：scores数组中的maxScore由标尺动态计算（如1-5标尺下maxScore = 5），因此前端渲染"7/10"或"4/5"时无需硬编码。

4.4 异常兜底

评测过程被try/catch包裹（direct-score.ts）：一旦裁判输出非 JSON、模型调用失败等，会返回success: false，scores为空数组、分数归零，并在summary.assessment中记录失败原因与耗时。这意味着提示词模板的"强制 JSON"在实际生产环境必须有容错设计，而不仅仅是提示词约束。

五、工具描述与 Skill 封装：让 Agent 学会调用

为了让 Agent 自主决定何时使用该工具，仓库在 tools/evaluation/direct-score.md 中给出了带语义描述的版本：

export const directScore = tool({ description: `Evaluate a response by scoring it against specific criteria. Use this for objective evaluations where you need to assess quality dimensions like accuracy, completeness, clarity, or task adherence. Returns structured scores with justifications.`, parameters: z.object({ response: z.string().describe("The LLM response to evaluate"), prompt: z.string().describe("The original prompt/instruction that generated the response"), context: z.string().optional() .describe("Additional context like retrieved documents or conversation history"), criteria: z.array(z.object({ name: z.string().describe("Name of the criterion (e.g., 'Accuracy')"), description: z.string().describe("What this criterion measures"), weight: z.number().min(0).max(1).default(1) .describe("Relative importance, weights should sum to 1") })).min(1).describe("Evaluation criteria to score against"), rubric: z.object({ scale: z.enum(["1-3", "1-5", "1-10"]).default("1-5"), levelDescriptions: z.record(z.string(), z.string()).optional() .describe("Optional descriptions for each score level") }).optional().describe("Scoring rubric configuration") }), execute: async (input) => evaluateWithLLM(input) });

关键点：

description中明确标注适用场景（"objective evaluations like accuracy, completeness, clarity"），帮助路由层将客观评测任务导向 direct scoring，将主观任务导向 pairwise-comparison-prompt.md；
每个字段的.describe()注释会作为 LLM 的函数调用 schema 提示，让 Agent 知道weight应满足"权重总和为 1"；
该工具在 src/tools/evaluation/index.ts 中统一导出，并经 src/index.ts 暴露为DirectScoreInput/DirectScoreOutput类型，供上层 Agent 与业务代码引用。

六、最佳实践：把提示词用对的关键约束

模板结尾给出了 5 条必须遵守的最佳实践，结合源码实现补充如下：

Evidence First（证据先行）：先收集证据再打分。模板在Your Task中强制了 1→2→3 的顺序；实现层面evidence数组是输出 schema 的必填字段，从结构上保证"先引证、后评分"。
Rubric Alignment（严格对齐评分标准）：Stick to rubric definitions, don't interpolate——裁判不得在档位之间自行内插分值。这与 tools/evaluation/direct-score.md 的 Implementation Notes 中"Calibration: Include few-shot examples of scores at each level"（在每档位附上 few-shot 示例以校准）互为补充。
Constructive Feedback（建设性反馈）：improvement字段必须可执行。模板要求输出Priority Improvements: Most impactful changes，实现中每个 criterion 都有独立improvement字符串，便于直接回流给生成模型迭代。
Consistency（跨评测一致性）：同一评测体系内使用相同的标准。实践中建议固定temperature: 0.3（见 direct-score.ts），并用 llm-evaluator.md 中提到的 Cohen's κ / Spearman's ρ 等指标监控裁判自身的一致性。
Calibration（校准）：以示例评测作为参照，减少裁判的随机漂移。

七、端到端运行：环境配置与测试验证

7.1 环境准备

仓库在 env.example 中定义环境变量，配置读取逻辑见 src/config/index.ts：

export const config = { openai: { apiKey: process.env.OPENAI_API_KEY || '', model: process.env.OPENAI_MODEL || 'gpt-4o' }, anthropic: { apiKey: process.env.ANTHROPIC_API_KEY || '' } } as const;

复制环境变量示例并填入OPENAI_API_KEY（可选覆盖OPENAI_MODEL，默认gpt-4o）；
依赖与脚本见 package.json，测试框架为 Vitest（配置见 vitest.config.ts）；
运行示例：npx tsx examples/basic-evaluation.ts；运行测试：npm test或npx vitest。

7.2 最小可运行示例

examples/basic-evaluation.ts 展示了完整调用方式：

import 'dotenv/config'; import { EvaluatorAgent } from '../src/agents/evaluator.js'; import { validateConfig } from '../src/config/index.js'; const agent = new EvaluatorAgent(); const result = await agent.score({ response: `...机器学习定义与三大类型的说明...`, prompt: 'Explain what machine learning is to a beginner', criteria: [ { name: 'Accuracy', description: 'Factual correctness of the explanation', weight: 0.4 }, { name: 'Clarity', description: 'Easy to understand for a beginner', weight: 0.3 }, { name: 'Completeness', description: 'Covers the key concepts adequately', weight: 0.3 } ], rubric: { scale: '1-5', levelDescriptions: { '1': 'Poor - Major issues', '2': 'Below Average - Several issues', '3': 'Average - Some issues', '4': 'Good - Minor issues only', '5': 'Excellent - No issues' } } }); // result.overallScore / result.weightedScore / result.summary...

创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考

企业官网建设流程全解析