Evals for AI agents: summarization evaluation

Evaluating AI Agents: A Systematic Approach to Meeting Summarization Quality

AI agents handling meetings often deliver inconsistent results: hallucinations, missed tasks, and bloated text. Evals fix this by providing objective quality metrics. These are tests for non-deterministic systems, much like unit tests in traditional code. They answer key questions: where does the agent break, has it improved after changes, and has anything working been broken?

This article covers eval types, error analysis, code-based assertions, and metrics for meeting summarization. The approach draws from experts like Hamel Husain and Eugene Yan.

Eval Types: From Simple Checks to LLM-as-Judge

An eval is a function f(output) → score that rates the agent's output. Three levels of complexity:

• Code-based assertions: Deterministic checks for formal properties (length, structure, keywords). Fast, free, but limited.
• Human evaluation: Nuanced human judgment. Accurate, but expensive and not scalable.
• LLM-as-Judge: LLM scores based on criteria. Scalable, cheap, needs validation.

The golden rule: combine all types. Code-based for every request, LLM-as-Judge on samples, human for calibration.

For meeting summaries, define metrics by priority:

• Faithfulness: Every claim backed by the transcript.
• Action item accuracy: Complete tasks with owner and deadline.
• Completeness: Coverage of key topics.
• Conciseness: Length <30% of transcript, density 0.15 entities/token.
• Coherence: Logical structure.

Error Analysis: The Foundation for Effective Evals

Manual error analysis is the first step before automation. Without it, evals are useless: teams build metrics ignoring real issues.

Error Analysis Process

Gather 20–50 transcript → summary pairs from production or synthetics. Read them, log issues without classifying:

• Hallucination: "Maria will handle refactoring" instead of "might look into."
• Missing action items at the end of long transcripts.
• Poor compression: summary longer than transcript.

Group into a taxonomy of failure modes:

FAITHFULNESS FAILURES:

• Hallucinated commitments.
• Wrong attribution.

COMPLETENESS FAILURES:

• Lost action items.
• Missed decisions.

CONCISENESS FAILURES:

• Retelling instead of summarizing.

Count frequencies: 2–3 top errors set eval priorities. Time: half-day to a week, maximum ROI.

Code-Based Assertions: Fast Pass/Fail Checks

Principle: binary pass/fail, no 1–5 scales. Forces clear definitions of acceptability.

Examples for meeting summaries:

interface Transcript {
  text: string;
  duration_minutes: number;
  participants: string[];
}

interface Summary {
  text: string;
  action_items: ActionItem[];
  decisions: string[];
}

interface ActionItem {
  description: string;
  owner?: string;
  deadline?: string;
}

interface AssertionResult {
  name: string;
  pass: boolean;
  reason: string;
}

function runAssertions(transcript: Transcript, summary: Summary): AssertionResult[] {
  const results: AssertionResult[] = [];
  
  // Compression ratio: summary < 30% transcript
  const transcriptTokens = transcript.text.split(' ').length;
  const summaryTokens = summary.text.split(' ').length;
  const ratio = summaryTokens / transcriptTokens;
  results.push({
    name: 'Compression ratio',
    pass: ratio < 0.3,
    reason: `Ratio: ${ratio.toFixed(2)}`
  });
  
  // Action items have owner and deadline
  const aiComplete = summary.action_items.every(ai => ai.owner && ai.deadline);
  results.push({
    name: 'Action items completeness',
    pass: aiComplete,
    reason: aiComplete ? 'OK' : 'Missing owner/deadline'
  });
  
  return results;
}

Additional assertions:

• Presence of action_items section.
• No small talk in summary.
• Valid JSON for structured output.

LLM-as-Judge and Advanced Scorers

For nuanced metrics, use LLM-as-Judge with prompts for faithfulness, completeness. Mastra has 16 scorers: faithfulness, completeness, conciseness.

Example faithfulness prompt:

Rate the faithfulness of this summary to the transcript. Every claim must be supported. Score 0–1.
Transcript: [text]
Summary: [text]
Score:```

Integrate into CI: run on 50 test cases before deploy. Improvement metric: faithfulness from 0.64 to 0.78.

**Key Takeaways:**
- Start with error analysis: 2–3 failure modes cover 60% of issues.
- Combine code-based (fast), LLM-as-Judge (scalable), human (validation).
- Priorities: faithfulness > action accuracy > completeness.
- Binary pass/fail for clear decisions.
- Automate in CI for systematic gains.

— Editorial Team