Every earlier module in this course built a stage of the pipeline: ASR, the dialogue core, TTS, telephony. This lesson is where you find out whether the stages, assembled, actually work for a real caller. A voice agent that looks flawless on a benchmark transcript can still fail the person on the phone, and the gap between those two pictures is exactly what a proper evaluation framework exists to close.
WER is a component metric, not a verdict
Word error rate tells you how faithfully Deepgram Nova-3 turned audio into text. It says nothing about whether the LLM core understood the resulting text, called the right tool, or produced a response the caller could act on. Treat WER the way Module 7’s earlier lessons treated recall in a RAG system: necessary, insufficient, and dangerous if it’s the only number on your dashboard.
# eval_metrics.py - component metrics, kept separate on purpose
def score_turn(turn: dict) -> dict:
return {
"wer": word_error_rate(turn["asr_transcript"], turn["reference_transcript"]),
"intent_correct": turn["predicted_intent"] == turn["gold_intent"],
"tool_call_correct": turn["tool_call"] == turn["gold_tool_call"],
"ttfa_ms": turn["time_to_first_audio_ms"],
}A 4% WER call can still fail if the LLM core, running on claude-sonnet-5 or GPT-5, misreads a correctly transcribed sentence, skips a required tool call, or the TTS output mispronounces an account number. Each of those is a different failure with a different fix, and a blended score hides which one happened.
Latency percentiles, not averages
Callers judge a system by its worst moments. Report p50, p95, and p99 time-to-first-audio (TTFA) separately, and alert on the tail, not the mean.
flowchart TB
T1["Caller finishes speaking"] --> T2["ASR endpoint detection\n~150ms (Nova-3 streaming)"]
T2 --> T3["LLM first token\np50 300ms / p95 900ms / p99 2100ms"]
T3 --> T4["TTS first audio chunk\np50 120ms / p95 280ms"]
T4 --> T5["Caller hears response\nTTFA p50 ~570ms / p95 ~1.3s / p99 ~2.6s"]
style T1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style T2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style T3 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style T4 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style T5 fill:#fff7ed,stroke:#f59e0b,color:#0F172AA p99 of 2.6 seconds means roughly 1 in 100 turns feels like a stall to the caller, and stalls are exactly what get repeated in a complaint transcript, even if the p50 dashboard looks pristine.
Task success rate and containment rate
These are the numbers that actually correlate with whether the agent is doing its job. Task success rate asks: did the caller’s stated goal get accomplished (a claim was filed, a balance was checked, a reservation was changed)? Containment rate asks: did the call finish without escalating to a human agent? Neither can be inferred from ASR or latency metrics alone; both require labeling the outcome of the whole conversation, not any single turn.
def containment_rate(calls: list[dict]) -> float:
resolved = sum(1 for c in calls if not c["escalated_to_human"])
return resolved / len(calls)Track these split by intent category. A blended 70% containment rate can hide a category (say, claims disputes) sitting at 30%, exactly the kind of stage-level signal a single top-line number erases.
Turn-level vs. conversation-level evaluation
Turn-level eval scores one exchange in isolation: was the intent classified correctly, was the tool call right, was the response accurate. Conversation-level eval scores the whole call: did the caller’s actual goal get met, was the tone appropriate throughout, did the agent recover gracefully from an early misunderstanding. A voice agent can pass every turn-level check and still fail conversation-level eval if it never resolves accumulated confusion from turn 3 by turn 8. Run both, and treat conversation-level scores as the one that gates launch, per the launch-gate pattern used in the capstone.
Building a golden set from real transcripts
Synthetic, scripted test calls are a starting point, not a golden set. Real callers interrupt, trail off, restate themselves, and have background noise. Sample your golden set from anonymized production or pilot-phase call transcripts, stratified across intent category, outcome (resolved vs. escalated), and audio conditions. Aim for 150-300 real conversations before you trust the eval numbers it produces.
Managed platforms often ship a coarser version of this signal automatically. NextNeural attaches a sentiment field (positive, neutral, or negative) to every call record, delivered via the call.ended webhook covered in Lesson 21, with no eval infrastructure required. It’s a useful first-pass filter for which calls are worth pulling into your golden set, not a substitute for the dimension-level rubric below, since “sentiment: negative” tells you a call went badly without telling you whether it was a tool-call error, a tone problem, or an unresolved intent.
LLM-as-judge scoring
Use a specific rubric, not a vague “rate this call” prompt. Score each dimension independently with claude-sonnet-5 as the judge:
JUDGE_RUBRIC = """
Score this call transcript on each dimension, 1-5, with a one-sentence justification:
1. task_completion: was the caller's stated goal accomplished?
2. tone_appropriateness: professional, empathetic where needed, no robotic repetition?
3. tool_use_correctness: were the right tools called with the right arguments?
4. recovery: if the agent misunderstood early, did it correct course by the end?
Return JSON only, one score object per dimension.
"""Independent, checkable dimensions is what makes judge scores discriminate between a genuinely good call and a mediocre one that merely sounded confident.