Anatomy of a Production Voice Pipeline

Every voice pipeline diagram you’ll see from a vendor shows a straight line: microphone in, response out. That line hides the part of the system that actually determines whether callers trust the agent: what happens in the 300 milliseconds after someone stops talking, and what happens six weeks later when a model upgrade quietly makes transcription 4% worse on accented speech. This lesson gives you the reference architecture the rest of the course builds against, stage by stage, the same way the anatomy lesson in a text-based RAG course gives you the map before you debug any specific box on it.

The three planes, for voice

A production voice system splits cleanly into three loosely coupled planes, each with a different tolerance for latency and a different failure mode.

1. The input plane. Audio arrives from a phone call, a browser microphone, or a mobile SDK, gets transported over Twilio Media Streams, LiveKit, or a raw WebSocket, and is chunked into frames for downstream processing. This plane deals with codecs, sample rates, and jitter buffers, not meaning.
2. The real-time serving plane. This is the hot path a caller actually feels: voice activity detection gates which audio gets transcribed, streaming ASR turns speech into partial and final transcripts, turn detection decides when the caller is actually done talking, the dialogue manager (an LLM core) reasons and optionally calls tools, and streaming TTS turns the response back into audio that gets sent out over the same transport. Every millisecond here is a millisecond the caller waits.
3. The control plane. Evaluation against golden call sets, latency and WER tracing, guardrail checks, and conversation-quality dashboards. Nothing here sits on a live call, but it’s the only reason you’d know six months from now that a routine ASR model upgrade quietly dropped accuracy on Southern-accented English by four points.

flowchart TB
    subgraph IN["Input plane"]
        MIC["Caller audio\nphone / browser / mobile"] --> TRANS["Transport\nTwilio Media Streams, LiveKit, WebSocket"]
        TRANS --> FRAME["Audio framing\n20ms chunks, resampling"]
    end

    subgraph RT["Real-time serving plane (hot path)"]
        VAD["VAD\nis this speech?"] --> ASR["Streaming ASR\nNova-3 partial + final transcripts"]
        ASR --> TURN["Turn detection\nis the caller done?"]
        TURN --> DM["Dialogue manager\nclaude-sonnet-5 / GPT-5, tool calls"]
        DM --> TTS["Streaming TTS\neleven_flash_v2_5 / sonic-2"]
        TTS --> OUT["Audio egress\nback over transport"]
    end

    FRAME --> VAD

    subgraph CTL["Control plane"]
        EVAL["Golden-set WER\n+ latency audits"] -.-> ASR
        GUARD["Guardrails\n+ safety checks"] -.-> DM
        TRACE["Tracing + dashboards"] -.-> RT
    end

    style MIC fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style TRANS fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style FRAME fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style VAD fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style ASR fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TURN fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style DM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style OUT fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style EVAL fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style GUARD fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style TRACE fill:#fff7ed,stroke:#f59e0b,color:#0F172A

Why turn detection is its own stage

New teams often assume VAD alone tells you when a caller is done talking: silence starts, silence means done, respond. That assumption is wrong often enough to break a product. VAD answers a narrower question, is there speech energy in this audio frame right now, and it answers it in milliseconds using signal-processing features, with no idea whether “um, so my account number is…” followed by 400ms of silence means “I’m done” or “I’m looking up the number.”

Turn detection sits downstream of VAD and upstream of the dialogue manager specifically because it needs information neither stage alone has: VAD’s silence signal, plus the semantic content of the partial transcript, plus (in the best systems) prosodic cues like trailing intonation. Getting this stage wrong produces the two most caller-visible failure modes in voice AI: the agent interrupting a caller who paused to think, or the agent leaving three seconds of dead air after a caller finishes because the threshold was tuned too conservatively. We dedicate a full lesson (Module 2, Lesson 6) to the decision logic here because it disproportionately drives perceived quality relative to how small the code footprint is.

The latency budget, stage by stage

Here’s a rough production latency budget for a cascaded pipeline handling a support call, the kind of numbers you’ll be trading against each other for the rest of this course:

# Rough per-stage latency budget for a cascaded voice pipeline.
# Total target: under 900ms from "caller stops talking" to
# "caller hears the first word of the response."

LATENCY_BUDGET_MS = {
    "vad_endpointing":       150,  # silence threshold + turn-detection model
    "asr_final_transcript":  120,  # Nova-3 streaming, last-chunk finalization
    "dialogue_manager_ttft": 350,  # LLM time-to-first-token, claude-sonnet-5 / GPT-5
    "tts_first_audio_chunk": 180,  # eleven_flash_v2_5 streaming synthesis
    "network_egress":         60,  # transport back to caller
}

def total_budget_ms(budget: dict[str, int]) -> int:
    return sum(budget.values())

# total_budget_ms(LATENCY_BUDGET_MS) == 860ms, just under the
# ~900ms line where conversations start to feel laggy rather than live.

Notice what’s absent: no document parsing, no batch anything. Everything in the real-time plane runs streaming, because a caller has no patience for a spinner, unlike a chat user who will tolerate a few seconds for a thoughtful answer. This is the single biggest architectural difference between voice systems and the text-based RAG or agent systems you may have built before: nearly every stage must support incremental, streaming output, or the budget above simply doesn’t close.

The control plane is where voice quality actually gets managed

Teams that skip the control plane still ship a working demo, the input and real-time planes are enough to answer calls. What they can’t do is tell whether last Tuesday’s ASR model bump degraded word error rate on their specific caller population, whether the dialogue manager started hallucinating account details after a prompt tweak, or whether p95 latency crept from 850ms to 1.4 seconds over a month of adding tool integrations. Module 7 builds this plane in full: golden call sets scored for WER and task success, real-time latency percentile tracking, and guardrail suites that catch unsafe or off-brand responses before a real caller does. The habit worth starting now, in lesson two of the course, is the same one that pays off in every production system: log every turn (audio in, transcript, model response, audio out, and every stage’s latency) from day one, because that log is the only ground truth you’ll have when something goes quietly wrong.

One shortcut worth flagging before Module 2 builds each of these planes by hand: platforms like NextNeural implement all three as one managed service, an agent’s voice_config standing in for the real-time plane and per-call transcripts, sentiment, and webhooks standing in for the control plane, all behind a handful of REST calls instead of the pipeline the rest of this course builds. Lesson 3 covers exactly when that trade is the right one.