Latency Budgeting End to End

Every previous lesson in this course optimized one stage of the pipeline in isolation: pick a fast STT model, chunk synthesis for early TTS start, choose a full-duplex transport. This lesson is where those choices get added up into a single number that determines whether your product feels like a conversation or feels like a walkie-talkie with a satellite delay. Latency budgeting is the discipline of treating end-to-end response time as a fixed resource you allocate deliberately across stages, rather than an emergent property you discover after launch when users start complaining.

Why 800-1200ms is the target, not an arbitrary nice-to-have

Human conversational turn-taking has a well-studied natural rhythm: in ordinary human-to-human conversation, the gap between one person finishing a sentence and the other starting to respond averages a few hundred milliseconds. Past roughly 1200ms of silence after a user stops talking, most people register a pause as unnatural, start wondering if they were heard, and often start talking again, which is exactly the collision that makes voice interfaces feel broken. That’s not a soft UX preference, it’s the actual perceptual threshold your architecture is competing against, and it’s why 800-1200ms end-to-end is treated as the de facto target for conversational voice AI rather than a number pulled from a vendor’s marketing page.

The four-stage budget

Every voice agent turn, cascaded or partially native, moves through roughly the same four stages, and each one has a realistic latency range worth knowing cold when you’re staring at a trace trying to figure out where the time went.

flowchart TB
    A["User stops speaking"] --> STT["STT: audio to text\n~150-300ms"]
    STT --> LLM["LLM: time-to-first-token\n~200-600ms (fast models)\nto 1000ms+ (slow/long prompts)"]
    LLM --> TTS["TTS: time-to-first-audio-byte\n~50-150ms (flash tier)\nto 400-900ms (quality tier)"]
    TTS --> NET["Network + jitter buffer\n~50-150ms"]
    NET --> B["User hears response start"]

    style A fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style STT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style LLM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style NET fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style B fill:#fff7ed,stroke:#f59e0b,color:#0F172A

Add the low end of every range and you land around 450-600ms, comfortably inside the target. Add the high end and you’re past 2000ms, well outside it. The entire job of latency budgeting is choosing, deliberately, which stages you can afford to run at the high end (because quality matters more there) and which stages need to run at the low end no matter what.

Reading a latency trace

When a pipeline’s total latency blows the budget, the instinct is often to optimize whatever stage feels intuitively slow, TTS is a common scapegoat because “generating audio” sounds expensive. The correct approach is to instrument every stage independently and let the trace tell you where the time actually went, because the dominant cost is frequently the LLM’s time-to-first-token, particularly when a system prompt has grown large with tool definitions, few-shot examples, and conversation history that gets reprocessed on every single turn. A pipeline spending 1400ms on LLM time-to-first-token and 120ms on TTS has a prompt engineering and model-selection problem, not a TTS problem, and shipping a faster TTS engine in that situation is optimizing a stage that was never the bottleneck.

import time

class LatencyTracer:
    def __init__(self):
        self.marks: dict[str, float] = {}

    def mark(self, stage: str):
        self.marks[stage] = time.monotonic()

    def report(self) -> dict[str, float]:
        stages = list(self.marks.items())
        durations = {}
        for (name_a, t_a), (name_b, t_b) in zip(stages, stages[1:]):
            durations[f"{name_a} -> {name_b}"] = round((t_b - t_a) * 1000, 1)
        return durations


# Usage inside the turn handler:
tracer = LatencyTracer()
tracer.mark("user_stopped_speaking")
# ... STT ...
tracer.mark("stt_complete")
# ... LLM first token ...
tracer.mark("llm_first_token")
# ... TTS first byte ...
tracer.mark("tts_first_byte")
# ... audio reaches playback buffer ...
tracer.mark("audio_playing")

log.info("turn_latency", **tracer.report())

Shipping this kind of per-turn tracing to a dashboard from day one, rather than adding it after users start complaining, is what turns “the assistant feels slow sometimes” into an actionable, stage-attributed number your team can actually fix.

Strategies at each stage

At the STT stage, use streaming transcription with partial hypotheses (Deepgram Nova-3 and similar streaming ASR emit interim results before the final transcript), so downstream processing can begin reasoning about likely intent before the user has even finished the sentence. At the LLM stage, the biggest lever is usually prompt size: a bloated system prompt with verbose tool schemas and long few-shot examples adds real, measurable milliseconds to every single turn, and prompt caching for the static portions of that context (available on both Claude and GPT-5 class models) can cut that cost dramatically on the second and subsequent turns of a session. At the TTS stage, sentence-chunked streaming synthesis (Module 4) overlaps generation and speech rather than waiting for a complete response, and picking a flash-tier model over a quality-tier one is the single highest-leverage lever if the current stage is quality-tier by default. At the network stage, terminate connections at a region close to your users, and prefer WebRTC’s purpose-built jitter buffering over ad hoc buffering logic for any client-facing leg, since reinventing jitter handling rarely beats a stack designed for exactly that problem.