Streaming ASR Fundamentals

Every millisecond in the latency budget from Lesson 2 depends on one assumption holding: that speech becomes text as it’s spoken, not after. This lesson is where that assumption gets built, with real streaming ASR code you’ll extend for the rest of Module 2. Get this stage wrong and every later stage, turn detection, the dialogue manager, TTS, inherits a delay it can never make up.

Batch vs streaming, and why voice needs the latter

Batch transcription APIs (upload an audio file, get a transcript back) are built for a different problem: transcribing a podcast, a meeting recording, a voicemail, where the whole file exists before transcription starts and a few seconds of processing time is invisible to anyone. Feed a batch API into a live conversation and you inherit its worst property, it can’t return anything until it has received and processed the complete audio segment, which means the caller experiences dead air for at least as long as they were talking, plus processing time.

Streaming ASR flips this. Audio is sent as a continuous stream of small frames, typically 20-100ms each, over a persistent connection, and the model returns transcript fragments incrementally as it gains confidence in what was said. The practical effect: by the time a caller finishes a sentence, most of it is already transcribed, and only the last fragment needs finalizing. This is what makes the ~120ms ASR line item in Lesson 2’s latency budget realistic instead of fictional.

Interim vs final transcripts

Streaming ASR emits two kinds of results, and conflating them is the single most common bug in early voice pipelines. Interim (partial) transcripts are best-effort guesses that can still change as more audio provides context; a word like “there” might become “their” once the next word arrives. Final transcripts are locked in and won’t be revised. Nova-3 marks this explicitly with an is_final flag on every message.

The rule that matters in practice: use interim transcripts for anything that benefits from being fast and forgiving of being wrong, live captions on a screen, coarse activity signals for turn detection. Use only final transcripts to trigger anything expensive or hard to undo, starting the dialogue manager’s reasoning call, logging the turn, running a tool call. Acting on an interim transcript for those is how you end up generating a confident, well-formed answer to a question the caller hadn’t finished asking.

flowchart TB
    AUDIO["Caller audio\n20ms frames"] --> WS["Nova-3 streaming\nWebSocket connection"]
    WS --> INT["Interim transcript\nis_final: false"]
    WS --> FIN["Final transcript\nis_final: true"]
    INT --> CAP["Live captions /\ncoarse turn signal"]
    FIN --> TURN["Turn detection +\ndialogue manager trigger"]

    style AUDIO fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style WS fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style INT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style FIN fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style CAP fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style TURN fill:#fff7ed,stroke:#f59e0b,color:#0F172A

Wiring up Deepgram Nova-3

Here’s a minimal, real streaming client against Deepgram’s wss://api.deepgram.com/v1/listen endpoint. This is close to what you’d actually deploy behind an audio ingress service.

import asyncio
import json
import websockets

DEEPGRAM_URL = (
    "wss://api.deepgram.com/v1/listen"
    "?model=nova-3"
    "&encoding=linear16"
    "&sample_rate=16000"
    "&interim_results=true"
    "&endpointing=300"
)

async def stream_transcription(audio_frame_queue: asyncio.Queue, api_key: str):
    headers = {"Authorization": f"Token {api_key}"}

    async with websockets.connect(DEEPGRAM_URL, extra_headers=headers) as ws:
        async def send_audio():
            while True:
                frame = await audio_frame_queue.get()
                if frame is None:  # sentinel: caller hung up
                    await ws.send(json.dumps({"type": "CloseStream"}))
                    break
                await ws.send(frame)  # raw PCM16 bytes, 20ms each

        async def receive_transcripts():
            async for message in ws:
                data = json.loads(message)
                if data.get("type") != "Results":
                    continue
                alt = data["channel"]["alternatives"][0]
                transcript = alt["transcript"]
                if not transcript:
                    continue
                if data["is_final"]:
                    print(f"[FINAL] {transcript}")
                    # safe to trigger turn detection / dialogue manager here
                else:
                    print(f"[interim] {transcript}")

        await asyncio.gather(send_audio(), receive_transcripts())

The endpointing=300 parameter tells Deepgram to flag a likely end-of-utterance after 300ms of silence, which is a useful signal but not a substitute for the semantic turn detection layer we build in Lesson 6, since it knows nothing about sentence completeness. Note also interim_results=true, without it you silently fall back to something closer to batch behavior on each utterance.

Open-source and self-hosted alternatives

Not every production ASR call has to go through a hosted API. Mistral’s Voxtral (Voxtral Mini, 3B parameters, and Voxtral Small, 24B, both released under Apache 2.0 in 2025) is trained for speech transcription and broader audio understanding, and benchmarks competitively with gpt-4o-transcribe on multilingual transcription accuracy while running entirely on infrastructure you control. OpenAI’s Whisper large-v3, open-weight since 2023, is still the baseline most self-hosted stacks compare against. Neither ships the batteries-included streaming WebSocket and diarization layer Nova-3 does out of the box; you’re responsible for building the chunking, buffering, and partial-result logic this lesson’s Deepgram client gets for free, typically behind a serving framework like vLLM or TensorRT-LLM. What you get back is no per-minute API bill, no audio leaving your infrastructure, and a cost curve that’s flat with GPU capacity instead of linear with call volume, the self-hosting tradeoff Lesson 22 works through in dollar terms.

WER benchmarks that matter in production

Word error rate (WER), the percentage of words a transcript gets wrong relative to a human-verified reference, is the headline metric for ASR quality, and Nova-3 benchmarks in the 5-8% range on clean, general American English audio as of 2026, competitive with or ahead of gpt-4o-transcribe on the same conditions. That number is close to meaningless in isolation, though, because WER is conditional on the audio distribution you actually feed it. Clean, single-speaker, native-accent audio might see 4-6% WER; a noisy call center line with cross-talk and regional accents can push past 15-20% on the same model without any bug, just a harder input distribution, which is exactly why Lesson 5 exists.

A useful production rule of thumb: WER under 8% is generally fine for a dialogue manager to reason over confidently, since LLMs tolerate minor transcription noise well. Above 12-15%, error compounds visibly, the LLM starts responding to garbled input, and callers notice the agent “not listening.” When you see WER creep into that range in your control-plane dashboards (Module 7), the fix is rarely swapping ASR vendors; it’s almost always tuning for the specific noise, accent, or audio-quality conditions actually present in your traffic, the subject of the next lesson.