Streaming & Chunked Synthesis

The previous lesson picked a TTS engine on the assumption that “fast” is a property of the engine alone. It isn’t. Even the fastest engine sounds slow if you feed it a fully-formed paragraph and wait for the whole thing to synthesize before playing a single sample. This lesson is about the orchestration pattern that actually delivers low perceived latency in production: chunking the LLM’s streaming output at sentence boundaries and pipelining synthesis so the caller hears the first words while the model is still writing the last ones.

Why full-response synthesis is a latency bug, not a quality feature

Picture the naive pipeline: the LLM streams tokens, your code accumulates them into a buffer, waits for a stop token, then hands the complete string to the TTS engine, then plays the resulting audio file. Every stage here is serial, and the user experiences the sum of all three delays stacked end to end, even though two of the three (LLM generation and TTS synthesis) could be running concurrently on different parts of the response. For a 3-sentence reply that takes 1.5 seconds to generate and 400ms to synthesize as one whole clip, the caller sits in dead silence for close to 2 seconds. That’s an eternity in a phone conversation, well past the point where humans start repeating themselves or hanging up.

The fix is architecturally simple and operationally the single highest-leverage change you can make to a voice pipeline’s felt responsiveness: synthesize each sentence as soon as it’s complete, not after the whole response finishes.

flowchart TB
    LLM["LLM token stream"] --> CHUNK["Sentence-boundary chunker"]
    CHUNK --> S1["Sentence 1 ready"]
    CHUNK --> S2["Sentence 2 ready"]
    CHUNK --> S3["Sentence 3 ready"]
    S1 --> TTS1["TTS stream: sentence 1"]
    S2 --> TTS2["TTS stream: sentence 2"]
    S3 --> TTS3["TTS stream: sentence 3"]
    TTS1 --> BUF["Playback buffer"]
    TTS2 --> BUF
    TTS3 --> BUF
    BUF --> OUT["Caller hears continuous audio"]

    style LLM fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style CHUNK fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style S1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style S2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style S3 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS3 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style BUF fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style OUT fill:#fff7ed,stroke:#f59e0b,color:#0F172A

Chunking at the right granularity

The chunker’s job is to find natural break points in a token stream that hasn’t finished yet, without waiting so long that you lose the latency win, and without cutting so aggressively that prosody breaks. Sentence-final punctuation (., ?, !) is the standard boundary because TTS models use sentence-level context to shape intonation; splitting mid-sentence on commas or clause boundaries routinely produces flat or oddly emphasized audio because the model no longer has the full clause to inform its pacing decisions. Most production chunkers also enforce a minimum chunk length (skip synthesizing a lone “Ok.” as its own chunk if a longer sentence is one token away) and a maximum wait timeout, so a pathologically long sentence doesn’t stall the whole pipeline waiting for a period that never arrives soon enough.

import re
import asyncio

SENTENCE_END = re.compile(r'(?<=[.!?])\s+')

async def stream_sentences(token_stream):
    """Yield complete sentences as they become available from an
    LLM token stream, without waiting for the full response."""
    buffer = ""
    async for token in token_stream:
        buffer += token
        parts = SENTENCE_END.split(buffer)
        # Everything but the last piece is a complete sentence;
        # the last piece is a possibly-incomplete tail we keep buffering.
        for sentence in parts[:-1]:
            if sentence.strip():
                yield sentence.strip()
        buffer = parts[-1]
    if buffer.strip():
        yield buffer.strip()

Streaming synthesis over a persistent connection

Chunking text is only half the pattern. The other half is keeping a persistent WebSocket open to the TTS engine so each sentence chunk can be pushed in without paying connection setup cost per sentence. Both ElevenLabs and Cartesia expose exactly this shape of API.

import asyncio
import json
import websockets

async def stream_tts(sentence_queue: asyncio.Queue, voice_id: str, api_key: str, playback):
    uri = f"wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input?model_id=eleven_flash_v2_5"
    async with websockets.connect(uri, extra_headers={"xi-api-key": api_key}) as ws:
        await ws.send(json.dumps({"text": " ", "voice_settings": {"stability": 0.5}}))

        async def sender():
            while True:
                sentence = await sentence_queue.get()
                if sentence is None:
                    await ws.send(json.dumps({"text": ""}))  # close signal
                    break
                await ws.send(json.dumps({"text": sentence + " "}))

        async def receiver():
            async for message in ws:
                data = json.loads(message)
                if data.get("audio"):
                    await playback.enqueue(data["audio"])  # base64 PCM/mp3 chunk

        await asyncio.gather(sender(), receiver())

Cartesia’s wss://api.cartesia.ai/tts/websocket follows the same shape: open once, push text as it becomes available, receive audio chunks as they’re synthesized, and close when the sentence queue is exhausted. The sonic-2 model is tuned specifically for this incremental-input pattern, often returning the first audio chunk before the input sentence has finished transmitting.

The playback buffer: absorbing jitter without gaps

Sentence chunks won’t arrive at perfectly even intervals. Synthesis time varies with sentence length and momentary API load, and network jitter adds its own variance. Playing each chunk the instant it arrives, with no buffer, produces audible gaps whenever a chunk is a beat late. The standard fix, borrowed directly from real-time video streaming, is a small jitter buffer: hold 200-400ms of synthesized audio in a queue before starting playback, and keep that lead as long as chunks keep arriving faster than playback consumes them. This buffer is a tuning knob, not a fixed constant: too small and you reintroduce gaps under load, too large and you’ve re-added the latency this entire pattern was built to eliminate.