Choosing a TTS Engine for Production

Every earlier lesson in this course has been about getting words out of audio and words out of a language model. This lesson flips the pipeline: getting words back into audio that a human will actually listen to, judge, and decide whether to trust. The engine you pick here determines whether your agent sounds like a person mid-conversation or like a phone tree reading a script, and it sets a hard floor on how fast your whole system can respond, because no amount of clever orchestration elsewhere in the pipeline rescues a TTS call that takes 900ms just to start speaking.

The three axes that actually matter

Vendors market TTS on “quality,” but production teams need three separate, sometimes conflicting numbers: time-to-first-audio-byte (how long before the caller hears anything), expressiveness (does it sound like a person having this specific conversation, with appropriate pacing and emotional color), and cost per unit of audio produced. A model that wins on one axis routinely loses on another, and the entire skill of choosing a TTS engine is knowing which axis your product actually needs to win.

Time-to-first-audio-byte, not total generation time, is the number worth obsessing over for any live conversational surface. A caller doesn’t experience “the model finished generating 8 seconds of audio in 1.2 seconds,” they experience “how long was I sitting in silence after I stopped talking.” Streaming-first architectures return the first audio chunk before the rest of the utterance is even synthesized, and the gap between flash-tier and quality-tier models on this specific number is often 5-10x, even when total throughput is comparable.

Comparing the field

These numbers move constantly as vendors ship new checkpoints, so treat this table as a way of reasoning about tradeoffs rather than a permanent scorecard. What doesn’t move is the underlying pattern: flash-tier and Cartesia’s sonic line are architected around streaming a small look-ahead window of text, producing the first phoneme’s audio before the sentence is fully known, while eleven_v3 does more work per chunk to get prosody and emotional inflection right, which necessarily costs time.

Kokoro deserves a second look for teams past the prototype stage: it’s an 82-million-parameter, Apache-2.0-licensed TTS model, small enough to run at real-time speed on a single modest GPU, and it costs nothing per character since you own the inference. It won’t match eleven_v3’s expressiveness or ElevenLabs’ voice catalog depth, but at high enough call volume the compute cost of self-hosting it can undercut every hosted option in this table, the same crossover Lesson 22 works out in dollar terms.

Language coverage and voice catalog

Beyond latency and quality, production teams in multi-market products care about two things vendors handle very differently: language coverage and voice catalog depth. ElevenLabs supports dozens of languages with reasonable quality across all of them and a large marketplace of pre-built and custom voices, which matters if your product needs a consistent brand voice across English, Spanish, and Hindi calls. Cartesia’s language coverage is narrower but its low-latency models are extremely competitive in the languages they do support, which is the right tradeoff if you’re serving a single-language, latency-critical market like US English telephony support.

import httpx

async def synthesize_flash(text: str, voice_id: str, api_key: str) -> bytes:
    """Low-latency, non-streaming synthesis for short conversational turns."""
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            url,
            headers={"xi-api-key": api_key},
            json={
                "text": text,
                "model_id": "eleven_flash_v2_5",
                "voice_settings": {"stability": 0.5, "similarity_boost": 0.8},
            },
        )
        resp.raise_for_status()
        return resp.content


async def synthesize_narration(text: str, voice_id: str, api_key: str) -> bytes:
    """Higher-quality synthesis for cached, offline narration content."""
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(
            url,
            headers={"xi-api-key": api_key},
            json={"text": text, "model_id": "eleven_v3"},
        )
        resp.raise_for_status()
        return resp.content

Making the call for your product

The decision usually collapses to one question: is a human waiting on the other end of this specific audio, right now? If yes, and the surface is a phone call or a voice agent turn, eleven_flash_v2_5 or Cartesia sonic-2 are the only reasonable defaults, because every hundred milliseconds of TTS latency is a hundred milliseconds added to a conversational turn that users perceive as awkward silence past roughly 300-500ms of accumulated delay. If the audio is generated once and served many times, eleven_v3 wins on nearly every dimension that matters, since its latency cost is paid exactly once and amortized across every future playback. Teams building fully native speech-to-speech experiences with gpt-realtime or Gemini Live skip this decision for the conversational path entirely, since audio generation is bundled into the model’s output tokens, a tradeoff we cover in depth in Module 6.