Every lesson before this one assumed a cascaded architecture: a dedicated STT model converts speech to text, a text-based LLM reasons and generates a response, and a dedicated TTS model converts that response back to speech, three specialized models wired together in series. That’s still the dominant production pattern, and for good reason, but it’s not the only one. This lesson opens the module on native speech-to-speech models, a genuinely different architecture where a single model consumes audio and produces audio, with no separate transcription step in between, and understanding how that changes the game is essential before you build against one in the next two lessons.
What “native” actually means
In a cascaded pipeline, audio in and audio out are handled by three independently trained models that never see each other’s internal representations, only the flattened text that passes between them. A native speech-to-speech model, by contrast, is trained to process audio as tokens directly, similar in spirit to how a text LLM processes word or subword tokens, and to generate audio tokens directly as output, without ever materializing an intermediate plain-text transcript as part of its core generation path. Models like OpenAI’s gpt-realtime and Google’s gemini-2.5-flash-native-audio-preview-09-2025 are built this way: audio goes in, the model reasons about it, and audio comes out, all inside a single forward pass through one model rather than three round trips through three.
flowchart TB
subgraph CASCADE["Cascaded pipeline"]
A1["User audio"] --> STT1["STT model\ntext transcript"]
STT1 --> LLM1["LLM\ntext response"]
LLM1 --> TTS1["TTS model\naudio output"]
end
subgraph NATIVE["Native speech-to-speech"]
A2["User audio tokens"] --> M["Single model\naudio tokens in -> audio tokens out"]
M --> OUT["Audio output"]
end
style A1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style STT1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style LLM1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style TTS1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style A2 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style M fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style OUT fill:#fff7ed,stroke:#f59e0b,color:#0F172AWhat this buys you: latency and expressiveness
Two benefits fall directly out of collapsing three models into one. First, latency: a cascaded pipeline pays a serial hand-off cost at every boundary, STT has to finish (or at least produce a usable partial) before the LLM can meaningfully begin, and the LLM has to produce text before TTS can start synthesizing it, even with the streaming and chunking techniques from Module 4 smoothing that over. A single native model removes two of those hand-offs entirely, generating audio output as a direct continuation of processing audio input, which is a meaningfully different latency profile, not just an incremental improvement.
Second, and arguably more important for how the assistant actually feels to talk to: expressiveness. A cascaded pipeline’s LLM stage only ever sees flattened text, “that’s not what I meant,” with no signal about whether that was said flatly, sarcastically, or with real frustration, because STT’s entire job is to discard everything except the words. A native model conditions directly on the acoustic signal, so pacing, hesitation, pitch, and emphasis remain available information the model can use both to understand intent and to shape the prosody of its own response, producing audio that can pick up on and mirror emotional tone in a way a cascaded pipeline structurally cannot, no matter how good its individual TTS engine is.
What it costs you: control and observability
The same collapse that buys latency and expressiveness takes away something cascaded architectures give you almost for free: a legible, independently inspectable intermediate representation. In a cascaded pipeline, the text transcript between STT and LLM is a first-class artifact you can log, audit, redact, or even edit before it reaches generation, which matters enormously for compliance-sensitive domains like healthcare or financial services where “what exactly did the system believe the user said” needs to be a queryable fact, not an inference. Native models don’t expose an equivalent artifact as part of their core path, since there’s no separate transcription stage to intercept.
The same applies to swappability: a cascaded pipeline lets you upgrade just the STT model when a better one ships, or switch just the TTS engine when you need different voice characteristics, without touching the other two stages, because they’re genuinely independent components. A native model is one model; improving its transcription accuracy or its voice quality means waiting for the vendor’s next checkpoint, not swapping a component yourself.
An open-weight option: Moshi
gpt-realtime and Gemini Live are both closed, hosted models. Kyutai Labs’ Moshi, released as open weights in 2024 alongside its underlying Mimi neural audio codec, is a genuine native speech-to-speech model you can run yourself: a roughly 7B-parameter model trained to listen and speak in full duplex, processing two audio streams simultaneously rather than waiting for a turn boundary, and small enough to run inference on a single consumer-grade GPU. It won’t match gpt-realtime or Gemini Live on general reasoning depth or tool-calling sophistication, both are trained at frontier scale with far more parameters behind the reasoning, but it’s a real option when self-hosting, data residency, or per-minute cost independence matter more than best-in-class conversational intelligence, and it’s the clearest evidence that the native speech-to-speech architecture in this lesson isn’t limited to a handful of closed vendors.
Tool-calling and control flow
Both architectures support tool-calling, letting the model invoke functions like looking up an order or checking a calendar mid-conversation, but the mechanics differ. Cascaded pipelines route tool calls through the same text-based LLM interface every other lesson in this course has used, meaning existing tool-calling code, prompt patterns, and debugging habits transfer directly. Native speech-to-speech models like gpt-realtime support tool-calling within the realtime session itself, but because the underlying reasoning happens over audio tokens rather than an inspectable text stream, tracing exactly why the model chose to call a given tool at a given moment is less transparent than in a cascaded architecture, where you can point at the literal text the LLM saw and generated. Neither gap is disqualifying, but both are real, and they’re exactly the tradeoffs the next two lessons will walk through hands-on as we build against gpt-realtime and Gemini Live directly.