Cascaded vs Native Speech-to-Speech

Every team building a voice agent eventually hits the same fork in the road: chain together an ASR model, an LLM, and a TTS model, or hand the whole conversation to a single model that hears and speaks directly. Both are legitimate 2026 production choices, and the wrong choice shows up not as a crash but as a system that’s subtly worse than it needed to be, either slower than a competitor’s or less observable than your compliance team will accept. This lesson gives you the framework to choose correctly before you’ve written a line of pipeline code.

Two architectures

A cascaded pipeline runs audio through three discrete, swappable stages: streaming ASR (Deepgram Nova-3, or OpenAI’s gpt-4o-transcribe) converts speech to text, an LLM (claude-sonnet-5 or GPT-5) reasons over that text and optionally calls tools, and streaming TTS (ElevenLabs eleven_flash_v2_5 or Cartesia sonic-2) converts the response back to audio. Each stage is a separate API call, each has an independent latency and cost line item, and a text transcript exists at every hop as a natural place to log, guardrail, and debug.

A native speech-to-speech model processes audio in and audio out through a single model call, with no intermediate text representation exposed to the caller. OpenAI’s gpt-realtime and Google’s gemini-2.5-flash-native-audio-preview-09-2025 are the two production-grade hosted options as of 2026, and Kyutai’s open-weight Moshi is the self-hostable version of the same architecture, covered in Module 6. The model reasons over acoustic features directly, which means it can pick up on tone, hesitation, and emphasis that a transcript would flatten away, and it does the whole “understand and respond” step in one round trip instead of three.

flowchart TB
    subgraph CASC["Cascaded"]
        A1["Caller audio"] --> A2["ASR: Nova-3\ntext transcript"]
        A2 --> A3["LLM: claude-sonnet-5\ntext response"]
        A3 --> A4["TTS: eleven_flash_v2_5\naudio response"]
    end

    subgraph NATIVE["Native speech-to-speech"]
        B1["Caller audio"] --> B2["gpt-realtime /\ngemini-2.5-flash-native-audio\naudio in, audio out"]
        B2 --> B3["Audio response"]
    end

    style A1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style A2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style A3 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style A4 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style B1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style B2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style B3 fill:#fff7ed,stroke:#f59e0b,color:#0F172A

The tradeoffs, head to head

Neither column is strictly better. A cascaded stack wins on anything that needs a compliance-grade audit trail, fine-grained tool orchestration, or the freedom to mix best-in-class vendors per stage. Native speech-to-speech wins on raw conversational latency and naturalness, particularly for casual, low-stakes interactions where a caller would actually notice the difference between 400ms and 900ms of lag.

Choosing for your use case

Support agents handling account changes, refunds, or anything with a compliance requirement should default to cascaded: you need the transcript as a first-class artifact for guardrailing before speech and for audit logs after the call, both covered in Module 7. IVR replacement leans cascaded too, mostly because DTMF fallback and call-transfer logic are easier to wire into an explicit dialogue-manager code path than into an opaque model. In-app consumer assistants, where the interaction is casual, latency-sensitive, and low-stakes (a music app, a game companion), are the strongest case for native speech-to-speech, since the naturalness and speed advantage is what users will actually feel, and the observability tradeoff matters less when nothing is being audited. Outbound calling sits in between and usually goes cascaded, because compliance requirements (consent scripts, opt-out handling) again favor an explicit, guardrailable text layer.

A third option: not building the cascade yourself

Both columns above assume you’re assembling and operating every stage. A managed voice agent platform like NextNeural collapses agent creation, telephony, and post-call analytics behind one API, so a lot of what Modules 2 through 5 build by hand is available as configuration instead. Creating an agent is one call to POST /api/voice/agents with a voice_config that names a provider (nextneural, sarvam, or elevenlabs) and a speaker; wiring that agent into an outbound or inbound line is a second call to create a campaign; triggering an individual call is POST /v1/calls with a phone number and campaign ID:

curl -X POST https://api.nextneural.ai/api/voice/agents \
  -H "Authorization: Bearer nn_k_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Aria",
    "role": "Support",
    "voice_config": {
      "provider": "nextneural",
      "model": "neuralvoice",
      "speaker": "anushka",
      "language_code": "en-IN"
    }
  }'

This is a legitimate default for teams that don’t need per-stage vendor swapping or bespoke tool-orchestration code, and a real cost against the cascaded pipeline’s flexibility: you inherit the platform’s transcript, sentiment, and webhook shape (Lesson 21 covers this in depth) instead of designing your own control plane. The build-vs-buy question is really “does this product need a custom dialogue core,” and it’s worth asking before Module 2 rather than after.

A hybrid worth knowing

A pattern that’s become common by 2026 splits the difference: run native speech-to-speech for the live, caller-facing turn-taking to get its latency and naturalness benefits, while asynchronously piping the same audio through a cascaded ASR pass purely for logging, evaluation, and guardrail auditing after the fact, off the critical path. This gets you the best perceived experience and still produces the clean transcript your control plane needs for the evaluation techniques in Module 7. It costs an extra ASR call per minute of audio, typically a fraction of a cent, which is cheap insurance against flying blind on quality.

Module 6 returns to native speech-to-speech in depth, once you’ve built a full cascaded pipeline in Modules 2 through 5 and have a concrete baseline to compare it against. For now, the rule of thumb worth carrying forward: default to cascaded when you need control and auditability, reach for native speech-to-speech when you need speed and naturalness above all else, and don’t decide until you know which of those your specific use case actually needs more.