Audio-Native Model

Audio-native models — also called native speech-to-speech or audio-in/audio-out models — are AI systems that take raw audio as input and generate raw audio as output using a single network trained end-to-end, rather than chaining together separate speech-to-text, language, and text-to-speech models. Because the conversation never fully collapses into a text transcript, tone, pacing, breath, and emotion survive the round trip, and the model can decide what to say and how to say it in one forward pass.

The category became the default architecture behind realtime voice APIs shipped between 2024 and 2026, including OpenAI’s Realtime API (gpt-realtime), Google’s Gemini Live API with native audio, Amazon’s Nova Sonic, and Kyutai’s open-weight Moshi.

From Cascaded Pipelines to Native Audio

For most of the last decade, voice assistants ran a cascaded pipeline: automatic speech recognition (ASR) converts speech to text, a language model reasons over that text, and a text-to-speech (TTS) model converts the reply back into audio. Each stage is a separate model with its own latency, and the transcript handoff discards everything that isn’t words — sarcasm, hesitation, a laugh.

Audio-native models collapse this into one model operating on audio tokens directly:

graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    A([Microphone Input]):::data --> B(Neural Audio Codec: Waveform to Tokens):::process
    B --> C(Single Transformer: Audio Tokens In and Out):::process
    C --> D(Neural Audio Codec: Tokens to Waveform):::process
    D --> E([Streamed Speaker Output]):::output

Compared with a cascaded stack of ASR, LLM, and TTS, this removes two model handoffs and the lossy text bottleneck between them — which is where most of a cascaded pipeline’s latency and emotional flatness comes from.

The Neural Codec and Token Loop

The component that makes this possible is the neural audio codec — a learned encoder/decoder that compresses raw waveforms into a low-frame-rate stream of discrete tokens and back. Kyutai’s Mimi codec, for example, runs at 12.5 Hz and roughly 1.1 kbps while preserving both semantic content and acoustic detail.

• Encoding — the codec turns speech into discrete audio tokens, the audio equivalent of word tokens.
• Modeling — a transformer, often initialized from a text LLM backbone, predicts those tokens autoregressively, the same way it predicts word tokens.
• Grounding — some models (Moshi) generate a parallel text “inner monologue” token stream purely to keep the output linguistically coherent.
• Decoding — the codec converts predicted tokens back into a waveform, streamed to the speaker in small chunks as they’re produced.

Key Models and Players (2025–2026)

Figures are compiled from Artificial Analysis’ speech-to-speech benchmark and vendor documentation as of mid-2026; treat exact numbers as directional, since they shift with prompt length and network conditions.

Where It Falls Short

Native audio models are harder to steer with domain-specific rules than a text LLM in a cascaded stack, because there’s no intermediate transcript to prompt-engineer against — and tooling for inspecting why a native model said something is still immature relative to text-based debugging. Teams that need strict auditability, such as regulated call centers, sometimes keep a cascaded pipeline specifically so a transcript exists at every step.

Official Implementation Snippet

Below is a representative example of an audio-native realtime session using OpenAI’s Python SDK. Notice there’s no separate transcription call or TTS call — audio is simply the input and output modality of one session:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def run_voice_session():
    async with client.realtime.connect(model="gpt-realtime") as connection:
        await connection.session.update(session={
            "modalities": ["audio", "text"],
            "voice": "marin",
            "turn_detection": {"type": "semantic_vad"},
        })

        # Stream microphone audio in as it's captured
        async def send_audio(pcm_chunk: bytes):
            await connection.input_audio_buffer.append(audio=pcm_chunk)

        # Audio comes back already spoken — no transcript, no separate TTS call
        async for event in connection:
            if event.type == "response.output_audio.delta":
                play(event.delta)  # raw PCM chunk, ready for playback

asyncio.run(run_voice_session())

References

• Introducing next-generation audio models in the API — OpenAI
• Updates for developers building with voice — OpenAI Developers
• Gemini 2.5 Native Audio upgrade — Google
• How to use Gemini Live API native audio in Vertex AI — Google Cloud
• Speech-to-speech model benchmark — Artificial Analysis
• Neural audio codecs explained — Kyutai
• Moshi: a speech-text foundation model for real-time dialogue (arXiv)