Building the Full-Duplex Pipeline

Every stage covered so far, from STT through TTS through transport, has implicitly assumed a clean turn-taking structure: the user speaks, then the agent speaks, then the user speaks again. Real conversation doesn’t work that way. People interrupt, correct themselves mid-sentence, and talk over an assistant that’s said something wrong or something they already know. A pipeline that can’t handle that isn’t a lesser version of a real assistant, it’s a different, more frustrating product wearing the same UI. This lesson builds the full-duplex architecture that makes a voice agent feel like it’s actually listening, not just waiting for its turn to talk.

Half-duplex vs full-duplex

A half-duplex pipeline strictly alternates between two states: listening or speaking, never both. It’s simpler to build, and it’s the default shape you get if you wire STT, LLM, and TTS together without deliberately engineering for interruption, because the natural request-response mental model maps directly onto it. The problem is that it makes barge-in structurally impossible: if the microphone stream isn’t being processed while TTS is playing, no amount of user speech registers until the agent finishes talking, which in a long response can mean several seconds of a caller repeating “wait, no” into a system that isn’t listening.

Full-duplex means both directions are live simultaneously: the microphone is always streaming to the STT engine, even mid-TTS-playback, so voice activity from the user can be detected and acted on in real time regardless of what the agent is currently doing. This is a genuinely harder system to build correctly, because now two audio streams are active concurrently and the system needs a clear policy for what happens when they collide, but it’s the only architecture that supports natural interruption, which is why every production-grade voice agent framework treats it as a baseline requirement rather than an advanced feature.

Barge-in: canceling the agent mid-sentence

Barge-in handling is the concrete mechanism that makes full duplex useful rather than just architecturally interesting. The moment voice activity detection (VAD) on the incoming microphone stream crosses a confidence threshold indicating the user has started speaking, the system needs to immediately stop sending further TTS audio to playback, cancel any pending or in-flight synthesis requests for the current response, and transition state to actively processing the user’s new input. Every millisecond of delay here is a millisecond the agent spends visibly not listening, which is the exact failure mode that makes voice interfaces feel broken rather than conversational.

import asyncio
from enum import Enum, auto

class AgentState(Enum):
    LISTENING = auto()
    SPEAKING = auto()

class DuplexController:
    def __init__(self):
        self.state = AgentState.LISTENING
        self.playback_task: asyncio.Task | None = None
        self.tts_stream_task: asyncio.Task | None = None

    async def on_vad_speech_start(self):
        """Fires the instant VAD detects the user has started talking,
        regardless of what the agent is currently doing."""
        if self.state == AgentState.SPEAKING:
            await self.barge_in()
        self.state = AgentState.LISTENING

    async def barge_in(self):
        """Cancel in-flight synthesis and stop playback immediately."""
        if self.playback_task and not self.playback_task.done():
            self.playback_task.cancel()
        if self.tts_stream_task and not self.tts_stream_task.done():
            self.tts_stream_task.cancel()
        await self.playback.flush()  # drop any buffered audio, don't let it drain out

    async def on_agent_response_ready(self, sentence_stream):
        """Start speaking, but stay interruptible the whole time."""
        self.state = AgentState.SPEAKING
        self.tts_stream_task = asyncio.create_task(self._synthesize(sentence_stream))
        self.playback_task = asyncio.create_task(self._play())
        await asyncio.gather(self.tts_stream_task, self.playback_task, return_exceptions=True)
        if self.state == AgentState.SPEAKING:
            self.state = AgentState.LISTENING

The controller here treats SPEAKING and LISTENING as states that can be interrupted at any point rather than steps in a fixed sequence, and every task doing work on behalf of the current response is cancellable, not fire-and-forget. That cancellability is the entire architectural point: a half-duplex system has no equivalent of barge_in() because it never has an in-flight response to cancel while also listening.

Echo cancellation: not hearing yourself

Full duplex introduces a problem half-duplex never has to face: if the microphone is live while the agent is speaking, and the agent’s audio is coming out of a nearby speaker, the microphone will pick up the agent’s own voice. Without correction, that creates a nasty feedback scenario where the STT engine transcribes the agent’s own TTS output as if it were user speech, and the agent ends up responding to itself, sometimes triggering runaway conversational loops that are confusing to debug and embarrassing to demo.

Acoustic echo cancellation (AEC) solves this using a well-established signal processing technique: since the system knows exactly what audio it’s currently sending to the speaker (it’s the reference signal, generated by its own TTS pipeline), it can subtract a modeled version of that signal from what the microphone picks up, canceling out the echo while preserving genuine user speech. Browsers implement AEC natively as part of the WebRTC media stack (echoCancellation: true on getUserMedia constraints), and telephony and dedicated hardware setups typically rely on the device or carrier’s own AEC implementation. This is one of the clearest cases in this course where using WebRTC (Module 5’s earlier lesson) rather than raw audio sockets pays for itself immediately: AEC is exactly the kind of purpose-built real-time media handling that’s expensive to reimplement and comes free with a proper WebRTC stack.

Putting it together: the event loop

A production full-duplex event loop needs to run VAD continuously on the input stream, maintain the agent’s current state (listening or speaking), and route every event, whether it’s a completed user utterance, a new LLM token, or a detected barge-in, through a single arbitration point rather than scattering state transitions across independent handlers that can race each other. The pattern worth internalizing is that “the agent is speaking” is never an excuse to stop processing input; it’s just one of two states the input processor always has to account for, and the moment those two paths (input processing, output playback) stop being coupled through a shared, cancellable state machine is the moment race conditions start showing up as an agent that occasionally talks over itself or gets stuck mid-response with no way to recover.