Telephony & Multi-Channel Deployment

The dialogue core, guardrails, and evaluation framework this course has built don’t belong to any one channel. The business asked for a voice agent that answers the phone, but the same agent, unchanged in its reasoning and tool logic, should also work in a website widget and a mobile app. This lesson is about the transport layer that makes that true without three separate rebuilds.

One core, three transports

flowchart TB
    subgraph CHAN["Channel-specific transport"]
        PHONE["Telephony\nTwilio Media Streams\n/ SIP trunking, mulaw 8kHz"]
        WEB["Web\nWebRTC, 16-24kHz"]
        APP["Mobile\nNative SDK, 16-24kHz"]
    end

    subgraph CORE["Channel-agnostic core"]
        NORM["Audio normalization\n(resample to pipeline rate)"]
        ASR["Nova-3 ASR"]
        LLM["Dialogue core\nclaude-sonnet-5 / GPT-5\n+ tools + guardrails"]
        TTS["TTS\nsonic-2 / eleven_flash_v2_5"]
        DENORM["Resample to\nchannel output rate"]
    end

    PHONE --> NORM
    WEB --> NORM
    APP --> NORM
    NORM --> ASR --> LLM --> TTS --> DENORM
    DENORM --> PHONE
    DENORM --> WEB
    DENORM --> APP

    style PHONE fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style WEB fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style APP fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style NORM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style ASR fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style LLM fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style TTS fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style DENORM fill:#fff7ed,stroke:#f59e0b,color:#0F172A

The dividing line matters: everything above NORM and below DENORM is channel-specific transport, and everything between them, including guardrails from Lesson 20 and evaluation from Lesson 19, is shared. A conversation-quality fix ships to all three channels at once because it lives in one place.

Wiring a Twilio Media Stream into the pipeline

Twilio Media Streams delivers audio as base64-encoded mulaw frames over a WebSocket, at 8kHz, in 20ms chunks. That has to be decoded and resampled before it reaches an ASR provider expecting a different format.

# twilio_bridge.py - Twilio Media Stream to pipeline bridge
import base64, audioop

async def handle_twilio_stream(ws):
    async for message in ws:
        event = json.loads(message)
        if event["event"] == "media":
            mulaw_bytes = base64.b64decode(event["media"]["payload"])
            # Decode mulaw -> 16-bit linear PCM at 8kHz
            pcm_8k = audioop.ulaw2lin(mulaw_bytes, 2)
            # Resample 8kHz -> 16kHz for the ASR provider
            pcm_16k, _ = audioop.ratecv(pcm_8k, 2, 1, 8000, 16000, None)
            await asr_stream.send(pcm_16k)

        elif event["event"] == "start":
            session_id = event["start"]["callSid"]
            await start_session(session_id, channel="telephony")

On the way out, TTS-synthesized audio has to be resampled back down to 8kHz mulaw before it’s written to the Twilio stream, the mirror image of the inbound path.

Or skip the bridge: managed telephony platforms

Everything above is worth building when you need channel-agnostic control over the dialogue core. If the telephony leg specifically, rather than the whole pipeline, is what you’d rather not own, a platform like NextNeural handles the mulaw decode, campaign dialing, and call lifecycle for you, and exposes it as three calls: create an agent, create a campaign (an outbound dialer or an inbound line tied to a phone number), then trigger calls against it.

# Create an outbound campaign against an existing voice agent
curl -X POST https://api.nextneural.ai/api/voice/flows \
  -H "Authorization: Bearer nn_k_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "outbound",
    "name": "Support callback line",
    "voice_agent_id": "agt_a1b2c3d4",
    "phone_number": "+15550010000",
    "config": { "recording_disclosure": "This call is being recorded for quality." }
  }'

# Trigger an individual call against it
curl -X POST https://api.nextneural.ai/v1/calls \
  -H "Authorization: Bearer nn_k_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{ "to": "+918888888888", "campaign_id": "flw_f1g2h3", "contact_name": "Priya Kumar" }'

This is the right call for teams whose telephony volume doesn’t justify owning codec normalization and Media Stream bridging as in-house infrastructure, at the cost of the fine-grained transport control the DIY approach above gives you.

Codec and sample-rate considerations by channel

Telephony is stuck at 8kHz mulaw (sometimes A-law outside North America) for backward compatibility with the PSTN, regardless of how good the caller’s actual phone is. Web (WebRTC) and mobile (native SDKs using Opus or PCM) typically run 16-24kHz, giving noticeably better ASR accuracy and TTS fidelity on those channels for identical pipeline logic. This means WER benchmarks from Lesson 19 need to be tracked per channel, not blended, since an 8kHz phone call will structurally show higher WER than a 24kHz web session even with zero pipeline bugs, and conflating the two hides real regressions on either one.

What stays channel-specific vs. shared

Channel-specific: codec handling, sample-rate conversion, the transport protocol (Media Streams WebSocket vs. WebRTC peer connection vs. a native SDK’s audio session), and channel-appropriate latency tuning (telephony users tolerate slightly more latency than web users accustomed to text-chat speed). Shared: the dialogue core, tool integrations, guardrails, and the evaluation and observability stack from Module 7. Keeping that line firm is what turns “we shipped a phone bot” into “we shipped a voice agent that happens to also answer the phone.”