Every millisecond this course has spent optimizing STT, LLM inference, and TTS synthesis can be erased by a transport layer that adds its own latency, drops packets under real network conditions, or simply can’t reach the client because of a NAT it never accounted for. Transport is the least glamorous layer of a voice pipeline and the one most likely to be treated as an afterthought, right up until a demo that worked perfectly on office Wi-Fi falls apart on a user’s spotty 4G connection. This lesson is about choosing the right transport for the right leg of your system, because “voice transport” isn’t one decision, it’s several different decisions depending on which two endpoints are talking to each other.
What WebRTC actually gives you
WebRTC is not “a way to send audio over the internet” in the generic sense; it’s a specific, purpose-built stack for real-time media that solves three hard problems raw sockets leave to you. First, it encrypts media in transit using SRTP by default, so you’re not building your own encryption layer for audio packets. Second, it includes adaptive jitter buffering and packet loss concealment tuned for the actual statistical behavior of real-time audio loss, smoothing over the exact kind of network instability that makes phone calls on bad Wi-Fi usable rather than a string of dropouts. Third, and often underappreciated until you hit it in production, it solves NAT traversal: most consumer devices sit behind NATs that make direct peer-to-peer connections impossible without help, which is exactly what STUN and TURN servers provide.
STUN servers let a client discover its own public-facing address and port so a remote peer knows where to send packets; when a direct path still isn’t possible because of a symmetric NAT or restrictive firewall, TURN servers relay the media through a public server as a fallback, at the cost of an extra hop of latency. A production WebRTC deployment always needs both configured, because STUN alone fails for a meaningful percentage of real-world network topologies, particularly on corporate or carrier-grade NAT setups.
What raw WebSockets give you instead
Raw WebSocket audio streaming skips all of that machinery in exchange for simplicity: it’s a persistent, ordered, TCP-based connection where you push audio bytes in one direction and receive them in the other, with no encryption, jitter handling, or NAT traversal beyond what standard HTTP/TLS already provides. That sounds like a strict downgrade until you notice where it actually gets used in production: server-to-server integrations, like your backend talking to a TTS or STT vendor’s API, and telephony bridges, like Twilio Media Streams, where a carrier delivers wss:// connections carrying mulaw-encoded 8kHz audio. In both cases, the two endpoints are stable servers on well-provisioned networks, not a consumer device on unpredictable Wi-Fi, so WebRTC’s jitter buffering and NAT traversal solve problems that don’t really exist on that leg, and the operational simplicity of a plain WebSocket wins.
flowchart TB
subgraph BROWSER["Browser-based voice agent"]
MIC["User microphone"] --> WEBRTC["WebRTC peer connection\nSRTP, jitter buffer, STUN/TURN"]
WEBRTC --> SFU["SFU / media server\n(LiveKit, Pipecat)"]
end
subgraph TELE["Telephony integration"]
PSTN["Phone network (PSTN)"] --> CARRIER["Carrier (Twilio)"]
CARRIER --> WS["wss:// Media Streams\nmulaw 8kHz"]
end
subgraph SERVER["Server-to-server"]
AGENT["Voice agent backend"] --> API["wss:// to STT/TTS vendor APIs"]
end
SFU --> AGENT
WS --> AGENT
style MIC fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style PSTN fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style WEBRTC fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style SFU fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style CARRIER fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style WS fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style AGENT fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style API fill:#fff7ed,stroke:#f59e0b,color:#0F172AMatching transport to use case
The pattern that emerges is straightforward once you see it laid out: WebRTC for any leg where a consumer device on an unmanaged network is one of the endpoints, and WebSocket for any leg where both endpoints are servers on managed infrastructure. A browser-based voice assistant needs WebRTC from the user’s microphone to your media server, because that’s the leg where jitter, packet loss, and NAT traversal are real, everyday problems. A telephony integration receives audio from the carrier over wss:// because the carrier has already handled the hard real-time delivery problem on the PSTN side of the call, and the WebSocket leg is comparatively well-behaved. Your backend’s connections to STT, LLM, and TTS vendor APIs are WebSocket for the same reason: two data centers talking to each other rarely need SRTP and jitter buffering.
When to reach for LiveKit or Pipecat
Implementing WebRTC correctly from scratch means running your own STUN/TURN infrastructure, handling ICE candidate negotiation, managing SDP offer/answer exchanges, and building an SFU (selective forwarding unit) if you need more than a single peer-to-peer connection, none of which is specific to voice AI and all of which is genuinely hard to get right under real network conditions. LiveKit and Pipecat exist precisely to remove this burden: both provide production-grade WebRTC infrastructure with SDKs that expose “join a room, publish audio, subscribe to audio” as the primary interface, hiding STUN/TURN configuration, SFU routing, and reconnection logic behind a few function calls. Pipecat additionally provides orchestration primitives for wiring STT, LLM, and TTS stages together as a pipeline, which overlaps directly with patterns from the earlier lessons in this course. Unless your product’s core differentiation is the transport layer itself, which it almost never is for a voice AI company, building custom WebRTC infrastructure is time spent re-solving a problem these frameworks have already solved at scale, instead of time spent on the conversational logic that’s actually your product.