Scaling Concurrent Voice Sessions

Everything in this course so far has been validated one call at a time. Production means hundreds or thousands of those calls happening at once, and the failure modes at that scale are different in kind, not just degree, from anything a single-call test surfaces. This lesson is about the infrastructure layer that keeps per-call latency numbers from Module 7 holding steady as concurrency climbs.

Connection pooling for streaming APIs

Opening a fresh WebSocket to Nova-3 or your TTS provider for every call pays a real setup cost (TLS handshake, auth, provider session init) at the worst possible moment: exactly when a traffic spike needs capacity fastest. Pool and pre-warm connections instead.

# connection_pool.py - pre-warmed pool for the ASR provider
class ASRConnectionPool:
    def __init__(self, min_size=20, max_size=500):
        self.min_size = min_size
        self.max_size = max_size
        self._pool: list[Connection] = []

    async def warm(self):
        self._pool = [await open_nova3_stream() for _ in range(self.min_size)]

    async def acquire(self) -> Connection:
        if self._pool:
            return self._pool.pop()
        if self.active_count() < self.max_size:
            return await open_nova3_stream()
        raise PoolExhausted("at max concurrent ASR connections")

Autoscaling and session affinity for media servers

A live call’s WebSocket lives on one media server instance (running LiveKit or Pipecat) for its whole duration. Autoscaling on CPU or connection count adds capacity for new calls, but it cannot rebalance sessions already pinned to an overloaded instance, so capacity planning needs headroom below saturation, not just a reactive scale-out trigger.

flowchart TB
    LB["Load balancer /\nsession router"] --> M1["Media server 1\n(pinned sessions)"]
    LB --> M2["Media server 2\n(pinned sessions)"]
    LB --> M3["Media server N\n(new capacity,\nautoscaled)"]
    M1 --> ASR["ASR pool"]
    M2 --> ASR
    M3 --> ASR
    ASR --> LLM["LLM core"]
    LLM --> TTS["TTS pool"]

    style LB fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style M1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style M2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style M3 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style ASR fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style LLM fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style TTS fill:#fff7ed,stroke:#f59e0b,color:#0F172A

A practical rule: cap each media server instance at 60-70% of its measured max concurrent-session capacity, so a spike has room to land on existing instances while new ones spin up, typically 30-90 seconds behind demand.

Rate limits, backpressure, and queueing

Every upstream provider (ASR, LLM, TTS) has a rate limit, usually expressed as requests-per-second or concurrent-streams. Hitting it mid-call is worse than hitting it before a call starts, so apply backpressure at call admission, not mid-conversation.

async def admit_call(caller_id: str):
    if not await rate_limiter.try_acquire("llm_core", weight=1):
        # Queue briefly rather than drop; most callers tolerate
        # a short hold tone better than a dropped connection.
        await queue.enqueue(caller_id, max_wait_s=8)
        if not await rate_limiter.try_acquire("llm_core", weight=1):
            raise CapacityExceeded("route to overflow / voicemail")
    return await start_session(caller_id)

A short queue with a hold tone beats a dropped call; a hard reject with graceful overflow routing (voicemail or a callback offer) beats an unhandled timeout.

Load testing that reflects real traffic

A load test that opens N connections simultaneously with synchronized pre-recorded audio only tests instantaneous peak, not sustained load. Stagger call arrivals on a realistic distribution, vary turn timing and silence gaps, and run long enough (30+ minutes) to observe steady-state pool exhaustion, rate-limit backpressure, and autoscaling lag, not just the first burst.

# example: staggered load test driver (pseudocode via a load tool)
loadtest --concurrency 500 --arrival-distribution poisson \
  --mean-call-duration 240s --ramp-up 300s --duration 1800s \
  --target wss://voice-gateway.internal/session

Concurrency numbers from a synchronized burst test look better than production will actually behave; a staggered, sustained test is what predicts the p95 latency your Module 7 dashboards will actually show once real callers arrive.