High Availability & Multi-Region Topologies

Lesson 25 deployed the gateway with multiple replicas and autoscaling, which survives a pod or a node failing. It does not survive an entire region going down, a rarer but real event, and one a gateway sitting in front of every LLM call in a company cannot treat as acceptable downtime. This lesson designs the topology that does survive it, and the specific new problem multi-region introduces: state that used to be safely local now needs a real answer for how it behaves when split across regions.

What Needs to Replicate, and What Doesn’t

flowchart TD
    subgraph RegionA["Region: us-east"]
        GWA["Gateway replicas"]
        RedisA[("Local Redis:\ncache, region-local rate limits")]
    end
    subgraph RegionB["Region: eu-west"]
        GWB["Gateway replicas"]
        RedisB[("Local Redis:\ncache, region-local rate limits")]
    end
    GlobalState[("Global state store:\ncross-region key budgets\n(Postgres, or a per-region\nbudget allocation, see below)")]

    GWA --> RedisA
    GWB --> RedisB
    GWA -.->|budget checks| GlobalState
    GWB -.->|budget checks| GlobalState

    style GlobalState fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style RedisA fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style RedisB fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

Not all state carries the same replication requirement. The exact-match and semantic caches from Lesson 16 are safe to keep region-local: a cache miss in one region that would have been a hit in another just costs one extra provider call, a correctness non-issue, purely a minor efficiency loss. Rate-limit counters (Lesson 15) can also often stay region-local with an acceptable trade-off, a caller who happens to be load-balanced across regions gets a rate limit that’s effectively looser by a small factor, usually tolerable. Budget state (Lesson 18) is the one that actually needs a real answer: a $50 monthly cap enforced independently by each region’s local state could let a caller spend up to $50 in every region it touches, multiplying the real intended limit by the number of regions.

The Split-Brain Mitigation

Full synchronous cross-region budget checks on every request add real latency and a new failure mode (what happens to the request if the cross-region check itself times out?). The standard practical compromise is pre-allocating a fraction of the total budget to each region:

def region_local_budget(total_budget: float, region: str, region_traffic_weights: dict[str, float]) -> float:
    # Split a key's total budget across regions proportional to expected
    # traffic share, so each region enforces a fast, local check against
    # its own allocation rather than a synchronous cross-region lookup
    # on every single request.
    return total_budget * region_traffic_weights.get(region, 1 / len(region_traffic_weights))

# A key with a $500/mo budget, expected 70% traffic in us-east, 30% in eu-west
us_east_cap = region_local_budget(500, "us-east", {"us-east": 0.7, "eu-west": 0.3})   # $350
eu_west_cap = region_local_budget(500, "eu-west", {"us-east": 0.7, "eu-west": 0.3})   # $150

This trades perfect global precision for enforcement that stays fast and available: the true worst-case overspend is bounded by the sum of regional allocations rather than unbounded, and a periodic (not per-request) reconciliation job can true up actual global spend against the intended total and adjust regional allocations going forward. For most teams, this bounded imprecision is an acceptable trade against the latency and availability cost of synchronous cross-region coordination on every request; a company with a harder financial requirement can instead run budget checks against a strongly consistent global store and accept the added latency, a legitimate but different choice.

Detecting and Routing Around a Region Failure

Lesson 14’s fallback chains operate entirely inside one running gateway process: they choose among provider deployments, with no visibility into whether the gateway process itself, or the whole region hosting it, is reachable at all. Region-level failure needs a layer above the gateway:

# Example: a DNS-based health check configuration (conceptually,
# using a provider like Route 53 or Cloudflare's load balancer)
# routes callers to the nearest healthy region, and away from a
# region whose gateway health endpoint stops responding.
healthCheck:
  endpoint: /health/readiness   # the same readiness endpoint from Lesson 25's Kubernetes config
  regions: [us-east, eu-west]
  failoverPolicy: nearest-healthy

sequenceDiagram
    participant Client
    participant DNS as Geo DNS / Anycast LB
    participant USEast as us-east gateway
    participant EUWest as eu-west gateway

    Client->>DNS: resolve ai-gateway.company.com
    DNS->>USEast: health check
    USEast--xDNS: unreachable (region down)
    DNS->>EUWest: health check
    EUWest-->>DNS: healthy
    DNS-->>Client: route to eu-west
    Client->>EUWest: request proceeds normally

This is the same “chokepoint enforced once, benefits everyone” principle that has run through this entire course, applied at the infrastructure layer: application code never needs to know a region failed, it always resolves ai-gateway.company.com and the DNS/load-balancer layer handles routing it to whichever region is actually healthy. With deployment and multi-region resilience covered, Lesson 27 closes Module 9 by comparing LiteLLM against the other three gateways introduced in Lesson 2, Portkey, TensorZero, and Envoy AI Gateway, now that you have enough hands-on context to evaluate the trade-offs directly.