Anatomy of a Production AI Gateway

Lesson 1 established why a gateway exists. This lesson opens it up and names its parts, because every gateway you will touch in this course, LiteLLM, TensorZero, Portkey, Envoy AI Gateway, is built from the same six layers, just with different implementations and different trade-offs in speed, deployment model, and feature depth. Knowing the layers means you can evaluate any new gateway that shows up in 2026 in about five minutes, by asking what each layer does and how it’s implemented, instead of reading its entire marketing page.

The Six Layers

flowchart TD
    A["1. Ingress & Auth\n(TLS, virtual keys, rate limits)"] --> B["2. Request Normalization\n(OpenAI-shaped schema in)"]
    B --> C["3. Routing Engine\n(which model, which deployment)"]
    C --> D["4. Provider Adapters\n(schema out, error normalization)"]
    D --> E["5. Cache & Cost Layer\n(exact/semantic cache, budget check)"]
    E --> F["6. Observability\n(traces, logs, metrics per request)"]

    style A fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style B fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style C fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style D fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style E fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style F fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

Ingress and auth terminates the incoming connection, validates the caller’s virtual key (not a raw provider key, Lesson 19 covers why), and applies the first rate-limit check before any provider is touched. Request normalization validates the incoming payload against the gateway’s canonical schema, almost always OpenAI’s Chat Completions shape (Lesson 5), so every downstream layer works against one predictable structure regardless of which client sent the request. The routing engine is the decision-making core: given the requested model name (or an alias like "fast-cheap"), the caller’s policy, and live health/latency signals, it picks exactly which provider, model, and API key will actually serve this request, this is where Module 4’s routing strategies live. Provider adapters do the mechanical translation, taking the normalized request and turning it into whatever shape Anthropic, Gemini, or a self-hosted vLLM server expects, then translating the response and any errors back into the canonical shape. The cache and cost layer checks for a cache hit before spending a token, and after a real call, checks whether it fits within budget and increments the caller’s spend, this is where Module 6 lives. Observability wraps the whole request in a span, recording model, tokens, latency, and cost, this is Module 8.

Control Plane vs Data Plane

A second useful split cuts across those six layers: the control plane is configuration that changes rarely, the list of available models, which teams have which budgets, routing policy, and the data plane is the hot path that executes on every single request. This split matters practically because it dictates a performance requirement: the data plane must read control-plane state from memory (or a fast local cache), never make a synchronous database round-trip per request, or gateway overhead becomes the slowest part of the whole call. TensorZero’s marketing claim of sub-millisecond p99 overhead at 10,000+ queries per second, and it is written in Rust specifically to keep the data plane out of any interpreted-language garbage collector’s way, is a direct consequence of taking this split seriously.

Mapping Real Gateways onto This Architecture

All four implement the same six layers, they just draw different lines around deployment complexity versus configurability. This course builds hands-on with LiteLLM starting in Module 3 because its config format is the most direct way to see all six layers in one readable file, then compares it against the other three in Lesson 27 once you have enough context to evaluate the trade-offs for yourself. Next, Lesson 3 places the gateway relative to two alternatives you have probably already used: calling a provider SDK directly, and relying on a framework’s built-in fallback logic.