Request Lifecycle: Ingress, Auth, Routing, Egress

Lesson 2 named the six layers of a gateway. This lesson walks one real request through all of them in order, because the ordering is not arbitrary, cheap, reject-fast checks run before expensive ones, and understanding this sequence is what lets you reason about where a bug, a cost leak, or a latency spike is actually happening when something goes wrong in production.

The Seven Stages

sequenceDiagram
    participant Client
    participant Gateway
    participant Cache
    participant Provider

    Client->>Gateway: POST /v1/chat/completions (virtual key)
    Gateway->>Gateway: 1. TLS + parse + schema validation
    Gateway->>Gateway: 2. Auth: resolve virtual key -> team, budget
    Gateway->>Gateway: 3. Policy: budget check, rate limit check
    Gateway->>Cache: 4. Cache lookup (exact or semantic)
    Cache-->>Gateway: miss
    Gateway->>Gateway: 5. Routing: pick provider + model + deployment
    Gateway->>Provider: 6. Adapter translates request, calls provider
    Provider-->>Gateway: raw provider response
    Gateway->>Gateway: 7. Egress: normalize, cost, cache write, log/trace
    Gateway-->>Client: OpenAI-shaped response

1. TLS termination, parsing, and schema validation. The gateway accepts the connection and validates the incoming JSON against its canonical request schema (Lesson 5). A malformed request, a missing messages field, an unsupported parameter, is rejected here with a 400, before any auth lookup or provider work happens; this is the cheapest possible place to fail.

2. Authentication. The gateway resolves the caller’s virtual key (never a raw provider key, Lesson 19) to an identity: which team, which project, which policy applies. An invalid or revoked key is rejected here with a 401.

3. Policy checks. With identity resolved, the gateway checks whether this caller is within its rate limit and budget (Lesson 15, Lesson 18). This runs before routing deliberately: there is no reason to spend CPU picking a provider for a request that is about to be rejected for being over budget.

4. Cache lookup. Before any provider is contacted, the gateway checks its cache (Lesson 16) for an identical or semantically equivalent prior request. A hit skips straight to stage 7 with zero tokens spent and zero provider latency.

5. Routing. On a cache miss, the routing engine (Module 4) resolves the requested model alias (e.g. "production-chat") to a concrete provider, model, and deployment, taking into account health, latency, and the routing strategy in effect.

6. Provider adapter and outbound call. The adapter (Lesson 6) translates the normalized request into the chosen provider’s exact schema, adds that provider’s real API key (which the caller never sees), and makes the outbound HTTPS call. If it fails, this is where retries and circuit breakers (Lesson 13) engage before falling back to a different deployment.

7. Egress. The provider’s raw response is translated back into the canonical schema so the client gets an identical shape no matter which provider actually served the request, cost is computed from the token usage and charged against the caller’s budget, a cache write happens if the response is cacheable, and a trace span is emitted (Module 8) recording model, latency, tokens, and cost before the response finally reaches the client.

Seeing This in a Real Gateway

You do not need to build a gateway from scratch to see this lifecycle; the LiteLLM proxy (built hands-on starting Lesson 7) implements exactly these stages, and its request logs make the sequence visible:

import httpx

# A single call through a running LiteLLM proxy exercises every stage above.
response = httpx.post(
    "http://localhost:4000/v1/chat/completions",
    headers={"Authorization": "Bearer sk-gateway-team-fraud-detection"},  # stage 2
    json={
        "model": "production-chat",                                      # stage 5 target
        "messages": [{"role": "user", "content": "Classify this transaction as fraud risk: low."}],
    },
)
print(response.json()["choices"][0]["message"]["content"])
print(response.headers.get("x-litellm-response-cost"))  # stage 7: cost computed on egress

The x-litellm-response-cost header is emitted at egress, after the real provider call returned and LiteLLM computed cost from the actual token usage, it is not knowable any earlier in the pipeline. That single detail is a useful debugging habit: whenever a gateway behavior seems surprising (a request rejected, a cost that doesn’t match expectations, a cached response reused when it shouldn’t have been), map it back onto which of these seven stages produced it, and the fix is usually obvious. Lesson 5 goes deeper into stage 1’s schema, the one contract every client and every provider adapter has to agree on for this whole pipeline to work.