Every team that starts calling LLM APIs directly from application code eventually hits the same four walls: a provider outage takes down every service that hardcoded that provider, nobody can answer “how much did the marketing team spend on Claude last month” without grepping logs across a dozen repos, a new open-weight model turns out to be cheaper and just as good for one specific task but rewiring every call site to try it is a multi-sprint project, and a security review finds that some service is sending customer PII to a provider nobody approved for that data. None of these are solved by picking a better SDK. They are solved by putting a layer between application code and the providers, one that owns routing, credentials, cost, and policy so that application code only ever needs to know “call the gateway.” That layer is the AI gateway, and by 2025-2026 it has become as standard in an LLM-serving stack as an API gateway or a database connection pool.
What an AI Gateway Actually Is
At the network level, an AI gateway is a reverse proxy purpose-built for LLM traffic. It exposes one HTTP endpoint that speaks a single, stable request/response schema (almost always the OpenAI Chat Completions shape, covered in depth in Lesson 5), and behind that endpoint it holds the actual provider credentials, the routing logic, and the policy engine. Application code sends a request to the gateway and never touches a provider SDK directly.
flowchart LR
App1["Checkout Service"] --> GW["AI Gateway\n(one endpoint, one schema)"]
App2["Support Bot"] --> GW
App3["Internal Copilot"] --> GW
GW --> OpenAI["OpenAI"]
GW --> Anthropic["Anthropic"]
GW --> Ollama["Self-hosted\n(Ollama / vLLM)"]
GW --> Groq["Groq\n(open-weight, fast inference)"]
style GW fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
style App1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
style App2 fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
style App3 fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
style OpenAI fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
style Anthropic fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
style Ollama fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
style Groq fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8pxThis is not a new architectural idea, it is the same reasoning that put a database connection pool between application code and a database, or a service mesh sidecar between microservices. What is new is that the “backends” here are LLM providers, each with a different API shape, different rate-limit behavior, different failure modes, and wildly different cost per token, which is exactly why a naive HTTP reverse proxy (nginx, an API gateway like Kong without its AI plugins) is not enough on its own; it needs to understand tokens, streaming, and model identity, not just paths and headers.
The Four Failures a Gateway Fixes
Single points of failure become company-wide incidents. OpenAI, Anthropic, and every other frontier provider have had multi-hour outages in 2024-2026. If twelve services each call openai.chat.completions.create() directly, all twelve fail at once, and recovery means a code change and redeploy in each one. With a gateway, the fallback chain (Lesson 14) is one config change in one place, and it can be live before the incident channel finishes forming.
Cost has no floor and no attribution. Direct calls mean cost tracking lives in whatever billing dashboard each provider gives you, sliced by API key at best. There is no way to answer “what did the fraud-detection team spend this month” without instrumenting every call site individually. A gateway that issues one virtual key per team (Lesson 19) gets you that answer for free, because every request already flows through one place that can log it.
Trying a new or cheaper model is a project, not a config change. A new open-weight model lands, benchmarks show it is 80% cheaper and good enough for a specific classification task. Without a gateway, adopting it means finding every call site for that task and rewriting it against a new SDK. With a gateway routing by task (Lesson 12), it is a routing rule.
Security and compliance have no chokepoint. PII redaction, audit logging, and “which team is allowed to send data to which provider” are all policies that need to be enforced somewhere. Enforcing them per-service means enforcing them nowhere reliably. A gateway is the one place these controls can actually be guaranteed, covered across Module 7.
What a 2026 Gateway Routes Across
The provider landscape a gateway has to speak to has grown far past “OpenAI or Anthropic.” A production gateway in 2025-2026 typically routes across closed frontier APIs (OpenAI, Anthropic, Google Gemini), fast open-weight inference services (Groq, Together AI, Fireworks), aggregators that expose hundreds of models behind one API (OpenRouter), and self-hosted open-weight models the company runs itself (Ollama for development and small deployments, vLLM for production-scale serving). This course builds real, running examples against all of these, not just the closed APIs, because in practice a modern routing strategy leans on open-weight models for cost and data-residency reasons at least as often as it reaches for a frontier model.
The rest of this module covers what a gateway is made of internally (Lesson 2) and where it fits relative to calling SDKs directly or relying on a framework’s built-in router, like LangChain’s model fallbacks (Lesson 3), before Module 2 gets into the request lifecycle and the unified schema that makes all of this possible.