Gateway vs Direct SDK Calls vs Framework Routers

Before committing to a gateway architecture, it is worth being honest about the alternatives, because a gateway is not free, it is a real piece of infrastructure with its own latency and operational cost. This lesson writes the same multi-provider failover three ways: raw SDKs, a framework’s built-in router, and a gateway, so the trade-off is visible in code rather than abstract.

Option 1: Direct SDK Calls

The most common starting point. Each provider gets its own client, and application code chooses between them explicitly.

from openai import OpenAI
from anthropic import Anthropic

openai_client = OpenAI(api_key="sk-...")
anthropic_client = Anthropic(api_key="sk-ant-...")

def get_completion(prompt: str) -> str:
    try:
        resp = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.choices[0].message.content
    except Exception:
        # Manually shaped fallback: different SDK, different response shape
        resp = anthropic_client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.content[0].text

This works, but every service that needs failover has to write this same try/except, keep both SDKs installed and current, and handle the fact that OpenAI and Anthropic return meaningfully different response shapes (resp.choices[0].message.content vs resp.content[0].text). Duplicate this across twelve services and a change to the fallback policy, say, adding a third provider, means twelve pull requests.

Option 2: Framework-Level Routing (LangChain)

Frameworks improve on this by giving you one abstraction over multiple providers and a built-in fallback primitive. LangChain’s with_fallbacks is the standard example:

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic

primary = ChatOpenAI(model="gpt-4o")
backup = ChatAnthropic(model="claude-sonnet-4-5")

model = primary.with_fallbacks([backup])

# One call, one response shape, regardless of which provider actually served it
response = model.invoke("Summarize this incident report in two sentences.")
print(response.content)

This is real progress: one call site, one response shape (response.content regardless of provider), and the fallback logic is declared once instead of hand-rolled. But it is scoped to the process it runs in. with_fallbacks is a property of a specific LangChain Runnable; a different service, even one in the same company, that is not built with LangChain gets none of this for free and has to reimplement the same policy. And because it lives inside application code, it has no vantage point to answer questions that span services: total spend across every team, a rate limit shared across all callers of a given model, or an audit log that a security team can query centrally.

Option 3: A Gateway

A gateway moves the same decision out of every process and into one shared service that all of them call. Application code, regardless of language or framework, sends one HTTP request:

from openai import OpenAI

# Same OpenAI SDK, pointed at the gateway instead of api.openai.com.
# The gateway (built in Module 3) owns which provider actually serves this.
client = OpenAI(
    api_key="sk-gateway-team-fraud-detection",  # a virtual key, not a real provider key
    base_url="http://ai-gateway.internal:4000",
)

response = client.chat.completions.create(
    model="production-chat",  # an alias the gateway maps to a real model + fallback chain
    messages=[{"role": "user", "content": "Summarize this incident report in two sentences."}],
)
print(response.choices[0].message.content)

Notice the call site is nearly identical to Option 1’s OpenAI call, that is deliberate: because the gateway speaks the OpenAI schema (Lesson 5), any service that already used the OpenAI SDK adopts a gateway by changing two constructor arguments, no rewrite required. The fallback chain, the budget for the fraud-detection team, and the audit log entry all live in the gateway’s config, one place, visible to every service, every language, and every framework that calls it.

Choosing Between Them

A single small service with one provider and no compliance requirement genuinely does not need a gateway yet, Option 1 is fine. The moment a second service, a second provider, or a first cost/compliance question shows up, the duplication cost of Options 1 and 2 starts to outgrow the network-hop cost of Option 3. The rest of this course builds Option 3 for real, starting with the request lifecycle in Lesson 4 and the schema that makes the “change two constructor arguments” trick above possible in Lesson 5.