LiteLLM Proxy: Config, Providers & Virtual Keys

Everything through Module 2 was conceptual: the lifecycle, the schema, the adapter pattern. This lesson makes it real by standing up the LiteLLM proxy, the open-source gateway (BerriAI/litellm on GitHub) this course uses for every hands-on exercise from here through Module 9, and writing a config that routes across a cloud provider and a self-hosted model.

Install and Run

pip install 'litellm[proxy]'

A minimal config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: local-llama
    litellm_params:
      model: openai/llama3.2          # the 'openai/' prefix tells LiteLLM this is an OpenAI-compatible endpoint
      api_base: http://localhost:11434/v1
      api_key: "none"                 # Ollama's OpenAI-compatible endpoint doesn't check this

general_settings:
  master_key: sk-1234                 # root credential for the proxy itself, never hand this to application code

Start it:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
litellm --config config.yaml --port 4000

model_name is the alias callers request; litellm_params.model is the real, provider-qualified identifier LiteLLM actually calls. This indirection is the entire mechanism behind aliasing and, in the next lesson, load balancing: multiple entries can share one model_name while pointing at different real deployments.

Calling It

Because LiteLLM’s proxy speaks the canonical OpenAI schema from Lesson 5, the standard OpenAI SDK works against it unmodified, just point base_url at the proxy:

from openai import OpenAI

client = OpenAI(
    api_key="sk-1234",                       # the master_key, fine for this first local test
    base_url="http://localhost:4000",
)

response = client.chat.completions.create(
    model="local-llama",                     # routes to the self-hosted Ollama model, no code change needed
    messages=[{"role": "user", "content": "What is a reverse proxy, in one sentence?"}],
)
print(response.choices[0].message.content)

Swap model="local-llama" for model="claude-sonnet" and rerun, no other line changes, the proxy handles translating to Anthropic’s schema (using exactly the adapter logic from Lesson 6) transparently. This is the payoff of Modules 1 and 2 made concrete: application code targets one alias and one schema; the gateway owns which real provider actually answers.

Virtual Keys: Never Hand Out the Master Key

The sk-1234 master key above has full admin rights over the proxy: it can create and revoke other keys and see every team’s spend. Real application code should never hold it. LiteLLM’s /key/generate endpoint issues scoped virtual keys instead:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o", "local-llama"],
    "max_budget": 50,
    "metadata": {"team": "fraud-detection"}
  }'

This returns a new sk-... key scoped to only the gpt-4o and local-llama aliases, capped at $50 of spend, and tagged with a team label for the cost-attribution work in Lesson 18. That key, not the master key, is what a service like fraud-detection’s backend actually holds:

client = OpenAI(
    api_key="sk-the-newly-generated-virtual-key",
    base_url="http://localhost:4000",
)
# This call succeeds: gpt-4o is in this key's allowed models
client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": "hi"}])

# This call fails with a 403: claude-sonnet was never granted to this key
client.chat.completions.create(model="claude-sonnet", messages=[{"role": "user", "content": "hi"}])

flowchart LR
    Master["master_key\n(admin: create/revoke keys, see all spend)"] -->|/key/generate| VK1["Virtual key: fraud-detection\nmodels: [gpt-4o, local-llama], budget: $50"]
    Master -->|/key/generate| VK2["Virtual key: support-bot\nmodels: [claude-sonnet], budget: $200"]
    VK1 --> App1["fraud-detection service"]
    VK2 --> App2["support-bot service"]

    style Master fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style VK1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style VK2 fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style App1 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style App2 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

With a running proxy, a multi-provider config, and scoped keys in hand, Lesson 8 extends the model_list with a production-grade self-hosted deployment using vLLM instead of Ollama, and Lesson 9 turns the local-llama-style single-deployment aliases into real multi-deployment load balancing.