Self-Hosting Open-Weight Models: Ollama & vLLM Behind the Gateway

Lesson 7’s config included a local-llama deployment pointed at Ollama, useful for development, but not what most companies run in production. This lesson compares Ollama and vLLM head-on and adds a real vLLM deployment to the gateway, because self-hosting open-weight models is now a routine, not exotic, part of a 2025-2026 gateway’s model_list, driven by cost, data residency, and latency requirements that a hosted API cannot always meet.

Ollama vs vLLM: Different Points on the Scale Curve

Ollama optimizes for the fastest possible path to a running local model: ollama pull llama3.2 && ollama run llama3.2 and you have a model serving requests, with automatic quantization defaults and a tiny operational footprint. It is built around a single-user or small-team mental model, not concurrent multi-tenant load.

vLLM optimizes for the opposite end: serving many concurrent requests from many callers with high GPU utilization. Its core innovation, PagedAttention, manages the GPU memory used for each request’s attention key-value cache the way an OS manages virtual memory pages, which is what lets it batch dozens of concurrent, different-length requests onto the same GPU far more efficiently than naive batching. This is the standard choice when a company is actually serving production traffic against a self-hosted open-weight model, not just developing against one locally.

Launching vLLM

pip install vllm

# Serves an open-weight model on an OpenAI-compatible endpoint at :8000
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --max-model-len 8192

Verify it directly before wiring it into the gateway, exactly the request shape from Lesson 5:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "List three uses of a PagedAttention-style KV cache."}]
  }'

Adding It to the Gateway

Because vLLM’s endpoint is already OpenAI-compatible, adding it to config.yaml uses the identical litellm_params shape as any cloud provider, extending Lesson 7’s config:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: self-hosted-llama         # a new alias for the production self-hosted deployment
    litellm_params:
      model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct
      api_base: http://localhost:8000/v1
      api_key: "none"

general_settings:
  master_key: sk-1234

flowchart TD
    GW["LiteLLM Gateway"] --> Cloud["Cloud providers\n(OpenAI, Anthropic)\nmetered, pay-per-token"]
    GW --> SelfHosted["Self-hosted vLLM\n(meta-llama/Llama-3.1-8B-Instruct)\nfixed GPU cost, no per-token bill"]

    style GW fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Cloud fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style SelfHosted fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

Why Teams Actually Self-Host

Three concrete reasons show up repeatedly in production gateway configs, not as ideology but as line items: cost at volume, a self-hosted open-weight model on owned or reserved GPU capacity has a fixed cost regardless of token volume, which wins decisively once request volume is high and steady enough to keep the GPU busy. Data residency and compliance, some data categories (regulated health or financial data, certain government contracts) cannot leave a specific network boundary at all, making a hosted API a non-starter regardless of price. Latency for high-throughput internal tools, a self-hosted model on infrastructure close to the caller can beat a cloud API’s network round-trip for latency-sensitive internal tooling. None of these are universal, a low-volume workload is almost always cheaper on a hosted API once GPU idle time is accounted for, which is exactly the kind of trade-off the routing engine in Module 4 is built to make automatically rather than by hand. Lesson 9 finishes Module 3 by turning single-deployment aliases like this one into real multi-deployment load balancing and fallback chains using LiteLLM’s Router.