The LiteLLM Router: Load Balancing & Fallbacks

Lessons 7 and 8 built a model_list with distinct aliases, one deployment each. Real production traffic needs two more things from the routing layer: spreading load across multiple healthy deployments of the same logical model, and automatically recovering when a deployment fails. Both are the job of LiteLLM’s Router, and this lesson configures both, first as a standalone Python object, then as proxy config.

Load Balancing: Multiple Deployments, One Alias

from litellm import Router

model_list = [
    {
        "model_name": "production-chat",           # shared alias: this is the pool
        "litellm_params": {"model": "openai/gpt-4o", "api_key": "sk-openai-..."},
    },
    {
        "model_name": "production-chat",            # second deployment in the same pool
        "litellm_params": {
            "model": "hosted_vllm/meta-llama/Llama-3.1-8B-Instruct",
            "api_base": "http://localhost:8000/v1",
            "api_key": "none",
        },
    },
]

router = Router(model_list=model_list, routing_strategy="simple-shuffle")

response = router.completion(
    model="production-chat",                        # router picks which deployment actually serves this
    messages=[{"role": "user", "content": "Summarize this ticket in one sentence."}],
)

simple-shuffle is the router’s default: it distributes requests across the pool without needing external state. Two other strategies matter for production: latency-based-routing continuously tracks each deployment’s recent response time and biases traffic toward the faster one, useful when self-hosted and cloud deployments have meaningfully different latency profiles; usage-based-routing-v2 tracks per-deployment request/token volume in Redis and balances to stay under each deployment’s configured rate limit, which matters once a deployment’s own rpm/tpm ceiling, not just raw speed, is the binding constraint.

# Prefer the faster deployment when latencies diverge meaningfully
router = Router(model_list=model_list, routing_strategy="latency-based-routing")

# Balance to respect each deployment's own rate limits, needs Redis
router = Router(
    model_list=model_list,
    routing_strategy="usage-based-routing-v2",
    redis_host="localhost", redis_port=6379,
)

Fallbacks: Recovering from Failure

Load balancing assumes every deployment in the pool is healthy. Fallbacks handle the case where one is not. LiteLLM configures this as an explicit mapping from a primary model name to an ordered list of fallback model names:

router = Router(
    model_list=model_list,
    fallbacks=[{"production-chat": ["self-hosted-llama-fallback"]}],
    num_retries=2,
)

# The equivalent proxy config.yaml, extending Lesson 8's model_list
model_list:
  - model_name: production-chat
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: self-hosted-llama-fallback
    litellm_params:
      model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct
      api_base: http://localhost:8000/v1
      api_key: "none"

litellm_settings:
  fallbacks:
    - production-chat: ["self-hosted-llama-fallback"]
  num_retries: 2

sequenceDiagram
    participant Client
    participant Router
    participant GPT4o as gpt-4o (primary)
    participant Llama as self-hosted Llama (fallback)

    Client->>Router: completion(model="production-chat")
    Router->>GPT4o: call
    GPT4o-->>Router: 500 / timeout
    Note over Router: primary failed, walk fallback chain
    Router->>Llama: call
    Llama-->>Router: 200 OK
    Router-->>Client: response (from fallback deployment)

The router only tries self-hosted-llama-fallback after production-chat actually fails, this is deliberately not a load-balancing pool: the fallback is a last resort, not a peer sharing traffic on the happy path. Routing a fallback to a self-hosted deployment specifically, rather than a second cloud provider, buys resilience against a cloud-provider-wide outage, since the self-hosted server’s availability has no dependency on that provider’s infrastructure at all, at the cost of potentially different output quality from the fallback model, a trade-off Lesson 14 covers when it looks at graceful degradation in more depth.

With load balancing and a basic fallback chain running, Module 4 goes deeper on the actual decision logic behind routing, static priority, live latency and cost signals, and classifying a prompt to pick the right model for its task, rather than the single flat pool used here.