Rate Limiting & Load Shedding

Fallback chains (Lesson 14) protect against a provider being unavailable. This lesson covers the opposite direction: protecting the provider, and the gateway itself, from too much legitimate traffic arriving at once. Rate limiting and load shedding are related but distinct: rate limiting enforces a ceiling per caller against a provider’s real quota; load shedding protects the gateway’s own capacity by deliberately rejecting some traffic before it can overwhelm everything.

Why Token Volume, Not Just Request Count

A traditional API rate limit (100 requests per minute) assumes each request costs roughly the same. LLM requests do not: a request with max_tokens: 4000 consumes far more of a provider’s actual capacity, and a much larger share of a tokens-per-minute quota, than one with max_tokens: 50, even though both count as “one request.” A rate limiter that only counts requests will let a caller blow through a provider’s real token quota while looking well within limits.

# Per-key limits set at key-generation time, extending Lesson 7's /key/generate call
# rpm: requests per minute, tpm: tokens per minute, both enforced together

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["production-chat"],
    "rpm_limit": 60,
    "tpm_limit": 100000,
    "metadata": {"team": "support-bot"}
  }'

A caller with this key hitting either the 60-requests-per-minute or the 100,000-tokens-per-minute ceiling, whichever comes first, gets a 429 back from the gateway before the request is even routed to a provider. This matters for two reasons: it protects the caller’s own budget from a runaway loop, and it protects the gateway’s aggregate quota with a given provider from being exhausted by one misbehaving caller, leaving none for everyone else.

Implementing a Token-Bucket Limiter

LiteLLM enforces rpm/tpm limits internally; the underlying algorithm worth understanding is the token bucket, since it is the standard approach anywhere in this space, including at the deployment level (Lesson 9) and the per-team budget level (Lesson 19):

import time
from dataclasses import dataclass

@dataclass
class TokenBucket:
    capacity: float          # max tokens the bucket can hold, e.g. tpm_limit
    refill_rate: float       # tokens added per second, e.g. tpm_limit / 60
    tokens: float = 0
    last_refill: float = 0

    def __post_init__(self):
        self.tokens = self.capacity
        self.last_refill = time.time()

    def allow(self, cost: float) -> bool:
        now = time.time()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now

        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False

# tpm_limit=100_000 -> refill_rate ~1,666 tokens/sec
bucket = TokenBucket(capacity=100_000, refill_rate=100_000 / 60)

def check_and_consume(estimated_tokens: int) -> bool:
    return bucket.allow(cost=estimated_tokens)

A token bucket naturally handles bursty-but-average-compliant traffic better than a fixed window counter: it allows a short burst up to the full bucket capacity, then throttles smoothly as the bucket drains, rather than allowing a full quota’s worth of traffic in the last millisecond of one window and again in the first millisecond of the next, a real failure mode of naive fixed-window rate limiting.

Load Shedding: Protecting the Gateway, Not Just the Provider

Rate limiting protects a provider’s quota per caller. Load shedding is a different concern entirely: when the gateway itself is near its own capacity, ambient load from all callers combined threatening to degrade everyone, it deliberately rejects lower-priority traffic first to preserve capacity for what matters most.

from enum import IntEnum

class Priority(IntEnum):
    LOW = 0        # batch jobs, internal experimentation
    NORMAL = 1     # standard product traffic
    CRITICAL = 2   # user-facing, revenue-impacting paths

def should_shed(current_load_pct: float, request_priority: Priority) -> bool:
    # As load rises, shed progressively more from the bottom of the
    # priority scale, so CRITICAL traffic is the last thing ever dropped.
    if current_load_pct > 95 and request_priority < Priority.CRITICAL:
        return True
    if current_load_pct > 85 and request_priority < Priority.NORMAL:
        return True
    return False

def handle_request(request, current_load_pct: float):
    priority = Priority[request.headers.get("x-priority", "NORMAL")]
    if should_shed(current_load_pct, priority):
        raise RuntimeError("503: gateway shedding low-priority load, retry later")
    # proceed to routing (Module 4) as normal

flowchart TD
    Load["Gateway load"] --> Check{"Load > 85%?"}
    Check -->|No| Serve["Serve all traffic normally"]
    Check -->|"85-95%"| ShedLow["Shed LOW priority traffic\n(batch jobs, experiments)"]
    Check -->|">95%"| ShedNormal["Shed LOW + NORMAL priority\nCRITICAL traffic still served"]

    style Load fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Serve fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style ShedLow fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style ShedNormal fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px

The distinction to hold onto: rate limiting answers “is this specific caller within its own quota,” enforced consistently regardless of overall gateway health; load shedding answers “is the gateway as a whole healthy enough to serve this priority tier right now,” a decision that only kicks in under real system-wide pressure. A well-run gateway needs both, a caller that respects its rate limit can still get shed during a genuine capacity crunch if its priority is low enough, and that is the correct behavior, not a bug. With Module 5’s reliability toolkit complete, retries, timeouts, circuit breakers, fallback chains, rate limiting, and load shedding, Module 6 turns to the other lever for controlling cost: not serving a request to a provider at all when the answer is already known.