Scaling MCP Servers: Load Balancing, Caching & Cost Control

MCP servers reached by many concurrent agents across an organization need the same scaling discipline as any high-traffic backend service, tuned to the specific traffic shape agentic workloads produce: bursty, I/O-bound, and often repeating similar queries, plus a caching risk specific to agentic use that a purely web-traffic mental model would not surface.

Autoscaling for Bursty, I/O-Bound Traffic

Agent-driven traffic is bursty (a workflow triggers ten agents simultaneously, then goes quiet) and I/O-bound (most tool time is spent waiting on a downstream API or database, not computing). CPU-based autoscaling under-reacts to this shape, scaling on concurrent request count or queue depth responds far better.

# KEDA ScaledObject: scale on concurrent in-flight HTTP requests, not CPU
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: enterprise-mcp-server-scaler
spec:
  scaleTargetRef:
    name: enterprise-mcp-server
  minReplicaCount: 3
  maxReplicaCount: 30
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(http_requests_in_flight{app="enterprise-mcp-server"})
        threshold: "20"   # Scale up once average in-flight requests per pod exceeds 20

flowchart LR
    Metric["in-flight request count\n(Prometheus)"] --> KEDA["KEDA ScaledObject"]
    KEDA -->|"scale 0 to 30 replicas\nbased on load"| Deploy["enterprise-mcp-server\nDeployment"]

    style Metric fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style KEDA fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style Deploy fill:#f0fdf9,stroke:#0D9488,color:#0F172A

Caching Expensive, Repeated Read-Only Calls

Many enterprise tool calls repeat: multiple agent sessions querying the same account, the same policy document, the same warehouse aggregate, within a short window. A TTL-based cache keyed on tool name and arguments cuts this redundant downstream load with bounded staleness risk.

import hashlib, json
from functools import wraps

def cached_tool(ttl_seconds: int):
    """Cache a read-only tool's result by a hash of its arguments, TTL matched
    to how frequently the underlying data actually changes."""
    def decorator(fn):
        @wraps(fn)
        async def wrapper(input, **kwargs):
            cache_key = f"mcp:cache:{fn.__name__}:{hashlib.sha256(json.dumps(input.model_dump(), sort_keys=True).encode()).hexdigest()}"
            cached = await redis_client.get(cache_key)
            if cached is not None:
                return json.loads(cached)
            result = await fn(input, **kwargs)
            await redis_client.setex(cache_key, ttl_seconds, json.dumps(result))
            return result
        return wrapper
    return decorator

@mcp.tool(name="crm_search_accounts")
@cached_tool(ttl_seconds=90)  # CRM data changes infrequently enough that 90s staleness is acceptable
async def search_accounts(input: SearchAccountsInput) -> dict:
    return await crm_client.search(input.query, tier=input.tier)

Never apply this decorator to a write tool (create_ticket, update_user_address), caching must be scoped strictly to read-only, idempotent operations.

The Staleness Risk for Decision-Critical Reads, and Explicit Invalidation

A blanket TTL chosen for the common case, an agent casually looking up account info, is not automatically the right choice for every consumer of that same cached tool. Consider a cached crm_get_account read that an agent uses moments later to decide whether to approve a discount based on the account’s current contract status: if a different session just updated that contract via crm_update_contract, the cached read can hand the discount-approval agent stale data for up to the full TTL window, a consequence meaningfully more costly than a casual lookup being 90 seconds out of date. Two mitigations address this, and the right choice depends on how often this specific collision actually matters for a given tool: explicit cache invalidation, where a write tool proactively busts the relevant cache key rather than waiting for the TTL to expire naturally, or bypassing the cache entirely for calls a caller marks as decision-critical.

@mcp.tool(name="crm_update_contract")
async def update_contract(account_id: str, new_status: str) -> dict:
    """Updates contract status AND invalidates any cached reads for this
    account, so a subsequent decision-critical read is never served stale
    data left over from before this write."""
    result = await crm_client.update_contract(account_id, new_status)
    cache_key_prefix = f"mcp:cache:get_account:{account_id}"
    await redis_client.delete(*(await redis_client.keys(f"{cache_key_prefix}*")))
    return result

class GetAccountInput(BaseModel):
    account_id: str
    bypass_cache: bool = Field(False, description="Set true for decision-critical reads that cannot tolerate staleness")

@mcp.tool(name="crm_get_account")
async def get_account(input: GetAccountInput) -> dict:
    """Reads an account. Callers making a consequential decision based on
    this data (e.g. approving a discount) should set bypass_cache=true."""
    if input.bypass_cache:
        return await _get_account_uncached(input.account_id)
    return await _get_account_cached(input.account_id)

Explicit invalidation on the specific write that matters is the stronger fix where the relationship between a write and a decision-critical read is known ahead of time (as in the contract-update example); the bypass_cache escape hatch is a reasonable complement for cases where a caller (a human designing the agent’s prompt, or the agent’s own reasoning) can identify at call time that this particular read matters more than the tool’s default caching policy assumes.

Per-Tool and Per-Team Cost Attribution

Downstream API costs (LLM inference triggered by a tool, metered third-party API calls, compute-heavy warehouse queries) should be attributed back to the calling team so budget ownership is clear.

@mcp.middleware()
async def cost_attribution_middleware(request, call_next):
    result = await call_next(request)
    estimated_cost = _estimate_tool_cost(request.tool_name, request.arguments)
    await cost_ledger.record(
        team=get_caller_team(request.state.subject),
        tool=request.tool_name,
        cost_usd=estimated_cost,
        timestamp=time.time(),
    )
    return result

flowchart TD
    Ledger["Cost Ledger\n(per tool call)"] --> Dashboard["Monthly cost dashboard,\nby team and by tool"]
    Dashboard --> Sales["Sales team: $340\n(crm.* tools)"]
    Dashboard --> Support["Support team: $890\n(kb.query heavy usage)"]
    Dashboard --> Data["Data team: $2,100\n(warehouse queries)"]

    style Ledger fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style Dashboard fill:#fff7ed,stroke:#f59e0b,color:#0F172A

With scaling, caching, and cost control, and the specific caching risks that come with them, in place, the next lesson covers the CI/CD pipeline that ships changes to this server safely: schema contract checks as a merge gate, and canary releases for production rollout.