Auto-Scaling Agent Fleets

Enterprise agent fleets face a scaling paradox: the work is highly variable (bursts of thousands of tasks interspersed with quiet periods), but the agents themselves look idle to the metrics that Kubernetes uses by default. Getting scaling right determines both your platform’s responsiveness and your monthly cloud bill.

Why CPU Metrics Fail for LLM Agents

A traditional stateless web server earns its CPU during every request. An LLM agent does the opposite, it spends most of its time blocked on the network, waiting for a model API response that may take 5–30 seconds. During that wait, CPU utilisation sits near zero.

flowchart TD
    A["Task arrives in queue"] --> B["Agent dequeues task"]
    B --> C["Build prompt\n(~5ms CPU)"]
    C --> D["Send to LLM API\n(blocked ~15s network)"]
    D --> E["Parse response\n(~2ms CPU)"]
    E --> F["Write result to DB\n(blocked ~10ms I/O)"]
    F --> G["Acknowledge message"]
    style A fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style B fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style C fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style D fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style E fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style F fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style G fill:#f0fdf9,stroke:#0D9488,color:#0F172A

An agent processing 50 tasks per hour might show 4% average CPU. HPA’s default target of 60% will never fire, and the task queue grows without bound during load spikes. The correct scaling signal is not resource consumption, it is work backlog.

Event-Driven Scaling with KEDA

KEDA (Kubernetes Event-Driven Autoscaler) extends Kubernetes with ScaledObjects that can react to external event sources: Kafka consumer lag, Redis list length, Azure Service Bus queue depth, or any Prometheus metric. It acts as an external metrics provider to the standard HPA controller, so it integrates cleanly without replacing existing tooling.

flowchart TD
    K["Kafka Topic\nagent-tasks"] --> S["KEDA ScaledObject\nlagThreshold: 50"]
    R["Redis Queue\ntask:pending"] --> S2["KEDA ScaledObject\nlistLength: 20"]
    S --> HPA["HPA Controller\ntargetReplicas = lag / 50"]
    S2 --> HPA2["HPA Controller\ntargetReplicas = len / 20"]
    HPA --> D["agent-fleet\nDeployment"]
    HPA2 --> D
    style K fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style R fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style S fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style S2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style HPA fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style HPA2 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style D fill:#EEF0F7,stroke:#6366F1,color:#0F172A

A Kafka-triggered ScaledObject looks like this in a Kubernetes manifest:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: agent-fleet-scaler
spec:
  scaleTargetRef:
    name: agent-fleet
  minReplicaCount: 0          # scale-to-zero when queue is empty
  maxReplicaCount: 100
  pollingInterval: 15         # check every 15 seconds
  cooldownPeriod: 120         # wait 2 minutes before scaling down
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka:9092
        consumerGroup: agent-workers
        topic: agent-tasks
        lagThreshold: "50"    # one pod per 50 pending messages

With this configuration, a queue depth of 500 messages produces 10 replicas; 2500 produces 50. KEDA polls the Kafka broker every 15 seconds, so the fleet responds within one polling cycle of the backlog growing.

Custom Prometheus Metrics as KEDA Triggers

Queue depth is a coarse signal. For smarter scaling, expose domain-specific metrics from agent code and use them as KEDA triggers. Two metrics that work well are agent_tasks_in_flight (tasks being processed right now) and agent_queue_wait_p95 (the 95th-percentile wait time for a task to reach a worker).

from prometheus_client import Gauge, Histogram, start_http_server
import time

# WHY Gauge not Counter: tasks_in_flight can decrease (task completes),
# so we need a metric that goes up and down, not a monotonically
# increasing counter.
tasks_in_flight = Gauge(
    "agent_tasks_in_flight",
    "Number of tasks currently being processed by this pod",
)

# WHY Histogram: we care about tail latency (p95/p99), not just average.
# Histograms let Prometheus compute percentiles server-side from buckets.
queue_wait = Histogram(
    "agent_queue_wait_seconds",
    "Time a task spent waiting in queue before processing",
    buckets=[1, 5, 10, 30, 60, 120, 300],
)

def process_task(task):
    enqueued_at = task.metadata["enqueued_at"]
    # Record wait time the moment we dequeue: before any processing.
    # This isolates queue latency from execution latency.
    queue_wait.observe(time.time() - enqueued_at)

    tasks_in_flight.inc()
    try:
        result = run_agent_logic(task)
    finally:
        # WHY finally: decrement even if the task raises an exception
        # so the gauge stays accurate and doesn't drift high over time.
        tasks_in_flight.dec()
    return result

Wire this to KEDA using the Prometheus trigger type, targeting a queue_wait_p95 threshold of 30 seconds, meaning KEDA scales out whenever tasks are waiting more than half a minute.

Scale-to-Zero and Cold Start Management

Scale-to-zero is KEDA’s most cost-effective feature: when the queue is empty, the fleet shrinks to zero pods and pays nothing for idle compute. The catch is cold-start latency. An agent pod that must download 400 MB of model embeddings or establish 10 database connections at startup can take 60–120 seconds to become ready.

For overnight idle periods, scale-to-zero is almost always worth the cold-start cost for the first morning task. For predictable daily bursts (09:00 business-hours load), a warm pool strategy avoids the penalty entirely.

flowchart TD
    T1["KEDA CronTrigger\n08:55 Mon–Fri"] --> W["Set minReplicas = 10\n(warm pool)"]
    T2["KEDA CronTrigger\n19:00 Mon–Fri"] --> Z["Set minReplicas = 0\n(scale-to-zero)"]
    W --> P["10 pods initialised\nand ready"]
    K["Kafka lag rises at 09:00"] --> KEDA["KEDA scales\nbeyond warm pool"]
    KEDA --> F["Fleet at 10–100 pods\nno cold start for first tasks"]
    style T1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style T2 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style W fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style Z fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style P fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style K fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style KEDA fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style F fill:#f0fdf9,stroke:#0D9488,color:#0F172A

To speed up cold starts unconditionally, split agent initialisation into two phases: a fast readiness phase (only health endpoint available) and a warm phase (LLM client loaded, embeddings cached). Use Kubernetes readinessProbe to hold traffic until the warm phase completes, and use init containers to pre-pull large model artifacts from object storage during pod startup in parallel with container image pull.

Cost Impact of Scaling Strategies

Running 20 agent pods continuously at $0.08/hour each costs roughly $1,150/month. KEDA with scale-to-zero for 14 overnight hours and a warm pool of 5 pods during 10 business hours costs closer to $290/month for the same average throughput, a 75% reduction. The scaling strategy is not an infrastructure detail; it is a product economics decision that belongs in architecture reviews.

In the next lesson you will take the fleet you have scaled and distribute it across multiple cloud regions for resilience and low-latency global access.