Kubernetes-Native Agent Deployment

Running enterprise agent systems in Kubernetes is not simply a matter of containerizing a Python process and writing a Deployment manifest. LLM-backed agents have a resource profile unlike conventional microservices: they are I/O-bound rather than CPU-bound, they hold large context windows in memory, and their tasks can take 30-90 seconds to complete. Each of these properties demands specific Kubernetes configuration that generic tutorials overlook. This lesson covers the complete production deployment stack, from resource tuning through secret injection to graceful shutdown.

Deployment Architecture: One Deployment per Agent Type

The most maintainable Kubernetes architecture for multi-agent systems is one Deployment per agent type. This separates scaling dimensions, your orchestrator may need 3 replicas while your research agent needs 8 during peak load, and gives you clean blast radius isolation during incidents.

flowchart TD
    NS["Namespace: agent-prod"] --> ORCH["Deployment: orchestrator-agent\n3 replicas"]
    NS --> RESEARCH["Deployment: research-agent\n8 replicas"]
    NS --> WRITER["Deployment: writer-agent\n4 replicas"]
    NS --> BROKER["Service: message-broker\n(RabbitMQ or Kafka)"]
    ORCH --> BROKER
    RESEARCH --> BROKER
    WRITER --> BROKER
    ORCH --> SVC_O["Service: orchestrator-agent\n(ClusterIP)"]
    style NS fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style ORCH fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style RESEARCH fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style WRITER fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style BROKER fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style SVC_O fill:#f0fdf9,stroke:#0D9488,color:#0F172A

Use namespaces for environment isolation: agent-prod, agent-staging, agent-dev. Resource quotas at the namespace level prevent a staging misconfiguration from consuming resources needed by production pods on the same cluster.

Resource Limits Tuned for LLM Agents

LLM agents spend most of their time waiting for API responses. During that wait, they consume almost no CPU, but they hold the full conversation context in memory. A workflow processing a 50-page document through multiple agents can accumulate 200KB-2MB of context in memory per concurrent task, and a pod handling 20 concurrent tasks can reach 40MB just for context buffers before adding model client libraries and the Python runtime.

# research-agent-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: research-agent
  namespace: agent-prod
spec:
  replicas: 8
  selector:
    matchLabels:
      app: research-agent
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # WHY maxSurge 1, maxUnavailable 0: ensures capacity never drops
      # during a deploy. An agent with a 60s task budget cannot afford
      # requests being shed to an overloaded remaining pod.
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: research-agent
    spec:
      # WHY 120s: must exceed the maximum task duration (90s for research tasks)
      # so in-flight tasks complete before SIGKILL is sent after SIGTERM.
      terminationGracePeriodSeconds: 120
      containers:
        - name: research-agent
          image: registry.internal/research-agent:v1.4.2
          resources:
            requests:
              memory: "512Mi"
              cpu: "100m"   # low request: agents are I/O-bound
            limits:
              memory: "2Gi"  # hard limit: protects node from context buffer leaks
              # WHY no CPU limit: CFS throttling can spike latency even at
              # low average utilization. LLM agents have bursty CPU use
              # during JSON parsing and context management.
          env:
            - name: MAX_CONCURRENT_TASKS
              valueFrom:
                configMapKeyRef:
                  name: research-agent-config
                  key: max_concurrent_tasks

ConfigMaps for Agent Configuration

Prompt templates and model selection are operational configuration, not code. Storing them in ConfigMaps means you can update a system prompt or switch a model without rebuilding the container image, enabling rapid iteration on agent behavior during incidents.

apiVersion: v1
kind: ConfigMap
metadata:
  name: research-agent-config
  namespace: agent-prod
data:
  max_concurrent_tasks: "20"
  default_model: "claude-sonnet-5"
  # WHY store prompt templates in ConfigMap: allows prompt engineers to
  # iterate on system prompts via a ConfigMap update + rolling restart
  # without requiring a Docker build, CI pipeline, and image promotion.
  system_prompt_template: |
    You are a research specialist agent in an enterprise workflow system.
    Your task is to gather and synthesize information relevant to the
    user's query. Always cite sources. Maximum response length: 2000 tokens.
  task_timeout_seconds: "90"
  max_steps: "15"

Mount the ConfigMap as a volume (not environment variables) if you want updates to take effect without pod restart. Python’s importlib.reload or a file watcher can pick up the new values dynamically, though a rolling restart is safer for production prompt changes.

Vault Agent Injector for Secret Delivery

sequenceDiagram
    participant K8S as Kubernetes API
    participant VAULT as HashiCorp Vault
    participant INJECT as Vault Agent Sidecar
    participant AGENT as Agent Container

    K8S->>INJECT: Pod scheduled, init container starts
    INJECT->>VAULT: Authenticate via K8s Service Account JWT
    VAULT->>INJECT: Return short-lived secret lease
    INJECT->>INJECT: Write secrets to tmpfs /vault/secrets/
    INJECT->>K8S: Init container exits successfully
    K8S->>AGENT: Main container starts
    AGENT->>AGENT: Read /vault/secrets/anthropic-key at startup
    Note over INJECT, AGENT: Vault agent sidecar renews lease\nbefore expiry, no pod restart needed

Annotate your pod template to enable injection:

# Pod template annotations for Vault Agent Injector
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "research-agent"
  vault.hashicorp.com/agent-inject-secret-anthropic-key: "secret/agent-prod/anthropic"
  # WHY template annotation: formats the raw Vault secret into the exact
  # environment variable format the agent expects, avoiding extra parsing code.
  vault.hashicorp.com/agent-inject-template-anthropic-key: |
    {{- with secret "secret/agent-prod/anthropic" -}}
    export ANTHROPIC_API_KEY="{{ .Data.data.api_key }}"
    {{- end }}

The secret lands at /vault/secrets/anthropic-key as a shell-sourceable file, never appearing in the pod spec, the Kubernetes Secret store, or kubectl describe pod output.

Health Probes and Graceful Shutdown

Liveness and readiness probes for LLM agents require more thought than for stateless HTTP services. A pod that is alive (process is running) may not be ready (it is draining tasks and should not receive new ones). These two states must map to separate probe endpoints.

import signal, asyncio
from fastapi import FastAPI
from contextlib import asynccontextmanager

# WHY module-level state: signal handlers run outside the async event loop
# and cannot use async primitives; they must set synchronous flags.
_draining = False
_active_tasks: set[asyncio.Task] = set()

@asynccontextmanager
async def lifespan(app: FastAPI):
    def handle_sigterm(signum, frame):
        global _draining
        # Signal handler: set flag so readiness probe returns 503 immediately,
        # stopping the load balancer from sending new requests to this pod.
        _draining = True

    signal.signal(signal.SIGTERM, handle_sigterm)
    yield
    # Shutdown: wait for all in-flight tasks to complete before exiting.
    # Kubernetes will SIGKILL after terminationGracePeriodSeconds if this
    # takes too long: which is why that value must exceed max task duration.
    if _active_tasks:
        await asyncio.gather(*_active_tasks, return_exceptions=True)

app = FastAPI(lifespan=lifespan)

@app.get("/healthz/live")
async def liveness():
    # WHY simple: liveness failing causes pod restart. Only return 503
    # if the process is genuinely broken (event loop blocked, OOM, etc.).
    # Returning 503 during normal drain would cause a disruptive restart.
    return {"status": "alive"}

@app.get("/healthz/ready")
async def readiness():
    # WHY drain check here and not in liveness: readiness 503 removes the pod
    # from the Service endpoint list (no new requests), without restarting it.
    # This is exactly the behavior needed during graceful shutdown.
    if _draining:
        return JSONResponse({"status": "draining"}, status_code=503)
    if len(_active_tasks) >= MAX_CONCURRENT_TASKS:
        return JSONResponse({"status": "saturated"}, status_code=503)
    return {"status": "ready", "active_tasks": len(_active_tasks)}

Configure the probes in the Deployment spec to match this endpoint structure:

livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  # WHY failureThreshold 1 for readiness: remove from rotation immediately
  # when draining starts; don't wait for 3 failures = 15 seconds of new
  # requests landing on a pod that is trying to shut down.
  failureThreshold: 1

Namespace Isolation and Rolling Updates

flowchart TD
    PROD["Namespace: agent-prod\n(resource quota: 32 CPU, 128Gi RAM)"] --> PA["agent pods"]
    STAGING["Namespace: agent-staging\n(resource quota: 8 CPU, 32Gi RAM)"] --> SA["agent pods"]
    DEV["Namespace: agent-dev\n(resource quota: 2 CPU, 8Gi RAM)"] --> DA["agent pods"]
    PA --> RQ["ResourceQuota enforced\nPrevents staging from\nstarving production"]
    style PROD fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style STAGING fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style DEV fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style PA fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SA fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style DA fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style RQ fill:#fff7ed,stroke:#f59e0b,color:#0F172A

Rolling updates with maxUnavailable: 0 and maxSurge: 1 keep capacity constant throughout a deployment. New pods must pass their readiness probe before old pods receive SIGTERM. Old pods drain in-flight tasks during terminationGracePeriodSeconds. The result is a deployment with zero dropped tasks, even for 90-second workflows, as long as the grace period is correctly tuned.

Monitor rolling update progress with kubectl rollout status deployment/research-agent -n agent-prod and set a timeout in your CI/CD pipeline that fails the deployment if rollout does not complete within a reasonable window. A stalled rollout (new pods not becoming ready) is a production incident, not a deployment pipeline concern, treat it as one.

The patterns in this lesson: resource tuning, secret injection via Vault, SIGTERM-based draining, and namespace isolation, compose into a deployment foundation that handles the operational realities of enterprise agent systems: long-running tasks, sensitive credentials, and zero-downtime updates at scale.