Cloud Deployment & Auto-scaling

Kubernetes Deployment for Agent Services

Kubernetes is the standard for production workloads that need auto-scaling, rolling deployments, and self-healing. Here’s a complete deployment configuration.

Kubernetes Manifests

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-service
  namespace: production   # namespaces isolate resources, production is separate from staging
  labels:
    app: agent-service
    version: "1.0.0"
spec:
  replicas: 3   # three pods for high availability; if one crashes, two keep serving
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # allow 1 extra pod during update, briefly run 4 pods to avoid downtime
      maxUnavailable: 0  # never reduce below desired replicas, zero-downtime guarantee
  selector:
    matchLabels:
      app: agent-service   # Deployment manages pods whose labels match this selector
  template:
    metadata:
      labels:
        app: agent-service
    spec:
      serviceAccountName: agent-service-sa   # least-privilege service account, only the permissions this app needs
      
      # Security context for the pod
      securityContext:
        runAsNonRoot: true   # Kubernetes rejects the pod if the container tries to run as root
        runAsUser: 1000      # matches the UID set in the Dockerfile, consistency matters
        fsGroup: 1000        # mounted volumes are owned by this group, useful for shared file access
      
      containers:
        - name: agent-api
          image: registry.example.com/agent-service:1.0.0
          imagePullPolicy: Always   # Always pull ensures the latest digest is used even if the tag didn't change
          ports:
            - containerPort: 8000
          
          # Secrets from Kubernetes Secret: never put secret values directly in the manifest
          env:
            - name: ENVIRONMENT
              value: production
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: llm-api-keys       # name of the Kubernetes Secret object
                  key: openai-api-key      # key within that Secret
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: llm-api-keys
                  key: anthropic-api-key
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: database-credentials
                  key: url
          
          # Resource limits: both requests and limits must be set for HPA to work correctly
          resources:
            requests:
              cpu: "500m"    # 0.5 CPU cores, the amount Kubernetes reserves for this pod on a node
              memory: "512Mi"
            limits:
              cpu: "2000m"   # 2 CPU cores max, prevents one pod from starving other workloads
              memory: "2Gi"  # OOM killer triggers if the container exceeds this
          
          # Health probes
          livenessProbe:
            httpGet:
              path: /health        # calls the lightweight liveness endpoint
              port: 8000
            initialDelaySeconds: 30   # give uvicorn time to start before the first check
            periodSeconds: 10         # check every 10 seconds
            failureThreshold: 3       # restart pod after 3 consecutive failures
          
          readinessProbe:
            httpGet:
              path: /health/ready   # deeper check, verifies dependencies like OpenAI and DB are reachable
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 5          # check more frequently than liveness to react faster to transient failures
            successThreshold: 1       # one passing check restores the pod to the load balancer
            failureThreshold: 3       # remove from load balancer after 3 failures (but don't restart)
          
          # Graceful shutdown
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "10"]  # drain active connections before SIGTERM is sent to uvicorn
          
          terminationGracePeriodSeconds: 30   # Kubernetes waits 30s after SIGTERM before force-killing

# k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: agent-service
  namespace: production
spec:
  selector:
    app: agent-service   # routes traffic to all pods with this label, automatically updates as pods scale
  ports:
    - protocol: TCP
      port: 80           # external port clients use when calling the Service
      targetPort: 8000   # forwarded to the pod's container port
  type: ClusterIP        # ClusterIP = internal only; use LoadBalancer or Ingress to expose externally

---
# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: agent-service
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"  # for long SSE connections, prevent nginx from closing the stream
    nginx.ingress.kubernetes.io/proxy-buffering: "off"       # disable buffering for SSE, tokens stream to the client immediately
    cert-manager.io/cluster-issuer: "letsencrypt-prod"       # cert-manager auto-provisions and renews TLS certificates
spec:
  ingressClassName: nginx
  tls:
    - hosts: [api.yourdomain.com]
      secretName: agent-service-tls   # cert-manager writes the TLS cert into this Secret
  rules:
    - host: api.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: agent-service
                port:
                  number: 80

Horizontal Pod Autoscaling

# k8s/hpa.yaml
apiVersion: autoscaling/v2   # v2 supports multiple metrics; v1 only supported CPU
kind: HorizontalPodAutoscaler
metadata:
  name: agent-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: agent-service   # HPA controls the replica count of this Deployment
  minReplicas: 2   # always at least 2 for HA, one pod can restart without downtime
  maxReplicas: 20  # cap at 20 to control costs, prevent runaway scaling from a traffic spike
  
  metrics:
    # Scale on CPU usage: CPU-bound when LLM responses are being post-processed or streamed
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70  # scale up when avg CPU > 70% across all pods
    
    # Scale on memory: add memory metrics for memory-bound workloads like large context windows
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80  # scale up when average memory usage exceeds 80%
  
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60    # wait 60s before scaling up again, prevents flapping
      policies:
        - type: Pods
          value: 4                       # add at most 4 pods at once, gradual scale-up avoids node pressure
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5 min before scaling down, agents may have long-running tasks
      policies:
        - type: Percent
          value: 25                      # remove at most 25% of pods at once, cautious scale-down
          periodSeconds: 60

Kubernetes Secrets Management

# Create secrets (never store in source control)
# --from-literal reads values from shell variables: secrets are never written to disk
kubectl create secret generic llm-api-keys \
  --from-literal=openai-api-key="$OPENAI_API_KEY" \
  --from-literal=anthropic-api-key="$ANTHROPIC_API_KEY" \
  -n production

# For production: use External Secrets Operator with AWS Secrets Manager
# It syncs secrets from AWS Secrets Manager to Kubernetes secrets automatically
# This way secrets are managed centrally and rotated without redeploying pods

CI/CD Pipeline with GitHub Actions

# .github/workflows/deploy.yml
name: Build and Deploy Agent Service

on:
  push:
    branches: [main]       # deploy automatically on every merge to main
  pull_request:
    branches: [main]       # run tests on every PR but don't deploy

env:
  REGISTRY: ghcr.io                                    # GitHub Container Registry, free for public repos
  IMAGE_NAME: ${{ github.repository }}/agent-service

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
        with:
          python-version: "3.12"
      
      - name: Install dependencies
        run: uv sync --extra dev   # --extra dev includes test and lint packages
      
      - name: Lint
        run: uv run ruff check src/ tests/   # fail fast on lint errors before running slower tests
      
      - name: Unit tests
        run: uv run pytest tests/unit/ -v --asyncio-mode=auto
        env:
          OPENAI_API_KEY: "test-key"  # mocked in tests, no real API calls during CI

  build:
    needs: test   # build only runs if tests pass, prevents pushing a broken image
    runs-on: ubuntu-latest
    outputs:
      image_tag: ${{ steps.meta.outputs.tags }}   # pass the image tag to the deploy job
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3   # Buildx enables multi-platform builds and layer caching
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}   # GITHUB_TOKEN is auto-generated, no manual secret needed
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha, prefix={{branch}}-   # e.g. main-abc1234, unique per commit
            type=ref, event=branch         # e.g. main, always points to the latest build of a branch
      
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}   # push only on merge to main, not on PRs
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha          # read layer cache from GitHub Actions cache
          cache-to: type=gha, mode=max   # write all layers to cache, speeds up subsequent builds significantly

  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'   # only deploy when merging to main, not feature branches
    environment: production               # requires manual approval if branch protection is configured
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}   # base64-encoded kubeconfig stored as a GitHub secret
      
      - name: Deploy to Kubernetes
        run: |
          # Update image tag in deployment: kubectl patches the running Deployment in-place
          kubectl set image deployment/agent-service \
            agent-api=${{ needs.build.outputs.image_tag }} \
            -n production
          
          # Wait for rollout: exits non-zero if the rollout fails within 5 minutes
          kubectl rollout status deployment/agent-service -n production --timeout=5m
      
      - name: Run smoke tests
        run: |
          ENDPOINT="https://api.yourdomain.com"
          # curl -w "%{http_code}" captures the HTTP status code: fail the pipeline if not 200
          STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$ENDPOINT/health")
          [ "$STATUS" = "200" ] || (echo "Health check failed: $STATUS" && exit 1)

Serverless Alternative: Google Cloud Run

For simpler deployments or very bursty workloads:

# cloudbuild.yaml
steps:
  # Step 1: build the Docker image and tag it with the commit SHA for traceability
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'us-central1-docker.pkg.dev/$PROJECT_ID/agents/agent-service:$COMMIT_SHA', '.']
  
  # Step 2: push the image to Artifact Registry before deploying
  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', 'us-central1-docker.pkg.dev/$PROJECT_ID/agents/agent-service:$COMMIT_SHA']
  
  # Step 3: deploy to Cloud Run: managed serverless, no nodes to configure
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    args:
      - run
      - deploy
      - agent-service
      - --image=us-central1-docker.pkg.dev/$PROJECT_ID/agents/agent-service:$COMMIT_SHA
      - --region=us-central1
      - --platform=managed
      - --min-instances=1        # keep warm, prevents cold starts for the first request after inactivity
      - --max-instances=100      # Cloud Run scales to 100 instances automatically under load
      - --memory=2Gi
      - --cpu=2
      - --timeout=3600           # for long agent runs, override the default 300s Cloud Run timeout
      - --concurrency=10         # 10 concurrent requests per instance before a new instance is spun up
      - --set-secrets=OPENAI_API_KEY=openai-api-key:latest   # inject secret from Secret Manager at runtime

Cloud Run scales to zero automatically, bills per request-second, and handles all infrastructure. The tradeoff: cold start latency (typically 2-5 seconds for Python) and limited configuration versus Kubernetes.

The right choice: Cloud Run for startups and bursty workloads. Kubernetes for teams with existing cluster expertise, fine-grained control requirements, or multi-service architectures.