Self-Healing Agent Networks

Production agent networks fail in ways that no runbook author anticipated. A worker agent gets stuck on a malformed task. A model API starts returning 503s intermittently, causing consumer lag to spike. A bad deployment silently reduces throughput by 40%. Self-healing systems detect these conditions automatically and execute remediation without waking engineers at 3am, reserving human escalation for the genuinely novel failures that machines cannot classify.

Watchdog Agents as First-Line Monitors

A watchdog agent is a lightweight, dedicated agent whose sole responsibility is observing the health of peer agents and triggering remediation when anomalies are detected. Unlike infrastructure-level health checks (Kubernetes liveness probes), watchdog agents understand domain semantics: they know what “normal” throughput looks like for a specific agent type at a specific time of day.

flowchart TD
    W["Watchdog Agent\n(always-on)"] --> M1["Poll Prometheus metrics\nevery 30s"]
    M1 --> CMP["Compare to 7-day\nhourly baseline"]
    CMP --> OK["Within normal range\nNo action"]
    CMP --> ANOM["Anomaly detected\n(throughput, error rate, latency)"]
    ANOM --> DIAG["Diagnose:\ncheck logs, queue depth,\ndependency health"]
    DIAG --> PB["Select remediation\nplaybook"]
    PB --> REM["Execute remediation\nactions"]
    REM --> VER["Verify recovery\nwithin SLO window"]
    VER --> PASS["Recovery confirmed\nLog incident + actions"]
    VER --> FAIL["Recovery failed\nEscalate to human"]
    style W fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style M1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style CMP fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style OK fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style ANOM fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style DIAG fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style PB fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style REM fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style VER fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style PASS fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style FAIL fill:#fff7ed,stroke:#f59e0b,color:#0F172A

Watchdog agents subscribe to the same Prometheus metrics as your alerting stack but act autonomously. They can be implemented as a separate LangGraph graph that runs on a polling loop, with tool nodes that call the Kubernetes API, the Kafka admin API, and the PagerDuty incidents API.

Anomaly Detection on Agent Metrics

Static thresholds generate false positives. A 5% error rate is alarming at 2am during steady traffic; it is expected for 90 seconds during a rolling deployment at 10am. Use a dynamic baseline built from a rolling 7-day hourly window:

import numpy as np
from datetime import datetime, timedelta
from prometheus_api_client import PrometheusConnect

prom = PrometheusConnect(url="http://prometheus:9090")

def is_anomalous(agent_type: str, metric: str, current_value: float) -> bool:
    # WHY query the same hour-of-day, same day-of-week over 4 weeks:
    # agent workload is strongly periodic. Monday 9am always has higher
    # error rates than Sunday 3am. Comparing to the same time window
    # prevents time-of-day seasonality from triggering false alerts.
    now = datetime.utcnow()
    historical = []
    for weeks_back in range(1, 5):  # 4 weeks of history
        start = now - timedelta(weeks=weeks_back, minutes=30)
        end = now - timedelta(weeks=weeks_back) + timedelta(minutes=30)
        result = prom.custom_query_range(
            f'{metric}{{agent_type="{agent_type}"}}',
            start_time=start,
            end_time=end,
            step="60",
        )
        if result:
            values = [float(v[1]) for v in result[0]["values"]]
            historical.extend(values)

    if len(historical) < 10:
        return False  # insufficient history; skip detection to avoid noise

    mean = np.mean(historical)
    std = np.std(historical)
    # WHY z-score threshold of 3: captures values more than 3 standard
    # deviations from the historical norm. This corresponds to ~0.3%
    # false-positive rate under a normal distribution: acceptable for
    # automated remediation triggers.
    z_score = (current_value - mean) / (std + 1e-9)
    return abs(z_score) > 3.0

Remediation Playbooks as Code

Remediation playbooks must be version-controlled, tested, and idempotent. An idempotent action produces the same outcome whether it runs once or ten times, critical when multiple monitoring systems may fire the same playbook simultaneously.

sequenceDiagram
    participant WD as Watchdog Agent
    participant K8s as Kubernetes API
    participant KA as Kafka Admin API
    participant P as Prometheus

    WD->>P: Query agent_tasks_processed_rate (agent=worker-a)
    P-->>WD: 0.0 (stuck for 120s)
    WD->>KA: GetConsumerGroupLag(group=worker-a)
    KA-->>WD: lag=847 (queue growing)
    WD->>WD: Select playbook: "worker_stuck_with_backlog"
    WD->>KA: DeleteConsumerGroupOffsets (reset stuck partition)
    KA-->>WD: OK
    WD->>K8s: DeletePod(name=worker-a-xyz, gracePeriod=0)
    K8s-->>WD: Pod deleted, replacement scheduled
    Note over WD,P: Wait 90 seconds for pod ready
    WD->>P: Query agent_tasks_processed_rate (agent=worker-a)
    P-->>WD: 12.4 tasks/min, recovery confirmed
    WD->>WD: Log incident: auto-resolved in 3m 20s

Each playbook is a Python function decorated with metadata that declares its retry budget and escalation trigger:

from dataclasses import dataclass
from typing import Callable

@dataclass
class Playbook:
    name: str
    max_attempts: int          # how many times to try before escalating
    verify_timeout_seconds: int  # how long to wait for recovery confirmation
    action: Callable

def playbook(name: str, max_attempts: int = 3, verify_timeout: int = 120):
    """Decorator that registers a remediation playbook."""
    def decorator(fn):
        PLAYBOOK_REGISTRY[name] = Playbook(
            name=name,
            max_attempts=max_attempts,
            verify_timeout_seconds=verify_timeout,
            action=fn,
        )
        return fn
    return decorator

@playbook(name="worker_pod_restart", max_attempts=3, verify_timeout=90)
def restart_worker_pod(agent_id: str, k8s_client) -> bool:
    # WHY delete the pod rather than calling a rolling restart on the
    # Deployment: we want this specific misbehaving pod replaced immediately,
    # not a gradual rollout that would leave the stuck pod running while
    # other pods are restarted first.
    pod_name = find_pod_for_agent(agent_id, k8s_client)
    if pod_name:
        k8s_client.delete_namespaced_pod(name=pod_name, namespace="agents")
    # Idempotent: if pod is already gone, delete is a no-op (404 ignored)
    return True

Escalation Ladder and the Feedback Loop

Self-healing is not a binary (fix it or page someone). Define a ladder: first attempt automated remediation, then retry with a more aggressive action, then page the on-call engineer with full diagnostic context already collected.

After every incident, whether auto-resolved or human-escalated, write a structured incident record to a knowledge base. The watchdog agent queries this knowledge base when diagnosing future anomalies, building institutional memory over time: “the last 5 times worker-b showed this pattern, the root cause was Kafka partition leadership election and the fix was a 45-second wait, not a pod restart.” This is the learn step of the detect → diagnose → remediate → verify → learn loop.

In the next lesson you will explore how to structure agent networks into hierarchies that mirror enterprise organisational charts, enabling delegation, accountability, and human override at every tier.