Chaos engineering is not about breaking things, it is about discovering which things were already broken before your users find them. For enterprise agent networks, the failure modes are qualitatively different from conventional microservices: LLM APIs return malformed JSON mid-stream, agent loops exhaust their step budgets silently, and a single slow tool call can cascade into a workflow-wide stall. This lesson equips you to design hypothesis-driven chaos experiments, inject realistic failures, and integrate them into CI so resilience regressions surface in minutes rather than incidents.
Defining Steady-State Hypotheses
A chaos experiment without a steady-state hypothesis is just vandalism. The hypothesis defines what “normal” looks like, so you can measure whether the system returns to it after a fault. For agent networks, steady state typically spans three dimensions: throughput (tasks completed per minute), error rate (fraction returning structured errors vs. hanging), and user-visible latency (P95 end-to-end workflow duration).
flowchart TD
H["Steady-State Hypothesis\n(baseline metrics)"] --> I["Inject Fault"]
I --> O["Observe System Behavior"]
O --> C{"Returns to\nSteady State?"}
C -- "Yes" --> P["Hypothesis Confirmed\nSystem is resilient"]
C -- "No" --> F["Hypothesis Falsified\nFix the weakness"]
F --> H
style H fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style I fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style O fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style C fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style P fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style F fill:#fff7ed,stroke:#f59e0b,color:#0F172ACapture steady state programmatically before each experiment run. Query your Prometheus instance for baseline P95 latency, pull error-rate from the last 5 minutes of logs, and store these values. After fault injection and recovery, re-query and assert that the metrics returned within your defined tolerance bands. This turns subjective intuition (“it seemed fine”) into an objective pass/fail signal.
Failure Modes Specific to AI Agent Networks
Conventional chaos experiments target infrastructure: kill a pod, block a port, fill a disk. Agent networks expose additional failure modes at the AI layer that infrastructure-level tools miss entirely.
LLM API timeouts are the most common. The API gateway accepts your connection, sends the HTTP 200 header, then stalls mid-stream because the model backend is overloaded. Your agent’s streaming parser blocks waiting for the next token chunk that never arrives. Without an explicit read timeout, this hangs silently.
Malformed tool responses occur when a downstream tool agent returns structurally invalid JSON or omits required fields. If the orchestrator does not validate tool output before passing it back to the LLM as context, the model may hallucinate a correction or loop attempting to re-invoke the tool.
Agent loop exhaustion happens when an agent’s step limit is too high and a buggy tool keeps returning results that trigger another tool call. The agent runs to its limit, burns tokens, and returns nothing useful.
A mock LLM server lets you inject all three deterministically:
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import asyncio, json, itertools
app = FastAPI()
# Pre-script failure sequences: each entry is (response_type, payload).
# WHY a cycle iterator: ensures experiments are repeatable in the same order
# across CI runs without requiring external state or a database.
FAILURE_SCRIPT = itertools.cycle([
("ok", "The analysis is complete."),
("timeout", None), # stall indefinitely, tests read-timeout handling
("malformed", "}{bad json"), # tests JSON parse error recovery
("rate_limit", None), # tests 429 backoff logic
])
@app.post("/v1/messages")
async def mock_messages(request: Request):
response_type, payload = next(FAILURE_SCRIPT)
if response_type == "timeout":
# WHY sleep 300: we want the agent's own timeout to fire, not ours.
# If the agent doesn't time out within its configured window, the test fails.
await asyncio.sleep(300)
return
if response_type == "rate_limit":
from fastapi.responses import JSONResponse
return JSONResponse(status_code=429, content={"error": {"type": "rate_limit_error"}})
if response_type == "malformed":
# WHY HTTP 200 with bad body: real APIs sometimes do this during partial failures.
# The agent must handle parse errors independently of HTTP status codes.
return StreamingResponse(iter([payload]), media_type="application/json")
async def stream():
yield json.dumps({"type": "content_block_delta", "delta": {"text": payload}})
return StreamingResponse(stream(), media_type="text/event-stream")Network Partition Simulation with Traffic Control
Infrastructure-level failures require infrastructure-level tooling. Linux tc (traffic control) can add latency, drop packets, or corrupt packets on a specific network interface, simulating slow inter-agent communication or a message broker becoming unreachable. Unlike application-layer mocking, tc affects all processes using the interface, producing the most realistic simulation of an actual network fault.
sequenceDiagram
participant CI as CI Runner
participant TC as tc (traffic control)
participant Orch as Orchestrator Agent
participant Tool as Tool Agent
CI->>TC: netem delay 500ms loss 20%
Note over TC: Partition injected on loopback
CI->>Orch: Submit workflow task
Orch->>Tool: Tool invocation (now slow + lossy)
Tool-->>Orch: Response delayed / dropped
Orch->>Orch: Timeout fires, degraded response returned
CI->>TC: tc qdisc del (restore network)
CI->>CI: Assert: workflow returned structured error, not hangA typical experiment script pairs tc commands with assertions:
import subprocess, time, requests
def inject_network_delay(interface: str, delay_ms: int, loss_pct: float):
# WHY netem at the kernel level: all agent processes using this interface
# are affected without requiring any code changes in the agents themselves.
subprocess.run([
"tc", "qdisc", "add", "dev", interface, "root", "netem",
"delay", f"{delay_ms}ms", "loss", f"{loss_pct}%"
], check=True)
def restore_network(interface: str):
subprocess.run(
["tc", "qdisc", "del", "dev", interface, "root"],
check=True
)
def run_partition_experiment(workflow_endpoint: str):
inject_network_delay("lo", delay_ms=500, loss_pct=20)
try:
start = time.time()
resp = requests.post(
workflow_endpoint,
json={"task": "summarize_report"},
timeout=15 # outer guard so the test itself doesn't hang
)
duration = time.time() - start
# WHY assert on duration, not just status: a workflow that returns 200
# after 45 seconds still violates SLA even if the response is correct.
assert duration < 12, f"Workflow hung for {duration:.1f}s under network fault"
assert resp.status_code in (200, 503), f"Unexpected status {resp.status_code}"
if resp.status_code == 503:
assert "degraded" in resp.json().get("status", ""), "No degraded status returned"
finally:
# WHY always restore in finally: a failed assertion must not leave
# tc rules in place, which would corrupt subsequent test cases.
restore_network("lo")Automating Chaos Tests in CI Pipelines
A chaos experiment that lives only in a runbook gets run once at launch and forgotten. Integrating experiments into CI ensures every merge is validated against your resilience hypotheses.
flowchart TD
PR["Pull Request"] --> UT["Unit & Integration Tests"]
UT --> CE["Chaos Experiment Suite"]
CE --> LLM["LLM Mock: timeout + malformed + 429"]
CE --> NET["Network: 500ms delay + 20% loss"]
CE --> LOOP["Agent Loop: step-limit exhaustion"]
LLM --> ASSERT["Assert steady-state metrics restored"]
NET --> ASSERT
LOOP --> ASSERT
ASSERT --> GATE{"Pass?"}
GATE -- "Yes" --> MERGE["Allow Merge"]
GATE -- "No" --> BLOCK["Block + label failure mode"]
style PR fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style UT fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style CE fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style ASSERT fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style GATE fill:#EEF0F7,stroke:#6366F1,color:#0F172A
style MERGE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
style BLOCK fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style LLM fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style NET fill:#fff7ed,stroke:#f59e0b,color:#0F172A
style LOOP fill:#fff7ed,stroke:#f59e0b,color:#0F172AKeep chaos experiments fast by scoping them to the minimum surface area needed to validate the hypothesis. An LLM timeout experiment does not need a real database, run it against the mock server in an isolated Docker network. Reserve full-environment experiments for nightly runs against a staging cluster where tc and pod-killing are safe to execute without user impact.
Label each experiment with the failure mode it covers and the specific agent component under test. When a CI run fails, the label tells the engineer immediately which resilience property regressed, not just that something broke. Attach the steady-state delta (baseline P95 vs. post-fault P95) as a CI artifact so engineers can see how badly the system degraded, not just that it did.
The discipline of chaos engineering compounds over time. Each experiment you add encodes a lesson from a past incident or a failure mode you anticipated before it became one. As your agent network grows to dozens of specialized agents and workflows, this suite becomes the fastest way to answer the question every senior engineer should be asking: what happens to our users when the LLM provider has a bad five minutes?
The next lesson moves from active fault injection to passive observation, covering distributed tracing so you can correlate events across ten concurrent agents when the failure you did not predict shows up in production.