Metrics, Alerting & SLA Dashboards

Distributed traces tell you what happened in a single workflow. Metrics tell you what is happening across all workflows right now. For enterprise agent networks where dozens of teams depend on your infrastructure, you need both layers: traces for debugging specific incidents and metrics for detecting systemic degradation before it breaches SLA commitments. This lesson covers the full stack from instrumentation to dashboards to alerting that actually wakes the right person at the right time.

The Four Golden Signals for Agent Systems

Google’s Site Reliability Engineering book defines four golden signals: latency, traffic, errors, and saturation, as the minimum viable set of metrics for any production system. For agent networks, each signal has a specific interpretation worth making explicit.

Latency for agents means P95 and P99 end-to-end workflow duration, not just LLM call duration. A fast LLM call that triggers three slow tool agents still results in a slow workflow. Track latency at the workflow level, not the component level, as your primary SLA metric.

Traffic means workflows per minute per workflow type. Different workflow types have different resource profiles, a document summarization workflow is far cheaper than a research-and-synthesis workflow. Traffic breakdowns by type help you predict when resource limits will become binding.

Errors means the rate of workflows returning structured errors or timing out. Track separately from HTTP 5xx errors, because an agent that returns a well-formed error response (status 200, error field populated) is failing the user without failing the HTTP layer.

Saturation means LLM API rate limit headroom and message queue depth. When you are consistently at 80% of your token-per-minute limit, the next traffic spike will degrade latency before it errors, saturation is the leading indicator.

Prometheus Instrumentation

Instrument at three levels: individual LLM calls, tool invocations, and end-to-end workflow completion. Use labels to carry team and workflow type dimensions, these become the axes of your SLA dashboards.

from prometheus_client import Counter, Histogram, Gauge
import time, functools

# WHY separate counters for total and errors: Prometheus rate() over
# both lets you compute error_rate = errors/total without a separate
# calculation, and alerts can fire on the ratio directly.
workflow_total = Counter(
    "agent_workflow_total",
    "Total workflows submitted",
    ["team_id", "workflow_type"]
)
workflow_errors = Counter(
    "agent_workflow_errors_total",
    "Workflows that returned an error or timed out",
    ["team_id", "workflow_type", "error_type"]
)
workflow_duration = Histogram(
    "agent_workflow_duration_seconds",
    "End-to-end workflow duration",
    ["team_id", "workflow_type"],
    # WHY these specific buckets: they align with SLA thresholds (3s, 8s)
    # and give useful percentile resolution for P95/P99 calculations.
    buckets=[0.5, 1, 2, 3, 5, 8, 12, 20, 30, 60]
)
llm_tokens_used = Counter(
    "agent_llm_tokens_total",
    "LLM tokens consumed",
    ["team_id", "model", "token_type"]  # token_type: input or output
)
rate_limit_headroom = Gauge(
    "agent_llm_rate_limit_remaining_pct",
    "Remaining rate limit as percentage of maximum",
    ["model"]
)

def instrument_workflow(team_id: str, workflow_type: str):
    """Decorator that records all four golden signals for a workflow function."""
    def decorator(fn):
        @functools.wraps(fn)
        async def wrapper(*args, **kwargs):
            workflow_total.labels(team_id=team_id, workflow_type=workflow_type).inc()
            start = time.monotonic()
            try:
                result = await fn(*args, **kwargs)
                return result
            except Exception as exc:
                # WHY label by error_type: allows alert rules to target
                # specific error classes (timeout vs. validation vs. upstream)
                # rather than firing on all errors indiscriminately.
                error_type = type(exc).__name__
                workflow_errors.labels(
                    team_id=team_id,
                    workflow_type=workflow_type,
                    error_type=error_type
                ).inc()
                raise
            finally:
                duration = time.monotonic() - start
                workflow_duration.labels(
                    team_id=team_id,
                    workflow_type=workflow_type
                ).observe(duration)
        return wrapper
    return decorator

RED Method Dashboards in Grafana

The RED method (Rate, Errors, Duration) is a dashboarding pattern derived from the golden signals, optimized for service-level views. Each workflow type gets a RED row in your Grafana dashboard.

flowchart TD
    PROM["Prometheus\n(metrics store)"] --> GRA["Grafana Dashboard"]
    GRA --> R["Rate panel\nworkflows/min by team"]
    GRA --> E["Error Rate panel\n% errors by workflow type"]
    GRA --> D["Duration panel\nP50/P95/P99 heatmap"]
    GRA --> S["Saturation panel\nRate limit headroom %"]
    R --> SLA["SLA Status row\n99% < 8s target line"]
    E --> SLA
    D --> SLA
    SLA --> ANNO["Incident annotations\n(deployment markers)"]
    style PROM fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style GRA fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style R fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style E fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style D fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style S fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SLA fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style ANNO fill:#EEF0F7,stroke:#6366F1,color:#0F172A

The PromQL for a P95 latency panel per team looks like this:

histogram_quantile(
  0.95,
  sum by (team_id, le) (
    rate(agent_workflow_duration_seconds_bucket[5m])
  )
)

Add a constant threshold line at your SLA target (8 seconds) on the same panel. When P95 crosses that line, the panel turns red before any alert fires, giving engineers visual context during incident response without needing to remember the SLA number.

Multi-Window Burn Rate Alerts

Single-threshold alerting (“fire when any request exceeds 8 seconds”) creates alert fatigue. The Google SRE workbook’s multi-window burn rate approach is far more effective: it computes how fast you are consuming your error budget and fires only when the burn rate is unsustainable.

flowchart TD
    EB["Error Budget\n1% of requests may exceed SLA\nin a 30-day window"] --> BR["Burn Rate Calculation"]
    BR --> MW1["1h window burn rate\n(fast detection)"]
    BR --> MW2["6h window burn rate\n(false positive filter)"]
    MW1 --> AND{"Both windows\nexceed threshold?"}
    MW2 --> AND
    AND -- "Yes: 14x burn (1h) + 3x burn (6h)" --> PAGE["P1 Page:\nSLA breach imminent\n~2h of budget remaining"]
    AND -- "No" --> WATCH["No page\nContinue monitoring"]
    style EB fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style BR fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style MW1 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style MW2 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style AND fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style PAGE fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style WATCH fill:#f0fdf9,stroke:#0D9488,color:#0F172A

The Grafana alerting rule for a 1% error budget over 30 days with a 14x burn rate threshold:

# In Grafana alerting rules (YAML provisioning)
groups:
  - name: agent_sla
    rules:
      - alert: AgentWorkflowSLABurnRateHigh
        expr: |
          (
            sum(rate(agent_workflow_errors_total[1h])) by (team_id)
            /
            sum(rate(agent_workflow_total[1h])) by (team_id)
          ) > (14 * 0.01)
          and
          (
            sum(rate(agent_workflow_errors_total[6h])) by (team_id)
            /
            sum(rate(agent_workflow_total[6h])) by (team_id)
          ) > (3 * 0.01)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "SLA burn rate critical for team {{ $labels.team_id }}"
          description: "Error budget will be exhausted in ~2 hours at current rate."
          runbook_url: "https://wiki.internal/runbooks/agent-sla-breach"
          dashboard_url: "https://grafana.internal/d/agent-sla?var-team={{ $labels.team_id }}"

The runbook_url and dashboard_url annotations are not cosmetic, they are the difference between a 5-minute and a 25-minute mean time to resolution when an engineer is paged at 3am.

PagerDuty Escalation Policies

Alert routing in PagerDuty should match urgency to escalation tier. P1 burns (imminent SLA breach) go directly to the primary on-call engineer with a 5-minute acknowledgment window before escalating to the secondary. P2 burns (elevated but not critical) can route to a lower-urgency channel that does not wake anyone up.

Include in every P1 alert: the team affected, the current burn rate multiple, the estimated time until budget exhaustion, the runbook link, and the Grafana dashboard link pre-loaded to the incident time range. Engineers who receive pages with this context can begin remediation immediately rather than spending the first 15 minutes reconstructing what happened.

Review your alert firing history weekly for the first month after deployment. Alerts that fire and resolve without engineer action within 10 minutes are candidates for de-escalation to P2. Alerts that consistently fire together suggest they should be grouped into a single composite alert to reduce paging volume.

The next lesson moves from latency and error rate to a different dimension of production health: cost attribution across teams, so finance and engineering both understand where LLM spend is going and which teams are approaching their budgets.