Distributed Tracing Across Agent Networks

When a single user request fans out across ten specialized agents: each making LLM calls, invoking tools, and publishing results to a message broker, structured logs from each agent become ten isolated islands of information. You know what each agent did, but you cannot answer the question that matters during an incident: which agent, which call, and which milliseconds caused the user to wait 15 seconds instead of 3? Distributed tracing exists to answer exactly that question by propagating a shared trace ID through every agent boundary and stitching all events into a single causal timeline.

Why Logs Fail at Multi-Agent Scale

A log line from an orchestrator agent at 14:23:01.452 and a log line from a tool agent at 14:23:07.891 are unrelated strings unless something connects them. In single-process systems, a request ID in thread-local storage provides that connection. In a multi-agent network, the request fans out through HTTP calls, message queues, and async event streams, each hop potentially losing the correlation identifier unless it is explicitly propagated.

flowchart TD
    U["User Request"] --> O["Orchestrator Agent\ntrace_id=abc123"]
    O --> A1["Research Agent\n(no trace_id passed)"]
    O --> A2["Analysis Agent\n(no trace_id passed)"]
    O --> A3["Writer Agent\n(no trace_id passed)"]
    A1 --> L1["Log: task complete\n(orphaned)"]
    A2 --> L2["Log: task complete\n(orphaned)"]
    A3 --> L3["Log: task complete\n(orphaned)"]
    style U fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style O fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style A1 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style A2 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style A3 fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style L1 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style L2 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style L3 fill:#EEF0F7,stroke:#6366F1,color:#0F172A

The result is that debugging a multi-agent latency spike requires manually correlating timestamps across multiple log streams, a process that takes 30 minutes for an experienced engineer and is often inconclusive. Distributed tracing eliminates this by making propagation a first-class concern of the instrumentation layer.

Propagating W3C TraceContext Across Agent Boundaries

The W3C TraceContext specification defines two HTTP headers: traceparent (carrying trace ID, parent span ID, and sampling flags) and tracestate (carrying vendor-specific data). OpenTelemetry implements this spec and handles serialization and deserialization for you.

from opentelemetry import trace
from opentelemetry.propagate import inject, extract
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import httpx

# WHY BatchSpanProcessor: it buffers spans and flushes asynchronously,
# so trace export never blocks agent decision-making in the hot path.
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://tempo:4317"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent.orchestrator")

async def invoke_tool_agent(agent_url: str, payload: dict, parent_span) -> dict:
    with tracer.start_as_current_span(
        "tool_agent.invoke",
        context=trace.set_span_in_context(parent_span),
        attributes={
            "agent.url": agent_url,
            "agent.payload_keys": list(payload.keys()),
        }
    ) as span:
        headers = {}
        # WHY inject here: this writes traceparent/tracestate into the headers dict
        # so the receiving agent can call extract() and continue the same trace.
        inject(headers)

        async with httpx.AsyncClient() as client:
            resp = await client.post(agent_url, json=payload, headers=headers, timeout=10.0)
            span.set_attribute("http.status_code", resp.status_code)

            if resp.status_code != 200:
                span.set_status(trace.StatusCode.ERROR, f"HTTP {resp.status_code}")

            return resp.json()

# On the receiving agent side: extract the context before creating child spans
from fastapi import FastAPI, Request as FastAPIRequest

app = FastAPI()
tool_tracer = trace.get_tracer("agent.tool")

@app.post("/invoke")
async def handle_invocation(request: FastAPIRequest):
    # WHY extract from headers: reconstructs the trace context propagated
    # by the caller so our spans appear as children in the same trace tree.
    ctx = extract(dict(request.headers))
    with tool_tracer.start_as_current_span("tool.execute", context=ctx) as span:
        span.set_attribute("tool.name", "document_retrieval")
        result = await execute_tool(await request.json())
        return result

Creating Meaningful Spans for Agent Operations

The value of distributed tracing depends entirely on what you put in your spans. Generic spans like "agent.run" with no attributes are nearly useless for debugging. Every span should carry enough context that an engineer can understand what happened without reading source code.

sequenceDiagram
    participant U as User
    participant O as Orchestrator
    participant L as LLM API
    participant T as Tool Agent

    U->>O: Request [trace_id=abc]
    O->>L: LLM call [span: llm.complete, model=claude-sonnet-5, tokens=1200]
    L-->>O: Tool use decision [span ends, 2.1s]
    O->>T: Tool invoke [span: tool.invoke, tool=search, parent=abc]
    T->>L: LLM call [span: llm.complete, model=claude-haiku-4-5, tokens=400]
    L-->>T: Result [span ends, 0.8s]
    T-->>O: Tool result [span: tool.invoke ends, 1.3s]
    O->>L: LLM call [span: llm.synthesis, tokens=2800]
    L-->>O: Final response [span ends, 3.4s]
    O-->>U: Response [trace ends, 7.6s total]

Instrument LLM calls with token counts and model names, these are the two attributes you will filter on most often when debugging cost anomalies and latency spikes:

async def call_llm(messages: list, model: str) -> str:
    with tracer.start_as_current_span("llm.complete") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.message_count", len(messages))

        response = await anthropic_client.messages.create(
            model=model,
            messages=messages,
            max_tokens=2048,
        )

        # WHY record token counts on the span: this is the single most useful
        # attribute for debugging runaway costs and correlating latency with
        # prompt size. Without it, you know a call was slow but not why.
        span.set_attribute("llm.input_tokens", response.usage.input_tokens)
        span.set_attribute("llm.output_tokens", response.usage.output_tokens)
        span.set_attribute("llm.stop_reason", response.stop_reason)

        return response.content[0].text

Sampling Strategies for Production

Capturing 100% of traces in development is correct, you want complete visibility during development and testing. Production at enterprise scale changes the economics. At 500 workflows per minute, each producing 50 spans, you are generating 25,000 spans per minute. Full capture is feasible in Grafana Tempo (it stores traces efficiently), but you may still want tail-based sampling to reduce egress costs to cloud storage.

Tail-based sampling keeps every trace that ends in an error or exceeds a latency threshold, and samples a small percentage of healthy fast traces. The Grafana Tempo pipeline supports this through its tail_sampling processor:

flowchart TD
    SPANS["All Agent Spans\n(25k/min)"] --> COLLECTOR["OTel Collector"]
    COLLECTOR --> TS["Tail Sampling Policy"]
    TS --> ERR["Keep: error traces\n(100%)"]
    TS --> SLOW["Keep: P95 > 10s traces\n(100%)"]
    TS --> FAST["Sample: healthy fast traces\n(5%)"]
    ERR --> TEMPO["Grafana Tempo"]
    SLOW --> TEMPO
    FAST --> TEMPO
    TEMPO --> GRAFANA["Grafana Explore\n(trace search + flame graphs)"]
    style SPANS fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style COLLECTOR fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style TS fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style ERR fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SLOW fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style FAST fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style TEMPO fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style GRAFANA fill:#f0fdf9,stroke:#0D9488,color:#0F172A

The tail-based policy requires all spans from a trace to flow through the same collector instance so the policy can evaluate the complete trace. Use a consistent-hash load balancer in front of your collector fleet, keyed on trace_id, to guarantee this.

Building a Trace-Based Debugging Workflow

When an incident fires, your debugging sequence should be: query Jaeger or Grafana Explore for traces matching the impacted time window and service, sort by duration descending, open the slowest trace, and read the flame graph from left to right. The widest span is your bottleneck. Click into that span’s attributes, model name, token count, tool name, agent URL, and you have the evidence you need to write a post-mortem root cause in under five minutes.

This workflow only works if your spans have the right attributes from the start. Retro-fitting attribute coverage into production traces after an incident is a painful and slow process. Instrument comprehensively from the beginning: every LLM call, every tool invocation, every agent-to-agent HTTP request, and every message queue publish.

Baggage, OpenTelemetry’s mechanism for propagating arbitrary key-value pairs across process boundaries: is useful for carrying business-level correlation IDs (workflow ID, user ID, tenant ID) that survive the entire trace and appear on every span without you having to set them manually on each one. Set them once at the entry point and let the propagator carry them through every agent boundary automatically.

The next lesson builds on this observability foundation by covering Prometheus metrics and Grafana dashboards for SLA tracking, moving from per-request trace debugging to fleet-wide health monitoring.