Prefill-Decode Disaggregation (PD Disaggregation)

Prefill-decode disaggregation (PD disaggregation) is a way of serving large language models in which the two halves of every request, prefill (reading the whole prompt in one parallel pass) and decode (generating the answer one token at a time), run on different machines. The prefill machine builds the KV cache for the prompt and ships it across the network; the decode machine picks it up and streams tokens. It matters because the two phases stress hardware in opposite ways, and forcing them to share GPUs is the main reason a busy inference cluster either has slow first tokens, choppy streaming, or a lot of idle silicon.

Two Phases, Two Different Bottlenecks

Every LLM request is really two workloads glued together.

Prefill processes the entire prompt at once. Thousands of tokens flow through each layer as a large matrix multiplication, which keeps the GPU’s compute units saturated (kernels like FlashAttention exist largely to make this pass efficient on long prompts). It is compute-bound, and its cost grows with prompt length. Prefill is what determines time to first token (TTFT), and it writes the key and value vectors for every prompt token into the KV cache as a by-product.

Decode then generates output one token per step. Each step does very little arithmetic (one new token per sequence), but it must read the model’s weights and the sequence’s entire KV cache out of GPU memory to do it. It is memory-bandwidth-bound, and it is what determines the inter-token latency (ITL), also called time per output token (TPOT), the smoothness of the streamed answer.

This split is the whole premise of the architecture. Two 2023-2024 systems papers, Splitwise (Patel et al.) and DistServe (Zhong et al.), independently made the same observation and proposed the same fix: if the phases want different hardware, stop making them share it.

The Interference Problem With Colocation

The default way to serve a model is colocated: one pool of GPUs runs both phases, and a scheduler using continuous batching interleaves them. It works, and for short prompts it works well. The trouble is what happens when a long prompt arrives.

A new request’s prefill is a big, compute-heavy job. While it runs on a GPU, the decode steps of every other request on that GPU have to wait, so their next token arrives late. Users see a stream that runs smoothly and then freezes for a second. Going the other way, prefill work queues behind decode batches, and the new request’s TTFT grows. You can tune the scheduler to favor one metric, but you cannot make both good at once on shared hardware, and you must provision for the worst case of both.

DistServe frames this as goodput: not raw throughput, but the number of requests per second that meet both a TTFT target and a TPOT target. By assigning the phases to different GPUs and choosing the parallelism strategy for each independently, the DistServe authors report serving up to 7.4x more requests, or meeting a 12.6x tighter latency target, than the state-of-the-art colocated systems they compared against while staying within the latency constraints for more than 90% of requests.

%%{init: {'theme': 'base'}}%%
graph LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    subgraph CO["Colocated: one pool, both phases"]
      R1[Requests]:::data --> G1[GPU pool: prefill and decode interleaved]:::process
      G1 --> T1[Token stream stalls whenever a long prefill lands]:::output
    end

    subgraph DI["Disaggregated: two pools"]
      R2[Requests]:::data --> RT[Router]:::process
      RT --> P[Prefill pool: compute-dense]:::process
      P -->|"KV cache over NIXL (RDMA or TCP)"| D[Decode pool: memory-bandwidth-rich]:::process
      D --> T2[Steady token stream]:::output
    end

How Disaggregation Works

A disaggregated request follows four steps:

1. Route. A router (or the serving framework’s own scheduler) picks a prefill worker and a decode worker for the request.
2. Prefill. The prefill worker runs the prompt through the model, producing the first token and the full KV cache.
3. Transfer. The KV cache moves from the prefill worker’s GPU memory to the decode worker’s GPU memory. This step is the new cost that colocated serving does not pay.
4. Decode. The decode worker generates the remaining tokens, and the prefill worker is already free to take the next prompt.

The transfer step is handled by a dedicated library. In vLLM’s implementation, a connector such as NixlConnector uses NVIDIA’s NIXL (Inference Xfer Library) for asynchronous send and receive of KV blocks between the two instances, with UCX as the default transport underneath, so the cache can move over RDMA or TCP without going through the CPU-side serving stack. The prefill instance is launched with the role kv_producer and the decode instance with kv_consumer, exactly as in the launch commands above. Disaggregation is also supported in SGLang, NVIDIA Dynamo, llm-d, and Modular’s MAX, so the approach is not tied to one engine.

How Big Is the Handoff?

Whether the transfer is cheap or ruinous depends on how large the KV cache is. Per token, the cache holds one key and one value vector for every layer and every KV head:

# Scenario: sizing the network link between prefill and decode pools for a
# 70B-class model that uses grouped-query attention (80 layers, 8 KV heads,
# head dim 128, fp16 cache).
def kv_bytes(tokens, layers=80, kv_heads=8, head_dim=128, dtype_bytes=2):
    per_token = 2 * layers * kv_heads * head_dim * dtype_bytes  # 2 = key + value
    return tokens * per_token

def transfer_ms(tokens, link_gbps_bytes):
    return kv_bytes(tokens) / (link_gbps_bytes * 1e9) * 1000

print(kv_bytes(1))                 # 327,680 bytes: about 320 KB per token
print(kv_bytes(8_192) / 1e9)       # about 2.7 GB for an 8k-token prompt
print(transfer_ms(8_192, 50))      # about 54 ms over a 50 GB/s (400 Gb/s) link
print(transfer_ms(8_192, 12))      # about 224 ms over a ~100 Gb/s network

This is why grouped-query attention and other cache-shrinking designs matter more under disaggregation: every byte trimmed from the cache is a byte that no longer crosses the network. It is also why the interconnect is a first-class part of the design rather than an afterthought.

Interactive: When Does the Handoff Pay for Itself?

Disaggregation trades a small, certain cost (moving the cache, plus scheduling overhead, added to TTFT) for the removal of a large, unpredictable one (a long prefill freezing every other stream on the GPU). Whether that trade is good depends on prompt length, link speed, and how fast your prefill runs.

Push the prompt length down to a few hundred tokens and the verdict flips to “stay colocated”: the prefill is so short that it barely disturbs anyone, so paying for a handoff buys nothing. Push it past a few thousand tokens and the colocated stall grows in proportion while the handoff cost grows much more slowly, so disaggregation wins by a widening margin. Drop the bandwidth to 1 GB/s and the break-even length becomes “never”: if moving the cache is slower than simply recomputing it, the whole idea fails. The model is deliberately simple (one long request, unchunked prefill, no queueing, fixed 70B-class cache size), so treat the numbers as intuition for the shape of the trade-off rather than a benchmark.

Routing: Not Every Request Should Be Split

Because the handoff has a fixed cost, production systems do not disaggregate blindly. A router decides per request, and the two inputs it cares most about are prompt length and how much of the prompt is already sitting in a prefix cache.

# Scenario: a router in front of a mixed workload, short chat turns and long
# document-analysis prompts. Only long, mostly-uncached prompts justify a
# remote prefill; everything else prefills locally on the decode worker.
def choose_path(prompt_tokens, cached_prefix_tokens, min_remote_tokens=1024):
    uncached = prompt_tokens - cached_prefix_tokens
    if uncached < min_remote_tokens:
        return "local_prefill"    # short, or the prefix cache already covers most of it
    return "remote_prefill"       # long and cold: worth the KV handoff

print(choose_path(300, 0))         # local_prefill  (short chat turn)
print(choose_path(12_000, 11_500)) # local_prefill  (long, but 96% cached from earlier turns)
print(choose_path(20_000, 0))      # remote_prefill (large fresh diff)

This logic is the same idea behind Prefill-as-a-Service (see below): send only the long, uncached prompts to dedicated prefill capacity, and keep short or cache-heavy requests close to where they will decode. A client never sees any of it, since it talks to one OpenAI-compatible endpoint:

# Scenario: the code-review bot from earlier. The application code is
# unchanged; the proxy decides whether this request is split across pools.
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8192/v1", api_key="unused")
stream = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": open("big_diff.patch").read() + "\n\nReview this."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Sizing the Pools and Choosing Hardware

Once the phases are separate, they scale independently, which is the second big payoff after killing interference. A workload of many short prompts with long answers (a chat assistant) needs a small prefill pool and a large decode pool. A workload of long prompts with short answers (summarization, code review, retrieval-heavy RAG) flips the ratio. Multi-turn conversations with heavy prompt overlap tend to need more decode capacity, because the prefix cache absorbs much of the prefill. With one shared pool, you can only guess a single blend; with two, you follow the workload.

Each pool can also use its own parallelism strategy. Splitwise made the hardware argument explicit: token generation does not need the compute of the newest GPUs and can run on cheaper or lower-power hardware, and the authors report up to 1.4x higher throughput at 20% lower cost than a conventional design. That reasoning is now being pushed to specialized silicon: NVIDIA’s Rubin CPX is positioned around prefill throughput, while accelerators like Groq’s LPU are built around decode bandwidth.

The Alternative: Chunked Prefill on Colocated Hardware

Disaggregation is not the only answer to interference. Chunked prefill, introduced in Sarathi-Serve, slices a long prompt into fixed-size chunks and mixes each chunk into a decode batch, so a new request joins the batch without freezing running streams (“stall-free scheduling”). The authors report 2.6x higher serving capacity on Mistral-7B on one A100, up to 3.7x on Yi-34B on two A100s, and up to 5.6x on Falcon-180B with pipeline parallelism, versus vLLM at the time.

# Scenario: the same code-review bot, but on a single node where a KV network
# link isn't available. Keep prefill and decode colocated and bound the stall
# by capping how many prompt tokens are batched per step.
vllm serve Qwen/Qwen3-0.6B --enable-chunked-prefill --max-num-batched-tokens 2048

The two techniques are complementary rather than exclusive: many deployments use chunked prefill inside each pool and disaggregation between them.

When Not to Use It

Disaggregation is an operational commitment, and it can make things worse:

• Small or lightly loaded workloads. Guides report 20-30% degradation on workloads too small to keep two pools busy, since you pay for two pools’ worth of idle capacity plus the handoff.
• Short prompts. Local prefill with a high prefix-cache hit rate is often faster than shipping a cache, and the widget above shows why: the stall being removed is smaller than the handoff.
• Weak interconnects. If moving the cache costs more than recomputing it, the architecture has no upside.
• Heterogeneous caches. Prefill and decode workers must agree on KV layout, page size, data type, and attention variant, so mixing engine versions, quantization schemes, or attention kernels between pools takes care.
• Engineering overhead. You now operate a router, two autoscaling groups, a transfer fabric, and failure modes such as a lost or corrupted cache transfer mid-request.

What’s New (2025-2026)

• From research to default infrastructure. The idea began as research prototypes (Splitwise and DistServe in late 2023 and early 2024) and production systems (Moonshot AI’s Mooncake, which serves its Kimi chatbot with a disaggregated, KVCache-centric design and reports up to 525% higher throughput in simulation and 75% more requests handled in production). By 2026, vLLM, SGLang, NVIDIA Dynamo, llm-d, Modular’s MAX, and Ray Serve all document a prefill/decode split, with NIXL as the common KV-transfer layer.
• Cross-datacenter prefill. Prefill-as-a-Service (Qin et al., April 2026) argues that newer hybrid-attention models shrink the KV cache enough to make prefill in a separate, compute-dense cluster practical, even across datacenters, by offloading only long uncached prompts. They report 54% higher throughput and 64% lower P90 TTFT than a homogeneous baseline, and roughly 15% higher throughput at equal cost.
• Scheduling for agent workloads. A June 2026 paper (Observation, Not Prediction) treats a whole multi-turn agent conversation as the scheduling unit, observing that agent traffic looks like one big prefill followed by a long, memory-bound tail, and reports a 51% reduction in p95 time-to-first-effective-token. As agentic workloads with long shared contexts become the dominant traffic, routing by conversation rather than by request is becoming standard.
• Hardware split by phase. Prefill-optimized and decode-optimized accelerators are appearing, and disaggregation is the software layer that lets a fleet mix them.

Practical Guidance

Disaggregation does not make any single token cheaper to compute. What it changes is that the two very different jobs inside one request stop fighting over the same silicon, so a cluster can be sized, tuned and bought for what each phase actually needs.

# Scenario: a code-review assistant that receives 20k-token diffs and streams
# back a short answer. Long prefills on the same GPUs as live decodes make
# every other user's stream stutter, so run the two phases on separate GPUs.

# Prefill instance (KV producer): reads the prompt, builds the KV cache.
CUDA_VISIBLE_DEVICES=0 \
UCX_NET_DEVICES=all \
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve Qwen/Qwen3-0.6B \
  --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'

# Decode instance (KV consumer): receives the cache, streams the tokens.
CUDA_VISIBLE_DEVICES=1 \
UCX_NET_DEVICES=all \
VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
vllm serve Qwen/Qwen3-0.6B \
  --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

# A proxy sends each request to the prefiller first, then hands the KV cache
# reference to the decoder. Clients talk to one OpenAI-compatible endpoint.
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
  --port 8192 \
  --prefiller-hosts localhost --prefiller-ports 8100 \
  --decoder-hosts localhost --decoder-ports 8200