Speculative Decoding

Speculative decoding is an inference acceleration technique that breaks the one-token-at-a-time bottleneck of autoregressive generation by letting a small, fast draft model guess several tokens ahead, then having the large target model check all of those guesses in a single forward pass. Every guess that survives verification is a token the expensive model produced without ever running a separate step for it. The critical property, and the reason the technique spread so quickly, is that a carefully constructed acceptance rule makes the final output distribution provably identical to what plain autoregressive sampling from the target model would have produced. Introduced by Google Research in Fast Inference from Transformers via Speculative Decoding and independently by DeepMind in Accelerating Large Language Model Decoding with Speculative Sampling, it now ships as a first-class feature in vLLM, SGLang, and TensorRT-LLM, and is the default serving configuration for a large share of latency-sensitive large language model deployments.

The Bottleneck It Solves: Memory Bandwidth, Not Compute

Generating one token from a 70B-parameter model requires reading all 70 billion weights out of HBM into the GPU’s compute units. At batch size 1 the arithmetic is trivial relative to that data movement, so the GPU spends most of every decode step waiting on memory. This is what makes decoding memory-bandwidth bound: the tensor cores are largely idle, and adding more FLOPs does nothing.

The prefill phase has the opposite character. Processing a 4,000-token prompt runs 4,000 positions through the same weights in parallel, so a single weight read serves thousands of multiply-accumulate operations, and the GPU is compute bound and efficient. Speculative decoding exists to import that efficiency into the decode phase. If you can hand the target model several candidate tokens at once, verifying five tokens costs almost exactly what verifying one token costs, because the expensive part (streaming the weights) happens once either way.

# Scenario: measuring why decode is slow on a single-request chat endpoint.
# Standard autoregressive decoding: one full weight read per token.
for step in range(max_new_tokens):
    logits = target_model(tokens)      # reads ~140 GB of fp16 weights
    tokens.append(sample(logits[-1]))  # emits exactly 1 token

# 512 tokens of output means 512 full passes over the weights,
# and the tensor cores sit near-idle for almost all of it.

How It Works: Draft, Verify, Accept or Correct

%%{init: {'theme': 'base'}}%%
graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    CTX([Current context]):::data --> DRAFT[Draft model: gamma cheap sequential steps]:::process
    DRAFT -->|"candidate tokens x_1 ... x_gamma"| VER[Target model: ONE parallel forward pass]:::process
    VER -->|"target probs q at every position"| CHECK{Accept token i? r < q_i / p_i}:::process
    CHECK -->|accept| KEEP[Keep token, move to next position]:::data
    KEEP --> CHECK
    CHECK -->|first rejection| RESAMP[Resample from normalized q minus p]:::process
    CHECK -->|all gamma accepted| BONUS[Free bonus token from the same pass]:::data
    RESAMP --> OUT[Emit 1 to gamma+1 tokens this step]:::output
    BONUS --> OUT
    OUT -.->|new context| CTX

Each speculative step runs three phases:

1. Draft. A small model runs γ (gamma) cheap autoregressive steps, producing candidate tokens x_1 ... x_γ and its own probability p_i for each one. A 1B draft against a 70B target costs roughly one to three percent of a target step, so these steps are nearly free.
2. Verify. The target model runs those γ candidates plus the current context through one forward pass. Because of causal attention, that single pass yields the target’s true distribution q_i at every one of the γ + 1 positions simultaneously.
3. Accept or correct. Walk the candidates left to right. Accept x_i with probability min(1, q_i(x_i) / p_i(x_i)). On the first rejection, discard every remaining candidate and resample that position from the residual distribution norm(max(0, q_i - p_i)). If all γ candidates are accepted, the pass has already computed q_{γ+1}, so you get one extra bonus token.

That acceptance rule is the entire correctness argument. It is standard rejection sampling, and it guarantees the emitted token is distributed exactly as q, the target model’s own distribution.

# The acceptance rule, which is where "lossless" actually comes from.
# Scenario: verifying one drafted token against the target's true distribution.
import random

def accept_or_correct(token, p_draft, q_target):
    """p_draft, q_target: full probability vectors over the vocabulary."""
    r = random.random()
    if r < min(1.0, q_target[token] / p_draft[token]):
        return token, True                      # accepted, cost us nothing
    residual = [max(0.0, q_target[v] - p_draft[v]) for v in range(len(q_target))]
    total = sum(residual)
    residual = [v / total for v in residual]    # normalized q minus p
    return sample_from(residual), False         # corrected, still exactly q

Two consequences fall out of this. First, when the draft is confident and correct, q/p is near 1 and tokens sail through. Second, a bad draft model never corrupts output, it only wastes the drafting compute, because rejected tokens are thrown away and replaced by a sample from the target’s own corrected distribution. Speculative decoding trades compute for latency, and it cannot trade away quality.

The Speedup Formula

The Leviathan paper gives a closed form. Let α be the expected acceptance rate (the probability a drafted token survives verification), γ the draft length, and c the cost of one draft step as a fraction of one target step. The expected number of tokens emitted per verification step is:

E[tokens per step] = (1 - α^(γ+1)) / (1 - α)

speedup = E[tokens per step] / (1 + γ·c)

The numerator is a truncated geometric series: it saturates as γ grows, because accepting eight tokens in a row requires α^8, which decays fast. The denominator grows linearly in γ. That tension is why the optimal draft length is finite and usually small.

Interactive: Tuning Draft Length Against Acceptance Rate

γ is the knob every serving team actually turns, and its optimum depends entirely on α and c. Drag the sliders below to see the speedup curve reshape and the optimal draft length move.

Three behaviors are worth pulling out. At low α, the curve peaks at γ = 2 or 3 and then falls, because you are paying to draft tokens that get rejected. Raise α toward 0.9 and the optimum slides well past five, since long accepted runs become common enough to justify the extra draft steps. And notice the curve can dip below the dashed 1x line at high γ combined with low α and high c: speculative decoding with a poorly matched draft model is genuinely slower than not using it at all, which is the single most common reason a team enables it, measures no gain, and turns it back off.

One caveat about reading the optimum off this curve: it is idealized. The model prices a draft step at c and verification at 1, but real systems also pay for tree-attention bookkeeping, KV-cache rollback on rejection, and scheduler overhead that grows with γ. That is why production defaults cluster around γ = 3 to 5 even when the formula alone would suggest going higher. Treat the widget as showing the shape of the trade-off, and measure your own stack for the exact setting.

The Draft Model Family Tree

The original formulation used a separate small model from the same family, for example Llama 3.2 1B drafting for Llama 3.3 70B. That works, but it forces you to host two models and the vocabulary must match exactly. The field has moved steadily toward drafts that live inside the target model.

Knowledge distillation shows up repeatedly here: the best-performing draft heads are trained to match the target’s output distribution rather than to be good language models in their own right, because acceptance rate, not draft quality, is the objective.

# Scenario: comparing two serving configs for the same 70B target on an
# internal benchmark, to decide which draft strategy to ship.
configs = {
    # Separate draft model: easy to set up, a second checkpoint in VRAM.
    "two_model": {"method": "draft_model", "model": "meta-llama/Llama-3.2-1B-Instruct",
                  "num_speculative_tokens": 4},
    # EAGLE-3 head: trained against this exact target, much higher acceptance.
    "eagle3":    {"method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.3-Instruct-70B",
                  "num_speculative_tokens": 5},
    # N-gram lookahead: no training, useful when output repeats the prompt
    # heavily (structured JSON, code edits, retrieval-grounded answers).
    "ngram":     {"method": "ngram", "prompt_lookup_max": 4,
                  "num_speculative_tokens": 3},
}

Where It Helps and Where It Does Not

Speculative decoding converts spare compute into lower latency. That framing predicts exactly when it works.

• Low batch size wins big. At batch 1 to 8 the GPU is idle enough that free compute is genuinely free, and reported speedups of 2x to 4x are real.
• High batch size erodes the gain. Large batches already saturate the tensor cores by amortizing weight reads across many sequences. Speculation now competes for the same compute, and past a crossover point (often somewhere between batch 32 and 64, depending on hardware and model) it can reduce total throughput even as it keeps per-request latency low. Modern servers handle this with dynamic speculation, disabling or shortening the draft as load rises.
• Predictable text raises acceptance. Code completion, structured JSON, and retrieval-grounded answers that quote source text all draft well. Open-ended creative writing at high temperature drafts poorly.
• Long reasoning chains are a strong fit. Models doing extended test-time compute emit thousands of intermediate tokens, most of them formulaic, and those are precisely the tokens a draft model predicts easily.
• It is orthogonal to other optimizations. It composes with FlashAttention, grouped-query attention, and quantization schemes such as TurboQuant, because it attacks step count rather than step cost.

One subtlety that trips people up: the lossless guarantee holds for the sampling distribution, not bit-for-bit output. Different kernel paths and floating-point reduction orders mean a speculative run and a non-speculative run with the same seed can diverge. The distributions are identical, individual sampled strings need not be.

What’s New (2025-2026)

• EAGLE-3 became the production default. After landing in vLLM, SGLang, and TensorRT-LLM, EAGLE-3’s combination of direct token prediction and multi-layer feature fusion displaced Medusa and vanilla two-model drafting for most new deployments. SGLang has reported roughly 3x to 3.4x decode speedup on Llama-3.3-70B at batch 1.
• Pretrained multi-token prediction heads. DeepSeek-V3 popularized baking MTP heads into pretraining rather than fitting a draft head afterward, and heads trained jointly with the target reach meaningfully higher acceptance than post-hoc ones. Several 2025-2026 open-weight releases now ship MTP or EAGLE-3 draft checkpoints alongside the main weights, treating “speculation support” as part of the release rather than a community add-on.
• Load-adaptive speculation. Serving stacks stopped treating γ as a static config value. Schedulers now shrink or disable drafting as batch size climbs, and some track a running acceptance estimate per request to set draft length dynamically.
• Agent workloads pulled it forward. Long-horizon agent loops make many sequential model calls, so per-token latency compounds across an entire trajectory. Reported production deployments in commerce and support agents have used EAGLE-3 with fine-tuned drafts specifically to cut end-to-end agent turn time, not just single-response latency.
• Speculation beyond tokens. The same predict-then-verify pattern has spread to adjacent layers of the stack: prefetching KV-cache entries from disaggregated memory, and speculating tool-call arguments in agent frameworks. The token-level version remains the mature, well-understood case.

Practical Guidance

The lasting lesson of speculative decoding is architectural rather than algorithmic. Rejection sampling is an old idea, and drafting with a cheap model is an obvious one. What made the technique matter was noticing that autoregressive decoding leaves modern accelerators mostly idle, and that the idle capacity can be spent on guesses which are then verified rather than trusted. Guessing plus verification is now one of the standard shapes for making large models faster, and it is showing up well beyond the decode loop it started in.

from vllm import LLM, SamplingParams

# Scenario: a customer-support chat backend serving a 70B model at
# batch size 1-4, where every user is watching tokens stream in and
# per-token latency is the thing they actually feel.
llm = LLM(
    model="meta-llama/Llama-3.3-70B-Instruct",
    speculative_config={
        "method": "eagle3",
        # A trained EAGLE-3 draft head for this exact target model.
        # The draft must share the target's tokenizer.
        "model": "yuhuili/EAGLE3-LLaMA3.3-Instruct-70B",
        # num_speculative_tokens is gamma: how far ahead to draft
        # before each verification pass. 3-5 is the usual sweet spot.
        "num_speculative_tokens": 5,
    },
    tensor_parallel_size=4,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
out = llm.generate("Why was my order refunded twice?", params)
print(out[0].outputs[0].text)

# The text above is drawn from exactly the same distribution as it
# would be with speculation disabled. Only the wall-clock time changes.