Chain of Draft (CoD)

Chain of Draft (CoD) is a prompting technique from Zoom researchers (Xu, Xie, Zhao, and He), published in February 2025, that sits between terse direct answering and verbose Chain-of-Thought. It keeps the idea that a model should reason in explicit intermediate steps, but instructs it to write each step as a compressed draft of a few words rather than a full sentence or paragraph. The motivation is economic: CoT’s accuracy gains come with a large token bill, since every reasoning token is generated one at a time (driving latency) and billed as output (driving cost). CoD’s claim, backed by the paper’s experiments, is that most of those tokens are packaging rather than reasoning, and that stripping them keeps accuracy while cutting output to a small fraction, reported as low as 7.6% of CoT’s token count on some tasks.

The Prompt

CoD is a single instruction, not a new decoding method or a fine-tune. The paper’s exact wording:

Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most. Return the answer at the end of the response after a separator ####.

That is the whole technique. The “5 words at most” cap is the active ingredient; the #### separator just makes the final answer trivial to parse out of the short trace.

A CoT trace and a CoD trace for the same problem look like this:

# Chain-of-Thought
Roger starts with 5 tennis balls. He buys 2 cans, and each can has 3 tennis
balls, so that is 2 times 3, which equals 6 new tennis balls. Adding the 6
new balls to his original 5 gives 5 + 6 = 11 tennis balls in total.
#### 11

# Chain of Draft
5 balls start.
2 cans x 3 = 6.
5 + 6 = 11.
#### 11

Same arithmetic, same answer, roughly a quarter of the tokens.

Why It Works

CoT improves reasoning because writing intermediate steps gives the model more forward passes to compute over and a scratchpad to store partial results. CoD’s bet is that the natural-language framing of those steps (“so that is”, “which equals”, “adding the … gives”) contributes nothing to either function. Humans solving the same problem on paper jot “2x3=6, +5=11”, not full sentences. CoD asks the model to do the same: keep the computation and the partial results, drop the prose.

The tokens saved are almost all output tokens, which is where CoD’s two wins come from:

• Cost. Output tokens are billed at a higher rate than input tokens on most APIs, and CoD cuts the output stream by roughly 70 to 90% on the paper’s tasks.
• Latency. Output tokens are produced sequentially, so time-to-final-answer scales with how much the model writes before ####. A shorter trace is a faster answer, independent of price.

Input tokens (the prompt, few-shot exemplars) are unchanged, so CoD does nothing for prompt-heavy workloads; its lever is purely the length of what the model generates.

The Numbers

From the paper, using GPT-4o and Claude 3.5 Sonnet with few-shot exemplars:

The sports-understanding row is the headline case: a 92.4% token reduction and a small accuracy gain, because the shorter trace gave the model less room to talk itself out of a correct intuition. GSM8K is the more typical case: a few points of accuracy traded for an ~80% token cut. The abstract’s “as little as 7.6% of the tokens” is the best-case ratio across the suite, not the average.

The Word-Budget Trade-off

The “5 words” cap is a knob. Tighter budgets emit fewer tokens but eventually starve the model of the scratchpad space it needs to hold partial results; looser budgets approach CoT’s token count with diminishing accuracy return. The paper found ~5 words a good operating point for GSM8K-style tasks; the widget models that shape.

When CoD Helps and When It Hurts

CoD is not a free swap for CoT. The paper and follow-up work flag several conditions:

• Few-shot exemplars matter. CoD’s gains are strongest when the prompt includes a few worked examples in draft style. Zero-shot CoD (just the instruction, no examples) is less reliable, and on some tasks the model reverts to verbose reasoning or loses accuracy.
• Small models benefit less. Models below roughly 3B parameters showed smaller token savings and larger accuracy drops under CoD, likely because they lean more on the verbal scaffold to stay on track.
• Working-memory-heavy problems. Tasks that need many partial results held at once (long multi-hop arithmetic, complex proofs) can degrade under a hard word cap, since the draft cannot hold everything the model needs to carry forward.
• Code tasks. A 2025 study applying CoD to software-engineering benchmarks found the concise-reasoning transfer weaker there: code problems often need explicit intermediate state (variable values, edge cases) that a five-word draft cannot express, so token savings came with real accuracy loss.
• Reasoning models with hidden thinking. For models that already do internal reasoning in a separate hidden channel (o-series, extended-thinking modes), CoD’s instruction applies to the visible output only and has little effect on the hidden reasoning cost.

CoD vs. CoT vs. Other Concise-Reasoning Methods

CoD and token-budget-aware reasoning attack the same problem from different ends: CoD caps the granularity of each step, budget-aware prompting caps the total and lets the model spend it however it likes. They compose.

Implementation Notes

# Few-shot exemplar, CoD style (put 2-4 of these in the prompt)
Q: A store had 24 apples, sold 9, then received 15 more. How many now?
A: 24 - 9 = 15.
   15 + 15 = 30.
   #### 30

# Scenario: guard against a draft trace that came back too terse to trust,
# and retry that one question with full CoT.
def solve_with_fallback(question, client):
    ans = solve(question)                      # CoD attempt (see frontmatter)
    if ans == "" or not any(c.isdigit() for c in ans):
        return solve_cot(question, client)     # spend the tokens only when needed
    return ans

• Keep the #### separator (or a JSON schema) so parsing does not depend on the draft’s format.
• Draft-style few-shot exemplars are doing real work; do not reuse verbose CoT exemplars with the CoD instruction.
• CoD changes output length, not sampling. Temperature, self-consistency voting, and structured output all still apply on top.
• Measure on your own task before rolling out. The accuracy delta ranges from slightly positive to clearly negative depending on how much working memory the task needs.

What’s New (2025-2026)

• Adoption as a default cost lever. CoD spread quickly through cost-sensitive production pipelines (classification, extraction, routing, grading) where CoT-level accuracy was needed but its token bill was not, precisely because it is a one-line prompt change with no retraining.
• RL-guided CoD for code. Later work trained models with reinforcement learning to produce token-efficient draft reasoning specifically for code generation, addressing the weakness plain CoD showed on software tasks.
• Interaction with reasoning models. As extended-thinking models became common, attention shifted to controlling hidden reasoning length (thinking budgets, effort parameters); CoD remains the tool for the visible-output case and for models without a separate thinking channel.
• Concise-reasoning research line. CoD, token-budget-aware reasoning, and related “think less” methods formed a small literature on trading reasoning verbosity for cost, with growing agreement that the right budget is task-dependent and worth tuning rather than fixing at five words everywhere.

Bottom Line

Chain of Draft is a small, high-leverage prompt change: keep the reasoning steps, shrink each to a draft, parse the answer after a separator. On arithmetic and commonsense reasoning it holds most of CoT’s accuracy for a fraction of the output tokens, which is a direct cut to both API cost and latency. It is weaker zero-shot, on small models, on working-memory-heavy problems, and on code, so it belongs in workloads where you can measure the accuracy trade and where the token bill is the actual constraint.

# Scenario: a high-volume math-word-problem grader where CoT reasoning
# is accurate but the per-call token bill and latency are the problem.
from openai import OpenAI
client = OpenAI()

COD_SYSTEM = (
    "Think step by step, but only keep a minimum draft for each thinking "
    "step, with 5 words at most. Return the answer at the end of the "
    "response after a separator ####."
)

def solve(question: str) -> str:
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": COD_SYSTEM},
            {"role": "user", "content": question},
        ],
    )
    text = r.choices[0].message.content
    return text.split("####")[-1].strip() if "####" in text else text.strip()

print(solve("Roger has 5 balls. He buys 2 cans of 3 balls each. How many now?"))
# draft trace is ~4 short lines instead of a paragraph; answer after ####