Context Rot

Context rot is the tendency for a large language model’s accuracy and reliability on a task to decline as the amount of input in its context window grows, even when the specific fact or instruction the model needs is present somewhere in that input and even far short of the model’s stated maximum context length. The term was coined and quantified by Chroma Research’s July 2025 report, which tested 18 frontier models, including Claude, GPT, Gemini, and Qwen3 variants, and found the same pattern across all of them: a 200K-token window can show real accuracy loss by 50K tokens, and a 1M-token window does not reliably reason across anything close to 1M tokens of actual content. Marketing copy quotes a context window’s maximum size; context rot is the reminder that the usable, reliable size is usually much smaller, and shrinks further depending on what’s in that context, not just how much of it there is.

Why This Isn’t the Same as Running Out of Context

It’s tempting to file context rot under “the context window was too small,” but that’s precisely the failure mode it isn’t. A model running out of context throws a clear, loud error, or truncates and drops content, either way, the failure is visible. Context rot happens with every token the model needs still inside the window, no truncation, no error, and the model’s own confidence in its (increasingly wrong) answer doesn’t reliably drop to warn you. It fails quietly, which is exactly why it stayed unmeasured for as long as it did: nothing in a typical eval harness distinguishes “the model got it right because the task was easy” from “the model got it right despite 80K tokens of surrounding text,” until someone runs the same task at multiple context lengths and watches accuracy fall.

What Chroma’s Study Actually Measured

The report ran three task families designed to isolate context length as the only variable, holding task difficulty constant so any accuracy change could only be attributed to the amount of surrounding context:

• Needle-in-a-haystack, semantic variant: instead of a needle the model can find by exact string match, questions require semantic matching between the question and the answer buried in the haystack. Performance declined faster when needle-question semantic similarity was lower, meaning the model had to work harder to recognize the relevant sentence among the surrounding text, and that extra difficulty compounded with length.
• LongMemEval: a conversational question-answering benchmark. Every model scored dramatically higher on a focused ~300-token prompt containing just the relevant turn than on the full 113K-token conversation history containing that same turn, with Claude Opus 4 and Sonnet 4 showing the widest gaps, most often by abstaining rather than answering wrong.
• Repeated Words: a minimal task, exactly reproduce a sequence of words. Even here, with nothing to “understand,” accuracy fell as sequence length grew, and models increasingly under-generated, generated random substitutions, or in GPT-3.5 Turbo’s case refused the task outright over 60% of the time at long lengths.

Distractors Make It Worse, Non-Uniformly

Adding even one distractor, a plausible-looking but irrelevant passage near the needle, measurably reduced accuracy versus a clean haystack. Four distractors compounded the effect further, but not in a smooth, predictable way: which specific distractor caused the most damage varied by combination of haystack source and distractor content, meaning the failure isn’t just “more noise is bad” but something closer to specific, hard-to-predict interference between passages competing for the model’s attention.

# Scenario: stress-testing a customer-support RAG system with irrelevant
# but plausible-looking passages mixed into the retrieved context
def inject_distractors(needle: str, distractors: list[str], filler: list[str], n: int):
    chosen = distractors[:n]                 # e.g. 0, 1, or 4 distractors
    pool = filler + chosen
    random.shuffle(pool)
    insert_at = random.randrange(len(pool))
    pool.insert(insert_at, needle)
    return "\n\n".join(pool)

# Accuracy with 0 distractors vs 4 distractors at the same context length
# reveals how much of a system's real-world degradation is driven by
# irrelevant retrieved chunks rather than length alone.

The Counterintuitive Finding: Coherence Hurts

The single most surprising result in the study inverts a common assumption: models perform worse, not better, when the haystack preserves a logical, coherent flow of ideas, and shuffling the haystack into incoherent order, destroying that narrative structure, consistently improved accuracy. The likely mechanism is that a coherent surrounding narrative behaves like a stronger distractor: nearby sentences that build logically toward something plausible pull the model’s attention along with them, competing more effectively for the model’s limited “budget” of relevant-passage recognition than a jumble of unrelated sentences would. A shuffled haystack has no narrative gravity to compete with the needle; a coherent one does.

Interactive: Simulated Accuracy vs. Context Length

The widget below reproduces the qualitative shape of the Chroma findings: accuracy declines as context length grows, distractors accelerate the decline, and shuffling the haystack (removing coherence) recovers some of that lost accuracy. Drag the length slider, add distractors, and toggle structure to see how the three factors interact:

Push the length slider to 150K+ with two distractors and a coherent (unshuffled) haystack, then flip to shuffled at the same settings: the recovered accuracy in the shuffled case is the exact effect Chroma’s report highlighted as counterintuitive, since the naive intuition is that removing structure should make a task harder, not easier.

How This Differs From “Lost in the Middle”

Context rot is often discussed alongside, and sometimes conflated with, the earlier and narrower “lost in the middle” finding (Liu et al., 2023): that models retrieve information best when it sits at the very start or very end of the context, and worst when it’s buried in the middle, producing a U-shaped accuracy curve by position. That’s a real, well-replicated effect, but it’s about where in the context the answer sits. Context rot is broader: it shows degradation driven by how much context there is, independent of where the needle sits, plus interaction effects from distractors and coherence that positional bias alone doesn’t explain. A model can suffer both at once: worse at long lengths generally, and worse still if the answer happens to land in the middle of that long context.

Mitigation: Keep the Effective Context Small

Because context rot scales with the volume of tokens the model actually has to reason across, not the size of the window it’s technically capable of holding, the practical fix in production systems is architectural: don’t hand the model everything, hand it the smallest sufficient slice.

flowchart LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef bad     fill:#EEF0F7,stroke:#94A3B8,stroke-width:2px,color:#475569,rx:8px,ry:8px;

    Q([Query]) --> A["Anti-pattern:<br/>dump entire corpus into context"]:::bad
    A --> M1["Model reasons over<br/>100K+ tokens, most irrelevant"]:::bad
    M1 --> R1["Accuracy erodes with length<br/>and distractor density"]:::bad

    Q --> B["Retrieve + rerank<br/>top-k relevant chunks"]:::process
    B --> M2["Model reasons over<br/>a few thousand relevant tokens"]:::data
    M2 --> R2["Effective context stays short;<br/>rot pressure stays low"]:::process

# Scenario: replacing "paste the whole knowledge base into the prompt"
# with a retrieval step that keeps the model's effective context small,
# even though the underlying corpus is far larger than any context window
def answer_with_bounded_context(question, retriever, reranker, model, top_k=8):
    candidates = retriever.search(question, k=50)      # cheap, wide net
    ranked = reranker.rerank(question, candidates)      # precise, narrow
    context = "\n\n".join(c.text for c in ranked[:top_k])
    # A few thousand relevant tokens, not the full corpus, is the point:
    # smaller, denser context resists context rot far better than "just
    # make the window bigger" resists it.
    return model.generate(f"{context}\n\nQuestion: {question}")

Other mitigations reported across the 2025-2026 literature follow the same principle from different angles: periodic summarization or compaction of long agent transcripts so old, low-value turns don’t linger as accumulating distractors; splitting a task across isolated sub-agents so no single model call ever has to reason over the full combined history (see Subagent Orchestration); and treating a model’s advertised context window as a ceiling, never a target, budgeting real production prompts to a fraction of it.

What’s New (2025-2026)

• Follow-up diagnostics. Research such as “Diagnosing and Mitigating Context Rot in Long-horizon Search” has extended the original finding from single-turn QA into long, multi-step agentic search trajectories, where context accumulates turn over turn and the rot pressure compounds across an entire session rather than a single prompt.
• “Safe context budget” as a planning term. Teams building on 1M+ token windows in 2025-2026 increasingly report a practical safe-context budget of roughly 150K-400K tokens for high-accuracy workloads, well under the marketed maximum, and treat that budget as a hard design constraint rather than an incidental observation.
• Agent harness awareness. Agent harness design (see Agent Harness) has started explicitly accounting for context rot in its context-management responsibilities: compaction, summarization, and sub-agent isolation are increasingly framed as rot-mitigation, not just cost-saving, features.
• Model-specific behavior differences persist. Chroma’s data showed Claude models trending toward cautious abstention under long-context ambiguity while GPT models trended toward higher hallucination rates under the same conditions, a difference worth accounting for when choosing which model sits at the end of a long, distractor-heavy pipeline.

Practical Guidance

Context rot reframes what a context window actually promises: capacity to hold tokens, not a guarantee of reliable reasoning across all of them. The systems that hold up under real production load are the ones that treat “how much context” as a variable to budget and measure, not a number to maximize.

# Scenario: before shipping a RAG pipeline that stuffs retrieved
# chunks into a 100K-token prompt, check whether accuracy actually
# holds up at that length, rather than trusting the model's max
# context window as a proxy for reliable context.
import random

def build_haystack(needle: str, filler_docs: list[str], target_tokens: int) -> str:
    docs = filler_docs.copy()
    random.shuffle(docs)                       # shuffled beats coherent, see below
    insert_at = random.randrange(len(docs))
    docs.insert(insert_at, needle)
    haystack = "\n\n".join(docs)
    return haystack[: target_tokens * 4]        # ~4 chars/token, rough cut

def eval_at_length(model, needle, question, filler_docs, token_budget):
    prompt = build_haystack(needle, filler_docs, token_budget)
    answer = model.generate(f"{prompt}\n\nQuestion: {question}")
    return needle_fact in answer                # simple substring check

results = {}
for budget in [2_000, 8_000, 32_000, 64_000, 128_000]:
    hits = [eval_at_length(model, needle, question, filler_docs, budget)
            for _ in range(20)]
    results[budget] = sum(hits) / len(hits)
# Don't assume accuracy at 2K tokens holds at 128K; measure it.