Contextual Retrieval

Contextual Retrieval is a retrieval-augmented generation indexing technique that fixes a specific, well-known weakness of chunk-based RAG: when a long document is split into small pieces for embedding, each piece loses the surrounding context that made it meaningful. A chunk that reads “The company’s revenue grew by 3% over the previous quarter” gives a retriever nothing to work with if the query is “how did ACME Corp perform in Q2 2023,” because the chunk never names the company or the quarter. Contextual Retrieval, introduced by Anthropic in September 2024, addresses this by using a cheap language model to write a one- or two-sentence description of where each chunk sits in its parent document, prepending that description to the chunk, and only then building the vector embedding and the BM25 index. The reported effect is large: a 49 percent reduction in the rate of failed retrievals, rising to 67 percent when a reranking step is added on top.

The Problem: Chunks Lose Their Context

Standard RAG splits source documents into chunks of a few hundred tokens, embeds each chunk into a vector, and also builds a keyword index over them. At query time it retrieves the chunks whose vectors are nearest the query vector, or whose keywords best match, and passes them to the model. This works well until a chunk depends on information that lives elsewhere in the document: an earlier sentence that established the subject, a section heading three paragraphs up, a date in the document’s preamble.

Two things go wrong when that context is stripped:

• Semantic search misses. The embedding of “revenue grew by 3%” lands near other generic financial statements, not near “ACME Corp Q2 2023 performance,” so the nearest-neighbor search never surfaces it.
• Keyword search misses. BM25 matches on exact terms. If the query contains “ACME” and “Q2” but the chunk contains neither, there is no lexical overlap to score.

Larger chunks reduce the problem but create others: they dilute the embedding (one vector now has to represent several ideas), they waste context-window budget at generation time, and they still break at document boundaries. Chunk overlap, covered in chunk overlap, helps at the seams but does nothing for a dependency that spans half the document.

The Fix: Situate Each Chunk Before Indexing

Contextual Retrieval runs an extra step during indexing. For every chunk, it sends the whole document plus that chunk to a small, fast model with a prompt that asks for a locating sentence:

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk.

The model returns something like “This chunk is from ACME Corp’s Q2 2023 earnings report and describes quarter-over-quarter revenue growth.” That 50 to 100 token string is prepended to the original chunk. The combined text is what gets embedded (Contextual Embeddings) and what goes into the keyword index (Contextual BM25). Both retrieval paths now have the company name, the quarter, and the document type to match against, even though none of that was in the raw chunk.

# Scenario: what actually changes in the index. Same chunk, before and after.
raw_chunk = "The company's revenue grew by 3% over the previous quarter."

# Before: this is what gets embedded and BM25-indexed. Ambiguous.
index_entry_before = raw_chunk

# After: a locating sentence from Claude Haiku is prepended first.
situating = "This chunk is from ACME Corp's Q2 2023 SEC filing, in the section on quarterly financial results."
index_entry_after = f"{situating}\n\n{raw_chunk}"
# A query for "ACME Corp Q2 2023 revenue" now matches on both vector and keyword.

Crucially, the situating text is used only for retrieval. Once the chunk is retrieved, you can pass either the contextualized version or the raw chunk to the generation model, depending on whether the extra sentence helps or just costs tokens.

The Numbers

Anthropic reported failure rates measured as the share of queries where the correct chunk was not in the top 20 retrieved.

Two secondary findings are worth carrying forward. Passing the top 20 chunks to the model outperformed passing the top 10 or top 5, so contextualization pairs well with retrieving generously and letting a reranker or the model itself sort it out. And the choice of embedding model mattered: Gemini and Voyage embeddings performed best among those tested, which is a reminder that this technique sits on top of your embedding choice rather than replacing the need to make a good one.

The widget below lets you move the two levers that matter, the pipeline stage and how many chunks you retrieve, and see the failure rate respond.

Try this: set the pipeline to the naive baseline, then step it up one stage at a time. Each stage shortens the highlighted bar. Then drag top-K down to 5 and watch every bar grow, which is why the technique is usually paired with retrieving 20 chunks, not 5.

Cost, and Why Prompt Caching Makes It Viable

The obvious objection is that contextualizing every chunk means an extra model call per chunk, and a large corpus has millions of chunks. The technique became practical because of prompt caching. Within a single document, the document text sent to the contextualizer is identical for every one of its chunks. Marking it as cached means you pay the full input price for it once and roughly a tenth of that for each subsequent chunk in the same document. Anthropic put the resulting cost at about 1.02 US dollars per million document tokens processed, using a small model for the contextualizing pass. For most knowledge bases that is a one-time indexing cost measured in tens of dollars, not a recurring one.

# Scenario: batch-contextualizing a corpus. Group chunks by parent document so
# the cached document prefix is reused across all of a document's chunks before
# the cache entry expires.
from collections import defaultdict

def build_contextual_index(documents: dict[str, str], chunker, embed, bm25):
    by_doc = defaultdict(list)
    for doc_id, text in documents.items():
        for chunk in chunker(text):                 # a few hundred tokens each
            by_doc[doc_id].append(chunk)

    for doc_id, chunks in by_doc.items():
        doc_text = documents[doc_id]
        for chunk in chunks:                         # doc_text prefix stays cached across this loop
            enriched = contextualize(doc_text, chunk)
            embed.add(doc_id, enriched)             # Contextual Embeddings
            bm25.add(doc_id, enriched)              # Contextual BM25

Where It Fits in the Pipeline

Contextual Retrieval is an indexing-time transformation. It composes with, rather than competes against, the other RAG improvements.

flowchart TD
    A["Source documents"] --> B["Chunker<br/>(few hundred tokens each)"]
    B --> C["For each chunk: small model writes<br/>a 50-100 token situating sentence"]
    C --> D["Prepend sentence to chunk"]
    D --> E["Embed enriched chunk<br/>(Contextual Embeddings)"]
    D --> F["Index enriched chunk<br/>(Contextual BM25)"]
    E --> G["Hybrid retrieval:<br/>merge vector + keyword hits"]
    F --> G
    G --> H["Rerank top candidates"]
    H --> I["Top ~20 chunks to the generator"]

    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;
    class A data;
    class B,C,D,E,F,G,H process;
    class I output;

The hybrid search entry covers the vector-plus-keyword merge, and reranking covers the final sort. Contextual Retrieval is upstream of both: it improves the quality of what each of them has to work with. It is also complementary to semantic chunking, which decides where the chunk boundaries fall; contextualization then repairs whatever cross-boundary context those boundaries severed.

What’s New (2025 to 2026)

Contextual Retrieval landed in late 2024 and became a standard part of the RAG toolkit through 2025. The main developments since:

• It is now a built-in in major frameworks. LlamaIndex, LangChain, and several vector database vendors ship a contextual-chunking pipeline out of the box, so it is a configuration choice rather than custom code.
• Cheaper contextualizer models. The pass was designed around a small, fast model. The 2025 to 2026 generation of small models (including distilled and flash-tier variants) made the indexing pass faster and cheaper still, which pushed it into use for larger and more frequently reindexed corpora.
• Convergence with context engineering. The broader discipline of context engineering frames Contextual Retrieval as one instance of a general principle: a retrieved chunk is only as useful as the context that travels with it. The same logic now shows up in agent memory systems and in just-in-time retrieval, where a fetched artifact is annotated with why it was fetched.
• Late-interaction alternatives. Methods like ColBERT attack the same lost-context problem differently, by matching at the token level so that a query term can match a chunk even without a document-level summary. In practice teams often use Contextual Retrieval because it drops into an existing single-vector pipeline with no change to the retrieval engine.

When to Use It, and When Not To

Contextual Retrieval earns its keep when documents are long, internally referential, and queried by attributes that are stated once and then assumed (client names, dates, product versions, case numbers). Legal documents, financial filings, technical manuals, and support-ticket histories are the canonical fits.

It is less useful when chunks are already self-contained (a FAQ where every entry names its own subject), when the corpus is small enough that recall is already near perfect, or when the corpus changes so fast that reindexing cost becomes a real recurring expense rather than a one-time one. As with every RAG change, the technique’s own documentation is emphatic on one point: run a domain-specific evaluation before and after, because the size of the gain depends heavily on how referential your particular documents are.

# Scenario: indexing a 500-page product manual for a support assistant.
# Raw chunks like "It increased by 3%." are useless out of context. This
# pass rewrites each one with a locating sentence before it is embedded.
import anthropic

client = anthropic.Anthropic()

CONTEXT_PROMPT = """<document>
{doc}
</document>
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>
Give a short succinct context to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else."""

def contextualize(doc: str, chunk: str) -> str:
    msg = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=100,
        messages=[{
            "role": "user",
            "content": [
                # The document is identical for every chunk in it, so mark it
                # for caching: you pay full price once, then ~10% per chunk.
                {"type": "text", "text": f"<document>\n{doc}\n</document>",
                 "cache_control": {"type": "ephemeral"}},
                {"type": "text", "text": CONTEXT_PROMPT.format(doc="(above)", chunk=chunk)},
            ],
        }],
    )
    situating = msg.content[0].text.strip()
    return f"{situating}\n\n{chunk}"   # this string is what gets embedded + BM25-indexed