Contextual Retrieval & Late Chunking

Lesson 7 ended with a confession worth restating plainly: even perfect structural chunking leaves a residue of failures, chunks that are true, well-formed, and completely unfindable, because the words that would connect them to a query live in a different chunk than the one containing the answer. “The fee is waived for accounts above tier 2.” Which fee? Which product? The author said so two paragraphs earlier, and two paragraphs earlier is, after chunking, a different chunk entirely, with no way for the embedding model to know the two are connected.

Two techniques attack this directly, from opposite directions. Both went from research papers to production defaults between late 2024 and 2026, and understanding both, along with the honest tradeoffs between them, is what separates a chunking pipeline that mostly works from one that reliably works.

Contextual retrieval: explain the chunk before embedding it

Anthropic’s contextual retrieval, published September 2024, is almost embarrassingly direct as an idea: for every chunk, ask an LLM to write a short sentence situating that chunk within its parent document, prepend that sentence to the chunk’s text, and then embed the combined result. The unfindable chunk from above becomes something like:

This chunk is from the Premium Checking fee schedule and describes the monthly maintenance fee waiver. The fee is waived for accounts above tier 2.

Now a search for “monthly maintenance fee waiver premium checking” retrieves it easily, because those exact words now live inside the chunk’s own embedded text rather than sitting two paragraphs away in a different chunk. In Anthropic’s original evaluation, contextual embeddings alone cut retrieval failures by 35 percent; combined with contextual BM25 (running the same enriched text through the lexical index too) and reranking (which Module 4 covers), the combination cut failures by up to 67 percent. Two years on, those numbers have held up well enough across independent replications that “contextual chunks” is simply how careful teams now ingest ambiguous, pronoun-heavy corpora, rather than being treated as an exotic optimization reserved for special cases.

# Contextual enrichment pass. The trick that makes it cheap:
# the full document is a CACHED prefix, reused across all of
# that document's chunks. Chunks 2..N pay ~10% for it.

SITUATE = """Here is a chunk from the document above:
<chunk>
{chunk_text}
</chunk>
Write 1-2 sentences situating this chunk within the document
(what product/section/topic it belongs to), to improve search
retrieval. Answer with only those sentences."""

async def contextualize(doc_text: str, chunks: list[Chunk]):
    for c in chunks:
        resp = await client.messages.create(
            model="claude-haiku-4-5",   # small model, simple task
            max_tokens=150,
            system=[{
                "type": "text",
                "text": f"<document>\n{doc_text}\n</document>",
                # The whole document is marked cacheable: the
                # first chunk pays full price to write the cache,
                # every later chunk reads it at ~10% of the rate.
                "cache_control": {"type": "ephemeral"},
            }],
            messages=[{"role": "user",
                       "content": SITUATE.format(chunk_text=c.text)}],
        )
        c.text = resp.content[0].text.strip() + "\n" + c.text

Run this through the batch API, since ingestion, as established in earlier lessons, is never latency-sensitive and batch pricing is meaningfully cheaper. The one-time cost lands in the low single dollars per million document tokens once caching is factored in. For a genuinely large 500-million-token corpus, that’s a few thousand dollars, spent once, to permanently lift the floor of every future query against that corpus, which is a favorable trade against the alternative of chasing individual context-loss failures one support ticket at a time forever. One detail worth deliberately not skipping: the same enriched text should also feed your BM25 lexical index, not just the vector index, because the situating sentences add exactly the vocabulary, product names, section topics, that lexical matching depends on to find rare-term queries. Module 4 covers hybrid search in depth, and this enrichment is where a meaningful share of its published gains actually come from.

Late chunking: embed the document, then cut

Late chunking, introduced by Jina AI in 2024, solves the same underlying problem inside the embedding model itself, with zero LLM calls required and therefore a different cost profile entirely from contextual retrieval. Normal pipelines chunk a document first and then embed each resulting piece in isolation, one at a time, with no visibility into the rest of the document. Late chunking flips that order around:

1. Feed the whole document (up to the embedding model’s context limit, typically 8K to 32K tokens on modern long-context embedders) through the transformer in a single pass.
2. Every token’s output vector now carries context from the entire document, because self-attention saw everything in that single pass, not just the tokens near it.
3. Apply your chunk boundaries after encoding is already complete, mean-pooling each chunk’s span of token vectors into that chunk’s final embedding.

The chunk containing “it increases to 45 days” ends up pooling token vectors that already attended to “notice period” from pages earlier in the same document, during that single full-document forward pass. Anaphora gets resolved essentially for free, as a side effect of how attention works, rather than requiring an explicit LLM call to spell out the connection in words.

flowchart TB
    subgraph naive["Naive order"]
        A["Document"] --> B["Chunk"] --> C["Embed each chunk\nin isolation"]
    end
    subgraph late["Late chunking"]
        D["Document"] --> E["Embed ALL tokens in\none contextual pass"] --> F["Pool token vectors\nper chunk span"]
    end
    C --> G["Context-blind vectors"]
    F --> H["Context-aware vectors"]

    style A fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style B fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style C fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style D fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style E fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style F fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style G fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style H fill:#f0fdf9,stroke:#0D9488,color:#0F172A

# Late chunking with a long-context embedding model. The chunk
# boundaries from Lesson 7's structure-aware chunker are reused
# here unchanged; only the ORDER of chunking vs embedding flips.

async def late_chunk_embed(doc: ParsedDoc, chunk_spans: list[tuple[int, int]]):
    full_text = "\n\n".join(b.text for b in doc.blocks)

    # One forward pass over the WHOLE document, returning
    # per-token hidden states rather than one pooled vector.
    token_embeddings, token_offsets = await embed_model.encode_tokens(full_text)

    chunk_vectors = []
    for start_char, end_char in chunk_spans:
        # Select the token vectors whose text offsets fall
        # inside this chunk's span, then mean-pool them.
        span_vecs = [
            token_embeddings[i] for i, (s, e) in enumerate(token_offsets)
            if s >= start_char and e <= end_char
        ]
        chunk_vectors.append(mean_pool(span_vecs))
    return chunk_vectors

Choosing between them

They’re not really rivals so much as two different line items in your budget, each with a different shape of cost and a different set of prerequisites. Contextual retrieval costs LLM tokens at ingestion time, works with any embedding model regardless of whether it supports long-context encoding, improves your BM25 lexical index as a direct side effect since the enrichment is real readable text, and the enrichment itself is human-readable, which helps enormously when debugging a specific retrieval failure months later. It shines on corpora where chunks reference their parent document constantly: contracts, policies, technical specifications.

Late chunking costs nothing extra at ingestion beyond the single embedding pass you’d need to run anyway, but it requires an embedding model that explicitly supports the technique, meaning it can return per-token representations rather than only a single pooled vector, plus documents that fit within that embedder’s context window. Jina’s v3 and v4 model line and a growing set of open models support this. It does nothing at all for your lexical index, since there’s no generated text artifact for BM25 to index.

Both together is a legitimate, if more expensive, choice on your highest-value collections. The gains overlap somewhat but don’t fully cancel each other out, since they attack the same underlying problem through genuinely different mechanisms, one generative and one architectural.

And a scope note worth remembering before either technique tempts you into skipping earlier work: neither technique replaces good structural chunking from Lesson 7. They repair the context a chunk lost after the fact; they cannot repair a chunk that was cut mid-table or mid-sentence in the first place, because by the time either technique runs, that damage is already baked into the chunk boundaries. Structure first, context second, and then the resulting vectors are only as good as the embedding model that produces them, which is exactly the next lesson’s problem to solve: choosing embedding models deliberately and living comfortably with the tradeoffs of quantization.