Chunking Strategies That Survive Production

Here’s an uncomfortable experiment that has embarrassed a lot of expensive systems, and one worth running yourself on your own corpus before trusting anyone’s advice on this topic, including this lesson’s: hold the embedding model constant and vary only the chunking strategy; then hold chunking constant and vary the embedding model. On most enterprise corpora, the chunking change moves retrieval metrics more than the embedding change does, often substantially more. The industry spent 2023 arguing about vector databases and 2024 arguing about embedding models, and quietly, the teams that actually won on retrieval quality were the ones that got chunking right first and treated everything else as secondary tuning.

Why chunking dominates

A chunk is the unit of everything downstream in this course: it’s what gets embedded, what gets retrieved, what gets reranked, and what the model ultimately reads when generating an answer. Every chunking mistake propagates through the entire rest of the stack, and no amount of retrieval or reranking sophistication downstream can fully undo a bad chunk boundary:

• Split a policy mid-sentence and neither resulting half embeds to anything close to what a real query about that policy looks like, so both halves become effectively unfindable.
• Merge three unrelated FAQ answers into one oversized chunk and its embedding becomes an average of three different topics, matching none of them particularly well and diluting the specificity that made each answer individually findable.
• Orphan a pronoun from its subject, as in “it increases to 45 days after the first year,” and the chunk becomes unfindable for exactly the queries it should be winning, because the words a searcher would actually type never appear in the chunk’s own text.

The goal, stated once here because it’s worth internalizing before writing any chunking code: each chunk should be a self-contained statement of one thing, sized so its embedding is specific and its text is complete. Every technique in this lesson is a different tactic toward that one goal.

Structure-aware splitting is the baseline

The 2023 tutorial approach split documents on raw character counts, ignoring the document’s own organization entirely. The 2026 baseline splits on the document’s own structure instead, which you preserved carefully during parsing in Module 2 precisely for this moment, and throwing that structure away now would waste the work Lesson 4 put into extracting it:

# Structure-aware chunker over the ParsedDoc block model.
# Strategy: walk blocks in order, start a new chunk at every
# heading boundary, pack blocks until the size budget is hit,
# and NEVER split tables or code blocks mid-object.

MAX_TOKENS = 600   # budget per chunk (see sizing table below)

def chunk(doc: ParsedDoc) -> list[Chunk]:
    chunks, current, path = [], [], []  # path = live heading trail

    for block in doc.blocks:
        if block.type == "heading":
            flush(chunks, current, path, doc)      # close open chunk
            path = path[: block.level - 1] + [block.text]
            continue

        # Atomic blocks ride whole, even if oversized.
        if block.type in ("table", "code"):
            flush(chunks, current, path, doc)
            chunks.append(make_chunk([block], path, doc))
            continue

        if tokens(current) + tokens([block]) > MAX_TOKENS:
            flush(chunks, current, path, doc)
        current.append(block)

    flush(chunks, current, path, doc)
    return chunks

def make_chunk(blocks, path, doc) -> Chunk:
    # The heading path travels WITH the text. "Security >
    # Data Retention > Backups" disambiguates the chunk for
    # the embedding model and becomes filterable metadata.
    header = " > ".join(path)
    body = "\n\n".join(b.text for b in blocks)
    return Chunk(
        text=f"[{doc.title} | {header}]\n{body}",
        doc_id=doc.doc_id, section=header,
        page=blocks[0].page, acl_tags=doc.acl_tags,
    )

def flush(chunks, current, path, doc) -> None:
    """Close out the in-progress chunk, if any content has
    accumulated, and reset the working buffer."""
    if current:
        chunks.append(make_chunk(current, path, doc))
        current.clear()

Two details in that code carry most of the value, and both are easy to skip if you’re moving fast, which is exactly why they’re worth calling out explicitly. First, headings are treated as split points, so chunks naturally align with the author’s own topic boundaries rather than an arbitrary character count that has no relationship to the document’s actual organization. Second, the title-plus-section prefix rides inside the embedded text itself, not just as separate metadata, which single-handedly rescues a large share of chunks full of pronouns and unexplained jargon, because the prefix supplies exactly the context a bare fragment was missing.

Sizing: the sweet spot by content type

There’s no single universal number that works for every corpus, but there are defensible defaults, converged on across many published evaluations and a considerable amount of production scar tissue accumulated by teams who tried both extremes and measured the results:

Under roughly 200 tokens, chunks tend to lose the surrounding context that would have made them answerable on their own, even when the core fact they contain is correct. Over roughly 1,000 tokens, embeddings start to blur into an average across multiple sub-topics and retrieval precision measurably drops, because the vector no longer points sharply at any one specific thing. When you’re genuinely unsure and need a starting point to test against your own eval set, 512 tokens remains a boring, well-tested, defensible default for general prose.

Overlap, semantic chunking, and other folklore

Overlap exists as insurance against bad split points, nothing more. If your split points are already structural, meaning they land on actual section and paragraph boundaries rather than mid-sentence character cuts, you need comparatively little insurance: 0 to 15 percent overlap covers the realistic edge cases. The widespread folklore recommending 50 percent overlap dates from the character-count-splitting era, where every single boundary was effectively a coin flip that might land mid-word or mid-idea, and heavy overlap was the only defense available against that randomness.

Semantic chunking, splitting at points where consecutive-sentence embedding similarity dips below a threshold, is appealing in theory and genuinely mixed in practice. It costs an extra embedding pass at ingestion time just to find the split points, it produces unpredictable and sometimes awkward chunk sizes, and on well-structured enterprise documents it mostly ends up rediscovering the same headings you already had available for free from parsing. It earns its keep specifically on unstructured walls of text, such as raw meeting transcripts or exported email threads, where genuinely no author-provided structure survives the parsing step for a structure-aware chunker to lean on.

Page-based chunking is worth knowing about for one specific, high-value reason: citations. When users need to see “page 12” to actually trust an answer enough to act on it, particularly in legal or compliance contexts, chunk-per-page (or chunk-with-page-spans, which the block model from Lesson 4 already supports) keeps provenance exact and directly verifiable against the source document.

The honest summary, worth carrying forward into the next lesson: structure-aware chunking with sane, content-type-appropriate sizes gets you roughly 90 percent of what’s achievable through chunking alone. The last 10 percent comes not from a better splitting algorithm, but from enriching each chunk with context about where it lives within the larger document, and that technique, contextual retrieval, along with its cousin late chunking, changed enough between 2024 and 2026 to deserve a full lesson of its own. That’s next.