Semantic Chunking

Semantic chunking is a document-splitting strategy for retrieval pipelines that draws chunk boundaries where the meaning of the text actually shifts, rather than every N characters or tokens. Instead of a ruler, it uses an embedding model: every sentence (or small group of sentences) gets embedded, adjacent embeddings are compared for similarity, and a boundary is drawn wherever that similarity drops sharply, on the theory that a sharp drop marks a topic change a fixed-size splitter would otherwise cut straight through. It sits in the same part of a retrieval-augmented generation pipeline as any other text splitter, upstream of embeddings and vector similarity search, but it changes how the chunks it produces are decided rather than what happens to them afterward.

The Problem It’s Solving

Fixed-size and recursive character splitters (LangChain’s CharacterTextSplitter, RecursiveCharacterTextSplitter) cut text at a target character or token count, optionally falling back through a list of separators (paragraph, then sentence, then word) to avoid slicing mid-word. That works fine when a document’s structure roughly lines up with its length, but it has no idea what the text actually says. A 512-token window can easily start halfway through one idea, run through a paragraph break, and end halfway through the next, so a single retrieved chunk mixes two unrelated topics, or a fact and its explanation land in different chunks and neither one alone answers the query. Semantic chunking targets exactly that failure mode: it groups sentences that are about the same thing and cuts between groups that aren’t, so a retrieved chunk is more likely to be a complete, self-contained unit of meaning rather than an arbitrary slice.

The Core Algorithm: Embedding-Similarity Breakpoints

The approach popularized by Greg Kamradt and adopted (with variations) by LangChain’s SemanticChunker and LlamaIndex’s SemanticSplitterNodeParser follows the same basic recipe:

1. Split the source text into sentences. A regex or NLP sentence tokenizer breaks the document into its smallest reasonable units.
2. Group sentences into small windows. Each sentence is usually combined with a buffer of k neighbors on either side before embedding, since embedding a single short sentence in isolation produces a noisy, less meaningful vector than embedding it with a little surrounding context.
3. Embed every window. Each sentence-plus-buffer group gets its own embedding vector, computed with the same embedding model that will later embed the finished chunks and queries.
4. Compute cosine distance between consecutive windows. Walking through the document in order, the algorithm compares each window’s embedding to the next window’s embedding and records how far apart they are.
5. Flag breakpoints where distance spikes. A window pair whose distance clears a threshold is marked as a chunk boundary; everything between two breakpoints becomes one chunk.

# Scenario: manually walking the breakpoint logic on a short technical doc,
# the same core loop SemanticChunker and SemanticSplitterNodeParser run internally
import numpy as np

def cosine_distance(a, b):
    return 1 - np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def find_breakpoints(sentence_embeddings, threshold_percentile=95):
    distances = [
        cosine_distance(sentence_embeddings[i], sentence_embeddings[i + 1])
        for i in range(len(sentence_embeddings) - 1)
    ]
    cutoff = np.percentile(distances, threshold_percentile)
    # A breakpoint after sentence i means a new chunk starts at sentence i+1
    return [i for i, d in enumerate(distances) if d > cutoff]

Choosing the Breakpoint: Threshold Methods

The distance-spike step needs a cutoff, and how that cutoff is chosen is the one genuinely tunable knob in semantic chunking. LangChain’s SemanticChunker exposes four:

• Percentile (the default): compute every consecutive-sentence distance in the document, then flag the top X% as breakpoints. A breakpoint_threshold_amount of 95 means only the sharpest 5% of jumps become boundaries, so a higher percentile produces fewer, larger chunks.
• Standard deviation: flag any distance more than k standard deviations above the document’s mean distance, which adapts automatically to how “jumpy” a particular document already is.
• Interquartile: use the interquartile range of the distance distribution to set the cutoff, which is more robust to a handful of extreme outlier distances than a raw standard-deviation cutoff.
• Gradient: look at the gradient of consecutive distances rather than the raw distances themselves, aimed at documents (legal or highly technical text, in LangChain’s own framing) where topically-similar sentences still sit at a high absolute distance from each other, drowning out the real spikes.

LlamaIndex’s SemanticSplitterNodeParser uses a similar breakpoint_percentile_threshold parameter (95 by default) alongside a buffer_size that controls how many neighboring sentences get folded into each embedded window before the distance comparison runs.

# Scenario: a policy document where paragraph breaks alone are unreliable signals,
# so the splitter needs a percentile threshold instead of a fixed word count
from llama_index.core.node_parser import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding

splitter = SemanticSplitterNodeParser(
    buffer_size=1,                       # fold 1 neighbor sentence in on each side before embedding
    breakpoint_percentile_threshold=90,  # lower than default: more boundaries, smaller chunks
    embed_model=OpenAIEmbedding(),
)
nodes = splitter.get_nodes_from_documents(documents)

The widget below makes this concrete: it plots a synthetic sequence of sentence-to-sentence similarity scores and lets you drag the drop threshold to see exactly which dips get promoted to chunk boundaries.

Semantic Chunking vs. Fixed-Size and Recursive Splitting

graph LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    DOC([Document]):::data --> FIX[Fixed-size split:<br/>every N tokens]:::process --> FCH[Chunks of equal size,<br/>boundaries ignore meaning]:::output
    DOC --> SEM[Sentence-embed +<br/>compare adjacent similarity]:::process --> SCH[Chunks of variable size,<br/>boundaries follow topic shifts]:::output

Recursive splitting (LangChain’s RecursiveCharacterTextSplitter) is a step up from naive fixed-size splitting because it tries paragraph and sentence separators before falling back to a hard character cut, but it’s still driven by a target size, not by meaning. Semantic chunking is the more expensive alternative that removes the size target entirely and lets the content decide where boundaries fall. It solves a related but distinct problem to chunk overlap: overlap patches boundary loss after the fact by duplicating text across a fixed-size cut, while semantic chunking tries to avoid cutting through meaningful content in the first place. The two aren’t mutually exclusive: a semantically-chunked pipeline can still apply a small overlap as a second line of defense against boundaries the similarity signal missed.

Trade-offs: When It Helps and When It’s Overkill

Semantic chunking’s cost is upfront and unavoidable: every sentence (or sentence window) in the corpus has to be embedded once at ingest time purely to decide where to cut, on top of the embedding calls the pipeline would make anyway to index the final chunks. For a million-document corpus that’s a meaningfully larger indexing bill and a slower ingest pipeline than a splitter that just counts characters. It also produces uneven chunk sizes: a document with one long, tightly-focused section and one section that jumps between many short ideas will yield wildly different chunk lengths, which can complicate downstream context-window budgeting if a pipeline assumes roughly uniform chunk sizes.

Independent evaluation has found the payoff is inconsistent rather than automatic. The paper “Is Semantic Chunking Worth the Computational Cost?” (Qu et al.) benchmarked semantic chunking against simple fixed-size splitting across multiple retrieval datasets and found the retrieval-quality gains were often small and did not reliably justify the added compute, particularly against a well-tuned fixed-size baseline with reasonable chunk overlap. In practice, semantic chunking tends to earn its cost on long-form, topically dense documents (research papers, policy manuals, meeting transcripts, contracts with many distinct clauses) where a fixed window is very likely to straddle a real topic boundary, and it tends to be overkill on short, already-structured documents (FAQ entries, product listings, API reference pages) where a paragraph or heading-based split already lines up with meaning at essentially zero extra cost.

What’s New (2025-2026)

• Threshold methods have diversified. Beyond the original percentile cutoff, LangChain’s SemanticChunker now ships standard-deviation, interquartile, and gradient breakpoint types, reflecting community findings that a single global percentile doesn’t generalize well across document types with very different baseline “jumpiness.”
• LLM-based chunk proposal as an alternative to pure embedding distance. Instead of (or in addition to) a cosine-distance walk, some pipelines now prompt an LLM directly to propose chunk boundaries or summarize-then-split, trading a cheaper embedding pass for a more expensive but semantically richer judgment call, particularly for documents where topic shifts are subtle enough that embedding distance alone under- or over-triggers.
• Skepticism has grown alongside adoption. As semantic chunking has become a default option in every major RAG framework, benchmark work like the “Is Semantic Chunking Worth the Computational Cost?” study has pushed teams to treat it as one configuration to A/B test against a well-tuned fixed-size-plus-overlap baseline, rather than an automatic upgrade, especially now that larger context windows and better rerankers reduce how much a single chunk’s boundary quality matters downstream.

Choosing a Strategy

Semantic chunking is not a strict upgrade over fixed-size splitting; it is a different bet, spending embedding compute at ingest time in exchange for chunk boundaries that are more likely to respect the document’s actual structure. Whether that bet pays off depends heavily on how topically dense the source documents are, which is why the strongest recommendation across both the LangChain and LlamaIndex documentation and independent benchmarks is to measure retrieval quality on the target corpus rather than assume semantic chunking is automatically better.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

# A support runbook where fixed-size splitting cuts a troubleshooting
# procedure in half partway through step 3
text = open("runbook.txt").read()

chunker = SemanticChunker(
    OpenAIEmbeddings(model="text-embedding-3-small"),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,  # only the sharpest 5% of similarity drops become boundaries
)

docs = chunker.create_documents([text])
for d in docs:
    print(len(d.page_content), d.page_content[:80])