Matryoshka Embeddings

Matryoshka embeddings are vector embeddings trained so that truncating the vector down to a short prefix, the first 64, 128, or 256 numbers out of a full 1536 or 3072, still produces a usable, accurate embedding on its own. The name comes from the Russian nesting dolls: a coarse, low-dimensional representation sits nested inside a finer, high-dimensional one, and cracking the doll open at any point still reveals something complete rather than a broken fragment. This one property collapses a problem every team building semantic search or RAG eventually hits: do you index at high dimensionality for accuracy, or low dimensionality for speed and storage, when different queries and different budgets want different answers? Matryoshka Representation Learning (MRL), introduced by Kusupati et al. in 2022, means the answer is no longer “pick one and re-embed everything if you’re wrong.”

The Problem MRL Solves

A standard embedding model is trained with a single loss applied to the full output vector. Every dimension is optimized jointly, and nothing guarantees that any particular subset of those dimensions means anything on its own. If a team later decides 3072 dimensions is too expensive to store and search across 500 million vectors, the obvious move, just take the first 256 numbers and throw the rest away, produces something close to noise. The model was never told those 256 numbers had to carry meaning by themselves; it only optimized the full vector as one unit.

This forces an uncomfortable choice in production: train and serve one large, accurate, expensive model, or train and serve a separate smaller model for the cheaper tier, doubling the training and maintenance burden, and still leaving nothing in between. MRL’s contribution is a training objective that removes the choice entirely, by making every prefix length independently meaningful in a single training run.

How the Training Objective Works

MRL modifies the loss function, not the model architecture. Instead of computing one contrastive or classification loss on the full-dimensional output, it computes the same loss independently on several nested prefixes of that output, typically a geometric sequence like 32, 64, 128, 256, 512, 1024, and the full dimension, then sums (usually with equal or tunable weights) across all of them before backpropagating.

# Scenario: fine-tuning a sentence embedding model so 8 nested dimensions
# are all independently searchable, not just the full 768-dim output
from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer
from sentence_transformers.losses import MatryoshkaLoss, MultipleNegativesRankingLoss

model = SentenceTransformer("microsoft/mpnet-base")

# The base loss (here, in-batch negatives) is computed at each nested size
base_loss = MultipleNegativesRankingLoss(model)
matryoshka_dims = [768, 512, 256, 128, 64, 32]
train_loss = MatryoshkaLoss(model, base_loss, matryoshka_dims=matryoshka_dims)

trainer = SentenceTransformerTrainer(
    model=model,
    train_dataset=train_dataset,   # (anchor, positive) pairs
    loss=train_loss,
)
trainer.train()
# The resulting model.encode(text)[:128] is now a valid 128-dim embedding
# on its own: no separate 128-dim model, no retraining, no re-indexing.

Because the same underlying network produces every prefix, and the network is pushed to front-load the most discriminative signal into the earliest dimensions, this is sometimes described as coarse-to-fine encoding: dimension 1 through 32 capture the broad topic, and each additional block of dimensions refines that with progressively finer distinctions, the same way the first few digits of a GPS coordinate narrow you to a city and the following digits narrow you to a street. Critically, MRL adds essentially no training or inference cost over a normal model: the forward pass is unchanged, only the loss computation during training gets a few extra terms.

Why Naive Truncation Fails and MRL Doesn’t

The gap between “just slice a normal embedding” and “slice a Matryoshka-trained embedding” is the entire point of the technique, and it’s large enough to see clearly even without exact benchmark numbers: a normal model’s accuracy collapses quickly as you cut dimensions, because nothing during training rewarded the early dimensions for standing alone, while an MRL-trained model’s accuracy degrades gracefully, because every prefix length was directly optimized.

At 768 (no truncation) the two curves meet, since there’s nothing to lose yet. Drag down toward 128 or 64 and the gap opens fast: the Matryoshka curve is still close to full accuracy, while the naive-truncation curve has fallen off a cliff, because a normal model crammed all its useful signal across the full width and gave the first 64 numbers no reason to be self-sufficient. This is the entire commercial argument for MRL in one picture: the same storage and latency budget buys dramatically more retrieval quality when the model was trained to expect truncation.

What Gets Truncated, and How

Truncation itself is nothing more than slicing the vector and, for models trained with cosine-similarity objectives, re-normalizing so the shorter vector still has unit length:

# Scenario: you already have full-dimension embeddings stored and want
# a cheap low-dim index without re-calling the embedding API
import numpy as np

def truncate_and_normalize(embedding: np.ndarray, dims: int) -> np.ndarray:
    truncated = embedding[:dims]
    return truncated / np.linalg.norm(truncated)

full_vec = np.array(stored_embedding)          # e.g. 3072-dim
small_vec = truncate_and_normalize(full_vec, 256)
# small_vec is now a valid 256-dim embedding, comparable by cosine
# similarity to any other 256-dim truncation of the same model's output

Most production APIs skip this manual step and expose it as a first-class parameter: OpenAI’s text-embedding-3-large and text-embedding-3-small accept a dimensions argument that truncates and renormalizes server-side, so the API never even returns the unused tail of the vector.

Nested, Not Sharded

It’s worth being precise about what “nested” means here, since it’s easy to confuse with sharding or splitting a vector into independent chunks. In MRL, dimension 200 is not independent of dimensions 1 through 199, it’s additional refinement layered on top of them, in the same way each additional GPS decimal digit refines rather than replaces the ones before it.

flowchart LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    TXT([Input text]):::data --> ENC[MRL-trained encoder<br/>single forward pass]:::process
    ENC --> V32[dims 1-32<br/>coarse topic]:::output
    ENC --> V128[dims 1-128<br/>+ finer distinctions]:::output
    ENC --> V768[dims 1-768<br/>full precision]:::output
    V32 -.nested inside.-> V128 -.nested inside.-> V768

Where It’s Used in 2025-2026

MRL has moved from a research idea to a default feature of nearly every major embedding API in the space of about two years:

Beyond text search, MRL has been extended well past its original scope: the original paper already demonstrated it on ImageNet-1K image classification, and 2025-2026 follow-up work applies the same nested-loss idea to multimodal visual document retrieval and to speaker embeddings for voice systems, wherever a single model needs to serve both a cheap coarse tier and an expensive precise tier.

The Two-Stage Retrieval Pattern

The most common production use of MRL isn’t picking one dimension and living with it, it’s using several dimensions of the same embedding together, exactly as shown in the frontmatter example above: search a large candidate pool with a short, cheap prefix, then re-score only the survivors with the full vector. Because both vectors come from one model and one API call’s worth of underlying computation, this avoids the two-model, two-pipeline complexity that a bi-encoder-plus-cross-encoder reranking setup usually requires, while still getting most of the accuracy benefit of full precision on the results that matter.

Trade-offs and Where MRL Doesn’t Help

MRL isn’t free of caveats, and the 2025-2026 literature has started pushing back on how far it should be trusted. A 2026 study, “To MRL or Not to MRL,” found that many modern text embedding models are already fairly robust to post-hoc truncation even without explicit Matryoshka training, and that the real, unambiguous MRL advantage shows up specifically in heavy-truncation regimes, cutting a vector down to a small fraction of its original size, rather than moderate truncation. In other words, if a team only ever needs to cut dimensions by half, plain truncation of a strong modern model may already be adequate; MRL earns its keep when the target is 1/8th the size or smaller, which is exactly the regime the interactive widget above is built to show.

The other trade-off is upfront: MRL is a training-time decision. A team cannot bolt MRL onto an already-trained, non-Matryoshka model after the fact and expect the nested-prefix property to appear; the naive-truncation curve in the widget above is what happens when you try. Choosing an MRL-trained model (or fine-tuning one with MatryoshkaLoss) has to happen before the embeddings are generated at scale.

Practical Guidance

Matryoshka embeddings turn a once-and-done architectural decision, how many dimensions to commit to, into a runtime knob that can be tuned per query, per tier of service, or per stage of a retrieval pipeline, without retraining or maintaining a second model.

from openai import OpenAI
import numpy as np

client = OpenAI()

# Scenario: a product catalog with 5M items, indexed once but queried
# thousands of times a second, where full 3072-dim vectors would be
# too slow and too large to hold in memory for every candidate.

def embed(texts, dimensions=None):
    resp = client.embeddings.create(
        model="text-embedding-3-large",
        input=texts,
        dimensions=dimensions,  # None = full 3072 dims, MRL-truncated otherwise
    )
    return np.array([e.embedding for e in resp.data])

# Stage 1: index and search with a short, cheap Matryoshka prefix
catalog_short = embed(catalog_texts, dimensions=256)
query_short = embed([query], dimensions=256)[0]
coarse_scores = catalog_short @ query_short
top_500 = np.argsort(coarse_scores)[::-1][:500]

# Stage 2: re-embed only the survivors at full precision and rerank
candidates_full = embed([catalog_texts[i] for i in top_500])
query_full = embed([query])[0]
fine_scores = candidates_full @ query_full
ranked = [top_500[i] for i in np.argsort(fine_scores)[::-1]]