Everything before this lesson produced enriched, well-bounded chunk text, first through careful parsing, then through structure-aware chunking, then through contextual enrichment or late chunking to fix the context-loss problem. Now that text becomes vectors, and three decisions get made in this lesson that you’ll live with for a long time afterward: which model, how many dimensions, and how much numeric precision. Get them wrong and you’ll either overpay monthly forever, since storage and RAM scale directly with corpus size for as long as the system runs, or face the one migration in this entire stack that touches every single stored byte in the index.
Choosing the model: the criteria that actually differ
Leaderboard score differences between the top embedding models are usually small on general-purpose text benchmarks, and those small differences often shrink further or vanish entirely once measured against your specific corpus rather than a public benchmark, so the real decision criteria that should drive your choice are mostly operational rather than purely about raw quality.
API models, Voyage, OpenAI’s text-embedding-3 line, Cohere Embed, and Gemini embeddings, offer zero infrastructure to manage, strong multilingual coverage out of the box, and simple per-token pricing you can budget against easily. Voyage’s domain-specific variants for code, finance, and law provide a genuine measurable edge on specialized corpora where general-purpose models tend to underperform. The costs on the other side of the ledger: your text transits a third-party vendor for every embedding call, per-token fees recur on every reindex you ever run, and the model can be deprecated on someone else’s schedule rather than yours, which is precisely the forced-migration scenario the quiz above walks through.
Open-weight models, the Qwen3-Embedding family (which topped MTEB’s multilingual leaderboards through much of 2025), BGE-M3, EmbeddingGemma for edge deployment, and Jina v4 for late chunking and multimodal use cases, run entirely inside your own infrastructure, carry no per-token fees, face no forced deprecations on an external timeline, and can be fine-tuned on your own domain data if general-purpose performance falls short. The costs: you now operate GPU inference yourself, and you own its latency characteristics and its scaling behavior under load, which is a real operational responsibility that API models hand off to the vendor.
The practical tiebreaker in most enterprise settings ends up being data residency, meaning can chunk text legally and contractually leave your infrastructure boundary at all, and migration control, meaning who actually decides when you reindex 100 million chunks: you, on your own schedule with proper validation time, or a deprecation email from a vendor with a fixed external deadline attached.
Whatever you choose, stamp embedding_model and embedding_version into every stored chunk’s metadata from day one. Mixed-model vectors sitting in one index is a silent corruption bug that you cannot detect from ranking output alone; results just get quietly, inexplicably worse, and without the version stamp you have no fast way to even confirm that mixing is the cause.
Dimensions: the Matryoshka dial
Modern embedding models are commonly trained with Matryoshka representation learning, MRL for short: the training loss is applied simultaneously to multiple prefix lengths of the output vector, so the first 256 or 512 dimensions of a 1536-dimension vector form a legitimate, independently usable coarse embedding entirely on their own, not just a truncated and degraded version of the full vector. That training choice turns dimensionality from a fixed, take-it-or-leave-it property of the model into a genuine dial you can tune per deployment.
- 3072 dims x float32 = 12 KB per chunk. At 50 million chunks, a realistic large-enterprise corpus size, that’s 600 GB of vectors, and most of that needs to sit in RAM for HNSW graph traversal to stay fast.
- Truncate to 1024 dims: 4 KB per chunk, 200 GB total. Typical measured recall loss from this truncation: low single digits in percentage terms, and often recovered close to fully by the rescoring pattern covered below.
For most enterprise corpora, 768 to 1024 MRL dimensions represents the pragmatic sweet spot between storage cost and retrieval quality. Keep the full-width vector available on cheaper cold storage if you want to preserve the option to rescore or re-cut chunk boundaries later without a full re-embed.
Precision: quantization with a safety net
The second dial available to you is bytes per dimension, and the pattern that makes aggressive precision settings safe in production is always the same shape: oversample with the cheap, low-precision representation, then rescore the survivors with the accurate, full-precision one.
# Two-phase search with binary quantization (Qdrant flavor;
# pgvector and Weaviate have equivalents).
#
# Phase 1: Hamming search over 1-bit vectors held in RAM.
# 32x smaller than float32, and register-speed math.
# Phase 2: rescore the candidate pool with original float
# vectors from disk, restoring ranking precision.
hits = client.query_points(
collection_name="kb_chunks",
query=query_vector,
limit=20, # what we actually want
search_params={
"quantization": {
"rescore": True, # phase 2 on
"oversampling": 3.0, # phase 1 fetches 3x20 = 60 candidates
}
},
)Rules of thumb that have survived real contact with production traffic, rather than just looking good in a benchmark: int8 quantization is nearly free, roughly 4x smaller than float32, with recall loss usually under 1 percent even without rescoring turned on, making it close to a strict upgrade for most deployments. Binary quantization is the aggressive setting, 32x smaller, but it genuinely needs oversampling in the 2x to 4x range plus rescoring to hold up, and it works best paired with higher-dimensional MRL models at 1024 dimensions or more, since more starting dimensions gives the binary encoding more signal to preserve even after crushing each dimension down to a single bit. Usefully, high-dimensional MRL models also tend to quantize better than lower-dimensional ones, so the two dials compound favorably rather than fighting each other: 1024-dimension int8 is a boring, well-tested, excellent default that cuts total storage by roughly 12x compared to naive 3072-dimension float32, for a recall cost that’s very hard to measure against your eval suite’s noise floor.
The migration tax
One more time, stated directly, because it’s the trap this entire lesson has been building toward: vectors are only ever comparable within the single model that produced them. The day you change models, or change dimensions in a way that isn’t a simple MRL truncation, or even change certain quantization settings, every chunk must re-embed and the index must rebuild from scratch. That’s the blue-green pattern established in Lesson 6, gated by the evaluation suite covered fully in Module 8, and it costs real time and real money at corpus scale, often more of both than teams budget for when they make the original model choice.
So when comparing embedding providers up front, price the eventual reindex into your decision, not just the steady-state per-token cost. A model that scores 20 percent better on a benchmark today but gets deprecated by its vendor in 18 months can end up costing more in total, once you account for the forced migration, than the “worse” model you retain full, permanent control over. This is the strongest practical argument for open weights specifically at large corpus sizes, where a reindex is expensive enough to genuinely hurt, and for keeping raw chunk text stored alongside vectors at all times, never just the vectors alone, so that a future re-embed never also requires re-parsing every source document from scratch.
Vectors are now stored, sized deliberately, and quantized with a safety net. Module 4 turns to making them findable at query time, starting with the store that holds them and the HNSW knobs that trade recall against speed under real query load.