ColBERT (Late Interaction Retrieval)

ColBERT is the retrieval architecture that introduced late interaction: instead of compressing a query or document into a single vector, it keeps one vector per token from each side, and defers all comparison between them until search time, using a lightweight operator called MaxSim. Introduced by Stanford’s Omar Khattab and Matei Zaharia in 2020, it sits between the two dominant alternatives (bi-encoders and cross-encoders), and has become the default reranking and multi-vector retrieval layer inside open-source RAG and search pipelines through 2025 and 2026.

Three Architectures, Three Trade-offs

Bi-encoders pool every token into a single vector per side and compare with one dot product, fast enough to search millions of documents, but blind to which specific words aligned. Cross-encoders concatenate query and document and run one transformer over the pair, highly accurate, but must be re-run for every candidate at query time, which doesn’t scale past a few hundred candidates. Late interaction keeps independent, precomputed token vectors and defers comparison to a cheap operator run only at query time, getting most of a cross-encoder’s precision at close to a bi-encoder’s cost.

The MaxSim Operator

For every query token, MaxSim takes its maximum cosine similarity against every document token vector, then sums those maximums across all query tokens. That sum is the entire ColBERT relevance score: no learned layer sits on top of it. Below, four query tokens are compared against every token of a candidate document; click a cell to inspect its similarity, then reveal which one wins per row.

Multi-Vector Storage: The Cost of Precision

Where a bi-encoder stores one 768-dimensional vector per passage, ColBERT stores one small vector (typically 128 dimensions) per token, meaning dozens to hundreds of vectors per document. That’s the price of keeping token-level detail, and it’s the entire reason compression schemes like PLAID’s residual quantization exist: without them, a multi-vector index for a large collection would be too large to serve economically.

Indexing Offline, Scoring Online

Late interaction splits neatly into a heavy step done once per collection and a light step done per query. Document token vectors are computed and compressed offline; at query time, the much shorter query is encoded once, a small set of candidates is shortlisted by comparing query tokens against precomputed centroids rather than every stored token, and only that shortlist is scored with exact MaxSim.

flowchart TB
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    subgraph IDX["Indexing: offline, once per collection"]
        A([Documents]):::data --> B(Token Encoder):::process
        B --> C(Per-token Embeddings, 128d):::process
        C --> D(Centroid Clustering):::process
        D --> E(2-bit Residual Compression):::process
        E --> F([PLAID Index]):::output
    end
    subgraph QRY["Query: online, per request"]
        G([Query]):::data --> H(Token Encoder):::process
        H --> I(Query Token Embeddings):::process
        I --> J{Candidate Generation}:::process
        J --> K(Exact MaxSim Scoring):::process
        K --> L([Ranked Results]):::output
    end
    F -.->|reads index| J
    F -.->|reads index| K

ColBERTv2 and the PLAID Engine

The original ColBERT stored full-precision token vectors, which made large-scale indexes prohibitively large. ColBERTv2 (2021) fixed this with denoised supervision (distilling relevance signal from a cross-encoder teacher) and residual compression: each token vector is stored as its nearest centroid id plus a small quantized residual, roughly 1–2 bits per dimension, cutting index size 6–10× with little quality loss. The PLAID engine built on top clusters document vectors into centroids offline, then prunes almost all candidates at query time using fast centroid-level comparisons before running exact MaxSim only on the survivors, the step that made late interaction fast enough for production-scale collections.

What’s New in Late Interaction (2025–2026)

• PyLate, a Sentence-Transformers-based library from LightOn, has become the standard way to train and serve ColBERT-family models, paired with a Rust fast-plaid engine and a WASM pylate-rs runtime for edge inference, succeeding the earlier RAGatouille tooling that first made ColBERT easy to fine-tune.
• GTE-ModernColBERT, trained on PyLate using the ModernBERT backbone (long context, FlashAttention, rotary position embeddings), is currently among the strongest open multi-vector retrievers on out-of-domain benchmarks.
• ColPali and ColQwen2 extend late interaction to documents-as-images: a vision-language encoder produces one vector per image patch instead of per text token, so a PDF page with charts and layout can be retrieved directly via MaxSim, no OCR pipeline required.
• ColBERT-Att explicitly integrates the late-interaction mechanism with attention weights for improved accuracy on out-of-domain retrieval tasks.
• The first dedicated Late Interaction and Multi-Vector Retrieval workshop convenes at ECIR 2026, and multi-vector search tooling is starting to target code-search use cases for coding agents, not just document retrieval.

Where ColBERT Fits in a Retrieval Stack

Because MaxSim is both accurate and cheap relative to a cross-encoder, late interaction is most often deployed as either a first-stage retriever over a PLAID index, or a lightweight reranker sitting between a fast ANN search pass and the LLM, narrowing millions of candidates down to a shortlist with a bi-encoder or ANN index, then reordering that shortlist with token-level precision before it reaches generation.

from pylate import indexes, models, retrieve

model = models.ColBERT(model_name_or_path="lightonai/GTE-ModernColBERT-v1")

# Offline: encode each document once, build a compressed PLAID index
index = indexes.PLAID(index_folder="pylate-index", index_name="docs", override=True)
doc_embeddings = model.encode(documents, is_query=False)
index.add_documents(documents_ids=doc_ids, documents_embeddings=doc_embeddings)

# Online: encode the query, score candidates with MaxSim against the index
retriever = retrieve.ColBERT(index=index)
query_embeddings = model.encode([query], is_query=True)
results = retriever.retrieve(queries_embeddings=query_embeddings, k=10)