Cross-Encoder vs. Bi-Encoder

Ask a model to judge whether a document answers a query, and there are two fundamentally different ways to architect that judgment. A bi-encoder runs the query and the document through the model separately, producing two independent vectors that are then compared with a cheap distance metric. A cross-encoder runs the query and the document through the model together, as one concatenated input, so every token of the query can attend to every token of the document (and vice versa) before a single relevance score comes out. That one architectural decision, encode-then-compare versus encode-jointly, is the difference between a system that can search a billion documents in milliseconds and one that produces the most accurate relevance judgment possible on a handful of candidates at a time. Production retrieval systems typically need both.

Bi-Encoders: Encode Separately, Compare Cheaply

A bi-encoder (also called a dual encoder or Siamese network, since the same weights are usually shared between the query-side and document-side towers) maps each input to a fixed-size vector independently. This is precisely what an embedding model does, so “bi-encoder” and “embedding model” describe the same architecture from two different angles: bi-encoder emphasizes the shape of the network, embedding emphasizes the output it produces.

The critical consequence of independence is that document vectors can be computed once, offline, in advance, and stored in a vector index such as HNSW. At query time, only the query needs to be encoded; comparing it against millions or billions of pre-computed document vectors is then a fast nearest-neighbor lookup, not millions of fresh model inferences. Sentence-BERT (SBERT), introduced in 2019, was the paper that made this practical for transformer models: it showed that finding the closest pair among 10,000 sentences dropped from roughly 65 hours of pairwise BERT inference to about 5 seconds using pre-computed sentence embeddings and cosine similarity, with accuracy close to BERT’s pairwise scoring. That speedup is the entire reason bi-encoders, not cross-encoders, sit at the front of every large-scale retrieval pipeline; see Dense Retrieval vs. Sparse Retrieval for how this plays out against classical term-matching retrieval.

The cost of that speed is that the model never sees the query and document together. It has to compress each one into a single vector before knowing what it will be compared against, so fine-grained interactions, exactly how a specific query term relates to a specific document phrase, are lost in that compression.

Cross-Encoders: Encode Jointly, Compare Precisely

A cross-encoder takes the query and document as a single concatenated input ([CLS] query [SEP] document [SEP]), passes it through the transformer together, and outputs one relevance score, typically via a classification or regression head on the final hidden state. Because self-attention operates over the whole sequence, every query token can directly attend to every document token, letting the model reason about their relationship rather than just the geometric distance between two pre-summarized vectors.

This joint attention is what makes cross-encoders substantially more accurate: Passage Re-ranking with BERT (Nogueira & Cho, 2019) was the paper that established this, taking the top MS MARCO passage-ranking leaderboard spot with a 27% relative improvement in MRR@10 simply by fine-tuning BERT to score query-passage pairs jointly. The tradeoff is unavoidable: nothing about a cross-encoder’s output can be pre-computed, since the score depends on the specific query-document pair. Scoring N documents against a query means N full forward passes through the model at query time, which is far too slow to run over an entire corpus, but entirely practical over a shortlist of candidates that a faster method has already narrowed down.

Side by Side

graph TB
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    subgraph BE["Bi-Encoder"]
        Q1([Query]):::data --> E1[Encoder]:::process --> VQ[(Query Vector)]:::data
        D1([Document]):::data --> E2[Encoder]:::process --> VD[(Doc Vector)]:::data
        VQ --> SIM[Cosine Similarity]:::output
        VD --> SIM
    end

    subgraph CE["Cross-Encoder"]
        Q2([Query]):::data --> JOIN["Query + Document<br/>concatenated"]:::process
        D2([Document]):::data --> JOIN
        JOIN --> SCORE[Relevance Score]:::output
    end

Why the Combination, Not the Choice, Is the Point

Because the two architectures fail in opposite directions, retrieve-then-rerank pipelines use both in sequence rather than picking one: a bi-encoder narrows a huge corpus down to a manageable candidate set (optimizing for recall, don’t miss the right answer), and a cross-encoder re-scores only that shortlist (optimizing for precision, put the truly best answer first). This is the same two-stage pattern covered in more depth in the reranking entry, which focuses on that pipeline; this entry focuses on the underlying architectural distinction that makes the pipeline necessary in the first place.

A widely cited 2022 study, “In Defense of Cross-Encoders for Zero-Shot Retrieval”, found that scale matters more for cross-encoders than for bi-encoders when generalizing to new domains: increasing a bi-encoder’s parameter count produced smaller out-of-domain gains than increasing a cross-encoder’s, and a sufficiently large cross-encoder outperformed a strong bi-encoder by more than four points on the BEIR out-of-domain benchmark. The practical takeaway that shaped subsequent reranker design: if a system can only afford to scale up one of the two stages, the cross-encoder stage tends to reward that investment more.

A Middle Ground: Late Interaction

Between the two extremes sits late interaction, pioneered by ColBERT (2020). Instead of collapsing a document into one vector like a bi-encoder, or requiring joint encoding like a cross-encoder, late-interaction models keep one vector per token and compute relevance with a fast “MaxSim” operation between query and document token vectors at query time. This preserves much of a cross-encoder’s fine-grained matching while remaining fast enough that some 2025-era vector databases now index ColBERT-style multi-vectors directly. The reranking entry covers this pattern and its 2025-2026 vector-database support in more detail.

What’s New (2025-2026)

• LLM-as-cross-encoder. Rather than fine-tuning a dedicated BERT-scale cross-encoder, teams increasingly prompt a general-purpose LLM with the query and a candidate document (or a batch of candidates) and ask for a relevance judgment directly. This is architecturally still a cross-encoder, query and document are processed jointly by one model, but it trades a purpose-trained small model for the flexibility of a model already in the pipeline.
• Long-context cross-encoders. Newer rerankers such as BGE-reranker-v2-gemma extend cross-encoder input limits to 8192 tokens, up from the roughly 512-token limits of BERT-era cross-encoders, reducing how often long documents must be truncated or chunked before joint scoring.
• Instruction-aware bi-encoders. 2025-2026 embedding models (Qwen3-Embedding, Gemini Embedding 2) increasingly accept a task instruction alongside the text being embedded, letting one bi-encoder produce different vectors for the same text depending on whether it’s being used for retrieval, classification, or clustering, narrowing part of the accuracy gap that used to favor cross-encoders for task-specific judgments.
• Distilled cross-encoders for latency-sensitive reranking. As agentic pipelines issue retrieval calls repeatedly across multiple reasoning steps, demand has grown for cross-encoders small enough to run in single-digit milliseconds per pair, trading some accuracy from the 2022-era scaling results for latency low enough to sit inside a multi-hop agent loop.

Practical Guidance

Neither architecture is a strict upgrade of the other: a bi-encoder’s independence is precisely what makes web-scale search possible, and a cross-encoder’s joint attention is precisely what makes its judgments more trustworthy. The question worth asking for any retrieval system isn’t “bi-encoder or cross-encoder,” it’s which stage of the pipeline each one belongs in.

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

query = "What causes a 429 error from an API?"
document = "A 429 status code means the client sent too many requests in a given time window; the server is rate-limiting the caller."

# Bi-encoder: encode query and document independently, then compare
bi_encoder = SentenceTransformer("all-MiniLM-L6-v2")
q_vec = bi_encoder.encode(query, normalize_embeddings=True)
d_vec = bi_encoder.encode(document, normalize_embeddings=True)
bi_score = float(np.dot(q_vec, d_vec))  # cosine similarity, since both are normalized

# Cross-encoder: encode the pair jointly, with full cross-attention
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
cross_score = cross_encoder.predict([(query, document)])[0]

print(f"Bi-encoder cosine similarity: {bi_score:.4f}")
print(f"Cross-encoder relevance score: {cross_score:.4f}")

# In production: bi-encoder pre-embeds millions of documents offline for fast
# ANN search, then a cross-encoder re-scores only the top few dozen candidates.