Reranking

Retrieving the right documents is not the same as retrieving similar-looking ones. A vector search over embeddings, powered by an index like HNSW, is extremely fast and casts a wide net — but cosine similarity between a query embedding and a document embedding is a coarse, single-pass approximation of true relevance. Reranking is the correction step: a second, more computationally expensive model re-examines a shortlist of candidates and reorders them by how genuinely relevant each one is to the query, before that shortlist is handed to an LLM or shown to a user.

By 2025–2026, reranking has gone from an optional accuracy boost to a near-mandatory stage in production RAG and enterprise search systems — cited repeatedly as the single highest-leverage change teams make when a RAG pipeline’s answers are technically retrieved but subtly wrong.

Why a Second Stage Is Necessary

Embedding-based retrieval (the “bi-encoder” approach) encodes the query and every document independently, then compares them with a simple distance metric. This is what makes it fast enough to search billions of vectors — the documents are pre-embedded once, offline, and only the query needs to be embedded at request time.

But independent encoding has a cost: the model never actually looks at the query and the document together. It can’t reason about how specific terms in the query interact with specific terms in the document. This causes two common failure modes:

• Semantic near-misses — a document that shares topical vocabulary with the query (e.g., mentions “tax,” “state,” and “remote”) but doesn’t actually answer the question ranks highly by cosine similarity, while the actually-correct passage (using different phrasing) ranks lower.
• Keyword-vs-intent mismatches — short or ambiguous queries produce embeddings that don’t discriminate well between subtly different documents, especially in domains with dense, overlapping jargon (legal, medical, financial, technical support).

Reranking models solve this by processing the query and each candidate document jointly, allowing full cross-attention between every query token and every document token — at the cost of being far too slow to run over an entire corpus. The standard solution: use cheap bi-encoder retrieval to narrow millions of documents down to a shortlist (typically 20–200), then apply the expensive, accurate reranker only to that shortlist.

Bi-Encoders vs. Cross-Encoders

graph TB
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef ai fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;
    classDef data fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef result fill:#0D9488,stroke:#0D9488,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    subgraph BE["Bi-Encoder — Stage 1 (retrieval)"]
        Q1([Query]):::data --> EQ(Encoder):::ai --> VQ[Query Vector]:::result
        D1([Document]):::data --> ED(Encoder):::ai --> VD[Doc Vector]:::result
        VQ -.cosine similarity.-> VD
    end

    subgraph CE["Cross-Encoder — Stage 2 (reranking)"]
        Q2([Query]):::data --> JOINT["Query + Document<br/>(concatenated, jointly encoded)"]:::ai
        D2([Document]):::data --> JOINT
        JOINT --> Score[Relevance Score]:::result
    end

The Two-Stage Retrieve-and-Rerank Architecture

flowchart LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef store fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef ai fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;
    classDef result fill:#0D9488,stroke:#0D9488,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    Query([User Query]) --> Embed(Query Embedding):::ai
    Embed --> ANN{HNSW / Vector Search}:::store
    ANN -->|"Top-100 candidates<br/>(fast, approximate)"| Candidates[Candidate Set]:::data
    Candidates --> Reranker(Cross-Encoder Reranker):::ai
    Reranker -->|"Top-5 reordered<br/>(slow, precise)"| Final[Final Context]:::result
    Final --> LLM(LLM Generation):::ai
    LLM --> Answer([Grounded Answer]):::result

    classDef data fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;

This pattern — sometimes called retrieve-then-rerank or a “funnel” architecture — is now the standard for serious production RAG systems. The retrieval stage optimizes for recall (don’t miss the right answer among the top-N), while the reranking stage optimizes for precision (put the truly best answers at the very top, where LLMs pay the most attention and where “lost in the middle” context effects matter least).

How Reranker Models Are Built

Most modern rerankers are cross-encoders fine-tuned on large-scale relevance-judgment datasets (e.g., MS MARCO, and increasingly synthetic relevance data generated by LLMs). Architecturally, they’re typically a transformer encoder (BERT-style, or a decoder-based LLM used in encoder mode) that takes the query and document concatenated as a single input sequence, and outputs a single relevance score via a classification head.

# Minimal cross-encoder reranking with a local model (sentence-transformers)
from sentence_transformers import CrossEncoder

model = CrossEncoder("BAAI/bge-reranker-v2-m3")

query = "How does HNSW handle incremental inserts?"
candidates = [
    "HNSW supports live insertion without a full index rebuild.",
    "IVF-PQ requires periodic re-clustering to add new vectors efficiently.",
    "HNSW was introduced by Malkov and Yashunin in 2016.",
]

pairs = [(query, doc) for doc in candidates]
scores = model.predict(pairs)  # higher = more relevant

ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
for doc, score in ranked:
    print(f"{score:.4f}  {doc}")

A Middle Ground: Late Interaction (ColBERT-style) Models

Between bi-encoders and full cross-encoders sits late interaction, pioneered by ColBERT (2020) and refined through ColBERT v2 and 2025’s Jina-ColBERT-v2. Instead of collapsing a document into one vector, late-interaction models keep a vector per token, and compute relevance via a fast “MaxSim” operation between query token vectors and document token vectors at query time.

MaxSim score = Σ_(query tokens) max_(doc tokens) similarity(q_token, d_token)

This preserves much of the fine-grained matching power of cross-encoders while remaining fast enough to sometimes skip the separate reranking stage entirely — some 2025-era vector databases (Qdrant, Vespa) now index ColBERT-style multi-vectors directly, blurring the line between “retrieval” and “reranking” into a single stage.

What’s New in Reranking (2025–2026)

Reranking has seen substantial model and architecture progress in the last two years:

• Cohere Rerank 3.5 (2024–2025) — improved reasoning-aware relevance scoring, notably better at matching queries that require inference (e.g., numerical reasoning, date comparisons, implicit intent) rather than surface-level keyword overlap. Widely adopted as an API-based default for enterprise RAG.
• Jina Reranker v2 (multilingual, agentic-RAG-oriented) — optimized specifically for agentic RAG use cases: function-calling relevance (ranking which tool/API a query should invoke), code search reranking, and ultra-low latency (claimed 15× faster inference than v1) across 100+ languages.
• BGE-reranker-v2-m3 and BGE-reranker-v2-gemma (BAAI, open-weight) — the leading open-source rerankers, supporting multilingual and long-context (8192-token) reranking, widely self-hosted alongside the bge-m3 embedding model for fully open retrieval stacks.
• LLM-as-reranker — Using instruction-tuned LLMs (via listwise or pairwise prompting) directly as rerankers has become a mainstream technique in 2025–2026, especially with fast, cheap models. Instead of a dedicated cross-encoder, the LLM is prompted with the query and a batch of candidates and asked to output a ranked list or per-document scores. This trades latency and cost for flexibility — no need to fine-tune or host a separate reranker model — and integrates naturally with agentic pipelines that already have an LLM in the loop.
• Listwise and setwise reranking — Rather than scoring documents one pair at a time (pointwise), 2025-era research and production systems increasingly rerank an entire candidate list at once (listwise), letting the model directly compare candidates against each other rather than against the query in isolation — often producing better top-1 precision, especially for LLM-based rerankers.
• Reranking in agentic and multi-hop RAG — As agentic RAG systems (which issue multiple retrieval calls, refine queries, and chain sub-questions) have become standard, reranking is now applied at each hop rather than once at the end, and reranker latency has become a first-class optimization target — driving the demand for smaller, distilled rerankers (e.g., ms-marco-MiniLM successors) that trade a little accuracy for sub-10ms inference.
• Native reranking support in vector databases — Weaviate, Qdrant, Elasticsearch, and Vespa have all shipped built-in reranking modules or integrations (calling out to Cohere, Jina, or local cross-encoder models) directly from the query API in 2025, removing the need for a separate orchestration step in application code.

Evaluating Reranker Quality

Reranker quality is typically measured with standard information-retrieval metrics on labeled relevance datasets:

The BEIR and MTEB (Massive Text Embedding Benchmark, which added a dedicated reranking track) benchmarks remain the standard public references for comparing reranker models across domains.

Practical Guidance: When and How to Rerank

A common rule of thumb: retrieve roughly 10–20× more candidates than you ultimately need (e.g., pull the top 50–100 via HNSW to end up with a final top 5), since reranking only helps if the correct document is somewhere in the candidate set to begin with — no reranker can recover a relevant document that retrieval never surfaced.

A Brief History: From Learning-to-Rank to Neural Cross-Encoders

Reranking as a concept predates modern embeddings by well over a decade. Classical learning-to-rank (LTR) systems — used by web search engines in the 2000s and 2010s (RankNet, LambdaMART, and gradient-boosted tree rankers like those in Bing and early Google search) — combined hundreds of hand-engineered features (click-through rate, term frequency, page authority, freshness) into a model trained to predict relevance ranking, applied as a second pass over a candidate set produced by a cheaper first-stage retrieval system (often classic inverted-index keyword search). The two-stage “retrieve cheaply, rank expensively” pattern reranking uses today is a direct descendant of that architecture.

The shift to neural rerankers began with BERT-based cross-encoders around 2019 (Nogueira & Cho’s passage reranking with BERT was an early influential result on the MS MARCO benchmark), which replaced hand-engineered features with a transformer that jointly attends over the query and document text. This produced large accuracy jumps on standard IR benchmarks but at a steep latency cost per pair scored — motivating the retrieve-then-rerank funnel that keeps the expensive model off the critical path for the full corpus. ColBERT (2020) then introduced late interaction as a middle ground, and the 2022–2024 period saw the rise of commercial reranking APIs (Cohere Rerank launched in 2023) purpose-built for the RAG use case, rather than web search. The 2025–2026 generation of rerankers (Rerank 3.5, Jina Reranker v2, BGE-reranker-v2 family) represents the maturation of that RAG-specific lineage — trained on synthetic and human relevance data that reflects how LLMs actually consume retrieved context, rather than how a search engine results page is browsed.

Reranking Beyond Text: Multimodal and Structured Reranking

As retrieval systems have expanded beyond plain text — into code search, image retrieval, and structured/tabular data — reranking has followed:

• Code reranking — Code search RAG (used in coding assistants and agentic developer tools) benefits enormously from reranking, since embedding similarity between code snippets often reflects surface syntax more than functional relevance. Jina Reranker v2 and several 2025-era open rerankers explicitly target code search, scoring candidate functions or files against a natural-language or code query.
• Multimodal reranking — With natively multimodal embedding models (text-image-audio-video) now common, cross-encoder-style rerankers that jointly score a text query against an image or video candidate have started appearing in 2025–2026 multimodal search products, following the same two-stage logic: cheap multimodal embedding retrieval, then a heavier joint model for final ordering.
• Structured/tabular reranking — Enterprise RAG over structured data (database rows, spreadsheets, knowledge-graph triples) increasingly reranks candidate rows or subgraphs using either a fine-tuned cross-encoder over serialized row text, or an LLM prompted with the schema and the candidate rows — extending the retrieve-then-rerank pattern outside of pure unstructured-text search.

Cost and Latency Considerations

Reranking is not free, and its cost structure differs meaningfully from retrieval:

Because reranking cost scales linearly with the number of candidates scored, the size of the candidate set passed from retrieval to reranking is the single biggest latency and cost lever available to an engineering team. Reranking the top 20 candidates instead of the top 200 can cut reranking latency by 10× with often minimal recall loss, provided the first-stage retrieval is good enough that the right answer is reliably within the smaller candidate set. This is why tuning first-stage recall (e.g., HNSW’s ef parameter) and reranker candidate-set size are usually optimized together, not independently — a stingy retrieval stage forces the reranker to work with a candidate set that may already be missing the best answer, while an overly generous one wastes reranking budget on candidates with no realistic chance of being relevant.

Common Pitfalls in Production

• Reranking too small a candidate set. If first-stage retrieval only returns the top 5–10 candidates, reranking has nothing meaningful to correct — the relevant document may never have made it into the set at all. Reranking adds the most value when applied to a moderately generous candidate pool (commonly 50–100).
• Ignoring reranker input length limits. Cross-encoders have maximum sequence lengths (often 512 tokens for older models, up to 8192 for newer long-context rerankers like BGE-reranker-v2-gemma). Silently truncating long documents before reranking can cut off the exact passage that would have justified a high relevance score.
• Treating reranker scores as calibrated probabilities. Relevance scores from different reranker models are not necessarily comparable in absolute terms, and thresholds tuned for one model (e.g., “only keep results scoring above 0.5”) often need to be re-tuned when switching reranker providers or versions.
• Not re-evaluating reranking gains over time. As the underlying embedding model or corpus changes, the marginal benefit of reranking can shift — teams should periodically re-measure NDCG/MRR with and without reranking rather than assuming a one-time evaluation still holds.

Why Reranking Matters More as RAG Matures

Early RAG systems in 2023–2024 often shipped with retrieval-only pipelines, trusting embedding similarity to be “good enough.” As RAG has moved into higher-stakes enterprise deployments — legal research, clinical decision support, financial analysis, customer support with regulatory exposure — the cost of subtly wrong retrieval (a plausible-but-irrelevant chunk feeding a confident, wrong LLM answer) has made reranking close to non-negotiable. Combined with the rise of agentic RAG (where retrieval happens repeatedly and errors compound across hops) and the maturing ecosystem of fast, purpose-built reranker models, reranking has solidified its position in 2025–2026 as the standard second stage of any serious retrieval pipeline — the precision layer sitting between fast approximate retrieval and the LLM that ultimately has to reason over what it’s given.

import cohere

co = cohere.Client("YOUR_API_KEY")

query = "What are the tax implications of remote work across state lines?"

# Stage 1: cheap, high-recall candidates from a vector DB (e.g. HNSW top-50)
candidates = [
    "Remote employees may owe income tax in both their home and work states.",
    "Our office is located in downtown Austin, Texas.",
    "Reciprocity agreements between states can eliminate double taxation.",
    "The company offers a hybrid work policy of three days in-office.",
    "Nexus rules determine whether an employer must withhold state tax.",
    # ... up to 50 candidate chunks from HNSW retrieval
]

# Stage 2: precise reranking of the candidate set
results = co.rerank(
    model="rerank-v3.5",
    query=query,
    documents=candidates,
    top_n=3,             # only keep the best 3 after reranking
)

for hit in results.results:
    print(f"[{hit.relevance_score:.4f}] {candidates[hit.index]}")

# Output is reordered by true relevance, not just embedding cosine similarity —
# e.g. the nexus/reciprocity/double-taxation chunks now rank above the
# superficially similar but irrelevant "office location" chunk.