Rerankers & Two-Stage Retrieval

Hybrid fusion hands you 40 candidates, and somewhere in them, hopefully, sits the answer. But “somewhere in the top 40” isn’t good enough on its own: the generation model reads maybe 8 chunks in its final prompt, and users reading citations pay real attention mostly to the top 3. The gap between “present in the pool” and “sitting at the top of the pool” is where rerankers live, and closing that gap is the single highest-leverage upgrade in the entire standard retrieval stack, more impactful per unit of engineering effort than almost anything else covered in this course.

Why rerankers see what embeddings can’t

Embedding retrieval is fundamentally a bi-encoder architecture: the chunk was compressed into one fixed vector months ago during ingestion, the query gets compressed into another vector just now at request time, and relevance is computed as a dot product between two summaries that have literally never been processed together, never seen each other during encoding. That independence is exactly what makes bi-encoders fast, you can compare a query against 50 million pre-computed chunk vectors in milliseconds, and exactly what caps their precision: all the fine-grained nuance of this specific query against this specific text got averaged away and lost at the moment the chunk was originally embedded, long before this particular query even existed.

A cross-encoder reranker refuses that compression entirely. It takes the query and one candidate chunk together as a single combined input, runs full bidirectional attention across both simultaneously, and outputs one relevance score for that specific pair. Every query token gets to directly interrogate every chunk token in the same forward pass: does “after one year” actually align with “12 months of employment” elsewhere in this chunk? Is this particular chunk fundamentally about canceling the policy or about enrolling in it, a distinction that matters enormously but that two independently-computed vectors might miss entirely? That’s a reading-comprehension judgment being made fresh for this exact query, not a fixed geometric relationship computed once and reused forever, and on ordering quality it reliably and measurably beats any pure similarity score.

The catch is equally obvious once you see it: you genuinely cannot cross-encode a query against 50 million chunks for every single request, the compute cost would be prohibitive. Hence two-stage retrieval, the architecture this entire module has been building toward piece by piece: a cheap, wide first stage (hybrid search plus RRF fusion from Lesson 11) guarantees the answer is somewhere in the room, top 40 candidates, and an expensive, narrow second stage decides who actually speaks first, reranking down to the top 8 that matter most.

flowchart LR
    P["40 fused\ncandidates"] --> CE["Cross-encoder:\nquery + chunk together,\none forward pass each"]
    CE --> S1["chunk A: 0.94"]
    CE --> S2["chunk B: 0.31"]
    CE --> S3["... 38 more scores"]
    S1 --> SORT["Sort by\nrelevance score"]
    S2 --> SORT
    S3 --> SORT
    SORT --> TOP["Top 8, reordered\nfor generation"]

    style P fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style CE fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style S1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style S2 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style S3 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SORT fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style TOP fill:#f0fdf9,stroke:#0D9488,color:#0F172A

The 2026 reranker menu

• API rerankers: Cohere Rerank 3.5 and Voyage’s rerank-2.5 line are the workhorses most teams reach for first: one straightforward HTTPS call, strong multilingual quality out of the box, and tens of milliseconds of latency per candidate batch. This is where most teams should start, since the integration cost is minimal and the quality bar is high.
• Open-weight: BGE-reranker-v2 and the Qwen3-Reranker family, which topped public reranking benchmarks through much of 2025, run comfortably in your own VPC for data-residency-sensitive deployments, on relatively modest GPU hardware compared to what generation models require.
• LLM-as-reranker: prompting a small LLM to score or explicitly order a list of candidates. More flexible than a purpose-built cross-encoder, since you can encode arbitrary instructions like “prefer newer policies over older ones when both seem relevant,” but meaningfully slower and noisier per dollar spent than a dedicated cross-encoder trained specifically for this task. It earns its place mostly inside the agentic loops covered in Module 6, where the LLM is already reading candidate passages as part of its reasoning process anyway, making relevance judgment a near-free byproduct of work it was doing regardless.

# Second stage: rerank the fused pool, with the two production
# details that matter: a timeout, and a fallback that keeps
# the system alive when the reranker is not.

import asyncio

async def rerank_top(query: str, candidates: list[Chunk],
                     final_k: int = 8) -> list[Chunk]:
    try:
        resp = await asyncio.wait_for(
            reranker.rerank(
                query=query,
                documents=[c.text for c in candidates],
                top_n=final_k,
            ),
            timeout=0.4,   # the latency budget line from Lesson 2
        )
        # Engine returns indices into our candidate list, best first.
        return [candidates[r.index] for r in resp.results]

    except (asyncio.TimeoutError, RerankerError) as e:
        # Fail open: fused order is still a decent order.
        # The alert (not the user) is who should feel this.
        log.warning("rerank_fallback", error=str(e))
        return candidates[:final_k]

That fallback path deserves a moment of attention beyond the code comment, because it’s the same architectural principle from Lesson 2’s safe() wrapper applied specifically here: reranking is an enhancement over an already-usable baseline, the RRF-fused order, and its failure should degrade the answer’s quality slightly, never take the request down entirely. A user who gets a slightly less well-ordered set of citations because the reranker timed out is having a fine experience; a user who gets an error page because a non-critical ranking service hiccuped is not, and the difference between those two outcomes is exactly this six-line try/except block.

Budgeting it honestly

Reranking sits squarely on the critical path of every request, so it pays real rent measured in milliseconds, and that rent needs to be budgeted deliberately rather than discovered accidentally in production. The arithmetic is friendly at the right pool size: running 40 candidates through a hosted reranker typically adds somewhere between 100 and 300 milliseconds of latency, and costs fractions of a cent per query. The published quality gains, Anthropic’s contextual retrieval study measured reranking contributing an additional third of failure reduction on top of what hybrid search alone achieved, and typical eval-suite lifts in the range of 10 to 25 percent on nDCG@10 are widely reported across independent teams, usually dwarf anything you’d get from chasing another embedding-model upgrade instead, at far higher migration cost per the discussion in Lesson 9.

Three practical rules keep this component honest in production rather than letting it become an unexamined tax on every request:

1. Keep the pool size in the 30-60 range. Below 30 candidates you starve the reranker of enough coverage to find genuinely good options; well past 60 you’re paying linearly for ordering improvements that have already flattened out, exactly as the quiz above makes you work through mathematically.
2. Rerank on the enriched text, not the bare chunk. The contextual prefixes established in Module 3 help the cross-encoder in exactly the same way they helped BM25 in Lesson 11: more disambiguating evidence packed into each candidate produces better relevance judgments from the model doing the scoring.
3. Let the evaluation suite, not intuition, decide whether the reranker earns its keep. Run your golden set from Module 8 both with and without the reranker enabled, on a quarterly cadence at minimum. If the measured quality delta sits inside the noise floor for your specific traffic pattern, that 250 milliseconds of latency rightfully belongs back to your users instead.

With retrieval and ranking now solid across both stages, the remaining major quality lever moves upstream of the index entirely, to the query itself. Real users ask vague, compound, pronoun-riddled questions that no amount of retrieval sophistication alone can fully compensate for, and the next module is about turning those messy real questions into queries this carefully built stack can actually win against.