Building a RAG Evaluation Suite

RAG fails quietly, more quietly than almost any other kind of production system failure a team is likely to have dealt with before. A retrieval regression does not throw an exception that pages someone; it simply returns plausible-looking chunks instead of the genuinely right ones, and the generation model then writes a perfectly fluent answer from whatever it received, and the only real signal anyone gets is a slow, hard-to-attribute drift in whether people still trust the assistant enough to keep opening it. Every other module in this course handed you levers to pull. This final module gives you the instrument that tells you which direction to pull them in, and without it, you are tuning a system you genuinely cannot hear.

The golden set

Start exactly where the real answers already live: your production logs. Lesson 21’s audit records contain genuine questions asked by genuine users, which reliably beats synthetic questions written internally by the same team that built the system, since self-written test questions tend to accidentally encode the system’s own internal vocabulary and blind spots rather than testing against how real users actually phrase things.

A working golden evaluation set runs somewhere between 200 and 500 items, and should deliberately cover several distinct categories rather than being one undifferentiated pile of questions:

• Real traffic, sampled across the genuine mix of query types your users actually send: simple lookups, comparisons, multi-hop questions, weighted roughly toward what people actually ask rather than toward what’s interesting to test.
• Known-hard cases: the specific failures that originally prompted engineering work on this system. Every bug fixed anywhere in this course’s pipeline becomes a permanent test case added here, so it can never silently return unnoticed.
• Unanswerable questions (roughly 10 to 20 percent of the set): plausible-sounding questions with genuinely no supporting document anywhere in the corpus. These specifically measure the refusal path built in Lesson 16, and they’re the category teams most often skip when building their first evaluation set, precisely because writing a question you know the system can’t answer feels unproductive compared to writing one it should get right.
• Permission-scoped questions: the identical question text asked as two different users, where the objectively correct answer legitimately differs between them because of their respective ACLs. This is the only automated test shape that can catch an ACL regression before an actual human notices something has leaked.

Each item in the set carries the question itself, the expected supporting chunk or document IDs for scoring retrieval, and a reference answer or scoring rubric for scoring generation. Labeling a set like this is genuinely a day or more of tedious work, but it pays for itself the very first time it catches a bad deploy before that deploy ever reaches a real user.

Measure the stages separately

The single most useful structural decision in the entire evaluation design, worth treating as the organizing principle for everything else in this lesson: score retrieval and generation completely independently, so that any red number on a dashboard points unambiguously at one specific module to go investigate rather than leaving you guessing across the whole pipeline.

Retrieval metrics require no LLM call at all, making them cheap enough to run automatically on every single commit to the retrieval code:

• Recall@k: did the genuinely supporting chunk actually make it into the top-k candidate set at all? This number is the hard ceiling on everything downstream of it; if this is low, nothing generation does afterward can possibly rescue the answer, no matter how good the prompt is.
• nDCG@k: is the good evidence sitting near the actual top of the ranked list, not merely present somewhere within it? This is specifically what your reranker from Module 4 is being paid, in latency and dollars, to improve.
• MRR: how high up the ranking did the first genuinely correct chunk land, on average across the whole set?

Generation metrics require an LLM-as-judge and are typically run nightly or specifically gating a pre-deploy check rather than on every commit:

• Faithfulness: is every individual claim in the answer actually supported by the retrieved context it cites? This is your hallucination detector, and it’s mechanically checkable precisely because Lesson 16 forced every answer to carry inline citation markers that a judge can verify against.
• Answer relevance: does the generated answer genuinely address the specific question that was asked, rather than a nearby but different question?
• Honest refusal rate: specifically on the unanswerable slice of the golden set, did the system correctly decline to answer rather than inventing something plausible?

# The gate: retrieval metrics are cheap and deterministic,
# so they run on EVERY change to the retrieval stack.

def eval_retrieval(golden: list[GoldenItem], k: int = 8) -> dict:
    recalls, ndcgs = [], []
    for item in golden:
        hits = retrieve_and_rerank(
            item.question, acl=item.as_user.principals, final_k=k,
        )
        got = [c.chunk_id for c in hits]
        want = set(item.supporting_chunk_ids)

        # Did the evidence arrive at all? (ceiling on quality)
        recalls.append(len(want & set(got)) / max(len(want), 1))
        # Did it arrive near the top? (reranker's job)
        ndcgs.append(ndcg(got, want, k))

    return {"recall@k": mean(recalls), "ndcg@k": mean(ndcgs)}

# CI gate: block the deploy on a real regression, not on noise.
BASELINE = {"recall@k": 0.87, "ndcg@k": 0.71}

def gate(results: dict, tolerance: float = 0.02) -> None:
    for metric, baseline in BASELINE.items():
        if results[metric] < baseline - tolerance:
            raise SystemExit(
                f"BLOCKED: {metric} {results[metric]:.3f} "
                f"below baseline {baseline:.3f}"
            )

That gate function is precisely what makes the blue-green reindex pattern from Lesson 6 genuinely safe to run: you build the candidate new index, you run this exact check against it, and the serving alias only ever swaps over if the numbers hold up against the established baseline. It’s also what turns a question like “should we keep the reranker enabled?” from Lesson 12 from a matter of engineering opinion into a directly measured decision backed by real numbers.

LLM-as-judge, used carefully

Generation quality genuinely needs a judge somewhere in the loop, and human labeling simply doesn’t scale to running against every single deploy. LLM-as-judge, the pattern underlying tools like RAGAS and TruLens along with the built-in eval features shipped by most observability vendors by 2026, works well in practice as long as you respect its several well-documented pathologies rather than trusting it blindly:

• Position bias: judges systematically prefer whichever candidate answer they happen to see listed first in a pairwise comparison. Randomize or explicitly swap the presentation order to control for this.
• Verbosity bias: judges systematically reward longer answers, independent of whether the extra length adds genuine value. Rubrics that score specific, itemized criteria, such as “is each individual claim properly cited?” and “is the refusal, where applicable, actually correct?”, resist this bias considerably better than an open-ended “which answer is better?” framing does.
• Self-preference: models tend to rate outputs from their own model family more generously than outputs from competing families. Use a genuinely different model family as the judge than whatever family generates the answers being judged, wherever that’s practically feasible.
• Version drift: upgrading the judge model silently rebases every historical score it has ever produced, without any obvious signal that this happened. Pin the specific judge model and version explicitly in your eval configuration, and when you must eventually upgrade it, re-baseline the entire historical score record in one single, deliberate run rather than letting old and new scores mix silently.

Then calibrate the judge against reality before trusting it: hand-label somewhere between 100 and 200 items yourself or with a small human review team, measure the judge’s agreement rate with those human labels, and let that measured agreement number set exactly how much weight the automated score should carry in your process. A judge measured at 90 percent agreement with humans is trustworthy enough to gate real deploys on its own; a judge measured at only 60 percent agreement is, honestly, more of a suggestion box than a gate.

Two closing habits separate teams whose systems genuinely improve over time from teams that just churn in place without making real progress. Run the full suite on a fixed schedule, not only in response to code changes, because the corpus itself drifts over time even when your code doesn’t change at all, which is exactly the subject Lesson 24 covers in depth. And make every single production failure into a permanent test case the moment it’s discovered: the golden set should grow steadily every month as a matter of process, and each addition is a specific regression that, once captured, can never again silently return unnoticed.

Quality is now genuinely measurable across both stages of the pipeline. The other two dimensions users feel directly, whether they consciously notice it or not, are how fast the system responds and, eventually, what it costs the business to keep running. Those are next.