The most demoralizing kind of RAG failure to diagnose is the one with no apparent cause at all. Nothing was deployed recently. Latency graphs are flat. Every dashboard reads green. And yet the assistant that people genuinely loved back in March is one they’ve quietly stopped opening by September, with no single incident anyone can point to as the moment it broke. This closing lesson of the course is about seeing that decline coming before it fully happens, because the two most important inputs to the entire system, the documents themselves and the questions being asked of them, are both alive and constantly changing, whether or not the code that processes them changes along with them.
Trace every stage
The natural unit of observability in a RAG system is the individual request, decomposed into the exact stages this course built module by module. OpenTelemetry-based tracing, now the common technical substrate underneath tools like Langfuse, Phoenix, LangSmith, and the various vendor-provided observability tools, gives every single request a tree of spans that mirrors the pipeline’s own structure:
# One span per stage, with the inputs and outputs that make
# a bad answer debuggable months later. This IS the audit
# record from Lesson 21, viewed as telemetry.
with tracer.start_as_current_span("rag.answer") as root:
root.set_attribute("user.principal_hash", user.principal_hash)
root.set_attribute("tenant.id", user.tenant_id)
with tracer.start_as_current_span("query.rewrite") as s:
plan = await rewrite_and_plan(question, history)
s.set_attribute("query.raw", question)
s.set_attribute("query.rewritten", plan.rewritten)
s.set_attribute("query.filters", json.dumps(plan.filters))
with tracer.start_as_current_span("retrieve.hybrid") as s:
cands = await hybrid_search(plan, acl=user.principals, top_k=40)
# Chunk IDs plus scores: the difference between "we can
# see it was not retrieved" and "we have no idea".
s.set_attribute("retrieval.chunk_ids", [c.id for c in cands[:10]])
s.set_attribute("retrieval.top_score", cands[0].score if cands else 0)
s.set_attribute("retrieval.count", len(cands))
with tracer.start_as_current_span("rerank") as s:
top = await rerank_top(plan.rewritten, cands, final_k=8)
s.set_attribute("rerank.kept_ids", [c.id for c in top])
s.set_attribute("rerank.top_score", top[0].rerank_score)
with tracer.start_as_current_span("generate") as s:
answer = await generate_grounded(question, top, user)
s.set_attribute("gen.model", answer.model)
s.set_attribute("gen.citation_count", len(answer.citations))
s.set_attribute("gen.refused", answer.refused)With that tree of spans in place, any user complaint becomes roughly a five-minute investigation rather than an open-ended debate: pull the trace by its ID, walk down through the stages in order, find precisely where the right chunk fell out of contention along the way. Without this instrumentation, teams end up arguing in a meeting about whether the embedding model needs replacing, when the actual culprit turns out to be a date filter incorrectly extracted from the word “recent” three stages earlier in the same request.
The four drifts
Corpus drift. The underlying documents themselves change: a company reorg renames entire teams and processes throughout the wiki, a bulk migration imports 40,000 pages that got parsed badly by an edge case in your pipeline, a new product launch introduces vocabulary your index has genuinely never encountered before. Watch for it via: ingestion volume broken down by source, the parse failure rate over time, and the share of queries whose top retrieval score falls below your established floor, since a rising share of low-confidence retrievals is a strong signal the corpus has stopped covering what people are actually asking about.
Query drift. The users themselves change: a newly onboarded team brings genuinely different vocabulary into their questions than your existing user base ever used, a new feature launch spawns an entirely new category of question your chunking strategy was never designed to serve well. Watch for it via: the honest refusal rate tracked over time, and periodic clustering of the last 30 days of real questions to surface genuinely new intents appearing in the traffic that weren’t present when the system was originally tuned.
Embedding drift. Not the stored vectors themselves changing, those are frozen once written, but rather the model’s fit to current reality gradually loosening: an embedding model trained on data through 2024 slowly comes to know progressively less about your organization’s 2026 jargon and newly introduced concepts as more of the corpus shifts toward vocabulary the model never saw during its own training. This shows up as slowly declining recall specifically on your newer documents when measured against the eval suite, and the remedy is exactly the blue-green reindex pattern established back in Lesson 6.
Model drift. Your generation model itself gets upgraded by the provider, and observable behavior shifts as a direct result: different default verbosity, different calibration around when to refuse, different fidelity to the citation contract your prompts depend on. This is precisely why prompt and model versions are pinned explicitly in the audit record from Lesson 21, and precisely why the full evaluation suite must run against every single model upgrade before that upgrade is allowed to reach real users, never assumed safe by default just because the provider labeled it an improvement.
Which yields the single operational rule this entire module has been circling around from several different angles: the evaluation suite runs on a fixed schedule, not merely in response to deploys. Nightly retrieval metrics, weekly generation metrics, both plotted over time on a shared dashboard. What you’re watching for here is slopes trending in the wrong direction over weeks, not simple threshold breaches that trigger a single alert and then get dismissed as a one-off.
Close the loop with users
Your users generate genuinely valuable improvement signal continuously, all day, every day, and most teams end up collecting almost none of it in a form that’s actually usable.
- Thumbs up and down: sparse in volume and negatively biased toward capturing frustration rather than satisfaction, but a down-vote paired with its full trace attached is a genuine gift to whoever triages it. Route every single down-vote into a triage queue with its complete span tree attached automatically, rather than as a bare rating number with no context.
- Citation clicks: a dense, implicit feedback signal that most teams underuse. Answers whose citations get clicked are answers users are actively bothering to verify, and citations that get clicked and then immediately abandoned, meaning the user clicked through and then quickly left, hint that the underlying evidence looked wrong once actually inspected up close, even though it looked plausible enough in the answer to warrant a click.
- Refusals: every single honest “not in the knowledge base” response, the refusal path built deliberately in Lesson 16, is effectively a labeled coverage gap reported automatically by the system itself. Aggregate these weekly; the resulting top clusters point directly at either documents that genuinely need to be ingested or a query type that needs dedicated support, no guessing required. This is arguably the single most directly actionable feedback stream available anywhere in the whole system, and it exists purely as a side effect of having built the refusal path instead of letting the model bluff its way through thin evidence.
- Rephrasings: a user who asks essentially the same underlying question three different ways within one session is telling you, quite clearly, that retrieval failed them the first two times, entirely without ever touching a thumbs-down button or leaving any explicit rating at all.
Every one of these four signal types flows to the identical destination: the golden set built back in Lesson 22. A failure becomes a labeled test case, that test case becomes a permanent gate in the CI pipeline, and that gate prevents the exact same regression from ever silently returning unnoticed. That loop, running reliably on a weekly cadence, is what separates a knowledge system that genuinely compounds in quality over time from one that slowly, invisibly decays despite nobody ever intending for that to happen.
Where you are now
Across this course you’ve built the capability to take a corpus of genuine enterprise documents and construct a system that parses them faithfully regardless of format, keeps that parsed content fresh as sources change underneath it, chunks it so it stays reliably findable rather than fragmenting into useless pieces, retrieves candidates with hybrid search and reorders them with reranking, understands what users are actually asking even when they ask it badly, generates answers that cite their real evidence and honestly admit the limits of what that evidence supports, reaches into live systems through agents and MCP when the static index genuinely isn’t enough on its own, enforces the organization’s permission model consistently on every single path through the system, resists the attacker who edits your wiki instead of typing a malicious query, satisfies the auditor who shows up asking hard questions six months after the fact, and measures all of the above well enough to know, proactively, when any part of it has started to quietly rot.
The specific techniques covered in this course will keep moving forward, as they always do in a field this active. The underlying structure will not: retrieve the right evidence, ground the generated answer firmly in it, prove exactly where that evidence came from, and continuously measure whether you actually did all three of those things correctly. Everything else covered across these 24 lessons is, in the end, implementation detail in service of that one durable structure.