Everything until now in this course was logistics: parse the documents, chunk them well, embed them, retrieve the right candidates, rank them properly. The user sees essentially none of that machinery directly. They see exactly one thing, the generated answer, and they extend trust in the entire system exactly as far as that single answer earns it, every single time. Generation is where a technically excellent pipeline, built carefully across the previous five modules, can still fail badly as a product, if the model answers from its own imagination instead of the retrieved evidence, cites nothing a user can actually check, or bluffs confidently when the corpus genuinely had nothing relevant to say.
Assembling the context
You have 8 reranked chunks from Module 4’s pipeline and a question to answer. How those chunks enter the final prompt matters considerably more than most teams expect going in:
Order by relevance, and consider placing the strongest evidence at both the top and the bottom if you’re passing a larger set. Long-context attention still measurably favors the edges of a prompt over its middle, the same positional effect discussed back in Lesson 1 for full-corpus stuffing, and it applies at a smaller scale inside individual prompts too. With a tight set of 8 chunks this effect matters relatively little; if you’re passing 30 or more chunks to a long-context model for a deep-reading task, deliberately put your strongest evidence at the top and bottom of that list rather than trusting a flat relevance ordering.
Keep provenance visible inside the prompt, not just in a separate metadata field. Each chunk arrives wrapped with its own identity: a number, its document title, its section path, its last-updated date. The model actively uses these signals to arbitrate conflicts between sources, and dates especially matter here: policies routinely supersede earlier versions of themselves, and “the March version says X, the January version said Y” is only a correct, useful answer if the model can actually see both documents’ dates and reason about which one is current.
Don’t strip the metadata header to save a few tokens. Roughly fifty tokens of header per chunk buys real conflict-resolution capability and genuine citeability. It is consistently the single best token expenditure in the entire prompt, cheap relative to the chunk text itself and disproportionately valuable for the trust the resulting citations earn.
GROUNDING_SYSTEM = """You are a knowledge assistant. Answer using
ONLY the numbered context below.
Rules:
- After every factual claim, cite its source inline: [2].
- Multiple sources: [1][4]. Never invent a number.
- If sources conflict, prefer the most recent and note the
conflict explicitly.
- If the context does not contain the answer, say so plainly.
Never fill gaps from general knowledge. Point to whichever
retrieved source is closest to the topic instead.
- Quote exact figures, dates, and names as written."""
def build_context(chunks: list[Chunk]) -> str:
parts = []
for i, c in enumerate(chunks, 1):
# Provenance header per chunk: the model reads it for
# recency arbitration, the citation renderer for links.
parts.append(
f"[{i}] {c.doc_title} > {c.section} "
f"(updated {c.updated_at:%Y-%m-%d})\n{c.text}"
)
return "\n\n---\n\n".join(parts)Citations that survive checking
Chunk-level inline markers, exactly as shown above, are the floor everyone should build to at minimum. Every [3] renders in the UI as a link built directly from chunk 3’s stored url, page, and section fields, all preserved carefully back in Lesson 4’s parsing work specifically for this purpose. Because that block-level provenance was preserved, the link can open the actual source PDF at the actual page the claim came from. Users click these constantly during the first weeks of any real deployment, that’s the trust audit happening in real time, visibly, across your whole user base, and it’s exactly why fake-able citations, bare document names the model could simply invent from a plausible-sounding title, are meaningfully worse than having no citations at all: they create the appearance of verifiability without the substance.
The ceiling above chunk-level citation is span-level citation: the answer ties each individual claim to the exact supporting sentence within its source chunk, not just to the chunk as a whole. Two roads lead there in practice. Provider-native citation modes, Anthropic’s citations API being the mature example as of this writing, return structured claim-to-span mappings with actual character offsets, handling the bookkeeping entirely server-side rather than requiring you to build it yourself. Or you can enforce a quote-first output schema on your own: the model must emit structured {"quote": "...verbatim...", "claim": "...", "source": 3} triples for every claim, and your rendering layer verifies each quote actually appears verbatim in chunk 3 through plain string matching before trusting it. That verification step catches the specific failure where a model paraphrases loosely while claiming to quote directly, which is exactly the failure mode you most want surfaced and caught before it reaches a user.
Span-level citation costs more output tokens per answer and requires stricter, more demanding prompting to get right consistently. Legal, medical, and financial deployments pay that cost without hesitation, since the verification value clearly outweighs the extra tokens for their use case. An internal engineering wiki assistant usually doesn’t need to go this far, and chunk-level citation serves it perfectly well.
The refusal path is a feature you must build
Sometimes retrieval comes back thin: eight chunks that are all adjacent-but-not-actually-answering the specific question asked. What happens next defines your system’s integrity as a product, and by default, with no deliberate engineering effort, what happens is bad: LLMs are trained to be helpful to a fault, and left unguided they will construct a fluent, well-organized answer from pretraining knowledge instead, presented in a tone completely indistinguishable from a genuinely grounded one built from real retrieved evidence.
Build the refusal path deliberately, as its own first-class piece of the system rather than an afterthought:
- Instruct it explicitly, exactly as the system prompt above does, and then actually test that the instruction holds up under pressure using deliberately unanswerable questions planted in your evaluation set, which Module 8 covers building in detail.
- Detect thin evidence upstream when possible, before generation even runs: if the reranker’s top score for a given query sits below a floor you’ve calibrated against your own data, skip straight to the honest no-evidence response and save the generation tokens entirely rather than generating an answer you already have reason to distrust.
- Design the honest response as genuine UX, not as an apology to minimize. “I couldn’t find this in the knowledge base. Closest matches: [links]. If it should exist, it may not be indexed yet.” That final sentence turns a dead end into a coverage-gap report; route these events to a feedback queue and they become a genuinely valuable, self-generating ingestion backlog pointing directly at what your corpus is actually missing.
One paragraph on the temperature question specifically, because someone on every team eventually asks it: low temperature does not fix grounding, and this is worth stating plainly since it’s a common and reasonable-sounding misconception. Sampling temperature governs how much variance exists among plausible next-token choices; it has essentially no bearing on whether the model actually respects the context contract laid out in your system prompt. Whether the model stays faithfully within the provided evidence is set by the clarity of your instructions, the genuine quality of the evidence retrieval handed it, and the underlying model’s instruction-following capability, none of which temperature touches. A well-grounded pipeline running at default temperature reliably beats an ungrounded, poorly-instructed one running at temperature zero, every time this comparison has been tested.
Single-pass generation over single-pass retrieval is now about as good as this architecture gets. The next real step up in capability, and the defining RAG pattern of the 2025-26 period, is letting the model drive retrieval itself: judging the evidence it received, reformulating its own searches when that evidence falls short, and looping until it genuinely has enough to answer well. That’s agentic RAG, and it’s next.