Capstone: Production Knowledge Platform

Every lesson in this course built one piece of a production RAG system. This capstone puts all eight modules back together as one coherent build, with the level of detail you’d actually want walking into week one: a realistic brief, a phased checklist with acceptance criteria you can check off, a launch gate that blocks going live until the system earns it, and guidance for defending the architecture to the people who fund it.

The brief

You’re the engineering lead for a knowledge assistant at Meridian Manufacturing, a mid-size industrial parts maker with roughly 2,200 employees across three sites. The assistant needs to answer questions against four genuinely different sources: the engineering wiki on Confluence (design specs, changelog history), HR policy PDFs on SharePoint (benefits, leave, safety procedures), signed vendor and customer contracts (PDF, some scanned amendments), and the support ticket history in the helpdesk system. Roughly 400,000 documents in total, growing by a few thousand a week.

Three constraints make this a genuine enterprise build rather than a demo, and each one traces directly back to a specific module of this course. Engineering staff and HR staff must never see each other’s restricted documents through the assistant, even though both groups’ content lives in the same deployment (Module 7). Contract terms cited in an answer must be traceable to the exact clause and document version, because procurement will act on what the assistant tells them (Module 6’s citation work, Module 7’s audit trail). And the system has to stay useful as the wiki and contract set keep changing weekly, not just look good in a launch demo (Module 2, Module 8).

Architecture overview

Every box in this diagram is something you built, named, and tested in an earlier lesson. Nothing here is new engineering; this is assembly.

flowchart TB
    subgraph SRC["Sources"]
        S1["Confluence\n(engineering wiki)"]
        S2["SharePoint\n(HR policy PDFs)"]
        S3["Contract PDFs\n(signed, some scanned)"]
        S4["Helpdesk\n(ticket history)"]
    end

    subgraph ING["Ingestion (Module 2)"]
        CONN["Connectors +\nchange detection"]
        PARSE["Docling + VLM fallback\n(Lessons 4-5)"]
        CHUNK["Structure-aware chunking +\ncontextual enrichment (Module 3)"]
    end

    subgraph IDX["Indexes"]
        VEC[("Qdrant: dense +\nBM25, per-collection\nby source (Module 4)")]
    end

    subgraph SERVE["Serving (Modules 5-6)"]
        QI["Rewrite + filter\nextraction"]
        RET["Hybrid retrieval,\nACL-filtered (Module 7)"]
        RR["Rerank"]
        GEN["Grounded generation\n+ citations, agentic\nfallback for gaps"]
    end

    subgraph CTL["Control plane (Module 8)"]
        EVAL["Eval suite +\nlaunch gate"]
        TRACE["Tracing +\naudit log"]
    end

    SRC --> CONN --> PARSE --> CHUNK --> VEC
    VEC --> RET
    QI --> RET --> RR --> GEN
    EVAL -.-> RET
    EVAL -.-> GEN
    TRACE -.-> SERVE

    style SRC fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style ING fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style IDX fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SERVE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style CTL fill:#fff7ed,stroke:#f59e0b,color:#0F172A

The one design decision worth calling out explicitly: contracts and tickets get their own Qdrant collections rather than sharing one with the wiki and HR policies, following Lesson 14’s routing criteria. Contracts need page-precise citations and occasional whole-document reading via long context; tickets churn fast enough that their sync loop runs on a much shorter interval than the relatively stable wiki. Everything else lives in shared collections filtered by ACL tags and source metadata, per Lesson 19.

Build checklist, phase by phase

Work through these in order. Each phase has a concrete acceptance criterion; do not start the next phase until the current one’s criterion is genuinely met, not just “looks about right.”

Two phases are easy to underestimate going in. Phase 1 typically eats more calendar time than Phases 2 and 3 combined, because the scanned contract amendments and inconsistent HR PDF layouts are where real corpora fight back, exactly as Lessons 4 and 5 warned. And Phase 4 is where teams are most tempted to cut corners under deadline pressure, which is precisely backwards: a security gap discovered after launch costs far more than the week it takes to build permission-scoped eval items now.

The launch gate

Borrow directly from Lesson 22’s gate pattern, but apply it across every dimension this capstone touches, not retrieval alone. A launch that passes on quality but fails on security, or passes on both but blows the latency budget, is not ready:

# capstone_gate.py - run this before any go-live decision.
# Every threshold maps to a phase above; a failure here means
# a specific phase's acceptance criterion regressed.

LAUNCH_THRESHOLDS = {
    "recall_at_8": 0.85,
    "ndcg_at_8": 0.70,
    "faithfulness": 0.90,
    "honest_refusal_rate": 0.90,
    "acl_violations": 0,          # hard zero, not a threshold
    "p95_latency_seconds": 4.0,
}

def launch_gate(results: dict) -> bool:
    failures = []
    for metric, threshold in LAUNCH_THRESHOLDS.items():
        value = results[metric]
        # Direction depends on the metric: latency and violations
        # must be BELOW threshold; everything else must be ABOVE.
        bad = (value > threshold if metric in
               ("p95_latency_seconds", "acl_violations")
               else value < threshold)
        if bad:
            failures.append(f"{metric}: {value} (need {threshold})")

    if failures:
        print("LAUNCH BLOCKED:\n" + "\n".join(failures))
        return False
    print("Launch gate passed.")
    return True

Run this against the full golden set, including the permission-scoped pairs and the unanswerable-question slice, not a convenient subset. If acl_violations is anything other than zero, nothing else on the list matters until it’s fixed; that is the one metric on this list with no acceptable partial credit.

Presenting to stakeholders

Meridian’s engineering VP and CFO are not going to read this course. They’re going to ask three questions, and the answers should be ready before the architecture review, not improvised during it.

“Why not just use ChatGPT with our documents pasted in?” This is Module 1’s argument, compressed: permissions, because HR and engineering content can’t share one undifferentiated context; freshness, because a contract signed this morning has to be answerable this afternoon; and cost, because 400,000 documents don’t fit in any prompt at any price that scales to daily use.

“What happens if it gets something wrong?” This is where the citation work from Lesson 16 earns its keep in a boardroom, not just in code. Every answer links back to the specific document, page, and version it came from, so a wrong answer is checkable in seconds rather than trusted blindly, and the audit trail from Lesson 21 means any disputed answer can be reconstructed exactly, including who asked and what they were authorized to see.

“Why did this take six weeks instead of the two-day hackathon demo we saw?” The honest answer is the difference between this capstone and a notebook: permission enforcement, evaluation gating, and the operational discipline in Module 8 don’t show up in a demo, but they’re the entire difference between a system that survives contact with 2,200 real employees and one that leaks the salary review folder in its third week.

Frame every technical decision this way: not what it is, but what it protects or what it costs to skip. That’s the difference between an architecture review that approves a launch and one that stalls in questions.

What to build next

Once Meridian’s assistant is live and the eval suite is holding steady week over week, the natural extensions follow the same cost-benefit discipline Module 5 taught for GraphRAG: adopt them when traffic data justifies them, not because they’re available. A scoped GraphRAG index over the contract collection alone would serve the recurring “which vendors are connected to this dispute” questions procurement actually asks. An MCP server exposing search_knowledge_base would let the same retrieval stack power an IDE assistant for engineering without a second integration. And a deep-research mode, gated behind explicit user request per Lesson 17’s cost discipline, would serve the quarterly “compile everything on vendor X” requests that come from procurement without becoming the default, expensive path for an ordinary employee’s Tuesday-afternoon question.

The system you’ve now designed end to end, across nine modules, is not a finished artifact. It’s a platform with a measurement loop built in from day one, which is exactly what lets it keep earning trust instead of slowly losing it the way an unmeasured system inevitably does.