Anatomy of a Production RAG System

Ask five vendors what RAG is and you’ll get five diagrams with an arrow from “your data” to “magic” to “answer.” Before we build anything, let’s replace that with the actual reference architecture that production teams converged on by 2026. Every later lesson in this course slots into one box of this picture, and when you’re debugging a system six months from now, this diagram is what you’ll mentally walk through to localize the problem.

The three planes

A production RAG system is really three loosely coupled systems wearing one name:

1. The ingestion plane (offline). Connectors pull documents from sources, parsers turn them into clean text and structure, chunkers split them, embedders vectorize them, and writers upsert into the indexes. It runs on queues and workers, tolerates retries, and never blocks a user. Nothing here has a human waiting on the other end of an HTTP request.
2. The serving plane (online). A user question comes in, gets rewritten and classified, hits the indexes through hybrid search with permission filters, candidates get reranked, the best chunks are assembled into a prompt, and the model generates a grounded, cited answer. Every millisecond here is felt by a human, and every stage is a candidate for the latency budget.
3. The control plane. Evaluation suites, tracing, quality dashboards, user feedback, and drift detection. It touches no request directly, but it’s the difference between a system you operate with confidence and a system that quietly operates you, degrading for weeks before anyone notices.

flowchart TB
    subgraph OFF["Ingestion plane (async)"]
        SRC["Sources\nDrive, Confluence, Slack, tickets"] --> CONN["Connectors +\nchange detection"]
        CONN --> PARSE["Parsing\nlayout-aware, VLM OCR"]
        PARSE --> CHUNK["Chunking +\ncontextual enrichment"]
        CHUNK --> EMB["Embedding\n+ metadata, ACLs"]
        EMB --> IDX[("Indexes\nvector + lexical")]
    end

    subgraph ON["Serving plane (request path)"]
        Q["User question"] --> QI["Query intelligence\nrewrite, filters, route"]
        QI --> RET["Hybrid retrieval\nwith ACL filter"]
        RET --> RR["Reranker"]
        RR --> GEN["Grounded generation\nwith citations"]
        GEN --> A["Answer + sources"]
    end

    IDX --> RET

    subgraph CTL["Control plane"]
        EV["Evaluation suite"] -.-> RET
        EV -.-> GEN
        TR["Tracing + dashboards"] -.-> ON
        FB["User feedback"] -.-> EV
    end

    style SRC fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style CONN fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style PARSE fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style CHUNK fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style EMB fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style IDX fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style Q fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style QI fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style RET fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style RR fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style GEN fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style A fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style EV fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style TR fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style FB fill:#fff7ed,stroke:#f59e0b,color:#0F172A

Why the planes must stay separate

The ingestion plane is bursty and slow, by nature rather than by poor engineering. A Confluence migration might dump 80,000 pages on you in an hour; a single scanned PDF might take 40 seconds in a VLM parser; a bulk permission change might require touching every chunk of a large space. The serving plane is the opposite: steady, high-frequency traffic with a hard latency budget of a few seconds end to end, where a single slow request is a visible incident, not a background job you can queue and forget.

Couple them and each inherits the other’s worst property. Parse at question time and users wait 40 seconds for a scanned PDF to become text before they even get an answer. Serve queries from your ingestion workers’ database and a reindex operation contends for the same resources your live users need, taking your assistant down during the exact window you’re trying to improve it. Keep them separate and each plane scales along the dimension that actually matters to it: ingestion scales in worker count during a migration and idles the rest of the time, serving scales in query replicas during business hours and can shrink overnight. They also deploy independently: a new chunking strategy rolls out by reindexing in the background and cutting over atomically, never by redeploying the customer-facing API and risking a live outage over a data pipeline change.

The one place the two planes meet is the index, and it deserves to be treated with the same rigor as a database schema, because that’s functionally what it is. Every chunk written by ingestion and read by serving carries an implicit contract: chunk text, source metadata, ACL tags, timestamps, and an embedding model version. Both planes must agree on that contract’s current shape. Change it (a new embedding model, a new chunking strategy, an added metadata field) and you’re making a schema migration, which is why Module 2’s incremental sync lesson spends real time on blue-green reindexing rather than treating index updates as a detail.

The serving path owns the latency budget

Here’s the skeleton of the serving path as code. The point isn’t the framework, it’s the shape: every stage is optional-on-failure except retrieval itself, and every stage’s timing is something you will eventually plot on a dashboard.

# serving.py - the request path in ~30 lines.
# Each stage degrades gracefully: if an enhancement fails,
# we log it and continue with what we have.

async def answer(question: str, user: User) -> Answer:
    # 1. Query intelligence (fast LLM call, ~150ms).
    #    On failure: use the raw question unchanged.
    plan = await safe(rewrite_and_plan, question, default=QueryPlan(question))

    # 2. Hybrid retrieval with the user's ACL filter.
    #    This is the one stage that must succeed; without
    #    candidates there is nothing to ground on.
    candidates = await retrieve(
        query=plan.rewritten,
        filters=plan.filters,
        acl=user.permission_tags,   # enforced IN the index query
        top_k=40,
    )

    # 3. Rerank 40 candidates down to the best 8.
    #    On failure: keep the fusion order, take the top 8.
    top = await safe(rerank, plan.rewritten, candidates, default=candidates[:8])

    # 4. Grounded generation with citations, streamed to the user.
    return await generate_grounded(question=question, chunks=top, user=user)


async def safe(fn, *args, default, timeout: float = 0.4):
    """Every enhancement stage runs through this wrapper.
    A slow or broken enhancement degrades the answer; it
    never takes the whole request down with it."""
    try:
        return await asyncio.wait_for(fn(*args), timeout=timeout)
    except Exception as e:
        log.warning("stage_degraded", stage=fn.__name__, error=str(e))
        return default

Notice what’s absent from this function: no parsing, no embedding of documents, no index writes. Those all happened hours ago on the ingestion plane, asynchronously, with nobody waiting. The only embedding computed at request time is the query’s own, which is why query embedding latency (typically 20 to 60ms for a hosted API) is worth tracking separately from retrieval latency in your traces.

The safe wrapper is doing more architectural work than its five lines suggest. It’s the mechanism that turns “reranker down” from an outage into a quality degradation, and it’s the same pattern every enhancement stage in this course will use: query rewriting (Module 5), filter extraction (Module 5), and reranking (Module 4) are all improvements over a working baseline, never single points of failure. The one exception, deliberately, is the retrieval call itself: there is no graceful degradation from “we have candidates” to “we have none,” so retrieval gets a real, generous timeout and genuine error handling rather than a silent default.

The control plane is not optional

Every team ships the first two planes; the business requirement to answer questions forces that much. The teams that succeed past the first few months also ship the third, and the gap between those two groups is almost entirely explained by what happens after launch, not by the initial build quality.

Without evaluation and tracing, quality regressions are invisible from the inside. You swap an embedding model because a benchmark looked good; retrieval recall drops 12 percent specifically on legal documents, because that collection’s vocabulary didn’t transfer well to the new model; and you find out three weeks later from an angry Slack thread, by which point dozens of users have quietly stopped trusting the tool and won’t come back even after you fix it. The control plane exists to catch that regression on day one, ideally before it ships.

We dedicate all of Module 8 to this plane, building a golden evaluation set, component-level metrics that isolate which stage broke, and drift detection that catches slow decay rather than sudden breaks. For now, one rule worth adopting from lesson one of your build: from day one, log every query, its rewritten form, its retrieved chunk IDs with scores, and the final answer with citations. That log costs almost nothing to start collecting, and it is the raw material every later evaluation technique in this course depends on. Teams that start logging on day 90 lose 90 days of real production failure cases they can never fully reconstruct.