Cost, Latency & Caching

Two numbers ultimately decide whether a genuinely good RAG system, built carefully across the previous seven modules of this course, actually survives contact with the business it serves: how long users have to wait, and what finance ends up paying every month. Both numbers are shaped directly by choices already made throughout this course, and both tend to get optimized in the wrong place, because teams instinctively tune whatever they find intellectually interesting rather than tuning whatever the actual measured profile says is the real bottleneck.

Where the milliseconds go

A typical serving path, measured directly rather than estimated from intuition:

Generation dominates the latency budget, essentially always, in every real deployment measured this way. Which means the retrieval optimizations that feel most engineering-satisfying to chase, shaving 30 milliseconds off an HNSW query, for instance, are pure noise against the overall number, and the moves that genuinely matter concentrate almost entirely on the generation stage:

Stream the response, always, without exception. Time-to-first-token is what users actually experience as speed, far more than total completion time. A 4-second answer that starts visibly appearing at the 600-millisecond mark feels genuinely fast to a user; the identical 4-second answer delivered as one solid block at the 4-second mark feels broken and slow. This is the single highest-leverage latency change available anywhere in the entire stack, and notably it’s purely a UI and API-shape decision, not a model or infrastructure change at all.

Show the work during genuinely long operations. The agentic loop from Lesson 17 can legitimately take upward of 15 seconds to complete. Streaming its intermediate steps to the user, “Searching policies… reading 3 documents…”, converts what would otherwise feel like a hang into visibly productive work happening, which measurably changes perceived quality even though the actual wall-clock time hasn’t changed at all.

Trim the context down to what actually earns its place. Every chunk passed into the final prompt costs both generation time and real money, proportionally. Eight well-reranked, genuinely relevant chunks beat thirty mediocre ones on latency, on cost, and, per Lesson 1’s discussion of context rot, frequently on answer accuracy too.

Parallelize whatever can genuinely run concurrently. Sub-queries produced by decomposition in Lesson 13 can run concurrently rather than sequentially; both legs of hybrid retrieval already run concurrently by design. Any serial stage in your pipeline that could technically run in parallel but currently doesn’t is free latency you’re simply donating away for no benefit.

Three caches, two of them dangerous

1. Exact-match cache. Key: the normalized question text plus a hash of the asker’s principal set. This is trivially safe as long as the key is constructed correctly, and in real enterprise traffic the hit rate tends to be considerably better than intuition would suggest, since everyone in a company genuinely does ask about the holiday policy in the same week every December. Invalidate cache entries whenever any document in their cited evidence set changes, or more simply, set a short TTL and accept the resulting small staleness window as an acceptable tradeoff.

2. Semantic cache. Key: the query’s embedding vector, matched against stored entries by similarity above a chosen threshold. This achieves a meaningfully higher hit rate than exact matching, and it’s also the source of the most instructive security bug covered anywhere in this course:

# Semantic cache with the permission-aware key. The principal
# hash is not an optimization detail, it is the security boundary
# from Module 7 surviving contact with a performance feature.

def cache_key(question_vec, user: User) -> str:
    # Two users with DIFFERENT permissions must never share an
    # entry, because the cached answer was built from evidence
    # retrieved under one specific ACL.
    return hashlib.sha256(
        b"|".join(sorted(p.encode() for p in user.principals))
    ).hexdigest()

async def cached_answer(question: str, user: User) -> Answer | None:
    qv = await embed(question)
    hit = await cache.search(
        namespace=cache_key(qv, user),   # partition BY permissions
        vector=qv,
        min_similarity=0.97,             # strict: paraphrase, not topic
    )
    if not hit:
        return None
    # Documents change. A stale-but-similar answer is still wrong.
    if await any_source_changed(hit.cited_doc_ids, since=hit.created_at):
        await cache.invalidate(hit.id)
        return None
    return hit.answer

Two guards appear in that code, both learned the hard way by teams who shipped without them first. The namespace partitions the entire cache by resolved permission set, so this performance-motivated layer cannot quietly undo the careful permission work established in Module 7. And the similarity threshold is deliberately strict, 0.97 rather than something looser like 0.85: a loose threshold ends up serving the cached answer to a genuinely different question that merely sounds superficially similar, which users experience not as a helpful shortcut but as the system being confidently, randomly wrong in a way that’s hard to diagnose from the outside.

3. Prompt caching (provider-side). Stable prefixes, your system prompt, your standing instructions, your tool schemas, get billed at roughly a tenth of the normal input token rate on reuse across requests. This is close to free money sitting on the table in serving, and it’s structurally load-bearing back in ingestion too: it’s exactly what makes contextual retrieval from Lesson 8 cost a few thousand dollars for a large corpus instead of an absurd, prohibitive sum, because each source document effectively becomes a cached prefix reused across every one of its own chunks during enrichment.

Where the dollars go, and how to spend fewer of them

Ingestion is fundamentally a capital cost, paid once per corpus, or again on each full reindex event; serving is fundamentally an operating cost, paid continuously, per query, for as long as the system runs. They optimize along genuinely different dimensions, and it’s worth keeping that distinction explicit when deciding where to invest engineering time.

On serving, the single biggest lever available is model tiering: use a small, fast model for the purely mechanical stages, query rewriting, filter extraction, evidence grading inside the self-corrective loop, and reserve your most expensive frontier model specifically for final generation, which is where quality differences are actually visible to the end user. Most systems can push this further and tier generation itself by query class: simple lookups over clean, unambiguous evidence generally don’t need your most expensive model at all, and the evaluation suite from Lesson 22 is precisely the tool that proves which query classes can be safely tiered down without any measurable quality drop. From there, trim context to fewer, better chunks as discussed above, and let prompt caching absorb your stable, repeated prefixes automatically.

On ingestion, the levers are different in character: batch APIs (typically around half the price of synchronous calls, and nothing about ingestion, established repeatedly throughout this course, is genuinely latency-sensitive), quantization and Matryoshka dimension truncation from Lesson 9 (roughly 12x less vector storage, which directly translates into a recurring RAM bill that shrinks proportionally), and hashing before doing any real work, established back in Lesson 6 (never re-embed a document whose underlying text genuinely didn’t change, which on real-world corpora skips the clear majority of incoming change events that turn out to be metadata-only touches).

The discipline that ties all of this together, worth adopting as a standing practice rather than a one-time exercise: put cost and latency on the exact same dashboard as quality, broken down per query class, and require every proposed optimization to prove itself against the evaluation suite before it ships, exactly as any other change to the pipeline would be gated. Which leaves one final question for this course to address: how do you actually know when a system that’s running perfectly fine today has quietly started to rot underneath you? That’s the subject of the final lesson.