Incremental Sync & Index Freshness

Lesson 1 called freshness a first-class constraint, alongside permissions, auditability, and scale. Here’s the uncomfortable arithmetic that makes it one: parsing plus chunking plus embedding a large corpus costs hours of compute time and real API dollars, as the previous two lessons made concrete. Do that nightly as a full rebuild, the naive approach most teams reach for first, and you get the worst of both worlds simultaneously: answers up to 24 hours stale, because the rebuild only runs once a day, and a five-figure monthly ingestion bill, most of it spent re-processing documents that did not actually change since yesterday’s run.

The fix is to stop treating ingestion as a batch job that runs on a clock and start treating it as a change stream that runs on events, which is a genuinely different architecture, not just a scheduling tweak.

Change-driven sync

Every serious source system offers some way to learn what changed without re-reading everything from scratch, and the specific mechanism varies by source but the pattern is consistent:

• Webhooks: Slack, Notion, GitHub, and Jira push change events to you within seconds of the change happening, which is the fastest and most efficient mechanism where it’s available.
• Delta APIs: Google Drive’s changes feed and Microsoft Graph’s delta queries return “everything since cursor X” in one cheap, paginated call, letting you poll efficiently even on sources without webhook support.
• Polling with cursors: for sources with neither webhooks nor a delta API, poll a modified_since listing on a short interval. Crude compared to the other two, but still meaningfully incremental compared to a full nightly crawl of everything.

Events land in a queue, and workers process one document at a time through the same parse-chunk-embed path built in the previous two lessons, unchanged. The queue absorbs bursts gracefully: a folder migration that suddenly emits thousands of change events at once doesn’t overwhelm the system, it just takes the workers longer to drain the backlog, and the queue gives you retries for free when an individual document fails to process.

flowchart LR
    W["Webhooks +\ndelta APIs"] --> Q[("Change queue")]
    P["Cursor polling\n(legacy sources)"] --> Q
    Q --> WK["Ingestion workers"]
    WK --> H{"Content hash\nchanged?"}
    H -- "No" --> M["Update metadata\n+ ACLs only"]
    H -- "Yes" --> PIPE["Parse, chunk,\nembed, upsert"]
    WK --> DEL{"Delete event?"}
    DEL -- "Yes" --> TOMB["Tombstone: remove chunks\nfrom every index"]

    style W fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style P fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style Q fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style WK fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style H fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style M fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style PIPE fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style DEL fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style TOMB fill:#EEF0F7,stroke:#6366F1,color:#0F172A

The three change types, and the one everyone forgets

Creates are easy: run the full pipeline, upsert the resulting chunks. There’s no prior state to reconcile against, so there’s little to get wrong here beyond ordinary error handling.

Updates hide a trap that catches teams who build this for the first time. You can’t just add the new version’s chunks alongside the old ones; the old version’s chunks must be removed, or a stale duplicate sits in the index forever, silently competing for retrieval rank against the current version and sometimes winning. The clean pattern is delete-then-insert keyed by doc_id: remove every chunk whose doc_id matches, then write the new set. Do it as a single transaction, or with a version tag that lets you filter atomically, so queries never see a half-updated document with half its chunks from the old version and half from the new. And hash before you do any of that expensive work, because a large share of “modified” events in practice change nothing about the actual text:

import hashlib

async def handle_update(event: ChangeEvent):
    raw = await fetch_from_source(event.doc_id)
    parsed = await parse(raw)  # the ParsedDoc from Lesson 4

    # Hash the normalized CONTENT, not the raw file bytes.
    # Raw bytes change when metadata changes; content does not.
    content = "\n".join(b.text for b in parsed.blocks)
    new_hash = hashlib.sha256(content.encode()).hexdigest()

    stored = await index.get_doc_meta(event.doc_id)

    if stored and stored.content_hash == new_hash:
        # Text unchanged: skip parse->embed cost entirely,
        # but ACLs or location may have moved, so sync those.
        await index.update_metadata(
            event.doc_id, acl_tags=parsed.acl_tags,
            url=parsed.url, updated_at=parsed.updated_at,
        )
        return

    # Text changed: atomically replace this doc's chunks.
    chunks = chunk(parsed)                  # Module 3
    vectors = await embed([c.text for c in chunks])
    await index.replace_doc(event.doc_id, chunks, vectors,
                            content_hash=new_hash)

Hashing normalized content rather than raw file bytes is the detail that makes this efficient in practice. Raw bytes change whenever anything about the file changes, including things that have nothing to do with the text: a permission grant, a folder move, a tag added by an automation bot. Content hashing sees straight through all of that noise and only triggers the expensive re-parse-chunk-embed path when the words a reader would actually see have genuinely changed.

Deletes are the change type everyone forgets, and the one with real legal teeth attached. A document deleted upstream, for any reason including a legal hold expiring or a compliance-driven purge, must be tombstoned out of the vector index, the lexical index, any caches holding derived answers, and any other derived artifacts built from it. Many delta APIs report deletions grudgingly, incompletely, or not at all, treating them as a second-class event type compared to creates and updates. So belt-and-suspenders systems also run a periodic reconciliation job: list all currently-live document IDs at the source, diff that list against what the index currently holds, and purge the orphans, the entries whose source document no longer exists anywhere. When legal deletes a document, “it’s still in the vector database somewhere” is not a sentence you want to be explaining to anyone.

Zero-downtime reindexing

Sometimes everything must be rebuilt at once, not incrementally: a new embedding model that isn’t compatible with existing vectors, a new chunking strategy, a new parser that handles tables better than the old one. This is the moment the incremental machinery above doesn’t help you, and you need a different pattern entirely. It’s blue-green, borrowed directly from web deployment practice:

1. Build index_v2 completely in the background, running the full corpus through the new pipeline, while index_v1 keeps serving all live traffic uninterrupted.
2. Run your evaluation suite from Module 8 against v2 before anyone sees it. This is the gate: no eval pass, no swap, full stop, regardless of how confident the change felt during development.
3. Point the serving alias at v2. This is one atomic configuration change, not a data migration, so it happens with zero downtime.
4. Keep v1 around for a day or two after the swap. Rollback, if the new version misbehaves under real traffic in a way the eval suite didn’t catch, is simply repointing the alias back.

One detail that trips up teams running this for the first time: keep live change events flowing into both indexes during the build phase, not just v1. Otherwise v2 launches already stale by however long the build took, sometimes hours, which defeats a meaningful part of the point of doing a careful, validated cutover.

Freshness earns a number on your dashboard

All of this machinery exists to serve one measurable outcome: sync lag, the elapsed time from a source change happening to that change being reflected in a queryable chunk. This number belongs on your operational dashboard from day one, not added later once someone complains about stale answers. Webhook-backed sources should sit comfortably under a minute of lag in steady state; polled sources should sit under their configured poll interval, with some margin. When lag spikes, meaning the queue is backing up or a source’s webhook silently stopped firing, you want an alert firing on your dashboard, not a user report arriving in a support channel days later as the first signal something went wrong.

With documents now flowing in clean, current, and correctly tombstoned when removed, Module 3 turns to the question that decides retrieval quality more than any single model choice you’ll make: how to cut these documents into chunks that are actually findable.