Every RAG failure post-mortem eventually reaches the same sentence: “we looked at what was actually in the index.” Wrong prices from shredded tables, policies attributed to the wrong section, footers stitched into the middle of paragraphs, entire tables reduced to a soup of numbers with no row or column meaning left. Retrieval and generation were fine, tuned carefully according to every later module in this course. The data was ruined at minute one, by parsing, and nothing downstream could recover what parsing threw away.
This lesson is about not ruining it, and it’s deliberately the first hands-on lesson in the course, because everything else assumes clean, structured input exists to work with.
Why PDFs fight you
A PDF is not a document in the semantic sense, whatever its file extension suggests. It’s a set of drawing instructions: place glyph, move cursor, draw line, fill rectangle. Reading order, paragraphs, tables, headers, footnotes: none of that exists in the format as data. It exists only in the visual arrangement on the page, and a parser has to reconstruct meaning from pixels and coordinates, which makes this fundamentally a computer vision and layout-analysis problem as much as a text-extraction one.
Naive extractors, the get_text() school of PDF handling that most quick-and-dirty scripts still use, read glyphs in stream order or raw coordinate order. On a clean single-column report with no tables, they mostly work, which is exactly why teams underestimate the problem until they hit a real corpus. On the documents enterprises actually have, full of tables, sidebars, and multi-column layouts, they fail in ways that are invisible until an answer built from their output is confidently wrong:
- Multi-column layouts interleave: line one of column A, line one of column B, line two of column A, producing a document that reads as gibberish to both humans and embedding models.
- Tables collapse into a run of cell values with no row or column boundaries, so a three-column pricing table becomes an unstructured sequence of numbers that could mean almost anything.
- Headers and footers repeat on every page and get spliced into sentences mid-paragraph, since the extractor has no concept of “this text repeats on every page and is furniture, not content.”
- Reading order breaks around figures, sidebars, pull quotes, and text boxes, often interleaving a caption with the paragraph that follows it on the page.
- Footnotes and endnotes get pulled out of their anchor context entirely, floating as isolated fragments with no link back to the claim they were qualifying.
Layout-aware parsing with Docling
By 2026 the open-source default for this job is Docling (started at IBM Research, now a Linux Foundation AI project with wide community adoption). It runs layout detection and table-structure recognition models over each page, reconstructs reading order using the detected layout regions rather than raw coordinate order, classifies blocks by type, and exports clean structure. It handles PDF, DOCX, PPTX, XLSX, HTML, and images through one consistent interface, and it runs entirely locally, which matters enormously when documents contractually or legally cannot leave your infrastructure to reach a hosted API.
# pip install docling
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
# Docling runs layout analysis + table structure recognition
# (its TableFormer model) on each page, then rebuilds the
# document as a tree of typed blocks in reading order.
result = converter.convert("q3-pricing-policy.pdf")
doc = result.document
# Export 1: markdown for humans, debugging, and LLM prompts.
# Tables come out as real markdown tables, not glyph soup.
markdown = doc.export_to_markdown()
# Export 2: the structured tree for your pipeline. Walk it to
# build your own document model with types and hierarchy intact.
for item, level in doc.iterate_items():
# item.label is the block type: section_header, text,
# table, list_item, picture, code, footnote...
print(level * " ", item.label, getattr(item, "text", "")[:60])The exported markdown is what you’ll feed to chunking in Module 3, and it’s genuinely readable if you print it out mid-pipeline for debugging, which is worth doing on your first ten documents to build intuition for what “good parsing” looks like on your actual corpus. The structured tree, walked with iterate_items(), is what you’ll use to build the metadata every later lesson depends on: section paths for context, page numbers for citations, table boundaries for atomic chunking.
Normalize everything into one document model
Docling covers files well. But your corpus also has Confluence pages arriving as HTML, Slack threads arriving as JSON with a completely different shape, Notion exports, and ticket bodies from your helpdesk system. Handling each source format with its own bespoke downstream logic is how ingestion pipelines become unmaintainable within a year. The winning move, and the one every subsequent module in this course assumes you made, is to normalize every source into one internal schema before chunking, so every downstream stage handles exactly one shape regardless of where the document came from:
# The document model every connector must produce.
# One schema in, regardless of source format.
from dataclasses import dataclass, field
@dataclass
class Block:
type: str # "heading" | "paragraph" | "table" | "list" | "code"
text: str # markdown-rendered content of the block
level: int = 0 # heading depth, 0 for non-headings
page: int | None = None # provenance for citations
@dataclass
class ParsedDoc:
doc_id: str # stable ID from the source system
source: str # "confluence" | "drive" | "slack" | ...
title: str
url: str # deep link users can open from a citation
updated_at: str # ISO timestamp, drives freshness logic
acl_tags: list[str] # who may see this (Module 7 depends on this!)
blocks: list[Block] = field(default_factory=list)Two fields here quietly carry half the course’s later material, and skipping them at parse time means re-crawling every single source later to backfill them, which on a large corpus is a multi-day operation you want to avoid ever repeating. updated_at powers incremental sync in Lesson 6, telling the system when a document last changed so it knows whether to re-process it. acl_tags powers permission-aware retrieval in Module 7, and it is the single field that, if missing or wrong, turns your knowledge assistant into a data leak.
Applying this to a non-PDF source is mostly a matter of writing a thin adapter. A Confluence connector, for instance, walks the page’s HTML (or uses the Confluence API’s structured content model directly, which is cleaner where available), maps <h2> to a heading block at level 2, maps <table> to a table block, and maps the page’s own last-modified timestamp and space-level permissions into updated_at and acl_tags. The adapter is source-specific; the output it produces is not, and that’s the entire point.
Practical rules that save reindexing later
Strip repeated furniture. Detect headers and footers by cross-page repetition (the same text appearing in the same position across most pages of a document) and drop them from the block stream entirely. They add noise to embeddings and waste tokens in every prompt they sneak into, and worse, a footer containing a copyright year can confuse a chunk about which year the surrounding content actually refers to.
Keep tables intact and typed. A table exported as markdown, with its caption and nearest preceding heading attached as context, is one of the most retrieval-valuable objects in an enterprise corpus, because tables are exactly where the specific numbers users search for tend to live. Never let a chunker split a table across chunk boundaries; Module 3 treats tables as atomic units for precisely this reason.
Preserve provenance per block. Page number and section path per block make span-level citations possible later, in Module 6. Users trust “page 12, Section 4.2, updated March 2026” far more than a bare document link with no indication of where in a 40-page document the relevant text actually sits, and that trust difference shows up directly in whether people click citations to verify an answer.
Route by confidence. Docling reports when a page has no extractable text layer at all, or when layout confidence is low (a common signal on documents with unusual layouts, heavy scanning artifacts, or handwritten annotations mixed with print). Send those pages to the VLM path covered in the next lesson, rather than accepting a low-confidence parse into your index silently. A parsing pipeline that fails loudly on hard pages is more valuable than one that fails quietly and lets bad chunks through.