VLM Parsing & Modern OCR

Lesson 4 handled the 90-something percent of documents with a real embedded text layer, where a fast structural parser does the whole job in milliseconds. This lesson handles the rest: scans, faxes (yes, still, especially in healthcare and legal), photographed whiteboards, stamped invoices, handwritten margins, and PDFs whose layout defeats even a good structural parser. In 2023 this was Tesseract territory, and results on anything but clean printed text were genuinely rough. By 2025, vision-language models rewrote the category almost entirely.

What VLMs changed

Classical OCR answers a narrow question: “what characters are on this page and where are they positioned.” Everything else, columns, tables, reading order, distinguishing a signature from a stamp from a handwritten note, was left as your problem to solve downstream. Vision-language models answer a fundamentally better question: “what does this page say, as a structured document.” They read the page holistically, the way a person would, and can emit markdown with intact tables directly, without a separate structure-reconstruction step. Handwriting, rotation, poor contrast, coffee stains, and mixed languages on the same page degrade them far less than they degraded classical OCR engines, because the model is reasoning about the whole image rather than classifying isolated character glyphs one at a time.

The 2025 wave made this practical at real corpus scale, not just as a demo. Mistral OCR shipped as a dedicated document-understanding API priced per page specifically for this workload. The general frontier models, the Gemini and GPT families and Claude, all became good enough at document images that many teams simply route hard pages to whichever model provider they already have a relationship with, rather than standing up a separate specialized service. Open-weight options, the Qwen-VL derivatives and purpose-built small document models like the DeepSeek-OCR line, cover the case where document text genuinely cannot leave your infrastructure, running on modest local GPU hardware.

Prompt for structure, not prose

The biggest practical lever, and the one teams underuse, is the prompt itself. Ask a VLM “what does this document say?” and you get a chatty summary that loses exactly the structural detail your pipeline needs. Ask it for a transcription contract instead, with explicit rules, and you get pipeline-grade output that slots directly into the same document model from Lesson 4:

# VLM parsing call: the prompt is a transcription CONTRACT,
# not an open question. Works with any multimodal API.

PARSE_PROMPT = """Transcribe this document page to markdown.

Rules:
- Preserve the reading order a careful human would use.
- Render every table as a markdown table. Never summarize
  or omit rows. Keep cell values exactly as printed.
- Mark headings with #, ##, ### matching their visual level.
- Transcribe handwriting where legible. If a word is not
  legible, write [illegible], never a guess.
- Do not add commentary, descriptions, or summaries.
- If part of the page is a photo or diagram, insert
  [figure: one-line description] at its position."""

async def vlm_parse_page(page_image: bytes) -> str:
    response = await client.messages.create(
        model="claude-sonnet-5",
        max_tokens=4096,
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": b64(page_image),
                }},
                {"type": "text", "text": PARSE_PROMPT},
            ],
        }],
    )
    return response.content[0].text

The [illegible] rule is doing quiet, important work that’s easy to overlook. VLMs are fundamentally generative models: faced with ambiguous pixels, they emit the most plausible text rather than admitting uncertainty, exactly as a language model completes a sentence with the most likely next words. A plausible wrong invoice total is more dangerous to your downstream users than an obviously broken OCR artifact, because a broken artifact gets caught and questioned while a plausible wrong number gets trusted and acted on. Giving the model explicit, prompted permission to say “I don’t know” for a specific word measurably reduces this failure mode in practice.

Route by cost tier

A frontier VLM pass costs real money and real seconds per page, both of which add up fast at corpus scale. Docling’s structural parse from Lesson 4 costs milliseconds and effectively nothing. The production pattern is a routing tier that spends the expensive path only where it actually pays for itself, keeping the vast majority of your corpus on the cheap path:

flowchart TD
    IN["Incoming document"] --> T{"Has embedded\ntext layer?"}
    T -- "Yes" --> DL["Docling structural parse\n(fast, local, free)"]
    DL --> C{"Layout confidence OK?\nTables plausible?"}
    C -- "Yes" --> OUT["Normalized ParsedDoc"]
    C -- "No" --> VLM["VLM parse\n(per-page, paid)"]
    T -- "No (scan/photo)" --> VLM
    VLM --> HS{"High-stakes doc with\nlow-confidence fields?"}
    HS -- "Yes" --> REV["Human review queue"]
    HS -- "No" --> OUT
    REV --> OUT

    style IN fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style T fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style DL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style C fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style VLM fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style HS fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style REV fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style OUT fill:#f0fdf9,stroke:#0D9488,color:#0F172A

Both branches emit the same ParsedDoc schema defined in Lesson 4. This is a deliberate architectural choice, not a coincidence: downstream stages, chunking, embedding, indexing, never know or care which parsing path a given page took, which means you can change the routing thresholds or swap VLM providers without touching a single line of Module 3’s code. In a typical corporate corpus, somewhere between 4 and 10 percent of pages end up on the VLM branch, which keeps per-corpus parsing cost in the hundreds of dollars rather than the tens of thousands a naive “VLM everything” approach would run up.

Two practical notes from the trenches

Resolution is a real cost dial, not a free quality knob. Render pages at roughly 150 to 200 DPI for VLM input. Below that range, small fonts start to smear and even a strong VLM begins to misread digits and punctuation. Above it, you pay meaningfully more in image tokens for detail the model’s vision encoder generally cannot use any better, since transcription accuracy on real documents tends to plateau well before you reach print-quality resolutions. Test this on a sample of your own hardest documents (dense tables, small print) rather than trusting a single blanket setting.

Batch APIs fit ingestion perfectly, and using the synchronous endpoint here is leaving money on the table. Most providers discount asynchronous batch processing by roughly half versus their synchronous per-token pricing, and nothing on the ingestion plane, as established in Lesson 2, has a latency deadline. Submitting a batch job for a backlog of scanned documents and collecting results over the following hours costs meaningfully less than looping through the same documents with synchronous calls, for identical output quality.

With clean, structured, correctly routed documents now flowing out of parsing in both its fast and expensive forms, the next problem is keeping that pipeline fed with current data as source documents keep changing underneath you, which is exactly where Lesson 6 picks up.