OCR & Document Intelligence

Reading Text from Images

You receive a scanned invoice. You need the vendor name, invoice number, date, and total amount, all entered into your accounting system automatically. The invoice exists as a JPEG. How do you get the data out?

This is the problem OCR (Optical Character Recognition) solves. OCR converts images containing text into machine-readable strings. It’s one of the most commercially valuable applications of computer vision, deployed everywhere from banking (reading cheques) to logistics (reading shipping labels) to healthcare (digitising paper records).

The challenge is harder than it sounds. Text in real-world images comes in:

• Dozens of different fonts
• Various sizes and colours
• Different languages and scripts
• Distorted by scanning, photographing, or printing imperfections
• Mixed with graphics, logos, and other visual noise

Modern deep learning OCR handles most of this: but preprocessing still makes an enormous difference.

How OCR Works Internally

At a high level, OCR pipelines do three things:

1. Text detection. Find where in the image text regions are. This is essentially object detection, but the “objects” are text blocks, lines, and words.

2. Character recognition. For each detected text region, classify what character(s) it contains. A deep learning model trained on millions of text samples recognises individual characters.

3. Language model post-processing. Use linguistic knowledge to correct errors. If a character was recognised as “rn”, a language model might know it’s more likely “m” based on surrounding words.

You don’t need to implement any of this, libraries handle it all. But knowing the stages helps you debug: if your text is detected correctly but recognised poorly, it’s a Stage 2 problem (preprocessing might help). If text regions are missed entirely, it’s a Stage 1 problem (resolution might be too low).

Method 1: Tesseract: The Classic

Tesseract is an open-source OCR engine originally developed by HP and now maintained by Google. It’s been around since 1985 and is the most widely used OCR tool in the world.

# Install the Python wrapper
pip install pytesseract pillow

# Also install the Tesseract binary (the actual OCR engine)
# Ubuntu/Debian:
sudo apt install tesseract-ocr
# macOS:
brew install tesseract
# Windows: download installer from GitHub

Basic usage: extract all text from an image:

import pytesseract
from PIL import Image

img = Image.open('invoice.jpg')
text = pytesseract.image_to_string(img)
print(text)

image_to_string runs the full OCR pipeline and returns the extracted text as a Python string. That’s it for basic usage, impressively simple.

For more control, get detailed information about each detected word and its position:

# Get detailed output: word, confidence, and bounding box
data = pytesseract.image_to_data(img, output_type=pytesseract.Output.DICT)

for i, word in enumerate(data['text']):
    if word.strip():   # skip empty strings
        conf = data['conf'][i]
        x, y, w, h = data['left'][i], data['top'][i], data['width'][i], data['height'][i]
        print(f"'{word}' at ({x},{y}), confidence: {conf}")

Confidence scores range 0-100. Words below 60 are often recognition errors.

Method 2: PaddleOCR: More Accurate

PaddleOCR (from Baidu) is a modern deep learning OCR library that consistently outperforms Tesseract on challenging documents: rotated text, complex layouts, handwriting, and multi-language content.

pip install paddlepaddle paddleocr

Setup and basic usage:

from paddleocr import PaddleOCR

# use_angle_cls=True: automatically handles rotated/upside-down text
# lang='en': specify the language (supports 80+ languages)
ocr = PaddleOCR(use_angle_cls=True, lang='en')

Creating the PaddleOCR object downloads the model weights on the first run. This takes a moment but only happens once.

# Run OCR on an image
result = ocr.ocr('invoice.jpg', cls=True)

The result is a nested list. Let’s unpack it clearly:

for block in result:       # result is a list of blocks
    for line in block:     # each block is a list of lines
        bbox = line[0]     # bounding box: [[x1,y1],[x2,y1],[x2,y2],[x1,y2]]
        text, confidence = line[1]
        print(f"'{text}', {confidence:.1%} confidence")
        print(f"  Position: top-left=({bbox[0][0]},{bbox[0][1]})")

PaddleOCR returns bounding boxes as four corner points (not x,y,w,h), which handles rotated text boxes. The confidence score tells you how reliable each recognised string is.

Preprocessing: The Hidden Multiplier

The difference between 70% OCR accuracy and 95% accuracy is often entirely in preprocessing, not the OCR model. Here’s why: OCR models are trained on clean, well-formatted text. When your input is a low-quality scan with shadows and skew, the model is doing its best with bad data.

Fix the data first, then run OCR.

import cv2
import numpy as np

def preprocess_for_ocr(image_path: str) -> np.ndarray:
    """Preprocess a document image for better OCR results."""
    img = cv2.imread(image_path)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    return gray

Start with grayscale. Most OCR models work on single-channel images.

    # Otsu's threshold: automatically separates text (dark) from background (light)
    # THRESH_OTSU finds the optimal threshold automatically
    _, binary = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)

Binarisation (black-and-white) makes text much crisper. Otsu’s method automatically finds the right threshold.

    # Remove "salt" noise: tiny random white specks
    # fastNlMeansDenoising: h=10 controls aggressiveness
    denoised = cv2.fastNlMeansDenoising(binary, h=10)
    
    return denoised

Denoising cleans up scanning artifacts. Return the cleaned image and use it as input to OCR.

Try preprocessing your document before OCR and compare the results, the improvement is often dramatic.

Extracting Structured Fields with Regex

OCR gives you raw text: a string like:

INVOICE
Acme Corp
123 Business Street
Invoice Number: INV-2024-0891
Date: March 15, 2024
Item           Qty    Price
Widget A        10    $25.00
Widget B         5    $48.00
                   TOTAL: $298.00

You need to extract specific fields: invoice number, date, and total. Regular expressions (regex) are the right tool, they find text patterns regardless of exact formatting.

import re

def extract_invoice_fields(ocr_text: str) -> dict:
    """Extract key fields from raw OCR text using regex patterns."""
    
    # Invoice number: look for "INV-" followed by digits and dashes
    inv_pattern = r'INV-[\d\-]+'
    inv_match = re.search(inv_pattern, ocr_text)
    invoice_number = inv_match.group() if inv_match else None
    return {'invoice_number': invoice_number}

Build on this with more patterns:

    # Date: look for common date formats
    date_pattern = r'\b\d{1,2}[\/\-]\d{1,2}[\/\-]\d{2,4}\b'
    date_match = re.search(date_pattern, ocr_text)
    
    # Total: look for "total" keyword followed by a currency amount
    total_pattern = r'(?:total|amount due)[\s:]*\$?([\d,]+\.?\d*)'
    total_match = re.search(total_pattern, ocr_text, re.IGNORECASE)
    
    return {
        'invoice_number': invoice_number,
        'date': date_match.group() if date_match else None,
        'total': total_match.group(1) if total_match else None,
    }

re.IGNORECASE makes “TOTAL”, “Total”, and “total” all match the same pattern. group(1) extracts just the captured group (the amount, without the “total:” prefix).

When to Use Cloud Document AI Instead

For high-volume production use, cloud document AI services often outperform local OCR:

• Google Document AI, state-of-the-art accuracy, pre-built models for invoices/receipts/contracts
• AWS Textract, automatically identifies form fields and tables
• Azure Form Recognizer, similar, with strong table extraction

Use local OCR (Tesseract, PaddleOCR) when: you have privacy requirements, low volume, or need offline capability. Use cloud services when: accuracy is critical, you need structured field extraction, or you’re processing standard document types.

Exercise: Download any invoice or receipt as a PDF or image (you can photograph a real receipt or find a sample online). Run it through PaddleOCR. Then write a regex to extract the total amount. How accurate is the OCR raw output? Does preprocessing improve it? Try the same image with and without preprocessing and compare character error rates.