Capstone: Invoice OCR + Defect Detector

Putting It All Together

You’ve learned pixels, preprocessing, CNNs, detection, segmentation, OCR, and APIs. This capstone module ties everything together into two complete, real-world projects.

Project 1: Invoice Data Extractor: given an image of an invoice, output a structured dictionary with the invoice number, date, vendor name, and total amount. This represents millions of real business workflows where paper documents need to be digitised.

Project 2: Product Defect Classifier, given a product photo, classify it as “good, ” “scratched, ” or “dented.” Train the model, evaluate it properly, and wrap it in a production inference function. This represents the quality control CV use case in manufacturing.

Let’s build both.

Project 1: Invoice Data Extractor

Step 1: Preprocess the Document

The first step is always cleaning the image. For document OCR, this means converting to grayscale, binarising, and denoising:

import cv2
import numpy as np

def preprocess_document(image_path: str) -> np.ndarray:
    """Prepare a document image for OCR."""
    img = cv2.imread(image_path)
    
    # Convert to grayscale: single channel is cleaner for OCR
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    
    return gray

Now add binarisation: converting to pure black-and-white:

    # Otsu's method finds the optimal threshold automatically
    # THRESH_BINARY: text (dark) becomes black, background becomes white
    _, binary = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
    
    # Denoise: remove scanning artifacts
    denoised = cv2.fastNlMeansDenoising(binary, h=10)
    
    return denoised

These two preprocessing steps frequently improve OCR accuracy by 20-30 percentage points on real scanned documents.

Step 2: Run OCR

Pass the preprocessed image to PaddleOCR:

from paddleocr import PaddleOCR
import cv2, tempfile, os

ocr_engine = PaddleOCR(use_angle_cls=True, lang='en')

def extract_text(image_path: str) -> str:
    """Extract all text from a document image."""
    # Preprocess first
    cleaned = preprocess_document(image_path)
    
    # Save preprocessed image to temp file (PaddleOCR takes a path)
    with tempfile.NamedTemporaryFile(suffix='.jpg', delete=False) as tmp:
        cv2.imwrite(tmp.name, cleaned)
        tmp_path = tmp.name
    
    result = ocr_engine.ocr(tmp_path, cls=True)
    os.unlink(tmp_path)   # clean up temp file

Now assemble all recognised text into a single string:

    all_lines = []
    for block in result:
        for line in block:
            text, confidence = line[1]
            if confidence > 0.7:   # only include high-confidence text
                all_lines.append(text)
    
    return '\n'.join(all_lines)

We only keep text with confidence above 70%. Low-confidence OCR output is often garbled and can confuse downstream extraction.

Step 3: Extract Structured Fields

Use regular expressions to find specific fields in the raw OCR text:

import re

def extract_invoice_fields(raw_text: str) -> dict:
    """Parse key invoice fields from raw OCR text."""
    
    # Invoice number: look for patterns like INV-001, #2024-891, etc.
    inv_match = re.search(
        r'(?:invoice|inv|#)\s*[:\-]?\s*([A-Z0-9][\w\-]{2,20})',
        raw_text, re.IGNORECASE
    )
    invoice_number = inv_match.group(1) if inv_match else None

Each field gets its own pattern. The key is to look for a nearby keyword (like “invoice” or “total”) that signals what the number means:

    # Date: common formats like 15/03/2024 or March 15, 2024
    date_match = re.search(
        r'\b(\d{1,2}[\/\-]\d{1,2}[\/\-]\d{2,4}|\w+ \d{1,2},\s*\d{4})\b',
        raw_text
    )
    
    # Total amount: look for "total" near a currency amount
    total_match = re.search(
        r'(?:total|amount due|grand total)[\s:]*\$?([\d,]+\.?\d*)',
        raw_text, re.IGNORECASE
    )
    
    # Vendor: look for company name indicators
    vendor_match = re.search(
        r'(?:from|vendor|bill from|company)[\s:]+([A-Z][A-Za-z\s]+(?:Ltd|LLC|Inc|Corp)?)',
        raw_text, re.IGNORECASE
    )
    
    return {
        'invoice_number': invoice_number,
        'date':           date_match.group(1).strip() if date_match else None,
        'total':          total_match.group(1).replace(',', '') if total_match else None,
        'vendor':         vendor_match.group(1).strip() if vendor_match else None,
    }

Step 4: The Complete Pipeline Function

Wire it all together into one clean function:

import json

def process_invoice(image_path: str) -> dict:
    """
    Full pipeline: image path → structured invoice data.
    Returns a dict with extracted fields, or None for fields not found.
    """
    print(f"Processing: {image_path}")
    
    raw_text = extract_text(image_path)
    print(f"Extracted {len(raw_text)} characters of text")
    
    fields = extract_invoice_fields(raw_text)
    
    # Add extraction metadata
    fields['_source_file'] = image_path
    fields['_raw_text_preview'] = raw_text[:200]
    
    return fields

# Run it
result = process_invoice('invoice.jpg')
print(json.dumps(result, indent=2))

Project 2: Product Defect Classifier

Dataset Structure

Organise your labelled images in PyTorch’s expected folder structure:

data/
  train/
    good/        (e.g., 300 images of undamaged products)
    scratched/   (e.g., 100 images of scratched products)
    dented/      (e.g., 80 images of dented products)
  val/
    good/        (e.g., 80 images)
    scratched/   (e.g., 25 images)
    dented/      (e.g., 20 images)
  test/           ← NEVER touch this during training
    good/
    scratched/
    dented/

The test set is sacred: you look at its performance only once, after you’ve finished all training decisions. This simulates real deployment performance.

Training

import torch, torch.nn as nn
from torchvision import models, transforms
from torch.utils.data import DataLoader, ImageFolder

# Load pretrained EfficientNet-B2
model = models.efficientnet_b2(weights='IMAGENET1K_V1')
CLASS_NAMES = ['dented', 'good', 'scratched']

# Replace the head for 3 classes
model.classifier[1] = nn.Linear(model.classifier[1].in_features, len(CLASS_NAMES))

# Freeze backbone for Stage 1
for param in model.features.parameters():
    param.requires_grad = False

Set up data loaders:

train_transform = transforms.Compose([
    transforms.Resize((260, 260)),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(10),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

val_transform = transforms.Compose([
    transforms.Resize((260, 260)),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

train_loader = DataLoader(ImageFolder('data/train', transform=train_transform), batch_size=32, shuffle=True)
val_loader   = DataLoader(ImageFolder('data/val',   transform=val_transform),   batch_size=32)

Train Stage 1 (head only):

device    = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model     = model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.classifier.parameters(), lr=1e-3)

for epoch in range(10):
    model.train()
    total_loss = 0
    for images, labels in train_loader:
        images, labels = images.to(device), labels.to(device)
        optimizer.zero_grad()
        loss = criterion(model(images), labels)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    print(f"Epoch {epoch+1}: loss={total_loss/len(train_loader):.4f}")

Evaluation with a Confusion Matrix

Don’t stop at “80% accuracy.” Look at a confusion matrix to understand exactly which mistakes the model makes:

from sklearn.metrics import confusion_matrix, classification_report
import numpy as np

model.eval()
all_preds, all_labels = [], []

with torch.no_grad():
    for images, labels in val_loader:
        outputs = model(images.to(device))
        preds   = outputs.argmax(dim=1).cpu().numpy()
        all_preds.extend(preds)
        all_labels.extend(labels.numpy())

cm = confusion_matrix(all_labels, all_preds)
print("Confusion Matrix:")
print(f"{'':>10}", '  '.join(CLASS_NAMES))
for i, row in enumerate(cm):
    print(f"{CLASS_NAMES[i]:>10}", '  '.join(f'{v:>8}' for v in row))

The confusion matrix rows are actual classes; columns are predicted classes. Read the diagonal for correct predictions. Off-diagonal elements are mistakes. For a defect detector, pay special attention to the “defective classified as good” cells, these are the dangerous misses.

Also print the full classification report:

print("\nDetailed Report:")
print(classification_report(all_labels, all_preds, target_names=CLASS_NAMES))

This shows precision, recall, and F1 score per class, the metrics that matter most in imbalanced classification.

Save and Use Your Model

# Save the trained weights
torch.save(model.state_dict(), 'defect_classifier.pth')

# Inference function: ready to use in production
def classify_product(image_path: str) -> dict:
    """Classify a product image. Returns prediction and confidence scores."""
    from PIL import Image
    
    img = Image.open(image_path).convert('RGB')
    tensor = val_transform(img).unsqueeze(0).to(device)
    
    model.eval()
    with torch.no_grad():
        probs = torch.softmax(model(tensor), dim=1)[0]
    
    best_idx = probs.argmax().item()
    return {
        'prediction': CLASS_NAMES[best_idx],
        'confidence': round(probs[best_idx].item(), 3),
        'all_scores': {name: round(probs[i].item(), 3) for i, name in enumerate(CLASS_NAMES)},
    }

# Test it
result = classify_product('new_product.jpg')
print(result)
# {'prediction': 'good', 'confidence': 0.981, 'all_scores': {...}}

Congratulations, you’ve completed the Vision AI course and built two production-ready pipelines. The skills you’ve used here: preprocessing, transfer learning, evaluation, and inference APIs, are the same ones used in real commercial systems.

Exercise: Choose one of the two projects and push it further. For the invoice pipeline: try it on 5 different invoice images (you can find samples online). What percentage of fields does it extract correctly? For the defect classifier: try to get images of a real product or use a public dataset (MVTec AD on Kaggle). Train and evaluate, what’s your test set accuracy? Which defect type is hardest to classify?