Image Processing Fundamentals with OpenCV

What Is Image Processing (and Why Do We Need It)?

Before feeding an image to a neural network or running any detection algorithm, you almost always need to pre-process it. Image processing is the step where you prepare raw visual data so that downstream algorithms can work better.

Think of it like preparing ingredients before cooking. A good chef doesn’t throw raw, unwashed vegetables directly into the pot. They clean, chop, and season first. Similarly, images fresh from a camera often have noise, inconsistent sizes, or distracting details that would confuse a model. Preprocessing fixes these problems.

There are two flavours of image processing:

Classical processing applies fixed mathematical rules, blur this, detect edges here, threshold at this value. The rules are written by humans and don’t change with data. Fast, predictable, and interpretable.

Deep learning learns the rules from data automatically. Powerful and flexible, but needs labelled examples.

In practice, you use both together: classical processing for preprocessing and post-processing, deep learning for the core prediction. This module covers the classical toolkit every CV engineer needs.

The Tool: OpenCV

OpenCV (Open Source Computer Vision Library) is the standard library for image processing in Python. It’s been around since 1999 and contains hundreds of functions for every image processing task imaginable. Installation is simple:

pip install opencv-python

Every example in this module uses OpenCV together with NumPy.

Noise Reduction: Why Images Are Never Perfectly Clean

Real images contain noise, random variations in pixel values caused by camera sensors, compression, or poor lighting. Zoom into any photo at 400% and you’ll see individual pixels that look slightly different from their neighbours for no visual reason. This noise causes problems for algorithms that look for patterns.

Gaussian blur is the standard solution. It works by replacing each pixel with a weighted average of itself and its neighbours, pixels closer to the centre contribute more to the average. The result is a smoothed image where random noise is averaged out.

import cv2
import numpy as np

img = cv2.imread('photo.jpg')
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

We load the image and convert to grayscale. Most classical processing algorithms work on single-channel images.

# Apply Gaussian blur
# (5, 5) is the kernel size: must be odd numbers
# 0 means "calculate sigma automatically from kernel size"
blurred = cv2.GaussianBlur(img_gray, (5, 5), 0)

The kernel size (5, 5) controls how much blurring happens. A 5×5 kernel means each pixel is averaged with a 5×5 neighbourhood of pixels. Larger kernels = more blurring. Try (3, 3) for mild smoothing or (15, 15) for heavy blurring.

Edge Detection: Finding Where Things End

An edge in an image is a place where pixel brightness changes rapidly, the boundary between a dark object and a light background, for example. Detecting edges is one of the most fundamental operations in CV, used for everything from reading text to detecting objects.

The Canny algorithm is the gold standard for edge detection. It works in four steps internally:

1. Blur the image (to remove noise: this is why we blurred first)
2. Find the gradient (how much and in what direction brightness changes at each pixel)
3. Keep only pixels that are local maxima in gradient direction (thin the edges)
4. Apply two thresholds to keep only strong, well-connected edges

# Canny edge detection
# threshold1=50: lower bound: edges weaker than this are discarded
# threshold2=150: upper bound: edges stronger than this are definitely kept
# Edges between 50-150 are kept only if connected to strong edges
edges = cv2.Canny(blurred, threshold1=50, threshold2=150)

The two thresholds give you a lot of control. Lower both values and you’ll detect more edges (including noise). Raise them and you’ll only detect strong, clear boundaries. A ratio of 1:3 (like 50 and 150) is a common starting point.

The output edges is a binary image: white (255) at edge pixels, black (0) everywhere else.

Thresholding: Creating Clean Black-and-White Images

Thresholding converts a grayscale image to pure black-and-white: every pixel either becomes 0 (black) or 255 (white) based on whether it was above or below a cutoff value.

This is incredibly useful when you want to separate a foreground object from a background, for example, isolating text on a document page, or separating a product from its packaging.

# Simple global threshold
# If pixel > 127: set to 255 (white)
# If pixel <= 127: set to 0 (black)
ret, binary = cv2.threshold(img_gray, 127, 255, cv2.THRESH_BINARY)

cv2.threshold returns two things, the threshold value used (stored in ret) and the resulting binary image (stored in binary). The _ in _, binary = ... is a Python convention for “I’m not using this value.”

For documents or images with uneven lighting, Otsu’s method automatically finds the optimal threshold:

# Otsu's method: automatically calculates best threshold
ret, binary_otsu = cv2.threshold(img_gray, 0, 255,
                                  cv2.THRESH_BINARY + cv2.THRESH_OTSU)
print(f"Otsu chose threshold: {ret}")  # e.g., 127.0

Otsu analyses the pixel value distribution and finds the threshold that best separates the two groups (foreground and background). For document OCR, always try Otsu first.

Morphological Operations: Cleaning Up Masks

After thresholding or masking, you often get a messy binary image with small holes, stray dots, and jagged edges. Morphological operations are a set of tools for cleaning this up.

They work by moving a small shape (called a structuring element or kernel) over every pixel and applying a rule.

First, define a kernel:

# A 5×5 square of ones: the shape moved over the image
kernel = np.ones((5, 5), np.uint8)

Dilation expands white regions. It turns nearby black pixels white. Useful for filling small holes inside objects and connecting nearby components:

dilated = cv2.dilate(binary, kernel, iterations=1)

Erosion shrinks white regions. It turns white pixels black if they’re too close to a black region. Useful for removing tiny noise specks and thinning objects:

eroded = cv2.erode(dilated, kernel, iterations=1)

A very common pattern is opening (erosion then dilation) to remove noise while preserving shape, or closing (dilation then erosion) to fill holes while preserving size:

# Opening: removes small noise blobs
opened = cv2.morphologyEx(binary, cv2.MORPH_OPEN, kernel)

# Closing: fills small holes
closed = cv2.morphologyEx(binary, cv2.MORPH_CLOSE, kernel)

Geometric Transforms: Resize and Rotate

Neural networks expect a fixed input size, so resizing is always part of your pipeline:

# Resize to 224×224 (note: width first, then height)
resized = cv2.resize(img, (224, 224))

# Resize while preserving aspect ratio (scale to height=224)
h, w = img.shape[:2]
scale = 224 / h
resized_proportional = cv2.resize(img, (int(w * scale), 224))

Rotation is useful for data augmentation during training:

# Get the centre point of the image
center = (img.shape[1] // 2, img.shape[0] // 2)

# Build a rotation matrix: rotate 45 degrees, no scaling
M = cv2.getRotationMatrix2D(center, angle=45, scale=1.0)

# Apply the rotation
rotated = cv2.warpAffine(img, M, (img.shape[1], img.shape[0]))

Putting It All Together: A Preprocessing Pipeline

Here’s a complete, reusable function that standardises images before feeding them to a neural network. This is the kind of function that lives at the beginning of every production CV pipeline:

def preprocess_for_model(image_path: str, target_size=(224, 224)) -> np.ndarray:
    """Load and preprocess an image for neural network inference."""
    # Step 1: Load
    img = cv2.imread(image_path)
    if img is None:
        raise ValueError(f"Could not load image: {image_path}")
    
    # Step 2: Convert BGR → RGB
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    
    # Step 3: Resize to model's expected input size
    img = cv2.resize(img, target_size)
    
    return img

Now add normalisation, neural networks expect pixel values in a specific range, not raw 0-255:

def normalise(img: np.ndarray) -> np.ndarray:
    """Normalise to ImageNet statistics (required for pretrained models)."""
    img = img.astype(np.float32) / 255.0  # scale 0-255 → 0.0-1.0
    
    # Subtract mean and divide by std (ImageNet statistics)
    mean = np.array([0.485, 0.456, 0.406])
    std  = np.array([0.229, 0.224, 0.225])
    img  = (img - mean) / std
    
    return img

The ImageNet mean and standard deviation values ([0.485, 0.456, 0.406] and [0.229, 0.224, 0.225]) are fixed constants, they were computed from the 1.2 million ImageNet training images. Any pretrained model trained on ImageNet expects these exact statistics. Using the wrong values is a very common cause of poor inference performance.

Exercise: Load a photo and apply the full pipeline: convert to grayscale → Gaussian blur → Canny edges. Experiment with the two Canny threshold values, what happens when you set both very low (like 10 and 30)? What about very high (like 200 and 250)? Try also applying Otsu thresholding to a document scan and see how well it isolates the text.