Convolutional Neural Networks (CNNs)

Why Can’t We Just Use Regular Neural Networks?

Imagine you wanted to build a neural network to classify photos of cats and dogs. The most obvious approach, a regular “fully connected” neural network, would treat an image as a long list of numbers: 224×224×3 = 150,528 individual values all fed in at once.

The problem? Two massive issues.

Issue 1: Too many parameters. A single hidden layer with 1000 neurons connecting to 150,528 inputs needs 150 million parameters. That’s just one layer. The network would take forever to train and would likely memorise the training data rather than learn anything useful.

Issue 2: No spatial awareness. A regular network treats the pixel at position (0, 0) and the pixel at position (100, 50) as completely unrelated. But that’s nonsense, a cat’s eye at the top-left of a photo is still a cat’s eye. The spatial arrangement of pixels matters enormously, and regular networks throw that information away.

Convolutional Neural Networks (CNNs) solve both problems elegantly. Instead of connecting every neuron to every pixel, CNNs use small filters that slide across the image, looking for patterns wherever they appear.

The Core Idea: Filters That Slide

A filter (also called a kernel) is a small grid of numbers, typically 3×3 or 5×5. The filter slides across the entire image, one position at a time, and at each position computes the dot product between itself and the patch of image it’s currently over.

Here’s an intuition for what this looks like:

Patch of image:    Filter (3×3):      Output value:
[120, 130, 110]   [0,  1,  0]
[125, 200,  90] · [1, -4,  1]  →  -680  (large negative = sharp brightness change)
[115, 140, 100]   [0,  1,  0]

The output is high (positive or negative) when the image patch matches the filter pattern, and near-zero when it doesn’t. By learning different filters, a CNN learns to detect different features:

• One filter might respond strongly to vertical edges
• Another to horizontal lines
• Another to a specific colour gradient

A layer with 64 filters produces 64 different “feature maps”, one per filter, each highlighting where in the image that filter’s pattern was found. The crucial insight: a filter uses the same weights everywhere in the image, so it detects the same feature regardless of where in the image it appears.

From Edges to Faces: Hierarchical Features

CNNs stack these convolutional layers. Early layers detect simple things; later layers combine them into increasingly complex concepts:

• Layer 1: Edges at different angles, colour gradients, simple blobs
• Layer 3-5: Textures, corners, basic shapes
• Layer 8-12: Eyes, wheels, windows, recognisable parts of objects
• Final layers: Faces, cars, cats, full object categories

This hierarchical feature learning is the magic that makes CNNs so powerful. No one programmed “a face has two eyes, a nose, and a mouth.” The network discovered this structure from millions of training examples.

Pooling: Compressing What We Know

After each convolutional layer, the feature maps are often large. Max pooling reduces their size while keeping the important information: it takes the maximum value in each small region.

import torch.nn as nn

# Max pooling with 2×2 window, stride 2
# Result: halves both height and width
max_pool = nn.MaxPool2d(kernel_size=2, stride=2)

Think of it like glancing at a photo from further away. You lose fine detail, but the important shapes and patterns remain visible. This makes subsequent layers faster and gives the network some tolerance for small shifts in object position.

Batch Normalisation and Skip Connections

Two more key ingredients in modern CNNs:

Batch Normalisation stabilises training by normalising the activations within each batch. Without it, values can grow wildly large or small as they pass through many layers, making training unstable. It’s almost always placed after convolution and before the activation function.

Skip connections (invented in ResNet) are even more clever. Instead of learning a direct transformation output = layer(input), the layer learns a correction: output = layer(input) + input. Adding the input directly creates a “shortcut” that lets gradients flow back to early layers without disappearing. This is why networks can now have 50, 100, or even 150 layers, skip connections make deep training possible.

Transfer Learning: Standing on Giants’ Shoulders

Training a CNN from scratch on ImageNet took Google researchers weeks on dozens of GPUs. You don’t need to do that.

Transfer learning starts with a pretrained model and adapts it to your task. The analogy: it’s like learning to drive after you already know how to ride a bicycle. You don’t have to relearn balance, road sense, or how to steer, you just need to learn what’s different.

A pretrained model (like EfficientNet trained on ImageNet’s 1.2 million images) has already learned to detect edges, textures, shapes, and complex objects. You just need to teach it your specific classes.

The strategy has two stages. First, load the model and swap out just the final prediction layer:

import torch
import torch.nn as nn
from torchvision import models

# Load EfficientNet-B2 pretrained on ImageNet
model = models.efficientnet_b2(weights='IMAGENET1K_V1')

# The final layer currently predicts 1000 ImageNet classes
# Replace it with a layer for YOUR number of classes
num_classes = 3   # e.g., "good", "scratched", "dented"
model.classifier[1] = nn.Linear(
    model.classifier[1].in_features,  # same input size
    num_classes                         # your output size
)

This replaces the last linear layer. The new layer starts with random weights, but everything before it keeps the pretrained ImageNet weights.

Next, freeze the backbone so you don’t accidentally destroy those pretrained features:

# Freeze all layers in the feature extractor
for param in model.features.parameters():
    param.requires_grad = False

# Only the new classifier head will be trained
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)

param.requires_grad = False tells PyTorch: don’t compute gradients for this parameter, and don’t update it during training. The backbone is frozen, only your new head will learn.

Setting Up Data Loading

PyTorch’s ImageFolder expects your images organised in this folder structure:

data/
  train/
    class_1/   (e.g., "good")
      img001.jpg
      img002.jpg
    class_2/   (e.g., "scratched")
      img003.jpg
  val/
    class_1/
      img100.jpg
    class_2/
      img101.jpg

from torchvision import transforms
from torch.utils.data import DataLoader, ImageFolder

# Training transforms include augmentation to help the model generalise
train_transform = transforms.Compose([
    transforms.Resize((260, 260)),
    transforms.RandomHorizontalFlip(),   # randomly mirror images
    transforms.RandomRotation(15),       # randomly rotate ±15 degrees
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

The Normalize values are ImageNet statistics, fixed constants that this pretrained model expects. They don’t change regardless of your data.

# Validation transforms: NO augmentation: test on real images
val_transform = transforms.Compose([
    transforms.Resize((260, 260)),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

train_ds = ImageFolder('data/train', transform=train_transform)
val_ds   = ImageFolder('data/val',   transform=val_transform)

train_loader = DataLoader(train_ds, batch_size=32, shuffle=True)
val_loader   = DataLoader(val_ds,   batch_size=32, shuffle=False)

# Check that classes were found correctly
print(train_ds.classes)   # e.g., ['dented', 'good', 'scratched']
print(train_ds.class_to_idx)  # e.g., {'dented': 0, 'good': 1, 'scratched': 2}

Training: Stage 1: Head Only

Now train just the new classification head for 10 epochs:

criterion = nn.CrossEntropyLoss()

# Only pass the head's parameters to the optimizer
optimizer = torch.optim.AdamW(model.classifier.parameters(), lr=1e-3)

CrossEntropyLoss is the standard loss function for classification. AdamW is the most popular optimiser. The learning rate 1e-3 (0.001) is a good starting point for head training.

for epoch in range(10):
    model.train()
    total_loss = 0

    for images, labels in train_loader:
        images, labels = images.to(device), labels.to(device)

        optimizer.zero_grad()          # clear gradients from last step
        outputs = model(images)        # forward pass
        loss = criterion(outputs, labels)  # compute loss
        loss.backward()                # compute gradients
        optimizer.step()               # update weights
        total_loss += loss.item()

    avg_loss = total_loss / len(train_loader)
    print(f"Epoch {epoch+1}/10: loss = {avg_loss:.4f}")

Watch the loss: it should decrease each epoch. If it plateaus, the head has learned as much as it can with the frozen backbone.

Stage 2: Unfreeze and Fine-Tune

After Stage 1, unfreeze everything and train the whole network with a much smaller learning rate:

# Unfreeze all parameters
for param in model.parameters():
    param.requires_grad = True

# Use a very small learning rate: we don't want to destroy pretrained features
optimizer2 = torch.optim.AdamW(model.parameters(), lr=1e-4)

# Train for 5-10 more epochs
for epoch in range(7):
    model.train()
    # ... same training loop as above

The tiny learning rate (1e-4 vs 1e-3) is critical. We’re making fine adjustments to features that already work well, large updates would undo the pretraining.

Running Inference

After training, switch to evaluation mode before predicting:

model.eval()

with torch.no_grad():   # no gradient computation needed at inference
    outputs = model(images)
    probs = torch.softmax(outputs, dim=1)     # convert to probabilities
    predicted_classes = torch.argmax(probs, dim=1)  # pick highest prob

# Map class index back to name
for i, (pred, prob) in enumerate(zip(predicted_classes, probs)):
    class_name = train_ds.classes[pred.item()]
    confidence = prob[pred].item()
    print(f"Image {i}: {class_name} ({confidence:.1%} confidence)")

The two critical lines: model.eval() and torch.no_grad(). Without eval(), BatchNorm uses batch statistics instead of the learned running stats, and Dropout randomly zeroes out neurons, both cause wrong predictions. Without no_grad(), PyTorch wastes time tracking gradients you’ll never use.

Exercise: Download any small image classification dataset from Kaggle (cats vs dogs, flowers, or even 50 images of 3 things you photograph yourself). Set up the data/train/class/ folder structure. Load EfficientNet-B2 pretrained, swap the head, freeze, train for 5 epochs. What’s your validation accuracy? Then unfreeze and fine-tune for 3 more epochs, does it improve?