Image Segmentation

When a Bounding Box Isn’t Enough

Object detection gives you rectangles. That’s often enough, counting cars on a road, finding faces in a photo, flagging defective products on a conveyor belt.

But sometimes, a rectangle is too imprecise. Consider these scenarios:

Measuring defect area on a solar panel. A bounding box includes background pixels inside the rectangle. You need to know exactly which pixels are defect vs. panel, so you can calculate the defect’s actual surface area.

Removing the background from a product photo. E-commerce sites need clean product images on white backgrounds. A bounding box crop would include background corners. You need the exact outline of the product.

Medical imaging. A radiologist’s report might say “tumour volume is 2.3 cm³.” Computing volume requires knowing which exact voxels (3D pixels) belong to the tumour, not a rough bounding box.

For all these cases, you need image segmentation: assigning a label to every single pixel in the image.

Two Flavours of Segmentation

Semantic Segmentation

Semantic segmentation classifies every pixel into a category, but doesn’t distinguish between individual objects.

Imagine a photo of a parking lot. Semantic segmentation might label every pixel as either “car, ” “road, ” “building, ” or “sky.” But it treats all cars as the same, you can’t tell car #1 from car #2.

This is useful when you care about categories but not individual instances: “how much of this frame is road?” “what percentage of pixels are vegetation?”

Instance Segmentation

Instance segmentation treats every individual object as separate, even within the same class. Each car in the parking lot gets its own unique mask: car #1, car #2, car #3…

This is more powerful, you can count individual objects, measure each one separately, or track specific instances across video frames. The trade-off: it’s computationally heavier and harder to train.

Which Should You Use?

The Traditional Approach: U-Net

For many years, the workhorse of segmentation was U-Net, an architecture originally designed for medical imaging. It has an encoder (like a CNN that compresses the image) and a decoder (that expands back to the original size, producing a per-pixel prediction).

U-Net and its descendants are still used extensively in medical imaging. But for general-purpose segmentation, newer foundation models have changed the game.

SAM: Segment Anything, No Training Required

In 2023, Meta AI released SAM (Segment Anything Model), and it genuinely lives up to its name. SAM was trained on 11 billion mask-image pairs, giving it the ability to segment virtually any object in any image, without any task-specific training on your part.

You don’t need labelled data. You don’t need to fine-tune. You just point at an object (with a click or a box), and SAM returns a precise mask.

First, install SAM and download the model weights:

pip install segment-anything
# Download the model checkpoint (ViT-H is the largest and most accurate)
# wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

Now set it up in Python:

from segment_anything import SamPredictor, sam_model_registry
import cv2
import numpy as np

# Load the model: "vit_h" is the largest variant
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

SamPredictor wraps the model and provides an easy interface for interactive segmentation. The model loads its weights from the checkpoint file you downloaded.

Segmenting with a Point Click

The most intuitive way to use SAM: click a point on the object you want to segment.

# Load your image
image = cv2.imread("product.jpg")
image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Tell SAM which image to work with (it extracts features once)
predictor.set_image(image_rgb)

Calling set_image runs the encoder part of SAM, it computes a feature representation of the image. This takes a moment, but you only need to do it once per image. After this, you can run multiple prompts quickly.

# Define a prompt: click a point on the object
# (320, 240) = x=320, y=240 in pixel coordinates
input_point = np.array([[320, 240]])
input_label = np.array([1])    # 1 = foreground (I want this object)
                                # 0 = background (I don't want this)

# Get predictions: returns 3 candidate masks by default
masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,   # gives 3 options at different scales
)

print(f"Returned {len(masks)} masks with scores: {scores}")

SAM returns three candidate masks: a small, medium, and large interpretation of what you clicked on. This is SAM being helpful: maybe you clicked on a button on a shirt; the small mask covers just the button, the medium covers the whole shirt, the large might cover the person. You choose which scale makes sense for your use case.

Using the Mask

# Pick the highest-confidence mask
best_idx  = np.argmax(scores)
best_mask = masks[best_idx]   # shape: (height, width), dtype: bool

print(f"Best mask score: {scores[best_idx]:.3f}")
print(f"Mask shape: {best_mask.shape}")
print(f"Object area: {best_mask.sum():,} pixels")

# Express as percentage of total image
total_pixels = best_mask.shape[0] * best_mask.shape[1]
coverage = best_mask.mean() * 100
print(f"Object covers {coverage:.1f}% of the image")

best_mask is a boolean array, True where the object is, False everywhere else. .sum() counts the True pixels (the object’s area). .mean() gives the fraction of the image covered.

Now remove the background for a clean product photo:

# Create a white-background version
result = image_rgb.copy()
result[~best_mask] = 255   # set background pixels to white

# Or create a transparent PNG (requires RGBA)
rgba = cv2.cvtColor(image, cv2.COLOR_BGR2BGRA)
rgba[:, :, 3] = (best_mask * 255).astype(np.uint8)  # alpha = mask
cv2.imwrite('product_transparent.png', rgba)

~best_mask means “everywhere the mask is False (background).” Setting those pixels to 255 (white) gives you a clean product image. The RGBA version uses the mask as an alpha channel, producing a transparent background PNG.

When to Use Segmentation

Segmentation adds significant complexity. Don’t reach for it unless you truly need pixel-level precision. Ask yourself:

• Is a bounding box good enough for my use case?
• Do I need to know the exact area/shape of the object?
• Am I doing background removal or precise object isolation?

If the answer to the first question is “yes,” use detection. If the answers to the second or third are “yes,” segmentation is the right choice.

Exercise: Find an image with a clear subject (a product, an animal, a piece of equipment). Use SAM with a point prompt to segment the main object. Try clicking different parts of the object, what different masks does SAM return? Now try multimask_output=False, what single mask does SAM choose? Use the mask to remove the background and save as a transparent PNG.