Face Detection & Recognition

Two Very Different Problems

When people say “face recognition,” they often mean two different things that are frequently confused. Let’s separate them clearly from the start.

Face detection answers the question: “Are there any faces in this image, and if so, where are they?” The output is a set of bounding boxes, one per face. The system has no idea who these people are; it just knows they’re faces. This is what happens when your phone camera draws a square around faces before you take a photo.

Face recognition answers the question: “Who is this person?” It takes a detected face and identifies the individual. This is what happens when Facebook suggests tagging people in your photos, or when an airport uses biometric gates.

Most face-based systems do both, in sequence: first detect all faces in the image, then run recognition on each one.

Understanding this distinction matters because they’re completely different algorithms with different data requirements, different use cases, and different ethical implications.

Face Detection: Finding the Faces

Method 1: OpenCV Haar Cascades (Old but Fast)

The oldest practical approach uses Haar cascades, a series of simple filters that look for facial features like eye-nose patterns. This method dates from 2001 and still works well when faces are front-facing and well-lit.

import cv2

# Load the pretrained face detector cascade
# OpenCV includes this file: cv2.data.haarcascades points to it
detector = cv2.CascadeClassifier(
    cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'
)

The cascade file contains hundreds of pre-learned facial feature detectors. It loads instantly and requires no GPU.

img = cv2.imread('group_photo.jpg')
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Detect faces
# scaleFactor: how much to shrink image at each scale (1.1 = 10% shrink per step)
# minNeighbors: how many rectangles must agree to call it a face (higher = fewer false detections)
faces = detector.detectMultiScale(img_gray, scaleFactor=1.1, minNeighbors=5)

print(f"Found {len(faces)} faces")

for (x, y, w, h) in faces:
    cv2.rectangle(img, (x, y), (x+w, y+h), color=(0, 255, 0), thickness=2)

detectMultiScale returns a list of (x, y, width, height) tuples. Each tuple is one detected face. The (x, y) is the top-left corner; w and h are the face width and height.

Limitations: Haar cascades struggle with faces that are rotated, partially obscured, or in poor lighting. For production systems, use the deep learning approach below.

Method 2: MTCNN (Better Accuracy)

MTCNN (Multi-Task Cascaded Convolutional Network) uses deep learning and handles more challenging conditions: faces at angles, smaller faces, partial occlusion. It also detects facial landmarks (eyes, nose, mouth corners).

# pip install mtcnn tensorflow
from mtcnn import MTCNN
import cv2

detector = MTCNN()

img = cv2.imread('photo.jpg')
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

results = detector.detect_faces(img_rgb)

for face in results:
    x, y, w, h = face['box']
    confidence = face['confidence']
    keypoints  = face['keypoints']   # eyes, nose, mouth positions
    
    print(f"Face at ({x},{y}), size {w}×{h}, confidence {confidence:.1%}")
    print(f"  Left eye: {keypoints['left_eye']}")
    print(f"  Right eye: {keypoints['right_eye']}")

MTCNN returns richer information: confidence score per face, and keypoint coordinates. The keypoints are useful for alignment, before recognition, you typically align all faces to a standard pose by rotating based on eye positions.

Face Recognition: The Embedding Approach

Modern face recognition doesn’t work by comparing photos pixel-by-pixel (which would be fragile and storage-intensive). Instead, it uses face embeddings.

Here’s the core idea: a neural network is trained to convert any photo of a face into a vector of numbers, typically 128, 256, or 512 numbers. This vector is called an embedding. The network is trained so that:

• Two photos of the same person produce very similar vectors (small distance between them)
• Photos of different people produce very different vectors (large distance between them)

To recognise someone, you compare their face’s embedding to a database of known embeddings. If any stored embedding is close enough, you’ve found a match. You never need to store the actual photos, just the numerical vectors.

This has a beautiful property: once you have someone’s embedding, you only need to compute it once. Every subsequent verification is just a vector comparison (milliseconds), not a full neural network inference on both images.

DeepFace: A Unified Library

DeepFace is a Python library that wraps multiple state-of-the-art face analysis models behind a clean API. It handles detection, alignment, embedding, and identity verification.

pip install deepface

Let’s start with verification: checking if two photos are of the same person:

from deepface import DeepFace

# Are these two photos the same person?
result = DeepFace.verify(
    img1_path='person_a_photo1.jpg',
    img2_path='person_a_photo2.jpg',
    model_name='Facenet512'   # one of several available models
)

print(f"Same person: {result['verified']}")   # True or False
print(f"Distance: {result['distance']:.4f}")  # lower = more similar
print(f"Threshold: {result['threshold']:.4f}")  # threshold for this model

distance is the similarity score, how far apart the two embeddings are. If the distance is below the threshold, they’re classified as the same person.

Now try full face analysis: age, gender, and emotion estimation from a single image:

# Analyse a single face image
analysis = DeepFace.analyze(
    img_path='portrait.jpg',
    actions=['age', 'gender', 'emotion'],
)

# analyse returns a list (one dict per face found)
face_data = analysis[0]
print(f"Estimated age: {face_data['age']}")
print(f"Gender: {face_data['dominant_gender']}")
print(f"Dominant emotion: {face_data['dominant_emotion']}")
print(f"All emotions: {face_data['emotion']}")

Note: these are estimates with real uncertainty. Age estimation is typically accurate to ±5 years. Emotion detection is notoriously unreliable in practice, the same face at different angles can show wildly different “emotions.” Use these features for statistical insights across populations, not for individual-level decisions.

A Critical Note on Ethics

Face recognition is one of the most ethically sensitive AI technologies in existence. Before building anything with it, you should understand the concerns clearly.

Consent and surveillance. Face recognition can identify individuals at scale without their knowledge or consent. This enables mass surveillance in ways that fundamentally change the dynamics of public space.

Demographic bias. Multiple studies have found that commercial face recognition systems have significantly higher error rates for people with darker skin tones and for women. A system that’s 98% accurate overall might be only 89% accurate for Black women, meaning it misidentifies 1 in 9 Black women it encounters. Deploying such a system in a security context creates serious harm.

Legal landscape. Many cities and countries are actively regulating or banning certain uses of face recognition, particularly for law enforcement. Before deploying, verify your local regulations.

Practical guidelines:

• Obtain explicit consent before collecting anyone’s face data
• Never use face recognition for surveillance of people in public spaces without legal authorisation
• Test your system’s accuracy across all demographic groups before deployment
• Provide humans with meaningful oversight of automated decisions

Exercise: Use MTCNN to detect faces in a group photo (you can use any team photo or celebrity image from the web). Print the number of faces found, the confidence score for each, and the coordinates of the eye keypoints. Then use DeepFace.analyze on a single-person portrait. Are the age/emotion estimates reasonable? Try different angles or expressions of the same person, how consistent are the results?