Video Is Just Very Fast Photography
This might seem obvious, but it’s the single most important insight for video computer vision: a video is simply a sequence of still images played fast enough to look like continuous motion.
A typical video at 30 fps (frames per second) shows 30 separate photos per second. Your eye and brain blend these into the illusion of smooth movement. Computationally, this means:
- Everything you learned about image processing applies directly to video
- Video CV = applying image analysis to each frame, then connecting results across time
- A 1-minute video at 30fps contains 1,800 frames to process
This creates two key challenges that don’t exist for single images:
Efficiency. You can’t run a 200ms detection model on every frame at 30fps, that would take 60 seconds to process 1 second of video. You need to be selective about which frames to process and how.
Temporal consistency. Objects don’t teleport between frames. An object detected in frame 100 is (almost certainly) the same object appearing in frame 101. Tracking algorithms use this to maintain consistent IDs across frames.
Reading a Video File
OpenCV’s VideoCapture class handles both video files and live camera feeds with the same API:
import cv2
# Open a video file
cap = cv2.VideoCapture('footage.mp4')
# Or open a webcam (0 = first camera, 1 = second camera, etc.)
# cap = cv2.VideoCapture(0)First, inspect the video’s properties:
fps = cap.get(cv2.CAP_PROP_FPS)
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
duration_sec = total_frames / fps
print(f"Video: {width}×{height} at {fps:.1f}fps")
print(f"Duration: {duration_sec:.1f} seconds ({total_frames} frames)")These properties help you plan your processing strategy.
Now read and process frames:
frame_num = 0
results = []
while cap.isOpened():
ret, frame = cap.read() # ret=True if frame read successfully
if not ret:
break # end of video or read error
# Process only every 5th frame for efficiency
if frame_num % 5 == 0:
# frame is a NumPy array (H, W, 3): treat like any image
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# ... run your analysis here ...
results.append({'frame': frame_num, 'data': 'your result'})
frame_num += 1
cap.release() # always release when done
print(f"Processed {frame_num} frames total")cap.read() returns a boolean (ret) and the frame as a NumPy array. Once ret is False, the video has ended. Always call cap.release() to free the camera/file resource.
Saving a Processed Video
If you want to save your annotated or processed frames back as a video:
# Set up the output video writer
fourcc = cv2.VideoWriter_fourcc(*'mp4v') # codec for .mp4
out = cv2.VideoWriter('output.mp4', fourcc, fps, (width, height))
# In your processing loop, after modifying `frame`:
out.write(frame) # write each processed frame
# After the loop:
out.release()
print("Saved to output.mp4")The fourcc (four-character code) specifies the video codec. mp4v works well for .mp4 files. XVID works for .avi.
Motion Detection with Background Subtraction
One of the most common video analysis tasks is motion detection, finding which parts of each frame are moving. The technique is called background subtraction: build a model of what the background looks like when nothing is happening, then flag any pixel that differs significantly from that background model.
OpenCV’s MOG2 algorithm builds a dynamic background model that adapts over time:
# Create the background subtractor
# history=500: number of frames used to build background model
# varThreshold=50: how different a pixel must be to count as "foreground"
backSub = cv2.createBackgroundSubtractorMOG2(history=500, varThreshold=50)The background model automatically adapts: if you put down a box and leave it there for 30 seconds, it eventually gets incorporated into the “background.” This handles slow environmental changes like clouds moving across a window.
Now apply it frame by frame:
cap = cv2.VideoCapture('parking_lot.mp4')
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
# Apply background subtraction
fg_mask = backSub.apply(frame)
# fg_mask: 255 (white) = moving pixel, 0 (black) = backgroundNow find the moving regions:
# Find contiguous regions of foreground pixels
contours, _ = cv2.findContours(
fg_mask,
cv2.RETR_EXTERNAL, # only outermost contours
cv2.CHAIN_APPROX_SIMPLE # compress contour points
)
for c in contours:
area = cv2.contourArea(c)
if area < 500: # filter out tiny noise blobs
continue
# Get bounding box of this moving region
x, y, w, h = cv2.boundingRect(c)
cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)
print(f"Motion detected: area={area:.0f} pixels at ({x},{y})")
cap.release()cv2.findContours finds connected regions of white pixels in the mask. Each contour is a moving object (or a group of nearby moving objects). The area filter removes small noise blobs, you typically don’t care about a 20-pixel blob, but you do care about a 500+ pixel blob.
Object Tracking: Remembering Who’s Who
Background subtraction tells you where things are moving. Tracking goes further: it gives each moving object a consistent ID across frames, so you know “object #3 started at position (100, 200) and is now at (150, 220).”
OpenCV includes several tracking algorithms. Here’s the simplest approach: centroid tracking: we represent each object as its centre point (centroid), and associate objects across frames by finding the nearest centroid:
# OpenCV's built-in tracker: CSRT is accurate, KCF is faster
tracker = cv2.TrackerCSRT_create()
# Initialise with the first frame and a bounding box
# bbox format: (x, y, width, height)
first_frame = ... # your first video frame
initial_bbox = (100, 200, 80, 120) # x, y, w, h of initial detection
tracker.init(first_frame, initial_bbox)Then update the tracker on each subsequent frame:
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
# Update tracker: returns success flag and new position
success, bbox = tracker.update(frame)
if success:
x, y, w, h = [int(v) for v in bbox]
cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)
cv2.putText(frame, "Tracking", (x, y-10),
cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
else:
cv2.putText(frame, "Lost", (50, 50),
cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 0, 255), 2)If success is False, the tracker lost the object, perhaps it left the frame or became fully obscured. You’d then re-run detection to find it again.
Putting It Together: A Simple People Counter
Here’s a minimal but real example: counting people crossing a line in a video:
import cv2
from ultralytics import YOLO
model = YOLO('yolov8n.pt')
cap = cv2.VideoCapture('entrance.mp4')
count = 0
line_y = 300 # y-coordinate of counting line
frame_num = 0
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
if frame_num % 3 == 0: # detect every 3rd frame
results = model(frame, classes=[0]) # class 0 = person
for box in results[0].boxes:
x1, y1, x2, y2 = [int(v) for v in box.xyxy[0]]
cy = (y1 + y2) // 2 # centroid y coordinate
if abs(cy - line_y) < 10: # crossing the line
count += 1
cv2.line(frame, (0, line_y), (frame.shape[1], line_y), (0, 255, 255), 2)
cv2.putText(frame, f"Count: {count}", (30, 50),
cv2.FONT_HERSHEY_SIMPLEX, 1.5, (0, 255, 0), 3)
frame_num += 1
cap.release()
print(f"Total count: {count}")This is a simplified counter, a production version would use proper tracking to avoid counting the same person twice (as they move slowly across the line). But for orientation purposes or high-level counting, this pattern works well.
Exercise: Get any video clip (even a short recording from your phone). Open it with OpenCV, print its FPS and dimensions, then extract every 10th frame and save them as individual JPEG files. Then apply background subtraction to detect which frames have motion. Which percentage of frames contain motion? What minimum contour area makes sense for filtering out noise?