Computer Vision, Explained.
A living knowledge publication dedicated to computer vision.
Articles
All articles →Motion & Tracking
A New Approach to Linear Filtering and Prediction ProblemsRudolf Kalman's 1960 paper that recast Wiener's filtering problem in state-space form and solved it with a recursive estimator, the origin of the Kalman filter.
Motion & Tracking
An Iterative Image Registration Technique with an Application to Stereo VisionThe 1981 IJCAI paper by Lucas and Kanade that replaced exhaustive search in image registration with a gradient-guided Newton–Raphson-type iteration, the origin of the Lucas–Kanade method.
Motion & Tracking
Aperture ProblemWhy motion seen through a small window is ambiguous along edges, so that only the component of motion across an edge can be measured locally.
Motion & Tracking
Bayesian FilteringRecursive estimation of the probability distribution of a hidden, changing state from a sequence of noisy measurements, by alternating a motion-model prediction with a Bayes' rule update.
Motion & Tracking
Brightness Constancy AssumptionThe assumption that a scene point keeps the same image intensity as it moves between frames, which turns motion estimation into an intensity-matching problem.
Motion & Tracking
ByteTrackA multi-object tracker that associates high-score detections first and then matches the remaining tracks to low-score detections, recovering occluded objects that score thresholds would discard.
Motion & Tracking
ByteTrack: Multi-Object Tracking by Associating Every Detection BoxThe ECCV 2022 paper by Zhang et al. that introduced BYTE, a second association round for low-confidence detections, and the ByteTrack tracker built on it with a YOLOX detector.
Motion & Tracking
Coarse-to-Fine EstimationEstimating large motions with small-motion methods by solving on an image pyramid, from the coarsest level to full resolution, and refining the estimate at each level.
Motion & Tracking
CONDENSATION—Conditional Density Propagation for Visual TrackingMichael Isard and Andrew Blake's 1996 conference paper and 1998 journal paper that tracked object outlines through dense clutter by propagating a weighted random sample set over time, bringing particle filtering to computer vision.
Geometry & 3D Vision
Cost VolumeA tensor of matching costs or similarities between each pixel of one image and its candidate correspondences in another, indexed by pixel position and candidate disparity, displacement, or depth.
Motion & Tracking
Data AssociationDeciding which measurements or detections belong to which tracked targets, and which are false alarms, missed detections, new targets, or targets that have disappeared.
Motion & Tracking
Determining Optical FlowThe 1981 paper by Horn and Schunck that computed dense optical flow by combining the brightness change constraint with a global smoothness assumption, founding the variational approach to motion estimation.
Evaluation & Metrics
Endpoint ErrorThe standard accuracy measure for optical flow, the distance in pixels between an estimated flow vector and the true one, averaged over the image.
Motion & Tracking
Extended Kalman FilterA nonlinear extension of the Kalman filter that linearizes the motion and measurement models around the current estimate, widely used for camera pose estimation, visual-inertial odometry, and tracking with range or bearing sensors.
Features & Representation
Feature MatchingFinding corresponding keypoints between two or more images of the same scene, from descriptor nearest-neighbor search to learned matchers.
Motion & Tracking
Feature TrackingHow distinctive image points are selected and followed across video frames, using the classic KLT tracker as the main example.
Datasets & Benchmarks
Flying ChairsA synthetic optical flow dataset of rendered chairs moving over Flickr photographs, created to train FlowNet and still used to pretrain flow networks.
Foundations
Gaussian DistributionThe bell-shaped probability distribution defined by a mean and a covariance, the default model for noise and uncertainty in estimation and tracking.
Evaluation & Metrics
Higher Order Tracking AccuracyA multi-object tracking metric that combines detection accuracy and association accuracy through their geometric mean, averaged over localization thresholds.
Motion & Tracking
Horn–Schunck MethodA global variational method that computes dense optical flow by minimizing brightness constancy errors together with a penalty on spatial variation of the flow, solved by a simple iterative averaging scheme.
Motion & Tracking
Hungarian AlgorithmAn algorithm that finds the minimum-cost one-to-one matching between two sets, used in computer vision to match detections to tracks and predictions to ground truth.
Evaluation & Metrics
IDF1An identity-based score for multi-object tracking that matches whole ground-truth and predicted trajectories one-to-one and reports the F1 score of correctly identified detections.
Image Processing & Computational Photography
Image GradientsHow image gradients measure the direction and strength of intensity change, and why they underpin edges, corners, and motion estimation.
Image Processing & Computational Photography
Image PyramidA multi-scale representation of an image built by repeatedly smoothing and subsampling it, so that each level holds a coarser copy at half the resolution of the one below.
Image Processing & Computational Photography
Image WarpingTransforming the geometry of an image by a coordinate mapping, usually computed by sampling the input image at the inverse-mapped position of every output pixel.
Motion & Tracking
Kalman FilterA recursive algorithm that estimates the hidden state of a linear dynamic system from a sequence of noisy measurements, widely used to smooth and predict object positions in tracking.
Datasets & Benchmarks
KITTIA benchmark suite of real driving data from Karlsruhe, with stereo, optical flow, scene flow, odometry, object detection, and tracking benchmarks built from camera, LiDAR, and GPS/IMU recordings.
Foundations
Least SquaresThe method of fitting a model to more measurements than unknowns by minimizing the sum of squared residuals, the workhorse of estimation in computer vision.
Motion & Tracking
Lucas–Kanade MethodA local, gradient-based method that estimates the displacement of an image window by assuming constant motion within it and solving a small least-squares problem, iterated with warping.
Datasets & Benchmarks
Middlebury BenchmarksSmall, high-accuracy stereo and optical flow benchmarks from Middlebury College and Microsoft Research that defined how dense correspondence methods were evaluated in the 2000s.
Datasets & Benchmarks
MOTChallengeA benchmark for multiple pedestrian tracking, released as MOT15, MOT16, MOT17, and MOT20, with hidden test annotations and a central evaluation server.
Motion & Tracking
Motion EstimationRecovering how image content, objects, or the camera moved from a sequence of images, from per-pixel flow to global and 3D motion.
Datasets & Benchmarks
MPI SintelAn optical flow benchmark rendered from the open-source animated film Sintel, with long sequences, large motions, motion blur, and atmospheric effects.
Motion & Tracking
Multi-Object TrackingEstimating the trajectories of a varying, unknown number of objects in a video while keeping each object's identity consistent over time.
Evaluation & Metrics
Multiple Object Tracking AccuracyThe CLEAR MOT accuracy score for multi-object tracking, which counts misses, false positives, and identity switches over a sequence and normalizes them by the number of ground-truth objects.
Motion & Tracking
Object TrackingEstimating the position, extent, or state of one or more objects in every frame of a video, keeping each object's identity over time.
Motion & Tracking
Optical FlowThe apparent motion of image content between two frames, represented as a two-dimensional displacement at every pixel.
Motion & Tracking
Optical Flow EstimationComputing a dense field of pixel displacements between two video frames, from classical variational methods to learned networks such as RAFT.
Motion & Tracking
Particle FilterA sequential Monte Carlo method that represents the probability distribution of a hidden state with weighted random samples, so it can track through nonlinear models and ambiguous, multimodal beliefs.
Motion & Tracking
RAFTA deep network for optical flow that matches all pairs of pixels once and refines a single flow field with a recurrent update operator.
Motion & Tracking
RAFT: Recurrent All-Pairs Field Transforms for Optical FlowThe ECCV 2020 best paper by Teed and Deng that replaced coarse-to-fine flow networks with an all-pairs correlation volume and a weight-tied recurrent update.
Motion & Tracking
Simple Online and Realtime TrackingThe 2016 paper by Bewley et al. that introduced SORT, showing that a Kalman filter and Hungarian matching on strong CNN detections rival far more complex online multi-object trackers.
Motion & Tracking
Simple Online and Realtime Tracking with a Deep Association MetricThe 2017 paper by Wojke, Bewley, and Paulus that introduced DeepSORT, extending SORT with a CNN appearance descriptor trained for person re-identification and a matching cascade, and reporting about 45% fewer identity switches on MOT16.
Motion & Tracking
SORTSimple Online and Realtime Tracking, a multi-object tracker that links per-frame detections into tracks using a constant-velocity Kalman filter on each box and Hungarian matching on box overlap.
Motion & Tracking
Tracking by DetectionBuilding object tracks by running a detector on every frame and linking its detections over time with a motion model and data association.