All articles
Every Computer Vision Online article, grouped by subject.
Foundations
- A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
The 1997 paper by Freund and Schapire that analysed the Hedge algorithm for on-line allocation and reversed it into AdaBoost, with its training-error and VC bounds, the multiclass and regression versions, the 1995 conference version, and its reception and later corrections.
- Aperture and F-Number
The opening that limits the cone of light a lens accepts, its f-number N = f/d, why N rather than the diameter sets exposure, the marked stop sequence and its rounding, the working f-number at close focus, and how the aperture trades light against depth of field, diffraction, vignetting and the shape of out-of-focus blur.
- Bayer Filter
The colour filter array over a single-chip colour sensor, seen as a sampling design, from the Bayer pattern and its four phases to X-Trans, RGBW and Condat's 2 × 3 array, with the Fourier view that places luminance at the origin and colour on carriers.
- Bayesian Inference
Treating unknown quantities as random variables and reasoning with their whole posterior distribution; conjugate priors, posterior summaries and credible intervals, the loss that picks a point estimate, prediction, model comparison by the marginal likelihood, approximate computation and where vision uses it.
- Bias–Variance Tradeoff
How the expected error of a learned predictor or estimator splits into noise, squared bias and variance, how the split is derived, computed in closed form for linear smoothers and estimated by resampling, why the tradeoff is a tendency rather than a law, and how it changes under the 0–1 loss of classification and segmentation.
- Boosting
How boosting turns many weak classifiers, such as single-feature decision stumps on image patches, into an accurate weighted vote, covering AdaBoost and its training-error bound, the exponential-loss and margin explanations, gradient boosting, sensitivity to label noise, scikit-learn's AdaBoostClassifier, and boosted feature selection in face and object detection.
- Camera Intrinsic and Extrinsic Parameters
The parameters of the pinhole camera matrix P = K[R | t], split into intrinsics K, which map camera coordinates to pixels, and extrinsics R and t, which place the camera in the world.
- Clustering
Grouping unlabelled data so that similar items share a group; why the task is undefined until a similarity and a scale are chosen, the main families of methods (centroid, mixture, density and mode seeking, hierarchical, graph-based), how a clustering is evaluated with and without ground truth, and where clustering is used in computer vision.
- Color Spaces
A color space assigns numbers to colors; choosing and converting between RGB, CIE XYZ, CIELAB, YCbCr, and HSV correctly matters for both image processing and learning.
- Colorimetry
The measurement of colour as seen by a standard human observer, from colour matching as linear algebra and the CIE 1931 colour-matching functions to tristimulus values, chromaticity diagrams, white points, Planckian light, RGB-to-XYZ matrices and metamerism.
- Computer Vision
What computer vision is, why recovering a scene from its images is hard, how the field relates to image processing, graphics, machine learning, robotics and photogrammetry, its main tasks, a dated history from the 1960s to foundation models, and a map of this site's categories.
- Conditional Random Field
A Markov random field over output labels whose potentials, and partition function, depend on the observed image, so that it models the conditional distribution of the labels given the input; with the linear-chain CRF and the label bias problem, learning by conditional likelihood, grid and fully connected CRFs for segmentation, mean-field inference as Gaussian filtering, and CRFs inside networks.
- Curse of Dimensionality
Why data, volume and distances behave against intuition when each example is described by many numbers, which forms of the problem matter for learning from images, and why the dimension that counts is the intrinsic dimension of the data rather than the number of pixels.
- Decision Tree
How a decision tree classifies or regresses by a sequence of threshold tests, how CART grows one greedily with Gini or entropy impurity and prunes it by cost complexity, what scikit-learn's DecisionTreeClassifier actually computes, and why trees on raw pixels need many leaves and change with every resample of the training images.
- Depth of Field
The range of object distances that a focused lens images with blur no larger than an acceptable circle of confusion, its exact near and far limits under the thin-lens model, the hyperfocal distance, Levoy's approximations and how depth of field scales with f-number, distance and focal length.
- Digital Camera
The camera as an imaging system, from lens, shutter and sensor to the image signal processor that demosaics, white-balances, colour-corrects, tone-maps and compresses its output; RAW against processed images, metadata, machine-vision interfaces, camera classes, and where processing breaks the assumptions of vision methods.
- Digital Image
A digital image is a grid of samples of a continuous scene, each quantized to a finite set of values and stored as an array of numbers.
- Dimensionality Reduction
How images and feature vectors with many numbers are described by fewer, either by keeping some of the original variables (feature selection) or by building new ones (feature extraction); the taxonomy of methods, filters, wrappers and embedded selection, classical MDS, Isomap and LLE in outline, how to choose the target dimension, and an example in which the directions of largest variance carry no class information.
- Dynamic Range
The ratio between the largest and the smallest signal a sensor, scene, file or display can handle, how EMVA 1288, Levoy's course notes and ISETCam define it for a camera, how to express it in ratios, decibels, stops and bits, and why bit depth, gain and exposure change what a capture keeps.
- Eigenvalues and Eigenvectors
The directions a square matrix only scales and the factors it scales them by; the characteristic polynomial, multiplicities and defective matrices, the spectral theorem and Rayleigh quotient for symmetric matrices, how far noise moves eigenvalues and eigenvectors, how libraries compute them, and an example on structure tensors of gratings that shows the eigengap setting eigenvector accuracy.
- Event Camera
A sensor whose pixels each report a timestamped event whenever their log intensity changes by a set contrast threshold, instead of capturing whole frames at a fixed rate; how the pixel works, the DVS, ATIS and DAVIS designs, the data and its representations, noise, simulation, and what the event stream can and cannot reveal about motion.
- Expectation–Maximization
An iterative method for maximum likelihood estimation when part of the data is hidden or missing, alternating an E step that infers the hidden variables with an M step that refits the parameters; its lower-bound derivation, the monotone increase of the likelihood and what is proved about convergence, the full updates for Gaussian mixtures, its variants, and its uses in vision from colour models and background subtraction to mixtures of motions.
- Exposure
How much light a camera records in one frame, set by shutter time, aperture and gain; how exposure is counted in stops, metered and controlled automatically, and what each setting costs a vision system in motion blur, noise, depth of field and clipping.
- Focal Length and Field of View
How a lens's focal length and the sensor's size set the angle of view of a camera, in horizontal, vertical and diagonal forms, and how sensor formats, crop factors, 35 mm equivalent focal lengths and the focus distance change it.
- Gaussian Distribution
The bell-shaped probability distribution defined by a mean and a covariance, the default model for noise and uncertainty in estimation and tracking.
- Gaussian Mixture Model
A probability density built as a weighted sum of Gaussians, read as data generated by first picking a component and then sampling from it; used in vision to model multimodal colour and intensity distributions, from per-pixel background models to the colour models of interactive segmentation.
- Gradient Descent
An iterative method that minimizes a differentiable function by repeatedly stepping in the direction of the negative gradient, the basic optimizer behind variational image models and neural network training.
- History of Computer Vision
The theories of vision that came before computers, from Greek optics to Helmholtz, Gestalt psychology and Gibson, a dated timeline from antiquity to foundation models with the article that owns each event, how the textbooks divide the field's past, and the themes that recur.
- Human Visual System
The eye, retina, lateral geniculate nucleus and visual cortex as a chain of optical and neural stages, what each stage does to the image, how human contrast sensitivity is measured, and which of these ideas computer vision borrowed.
- Image Formation
How light leaving a scene becomes an array of pixel values, through illumination and reflectance, the lens, the sensor and its colour filters, sampling and quantization, and the camera's own processing.
- Image Noise
The random variation in recorded pixel values that does not come from the scene, its physical sources in the camera, and how it is modeled and measured.
- Image Sensor
How the chip behind a camera lens turns light into numbers, from photosites and quantum efficiency to CCD and CMOS readout, full-well capacity, noise, gain and the ADC, colour filters, rolling and global shutters, and EMVA 1288 measurements.
- Information Theory for Computer Vision
The quantities information theory supplies to vision, entropy, conditional entropy, mutual information, differential entropy and the KL divergence, how they relate, why forward and reverse KL fit different models, what the data processing inequality and Fano's inequality say about processing images, and the conventions (log base, bin width, normalization) that change the numbers.
- Inverse Problems in Vision
Recovering causes from their effects, the shape of the problem behind restoration, reconstruction and much of geometry; forward models, well-posedness and ill-conditioning, why finer sampling can make a problem worse, and the remedies of regularization, Bayesian priors and learned priors.
- k-Means Clustering
The clustering method that splits data into k groups by alternately assigning each point to the nearest of k centres and moving each centre to the mean of its points; its sum-of-squares objective, Lloyd's algorithm and why it stops at a local minimum, k-means++ seeding and restarts, how to choose k, its spherical-cluster assumptions, and its uses in colour quantization, segmentation, superpixels and visual vocabularies.
- k-Nearest Neighbors
The classifier and regressor that predict from the k most similar stored examples, with no training step; how the vote, the number of neighbours, the distance and the feature scaling decide its accuracy, the Cover–Hart bound on its error, why it struggles in high dimensions, and where nearest-neighbour methods are used in computer vision.
- Kernel Methods
Learning algorithms that touch the data only through a positive-definite kernel, a similarity that equals an inner product in a feature space never built; the kernel trick, the rules for making valid kernels, kernel ridge regression and its link to Gaussian processes, kernel PCA with centring in feature space, the cost of the n × n Gram matrix, and the Nyström and random-feature approximations that remove it, with an example on oriented edge patches.
- Least Squares
The method of fitting a model to more measurements than unknowns by minimizing the sum of squared residuals, the workhorse of estimation in computer vision.
- Lens Distortion
The systematic deviation of a real lens from the pinhole model's straight-line projection, which bends straight lines in the image and is modeled, estimated, and removed as part of camera calibration.
- Levenberg–Marquardt Algorithm
The damped Gauss–Newton method for nonlinear least squares as a procedure, with its accept-and-reject logic, damping updates, scaling matrix, trust-region form and stopping tests, as MINPACK, SciPy and Ceres Solver implement them.
- LiDAR
An active sensor that measures distance by timing laser light reflected from surfaces, and sweeps its beams across the scene to produce a point cloud with range, intensity and a timestamp for every return.
- Linear Algebra
The mathematics of vectors, matrices and linear maps that computer vision is written in, from images stored as vectors and data matrices to camera projections, homographies, rank, null spaces, eigenvalues and the numerical care needed to solve linear systems.
- Linear Discriminant Analysis
The classifier that models each class as a Gaussian with a covariance matrix shared by all classes, which makes its decision boundaries linear; Fisher's 1936 ratio criterion and its equivalence to the two-class rule, projection to at most c − 1 dimensions, quadratic discriminant analysis, shrinkage of the covariance for small samples, and scikit-learn's solvers.
- Linear Regression
The model that predicts a real-valued target as a weighted sum of input features, fitted by ordinary least squares, ridge, lasso or elastic net; what each estimator does to the coefficients, how scikit-learn scales its penalties, and how lasso selects pixels from image patches.
- Logistic Regression
The linear classifier that models the probability of a class as a sigmoid (or, for several classes, a softmax) of a weighted sum of image features, fitted by maximum likelihood; its log-odds reading, Newton's method as iteratively reweighted least squares, what happens on separable data, regularization, multiclass schemes, calibration and class imbalance.
- Machine Learning
How a program learns a task such as recognizing objects from examples instead of hand-written rules, why doing well on the training images is not the goal, and how features, models, data and evaluation fit together in computer vision.
- Markov Random Field
A probability model over the pixels or labels of an image in which each variable depends directly only on its neighbours, so that the joint distribution is a Gibbs distribution over the graph's cliques; with the Ising and Potts models, Gaussian Markov random fields, MAP and MPM estimates, and the optimizers used on them, from ICM and annealing to graph cuts and loopy belief propagation.
- Maximum a Posteriori Estimation
Choosing the single most probable value of an unknown under its posterior distribution; what loss it answers to, how it differs from the posterior mean and median, why sparse priors give exact zeros and why the answer depends on how the unknown is parameterized.
- Maximum Likelihood Estimation
The method of choosing model parameters under which the observed data are most probable; it turns noise models into losses such as least squares and cross-entropy, and its large-sample properties explain when estimates from images can be trusted.
- Maximum Likelihood from Incomplete Data via the EM Algorithm
The 1977 paper by Dempster, Laird and Rubin, read before the Royal Statistical Society, that defined the EM algorithm at three levels of generality, named it and generalized EM, proved that every iteration increases the likelihood, related its rate to the fraction of missing information, and collected earlier special cases from missing values to mixtures and factor analysis; with the discussion it drew and the error in its convergence proof that Wu found in 1983.
- Mean Shift
A mode-seeking procedure that climbs a kernel density estimate by repeatedly moving a window to the weighted mean of the data inside it; why that step is a normalized gradient, when it converges, how the bandwidth decides what counts as a mode, and how it clusters colours, filters and segments images, and tracks objects.
- Mean Shift: A Robust Approach Toward Feature Space Analysis
The 2002 IEEE TPAMI paper by Comaniciu and Meer that recast an old mode-seeking procedure as a general tool for feature space analysis, gave a convergence theorem for kernels with convex profiles, and applied it in the joint spatial–range domain to discontinuity-preserving smoothing and image segmentation.
- Nearest Neighbor Pattern Classification
The 1967 paper by Cover and Hart that proved the large-sample error of the nearest-neighbour rule is at most about twice the Bayes error for any number of classes, with the assumptions behind the bound, its k-NN extension, the paper's worked examples, what it does not claim, and the slips found in it since.
- Nonlinear Least Squares
Fitting a model that is nonlinear in its parameters by iteratively minimizing a sum of squared residuals, the optimization behind calibration, triangulation, homography refinement, and bundle adjustment.
- Omnidirectional Camera
A camera whose field of view covers a full horizontal circle, a hemisphere or nearly the whole sphere, built from fisheye lenses, curved mirrors or several cameras, and described by spherical rather than pinhole projection models.
- Optimization
Choosing the parameters that minimize an objective, possibly under constraints, the common form of fitting, calibration, registration, segmentation and network training in computer vision, and the convexity, optimality conditions and solver families that decide how such problems are solved.
- Overfitting and Underfitting
How to tell whether a vision model fits its training images too loosely or too closely, by reading training and validation error against capacity and against the amount of data, and which lever fixes each failure.
- Perceptron
Rosenblatt's perceptron, a linear threshold unit trained by the error-correction rule that adds a misclassified example to the weights; the convergence theorem and its mistake bound for linearly separable data, why it cycles on non-separable data, XOR and what Minsky and Papert actually proved, the Mark I machine, and the pocket, voted, averaged and kernel variants.
- Pinhole Camera Model
The ideal camera in which every scene point projects through a single center onto an image plane, giving the perspective projection x = fX/Z, y = fY/Z.
- Pixel
The pixel as picture element, where the word came from, the different things it names in images, sensors and displays, how pixels are counted, and why the position of a pixel's centre changes every resize.
- Point Spread Function
The image a camera forms of a single point of light, how diffraction, defocus, aberrations and the pixel aperture combine into it, its Fourier transform the optical transfer function and its magnitude the MTF, and how the MTF is measured from a slanted edge.
- Principal Component Analysis
The linear method that finds the directions along which a set of images or feature vectors varies most, and uses them to compress, denoise, decorrelate and whiten the data; computed from the SVD of the centred data matrix.
- Probabilistic Graphical Models
Graphs that encode which variables of a joint probability distribution depend directly on which, so that the distribution factorizes into small pieces; with Bayesian networks, d-separation and explaining away, undirected models and factor graphs, exact inference and its cost, belief propagation on trees and loopy graphs, mean field, Gibbs sampling, learning, and the vision models built on them.
- Probability and Statistics
The mathematics of uncertainty that vision uses everywhere, from noisy pixels to classifier outputs; random variables, distributions, conditional probability and Bayes' rule, expectation and covariance, sums and averages, and estimation from samples.
- Radiometry and Photometry
How light is measured in physical units (flux, radiant intensity, irradiance and radiance) and in units weighted by human vision (lumens, candelas, lux and candelas per square metre), with solid angle, the cosine and inverse-square laws, Lambertian surfaces, why radiance is what a pixel measures, and the SI definition of the candela.
- Random Forest
How a random forest combines many decision trees, each grown on a bootstrap sample with a random subset of features tried at every split, how Breiman's strength and correlation explain its error, how out-of-bag votes estimate error without a test set, and how scikit-learn's averaged probabilities and feature importances differ from Breiman's original procedure.
- Reflectance Models
How the bidirectional reflectance distribution function (BRDF) describes a surface, the properties a physically plausible BRDF must have, and the models built on it, from Lambertian and Phong to Blinn–Phong and microfacet models with GGX, Smith masking and Fresnel, with tests of reciprocity, energy and normalization.
- RGB-D Camera
A depth camera and a colour camera built into one unit, whose depth map must be registered to the colour image through the two cameras' intrinsics and extrinsics before colour and depth can be used pixel by pixel.
- Robust Estimation
Fitting a model when some measurements are gross errors, with M-estimators such as Huber's and Tukey's biweight solved by iteratively reweighted least squares, a robust scale such as the MAD, high-breakdown starts such as least median of squares, and the trade-off between efficiency and breakdown.
- Rolling Shutter
How a sensor that exposes and reads its rows one after another turns camera motion into image distortion; the projection models, the rolling-shutter constraint, line-delay calibration, rolling-shutter-aware pose and odometry, correction to global-shutter images, and flicker bands.
- Singular Value Decomposition
The factorization A = UΣVᵀ of any matrix into two orthogonal matrices and a diagonal matrix of singular values, and how vision uses it for rank, pseudoinverses, low-rank approximation, homogeneous systems and projection onto constrained matrices.
- Spectral Clustering
Clustering by the eigenvectors of a similarity graph's Laplacian; the unnormalized, random-walk and symmetric Laplacians, the Ng–Jordan–Weiss algorithm and its analysis, the eigengap, the choice of graph and scale, what scikit-learn computes, and an example on grating patches that k-means cannot separate.
- Stereo Camera
A depth sensor built from two or more matched, synchronized and factory-calibrated imagers, sometimes with an on-board matcher and an infrared texture projector, that outputs disparity or depth computed by triangulation.
- Structured Light
An active depth sensor that replaces one camera of a stereo pair with a projector, labels every illuminated point with a projected code such as a Gray-code sequence, phase-shifted fringes or a pseudo-random dot pattern, and triangulates each camera ray against the projector plane or ray the code identifies.
- Supervised Learning
Learning to predict a label or a measured quantity from images by fitting a model to examples whose correct outputs are given, with the choice between classifying and regressing, the output heads each implies, and where the labels come from.
- Support Vector Machine
The classifier that separates two classes of images or image features by the hyperplane with the widest margin, its soft-margin and hinge-loss forms, the dual problem and kernels such as the RBF, histogram intersection and χ² kernels, its solvers and library defaults, and its use with HOG features, deformable part models, bags of visual words and CNN features.
- Support-Vector Networks
Cortes and Vapnik's 1995 Machine Learning paper that added the soft margin to the optimal-hyperplane method, combined it with kernels ("convolution of the dot product") into the support-vector network, and tested it on US Postal Service and NIST digits, with its derivations, tables, inconsistencies and reception.
- t-SNE
t-distributed stochastic neighbor embedding, the nonlinear method that turns distances between images or feature vectors into neighbor probabilities and lays the points out in two or three dimensions so that a heavy-tailed Student-t distribution reproduces them; its cost and gradient, perplexity, the crowding problem, optimization tricks, Barnes-Hut acceleration, scikit-learn's defaults, and what a map does and does not show.
- Thermal and Infrared Camera
Cameras that image infrared light, from near-infrared cameras built on silicon to long-wave thermal cameras that see the heat objects emit, with Planck's law, emissivity, NETD and temperature measurement
- Thin Lens Model
The idealized lens of first-order optics, which focuses every ray from a point onto one conjugate point by the thin-lens equation 1/z_o + 1/z_i = 1/f, and which explains magnification, focusing, the pinhole equivalence of a focused camera and the size of defocus blur.
- Time-of-Flight Camera
A depth camera that lights the scene with its own modulated or pulsed light and measures, at every pixel, how long the light takes to return, from the phase of a continuous-wave signal or from photon arrival times.
- Unsupervised Learning
Learning from images without labels by modelling the distribution of the inputs; the main tasks (density estimation, clustering, dimensionality reduction, learned representations), how such models are evaluated when there are no correct answers, and kernel density estimation in outline.
Image Processing & Computational Photography
- A Non-Local Algorithm for Image Denoising
The 2005 CVPR paper by Buades, Coll and Morel that introduced non-local means, averaging pixels whose surrounding patches look alike anywhere in the image, and proposed method noise as a way to evaluate denoisers.
- Bilateral Filter
An edge-preserving smoothing filter that averages each pixel with neighbors that are both nearby and similar in value, so noise and texture are removed while strong edges stay sharp.
- Bilateral Filtering for Gray and Color Images
The 1998 ICCV paper by Tomasi and Manduchi that combined domain and range weighting into a single non-iterative, edge-preserving smoothing filter, named it bilateral filtering, and applied it to color images in CIE-Lab.
- BM3D
The patch-based denoiser that groups similar blocks into 3D stacks, shrinks the stacks jointly in a separable 3D transform and averages the overlapping estimates, first with hard thresholding and then with an empirical Wiener filter guided by the first result.
- Burst Photography
Capturing several short, equally exposed frames for one picture, aligning them to a reference and merging them robustly, as in Google's HDR+ pipeline; its tile-pyramid alignment, pairwise frequency-domain Wiener merge, tuning factor and artifacts, burst super-resolution and learned burst denoising, with a worked comparison against the temporal mean and 3-D DFT shrinkage.
- Color Constancy
Estimating the colour of the light that illuminates a scene from a single image so that it can be divided out, from gray world, white patch, shades of gray and gray-edge to gamut mapping, corrected moments and learned estimators such as CCC, FFCC and FC4, measured by the angular error.
- Computational Photography
Making photographs by combining capture and computation, from burst align-and-merge, HDR, multi-frame denoising and super-resolution to synthetic depth of field, panoramas, flash/no-flash pairs, light-field cameras and coded apertures and exposures; how these reached phones, and why phone images break the single-exposure assumptions of vision methods.
- Connected Component Labeling
Assigning one label to every set of foreground pixels joined by a chain of neighbours, for example with a two-pass raster scan that records label equivalences in a union-find structure.
- Convolution
The operation that replaces every pixel by a weighted sum of its neighbors, with the weights given by a kernel; the basis of linear filtering and of convolutional networks.
- Demosaicing
Reconstructing a full-colour image from the single colour that each photosite of a Bayer colour filter array records; bilinear, gradient-corrected (Malvar–He–Cutler), directional (Hamilton–Adams), colour-difference, residual-interpolation and learned methods, with their exactness, false colour and noise gain compared on images with known ground truth.
- Discrete Cosine Transform
The orthogonal transform that writes a block of pixels as a sum of sampled cosines, why its mirror-symmetric boundary lets it pack the energy of smooth image blocks almost as well as the optimal Karhunen–Loève transform, and how JPEG and SciPy define it.
- Distance Transform
An image transform that gives every pixel its distance to the nearest feature pixel; two-pass chamfer transforms with local masks, exact Euclidean transforms, the feature transform, and the error each chamfer mask makes.
- ESRGAN
The Enhanced Super-Resolution GAN of Wang et al. (2018), which rebuilt SRGAN with residual-in-residual dense blocks without batch normalization, a relativistic average discriminator and a perceptual loss before activation, and its successor Real-ESRGAN (2021), which trains the same generator on a second-order synthetic degradation with a U-Net discriminator.
- Eulerian Video Magnification for Revealing Subtle Changes in the World
The SIGGRAPH 2012 paper by Wu, Rubinstein, Shih, Guttag, Durand and Freeman that amplified invisible colour changes and small motions in ordinary video by band-passing each pixel's time series in a spatial pyramid and adding it back, without tracking or optical flow, and derived the bound (1 + α)δ(t) < λ/8 that limits how far the motion can be magnified.
- Fourier Transform
The decomposition of an image into two-dimensional sinusoidal gratings of different frequencies, orientations, amplitudes, and phases, and the frequency-domain view of filtering, sampling, and registration that it provides.
- Frequency Domain Filtering
Designing and applying linear filters as transfer functions on the discrete Fourier grid; ideal, Gaussian and Butterworth low-pass filters compared at a matched cutoff for ringing, edge width and passed noise, the two Butterworth gain conventions, and homomorphic filtering of illumination and reflectance in the log domain.
- Gamma Correction
The nonlinear transfer functions that map light to stored code values and code values back to light, such as sRGB, BT.709 and BT.1886, and why they decide how many visible steps an 8-bit image has.
- Gaussian Filter
A linear smoothing filter that replaces each pixel by an average of its neighbors weighted by a Gaussian, whose standard deviation sets the scale of the detail removed.
- High Dynamic Range Imaging
Capturing scenes whose brightness range exceeds what one exposure can record, by bracketing exposures, recovering the camera response and a radiance map, merging with noise-aware weights and removing ghosts, or with single-shot HDR sensors and learned single-image reconstruction, and compressing the result for display by tone mapping or exposure fusion.
- Histogram Equalization
A contrast enhancement method that maps each intensity level through the image's cumulative histogram so that output levels are used approximately uniformly, with adaptive variants such as CLAHE that equalize local regions.
- Image Blending
Joining two images, or parts of them, so that the transition is invisible; hard cuts, feathering, multiband (Laplacian-pyramid) blending and Poisson (gradient-domain) blending, and the trade-off each makes between a visible seam, overlapping textures and a shift in brightness.
- Image Compression
How images are stored in fewer bits, losslessly or with controlled loss; the transform, quantization and entropy-coding stages of JPEG, how a quality setting becomes a quantization table, what JPEG 2000 changes, and how rate is traded against distortion.
- Image Deblurring
Recovering a sharp image from a blurred one, covering the convolution model and point-spread function, inverse, Wiener and Tikhonov filtering, Landweber and Richardson–Lucy iterations, regularization and stopping, boundary handling and ringing, blind deconvolution, motion and defocus blur, and learned deblurring in outline.
- Image Dehazing
Removing haze from a single image, covering the haze imaging model with transmission and atmospheric light, the dark channel prior and its derivation, atmospheric-light estimation, soft-matting and guided-filter refinement, failure cases, other priors such as haze-lines, learned dehazing networks, and how dehazing is evaluated on synthetic haze.
- Image Denoising
Estimating a clean image from a noisy observation, removing random variation while keeping edges, texture, and fine detail.
- Image Enhancement
Making an image easier to see or more pleasing without a model of how it was degraded, by point, histogram, spatial, frequency, colour-balance and local-contrast operators, and how such results are judged when there is no ground truth.
- Image Filtering
Computing each output pixel from a neighborhood of input pixels, the family of operations behind smoothing, sharpening, edge detection, and denoising.
- Image Gradients
How image gradients measure the direction and strength of intensity change, and why they underpin edges, corners, and motion estimation.
- Image Histogram
The count of pixels at each intensity or color level, an empirical distribution that summarizes exposure and contrast and underlies thresholding, equalization, and histogram-based recognition and tracking.
- Image Interpolation
Estimating an image between its pixels by convolving the samples with a kernel; nearest, linear, cubic convolution, Mitchell–Netravali, Lanczos and B-spline kernels compared by their frequency response, ringing, reproduction of constants and the error they accumulate.
- Image Processing
Computing new images, or simple measurements, from images by point, neighborhood, global and geometric operations; how enhancement, restoration and learned methods build on them, and where image processing meets computer vision.
- Image Pyramid
A multi-scale representation of an image built by repeatedly smoothing and subsampling it, so that each level holds a coarser copy at half the resolution of the one below.
- Image Restoration
Estimating a clean image from a degraded observation through a model of the degradation; degradation models as a family, MAP estimation and the null space, deraining, desnowing and adverse-weather removal, all-in-one restoration of unknown corruptions, general restoration networks such as MPRNet and Restormer, and how results are compared.
- Image Sharpening
Raising the contrast of fine detail by adding back a multiple of what a blur removes; unsharp masking, Laplacian and fixed-kernel sharpening analysed exactly for their overshoot, halo width and noise gain, with the clipping and threshold rules of scikit-image and Pillow.
- Image Signal Processing Pipeline
The camera's processing chain from raw mosaic to finished picture as a designed system, covering why the order of its stages matters, how clipping, white balance, demosaicing and denoising interact, hardware against software pipelines, the HDR+ finishing pipeline, and inverse pipelines that unprocess images into raw data.
- Image Stitching
Combining overlapping photographs into one panorama, covering Brown and Lowe's automatic pipeline of feature matching, probabilistic image-match verification, rotation-and-focal bundle adjustment, straightening, gain compensation and multi-band rendering, together with projection surfaces, dynamic-programming seams and the defaults of OpenCV's stitching sample.
- Image Warping
Transforming the geometry of an image by a coordinate mapping, usually computed by sampling the input image at the inverse-mapped position of every output pixel.
- Integral Image
A table of cumulative sums, also called a summed-area table, from which the sum over any rectangle or box takes a fixed number of lookups in any number of dimensions, and the integer range and floating-point precision it needs.
- Laplacian Operator
The sum of the second partial derivatives of an image, an isotropic operator whose zero crossings mark edges, whose Gaussian-smoothed form detects blobs across scales, and which appears in sharpening, pyramids, and variational methods.
- Low-Light Image Enhancement
Recovering a well-exposed, natural-looking image from one taken in poor light; global curves, Retinex illumination estimation with max-RGB and edge-aware refinement, Zero-DCE's light-enhancement curves and non-reference losses, Retinexformer, and raw-to-RGB learning in the dark, with the benchmarks and metrics used to compare them.
- Mathematical Morphology
A theory and toolbox for processing images by shape, probing them with a small set called a structuring element through erosion, dilation, and the operators built from them, such as opening, closing, top-hats, and morphological reconstruction.
- Median Filter
A nonlinear filter that replaces each pixel by the median of its neighborhood, removing impulse noise while keeping step edges sharp.
- Non-Local Means
A denoising algorithm that replaces each pixel with a weighted average of pixels whose surrounding patches look similar, wherever they are in the image.
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig and colleagues' CVPR 2017 paper from Twitter that introduced SRResNet and SRGAN, trained a super-resolution network with a VGG feature loss plus an adversarial loss, and used a 26-rater mean opinion score study to argue that PSNR and SSIM miss what makes a 4× upscaled image look real.
- Point Operations
Operations that compute each output pixel from the input value at the same pixel alone, such as negatives, log and power-law maps, contrast stretches, windows and thresholds; why every one is a look-up table, and what an 8-bit table merges, skips and cannot invert.
- Recovering High Dynamic Range Radiance Maps from Photographs
Debevec and Malik's SIGGRAPH 1997 paper, which recovered a camera's or film's response curve from photographs taken at known exposure times by a smoothness-regularized linear least-squares fit, merged the photographs into a floating-point radiance map, and demonstrated it on a digital camera, scanned print film and synthetic motion blur.
- Sampling and Aliasing
How a continuous image becomes a grid of samples, when the samples determine it completely, and how detail too fine for the grid turns into false, coarser patterns.
- Sobel Operator
A pair of 3×3 kernels that estimate the horizontal and vertical image derivatives by combining a central difference with smoothing across it; the most common way to compute image gradients.
- Super-Resolution
Recovering a higher-resolution image from one or more low-resolution images; the degradation model, interpolation baselines, multi-frame reconstruction, example-based and sparse-coding methods, CNNs from SRCNN to transformers, perceptual, adversarial and diffusion approaches, video super-resolution, the standard benchmarks, and why PSNR and perceived quality disagree.
- Tone Mapping
Compressing a high-dynamic-range radiance map into the range a display or print can show, with global curves such as Reinhard's photographic operator, local operators such as automatic dodging-and-burning, and base–detail decompositions such as Durand and Dorsey's bilateral-filter method, and why local operators produce halos.
- Total Variation Denoising
Removing noise by minimizing the total variation of the image plus a quadratic fidelity term, the Rudin–Osher–Fatemi model, which keeps edges but lowers the contrast of small features and turns noisy ramps into staircases.
- Wavelet Transform
The multiresolution decomposition of an image into a coarse approximation and detail coefficients at octave-spaced scales and three orientations, computed by a cascade of two-channel filter banks; its orthogonal and biorthogonal forms, vanishing moments, boundary handling, and why images are sparse in it.
- Wiener Filter
The linear filter that minimizes the expected mean squared error of a restored image, built from the power spectra of signal and noise, for denoising and deconvolution, with its closed-form error, its estimated and parametric forms, and its local and transform-domain variants.
Features & Representation
- A Combined Corner and Edge Detector
Chris Harris and Mike Stephens's 1988 Alvey Vision Conference paper that replaced Moravec's shift test with the local auto-correlation matrix, introduced the det − k·trace² response, and used it to find corners and edges together for 3D interpretation of image sequences.
- A Computational Approach to Edge Detection
John Canny's 1986 IEEE TPAMI paper that stated explicit performance criteria for edge detectors, derived optimal operators from them, and introduced the detector built on Gaussian derivatives, non-maximum suppression, and hysteresis thresholding.
- Approximate Nearest Neighbor Search
Finding the database vectors closest to a query without comparing it with every one, by accepting an occasional wrong answer; why exact search stops paying off in high dimensions, the four families of indexes (randomized k-d trees with best-bin-first search, locality-sensitive hashing, inverted files with product quantization, and navigable small-world graphs), how their recall, speed and memory are measured, and the libraries that implement them.
- Bag of Visual Words
Representing an image by counts of quantized local descriptors ("visual words") from a learned vocabulary; how vocabularies are built, how descriptors are assigned to words, how tf-idf weighting and inverted files make retrieval fast, and the vocabulary tree, approximate k-means and soft assignment that scaled it to millions of images.
- Binary Descriptors
Local feature descriptors made of bits, each bit the outcome of an intensity comparison or another binary test around a keypoint, compared by the Hamming distance; covers BRIEF, BRISK and FREAK sampling patterns, the selection and learning of tests, matching in practice, and what bit length buys.
- Blob Detection
Finding compact regions that are brighter or darker than their surroundings, together with their size, from extrema of scale-normalized second-derivative measures such as the Laplacian of Gaussian, the difference of Gaussians and the determinant of the Hessian.
- BYOL
Bootstrap Your Own Latent, a self-supervised method in which an online network with a predictor learns to predict a moving-average target network's embedding of another view of the same image, with no negative pairs; how the predictor and stop-gradient keep it from collapsing, and SimSiam, which drops the moving average.
- Canny Edge Detector
The classic edge detection algorithm that smooths with a Gaussian, takes the gradient, thins edges by non-maximum suppression along the gradient direction, and links them with two thresholds and hysteresis.
- Contrastive Learning
Training an encoder so that embeddings of matching pairs, such as two augmented views of one image, are close and embeddings of non-matching pairs are far apart, through margin losses, the InfoNCE and NT-Xent softmax losses, and recipes such as SimCLR and MoCo.
- Contrastive Loss
The margin loss on pairs that pulls the embeddings of similar images together and pushes dissimilar ones apart until they are a margin away, from Chopra, Hadsell and LeCun's energy-based form (2005) to Hadsell, Chopra and LeCun's squared hinge (2006), with its gradients, why the push term prevents collapse, what the margin does and does not set, and its variants.
- Deep Features
The activations of a trained network's hidden layers used as general-purpose image descriptors, how they change from layer to layer, how to choose and read out a layer, and how representations at different layers or in different networks are compared, with centered kernel alignment and canonical correlation.
- DINO
A family of self-supervised models from Meta in which a student network learns to match the softmax output of a momentum teacher on different crops of the same image, with centering and sharpening to prevent collapse; DINOv2 and DINOv3 scale the recipe into frozen general-purpose backbones.
- Distinctive Image Features from Scale-Invariant Keypoints
David Lowe's 1999 ICCV paper and 2004 IJCV journal paper that introduced SIFT, a scale- and rotation-invariant local feature, and used it to recognize objects in cluttered, partly occluded images.
- Edge Detection
Locating the pixels where image intensity changes sharply, from derivative filters and the Canny detector to learned boundary detectors evaluated against human annotations.
- Embeddings
Fixed-length vectors that a learned or hand-designed map assigns to images so that similar images get nearby vectors, where they come from (classifier features, metric and contrastive training, joint image–text models), how they are normalized and whitened, the geometry of the spaces they live in, and what they are used for.
- Emerging Properties in Self-Supervised Vision Transformers
The ICCV 2021 paper by Caron, Touvron, Misra, Jégou, Mairal, Bojanowski and Joulin that introduced DINO, self-distillation with no labels, and reported that self-supervised Vision Transformers produce attention maps that outline objects and features that work as nearest-neighbor classifiers.
- Evaluating Representations
How the features of a pretrained encoder are judged by what a cheap readout can do with them, covering linear and least-squares probes, weighted nearest neighbours, low-shot probes and fine-tuning, the solvers and hyperparameters that move their results, and which changes of feature coordinates each protocol ignores.
- FAST Corner Detector
A corner detector that compares a pixel with the 16 pixels on a circle around it and accepts it when a long enough contiguous arc is all brighter or all darker by a threshold, with a learned decision tree that makes the test fast.
- Feature Descriptors
Vectors that summarize the image appearance around a keypoint so that the same scene point seen in different images gets similar vectors, from raw patches and gradient histograms to binary strings and learned embeddings.
- Feature Detection
Finding distinctive, repeatable points and regions in an image, such as corners, blobs, and stable regions, so that the same scene locations can be found again in other images.
- Feature Matching
Finding corresponding keypoints between two or more images of the same scene, from descriptor nearest-neighbor search to learned matchers.
- Gabor Filter
A linear filter whose kernel is a Gaussian envelope multiplied by a sinusoidal carrier, tuned to one spatial frequency and orientation, and the function that attains the joint space–frequency uncertainty bound.
- Global Image Descriptors
Descriptors that summarize a whole image in one vector, from colour histograms compared by Swain and Ballard's intersection to chromaticity and colour-ratio histograms, colour moments, spatial colour layouts and GIST; what each is invariant to, what that invariance costs, and how such vectors are compared and searched.
- Haar-like Features
Rectangle features that compare the summed intensities of adjacent image regions, computed in constant time at any position and scale from an integral image, the representation behind the Viola–Jones face detector.
- Harris Corner Detector
A corner detector that scores each pixel by how much a small window's intensities change under shifts in every direction, using the determinant and trace of the gradient structure tensor.
- Histogram of Oriented Gradients
A dense descriptor that summarizes an image window by histograms of gradient orientations in small cells, contrast-normalized over overlapping blocks; how it is computed, why its parameters are what they are, and how its variants and implementations differ.
- Histograms of Oriented Gradients for Human Detection
Dalal and Triggs's CVPR 2005 paper, which showed that dense grids of locally normalized gradient-orientation histograms with a linear SVM detect people far better than earlier feature sets, measured each design choice, and introduced the INRIA person dataset.
- Hough Transform
A voting method that finds lines, circles and other parametric shapes in an edge map by letting every edge pixel vote for all shapes that pass through it and taking the peaks of the vote accumulator.
- I-JEPA
A self-supervised method for Vision Transformers that predicts, from one visible block of an image, the target encoder's representations of several hidden blocks, learning semantic features without hand-crafted view augmentations or pixel reconstruction.
- Image Features
What an image feature is, how local features (detected keypoints with descriptors) differ from global ones that summarize a whole image, sparse and dense sampling, handcrafted and learned features, what each invariance buys and costs, and how features are used for matching, retrieval, recognition, tracking and 3D vision.
- Image Moments
Weighted sums of pixel coordinates that summarize a region's area, centroid, orientation and shape; central and normalized moments, Hu's seven invariants, complex and Zernike moments, and how far the invariants hold on real pixels.
- InfoNCE Loss
The softmax loss that asks a model to pick the true partner of a sample, such as the matching view of an image patch, among K candidates; it is a lower bound on mutual information that can never exceed log K, descends from noise-contrastive estimation, and underlies CPC, SimCLR, MoCo and CLIP.
- Invariance and Equivariance
How a feature or network responds when its input is transformed, either unchanged (invariance) or transformed in a predictable way (equivariance or covariance); the group-theoretic definitions, the ways each is built (weight sharing, pooling over a group, canonicalization, normalization), and why they hold exactly on pixels only for some transformations.
- Learned Local Features
Keypoint detectors and local descriptors trained from data rather than designed by hand, from patch descriptors such as L2-Net and HardNet to networks that detect and describe in one pass such as D2-Net, and why a point that is detected again is not necessarily one that can be matched.
- Local Binary Patterns
A texture operator that codes each pixel by which of its neighbours are at least as bright as it, and describes a region by the histogram of those codes; the operator, its uniform and rotation-invariant mappings, what it is and is not invariant to, and its implementations.
- LoFTR
A detector-free local feature matcher (Sun et al., CVPR 2021) that transforms coarse convolutional feature maps of two images with interleaved self- and cross-attention, matches every coarse position by dual-softmax or optimal transport, and refines the confident matches to sub-pixel accuracy by an expectation over a small correlation window.
- Masked Autoencoder
A self-supervised pre-training method for Vision Transformers that hides 75% of an image's patches, encodes only the visible ones, and trains a light decoder to reconstruct the missing pixels; the encoder is then fine-tuned for recognition, detection and segmentation.
- Masked Image Modeling
Self-supervised pre-training that hides part of an image and trains a network to predict what is hidden, with design choices in how to mask, what to predict and how heavy the prediction head is.
- Metric Learning
Learning a distance function from examples of what should count as similar, from Mahalanobis metrics learned with convex programs (Xing et al., LMNN, ITML) and NCA to deep siamese networks trained with pair, ranking and proxy losses, with the sampling problem, the open-set evaluation protocol and what a fair comparison of the losses found.
- MoCo
Momentum Contrast, a self-supervised contrastive method from Facebook AI Research (2020) that keeps a large queue of negative keys encoded by a slowly moving average of the query encoder; MoCo v2 changed the recipe, and MoCo v3 (2021) dropped the queue and trained Vision Transformers.
- ORB
Oriented FAST and Rotated BRIEF, a fast binary keypoint detector and descriptor that finds FAST corners on an image pyramid, ranks them by the Harris measure, orients them by the intensity centroid, and describes each with 256 learned intensity comparisons rotated to that orientation and matched by Hamming distance.
- ORB: An Efficient Alternative to SIFT or SURF
Rublee, Rabaud, Konolige and Bradski's ICCV 2011 paper that combined an oriented FAST detector with a rotation-aware, decorrelated BRIEF descriptor, reported it two orders of magnitude faster than SIFT, and released it in OpenCV.
- Pretext Tasks
Hand-designed prediction problems whose labels are computed from unlabeled images, such as the relative position of two patches, a shuffled jigsaw, a hidden region, the colors of a gray image or an applied rotation, solved only so that the network learns features; how they are built, why they invite shortcuts, and how to tell whether one teaches anything.
- Product Quantization
Compressing vectors such as image descriptors and embeddings into short codes by splitting each vector into subvectors and quantizing each with its own small k-means codebook; how the quantizer is trained, how distances are estimated from codes and how biased those estimates are, why the split and the rotation before it decide the error (optimized product quantization and eigenvalue allocation), and the locally optimized, additive and anisotropic variants.
- Representation Learning
Learning the features a vision system works with instead of designing them by hand, what makes a learned representation good, the supervised, self-supervised and image–text routes to one, and how representations are compared with linear probes, nearest neighbors, fine-tuning and transfer benchmarks.
- Scale Space
The family of Gaussian-smoothed copies of an image indexed by a continuous scale parameter, with the axioms that single out the Gaussian and the normalized derivatives that select a characteristic scale for each image structure.
- Self-Supervised Learning
Learning image representations from unlabeled images by deriving the training signal from the images themselves, through pretext tasks, agreement between augmented views, or masked prediction.
- Shape Descriptors
Numbers and histograms that summarize the outline or silhouette of an object (region measures such as circularity and solidity, Freeman chain codes, Fourier descriptors and shape contexts), how each is made invariant to position, size, rotation and starting point, and what that invariance costs.
- Siamese Network
A network that compares inputs by passing each one through the same weights and comparing the outputs; what weight sharing guarantees, pseudo-Siamese and 2-channel alternatives, learned comparison heads, triplet networks, the cost of comparing many items, and its uses from face verification and stereo matching to self-supervised learning.
- SIFT
The Scale-Invariant Feature Transform, which detects blob-like keypoints at extrema of a difference-of-Gaussians scale space and describes each with a 128-dimensional histogram of gradient orientations that is invariant to scale and rotation.
- SimCLR
The contrastive self-supervised method of Chen, Kornblith, Norouzi and Hinton (2020), which trains a ResNet so that two augmented views of one image agree under the NT-Xent loss with in-batch negatives, through strong crop-and-colour augmentation, a nonlinear projection head and large batches; SimCLR v2 scaled it up for semi-supervised learning.
- Similarity and Distance Measures
The functions that score how alike two feature vectors, histograms or binary codes are (Euclidean and other ℓp distances, cosine similarity and the dot product, Mahalanobis, Hamming, histogram measures such as χ², intersection and Hellinger, and the earth mover's distance), which of them are true metrics, and where each is used in descriptor matching, histogram comparison and embedding retrieval.
- SuperGlue
A learned feature matcher (Sarlin et al., CVPR 2020) that refines two sets of keypoint descriptors with a graph neural network of alternating self- and cross-attention and turns their similarities into a partial assignment by optimal transport with dustbins, solved with the Sinkhorn algorithm.
- SuperPoint
A fully convolutional network from Magic Leap (2018) that detects interest points and computes 256-dimensional descriptors for a whole image in one forward pass, trained without manual labels through synthetic pre-training and homographic adaptation.
- SURF
Speeded-Up Robust Features, a scale- and rotation-invariant keypoint detector and descriptor that replaces Gaussian derivatives with box filters and Haar wavelets evaluated on an integral image, so that every filter costs the same at every scale.
- Template Matching
Finding where a small reference image, the template, occurs in a larger image by scoring every candidate position with a similarity measure such as the sum of squared differences or normalized cross-correlation; how the scores relate, how they are computed fast with the FFT and integral images, how template size and texture set localization accuracy, and how rotation, deformation and clutter break the method.
- Texture Analysis
The description of image regions by the statistics of their local structure, with co-occurrence matrices, filter-bank energies, local binary patterns, textons and learned filter banks, and what each statistic can and cannot tell apart.
- Triplet Loss
A loss for learning embeddings that asks an anchor image to be closer to a positive of its own class than to a negative of another class by a margin, how it grew out of relative-comparison metric learning, and how the choice of triplets (random, semi-hard, batch-hard, distance-weighted) decides whether training learns, stalls or collapses.
- Video Google: A Text Retrieval Approach to Object Matching in Videos
Sivic and Zisserman's ICCV 2003 paper, extended in IEEE TPAMI in 2009, that quantized affine covariant region descriptors into visual words and searched feature films for a user-outlined object with the tf-idf weighting, stop lists and inverted files of text search engines.
- VLAD
The vector of locally aggregated descriptors, which describes an image by summing, for each visual word of a small k-means vocabulary, the residuals of the local descriptors assigned to it; the encoding, its L2, signed square-root and intra-normalizations and why burstiness motivates them, vocabulary adaptation, its relation to bag of words and Fisher vectors, and its variants up to NetVLAD.
Geometry & 3D Vision
- 2D and 3D Geometric Transformations
The nested families of Euclidean, similarity, affine and projective transformations of the plane and of space, their degrees of freedom, what each preserves, why each forms a group, and the Lie groups SO(3), SE(3) and Sim(3) used to estimate rotations, poses and similarities.
- 3D Foundation Models
Large feed-forward networks that regress pointmaps, depth and cameras for one, two or hundreds of uncalibrated images, from DUSt3R and MASt3R to VGGT, Fast3R, CUT3R and MoGe; their training data, the alignment conventions under which they are evaluated, and their documented limits.
- 3D Gaussian Splatting
A radiance-field model that represents a scene as millions of anisotropic 3D Gaussians, rasterizes them by sorting and alpha blending, and fits them to posed photographs; the 2023 model, its training and limits, and the variants for anti-aliasing, surfaces, compression, motion, feed-forward prediction and SLAM.
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering
Kerbl, Kopanas, Leimkühler and Drettakis's 2023 ACM Transactions on Graphics paper, a SIGGRAPH 2023 Best Paper, which replaced the radiance field's network and ray marching with optimized anisotropic 3D Gaussians and rendered unbounded real scenes in real time.
- 3D Object Detection
Finding objects in 3D space and describing each by a class, a score and an oriented 3D box, from LiDAR point clouds, stereo or single images, several cameras, RGB-D scans or combinations of sensors, and how the KITTI, nuScenes, Waymo Open and indoor benchmarks decide which detections count.
- 3D Reconstruction
Recovering the 3D shape of objects and scenes from images, depth maps, or range scans, as point clouds, meshes, volumes, or implicit and radiance fields, and measuring how accurate and complete the result is.
- 3D Representations
The main ways to store 3D shape and scenes, from depth maps, point clouds and meshes to voxel grids, distance and occupancy fields, radiance fields and 3D Gaussians, and what each makes easy or hard.
- 6-DoF Object Pose Estimation
Estimating the 3D rotation and translation of a known rigid object, or of an object seen only at test time, from an RGB or RGB-D image; template, point-pair, keypoint, dense-coordinate, direct-regression and render-and-compare methods, symmetry and pose ambiguity, the BOP benchmark with its datasets and its VSD, MSSD and MSPD errors, and category-level and model-free pose in outline.
- A Computer Algorithm for Reconstructing a Scene from Two Projections
H. C. Longuet-Higgins's 1981 Nature letter that introduced to computer vision the matrix now called the essential matrix and the linear eight-point algorithm for recovering relative pose and scene structure from two calibrated views.
- A Flexible New Technique for Camera Calibration
Zhengyou Zhang's 1999 ICCV paper and 2000 IEEE TPAMI paper that calibrated a camera from a few views of a printed planar pattern, using a closed-form solution from plane homographies followed by maximum-likelihood refinement with radial distortion.
- A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
Scharstein and Szeliski's 2002 IJCV paper that broke dense stereo algorithms into four building blocks, compared them on images with ground truth, and started the Middlebury stereo benchmark.
- A Volumetric Method for Building Complex Models from Range Images
Brian Curless and Marc Levoy's SIGGRAPH 1996 paper that merged aligned laser range scans into a cumulative weighted signed distance function on a voxel grid, filled holes by carving empty space, and built the Stanford dragon and Happy Buddha models.
- Alignment by Maximization of Mutual Information
Paul Viola and William Wells's ICCV 1995 paper and its 1997 journal version, which aligned a model to an image, or one imaging modality to another, by maximizing the mutual information between their intensities, estimated with Parzen windows from small random samples and maximized by stochastic gradient ascent.
- Augmented Reality
Overlaying computer-generated content on a live view of the real world, registered in 3D and updated in real time, which requires tracking the camera pose, understanding the scene's geometry and lighting, and displaying the result with low latency.
- Bundle Adjustment
The joint nonlinear least-squares refinement of all camera parameters and 3D points in a reconstruction, made practical by the sparse structure of the problem and the Schur complement.
- Camera Calibration
Estimating a camera's intrinsic parameters, lens distortion, and pose from images of a target with known geometry, usually by a closed-form initialization followed by minimizing reprojection error.
- Camera Pose
The position and orientation of a camera as a rigid transform in SE(3), the two directions it is stored in (world-to-camera and camera-to-world), the axis conventions of OpenCV, COLMAP, OpenGL and NeRF, and how pose errors are measured.
- Cost Volume
A tensor of matching costs or similarities between each pixel of one image and its candidate correspondences in another, indexed by pixel position and candidate disparity, displacement, or depth.
- Deep Learning on 3D Data
How neural networks consume 3D data as voxel grids, rendered views, point sets, meshes and graphs, or implicit functions, the main designs for each, the tasks and benchmarks they are measured on, and the trade-offs between them.
- Depth Anything
A family of monocular depth models that predict relative inverse depth with a DINOv2 encoder and a DPT decoder, trained by teacher–student pseudo-labelling of 62 million unlabeled images, with a synthetic-only teacher in V2.
- Depth Estimation
Recovering the distance to the scene at every pixel of an image, from stereo pairs, many views, active sensors, or a single image.
- Depth Map
An image whose pixels store the distance to the scene, usually the depth Z along the optical axis, so that each pixel with a known camera can be lifted to a 3D point.
- Differentiable Rendering
Rendering that also returns the derivatives of pixel values with respect to geometry, camera, materials and lighting, so that scene parameters can be fitted to images by gradient descent; why rasterization breaks gradients at visibility edges, and the approximate, soft, physically based and volumetric renderers that restore them.
- Direct Linear Transform (DLT)
A linear method that estimates a projective mapping, such as a homography or a camera matrix, from point correspondences by stacking two equations per point into a homogeneous system and solving it with the singular value decomposition.
- Direct Methods
Estimating camera motion and scene structure by minimizing the photometric error between images, the differences of their pixel intensities, instead of the reprojection error of matched features.
- Direct Sparse Odometry
A monocular visual odometry algorithm that minimizes a photometric error over a sparse, evenly sampled set of pixels, jointly optimizing poses, inverse depths, affine brightness parameters and intrinsics in a sliding window of keyframes, on images corrected with a photometric calibration.
- Disparity
The horizontal shift, in pixels, between the images of the same scene point in a rectified stereo pair; it is inversely proportional to depth.
- DROID-SLAM
A deep visual SLAM system that recurrently revises dense optical flow between frames of a frame graph and turns each revision into camera-pose and per-pixel inverse-depth updates through a differentiable dense bundle adjustment layer.
- DUSt3R
A transformer that regresses, for a pair of uncalibrated and unposed images, a 3D point for every pixel of both images in the first camera's frame, from which depth, focal length, relative pose and matches are read off and many pairs are globally aligned; MASt3R adds a matching head and metric pointmaps.
- Epipolar Geometry
The geometry of two views of the same scene, in which the two camera centers and any scene point span a plane, so that a point in one image can only match points on a single line in the other.
- Essential Matrix
The 3 × 3 matrix E = [t]×R that encodes the relative rotation and translation direction between two calibrated views, how it is estimated from point matches, and how the camera pose is recovered from it.
- Factor Graphs
A graphical model that writes an estimation problem as a product of factors over unknown variables, the standard way to formulate SLAM, structure from motion and sensor fusion as sparse nonlinear least squares and to solve them in batch or incrementally.
- Five-Point Algorithm
The minimal solver for the relative pose of two calibrated cameras, which computes every essential matrix consistent with five point matches by reducing the problem to the roots of a tenth-degree polynomial.
- Fundamental Matrix
The rank-2 3 × 3 matrix F that encodes the epipolar geometry of two uncalibrated views through the constraint x'ᵀFx = 0, and how it is estimated from point correspondences.
- Hand-Eye Calibration
Estimating the fixed rigid transform between a robot's hand and a camera mounted on it from pairs of hand and camera motions, by solving AX = XB or AX = YB with closed-form solvers such as Tsai–Lenz, Park–Martin, Horaud–Dornaika and dual quaternions and refining nonlinearly, with the motion conditions for a unique solution.
- Homogeneous Coordinates
Representing a point by a vector with one extra coordinate, defined only up to scale, so that translations, perspective projections, and points at infinity can all be handled with matrix multiplication.
- Homography
An invertible projective transformation of the plane, a 3×3 matrix defined up to scale, that relates two images of a planar scene or two images taken by a camera rotating about its center.
- Image Registration
Aligning two images of the same scene or object, taken at different times, from different viewpoints or with different sensors, by finding the geometric transformation that maps points in one onto the corresponding points in the other.
- In Defense of the Eight-Point Algorithm
Richard Hartley's 1995 ICCV paper and 1997 journal paper, which showed that translating and scaling image coordinates before running the linear eight-point algorithm makes it a practical estimator of the fundamental matrix, nearly as accurate as iterative methods.
- Iterative Closest Point (ICP)
An algorithm that rigidly aligns two point clouds without known correspondences by alternating closest-point matching with a least-squares estimate of the rotation and translation.
- Kabsch Algorithm
The closed-form least-squares rotation, translation and optional scale that align two sets of corresponding 3D points, computed from an SVD (Kabsch, Arun et al., Umeyama) or a unit-quaternion eigenvector (Horn), with the reflection correction, weighting and degenerate configurations.
- KinectFusion
A real-time dense RGB-D SLAM system that fuses every depth frame into a TSDF volume on the GPU and tracks the camera by point-to-plane ICP against a surface ray cast from that volume.
- Loop Closure
Recognizing that a camera or robot has returned to a place it has already mapped and turning the revisit into a constraint that removes accumulated drift; appearance-based detection with bags of words (DBoW2, FAB-MAP, Bayesian filtering), geometric verification, LiDAR loop closure, perceptual aliasing and the cost of false positives, and evaluation by precision and recall.
- Marching Cubes
An algorithm that turns a scalar field sampled on a 3D grid, such as a signed distance volume, into a triangle mesh of one level set, cube by cube, with a lookup table and linear interpolation along edges.
- Marching Cubes: A High Resolution 3D Surface Construction Algorithm
William Lorensen and Harvey Cline's SIGGRAPH '87 paper that built triangle models of constant-density surfaces from CT, MR and SPECT slices with a cube-by-cube case table, linear interpolation along edges and normals from the data gradient.
- MiDaS
A family of monocular depth models from Intel Labs that predict relative inverse depth, trained on several datasets at once with a scale-and-shift-invariant loss and judged by zero-shot transfer to datasets never seen in training.
- Monocular Depth Estimation
Predicting a depth map from a single image, why one view fixes depth only up to scale, and how relative and metric models are trained with supervised, scale-and-shift-invariant and self-supervised losses and then evaluated.
- Multi-View Geometry
The geometry that relates cameras, scene points, and their images across one, two, three, or many views, and that determines what can be reconstructed from images and up to which ambiguity.
- Multi-View Stereo
Recovering dense 3D geometry, as depth maps, point clouds, meshes, or volumes, from many images whose camera poses and intrinsics are known.
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng's ECCV 2020 paper, a Best Paper Honorable Mention, which fitted a multilayer perceptron of density and colour to one scene's posed photographs through volume rendering and outperformed earlier view synthesis methods.
- Neural Implicit Representations
Coordinate-based networks that store a shape, an image or another field in their weights, answering a query at any continuous point; how they are conditioned, trained from 3D data, raw points or images, why plain ReLU networks blur detail, and how surfaces are extracted from them.
- Neural Radiance Fields
NeRF, a network that maps a 3D position and viewing direction to density and colour and is fitted to posed photographs through volume rendering; the original 2020 model, how it is trained and rendered, its limits, and the family that made it anti-aliased, unbounded, fast, robust to photo collections, pose-free and dynamic.
- Neural Rendering
Image synthesis that combines learned components with the physics of rendering, from 2D networks that turn rasterized features into photographs to 3D scene representations fitted to photographs through a differentiable renderer; the main families, why gradients through rendering matter, and how results are measured.
- Novel View Synthesis
Rendering a scene from a camera that took no picture of it, given images and usually their camera poses; the variants of the problem, the families of methods from image warping to radiance fields and diffusion models, the benchmarks, and why the results are hard to measure.
- ORB-SLAM
The feature-based visual SLAM system that uses ORB features for tracking, mapping, relocalization and loop closing in three parallel threads, with covisibility and Essential Graphs, automatic monocular initialization, Sim(3) loop correction, and its successors ORB-SLAM2 (stereo and RGB-D) and ORB-SLAM3 (visual-inertial and multi-map).
- Parametric Human Body Models
Statistical models that turn a small vector of shape and pose parameters into a full 3D body mesh; SCAPE and SMPL, linear blend skinning on a kinematic tree, shape and pose blend shapes, the hand, face and whole-body models MANO, FLAME and SMPL-X, fitting and regressing them from images, and their licence terms.
- Perspective-n-Point (PnP)
Estimating the position and orientation of a calibrated camera from known 3D points and their 2D image projections, the absolute pose problem behind visual localization, augmented reality, and camera tracking.
- Photo Tourism: Exploring Photo Collections in 3D
Snavely, Seitz and Szeliski's 2006 SIGGRAPH paper and 2008 IJCV journal paper that recovered camera poses and sparse 3D points from internet photos of landmarks and used them to browse the photos in 3D.
- Photogrammetry
Measurement from photographs as a surveying discipline, covering aerial and close-range image networks, orientation and its vocabulary, ground control and georeferencing, camera self-calibration and the doming error, accuracy assessment at independent check points, orthophotos, surface models and point clouds, and heritage, geomorphology and engineering uses.
- Photometric Stereo
Recovering surface orientation and albedo at every pixel from several images taken from a fixed viewpoint under different lighting, from Woodham's Lambertian least squares to uncalibrated, robust and learned methods and the DiLiGenT benchmark.
- Plane Sweep Stereo
A multi-view matching algorithm that sweeps a family of planes through the scene, warps every image onto each plane by the homography the plane induces, and keeps, for each pixel, the plane on which the images agree best.
- Point Cloud
A set of 3D points sampled from the surfaces of a scene, possibly with color, intensity, or normals per point, and the most direct output of 3D sensors and reconstruction.
- Point Cloud Registration
Estimating the transformation that aligns two or more point clouds of the same object or scene, by local refinement such as ICP, by global methods built on 3D feature correspondences, by globally optimal search, or by probabilistic and non-rigid models, with the benchmarks and metrics used to compare them.
- Point Cloud Segmentation
Assigning a label to every point of a 3D point cloud, by semantic class, by object instance or by object part, with point-based, sparse-voxel, range-image and transformer networks, and the indoor, outdoor and object benchmarks on which they are compared.
- PointNet
A deep network that classifies and segments raw point clouds by applying one shared MLP to every point and combining the results with max pooling, a symmetric function, together with its hierarchical successor PointNet++.
- PointPillars
A LiDAR 3D object detector that groups points into vertical pillars, encodes each pillar with a simplified PointNet, scatters the features into a bird's-eye-view pseudo-image and detects boxes with a 2D convolutional backbone and an SSD-style head, avoiding the 3D convolutions of VoxelNet and SECOND.
- Poisson Surface Reconstruction
A method that turns an oriented point cloud into a watertight mesh by finding the indicator function whose gradient best matches the point normals, which reduces to solving a Poisson equation.
- Polygon Mesh
A surface stored as vertices joined into polygonal faces, usually triangles, with the connectivity, orientation, normals and texture coordinates that rendering, simulation and 3D reconstruction rely on.
- Pose Graph Optimization
Estimating a trajectory of poses from noisy relative-pose measurements by nonlinear least squares on SE(2), SE(3) or Sim(3), with the initialization, robust back-ends and certifiable solvers that decide whether the optimizer finds the right answer.
- Projective Geometry
The geometry of properties that survive perspective projection, such as straightness, incidence, and the cross-ratio, which underlies how computer vision models cameras and relates images.
- Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography
Martin Fischler and Robert Bolles's 1981 Communications of the ACM paper that introduced RANSAC and used it to solve the location determination problem, including a closed-form analysis of the perspective-three-point problem.
- RANSAC
Random sample consensus, a randomized method that fits a model to data containing many outliers by repeatedly fitting minimal random samples and keeping the model that the most data points agree with.
- Reprojection Error
The image distance, in pixels, between where a point was measured and where the estimated cameras and 3D structure say it should appear; the standard cost minimized in calibration, triangulation, pose estimation, and bundle adjustment.
- Rotation Representations
The ways a 3D rotation can be written down (rotation matrices, Euler angles, rotation vectors, unit quaternions, 6D vectors and the Lie algebra), how they convert into each other, and where each one fails.
- Self-Supervised Depth Estimation
Training a single-image depth network without depth labels, by warping one view of a stereo pair or video into another and minimizing the photometric error, with the losses, masks and evaluation protocol that make it work.
- Semi-Global Matching (SGM)
A dense stereo algorithm that approximates a global smoothness energy by summing one-dimensional dynamic-programming path costs from several directions through every pixel.
- Shape from Shading
Recovering the shape of a smooth surface from the brightness variation in a single image under known lighting and reflectance, from Horn's image irradiance equation and its ambiguities to characteristic strips, variational, local, linear and viscosity-solution methods and their benchmarks.
- Signed Distance Function
A scalar field giving the distance to the nearest surface point, negative inside and positive outside, whose zero level set is the surface and whose gradient is the surface normal.
- Simultaneous Localization and Mapping (SLAM)
Building a map of an unknown environment while estimating the trajectory of the sensor moving through it, and using recognized revisits to keep both globally consistent.
- Sparse Convolutional Networks
Convolutional networks that store only the occupied sites of a voxel grid, as coordinates and feature rows, and convolve them through precomputed input–output pairs, with regular, submanifold and generalized sparse convolutions and the libraries that run them on LiDAR and indoor scans.
- Stereo Matching
Finding, for every pixel of one rectified stereo image, its corresponding pixel on the same row of the other image, producing a disparity map from which depth follows.
- Stereo Processing by Semiglobal Matching and Mutual Information
Heiko Hirschmüller's 2005 CVPR paper and 2008 TPAMI journal paper that introduced semi-global matching, a dense stereo method combining a pixel-wise mutual information cost with a smoothness energy approximated along many one-dimensional paths.
- Stereo Rectification
Warping a pair of stereo images by one homography each so that corresponding epipolar lines become the same image row, turning the correspondence search into a scan along rows.
- Stereo Vision
Recovering depth from two images taken by cameras with a known relative pose, by measuring how far each point shifts between the views.
- Structure from Motion
Recovering the poses and intrinsics of the cameras that took a set of images, together with a sparse 3D point cloud of the scene, from the images alone, by incremental or global reconstruction.
- Surface Reconstruction
Turning points, depth maps or implicit functions into a continuous surface, usually a triangle mesh, by interpolating or approximating the samples under assumptions about smoothness, visibility or learned shape, and measuring the result against a known surface.
- Triangulation
Recovering the 3D position of a point from its projections in two or more images taken by cameras whose calibration and poses are known.
- TSDF Fusion
Fusing many depth maps with known camera poses into one voxel grid of truncated signed distances by a per-voxel weighted average, then extracting the surface at the zero level set.
- Vanishing Points and Lines
The image points where parallel scene lines appear to meet, and the image lines that collect the vanishing points of a plane, with how they are detected and what they reveal about a camera's orientation, focal length and the scene's dimensions from a single image.
- Variational Shape from Shading
Iterative grid solvers for shape from shading that minimize the brightness error together with smoothness or integrability terms, from Ikeuchi and Horn's stereographic scheme to Horn's coupled height-and-gradient method, with a worked example.
- Visual Localization
Estimating the full 6-DoF position and orientation of a camera from an image, against a map of a known scene; structure-based 2D–3D matching with PnP and RANSAC, hierarchical localization, scene coordinate regression, absolute and relative pose regression and their documented limits, and the 7-Scenes, Cambridge Landmarks and Aachen Day-Night benchmarks with their metrics.
- Visual Odometry
Estimating the motion of a camera, and of the vehicle or device carrying it, incrementally from a video stream, frame after frame, without building a globally consistent map.
- Visual-Inertial Odometry
Estimating the motion of a camera rigidly attached to an inertial measurement unit by fusing image features with gyroscope and accelerometer readings, which makes metric scale and the direction of gravity observable for a single camera.
- Volume Rendering
How an image is formed from a field of density and colour by integrating emission and absorption along each camera ray, and the alpha-compositing sum that NeRF and 3D Gaussian splatting use to approximate it.
- Voxel Grid
A regular 3D grid of small cubes that stores occupancy, signed distance or other values per cell, and the dense, octree, hashed and VDB structures that make such grids affordable at high resolution.
Deep Learning for Vision
- Activation Function
The fixed nonlinear function a neural network applies to each unit's weighted sum, such as ReLU or GELU, without which any stack of layers would collapse into a single linear map.
- Adam
A stochastic gradient method that gives every network weight its own step size, dividing a running average of the gradient by the root of a running average of its square; the Vision Transformer, for example, was pre-trained with it.
- Adversarial Examples
Inputs changed by a small, deliberately chosen perturbation so that a classifier gets them wrong; covers the linear explanation, Lp threat models, FGSM, PGD, DeepFool and Carlini–Wagner attacks, universal, physical and black-box attacks, adversarial training, gradient masking, evaluation with AutoAttack, certified defences and the trade-off between robustness and accuracy.
- AlexNet
The eight-layer convolutional network of Krizhevsky, Sutskever and Hinton (2012), five convolutional and three fully connected layers with about 60 million parameters, almost all in the fully connected head; its two-GPU layout survives as grouped convolution, and a single-tower variant is the AlexNet that libraries ship today.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
The ICLR 2021 paper by Dosovitskiy and eleven Google Research colleagues that introduced the Vision Transformer (ViT) and showed that a plain transformer on image patches, pre-trained on 14 to 300 million images, matches or beats the best convolutional networks at lower pre-training cost.
- Attention Is All You Need
The 2017 NIPS paper by Vaswani and colleagues at Google that introduced the transformer, an encoder–decoder for machine translation built from attention alone, which later became a standard architecture for image classification and object detection.
- Attention Mechanism
A neural network operation that answers a query by taking a weighted average of a set of values, with weights computed from how well the query matches each value's key, so that a model can focus on the relevant image regions, words or features for each output; covers additive and dot-product scoring, soft and hard attention, and its history from machine translation to vision.
- Autoencoder
A network trained to reconstruct its input through an encoder and a decoder, so that the code between them captures the structure of the data; used for compression, denoising, anomaly detection, pretraining and the latent spaces of latent diffusion models.
- Automatic Differentiation
Computing exact derivatives of a program by applying the chain rule to each elementary operation as it runs, in forward mode (dual numbers, Jacobian-vector products) or reverse mode (a tape, vector-Jacobian products), as deep-learning libraries do.
- Backbone Network
The pretrained feature-extracting network at the base of a detector, segmenter or other vision model, which turns an image into feature maps at several resolutions for task-specific necks and heads to read.
- Backpropagation
The algorithm that computes the gradient of a network's loss with respect to every weight by applying the chain rule backward through the computation graph, at a cost comparable to one forward pass.
- Batch Normalization
A layer that normalizes each channel of a feature map with the mean and variance of the current mini-batch, then rescales it with learned parameters; it allows higher learning rates in convolutional networks but depends on the batch size and behaves differently in training and inference.
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe and Christian Szegedy's 2015 ICML paper that introduced batch normalization and used it to train an Inception image classifier in a fraction of the steps, with higher learning rates and without dropout.
- Class Imbalance
When some classes have far more training images than others, as in long-tailed recognition, classifiers favour the frequent classes; how imbalance is measured, why it hurts, and the remedies of resampling, reweighting, rebalanced losses, decoupled training and post-hoc logit adjustment for a prior shift.
- Computer Vision Foundation Models
Large models pretrained once on broad image, image–text or mask data and adapted to many vision tasks, covering where the term comes from, the main families (contrastive image–text encoders, self-supervised encoders, promptable segmenters, open-set detectors, vision–language models and generative models), how they are adapted and evaluated, how they scale, and their limits and risks.
- ConvNeXt
A pure convolutional network family (Liu et al., CVPR 2022) obtained by modernizing ResNet-50 step by step toward the design of a hierarchical vision transformer, with patchify stem, 7×7 depthwise convolutions, inverted bottlenecks, GELU and LayerNorm, and its successor ConvNeXt V2 (CVPR 2023), which adds global response normalization and fully convolutional masked-autoencoder pre-training.
- Convolutional Neural Network
A neural network built from convolutional layers, small learned filters applied at every image position, interleaved with nonlinearities and downsampling, so that features grow from edges to object parts while the parameter count stays independent of image size.
- Cross-Attention
Attention in which one set of tokens asks the questions and another set supplies the answers, so that image features can read a text prompt, learned object queries can read an image, or a small set of latents can summarize a large input; the output has one row per query and the cost grows with the product of the two set sizes.
- Cross-Entropy Loss
The loss that scores a predicted class distribution by the negative log-probability it gives the true label; it is the standard loss for image classification and per-pixel segmentation, the negative log-likelihood of a softmax or sigmoid output, and differs from the KL divergence by a constant.
- Data Augmentation
Training a network on randomly transformed copies of its training images, such as crops, flips, colour changes, erasing and mixing, so that it learns invariances the small training set cannot show, provided each transformation keeps the label correct.
- Decoupled Weight Decay Regularization
Ilya Loshchilov and Frank Hutter's ICLR 2019 paper, first posted to arXiv in November 2017 as "Fixing Weight Decay Regularization in Adam", which showed that an L2 penalty is not weight decay under Adam, introduced AdamW and SGDW, and tested them on CIFAR-10 and ImageNet32x32.
- Deep Learning
Learning many layers of representation directly from pixels with neural networks trained by gradient descent; what makes depth useful, the history from the perceptron through the 2012 ImageNet result to foundation models, the main architecture families, how training at scale works, and a map of the deep-learning articles on this site.
- Deep Learning (2015 paper)
The 2015 Nature review by LeCun, Bengio and Hinton that defined deep learning as representation learning with many levels, explained backpropagation, convolutional and recurrent networks for a general scientific audience, and forecast the rise of unsupervised learning and attention; its argument, figures, claims, errors and reception.
- Deep Residual Learning for Image Recognition
The 2016 CVPR paper by He, Zhang, Ren and Sun of Microsoft Research that introduced residual networks (ResNets), trained them to 152 layers on ImageNet, and won the ILSVRC 2015 classification task.
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
The 2015 ICCV paper by He, Zhang, Ren and Sun of Microsoft Research that introduced the parametric ReLU (PReLU) and the rectifier-aware initialization now called He or Kaiming initialization, and reported 4.94% top-5 error on the ImageNet test set, below the 5.1% reported for a trained human annotator.
- DenseNet
The densely connected convolutional network of Huang, Liu, van der Maaten and Weinberger (CVPR 2017), in which every layer of a dense block receives the concatenated feature maps of all earlier layers, with its growth rate, bottleneck and compression (DenseNet-BC), configurations, results, memory cost and descendants.
- Depthwise Separable Convolution
A convolutional layer factored into a depthwise convolution, one spatial filter per input channel, and a pointwise 1×1 convolution that mixes channels, cutting parameters and multiply-adds by a factor of about k² at the price of a restricted kernel; the building block of Xception, MobileNet and EfficientNet.
- Dilated Convolution
The convolution whose filter taps are spaced r pixels apart, how it widens the receptive field without extra weights or lost resolution, its roots in the à trous wavelet algorithm, the gridding that stacked dilations cause and how degridding removes it, and its uses in segmentation, optical flow, tracking, audio and temporal models.
- Distributed Training
Training one neural network on many devices at once, by splitting the batch (data parallelism), the training state (sharded data parallelism, ZeRO and FSDP) or the model itself (tensor and pipeline parallelism), so that gradients match those of training on a single device while memory and communication are divided.
- Domain Adaptation
Adapting a model trained on labelled images from one domain to a shifted target domain, usually with only unlabelled target images; the settings, the theory of what alignment can and cannot guarantee, discrepancy, adversarial, pixel-level, self-training and test-time methods, domain generalization, and the benchmarks.
- Dropout
A regularization technique that randomly zeroes units of a neural network during training, with structured variants for convolutional feature maps and a test-time form for uncertainty estimates.
- Early Stopping
Ending training at the checkpoint with the lowest validation error rather than at convergence, with patience rules, checkpoint selection, retraining on all the data, and its equivalence to L2 regularization for linear models trained by gradient descent.
- EfficientNet
Google's family of convolutional networks built by scaling one searched baseline, EfficientNet-B0, in depth, width and input resolution together with a single compound coefficient (B1 to B7, 2019), followed by EfficientNetV2 (2021), which mixes fused and depthwise blocks and grows the image size during training to train faster.
- Encoder–Decoder
A network design in which an encoder compresses the input into a coarse, many-channel representation and a decoder expands it back into the output space, usually at the input's resolution; the standard architecture for segmentation, depth, flow, image translation and pose heatmaps, and, without the spatial grid, for sequence-to-sequence models.
- Feature Map
One channel of a convolutional layer's output, a grid that records how strongly one learned filter responds at each position of the image, together with how its size is set by kernel, stride and padding and how maps change from edges to objects with depth.
- Feature Visualization
Methods that show what a unit, channel, class or layer of a vision network responds to, by synthesizing an input that maximizes it (activation maximization with its regularizers, image parameterizations and learned priors), by ranking real images that activate it, by projecting or inverting activations, or by scoring it against labelled concepts (Network Dissection and its successors), with the evidence on how far such pictures can be trusted.
- Fine-Tuning
Adapting a pretrained network to a new task by continuing to train some or all of its weights, and the choices that decide how well it works, including which layers to freeze, how fast each layer learns, how the new head starts, and how to keep what pretraining learned.
- Focal Loss
A reshaped cross-entropy loss that multiplies each example's loss by (1 − p_t)^γ, so that the many well-classified examples, such as the easy background of a dense detector, contribute little; its definition, α and γ, its gradient and minimiser, the under-confident probabilities it produces, and its variants for heatmaps, quality scores, multi-label classification and long-tailed detection.
- Fully Connected Layer
The layer that connects every input to every output through one weight matrix and a bias, how many parameters it holds in classic image classifiers, how global average pooling replaced it in their heads, why it is the same map as a convolution whose kernel covers its whole input or a 1×1 convolution applied at every position, and where it survives in modern networks.
- Going Deeper with Convolutions
The 2015 CVPR paper by Szegedy and colleagues at Google that introduced the Inception architecture and GoogLeNet, a 22-layer network designed for a budget of 1.5 billion multiply-adds, which won ILSVRC 2014 classification and the detection track with outside training data.
- Grad-CAM
A method that shows where in an image a convolutional network found evidence for a class, by weighting a layer's feature maps with the average gradient of the class score; it generalizes class activation mapping (CAM), and its variants include Score-CAM and Grad-CAM++.
- Graph Neural Network
A neural network that works on graphs by passing messages between neighbouring nodes with weights shared across the graph; message passing, GCN and SGC, attention on graphs, readout, permutation equivariance, over-smoothing, and uses in vision for superpixels, scene graphs, skeletons, point clouds, meshes and video.
- Hyperparameter Tuning
How the settings that training does not learn, such as the learning rate, regularization strength and augmentation, are chosen by search on validation data, from grid and random search through Bayesian optimization, successive halving, Hyperband, BOHB and population-based training, and what tuning costs and does to reported results.
- ImageNet Classification with Deep Convolutional Neural Networks
Krizhevsky, Sutskever and Hinton's 2012 NIPS paper, often called the AlexNet paper, which trained a large convolutional network on two GPUs, won ILSVRC-2012 with a top-5 test error of 15.3% against 26.2% for the next entry, and turned computer vision toward deep learning.
- Inception
The family of convolutional networks built from Inception modules, which run 1×1, 3×3 and 5×5 convolutions and a pooling path side by side and concatenate their outputs; from GoogLeNet (2014) through BN-Inception, Inception-v3 and Inception-v4 to Inception-ResNet, with 1×1 reductions, auxiliary classifiers and factorized convolutions.
- Inductive Bias
The assumptions a learning system makes about data it has not seen, which in vision networks are built into the architecture as locality, weight sharing and equivariance to translations, permutations or rotations, and which trade sample efficiency against what the model can represent.
- Interpretability of Neural Networks
How to find out what a trained vision network has learned and why it made a prediction, the field also called explainability, explainable AI or XAI, covering saliency maps and gradient attribution, class activation maps such as Grad-CAM, occlusion and other perturbation methods, feature visualization, concept-based explanations, attention maps, models designed to be interpretable, and how explanations are tested.
- Knowledge Distillation
Training a student network on a teacher network's softened predictions, or on its features or the relations between its outputs; temperature and the T² factor, why soft targets help, logit, feature, relational and self-distillation, born-again networks, the teacher–student capacity gap, function matching with consistent views and long schedules, DeiT's distillation token, and a worked example with an exact teacher.
- L1 and L2 Losses
The squared (L2) and absolute (L1) errors used to train networks that output numbers, such as box offsets, depth, flow and pixel values, with the robust losses between them (Huber, smooth L1, Charbonnier and Barron's general loss); what each one predicts when the target is uncertain, and how its gradient treats large errors.
- Label Smoothing
Training a classifier on targets that mix the one-hot label with a uniform distribution, so the cross-entropy loss has a finite minimum with a fixed gap between the true-class logit and the others; its definition in Inception-v3, the equivalent penalty against a uniform prior, its effects on accuracy, calibration, label noise, transfer and knowledge distillation, and its place in ImageNet and transformer training recipes.
- Layer Normalization
A normalization layer that standardizes each example's (in transformers, each token's) features with their own mean and variance and applies a learned per-feature scale and shift; where it sits in a residual block (post-norm or pre-norm) controls the size of gradients at initialization and the need for learning-rate warmup.
- Learning and Transferring Mid-Level Image Representations Using Convolutional Neural Networks
Oquab, Bottou, Laptev and Sivic's CVPR 2014 paper, which reused the layers of an ImageNet-trained convolutional network, added two trained adaptation layers and a sliding-window patch scheme, and reported state-of-the-art Pascal VOC object and action classification.
- Learning Rate Schedule
A rule that changes the learning rate during training, usually a short warmup followed by a decay (step, exponential, polynomial or cosine), so that a network makes fast progress early and settles into a good minimum late.
- LeNet
The family of small convolutional networks for handwritten character recognition that LeCun and colleagues built at Bell Labs from 1989 to 1998, from the zip-code network to LeNet-5, with its trainable subsampling, partial connection table and fixed RBF output codes, and how today's re-implementations differ.
- LoRA
Low-rank adaptation, which fine-tunes a frozen pretrained network by learning a low-rank update BA for chosen weight matrices; its initialization and α/r scaling, which layers receive it, merging for serving without added latency, the evidence on rank, QLoRA and DoRA, and its use in vision transformers, vision–language models and diffusion models.
- Loss Function
The function that scores how far a network's output is from the desired output on one example; training minimizes its average over the training images, and the choice of loss encodes what counts as a good prediction for each vision task.
- Mixed-Precision Training
Training a network with most of its arithmetic and storage in 16-bit (FP16 or BF16) or 8-bit floating point while keeping the numerically sensitive parts in FP32, using an FP32 master copy of the weights, loss scaling, FP32 accumulation and per-tensor scales for FP8.
- Mixup and CutMix
Training an image classifier on pairs of training images combined into one, by blending them (mixup), by pasting a rectangle of one into the other (CutMix), or by mixing hidden features (Manifold Mixup), with the labels mixed in the same proportion; how the mixing and the label weights are computed, how they relate to Cutout and Random Erasing, and what their papers report on accuracy, calibration, localization and transfer.
- MobileNet
Google's family of convolutional networks for phones and other small devices, from MobileNet v1 (2017), built from depthwise separable convolutions with width and resolution multipliers, through MobileNetV2's inverted residuals with linear bottlenecks and MobileNetV3's latency-driven architecture search, to MobileNetV4's universal inverted bottleneck and mobile attention (2024).
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
The 2017 arXiv paper by Howard and colleagues at Google that built small, low-latency image classifiers from depthwise separable convolutions, added width and resolution multipliers to fit them to a budget, and tested them on ImageNet, fine-grained recognition, geolocalization, face attributes, detection and face embeddings.
- Multi-Task Learning
Training one network for several tasks at once, with a shared representation and a head per task; hard and soft parameter sharing, cross-stitch units and multi-head vision networks, loss weighting by uncertainty and GradNorm, gradient conflict, which tasks to train together, and the evidence on when sharing helps and when tasks interfere.
- Multilayer Perceptron
A neural network built from a stack of fully connected layers with nonlinearities between them; a universal approximator in principle, but on raw pixels it needs huge numbers of weights and has no notion of where a pattern sits, which is why vision models use it inside larger designs, as classifier heads, transformer blocks and coordinate networks.
- Neural Architecture Search
Automating the design of network architectures by searching a defined space of candidate networks, with the choice of search space (macro, cell-based, hierarchical), search strategy (reinforcement learning, evolution, random search, gradient-based and one-shot methods) and performance estimation, hardware-aware search for mobile vision, benchmarks such as NAS-Bench-101, and what the evidence says about cost, baselines and reproducibility.
- Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, Cho and Bengio's ICLR 2015 paper, first posted in September 2014, which let a translation decoder soft-search the source sentence for each word it writes instead of reading one fixed-length summary vector; the form of attention it introduced was carried to image captioning in 2015 and later into the transformer, although the paper itself reports English–French translation only.
- Neural Network
A function built from layers of simple units, each computing a weighted sum of its inputs plus a bias followed by a nonlinearity, whose weights are learned from data, for example to map an image's pixels to class scores.
- Normalization Layers
Layers that standardize a network's activations with a mean and variance computed over a chosen set of tensor axes, then apply a learned scale and shift; batch, layer, instance, group and RMS normalization differ only in those axes, and weight normalization rescales the weights instead.
- Parameter-Efficient Fine-Tuning
Adapting a frozen pretrained network by training only a small added or selected set of parameters per task; adapters, prefix and prompt tuning, visual prompt tuning, BitFit and other selective methods, low-rank updates, how the methods relate, what they save in storage, memory and compute, and the evidence on accuracy in vision.
- Patch Embedding
The first layer of a vision transformer, which cuts an image into patches and maps each one to a token vector; how its width bounds what the tokens can keep, why it is a strided convolution that tracks image shifts only in whole-patch steps, and how overlapping and convolutional stems, class, distillation and register tokens, patch merging and patch-size resizing modify it.
- Pooling
A fixed, parameter-free layer that summarizes each small window of a feature map by its maximum or average, shrinking the map, widening what later layers see, and tolerating small shifts at the cost of resolution and aliasing.
- Positional Encoding
How transformers and coordinate networks are told where each token or pixel is, from the fixed sinusoids of the original transformer and learned patch-position embeddings to 2D image encodings, relative position biases, rotary embeddings and random Fourier features.
- Receptive Field
The region of the input image that can affect one unit of a convolutional network, how kernel size, depth, stride, pooling and dilation make it grow, and why the effective, gradient-weighted region is much smaller than the theoretical one.
- Recurrent Neural Network
A neural network that reads a sequence one step at a time while carrying a state vector, trained by backpropagation through time; why its gradients vanish or explode over time, gradient clipping, the LSTM and GRU, bidirectional, stacked and convolutional variants, and their use for captioning, video, text recognition and optical flow.
- Regularization
Any change to how a network is trained that aims to lower its error on new images rather than on the training images, from weight penalties and data augmentation to early stopping and the implicit bias of SGD.
- Residual Connection
A shortcut that adds a block's input to its output, y = x + F(x), so that the layers learn only a correction to the identity; it lets very deep convolutional networks and transformers train, because signals and gradients always have a path that bypasses the weights.
- ResNet
The family of residual convolutional networks introduced by He et al. in 2015, a 7×7 stem and four stages of basic or bottleneck residual blocks at strides 4 to 32, in depths from ResNet-18 to ResNet-152, with its main variants (v1.5, pre-activation, Wide ResNet, ResNeXt, ResNet-D) and the training recipes that changed its accuracy.
- Saliency Maps
Gradient-based attribution maps that show which pixels a network's class score depends on, from plain input gradients, gradient times input, deconvnet and guided backpropagation to integrated gradients and its axioms, SmoothGrad, and attention rollout and flow for transformers, with the evidence on when they can be trusted.
- Scaling Laws
Empirical laws that describe how a network's error falls as training data, model size and compute grow, covering power laws and their saturating forms, compute-optimal allocation, the evidence from vision (JFT-300M, Instagram hashtags, Big Transfer, ViT scaling, ViT-22B, OpenCLIP), downstream and transfer scaling, where the laws break, and how to fit and extrapolate them.
- Self-Attention
An operation in which every token of a set, such as the patches of an image, computes its output as a weighted average of all tokens, with weights given by a softmax over scaled dot products between learned queries and keys.
- Semi-Supervised Learning
Training a classifier on a few labelled images together with many unlabelled ones, through pseudo-labels, consistency between perturbed views, teacher–student training or label propagation on a similarity graph, and the assumptions and failure modes that decide whether the unlabelled images help.
- Softmax
The function that turns a vector of real-valued scores (logits), such as a classifier's class scores for an image, into a probability distribution by exponentiating and normalizing them; a temperature controls how peaked the distribution is.
- Squeeze-and-Excitation
A small block that averages each channel of a feature map over the whole image, passes the resulting vector through a two-layer bottleneck ending in a sigmoid, and multiplies every channel by its gate, so that a convolutional network can reweight its features using global context at a cost of about 10% more parameters and 0.26% more computation in ResNet-50.
- Stochastic Gradient Descent
Gradient descent in which each step uses the average gradient over a small random mini-batch of training images instead of the whole dataset, usually with momentum; it is the basic optimizer for training neural networks.
- Swin Transformer
A hierarchical vision transformer (Liu et al., ICCV 2021) that computes self-attention inside local windows shifted between consecutive blocks and merges patches between stages, giving CNN-like feature maps at linear cost; with Swin Transformer V2's res-post-norm, scaled cosine attention and log-spaced continuous position bias (CVPR 2022).
- Training a Neural Network
The practical workflow for fitting a neural network to labeled images, from splitting the data and normalizing pixels through the training loop, learning curves and early stopping to hyperparameter search, debugging checks, reproducibility and memory.
- Transfer Learning
Reusing what a network learned on a large source task, typically by pretraining on many labelled images and then adapting it to a target task with few labels, through a linear probe on frozen features, partial fine-tuning or full fine-tuning.
- Transformer
A neural network architecture that stacks identical blocks of multi-head self-attention and a position-wise MLP, each wrapped in a residual connection with layer normalization; introduced for machine translation and now used for image classification, detection and segmentation on sequences of image patches.
- Transposed Convolution
The layer that maps a small feature map to a larger one by applying the transpose of a strided convolution's matrix, equivalently the convolution's gradient with respect to its input; the learned upsampling step of segmentation decoders and image generators.
- Uncertainty Estimation
How a vision model can report how far its prediction may be wrong; aleatoric and epistemic uncertainty, heteroscedastic regression, Bayesian neural networks, Monte Carlo dropout, deep ensembles, single-pass methods, calibration and recalibration, out-of-distribution detection, evaluation under shift, and uses in depth, flow, detection, segmentation and medical imaging.
- Vanishing and Exploding Gradients
Why the gradient that backpropagation sends to the early layers of a deep network can shrink or grow geometrically with depth, how saturating activations and weight scale cause it, and the remedies that made deep vision networks trainable.
- Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan and Zisserman's ICLR 2015 paper, first posted in September 2014, which tested plain convolutional networks of 11 to 19 layers built from 3×3 filters, won ILSVRC 2014 localization and placed second in classification, and released the VGG-16 and VGG-19 models.
- VGG
The very deep convolutional networks of Simonyan and Zisserman (2015), plain stacks of 3×3 convolutions with 11 to 19 weight layers; VGG-16 and VGG-19 became standard backbones and feature extractors, at a cost of about 140 million parameters and 15 to 20 billion multiply-adds per image.
- Vision Transformer
A transformer encoder applied to an image cut into fixed-size patches, with a class token and learned position embeddings; it matches or beats convolutional networks at image classification when pre-trained on enough data, and is now a standard backbone for detection, segmentation and vision–language models.
- Visualizing and Understanding Convolutional Networks
Zeiler and Fergus's 2014 ECCV paper, first posted to arXiv in November 2013, which projected activations of an ImageNet-trained convolutional network back to pixels with a deconvnet, used occlusion to test what the classifier relied on, changed AlexNet's first layers in response, and showed that the network's features transferred to Caltech-101 and Caltech-256.
- Weight Decay
Shrinking a network's weights by a small factor at every training step, either through an L2 penalty in the loss or applied directly to the weights (decoupled, as in AdamW); the two coincide for plain SGD but not for Adam, and with normalization layers the decay mainly sets the effective learning rate.
- Weight Initialization
How the starting values of a network's weights are chosen, from breaking symmetry to the variance-preserving rules of Glorot and Bengio and of He et al., orthogonal and data-driven schemes, zero-initialized residual branches, and what PyTorch does by default.
Recognition & Detection
- 3D Human Pose Estimation
Recovering the 3D positions of a person's body joints from one image, a video or several calibrated cameras; lifting 2D joints, volumetric heatmaps, temporal convolutions, learnable triangulation, SMPL body models in outline, the Human3.6M, MPI-INF-3DHP and 3DPW protocols, and MPJPE with its Procrustes-aligned variant.
- A Discriminatively Trained, Multiscale, Deformable Part Model
The CVPR 2008 paper by Felzenszwalb, McAllester and Ramanan that introduced the deformable part model detector, a HOG root filter plus higher-resolution movable part filters trained from bounding boxes as a latent SVM with margin-sensitive hard-negative mining; its method as published, its PASCAL VOC 2006 and 2007 results, and how the 2010 journal version changed it.
- Anchor Boxes
Reference boxes of fixed sizes and aspect ratios, tiled at every feature-map position, that a detector classifies and refines, and that decide which of its predictions are trained on which objects.
- Anchor-Free Object Detection
How object detectors work without anchor boxes, by predicting boxes from keypoints, from points inside or at the centre of an object, or as a set matched one-to-one to the objects, and what changes, and what does not, when the anchors are removed.
- ArcFace
The margin-based softmax losses for learning face embeddings, from normalized softmax and the multiplicative angular margin of L-Softmax and SphereFace to the additive cosine margin of CosFace and the additive angular margin of ArcFace, with their geometry on the hypersphere, the scale parameter, implementation details, failure modes and reported results.
- Biometric Recognition Systems
Recognizing people automatically from their face, iris, fingerprints or other body traits; how systems enrol people and then verify or identify them, how their errors are measured (FMR and FNMR, FAR and FRR, FPIR and FNIR, DET curves and the equal error rate), and how presentation attacks, template protection, demographic differentials and regulation shape their use.
- Bounding Box
The smallest axis-aligned rectangle that encloses an object in an image, the standard way to say where an object is in detection, tracking, and annotation.
- Bounding Box Regression
How object detectors learn to predict box coordinates, from the offsets and log scales of R-CNN to smooth L1, IoU-based losses such as GIoU, DIoU and CIoU, and box distributions.
- CenterNet
The 2019 detector of Zhou, Wang and Krähenbühl, “Objects as Points”, which finds each object as the peak of a per-class heatmap of box centres, regresses its size and a sub-cell offset at that peak, replaces box-based non-maximum suppression with a 3×3 max-pooling, and extends the same heads to 3D boxes and human pose.
- Connectionist Temporal Classification
An alignment-free sequence loss that lets a network trained on whole transcriptions label every frame of a text line, handwriting or speech signal, with its blank label, many-to-one collapsing map, forward–backward algorithm in log space, decoders and assumptions.
- CRNN
The convolutional recurrent neural network of Shi, Bai and Yao, which turns the columns of a CNN feature map into a sequence, reads it with a deep bidirectional LSTM and transcribes it with CTC, so that a word image is read without character segmentation; its official configuration, training, decoding with and without a lexicon, results as reported and its role as the baseline of later scene text recognizers.
- DeepFace
Taigman, Yang, Ranzato and Wolf's 2014 face verification system from Facebook AI Research, which frontalized each face with a 3D model fitted to 67 landmarks, mapped it with a nine-layer network with three locally connected layers, trained to classify 4,030 identities, and compared the resulting features with an inner product, a weighted chi-square SVM or a Siamese network, reaching 97.35% on LFW and 91.4% on YouTube Faces.
- Deformable Part Models
The HOG-based object detector of Felzenszwalb, Girshick, McAllester and Ramanan, which scores a coarse root filter plus higher-resolution part filters that may move at a quadratic cost, matched with generalized distance transforms and trained as a latent SVM; its model, training, releases, results and relation to convolutional networks.
- DETR
A transformer-based object detector that predicts a fixed-size set of boxes in parallel and is trained with a one-to-one Hungarian matching loss, so it needs neither anchors nor non-maximum suppression.
- Document Image Analysis
Computer vision for scanned and photographed documents, from binarization, skew and warp correction and page layout analysis to table and form recognition and document understanding models, with the DIBCO, page segmentation, PubLayNet, DocBank, FUNSD, RVL-CDIP and table benchmarks and the protocols that score them.
- Eigenfaces
The appearance-based face recognition method of Turk and Pentland, which projects aligned face images onto the principal components of a training set of faces, recognizes a face by its nearest neighbour among the projected coefficients and detects faces by their distance from that subspace; with Fisherfaces as its discriminant variant and the effects of lighting, pose and alignment.
- Face Detection
Finding every face in an image and returning a box, a confidence score and often a few facial landmarks for each, from boosted cascades and part models to the single-stage CNN detectors evaluated on FDDB and WIDER FACE.
- Face Recognition
Deciding whether two face images show the same person (verification) or which enrolled person a face belongs to (identification, closed- or open-set), with the detect, align, embed and compare pipeline, its benchmarks and evaluation protocols, and NIST's measurements of demographic differentials.
- FaceNet
The face recognition model from Google (2015) that maps a face crop to a 128-dimensional unit vector compared by squared Euclidean distance, trained with a triplet loss and online semi-hard negative mining; its networks, the readings of semi-hard mining in open reimplementations, its uses for verification, identification and clustering, and why margin-based softmax losses later replaced its training objective.
- FaceNet: A Unified Embedding for Face Recognition and Clustering
Schroff, Kalenichenko and Philbin's CVPR 2015 paper from Google, which trained a convolutional network with a triplet loss and online semi-hard negative mining to map face crops to 128-dimensional unit vectors, reported 99.63% on LFW and 95.12% on YouTube Faces from 100 to 200 million private training images, and, in its later arXiv versions, added harmonic embeddings.
- Facial Expression Recognition
Assigning expression labels, facial action units or valence and arousal to face images and videos; basic-emotion categories and FACS as the datasets define them, posed against spontaneous data, landmark, appearance and deep approaches, the CK+, FER2013, AffectNet, RAF-DB and DISFA benchmarks with their protocols, and the documented limits of inferring emotion from facial movements.
- Facial Landmark Detection
Locating a fixed set of named points on a face, such as the eye corners, nose tip, mouth corners and jaw line; landmark schemes, active shape and appearance models, cascaded shape regression (ESR, SDM, LBF), heatmap and coordinate networks, 3D face alignment, and the 300-W, WFLW, COFW and AFLW benchmarks with their normalized errors.
- Fast R-CNN
The 2015 region-based detector that computes one convolutional feature map per image, pools a fixed-size feature for each proposal with RoI pooling, and trains classification and box regression together in one network with a multi-task loss.
- Faster R-CNN
A two-stage object detector in which a region proposal network, sharing the detector's convolutional features, replaces hand-designed region proposals, making proposal generation nearly free and the whole detector one network.
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
The 2015 NIPS paper by Ren, He, Girshick and Sun that replaced hand-designed region proposals with a region proposal network sharing the detector's convolutional features, making proposals nearly free and the whole detector one network.
- FCOS
A one-stage, anchor-free object detector that predicts, at every feature-map location inside an object, the distances to the four sides of its box, assigns objects to feature-pyramid levels by the size of those distances, and down-weights boxes from off-centre locations with a center-ness score.
- Feature Pyramid Network
A network design that turns a convolutional backbone's feature maps at strides 4 to 32 into a pyramid with strong semantic features at every scale, through a top-down pathway and lateral connections, so a detector can find small and large objects from one image scale.
- Few-Shot Recognition
Recognizing new image classes from one or a few labelled examples each, set up as N-way K-shot episodes with support and query sets; metric-based methods (siamese, matching, prototypical and relation networks), optimization-based meta-learning (MAML) and pretrained-embedding baselines, few-shot detection, and the Omniglot, miniImageNet, tieredImageNet and Meta-Dataset protocols.
- Fine-Grained Image Classification
Telling apart subordinate categories such as bird species, car models or aircraft variants, where classes differ in small local details; from part annotations and part detectors to bilinear pooling and learned attention, with the CUB-200-2011, Stanford Cars, FGVC-Aircraft, Stanford Dogs and iNaturalist benchmarks.
- Handwriting Recognition
Reading handwritten text from pen trajectories (online) or scanned images (offline), from isolated characters and words to whole lines and pages, with HMM, CTC (BLSTM and MDLSTM) and transformer recognizers, the IAM, RIMES and READ benchmarks and their protocols, and the role of lexicons and language models in character and word error rates.
- Hard Negative Mining
Training a detector's classifier on the background examples it gets wrong rather than on a random sample of background, from bootstrapping in sliding-window and part-model training to online hard example mining, SSD's fixed-ratio selection, and the focal loss that reweights instead.
- Heatmap Regression
Locating keypoints by training a network to output one image-like map per keypoint, usually a Gaussian bump at the true position, and reading the coordinate off the map; target design, resolution and quantization error, sub-pixel decoding (quarter shift, DARK, offsets), integral regression, and the coordinate alternatives DeepPose, RLE and SimCC.
- HRNet
The High-Resolution Network, a convolutional backbone that keeps a high-resolution branch through the whole network and repeatedly exchanges information with parallel lower-resolution branches, used for human pose estimation (HRNet), semantic segmentation and detection (HRNetV2, HRNetV2p) and bottom-up pose estimation (HigherHRNet).
- Human Pose Estimation
Locating the body joints of every person in an image, such as shoulders, elbows, knees and ankles, to recover each person's 2D skeleton; keypoint definitions, part models, heatmap networks, top-down and bottom-up multi-person methods, and the PCKh and COCO keypoint metrics.
- Image Classification
Assigning one or more category labels to a whole image, usually by predicting a probability for each of a fixed set of classes.
- Image Retrieval
Finding the images in a collection that show the same object, place or kind of thing as a query image; instance and category retrieval, the pipeline of descriptors, indexing and ranking, spatial verification, query expansion and diffusion, deep global descriptors, and the benchmarks (UKBench, Holidays, Oxford and Paris, their revisited versions, Google Landmarks v2) with their official evaluation code.
- Industrial Visual Inspection
Using cameras and vision algorithms to check manufactured parts on a production line, from lighting and optics to golden-template comparison, supervised defect models and anomaly detection trained on good parts only, judged by how many defects escape and how many good parts are rejected.
- Keypoint Detection
Locating a fixed set of named points, such as an animal's paws, a hand's fingertips or a car's headlights, on every object of a category, or on categories never seen in training; keypoint definitions, transfer from human pose models, multi-view labelling, category-agnostic methods, benchmarks for animals, hands, faces and vehicles, and the PCK and OKS metrics.
- Label Assignment in Object Detection
How a detector decides, during training, which of its predictions are trained as positives for which ground-truth object and which as background, from IoU thresholds to ATSS, optimal transport, task-aligned and one-to-one Hungarian matching.
- Non-Maximum Suppression
The post-processing step that removes duplicate detections by repeatedly keeping the highest-scoring box and discarding the boxes that overlap it beyond an IoU threshold, with its soft, distance-aware and learned variants.
- Object Counting
Estimating how many objects of a kind an image contains, by detecting and counting them, by regressing the count from global features, or by predicting a density map whose integral is the count; with crowd counting networks (MCNN, CSRNet), point-supervised losses (Bayesian loss, DM-Count), class-agnostic counting (FamNet and FSC-147), and the UCSD, ShanghaiTech, UCF-QNRF, JHU-CROWD++ and FSC-147 benchmarks scored by MAE and root mean squared error.
- Object Detection
Finding every object of a set of categories in an image and returning, for each one, a bounding box, a class label and a confidence score.
- One-Stage Object Detectors
Object detectors that predict class scores and box offsets densely at every position of a feature map, or of a feature pyramid, in a single network pass, with no region-proposal stage.
- Open-Vocabulary Object Detection
Detecting objects of categories named in text at test time, including categories that had no box annotations in training, by scoring image regions against text embeddings, from zero-shot detection with word vectors to detectors built on CLIP and on grounding data, with the OV-COCO and OV-LVIS protocols.
- OpenPose
A bottom-up multi-person 2D pose estimator from Carnegie Mellon University that detects all body keypoints in an image as heatmaps and groups them into people with part affinity fields, 2D vector fields that run along each limb; the CVPR 2017 model, the journal version with foot keypoints and PAF-first refinement, and the released OpenPose library.
- Optical Character Recognition
Turning images of text into machine-readable characters, from scanned printed pages and handwriting to text in street scenes, with the classical document pipeline, segmentation-free line recognizers, scene text methods, benchmarks and the edit-distance metrics used to score them.
- Pedestrian Detection
Finding every person on foot in an image, usually from a vehicle or a fixed camera; from HOG with a linear SVM and boosted channel features to adapted convolutional detectors and crowd-aware losses and suppression, with the Caltech, CityPersons and CrowdHuman benchmarks and the log-average miss rate.
- Person Re-Identification
Finding the images of one person across cameras whose views do not overlap; hand-crafted colour and texture features with learned metrics (ELF, LOMO and XQDA, KISSME), deep identity embeddings, part-based and attention models, triplet losses, the strong baseline and k-reciprocal re-ranking, with the Market-1501, DukeMTMC-reID, CUHK03 and MSMT17 protocols, CMC and mAP, and the withdrawal of DukeMTMC.
- Pictorial Structures
The part-based object model in which parts with their own appearance models are joined by spring-like deformation costs; Fischler and Elschlager's energy, Felzenszwalb and Huttenlocher's exact tree inference with generalized distance transforms, constellation models, pictorial structures revisited, flexible mixtures of parts, and deep networks with structured inference.
- R-CNN
The region-based CNN detector of 2014, which classified about 2,000 warped region proposals per image with CNN features, linear SVMs and box regression, and its direct successor SPPnet, which shared the convolutional features across proposals; Fast R-CNN, which trained the detector in one stage, is summarized.
- Rapid Object Detection using a Boosted Cascade of Simple Features
The CVPR 2001 paper by Viola and Jones that combined the integral image, AdaBoost feature selection and an attentional cascade into a frontal face detector running at 15 frames per second, with its 2004 journal version and the comparisons, inconsistencies and influence of both.
- Region Proposals
Class-agnostic candidate boxes that a detector examines instead of every window, how objectness, grouping, window scoring and learned networks produce them, how their recall is measured, and why the region proposal network replaced hand-designed methods.
- RetinaNet
A one-stage object detector from 2017 that predicts class scores and box offsets for dense anchors on a feature pyramid and trains its classifier with the focal loss, which down-weights the many easy background examples so that a single dense network matches two-stage accuracy.
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
The CVPR 2014 paper by Girshick, Donahue, Darrell and Malik that introduced R-CNN, classifying bottom-up region proposals with fine-tuned ImageNet CNN features, and raised PASCAL VOC detection mAP by more than 30% relative to the previous best.
- RoI Pooling
How a region-based detector turns each candidate box on a shared feature map into a fixed-size feature grid, from RoI pooling in SPP-net and Fast R-CNN through RoI warping, RoIAlign and precise RoI pooling to position-sensitive, multi-level and rotated variants.
- Scene Recognition
Naming the kind of place a photograph shows, such as a kitchen, a coast or a street; from holistic GIST descriptors and spatial pyramids of visual words to object-based features and CNNs trained on Places, with scene attributes and the 15-Scene, MIT Indoor-67, SUN397 and Places365 benchmarks and their protocols.
- Scene Text Detection
Finding where words and text lines are in photographs of the world, as rotated boxes, quadrilaterals or polygons; from connected-component methods such as MSER and the stroke width transform to regression detectors such as TextBoxes and EAST and segmentation detectors such as PixelLink, PSENet and DBNet, with the benchmarks and matching rules used to score them.
- Scene Text Recognition
Reading the word in a cropped image of text from a photograph, with CTC and attention decoders, rectification of curved and perspective text by spatial transformers (RARE, ASTER, MORAN), language models (SRN, ABINet, PARSeq), synthetic training data, the cropped-word benchmarks and the evaluation conventions that make their numbers comparable or not.
- Selective Search
A class-agnostic region proposal algorithm that over-segments an image, greedily merges neighbouring regions by colour, texture, size and fill similarity, and returns the bounding box of every region in the hierarchy, diversified over colour spaces, similarity sets and starting segmentations.
- Sliding Window Detection
Detecting objects by running a fixed-size window classifier at every position and scale of an image, how the stride, the pyramid's scale step and the window shape set what is found and what it costs, and how the idea survives inside modern detectors.
- SSD (Single Shot MultiBox Detector)
A one-stage object detector from 2016 that predicts class scores and box offsets for fixed default boxes on several feature maps of decreasing resolution, in a single network pass with no proposal stage.
- SSD: Single Shot MultiBox Detector
The 2016 ECCV paper by Liu, Anguelov, Erhan, Szegedy, Reed, Fu and Berg that introduced SSD, a one-stage detector predicting from default boxes on several feature maps of one network, and whose five arXiv versions changed the network, the results and the labels of its own tables.
- Stacked Hourglass Network
The 2016 pose estimation network of Newell, Yang and Deng, which repeats an hourglass of pooling, upsampling and skip branches several times, predicts joint heatmaps after every hourglass and supervises each prediction; its design, training, results as reported, and its reuse in associative embedding, CornerNet and CenterNet.
- Two-Stage Object Detectors
Object detectors that first propose a few thousand class-agnostic candidate regions and then classify and refine each one with features pooled from a shared CNN feature map, the design line of R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN.
- Viola-Jones Object Detection
The boosted cascade detector of Viola and Jones, which selects a few rectangle features with AdaBoost and chains the resulting classifiers so that most background windows are rejected after a handful of feature evaluations.
- Visual Anomaly Detection
Detecting and localizing images, or image regions, that depart from a set of normal training images, with no examples of the anomalies; reconstruction, nearest-neighbour and Gaussian models on pretrained features, normalizing flows and student–teacher networks, with the MVTec AD, VisA and MVTec LOCO AD benchmarks and the image AUROC, pixel AUROC and PRO metrics.
- Visual Place Recognition
Recognizing where a photograph was taken by retrieving geotagged images of the same place; the retrieval formulation, FAB-MAP, aggregated descriptors (VLAD, NetVLAD, GeM), geometric re-ranking, appearance and viewpoint change, its role in SLAM loop closure, foundation-model features, and the Pittsburgh, Tokyo 24/7, Nordland and Mapillary SLS benchmarks with Recall@N and the localization radius.
- YOLO
A family of real-time one-stage object detectors, begun by Redmon et al. in 2016, that predict every box and class probability of an image in a single network pass; later versions from several groups added anchors, multi-scale heads, anchor-free heads and NMS-free training.
- You Only Look Once: Unified, Real-Time Object Detection
The 2016 CVPR paper by Redmon, Divvala, Girshick and Farhadi that introduced YOLO, recasting object detection as one regression from image pixels to boxes and class probabilities, solved by a single network fast enough for live video.
- Zero-Shot Recognition
Recognizing image classes that had no training images, by relating images to descriptions of the classes such as attributes, word vectors or text; from attribute classifiers and compatibility functions to generated features and vision–language models, with the standard and generalized settings, the proposed splits of AwA2, CUB, SUN and aPY, and the harmonic mean.
Segmentation
- Active Contours
Curves that move through an image to minimize an energy made of internal smoothness terms and image terms, from Kass, Witkin and Terzopoulos's snakes to balloon forces, gradient vector flow and geometric contours.
- DeepLab
A family of semantic segmentation networks from Google and UCLA (2015–2018) built on atrous (dilated) convolution, which keeps feature maps dense without extra parameters, atrous spatial pyramid pooling for multi-scale context, a fully connected CRF for boundaries in its first two versions, and a light decoder in DeepLabv3+.
- FCN
The fully convolutional network of Long, Shelhamer and Darrell (2015), a semantic segmentation network trained end to end from pixels to pixels; it recasts a classification CNN's fully connected layers as convolutions, upsamples coarse class scores in-network, and fuses scores from finer layers (FCN-32s, FCN-16s, FCN-8s).
- Fully Convolutional Networks for Semantic Segmentation
The CVPR 2015 paper by Long, Shelhamer and Darrell that recast ImageNet classification networks as fully convolutional networks and trained them end to end, pixels to pixels, for semantic segmentation, replacing patchwise and proposal-based pipelines.
- GrabCut
Interactive foreground extraction from a rectangle drawn around an object, which alternates between fitting Gaussian mixture colour models to the current foreground and background and recomputing the segmentation with a minimum graph cut, then lets the user correct it with brush strokes.
- Graph Cuts
Energy minimization by minimum s–t cut, which labels the pixels of an image optimally for binary energies with submodular pairwise terms and approximately, through expansion and swap moves, for many labels.
- Graph-Based Image Segmentation
The Felzenszwalb–Huttenlocher algorithm, which scans the edges of a pixel graph in order of increasing weight and merges two regions unless the edge between them exceeds one region's internal variation plus a size-dependent tolerance set by a scale parameter k.
- Image Matting
Estimating a soft opacity (alpha) value and the foreground colour at every pixel so that an object can be cut out and composited onto a new background; covers the compositing equation and why it is under-constrained, trimaps and scribbles, blue screen, sampling-based, closed-form and deep matting, trimap-free and portrait matting, and the alphamatting.com and Composition-1k benchmarks with SAD, MSE, gradient and connectivity errors.
- Image Segmentation
Dividing an image into regions by assigning every pixel to a segment, a class, or an object instance, the task behind semantic, instance, panoptic and interactive segmentation.
- Instance Segmentation
Detecting every object of a fixed set of "thing" classes and giving each one its own pixel mask, class label and confidence score, so that two touching cars come out as two cars rather than one car-colored region.
- Interactive Segmentation
Extracting the mask of an object that a user indicates with clicks, scribbles or a box and corrects in further rounds, from seeded graph-based methods to click-trained networks and promptable models, and how such systems are compared by simulating clicks and counting how many are needed to reach a target IoU.
- Level Set Methods
Segmentation by moving a contour that is stored implicitly as the zero level set of a function on the pixel grid, so that it can split and merge, covering the level set equation and its numerics, geodesic active contours, Chan–Vese and morphological approximations.
- Mask R-CNN
An instance segmentation model that adds a small fully convolutional mask branch and RoIAlign to the Faster R-CNN detector, predicting a box, a class and a binary mask for every detected object.
- Mask R-CNN (2017 paper)
The 2017 ICCV paper by He, Gkioxari, Dollár and Girshick that turned Faster R-CNN into an instance segmentation system by adding a parallel mask branch and RoIAlign, won the ICCV 2017 Best Paper award (Marr Prize), and became a widely used baseline for instance-level recognition.
- Mask2Former
The Masked-attention Mask Transformer (Cheng et al., CVPR 2022), a mask classification model whose Transformer decoder restricts cross-attention to each query's predicted mask, so that one architecture reaches strong results on panoptic, instance and semantic segmentation.
- Normalized Cuts
A graph-partitioning method for image segmentation that measures a cut relative to the total connection of each side, and approximates the best partition with the second eigenvector of a generalized eigenvalue problem; the basis of spectral clustering in vision.
- Normalized Cuts and Image Segmentation
The 2000 IEEE TPAMI paper by Shi and Malik, first presented at CVPR 1997, that cast image segmentation as partitioning a weighted pixel graph, defined the normalized cut criterion, and approximated it with a generalized eigenvector, bringing spectral graph partitioning into computer vision.
- Open-Vocabulary Segmentation
Labelling every pixel of an image with classes named in text at test time, including classes with no training masks, by matching pixels or mask proposals against text embeddings; from zero-shot segmentation with word vectors to CLIP-based two-stage and single-stage models, with the zero-shot splits and the ADE20K-847, PASCAL Context-459, ADE20K-150, PASCAL Context-59 and PASCAL VOC protocols.
- Panoptic Segmentation
Giving every pixel of an image a class label and, for countable objects, an instance id, so that stuff such as sky and road and things such as cars and people are segmented together without overlaps, and scoring the result with panoptic quality (PQ).
- PSPNet
The Pyramid Scene Parsing Network (Zhao et al., CVPR 2017), a semantic segmentation model that appends a pyramid pooling module to a dilated ResNet so that every pixel's prediction sees global and sub-region context, trained with an auxiliary loss; it won the ImageNet Scene Parsing Challenge 2016.
- Region Growing
Segmenting an image by growing regions from seed pixels for as long as a homogeneity criterion holds, as in seeded region growing, or by splitting non-uniform quadtree blocks and merging similar neighbours.
- Salient Object Detection
Finding and segmenting the object or objects in an image that stand out most to a human observer, producing a per-pixel saliency map; also called salient object segmentation or salient region detection, distinct from fixation prediction, with contrast-based and deep encoder–decoder methods, co-saliency, and the MSRA, ECSSD, PASCAL-S and DUTS benchmarks scored by MAE, F-measure, S-measure and E-measure.
- SAM
The Segment Anything Model, a promptable segmentation model that returns object masks for points, boxes or rough masks, trained with a model-in-the-loop data engine on the SA-1B dataset of 1.1 billion masks, and the starting point of a family of faster, sharper, video and medical variants.
- Semantic Segmentation
Labeling every pixel of an image with one of a fixed set of classes, such as road, sky, car or person, without separating individual objects; the dense-prediction counterpart of image classification, also called scene parsing.
- Superpixels
An oversegmentation of an image into a few hundred compact, roughly uniform regions that follow image boundaries, used in place of pixels to make later processing cheaper.
- Thresholding
Segmenting an image by comparing each pixel's intensity with a threshold, chosen by hand, from the histogram as in Otsu's method, or from each pixel's neighborhood as in Sauvola's method.
- U-Net
A convolutional encoder–decoder for segmentation whose contracting path gathers context and whose symmetric expanding path, fed by concatenated skip connections, restores pixel-accurate boundaries; designed in 2015 for biomedical images with few labels, and since the base of 3D U-Net, Attention U-Net, UNet++, nnU-Net and the denoisers of diffusion models.
- U-Net: Convolutional Networks for Biomedical Image Segmentation
The 2015 MICCAI paper by Ronneberger, Fischer and Brox that introduced U-Net, a fully convolutional network with a symmetric expanding path, and showed that with elastic-deformation augmentation it could learn microscopy segmentation from 20 to 35 annotated images.
- Video Object Segmentation
Segmenting chosen objects in every frame of a video, given their masks in the first frame, from user clicks or scribbles, or with no input at all, and measuring the result with the DAVIS region and boundary scores J and F.
- Watershed
A morphological segmentation algorithm that treats an image, usually its gradient magnitude, as a relief and floods it from its minima or from markers, so that regions are catchment basins and boundaries are the lines where floods meet.
Motion & Tracking
- A New Approach to Linear Filtering and Prediction Problems
Rudolf Kalman's 1960 paper that recast Wiener's filtering problem in state-space form and solved it with a recursive estimator, the origin of the Kalman filter.
- An Iterative Image Registration Technique with an Application to Stereo Vision
The 1981 IJCAI paper by Lucas and Kanade that replaced exhaustive search in image registration with a gradient-guided Newton–Raphson-type iteration, the origin of the Lucas–Kanade method.
- Aperture Problem
Why motion seen through a small window is ambiguous along edges, so that only the component of motion across an edge can be measured locally.
- Background Subtraction
Separating moving objects from a static camera's scene by comparing each frame with a model of the background, from frame differencing and running averages to per-pixel Gaussian mixtures, kernel density estimates, ViBe and trained networks, with the problems that break them and the CDnet benchmark that measures them.
- Bayesian Filtering
Recursive estimation of the probability distribution of a hidden, changing state from a sequence of noisy measurements, by alternating a motion-model prediction with a Bayes' rule update.
- Block Matching
Estimating motion by dividing a frame into blocks and finding, for each block, the displacement whose block in a reference frame matches it best, by exhaustive search or by fast search patterns, as video encoders do.
- Brightness Constancy Assumption
The assumption that a scene point keeps the same image intensity as it moves between frames, which turns motion estimation into an intensity-matching problem.
- ByteTrack
A multi-object tracker that associates high-score detections first and then matches the remaining tracks to low-score detections, recovering occluded objects that score thresholds would discard.
- ByteTrack: Multi-Object Tracking by Associating Every Detection Box
The ECCV 2022 paper by Zhang et al. that introduced BYTE, a second association round for low-confidence detections, and the ByteTrack tracker built on it with a YOLOX detector.
- Coarse-to-Fine Estimation
Estimating large motions with small-motion methods by solving on an image pyramid, from the coarsest level to full resolution, and refining the estimate at each level.
- Computer Vision in Sports
Computer vision applied to sports video, from player and ball tracking, pitch registration and action spotting to officiating technology; the broadcast and fixed-camera settings, the tracking and event data they produce, the SoccerNet and SportsMOT benchmarks with their protocols, and the documented limits.
- CONDENSATION—Conditional Density Propagation for Visual Tracking
Michael Isard and Andrew Blake's 1996 conference paper and 1998 journal paper that tracked object outlines through dense clutter by propagating a weighted random sample set over time, bringing particle filtering to computer vision.
- Correlation Filter Tracking
Tracking a target by learning, online and in the Fourier domain, a filter whose response peaks on the target, from MOSSE and kernelized correlation filters to scale estimation, spatial regularization against the boundary effect, ECO, and failure detection with the peak-to-sidelobe ratio.
- Data Association
Deciding which measurements or detections belong to which tracked targets, and which are false alarms, missed detections, new targets, or targets that have disappeared.
- Determining Optical Flow
The 1981 paper by Horn and Schunck that computed dense optical flow by combining the brightness change constraint with a global smoothness assumption, founding the variational approach to motion estimation.
- Extended Kalman Filter
A nonlinear extension of the Kalman filter that linearizes the motion and measurement models around the current estimate, widely used for camera pose estimation, visual-inertial odometry, and tracking with range or bearing sensors.
- Feature Tracking
How distinctive image points are selected and followed across video frames, using the classic KLT tracker as the main example.
- FlowNet
The first convolutional networks trained end to end to predict optical flow, FlowNetSimple and FlowNetCorr, and their stacked successor FlowNet 2.0.
- Horn–Schunck Method
A global variational method that computes dense optical flow by minimizing brightness constancy errors together with a penalty on spatial variation of the flow, solved by a simple iterative averaging scheme.
- Hungarian Algorithm
An algorithm that finds the minimum-cost one-to-one matching between two sets, used in computer vision to match detections to tracks and predictions to ground truth.
- Kalman Filter
A recursive algorithm that estimates the hidden state of a linear dynamic system from a sequence of noisy measurements, widely used to smooth and predict object positions in tracking.
- Lucas–Kanade Method
A local, gradient-based method that estimates the displacement of an image window by assuming constant motion within it and solving a small least-squares problem, iterated with warping.
- Mean Shift Tracking
Kernel-based tracking of a target by its colour histogram, in which the Bhattacharyya coefficient between kernel-weighted histograms is maximized by mean shift steps; with CAMShift's back-projection and moment-based window, scale adaptation, background-weighted histograms, and where these trackers fail.
- Motion Estimation
Recovering how image content, objects, or the camera moved from a sequence of images, from per-pixel flow to global and 3D motion.
- Motion Models for Tracking
The dynamic models a tracker uses to predict where a target will be next, from constant-position, constant-velocity, constant-acceleration and turn models with their process noise to parametric image motion, multiple-model filters, and the consistency checks that tell whether a model fits.
- Motion Segmentation
Dividing a video into regions or point trajectories that move differently, from layered motion and robust parametric fitting through subspace clustering of feature trajectories and long-term trajectory grouping to networks that segment moving objects from optical flow, with the Hopkins 155, FBMS-59 and DAVIS-Moving benchmarks and their measures.
- Multi-Object Tracking
Estimating the trajectories of a varying, unknown number of objects in a video while keeping each object's identity consistent over time.
- Object Tracking
Estimating the position, extent, or state of one or more objects in every frame of a video, keeping each object's identity over time.
- Optical Flow
The apparent motion of image content between two frames, represented as a two-dimensional displacement at every pixel.
- Optical Flow Estimation
Computing a dense field of pixel displacements between two video frames, from classical variational methods to learned networks such as RAFT.
- Particle Filter
A sequential Monte Carlo method that represents the probability distribution of a hidden state with weighted random samples, so it can track through nonlinear models and ambiguous, multimodal beliefs.
- Point Tracking (Tracking Any Point)
Following arbitrary query points through a whole video, including through occlusion, with a visibility flag in every frame; the TAP-Vid benchmark and its measures, and trackers from Particle Video and PIPs to TAP-Net, TAPIR and CoTracker.
- PWC-Net
A compact optical flow network that estimates flow coarse to fine on a learned feature pyramid, warping the second image's features and building a small cost volume at every level.
- RAFT
A deep network for optical flow that matches all pairs of pixels once and refines a single flow field with a recurrent update operator.
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
The ECCV 2020 best paper by Teed and Deng that replaced coarse-to-fine flow networks with an all-pairs correlation volume and a weight-tied recurrent update.
- Scene Flow
Estimating the 3D motion of every visible scene point between two time steps, from stereo video, RGB-D frames or lidar point clouds, and how it relates to optical flow and disparity.
- Siamese Trackers
Single-object trackers that compare a target template with a search region through one shared, offline-trained network; SiamFC's fully convolutional cross-correlation and logistic loss, GOTURN's regression, SiamRPN's proposal head, DaSiamRPN's distractor-aware training, SiamRPN++'s padding argument and depthwise correlation, SiamMask's masks, and the centre bias that padding causes.
- Simple Online and Realtime Tracking
The 2016 paper by Bewley et al. that introduced SORT, showing that a Kalman filter and Hungarian matching on strong CNN detections rival far more complex online multi-object trackers.
- Simple Online and Realtime Tracking with a Deep Association Metric
The 2017 paper by Wojke, Bewley, and Paulus that introduced DeepSORT, extending SORT with a CNN appearance descriptor trained for person re-identification and a matching cascade, and reporting about 45% fewer identity switches on MOT16.
- Single Object Tracking
Following one arbitrary target through a video from a single box in the first frame, and how trackers are compared, from OTB's one-pass evaluation and VOT's reset-based protocol to LaSOT, GOT-10k, TrackingNet and transformer trackers.
- SORT
Simple Online and Realtime Tracking, a multi-object tracker that links per-frame detections into tracks using a constant-velocity Kalman filter on each box and Hungarian matching on box overlap.
- Tracking by Detection
Building object tracks by running a detector on every frame and linking its detections over time with a motion model and data association.
- Video Surveillance and Security
Computer vision applied to video from fixed security and traffic cameras as its research literature describes it; the processing chain from detection to multi-camera tracking, re-identification, attributes, events and search, the PETS, DukeMTMC, CityFlow, AI City Challenge, VIRAT and MEVA benchmarks with their protocols and metrics, documented failure modes, NIST's accuracy and demographic findings, and the privacy concerns stated in official documents.
Video Understanding
- 3D Convolutional Networks
Video networks whose convolutions span time as well as space, from C3D's homogeneous 3×3×3 kernels to mixed and factorized (2+1)D, pseudo-3D and separable designs, inflation of image networks, and efficient channel-separated and expanded architectures such as CSN and X3D.
- Action Recognition
Naming the action that a video clip shows, also called video classification; the task's definition and variants, the model families, how models are trained and evaluated on Kinetics, UCF101 and HMDB51, representative results, and the pitfalls in comparing them.
- Action Recognition with Improved Trajectories
Wang and Schmid's ICCV 2013 paper that estimated camera motion as a homography from SURF and optical-flow matches, masked out detected humans, and used the estimate to warp the flow and prune background trajectories; winner of the 2023 Helmholtz Prize.
- Dense Trajectories
A hand-crafted video representation that samples points densely, tracks them with median-filtered optical flow for 15 frames, and describes the space-time tube around each track; the improved version cancels camera motion with a homography.
- I3D
The Two-Stream Inflated 3D ConvNet, which turns an ImageNet-trained Inception-v1 into a 3D video network by inflating its filters and pooling kernels in time, initializes it so that a static video reproduces the image network, and averages an RGB and an optical-flow stream.
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira and Zisserman's CVPR 2017 paper, which compared five video architectures on the new Kinetics dataset, measured how much Kinetics pretraining helped on UCF101 and HMDB51, and introduced the Two-Stream Inflated 3D ConvNet (I3D).
- Skeleton-Based Action Recognition
Recognizing human actions from sequences of body-joint coordinates instead of pixels; where skeletons come from, the NTU RGB+D and Kinetics-Skeleton benchmarks, recurrent and CNN models, spatial-temporal graph convolution from ST-GCN to adaptive graphs, heatmap-based 3D CNNs, and what skeletons cannot see.
- SlowFast
A two-pathway 3D convolutional video network in which a Slow pathway reads few frames with many channels for appearance, a lightweight Fast pathway reads α times more frames with a fraction β of the channels for motion, and lateral connections fuse Fast into Slow; used for action classification and, with a Faster R-CNN-style head, for AVA action detection.
- SlowFast Networks for Video Recognition
The ICCV 2019 paper by Feichtenhofer, Fan, Malik and He that split a video network into a slow, wide pathway for appearance and a fast, thin pathway for motion, trained it on Kinetics without ImageNet pretraining, and reported state-of-the-art results on Kinetics, Charades and AVA.
- Spatio-Temporal Action Detection
Finding every person in a video, boxing them over time and labelling what each one does, either on sparse keyframes with several labels per person (AVA) or as linked action tubes (UCF101-24, J-HMDB); the datasets, the frame-mAP and video-mAP protocols, and the main method families.
- Temporal Action Localization
Finding the start, end and class of every action in an untrimmed video, also called temporal action detection; how the task relates to object detection, the families of methods from sliding windows and proposals to anchor-free transformers, post-processing, the THUMOS14 and ActivityNet protocols, and how results are compared.
- Temporal Action Segmentation
Labelling every frame of a long, untrimmed video, typically a procedure such as cooking, with the action being performed; the problem and its relation to temporal action localization, over-segmentation, temporal convolutional models from ED-TCN to MS-TCN and their successors, weakly supervised settings, the Breakfast, 50Salads and GTEA benchmarks, and frame accuracy, edit score and segmental F1 as the official evaluation code computes them.
- Temporal Modeling
How video models combine information across frames, from order-blind pooling of per-frame features to recurrence, temporal and 3D convolution, optical flow, space-time attention and temporal shifts, and how frames are sampled from a video for training and testing.
- Temporal Segment Networks
A 2016 video-level training framework for two-stream action recognition networks that splits each video into K equal segments, takes one short snippet from each, scores every snippet with a shared ConvNet and trains on the combined (consensus) score, together with a set of practices for training deep networks on small video datasets.
- TimeSformer
A convolution-free video classifier that applies ViT-Base to the 16×16 patches of every frame and splits each block's attention into a temporal step and a spatial step; the model, the five attention schemes its paper compared, its parameter and FLOP costs, its training recipe, its results on Kinetics, Something-Something V2, Diving-48 and HowTo100M, and its long-clip variants.
- Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan and Zisserman's NIPS 2014 paper that paired an ImageNet-pretrained frame network with a network trained on stacked optical flow, and matched improved dense trajectories on UCF101, which earlier deep video networks had trailed.
- Two-Stream Networks
Video recognition networks that run one CNN on single RGB frames for appearance and a second on stacked optical flow for motion, then fuse the two, from Simonyan and Zisserman's 2014 design to convolutional fusion, temporal segment networks and two-stream I3D.
- Video Anomaly Detection
Scoring each frame of a video for events that depart from what normal training video shows; hand-crafted normality models, reconstruction and future-frame prediction, object-centric self-supervised methods and weakly supervised multiple-instance learning, with the UCSD Ped1 and Ped2, CUHK Avenue, ShanghaiTech Campus, UCF-Crime and XD-Violence benchmarks, micro and macro frame-level AUC, and the limits of what the benchmarks call abnormal.
- Video Foundation Models
Large video encoders pretrained once on broad unlabelled or weakly captioned video and reused for many tasks; their training objectives (masked pixel modelling, masked feature distillation, contrastive video–text learning, prediction in latent space), the main model families as their papers describe them, evaluation by frozen probes and fine-tuning, and the data and compute they report.
- Video Summarization
Shortening a video to its important parts, as a set of keyframes or a skim of short clips; how segments are formed, scored and selected under a length budget, the clustering, diversity and supervised methods, highlight detection and query-focused summaries, the SumMe and TVSum benchmarks with their F-score protocol, and why random summaries score close to published methods under it.
- Video Transformers
Transformer networks that classify video from space-time patch tokens; how frames are tokenized, why attention over all patches of all frames is costly, and how TimeSformer, ViViT, MViT and Video Swin Transformer factorize, pool or localize that attention.
- Video Understanding
Recognizing, locating and describing what happens in video, and how it differs from understanding single images; the overview of the tasks, the model eras from space-time features to video transformers, the datasets and how results are measured.
- VideoMAE
A self-supervised pretraining method that masks 90–95% of a video's space-time tubelets with the same spatial mask in every frame, encodes only the visible ones with a plain ViT and reconstructs the missing pixels; it trains video transformers from the target dataset alone, and VideoMAE V2 scales it to a billion parameters with decoder masking.
Generative Vision
- 3D Generation
Producing new 3D objects or scenes, from noise, a text prompt or a single image, as representations that can be rendered from any viewpoint; 3D-aware GANs, score distillation from 2D diffusion models, view-conditioned and multi-view diffusion, feed-forward reconstruction, output representations, the Janus problem and other failures, and evaluation.
- Auto-Encoding Variational Bayes
Diederik Kingma and Max Welling's ICLR 2014 paper that introduced the SGVB estimator and the AEVB algorithm, built on the reparameterization trick, for continuous latent-variable models, with the variational autoencoder as its worked example, trained on MNIST and Frey Face images.
- Autoregressive Image Model
A generative model that writes the probability of an image as a product of conditional distributions, one pixel or token at a time, and samples by generating them in order; PixelRNN, PixelCNN and its blind spot, the gated PixelCNN, Image GPT, two-stage token models such as DALL·E, masked-token decoding (MaskGIT), next-scale prediction (VAR), and the cost of sampling.
- BigGAN
A large class-conditional GAN for ImageNet (Brock, Donahue and Simonyan, 2019) that combined a residual generator with shared class embeddings and hierarchical latents, a projection discriminator, self-attention, spectral normalization and very large batches; its truncation trick and orthogonal regularization, its late training collapse, BigGAN-deep, published results, and later use as a baseline and a source of features.
- Classifier-Free Guidance
A sampling technique for conditional diffusion models that trains one network with and without the condition and extrapolates from the unconditional prediction toward the conditional one, trading sample diversity for fidelity; classifier guidance, the implicit-classifier view, the guidance-scale conventions, negative prompts, dynamic thresholding and the cost of two passes.
- Conditional GAN
A generative adversarial network whose generator and discriminator both receive a condition, such as a class label or an image; conditioning by concatenation, auxiliary classifiers (AC-GAN) and their pull away from class boundaries, projection discriminators, conditional batch normalization, image conditions, and class-conditional ImageNet results.
- Consistency Model
A generative model that maps any point on a diffusion model's probability-flow ODE trajectory straight to the trajectory's origin, so it can sample in one network evaluation; the consistency function and its boundary condition, consistency distillation and consistency training, multistep sampling, zero-shot editing, the published results, and later variants such as improved consistency training and latent consistency models.
- Controllable Image Generation
Steering a text-to-image diffusion model with conditions beyond the prompt, such as boxes, sketches, edges, depth, pose, segmentation maps, reference images and per-region prompts; the kinds of control, conditioned training, adapters on frozen models, guidance on conditions, training-free attention and energy methods, regional composition with MultiDiffusion, and how control is evaluated.
- ControlNet
A network that adds spatial conditions such as edges, depth, segmentation maps or human pose to a frozen text-to-image diffusion model through a trainable copy of its encoder joined by zero-initialized convolutions; the locked and trainable copies, zero convolutions and their gradients, the conditions and training data, sudden convergence, guess mode and multiple ControlNets, ControlNet 1.1, adapters in outline, and reported results.
- CycleGAN
The unpaired image-to-image translation model of Zhu, Park, Isola and Efros, which trains two generators and two discriminators with adversarial losses in both domains and a cycle-consistency loss; its identity loss, ResNet generators, PatchGAN discriminators, least-squares loss and image buffer, its results and failure cases, and the related unpaired methods DiscoGAN, DualGAN, UNIT, MUNIT and CUT.
- DALL·E
OpenAI's family of text-to-image models. The 2021 DALL·E compresses images with a discrete VAE trained through the Gumbel-softmax relaxation, models caption and image tokens with a 12-billion-parameter transformer, and reranks samples with CLIP. GLIDE, DALL·E 2 (unCLIP) and DALL·E 3 followed; this article describes the last two only through later papers that report them.
- DCGAN
The deep convolutional GAN of Radford, Metz and Chintala, an all-convolutional generator and discriminator with batch normalization, ReLU and LeakyReLU that made GAN training on images reliable enough to become the standard baseline; its architecture and training recipe as the official code shows them, its results as later papers report them, and its discriminator as a feature extractor.
- Deepfake Detection
The task of deciding whether an image or video of a face, or any image, was generated or manipulated, from its pixels; the face-forgery benchmarks, the cues detectors use (blending boundaries, frequency spectra, mouth motion), detection of generated images in general, generalization to unseen generators and robustness to compression, how detectors are scored, and the watermarking and provenance that complement them.
- Deepfakes
Realistic synthetic or manipulated images, video and audio of real people; what counts as a deepfake, the method families (face swapping, reenactment, lip sync, attribute editing, full-face synthesis) at survey level, their history, documented uses and harms, consent, legal responses in the EU, the US and the UK as of November 2026, and detection and provenance in outline.
- Denoising Diffusion Probabilistic Models
The NeurIPS 2020 paper by Ho, Jain and Abbeel that showed diffusion models can generate high-quality images, connected them to denoising score matching and Langevin dynamics, reported a CIFAR-10 FID of 3.17 and LSUN samples it likened to ProgressiveGAN's, and read the variational bound as a progressive lossy code.
- Denoising Diffusion Probabilistic Models (DDPM)
The training and sampling algorithm of Ho, Jain and Abbeel for diffusion models; the variational bound split into per-step KL terms, the forward-process posterior, the noise-prediction parameterization and simple loss, ancestral sampling with fixed or learned variances, and the improvements of Nichol and Dhariwal.
- Diffusion Distillation
Training a fast student to replace the many network evaluations of a diffusion sampler; why sampling is slow, progressive and guided distillation, consistency distillation in outline, adversarial diffusion distillation and the Turbo models, distribution matching distillation, latent consistency models in outline, and how distilled models are evaluated and what they lose in diversity.
- Diffusion Model
A generative model that learns to reverse a gradual Gaussian noising of images, generating by starting from pure noise and denoising step by step; the forward and reverse processes, the noise, data and score parameterizations, the U-Net and transformer denoisers, noise schedules, conditioning and the unifying score-SDE view.
- Diffusion Sampling Methods
How a trained diffusion model is turned into images; ancestral sampling, DDIM with its deterministic and invertible update, Euler and Heun steps on the probability-flow ODE in the EDM parameterization, DPM-Solver's exponential integrators, predictor–corrector samplers, and the trade-off between network evaluations and sample quality.
- Diffusion Transformer
A diffusion model whose denoiser is a plain transformer over patches of a latent image instead of a convolutional U-Net; patchify and patch size, the in-context, cross-attention, adaLN and adaLN-Zero blocks, scaling by forward-pass Gflops, U-ViT's long skips, Stable Diffusion 3's joint-attention MM-DiT, video transformers in outline, and published results.
- Evidence Lower Bound (ELBO)
The tractable lower bound on the log-likelihood of an image that latent-variable generative models maximize instead of the likelihood itself; its two derivations, why the gap equals the KL divergence to the true posterior, the reconstruction and KL terms, the reparameterization trick, importance-weighted bounds and its role in VAEs and diffusion models.
- Flow Matching
A simulation-free way to train continuous normalizing flows by regressing a network onto the velocity of a chosen path between noise and data; conditional flow matching and why it equals the marginal objective, straight-line (rectified flow) paths, minibatch optimal-transport couplings, reflow, the link to diffusion models, and its use in Stable Diffusion 3.
- GAN Inversion
Finding the latent code from which a pretrained GAN generator reproduces a given image, so that the image can be edited with the generator's latent directions; optimization-based, encoder-based and hybrid inversion, the inversion spaces Z, W, W+ and StyleGAN's noise maps, the distortion–editability trade-off, generator tuning, and editing after inversion.
- Generative Adversarial Nets
The 2014 NIPS paper by Goodfellow and colleagues at the Université de Montréal that trained a generator against a discriminator in a minimax game, proved that the game's global optimum recovers the data distribution, and evaluated the samples on MNIST, TFD and CIFAR-10 with Parzen-window likelihood estimates.
- Generative Adversarial Network
A generative model trained as a game between a generator that turns random noise into images and a discriminator that tries to tell them from real images; the minimax objective and its Jensen–Shannon interpretation, the non-saturating loss, DCGAN, mode collapse and training instability, Wasserstein and other remedies, conditional and large-scale GANs, and why diffusion models overtook them.
- Generative Model
A model of the probability distribution of images that can draw new samples from it; the overview of the main families (autoregressive models, variational autoencoders, normalizing flows, GANs and diffusion models), how they differ, and how they are compared.
- High-Resolution Image Synthesis with Latent Diffusion Models
The CVPR 2022 paper by Rombach, Blattmann, Lorenz, Esser and Ommer that trained diffusion models in the latent space of a pretrained autoencoder, added cross-attention conditioning, and reported results on unconditional, class-, text- and layout-conditioned generation, super-resolution and inpainting at a fraction of pixel-space compute.
- Image Editing
The task of changing a given image according to an instruction, a mask, a text prompt, strokes or dragged points while keeping everything else intact; the tension between fidelity and editability, GAN inversion, SDEdit, DDIM and null-text inversion, attention and feature control, blended and mask-based editing, instruction-following models, personalization, drag-based editing, how edits are evaluated, and the risks of manipulated images.
- Image Generation
The task of producing new images from random noise, or from noise and a class label, that look as if they came from the training distribution; how it is defined, the unconditional and class-conditional variants, the benchmarks it is measured on, and its failure modes, from mode dropping to memorized training images.
- Image Inpainting
The task of filling missing or unwanted regions of an image so that the result looks natural, and of extending an image beyond its borders (outpainting); the mask types, the move from diffusion PDEs and patch copying to convolutional networks, GANs and diffusion models, the datasets used, and why pixel metrics mislead for large holes.
- Image-to-Image Translation
The task of mapping an image from one domain to a corresponding image in another, such as label maps to photos, maps to aerial views or day to night; paired and unpaired training, pix2pix, CycleGAN and their successors, semantic image synthesis, diffusion-based translation, and why per-pixel losses and metrics favour blurry averages.
- Imagen
Google Research's 2022 text-to-image diffusion system, a frozen T5-XXL text encoder feeding a 64×64 base diffusion model and two text-conditioned super-resolution diffusion models; its Efficient U-Net, noise conditioning augmentation, guidance and dynamic thresholding, training, results and ablations, and the models built on it.
- Latent Diffusion Model
A diffusion model that runs in the compressed latent space of a pretrained autoencoder instead of in pixels; the two-stage design, the downsampling factor and its trade-off, the latent scaling factor, cross-attention and concatenation conditioning, the compute savings, and successors from SDXL to latent transformers and rectified flow.
- Latent Space
The space of codes from which a generative model produces images; its geometry for Gaussian latents, linear and spherical interpolation, latent arithmetic, semantic directions found with labels (InterFaceGAN) or by PCA (GANSpace), StyleGAN's W and W+ spaces, disentanglement and how it is measured, and the spatial latents of autoencoders used by latent diffusion.
- Mode Collapse
The failure in which a GAN's generator covers only part of the data distribution, and the training instability and non-convergence that come with it; the analyses behind both (the Helvetica scenario, limited discriminators, the Dirac-GAN and local convergence), the remedies (minibatch discrimination, unrolled GANs, PacGAN, VEEGAN, gradient penalties, Wasserstein losses) and how collapse is measured.
- Neural Style Transfer
The task of rendering the content of one image in the visual style of another with deep network features; Gatys et al.'s optimization, feed-forward stylization networks, instance normalization, arbitrary style transfer with AdaIN and WCT, photorealistic and video style transfer, and how results are evaluated.
- Normalizing Flow
A generative model that turns a simple random code into an image through an invertible network, so that the exact likelihood of any image can be computed; change of variables, coupling and autoregressive layers, Glow's actnorm and invertible 1 × 1 convolutions, continuous flows, dequantization and bits per dimension, and uses in anomaly detection and pose estimation.
- Perceptual Loss
A training loss that compares two images through the activations of a fixed pretrained network instead of their pixels; the feature reconstruction and Gram-matrix style losses, the layers they use, their use in super-resolution, style transfer, autoencoders and inpainting, the artefacts they introduce, and how they relate to the LPIPS metric.
- Personalized Image Generation
Teaching a pretrained text-to-image diffusion model a specific subject from a few images so that it can draw that subject in new scenes; textual inversion, DreamBooth with its prior-preservation loss, Custom Diffusion's cross-attention fine-tuning, low-rank and other compact updates, encoder-based methods without test-time tuning, multi-subject composition, the DreamBooth benchmark with its DINO, CLIP-I and CLIP-T measures, and the risks.
- pix2pix
The conditional GAN of Isola, Zhu, Zhou and Efros for paired image-to-image translation, such as label maps to photos or maps to aerial views; its objective of a conditional adversarial loss plus L1, the U-Net generator, the PatchGAN discriminator and its receptive field, dropout as noise, the FCN-score and perceptual studies, and its high-resolution successor pix2pixHD with multi-scale discriminators and a feature-matching loss.
- Score Matching
A way to learn the gradient of the log-density of images, the score, without ever computing a normalizing constant; Hyvärinen's explicit and implicit objectives, denoising and sliced score matching, Langevin sampling, noise-conditional score networks and the score-SDE view behind diffusion models.
- Stable Diffusion
The family of open-weight text-to-image models from CompVis, Runway and Stability AI, from the latent-diffusion U-Nets of versions 1.x, 2.x and SDXL to the rectified-flow transformers of Stable Diffusion 3 and 3.5; their training data, the distilled Turbo models, licences, the fine-tuning and control ecosystem, reported results, and known risks.
- StyleGAN
NVIDIA's family of style-based GAN generators, in which a mapping network turns the latent code into per-layer styles that modulate every convolution; the original StyleGAN (2019), StyleGAN2's weight demodulation and path length regularization, adaptive discriminator augmentation for small datasets, alias-free StyleGAN3, and StyleGAN-XL.
- Text-to-Image Generation
The task of generating an image from a natural-language description; how it is posed, the GAN, autoregressive and diffusion approaches from 2016 to latent and transformer diffusion models, the text encoders and web-scale training data they rely on, how prompt fidelity and image quality are measured, and the open problems from attribute binding and text rendering to bias, memorization and copyright.
- Variational Autoencoder
An autoencoder made probabilistic, whose encoder outputs a Gaussian distribution over latent codes and whose decoder defines a likelihood for the image, trained by maximizing the evidence lower bound so that codes drawn from a standard normal prior decode to new images; its architecture, training difficulties such as posterior collapse, why its samples are blurry, its main variants and its role as the first stage of latent diffusion.
- Video Diffusion Model
The denoising network behind diffusion-based video generation, a U-Net or transformer that factorizes space and time; how a video U-Net is built from an image U-Net, how temporal attention and temporal convolution divide the work, how one network trains on images and videos, how an unconditionally trained model is conditioned on known frames by replacement or reconstruction guidance, and the latent, cascaded, correlated-noise and factorized variants that followed.
- Video Generation
The task of synthesizing video, unconditionally or from text, an image or past frames; why time makes it harder than image generation, the GAN, autoregressive and diffusion families from 2016 to latent and transformer diffusion models, the factorized and latent architectures they share, long-video and consistency methods, video prediction and world models, how results are measured with FVD, CLIP scores, VBench and human studies, the training data, and the risks from deepfakes to provenance.
- Video Prediction
The task of generating the future frames of a video from its past frames, optionally given the actions of an agent; why a deterministic predictor trained with squared error blurs uncertain futures, the pixel, motion, structure and adversarial predictors, the latent-variable models with a learned prior such as SVG, autoregressive, token and diffusion predictors, action-conditioned prediction for robot planning, the Moving MNIST, KTH, BAIR and Kinetics-600 benchmarks, and best-of-K evaluation with PSNR, SSIM, LPIPS and FVD.
- VQ-VAE
The vector-quantized variational autoencoder, which replaces a continuous latent code with the nearest entries of a learned codebook, trains through the quantizer with a straight-through gradient and codebook and commitment losses or moving-average codebook updates, and generates by fitting an autoregressive prior over the discrete codes; the hierarchical VQ-VAE-2, codebook collapse and its remedies.
- VQGAN
The vector-quantized autoencoder of “Taming Transformers”, trained with a perceptual loss and a patch-based adversarial loss balanced by an adaptive weight so that images compress into short grids of discrete tokens, and the transformer that models those tokens for unconditional, conditional and megapixel synthesis with a sliding attention window; later image tokenizers in outline.
- Wasserstein GAN
A GAN trained to minimize the Earth Mover's (Wasserstein-1) distance instead of the Jensen–Shannon divergence, using a Lipschitz-constrained critic in place of a classifier; the Kantorovich–Rubinstein duality behind it, weight clipping and its failure modes, the gradient penalty of WGAN-GP, the critic's loss as a training signal, and the published results.
- World Models
Learned models that predict how an environment's observations, and often its rewards, change in response to an agent's actions, so that the agent can plan or learn in imagination; the latent dynamics models of Ha and Schmidhuber, PlaNet and the Dreamer line, generative interactive environments such as Genie, video generators used as simulators and their limits, world models for driving and robots, how they are evaluated, and their open problems.
Multimodal & Vision-Language
- BLIP
A family of vision–language models from Salesforce Research that pretrain on image–text pairs for both understanding and generation, from BLIP (2022), with its shared encoder–decoder and caption bootstrapping (CapFilt), to BLIP-2 (2023), which connects a frozen image encoder to a frozen large language model through a Querying Transformer, and InstructBLIP.
- CLIP
A dual-encoder model from OpenAI that trains an image encoder and a text encoder contrastively on 400 million image–caption pairs from the internet, so that images can be classified "zero-shot" by comparing them with embedded text prompts such as "a photo of a dog."
- Document Understanding
Extracting meaning from document images, such as key–value fields from forms and receipts, answers to questions about pages, infographics and charts, and document types, with layout-aware encoders (LayoutLM family), OCR-free generators (Donut, Pix2Struct) and vision–language models, and the FUNSD, CORD, RVL-CDIP, DocVQA, InfographicVQA and ChartQA benchmarks with entity-level F1, ANLS and relaxed accuracy as their evaluation code computes them.
- Flamingo
DeepMind's 2022 family of vision–language models that joins a frozen contrastively pretrained NFNet image encoder to a frozen Chinchilla language model through a Perceiver Resampler and zero-initialized, tanh-gated cross-attention layers, trained on interleaved web pages so that it can learn image and video tasks from a few examples in its prompt.
- Grounding DINO
An open-set object detector that adds a BERT text encoder to the DINO detection transformer and fuses language into its encoder, query initialization and decoder, so that it detects whatever objects a text prompt names, from a list of category names to a referring expression.
- Image Captioning
Generating a natural-language sentence that describes an image, with encoder–decoder networks, attention over image regions and pretrained vision–language models; evaluated with n-gram metrics such as BLEU and CIDEr, semantic and reference-free metrics such as SPICE and CLIPScore, and human judgment.
- Image–Text Retrieval
Ranking a collection of images for a text query, or a collection of captions for an image query, with dual encoders, cross-encoders or a combination of both, evaluated by Recall@K on Flickr30K and MS-COCO.
- Instruction Tuning for Vision–Language Models
Fine-tuning a vision–language model on (image, instruction, response) examples so that it follows requests about images, and then on preference pairs so that it says less that the image does not support; how the data is generated, what is trained and frozen, how mixtures are balanced, and what RLHF, DPO and AI feedback change.
- Learning Transferable Visual Models From Natural Language Supervision
The 2021 ICML paper by Radford and eleven OpenAI colleagues that trained CLIP on 400 million web image–text pairs and studied its zero-shot transfer, linear-probe features, robustness to distribution shift, data overlap and social biases across more than 30 datasets.
- LLaVA
The Large Language and Vision Assistant, an open family of vision–language models that connects a pretrained image encoder to an open language model through a small projector and trains it on visual instruction data, from the 2023 original and LLaVA-1.5 to LLaVA-NeXT, LLaVA-OneVision and LLaVA-Med.
- Multimodal Fusion
Combining information from two or more modalities, such as an image and a question or color and depth, inside one model; where the streams meet (early, intermediate or late) and the operator that joins them (concatenation, products, bilinear pooling, FiLM, cross-attention).
- Multimodal Hallucination
Generated text that describes objects, attributes, relations or counts that the image does not support, in image captioners and vision–language models; how CHAIR, POPE and newer benchmarks measure it, what causes it, and how data, preference tuning, decoding and post-hoc correction reduce it.
- Multimodal Learning
Learning from and relating two or more kinds of data, such as images and text, through shared representations, fusion, alignment and translation between modalities; the overview of vision–language methods, tasks and benchmarks.
- Prompt Engineering
Steering pretrained vision–language models through their inputs instead of their weights; hand-written and ensembled text templates for CLIP-style classifiers, class descriptions written by language models, learned context vectors (CoOp, CoCoOp) and their base-to-new trade-off, visual prompts added to or drawn on the image, in-context examples and chain-of-thought for generative vision–language models, and rewritten prompts for text-to-image models, with the documented gains and sensitivities of each.
- Show and Tell: A Neural Image Caption Generator
The CVPR 2015 paper by Vinyals, Toshev, Bengio and Erhan that generated image captions with a single network, a pretrained CNN feeding an LSTM language model trained by maximum likelihood, and its 2017 journal version on the shared win of the 2015 COCO Captioning Challenge.
- Text Conditioning for Image Generation
How a text encoder's output steers an image generator; per-token embeddings read by cross-attention, appended to attention keys and values, or joined with image tokens in one attention, and pooled caption vectors added to the timestep embedding or used to modulate normalization; what papers report about the choice and size of encoder, the quality of training captions and recaptioning, and how prompt adherence is measured with CLIPScore, T2I-CompBench and GenEval.
- Video Captioning
Generating natural-language descriptions of video, one sentence per trimmed clip or a timestamped sentence for every event in an untrimmed video; encoder–decoder models from mean pooling, S2VT and temporal attention to dense captioners such as PDVC and Vid2Seq and end-to-end video transformers, the MSVD, MSR-VTT, ActivityNet Captions, YouCook2 and VATEX benchmarks, and what BLEU, METEOR, CIDEr and SODA measure.
- Video–Language Model
A vision–language model for video, which samples frames, turns each into visual tokens with an image encoder, compresses and orders those tokens within a budget the language model can afford, and answers questions about the video or describes it; covers frame sampling, temporal tokens, token compression, video question answering, text–video retrieval, long-video benchmarks and the single-frame bias critique.
- Vision–Language Model
A model that reads images, usually together with text, and writes text with a language model, built from a vision encoder, a connector that turns image features into tokens or cross-attention inputs, and a large language model; also called a multimodal large language model.
- Vision–Language Pretraining
Pretraining image–text models on large collections of paired images and captions, with objectives such as image–text contrastive learning, image–text matching, masked language modeling and captioning, so that the models transfer to retrieval, visual question answering and captioning.
- Visual Grounding
Localizing the image region that a piece of text refers to, as one box for a referring expression, a box for every phrase of a caption, or a mask; usually scored as accuracy at IoU 0.5 on RefCOCO-style benchmarks.
- Visual Instruction Tuning
The 2023 NeurIPS paper by Liu, Li, Wu and Lee that had a text-only GPT-4 write visual instruction data from COCO captions and boxes, trained the LLaVA assistant on it, built the GPT-4-judged LLaVA-Bench, and reached 92.53% on ScienceQA by using GPT-4 to settle disagreements with LLaVA.
- Visual Question Answering
Answering a natural-language question about an image, as a classifier over frequent answers or by generating free-form text; benchmarks from VQA and VQA v2 to CLEVR, GQA, OK-VQA, VizWiz and TextVQA, the consensus-based VQA accuracy metric, and the language priors that let models answer without looking.
- Visual Reasoning
Answering questions about, or judging statements against, images when the answer requires composing several steps of perception, such as finding objects, comparing attributes, counting and following spatial relations; the diagnostic benchmarks CLEVR, GQA, NLVR2, SNLI-VE, VCR and Winoground with their protocols, and modular, neuro-symbolic, program-generating and vision–language-model approaches with their documented limits.
Medical & Scientific Vision
- Aerial and Drone Image Analysis
How low-altitude images from drones and aircraft are planned, captured and analysed for mapping and inspection, from platforms, cameras, ground sampling distance and overlap to orthomosaics and surface models, inspection tasks, the VisDrone, UAVDT and UAVid benchmarks, and flight rules as official sources state them.
- Aerial Object Detection
Detecting vehicles, ships, aircraft and other objects in overhead satellite and aerial images, where objects are small, densely packed and arbitrarily oriented; oriented bounding boxes and their angle conventions, rotated IoU and NMS, tiling of large images, the main method families, and benchmarks such as DOTA, xView, HRSC2016 and DIOR with their protocols.
- Bioimage Analysis
How biologists turn microscopy images into measurements, covering the imaging modalities from brightfield to electron microscopy, the tasks of segmentation, tracking, classification, restoration and measurement, the open tools from ImageJ and CellProfiler to napari, deep learning and model zoos, open file formats, FAIR data and image archives, and the reporting practices that make an analysis reproducible.
- Cell Detection
Locating every cell, nucleus or mitotic figure in a microscopy or histology image as a point, with blob detectors, learned proximity and density maps and point detectors, scored by matching points within a radius, and benchmarked in mitosis challenges from MITOS 2012 to MIDOG.
- Cell Segmentation
Finding every cell or nucleus in a microscopy or histology image and giving each its own mask, so that cells can be counted, measured and tracked, with classical threshold-and-watershed pipelines, learned methods such as StarDist, Cellpose and HoVer-Net, and the instance-matching metrics and benchmarks used to compare them.
- Cellpose
A generalist deep-learning model for cell and nucleus segmentation that predicts, for every pixel, a flow towards the centre of its cell, derived from simulated heat diffusion, and recovers each cell as the set of pixels whose flows converge to the same point; with its training set, versions from Cellpose 1 to Cellpose-SAM, and reported results.
- Change Detection
Finding where the Earth's surface changed between co-registered images taken at different dates; the preprocessing it depends on (co-registration and radiometric normalization), classical methods from image differencing and ratioing to change vector analysis, MAD and post-classification comparison, Siamese and other learned networks, and benchmarks such as OSCD, LEVIR-CD, HRSCD and xBD with their protocols.
- Chest X-ray Analysis
Computer vision on chest radiographs; the projections and what they change in the image, the findings that label sets name, labels mined from radiology reports and how noisy they are, the public datasets with their reference standards, localization, shortcuts and failures in subgroups, and published results with their study designs.
- Clinical Validation and Regulation of Medical Imaging AI
How the claims made for a medical image analysis model are tested, from internal, temporal and external validation through reader studies, prospective studies and randomized trials, with the reporting guidelines that describe such studies, the statistics of rare findings, the regulatory pathways in outline, and monitoring after deployment.
- Computed Tomography (CT)
An X-ray imaging method that measures attenuation along many lines through the body and reconstructs cross-sectional images from them, with each voxel calibrated in Hounsfield units.
- Computer Vision in Agriculture
Computer vision for crops and fields, from satellite time series to plant-level images; crop monitoring, yield estimation, weed and disease detection, fruit and wheat head counting, phenotyping in outline, splits by field, region and season, and datasets with their protocols and results.
- Computer Vision in Radiology
How computer vision is used on radiological images such as chest X-rays, mammograms, CT, MRI and ultrasound, for computer-aided detection and diagnosis, triage of reading worklists, quantification and report generation, and how such systems are evaluated, why they fail through shortcut learning and dataset shift, and how regulators classify them.
- Digital Pathology
Computer vision for digitized microscope slides of stained tissue, from whole-slide scanners, pyramidal image formats and stain variation to nucleus segmentation, tissue classification, slide-level prediction by multiple-instance learning, benchmarks such as CAMELYON and PANDA, regulatory milestones and pathology foundation models.
- Fluorescence Microscopy
Fluorescence microscopy as an imaging sensor, covering fluorescence and fluorophores, excitation and emission filters and channels, widefield, confocal, light-sheet and related variants, photon-limited noise, photobleaching and phototoxicity, spectral bleed-through and unmixing, and deconvolution in outline.
- Hounsfield Units
The scale on which CT scanners report attenuation, fixed at 0 for water and −1000 for air, how scanners are calibrated to it, why tissue values still depend on the X-ray energy and the scanner, and how windowing turns it into a displayed image.
- Land Cover Classification
Assigning a land cover or land use class to every patch or pixel of satellite and aerial images; class schemes such as CORINE, ESA WorldCover and Dynamic World, spectral, temporal and spatial features, classical and learned methods, label sources and their errors, design-based accuracy and area estimation, spatial autocorrelation in train and test splits, and benchmarks with their protocols.
- Lesion Detection
Finding and marking every lesion of interest, such as a lung nodule or a mass, in a radiology image or volume; detection and segmentation formulations, candidate generation and false-positive reduction, free-response ROC (FROC) analysis and the competition performance metric, matching criteria, and the LUNA16, DeepLesion and VinDr-CXR benchmarks with their protocols.
- Magnetic Resonance Imaging (MRI)
A medical imaging method that uses a strong magnet, magnetic field gradients and radio waves to sample the spatial Fourier transform (k-space) of the body's hydrogen magnetization, producing cross-sectional images whose contrast is set by tissue relaxation times and the scanner's sequence parameters.
- Mammography and Breast Cancer Screening
Computer vision for screening mammography, covering screening programmes and reading protocols, mammographic views and digital breast tomosynthesis, breast density and BI-RADS categories, screening metrics such as recall rate, cancer detection rate and interval cancers, the public datasets and their access terms, and how AI in screening has been evaluated in retrospective, reader, prospective and randomized studies.
- Medical Computer Vision
An overview of computer vision on medical images, from X-ray, CT, MRI, ultrasound and microscopy to the tasks of reconstruction, segmentation, registration, detection and slide-level prediction, and the problems that set the field apart from vision on photographs, namely scarce and disputed labels, privacy, shift between scanners and sites, evaluation against clinical reference standards, and regulation.
- Medical Foundation Models
Large encoders pretrained once on medical images, image–text pairs or masks and adapted to many clinical imaging tasks, covering how they are pretrained, cases in pathology, radiology, ophthalmology and segmentation (UNI, CONCH, RETFound, CheXzero, BiomedCLIP, MedSAM), how they are evaluated, contamination and benchmark problems, and the evidence on clinical use.
- Medical Image Classification
Assigning a finding, diagnosis or grade to a medical image, study or patient; the unit a label belongs to, weak labels from reports, transfer from natural images and in-domain pretraining, class imbalance, splits by patient and site, evaluation at operating points and calibration, and benchmarks with their protocols.
- Medical Image Preprocessing
The steps that turn stored CT and MRI scans into consistent model inputs, from reading DICOM and NIfTI geometry and orientation to resampling images and labels, normalizing intensities by modality and cropping to the foreground, and how each choice changes the result.
- Medical Image Reconstruction
Computing a cross-sectional image from the indirect measurements a scanner records, such as CT projections or MRI k-space samples, by inverting a model of the acquisition with knowledge of the noise and of what images look like.
- Medical Image Registration
Aligning medical images of the same patient over time or across modalities, or of different patients to an atlas, with rigid, affine and deformable transforms; similarity measures, regularization and diffeomorphic methods, atlas-based segmentation, learned registration such as VoxelMorph, and evaluation by landmarks, propagated labels and Jacobian folding.
- Medical Image Segmentation
Delineating organs, tumors, lesions and other structures voxel by voxel in CT, MRI and other medical images; the 2D, 2.5D and 3D approaches, losses for very small targets, voxel spacing and anisotropy, metrics such as Dice and surface distances with their pitfalls, and benchmarks such as the Medical Segmentation Decathlon.
- Multispectral Satellite Imaging
How passive optical satellites record the Earth in a handful of spectral bands, from whiskbroom and pushbroom scanners and spectral response functions to the four resolutions, calibration from digital numbers to radiance and reflectance, atmospheric correction, band-to-band registration and spectral indices, with hyperspectral imaging in outline and the instruments of Landsat 8 and 9 and Sentinel-2.
- Neuroimaging Analysis
Computer vision on brain MRI; the structural pipeline of skull stripping, bias-field correction, registration to a template, tissue segmentation and cortical parcellation, lesion and tumor analysis, functional and diffusion MRI in outline, the main software suites, the large datasets and their access terms, and published results with their study designs.
- nnU-Net
A self-configuring framework for biomedical image segmentation that keeps a plain U-Net and sets everything around it, from target spacing, normalization, patch and batch size and network depth to the choice of 2D, 3D and cascaded configurations, from a fingerprint of the training data by fixed parameters, rules and cross-validation.
- Ophthalmic Image Analysis
Computer vision for images of the eye; fundus photography and OCT as sensors, diabetic retinopathy grading scales, AI screening including autonomous systems and their US regulatory history, vessel, lesion and OCT layer segmentation, datasets with access terms, and results with their study designs.
- Optical Microscopy
Light microscopy as an imaging sensor, covering the optical path and Köhler illumination, objectives and numerical aperture, the diffraction-limited point-spread function and the Abbe, Rayleigh and Sparrow resolution criteria, brightfield, darkfield, phase contrast and differential interference contrast, cameras and Nyquist sampling, and the artifacts that image analysis has to handle.
- Organ Segmentation
Delineating many organs at once in CT and MRI volumes; atlas-based and multi-atlas segmentation with label fusion, learned networks and whole-body models such as TotalSegmentator, per-organ evaluation in physical units, and the BTCV, CHAOS and AMOS benchmarks with their protocols.
- Remote Sensing and Earth Observation
How satellites and aircraft image the Earth in many spectral bands and with radar, what spatial, spectral, temporal and radiometric resolution mean, how the images are corrected and analysed with computer vision, and the public missions, benchmarks and pitfalls of the field, from Landsat 1 in 1972 to geospatial foundation models.
- Stain Normalization
Standardizing the color of histology images that were stained or scanned under different conditions, by matching color statistics (Reinhard), estimating stain vectors in optical density space (Macenko, Vahadane) or learning the mapping with a network, and how this compares with stain augmentation during training.
- Surgical Computer Vision
Computer vision on video recorded during surgery, mainly laparoscopic and robot-assisted operations; the visual data, instrument presence, segmentation and tracking, phase, step and action recognition, skill assessment, the Cholec80, EndoVis, HeiChole and related datasets with their protocols, temporal models in outline, how frame-level and segment-level metrics differ, and the evidence on use in the operating room with its study designs.
- Synthetic Aperture Radar (SAR) Imaging
An active microwave sensor that images the ground sideways from a moving platform and focuses its echoes into a complex-valued image whose resolution does not depend on range, with speckle, slant-range geometry, polarimetry and interferometry as the properties that set it apart from optical images.
- Tumor Segmentation
Delineating tumors and their sub-regions in CT and MRI, with brain, liver and kidney tumors as cases; nested regions such as whole tumor, tumor core and enhancing tumor, small and absent targets and how they move Dice, disagreement between annotators, and the BraTS, LiTS and KiTS challenges with their protocols and rankings.
- Ultrasound Imaging
A medical imaging method that sends short sound pulses into the body and forms images from their echoes, converting echo time to depth with an assumed speed of sound, building B-mode images by beamforming, envelope detection and log compression, and producing a speckle texture whose statistics follow from the physics.
- Whole-Slide Image Classification
Predicting one label for a whole microscope slide, specimen or patient from thousands of tiles, trained with slide-level labels only; multiple-instance learning formulations, attention-based and transformer aggregation, tile features from pretrained encoders, weak labels from pathology reports, and the CAMELYON and PANDA protocols for evaluation at slide and patient level.
- Whole-Slide Imaging
How slide scanners digitize glass microscope slides into gigapixel images, from tile and line scanning, stitching and focus to resolution in micrometers per pixel, pyramidal vendor, DICOM and OME-TIFF files, compression, color calibration and image quality, and how vision code reads them.
- X-ray Radiography
How a projection radiograph is made, from the X-ray tube and its spectrum through attenuation in the body, scatter and the detector to the stored DICOM image, and what that physics means for vision methods that read radiographs.
Robotics & Embodied Vision
- 3D Occupancy Prediction
Predicting, for every voxel of the space around a vehicle or robot, whether it is free or occupied and by which class, from cameras or LiDAR; covers occupancy grids and semantic scene completion, camera-based occupancy networks, the Occ3D, SurroundOcc and OpenOccupancy labels, visibility masks, and the IoU, mIoU and RayIoU scores.
- Active Perception
The idea that a perceiving agent should decide why, what, how, when and where to sense, from Bajcsy's 1988 definition and the active vision of the late 1980s to next-best-view planning by information gain, active recognition and object search, learned exploration policies and active perception in embodied AI.
- Affordance Detection
Finding where on an object an action is possible, from Gibson's concept of affordances to pixel-wise and part-level affordance segmentation (UMD, IIT-AFF, AffordanceNet) and weakly supervised affordance grounding from videos and exocentric images (interaction hotspots, AGD20K), with the weighted F-measure, KLD, SIM and NSS as the benchmarks' code defines them, and documented limits.
- Autonomous Driving Perception
How a self-driving vehicle turns camera, LiDAR, radar and map data into a 3D description of the road, its own position and the moving objects around it, from Dickmanns's vision-guided van and the DARPA challenges to today's learned 3D detectors, bird's-eye-view models and benchmarks.
- Bird's-Eye-View Perception
Turning images from one or more cameras into a metric top-down grid around the vehicle, by inverse perspective mapping, depth-based lifting or attention, and using that grid for 3D detection, map segmentation and planning.
- Drivable Area Segmentation
Labeling the parts of a driving scene that a vehicle can drive on, from road and free-space segmentation to the directly and alternatively drivable areas of BDD100K, covering flat-ground and stereo (v-disparity) methods, appearance models, segmentation and multi-task networks, the KITTI road and BDD100K protocols and metrics, and documented limits.
- Embodied AI
The study of agents that perceive their surroundings through their own sensors and act in them, covering where the idea comes from, simulators used as research platforms, embodied tasks from navigation and question answering to instruction following and rearrangement, learning by imitation, reinforcement and large pretrained models including vision-language-action models, transfer from simulation to real robots, and how embodied agents are evaluated.
- End-to-End Autonomous Driving
Learning a driving policy as one model trained from sensor data to controls or a planned trajectory, instead of a chain of separately built perception, prediction and planning modules, covering ALVINN and behaviour cloning, compounding error and recovery data, conditional imitation, privileged teachers, sensor-fusion and planning-oriented models, and why open-loop and closed-loop evaluation disagree.
- Lane Detection
Finding the lane markings or lane boundaries of a road in camera images and returning each lane as a curve, covering lane representations, classical edge-and-fitting pipelines, segmentation, row-anchor and curve-based networks, 3D lanes, the TuSimple and CULane benchmarks and their metrics, and the cases where detectors fail.
- Motion Forecasting for Autonomous Driving
Predicting the future trajectories of vehicles, pedestrians and cyclists from their tracked past and the road map, covering multimodal futures and why single-trajectory regression averages them, physics baselines, raster and vectorized scene encoders, anchor-, goal- and transformer-based predictors, joint prediction, and the Argoverse, Waymo Open Motion and nuScenes benchmarks with the exact definitions of their metrics.
- Obstacle Detection and Avoidance
How mobile robots and drones find obstacles in camera images and turn them into avoidance commands, covering stereo and monocular depth, optical flow and time to contact, insect-inspired flow balancing, learned collision prediction, reactive avoidance schemes, how methods are evaluated, and the documented failures on thin, textureless, blurred and transparent obstacles.
- Robot Perception
How robots use cameras, depth sensors, LiDAR, inertial and joint sensors and touch to estimate their own state and the world around them fast enough to act on it, covering the perception–action loop, state estimation, latency and robustness, and the main robot settings from mobile robots and arms to drones, legged robots and cars.
- Robotic Grasping
Finding where and how a robot gripper should take hold of an object from depth images or point clouds, covering gripper types, force closure and grasp quality metrics, data-driven grasp detection with planar rectangles and 6-DoF poses, the Cornell, Jacquard and GraspNet-1Billion benchmarks, and why physical success rates depend on the objects, clutter and trial protocol.
- Sensor Fusion for Autonomous Driving
Combining camera, LiDAR and radar measurements into one estimate of the scene around a vehicle, covering calibration and projection between sensors, time synchronization and ego-motion compensation, early, mid-level, bird's-eye-view and late fusion, and fusion over time through tracking.
- Sim-to-Real Transfer
Training perception models and robot policies in simulation and making them work on real hardware, covering the reality gap, visual and dynamics randomization, system identification, pixel- and feature-level domain adaptation, translating real images into a canonical simulation, synthetic data for detection and pose estimation, and what the published results show, each within its own set-up.
- Traffic Sign Recognition
Finding traffic signs in road images and naming each one, from colour thresholds, shape voting and Haar cascades to convolutional networks; covers the GTSRB classification benchmark and its IJCNN 2011 competition, the GTSDB, Tsinghua-Tencent 100K and Mapillary detection benchmarks with their protocols, physical adversarial attacks on signs, and the limits the sources document.
- Vision for Drones and UAVs
How small unmanned aircraft use cameras to fly and to observe, covering onboard state estimation, obstacle sensing, tracking and following, vision-based landing, inspection, detection and tracking benchmarks recorded from drones such as VisDrone and UAVDT, drone racing and event cameras, the compute, payload and latency budgets, and the documented limits of small objects, motion blur, altitude and viewpoint change and poor light.
- Vision-and-Language Navigation
Following natural-language route instructions through buildings an agent has never seen, covering Room-to-Room and its successors (R4R, RxR, REVERIE), continuous environments (VLN-CE), the goal and path-fidelity metrics (SR, SPL, CLS, nDTW, SDTW) as the official code computes them, seen and unseen splits, and methods from sequence-to-sequence agents through speaker-follower models and pretrained transformers to LLM-based agents.
- Vision-Based Robotic Manipulation
How robots use cameras and depth sensors to pick, place, insert and rearrange objects, covering 6D object pose estimation for picking (PoseCNN, DenseFusion, FoundationPose and the BOP benchmark with its VSD, MSSD and MSPD errors), model-based and model-free pick-and-place, keypoint and affordance representations, assembly, deformable objects, learned visuomotor policies, and how task success is measured.
- Vision-Language-Action Models
Robot policies that map camera images and language instructions to actions using models pretrained on web-scale vision and language data, covering action representations from discretized action tokens to regression heads, diffusion and flow-matching action experts, RT-1, RT-2 and OpenVLA, cross-embodiment training on Open X-Embodiment, fine-tuning, real-robot and simulated evaluation with SIMPLER and LIBERO, and the limits their authors report.
- Visual Navigation
Moving a robot or simulated agent to a goal using what its cameras and depth sensors see, covering point-goal, object-goal, image-goal and instruction-following navigation, map-based pipelines against learned policies, the success and SPL metrics, simulators such as Habitat, and the gap between simulation and real robots.
- Visual Servoing
Control of a robot's motion by feedback from camera images, in which an error between measured and desired visual features is driven to zero through the interaction matrix, in image-based, position-based and hybrid (2.5D) forms.
- Visuomotor Policy Learning
Learning robot control directly from camera images, covering end-to-end training with guided policy search, behaviour cloning and its compounding error with DAgger, reinforcement learning from pixels with QT-Opt and image augmentation (RAD, DrQ), implicit and diffusion policies, action chunking with ACT, and the robomimic and related benchmarks with their protocols.
Vision Systems & Engineering
- Active Learning
How a model chooses which unlabelled images to send for labelling next, by uncertainty, disagreement, coverage or typicality, and why the best choice depends on how many labels have been bought.
- Annotation Formats
How COCO JSON, Pascal VOC XML, YOLO text files and related formats store boxes, polygons and masks, how COCO's compressed run-length encoding works, and what is lost when annotations are converted from one format to another.
- Annotation Quality and Label Noise
How to measure and repair the quality of human labels for vision datasets; raw and chance-corrected agreement (Cohen's kappa, Scott's pi, Fleiss' kappa, Krippendorff's alpha) and why agreement is not accuracy, latent-truth aggregation with Dawid–Skene EM against the majority vote, finding wrong labels with confident learning, and what label noise does to training and evaluation.
- Bias and Fairness in Computer Vision
What it means for a vision system's decisions to be fair to different groups of people, the formal criteria (demographic parity, equalized odds and equal opportunity, calibration within groups), why they cannot all hold at once, how post-processing and training-time methods trade one against another, and what an audit of a deployed system reports.
- Camera Interfaces
The link between a camera and its host, one layer below the interface standards, covering what a frame costs in bytes, pixel-format packing in GenICam's PFNC and in Linux V4L2, the MIPI CSI-2 link-rate formula, frame grabbers against network and USB hosts, GenTL producers and consumers, and the open-source acquisition libraries, with an example that packs, unpacks and cross-decodes two 10-bit layouts.
- CARLA
An open-source driving simulator built on Unreal Engine, with a server–client architecture, synchronous stepping, camera, depth, segmentation and LiDAR sensors, a Traffic Manager and a recorder, described from its 0.9.16 documentation and LibCarla source, with an example that reproduces its depth encoding and camera coordinates.
- COLMAP
An open-source structure-from-motion and multi-view stereo pipeline that recovers camera poses, intrinsics, sparse points and dense geometry from image collections, with a GUI, a command-line interface and Python bindings.
- CUDA
NVIDIA's platform and programming model for general-purpose computing on its GPUs, covering kernels and their launch, the grid, block and thread hierarchy, grid-stride loops, host, device, shared and pinned memory, barriers, atomics, streams, events and CUDA graphs, the toolkit and its libraries, the Python routes and the alternatives, with a privatized image histogram run in numba's CUDA simulator.
- Dataset Curation
How a training set for a vision model is assembled and maintained, from defining what it should represent through filtering, deduplication, balancing, splitting, documentation and versioning, and how data pruning ranks examples so that a smaller set trains as well as a larger one.
- Deploying Vision Models
Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.
- Designing a Computer Vision System
How a working vision system is designed from its requirements, through the choice of sensor, optics, lighting, compute and model, to the data it is trained on, its validation in the field and its maintenance once deployed, with the requirement arithmetic for a measurement station worked through in code.
- Detectron2
Facebook AI Research's PyTorch library for object detection and segmentation, covering its dataset dicts, mappers and augmentations, aspect-ratio batching, in-model normalization and padding, configs, trainers, model zoo, deployment and status as of November 2026, with a NumPy port of its resize, grouping and padding rules.
- DICOM
The standard for storing and exchanging medical images, from its information model, data elements, value representations and transfer syntaxes to Part 10 files, UIDs, encapsulated pixel data and network services, with a section on the NIfTI-1 and NIfTI-2 headers, checked by writing a synthetic CT series byte by byte and converting it to NIfTI.
- Edge AI Accelerators
The inside of the neural processing units and other fixed-function accelerators that run vision models on devices, covering arrays of multiply-accumulate units, systolic arrays, weight- and output-stationary dataflows, the on-chip memory hierarchy, tiling a convolution onto a fixed array, utilization lost to shape mismatch and fill and drain, operator coverage, and why a peak TOPS figure is not a throughput.
- Edge Inference
Running a vision model on the device that captures the images, such as a phone, camera, robot or microcontroller, instead of in a data centre, covering why it is done, the limits of compute, memory, power and heat, the classes of hardware and runtimes, the roofline model for finding memory-bound layers, split and offloaded computation, and how latency and energy are measured.
- Ethics of Computer Vision
The ethical questions that systems which see people raise (capture without consent, identification at scale, inference of what people do not show, errors that fall on people, dual use), the NIST AI Risk Management Framework and the EU AI Act in outline with their dates as of November 2026, research practice, and the limits of technical fixes.
- FFmpeg
The open-source command-line tool and libraries that demux, decode, filter, encode and mux video, with packets and frames, timestamps and time bases, GOPs and frame types, seeking, frame extraction, pixel formats, CRF, ffprobe and licensing, checked on synthetic video whose frames carry their own index.
- GPUs for Computer Vision
The graphics processing unit as a machine for vision workloads, covering SIMT execution in warps, latency hiding by many resident threads, the memory hierarchy from registers to host memory, data reuse by tiling, why convolutions suit GPUs and element-wise operations are limited by bandwidth, TF32 and reduced precision, memory capacity, host–device transfer, GPUs from other vendors and how to choose one.
- Image Annotation
How people attach labels to images for training and evaluating vision models, from image tags and bounding boxes to masks, keypoints, tracks and 3D cuboids, and how guidelines, agreement checks, quality control, model assistance and fair pay decide whether those labels can be trusted.
- Image File Formats
What PNG, TIFF, JPEG files, OpenEXR and RAW containers store, including chunks, tags, sample types, bit depths and metadata; how far each one's dynamic range reaches; and what Pillow, imageio and tifffile return when they read them.
- ImageJ and Fiji
The public-domain Java program for scientific images, its ImageJ2 redesign and the Fiji distribution, with their architecture, update sites, licences and algorithm conventions, and a re-implementation of Subtract Background's rolling ball checked on a synthetic image.
- Industrial Machine Vision
The system engineering of vision on the factory floor, covering machine-vision cameras and their interfaces, GenICam, triggering from encoders and line-scan acquisition, strobed lighting and optics selection, integration with PLCs and robots, gauging and measurement uncertainty in production, and smart cameras against PC-based systems.
- Inference Optimization
Transformations that make a trained vision network run faster or in less memory without changing the function it computes, from constant folding and operator fusion to layout changes, compilation, kernel selection and memory planning, with a worked example of what fusion stops writing to memory.
- ITK and SimpleITK
The Insight Toolkit for N-dimensional scientific image analysis and its simplified interface SimpleITK, with ITK's data-flow pipeline, requested regions and streaming, its physical-space image model, IO factories and wrapping, and a port that streams ITK's discrete Gaussian through padded regions.
- Latency and Throughput
The two measures of a vision system's speed, how waiting, service time and batching set them, why percentiles rather than means describe latency, what parameters, multiply-adds and activations do and do not say about runtime, and how to time a component honestly.
- Lighting for Machine Vision
How the geometry, pattern, wavelength and filtering of the light decide which features of a part have contrast in an inspection image, from the law of reflection behind bright field and dark field to domes, coaxial light, colour, pass filters, polarizers and collimation, with an exact rendering of four defects under six lighting geometries.
- LiteRT (TensorFlow Lite)
Google's on-device runtime for machine-learning models, formerly TensorFlow Lite, covering the .tflite FlatBuffer format, the Interpreter and CompiledModel APIs, CPU, GPU and NPU acceleration, converters and the arena planner that packs a network's tensors into one buffer, with a NumPy port of that planner run on a small CNN.
- Machine Vision Cameras
The industrial camera as a class of device, covering area-scan, line-scan and TDI sensors, shutter modes, binning, decimation and regions of interest, pixel formats and bit depth, how to read an EMVA 1288 datasheet, cooled, SWIR and polarization variants, and a checklist for choosing one.
- MLOps for Computer Vision
The engineering practices that keep a vision model reproducible and maintainable after its first release, from versioning images, annotations and models by their content through pipelines that re-run only the stages a change affects, experiment tracking and the model registry to continuous training and lineage, with an example that caches a shape classifier's pipeline by content hash.
- Model Compression
Making a trained vision model smaller and cheaper to run by pruning weights or filters, quantizing, distilling it into a smaller network, factorizing its layers or choosing a compact architecture, and how to compare the results fairly in accuracy, size, operations and measured latency.
- Model Monitoring and Data Drift
How a deployed vision system is watched for change once it is in service, from what can be observed without labels to sequential detectors (Shewhart, CUSUM and EWMA charts), windowed two-sample tests, label-shift estimation with a black-box classifier, accuracy estimates without labels, feedback loops and failure triage, with an inspection-camera example in which three drifts meet three detectors.
- Model Quantization
Storing and computing a trained network's weights and activations in low-precision formats such as INT8, INT4 or FP8, with a scale and zero point per tensor, channel or group; the integer arithmetic behind it, post-training calibration, quantization-aware training with the straight-through estimator, and what it costs in accuracy.
- MONAI
The MONAI Consortium's open-source, PyTorch-based framework for deep learning on medical images, with its transforms and MetaTensor, datasets, networks, losses and metrics, sliding-window inference, workflows, bundles and Auto3DSeg, the defaults that change results in release 1.6.1, and a NumPy rebuild of its blended sliding-window inference.
- Multi-Camera Systems
Rigs of several cameras treated as one instrument, covering topologies, the overlap graph between views, joint extrinsic calibration of more than two cameras and its upkeep in service, the synchronization budget, clock mapping of many streams, and bandwidth on shared links.
- Network Pruning
Removing weights, filters or channels from a trained network by a score of how much each one matters, from magnitude and Optimal Brain Damage to Optimal Brain Surgeon's inverse-Hessian update, Taylor and redundancy criteria for filters, channel selection by reconstruction, learned channel scales and lottery tickets.
- NumPy
The Python library of n-dimensional arrays in which Python vision libraries exchange images, with the array model, views and copies, broadcasting, accumulator types and NumPy 2's promotion rules that decide what image arithmetic returns.
- ONNX
An open file format and operator standard for trained machine-learning models, used to move a vision model from the framework that trained it, such as PyTorch, to the runtimes and compilers that execute it.
- ONNX Runtime
Microsoft's open-source inference engine for ONNX models, which optimizes a model's graph, partitions it among execution providers for the hardware present, inserts the copies between devices and runs it, with a worked example of partitioning and copy insertion.
- Open3D
An open-source C++ and Python library for 3D data, with legacy and tensor geometry types, neighbour search, normals, registration and reconstruction pipelines and visualization, and a port of its colored ICP that separates what the geometric and photometric terms constrain on a textured plane.
- OpenCV
The open-source C++ library of computer vision and image processing functions, with Python bindings that work on NumPy arrays; its modules, conventions, versions from 1.0 to 5.0, contrib modules and licence.
- Point Cloud Library (PCL)
The open-source C++ template library for 2D/3D image and point cloud processing, with its PointCloud<PointT> data model, its modules, releases and licence, and a port of its plane segmentation and Euclidean cluster extraction checked against the exact single-linkage partition of a synthetic tabletop.
- Privacy in Computer Vision
What images reveal about the people and places in them, how detection followed by blurring, pixelation or a solid box tries to remove it and how an attacker who can model the redaction undoes blur and pixelation, and where federated learning, on-device processing, differential privacy and data protection law fit.
- PyTorch
An open-source Python library for building and training neural networks with tensors, automatic differentiation and GPU acceleration, run eagerly as ordinary Python code and optionally compiled or exported for deployment.
- Real-Time Vision Pipelines
How a vision system turns a stream of camera frames into timely decisions, covering frame budgets, latency against throughput, pipelining and threads, queues and dropped frames, timestamps and clock synchronization, and how end-to-end latency is measured.
- Robot Operating System (ROS)
The open-source robotics middleware ROS 2 as of its Jazzy distribution, covering nodes, topics, services and actions, DDS and the RMW layer, quality of service, discovery, launch and composition, tf2 and the REP 105 frames, the image and camera-info messages, message_filters, bags and distributions, with an example that ports the Python time synchronizers.
- scikit-image
The Python image processing library that works on NumPy arrays, with its modules, its conventions for data types and coordinates, its versions and licence, and an example that checks those conventions on a synthetic image.
- Synthetic Data for Computer Vision
Training and testing vision models on images produced by a renderer, a compositor or a generative model, with labels that come from the generator itself, and the conventions, gaps and evidence that decide whether such data helps.
- TensorFlow
Google's open-source machine learning framework, which runs computations as dataflow graphs, executes eagerly by default since version 2.0, and pairs Keras for model building with tf.data for input pipelines, SavedModel for export, TensorFlow Serving for servers and LiteRT for devices.
- TensorRT
NVIDIA's inference builder and runtime for NVIDIA GPUs, which turns an ONNX file, a PyTorch program or a network built through its API into a serialized engine, with the 11.x move to strongly typed networks, mixed precision prepared by Model Optimizer's AutoCast, tactics and timing caches, optimization profiles, plugins, trtexec and Polygraphy, and a worked example of AutoCast's precision rule.
- Testing and Validating Vision Systems
How a vision system is shown to meet its requirements before and after it goes into service, from the test plan and held-out tests to slice analysis, behavioural and metamorphic tests, stress tests at the requirement's limits, regression tests and the field acceptance test, with an inspection example in which the aggregate passes, one slice fails and a label-free test points to it.
- torchvision
The computer vision library of the PyTorch project, with its v2 transforms and TVTensors, box bookkeeping under crops, the models and weights API, detection models that resize and pad inside the model, operators, datasets and reference scripts, the 0.29 versions, and a NumPy port of RandomIoUCrop, box clamping and SanitizeBoundingBoxes.
- Video Analytics Pipelines
How systems that analyse many live camera streams share ingest, decoding and compute among them, using per-stream knobs and resource–quality profiles, frame sampling and filtering, scheduling against quality and lag goals, and indexes for later queries; with an example that scores frame-filtering policies on sixteen synthetic streams by events found, detector calls and delay.
Datasets & Benchmarks
- A Benchmark for the Evaluation of RGB-D SLAM Systems
Sturm, Engelhard, Endres, Burgard and Cremers's IROS 2012 paper that introduced the TUM RGB-D benchmark, Kinect sequences with calibrated, time-synchronized motion-capture ground truth, and defined two measures already in use, the relative pose error and the absolute trajectory error, for scoring visual odometry and SLAM.
- A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics
Martin, Fowlkes, Tal and Malik's ICCV 2001 paper that collected several human segmentations per natural image, showed with refinement-tolerant error measures that people segment consistently, scored Normalized Cuts against them, and measured how Gestalt grouping cues behave in natural scenes.
- ActivityNet
A benchmark of untrimmed YouTube videos of daily human activities, organized by a taxonomy taken from the American Time Use Survey and annotated with the start and end of each activity; its 200-class release 1.3 is a standard test set for temporal action localization.
- ADE20K
A scene-centric dataset from MIT CSAIL, first released in 2016, in which a single expert annotator labeled every object and stuff region, and many object parts, in more than 20,000 images with an open vocabulary of thousands of classes; its SceneParse150 benchmark, scored by pixel accuracy and mean IoU over 150 classes, became a standard test of semantic segmentation.
- AVA
Google's dataset of 15-minute movie segments in which every person in keyframes sampled once per second has a box and one or more of 80 atomic action labels; the standard benchmark for spatio-temporal action detection, scored by frame-mAP at IoU 0.5 on 60 classes, released as versions v1.0 to v2.2 and extended by AVA-Kinetics.
- AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
Gu, Sun, Ross and colleagues' CVPR 2018 paper that introduced AVA, a dataset of 15-minute movie segments in which every person in keyframes sampled once per second is boxed and labelled with atomic actions, together with an I3D-on-Faster-R-CNN baseline whose 15.8% frame-mAP showed how hard person-level action detection was.
- Benchmarks and Leaderboards
What a computer vision benchmark consists of (a task, data, a protocol and a metric), how held-out and sequestered test sets, evaluation servers, submission limits, challenges and leaderboards keep its scores honest, and the ways those scores mislead.
- Berkeley Segmentation Dataset
Natural photographs segmented by several people each, released as BSDS300 in 2001 and extended to BSDS500 in 2011, with the benchmark that scores boundary detectors by matched precision and recall (ODS, OIS, AP) and segmentations by covering, Rand index and variation of information.
- BraTS
The Brain Tumor Segmentation challenge series, held at MICCAI every year since 2012, with its multi-institutional MRI data, changing labels and sub-challenges, and three successive ways of scoring a segmentation.
- Caltech Pedestrian
The Caltech Pedestrian Dataset of Dollár, Wojek, Schiele and Perona, about 10 hours of driving video from Los Angeles with 350,000 pedestrian boxes, its 2009 benchmark and 2012 re-evaluation, the rules that filter detections and standardize box widths, and the 2016 re-annotation.
- Caltech-101
A dataset of web images of 101 object categories and a background class, collected at Caltech in 2003, whose fixed-number-of-training-images protocol and mean per-class accuracy made it the standard multi-class recognition benchmark of the mid-2000s.
- Caltech-UCSD Birds (CUB-200)
A dataset of Flickr photographs of 200 bird species from Caltech and UC San Diego, released as CUB-200 in 2010 and extended in 2011 as CUB-200-2011 with 11,788 images, boxes, 15 part locations with visibility and 312 attributes; part localization on it is scored in units of a per-part standard deviation.
- CelebA
The CelebFaces Attributes dataset of 202,599 web images of 10,177 celebrities, each labelled with 40 binary face attributes and five landmarks, which became the standard benchmark for face attribute prediction, the source of CelebA-HQ, and one of the most used training sets in generative modelling.
- ChestX-ray14
The NIH Clinical Center's chest radiograph dataset with labels mined from radiology reports; its two versions (ChestX-ray8 with eight findings and ChestX-ray14 with fourteen), the 1,600 radiologist-drawn boxes, the splits, and the localization protocol with Acc., AFP and the IoBB criterion worked through on synthetic heatmaps.
- CheXpert
Stanford's chest radiograph dataset with positive, negative and uncertain labels mined from reports and radiologist-labelled validation and test sets; its label format, splits, competition protocol and access, with the five uncertainty policies worked through at their population optimum and scored under a reading of the leaderboard's two columns.
- CIFAR
CIFAR-10 and CIFAR-100, two labelled sets of 60,000 colour images of 32×32 pixels taken from the 80 Million Tiny Images collection in 2009, with 10 classes or 100 classes in 20 superclasses, the standard small-scale benchmarks for image classification.
- Cityscapes
A dataset of stereo video from the streets of 50 cities, mostly in Germany, with 5,000 finely and 20,000 coarsely annotated frames, whose benchmarks for pixel-level, instance-level, and panoptic segmentation became the standard for urban scene understanding.
- COCO
Microsoft's Common Objects in Context, a dataset of everyday scenes with per-instance segmentation masks for 80 object categories, plus keypoints, captions, stuff and panoptic labels, whose evaluation protocol became the standard benchmark for object detection.
- Dataset Bias
How the way a dataset's images are chosen, photographed and labelled makes it an unrepresentative sample of the visual world, how cross-dataset tests measure the damage, and what bias means when the images show people, from demographic representation to withdrawn datasets.
- DAVIS
The Densely Annotated VIdeo Segmentation benchmark, from the 50 single-object clips of DAVIS 2016 to the multi-object, interactive and unsupervised DAVIS 2017–2020 challenges, with its formats, licence and official evaluation code.
- DTU MVS
The DTU Robot Image Data multi-view stereo set, objects photographed by a robot arm from 49 or 64 positions under seven lightings with structured-light reference scans; its accuracy and completeness protocol as the official MATLAB code computes it, the scene lists used by learned methods, and its terms.
- Ego4D
A 3,670-hour dataset of first-person video from 931 camera wearers in 74 locations and 9 countries, with five benchmark families (episodic memory, hands and objects, audio-visual diarization, social interactions and forecasting); its versions, per-institution licence agreements, and the natural language query protocol as the official code computes it.
- EPIC-KITCHENS
An egocentric video dataset recorded with head-mounted cameras in participants' own kitchens and labelled from their spoken narrations as verb–noun actions; its 55-hour (2018) and 100-hour (2020) versions, the recognition, anticipation, detection, retrieval and domain-adaptation challenges, how the official code scores verb, noun and action accuracy, the seen/unseen splits, and its licence and privacy measures.
- ETH3D
Laser-scanned indoor and outdoor scenes from ETH Zürich with high-resolution multi-view, low-resolution many-view and two-view stereo benchmarks, scored by accuracy, completeness and F1 at tolerances from 1 cm to 50 cm, plus an RGB-D SLAM benchmark.
- EuRoC MAV
Eleven visual-inertial sequences flown by a hexacopter in an ETH Zurich machine hall and a motion-capture room, with stereo images, IMU data and laser-tracker or Vicon ground truth; the standard test set for visual-inertial odometry and SLAM, and how trajectory alignment changes the errors reported on it.
- FFHQ
Flickr-Faces-HQ, NVIDIA's 2019 dataset of 70,000 aligned 1024×1024 face crops from Creative Commons Flickr photos, built for StyleGAN and since the standard benchmark for unconditional face generation; how the crops were made from facial landmarks, the image and dataset licences, the opt-out process, and the conventions that make FID on it differ between papers.
- Flickr30k
A 2014 corpus of 31,783 Flickr photographs of everyday activities with five crowdsourced English captions each, extending the 8,092-image Flickr8k and later annotated with phrase-to-box links as Flickr30k Entities; its splits, the 1K-test Recall@K retrieval protocol as the VSE++ code computes it, caption metrics, phrase localization, terms of use and known biases.
- Flying Chairs
A synthetic optical flow dataset of rendered chairs moving over Flickr photographs, created to train FlowNet and still used to pretrain flow networks.
- HMDB: A Large Video Database for Human Motion Recognition
Kuehne, Jhuang, Garrote, Poggio and Serre's ICCV 2011 paper that collected 51 action classes from movies and web video, tested how far low-level colour and scene features could classify existing action datasets, and measured two baselines, a bag of HOG/HOF words and a model of the visual cortex, across camera motion, viewpoint, occlusion and video quality.
- HMDB51
A 2011 dataset of about 6,800 short clips of 51 human actions, cut mostly from movies, with meta-labels for body visibility, camera motion, viewpoint, number of people and quality; its three 70/30 splits that keep clips of one source video on one side, mean accuracy over the splits, the stabilized version, and the results that mattered.
- Human3.6M
The laboratory motion-capture dataset of 3.6 million 3D human poses with synchronized video, depth and body scans, its capture record, licence, split and evaluation server, the paper's own error measures, and how its headline numbers are aggregated.
- ICDAR Robust Reading Benchmarks
The Robust Reading Competition series run at the International Conference on Document Analysis and Recognition since 2003, whose Focused Scene Text (2013) and Incidental Scene Text (2015) datasets became the standard scene text benchmarks, with their DetEval and IoU matching rules, word recognition and end-to-end protocols and online evaluation server.
- ImageNet
A large database of web images organized by the noun hierarchy of WordNet, whose 1,000-class ILSVRC subset became the standard benchmark for image classification and the default source of pretrained networks.
- ImageNet: A Large-Scale Hierarchical Image Database
Deng, Dong, Socher, Li, Li and Fei-Fei's CVPR 2009 paper that introduced ImageNet, a database of web images organized by the WordNet noun hierarchy and verified by crowd workers, which later became the basis of the ILSVRC benchmark.
- iNaturalist
Species classification datasets built from research-grade observations on the iNaturalist citizen-science platform, released with the FGVC competitions of 2017, 2018, 2019 and 2021, with observer-based and date-based splits, long-tailed training sets, top-k scoring and non-commercial terms of use.
- Kinetics
DeepMind's family of human action datasets, Kinetics-400, -600, -700 and -700-2020, made of about 10-second YouTube clips with one action label each and distributed as links that disappear over time; the standard large-scale benchmark and pretraining set for action recognition.
- KITTI
A benchmark suite of real driving data from Karlsruhe, with stereo, optical flow, scene flow, odometry, object detection, and tracking benchmarks built from camera, LiDAR, and GPS/IMU recordings.
- Labeled Faces in the Wild
The 2007 benchmark of 13,233 news photographs of 5,749 people for deciding whether two face images show the same person, with its development and ten-fold views, the image-restricted and unrestricted training settings, the 2014 outside-data categories, and the mean accuracy, standard error and ROC curves its protocol asks for.
- LAION
LAION-400M and LAION-5B, open collections of hundreds of millions to billions of image URLs paired with their web alt-text and filtered by CLIP similarity, which trained OpenCLIP and Stable Diffusion, were withdrawn in December 2023 over links to suspected child sexual abuse material, and were re-released as Re-LAION-5B in August 2024.
- LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models
Schuhmann and colleagues' NeurIPS 2022 Datasets and Benchmarks paper that released LAION-5B, 5.85 billion CLIP-filtered image–text pairs from Common Crawl, with the tools to download, embed and search them, open reproductions of CLIP trained on its subsets, and a datasheet, and that won an Outstanding Datasets and Benchmarks Paper award.
- Learning Multiple Layers of Features from Tiny Images
Alex Krizhevsky's 2009 University of Toronto report on training restricted Boltzmann machines on two million tiny web images, which introduced CIFAR-10 and CIFAR-100 to measure whether the learned features help object recognition.
- LSUN
The Large-scale Scene Understanding dataset from Princeton, about ten million web images in 10 scene categories and 59 million in 20 object categories labelled by a cascade of deep classifiers with crowd workers in the loop; its label precision, the LSUN challenges at CVPR 2015–2017, and its second life as a generative-modelling benchmark (bedrooms, churches, cats, cars), where FID protocols differ from paper to paper.
- LVIS
A large-vocabulary instance segmentation dataset of over 1,000 object categories on the COCO images, built as a federated dataset with positive and negative image sets per category, whose evaluation reports AP for rare, common and frequent categories.
- Market-1501
The 2015 person re-identification benchmark of 32,668 detector boxes of 1,501 people seen by six cameras outside a campus supermarket, with its 500,000-image distractor set, its good, distractor and junk labels and their codes in the release, the single- and multi-query protocol and mAP as third-party code computes them, its unread licence, and the withdrawal of DukeMTMC.
- Microsoft COCO: Common Objects in Context
Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár and Zitnick's ECCV 2014 paper that introduced COCO, a dataset of non-iconic everyday scenes with a segmentation mask for every instance of 91 object categories, built through a three-stage crowdsourcing pipeline.
- Middlebury Benchmarks
Small, high-accuracy stereo and optical flow benchmarks from Middlebury College and Microsoft Research that defined how dense correspondence methods were evaluated in the 2000s.
- MNIST
The dataset of 70,000 handwritten digits, 28 × 28 pixels, that LeCun, Cortes and Burges built from NIST's handwriting collections at Bell Labs in 1994, how its writers were split and its images normalized, why its 10,000-image test set is saturated, and the EMNIST, QMNIST and Fashion-MNIST datasets that extend, reconstruct or replace it.
- ModelNet
A collection of 3D CAD models built at Princeton for 3D ShapeNets in 2015, whose 10- and 40-category subsets, ModelNet10 and ModelNet40, became widely used benchmarks for 3D shape classification and retrieval; its original protocol tested 12 rotated copies of 20 models per category and ranked retrieval results over “the remaining test data”.
- MOTChallenge
A benchmark for multiple pedestrian tracking, released as MOT15, MOT16, MOT17, and MOT20, with hidden test annotations and a central evaluation server.
- MPI Sintel
An optical flow benchmark rendered from the open-source animated film Sintel, with long sequences, large motions, motion blur, and atmospheric effects.
- MPII Human Pose
A benchmark for 2D human pose estimation from the Max Planck Institute for Informatics, released in 2014, with about 25,000 YouTube frames and 40,000 people annotated with 16 body joints, collected through a taxonomy of everyday activities; its single-person track is scored with PCKh, a joint-distance measure normalized by head size, and its multi-person track with average precision over body joints.
- MVTec AD
An industrial anomaly detection dataset of 5,354 images in 15 texture and object categories, with defect-free training images and pixel-precise defect masks, released by MVTec in 2019 and scored by an official evaluation script that computes the area under the per-region overlap curve and image-level ROC AUC.
- nuScenes
A multimodal driving dataset from Motional (formerly nuTonomy) with 1,000 annotated 20-second scenes from Boston and Singapore, recorded with six cameras, five radars and a LiDAR, whose detection benchmark ranks methods by the nuScenes detection score (NDS) and whose tracking benchmark ranks them by AMOTA.
- NYU Depth v2
An indoor RGB-D dataset recorded with a Microsoft Kinect, with 1,449 densely labeled frames and about 400,000 raw video frames, used as the standard indoor benchmark for single-image depth estimation and RGB-D semantic segmentation.
- Object Retrieval with Large Vocabularies and Fast Spatial Matching
Philbin, Chum, Isard, Sivic and Zisserman's CVPR 2007 paper that scaled visual-word retrieval to a million Flickr images with approximate k-means vocabularies and spatial re-ranking by affine transformations hypothesized from single region correspondences, and introduced the Oxford 5K benchmark.
- Open Images
Google's dataset of about 9 million Flickr images released in seven versions from 2016 to 2022, with verified positive and negative image-level labels, 16 million boxes on 600 classes, instance masks, visual relationships, localized narratives and point labels, and a challenge metric built for incomplete, hierarchical labels.
- Oxford Buildings
The Oxford 5K and Paris 6K landmark retrieval benchmarks of Flickr photographs, with 55 query regions each, Good, OK and Junk labels and a trapezoidal mAP; their distractor sets; the 2018 Revisited Oxford and Paris with Easy, Medium and Hard protocols and the R1M distractors; and INRIA Holidays as a related benchmark.
- PASCAL VOC
The PASCAL Visual Object Classes challenge (2005–2012), whose 20-class datasets of Flickr photographs with bounding boxes and segmentations set the standard protocol for evaluating object detection.
- Places
A scene-centric database of millions of web photographs labeled with the type of place they show, built at MIT from the SUN category list, whose Places205 and Places365 subsets and released Places-CNNs became the standard training data, benchmark and pretrained models for scene recognition.
- SA-1B
Meta AI's Segment Anything 1-Billion dataset of 11 million licensed photographs and 1.1 billion class-agnostic masks, all generated automatically by SAM, with its released format, the automatic-generation recipe and how the released code's defaults differ from it, the mask-quality audit and its research licence.
- ScanNet
A dataset of about 1,500 RGB-D video scans of indoor rooms, captured with a tablet depth sensor, reconstructed into meshes and labelled by crowd workers, whose benchmark made it the standard test of 3D semantic and instance segmentation of indoor scenes.
- ShapeNet
A repository of 3D CAD models organized by WordNet synsets, whose subsets ShapeNetCore (55 categories) and ShapeNetSem (270 categories), and the part labels added to 16 of its categories, became the standard training and benchmark data for 3D shape classification, part segmentation, single-view reconstruction and shape generation.
- Something-Something
A crowd-acted video dataset of people performing 174 caption templates such as "Pushing something from left to right" with everyday objects, built so that classes differ in what happens rather than in the objects or scenes; its second version (220,847 videos) is the standard benchmark for temporal reasoning in action recognition.
- Tanks and Temples
A benchmark of 14 large indoor and outdoor scenes, recorded as 4K video and scanned with an industrial laser scanner, that scores complete image-based 3D reconstruction pipelines by precision, recall and F-score at a per-scene distance threshold.
- The Pascal Visual Object Classes (VOC) Challenge
Everingham, Van Gool, Williams, Winn and Zisserman's 2010 IJCV paper that described the PASCAL VOC challenge up to 2007, its Flickr dataset, annotation rules and evaluation, and analysed what the 2007 entries had learned.
- TUM RGB-D
The TUM RGB-D SLAM dataset and benchmark, Kinect and Asus Xtion sequences at 30 Hz with motion-capture ground truth at 100 Hz; its sequence categories, file formats and calibration, timestamp association, and the absolute trajectory error and relative pose error as the official scripts compute them.
- UCF101
A dataset of 13,320 YouTube clips of 101 human actions, released in 2012 with three official train/test splits that keep clips of the same group apart, a standard action recognition benchmark of the 2010s, later widely used for fine-tuning after Kinetics pretraining.
- Unbiased Look at Dataset Bias
Torralba and Efros's CVPR 2011 paper that asked whether recognition datasets represent the visual world, showed that a classifier can tell twelve popular datasets apart, measured how much "car" and "person" models lose when tested on other datasets, priced a training example from one dataset in examples of another, and named four kinds of dataset bias.
- Visual Genome
Stanford's dataset of about 108,000 Flickr images densely annotated by crowd workers with region descriptions, objects, attributes, relationships, scene graphs and question–answer pairs mapped to WordNet; versions 1.0 to 1.4, the filtered 150-object, 50-predicate split that scene graph generation is scored on, Recall@K with and without the graph constraint as the widely used code computes it, and its role in dense captioning and in the detector behind bottom-up features.
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna and colleagues' 2017 IJCV paper that introduced Visual Genome, about 108,000 COCO and YFCC100M images annotated by crowd workers with region descriptions, objects, attributes, relationships, scene graphs and question–answer pairs mapped to WordNet, with baseline experiments on attributes, relationships, region captions and question answering.
- Visual Object Tracking Challenge
The annual VOT benchmark for single-target tracking, run since 2013 and continued as VOTS from 2023, with curated sequences, rotated-box and mask annotation, an open toolkit, and the expected average overlap measure.
- VQA Dataset
The VQA dataset of open-ended questions about COCO images and abstract scenes, each with ten crowd answers; VQA v1 (2015) and the balanced VQA v2 (2017), the consensus accuracy as the official evaluation code computes it (answer normalization, averaging over annotator subsets, the 2021 fix), the test-dev and test-standard servers, and the language-prior analyses.
- VQA: Visual Question Answering
Antol, Agrawal, Lu, Mitchell, Batra, Zitnick and Parikh's ICCV 2015 paper that proposed free-form, open-ended visual question answering as an automatically scorable task, collected the VQA dataset of questions about COCO images and abstract scenes with ten answers each, analysed it with human studies and reported the first baselines.
- Waymo Open Dataset
Waymo's family of autonomous-driving datasets, whose perception set of 1,150 twenty-second scenes from San Francisco, Mountain View and Phoenix carries LiDAR and camera boxes, and whose 3D detection benchmark scores AP and the heading-weighted APH at two difficulty levels.
- WIDER FACE
A face detection benchmark from the Chinese University of Hong Kong, released in 2015, with 32,203 web images of 61 event classes (60 in the paper) and 393,703 labelled faces, scored by average precision on nested Easy, Medium and Hard settings computed by its official MATLAB code.
Evaluation & Metrics
- Absolute Trajectory Error
The root-mean-square distance between estimated and ground-truth camera positions after the two trajectories are aligned; how the alignment (SE(3), Sim(3), yaw-only, all poses or the first), timestamp association and reference frames change the number, and how it relates to the relative pose error and the KITTI odometry metric.
- Accuracy
The fraction of test examples a classifier labels correctly, with its top-k, balanced, and per-pixel variants, and why it can mislead on imbalanced data or small test sets.
- Average Precision
The area under a detector's precision–recall curve for one class, computed from score-ranked detections matched to ground truth by IoU, and its averages over classes and IoU thresholds (mAP and COCO AP).
- Benchmark Overfitting
How reusing one test set for years of model selection can inflate benchmark scores, what the theory of adaptive data analysis says about it, and what replication studies of CIFAR-10, ImageNet, MNIST and Kaggle found when they measured it.
- BLEU
The n-gram precision metric of Papineni et al. (2002), with its brevity penalty, corpus aggregation, smoothing for single sentences, and the parameters that make two “BLEU” scores incomparable.
- Chamfer Distance
The average distance from each point of one point set to the nearest point of another, in both directions; the conventions that make published values incomparable, its sensitivity to sampling density and outliers, and how it relates to the Hausdorff distance, the Earth Mover's Distance and the F-score.
- Character Error Rate
The edit distance between recognized and reference text divided by the reference length, with the Levenshtein recurrence, substitution, deletion and insertion counts, word error rate, pooled and per-line averages, normalized edit distance, MER, WIL, character accuracy and the text normalizations that change the numbers.
- CIDEr
The consensus-based caption metric of Vedantam, Zitnick and Parikh (2015), which compares a candidate's TF-IDF n-gram vectors with those of many human references, its defended variant CIDEr-D, the human study behind it, and how the official code departs from the paper.
- Confusion Matrix
The table that counts how often each true class is predicted as each class, with true and false positives and negatives, row and column normalization, the metrics read from it, and how to see from it which classes a vision model confuses.
- Cross-Validation
How k-fold cross-validation and its variants (leave-one-out, repeated, stratified, grouped, time-ordered and nested) reuse a small labelled image set for both training and evaluation, what the estimate measures, how noisy it is, and why tuning and reporting on the same folds is optimistic.
- Depth Estimation Error Metrics
The standard error measures for predicted depth maps (absolute and squared relative error, RMSE, log RMSE, threshold accuracy, the scale-invariant log error and the log10 error), which errors each one rewards, and the evaluation conventions that change the reported numbers.
- Dice Coefficient
Twice the overlap of a predicted and a ground-truth mask divided by their total size, equal to the F1 score over pixels; the standard overlap score of medical image segmentation and, in soft form, a training loss.
- Do ImageNet Classifiers Generalize to ImageNet?
Recht, Roelofs, Schmidt and Shankar's ICML 2019 paper that rebuilt the CIFAR-10 and ImageNet test sets by repeating the original collection procedures, found accuracy drops of several to fourteen points for every model with the ranking almost unchanged, and traced the drop to harder images rather than test-set reuse.
- Endpoint Error
The standard accuracy measure for optical flow, the distance in pixels between an estimated flow vector and the true one, averaged over the image.
- Expected Calibration Error
A measure of how far a classifier's confidence departs from its accuracy, computed by binning predictions by confidence and averaging the gap between accuracy and mean confidence in each bin; its definition, reliability diagrams, the maximum calibration error, how binning changes its value, and why it is read alongside the Brier score and negative log-likelihood.
- F1 Score
The harmonic mean of precision and recall, with its weighted F-beta form, macro, micro, weighted and samples averaging over classes, its dependence on the decision threshold, its identity with the Dice coefficient on binary masks, and the cases where it misleads.
- Fréchet Inception Distance
The standard sample-based score for image generative models, the Fréchet (2-Wasserstein) distance between Gaussians fitted to Inception-v3 features of real and generated images; its formula, its small-sample bias, its sensitivity to resizing and compression, its ImageNet bias, and the related metrics IS, KID, precision and recall, and FD-DINOv2.
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
The NIPS 2017 paper by Heusel, Ramsauer, Unterthiner, Nessler and Hochreiter that gave the GAN generator and discriminator separate learning rates, proved convergence to a local Nash equilibrium by two time-scale stochastic approximation, read Adam as a heavy ball with friction, and introduced the Fréchet Inception Distance to measure the results.
- Hausdorff Distance
The largest distance from a point of one set to the nearest point of the other, taken in both directions; its directed, percentile, modified and partial (ranked) forms, why filled regions and their boundaries give different values, how libraries extract boundaries and take percentiles, and matching a template under translation with one distance transform.
- Higher Order Tracking Accuracy
A multi-object tracking metric that combines detection accuracy and association accuracy through their geometric mean, averaged over localization thresholds.
- IDF1
An identity-based score for multi-object tracking that matches whole ground-truth and predicted trajectories one-to-one and reports the F1 score of correctly identified detections.
- Image Quality Assessment
Predicting how good an image looks to people, with or without a pristine reference; how human quality scores are collected (ITU-R BT.500 methods, MOS and DMOS, paired comparisons), the benchmark databases, how a metric is validated by rank and linear correlation after a logistic fit, the families of metrics from PSNR to deep no-reference models, and why metrics trained on one kind of distortion fail on another.
- Image Quality Assessment: From Error Visibility to Structural Similarity
Wang, Bovik, Sheikh and Simoncelli's 2004 IEEE Transactions on Image Processing paper, which argued that image quality should be judged by the loss of structural information rather than by the visibility of errors, defined the SSIM index, and tested it on 344 JPEG and JPEG2000 images rated by human observers.
- Inception Score
A sample-only score for image generative models, the exponential of the mean KL divergence between an ImageNet classifier's label distribution for each generated image and the label marginal over all of them; its formula and mutual-information form, the 50,000-sample, 10-split protocol, what it rewards and misses, and how it relates to FID.
- Intersection over Union
The area where a predicted region and the ground truth overlap, divided by the area they cover together, the standard way to score localization in detection, segmentation, and tracking.
- LPIPS (Learned Perceptual Image Patch Similarity)
A full-reference perceptual distance that compares unit-normalized deep-network features of two images, with per-channel weights calibrated on human similarity judgments; its formula and lin, tune and scratch variants, the 2AFC and JND evaluation on BAPPS, the conventions that change its value, its weaknesses as a loss, and its successors E-LPIPS, ST-LPIPS and DISTS.
- Mean Per Joint Position Error
The average Euclidean distance between predicted and true 3D joints, and the family built on it; absolute and root-relative error, the choice of root and joint set, scale-normalized N-MPJPE, Procrustes-aligned PA-MPJPE, root position error and per-vertex error, with what each alignment removes and what it hides.
- Mean Squared Error
The average squared difference between a prediction and its reference, over pixels, channels or values, with its square root RMSE and the mean absolute error MAE; how they are computed for images, what their units and statistical meaning are, and why very different distortions can share one MSE.
- Multiple Object Tracking Accuracy
The CLEAR MOT accuracy score for multi-object tracking, which counts misses, false positives, and identity switches over a sequence and normalizes them by the number of ground-truth objects.
- NIQE (Natural Image Quality Evaluator)
A no-reference image quality score that fits a multivariate Gaussian to natural scene statistics of a test image's patches and measures its distance from a model fitted to pristine natural images; its MSCN features, formula, results on LIVE, use in perceptual super-resolution, its relatives BRISQUE, IL-NIQE and PIQE, and where it fails.
- Object Keypoint Similarity
The per-object similarity between predicted and annotated keypoints that COCO uses in place of IoU, its Gaussian keypoint similarity and constants, what it implies under Gaussian error, Ronchi and Perona's error taxonomy and rescoring analysis, and how evaluation code, toolkit forks and OKS-based NMS compute it.
- Panoptic Quality
The standard score for panoptic segmentation, which matches predicted and ground-truth segments at IoU above 0.5 and combines the matched IoUs with the unmatched segments into an IoU-weighted F1, per class and averaged over classes.
- Peak Signal-to-Noise Ratio
The mean squared error between an image and its reference expressed in decibels relative to the largest possible pixel value; how the peak value, colour channels, border cropping and averaging over a test set change the number, and why a higher PSNR does not always mean a better-looking image.
- Percentage of Correct Keypoints
The family of hit-rate metrics for keypoints, PCK, PCKh, PCP and their 3D and correspondence forms, which count a predicted point or limb as correct when its error falls under a threshold scaled by the object, a body part or the limb itself; their definitions, implementations and the conventions that make numbers incomparable.
- Precision and Recall
Two complementary measures of a classifier or detector, the fraction of its positive predictions that are correct and the fraction of true positives it finds, with the F-score, the confusion matrix, and the precision–recall and ROC curves built from them.
- Recall@K
The fraction of queries whose ranked results contain a correct item within the first K, the standard score of image–text retrieval, metric learning, place recognition and re-identification, with its two definitions, median rank, rsum, and the alternatives that look beyond the first hit.
- ROC Curve and AUC
The area under the receiver operating characteristic curve, a threshold-free measure of how well a classifier's scores rank positives above negatives, with its probabilistic meaning, computation with tied scores, partial and multi-class variants, and the operating-point summaries used in vision.
- Statistical Significance in Evaluation
How much of a reported difference between two vision models is noise from the finite test set or from training randomness, and the confidence intervals, paired tests, multiple-comparison corrections and reporting practices that tell a real improvement from a lucky seed.
- Structural Similarity Index
A full-reference image quality index that compares local means, contrasts and correlations of two images in a sliding Gaussian window and averages the result; its formula and constants, the multi-scale MS-SSIM, the implementation conventions that change its value, its relation to the MSE, and where it disagrees with what viewers see.
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, Isola, Efros, Shechtman and Wang's CVPR 2018 paper, which collected the BAPPS dataset of human patch-similarity judgments, found that deep features of supervised, self-supervised and unsupervised networks agree with people far more often than PSNR or SSIM, and introduced the calibrated LPIPS metric.
- Training, Validation, and Test Sets
Why labelled images are divided into a training set that fits the model, a validation set that selects it and a test set that reports it once, how large each must be, and how leakage between them, from near-duplicate images to frames of one video, inflates the reported score.