Computer Vision

Computer vision is the field that builds programs to extract information from images and video: what objects a photograph contains and where they are, how far away each surface is, how the camera and the things in front of it are moving, what a scene would look like from another viewpoint, or how to describe a picture in words. Related names include machine vision, image understanding and image analysis, and the field overlaps with pattern recognition. Its inputs are usually photographs and video from ordinary cameras, but the same methods apply to depth sensors, LiDAR, medical scanners, microscopes and satellites.

This article is the overview of the field: what computer vision is and why it is hard, how it relates to its neighbours, the main tasks, a short history with dates, and a map of the categories this site is organized into. Each subject it mentions has its own article, linked where it first appears, and is summarized here only in a sentence or two.

Definition

Textbooks define the field by its goal. Ballard and Brown (1982, §1.1) call computer vision the enterprise of automating and integrating the many processes and representations used for visual perception. In their account it includes techniques that are useful on their own, image processing and statistical pattern classification, but also, more importantly, geometric modeling and cognitive processing. Szeliski (2022, §1.1) states the goal as describing the world seen in one or more images and reconstructing its properties, such as shape, illumination and the distribution of colours. Torralba, Isola and Freeman (2024, §1.2) start from David Marr’s definition of vision, “to know what is where by looking” (Marr, 1982, as they quote it), and stress that the goal is not to measure the light reaching the camera but to recover the properties of the scene that matter to an observer: surfaces, objects, materials, free space, other agents.

Two points follow from these definitions. First, the input is an image, a digital image in practice: an array of numbers sampled and quantized from the light that reached a sensor. Second, the output is a statement about the world, not about the array: a label, a position, a depth, a motion, a 3D model, a sentence. Ballard and Brown’s examples already span several sensing modalities: their table of image analysis tasks lists X-rays, ultrasound, radar, infrared and electron microscopy alongside visible light (Ballard & Brown, 1982, Table 1.1).

Intuition

People see without effort, which hides how much work seeing is. Szeliski (2022, §1.1) calls vision an inverse problem: it recovers unknowns from information that is insufficient to determine them fully. Three kinds of loss make the problem hard.

Projection loses depth. A camera maps a 3D scene onto a 2D image, so a small object close to the camera and a large one far away can produce the same picture. The pinhole camera model makes this exact: every point along a ray through the camera centre lands on the same pixel.
A pixel mixes several causes. A pixel’s value depends on the surface’s reflectance, the illumination, the viewing direction and the camera’s processing (see image formation). Dark paper in bright light can give the same value as white paper in shadow.
Most of the scene is hidden. Torralba, Isola and Freeman (2024, §1.2.1) point out that occlusion is common, and that for any observer more surfaces are occluded than visible.

A vision system must therefore add knowledge that the image does not contain: physical models of how images form, assumptions about which scenes are likely, or regularities learned from many examples. Szeliski names exactly these three resources (physics-based models, probabilistic models and machine learning) as the ways to choose among the possible solutions. The example below shows the first two losses in a few lines of code, and how a second view removes one of them.

Formal Definition

Write the image as the output of a forward model,

\mathbf{I} = \mathcal{F}(\mathbf{s}) + \boldsymbol{\varepsilon},

where $\mathbf{s}$ collects the scene’s properties (geometry, materials, lighting, the camera’s pose and parameters), $\mathcal{F}$ is image formation, the process that physics and computer graphics describe, and $\boldsymbol{\varepsilon}$ is noise. Computer vision estimates either $\mathbf{s}$ itself or a quantity $y = g(\mathbf{s})$ that a task needs: a class label, a set of boxes, a depth map, a motion field.

The difficulty is that $\mathcal{F}$ is many-to-one: many scenes produce the same image, so $\mathcal{F}$ has no inverse. Two standard ways out correspond to the two halves of the field’s history.

Models and priors. A probabilistic formulation puts a prior $p(\mathbf{s})$ on scenes and chooses, for example, the most probable scene given the image,

\hat{\mathbf{s}} = \arg\max_{\mathbf{s}} \; p(\mathbf{I} \mid \mathbf{s}) \, p(\mathbf{s}),

where the likelihood $p(\mathbf{I} \mid \mathbf{s})$ encodes the forward model and the noise. Szeliski (2022, §1.1) describes this Bayesian approach, with a loss function on wrong answers and inference that minimizes the expected loss, as an approach his book uses often. The likelihood is often maximized on its own (maximum likelihood estimation), and the prior often appears as a regularization term in an optimization problem.

Learning from examples. Instead of writing $\mathcal{F}$ and $p(\mathbf{s})$ down, collect $N$ images $\mathbf{I}_i$ with the desired outputs $y_i$ and fit a function $f$ that reproduces the mapping:

\hat{f} = \arg\min_{f \in \mathcal{H}} \; \frac{1}{N} \sum_{i=1}^{N} \ell\big(f(\mathbf{I}_i), y_i\big).

Torralba, Isola and Freeman (2024, §1.3.8) describe learning-based vision this way, as programming vision algorithms with data, and note that it does not change the input, the output or the role of the mapping between them, only how the mapping is found. The prior has not disappeared: it lives in the training data, the loss $\ell$ and the hypothesis class $\mathcal{H}$ , such as the convolutions of a convolutional neural network. The machine learning article covers the framework, including why fitting the training images is not the goal.

Example: Two Scenes, One Image

The example renders two different scenes through the same pinhole camera, as 8-bit arrays of the kind every vision system receives. Scene A is a square card 0.2 m wide at 2 m from the camera, with reflectance 0.8 under unit illumination. Scene B is a card twice as large and twice as far, with half the reflectance under twice the light. In this simplified model a pixel records reflectance times illumination, averaged over the part of the pixel that the card covers. The code then moves the camera 10 cm to the right and renders each scene again.

import numpy as np

F, SIZE, SS = 100.0, 64, 8          # focal length (px), image size, subsamples per pixel

def render(side, depth, albedo, light, cam_x=0.0):
    """Pinhole image of a fronto-parallel square card on a black background."""
    # pixel centres on a supersampled grid, principal point at the image centre
    u = (np.arange(SIZE * SS) + 0.5) / SS - SIZE / 2
    X = u * depth / F + cam_x        # back-project each sample onto the card's plane
    Y = u * depth / F
    inside = (np.abs(X)[None, :] <= side / 2) & (np.abs(Y)[:, None] <= side / 2)
    radiance = albedo * light * inside            # Lambertian, light along the view axis
    pixel = radiance.reshape(SIZE, SS, SIZE, SS).mean(axis=(1, 3))
    return np.round(255 * np.clip(pixel, 0, 1)).astype(np.uint8)

scenes = {"A": dict(side=0.2, depth=2.0, albedo=0.8, light=1.0),   # small, near, bright paper
          "B": dict(side=0.4, depth=4.0, albedo=0.4, light=2.0)}   # large, far, grey paper, strong light

left = {k: render(**s) for k, s in scenes.items()}
print("image:", left["A"].shape, left["A"].dtype, "| row 32, columns 25-39 of A:")
print(left["A"][32, 25:40])
print("A and B identical:", np.array_equal(left["A"], left["B"]))

baseline = 0.1                                    # move the camera 10 cm to the right
for k, s in scenes.items():
    right = render(**s, cam_x=baseline)
    cols = np.arange(SIZE)
    centre = lambda img: (img[32].astype(float) * cols).sum() / img[32].sum()
    disparity = centre(left[k]) - centre(right)
    print(f"{k}: disparity {disparity:.2f} px -> depth {F * baseline / disparity:.2f} m,"
          f" brightest pixel {right.max()}")

image: (64, 64) uint8 | row 32, columns 25-39 of A:
[  0   0 204 204 204 204 204 204 204 204 204 204   0   0   0]
A and B identical: True
A: disparity 5.00 px -> depth 2.00 m, brightest pixel 204
B: disparity 2.50 px -> depth 4.00 m, brightest pixel 204

The first view is a $64 \times 64$ array of 8-bit integers, and the two scenes produce exactly the same array: size and distance trade off in projection, and reflectance and illumination trade off in the pixel value, so no program that sees only this image can tell the scenes apart. The second view separates their depths. The near card shifts twice as far across the image as the far one, and dividing the baseline times the focal length by that shift, the disparity, recovers 2 m and 4 m, the principle behind stereo vision and structure from motion. The brightness ambiguity survives the second view: both cards still give 204, so telling grey paper in strong light from white paper in weak light needs another assumption, such as a known light source or a learned sense of which materials and lights are common. Real systems face both ambiguities at once, together with noise, occlusion and clutter.

Relation to Neighbouring Fields

Image processing

Image processing computes new images, or simple measurements, from images: denoising, enhancement, resampling, filtering. Szeliski (2022, §1.2) writes that what distinguished computer vision from the already existing field of digital image processing was the aim of recovering the three-dimensional structure of the world from images, as a step towards full scene understanding. Ballard and Brown (1982, §1.1) instead count image processing as one of computer vision’s parts. Both views are consistent with the continuum that the image processing article describes: there is no sharp line, vision systems run image processing inside them, and image processing increasingly uses learned vision models.

Computer graphics

Graphics and vision are each other’s inverse. Graphics renders images from a description of a scene, the forward model $\mathcal{F}$ above, and Szeliski (2022, §1.1) notes that the forward models vision uses come from physics and from computer graphics; vision tries to go the other way, from images to a description of the world. This is why vision is sometimes called inverse graphics, and why methods that fit a scene model by rendering it and comparing the result with the photographs recur throughout the field. The link is as old as the field: Roberts’s 1963 thesis built a 3D description of polyhedral objects from a photograph and then displayed that structure from any viewpoint with hidden lines removed (Roberts, 1963, abstract). Szeliski (2022, §1.2) singles out the growing interaction with graphics as perhaps the most notable development of the 1990s, through image-based modeling and rendering. Neural rendering, neural radiance fields and 3D Gaussian splatting are its current form: they fit a scene representation to photographs through a differentiable renderer. Generative models, which synthesize images, now sit inside computer vision too: Torralba, Isola and Freeman (2024) give them a part of their textbook (chapters 32–34).

Machine learning and pattern recognition

Pattern recognition was part of the field’s vocabulary from the start: Papert (1966) hoped the Summer Vision Project would be a landmark in the development of “pattern recognition”, in his quotation marks, and Ballard and Brown (1982) include statistical pattern classification as a component of vision. Torralba, Isola and Freeman (2024, §1.3.8) trace learning in vision to Rosenblatt’s perceptron of 1958, through hand-designed features fed to shallow classifiers in the 2000s, to networks trained end to end from pixels after 2012; they observe that as of the 2020s it is hard to find a vision system that does not use deep learning somewhere. Machine learning supplies the framework (training, generalization, evaluation), the neural network the dominant model family, and representation learning the idea that the features a vision system works with can themselves be learned. Vision has in turn given machine learning many of its benchmarks, from MNIST and ImageNet onwards.

Robotics

Early computer vision was robot perception. Szeliski (2022, §1.2) describes the field’s beginnings as the visual perception part of an agenda to give robots intelligent behaviour, at laboratories such as MIT, Stanford and CMU, and the image features article recounts how Moravec’s interest operator came from a robot rover that navigated by vision. Robots need vision that runs in real time on a moving platform and supports action: visual odometry and SLAM estimate the robot’s motion and map its surroundings, object detection and depth estimation find what to grasp or avoid, and camera calibration relates the camera to the robot’s body. Torralba, Isola and Freeman (2024, §1.2) note that computer vision mostly studies perception from a disembodied perspective, without an agent or a task, and list a theory of embodied perception among what is still missing (§1.4); robotics is where that gap shows.

Photogrammetry

Photogrammetry, measurement from photographs, is older than computer vision and grew up with aerial mapping. Szeliski (2022, §1.2) records that the global optimization methods that structure from motion adopted in the 1990s were later recognized as the bundle adjustment long used in photogrammetry; the bundle adjustment article traces the method to aerial mapping in the late 1950s. The standard lens distortion model is named in part after the photogrammetrist Duane Brown. In our reading the fields differ mainly in emphasis: photogrammetry puts metric accuracy and error analysis with calibrated cameras first, while computer vision has emphasized automation, uncalibrated and casual imagery, and recognition. In surveying, 3D reconstruction from photographs is itself called photogrammetry.

Human vision

The study of biological vision shaped the field and still supplies questions. As Szeliski and Torralba and colleagues summarize Marr (1982), he proposed analysing any information-processing system at three levels: the computational theory (what is computed and why), the representation and algorithm, and the hardware implementation (Szeliski, 2022, §1.2; Torralba, Isola & Freeman, 2024, §1.3.6). Optical illusions, such as the checker-shadow illusion, in which two squares of the same gray look different because the visual system discounts the shadow, show the visual system solving the same inverse problem with its own assumptions (Szeliski, 2022, Figure 1.3; Torralba, Isola & Freeman, 2024, §1.2.2). The term receptive field comes from neurophysiology: Hubel and Wiesel recorded simple and complex cells in the cat’s visual cortex from 1959 (Torralba, Isola & Freeman, 2024, §1.3.5), and Fukushima’s neocognitron (1980), a precursor of the convolutional network, modeled its cells on them.

The Main Tasks

Torralba, Isola and Freeman (2024, §1.2.2) list tasks as varied as detecting changes, estimating motion, recognizing and locating objects, recognizing materials, reading text, building 3D models, finding free space and understanding the behaviour of animals, and note that some seem to need much outside knowledge while others seem solvable from images alone. Szeliski (2022, §1.3, Figure 1.12) arranges his book along an axis from images and semantics (“what”) to 3D geometry (“where”), a reference, he writes, to separate pathways in the human visual system that should not be taken too seriously. The tasks below follow that division, with generation and language added.

Task family	Input	Output	Examples
Recognition	Image	Labels, boxes, identities	Classification, detection, face and pose recognition, text reading, retrieval
Segmentation	Image	A label or segment for every pixel	Semantic, instance and panoptic segmentation
Correspondence	Two or more images	Matched points, an alignment	Feature matching, image registration
3D vision	Images, depth or range data	Depth, camera poses, 3D models	Stereo, structure from motion, SLAM, 3D reconstruction, novel view synthesis
Motion and tracking	Video	Motion fields, object tracks	Optical flow estimation, single- and multi-object tracking
Video understanding	Video	Actions and events, with times and places	Action recognition, temporal action localization
Generation and restoration	Noise, labels, text or a degraded image	A new or improved image	Image generation, text-to-image generation, inpainting, super-resolution, denoising
Vision and language	Images with text	Text, or a match between the two	Captioning, visual question answering, grounding, image–text retrieval

Recognition answers what is in an image. Image classification assigns labels to the whole image; object detection adds a bounding box for each object; face recognition, human pose estimation and optical character recognition specialize recognition to people and writing, and image retrieval finds images of the same object or place in a collection.

Segmentation assigns every pixel: to a class (semantic segmentation), to an object (instance segmentation), or both (panoptic segmentation); image segmentation is the task overview. Papert’s 1966 memo already set figure–ground separation, dividing a picture into likely objects, likely background and “chaos”, as the project’s primary goal, ahead of region description and object identification (Papert, 1966, p. 2).

Correspondence finds the same physical point in different images, the step that most geometric methods start from. Feature matching pairs detected image features, and image registration aligns whole images.

3D vision recovers geometry: depth estimation from one image or by stereo matching, camera poses and sparse points from many photographs by structure from motion, the sensor’s path and a map by SLAM, dense surfaces by 3D reconstruction, and new views of the scene by novel view synthesis. Multi-view geometry states what can be recovered from how many views.

Motion and tracking estimate how image content moves between frames (optical flow and its estimation) and follow objects over time, keeping their identities (object tracking, multi-object tracking).

Video understanding recognizes and locates actions and events (action recognition, temporal action localization), where the answer depends on how frames change, not only on what one frame shows.

Generation runs the problem in the forward direction with learned models: image generation produces images from noise or class labels, text-to-image generation from text, and inpainting fills in missing regions. Restoration tasks such as super-resolution and denoising sit between image processing and generation.

Vision and language connects images to text: describing them (image captioning), answering questions about them (visual question answering), finding the region a phrase refers to (visual grounding), and retrieving images for a query (image–text retrieval). Vision–language models now handle many of these with one model that reads images and writes text.

History

The field’s history is usually told in decades, and the two textbooks used here tell it differently in places. The account below follows Szeliski (2022, §1.2) and Torralba, Isola and Freeman (2024, §1.3), with primary sources where they could be read, and says where the sources disagree.

Beginnings (1960s)

Computers able to process images appeared in the early 1960s, and digital image processing grew alongside the space program (see image processing). On 10 May 1963 Lawrence G. Roberts submitted his MIT PhD thesis, Machine Perception of Three-Dimensional Solids, for a degree dated June 1963. His program processed a photograph into a line drawing, turned the line drawing into a three-dimensional description of the objects, and displayed that description from any viewpoint with hidden lines removed. It assumed that the photograph was a perspective projection of objects built from transformations of known 3D models, each supported by other objects or by a ground plane (Roberts, 1963, abstract). Torralba, Isola and Freeman (2024, §1.3.7) call this the first computer vision Ph.D.; its world of simple polyhedra became known as the blocks world.

On 7 July 1966, Seymour Papert’s Vision Memo No. 100 at MIT’s Project MAC proposed the Summer Vision Project: the group’s summer workers would build a significant part of a visual system, split into sub-problems that individuals could work on independently. Its goals were figure–ground analysis, region description and, finally, object identification by matching against a vocabulary of known objects, with Gerald Sussman coordinating the project meetings (Papert, 1966, pp. 1–2). MIT’s repository records the memo’s date as 1 July 1966; the memo itself is dated 7 July. A popular version of the story has Marvin Minsky asking a single undergraduate, Sussman, to link a camera to a computer over a summer and have it describe what it saw. Szeliski (2022, §1.2) repeats that story from Boden (2006), who cites Crevier (1993) for it, and adds in a footnote that the actual memo was written by Papert and involved a whole cohort of students. Torralba, Isola and Freeman (2024, §2.1) take the project as a sign of the optimism of the early days; the task proved harder than anyone expected.

Where the sources disagree on the start. Szeliski (2022, §1.2) writes that computer vision first started out in the early 1970s, as the perception component of the artificial intelligence agenda, and cites Roberts’s blocks-world work in its 1965 published form. Torralba, Isola and Freeman (2024, §1.3.7) begin with Roberts’s 1963 thesis and describe 1960–1990 as one period dominated by the geometric perspective. The 1963 thesis and the 1966 memo show vision problems being worked on at MIT in the 1960s; whether that is the start of the field or its prehistory is a matter of definition.

Geometry and early vision (1970s)

Early scene understanding extracted edges and inferred the 3D structure of blocks-world scenes from the arrangement of 2D lines; several line-labeling algorithms were developed, and objects that were not polyhedra were modeled with generalized cylinders and arrangements of parts (Szeliski, 2022, §1.2). Barrow and Tenenbaum’s intrinsic images explained variations of shading and intensity by image formation effects such as surface orientation and shadows (Szeliski, 2022, §1.2). Quantitative methods appeared in the same period: the first feature-based stereo algorithms, intensity-based optical flow such as the Horn–Schunck and Lucas–Kanade methods (both 1981), and the recovery of structure and camera motion together, with Ullman (1979) and Longuet-Higgins (1981) (Szeliski, 2022, §1.2).

Marr’s book Vision, published posthumously in 1982, summarized much of this period’s thinking (Szeliski, 2022, §1.2; Torralba, Isola & Freeman, 2024, §1.3.6). Besides the three levels of analysis described above, it proposed, as Torralba, Isola and Freeman (2024, §1.3.6) summarize it, a sequence of representations from the image through a primal sketch of edges and junctions and a 2½-D sketch of the depth and orientation of the visible surfaces (a depth map with discontinuities marked) to 3D models of objects. Torralba, Isola and Freeman (2024, §1.3.6) judge that it is hard to overstate the book’s influence, and suggest that its shortcoming may have been underestimating the role of learning in building representations, which is the premise of deep learning.

Mathematical methods (1980s)

Szeliski (2022, §1.2) characterizes the 1980s by more sophisticated mathematics for quantitative image and scene analysis: image pyramids and scale space for coarse-to-fine search, wavelets towards the end of the decade, shape from shading, texture and focus, better edge detection, including Canny’s detector (1986), and active contours. Stereo, flow, shape-from-X and edge detection were found to share one framework as variational optimization problems made well posed by regularization; Geman and Geman (1984) posed such problems as Markov random fields, and Kalman filters were used to update estimates over time. The Kalman filter and Bayesian filtering articles cover their use in tracking.

Projective geometry, physics and statistics (1990s)

In the 1990s, projective invariants for recognition grew into a concerted effort on structure from motion; projective reconstruction without calibration and factorization methods were developed, and global optimization was adopted and later recognized as photogrammetry’s bundle adjustment (Szeliski, 2022, §1.2). Dense stereo advanced, with global optimization by graph cuts perhaps the biggest breakthrough in Szeliski’s view; multi-view stereo, tracking with active contours and particle filters, and segmentation by normalized cuts and mean shift followed. Statistical learning entered, in Szeliski’s account, through eigenfaces, principal component analysis applied to face recognition by Turk and Pentland (1991). Torralba, Isola and Freeman (2024, §1.3.7) add that digital cameras became common in this decade and that learning-based object detection, in particular face detection, gained attention; the Viola–Jones detector (2001) used boosting to detect faces, outperformed the hand-engineered detectors of the time in speed and detection rate, and became popular in consumer cameras (§1.3.8).

Features, benchmarks and learning (2000s)

Two trends defined the 2000s. Recognition became feature-based: local descriptors such as SIFT and dense ones such as HOG, pooled into a bag of visual words or fed to a support vector machine or boosting (Szeliski, 2022, §1.2; Torralba, Isola & Freeman, 2024, §1.3.7). And progress began to be driven by shared benchmarks: Caltech-101, PASCAL VOC, LabelMe and CIFAR, and the first very large image collections, with millions of images, such as Tiny Images and ImageNet (Torralba, Isola & Freeman, 2024, §1.3.7). Torralba and colleagues note that although learning played an important role, most architectures and features were still designed by hand. In parallel, techniques from image-based rendering moved into everyday cameras as computational photography: panoramic stitching, high-dynamic-range imaging by exposure bracketing, and tone mapping (Szeliski, 2022, §1.2).

Deep learning (2012 onwards)

In 2012 an ensemble of convolutional networks trained on ImageNet data won the ILSVRC-2012 classification competition with a top-5 test error of 15.3%, against 26.2% for the second-best entry (Krizhevsky et al., 2012). AlexNet, as it came to be called, relied on GPU training; Szeliski (2022, §1.2) calls it the first neural network to win the ImageNet challenge and credits the change that followed to large labeled datasets, general-purpose computing on graphics processors, fast dissemination on arXiv and open-source frameworks and models. Torralba, Isola and Freeman (2024, §1.3.8) note that the event has come to be called the “ImageNet moment”, and that it was a system trained end to end, from pixels to predictions.

Within a few years deep networks took over most tasks: detection with R-CNN (2014) and its successors, segmentation with fully convolutional networks (2015), deeper classification networks such as ResNet, optical flow, depth and pose estimation, and action recognition. Neural networks themselves were not new: Torralba, Isola and Freeman (2024, §1.3.8) trace them through the perceptron, Fukushima’s neocognitron, which they place in the 1980s and cite in its 1980 publication, and the convolutional networks that LeCun and colleagues trained in 1989, whose system the US Postal Service put into use to read handwritten digits (see LeNet), and explain the delay by the lack of data and compute before the 2010s. Generative models arrived in the same decade, with generative adversarial networks (2014) and diffusion models (2015), whose widely used DDPM formulation followed in 2020. The transformer, introduced for language in 2017, was applied directly to image patches by the Vision Transformer (2021).

Foundation models (2020s)

Bommasani et al. (2021) proposed the name foundation models for models trained on broad data at scale that can be adapted to a wide range of downstream tasks. They pointed to both the leverage and the risk of many applications inheriting the defects of one model. In vision the pattern appears as large models pretrained once and reused: CLIP (2021) learned a joint image–text embedding from 400 million image–caption pairs and classifies images by comparing them with text prompts; self-supervised models such as DINO and the masked autoencoder learn features without labels; SAM (2023) returns object masks for points, boxes or rough masks; Stable Diffusion generates images from text; and vision–language models such as LLaVA answer questions about images in natural language. Training data grew from curated datasets to web-scale collections such as LAION, which brought new questions about dataset bias, licensing and evaluation. As of October 2026 these models are commonly fine-tuned or used frozen as the starting point for specific tasks, as transfer learning and representation learning describe.

Map of This Site

Every article on this site belongs to one of sixteen categories. The list below summarizes each and names a few articles to start from.

Foundations. The mathematics and physics that the rest builds on: linear algebra, probability and statistics, optimization, how a camera forms an image (image formation, the pinhole camera model, color spaces, image noise), sensors such as LiDAR, and classical machine learning, from k-nearest neighbors to principal component analysis.
Image Processing & Computational Photography. Operations that compute images from images: convolution and filtering, the Fourier transform, histogram equalization, mathematical morphology, image warping, denoising and super-resolution. Image processing is the category’s overview.
Features & Representation. What a vision system computes from pixels to work with: edge detection, feature detection, descriptors and matching, learned embeddings, and self-supervised and contrastive learning. Image features and representation learning are its overviews.
Geometry & 3D Vision. Cameras and the 3D world: projective geometry, camera calibration, homographies, epipolar geometry, stereo, structure from motion, SLAM, 3D reconstruction and neural rendering. Multi-view geometry is its hub.
Deep Learning for Vision. Networks and how they are trained: deep learning is the category’s overview, with the neural network, backpropagation, convolutional networks, transformers, batch normalization, data augmentation, and model families from AlexNet and ResNet to the Vision Transformer.
Recognition & Detection. Naming and locating: image classification, object detection with two-stage and one-stage detectors such as Faster R-CNN, YOLO and DETR, faces, human pose, text and retrieval.
Segmentation. Labeling every pixel, from classical thresholding, watershed and graph-based methods to U-Net, Mask R-CNN and SAM; image segmentation is the task overview.
Motion & Tracking. Optical flow, motion estimation, feature tracking and object tracking, including tracking by detection and the filters it uses.
Video Understanding. Recognizing what happens in video; video understanding is the category’s overview, with temporal modeling and the video model families.
Generative Vision. Generative models of images, such as the variational autoencoder, GANs and diffusion models, and the tasks they serve, from image generation to image-to-image translation.
Multimodal & Vision-Language. Models that connect images with text: multimodal learning, vision–language pretraining, CLIP, vision–language models and their failure modes such as multimodal hallucination.
Medical & Scientific Vision. Vision on medical scans, microscopy, astronomy and remote sensing, where images often come from sensors other than cameras and errors carry high costs. As of October 2026 the general methods it draws on are filed under other categories: U-Net, introduced for biomedical images, under Segmentation, and image registration under Geometry & 3D Vision.
Robotics & Embodied Vision. Perception for agents that act: robot navigation and manipulation and autonomous driving. As of October 2026 the articles it builds on are filed under other categories: SLAM and visual odometry under Geometry & 3D Vision, LiDAR under Foundations, and driving datasets such as KITTI under Datasets & Benchmarks.
Vision Systems & Engineering. Software libraries, deployment, efficiency and hardware: running models on phones and embedded devices. As of October 2026 the efficient architectures this relies on, such as MobileNet and depthwise separable convolutions, are filed under Deep Learning for Vision.
Datasets & Benchmarks. The datasets that train and test vision systems, from MNIST and ImageNet to COCO, and how benchmarks and leaderboards work and mislead.
Evaluation & Metrics. How results are measured: accuracy, precision and recall, the confusion matrix, the F1 score, intersection over union, average precision, image quality measures such as PSNR and SSIM, and the methodology of training, validation and test sets and statistical significance.

Common Misconceptions

“Vision is easy, because people do it effortlessly.” The Summer Vision Project planned a significant part of a visual system for one summer (Papert, 1966). Szeliski (2022, §1.1) traces the misperception to early artificial intelligence, when the cognitive parts of intelligence were believed to be harder than the perceptual ones.
“Computer vision is image processing.” Image processing mostly maps images to images; vision maps images to statements about the scene. The two overlap and depend on each other, but the goal differs.
“Computer vision is recognizing objects.” Recognition is one family of tasks. Recovering geometry and motion, which needs no names for things, has been central since Roberts’s thesis, and much of the field, from stereo to SLAM, is about where rather than what.
“Deep learning solved computer vision.” Learned models dominate most benchmarks, but they inherit the inverse problem’s ambiguity and the biases of their training data and can fail on images unlike those they were trained on, as the benchmark overfitting and dataset bias articles discuss. Torralba, Isola and Freeman (2024, §1.4) list visual common sense, integration with other senses and embodied perception among what is still missing.
“Images are photographs in visible light.” Vision methods run on X-ray, ultrasound, radar, infrared and electron-microscope images, as Ballard and Brown’s table already showed in 1982, and on depth maps, point clouds and event streams today.

Where It Is Used

Szeliski (2022, §1.1) lists industrial applications including optical character recognition of postal codes and number plates, machine inspection, retail checkout, warehouse logistics, medical imaging, self-driving vehicles, photogrammetry from aerial and drone images, match moving for film, motion capture, surveillance, and fingerprint and biometric recognition; and consumer applications including panorama stitching, exposure bracketing, 3D modeling from snapshots, video stabilization and face detection. On this site, phones and cameras appear through face detection, stitching with homographies, denoising and super-resolution; augmented reality combines camera tracking, scene understanding and rendering in real time; driving and robotics combine detection, tracking, depth estimation and SLAM on camera and LiDAR data, benchmarked on datasets such as KITTI; and mapping, document reading, identity verification and search draw on 3D reconstruction, optical character recognition, face recognition, and image and image–text retrieval.