History of Computer Vision

The textbooks start the history of computer vision as a field in the 1960s or the early 1970s, with the first programs that interpreted photographs. The questions those programs tried to answer are much older: how light carries information about the world into the eye, and how an observer turns that light into a world of objects. This article covers both. It describes the theories of vision that came before computers, as Torralba, Isola and Freeman (2024, §1.3) summarize them; gives a dated timeline from Greek optics to foundation models, with the site article that owns each event; compares the ways the textbooks divide the field’s past; and draws out the themes that recur across periods.

The decade-by-decade account of the field itself, from Roberts’s blocks world and the Summer Vision Project to foundation models, is the History section of computer vision. The histories of particular methods live in their own articles: deep learning, image processing, image features, and each model and dataset article. The neuroscience that shaped the field, from Ramón y Cajal’s anatomy of the retina to Hubel and Wiesel’s oriented cells, is in human visual system and receptive field.

Definition

In this article the history of computer vision means two things. The first is the history of the field: the programs, methods, datasets and models built since the early 1960s to extract information from images. The second is the history of the theories of vision on which the field draws. Torralba, Isola and Freeman (2024, §1.3.7) argue that computer vision is not only an engineering discipline but another part of an interdisciplinary scientific effort to understand how natural intelligence and perception work, and they introduce the field through that longer history, from the Greek philosophers through Helmholtz, Gestalt psychology, Gibson, neuroscience and Marr to learning-based vision (§1.3.1–§1.3.8). They describe their own account as a set of snapshots, not a replacement for the books and papers it mentions (§1.3).

Two cautions apply to any such history. Torralba et al. (§1.3.7) note that recent history is hard to summarize because it needs fine temporal resolution and because the crucial discoveries are not yet clear. And credit is often uncertain: they note, for instance, that early versions of the law of reflection are credited to Hero, Ptolemy, Archimedes and Plato, and were perhaps known before them (§1.3.1).

Before Computers: Theories of Vision

Rays from the eye or to the eye

The Greek philosophers had two competing theories of how the eyes pick up information about the world, as Torralba et al. (§1.3.1) describe them. In intromission theories, defended by Demokritos, Epicurus and Lucretius, with early versions dated by Torralba et al. to 425 BC, objects emitted copies of themselves (eidola or simulacra) that entered the eye. The theory left open how one object could send copies to many observers without the copies interfering, and how the copy of a large object could fit into the eye. In extramission theories, which began with Empedocles (about 494–434 BC) and were followed by Plato and Euclid among others, the eye sent out rays that sensed the world like fingers; Empedocles held that the fire in the eye produced them, which also explained why cats’ eyes seem to shine at night. Plato’s theory combined elements of both. Aristotle criticized extramission, pointing out, as Euclid also did, that the stars were too far away for rays from the eye to reach them, and suggested that only some objects are sources of light while others reflect it.

Euclid (dated 325 BC by Torralba et al.) gave the first mathematical theory of vision. In his Optics, the rays from the eye travel in straight lines and form a cone with its apex in the eye; things seen under a larger angle appear larger; and geometric reasoning with similar triangles gives the size of objects, such as the height of an object from the length of its shadow. Torralba et al. call the straight-line model of light a major discovery and the foundation of image formation, scene geometry and computer vision, and write that Euclid’s work set the basis of perspective. The pinhole camera model is the modern form of the straight-ray cone. In our reading the geometry does not depend on which way the rays run: a cone of rays from the eye and a cone of rays converging on it are the same set of lines. Hero of Alexandria postulated straight-line propagation and an early law of reflection, and Ptolemy described refraction (§1.3.1).

Ibn al-Haytham (965–1040), known in the Latin world as Alhacen, transformed these theories into what Torralba et al. (§1.3.1) call a true scientific discipline, in his Book of Optics (Kitāb al-Manāẓir, known in Latin as De aspectibus), published between 1028 and 1038. He rejected extramission: sight could be explained by light coming from objects, with no need for rays from the eye. He described the pinhole camera, and Torralba et al. credit him with inventing the camera obscura. Where Ptolemy believed that the distance of an object could be measured with one eye, by feeling the length of a ray, Ibn al-Haytham showed experimentally that both eyes were needed to perceive depth, the cue that stereo vision exploits. Kepler, building on his work in 1604, gave the first complete description of how images form on the retina: the eye’s lens projects a reversed picture there, where earlier theories had taken the crystalline lens to be the sensitive element (§1.3.1). The human visual system article describes the eye as an optical instrument.

Helmholtz: perception as inference

Hermann von Helmholtz (1821–1894) trained as a physician and contributed, in Torralba et al.’s list, to philosophy, physics, audition, colour theory and visual perception (§1.3.2). With Thomas Young he developed the theory of trichromacy, that the eye has three classes of receptors sensitive to different wavelengths, and he invented the ophthalmoscope for viewing the retina, at 29 by Torralba et al.’s account (§1.3.2). His general rule for vision, as Torralba et al. quote it from his Treatise on Physiological Optics, is that we perceive whatever objects would have had to be present to produce the impression made on the eye. The mind makes perceptions out of sensations by finding the representations most likely to explain the sensory input; Helmholtz called these psychic acts unconscious conclusions, unlike the conscious conclusions of an astronomer computing the position of a star. Torralba et al. write that this unconscious inference is the topic of computer vision and that the viewpoint relates to Bayesian methods for vision (Bayesian inference). The color constancy article quotes his phrase, through Barron (2015), for the way a person “discounts the illuminant”.

Helmholtz also stressed the active observer, in a passage Torralba et al. quote. He argued that we learn that a table is a stationary form in space, the cause of the changing images in our eyes, because we can change those images at will by moving. Torralba et al. (§1.3.2) note that this foreshadows some of the unsupervised learning methods used in computer vision today.

Gestalt psychology and perceptual organization

Gestalt psychology began, in Torralba et al.’s account (§1.3.3), with Max Wertheimer’s 1912 paper on the perception of movement, which introduced the phi phenomenon: two bars flashed at different places in quick succession are seen as one bar moving continuously between them. Wertheimer, Kurt Koffka and Wolfgang Köhler held that the perceived motion is a new phenomenon, not present in either frame, and that perception is about wholes more than parts. This was a reaction against the view that a scientific psychology had to decompose stimuli into their elements.

Wertheimer posed the problem of perceptual organization: how the visual system groups features such as colours and lines into coherent objects. The grouping cues the Gestalt psychologists found, which they called laws, include proximity, similarity, closure, good continuation, parallelism and symmetry, common fate (elements that move together group together) and past experience (§1.3.3). Torralba et al. caution that these are not strictly laws and are not all equally strong; Gestalt theory did not say quantitatively how conflicting cues balance, and Stephen Palmer later measured their strengths and added new ones. In vision papers the cues reappear directly: Shi and Malik open their normalized cuts paper with Wertheimer’s factors of grouping, as image segmentation reports, and motion segmentation rests on common fate. Torralba et al. write that the Gestalt principles have influenced much modern research in computer vision, in particular image segmentation.

The Gestalt psychologists also studied lightness, the perceived quantity influenced by both the perceived reflectance and the perceived illumination of a surface. Koffka’s ring, from his 1935 Principles of Gestalt Psychology, is a ring of uniform grey whose two halves look different once a white gap splits it in two; Torralba et al. conclude that lightness perception is not local and depends on context and grouping (§1.3.3). Amodal completion, seeing a whole object when only part of it is visible, and modal completion, which produces illusory contours such as the sides of the Kanizsa triangle, are further cases of organization; Torralba et al. note that we use amodal completion all the time, since most objects we see are partly occluded.

Gibson: the ecological approach

In Torralba et al.’s account, James J. Gibson argued that vision should be studied ecologically: the eye in the context of the body that carries it and the world it lives in, with experiments that resemble real life; Torralba et al. (§1.3.4) add that many Gestalt displays were too simple and unlikely to occur in the real world. He described the environment in terms relevant to an observer, a medium (air or water), substances and the surfaces that separate them, with the ground plane as the most important surface for a terrestrial animal. His ecological optics studies not the light of physics textbooks, from point sources interacting once with a surface or lens, but the light converging on a point of observation after countless interactions in a cluttered environment: the ambient optic array. When an observer moves, the changing array carries information about the environment and about the observer, as the flow pattern seen by a bird flying over the ground does (§1.3.4). In our reading, optical flow is the computational counterpart of that flow pattern.

Gibson’s affordances are, in Torralba et al.’s examples, what the environment offers an animal: a path affords locomotion, an obstacle collision, an object with a sharp edge cutting; the vision papers that affordance detection cites paraphrase them as action possibilities. The sources date the concept differently. Torralba et al. (§1.3.4) write that Gibson introduced it in 1966, citing The Senses Considered as Perceptual Systems; the affordance detection article reports that vision papers cite either his 1979 book, The Ecological Approach to Visual Perception, or his 1977 paper “The Theory of Affordances”. Gibson also held, in their account, that the optic array is rich enough to specify the environment, so that perception is the direct pickup of information from it, against the more common view that the input is impoverished and perception must complete it. Torralba et al. draw a design lesson from this for perceptual systems: the richer the input, the simpler the solution one may hope for. Marr, they add, thought Gibson was asking the right questions but had a naive view of how hard they would be to answer (§1.3.6).

From theories to programs

Two further strands lead from these theories to computer vision. Neuroscience, from the late nineteenth century on, mapped the circuits of the retina and the visual cortex and measured what single cells respond to; the human visual system article tells that story, and receptive field traces the term into neural networks. And Marr’s computational theory, which Torralba et al. (§1.3.6) describe as influenced by work in computer vision, cognitive psychology and neurophysiology, tied the questions, in their account, to information processing; computer vision describes his levels of analysis and his sequence of representations. Among the computer vision work that inspired Marr, Torralba et al. name David Waltz’s interpretation of blocks-world drawings with shadows, Land and McCann’s Retinex theory of colour perception and Horn’s theory of shape from shading.

How the History Is Divided

The textbooks and the site’s history sections divide the past at different points. Each division below is given as the source states it; the table compares divisions, not events.

Source	Division
Torralba, Isola & Freeman (2024, §1.3.7)	1960–1990, dominated by the geometric perspective; 1990–2000, an acceleration, in part because digital cameras became common, with learning-based detection gaining attention; 2000–2010, image classification, recognition and medium-size benchmarks; 2010–2020, dominated by deep learning
Torralba, Isola & Freeman (2024, §1.3.8, Figure 1.16)	Enthusiasm for neural networks rising and falling with a period of roughly 25 years, partly but not always in step with interest in learning in general
Szeliski (2022, §1.2), as computer vision reports	Computer vision first started out in the early 1970s, as the perception component of the artificial intelligence agenda
Goodfellow et al. (2016, §1.2.1), as deep learning reports	Three waves of neural network research: cybernetics (1940s–1960s), connectionism (1980s–1990s) and deep learning (from 2006)
Gonzalez & Woods (2018, §1.2), as image processing reports	Digital pictures from the early 1920s, not processed by computer; computer image processing from the early 1960s, growing with the space program

The divisions answer different questions. The two computer vision textbooks disagree about when the field began, and the computer vision article discusses that disagreement; in its account, whether the 1960s work at MIT is the start of the field or its prehistory is a matter of definition. The deep learning waves date one family of methods, and Torralba et al.’s 25-year cycle describes the interest in that family rather than the field as a whole; the 1960–1990 “geometric” period of one account overlaps the connectionist wave of the other. Image processing, by Gonzalez and Woods’s account as the image processing article reports it, began decades before computers could interpret images.

Timeline

Each row gives the year as the named source prints it, and the article that owns the event, where the full story and the primary references are. “Torralba et al.” is Torralba, Isola and Freeman (2024). Where sources give different years, the row gives both.

Year	Event	Date from	Owner
c. 494–434 BC (Empedocles’ life)	Extramission: rays from the eye sense the world	Torralba et al. §1.3.1	this article
425 BC	Early intromission theories: objects send copies of themselves into the eye	Torralba et al. §1.3.1	this article
325 BC	Euclid’s Optics: straight rays forming a cone from the eye	Torralba et al. §1.3.1	this article; pinhole camera model
1028–1038	Ibn al-Haytham’s Book of Optics: light from objects, the pinhole camera, two eyes for depth	Torralba et al. §1.3.1	this article
1435	Alberti’s rules of perspective drawing, which Criminisi, Reid and Zisserman’s single-view metrology paper frames itself as reversing	Vanishing points	vanishing points
1604	Kepler: the reversed retinal image, focused by the lens	Torralba et al. §1.3.1	this article
1639; 1822	Desargues treats parallel lines as meeting at infinity; Poncelet’s treatise on projective properties	Projective geometry	projective geometry
1821–1894 (Helmholtz’s life)	Trichromacy with Young; perception as unconscious inference	Torralba et al. §1.3.2	this article; colorimetry
1912	Wertheimer’s phi phenomenon; the start of Gestalt psychology	Torralba et al. §1.3.3	this article; image segmentation
Early 1920s	Bartlane cable pictures, digital images not processed by computer	Image processing	image processing
1935	Koffka’s Principles of Gestalt Psychology	Torralba et al. §1.3.3 and reference list	this article
1938	Hartline records single optic nerve fibres and popularizes the receptive field	Torralba et al. §1.3.5	human visual system
1953	Kuffler: centre–surround receptive fields of ganglion cells	Torralba et al. §1.3.5	human visual system
1957; 1958	Rosenblatt’s perceptron, in a January 1957 report and a 1958 paper (Torralba et al. give 1958)	Perceptron; Torralba et al. §1.3.8	perceptron
1957–59	Brown’s bundle method for aerial mapping	Bundle adjustment	bundle adjustment
1959	Hubel and Wiesel record cells in the cat’s striate cortex	Torralba et al. §1.3.5; receptive field	human visual system; receptive field
1963	Roberts’s MIT thesis on machine perception of three-dimensional solids: the blocks world	Computer vision; Torralba et al. §1.3.7	computer vision
1964	JPL corrects images of the Moon from Ranger 7 by computer	Image processing	image processing
1966	Papert’s Summer Vision Project memo	Computer vision	computer vision
1966; 1977; 1979	Gibson’s affordances: 1966 in Torralba et al.; the 1977 paper or the 1979 book in the vision papers Affordance Detection cites	Torralba et al. §1.3.4; affordance detection	this article; affordance detection
1969	Minsky and Papert’s Perceptrons	Torralba et al. §1.3.8; deep learning	perceptron; deep learning
1970; 1975	Horn’s shape from shading	Shape from shading	shape from shading
1971	Binford’s generalized cylinders	Torralba et al. §1.3.7 and reference list	computer vision
1973	Fischler and Elschlager’s pictorial structures	Pictorial structures	pictorial structures
1980 (1979; 1982)	Fukushima’s neocognitron: 1980 in Torralba et al.’s reference list and in Goodfellow et al.; 1979 in Schmidhuber and 1982 in LeCun et al. (2015), as Deep Learning reports	Deep learning; Torralba et al. reference list	deep learning
1981	Horn–Schunck and Lucas–Kanade optical flow; Longuet-Higgins’s eight-point algorithm	Horn–Schunck, Lucas–Kanade, Longuet-Higgins	the three articles
1982	Marr’s Vision, published posthumously	Computer vision; Torralba et al. §1.3.6	computer vision
1983	Burt and Adelson’s pyramid; Witkin names scale-space	Image pyramid; scale space	the two articles
1984	Geman and Geman pose restoration with Markov random fields	Computer vision; Markov random field	Markov random field
1986	Rumelhart, Hinton and Williams popularize backpropagation; Rumelhart and McClelland’s Parallel Distributed Processing	Backpropagation; Torralba et al. reference list	backpropagation
1986	Canny’s edge detector	Canny edge detector	Canny edge detector
1988	Kass, Witkin and Terzopoulos’s snakes; Harris and Stephens’s corner detector	Active contours; Harris corner detector	the two articles
1989	LeCun et al. train a convolutional network on handwritten zip-code digits	LeNet; Torralba et al. §1.3.8	LeNet
1991	Turk and Pentland’s eigenfaces	Eigenfaces	eigenfaces
1999; 2004	Lowe’s SIFT, in full in 2004	SIFT	SIFT
2000	Shi and Malik’s normalized cuts	Normalized cuts	normalized cuts
2001	Viola–Jones boosted face detector; Boykov, Veksler and Zabih’s graph-cut moves for multi-label energies	Viola–Jones; graph cuts	the two articles
2002	Comaniciu and Meer make mean shift a general vision tool	Mean shift	mean shift
2003; 2004	Caltech-101 gathered (September 2003) and introduced (CVPR 2004 workshop)	Caltech-101	Caltech-101
2005	Dalal and Triggs’s histogram of oriented gradients	HOG	HOG
2005–2012	The PASCAL VOC challenge	PASCAL VOC	PASCAL VOC
2009	ImageNet presented at CVPR 2009; the CIFAR report (April 2009)	ImageNet; CIFAR	the two articles
2012	An ensemble of convolutional networks wins the ILSVRC-2012 classification task with 15.3% top-5 test error, against 26.2% for the second-best entry	Computer vision; AlexNet	AlexNet
2013; 2014	R-CNN (arXiv, November 2013; CVPR 2014)	R-CNN	R-CNN
2014	COCO (ECCV 2014); generative adversarial networks	COCO; GAN	the two articles
2015; 2016	Fully convolutional networks (CVPR 2015); ResNet (December 2015 report, CVPR 2016)	FCN; ResNet	the two articles
2017	The transformer	Transformer	transformer
2020; 2021	Vision Transformer (October 2020 preprint, ICLR 2021)	Vision Transformer	Vision Transformer
2021	CLIP (February 2021 preprint, ICML 2021); the term “foundation model”	CLIP; foundation models	the two articles
2023	SAM (April 2023; ICCV 2023)	SAM	SAM

The 15.3% of 2012 came from an ensemble that included networks pretrained on a larger ImageNet release; the deep learning article gives the breakdown and the 16.4% of the networks trained on the challenge data alone. The neocognitron’s dates differ because the accounts cite different publications.

Recurring Themes

Hand design against learning. Torralba et al. (§1.3.8) describe three ways to obtain a vision algorithm: write it directly, step by step; derive it as the rational thing to do under a model of the world; or collect examples of inputs and correct outputs and fit a function to them. They place learning in the philosophical tradition of empiricism, knowledge acquired by experience, against rationalism, knowledge acquired by reason, and write that the modern word for experience is data. They also argue that learning-based vision does not change the structure of the problem, the input, the output and the mapping between them, but only how the mapping is found: learners play the role that human vision scientists played in earlier eras. Their judgement on Marr points the same way: his shortcoming may have been underestimating the role of learning in building representations (§1.3.6).

Learning was there from the start. In Torralba et al.’s account (§1.3.8), learning algorithms have been part of computer vision since its origins: Rosenblatt’s perceptron of 1958 could in principle learn simple visual patterns. What changed was the share of the pipeline that was learned. In the 2000s, they write, a common pattern fed the best hand-engineered features to a shallow learner such as a support vector machine, and more and more of the pipeline was gradually replaced by learned modules until AlexNet learned it end to end (§1.3.8).

Cycles of enthusiasm. Interest in neural networks has risen and fallen, with a period of roughly 25 years in Torralba et al.’s Figure 1.16. In their account Minsky and Papert’s Perceptrons (1969) largely killed the excitement around perceptrons; interest surged again in the 1980s, when stacked perceptrons were shown to perform complex operations and Rumelhart and McClelland’s Parallel Distributed Processing appeared; it waned in the 1990s; and by 2000 the field of AI as a whole was in a “winter” (§1.3.8). The deep learning article gives Goodfellow et al.’s dating of the same ebbs and flows, and Schmidhuber’s earlier dates for several milestones.

Compute and data. Torralba et al. (§1.3.8) explain why the networks of the 1980s did not take over earlier: the compute and data needed to make them work did not yet exist. In their account of 2000–2010 the data arrived: medium-size benchmarks with tens of thousands of images, then the first very large databases with millions, built on the hypothesis that most vision tasks need to be learned (§1.3.7).

Benchmarks as drivers. The computer vision article names the benchmarks that, in Torralba et al.’s account (§1.3.7), began to drive progress in the 2000s; Torralba et al. add that the benchmarks popularized new metrics and guidelines for reproducible research. The benchmark overfitting and dataset bias articles describe the costs of that arrangement.

Geometry first, then statistics. Torralba et al. (§1.3.7) characterize 1960–1990 by geometric concepts (stereo, shape from shading, multi-view geometry, motion estimation, model-driven recognition) that turned old hypotheses into algorithms, and the 1990s by a move away from rigid model-driven approaches toward ecological image properties, perceptual properties such as segmentation and texture, and learned detection. In our reading this is the empiricist turn of the previous theme seen from the side of models: the knowledge built into a method moved from explicit geometry and physics toward statistics estimated from images, while geometry stayed in multi-view geometry and bundle adjustment.

Perception as inference. Helmholtz’s unconscious inference, the choice of the scene that best explains the image, is the idea Torralba et al. (§1.3.2) connect to Bayesian methods in vision. In our reading it runs through the field’s formulations with priors and likelihoods: Geman and Geman’s Markov random fields of 1984, the regularized formulations of the 1980s that computer vision describes, and the view of vision as an inverse problem.

Optimism. The computer vision article recounts how Torralba et al. (§2.1) read Papert’s 1966 summer proposal as a sign of early optimism; their own chapter 2 begins by embracing that optimism and hand-designing a complete system for a simplified blocks world (§2.1–§2.2). Their remark that Marr found Gibson naive about the difficulty of the questions he asked (§1.3.6) is, in our reading, the same lesson one level up. And the separation of what is computed from how, which Torralba et al. credit to Marr together with Poggio and others, is in their view often ignored by current practice, where the task is tied to an architecture without a clear understanding of what is being computed (§1.3.6).

Common Misconceptions

• “Computer vision started with the Summer Vision Project.” Roberts’s thesis is three years older, and the textbooks disagree about where the field starts; the computer vision article sets out both views. The questions the field asks are older still: Torralba et al. (§1.4) describe a span of more than 2,000 years from the Greeks’ extramission theories to the engineering of computer vision systems.
• “Learning arrived with deep learning in 2012.” Learning algorithms have been part of computer vision since its origins, and the boosted face detector of 2001 became popular in consumer cameras (Torralba et al., §1.3.8; see Viola–Jones). What 2012 changed was end-to-end learning from pixels to predictions.
• “Euclid’s optics was wrong because the rays went the wrong way.” Ibn al-Haytham showed that rays from the eye are unnecessary, but Euclid’s straight-line geometry is what Torralba et al. (§1.3.1) credit as the foundation of image formation and perspective, and it is unchanged when the rays are reversed (our reasoning).
• “The Gestalt laws are laws.” Torralba et al. (§1.3.3) write that they are not strictly laws, that the cues differ in strength, and that Gestalt theory did not say how they combine when they conflict; that was measured later.
• “Interest in neural networks grew steadily until it won.” It rose and fell more than once, and some periods of low interest in neural networks were periods of high interest in other learning algorithms (Torralba et al., §1.3.8).

Where It Is Used

The timeline points to the articles that own each period’s history:

• The field, decade by decade: computer vision, including the blocks world, the Summer Vision Project and the disagreement over the start.
• Neural networks and deep learning: deep learning, neural network, perceptron, backpropagation, LeNet and AlexNet.
• Image processing: image processing, from the Bartlane cable to learned restoration.
• Features: image features, from Moravec’s interest operator to learned features.
• Biological vision: human visual system and receptive field.
• Geometry: projective geometry, photogrammetry, bundle adjustment and structure from motion.
• Datasets and benchmarks: the Historical Milestones sections of ImageNet, PASCAL VOC, COCO, Caltech-101 and CIFAR.
• Foundation models: foundation models.