Deep Learning

Deep learning is the part of machine learning that learns a task with a neural network of many layers, trained end to end from raw data, so that the features a classical vision system would design by hand are learned instead. For an image, the input is the array of pixels and every intermediate representation, from edge responses to object categories, is a set of numbers computed by layers whose weights were fitted to examples. LeCun, Bengio and Hinton (2015), in a widely cited Nature review, define deep-learning methods as representation-learning methods with multiple levels of representation, obtained by composing simple non-linear modules that each transform the representation at one level into a slightly more abstract one. This article is the overview of the subject for computer vision: what depth buys, how the field developed from the perceptron to foundation models, the main families of architectures, what training at scale involves, and where each topic is explained in more detail on this site.

Definition

A deep network computes its output by passing the input through a sequence of learned transformations, or layers, so that each layer’s output is the next layer’s input; deep learning fits all of the layers’ parameters jointly to data, almost always by gradient descent on a loss, with gradients computed by backpropagation. Three things distinguish it from earlier machine learning in vision:

• Learned features. The layers that turn pixels into a useful description are trained, not designed. LeCun et al. (2015) call this the key aspect: the layers of features are learned from data with a general-purpose learning procedure rather than designed by engineers.
• Composition. The function is a composition of many simple functions, which can represent intricate dependencies with fewer units than a shallow model would need (see Properties).
• End-to-end training. One loss at the output trains every layer at once, so the early layers learn whatever helps the final task.

How many layers make a network “deep” has no agreed answer. Schmidhuber (2015), in his historical survey, reports that discussions with experts had not settled where shallow learning ends and deep learning begins, and for his own purposes calls problems with credit-assignment paths longer than 10 links “very deep”. LeCun et al. (2015) speak of depths of 5 to 20 non-linear layers; later that year, ResNet’s winning ILSVRC 2015 entry used networks of up to 152 layers (He et al., 2016).

Intuition

An image classifier has to be selective and invariant at once. It must separate a Samoyed from a white wolf, which may differ in a few details, while giving the same answer for two Samoyeds in different poses, places and lighting, whose pixels differ almost everywhere. LeCun et al. (2015) call this the selectivity–invariance dilemma: a linear classifier on raw pixels cannot solve it, so classical systems relied on a hand-designed feature extractor to do so before the classifier.

A deep network solves the dilemma in stages. The review describes the typical result for images: the first layer of learned features responds to edges at particular orientations and positions, the second to motifs formed by arrangements of edges, the third to larger combinations that correspond to parts of objects, and later layers to objects as combinations of parts. Each stage gains a little invariance (to small shifts of the edges inside a motif, for instance) and a little selectivity, and no single stage has to solve the whole problem. The feature map article shows such learned features, and representation learning covers what makes a learned representation good and how representations are compared.

Learning the stages needs many labelled examples, because the features must be estimated rather than supplied, and much computation, because every example passes through every layer many times. Goodfellow, Bengio and Courville (2016, §1.2.2–1.2.3) give these two, larger datasets and the resources to run larger models, as the main reasons deep learning succeeded only decades after its first experiments, and the history below is largely the story of both requirements being met.

Formal Definition

A deep network with $L$ layers is a composition

$$f_\theta(\mathbf{x}) = f_L\big(f_{L-1}(\cdots f_1(\mathbf{x};\, \theta_1) \cdots;\, \theta_{L-1});\, \theta_L\big),$$

where $\mathbf{x}$ is the input image, $f_l$ is the $l$-th layer with parameters $\theta_l$, and $\theta = (\theta_1, \dots, \theta_L)$. Writing $\mathbf{h}_0 = \mathbf{x}$ and $\mathbf{h}_l = f_l(\mathbf{h}_{l-1}; \theta_l)$, the intermediate values $\mathbf{h}_l$ are the representations at each level. A layer is typically an affine map followed by an elementwise activation function, $f_l(\mathbf{h}) = \phi(W_l \mathbf{h} + \mathbf{b}_l)$, where $W_l$ may be a dense matrix or a convolution; attention layers depart from this form, because their mixing weights are computed from the input.

Training minimizes the average loss over a labelled set $\{(\mathbf{x}_i, y_i)\}_{i=1}^N$, usually with a regularizer $R$:

$$\min_\theta \; \frac{1}{N} \sum_{i=1}^{N} \ell\big(f_\theta(\mathbf{x}_i),\, y_i\big) + \lambda R(\theta).$$

Stochastic gradient descent replaces the full average by the average over a small random mini-batch $B$ and steps against its gradient,

$$\theta \leftarrow \theta - \eta \, \nabla_\theta \frac{1}{|B|} \sum_{i \in B} \ell\big(f_\theta(\mathbf{x}_i),\, y_i\big),$$

with learning rate $\eta$. Backpropagation computes the gradient with respect to every layer’s parameters in one backward pass by the chain rule,

$$\frac{\partial \ell}{\partial \mathbf{h}_{l-1}} = \left(\frac{\partial \mathbf{h}_l}{\partial \mathbf{h}_{l-1}}\right)^{\!\top} \frac{\partial \ell}{\partial \mathbf{h}_l},$$

at a cost comparable to one forward pass. The product of $L$ such Jacobians in this recursion is the source of the vanishing and exploding gradients that made deep networks hard to train (see Properties). The backpropagation and neural network training articles develop these steps.

Properties

Depth is efficient for compositional functions. Bengio, Lamblin, Popovici and Larochelle (2007) summarize results from circuit complexity: some functions that a deep circuit represents with a number of elements polynomial in the input size need exponentially many elements in a circuit of two layers. Their example is parity of $d$ inputs, which needs $O(2^d)$ examples and parameters for a Gaussian-kernel SVM, $O(d^2)$ parameters for a network with one hidden layer, and $O(d)$ for a network of $O(\log_2 d)$ layers. LeCun et al. (2015) state the same idea for natural signals: they are compositional hierarchies, in which edges form motifs, motifs form parts and parts form objects, and a deep network can exploit that structure. These are statements about what depth makes possible; they do not guarantee that training finds such a representation.

Distributed representations generalize to new combinations. A layer whose units can be active together encodes an input by a pattern over many features, and each feature takes part in many inputs. Goodfellow et al. (2016, §1.2.1) give a vision example: for cars, trucks and birds in three colours, three units for colour and three for object identity suffice, where a separate unit for each combination would need nine, and the unit for redness learns from red objects of every kind. LeCun et al. (2015) describe such representations as allowing generalization to combinations of feature values not seen in training.

Deep networks are hard to optimize, for reasons that are now mostly understood. Errors propagated backwards through many layers shrink or grow geometrically, the vanishing gradient problem, which Hochreiter identified in 1991 and Bengio, Simard and Frasconi analysed in 1994, by the accounts of Goodfellow et al. (2016, §1.2.1) and Schmidhuber (2015). Bengio et al. (2007) add that deep networks had generally been found empirically to be no better, and often worse, than networks with one or two hidden layers. The fixes are the subject of several articles: rectified linear activations, careful weight initialization, batch normalization and other normalization layers, and residual connections. Even with them, depth is not free: He et al. (2016) found that adding layers to an already deep plain network raised its training error, a degradation that was not overfitting, and residual connections were their answer.

Local minima turned out to matter less than feared. A long-standing worry was that gradient descent would be trapped in poor local minima. LeCun et al. (2015) argue from results of 2014 that this is rarely a problem for large networks: the loss surface is dominated by saddle points, and most of them have similar values of the objective. This is their reading of then-recent theory and experiment, not a guarantee for any particular network.

Performance grows with data, model size and compute together. Sun, Shrivastava, Singh and Gupta (2017) found that performance on vision tasks increased logarithmically with the amount of pretraining data, up to the 300 million images they used. Zhai, Kolesnikov, Houlsby and Beyer (2022) found that the error of vision transformers followed a power law in training compute over more than two orders of magnitude, saturating at both ends, and that small models could not benefit from more data or compute; big models were also more sample-efficient. The section on training at scale returns to these results.

History

Historians of the field divide its past differently. Goodfellow et al. (2016, §1.2.1) describe three waves, cybernetics in the 1940s–1960s, connectionism in the 1980s–1990s, and deep learning from 2006, and stress that the ideas date back to the 1940s and only the name is recent. Schmidhuber’s (2015) survey, which sets out to assign credit, traces deep networks to the multi-layer networks trained by Ivakhnenko’s Group Method of Data Handling from 1965 and gives earlier dates than other accounts for several milestones. The summary below follows the dates of the primary papers and notes where these accounts differ.

Learning machines, 1943–1969

McCulloch and Pitts (1943) modelled neurons as threshold units whose weights were set by hand. Rosenblatt’s perceptron, described in a 1957 Cornell report (the version LeCun et al. 2015 cite) and a 1958 paper, learned its weights from examples; Goodfellow et al. call it the first model that could. Perceptrons and the related ADALINE were linear models, and such models cannot compute even the XOR of two binary inputs; Goodfellow et al. date the first decline of neural network research to criticism of these limits, especially a 1969 book by Minsky and Papert. The neural network article gives the details of this period.

Backpropagation and connectionism, 1970–1995

Training networks with hidden layers needed a way to assign credit to hidden units, which backpropagation provides. Accounts of its origin differ. LeCun et al. (2015) write that it was discovered independently by several groups in the 1970s and 1980s, citing Werbos (1974), Parker (1985), LeCun (1985) and Rumelhart, Hinton and Williams (1986). Schmidhuber (2015) traces efficient reverse-mode differentiation to Linnainmaa’s 1970 master’s thesis, which did not refer to neural networks, and the first network-specific application to Werbos (1981). Both accounts agree that Rumelhart et al. (1986) popularized the method: their short Nature paper showed that hidden units trained this way come to represent important features of the task, which, they wrote, distinguishes back-propagation from earlier, simpler methods such as the perceptron-convergence procedure. The backpropagation article covers the credit question in full.

Convolutional networks grew out of visual neuroscience. LeCun et al. (2015) describe their convolutional and pooling layers as directly inspired by the simple and complex cells of Hubel and Wiesel and trace their roots to Fukushima’s neocognitron, which had a similar architecture but no end-to-end supervised training. Its date varies between accounts, which cite different publications: Schmidhuber gives 1979, citing a paper in the Transactions of the IECE alongside the 1980 one; Goodfellow et al. cite the 1980 Biological Cybernetics paper also cited here; and LeCun et al. (2015) cite a 1982 paper by Fukushima and Miyake in Pattern Recognition. LeCun and colleagues at Bell Labs then trained convolutional networks with backpropagation to read handwritten zip codes (LeCun et al., 1989), the line of work that led to LeNet-5 and a 1998 overview of gradient-based learning for document recognition (LeCun, Bottou, Bengio & Haffner, 1998). According to LeCun et al. (2015), a check-reading system built on such a network was reading over 10% of all cheques in the United States by the late 1990s.

The lull, mid-1990s to mid-2000s

By the late 1990s, LeCun et al. (2015) write, neural networks and backpropagation were largely forsaken by the machine-learning community and ignored by computer vision, and it was widely thought that learning multistage feature extractors with little prior knowledge was infeasible. Goodfellow et al. (2016) date the decline from the mid-1990s to 2007 and attribute it to two factors: ventures based on neural networks made claims they could not meet, and kernel machines and graphical models achieved good results on many tasks. Vision in this period was dominated by hand-designed image features with learned classifiers, as the machine learning article describes.

Work on networks continued. Hochreiter and Schmidhuber (1997) introduced the long short-term memory (LSTM), a recurrent unit whose gated memory cell holds information over long time spans, in answer to the vanishing gradients of recurrent networks. Goodfellow et al. credit the Canadian Institute for Advanced Research (CIFAR) and its Neural Computation and Adaptive Perception programme, which united the groups of Geoffrey Hinton, Yoshua Bengio and Yann LeCun, with keeping the research alive.

Unsupervised pretraining, 2006

The next wave began with a way to train deep networks one layer at a time. Hinton, Osindero and Teh (2006) introduced a fast, greedy algorithm that learns a deep belief network one layer at a time, each layer modelling the activity of the layer below without labels, and then fine-tunes the whole network; their three-hidden-layer model reached 1.25% test error on MNIST handwritten digits, which they reported as better than the best backpropagation networks and support vector machines given no built-in knowledge of image geometry. Bengio et al. (2007) found that the same layer-wise strategy worked with autoencoders and that it mostly helped optimization, by starting the weights near a good solution. LeCun et al. (2015) describe this CIFAR-led revival as working remarkably well when labelled data was limited.

The name dates from this period, though the accounts differ in what they claim. Goodfellow et al. say the wave popularized the term “deep learning” to stress that deeper networks could now be trained; Schmidhuber’s 2014 preprint says the expression was coined around 2006 and cites earlier phrases about deep networks. Pretraining itself proved temporary: according to LeCun et al. (2015), rectified linear units allowed deep supervised networks to be trained without it, and once deep learning had been rehabilitated, pretraining was needed only for small datasets.

Compute changed at the same time. According to LeCun et al. (2015), graphics processing units (GPUs) that were convenient to program let researchers train networks 10 or 20 times faster, which made possible the first major application of the pretraining approach, speech recognition, from 2009. Cireşan and colleagues (2011) described a fast GPU implementation of convolutional networks trained purely by supervised backpropagation and reported the best published results on MNIST, NORB and CIFAR-10. A committee of 25 such networks won the final phase of the German traffic sign recognition benchmark, held at the IJCNN 2011 conference, with a recognition rate of 99.46%, against 98.84% for humans on the task (Cireşan, Meier, Masci & Schmidhuber, 2012); Schmidhuber (2015) describes the result as twice as good as human test subjects.

ImageNet and the turn of 2012

The decisive result for computer vision was on ImageNet. Krizhevsky, Sutskever and Hinton (2012) trained a convolutional network with 60 million parameters, five convolutional and three fully connected layers, on the 1.2 million training images of the ImageNet challenge, using two GTX 580 GPUs for five to six days. In the ILSVRC-2012 classification task its entry reached a top-5 test error of 16.4% with five averaged networks trained on the challenge data and 15.3% with two more networks pretrained on the larger Fall 2011 ImageNet release (15 million images in 22,000 categories), against 26.2% for the second-best entry (Krizhevsky et al., 2012, Table 2; Russakovsky et al., 2015, Table 5 of arXiv v3). LeCun et al. (2015) attribute the success to the efficient use of GPUs, ReLUs, dropout and data augmentation; the AlexNet article describes the network.

The challenge organizers call 2012 a turning point for large-scale object recognition, and in 2013 the vast majority of entries used deep convolutional networks (Russakovsky et al., 2015). Accounts differ on how new the result was. Schmidhuber (2015) presents it as one in a series of contest wins by GPU-trained convolutional networks since 2011; LeCun et al. (2015) emphasize that convolutional networks had been largely forsaken by mainstream computer vision until 2012 and call the report a breakthrough. In our reading the two accounts are compatible: the method was not new, but ImageNet was the benchmark the vision community used, and the margin was large.

Deeper networks and transfer, 2013–2019

The years after 2012 explored depth and reuse. VGG stacked small $3 \times 3$ convolutions, Inception combined filters of several sizes in parallel branches, and ResNet used residual connections to train networks of up to 152 layers; an ensemble of residual networks won the ILSVRC 2015 classification task with 3.57% top-5 error (He et al., 2016). On a sample of 1,500 test images, the challenge organizers had estimated a trained human annotator’s top-5 error at 5.1% (Russakovsky et al., 2015).

Features learned on ImageNet also transferred. R-CNN brought them to object detection and fully convolutional networks (FCN) to semantic segmentation, and pretraining on ImageNet and then fine-tuning on the target task became the standard recipe (transfer learning). The same period exposed a weakness: Szegedy et al. (2014) showed that an imperceptible, deliberately computed perturbation could make a network misclassify an image, and that the same perturbation could also make a network trained on a different subset of the data misclassify the image.

Attention, transformers and scale, 2017–2021

The transformer (Vaswani et al., 2017) replaced recurrence and convolution with self-attention for machine translation. Dosovitskiy et al. (2021) applied it almost unchanged to sequences of image patches, $16 \times 16$ pixels in the paper’s title, giving the Vision Transformer (ViT). Trained on ImageNet alone without strong regularization it fell a few percentage points short of ResNets of comparable size, which its authors attributed to the lack of the convolution’s built-in assumptions, translation equivariance and locality; pretrained on the 14-million-image ImageNet-21k or the in-house JFT-300M, it approached or beat the state of the art on several image recognition benchmarks.

Scale also changed what supervision looked like. Self-supervised learning learned features from unlabelled images, and CLIP (Radford et al., 2021) learned them from 400 million image–text pairs collected from the internet, which let it classify by naming classes in text: zero-shot, it matched the accuracy of the original ResNet-50 on ImageNet without using any of ImageNet’s 1.28 million training examples.

Foundation models, 2021 onward

Bommasani et al. (2021) introduced the term foundation model for any model trained on broad data, generally with self-supervision at scale, that can be adapted to a wide range of downstream tasks; they note that such models are based on standard deep learning and transfer learning, and that what is new is their scale. In vision, examples include CLIP, the self-supervised DINO family of feature extractors, and SAM, a segmentation model trained to respond to prompts on a dataset of more than 1 billion masks on 11 million images (Kirillov et al., 2023). Vision-language models connect such image encoders to language models. Computer vision foundation models extend the pretraining and adaptation ideas of the earlier periods to more data and larger models.

The architectures did not settle. Liu et al. (2022) modernized a standard ResNet step by step toward a transformer’s design and obtained a pure convolutional network, ConvNeXt, that competed favourably with transformers in accuracy and scaling, reaching 87.8% ImageNet top-1 accuracy after ImageNet-22K pretraining. The lesson the period suggests, in our reading, is that data, compute and training recipe matter as much as the choice between convolution and attention.

Architecture Families

Deep networks for vision fall into a few families, distinguished by how they share parameters and what structure they assume. The table summarizes them; the paragraphs after it link the articles that explain each.

Family	Basic operation	Built-in assumption	Typical use in vision
Multilayer perceptron	Fully connected layer	None specific to images	Classification heads; small inputs
Convolutional network	Small filters shared across positions	Locality and translation equivariance	Classification, detection, segmentation backbones
Recurrent network	State updated step by step	Sequential order	Video, image captioning
Transformer	Self-attention over tokens or patches	Few; position must be encoded	Backbones at scale, vision–language models
Generative architectures	Encoders, decoders, adversarial or denoising training	Depends on the model	Image synthesis, editing, restoration

Multilayer perceptrons. The multilayer perceptron connects every unit to every input. It is the simplest deep network, and fully connected layers reappear inside the other families, as classification heads or as the per-token MLP of a transformer; on whole images it ignores spatial structure and its parameter count grows with image size.

Convolutional networks. A convolutional neural network applies small learned filters at every position, so parameters are shared and features are local; LeCun et al. (2015) name local connections, shared weights, pooling and many layers as the four key ideas. Its receptive field grows with depth, which is how later layers see whole objects. The site’s model articles follow the family’s development: LeNet, AlexNet, VGG, Inception, ResNet, the efficient MobileNet and EfficientNet, and components such as depthwise separable convolution and squeeze-and-excitation. For dense outputs, an encoder–decoder with transposed convolutions upsamples features back to image resolution, as in U-Net.

Recurrent networks. A recurrent network processes a sequence one element at a time while carrying a state vector, and unrolled in time it is a very deep network that shares its weights across steps (LeCun et al., 2015). In vision, recurrent layers, usually LSTMs, were combined with convolutional encoders for image captioning, where a network reads the image and a recurrent decoder writes the sentence, and for video, as the temporal modeling article describes. The transformer was proposed for sequence tasks in place of recurrence (Vaswani et al., 2017).

Transformers. A transformer alternates self-attention, which lets every token combine information from every other token with weights computed from the content, and a per-token MLP. Because attention is indifferent to order, position is added with a positional encoding. The Vision Transformer treats an image as a sequence of patches; with fewer built-in assumptions than a convolutional network, it needs more data or stronger regularization, as the ViT results above show.

Generative architectures. Generative models learn to produce images rather than labels. The autoencoder learns a code by reconstructing its input; the variational autoencoder and the generative adversarial network, both from 2014, trained deep image generators end to end, one through a variational bound and one through a game against a discriminator; and diffusion models generate by reversing a gradual noising process. The generative model article compares the families and traces their history.

Two further notions cut across the families. A backbone network is the pretrained feature extractor on which task-specific heads are built, whatever its family. And the families mix: detectors such as DETR put a transformer on a convolutional backbone, and Stable Diffusion runs a diffusion model inside an autoencoder’s latent space.

Training at Scale

Deep learning’s results depend as much on how networks are trained as on their architecture. Five ingredients recur.

Data

The first ingredient is labelled data in quantity. Goodfellow et al. (2016, §1.2.2) trace benchmark datasets from the tens of thousands of examples of MNIST in the 1990s to the hundreds of thousands to tens of millions of the early 2010s, ImageNet among them, and argue that the burden of generalizing from little data had been considerably lightened; as a rough rule of thumb in 2016, they write, supervised deep learning reached acceptable performance with about 5,000 labelled examples per category and matched or exceeded human performance with at least 10 million. Sun et al. (2017) studied what happens beyond ImageNet with JFT-300M, a set of 300 million images with more than 375 million noisy labels: performance on classification, detection, segmentation and pose estimation kept rising, logarithmically in the amount of data, and the scale of the data seemed to overpower the noise in the labels.

Labels are the expensive part, which is why later work turned to images without labels (self-supervised learning, contrastive learning, the masked autoencoder) and to images paired with text (multimodal learning). Larger data also brings its own problems: collections reflect how they were gathered (dataset bias), and years of comparison on one test set raise the question of benchmark overfitting.

Compute

The second ingredient is computation. Krizhevsky et al. (2012) wrote that their network’s size was limited mainly by GPU memory and by the training time they were willing to accept, and that their results could be improved by waiting for faster GPUs and bigger datasets. Goodfellow et al. (2016, §1.2.3) report that artificial networks had doubled in size roughly every 2.4 years since the introduction of hidden units, driven by faster computers with more memory and by larger datasets. Training is spread over many devices by splitting each mini-batch among them, which raises the batch size: Goyal et al. (2017) trained a ResNet-50 on ImageNet with mini-batches of 8,192 images on 256 GPUs in one hour, without loss of accuracy, by scaling the learning rate linearly with the batch size and warming it up over the first epochs.

Scale has structure. In Zhai et al. (2022), the best error achievable with a given compute budget fell as a power law over more than two orders of magnitude, but small models saturated early, so model size, data and compute had to grow together; their largest ViT, with two billion parameters, reached 90.45% ImageNet top-1 accuracy. Such studies treat compute as a budget to be allocated, not just a constraint.

Optimization

The third ingredient is making the optimization work. Almost all deep networks are trained with variants of stochastic gradient descent: plain SGD with momentum, or adaptive methods such as Adam, with a learning-rate schedule that usually decays the rate over training and often warms it up at the start. The loss is chosen to fit the output, most often the cross-entropy loss of a softmax for classification. Depth makes the problem fragile, and the remedies listed under Properties (initialization, normalization, residual connections, rectified activations) are what make networks of dozens or hundreds of layers trainable. The general theory is in optimization and gradient descent; the neural network training article covers the practical workflow, from data splits and learning curves to debugging.

Regularization

The fourth ingredient is keeping a network with millions of parameters from memorizing its training images. Regularization covers the methods and the evidence: weight decay, early stopping, dropout, and above all data augmentation, which multiplies the training set with transformed copies of its images. Very large networks complicate the classical picture of overfitting, a point the regularization article also covers.

Reuse

The fifth ingredient is not training from scratch. A network pretrained on a large dataset is adapted to a new task with less data and computation, by training a new head on frozen features (linear probing) or by fine-tuning the weights, all of them or a small added set as in parameter-efficient methods such as LoRA; transfer learning covers when this helps, and fine-tuning the choices it involves. Knowledge distillation trains a small network to imitate a large one, which serves deployment. In our reading, the foundation models of the history above take this practice furthest: one expensive pretraining, many cheaper adaptations.

Examples

• Image classification. The task that started the deep-learning era in vision. A convolutional network or ViT maps the image to class scores, trained with cross-entropy on labelled images; the image classification article covers the task and its benchmarks.
• Object detection and segmentation. A pretrained backbone feeds heads that predict boxes or per-pixel labels: two-stage detectors from R-CNN onward, one-stage detectors such as YOLO, and segmentation networks such as FCN and U-Net. See object detection and semantic segmentation.
• Dense geometric estimates. The same machinery regresses continuous quantities per pixel, such as depth in monocular depth estimation and motion in learned optical flow estimation with RAFT.
• Generation. Diffusion models and GANs synthesize, edit and restore images, as in text-to-image generation.
• Vision and language. CLIP-style encoders and vision-language models answer questions about images, write captions and ground phrases in image regions.

Common Misconceptions

• “Deep learning began in 2012.” The methods are decades older: backpropagation-trained convolutional networks read cheques in the 1990s, and GPU-trained networks won competitions in 2011. What changed in 2012 was the scale of data and compute and the visibility of the result on the vision community’s main benchmark.
• “Deep networks are models of the brain.” Convolutional networks borrowed ideas from visual neuroscience, but Goodfellow et al. (2016, §1.2.1) caution against viewing deep learning as an attempt to simulate the brain: it draws on many fields, especially applied mathematics, and real neurons compute very different functions from the units of modern networks.
• “More layers always help.” Without residual connections, adding layers to an already deep plain network raised its training error (He et al., 2016). Depth helps when the architecture and training make the extra layers trainable.
• “Transformers made convolutional networks obsolete.” A convolutional network modernized with the same training recipes competed with transformers at scale (Liu et al., 2022), and many current systems combine both.
• “Deep learning needs millions of labelled images for every task.” Pretraining on many images, labelled or not, or on image–text pairs, followed by transfer, can make far fewer labels sufficient for a new task, as the transfer learning and self-supervised learning articles show.
• “A high benchmark score means human-like perception.” Small, deliberately crafted perturbations can change a network’s decision (Szegedy et al., 2014), and accuracy measured on one dataset may not hold on images collected differently.

Where It Is Used

Deep learning underlies nearly every current method in the site’s other categories, from recognition and segmentation to 3D vision, video, generation and vision–language systems. Within the deep-learning category, the articles form a map:

• Networks and their parts. Neural network and multilayer perceptron; activation functions, softmax, convolution layers, pooling, feature maps, receptive fields, residual connections, normalization layers, self-attention and positional encoding.
• Architectures. Convolutional neural networks, transformers, encoder–decoders, autoencoders and backbone networks.
• Models. LeNet, AlexNet, VGG, Inception, ResNet, MobileNet, EfficientNet and the Vision Transformer, most with a paper article for the original publication.
• Training. Backpropagation, loss functions and the cross-entropy loss, stochastic gradient descent, Adam, learning-rate schedules, weight initialization, the vanishing gradient problem, batch normalization and the practical workflow of neural network training.
• Generalization and reuse. Regularization, dropout, data augmentation and transfer learning, with representation learning and self-supervised learning in the features category.

Across all of these, interpretability of neural networks studies what the learned features and decisions mean.