Deep Learning (2015 paper)

“Deep learning” is a nine-page review by Yann LeCun, Yoshua Bengio and Geoffrey Hinton, published in Nature on 27 May 2015 (volume 521, pages 436–444, in the print issue of 28 May). It explains to a general scientific audience what deep learning is, how it is trained, why convolutional and recurrent networks had succeeded, and where the authors expected the field to go. It reports no new experiments. Its value lies in its definition of the subject, its account of how the field reached 2015, and its forecasts, made by three researchers who had done much of the work they describe. The review is widely cited, and in 2019 its three authors received the 2018 ACM A.M. Turing Award for their work on deep neural networks. This article covers the review itself: its argument section by section, its figures, the claims it made as of 2015 and how they have held up, the errors in the printed text, and its reception. The subject, from its history to its architecture families, is explained in deep learning.

The review was read in the publisher’s PDF hosted on Hinton’s University of Toronto page and compared with the open-access copy that HAL holds; page numbers below are Nature’s.

Context

The review appeared in a Nature “Insight” on machine intelligence; its pages carry the running head “Review Insight”. Crossref’s records for the issue list a one-page piece titled “Machine intelligence” by Chouard and Venema (p. 435), followed on consecutive pages by this review and five other articles: on reinforcement learning (Littman), probabilistic machine learning (Ghahramani), small autonomous drones (Floreano and Wood), soft robots (Rus and Tolley) and evolutionary computation (Eiben and Smith). The printed dates are “Received 25 February; accepted 1 May 2015”. As printed, LeCun was at Facebook AI Research and New York University, Bengio at the Université de Montréal, and Hinton at Google and the University of Toronto.

By early 2015 the evidence the review draws on was recent. Krizhevsky, Sutskever and Hinton’s 2012 ImageNet result (the AlexNet paper) had turned computer vision toward convolutional networks; by the review’s account, deep networks had set records in speech recognition from 2009 and were deployed in Android phones by 2012 (p. 439), and recurrent encoder–decoder networks had quickly become competitive in machine translation (p. 441). The review sets out to explain why these results came from one family of methods, and it tells the story largely through its authors’ own work, a point its critics made at once (see Reception).

The Argument

The review’s thesis is stated in its standfirst and introduction (p. 436). Conventional machine learning needed a hand-designed feature extractor to turn raw data, such as the pixel values of an image, into a representation a classifier could use. Representation learning lets a machine discover the needed representations from raw data, and deep-learning methods are representation-learning methods with multiple levels of representation, built by composing simple non-linear modules that each transform one level into a slightly more abstract one. For images, the review describes the typical hierarchy: edges in the first layer, motifs formed by arrangements of edges in the second, parts of objects in the third, and objects in later layers. It names the key aspect as the fact that these features are learned from data with a general-purpose procedure instead of being designed by engineers.

From this the review draws its central prediction, also on p. 436: because deep learning needs very little engineering by hand, it can take advantage of increases in computation and data, so the authors expected many more successes in the near future. The rest of the review supports these claims in three ways: it explains the machinery (supervised learning, backpropagation, convolutional and recurrent networks), it tells a history in which the machinery was neglected and then vindicated, and it surveys applications from image recognition to translation.

Its references carry part of the argument. Of the 103 references, ten carry a one-sentence annotation saying why the work mattered: for example, ref. 1 (Krizhevsky et al., 2012) is described as a breakthrough that precipitated the rapid adoption of deep learning by the computer vision community, and ref. 32 (Hinton, Osindero & Teh, 2006) as introducing layer-by-layer unsupervised pretraining.

Structure and Content

The review has no numbered sections. After the untitled introduction, it has seven headed sections.

Supervised learning (pp. 436–438). A system that classifies images of houses, cars, people and pets produces a vector of scores, an objective function measures their error, and training adjusts the weights against the gradient, in practice by stochastic gradient descent on small sets of examples. The review notes that a typical deep-learning system may have hundreds of millions of weights and hundreds of millions of labelled examples. It then poses the problem that motivates depth: a linear classifier divides its input space into half-spaces, but recognition must be insensitive to pose, lighting and background while sensitive to small differences, such as those between a Samoyed and a white wolf. The review calls this the selectivity–invariance dilemma (p. 438). Generic kernels do not generalize far from the training examples, and hand-designed features need engineering skill; a stack of learned non-linear modules, “say a depth of 5 to 20”, can be both selective and invariant.

Backpropagation to train multilayer architectures (pp. 438–439). Backpropagation is presented as a practical application of the chain rule, discovered independently by several groups in the 1970s and 1980s (refs 24–27). The section notes that the rectified linear unit was then the most popular non-linearity and lets deep supervised networks train without unsupervised pretraining (ref. 28). It then gives the history the rest of the review relies on: in the late 1990s neural networks were “largely forsaken” by machine learning and ignored by computer vision and speech recognition, partly from a fear that gradient descent would get stuck in poor local minima. The authors answer that fear with recent theoretical and empirical results (refs 29, 30): in large networks the loss landscape is dominated by saddle points whose objective values are mostly similar. They credit a group of researchers brought together by the Canadian Institute for Advanced Research (CIFAR) with reviving interest around 2006 through unsupervised layer-wise pretraining, followed by fine-tuning with backpropagation; they name speech recognition, made possible by programmable GPUs that trained networks 10 or 20 times faster, as the approach’s first major application, deployed in Android phones by 2012; and they state that once deep learning had been rehabilitated, pretraining turned out to be needed only for small datasets (p. 439).

Convolutional neural networks (p. 439). A convolutional network rests on four ideas: local connections, shared weights, pooling and many layers. Shared filters exploit two properties of images, that local groups of values are correlated and that local statistics do not depend on location; pooling merges semantically similar features and gives invariance to small shifts. The section ties the architecture to visual neuroscience (simple and complex cells, the ventral pathway) and cites a 2014 study in which high-level units of a convolutional network explained half the variance of random sets of 160 neurons in monkey inferotemporal cortex (ref. 45). It traces the roots to Fukushima’s neocognitron, which had a similar architecture but no end-to-end supervised learning, and lists applications from the early 1990s, including a document-reading system that by the late 1990s was reading over 10% of all cheques in the United States.

Image understanding with deep convolutional networks (pp. 439–440). This is the section most directly about computer vision. It lists successes since the early 2000s (traffic signs, biological image segmentation, faces, text, pedestrians and bodies, face recognition) and pixel-level labelling for robots and self-driving cars, then states that convolutional networks were largely forsaken by mainstream computer vision and machine learning until the ImageNet competition of 2012. There, the review says, they almost halved the error rates of the best competing approaches, a success it attributes to the efficient use of GPUs, ReLUs, dropout and data augmentation, and which it calls a revolution in computer vision. Convolutional networks, it continues, were by 2015 the dominant approach for almost all recognition and detection tasks and approached human performance on some; recent architectures had 10 to 20 layers of ReLUs and hundreds of millions of weights, and training times had fallen from weeks to a few hours in two years. The section names companies deploying the technology and building chips for it.

Distributed representations and language processing (pp. 440–441). Deep networks are said to have two exponential advantages over methods without distributed representations: features that combine freely ($2^n$ combinations of $n$ binary features) and depth itself. The worked example is the neural language model (ref. 71, Bengio et al., 2001), whose learned word vectors place Tuesday near Wednesday and Sweden near Norway. N-gram models, by contrast, count sequences whose number grows as $V^N$ with vocabulary size $V$ and treat each word as an atom. The section uses this to set the neural paradigm, fast inference with large activity vectors and weight matrices, against logic-inspired manipulation of symbols by rules of inference.

Recurrent neural networks (pp. 441–442). A recurrent network keeps a state vector and, unfolded in time, is a deep network that shares its weights across steps, so backpropagation applies, but its gradients explode or vanish over many steps (refs 77, 78; see vanishing gradient problem). The section describes encoder–decoder translation, in which an English encoder’s final state, a “thought vector”, initializes a French decoder; image captioning with a convolutional encoder and a recurrent decoder (image captioning); the long short-term memory (LSTM), whose memory cell has a self-connection of weight one gated by another unit; and the memory-augmented Neural Turing Machines and memory networks of the previous year.

The future of deep learning (p. 442). Four forecasts close the review. Unsupervised learning, catalytic in the 2006 revival but since overshadowed by supervised learning, would become “far more important in the longer term”, because human and animal learning is largely unsupervised; the authors add that the review did not focus on it. Vision would progress through systems trained end to end that combine convolutional networks with recurrent networks using reinforcement learning to decide where to look, as human vision samples a scene with a moving fovea. Language understanding would improve when recurrent networks learn to attend selectively to one part of a sentence or document at a time. And major progress in AI would come from combining representation learning with complex reasoning, which needs new paradigms that replace rule-based manipulation of symbols by operations on large vectors.

Figures

The review has five figures.

• Fig. 1 (p. 437), multilayer networks and backpropagation. Panel a, reproduced with permission from C. Olah, shows a network with two inputs, two sigmoid hidden units and one sigmoid output bending a regular grid so that two curved classes become linearly separable. Panel b explains the chain rule, and panels c and d give the forward and backward equations for a network with two hidden layers, with the output error derivative $y_l - t_l$ for the squared error $\tfrac{1}{2}(y_l - t_l)^2$.
• Fig. 2 (p. 438), inside a convolutional network. The feature maps (outputs, not filters) of each layer for a photograph of a Samoyed, from the red, green and blue input planes through convolution, ReLU and max-pooling stages to class scores. The printed scores rank Samoyed (16) first, then Papillon (5.7) and Pomeranian (2.7), with white wolf at 0.4.
• Fig. 3 (p. 440), from image to text. A convolutional network feeding a recurrent network that writes captions (top, from ref. 102), and captions whose words are generated while attending to different image regions (middle and bottom, from ref. 86, Xu et al., of which Bengio is an author).
• Fig. 4 (p. 441), learned word vectors. A t-SNE projection of learned word representations (left) and a two-dimensional map of phrase representations learned by an English-to-French encoder–decoder (right), showing semantically similar words and phrases close together.
• Fig. 5 (p. 442), a recurrent network unfolded in time, with the same matrices $U$, $V$ and $W$ used at every step.

Fig. 2 and Fig. 3 are the review’s computer-vision figures: one shows the representation hierarchy the introduction describes, and the other shows the pairing of a convolutional encoder with a language decoder that the review presents as a recent, striking demonstration.

Claims as of 2015

The review reports no results table of its own. The claims it makes, and on which its forecasts rest, are summarized below with their pages; the later evidence is discussed in the next section.

Claim (page)	As stated in 2015	Later evidence
Depth (p. 438)	Intricate functions with “say a depth of 5 to 20” non-linear layers	Residual networks of up to 152 layers on ImageNet; an ensemble of them won the ILSVRC 2015 classification task (He et al., 2016)
Pretraining (p. 439)	Layer-wise unsupervised pretraining needed only for small datasets	Layer-wise pretraining fell out of use (our reading); pretraining a whole network and adapting it became the dominant approach to transfer learning (Bommasani et al., 2021, §1.1)
ConvNet dominance (p. 440)	Dominant for almost all recognition and detection tasks	Still dominant in 2020 by the account of Dosovitskiy et al. (2021), whose Vision Transformer then competed with them
Unsupervised learning (p. 442)	Overshadowed by supervised learning, expected to become far more important	Self-supervision at scale became the usual way to train foundation models (Bommasani et al., 2021, §1)
Vision (p. 442)	Progress from ConvNets plus recurrent networks that use reinforcement learning to decide where to look	Not the main line of development (our reading; see below)
Language (p. 442)	Recurrent networks will improve by learning to attend	The transformer was built on attention alone, without recurrence (Vaswani et al., 2017)

How the Forecasts Held Up

Unsupervised learning. The forecast held up, under a new name. Bommasani et al. (2021, §1) define foundation models as models trained on broad data, generally with self-supervision at scale, and note in a footnote to §1.1 that self-supervised learning was dominant in the early days of deep learning but was for a decade largely overtaken by pure supervised learning as labelled datasets grew. In our reading this is the arc the review describes, followed by the turn it predicted. In vision, He et al. (2022) reported that pretraining by reconstructing masked image patches, with no labels, transferred to downstream tasks better than supervised pretraining. The methods are not those the review cites (refs 91–98: Boltzmann machines, autoencoders, sparse coding); the self-supervised learning and masked autoencoder articles describe what replaced them.

Pretraining. The review’s statement that pretraining was needed only for small datasets referred to initializing a network layer by layer without labels. In our reading that procedure fell out of use. Pretraining in a different sense, training a whole network on a surrogate task and adapting it to the target task, became what Bommasani et al. (2021, §1.1) call the dominant approach to transfer learning within deep learning, first with labelled ImageNet and then with unlabelled data or image–text pairs, as transfer learning and fine-tuning describe. The review anticipates this use in one sentence, noting that pretraining helps in a transfer setting with many examples for source tasks and few for target tasks (p. 439).

Attention and recurrence. The language forecast was half right. Attention, for which the review cites Bahdanau et al. (ref. 76) and Xu et al. (ref. 86), became central; in our reading, recurrence did not survive it. Vaswani et al. (2017) proposed the transformer, based solely on attention and dispensing with recurrence and convolution, and Dosovitskiy et al. (2021) applied it to image patches in the Vision Transformer.

Active vision. The vision forecast, systems that learn by reinforcement where to look, was not the route the field took. In our reading of the developments the deep learning article traces, the main gains after 2015 came from deeper convolutional networks, transformers that process all image patches at once, and pretraining at scale, all of which see the whole image in one pass. Attention arrived in vision as a learned weighting over all positions, not as a sequence of glimpses chosen by a policy.

Depth and scale. The review’s “5 to 20” layers was overtaken within the year: He et al. (2016) evaluated residual networks of up to 152 layers on ImageNet (and explored over 1,000 layers on CIFAR-10), and an ensemble of them won the ILSVRC 2015 classification task with 3.57% top-5 error on the test set. Its broader claim, that deep learning would keep improving as computation and data grew, is the premise of the scaling studies that the subject article summarizes.

Reasoning. The final forecast, that new paradigms would replace rule-based symbol manipulation with operations on large vectors, is a statement about AI as a whole; this article does not assess it.

Reception and Influence

The review is widely cited as a reference for the field as a whole; Bommasani et al. (2021, §1.1), for example, cite it for the revival of deep neural networks under the name deep learning.

It was also contested at once. In June 2015, Jürgen Schmidhuber posted a critique on his IDSIA web page objecting that the review credits mainly its authors’ own groups and omits earlier pioneers. Among its nine points: Ivakhnenko’s multilayer networks trained from 1965 are not mentioned; the review cites Werbos’s 1974 thesis and later work on backpropagation but not its earlier inventors, from continuous forms of the early 1960s to Linnainmaa’s efficient version of 1970; the 2006 revival is credited to CIFAR-funded researchers alone; the claim that convolutional networks were forsaken until 2012 ignores GPU-trained max-pooling networks that had won contests before it, including the IJCNN 2011 traffic-sign competition; and the outlook’s reinforcement-learning attention had been proposed by Schmidhuber and Huber in 1991. Schmidhuber’s own survey (Schmidhuber, 2015) gives this alternative history, and the deep learning article sets the accounts side by side; this article does not adjudicate the priority claims.

In 2019 the ACM named Bengio, Hinton and LeCun recipients of the 2018 A.M. Turing Award “for conceptual and engineering breakthroughs” that made deep neural networks a critical component of computing. The award honours the three researchers’ work, not this review, which the ACM announcement does not mention.

Limitations

• A partial history. By its critics’ account, the review’s history is largely that of its authors’ groups. Readers who want the dates of the primary papers, and where the accounts disagree, should use the history in the deep learning article. One instance: the review’s ref. 46 dates the neocognitron to a 1982 paper by Fukushima and Miyake, while Schmidhuber (2015) cites Fukushima’s papers of 1979 and 1980.
• Little on failure modes. The review does not mention adversarial examples, though Szegedy et al. (2014) had shown that imperceptible, deliberately computed perturbations make networks misclassify images; nor does it discuss interpretability or dataset bias. (A text search of the review finds none of these terms; “bias” occurs once, for bias terms in Fig. 1.)
• Selective by design. By the authors’ own statement, it does not focus on unsupervised learning, the topic it expects to matter most; and it treats reinforcement learning only in its outlook.
• Few quantitative statements. As a review for a general audience it gives directions and orders of magnitude (“almost halving the error rates”, “10 to 20 layers”), not results with datasets and metrics; the cited papers have to be read for those. The AlexNet figure, for example, is a top-5 error of 15.3% against 26.2% for the next entry in ILSVRC-2012 (Krizhevsky et al., 2012).

Errors in the Printed Text

No correction or erratum is registered for the review at Crossref (as of October 2026). The following errors occur in both the Nature PDF and the HAL copy:

• Fig. 1b caption (p. 437). The chain rule is described as multiplication by the product of $\partial y/\partial x$ and $\partial z/\partial x$; it should be $\partial z/\partial y$, as the figure’s own equations show. The same caption has “hyberbolic tangent” and the garbled “logistic function logistic”.
• Fig. 4 caption (p. 441). The phrase plot on the right is credited to ref. 75 (Mikolov et al., 2013, on word and phrase vectors), but it matches, in its axes and its phrases, a panel of Fig. 5 in Cho et al. (2014), the encoder–decoder paper that is ref. 72; the caption’s own description, an English-to-French encoder–decoder, is Cho et al.’s model. The caption also cites ref. 18, Bottou and Bousquet’s paper on the trade-offs of large-scale learning, for jointly learned word representations; in our reading it is a misnumbered citation.
• Ref. 1. The AlexNet paper is given pages 1090–1098; the proceedings PDF gives 1097–1105. The printed range is the same as that printed for ref. 94 (Kavukcuoglu et al., 2010).
• Ref. 86. Xu et al.’s “Show, attend and tell” is listed as an ICLR 2015 paper; it was published at ICML 2015 (Xu et al., 2015).
• Ref. 102. Vinyals et al.’s “Show and tell” is listed with the arXiv number of Xu et al. (1502.03044) and as an ICML paper; it is arXiv:1411.4555 and appeared at CVPR 2015 (Vinyals et al., 2015; see the Show and Tell paper).
• Ref. 91. The wake–sleep paper is given pages 1558–1161; they are 1158–1161 (Hinton, Dayan, Frey & Neal, 1995).
• Ref. 49. The authors’ initials are exchanged: the paper is by P. Y. Simard and D. Steinkraus (Simard, Steinkraus & Platt, 2003), printed as “Simard, D., Steinkraus, P. Y.”.
• Text (p. 442). “a 15-sentence version of the The Lord of the Rings”.

These are slips in captions, references and one sentence; none changes the review’s argument.

Versions

The publisher’s version of record is the Nature PDF, which Hinton hosts on his University of Toronto page; nature.com itself was not read for this article. HAL holds an open-access copy deposited in September 2023 (hal-04206682v1), which OpenAlex lists as a submitted version and as the review’s only open-access location. Its body text, captions and reference list match the Nature PDF; it lacks Nature’s running heads, the received and accepted dates, and the Author Information paragraph, and keeps the Acknowledgements.