ImageNet Classification with Deep Convolutional Neural Networks

“ImageNet Classification with Deep Convolutional Neural Networks” is a paper by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton of the University of Toronto, published in Advances in Neural Information Processing Systems 25 (NIPS 2012). It trained a convolutional neural network with 60 million parameters on 1.2 million ImageNet images using two GPUs, and a variant of it, entered as team SuperVision, won the ILSVRC-2012 classification task with a top-5 test error of 15.3%, against 26.2% for the second-best entry. The network is known as AlexNet, after its first author. The paper received the NeurIPS 2022 Test of Time Award and was republished in Communications of the ACM in 2017. This article covers the publication; how convolutional networks work today is in the convolutional neural network article.

Problem

The authors judged datasets of tens of thousands of images, such as Caltech-101 and CIFAR-10, too small for objects in realistic settings. ImageNet offered over 15 million labeled images in about 22,000 categories, and the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) used a 1,000-class subset with about 1.2 million training images. Learning so many categories, the authors argued, needs a model of large capacity that also carries prior knowledge about images. CNNs offer both, but had been prohibitively expensive to apply at scale to high-resolution images.

The systems the paper compares against were not neural networks: the ILSVRC-2010 winner, which averaged six sparse-coding models trained on different features, a later system using Fisher vectors computed from two types of densely sampled features, and an ILSVRC-2012 runner-up that averaged classifiers on Fisher vectors. Among neural networks, Cireşan and colleagues had described a GPU implementation of CNNs in 2011 with record results on NORB, CIFAR-10 and MNIST; the paper compares its two-GPU layout to their “columnar” CNN.

Contribution

• Scale. One of the largest CNNs trained to date, with by far the best results reported on the ILSVRC-2010 and ILSVRC-2012 data.
• A GPU implementation of convolution and the other training operations, released publicly as cuda-convnet.
• Architectural features that improved accuracy or cut training time: rectified linear units, training across two GPUs, local response normalization, and overlapping pooling.
• Defenses against overfitting, a serious problem even with 1.2 million images: data augmentation and dropout.

Depth mattered, the authors reported: removing any convolutional layer, each holding at most 1% of the parameters, made results worse.

Method

Architecture. Images were rescaled and center-cropped to $256 \times 256$, and the only other preprocessing was subtracting the mean training image. The network has five convolutional and three fully connected layers, ending in a 1,000-way softmax trained with the multinomial logistic regression objective (the cross-entropy loss), 60 million parameters and 650,000 neurons in all. The convolutional layers have 96 kernels of $11 \times 11 \times 3$ at stride 4, then 256 of $5 \times 5 \times 48$, 384 of $3 \times 3 \times 256$, 384 of $3 \times 3 \times 192$ and 256 of $3 \times 3 \times 192$; the fully connected layers have 4,096, 4,096 and 1,000 units. ReLU follows every layer except the output, response normalization follows the first two convolutional layers, and max pooling follows those two and the fifth.

The paper gives the input as $224 \times 224 \times 3$ and mentions no padding; by our arithmetic an unpadded $11 \times 11$ kernel at stride 4 then does not fit a whole number of times, since $(224 - 11)/4 + 1 = 54.25$. Widely used reference implementations, the Caffe AlexNet model and the Stanford CS231n notes, take $227 \times 227$ inputs, which give a $55 \times 55$ first-layer map.

The paper presents four “unusual features” in order of estimated importance:

1. ReLU. Units computing $\max(0, x)$, named after Nair and Hinton (see activation function). A four-layer CNN with ReLUs reached 25% training error on CIFAR-10 six times faster than with tanh units; with saturating units, the authors wrote, they could not have experimented with networks this large.

2. Two GPUs. A GTX 580 has 3 GB of memory, so half of each layer’s kernels sit on each GPU. The third convolutional layer and the fully connected layers see all maps of the previous layer; the second, fourth and fifth convolutional layers see only those on their own GPU. Against a one-GPU network with half as many kernels per convolutional layer, top-1 and top-5 error fell by 1.7 and 1.2 points.

3. Local response normalization (LRN). Each post-ReLU activity $a^i_{x,y}$ of kernel $i$ at position $(x, y)$ is divided by a term built from the $n$ neighboring kernel maps at the same position:

   $$b^i_{x,y} = a^i_{x,y} \Big/ \left(k + \alpha \sum_{j=\max(0,\, i-n/2)}^{\min(N-1,\, i+n/2)} \left(a^j_{x,y}\right)^2\right)^{\beta},$$

   with $N$ kernels in the layer and $k = 2$, $n = 5$, $\alpha = 10^{-4}$, $\beta = 0.75$ set on validation data. The authors call it a form of lateral inhibition. It lowered top-1 and top-5 error by 1.4 and 1.2 points.

4. Overlapping pooling. $3 \times 3$ max pooling at stride 2 lowered top-1 and top-5 error by 0.4 and 0.3 points against $2 \times 2$ at stride 2 (see pooling).

Reducing overfitting. Training used random $224 \times 224$ crops of the $256 \times 256$ images and their horizontal reflections; without them, the authors reported, the network overfit substantially. Test predictions averaged ten patches (four corners and center, plus reflections). A second augmentation added to each image a random combination of the principal components of the training set’s RGB values, scaled by their eigenvalues and by Gaussian factors with standard deviation 0.1, to model changes in the intensity and color of illumination; it lowered top-1 error by over 1%. Dropout, recently introduced by Hinton et al. (2012), was applied with probability 0.5 in the first two fully connected layers, at the cost of roughly doubling the iterations to converge.

Training. Stochastic gradient descent with batches of 128, momentum 0.9 and weight decay 0.0005, which the authors found also reduced training error. The learning rate started at 0.01 and was divided by 10 by hand when validation error stopped improving, three times in all. About 90 passes over the training set took five to six days on two NVIDIA GTX 580 3GB GPUs.

Worked Example: Local Response Normalization

The sketch applies eight oriented derivative filters to a synthetic image of a bright square and computes the LRN divisor, with the paper’s constants, at a point on the square’s left edge.

import numpy as np
from scipy.signal import correlate2d

# Synthetic 32x32 grayscale image on the 0-255 scale: a bright square on a dark background.
img = np.full((32, 32), 40.0)
img[8:24, 8:24] = 200.0

# Eight 3x3 oriented derivative filters, one per 45 degrees: a stand-in for a conv layer.
angles = np.arange(8) * np.pi / 4
dx = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]) / 8.0
maps = np.stack([correlate2d(img, np.cos(t) * dx + np.sin(t) * dx.T, mode="valid")
                 for t in angles])
a = np.maximum(maps, 0)                      # ReLU, shape (8, 30, 30)

def lrn_divisor(a, k=2.0, n=5, alpha=1e-4, beta=0.75):
    """Divisor of local response normalization across channels (Krizhevsky et al., 2012, Sec. 3.3)."""
    N = a.shape[0]
    d = np.empty_like(a)
    for i in range(N):
        lo, hi = max(0, i - n // 2), min(N - 1, i + n // 2)
        d[i] = (k + alpha * (a[lo:hi + 1] ** 2).sum(axis=0)) ** beta
    return d

y, x = 12, 7                                 # a point on the square's left edge
d = lrn_divisor(a)
print("a         =", np.round(a[:, y, x], 1))
print("b = a / d =", np.round((a / d)[:, y, x], 1))
print("d         =", np.round(d[:, y, x], 2))
print("d, image scaled to [0, 1]:", np.round(lrn_divisor(a / 255.0)[:, y, x], 2))

a         = [80.  56.6  0.   0.   0.   0.   0.  56.6]
b = a / d = [35.5 25.1  0.   0.   0.   0.   0.  30.1]
d         = [2.26 2.26 2.26 1.88 1.68 1.88 1.88 1.88]
d, image scaled to [0, 1]: [1.68 1.68 1.68 1.68 1.68 1.68 1.68 1.68]

Channels 1 and 7 respond equally, but channel 1 neighbors the strongly responding channel 0 and is divided by 2.26, while channel 7’s neighbors are quiet (the neighborhood does not wrap around) and it is divided by 1.88: the competition between kernel maps the authors describe. The last line is our own observation: with these fixed constants, activations 255 times smaller make LRN a uniform division by $2^{0.75} \approx 1.68$.

Results

On the ILSVRC-2010 test set, the only edition with public test labels, the network reached 37.5% top-1 and 17.0% top-5 error, against 47.1% and 28.2% for the competition’s winner and 45.7% and 25.7% for the best result published since.

On ILSVRC-2012 the authors used validation and test error interchangeably, having seen them differ by at most 0.1%. Top-5 error was 18.2% for one CNN and 16.4% for an average of five. One CNN with a sixth convolutional layer, pretrained on the full ImageNet Fall 2011 release (15 million images, 22,000 categories) and fine-tuned, reached 16.6%, and averaging two such networks with the five gave the winning 15.3% on the test set. Russakovsky et al. (2015) add that SuperVision also won that year’s single-object localization task.

Qualitatively, the first-layer kernels were frequency- and orientation-selective filters and colored blobs; with the restricted connectivity, one GPU’s kernels were largely color-agnostic and the other’s color-specific. Nearest neighbors by Euclidean distance between last-hidden-layer activations were training images far from the test image in pixel space but showing the same kind of object, such as dogs and elephants in varied poses.

The abstract on the proceedings web page differs from the PDF’s, giving 39.7% and 18.9% error and 500,000 neurons; the numbers here are from the PDF.

Impact

Russakovsky et al. call ILSVRC-2012 a turning point for large-scale object recognition and report that the vast majority of ILSVRC-2013 entries used deep convolutional networks. LeCun, Bengio and Hinton (2015), Hinton being one of the paper’s authors, write in Nature that mainstream computer vision had largely forsaken convolutional networks until the 2012 competition, credit the result to GPUs, ReLUs, dropout and data augmentation, and call convolutional networks the dominant approach for almost all recognition and detection tasks since.

The NeurIPS 2022 program chairs, who called it the “AlexNet paper”, unanimously selected it for the conference’s Test of Time Award, given to a NeurIPS paper from ten years earlier, noting that it far surpassed the state of the art of its time.

Limitations

• Hardware-driven design. The authors state that network size was limited mainly by GPU memory and training time; the two-GPU split followed from 3 GB cards, and its connectivity was chosen by cross-validation.
• Ingredients that did not last. Simonyan and Zisserman (2015) found that LRN did not improve their networks on ILSVRC while costing memory and computation. Zeiler and Fergus (2014) found that AlexNet’s first-layer filters covered few mid frequencies and that the stride-4 first layer caused aliasing in the second.
• Incomplete specification. The paper defers details to its code and parameter files, and its ILSVRC-2012 numbers mix validation and test errors.

What Came After

Zeiler and Fergus (2014) visualized AlexNet’s layers and, in response, used $7 \times 7$ first-layer filters at stride 2, reporting a single model 1.7 points better in top-5 test error and 14.8% with an ensemble. Simonyan and Zisserman (2015) largely followed Krizhevsky et al.’s training procedure but replaced the large early filters with stacks of $3 \times 3$ convolutions in networks of up to 19 layers. Szegedy et al. (2015) reported that GoogLeNet used 12 times fewer parameters than AlexNet while being significantly more accurate. He et al. (2016) still followed the 2012 ImageNet practice of $224 \times 224$ crops and flips, mean subtraction, the paper’s color augmentation and ten-crop testing, while using residual connections and batch normalization, and no dropout, in networks of over 100 layers (see the ResNet paper).