Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

“Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification” by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun of Microsoft Research was posted to arXiv on 6 February 2015 and presented at the IEEE International Conference on Computer Vision (ICCV) in Santiago, Chile, in December 2015. It made two changes to rectifier networks. The parametric rectified linear unit (PReLU) learns the slope of the negative part of the activation function instead of fixing it at zero, and a rectifier-aware initialization, now usually called He or Kaiming initialization, scales each layer’s random weights so that very deep ReLU networks can be trained from scratch. With these, an ensemble of six networks reached 4.94% top-5 error on the ImageNet 2012 classification test set, which the authors compared with a human top-5 error of 5.1% reported by Russakovsky et al. The derivation of the initialization and its use today are covered in the weight initialization article; this article covers the publication.

arXiv has a single version (v1), and the ICCV paper differs from it in places. The ICCV version adds a leaky-ReLU baseline, an analysis of PReLU through the Fisher information matrix, a plot of mean layer responses, a detection experiment on PASCAL VOC and a footnote on the concurrent batch normalization paper. The arXiv version alone describes the protocol of the human study, compares with a later result from Baidu and analyses the errors class by class. Figure and table numbers below refer to the ICCV version unless labelled.

Problem

By early 2015 rectified linear units were standard in image classifiers, and the authors observed that recent improvements to models, and theoretical guidelines for training them, had rarely looked at the properties of the rectifier itself. ReLU is not symmetric: its mean response is never below zero, and even with symmetric inputs and weights its outputs are asymmetrically distributed. The leaky ReLU of Maas, Hannun and Ng (2013) kept a fixed small slope of 0.01 for negative inputs to avoid zero gradients, but in their experiments, as He et al. note, it had a negligible effect on accuracy compared with ReLU.

Depth was the other open question. At ILSVRC 2014, GoogLeNet (Szegedy et al., 2015) won classification with 6.66% top-5 test error and VGG (Simonyan & Zisserman, 2015) placed second with 7.32%; the authors’ own entry, based on spatial pyramid pooling (SPP), had 8.06%. Both leading entries were deeper than earlier networks, and training them was not straightforward. With weights drawn from Gaussians of fixed standard deviation, such as 0.01 in AlexNet, networks with more than about eight convolutional layers had difficulty converging, as the VGG team reported and He et al. observed. The VGG team therefore trained an 8-convolutional-layer model first and used it to initialize deeper ones (see VGG), and GoogLeNet and deeply supervised nets added auxiliary classifiers to intermediate layers. He et al. argued that pre-training costs time and may lead to a poorer local optimum. The principled alternative, the “Xavier” initialization of Glorot and Bengio (2010), had been derived assuming linear activations, which the authors called invalid for ReLU and PReLU.

Contribution

• PReLU, a rectifier whose negative slope is learned jointly with the rest of the network, one slope per channel or one per layer, at a negligible cost in parameters and computation.
• An initialization derived for rectifiers, which takes the ReLU’s halving of the signal’s second moment into account and made models of up to 30 weight layers converge from scratch where the Xavier initialization stalled.
• A family of large networks built on these two, of which the best single model had 5.71% top-5 validation error, better than every multi-model result of ILSVRC 2014, and a six-model ensemble with 4.94% top-5 test error, a relative improvement of 26% over GoogLeNet.

Method

PReLU. For the input $y_i$ of the activation on channel $i$, the paper defines

$$f(y_i) = \max(0, y_i) + a_i \min(0, y_i),$$

which is ReLU for $a_i = 0$ and the leaky ReLU for a small fixed $a_i$. In the channel-wise version each channel has its own $a_i$, so the extra parameters equal the number of channels; the channel-shared version uses one $a$ per layer. The gradient with respect to $a_i$ sums the back-propagated gradient times $y_i$ over all positions of the feature map where $y_i \le 0$, and $a_i$ is updated by momentum SGD like the weights. The slopes start at 0.25 and are trained without weight decay, which in the authors’ words would push them toward zero and so bias PReLU toward ReLU; even without regularization, the learned slopes rarely exceeded 1 in magnitude.

In the ICCV version the authors analysed why PReLU might help optimization. For a two-layer network under independence assumptions, the off-diagonal blocks of the Fisher information matrix contain the product of the mean activations, which is positive for ReLU and can be closer to zero for PReLU; blocks closer to zero make first-order SGD closer to the natural gradient. They note that the assumptions do not hold as training evolves, report an evaluation on real data in their supplementary material, and show that the trained PReLU network had smaller mean responses than the ReLU network in most layers (Figure 2).

Initialization. Following Glorot and Bengio, the paper tracks the variance of a layer’s responses: for a convolution with $k \times k$ filters and $c$ input channels, a response sums $n = k^2 c$ products, and because ReLU zeroes half of a zero-mean symmetric input, the variance is preserved when $\tfrac{1}{2} n \operatorname{Var}[w] = 1$, giving zero-mean Gaussian weights of standard deviation $\sqrt{2/n}$ and zero biases. The large models were initialized with the backward form of this condition, and the three fully connected layers with smaller standard deviations of 0.01, 0.01 and 0.001, because the unnormalized input pixels, in about $[-128, 128]$, could otherwise overflow the softmax. The weight initialization article gives the derivation, the backward form, the correction for PReLU and the paper’s worked case of VGG’s model B, in which a fixed standard deviation of 0.01 would shrink the gradient reaching the second layer by a factor of about $1.7 \times 10^4$.

Architectures. The comparisons of activations used a 14-layer model from He and Sun (2015), with a $7 \times 7$ first layer, ten layers of $2 \times 2$ filters, an SPP layer and three fully connected layers (Table 1). The large models (Table 3) start from VGG-19. Model A (19 layers) uses a $7 \times 7$ first layer with stride 2, moves the other three convolutions that VGG-19 applies at $224 \times 224$ and $112 \times 112$ resolution to the $56$, $28$ and $14$ stages, and inserts a spatial pyramid pooling layer with $7 \times 7$, $3 \times 3$, $2 \times 2$ and $1 \times 1$ bins (63 in total) before the first fully connected layer. Its cost, $1.90 \times 10^{10}$ operations, is close to VGG-19’s $1.96 \times 10^{10}$, but it ran faster in practice, 2.6 s against 3.0 s for their reproduction of VGG-19 per mini-batch of 128 on four Nvidia K20 GPUs, because convolutions on large maps were slower than equally costly ones on small maps. The authors state that they have no evidence that model A is a better architecture than VGG-19. Model B (22 layers, $2.32 \times 10^{10}$ operations) adds three convolutional layers, and model C widens B to 384, 768 and 896 filters in the last three stages ($5.30 \times 10^{10}$ operations). They chose width over further depth because deeper models had shown diminishing or negative returns.

Training and testing. Training largely follows AlexNet, VGG and SPP-net: random $224 \times 224$ crops from an image whose shorter side is drawn from $[256, 512]$, horizontal flips and AlexNet’s colour alteration, with scale jittering from the start rather than only during fine-tuning as in VGG. Every model was trained from scratch with the new initialization, with mini-batches of 128, momentum 0.9, weight decay 0.0005, dropout of 50% in the first two fully connected layers, and a learning rate of $10^{-2}$, $10^{-3}$ and then $10^{-4}$, switched when the error plateaued, for about 80 epochs. Models A and B took three to four weeks on four K20 GPUs and model C as long on eight K40 GPUs, using data parallelism for the convolutional layers in a modified Caffe. At test time, the convolutional layers run on the whole resized image; each $14 \times 14$ window of the last feature map is pooled by the SPP layer and classified, and the scores of all windows, of the flipped image and of several scales are averaged.

Worked Example: What a Learned Slope Keeps

The authors found the largest learned slopes in the first layer (0.681 channel-shared, 0.596 averaged over channels) and read them as the network keeping both the positive and the negative responses of its edge-like filters, which they called economical given the limited number of filters. The PyTorch script below (PyTorch 2.14.1) is our illustration of that reading in its most extreme form: the first layer has a single $1 \times 1$ filter without bias, so a ReLU must discard one polarity of the image completely. The synthetic $12 \times 12$ images contain one horizontal or vertical bar (the class) of 4 to 8 pixels, either bright (+1) or dark (−1) on a background of 0. With a positive weight $w$, a dark bar becomes $\text{ReLU}(-w) = 0$, the same as the background, so after the first layer every dark-bar image is exactly zero and its class cannot be recovered. The rest of the network is two $3 \times 3$ ReLU convolutions with 8 channels, global max pooling and a linear classifier, trained with Adam and no weight decay; the script prints the test accuracy on bright and dark bars for seeds 0 to 4.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)

def bars(n, size=12, gen=None):
    """Class 0: horizontal bar, class 1: vertical bar (length 4-8 px, random place).
    Polarity is random: the bar is +1 (bright) or -1 (dark) on a background of 0."""
    y = torch.randint(0, 2, (n,), generator=gen)
    sign = torch.randint(0, 2, (n,), generator=gen) * 2 - 1           # +1 bright, -1 dark
    length = torch.randint(4, 9, (n,), generator=gen)
    a = torch.randint(0, size, (n,), generator=gen)
    b = (torch.rand(n, generator=gen) * (size - length + 1)).long()
    x = torch.zeros(n, 1, size, size)
    for i in range(n):
        if y[i] == 0:
            x[i, 0, a[i], b[i]:b[i] + length[i]] = sign[i]
        else:
            x[i, 0, b[i]:b[i] + length[i], a[i]] = sign[i]
    return x, y, sign

class Net(nn.Module):
    """One 1x1 filter without bias, then the activation under test, then a small ReLU CNN."""
    def __init__(self, act):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 1, 1, bias=False)   # a single first-layer filter
        self.act = act
        self.body = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(),
                                  nn.Conv2d(8, 8, 3, padding=1), nn.ReLU())
        self.head = nn.Linear(8, 2)

    def forward(self, x):
        h = self.body(self.act(self.conv1(x)))
        return self.head(h.amax(dim=(2, 3)))         # global max pooling

def run(name, seed, steps=600):
    torch.manual_seed(seed)
    gen = torch.Generator().manual_seed(seed)
    x_tr, y_tr, _ = bars(2048, gen=gen)
    x_te, y_te, s_te = bars(2048, gen=gen)
    act = {"ReLU": nn.ReLU(), "LReLU 0.25": nn.LeakyReLU(0.25), "PReLU": nn.PReLU(init=0.25)}[name]
    model = Net(act)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)  # no weight decay, as for the paper's a_i
    for _ in range(steps):
        idx = torch.randint(0, 2048, (64,), generator=gen)
        loss = F.cross_entropy(model(x_tr[idx]), y_tr[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    with torch.no_grad():
        ok = model(x_te).argmax(1) == y_te
    slope = f"{model.act.weight.item():+.2f}" if name == "PReLU" else "-"
    w = model.conv1.weight.item()                    # the single first-layer weight
    return w, ok[s_te == 1].float().mean().item(), ok[s_te == -1].float().mean().item(), slope

print(f"torch {torch.__version__}")
print("activation  seed  conv1 w  acc. bright bars  acc. dark bars  learned a")
for name in ["ReLU", "LReLU 0.25", "PReLU"]:
    for seed in range(5):
        w, br, dk, slope = run(name, seed)
        print(f"{name:10s}  {seed:4d}  {w:+7.2f}  {br:16.3f}  {dk:14.3f}  {slope:>9s}")

Output:

torch 2.14.1
activation  seed  conv1 w  acc. bright bars  acc. dark bars  learned a
ReLU           0    +0.73             1.000           0.490          -
ReLU           1    +0.97             1.000           0.491          -
ReLU           2    +0.71             1.000           0.500          -
ReLU           3    -1.32             0.491           1.000          -
ReLU           4    +0.59             1.000           0.500          -
LReLU 0.25     0    -0.82             1.000           1.000          -
LReLU 0.25     1    +1.03             1.000           1.000          -
LReLU 0.25     2    +0.86             1.000           1.000          -
LReLU 0.25     3    -1.47             1.000           1.000          -
LReLU 0.25     4    +0.70             1.000           1.000          -
PReLU          0    -0.63             1.000           1.000      +0.99
PReLU          1    +0.92             1.000           1.000      +0.70
PReLU          2    +0.73             1.000           1.000      +0.76
PReLU          3    -1.36             1.000           1.000      +0.66
PReLU          4    +0.58             1.000           1.000      +0.74

With ReLU, the network classifies one polarity perfectly and the other at chance, and which polarity is lost follows the sign of the first-layer weight: on seed 3 the weight is negative and the bright bars are the ones discarded. Any nonzero slope keeps the information, as the fixed leaky ReLU shows. PReLU, started at 0.25, raised its slope to between 0.66 and 0.99 on every seed, comparable to or above the paper’s first-layer values; in our reading, the gradient favours a larger slope in this deliberately starved network because it enlarges the dark bars’ responses. The paper’s own networks had 64 or 96 first-layer filters and could have covered both polarities with separate filters, so the example shows the mechanism, not the size of the effect. Running the script twice gave identical output.

Results

All results are on ImageNet 2012 (about 1.2 million training images, 50,000 validation and 100,000 test images), on the validation set except the final test-set comparison.

PReLU against ReLU. On the 14-layer model (Table 2; 10-view testing, shorter side 256 for training and testing, 75 epochs), the top-1/top-5 errors were 33.82/13.34% with ReLU, 33.80/13.56% with a leaky ReLU fixed at $a = 0.25$, 32.71/12.87% with channel-shared PReLU and 32.64/12.75% with channel-wise PReLU. The channel-shared version adds only 13 parameters, which the authors took as evidence that learning the shape of the activation matters. They also note that a fixed slope chosen carefully by cross-validation could beat ReLU, at the cost of repeated training. The learned slopes (Table 1) were largest in the first layer; for the channel-wise version, deeper convolutional layers generally had smaller slopes, from 0.124 to 0.198 in the last stage and about 0.075 in the fully connected layers, which the authors read as activations becoming “more nonlinear” with depth. The channel-shared slopes vary less regularly: 0.561 in the last layer of the second stage, then from 0.008 to 0.210 in the last stage. On the large model A (Table 4; dense testing), PReLU lowered the error at each of three test scales and, with the scales combined, from 24.02/6.51% to 22.97/6.28%, a reduction of 1.05 points in top-1 and 0.23 in top-5 error. PReLU had lower training and validation error than ReLU throughout training (Figure 4).

Initialization. The initialization was compared with Xavier’s on a 22-layer model, which converged with both but started reducing the error earlier with the new one, and on a 30-layer model, which converged only with the new one (Figure 3); the weight initialization article gives the numbers. The 30-layer model reached 38.56/16.59% error, worse than the 14-layer model’s 33.82/13.34%, a degradation discussed under Limitations.

Single models. With 10-view testing (Table 5), model C with PReLU had 24.27/7.38% top-1/top-5 validation error, against 28.07/9.33% for VGG-16, which the authors tested themselves with the released model, and 9.15% top-5 for GoogLeNet. With multi-scale dense testing (Table 6):

Model (validation set, multi-scale)	Top-1 error	Top-5 error
VGG (arXiv v5 of the VGG paper)	24.4%	7.1%
A, ReLU	24.02%	6.51%
A, PReLU	22.97%	6.28%
B, PReLU	22.85%	6.27%
C, PReLU	21.59%	5.71%

The authors attribute the gain of their ReLU baseline over VGG-19 mainly to end-to-end training without pre-training shallower models. The 19-layer A and 22-layer B performed about the same, while the wider C improved on B, from which they concluded that once models are deep enough, width becomes an essential factor.

Ensemble. Six models, only one of them with architecture C, reached 4.94% top-5 error on the test set (Table 7), whose labels are not published (the arXiv version adds that the ILSVRC server evaluated it), against 6.66% for GoogLeNet, 7.32% for VGG’s competition entry and 6.8% for VGG’s later arXiv v5 ensemble. The arXiv version’s tables also list Baidu’s 5.98%, which the ICCV version omits. A footnote in the ICCV version notes that batch normalization (Ioffe & Szegedy, 2015), posted to arXiv five days after this paper, achieved results on par: 5.82% for a single model and 4.82% for an ensemble.

Transfer to detection (ICCV version). As backbones for Fast R-CNN (Girshick, 2015), fine-tuned on PASCAL VOC 2007 trainval and evaluated on its test set, the networks reached a mean average precision of 67.6% for A with ReLU, 68.4% for A with PReLU and 69.2% for C with PReLU, against 66.9% reported for VGG-16 in the Fast R-CNN paper. The authors read the gain of PReLU over ReLU on the same model as a sign of better learned features for transfer learning.

The comparison with human performance

The 5.1% comes from a study by Russakovsky et al. (2015) of two expert annotators labelling ILSVRC classification test images with five guesses each, through an interface that listed all 1,000 classes, each with a row of 13 training images. The first annotator trained on 500 images and then labelled a random sample of 1,500 test images with a top-5 error of 5.1%, against 6.8% for GoogLeNet on the same images. The second trained on only 100 images and labelled 258 images with an error of about 12.0%. On the 204 images both labelled, counting an image as correct when either annotator got it right gave an error of about 2.4%, against 4.9% for GoogLeNet. “Human-level performance” in the title therefore means the error of one trained annotator on a 1,500-image sample. The 4.94% is a test-set result from the ILSVRC evaluation server, not a result on the same images; the paper does not give the number of images, and we take it to be the full 100,000-image test set. By our arithmetic, the binomial standard error of a 5.1% error rate estimated from 1,500 images is about 0.57 points, several times the 0.16-point difference.

The authors framed the result carefully. Following Russakovsky et al., who found that 37% of the annotator’s errors involved fine-grained classes, such as the more than 120 dog breeds, and 24% classes the annotator was not aware of, He et al. showed fine-grained images their network classified correctly, while noting that it still erred where humans find the task easy, especially when context or high-level knowledge is needed. They stated that the result does not indicate that machine vision outperforms human vision on object recognition in general, and that on everyday object categories, such as those of PASCAL VOC, machines still made obvious errors in cases trivial for humans.

Impact

The PAMI Technical Committee lists the paper as a 2025 winner of the Helmholtz Prize, alongside Fast R-CNN; the biennial prize, presented at each ICCV, recognizes ICCV papers from ten years earlier with significant impact on computer vision research.

The initialization became a library default under the authors’ names. In PyTorch 2.14, torch.nn.init.kaiming_normal_ and kaiming_uniform_ cite this paper and are documented as “also known as He initialization”; their mode argument chooses between the paper’s forward (fan_in) and backward (fan_out) conditions, and their gain for a leaky rectifier with slope $a$, $\sqrt{2/(1 + a^2)}$, is the paper’s PReLU correction. TorchVision’s ResNets (version 0.29.0 source) initialize every convolution with kaiming_normal_(mode="fan_out", nonlinearity="relu"), the backward form the paper used for its large models. PyTorch’s nn.PReLU keeps the paper’s initial slope of 0.25, defaults to a single shared slope (num_parameters=1) rather than the channel-wise version the paper used in its comparison on model A, and its documentation advises against weight decay on the slope.

The authors’ next paper, on residual networks, initialized its weights “as in” this one, trained all networks from scratch, and listed the PReLU network as a baseline in its ImageNet tables. Clevert et al. (2016) initialized the networks of their MNIST experiment on ELUs with He et al.’s method and placed an SPP layer, citing this paper, before the fully connected layers of their ImageNet network.

Limitations

• The human comparison is narrow. The reference point is one annotator on a 1,500-image sample, a second annotator was more than twice as wrong, and the two error rates were measured on different images. The authors themselves restricted the claim to this dataset.
• Depth was not solved. The initialization let 30-layer models converge but not improve: the 30-layer model was clearly worse than the 14-layer one, which the ICCV version attributes to higher training error and calls an open problem. The arXiv version had instead suggested that the way depth was added might be inappropriate or the task not complex enough, and mentions models with 3 to 9 layers added to B whose training and test errors degraded in the first 20 epochs. The large models were made wider rather than deeper for this reason. The ResNet paper took up this degradation problem.
• PReLU’s gains were modest and its robustness unclear. On ImageNet, PReLU lowered model A’s multi-scale top-5 error by 0.23 points. Xu et al. (2015) compared rectifiers in Network-in-Network models on CIFAR-10 and CIFAR-100: PReLU had the lowest training error, and lower test error than ReLU (11.79% against 12.45% on CIFAR-10, 41.63% against 42.90% on CIFAR-100), but higher test error than a very leaky ReLU with a fixed slope of $1/5.5$ and their randomized leaky ReLU (11.20% and 11.19% on CIFAR-10, 40.42% and 40.25% on CIFAR-100), which they read as a sign that PReLU may overfit on small datasets. The authors’ own ResNets used ReLU, as do TorchVision’s.
• The analyses assume independence. Both the variance derivation and the Fisher-information argument assume independent, identically distributed weights and inputs, and the authors acknowledge that the latter’s assumptions fail as training evolves. The initialization fixes the scale only at the start of training; the vanishing and exploding gradients article discusses what it does and does not address.
• Heavy machinery. The headline result combined six models, dense multi-scale testing and weeks of multi-GPU training per model.

What Came After

Batch normalization (Ioffe & Szegedy, 2015; see the batch normalization paper), posted days later, reached a comparable ensemble error by normalizing each layer’s inputs during training rather than only at initialization. In December 2015 the same four authors posted “Deep Residual Learning for Image Recognition” (He et al., 2016), which counted normalized initialization, including this paper’s, and batch normalization as having largely addressed vanishing gradients, and added identity shortcuts to overcome the degradation this paper had left open; a ResNet ensemble won ILSVRC 2015 with 3.57% top-5 test error.

Learned and negative-valued activations continued in other forms. Clevert et al. (2016) argued that PReLU and leaky ReLU, although they take negative values, do not saturate to a noise-robust deactivated state, and proposed the exponential linear unit (ELU), which saturates to a negative constant; later smooth activations such as GELU and SiLU are described in the activation function article. On the initialization side, data-driven schemes such as LSUV and schemes for residual networks such as zero-initialized residual branches and Fixup start from He et al.’s rule or replace it; the weight initialization article covers them.