Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

“Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift” is a nine-page paper by Sergey Ioffe and Christian Szegedy of Google. It was posted to arXiv in February 2015 and published at the 32nd International Conference on Machine Learning (ICML) in Lille that July. It proposed normalizing activations inside a network with statistics of each training mini-batch, as a layer that gradients flow through, and showed that a batch-normalized Inception network could be trained on ImageNet with learning rates up to 30 times higher, without dropout, and to the original’s accuracy in a fraction of the steps. Its explanation, a reduction of what it named internal covariate shift, has since been disputed, but the layer was soon taken up across convolutional networks. The technique as used today is covered in the batch normalization article. The paper received the ICML 2025 Test of Time Award.

Problem

By 2015, deep networks were trained with stochastic gradient descent and its variants, which, the paper notes, need careful tuning of the learning rate and initial parameters. The authors located the difficulty in depth: each layer’s inputs depend on the parameters of every layer below it, so small parameter changes are amplified as they propagate, and each layer must keep adapting to inputs whose distribution moves during training.

They borrowed the term covariate shift, a change in the input distribution of a learning system, usually handled by domain adaptation. The paper extends the notion from a whole model to its parts and calls the change in the distribution of a network’s internal activations during training internal covariate shift. They tied it to a second problem: with a saturating activation function such as the sigmoid, parameter updates can push many pre-activations into the flat regions where the derivative is near zero and gradients vanish. The usual remedies, the paper notes, were rectified linear units, careful initialization and small learning rates.

Whitening a network’s inputs (zero mean, unit variance, decorrelated) was known to speed up convergence, so the paper considers whitening every layer’s inputs, and identifies two obstacles. If the normalization is computed outside the gradient step, the optimizer does not see it: the paper’s example is a bias that grows without bound while the loss stays fixed, and the authors’ early models blew up this way. And full whitening of each layer over the whole training set needs a covariance matrix, its inverse square root, and their derivatives, which is too costly to repeat after every update.

Contribution

• Normalization inside the model. Each scalar activation is standardized independently rather than whitened jointly, and with mini-batch statistics rather than training-set statistics, so the normalization is differentiable and the gradient accounts for it. A learned scale $\gamma$ and shift $\beta$ per activation let the layer represent the identity, so normalization costs no expressive power.
• Inference and convolutions. After training, population estimates replace the batch statistics, so an output depends only on its input and the transform folds into a fixed affine map. In convolutional layers, statistics are pooled over the mini-batch and all positions of a feature map, with one $\gamma$ and $\beta$ per map.
• Claims about training. The paper argues that batch normalization allows much higher learning rates, reduces dependence on the parameter scale and initialization, makes networks with saturating nonlinearities trainable, and regularizes the model, in some cases removing the need for dropout.

Method

Algorithm 1 gives the transform and its backward pass through the batch statistics, and Algorithm 2 the training and inference procedure; the batch normalization article derives both. The transform sits between the affine map and the nonlinearity, $z = g(\mathrm{BN}(W\mathbf{u}))$, with the bias dropped, because the authors expect $W\mathbf{u}$, unlike a layer’s input (itself the output of a nonlinearity), to have a symmetric, non-sparse distribution that normalization can stabilize.

Section 3.3 argues why higher learning rates become safe. Normalization keeps small parameter changes from amplifying through depth. Scaling a layer’s weights by a scalar leaves its output and Jacobian unchanged, and the gradient with respect to the weights shrinks in proportion, so the authors argue that batch normalization stabilizes the growth of the parameters. They also conjecture, under assumptions they acknowledge are unrealistic, that it brings layer Jacobians closer to orthogonal, and leave this to further study.

Results

MNIST. To look at internal covariate shift directly, the authors trained a deliberately simple network on 28×28 binary MNIST digits: three fully connected hidden layers of 100 sigmoid units with small random Gaussian initial weights, trained for 50,000 steps with 60 examples per mini-batch. The batch-normalized version reached higher test accuracy sooner. Plotting the 15th, 50th and 85th percentiles of one typical sigmoid input in the last hidden layer over training, the paper shows the distribution drifting in both mean and spread without batch normalization and staying much more stable with it.

ImageNet, single networks. The main experiment used a new variant of Inception (Szegedy et al., 2015) in which each 5×5 convolution was replaced by two stacked 3×3 convolutions, with 13.6 million parameters and no fully connected layer except the softmax. It was trained with distributed asynchronous SGD with momentum and mini-batches of 32, and evaluated by single-crop top-1 accuracy on the validation set. Adding batch normalization alone, the authors write, does not take full advantage of it, so the accelerated networks also changed the training recipe:

• increase the learning rate;
• remove dropout, which the authors found raised validation accuracy, conjecturing that batch statistics supply a similar regularization;
• shuffle training examples more thoroughly, so the same examples do not keep sharing a mini-batch, worth about 1% in validation accuracy;
• reduce the L2 weight penalty fivefold;
• decay the learning rate six times faster;
• remove local response normalization;
• reduce photometric distortions, since the faster-training network sees each image fewer times.

The baseline Inception, with initial learning rate 0.0015, reached 72.2% after 31 million steps. The paper reports, for steps to reach 72.2% and maximum accuracy:

Network	Change from Inception	Steps to 72.2%	Max accuracy
BN-Baseline	batch normalization before each nonlinearity	13.3 million	72.7%
BN-x5	plus the changes above, learning rate ×5	2.1 million	73.0%
BN-x30	as BN-x5, learning rate ×30	2.7 million	74.8%
BN-x5-Sigmoid	as BN-x5, sigmoid instead of ReLU	not reached	69.8%

BN-x5 thus matched Inception in 14 times fewer steps. The same fivefold learning-rate increase without batch normalization drove the parameters to machine infinity, and Inception with sigmoid units stayed at chance accuracy. BN-x30 trained more slowly at first than BN-x5 but reached 74.8% after about 6 million steps, a result the authors call counterintuitive.

ImageNet, ensemble. Six networks based on BN-x30, variously modified with larger initial convolution weights, light dropout (5% or 10%, against 40% in the original Inception) and non-convolutional batch normalization in the last hidden layers, were combined by averaging class probabilities over 144 crops per image. The ensemble reached 4.9% top-5 error on the 50,000 validation images and 4.82% on the ILSVRC test server, against 4.94% reported by He et al. (2015) and 6.67% for the GoogLeNet ensemble. The paper says this exceeds the estimated accuracy of human raters reported in the ImageNet challenge paper; He et al.’s result, posted the same month, had already made that claim in its title. A single BN-Inception network had 7.82% top-5 validation error on one crop and 5.82% with 144 crops.

Impact

Adoption in computer vision was fast. ResNet (He et al., 2016), posted to arXiv ten months after the paper, put batch normalization after every convolution and before the activation, “following” Ioffe and Szegedy, and, like them, trained without dropout; in our reading, the paper’s training lessons traveled with the layer. By 2018, Ioffe (2017) had called batch normalization part of the standard toolkit for training deep networks, Santurkar et al. (2018) wrote that most deep learning models used it by default, and Wu and He (2018) called it a foundation of many state-of-the-art computer vision algorithms, crediting it with easing optimization and with a regularizing effect from the noise of batch statistics.

ICML gave the paper its 2025 Test of Time Award.

Limitations

The evidence for the central hypothesis is narrow. Internal covariate shift is shown directly only in the MNIST plot, for one activation of one small network; the ImageNet experiments measure speed and accuracy, which other mechanisms could explain. The Jacobian argument rests on assumptions the authors say do not hold, the BN-x30 behavior is unexplained, and the regularizing effect is a conjecture the conclusion says needs more study. All ImageNet results come from one architecture, and the accelerated networks changed seven parts of the recipe at once; apart from shuffling, the paper does not measure what each change contributed.

The explanation was later challenged. Santurkar et al. (2018) injected noise that shifted the distributions after each batch-normalization layer and found that the networks trained almost as well as with standard batch normalization; they argue that batch normalization helps by smoothing the loss landscape rather than by stabilizing input distributions (see the batch normalization article).

The method also depends on the batch. The paper requires mini-batches larger than one and notes that an activation’s value depends on the other examples in its batch. Later work found that accuracy falls when batches are small or not independent (Ioffe, 2017), as in detection and segmentation, where memory allows only a few images per GPU (Wu and He, 2018).

What Came After

The main alternatives avoid the batch dimension altogether. Layer normalization (Ba, Kiros and Hinton, 2016) normalizes over the units of a layer for each example, computes the same in training and testing, and was aimed at recurrent networks and small mini-batches; it is the normalization of the transformer. Group normalization (Wu and He, 2018) normalizes groups of channels within each image and stays accurate at batch sizes where batch normalization degrades. Ioffe’s own batch renormalization (2017) instead keeps batch statistics but corrects them toward the population estimates so that training and inference outputs match. The batch normalization article compares these variants.