Normalization Layers

A normalization layer standardizes the activations of a neural network, subtracting a mean and dividing by a standard deviation, and then applies a learned scale and shift. The common activation normalizations (batch, layer, instance and group normalization) compute exactly the same thing. What distinguishes them is which entries of a feature map share one mean and variance. That choice decides whether an image’s output depends on the other images in its batch, whether training and inference compute the same function, and which differences between images the layer erases. RMS normalization drops the mean, and weight normalization normalizes the weights rather than the activations. Batch normalization has its own article; this one compares the family.

Definition

An activation normalization layer takes a tensor of shape $(N, C, H, W)$ (batch, channels, height, width) and partitions its entries into sets. Within each set it subtracts the set’s mean and divides by the set’s standard deviation. It then multiplies every entry by a learned scale and adds a learned shift, usually one pair per channel. The method is named by the partition:

• Batch normalization (BN): one set per channel, spanning all images and positions, i.e. statistics over the axes $(N, H, W)$.
• Layer normalization (LN): one set per image, over $(C, H, W)$.
• Instance normalization (IN): one set per image and channel, over $(H, W)$.
• Group normalization (GN): one set per image and group of $C/G$ consecutive channels, over $(C/G, H, W)$.

RMS normalization (RMSNorm) divides by the root mean square over a set without subtracting the mean. Weight normalization (WN) is not an activation normalization at all: it rewrites each filter of a layer as a direction vector of unit length times a learned length.

Intuition

Picture the feature map as a block of numbers with four edges: images, channels, rows and columns. Each method cuts the block along some of those edges and standardizes each piece separately. Cut along channels only and every piece mixes all the images in the batch: that is BN, and its statistics describe the batch, not the image. Cut along images and the pieces never mix images, so the layer behaves the same for a batch of one as for a batch of a thousand. Wu and He’s Figure 2 draws the four cases as shaded regions of such a block.

Whatever falls inside a piece loses its overall level and spread. IN erases each channel’s mean and contrast in every image; Ulyanov, Vedaldi and Lempitsky (2016) wanted exactly this for style transfer, where the stylized image should not depend on the contrast of the content image. LN erases only the overall level and spread of the whole image, leaving the relative strengths of channels intact. GN sits between them: channels in the same group are compared with one another, channels in different groups are not.

Formal Definition

Index an entry by $i = (i_N, i_C, i_H, i_W)$ and let $\mathcal{S}_i$ be the set of entries normalized together with it. Following Wu and He (2018, Eqs. 1–2, 6),

$$\mu_i = \frac{1}{m}\sum_{k \in \mathcal{S}_i} x_k, \qquad \sigma_i^2 = \frac{1}{m}\sum_{k \in \mathcal{S}_i} (x_k - \mu_i)^2, \qquad y_i = \gamma_{i_C}\,\frac{x_i - \mu_i}{\sqrt{\sigma_i^2 + \epsilon}} + \beta_{i_C},$$

with $m = |\mathcal{S}_i|$ and a small constant $\epsilon$. The methods differ only in $\mathcal{S}_i$:

Method	$\mathcal{S}_i$: entries $k$ with	Axes reduced	Statistics per layer	Uses other images
Batch norm	$k_C = i_C$	$N, H, W$	$C$	yes, in training
Layer norm	$k_N = i_N$	$C, H, W$	$N$	no
Instance norm	$k_N = i_N,\ k_C = i_C$	$H, W$	$N C$	no
Group norm	$k_N = i_N,\ \lfloor k_C / (C/G) \rfloor = \lfloor i_C / (C/G) \rfloor$	$C/G, H, W$	$N G$	no
Layer norm over channels	$k_N = i_N,\ k_H = i_H,\ k_W = i_W$	$C$	$N H W$	no

GN with $G = 1$ is LN, and with $G = C$ it is IN; Wu and He use $G = 32$ by default. The last row is not in Wu and He’s table but is how transformers apply LN, as discussed under Common Misconceptions.

The affine parameters differ between definitions as well. Wu and He learn one $\gamma$ and $\beta$ per channel for all four methods. Ba, Kiros and Hinton (2016) give each hidden unit its own gain and bias, and PyTorch’s nn.LayerNorm follows them: with normalized_shape $(C, H, W)$ it learns $C \cdot H \cdot W$ scales and as many shifts. Ulyanov et al.’s instance normalization (Eq. 3) has no affine parameters, and nn.InstanceNorm2d defaults to affine=False.

RMSNorm (Zhang and Sennrich, 2019) keeps only the rescaling:

$$y_i = g_i\,\frac{x_i}{\mathrm{RMS}(\mathbf{x})}, \qquad \mathrm{RMS}(\mathbf{x}) = \sqrt{\frac{1}{n}\sum_{k=1}^{n} x_k^2} ,$$

over the $n$ summed inputs of a layer. Their Eq. 4 has no $\epsilon$ and no shift; the layer’s bias is added after normalization, as in their Eq. 2 for LN. PyTorch’s nn.RMSNorm adds $\epsilon$ inside the square root, by default the machine epsilon of the computation type. When the inputs have zero mean, RMSNorm equals LN. Zhang and Sennrich hypothesize that LN’s re-centering is dispensable and that its re-scaling is what helps.

Weight normalization (Salimans and Kingma, 2016) reparameterizes a filter $\mathbf{w}$, for a convolution the $k^2 C_\text{in}$ weights of one output channel, as

$$\mathbf{w} = \frac{g}{\lVert \mathbf{v} \rVert}\,\mathbf{v},$$

and trains $g$ and $\mathbf{v}$ instead of $\mathbf{w}$, so that $\lVert \mathbf{w} \rVert = g$ whatever $\mathbf{v}$ is. No activation statistics are involved. The authors show that for a single layer with whitened inputs, BN’s pre-activation statistics are $\mu = 0$ and $\sigma = \lVert \mathbf{v} \rVert$, so the two coincide there; in deeper networks they do not.

Properties

Training versus inference. Only BN uses statistics from other examples, so only BN needs a different rule at inference: it replaces batch statistics with running averages collected during training. LN, IN, GN and RMSNorm compute the same function in both modes, a point Ba et al. made for LN. PyTorch’s InstanceNorm2d can track running statistics, but track_running_stats is off by default. A consequence (our reasoning): with fixed statistics, BN at inference is a per-channel affine map that can be folded into the preceding convolution, while the batch-independent layers must compute their statistics on every input at inference too.

Dependence on batch size. BN’s accuracy drops as the number of images per device falls, and Wu and He designed GN for that regime; the batch normalization article gives their ImageNet numbers. Removing the batch from the statistics does not, in our reading, make training independent of the batch size: gradient noise still depends on it, and Wu and He scaled the learning rate linearly with the batch size in these experiments.

Invariances. A normalization layer’s output does not change when its input changes in ways the statistics absorb. LN is invariant to rescaling and shifting all of one image’s activations together, IN additionally to a separate rescaling and shift of each channel, and RMSNorm to rescaling but not to shifting. Ba et al. (Table 1) and Zhang and Sennrich (Table 1) tabulate these properties with respect to weights and data. The same invariance means that the information is lost: an IN layer cannot pass on how bright or high-contrast a channel was in an image unless another path carries it.

Accuracy in convolutional classifiers. Wu and He compared the four methods in ResNet-50 on ImageNet with 32 images per GPU (validation error, center crop): BN 23.6%, GN 24.1%, LN 25.3% and IN 28.4%. Among fixed numbers of groups, $G = 32$ was best and one group (LN) was 1.2 points worse; among fixed numbers of channels per group, one channel (IN) was 4.2 points worse than the best, 16 (Table 3). Ba et al. had already observed in preliminary experiments that LN sped up convolutional networks over no normalization but that BN did better, and suggested why: units whose receptive fields lie near the image border are rarely active, so their statistics differ from the rest of the layer, contradicting LN’s assumption that all units contribute similarly.

Placement in transformers. The original transformer applies LN after each residual addition (post-norm); Vision Transformers apply it to the input of each sub-layer (pre-norm). The transformer article gives the equations and Xiong et al.’s (2020) analysis of why post-norm needs a learning-rate warmup.

Examples

The script builds a feature map of shape $(N, C, H, W) = (4, 8, 16, 16)$ from four synthetic $16 \times 16$ images (a disc, a square, horizontal stripes and vertical stripes, each of period 4) filtered by eight random $3 \times 3$ kernels. It implements each normalization in NumPy as a choice of axes and checks the result against the PyTorch module. It was run with NumPy 2.5.4 and PyTorch 2.14.1, in double precision, with all scales $\gamma = 1$ and shifts $\beta = 0$ as at initialization.

import numpy as np
from scipy.signal import correlate2d

rng = np.random.default_rng(0)
S = 16
yy, xx = np.mgrid[:S, :S]

def image(kind, contrast=1.0, offset=0.0):
    """A 16x16 grayscale image of a known shape, with given contrast and offset."""
    shapes = {
        "disc":    ((yy - 7.5) ** 2 + (xx - 7.5) ** 2 < 25).astype(float),
        "square":  ((abs(yy - 7.5) < 4) & (abs(xx - 7.5) < 4)).astype(float),
        "hstripe": (yy // 2 % 2).astype(float),        # horizontal stripes, period 4
        "vstripe": (xx // 2 % 2).astype(float),        # vertical stripes, period 4
    }
    return offset + contrast * shapes[kind]

kernels = rng.normal(size=(8, 3, 3))                    # 8 random 3x3 filters -> C = 8
def features(imgs):
    return np.stack([[correlate2d(im, k, mode="same") for k in kernels] for im in imgs])

x = features([image("disc"), image("square"), image("hstripe"), image("vstripe")])
N, C, H, W = x.shape
G = 4                                                   # GroupNorm: 4 groups of 2 channels
eps = 1e-5

def standardize(x, axes):
    mu = x.mean(axis=axes, keepdims=True)
    var = x.var(axis=axes, keepdims=True)               # biased variance, as in training mode
    return (x - mu) / np.sqrt(var + eps), mu.size

def group_norm(x, G):
    xg, n = standardize(x.reshape(N, G, C // G, H, W), (2, 3, 4))
    return xg.reshape(N, C, H, W), n

def rms_norm(x, axes):
    ms = (x ** 2).mean(axis=axes, keepdims=True)
    return x / np.sqrt(ms + eps), ms.size

methods = {                                   # name: (output, number of statistics, axes reduced)
    "batch":          (*standardize(x, (0, 2, 3)), "N,H,W"),
    "layer":          (*standardize(x, (1, 2, 3)), "C,H,W"),
    "instance":       (*standardize(x, (2, 3)), "H,W"),
    "group":          (*group_norm(x, G), "C/G,H,W"),
    "layer (C only)": (*standardize(x, (1,)), "C"),
    "rms":            (*rms_norm(x, (1, 2, 3)), "C,H,W"),
}
print("feature map (N, C, H, W) =", x.shape)
for name, (y, n_stats, axes) in methods.items():
    print(f"{name:15s} reduces over {axes:8s} {n_stats:4d} sets of statistics, "
          f"{x.size // n_stats:5d} values each")

# ---- compare with PyTorch modules (gamma = 1, beta = 0 at initialization) ----
import torch, torch.nn as nn
torch.set_num_threads(1)
print("torch", torch.__version__)
xt = torch.from_numpy(x)                                # float64
to_last, to_first = (0, 2, 3, 1), (0, 3, 1, 2)
modules = {
    "batch":          nn.BatchNorm2d(C),                # training mode: batch statistics
    "layer":          nn.LayerNorm([C, H, W]),
    "instance":       nn.InstanceNorm2d(C),
    "group":          nn.GroupNorm(G, C),
    "layer (C only)": lambda t: nn.LayerNorm(C).double()(t.permute(to_last)).permute(to_first),
    "rms":            nn.RMSNorm([C, H, W], eps=eps),
}
with torch.no_grad():
    for name, m in modules.items():
        if isinstance(m, nn.Module):
            m = m.double()
        diff = np.abs(m(xt).numpy() - methods[name][0]).max()
        print(f"{name:15s} max |numpy - torch| = {diff:.1e}")
    # The two extremes of group normalization (Wu and He, 2018): G = 1 and G = C.
    print("GroupNorm(1, C) vs layer:    %.1e" % np.abs(nn.GroupNorm(1, C).double()(xt).numpy() - methods["layer"][0]).max())
    print("GroupNorm(C, C) vs instance: %.1e" % np.abs(nn.GroupNorm(C, C).double()(xt).numpy() - methods["instance"][0]).max())

Output:

feature map (N, C, H, W) = (4, 8, 16, 16)
batch           reduces over N,H,W       8 sets of statistics,  1024 values each
layer           reduces over C,H,W       4 sets of statistics,  2048 values each
instance        reduces over H,W        32 sets of statistics,   256 values each
group           reduces over C/G,H,W    16 sets of statistics,   512 values each
layer (C only)  reduces over C        1024 sets of statistics,     8 values each
rms             reduces over C,H,W       4 sets of statistics,  2048 values each
torch 2.14.1
batch           max |numpy - torch| = 8.2e-15
layer           max |numpy - torch| = 8.9e-16
instance        max |numpy - torch| = 6.2e-15
group           max |numpy - torch| = 1.3e-15
layer (C only)  max |numpy - torch| = 6.7e-16
rms             max |numpy - torch| = 4.4e-16
GroupNorm(1, C) vs layer:    8.9e-16
GroupNorm(C, C) vs instance: 1.1e-15

Each one-line NumPy reduction reproduces the PyTorch module to rounding error, and GroupNorm with one group or with one channel per group reproduces LN and IN. The two layer norms differ by a factor of 256 in the number of statistics: 4 (one per image) when nn.LayerNorm is given $(C, H, W)$, and 1,024 (one per pixel of each image) when it is applied to the channel vector at each position.

The second part, continuing the same script, puts the disc image in a different batch, then changes the disc’s feature map in three ways: one scale and shift for the whole map ($3x + 0.5$), a scale alone ($3x$), and a different scale and shift for each channel. For each method it prints the largest change in the disc’s normalized output. It ends with weight normalization of a convolution.

# ---- the same disc image in two different batches (continues the script above) ----
x_a = x                                                 # disc, square, horizontal, vertical stripes
x_b = features([image("disc"), image("disc", contrast=3.0),
                image("square", offset=2.0), image("hstripe", contrast=0.2)])
# ---- the disc's feature map with its contrast and brightness changed ----
x_c = x.copy(); x_c[0] = 3.0 * x[0] + 0.5                          # one scale and shift per image
x_d = x.copy(); x_d[0] = 3.0 * x[0]                                 # scale only
s, t = rng.uniform(0.5, 2.0, (C, 1, 1)), rng.uniform(-1, 1, (C, 1, 1))
x_e = x.copy(); x_e[0] = s * x[0] + t                               # a scale and shift per channel

def out0(m, batch):                     # normalized feature map of image 0 within a batch
    with torch.no_grad():
        return m(torch.from_numpy(batch)).numpy()[0]

print(f"{'max change in image 0':22s}{'other batch':>12s}{'a*x+b':>8s}{'a*x':>8s}{'per channel':>13s}")
for name, m in modules.items():
    if isinstance(m, nn.Module):
        m = m.double()
    ref = out0(m, x_a)
    d = [np.abs(out0(m, xb) - ref).max() for xb in (x_b, x_c, x_d, x_e)]
    print(f"{name:22s}" + "".join(f"{v:{w}.1e}" for v, w in zip(d, (12, 8, 8, 13))))

bn = modules["batch"].eval()                            # evaluation mode: running statistics
print("batch, eval mode      other batch: %.1e" % np.abs(out0(bn, x_b) - out0(bn, x_a)).max())

# ---- weight normalization reparameterizes a convolution's weights, not its outputs ----
torch.manual_seed(0)
conv = nn.utils.parametrizations.weight_norm(nn.Conv2d(1, 4, 3, padding=1).double())
g, v = conv.parametrizations.weight.original0, conv.parametrizations.weight.original1
print("weight norm g:", g.detach().flatten().numpy().round(4))
print("||w|| per filter:", conv.weight.detach().flatten(1).norm(dim=1).numpy().round(4))
img = torch.from_numpy(np.stack([image("disc"), image("vstripe")])[:, None])
with torch.no_grad():
    y1 = conv(img); v.mul_(10.0); y2 = conv(img)        # scale the direction parameters v by 10
print("output change after v *= 10: %.1e" % (y1 - y2).abs().max())

Output:

max change in image 0  other batch   a*x+b     a*x  per channel
batch                      2.2e+00 3.0e+00 2.9e+00      1.6e+00
layer                      0.0e+00 3.7e-06 3.7e-06      6.0e-01
instance                   0.0e+00 2.0e-05 2.0e-05      1.6e-05
group                      0.0e+00 1.4e-05 1.4e-05      8.0e-01
layer (C only)             0.0e+00 2.3e-05 2.3e-05      1.6e+00
rms                        0.0e+00 1.2e-01 3.5e-06      6.2e-01
batch, eval mode      other batch: 0.0e+00
weight norm g: [0.5125 0.4988 0.5706 0.6845]
||w|| per filter: [0.5125 0.4988 0.5706 0.6845]
output change after v *= 10: 1.1e-16

Only BN in training mode makes the disc’s output depend on its batch, and it changes in every column because altering one image moves the statistics of all of them. The other columns match the invariances above; the residuals of order $10^{-6}$ to $10^{-5}$ come from $\epsilon$, which is not rescaled. Weight normalization fixes $\lVert \mathbf{w} \rVert = g$ whatever the scale of $\mathbf{v}$, so multiplying $\mathbf{v}$ by 10 leaves the output unchanged.

Common Misconceptions

• “Layer normalization” names two different operations in vision. In Ba et al. and in Wu and He’s comparison, LN normalizes each image over all of $(C, H, W)$. In the Vision Transformer, LN is applied to each token, the feature vector of one patch, so it normalizes over channels separately at every position; ConvNeXt’s official code (its LayerNorm class with data_format="channels_first") computes the same per-position channel statistics on $(N, C, H, W)$ tensors, and its blocks permute to channels-last and normalize the last axis. The example shows the difference: 4 statistics against 1,024 for the same feature map.
• Batch-independent does not mean information-preserving. IN removes every channel’s mean and contrast from every image. This is the point in stylization and a cost in classification, where Wu and He measured IN 4.8 points worse than BN.
• Instance normalization is not batch normalization with a batch of one. At training time the computations are similar, as Wu and He note, but BN at inference uses running averages over the training set, while IN keeps normalizing each image by its own statistics.
• RMSNorm is not LN with a different denominator only. It removes the mean subtraction too, so it is not invariant to shifts (0.12 in the example) and matches LN only when the activations already have zero mean.
• Weight normalization does not fix the scale of activations. Because it ignores the data, Salimans and Kingma found it important to initialize $g$ and the biases from the statistics of one minibatch so that every layer’s pre-activations start with zero mean and unit variance on that minibatch (their §3).

Where It Is Used

• Convolutional classifiers trained with tens of images per device use BN, as in ResNet; see batch normalization and its paper.
• Detection and segmentation fit only one or two high-resolution images per GPU. Detectors such as Mask R-CNN then freeze BN into a fixed affine map. Wu and He replaced it with GN in a ResNet-50 FPN Mask R-CNN on COCO, fine-tuned with one image per GPU; it raised box AP on val2017 from 38.6 to 40.3 and mask AP from 34.5 to 35.7, and to 40.8 and 36.1 with longer training (their Table 6). Applying BN in the box head, whose 512 regions per image are not independent samples, was about 9 AP worse. With GN they also trained Mask R-CNN from scratch, reaching 39.5 box AP with ResNet-50.
• Video classification trades clip length against batch size. In Wu and He’s ResNet-50 I3D experiments on Kinetics-400 (10-clip validation accuracy), 64-frame clips at 4 clips per GPU gave 73.3% top-1 with BN and 74.5% with GN, both extended to normalize over time as well as space; see action recognition.
• Vision Transformers use pre-norm LN on every token. The ViT paper’s ResNet baselines (“ResNet (BiT)”) replaced BN with GN and used standardized convolutions, modifications that it says improve transfer.
• Modern convolutional networks have borrowed LN from transformers. In ConvNeXt (Liu et al., 2022), replacing BN by LN in the network modernized step by step from ResNet-50 gave 81.5% ImageNet-1K top-1 accuracy against 81.4% with BN; the authors note that substituting LN directly in the original ResNet is suboptimal.
• Style transfer and image generation use instance normalization. Ulyanov et al. replaced BN with IN, applied at test time as well, in feed-forward stylization networks and reported markedly better images. Huang and Belongie (2017) argued that IN normalizes style, carried by channel means and variances, and introduced adaptive instance normalization (AdaIN), which has no learned affine parameters and instead sets each channel’s mean and standard deviation to those of a style image. StyleGAN drives its generator with AdaIN, and StyleGAN2 replaced it with weight demodulation; SPADE modulates normalized features with a semantic map (image-to-image translation).
• RMSNorm was proposed for speed: in Zhang and Sennrich’s experiments, mostly on language tasks, it matched LN’s quality while reducing running time by 7–64%, depending on the model and implementation. In their CIFAR-10 experiment with a small convolutional network (Table 10), test error was 8.25% with BN, 8.28% with WN, 8.83% with RMSNorm, 10.49% with LN and 8.96% without normalization; the text puts RMSNorm’s margin over no normalization at 0.013%, where the table gives 0.13 points. Their Appendix A.4 (in the NeurIPS supplementary material and the arXiv version) says LN was applied there over the width and height dimensions.
• Weight normalization suits models where batch statistics are unwelcome, such as recurrent networks, reinforcement learning and generative models, its authors argue. On CIFAR-10 without data augmentation, they reported test errors of 8.46% with WN, 8.43% with the plain parameterization and 8.05% with BN; WN combined with a mean-only BN reached 7.31%. WN’s main benefit was speed: in those runs it sped up convergence over the plain parameterization nearly as much as BN, which made training about 16% slower in their implementation while WN did not.