Regularization

A network with hundreds of thousands of weights can memorize a few hundred training images outright. Regularization is everything that pushes training toward solutions that also work on unseen images: weight penalties, synthetic variations of the training images, injected noise, stopping early, and the biases of the optimizer itself. This hub surveys these methods and the evidence about how much they matter; the training workflow article covers how generalization is measured.

Definition

Goodfellow, Bengio and Courville (2016) define regularization, in §5.2.2 and again at the start of Chapter 7, as any change to a learning algorithm that is meant to reduce its generalization error but not its training error. The generalization error is the expected error on new data from the training distribution; its difference from the training error, the generalization gap, is estimated by the gap between validation and training performance.

Regularizers are often split into explicit ones, which change the objective or the data (a weight penalty, augmentation, label smoothing), and implicit ones, which are side effects of the training procedure (the choice of optimizer, its noise, the number of steps).

Intuition

Many weight settings fit a small training set perfectly. Some respond to the stroke shapes that distinguish a 3 from an 8; others also latch onto the exact pixel positions and noise of the particular training images. The training loss cannot choose between them. A regularizer adds a preference: for small weights, for answers that do not change when the image moves by a pixel, for less confident predictions. If the preference matches how real images vary, the chosen solution generalizes better.

Formal Definition

The most common explicit form adds a penalty $\Omega(\theta)$ to the empirical risk of the loss function article:

$$J(\theta) = \frac{1}{N}\sum_{i=1}^{N} \ell\big(f(\mathbf{x}_i; \theta), y_i\big) + \lambda\, \Omega(\theta).$$

L2 regularization uses $\Omega(\theta) = \tfrac{1}{2}\lVert \mathbf{w} \rVert_2^2$; Goodfellow et al. (§7.1) penalize only the weights and leave biases unregularized, because each bias controls a single variable and needs less data to fit. The penalty adds $\lambda\mathbf{w}$ to the gradient, so a gradient descent step with learning rate $\eta$ on the data loss $L$ becomes

$$\mathbf{w} \leftarrow (1 - \eta\lambda)\,\mathbf{w} - \eta\, \nabla_{\mathbf{w}} L,$$

which shrinks every weight by a constant factor before the usual step: hence the name weight decay. Near a minimum $\mathbf{w}^*$ of the unregularized loss with Hessian eigenvalues $h_i$, the penalty rescales the component of $\mathbf{w}^*$ along the $i$-th eigenvector by $h_i / (h_i + \lambda)$ (§7.1.1): directions the loss constrains strongly are barely changed, while weakly constrained ones shrink toward zero. In linear models the same penalty is ridge regression, the regularized least squares solution.

L1 regularization uses $\Omega(\theta) = \lVert \mathbf{w} \rVert_1$. Goodfellow et al. (§7.1.2) show that, unlike L2, it can drive some weights exactly to zero, a sparsity that has been used for feature selection.

Properties

Weight decay and the optimizer

For plain SGD, L2 regularization and weight decay are the same update, with decay $\eta\lambda$. Loshchilov and Hutter (2019) showed that this equivalence fails for adaptive methods such as Adam: the L2 gradient is divided by Adam’s per-weight scale, so weights with large past gradients are regularized less. Their AdamW applies the decay directly to the weights, outside the adaptive step, and they reported that this substantially improved Adam’s generalization on image classification. Weight decay can also matter for optimization: Krizhevsky et al. (2012) trained AlexNet with weight decay 0.0005 and found it reduced the training error, not just the test error.

Data augmentation

Goodfellow et al. (§7.4) call augmentation particularly effective for object recognition: translating training images by a few pixels can greatly improve generalization even for convolutional networks. The transformations must preserve the class; horizontal flips and 180° rotations are wrong for character recognition, where they confuse b with d and 6 with 9. AlexNet trained on random 224×224 crops of 256×256 images and their mirror images, without which, its authors reported, it overfitted substantially; perturbing the RGB colors along their principal components cut top-1 error by over 1% (Krizhevsky et al., 2012). In common practice, crops, flips and color jitter remain the default for natural images.

Later methods alter images more aggressively:

• Cutout (DeVries & Taylor, 2017) masks a random square of the input, 16×16 pixels for CIFAR-10 in their experiments.
• Mixup (Zhang et al., 2018) trains on convex combinations $\tilde{\mathbf{x}} = t\,\mathbf{x}_i + (1-t)\,\mathbf{x}_j$ of two images and the same combination of their one-hot labels, with $t \sim \mathrm{Beta}(\alpha, \alpha)$ (the paper calls the weight $\lambda$). The authors reported better generalization on ImageNet and CIFAR and less memorization of corrupted labels.
• CutMix (Yun et al., 2019) pastes a rectangle from one image into another and mixes the labels in proportion to the pasted area, keeping Cutout’s occlusion without wasting pixels on a blank patch.

Noise, normalization and soft targets

Dropout randomly zeroes units during training; AlexNet’s authors found that their network overfitted substantially without it in its first two fully connected layers. Batch normalization was designed to speed up training, but Zhang et al. (2017) observed that it also improved the generalization of their Inception network on CIFAR-10. Label smoothing replaces one-hot targets with slightly softened ones, penalizing overconfident logits; the cross-entropy loss article gives the formula and results.

Early stopping and pretraining

Early stopping, described in the training workflow article, returns the weights with the best validation error. For a linear model with a quadratic error trained by gradient descent, Goodfellow et al. (§7.8) show it is equivalent to L2 regularization, with the number of steps times the learning rate playing the role of the reciprocal of the weight-decay coefficient. In our reading, starting from weights pretrained on a large dataset (transfer learning) acts as a strong prior that keeps the weights near a solution learned from many images.

Rethinking generalization

Zhang et al. (2017) trained standard image classifiers on CIFAR-10 with the labels replaced by random ones. The networks reached zero training error, training only a small constant factor longer, with test accuracy at chance; they even fitted pure noise images. Inception V3 reached 95.20% top-1 training accuracy on ImageNet with random labels, and still about 90% with dropout and weight decay switched on. Conversely, with weight decay, dropout and augmentation all off, Inception V3 still reached 59.80% top-1 accuracy on true ImageNet labels, and on CIFAR-10 the gains from augmentation and weight decay were a few points. They concluded that explicit regularization can improve test performance but is “neither necessary nor by itself sufficient” for controlling generalization error.

Implicit regularization

Zhang et al. pointed to the optimizer instead. For a linear model with more parameters than examples, SGD started from zero stays in the span of the training inputs, so the interpolating solution it finds is the one with minimum L2 norm: a regularizer nobody wrote down. They cautioned that the norm alone did not predict generalization in their experiments. The SGD article summarizes the related evidence that small-batch gradient noise favors flatter minima. How these implicit biases work in deep networks remains an open research question.

Double descent

The classical picture is a U-shaped test error curve: test error falls as capacity grows, then rises once the model overfits. Belkin et al. (2019) proposed a double descent curve that continues past the interpolation threshold, where the model first fits the training set exactly: test error peaks there and then falls again as capacity grows further. They demonstrated it with minimum-norm random Fourier feature models on an MNIST subset, among others. Nakkiran et al. (2020) found model-wise double descent in ResNets and other networks on CIFAR-10 and CIFAR-100, most prominently with label noise, as well as epoch-wise double descent over training time. Data augmentation shifted the peak toward larger models, and varying weight decay produced a similar curve. Near the threshold, more capacity, data or training can hurt.

Summary

Method	Where applied	Typical setting in vision	Article
L2 / weight decay	Objective or update	$5 \times 10^{-4}$ for AlexNet with SGD; decoupled in AdamW	Adam, SGD
L1 penalty	Objective	Rare for whole networks; used for sparsity	This article
Crops, flips, color jitter	Input images	Default for natural-image classifiers	—
Cutout, Mixup, CutMix	Input images and labels	CIFAR and ImageNet classifiers	—
Dropout	Hidden units	Fully connected layers, as in AlexNet	Dropout
Batch normalization	Hidden activations	Common in convolutional networks	Batch normalization
Label smoothing	Targets	$\varepsilon = 0.1$ in Szegedy et al.’s ImageNet experiments	Cross-entropy loss
Early stopping	Training length	Patience on validation loss	Training
Pretraining	Initialization	ImageNet weights for small datasets	—

Examples

The script trains a fully connected network with 301,066 parameters on 200 of scikit-learn’s 8×8 digit images (20 per class) and tests it on 500 others. It compares no regularization, weight decay, and augmentation by random shifts of up to one pixel in each direction (flips would turn digits into other symbols), then fits shuffled labels. A second test set shifts each of the 500 test images once at random. Run with PyTorch 2.14.1, it takes a few seconds on a CPU.

import sys
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.manual_seed(SEED)

# 200 training images (20 per class) and 500 held-out test images, pixels in [0, 1].
digits = load_digits()
X = torch.tensor(digits.images / 16.0, dtype=torch.float32)        # (1797, 8, 8)
y = torch.tensor(digits.target)
i_tr, i_te = train_test_split(np.arange(len(y)), train_size=200, test_size=500,
                              random_state=SEED, stratify=digits.target)
X_tr, y_tr, X_te, y_te = X[i_tr], y[i_tr], X[i_te], y[i_te]

def shift(img, generator=None):
    """Move each 8x8 image by -1, 0 or +1 pixels in x and in y, filling with zeros."""
    padded = F.pad(img, (1, 1, 1, 1))                               # (B, 10, 10)
    dx, dy = torch.randint(0, 3, (2, len(img)), generator=generator)
    return torch.stack([p[r:r + 8, c:c + 8] for p, r, c in zip(padded, dy, dx)])

# A second test set: the same 500 test images, each shifted once at random.
X_te_shifted = shift(X_te, torch.Generator().manual_seed(1))

def evaluate(model, X, y):
    with torch.no_grad():
        logits = model(X.flatten(1))
    return (logits.argmax(1) == y).float().mean().item(), F.cross_entropy(logits, y).item()

def run(name, labels, weight_decay=0.0, augment=False, epochs=150, B=50):
    model = nn.Sequential(nn.Linear(64, 512), nn.ReLU(),
                          nn.Linear(512, 512), nn.ReLU(), nn.Linear(512, 10))
    opt = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9,
                          weight_decay=weight_decay)              # adds weight_decay * theta to the gradient
    for epoch in range(epochs):
        for b in torch.randperm(len(labels)).split(B):
            xb = shift(X_tr[b]) if augment else X_tr[b]             # a fresh random shift every time
            loss = F.cross_entropy(model(xb.flatten(1)), labels[b])
            opt.zero_grad()
            loss.backward()
            opt.step()
    tr_acc, _ = evaluate(model, X_tr, labels)
    te_acc, te_loss = evaluate(model, X_te, y_te)
    sh_acc, _ = evaluate(model, X_te_shifted, y_te)
    norm = torch.sqrt(sum((p ** 2).sum() for p in model.parameters()))
    print(f"{name:22s} {tr_acc:6.3f} {te_acc:6.3f} {te_loss:6.3f} {sh_acc:8.3f} {norm:7.1f}")

n_params = sum(p.numel() for p in nn.Sequential(nn.Linear(64, 512), nn.Linear(512, 512),
                                                 nn.Linear(512, 10)).parameters())
print(f"torch {torch.__version__}; {n_params:,} parameters, {len(y_tr)} training images\n")
print(f"{'':22s} {'train':>6s} {'test':>6s} {'test':>6s} {'shifted':>8s} {'weight':>7s}")
print(f"{'':22s} {'acc':>6s} {'acc':>6s} {'loss':>6s} {'test acc':>8s} {'norm':>7s}")
run("no regularization", y_tr)
run("weight decay 3e-3", y_tr, weight_decay=3e-3)
run("1-pixel shifts", y_tr, augment=True)
run("shifts + weight decay", y_tr, weight_decay=3e-3, augment=True)
run("random labels", y_tr[torch.randperm(len(y_tr))], epochs=500)

Output:

torch 2.14.1; 301,066 parameters, 200 training images

                        train   test   test  shifted  weight
                          acc    acc   loss test acc    norm
no regularization       1.000  0.934  0.338    0.358    21.7
weight decay 3e-3       1.000  0.932  0.257    0.360    12.2
1-pixel shifts          1.000  0.926  0.224    0.906    25.9
shifts + weight decay   0.995  0.922  0.249    0.900    16.3
random labels           1.000  0.094 10.300    0.086    28.7

With 1,500 parameters per training image, the unregularized network fits every training image yet scores 93.4% on the centered test images, a small version of Zhang et al.’s observation. Weight decay nearly halves the weight norm and lowers the test loss from 0.338 to 0.257, through less confident wrong answers, but leaves test accuracy unchanged. The shifted test set separates the methods: the network trained on centered digits drops to 35.8%, because a fully connected layer has no notion that a stroke one pixel to the right is the same stroke, while the shift-trained network keeps 90.6%. On shuffled labels the network again fits the training set perfectly, at chance on the test set.

Over six seeds the pattern held: centered test accuracy was 87–94% for every method, weight decay lowered the test loss in all six runs, and shifted-test accuracy was 36–39% without augmentation against 89–93% with it. The random-label runs always fitted the training set, with test accuracy of 9–18%. The network size, learning rate, number of epochs and weight decay were hand-picked for illustration, not tuned on a validation set; our reading is that augmentation helps most when it teaches an invariance the test images need.

Common Misconceptions

• “Regularization means a penalty term.” Augmentation, early stopping, dropout, pretraining and the optimizer itself also regularize, and in vision augmentation often matters more than penalties, as Zhang et al.’s ImageNet results suggest.
• “An over-parameterized network must overfit.” Networks that can memorize random labels still generalize well on real labels; the parameter count alone does not predict test error.
• “Weight decay and L2 regularization are the same.” Only for plain SGD; with Adam they differ, which is why AdamW exists.
• “Any transformation is a valid augmentation.” It must preserve the label; flipped digits and rotated characters change the class.
• “More regularization is always safer.” Too strong a penalty or too aggressive an augmentation underfits, as Zhang et al. (2018) reported for Mixup with large $\alpha$ on ImageNet.

Where It Is Used

Trained vision models typically combine several regularizers. Image classifiers use crop-and-flip augmentation, weight decay with SGD or Adam, and often label smoothing in the cross-entropy loss; detection and segmentation networks commonly start from pretrained backbones, with geometric augmentations that transform boxes and masks along with the images. Classical vision uses the same idea in another form: the smoothness terms of variational methods such as Horn–Schunck optical flow are Tikhonov regularizers, the image-domain relatives of weight decay.