Weight Decay

Weight decay multiplies a network’s weights by a factor slightly below one at every training step, so that weights the loss does not actively support drift toward zero. It is part of standard training recipes from ResNets trained with SGD to Vision Transformers trained with AdamW, usually as one number in the optimizer’s call. That number means different things in different optimizers. For plain SGD, decaying the weights is the same as adding an L2 penalty to the loss; for Adam it is not, and the difference changes results. In networks with normalization layers, most weights do not affect the output through their scale at all, and there decay acts mainly on the effective learning rate rather than as a penalty. This article covers these mechanisms, how weight decay is set in practice, which parameters are left out, and the L1 penalty; Regularization places weight penalties among the other regularizers.

Purpose

The classical purpose is a preference for small weights. Hanson and Pratt (1988) described adding a decay term to the delta rule that shrinks every weight geometrically at each step, and noted that the same update follows from a quadratic cost on the weights. Krogh and Hertz (1991) proved that for a linear network the decay suppresses weight components the training data do not determine and, with the right strength, part of the effect of noise on the targets. Read this way, weight decay is the L2 penalty of the regularization article, and in a Bayesian reading it is the MAP estimate under a Gaussian prior on the weights (Goodfellow et al., 2016, §5.6.1).

Later work found other roles. Zhang, Wang, Xu and Grosse (2019; read in arXiv v1) list three: in networks with batch normalization trained by SGD or Adam, decay raises the effective learning rate; with the second-order optimizer K-FAC it approximately regularizes the network’s input–output Jacobian; and it keeps K-FAC’s damping term from swamping the curvature. Which role dominates depends on the optimizer and the architecture.

When to Use It

• Training from scratch in image classification and other vision tasks. Every recipe in the table under How to Apply It sets a nonzero weight decay.
• With adaptive optimizers, use the decoupled form (AdamW) rather than an L2 term passed to Adam, for the reasons below.
• With normalization layers, expect decay on the normalized layers to act through the learning rate, so tune it together with the learning rate and schedule.
• Less clearly in fine-tuning: decay toward zero pulls the weights away from the pretrained ones, and that article covers penalties toward the starting point instead and recipes that drop decay altogether.

How It Works

The L2 penalty and the SGD update

With data loss $L(\theta)$ and the penalty $\tfrac{\lambda'}{2}\lVert\theta\rVert_2^2$, a gradient step with learning rate $\eta$ is

$$\theta_{t+1} = \theta_t - \eta\big(\nabla L(\theta_t) + \lambda'\theta_t\big) = (1 - \eta\lambda')\,\theta_t - \eta\,\nabla L(\theta_t).$$

Weight decay in Hanson and Pratt’s sense applies the shrink factor directly, $\theta_{t+1} = (1 - \lambda)\theta_t - \eta\nabla L(\theta_t)$. Loshchilov and Hutter (2019, Proposition 1) state the familiar consequence: for plain SGD the two are the same update when $\lambda' = \lambda/\eta$. The equivalence has a catch they emphasize. If one decay rate per step $\lambda$ is what matters, then the best L2 coefficient $\lambda'$ depends on the learning rate, and changing $\eta$ alone also changes the regularization. In their CIFAR-10 grids for a ResNet trained with SGD and an L2 term, the best settings lay along a diagonal of learning rate against $\lambda'$ (their Figure 2).

Momentum breaks even this equivalence. With an L2 term the penalty gradient enters the momentum buffer, $b_{t+1} = \mu b_t + \nabla L + \lambda'\theta_t$, and is applied again on later steps; by our reasoning, when the weights change slowly the per-step shrink approaches $\eta\lambda'/(1-\mu)$, ten times $\eta\lambda'$ for $\mu = 0.9$. Loshchilov and Hutter’s SGDW applies the decay outside the buffer instead (their Algorithm 1). The example below shows that the two then follow different trajectories.

Decoupled weight decay under Adam

Adam divides each coordinate of its averaged gradient by the root of its averaged squared gradient. Passed through Adam, an L2 term $\lambda'\theta$ is divided by the same per-weight scale as the loss gradient. Loshchilov and Hutter prove (Proposition 2) that for any optimizer whose preconditioner is not a multiple of the identity, no L2 coefficient reproduces weight decay. Their AdamW instead applies the decay outside the adaptive step:

$$\theta_t = \theta_{t-1} - \eta_t\left(\alpha\,\frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon} + \lambda\,\theta_{t-1}\right),$$

where $\alpha$ is Adam’s step size and $\eta_t$ a schedule multiplier that scales the decay too. The Adam article gives the moment estimates and the paper’s reported gains.

To see what the L2 form does instead, take a fixed diagonal preconditioner with entries $1/s_i$, where $s_i$ is weight $i$’s typical gradient magnitude. Loshchilov and Hutter’s Proposition 3 shows that decoupled decay then equals an L2 penalty in which each weight is weighted by its $s_i$; by our derivation the penalty is $\frac{\lambda}{2\alpha}\sum_i s_i\theta_i^2$, as in their proof (Appendix A), while the proposition as printed carries an extra factor $1/\alpha$. Seen from the other side, an ordinary L2 term under Adam regularizes weights with large gradients relatively less. By our reading it regularizes weights with tiny loss gradients relatively more: for such a weight the normalized penalty gradient dominates and moves it toward zero by up to about $\alpha$ per step, as the Adam article’s example shows for pixels that are zero in every training image. The paper’s Section 3 adds a Bayesian-filtering justification, due to Aitchison, in which decoupled decay appears naturally in the Gaussian state-transition model that carries the weights from one step to the next.

Weight decay with normalization layers

A convolution followed by batch normalization (or layer, group or instance normalization) computes the same function if its kernel $w$ is multiplied by any $c > 0$, because the normalization divides the scale out. Van Laarhoven (2017) drew the conclusion: an L2 penalty on such weights has no regularizing effect on the function, since any value of $\lambda$ can be absorbed by rescaling. The scale still matters to the optimizer. For a loss with $L(cw) = L(w)$, two identities hold (Li & Arora, 2020, Lemma 1.3):

$$\langle \nabla L(w), w \rangle = 0, \qquad \nabla L(cw) = \frac{1}{c}\,\nabla L(w).$$

The gradient is orthogonal to the weights and shrinks as they grow. Writing $w = \lVert w\rVert\,\hat w$, an SGD step of size $\eta$ turns the direction $\hat w$ by about $\eta/\lVert w\rVert^2$ times the gradient of the unit-norm problem, so van Laarhoven calls $\eta/\lVert w\rVert_2^2$ the effective learning rate (Zhang et al. reach the same from an update rule of Hoffer et al., their Eq. 9). Two consequences follow, the first by our arithmetic from the orthogonality:

• Without decay, plain SGD gives $\lVert w_{t+1}\rVert^2 = \lVert w_t\rVert^2 + \eta^2\lVert\nabla L\rVert^2$: the norm can only grow, so the effective learning rate falls on its own, an implicit learning-rate decay.
• With decay, shrinkage and growth balance. In van Laarhoven’s simple model of the stationary phase, the norm settles at a value proportional to $(\eta/\lambda)^{1/4}$ for SGD. Squaring it, the equilibrium effective learning rate $\eta/\lVert w\rVert^2$ scales as $\sqrt{\eta\lambda}$ (our arithmetic; the paper prints $\sqrt{\eta/\lambda}$ in the text and in Table 1, which contradicts its next sentence, that choosing $\eta \propto 1/\lambda$ keeps the effective rate constant). For Adam and RMSProp his Table 1 gives an effective rate of $\eta/\lVert w\rVert$ instead and a norm proportional to $(\eta/\lambda)^{1/3}$; the equilibrium rate printed there, $\sqrt[3]{\eta^2/\lambda}$, has the same inversion, and $\sqrt[3]{\eta^2\lambda}$ follows (our arithmetic).

Zhang et al. (2019) tested this account. In ResNet32 and VGG16 on CIFAR-100, decay on the convolutional layers, which are followed by batch normalization, produced nearly all of its generalization effect, while decay on the final fully connected layer alone mattered little (their Figure 2). Training without decay but rescaling each layer’s weights after every epoch to the norm of a decayed run, which leaves the predictions unchanged and only transfers the effective learning rate, closed almost the whole gap (Figure 4). They concluded that for batch-normalized networks trained with SGD or Adam, weight decay works mainly through the effective learning rate.

Li and Arora (2020) made the link exact. For a scale-invariant loss, SGD with learning rate $\eta$ and weight decay $\lambda$, without momentum, produces the same sequence of networks, up to a rescaling of the scale-invariant weights, as SGD without decay whose learning rate grows exponentially, $\tilde\eta_t = (1 - \lambda\eta)^{-2t-1}\eta$ (their Theorem 1.1, which also covers momentum; Theorem 2.12 extends it to multi-phase schedules). They trained a PreResNet32 on CIFAR-10 with such an exponentially increasing schedule. Their Lemma 2.7 implies that in such networks rescaling the initialization is equivalent to changing $\eta$ at fixed $\eta\lambda$, which in their reading is why training with normalization and decay is insensitive to initialization scale. In a toy example of theirs, a last linear layer trained with logistic loss, SGD with either normalization or decay alone reached the global minimum, but with both it failed to converge to it.

The parameters that are not scale-invariant (the final classifier, biases and the normalization layers’ own scales and shifts) still see decay as a true penalty.

How to Apply It

In PyTorch

As of PyTorch 2.14, the weight_decay argument means two different things (PyTorch documentation):

• torch.optim.SGD(weight_decay=λ) adds $\lambda\theta$ to the gradient before momentum: the L2 form.
• torch.optim.Adam(weight_decay=λ) also adds $\lambda\theta$ to the gradient, an L2 penalty, unless decoupled_weight_decay=True is passed. Its default weight_decay is 0.
• torch.optim.AdamW(weight_decay=λ) is Adam with decoupled_weight_decay=True. Each step first sets $\theta \leftarrow \theta - \gamma\lambda\theta$, with $\gamma$ the current learning rate, then takes the Adam step. Its default $\lambda$ is $10^{-2}$.

PyTorch’s decay is multiplied by the learning rate $\gamma$, where the paper multiplies it only by the schedule factor $\eta_t$. With PyTorch’s form, changing the learning rate at fixed weight_decay changes the per-step decay $\gamma\lambda$ in proportion, and a learning-rate scheduler anneals the decay along with the rate; with the paper’s form, $\lambda$ is a per-step decay independent of $\alpha$ (our comparison of the two formulas). A $\lambda$ quoted from the paper or from another library should be converted, not copied.

Which parameters to decay

model.parameters() hands every parameter to the optimizer, so by default biases, normalization scales and shifts, and transformer embeddings are all decayed. Common recipes exclude some of them through parameter groups:

• Goyal et al. (2017) used a weight decay of $10^{-4}$ for their large-batch ResNet-50 and, following the ResNet paper as they state, did not decay the batch-normalization scale and shift.
• He, T. et al. (2019) describe “no bias decay”: decay only the weights of convolutional and fully connected layers, and leave biases and the batch-normalization $\gamma$ and $\beta$ unregularized. In their own ablation on ResNet-50 it did not raise ImageNet top-1 accuracy: adding it moved validation accuracy from 76.19% to 76.16% at batch size 256 and from 76.37% to 76.03% at batch size 1,024 (Table 4, the same in the CVPR and arXiv versions).
• MAE’s fine-tuning code exempts every one-dimensional parameter, which covers biases and normalization scales and shifts, plus a model-specific list from the network (util/lr_decay.py).
• torchvision’s classification reference script (release v0.29.2) has separate options for normalization layers, biases, and transformer embeddings (the class token, position embeddings and relative position bias tables), each defaulting to the main value; its recipes set the normalization decay to 0 for models including EfficientNet-V2, ViT-L/16, ConvNeXt and ShuffleNet V2, and all three to 0 for Swin Transformer.

Goodfellow et al. (§7.1) give the classical reason for sparing biases, which the regularization article summarizes. For the normalization parameters, the scale-invariance analysis above adds that decaying them is a real penalty on the function rather than a change of effective learning rate (our reading).

Typical values

Recipes from torchvision’s classification reference README (release v0.29.2), for ImageNet-trained models, with the per-step shrink at the peak learning rate computed by us:

Model	Optimizer	Peak learning rate	weight_decay	Not decayed	Peak shrink per step
ResNet (script defaults)	SGD, momentum 0.9	0.1	$10^{-4}$	—	$10^{-5}$
MobileNetV2	SGD, momentum 0.9	0.045	$4 \times 10^{-5}$	—	$1.8 \times 10^{-6}$
EfficientNet-V2	SGD, momentum 0.9	0.5	$2 \times 10^{-5}$	normalization	$10^{-5}$
ViT-B/16	AdamW	0.003	0.3	—	$9 \times 10^{-4}$
ConvNeXt	AdamW	0.001	0.05	normalization	$5 \times 10^{-5}$
Swin Transformer	AdamW	0.001	0.05	normalization, biases, embeddings	$5 \times 10^{-5}$

The script passes the same --weight-decay value to SGD, where it is an L2 coefficient, and to AdamW, where it is decoupled, which is why the AdamW values are 500 to 15,000 times larger. For SGD the shrink per step is $\eta\lambda$ before momentum amplifies it. These are torchvision’s reproduction recipes; the original papers may use other values.

Interplay with the schedule

In both of PyTorch’s forms the decay follows the learning rate, so over a run the weights are multiplied by about $\prod_t (1 - \eta_t\lambda) \approx \exp(-\lambda\sum_t \eta_t)$ (our approximation, ignoring the loss gradient): a longer run or a higher peak rate decays more at the same $\lambda$. Loshchilov and Hutter found that the best decay shrinks as training gets longer and proposed normalizing it by the number of batch passes, a rule the learning-rate schedule article gives. For normalized layers, Li and Arora’s Lemma 2.7 and van Laarhoven’s equilibrium both point to the product $\eta\lambda$ as the quantity that matters, so lowering the learning rate during a schedule also lowers the decay’s effect on those layers (our reading).

Example

The script trains a small convolutional network on scikit-learn’s 8×8 digit images, 300 for training and 1,497 for testing. It runs in float64, so that rounding does not blur the comparison, and every run uses the same initialization and batch order. The first part compares L2 and decoupled decay with one coefficient under plain SGD, SGD with momentum and Adam, then sweeps the coefficient. It was run with PyTorch 2.14.1 and takes about 15 seconds on a laptop CPU.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

torch.set_num_threads(1)
torch.set_default_dtype(torch.float64)        # float64, so rounding does not hide the comparison

digits = load_digits()                        # 1,797 grayscale 8x8 images, pixel values 0..16
X = torch.tensor(digits.images / 16.0).unsqueeze(1)          # (1797, 1, 8, 8)
y = torch.tensor(digits.target)
i_tr, i_te = train_test_split(np.arange(len(y)), train_size=300, random_state=0,
                              stratify=digits.target)
X_tr, y_tr, X_te, y_te = X[i_tr], y[i_tr], X[i_te], y[i_te]

def make_cnn():
    return nn.Sequential(
        nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),     # 16 x 4 x 4
        nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),    # 32 x 2 x 2
        nn.Flatten(), nn.Linear(128, 10))

def train(opt_name, mode, wd, seed=0, epochs=60, B=30, lr=None, momentum=0.0):
    """mode: 'none', 'L2' (wd * theta added to the gradient by the optimizer)
    or 'decoupled' (theta <- (1 - lr * wd) * theta, applied outside the optimizer)."""
    torch.manual_seed(seed)
    model = make_cnn()
    params = list(model.parameters())
    l2 = wd if mode == "L2" else 0.0
    if opt_name == "SGD":
        lr = lr or 0.1
        opt = torch.optim.SGD(params, lr=lr, momentum=momentum, weight_decay=l2)
    elif mode == "decoupled":
        lr = lr or 1e-3
        opt = torch.optim.AdamW(params, lr=lr, weight_decay=wd)     # PyTorch's decoupled form
    else:
        lr = lr or 1e-3
        opt = torch.optim.Adam(params, lr=lr, weight_decay=l2)      # weight_decay = L2 penalty
    g = torch.Generator().manual_seed(seed)                         # same batches for every run
    for epoch in range(epochs):
        for b in torch.randperm(len(y_tr), generator=g).split(B):
            loss = F.cross_entropy(model(X_tr[b]), y_tr[b])
            opt.zero_grad()
            loss.backward()
            if opt_name == "SGD" and mode == "decoupled":
                with torch.no_grad():
                    for p in params:
                        p.mul_(1 - lr * wd)                         # decay first, then the loss step
            opt.step()
    with torch.no_grad():
        test_acc = (model(X_te).argmax(1) == y_te).double().mean().item()
        train_loss = F.cross_entropy(model(X_tr), y_tr).item()
    w_norm = torch.sqrt(sum((p ** 2).sum() for p in params)).item()
    return model, test_acc, train_loss, w_norm

def max_diff(m1, m2):
    return max((p - q).abs().max().item() for p, q in zip(m1.parameters(), m2.parameters()))

print(f"torch {torch.__version__}; {sum(p.numel() for p in make_cnn().parameters()):,} parameters, "
      f"{len(y_tr)} training and {len(y_te)} test images")

# 1. Same coefficient, two implementations: do the weights coincide?
print("\nmax |theta(L2) - theta(decoupled)| after 600 steps, same coefficient")
for opt_name, wd, mom in [("SGD", 1e-2, 0.0), ("SGD", 1e-2, 0.9), ("Adam", 1e-2, 0.0)]:
    a, *_ = train(opt_name, "L2", wd, momentum=mom)
    b, *_ = train(opt_name, "decoupled", wd, momentum=mom)
    label = f"{opt_name}{' + momentum 0.9' if mom else ''}, wd {wd:g}"
    print(f"  {label:28s} {max_diff(a, b):.1e}")

# 2. A sweep of the coefficient: test accuracy and weight norm for each rule.
wds = [0, 1e-3, 1e-2, 3e-2, 1e-1, 3e-1, 1, 3]
cols = [("SGD", "L2"), ("Adam", "L2"), ("Adam", "decoupled")]   # SGD decoupled = SGD L2
print("\ntest accuracy / weight norm (SGD lr 0.1, Adam lr 0.001, 60 epochs)")
print(f"{'wd':>7s}" + "".join(f"{o + ' ' + m:>18s}" for o, m in cols))
for wd in wds:
    cells = []
    for o, m in cols:
        _, acc, _, norm = train(o, m if wd else "none", wd)
        cells.append(f"{acc:.3f} / {norm:5.2f}")
    print(f"{wd:>7g}" + "".join(f"{c:>18s}" for c in cells))

Output:

torch 2.14.1; 6,090 parameters, 300 training and 1497 test images

max |theta(L2) - theta(decoupled)| after 600 steps, same coefficient
  SGD, wd 0.01                 3.3e-15
  SGD + momentum 0.9, wd 0.01  6.6e-01
  Adam, wd 0.01                4.4e-01

test accuracy / weight norm (SGD lr 0.1, Adam lr 0.001, 60 epochs)
     wd            SGD L2           Adam L2    Adam decoupled
      0     0.939 /  9.47     0.935 / 10.67     0.935 / 10.67
  0.001     0.940 /  9.20     0.934 /  9.92     0.935 / 10.67
   0.01     0.939 /  7.54     0.931 /  7.56     0.935 / 10.64
   0.03     0.925 /  6.04     0.914 /  5.97     0.935 / 10.58
    0.1     0.102 /  0.04     0.842 /  4.26     0.935 / 10.40
    0.3     0.102 /  0.02     0.102 /  0.05     0.933 /  9.92
      1     0.102 /  0.01     0.102 /  0.05     0.928 /  8.66
      3     0.102 /  0.01     0.099 /  0.05     0.908 /  6.68

Under plain SGD the two implementations end within $3 \times 10^{-15}$ of each other after 600 steps, the float64 rounding of computing the same update in two orders; the sweep therefore shows one SGD column. Add momentum, or switch to Adam, and the same coefficient gives a different network.

The sweep shows how differently the same number acts. Under SGD, $\lambda = 0.1$ shrinks the weights by $\eta\lambda = 1\%$ per step, faster than the loss gradient can restore them, and the network collapses to a constant prediction, whose accuracy is one class’s share of the test set (152 of 1,497 images, 10.2%). Adam with an L2 term collapses one step later, at 0.3; in our reading, once the penalty dominates a weight’s gradient, every normalized step moves that weight by about $\alpha$ toward zero. AdamW shrinks by only $\alpha\lambda = 10^{-3}\lambda$ per step: to reduce the weight norm as much as Adam’s L2 term at $\lambda = 0.001$ does (to 9.92), it needs $\lambda = 0.3$. No setting raised test accuracy by more than 0.001 over no decay here; with 300 images and no augmentation, the example shows the mechanics, not a benefit.

The second part, continuing the script, puts batch normalization after both convolutions, checks scale invariance on one batch, then trains with plain SGD at a learning rate of 0.3 and decays only the two convolution kernels, through parameter groups. It takes about 15 seconds.

# 3. Batch normalization: a scale-invariant layer, and what weight decay does to it.
def make_bn_cnn():
    return nn.Sequential(
        nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(16, 32, 3, padding=1, bias=False), nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Linear(128, 10))

torch.manual_seed(0)
net = make_bn_cnn()
W = net[4].weight                                   # second convolution, followed by BatchNorm2d
def out_and_grad():
    net.zero_grad()
    out = net(X_tr[:30])                            # training mode: batch statistics
    F.cross_entropy(out, y_tr[:30]).backward()
    return out.detach(), W.grad.clone()
out1, g1 = out_and_grad()
with torch.no_grad():
    W.mul_(10.0)
out10, g10 = out_and_grad()
with torch.no_grad():
    W.div_(10.0)
cos = (g1 * W).sum() / (g1.norm() * W.norm())
print(f"\nconv weights x10: max output change {(out10 - out1).abs().max():.1e}, "
      f"gradient norm ratio {g10.norm() / g1.norm():.4f}, cos(gradient, weights) {cos:.1e}")

def train_bn(wd, seed=0, epochs=60, B=30, lr=0.3):
    """Plain SGD; weight decay (L2 = decoupled here) on the two convolution kernels only."""
    torch.manual_seed(seed)
    model = make_bn_cnn()
    convs = [model[0].weight, model[4].weight]
    rest = [p for p in model.parameters() if all(p is not c for c in convs)]
    opt = torch.optim.SGD([{"params": convs, "weight_decay": wd},
                           {"params": rest, "weight_decay": 0.0}], lr=lr)
    g = torch.Generator().manual_seed(seed)
    history, grew = {0: model[4].weight.norm().item()}, True
    for epoch in range(1, epochs + 1):
        model.train()
        for b in torch.randperm(len(y_tr), generator=g).split(B):
            before = model[4].weight.norm().item()
            loss = F.cross_entropy(model(X_tr[b]), y_tr[b])
            opt.zero_grad()
            loss.backward()
            opt.step()
            grew &= model[4].weight.norm().item() >= before
        if epoch in (1, 5, 20, 60):
            history[epoch] = model[4].weight.norm().item()
    model.eval()                                    # running statistics, as at test time
    with torch.no_grad():
        acc = (model(X_te).argmax(1) == y_te).double().mean().item()
    return history, grew, acc

print("\nBN network, plain SGD lr 0.3, seed 0: ||W_conv2|| (lr / ||W_conv2||^2) after epoch")
print(f"{'wd on convs':>11s}" + "".join(f"{e:>16d}" for e in (0, 1, 5, 20, 60)))
for wd in [0.0, 1e-3, 1e-2, 3e-2]:
    history, grew, _ = train_bn(wd)
    cells = "".join(f"{n:7.2f} ({0.3 / n ** 2:.3f})" for n in history.values())
    print(f"{wd:>11g}" + cells + ("  norm grew at every step" if grew else ""))

print("\ntest accuracy over seeds 0-4 (eval mode)")
for wd in [0.0, 1e-3, 1e-2, 3e-2]:
    accs = [train_bn(wd, seed=s)[2] for s in range(5)]
    print(f"{wd:>11g}  median {np.median(accs):.3f}, range {min(accs):.3f}-{max(accs):.3f}")

Output:

conv weights x10: max output change 8.6e-05, gradient norm ratio 0.1000, cos(gradient, weights) 3.3e-06

BN network, plain SGD lr 0.3, seed 0: ||W_conv2|| (lr / ||W_conv2||^2) after epoch
wd on convs               0               1               5              20              60
          0   3.27 (0.028)   4.08 (0.018)   4.48 (0.015)   4.54 (0.015)   4.55 (0.015)  norm grew at every step
      0.001   3.27 (0.028)   4.08 (0.018)   4.42 (0.015)   4.29 (0.016)   3.81 (0.021)
       0.01   3.27 (0.028)   3.98 (0.019)   3.96 (0.019)   2.61 (0.044)   4.05 (0.018)
       0.03   3.27 (0.028)   3.79 (0.021)   3.29 (0.028)   1.43 (0.147)   1.73 (0.101)

test accuracy over seeds 0-4 (eval mode)
          0  median 0.963, range 0.958-0.975
      0.001  median 0.961, range 0.960-0.973
       0.01  median 0.957, range 0.751-0.965
       0.03  median 0.961, range 0.155-0.968

Multiplying the kernel by 10 changes the outputs only through batch normalization’s $\epsilon$, which is not rescaled; the gradient shrinks by exactly 10 and is orthogonal to the kernel, as the two identities predict. Without decay the second kernel’s norm grew at every one of the 600 steps, from 3.27 to 4.55, and the effective learning rate fell by almost half while the nominal rate stayed at 0.3. With decay the norm turns downward, and at $\lambda = 0.03$ it reaches 1.43 by epoch 20, raising the effective rate fivefold over its starting value; it then rises again as the larger steps add more norm than the decay removes, the balance van Laarhoven’s model describes. Decay barely changed the median test accuracy across five seeds, but at the two larger values at least one seed ended far below the others (minimum 75.1% and 15.5%), which in our reading is training destabilized by too large an effective learning rate. On the batch-normalized layers, the decay coefficient acted here as a learning-rate setting.

Variants

• L2 penalty in the loss. The coupled form: what weight_decay means in PyTorch’s SGD, Adam and RMSprop.
• Decoupled decay: SGDW and AdamW (Loshchilov & Hutter, 2019). The decay is applied to the weights directly. PyTorch implements AdamW, and the same switch, decoupled_weight_decay=True, in Adam, NAdam and RAdam; for SGD it can be applied by hand, as in the example.
• Normalized weight decay (Loshchilov & Hutter, 2019, Appendix B.1) divides $\lambda$ by the square root of the number of batch passes, so that one value serves runs of different lengths.
• Decay toward the pretrained weights (L2-SP) replaces $\lVert\theta\rVert^2$ with $\lVert\theta - \theta_0\rVert^2$ for fine-tuning; the fine-tuning article describes it.
• L1 penalty. $\lambda\lVert\theta\rVert_1$ adds $\lambda\,\mathrm{sign}(\theta)$ to the gradient, a constant pull toward zero rather than a proportional one, and corresponds to MAP estimation with a Laplace prior (Goodfellow et al., §7.1.2). It is the classical route to sparse weights, but plain SGD rarely produces them: a weight reaches zero only if a step happens to land on it exactly (Tsuruoka, Tsujii & Ananiadou, 2009). Their paper describes an alternative due to Carpenter (2008), in which the penalty is applied after each gradient step and the weight is clipped at zero when it would cross it, and adds a cumulative penalty to counter gradient noise.

Best Practices

• Use AdamW, not Adam with weight_decay, when training with Adam; in PyTorch 2.14 the two are different algorithms.
• Tune the learning rate and weight decay together, on a grid that covers both; with batch-normalized layers they act mainly through their product, and in PyTorch’s AdamW the per-step decay is the product of the two.
• Re-tune the decay when the training length changes, since its total effect grows with the summed learning rate.
• Build parameter groups deliberately. Decide whether biases, normalization parameters and embeddings are decayed, and record the choice with the decay value, since recipes differ.

Pitfalls

• Adam’s weight_decay is an L2 penalty. Setting it to a typical AdamW value such as 0.05 gives a very different, much stronger regularization; in the example, Adam with an L2 coefficient of 0.3 collapsed where AdamW was barely affected.
• Copying SGD values to AdamW or the reverse. The same number is a gradient coefficient in one and a shrink factor scaled by the learning rate in the other.
• Assuming decay regularizes a batch-normalized layer. It changes that layer’s effective learning rate; too much decay can raise it far enough to destabilize training, as in the example.
• Decaying everything by default. Passing model.parameters() decays normalization parameters and embeddings along with the weights.
• Expecting L1 under SGD to produce exact zeros. It needs clipping or a proximal step to do so.

Evidence and Impact

• Decoupling. Loshchilov and Hutter found that with Adam, an L2 term gave no better result than no regularization in their CIFAR-10 grids, while AdamW reached settings comparable to SGD with momentum and made the best decay largely independent of the learning rate (Figure 2). The Adam article reports their relative improvement in test error. The AdamW recipes in the table above use the decoupled form.
• Across optimizers. Zhang et al. (2019) found decay improved test accuracy for every optimizer and network they tried, and decoupled decay beat the L2 form wherever the two differ (their Figure 1). For a ResNet32 on CIFAR-10 with batch normalization and data augmentation, decay raised test accuracy from 92.95% to 95.14% with SGD and from 93.63% to 94.66% with Adam (Table 1, arXiv version). They found the largest gains with K-FAC when batch normalization was disabled.
• Exclusions. Excluding biases and normalization parameters is common in the recipes above, but the published ablation above found no accuracy gain from it on ResNet-50.