Decoupled Weight Decay Regularization

“Decoupled Weight Decay Regularization” is a paper by Ilya Loshchilov and Frank Hutter of the University of Freiburg. It was first posted to arXiv on 14 November 2017 under the title “Fixing Weight Decay Regularization in Adam”, revised twice, and presented as a poster at the International Conference on Learning Representations (ICLR) in May 2019. Its argument is short: for plain stochastic gradient descent, adding an L2 penalty to the loss and shrinking the weights directly are the same update, but for adaptive optimizers such as Adam they are not, and the deep-learning libraries of the time implemented the penalty while calling it weight decay. The authors proposed applying the decay to the weights outside the gradient step, which gave the variants they named AdamW and SGDW, and showed on CIFAR-10 and a 32×32 version of ImageNet that the change made Adam’s best learning rate and decay more independent and its test error lower. All its experiments are image classification with residual networks, the setting in which, as the paper notes, SGD with momentum still gave the best results. This article covers the publication; the technique as used today, with its derivations and a worked example, is in the weight decay article.

Problem

By 2017, adaptive methods such as AdaGrad, RMSProp and Adam were a common default for training networks, but, as the paper’s introduction notes, the best results on CIFAR-10 and CIFAR-100 still came from SGD with momentum. Wilson et al. (2017) had reported that adaptive methods generalized worse than SGD with momentum on several tasks, image classification among them (the Adam article gives their numbers). Explanations on offer included sharp minima and inherent properties of adaptive methods.

Loshchilov and Hutter proposed a more mundane cause. In the common implementations, weight decay was added to the gradient as $\lambda\theta$, the gradient of an L2 penalty $\tfrac{\lambda}{2}\lVert\theta\rVert^2$. For plain SGD this is exactly a multiplicative shrink of the weights, the decay that the paper attributes to Hanson and Pratt (1988). Adam, however, divides each coordinate of its gradient estimate by a running root-mean-square of that coordinate’s past gradients, and with the L2 form the penalty term is divided too. In the paper’s words, weights with large typical gradients are then “regularized by a smaller relative amount” than the rest. If L2 regularization helps SGD on a dataset but barely acts under Adam, the authors argued, Adam’s generalization gap on that dataset follows. A second observation concerned SGD itself: the equivalence holds only when the L2 coefficient is rescaled by the learning rate, so the best penalty and the best learning rate are tied together and must be tuned jointly.

Contribution

As published at ICLR 2019, the paper contributes:

• A formal comparison of the two forms (Propositions 1–3, proofs in Appendix A): equivalence for SGD after rescaling, no equivalence for any optimizer whose preconditioner is not a multiple of the identity, and, for a fixed diagonal preconditioner, an equivalence to an L2 penalty that weights each coordinate by its preconditioner scale.
• SGDW and AdamW (Algorithms 1 and 2): SGD with momentum and Adam with the decay applied directly to the weights, both with a schedule multiplier $\eta_t$ that scales the learning rate and the decay together.
• An empirical study on CIFAR-10 and ImageNet32x32 showing that decoupling makes the learning rate and decay more separable in hyperparameter search and lowers Adam’s test error, under fixed, step-drop and cosine learning-rate schedules and training budgets from 100 to 1,800 epochs.
• Two practical additions in Appendix B: normalized weight decay, which scales the decay with the number of batch passes, and AdamWR, AdamW with the cosine annealing and warm restarts of the authors’ earlier SGDR (Loshchilov & Hutter, 2017).
• A justification through Bayesian filtering (Section 3), summarizing Aitchison’s view of adaptive methods, in which decoupled decay, not an L2 term, appears in the model of how weights change between steps.

Method

The propositions. The paper writes decay as $\theta_{t+1} = (1-\lambda)\theta_t - \alpha\nabla f_t(\theta_t)$, with $\lambda$ a decay rate per step (its Equation 1). Proposition 1 states the known fact that SGD with this decay takes the same steps as SGD on $f_t + \tfrac{\lambda'}{2}\lVert\theta\rVert^2$ with $\lambda' = \lambda/\alpha$. Proposition 2 considers an optimizer with steps $\theta_{t+1} = \theta_t - \alpha M_t\nabla f_t$: if $M_t \ne kI$, no L2 coefficient reproduces decay, because matching the two updates would need $\lambda\theta = \alpha\lambda' M_t\theta$ for every $\theta$. Proposition 3 takes a fixed preconditioner $M_t = \mathrm{diag}(s)^{-1}$ and shows that decay then equals an L2 penalty on $\theta \odot \sqrt{s}$, so that weights with large $s_i$, which in practice means historically large gradients, are regularized more than a plain L2 term would regularize them. The weight decay article derives all three.

Proposition 3 as printed contains an error in a constant. Its statement gives the penalty as $\frac{\lambda'}{2\alpha}\lVert\theta\odot\sqrt{s}\rVert_2^2$ with $\lambda' = \lambda/\alpha$, while its proof (Appendix A, Equations 9–13) uses $\frac{\lambda'}{2}\lVert\theta\odot\sqrt{s}\rVert_2^2$. The proof’s version is the consistent one: the gradient of that penalty, divided by $s$ and multiplied by $\alpha$, is $\alpha\lambda'\theta = \lambda\theta$, the decay step. The equivalent penalty is therefore

$$\frac{\lambda}{2\alpha}\sum_i s_i\,\theta_i^2$$

(our derivation). With the statement’s extra factor $1/\alpha$, the implied step would shrink the weights by $(\lambda/\alpha)\theta$ instead of $\lambda\theta$, off by that factor. The same statement appears in arXiv v3 and in the ICLR copy. In v2, where the proposition first appeared, the penalty was written on $\theta \odot s$ without the square root, and the proof’s step from Equation 13 to 14 dropped a factor of $s$; v3 corrected the square root but kept the $1/\alpha$. arXiv v3 follows the proposition with a caveat: practical adaptive methods change the preconditioner at every step, so it gives intuition rather than an exact equivalence. v2 had a similar caveat; the ICLR copy has none.

The algorithms. Algorithm 1 presents SGD with momentum in both forms on one listing, with the lines that belong to each variant distinguished only by colour. The L2 variant adds $\lambda\theta_{t-1}$ to the gradient in line 6, before the momentum buffer; SGDW instead decays the weights in line 9, outside it:

$$m_t = \beta_1 m_{t-1} + \eta_t\,\alpha\,\nabla f_t(\theta_{t-1}), \qquad \theta_t = \theta_{t-1} - m_t - \eta_t\,\lambda\,\theta_{t-1}.$$

Algorithm 2 does the same for Adam, with defaults $\alpha = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$: the L2 variant puts $\lambda\theta_{t-1}$ into the gradient that feeds both moment estimates, while AdamW subtracts $\eta_t\lambda\theta_{t-1}$ alongside the normalized step $\eta_t\alpha\hat m_t/(\sqrt{\hat v_t}+\epsilon)$ in line 12. In both algorithms the decay is multiplied by the schedule multiplier $\eta_t$, so a schedule decays the regularization along with the learning rate, but not by the base learning rate $\alpha$; that is how the paper separates the two hyperparameters. PyTorch later chose differently: its AdamW multiplies the decay by the full learning rate, $\theta \leftarrow \theta - \gamma\lambda\theta$, so a $\lambda$ from the paper and one from PyTorch differ by a factor of the base learning rate, as the weight decay article explains.

Normalized weight decay (Appendix B.1). The authors’ preliminary experiments showed that the best decay depends on the training budget, and they replace $\lambda$ by a normalized value,

$$\lambda = \lambda_{\text{norm}}\sqrt{\frac{b}{BT}},$$

with batch size $b$, $B$ training points and $T$ epochs (in AdamWR, the epochs of the current restart). $\lambda_{\text{norm}}$ is then the decay for a single batch pass. They call this square-root rule one possibility informed by few experiments; arXiv v3 adds that even better scaling rules are likely to exist.

AdamWR (Appendix B.2). Following SGDR, the multiplier follows a cosine within each run, $\eta_t = 0.5 + 0.5\cos(\pi T_{\text{cur}}/T_i)$, and is reset to 1 at each restart while the weights are kept; the period $T_i$ starts small (1% to 10% of the budget, the paper suggests) and is multiplied by $T_{\text{mult}}$, for example 2, at each restart. Appendix C’s example, $T_0 = 100$ and $T_{\text{mult}} = 2$, restarts at epochs 100, 300, 700 and 1,500, and the solutions just before the restarts are the ones the optimizer recommends. AdamWR combines this schedule with AdamW and normalized decay, and SGDWR does the same for SGDW. The learning-rate schedule article covers cosine annealing and warm restarts as schedules.

Bayesian filtering (Section 3). The ICLR version adds a section, with full credit to Aitchison (2018), that views stochastic optimization as tracking the optimal weights with a Bayesian filter. With a Gaussian transition from step to step, $P(\theta_{t+1}\mid\theta_t) = \mathcal{N}((I - A)\theta_t, Q)$, the choice $A = \lambda I$ multiplies the mean by $1 - \lambda$ at every step, which is decoupled decay, and acts independently of each parameter’s uncertainty, which the L2 form would not.

Results

Setup. The experiments follow Gastaldi’s (2017) Shake-Shake setup, which had reached 2.86% test error on CIFAR-10, and reuse its code, based on fb.resnet.torch: batch size 128, the standard CIFAR augmentation, and two residual networks, a 26 2x64d ResNet (depth 26, two residual branches, first block 64 wide) with 11.6 million parameters and a 26 2x96d ResNet with 25.6 million. The second dataset is ImageNet32x32 (Chrabaszcz, Loshchilov & Hutter, 2017), the 1.2 million ImageNet training images downsampled to 32×32 pixels. In the ICLR version the CIFAR-10 results are reported as top-1 test error and the ImageNet32x32 ones as top-5 test error.

The paper has no results tables. Every comparison is a heatmap or a learning curve, and the text prints only a few numbers. Below, the figure numbers are the ICLR version’s, which match arXiv v3.

• Schedules (Section 4.1, Figure 1). Adam with L2 regularization and AdamW, each with a fixed learning rate, a step-drop schedule (drops at epochs 30, 60 and 80) and cosine annealing, were trained for 100 epochs on CIFAR-10 over grids of initial learning rate and decay. AdamW beat the L2 form under every schedule, with larger differences for better schedules, and its search space was more separable, especially with the two schedules; cosine annealing gave clearly the best results and was used in all later experiments.
• Separability (Section 4.2, Figure 2). For the same network, budget and cosine schedule, four heatmaps of test error compare SGD with an L2 term, SGDW, Adam with an L2 term and AdamW. For SGD with L2 the best settings lie along a diagonal: from the top-left of the ten best settings ($\alpha = 1/2$, $\lambda = 1/8 \times 0.001$), changing either value alone worsens the result, while changing both together can still improve it. The authors suggest that this coupling may have contributed to SGD’s reputation for sensitivity to hyperparameters. SGDW’s basin is aligned with the axes: even at a poorly chosen learning rate of 1/1024, tuning only the decay finds a good value ($1/4 \times 0.001$). Adam with L2 did clearly worse than SGD, and its best settings with a nonzero penalty were no better than with none. AdamW was substantially better than Adam with L2, rivaled SGD and SGDW, and largely decoupled the two hyperparameters.
• Long runs (Section 4.3, Figure 3). With the larger 26 2x96d network, 1,800 epochs and the learning rate fixed at Adam’s default of 0.001, the authors compared 12 L2 coefficients for Adam with 7 normalized decays for AdamW. The learning curves often coincided for the first half of training, after which AdamW tended to reach lower training loss and lower test error; at matched training loss AdamW still had lower test error, which the authors read as better generalization and not only better optimization. ImageNet32x32 gave the same qualitative result (SuppFigure 4).
• Warm restarts (Section 4.4, Figure 4). AdamWR sped up AdamW by up to a factor of 10, judged at the first restart, on both datasets. At the default learning rate of 0.001, AdamW achieved a 15% relative improvement in test error over Adam on CIFAR-10 and ImageNet32x32, and AdamWR reached the same results with much better anytime performance. The improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and gave performance comparable to SGDWR on ImageNet32x32. No absolute errors are printed for these runs.
• Normalized decay (Appendix D, SuppFigure 3). The best raw decay shrank as the budget grew, while the best normalized values stayed similar across budgets, across the two datasets, and across SGDW and AdamW (for example, $\lambda_{\text{norm}} = 0.025$ and $0.05$). An ImageNet32x32 epoch is 24 times longer than a CIFAR-10 epoch, and without the normalization, the authors write, the CIFAR-10 decay would have been roughly 5 times too large there; $\sqrt{24} \approx 4.9$ (our arithmetic), the factor the square-root rule removes. A 4×4 grid of “standard Adam” (L2 and a fixed learning rate) run for 1,800 epochs did at best about as well as AdamW with 18 times fewer epochs and a smaller network (SuppFigure 2).
• Other groups’ results (Section 4.5, added in arXiv v3 and present in the ICLR copy). The authors list early uses of AdamW: Wang et al. (2018) trained a face detector on WIDER FACE with it; Völker et al. (2018) trained networks on intracranial EEG and, in a comparison they supplied to the authors privately, found AdamW better than Adam with the same hyperparameters (73.68% against 71.37% test accuracy with Deep4Net, and a statistically significant 72.04% against 61.34% with a ResNet); Radford et al. (2018) used it to train the first GPT; and Zhang et al. (2018) found decoupled decay better than L2 regularization wherever the two differ.

The text and figures disagree in a few places, all in labels rather than results. The introduction cites Figures 2 and 3 for the 15% improvement, but Figure 2 holds the 100-epoch grids and the 15% statement in Section 4.4 refers to the default learning rate in the longer runs. Appendix D says the normalized decay is “defined in Eq. (15)”, which is the cosine multiplier (the definition in Appendix B.1 is unnumbered), and cites SuppFigure 4 for normalized decay, which is SuppFigure 3. The equation reference was wrong in the same way in v1 (Eq. 7 for Eq. 5) and v2 (Eq. 8 for Eq. 6).

Versions

The paper went through three arXiv versions, and the ICLR 2019 copy is a fourth text. The dates are from the arXiv submission history and the PDF metadata; the comparisons are ours, made word by word.

Version	Date	Title	Main changes
arXiv v1	14 Nov 2017	Fixing Weight Decay Regularization in Adam	Decoupling, normalized decay and AdamWR as Sections 2–4; no propositions; 13 pages
arXiv v2	14 Feb 2018	Fixing Weight Decay Regularization in Adam	Two-column layout; new Section 3 with Propositions 1–3; Adam with fixed versus cosine learning rate as a first experiment; 13 pages
arXiv v3	4 Jan 2019	Decoupled Weight Decay Regularization	ICLR 2019 header; notation $\theta$, $\lambda$; Bayesian-filtering section; normalized decay and AdamWR moved to an appendix; Section 4.5 on other groups’ use; 19 pages
ICLR (OpenReview PDF)	file dated 2 May 2019	Decoupled Weight Decay Regularization	Same sections and numbering as v3, with a few passages missing; 18 pages

v1 framed the problem as a defect: its abstract says common implementations of Adam limit the benefit of weight decay because the weights do not decay multiplicatively, and its introduction calls the standard implementation “dysfunctional”. It used weights $x$ and decay $w_t$, and showed the coupling with a large-$t$ form of the Adam update (its Equation 4) rather than a proposition. Its experiments already contained the decoupling grids, the 1,800-epoch comparison, normalized decay and AdamWR, with the same 15% statement; its abstract claimed state-of-the-art results for AdamWR on CIFAR-10 and ImageNet32x32.

v2 adopted the framing of the final abstract, L2 regularization and weight decay being equivalent for SGD but not for adaptive methods, and added the propositions, with a remark, removed later, that Proposition 2 showed the inequivalence “for what we believe to be the first time”. It also linked the Proposition 3 intuition to flat minima: penalizing weights with large gradients might steer the search toward flatter optima. A new first experiment compared Adam with a fixed learning rate and with cosine annealing. Its numbers disagree with each other: the contributions list says cosine annealing reduces Adam’s test error by a relative 15%, while Section 6.1 says roughly 20%.

v3 took the published title, removed the flat-minima paragraph and the “first time” remark, added the Bayesian-filtering section and the section on adoption, promoted v2’s supplementary comparison of fixed, step-drop and cosine schedules for Adam and AdamW to the main Figure 1, and moved normalized decay and AdamWR to Appendix B. It lists five observations in its introduction; the fifth, that Adam can benefit substantially from a scheduled learning-rate multiplier, is not in the other versions, and its Section 4.1 adds a sentence advising readers not to overlook schedules for adaptive methods.

The ICLR copy, read in the Internet Archive’s snapshot of the official OpenReview PDF, has v3’s structure but is missing several of v3’s sentences, despite its later file date: the fifth observation, the advice on schedules in Section 4.1, the caveat after Proposition 3, and the closing caution in Appendix D that the scaling experiments were not very comprehensive. Two of its figure references, which v3 corrects, point to the wrong supplementary figures. In our reading the OpenReview PDF is a camera-ready text prepared before v3’s final edits, so v3 is the fuller version; claims found only in v3 are labelled as such here.

Impact

The ICLR abstract already noted that decoupled weight decay had been adopted by many researchers and implemented by the community in TensorFlow and PyTorch, and the acknowledgments list implementations in PyTorch (a pull request), TensorFlow (a DecoupledWeightDecayExtension in tf.contrib), fast.ai, Keras and Caffe. PyTorch added torch.optim.AdamW in release 1.2.0 of 8 August 2019, citing the paper (release notes and version 1.2.0 documentation); the class is absent from the 1.1.0 documentation. From that first version its update multiplied the decay by the learning rate, p.data.mul_(1 - lr * weight_decay), the form that PyTorch 2.14 keeps with a default weight_decay of 0.01.

In computer vision, AdamW appears in many recipes of the transformer era: those described on this site use it for DETR, DeiT’s data-efficient Vision Transformer, MAE, DINO, ConvNeXt and SAM, several of them with a cosine schedule after warmup. The paper’s own experiments were convolutional, and its motivation was to let Adam compete with SGD where it had not, so that practitioners need not switch optimizers between tasks.

Later analyses of weight decay started from the paper’s finding. Zhang, Wang, Xu and Grosse (2019; read in arXiv v1 of October 2018) start from its finding that literal decay outperforms L2 regularization when the two differ, confirm it for SGD, Adam and the second-order optimizer K-FAC on CIFAR-10 and CIFAR-100, and then ask why decay helps at all. In networks with batch normalization, whose weights are scale-invariant, they find that decay works mainly by raising the effective learning rate; the weight decay article follows that line of work.

Limitations

• Narrow evidence. All main experiments use one architecture family (Shake-Shake ResNets) on two small-image datasets. The authors themselves say their results must be verified on a wider range of tasks, and they state only as a belief that other adaptive methods, such as AdaGrad and AMSGrad, behave similarly.
• Results only in figures. With no tables, error bars or absolute errors for the main comparisons, the 15% improvement cannot be checked against printed values, and the paper does not state how many runs each heatmap cell averages.
• Theory for a fixed preconditioner. Proposition 3 holds only for a constant diagonal preconditioner, which Adam does not have (a caveat stated in v2 and v3 but not in the ICLR copy), and its printed constant is off by a factor $1/\alpha$, as shown above. The propositions explain why the two forms differ, not why decoupled decay generalizes better; the Bayesian-filtering argument is a motivation borrowed from Aitchison.
• Heuristic normalization. The square-root rule for normalized decay was suggested by initial CIFAR-10 experiments and checked on ImageNet32x32, a limitation the authors acknowledge.
• Parameterization. The paper scales the decay by the schedule multiplier but not by the learning rate; PyTorch scales it by the full learning rate. Decay values are therefore not portable between the paper and common implementations without conversion.
• Normalization layers not discussed. No version of the paper mentions batch normalization or scale invariance. It treats decay only as a regularizer of the function, whereas in normalized layers decay acts mainly through the effective learning rate, as Zhang et al. (2019) later found.

What Came After

The questions the paper left open were taken up in the analyses of weight decay with normalization layers, beginning with Zhang et al. (2019) and continuing with work on effective learning rates; the weight decay article and the normalization layers article cover them. Decoupled decay spread beyond Adam: besides the AdamW class, PyTorch 2.14’s Adam, NAdam and RAdam take a switch, decoupled_weight_decay=True, that applies it. Cosine annealing, which the paper adopted for all its later experiments, accompanies AdamW, after a warmup, in several of the vision recipes above; the learning-rate schedule article tabulates DeiT’s and ConvNeXt’s.