Learning Rate Schedule

A learning rate schedule is a rule $\eta_t$ that sets the step size of stochastic gradient descent or Adam at iteration $t$ instead of keeping it fixed. AlexNet and the original ResNets divided the rate by 10 when progress stalled; recent recipes such as DeiT and ConvNeXt raise it over a few warmup epochs, then let it fall along a cosine to zero. The schedule interacts with the batch size, the weight decay and the training length, so it is tuned together with them, as part of training a neural network.

Purpose

A single learning rate has to serve two phases that want different things. Early on the weights are far from any good solution and the loss falls quickly along a few directions, so large steps make fast progress. Late in training the mini-batch gradient is dominated by noise, because individual images still disagree when their average gradient is near zero, and a constant step keeps the weights bouncing around a noise floor whose height grows with the step size; the SGD article derives this and shows it on digit images. Decreasing the rate lowers the floor.

The classical answer is a decaying step with $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$, conditions that go back to Robbins and Monro (1951) and that the SGD article discusses. Smith, Kindermans, Ying and Le (2018) summarize the intuition for strongly convex functions: the first condition lets the iterates reach the minimum however far away they start, and the second makes the rate decay fast enough to converge rather than bounce around the minimum because of gradient noise. These conditions are asymptotic. Training a network runs for a fixed budget of epochs, so practical schedules are designed for a known horizon $T$ and usually decay to zero (or near it) exactly at the end, a different goal from convergence in the limit (our reading).

A second purpose appeared with large batches and adaptive optimizers: the first few hundred or thousand updates can be unstable at the rate that is best later, and a short warmup that starts small avoids this. Smith et al. (2018) give a third, noise-based view. Following earlier work that treats SGD as integrating a stochastic differential equation, they describe its fluctuations by a noise scale $g \approx \epsilon N / B$ for learning rate $\epsilon$, training-set size $N$ and batch size $B \ll N$. Decaying the learning rate lowers $g$, and so does increasing the batch size: in their CIFAR-10 and ImageNet experiments, growing the batch by the factor by which a step schedule would have cut the rate gave nearly the same test accuracy after the same number of epochs, with fewer parameter updates.

When to Use It

• Training from scratch with SGD. The recipes cited here all decay the rate; in the example below a constant rate ends far behind.
• Adaptive optimizers need one too. Loshchilov and Hutter (2019) found that schedules substantially improved Adam on CIFAR-10 and advised not to overlook them for adaptive methods; the original Transformer and DeiT were trained with warmup followed by decay.
• Warmup matters most when the peak rate is large, as with large batches scaled by the linear scaling rule, and for Adam-type optimizers and post-norm Transformers early in training (sources below).
• Fine-tuning a pre-trained backbone (transfer learning) uses the same shapes at a much smaller peak rate; ConvNeXt’s ImageNet-22K-to-1K fine-tuning, for instance, used AdamW at $5 \times 10^{-5}$ with a cosine schedule and no warmup (Liu et al., 2022, supplementary material).
• A schedule does not rescue a wrong base rate. If the peak rate makes the loss diverge even after warmup, the base rate must come down.

How It Works

A schedule is usually written as a multiplier of a base (peak) rate $\eta_0$. With $T$ total iterations:

Schedule	$\eta_t / \eta_0$	Typical use
Step decay	$\gamma^{k}$ after the $k$-th milestone, $\gamma = 0.1$	AlexNet, ResNet, Goyal et al.
Exponential	$\gamma^{t/s}$ (decay by $\gamma$ every $s$ steps)	GoogLeNet, Inception-v3, MobileNetV2
Polynomial (“poly”)	$(1 - t/T)^{p}$, $p = 0.9$	DeepLab and other segmentation models
Cosine	$\tfrac{1}{2}\left(1 + \cos(\pi t / T)\right)$	Bag of Tricks, DeiT, ConvNeXt
Linear warmup	$(t+1)/W$ for $t < W$	before any of the above
Inverse square root	$\min\left(t^{-1/2}, \, t\,W^{-3/2}\right)$, rescaled	original Transformer

Step decay

The rate stays constant and drops by a fixed factor at milestones. Krizhevsky, Sutskever and Hinton (2012) trained AlexNet with a rate of 0.01, divided by 10 by hand whenever the validation error stopped improving, three times in about 90 epochs. He et al. (2016) used the same rule for ResNets on ImageNet, starting at 0.1. For their CIFAR-10 networks they fixed the milestones instead: 0.1, divided by 10 at 32k and 48k iterations, stopping at 64k. Later ImageNet recipes fixed them in epochs (Goyal et al. in the table under Best Practices). Fixed milestones make runs reproducible; plateau-triggered drops adapt to the run but depend on a noisy validation curve (our reading).

Exponential and polynomial decay

Exponential decay multiplies the rate by a constant factor at regular intervals. GoogLeNet was trained with a rate decreased by 4% every 8 epochs (Szegedy et al., 2015), a gentle decay: after 100 epochs the rate is still $0.96^{12.5} \approx 0.60$ of its start, whereas the factor of 0.94 every 2 epochs used for Inception-v3 (Szegedy et al., 2016) leaves $0.94^{50} \approx 0.045$ (our arithmetic). MobileNetV2 used 0.98 per epoch (Sandler et al., 2018).

The polynomial schedule reaches zero at $T$. DeepLab v2 adopted it with power 0.9, noting that ParseNet had found the same advantage: for DeepLab-LargeFOV with the same batch size of 30 and 6,000 iterations it gave 63.42% mean IoU on PASCAL VOC 2012 val against 62.25% for step decay, before the CRF (Chen et al., 2018, Table 2, read in arXiv v2); the DeepLab article gives the longer runs that the authors adopted.

Cosine annealing

Loshchilov and Hutter (2017) decay the rate along half a cosine,

$$\eta_t = \eta_{\min} + \tfrac{1}{2}\left(\eta_{\max} - \eta_{\min}\right)\left(1 + \cos\left(\pi \, \frac{T_{\text{cur}}}{T_i}\right)\right),$$

where $T_{\text{cur}}$ counts epochs since the last restart and is updated every batch, so it takes fractional values. With a single cycle ($T_i = T$, $\eta_{\min} = 0$) it is the cosine decay that He, T. et al. (2019) used in their ImageNet recipe. They describe its shape: it falls slowly at first, almost linearly in the middle and slowly again at the end, so compared with step decay it starts decaying immediately but stays large until step decay’s first tenfold drop. Cosine decay has two practical advantages over steps: one hyperparameter (the length $T$) instead of milestones and factors, and no sudden drops that require a momentum correction (in our reading; the SGD article explains why changing the rate interacts with PyTorch’s momentum buffer).

Warmup

Constant warmup trains at a low rate for a while, then switches to the full rate. He et al. (2016) needed it for their 110-layer CIFAR-10 ResNet, whose initial rate of 0.1 was slightly too large to start converging: they trained at 0.01 until the training error fell below 80% (about 400 iterations), then returned to 0.1.

Gradual (linear) warmup ramps the rate up instead. Goyal et al. (2017) proposed it together with the linear scaling rule, which the SGD article covers with their main results: they start from the small-batch rate $\eta$ and add a constant increment every iteration until they reach $k\eta$ after 5 epochs, then continue with the usual schedule. Constant warmup was not enough at large batches: for ResNet-50 with batches of 8,192 and a target rate of 3.2, switching from 0.1 to 3.2 after 5 epochs made the training error spike, and the top-1 validation error was 25.88% ± 0.56, worse than the 24.84% ± 0.37 without any warmup; gradual warmup gave 23.74% ± 0.09 (mean ± std of 5 runs, their Table 1). He, T. et al. (2019) start from zero, $\eta_t = t\eta/m$ for the first $m$ batches.

Warmup for Adam and Transformers

Vaswani et al. (2017) trained the Transformer with Adam and

$$\eta_t = d_{\text{model}}^{-1/2} \cdot \min\left(t^{-1/2}, \; t \cdot W^{-3/2}\right), \qquad W = 4000,$$

a linear rise for $W$ steps followed by decay proportional to $1/\sqrt{t}$. The two branches meet at $t = W$, where the rate peaks at $(d_{\text{model}} W)^{-1/2} \approx 7.0 \times 10^{-4}$ for the base model’s $d_{\text{model}} = 512$ (our arithmetic).

Why Adam needs warmup was studied by Liu et al. (2020). In their experiments, removing warmup from a Transformer for IWSLT’14 German–English translation left the training loss around 10 instead of below 3. They hypothesized that Adam’s adaptive rate, $1/\sqrt{v_t}$, has problematically large variance in the first steps because $v_t$ is estimated from very few gradients, and presumed that warmup works as a variance reduction technique. Two controls support this: freezing the weights for the first 2,000 iterations while $v_t$ accumulates, and a large $\epsilon = 10^{-4}$, both avoided the failure without warmup (the second at the cost of worse final results). From an approximation of that variance they derived RAdam, which multiplies the adaptive step by a rectification term that behaves like a warmup computed from $\beta_2$ and $t$ and uses plain momentum SGD while the variance is intractable. Xiong et al. (2020) give an architectural explanation for post-norm Transformers, covered in the Transformer article; the two are not exclusive (our reading).

How to Apply It

PyTorch implements schedules in torch.optim.lr_scheduler (version 2.14). Each scheduler wraps an optimizer and changes param_groups[i]["lr"] when its step() is called; since PyTorch 1.1.0 it must be called after optimizer.step(), otherwise the first value of the schedule is skipped (PyTorch documentation, torch.optim). Defaults as of version 2.14:

Class	Schedule	Key arguments (defaults)
StepLR, MultiStepLR	step decay	step_size or milestones, gamma=0.1
ExponentialLR	$\gamma^t$ per step()	gamma (required)
PolynomialLR	$(1 - t/T)^p$	total_iters=5, power=1.0
CosineAnnealingLR	one cosine, no restarts	T_max, eta_min=0.0
CosineAnnealingWarmRestarts	SGDR	T_0, T_mult=1, eta_min=0.0
LinearLR	linear ramp of the multiplier	start_factor=1/3, end_factor=1.0, total_iters=5
SequentialLR	switches schedulers at milestones	schedulers, milestones
OneCycleLR	1cycle	max_lr, total_steps or epochs and steps_per_epoch, pct_start=0.3, div_factor=25, final_div_factor=1e4
ReduceLROnPlateau	drop when a metric stalls	factor=0.1, patience=10, threshold=1e-4
LambdaLR	any multiplier function	lr_lambda

The counter $t$ is whatever the caller counts: T_max=90 means 90 epochs only if step() is called once per epoch. OneCycleLR is documented to be stepped after every batch, and ReduceLROnPlateau.step() takes the monitored metric.

Example: warmup and decay on stripe images

The script below (PyTorch 2.14.1) trains a small CNN with batch normalization to classify $16 \times 16$ images of sinusoidal stripes by their orientation (0°, 45°, 90° or 135°), with random frequency and phase and Gaussian pixel noise of standard deviation 2.5, more than twice the stripe amplitude. SGD with momentum 0.9 runs for 10 epochs of 24 batches of 50 images ($T = 240$ iterations), under four schedules applied per iteration through LambdaLR: constant, step decay (÷10 at 50% and 75% of training), cosine, and a 2-epoch linear warmup followed by cosine. Each runs at a sensible base rate of 0.1 and a too-high rate of 3.0, for five seeds.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)

def stripes(n, g, size=16, noise=2.5):
    """16x16 sinusoidal stripes at 0, 45, 90 or 135 degrees (the class), random
    frequency and phase, buried in Gaussian pixel noise of standard deviation 2.5."""
    y = torch.randint(0, 4, (n,), generator=g)
    theta = y.float() * math.pi / 4
    freq = 0.15 + 0.15 * torch.rand(n, generator=g)          # cycles per pixel
    phase = 2 * math.pi * torch.rand(n, generator=g)
    r = torch.arange(size).float() - size / 2
    yy, xx = torch.meshgrid(r, r, indexing="ij")
    u = xx * torch.cos(theta)[:, None, None] + yy * torch.sin(theta)[:, None, None]
    img = torch.sin(2 * math.pi * freq[:, None, None] * u + phase[:, None, None])
    return (img + noise * torch.randn(n, size, size, generator=g))[:, None], y

g = torch.Generator().manual_seed(1000)
X_tr, y_tr = stripes(1200, g)
X_te, y_te = stripes(1000, g)

def make_cnn():
    block = lambda c_in, c_out: [nn.Conv2d(c_in, c_out, 3, padding=1), nn.BatchNorm2d(c_out), nn.ReLU()]
    return nn.Sequential(*block(1, 8), nn.MaxPool2d(2), *block(8, 16), nn.MaxPool2d(2),
                         *block(16, 16), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(16, 4))

B, EPOCHS = 50, 10
STEPS = EPOCHS * len(X_tr) // B        # T = 240 iterations
WARM = 2 * len(X_tr) // B              # 2 epochs = 48 iterations of linear warmup

# Multipliers of the base learning rate at iteration t (applied by LambdaLR every batch).
SCHEDULES = {
    "constant": lambda t: 1.0,
    "step":     lambda t: 1.0 if t < STEPS // 2 else 0.1 if t < 3 * STEPS // 4 else 0.01,
    "cosine":   lambda t: 0.5 * (1 + math.cos(math.pi * t / STEPS)),
    "warmup+cosine": lambda t: (t + 1) / WARM if t < WARM
                     else 0.5 * (1 + math.cos(math.pi * (t - WARM) / (STEPS - WARM))),
}

def train(schedule, base_lr, seed):
    torch.manual_seed(seed)
    model = make_cnn()
    opt = torch.optim.SGD(model.parameters(), lr=base_lr, momentum=0.9)
    sched = torch.optim.lr_scheduler.LambdaLR(opt, SCHEDULES[schedule])
    order = torch.Generator().manual_seed(seed)
    first_epoch_loss = 0.0
    for epoch in range(EPOCHS):
        perm = torch.randperm(len(X_tr), generator=order)
        for i in range(0, len(X_tr), B):
            idx = perm[i:i + B]
            loss = F.cross_entropy(model(X_tr[idx]), y_tr[idx])
            opt.zero_grad()
            loss.backward()
            opt.step()
            sched.step()                # per iteration, after optimizer.step()
            if epoch == 0:
                first_epoch_loss += loss.item() * len(idx) / len(X_tr)
    model.eval()
    with torch.no_grad():
        train_loss = F.cross_entropy(model(X_tr), y_tr).item()
        test_acc = (model(X_te).argmax(1) == y_te).float().mean().item()
        model.train()                   # same weights, normalized with the batch's own statistics
        batch_stat_loss = F.cross_entropy(model(X_tr), y_tr).item()
    return first_epoch_loss, train_loss, batch_stat_loss, test_acc

print(f"torch {torch.__version__}; T = {STEPS} iterations, warmup = {WARM}")
marks = [0, WARM - 1, STEPS // 2, 3 * STEPS // 4, STEPS - 1]
print("multiplier at t =" + "".join(f"{t:>7d}" for t in marks))
for name, f in SCHEDULES.items():
    print(f"{name:16s} " + "".join(f"{f(t):7.3f}" for t in marks))

SEEDS = range(5)
for base_lr in [0.1, 3.0]:
    print(f"\nbase learning rate {base_lr}: median over seeds {list(SEEDS)} (test accuracy range)")
    for name in SCHEDULES:
        runs = [train(name, base_lr, s) for s in SEEDS]
        ep1, loss, bs_loss, acc = (sorted(r[k] for r in runs) for k in range(4))
        print(f"  {name:14s} epoch-1 loss {ep1[2]:5.2f}  train loss {loss[2]:5.3f} "
              f"(batch stats {bs_loss[2]:5.3f})  test acc {acc[2]:.3f} ({acc[0]:.3f}-{acc[-1]:.3f})")

Output (identical over two runs; about 25 seconds on a laptop CPU):

torch 2.14.1; T = 240 iterations, warmup = 48
multiplier at t =      0     47    120    180    239
constant           1.000  1.000  1.000  1.000  1.000
step               1.000  1.000  0.100  0.010  0.010
cosine             1.000  0.908  0.500  0.146  0.000
warmup+cosine      0.021  1.000  0.691  0.222  0.000

base learning rate 0.1: median over seeds [0, 1, 2, 3, 4] (test accuracy range)
  constant       epoch-1 loss  1.21  train loss 1.493 (batch stats 0.046)  test acc 0.623 (0.536-0.790)
  step           epoch-1 loss  1.21  train loss 0.083 (batch stats 0.083)  test acc 0.870 (0.862-0.886)
  cosine         epoch-1 loss  1.22  train loss 0.060 (batch stats 0.061)  test acc 0.865 (0.860-0.872)
  warmup+cosine  epoch-1 loss  1.38  train loss 0.061 (batch stats 0.059)  test acc 0.872 (0.863-0.876)

base learning rate 3.0: median over seeds [0, 1, 2, 3, 4] (test accuracy range)
  constant       epoch-1 loss  1.46  train loss 1.559 (batch stats 1.100)  test acc 0.303 (0.266-0.409)
  step           epoch-1 loss  1.46  train loss 1.018 (batch stats 1.018)  test acc 0.443 (0.263-0.456)
  cosine         epoch-1 loss  1.43  train loss 0.636 (batch stats 0.635)  test acc 0.615 (0.431-0.688)
  warmup+cosine  epoch-1 loss  1.08  train loss 0.268 (batch stats 0.266)  test acc 0.820 (0.773-0.843)

At the sensible rate of 0.1 every decaying schedule ends between 86% and 89% test accuracy and warmup changes little. The constant rate is far behind (median 62%, 54–79% across seeds), and the two loss columns show why: normalized with each batch’s own statistics, the constant-rate network fits the training set (median loss 0.046), but in evaluation mode, which uses the running averages of the batch statistics, its median loss is 1.49. In our reading, weights still taking large steps at the end of training leave the running averages describing a slightly older network; a decayed rate lets weights and statistics settle together.

At 3.0 the schedules separate. Without warmup the median first-epoch loss is 1.43–1.46, worse than $\ln 4 \approx 1.386$, the loss of a uniform guess over four classes: averaged over the first epoch, the full-rate steps leave the network worse than guessing. Cosine decay alone recovers partly (median 62%, 43–69%); step decay, whose rate stays at 3.0 for half of training, recovers less (44%). Warmup followed by cosine reaches 82% (77–84%): starting at $0.021 \times 3.0 = 0.0625$ and ramping over two epochs lets the network reach a region where the full rate is tolerable. It still falls short of the 87% at the sensible rate; warmup widens the range of usable peak rates but does not make the base rate irrelevant.

Stepping semantics

The next script records the learning rate each optimizer step actually uses for several built-in schedulers, with 10 epochs of 24 batches as above:

import math
import torch
from torch.optim.lr_scheduler import (CosineAnnealingLR, LambdaLR, LinearLR, OneCycleLR,
                                      ReduceLROnPlateau, SequentialLR)

EPOCHS, STEPS_PER_EPOCH = 10, 24
T, WARM = EPOCHS * STEPS_PER_EPOCH, 2 * STEPS_PER_EPOCH

def trajectory(make_scheduler, n_steps, lr=3.0):
    """Learning rate used by each of n_steps optimizer steps; scheduler.step() follows optimizer.step()."""
    opt = torch.optim.SGD([torch.nn.Parameter(torch.zeros(1))], lr=lr, momentum=0.9)
    sched = make_scheduler(opt)
    lrs = []
    for _ in range(n_steps):
        lrs.append(opt.param_groups[0]["lr"])
        opt.step()
        sched.step()
    return lrs

fmt = lambda xs, d=3: " ".join(f"{x:.{d}f}" for x in xs)

# 1. OneCycleLR counts batches: stepping it once per epoch leaves it in its warm-up phase.
one_cycle = lambda opt: OneCycleLR(opt, max_lr=3.0, epochs=EPOCHS, steps_per_epoch=STEPS_PER_EPOCH)
per_batch = trajectory(one_cycle, T)
print("OneCycleLR, lr at the start of each epoch")
print("  stepped per batch:", fmt(per_batch[::STEPS_PER_EPOCH]))
print("  stepped per epoch:", fmt(trajectory(one_cycle, EPOCHS)))
print(f"  per batch: peak {max(per_batch):.3f} at step {per_batch.index(max(per_batch))}, "
      f"last {per_batch[-1]:.2e}")

# 2. CosineAnnealingLR with T_max in epochs but stepped per batch: the cosine keeps cycling.
cos_epochs = trajectory(lambda opt: CosineAnnealingLR(opt, T_max=EPOCHS), T)
print("\nCosineAnnealingLR(T_max=10) stepped per batch, lr every 5 steps:")
print(" ", fmt(cos_epochs[:41:5]))

# 3. Linear warmup then cosine with built-in schedulers equals the LambdaLR of the training example.
seq = trajectory(lambda opt: SequentialLR(opt, [
    LinearLR(opt, start_factor=1 / WARM, end_factor=1.0, total_iters=WARM - 1),
    CosineAnnealingLR(opt, T_max=T - WARM)], milestones=[WARM]), T)
lam = trajectory(lambda opt: LambdaLR(opt, lambda t: (t + 1) / WARM if t < WARM
                                      else 0.5 * (1 + math.cos(math.pi * (t - WARM) / (T - WARM)))), T)
print(f"\nSequentialLR(LinearLR, CosineAnnealingLR) vs LambdaLR: max difference "
      f"{max(abs(a - b) for a, b in zip(seq, lam)):.1e}")

# 4. The first step uses the optimizer's lr times start_factor (0.05 * 0.05 = 0.0025).
print("\nLinearLR(start_factor=0.05, total_iters=40), lr=0.05, first 4 steps:",
      fmt(trajectory(lambda opt: LinearLR(opt, start_factor=0.05, total_iters=40), 4, lr=0.05), 6))

# 5. ReduceLROnPlateau is stepped with a validation metric, once per epoch.
opt = torch.optim.SGD([torch.nn.Parameter(torch.zeros(1))], lr=0.1)
plateau = ReduceLROnPlateau(opt, mode="min", factor=0.1, patience=2)
val_loss = [1.0, 0.8, 0.7, 0.71, 0.70, 0.72, 0.69, 0.70, 0.70, 0.71]
print("\nReduceLROnPlateau(patience=2): epoch, validation loss, lr for the next epoch")
for epoch, v in enumerate(val_loss):
    plateau.step(v)
    print(f"  {epoch} {v:.2f} {opt.param_groups[0]['lr']:.3g}")

Output:

OneCycleLR, lr at the start of each epoch
  stepped per batch: 0.120 0.858 2.316 3.000 2.839 2.413 1.806 1.139 0.543 0.137
  stepped per epoch: 0.120 0.121 0.126 0.133 0.142 0.155 0.170 0.189 0.209 0.233
  per batch: peak 3.000 at step 71, last 1.20e-05

CosineAnnealingLR(T_max=10) stepped per batch, lr every 5 steps:
  3.000 1.500 0.000 1.500 3.000 1.500 0.000 1.500 3.000

SequentialLR(LinearLR, CosineAnnealingLR) vs LambdaLR: max difference 1.3e-15

LinearLR(start_factor=0.05, total_iters=40), lr=0.05, first 4 steps: 0.002500 0.003688 0.004875 0.006063

ReduceLROnPlateau(patience=2): epoch, validation loss, lr for the next epoch
  0 1.00 0.1
  1 0.80 0.1
  2 0.70 0.1
  3 0.71 0.1
  4 0.70 0.1
  5 0.72 0.01
  6 0.69 0.01
  7 0.70 0.01
  8 0.70 0.01
  9 0.71 0.001

• OneCycleLR stepped per batch starts at $3.0/25 = 0.12$, peaks at step 71 (30% of the 240 steps) and ends at $0.12/10^4$. Stepped once per epoch, it covers only the first 10 of its 240 steps and never leaves the warmup.
• CosineAnnealingLR(T_max=10) meant as 10 epochs but stepped per batch reaches zero after 10 steps and climbs back to 3.0 at step 20, repeating with period $2 T_{\max}$.
• The warmup-plus-cosine schedule of the training example can be built from SequentialLR, LinearLR and CosineAnnealingLR and agrees with the LambdaLR to rounding error.
• The first step uses the base rate times start_factor, 0.0025. The comment in PyTorch’s own LinearLR example (version 2.14) lists 0.003687 for epoch 0, the value after the first step(), so the documented examples are offset by one step from the rate that trains the first epoch (our reading of that example).
• ReduceLROnPlateau counts epochs without improvement. Epoch 4’s 0.70 ties the best value and does not count as an improvement under the relative threshold of $10^{-4}$, so after the third such epoch (epoch 5) the rate drops tenfold; the new best of 0.69 at epoch 6 resets the count.

Variants

• Warm restarts (SGDR). Loshchilov and Hutter (2017) restart the cosine at $\eta_{\max}$ every $T_i$ epochs, keeping the weights, and let each period grow by a factor $T_{\text{mult}}$; with $T_0 = 10$ and $T_{\text{mult}} = 2$ the restarts come at epochs 10, 30, 70 and 150. Because a restart temporarily worsens the model, they recommend the weights at the end of a period, where $\eta = \eta_{\min}$. Following an approach proposed by Huang et al. (snapshot ensembles), they also ensembled the models obtained just before restarts. PyTorch’s CosineAnnealingWarmRestarts implements SGDR.
• Cyclical learning rates (CLR). Smith (2017) lets the rate oscillate between bounds, rising and falling linearly over a cycle (the triangular policy), with a half-cycle (stepsize) of 2–10 epochs; he found triangular, parabolic (Welch) and sinusoidal (Hann) windows equivalent. Variants halve the amplitude every cycle (triangular2) or shrink the bounds exponentially (exp_range). PyTorch’s CyclicLR implements all three.
• The learning-rate range test (Smith, 2017) chooses the bounds: train for a few epochs while the rate rises linearly from very small to large, plot accuracy against rate, and take the rate where accuracy starts to rise as the lower bound and the rate where its rise slows, becomes ragged or falls as the upper bound. Smith also gives a rule of thumb: the optimal rate is usually within a factor of two of the largest rate that converges, so the lower bound can be set to 1/3 or 1/4 of the upper.
• 1cycle and super-convergence. Smith and Topin (2019) use a single cycle shorter than training, rising from a low rate to a large maximum and back, followed by a final phase that lowers the rate several orders of magnitude below the start, while momentum cycles inversely (0.95 to 0.85 in many of their runs). They argue that large rates regularize training, so other regularization, such as weight decay, must be reduced to compensate. PyTorch’s OneCycleLR defaults to a two-phase version following fastai; three_phase=True mimics the paper.
• ReduceLROnPlateau automates AlexNet’s manual rule: drop the rate by factor when a monitored metric has not improved for patience epochs.
• Inverse square root decay after warmup, the original Transformer’s schedule, never reaches zero, so the training length need not be fixed in advance (our reading).

Best Practices

• Default to linear warmup plus cosine decay to zero for training a vision model from scratch; recent recipes do (table below). Step decay remains a fine baseline and is what many older papers report.
• Fix the schedule in iterations or epochs consistently and step the scheduler at the matching granularity. Per-iteration stepping gives smooth warmup even when warmup is shorter than an epoch.
• Scale the peak rate with the batch size (linear scaling rule) and lengthen warmup when the peak rate grows; DeiT scales its rate as $0.0005 \times B/512$ (Touvron et al., 2021, supplementary Table 9).
• Find the peak rate with a short range test or a coarse logarithmic sweep rather than tuning the schedule shape.
• Treat schedule, weight decay and training length as one setting. In SGD with an L2 penalty the best learning rate and weight decay lie on a diagonal and must change together; decoupled weight decay makes them more separable (Loshchilov & Hutter, 2019). In their AdamW and SGDW the schedule multiplier also scales the weight decay, as the learning rate does in PyTorch’s AdamW (Adam), so decaying the rate decays the weight decay. They found the best weight decay to depend on the number of batch passes and proposed scaling it as $\lambda = \lambda_{\text{norm}}\sqrt{b/(BT)}$ for batch size $b$, $B$ training images and $T$ epochs (their Appendix B.1). Smith and Topin (2019) cut weight decay to $3 \times 10^{-6}$–$10^{-5}$ for 1cycle on ImageNet.

Training schedules in recipes discussed in the ResNet and Transformer articles, as printed in each paper:

Recipe (ImageNet-1k)	Optimizer, batch	Peak rate	Warmup	Decay	Epochs
Goyal et al. (2017), ResNet-50	SGD, 8,192	3.2	5 epochs, linear from 0.1	÷10 at 30, 60, 80	90
He, T. et al. (2019), ResNet-50-D with refinements	SGD (Nesterov), 1,024	$0.1 \times 1024/256$	linear from 0	cosine	120 (200 with mixup)
DeiT-B (ViT; Touvron et al., 2021)	AdamW, 1,024	$0.0005 \times 1024/512$	5 epochs	cosine	300
ResNet-50 A1 (Wightman et al., 2021)	LAMB, 2,048	0.005	5 epochs	cosine	600
ConvNeXt (Liu et al., 2022)	AdamW, 4,096	0.004	20 epochs, linear	cosine	300

Bag of Tricks illustrates warmup with 5 epochs without stating the length of its own runs, and its baseline at batch 256 used step decay at epochs 30, 60 and 90. ConvNeXt’s settings are in the supplementary material of the CVPR paper; isotropic ConvNeXts used 50 warmup epochs.

Pitfalls

• Wrong stepping unit. A scheduler configured in epochs but stepped per batch runs through its schedule many times over; OneCycleLR stepped per epoch never leaves its warmup (both shown above).
• Calling scheduler.step() before optimizer.step() skips the first value of the schedule (PyTorch documentation).
• Changing the number of epochs without changing the schedule. Cosine, polynomial and 1cycle schedules are tied to $T$; stopping a cosine run early leaves the rate high, and a longer run needs a new $T$, not more epochs at the final rate.
• Evaluating mid-cycle. With restarts or cyclical rates, accuracy is best at the end of a cycle, where the rate is lowest (Loshchilov & Hutter, 2017; Smith, 2017).
• Constant warmup at large batch sizes. The jump from the low to the full rate can cause the spike that Goyal et al. (2017) observed; ramp linearly instead.
• Ignoring normalization statistics. In the example, a constant rate of 0.1 left batch-normalization running averages out of step with the weights, in our reading; with a decayed rate the two loss columns agree, and normalizing with statistics recomputed on the training images (the batch-statistics column) gives a low training loss even at a constant rate.
• Comparing models trained with different schedules. Published ImageNet results for the same ResNet-50 differ by several points with the training recipe, of which the schedule is one ingredient (ResNet has the recipe table and Wightman et al.’s observation on this).

Evidence and Impact

• Cosine and restarts. On CIFAR-10 and CIFAR-100 with a WRN-28-10, the step schedule of Zagoruyko and Komodakis (÷5 at epochs 60, 120 and 160 of 200), with an initial rate of 0.05 that did better than the original 0.1, gave 4.13% and 20.21% test error (median of 5 runs). A single 200-epoch cosine cycle ($T_0 = 200$, no restart) gave 3.86% and 19.98%, and SGDR with $T_0 = 10$, $T_{\text{mult}} = 2$ gave 4.03% and 19.58% while reaching about 4% and 20% two to four times sooner (Loshchilov & Hutter, 2017, Table 1). They mark better results in bold without statistical tests, noting too few runs.
• Cosine on ImageNet. Adding cosine decay to an otherwise fixed recipe raised top-1 validation accuracy from 77.26% ± 0.07 to 77.73% ± 0.13 for ResNet-50-D (mean of 4 runs), from 77.50% to 78.19% for Inception-V3 and from 71.90% to 72.83% for MobileNet (He, T. et al., 2019, CVPR version, Table 6; the arXiv version’s table gives 77.16% and 77.91% for ResNet-50-D). In the same paper, warmup raised ResNet-50 from 75.17% to 75.93% at batch 1,024 but only from 75.87% to 76.03% at batch 256 (Table 4).
• Cyclical rates. On CIFAR-10 with Caffe’s example network, the triangular2 policy reached the 81.4% test accuracy of the standard settings (labelled “fixed”) in 25,000 instead of 70,000 iterations; on ImageNet it gave 58.4% for AlexNet against 58.0%, and 64.4% for GoogLeNet against 63.0%, at the same number of iterations as the “fixed” baselines (Smith, 2017, Table 1, arXiv version).
• Super-convergence. In the arXiv version of Smith and Topin’s paper, a ResNet-56 on CIFAR-10 reached 92.4% test accuracy in 10,000 iterations with one cycle, against 91.2% after about 80,000 iterations of a piecewise-constant schedule, and a wide ResNet reached 91.9% ± 0.2 in 100 epochs with 1cycle against 90.3% ± 1.0 in 800 with step decay (Table 2). The gains were larger with less training data. On ImageNet the comparison is weaker: 1cycle trained ResNet-50 to 67.6% top-1 in 20 epochs with reduced weight decay, against 63.7% after 100 epochs for a baseline with Inception’s exponential schedule, whose curve the authors describe as still heading toward about 65% (curves averaged over two runs).
• Adaptive optimizers. RAdam (Liu et al., 2020), which has no warmup length to set, matched Adam with linear warmup on three translation tasks (BLEU 34.76 ± 0.003 against 34.66 ± 0.014 on IWSLT’14 German–English, 5 seeds; Table 2).