Early Stopping

Early stopping ends training at the point where the error on a held-out validation set is lowest, rather than when the training loss stops falling. The training loop stores a copy of the parameters whenever the validation error improves, stops once it has failed to improve for a set number of evaluations (the patience), and returns the stored copy. Goodfellow, Bengio and Courville (2016, §7.8) call it probably the most commonly used form of regularization in deep learning, and their practical advice (§11.2) is that it should be used almost universally. The training workflow article places it in the training loop and runs it on a small convolutional network; this article covers the procedure in detail, the theory that links it to weight decay, its interaction with learning-rate schedules and double descent, and the settings where it is unreliable.

Purpose

A network with enough capacity to memorize its training images can, if trained long enough, fit their noise as well as their content. Goodfellow et al. describe what is then often observed: the training loss keeps falling while the validation loss first falls and then rises, the asymmetric U-shaped curve of the Deep Learning book’s Figure 7.3, which the authors say occurs reliably. Early stopping returns the parameters from the bottom of that U.

Goodfellow et al. offer two views of what this does. As hyperparameter selection, the number of training steps is a hyperparameter like any other, with a U-shaped validation curve, but it is unusual in that a single training run tries out all its values; the only extra cost is evaluating the validation set periodically and keeping one copy of the best parameters. As regularization, limiting the number of steps limits how far the parameters can travel from their initialization, which for a linear model with a quadratic loss is equivalent to an L2 penalty (see How It Works).

The idea is older than deep learning. Raskutti, Wainwright and Yu (2014) trace it to the Landweber iteration in the 1970s and to neural network training by Morgan and Bourlard; they date the latter work to 1990, while Ali, Kolter and Tibshirani (2019) give 1989. The same semiconvergence appears in iterative deblurring, where the image deblurring article shows Landweber iteration improving an image and then fitting its noise.

When to Use It

Training from scratch or fine-tuning with limited data. Any run long enough to overfit benefits; with few labels, fine-tuning overfits quickly (transfer learning).
Alongside other regularizers. Goodfellow et al. note that it can be used alone or with other regularization strategies, and that even with regularizers that change the objective, the best generalization rarely comes at a minimum of the training objective.
Label noise. Li, Soltanolkotabi and Oymak (2020) trained a four-layer network on MNIST with a fraction of training labels replaced at random. Trained for many iterations it fitted the corrupted labels and its test accuracy fell; stopped early, its test accuracy stayed high even with a large fraction corrupted, and its accuracy on the training images, measured against the original labels, stayed around 100% even when half were corrupted (their Figure 1). They prove that for a one-hidden-layer network on clustered data, gradient descent first fits the correct labels and only fits the corrupted ones after moving far from its initialization.

It helps less, or misleads, when:

The training set is huge relative to the model. If the validation error never rises within the training budget, early stopping only ends training when the budget does; in our reading, the length of the learning rate schedule then decides when training ends.
The validation measure is not the goal. In closed-loop tasks such as robot control, the validation loss of a policy is only a surrogate for task success; see Pitfalls.
The validation set is small. Its minimum is then partly noise; the example below measures how much.

How It Works

The procedure

The Deep Learning book states the procedure as Algorithm 7.1. Let $n$ be the number of training steps between evaluations and $p$ the patience. Starting from the initial parameters $\theta_0$ , with best validation error $v = \infty$ and a counter $j = 0$ :

Train for $n$ steps; compute the validation error $v'$ .
If $v' < v$ , store $\theta^* \leftarrow \theta$ , the step count $i^*$ and $v \leftarrow v'$ , and reset $j \leftarrow 0$ ; otherwise increase $j$ by one.
Repeat until $j = p$ . Return $\theta^*$ and $i^*$ .

The stored copy is written rarely and never read during training, so it can sit in slower memory, such as host memory or disk. The algorithm returns $\theta^*$ , not the final parameters, and that distinction matters in practice: some libraries default to the final ones (see How to Apply It).

Using the validation data afterwards

Early stopping uses data that the model never trains on. Goodfellow et al. give two ways to recover it once the stopping point is known:

Retrain from scratch (Algorithm 7.2). Split the training set into subtrain and validation parts, run early stopping to find $i^*$ , reinitialize, and train on the whole training set for $i^*$ steps. It is unclear whether to keep the number of parameter updates or the number of passes through the data, since a larger training set needs more updates per pass.
Continue training (Algorithm 7.3). Keep $\theta^*$ , record the subtrain loss $\epsilon$ at which early stopping halted, and continue training on all the data until the loss on the validation part falls below $\epsilon$ . This avoids a second full run but is less well behaved: the validation loss may never reach $\epsilon$ , so the procedure is not guaranteed to end.

The published record shows both habits. The CIFAR article describes cuda-convnet’s documented procedure, which used one of the five training batches for early stopping and then retrained on all five; the arXiv revisions of EfficientNet hold out 25,000 ImageNet training images as a minival set for early stopping and evaluate the early-stopped checkpoint on the original validation set.

Early stopping as L2 regularization

Goodfellow et al. (§7.8) derive the equivalence for a quadratic approximation of the loss around its minimum $w^*$ , $\hat J(w) = J(w^*) + \tfrac{1}{2}(w - w^*)^\top H (w - w^*)$ , with $H = Q \Lambda Q^\top$ and eigenvalues $\lambda_i$ . Gradient descent with learning rate $\epsilon$ from $w^{(0)} = 0$ gives, after $\tau$ steps (their eq. 7.40),

Q^\top w^{(\tau)} = \left[I - (I - \epsilon\Lambda)^\tau\right] Q^\top w^*,

provided $|1 - \epsilon\lambda_i| < 1$ , while L2 regularization with coefficient $\alpha$ (a penalty $\tfrac{\alpha}{2} w^\top w$ ) gives (eq. 7.41)

Q^\top \tilde w = (\Lambda + \alpha I)^{-1} \Lambda\, Q^\top w^* .

Both shrink each eigen-direction of $w^*$ by a factor, which this article calls the filter factor (the book gives it no name; Ali et al., below, call the map a spectral shrinkage map): $1 - (1 - \epsilon\lambda_i)^\tau$ for early stopping and $\lambda_i / (\lambda_i + \alpha)$ for weight decay. Directions of high curvature are learned early and kept by both; directions of low curvature are learned late and shrunk by both. The two are equal when $(I - \epsilon\Lambda)^\tau = (\Lambda + \alpha I)^{-1}\alpha$ (eq. 7.43). If all $\lambda_i$ are small, in the book’s sense $\epsilon\lambda_i \ll 1$ and $\lambda_i/\alpha \ll 1$ , taking logarithms gives $\tau \approx 1/(\epsilon\alpha)$ (eq. 7.44), that is (eq. 7.45)

\alpha \approx \frac{1}{\tau \epsilon},

so the number of steps times the learning rate plays the role of the reciprocal of the weight-decay coefficient. The argument holds for any initial value, not only zero (the book’s footnote), but it is exact only for a quadratic loss.

Ali, Kolter and Tibshirani (2019) sharpened this for linear least squares, with loss $\lVert y - X\beta\rVert^2/2n$ . They identify gradient descent with step $\epsilon$ after $k$ steps with gradient flow at time $t = k\epsilon$ (here $t = \tau\epsilon$ ). In the limit of small steps (gradient flow), the factor becomes $1 - e^{-t s_i}$ , with $s_i$ the eigenvalues of $X^\top X/n$ , against $s_i/(s_i + \lambda)$ for ridge regression; their ridge objective, $\lVert y - X\beta\rVert^2/n + \lambda\lVert\beta\rVert^2$ , is twice the book’s form with $\alpha = \lambda$ . No single calibration $\lambda(t)$ makes the two maps equal, but with the calibration $\lambda = 1/t$ they prove that the estimation risk of gradient flow is at most 1.6862 times that of ridge, along the whole path and with no assumption on the features (their Theorem 1), and that the same holds for out-of-sample prediction risk on average over a spherical prior on the true coefficients. When both are tuned optimally, the ratio of their Bayes risks lies between 1 and 1.2147 (Theorem 3). In the numerical example they show (Gaussian features, $\Sigma = I$ , $n = 500$ , $p = 1000$ ), the ratio of the minimum Bayes risks was 1.0036. As a step of our own, write $x = t s_i$ : the two factors are $1 - e^{-x}$ and $x/(1+x)$ , which agree as $x \to 0$ and as $x \to \infty$ and differ most, by 0.2036, at $x \approx 2.5$ . That is the size of the mismatch the calibration leaves.

For deep networks the equivalence is a heuristic. The loss is not quadratic, the optimizer is not plain gradient descent, and the learning rate changes during training. The view that survives is the one Goodfellow et al. start from: if the gradient is bounded, the product of steps and step size bounds how far training can move from the initialization.

How to Apply It

A minimal example

The script below shows both halves of the technique on a convex problem whose iterates are known exactly. It learns a linear deblurring filter: an $11 \times 11$ filter that maps a patch of a blurred, noisy image to the clean centre pixel. The 14 synthetic $64 \times 64$ images each have a ramp background and eight discs or squares. They are blurred with a Gaussian of $\sigma = 1.5$ pixels, and noise of standard deviation 0.02 is added. Four images supply the 300 training pixels (the subtrain set) and two the 100 validation pixels, drawn from the 5,832 patch-centre pixels of those two images, which form the validation pool for Part 2; eight images supply 23,328 test pixels. Errors are mean squared errors over pixels. The filter $w$ (121 weights, no bias) minimizes $J(w) = \lVert Xw - y\rVert^2/2n$ by gradient descent from zero with step $\epsilon = 1/s_{\max}$ , where $s_{\max}$ is the largest eigenvalue of $H = X^\top X/n$ . Ridge adds $\tfrac{\alpha}{2}\lVert w\rVert^2$ . Both are computed in closed form in the eigenbasis of $H$ , after checking the closed form against 2,000 explicit iterations.

The test pixels are used here as ground truth, to measure what each rule achieves. A real training run cannot use them that way. In Part 1, $\tau^*$ and $\alpha^*$ are the test-optimal step count and ridge coefficient; the checkpoint grid places an evaluation every $\tau^*/20$ steps, 200 in all, so that the best checkpoint falls near the twentieth. Part 2 keeps that gradient descent trajectory fixed and simulates validation curves over repeated draws of validation pixels, drawn with replacement from the validation pool, in two ways that the sources leave open, so both are run. In the “fresh” case each evaluation draws new pixels, so the noise is independent from one checkpoint to the next, as it is when a new validation subset is drawn at each evaluation. In the “fixed” case one set is drawn per run and used at every checkpoint, the usual practice. Each rule is Algorithm 7.1 with patience $P$ , returning either the stored best checkpoint (“restore”) or the checkpoint at which it stopped (“last”). A rule succeeds when its checkpoint’s test error is within 1% of the best checkpoint’s. The script uses NumPy 2.3.5 and SciPy 1.16.3 and ran in about 4 to 5 seconds on a laptop CPU.

import numpy as np
from scipy.ndimage import gaussian_filter

SIZE, RAD, BLUR, NOISE = 64, 5, 1.5, 0.02   # image size, patch radius (11x11 filter), blur, noise sd
N_SUB, N_VAL, K = 300, 100, 200             # subtrain and validation pixels, checkpoints
R_DRAWS, SEEDS = 2000, range(10)            # 2,000 draws, data seeds 0-9

def make_image(rng):                         # ramp background plus 8 discs or squares
    yy, xx = np.mgrid[0:SIZE, 0:SIZE].astype(float)
    a, b = rng.uniform(0.2, 0.8, 2)
    th = rng.uniform(0, 2 * np.pi)
    img = a + (b - a) * (np.cos(th) * xx + np.sin(th) * yy + SIZE - 1) / (2 * SIZE - 2)
    for _ in range(8):
        cy, cx = rng.uniform(0, SIZE, 2)
        r, v = rng.uniform(4, 12), rng.uniform(0, 1)
        if rng.random() < 0.5:
            img[(yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2] = v
        else:
            img[(np.abs(yy - cy) <= r) & (np.abs(xx - cx) <= r)] = v
    return img

def dataset(seed):
    rng = np.random.default_rng(seed)
    clean = [make_image(rng) for _ in range(14)]
    blurred = [gaussian_filter(im, BLUR, mode="reflect") for im in clean]
    noisy = [b + NOISE * rng.standard_normal(b.shape) for b in blurred]
    def pairs(ids):                          # 11x11 degraded patch -> clean centre pixel
        k = 2 * RAD + 1
        X = np.vstack([np.lib.stride_tricks.sliding_window_view(noisy[i], (k, k)).reshape(-1, k * k)
                       for i in ids]) - 0.5
        y = np.concatenate([clean[i][RAD:-RAD, RAD:-RAD].ravel() for i in ids]) - 0.5
        return X, y
    Xtr, ytr = pairs(range(0, 4)); Xvp, yvp = pairs(range(4, 6)); Xte, yte = pairs(range(6, 14))
    i = rng.choice(len(ytr), N_SUB, replace=False)
    j = rng.choice(len(yvp), N_VAL, replace=False)
    info = (np.min(clean), np.max(clean), np.std(np.array(noisy) - np.array(blurred)))
    return Xtr[i], ytr[i], Xvp[j], yvp[j], Xvp, yvp, Xte, yte, info

def mse(X, y, W):                            # W is (p,) or (p, k): one column per model
    return np.mean((X @ W - (y if W.ndim == 1 else y[:, None])) ** 2, axis=0)

class LinearGD:
    """Gradient descent from zero on J(w) = |Xw - y|^2 / 2n, and ridge, in the eigenbasis of H."""
    def __init__(self, X, y, eps=None):
        n = len(y)
        self.H, self.g = X.T @ X / n, X.T @ y / n
        self.s, self.V = np.linalg.eigh(self.H)
        self.eps = 1.0 / self.s[-1] if eps is None else eps
        self.z = (self.V.T @ self.g) / self.s            # least-squares solution, eigenbasis
    def gd_factor(self, tau):
        return 1 - (1 - self.eps * self.s[:, None]) ** np.atleast_1d(tau)[None, :]
    def ridge_factor(self, alpha):
        return self.s[:, None] / (self.s[:, None] + np.atleast_1d(alpha)[None, :])
    def w_gd(self, tau):
        return self.V @ (self.gd_factor(tau) * self.z[:, None])
    def w_ridge(self, alpha):
        return self.V @ (self.ridge_factor(alpha) * self.z[:, None])

# ---------------- Part 1: early stopping against ridge (seed 0) ----------------
def part1(seed, show):
    X, y, Xv, yv, Xvp, yvp, Xte, yte, info = dataset(seed)
    m = LinearGD(X, y)
    kappa = m.s[-1] / m.s[0]
    T_MAX = int(10 * kappa)
    taus = np.unique(np.round(np.logspace(0, np.log10(T_MAX), 600)).astype(int))
    te_gd = mse(Xte, yte, m.w_gd(taus))
    te_r = mse(Xte, yte, m.w_ridge(1 / (m.eps * taus)))   # same grid, alpha = 1/(eps tau)
    k1, k2 = np.argmin(te_gd), np.argmin(te_r)
    tau_s, alpha_s = taus[k1], 1 / (m.eps * taus[k2])
    out = dict(ratio=te_gd[k1] / te_r[k2], aet=alpha_s * m.eps * tau_s)
    if not show:
        return out
    print(f"data: 14 images {SIZE}x{SIZE}, values in [{info[0]:.3f}, {info[1]:.3f}], noise sd {info[2]:.4f}; n={len(y)}, p={X.shape[1]},"
          f" validation {len(yv)}, validation pool {len(yvp)}, test {len(yte)}")
    print(f"H: s_max {m.s[-1]:.3f}, s_min {m.s[0]:.2e}, condition number {kappa:.0f}; eps = 1/s_max = {m.eps:.4f}")
    # checks: closed form against iteration, ridge eigenbasis against a direct solve
    w, d1 = np.zeros(X.shape[1]), 0.0
    for t in range(1, 2001):
        w = w - m.eps * (m.H @ w - m.g)
        if t in (1, 10, 100, 1000, 2000):
            d1 = max(d1, np.abs(w - m.w_gd(t)[:, 0]).max())
    al = np.array([1e-4, 1e-2, 1.0])
    ws = np.stack([np.linalg.solve(m.H + a * np.eye(len(m.s)), m.g) for a in al], 1)
    print(f"check: max |GD iterate - closed form| {d1:.1e}; max |ridge eigenbasis - solve| "
          f"{np.abs(ws - m.w_ridge(al)).max():.1e}")
    w_ols = m.V @ m.z
    print(f"baselines, test MSE: predict grey {np.mean(yte ** 2):.5f}, degraded centre pixel "
          f"{np.mean((Xte[:, X.shape[1] // 2] - yte) ** 2):.5f}, least squares {mse(Xte, yte, w_ols):.5f}"
          f" (train {mse(X, y, w_ols):.5f})")
    print("\n     tau   train MSE  val MSE   test MSE | ridge alpha=1/(eps tau)  test MSE")
    for t in [1, 10, 100, 1000, tau_s, 10_000, 100_000, T_MAX]:
        wt, a = m.w_gd(t)[:, 0], 1 / (m.eps * t)
        print(f"{t:8d}   {mse(X, y, wt):.5f}   {mse(Xv, yv, wt):.5f}   {mse(Xte, yte, wt):.5f} |"
              f"       {a:9.2e}          {mse(Xte, yte, m.w_ridge(a)[:, 0]):.5f}")
    print(f"\nbest GD: tau* = {tau_s}, test {te_gd[k1]:.5f};  best ridge: alpha* = {alpha_s:.2e}, "
          f"test {te_r[k2]:.5f};  ratio {out['ratio']:.3f};  alpha* eps tau* = {out['aet']:.3f}")
    f_gd = m.gd_factor(tau_s)[:, 0]
    f_r = m.ridge_factor(1 / (m.eps * tau_s))[:, 0]
    print(f"filter factors at tau*, alpha = 1/(eps tau*) = {1 / (m.eps * tau_s):.2e}:")
    print("  eigenvalue s   GD 1-(1-eps s)^tau   ridge s/(s+alpha)")
    for i in [120, 100, 80, 60, 40, 20, 0]:
        print(f"  {m.s[i]:11.2e}   {f_gd[i]:18.3f}   {f_r[i]:17.3f}")
    print(f"  max |GD - ridge| over all {len(m.s)} directions: {np.abs(f_gd - f_r).max():.3f}")
    # Algorithms 7.1-7.3 on a checkpoint grid
    n_eval = int(round(tau_s / 20))
    ck = n_eval * np.arange(1, K + 1)
    W = m.w_gd(ck)
    val, true = mse(Xv, yv, W), mse(Xte, yte, W)
    kv = np.argmin(val)
    print(f"\ncheckpoints every {n_eval} steps, {K} checkpoints; local minima of the test curve: "
          f"{np.sum((true[1:-1] < true[:-2]) & (true[1:-1] < true[2:]))}; best test checkpoint {np.argmin(true) + 1}")
    print(f"Alg. 7.1, best of {N_VAL} validation pixels: checkpoint {kv + 1} (tau {ck[kv]}), test {true[kv]:.5f}"
          f"  [test-optimal checkpoint: {true.min():.5f}]")
    Xa, ya = np.vstack([X, Xv]), np.concatenate([y, yv])
    ma = LinearGD(Xa, ya, eps=m.eps)
    print(f"Alg. 7.2, refit on all {len(ya)} pixels for {ck[kv]} steps (eps s_max = {m.eps * ma.s[-1]:.3f}):"
          f" test {mse(Xte, yte, ma.w_gd(ck[kv])[:, 0]):.5f}")
    target = mse(X, y, W[:, kv])           # subtrain loss when early stopping halted
    w, steps = W[:, kv].copy(), 0
    while mse(Xv, yv, w) > target and steps < K * n_eval:
        for _ in range(n_eval):
            w = w - m.eps * (ma.H @ w - ma.g)
        steps += n_eval
    print(f"Alg. 7.3, continue on all pixels until validation loss <= {target:.5f}: "
          f"{'reached' if mse(Xv, yv, w) <= target else 'not reached'} after {steps} more steps, test {mse(Xte, yte, w):.5f}")
    return out, (Xvp, yvp, Xte, yte, m, ck, true)

out, (Xvp, yvp, Xte, yte, m, ck, true) = part1(0, True)
rows = [part1(sd, False) for sd in SEEDS]
print(f"\nover data seeds {SEEDS.start}-{SEEDS.stop - 1}: best GD / best ridge test MSE "
      f"{min(r['ratio'] for r in rows):.3f}-{max(r['ratio'] for r in rows):.3f}; alpha* eps tau* "
      f"{min(r['aet'] for r in rows):.3f}-{max(r['aet'] for r in rows):.3f}")

# ---------------- Part 2: patience rules on noisy validation curves ----------------
def stop(v, P):                               # Algorithm 7.1: returns (best index, stop index)
    best, bi, wait = np.inf, 0, 0
    for k, vk in enumerate(v):
        if vk < best:
            best, bi, wait = vk, k, 0
        else:
            wait += 1
            if wait >= P:
                return bi, k
    return bi, len(v) - 1

rules = [(P, keep) for P in (1, 3, 10, 30) for keep in ("restore", "last")] + [(K, "full")]
names = [f"P={P} {keep}" if keep != "full" else "all 200, best" for P, keep in rules]
R2 = (Xvp @ m.w_gd(ck) - yvp[:, None]) ** 2   # squared validation residuals at every checkpoint
tol_ok = true <= true.min() * 1.01

def evaluate(curves):
    ok, exc, cost = np.zeros(len(rules)), np.zeros(len(rules)), np.zeros(len(rules))
    for v in curves:
        for r, (P, keep) in enumerate(rules):
            bi, si = stop(v, P)
            pick = si if keep == "last" else bi
            ok[r] += tol_ok[pick]; exc[r] += true[pick] / true.min() - 1; cost[r] += si + 1
    return 100 * ok / len(curves), 100 * exc / len(curves), cost / len(curves)

rng = np.random.default_rng(1)
print()
res = {"noise-free": evaluate([true])}
print("check, noise-free validation: every restore rule returns the best checkpoint:",
      all(stop(true, P)[0] == np.argmin(true) for P in (1, 3, 10, 30, K)))
for mode in ("fresh", "fixed"):
    for mv in (100, 1000):
        curves = []
        for _ in range(R_DRAWS):
            if mode == "fresh":                # new pixels at every checkpoint
                idx = rng.integers(0, len(yvp), (K, mv))
                curves.append(R2[idx, np.arange(K)[:, None]].mean(axis=1))
            else:                              # one validation set for the whole run
                curves.append(R2[rng.integers(0, len(yvp), mv)].mean(axis=0))
        res[f"{mode}, m={mv}"] = evaluate(curves)
print(f"\n{'':22s}" + "".join(f"{n:>15s}" for n in names))
for label, part in [("% within 1% of best", 0), ("mean excess test MSE %", 1), ("checkpoints evaluated", 2)]:
    print(label)
    for key, val in res.items():
        print(f"  {key:20s}" + "".join(f"{x:15.1f}" for x in val[part]))

Output:

data: 14 images 64x64, values in [0.000, 0.997], noise sd 0.0200; n=300, p=121, validation 100, validation pool 5832, test 23328
H: s_max 3.997, s_min 6.46e-05, condition number 61874; eps = 1/s_max = 0.2502
check: max |GD iterate - closed form| 3.9e-14; max |ridge eigenbasis - solve| 3.7e-13
baselines, test MSE: predict grey 0.04793, degraded centre pixel 0.00438, least squares 0.00469 (train 0.00124)

     tau   train MSE  val MSE   test MSE | ridge alpha=1/(eps tau)  test MSE
       1   0.00908   0.00786   0.00967 |        4.00e+00          0.01989
      10   0.00728   0.00662   0.00837 |        4.00e-01          0.00892
     100   0.00323   0.00303   0.00481 |        4.00e-02          0.00523
    1000   0.00208   0.00230   0.00387 |        4.00e-03          0.00396
    4131   0.00164   0.00206   0.00355 |        9.68e-04          0.00363
   10000   0.00143   0.00216   0.00365 |        4.00e-04          0.00362
  100000   0.00124   0.00326   0.00456 |        4.00e-05          0.00421
  618735   0.00124   0.00341   0.00469 |        6.46e-06          0.00458

best GD: tau* = 4131, test 0.00355;  best ridge: alpha* = 5.93e-04, test 0.00360;  ratio 0.985;  alpha* eps tau* = 0.613
filter factors at tau*, alpha = 1/(eps tau*) = 9.68e-04:
  eigenvalue s   GD 1-(1-eps s)^tau   ridge s/(s+alpha)
     4.00e+00                1.000               1.000
     1.12e-03                0.685               0.536
     6.03e-04                0.464               0.384
     4.11e-04                0.346               0.298
     2.73e-04                0.246               0.220
     1.59e-04                0.152               0.141
     6.46e-05                0.065               0.063
  max |GD - ridge| over all 121 directions: 0.200

checkpoints every 207 steps, 200 checkpoints; local minima of the test curve: 1; best test checkpoint 20
Alg. 7.1, best of 100 validation pixels: checkpoint 20 (tau 4140), test 0.00355  [test-optimal checkpoint: 0.00355]
Alg. 7.2, refit on all 400 pixels for 4140 steps (eps s_max = 0.978): test 0.00339
Alg. 7.3, continue on all pixels until validation loss <= 0.00164: reached after 828 more steps, test 0.00345

over data seeds 0-9: best GD / best ridge test MSE 0.979-1.002; alpha* eps tau* 0.563-0.694

check, noise-free validation: every restore rule returns the best checkpoint: True

                          P=1 restore       P=1 last    P=3 restore       P=3 last   P=10 restore      P=10 last   P=30 restore      P=30 last  all 200, best
% within 1% of best
  noise-free                    100.0          100.0          100.0          100.0          100.0          100.0          100.0            0.0          100.0
  fresh, m=100                    0.0            0.0            0.8            3.5           34.4           87.7           49.5            3.5           30.1
  fresh, m=1000                   0.0            0.0            2.7           13.1           54.7           88.4           57.7            0.1           45.7
  fixed, m=100                   64.4           65.4           65.0           69.2           65.0           62.0           65.0            0.3           65.0
  fixed, m=1000                  99.7           99.7           99.7           99.0           99.7           78.9           99.7            0.0           99.7
mean excess test MSE %
  noise-free                      0.0            0.0            0.0            0.1            0.0            0.6            0.0            3.1            0.0
  fresh, m=100                   22.6           16.9           14.5            6.0            5.2            0.4            2.5            3.6            4.6
  fresh, m=1000                  20.2           15.0            9.4            3.9            2.0            0.4            1.3            3.9            2.2
  fixed, m=100                    1.9            1.6            1.5            1.2            1.5            1.5            1.5            3.7            1.5
  fixed, m=1000                   0.1            0.1            0.1            0.2            0.1            0.7            0.1            3.2            0.1
checkpoints evaluated
  noise-free                     21.0           21.0           23.0           23.0           30.0           30.0           50.0           50.0          200.0
  fresh, m=100                    2.9            2.9            7.0            7.0           20.8           20.8           53.9           53.9          200.0
  fresh, m=1000                   3.3            3.3            8.9            8.9           24.5           24.5           56.5           56.5          200.0
  fixed, m=100                   26.3           26.3           28.7           28.7           35.6           35.6           55.5           55.5          200.0
  fixed, m=1000                  22.0           22.0           24.0           24.0           31.0           31.0           51.0           51.0          200.0

Part 1: the U-shape and the ridge correspondence. The training error falls at every step and reaches the least-squares value. The test error falls to its minimum at $\tau^* = 4{,}131$ and then rises. Run to convergence, the learned filter does worse on test images than simply copying the degraded centre pixel, whereas the early-stopped filter does about 19% better. The condition number of $H$ is about 62,000: the smooth, blurred patches leave many directions with tiny eigenvalues, and those are the directions in which least squares amplifies noise. Ridge at $\alpha = 1/(\epsilon\tau)$ follows the same U, and the two optima are close. Gradient descent’s best test error is 1.5% below ridge’s on this dataset, and over ten datasets the ratio ranges from 0.979 to 1.002. The calibration itself is looser: the best ridge coefficient is 0.56 to 0.69 times $1/(\epsilon\tau^*)$ , off by a factor of about 1.4 to 1.8. The filter-factor table shows why the two do not coincide. The factors agree in the strongest direction (both 1) and in the weakest (0.065 against 0.063), but in between gradient descent keeps more of each direction, by up to 0.200, close to the 0.2036 that the gradient-flow comparison above predicts. In our reading, ridge needs a smaller $\alpha$ than $1/(\epsilon\tau^*)$ to keep as much of those middle directions, which is the direction of the printed offset.

Algorithms 7.1–7.3. With this 100-pixel validation set, Algorithm 7.1 run over all 200 checkpoints (a patience as long as the budget) picked exactly the test-optimal checkpoint. Part 2 shows that this is a favourable draw: a fixed 100-pixel set lands within 1% of the best checkpoint only about 65% of the time. Using the validation pixels for training afterwards helped in both ways. Retraining from scratch on all 400 pixels for the same number of steps lowered the test error from 0.00355 to 0.00339. Continuing training until the validation loss reached the subtrain loss at the stop did terminate here, after 828 steps, and gave 0.00345.

Part 2: patience and noise. With a noise-free validation curve every rule finds the best checkpoint, and returning the last checkpoint costs nothing until the patience carries the run past the region near the minimum. With $P = 30$ , the “last” rule returns a checkpoint 30 evaluations late, 3.1% worse. A fixed validation set produces a smooth curve, so the rules stop near checkpoint 20 plus the patience, and the patience hardly matters. What matters is the size of the set. With 1,000 pixels every restore rule is within 1% in 99.7% of the draws; with 100 pixels, in 64% to 65%, whichever patience is used. In our reading, the minimum of a small set’s curve lies away from the test curve’s minimum in a third of the draws, and more patience does not move it. Independent noise from one evaluation to the next behaves differently. Patience 1 or 3 stops after 3 to 9 evaluations on average, and the stored checkpoints have test error 9% to 23% above the best. Even training for all 200 checkpoints and keeping the lowest validation value lands within 1% in only 30% to 46% of the draws. In our reading, the lowest of many independent noisy values is chosen as much by the noise as by the curve. In this run, the checkpoint at which the $P = 10$ rule stopped was within 1% far more often (88%) than the checkpoint it had stored (34% to 55%). The output does not show whether that holds on other grids. It was measured with one checkpoint spacing only, which put the best checkpoint at the twentieth evaluation, and the $P = 10$ runs stopped at evaluation 21 to 25 on average. For the restore rules, more pixels per evaluation help in both noise settings, and the cost in evaluations grows with the patience.

Two related examples on the site use other measures. Super-resolution fits a ridge-regression linear filter on similar synthetic shapes and compares upsamplers by PSNR, and Image Deblurring stops Landweber iteration by a discrepancy rule on a single image. The training workflow article’s example shows patience on a convolutional network, whose stopping epoch ranged from 19 to 92 over eight seeds.

In libraries

The common implementations differ in their defaults, which matters because the default is often what gets used. In the source code read for this article:

Keras 3.12.4 (keras.callbacks.EarlyStopping) monitors val_loss with patience=0, min_delta=0 and restore_best_weights=False. By default it therefore stops at the first epoch without improvement and leaves the model with the last weights, not the best. It also has a baseline the metric must beat and a start_from_epoch warm-up during which it does not stop.
PyTorch Lightning 2.6.6 (EarlyStopping) defaults to patience=3 and counts validation checks, not epochs: its docstring notes that with a check every 10 epochs and patience 3, training runs at least 40 epochs. It can also stop when the metric becomes non-finite (check_finite, on by default), passes a stopping_threshold or crosses a divergence_threshold. The callback itself does not restore weights; it only signals the trainer to stop.
scikit-learn 1.7.2’s multilayer perceptron has early_stopping=False by default; when enabled it holds out validation_fraction=0.1 of the training data (stratified for classification), stops after more than n_iter_no_change=10 epochs without an improvement of at least tol=1e-4 in the validation score, and restores the best coefficients. With the SGD solver and learning_rate="adaptive", a stall divides the learning rate by 5 instead of stopping, and training stops only once the rate has fallen to $10^{-6}$ or below.

Variants

Patience and minimum improvement. Patience $p$ in Algorithm 7.1, possibly with a threshold (min_delta, tol) below which a change does not count as an improvement. A threshold makes noise less likely to reset the counter but can stop a run that is still improving slowly.
Evaluation interval. Evaluating every $n$ steps rather than every step trades resolution against cost; Goodfellow et al. suggest evaluating less often, or on a smaller validation set, when evaluation is expensive. A patience counted in evaluations means a different number of epochs at each interval.
Thresholds and divergence checks. Stop as soon as the metric reaches a target, or as soon as it becomes non-finite or worse than a bound (Lightning’s options), which ends failed runs early rather than regularizing good ones.
Stopping without held-out data. For gradient descent on least squares in a reproducing kernel Hilbert space, Raskutti, Wainwright and Yu (2014) propose a data-dependent stopping rule that uses no hold-out or cross-validation data, prove that it attains minimax-optimal rates for several kernel classes, and report simulations in which it compared favourably with a hold-out rule and with one based on Stein’s unbiased risk estimate. In image restoration the analogous rule stops at the first iterate whose residual falls below the noise level, a discrepancy rule that the image deblurring article applies.
Plateau-based learning-rate drops. Instead of stopping, reduce the learning rate when the validation metric stalls, as PyTorch’s ReduceLROnPlateau does (described in the learning rate schedule article) and as scikit-learn’s adaptive mode combines with stopping.
Checkpoint averaging and best-of-run selection. Some fields report the best checkpoint over a run, or the mean of the last several, instead of stopping; see Pitfalls for why the choice matters.

Best Practices

Return the best checkpoint, not the last. Algorithm 7.1 returns $\theta^*$ . Check the library default (Keras’s is the last weights). The example’s noise-free row shows the cost of keeping the last weights growing with the patience; under independent noise the example’s $P = 10$ rule did better with its last checkpoint, on the one checkpoint spacing measured.
Use a validation set large enough for the differences you care about. In the example, a fixed 1,000-pixel set nearly always picked a checkpoint within 1% of the best, and a fixed 100-pixel set did so about two times in three, whatever the patience.
Set patience relative to the noise of the curve, and in evaluations. Under independent noise, a patience of 1 or 3 stopped on the first noisy upturn in the example. With a fixed validation set, larger patience mostly cost extra evaluations.
Keep the test set out of it. Stopping is a hyperparameter choice and belongs on validation data (train, validation and test split). With little data, choose the number of epochs by cross-validation and retrain.
Decide how to use the validation data afterwards. Retraining on all data for the selected number of steps (Algorithm 7.2) or continuing to a target loss (Algorithm 7.3) both recover the held-out examples; in the example both lowered test error. Note which you did, since step counts and epoch counts differ when the training set grows.
Treat stopping as one hyperparameter among several. The best number of steps depends on the learning rate, the schedule and the other regularizers, so it is tuned jointly with them (hyperparameter tuning).

Pitfalls

Stopping before a learning-rate drop. AlexNet’s authors lowered the rate when the validation error stopped improving (as the learning rate schedule article describes) rather than stopping. In our reading, a validation curve that has flattened at one learning rate can still fall after the rate is lowered. A patience shorter than the interval between scheduled drops can end the run on such a plateau. Plateau-based drops, and scikit-learn’s adaptive mode, build the waiting into the schedule instead.
Epoch-wise double descent. Pezeshki et al. (2022) describe epoch-wise double descent, which they credit to Nakkiran et al., as test error that undergoes two descents as training time increases (the regularization article covers double descent more broadly). They reproduce it in a linear teacher-student set-up and attribute it to features learned at different scales: as the fast-learning features overfit, slower-learning ones start to fit, giving a second descent. In our reading, a patience rule stops on the first rise and never sees the second descent, so a curve that might have one is worth running to the end of the budget at least once.
A validation loss that is not the goal. In robot learning from offline demonstrations, Mandlekar et al. (2021) evaluated every checkpoint by rollouts in simulation and found that choosing the policy with the lowest validation loss, or the last checkpoint, gave policies 10% to 100% worse than the best checkpoint (their §4.5). The visuomotor policy learning article recommends selecting checkpoints by closed-loop rollouts. The same applies whenever the validation loss is a surrogate for the measure that matters, for example a cross-entropy loss when the goal is a detection metric: monitor the metric itself if it can be computed.
Noisy validation curves. Small validation sets, augmentation at evaluation time or noisy training make the best validation value partly a matter of luck. In the example, independent noise drove short patience to stop far too early, and even full-length training with best-checkpoint selection was within 1% less than half the time.
Reporting the best checkpoint as a test result. Selecting the best checkpoint by test performance, as some benchmarks do, reports an optimistic number. Fine-tuning shows a related limit: in Kumar et al.’s experiments, summarized in the fine-tuning article, even stopping fine-tuning at the point best on out-of-distribution test data left it below a linear probe on out-of-distribution data on average.
Treating the ridge equivalence as exact. It holds for a quadratic loss and plain gradient descent from a fixed start; even there the filter factors differ by up to about 0.2 at the calibrated coefficient, and in the example the best ridge coefficient was off from $1/(\epsilon\tau^*)$ by a factor of 1.4 to 1.8.

Evidence and Impact

Theory for linear models. The quadratic-model equivalence to L2 regularization (Goodfellow et al., §7.8), and Ali, Kolter and Tibshirani’s (2019) bounds: gradient-flow estimation risk is at most 1.6862 times ridge risk under the calibration $\lambda = 1/t$ , with optimally tuned Bayes risks within a factor of 1.2147. In the example’s least-squares problem, the test error of optimally stopped gradient descent was between 2.1% below and 0.2% above that of optimally tuned ridge on ten datasets.
Statistical optimality without hold-out data. Raskutti, Wainwright and Yu (2014) show that their data-dependent stopping rule achieves minimax-optimal rates for several kernel classes, including Sobolev spaces.
Robustness to label noise. Li, Soltanolkotabi and Oymak (2020) give both the MNIST evidence and a proof for one-hidden-layer networks on clustered data that early-stopped gradient descent fits the correct labels before the corrupted ones.
Practice in vision. Early stopping is the default regularizer in Goodfellow et al.’s practical methodology, and dataset protocols record it: the CIFAR article reports cuda-convnet’s batch-5 procedure and ResNet’s 45,000/5,000 split for deciding the training length, and EfficientNet’s later arXiv versions describe a 25,000-image minival set used for early stopping.
Where it fails. The robomimic study is direct evidence that validation-loss selection can pick much worse policies when the training objective is a surrogate for the task. Double descent over training time shows that the U-shaped curve the procedure assumes is not universal.