Overfitting and Underfitting

A model underfits when it cannot fit even its own training images well, and overfits when it fits them much better than it fits new ones. Both are failures to generalize, but they have opposite causes and opposite cures, so the first practical question about any disappointing model, from a flat-field correction to an image classifier, is which of the two it suffers from. The answer comes from comparing two numbers, the error on the training data and the error on held-out data, and from watching how they move when one changes the model’s capacity or the amount of data.

The machine learning article defines risk, the generalization gap, capacity and the U-shaped curve of test error against capacity, and gives the approximation–estimation and bias–variance decompositions that underlie them. This article builds on those definitions and covers what they leave to practice: how to diagnose each failure from training and validation curves, how capacity and data act as two separate levers, the forms overfitting takes in vision, and which remedy, each with its own article, addresses which failure. An example fits a smooth illumination field from a few noisy background pixels and turns both levers with exact ground truth.

Definition

Goodfellow, Bengio and Courville (2016, §5.2) state the two central challenges of machine learning as making the training error small and making the gap between training and test error small. Underfitting occurs when the model cannot reach a sufficiently low error on the training set. Overfitting occurs when the gap between training and test error is too large. The test error here is the generalization error, the expected error on a new input drawn from the distribution the system will meet; it is estimated on a test set collected separately from the training set. Murphy (2022, §1.2.3, §5.4.1, eq. 5.84) calls the difference between the population risk and the empirical risk the generalization gap and reads a large gap as the sign of overfitting.

Two points of usage follow from these definitions. Overfitting is a statement about the gap, not about the test error rising: a model can overfit, with a large gap, and still be the best available. And underfitting is a statement about the training error, so it can be diagnosed without any held-out data at all.

Intuition

Suppose a microscope or an inspection camera images a scene under uneven illumination: the lens darkens the corners (vignetting) and the light falls off from one side. To correct the images, one estimates the smooth shading field from pixels known to be background and divides it out. A plane through those pixels cannot follow the falloff into the corners: it is wrong on the background pixels it was fitted to and on all the others, which is underfitting. A polynomial surface with as many coefficients as there are sample pixels can pass through every noisy sample and swing wildly between them, so it matches the samples better than the true field does and the rest of the background much worse, which is overfitting. Between the two lies a surface flexible enough to follow the vignetting but stiff enough that the samples pin it down.

Two things move that balance point. Capacity: a richer family of surfaces can follow more of the true field but needs more samples to fix its extra coefficients. Data: each additional sample rules out surfaces that merely happened to pass near the others, so more samples make a richer family affordable. Noise shifts the balance too: the noisier each sample, the more a flexible surface spends its freedom on fitting the noise.

Formal Definition

Let a model $f$ from a hypothesis space $\mathcal{H}$ be fitted to $n$ training examples $(\mathbf{x}_i, y_i)$ by minimizing the training error $\hat{R}_n(f) = \frac{1}{n}\sum_i \ell(y_i, f(\mathbf{x}_i))$, and let $R(f) = \mathbb{E}[\ell(y, f(\mathbf{x}))]$ be its error on new examples from the same distribution. For the fitted $\hat{f}$ the gap $R(\hat{f}) - \hat{R}_n(\hat{f})$ is non-negative in expectation: the expected test error is at least the expected training error, because the parameters were chosen to make the training error small (Goodfellow et al., §5.2).

The capacity lever. Let $\mathcal{H}_1 \subset \mathcal{H}_2 \subset \dots$ be nested hypothesis spaces, such as polynomials of increasing degree. Minimizing the training error over a larger set can only give a smaller or equal minimum, so the training error of the fitted model is non-increasing in capacity, exactly and for every training set. The test error has no such property: the machine learning article explains why it typically falls and then rises, the U-shaped curve. Training error therefore always prefers the most flexible model, which is why capacity is chosen on a validation set or by cross-validation (Goodfellow et al., §5.3; Murphy, §5.4.2, which calls the effect the optimism of the training error).

The data lever. For a fixed hypothesis space, more data does the opposite. Murphy (§4.5.7) shows learning curves of polynomial models of several degrees: the training error of the flexible models initially rises as the training set grows, because a larger set is harder to fit, while the test error falls toward the noise floor; the gap is larger for the more complex models and shrinks as $n$ grows. A model that is too simple keeps a high test error however much data it gets, which is underfitting. Goodfellow and colleagues (§5.2) add that the expected generalization error never increases with the number of training examples, and that a model can have the optimal capacity and still have a large gap, which gathering more training examples may then reduce.

How large the gap is for least squares. For a linear model with $p$ coefficients fitted by least squares to $y = f(\mathbf{x}) + \varepsilon$ with noise variance $\sigma^2$, a short calculation (ours) gives the gap exactly when the test inputs are the training inputs with fresh noise. With $H$ the projection onto the model’s column space, the expected training residual is $\lVert (I - H)\mathbf{f} \rVert^2 + \sigma^2 (n - p)$ and the expected error on fresh noise at the same inputs is $\lVert (I - H)\mathbf{f} \rVert^2 + \sigma^2 (n + p)$, so

$$\mathbb{E}\big[\text{test MSE}\big] - \mathbb{E}\big[\text{training MSE}\big] = \frac{2\sigma^2 p}{n},$$

whatever the true function $f$ is: the gap grows with the number of coefficients and with the noise, and shrinks with the number of examples, the three levers in one formula. At new inputs the gap is larger, and the variance term depends on where those inputs lie relative to the training inputs; the inductive bias article gives that out-of-sample variance formula and shows it exploding near the interpolation threshold, where the number of examples approaches the number of coefficients. Beyond the threshold, where flexible models fit the training set exactly, test error can fall again; the regularization article covers this double descent.

Properties

Diagnosis from two curves

Goodfellow and colleagues (§11.3, §11.4.1, §11.5) organize practical work around comparing the training error with the validation or test error and with a target. Their advice, with the remedies the site’s articles cover:

Training error	Validation error	Diagnosis	First levers
High	High, close to training	Underfitting, or a defect in the data or the code	More capacity, better features or inputs, longer or better-tuned optimization, less regularization
Low	Much higher	Overfitting	More data, augmentation, regularization, early stopping, a smaller model
Low	Low	Done, if both meet the target	None
Rising in training	—	Optimization failure (for example a learning rate that is too large)	Fix the optimizer before judging capacity

Several refinements matter. If the training error is too high, there is no reason to gather more data, because the model is not using the data it already has; the authors recommend a larger model or a better-tuned learning rate first, and, if that fails, suspecting the data: inputs too noisy, or not containing what is needed to predict the target (§11.3). If the training error is low and the test error high, more data is among the most effective remedies, with stronger regularization or a smaller model as a simple alternative (§11.3). High error on both sets can also be a software defect, which is separated from genuine underfitting by checking that the model can fit a tiny dataset, a test the neural network training article lists among its sanity checks (§11.5). And a low training error with a high test error can be an evaluation bug, a model saved and reloaded wrongly or test images preprocessed differently from training images (§11.5).

The book’s worked case is a vision one (§11.6): a network that transcribed address numbers from Street View photographs had nearly identical training and test errors, so its shortfall was underfitting or a problem with the training data, not overfitting. Inspecting the training images the model got most confidently wrong showed that the detector’s crops often cut off digits; widening the crop, a change to the input rather than to the model, added ten percentage points of coverage. Underfitting is not always a matter of capacity.

The curves themselves are cheap to produce. A validation curve plots training and validation error against one capacity hyperparameter, and a learning curve plots them against the training-set size. scikit-learn 1.7.2 provides both as validation_curve and learning_curve in sklearn.model_selection; in the installed source, learning_curve defaults to five training sizes from 10% to 100% of the training portion (train_sizes=np.linspace(0.1, 1.0, 5)), both use 5-fold cross-validation when cv=None, and learning_curve takes its smaller training sets as prefixes of the training portion, shuffled first only when shuffle=True; in our reading, a dataset sorted by class or by capture session then gives unrepresentative small subsets. Goodfellow and colleagues (§11.3) suggest spacing training-set sizes on a logarithmic scale, since adding a small fraction of the data rarely changes the generalization error noticeably. During the training of a network, the analogue is the curve of training and validation loss against epochs, which the neural network training article reads for the same two failures and early stopping uses to pick the weights.

Representational and effective capacity

Capacity is not only the size of the hypothesis space. Goodfellow and colleagues (§5.2) distinguish the representational capacity of the model family from its effective capacity, which can be lower because the optimizer finds only some of the functions the family contains. Their Table 11.1 (§11.4.1) lists how common hyperparameters move effective capacity: more hidden units and wider convolution kernels raise it, and so do a smaller weight-decay coefficient and a lower dropout rate, while a learning rate tuned correctly gives the highest effective capacity and one too large or too small lowers it. One consequence they point out: a hyperparameter such as weight decay can only subtract capacity, so if a model underfits with weight decay at zero, no setting of that coefficient will move it toward overfitting. They also write that neural networks typically do best with a very low training error, high capacity, and a gap that regularization then reduces, so that the best results usually come from a large, well-regularized model (§11.4.1).

Overfitting is not uniform

Caruana, Lawrence and Giles (2000) studied overfitting in multilayer perceptrons with one hidden layer, trained by backpropagation. Polynomials fitted to noisy samples of a smooth curve underfitted at low order and overfitted at high order (their Figure 1), while networks with far more weights than training points did not overfit significantly more than small ones. On data with one gently varying region and one strongly varying region, polynomials could not fit both: an order low enough for the gentle region underfitted the other, and an order high enough for the strong region overfitted the gentle one (§3). They write that regularization methods such as weight decay typically assume overfitting is a global phenomenon, whereas it can vary significantly in different regions of a model. Networks trained by backpropagation fitted both regions in their experiments, while conjugate gradient, which reached lower training error, overfitted the gently varying region. On seven problems, including vision data used to steer an autonomous car, and with early stopping, nets that were too large generalized worse than smaller ones on only three, and the loss was small; on some tasks small nets also overtrained considerably, and the authors conclude that regularization, for example early stopping, is critical for nets of all sizes (§4). Their experiments are with small fully connected networks, not convolutional ones.

For images the lesson is to look at where a model errs, not only at its average error. The example below shows a polynomial shading model whose overfitting concentrates at the image border.

Forms of Overfitting in Vision

• Fitting pixel noise. A flexible model of a smooth quantity, such as an illumination field, a background or a lens distortion, follows the noise in the pixels it was fitted to; the example below measures this.
• Memorizing label noise. A classifier with enough capacity reaches a training error below the error any classifier can have on new images by memorizing wrong labels, as the decision-tree example in the machine learning article shows; the regularization article summarizes the evidence that image classifiers can fit even randomly assigned labels.
• Learning accidents of the training set. A model can fit regularities that hold in its training images and not in the world: where objects sit in the frame, the background they appear on, the camera that took them. Such a model can score well on a test set drawn the same way and fail elsewhere; dataset bias covers how datasets acquire these regularities and how cross-dataset tests reveal them, and the machine learning article’s third example shows a pixel-based classifier failing when shapes move across the frame.
• Leakage between splits. Near-duplicate images or frames of one video in both the training and the test set make the test error optimistic, hiding overfitting; the training, validation and test sets article describes how to split.
• Overfitting the choice of model. Choosing among many models or hyperparameter settings by one validation score overfits that score; cross-validation covers this overfitting in model selection and nested cross-validation as its remedy, and benchmark overfitting covers the same effect for a community reusing one test set.

Remedies

The two failures call for opposite moves. In our reading, the remedy for one can worsen the other: added capacity widens an overfitting model’s gap, and added regularization raises an underfitting model’s training error further.

• For underfitting: increase capacity (a higher degree, more units or layers, a larger network), provide better inputs (features or crops that contain what the target depends on, as in the Street View case), optimize longer or tune the learning rate (hyperparameter tuning describes the order in which to tune), and reduce regularization that is stronger than the data require. Goodfellow and colleagues (§11.4.1) note that a model whose training error is above the target leaves no choice but to increase capacity, if the optimizer works correctly and no regularization is in use.
• For overfitting: more training data, which needs no change to the model; data augmentation, which creates label-preserving variations of the training images; explicit regularization such as weight decay or dropout; early stopping; a smaller model or lower degree; and starting from weights pretrained on other data, by transfer learning or fine-tuning, which carries prior knowledge in. The regularization article compares these methods and the evidence about how much each helps.

Which lever is cheaper depends on the problem: Goodfellow and colleagues (§11.3) contrast settings where gathering more data is cheap with medical applications, where it may be costly or infeasible.

Examples

The example is the estimation of an illumination field from the Intuition section, with exact ground truth. A $128 \times 128$ image has a known shading field

$$F(r, c) = 0.8 \left(1 + \frac{(r - 60)^2 + (c - 70)^2}{120^2}\right)^{-2} \left(1 + 0.25\,u\right),$$

where $r$ and $c$ are the row and column of a pixel (in pixels, measured at pixel centres from the top-left), the first factor is the $\cos^4$ law of natural vignetting that the image formation article gives, here for an optical axis at row 60 and column 70, slightly off the image centre, and a focal length of 120 pixels, and the second factor is a light falloff from right to left, with $u$ the column coordinate scaled to $[-1, 1]$. Three dark objects (reflectance 0.35) sit on a background of reflectance 1, and every pixel carries Gaussian noise of standard deviation $\sigma$; nothing is clipped or quantized. The model is a polynomial surface in the scaled coordinates $(u, v)$ of total degree $d$, with $p = (d+1)(d+2)/2$ coefficients, fitted by least squares to $n = 30$, 100 or 300 randomly chosen background pixels, nested so that the 30 are among the 100 and the 100 among the 300. Degrees stop below the interpolation threshold ($p < n$), so $n = 30$ uses degrees 0–6.

The training MSE is measured on the noisy training pixels and the test MSE against the noise-free field over every background pixel outside the 300-pixel training pool. The gap compares like with like: the expected error on fresh noisy pixels, which is the test MSE plus $\sigma^2$, minus the training MSE. The script also prints the condition number of the training design matrix, the in-sample prediction $2\sigma^2 p/n$ of the Formal Definition, and the approximation floor, the test MSE of the degree-$d$ surface fitted to the noise-free field over all background pixels (scored on seed 0’s test pixels), which is approximately the best that degree can do. It repeats everything for ten seeds (0–9), which change the sampled pixels and the noise, at $\sigma = 0.02$ and $\sigma = 0.04$ (the same noise draw scaled by two), and at the top degree for each $n$ it also fits ridge regression with a fixed penalty $\alpha = 4 \times 10^{-4}$ on every coefficient except the constant. That value is Murphy’s (§11.3, eq. 11.55) $\lambda = \sigma^2/\tau^2$ for $\sigma = 0.02$ and a prior standard deviation $\tau = 1$ on each coefficient, and it is used unchanged at $\sigma = 0.04$. Finally, it splits the test MSE between a 16-pixel band at the image border and the interior. One fit is checked against scikit-learn’s PolynomialFeatures with LinearRegression and Ridge; with fit_intercept=True, scikit-learn centres the data and so leaves the intercept unpenalized, as the hand ridge does by leaving the constant column out of the penalty. It runs in about 1–1.5 seconds with NumPy 2.3.5 and scikit-learn 1.7.2.

import os
for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[k] = "1"                       # single-threaded BLAS, set before NumPy loads
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge

# A 128x128 image: a smooth illumination field (cos^4 vignetting about an off-centre optical
# axis, f = 120 px, times a left-to-right ramp) on which three dark objects sit.
H = W = 128
r, c = np.mgrid[0:H, 0:W].astype(float)
u, v = (c + 0.5) / 64 - 1, (r + 0.5) / 64 - 1          # coordinates scaled to [-1, 1]
F = 0.8 * (1 + ((r - 60) ** 2 + (c - 70) ** 2) / 120.0 ** 2) ** -2 * (1 + 0.25 * u)
obj = (((r - 40) ** 2 + (c - 44) ** 2 <= 14 ** 2) | ((r - 90) ** 2 + (c - 84) ** 2 <= 18 ** 2)
       | ((r >= 18) & (r <= 37) & (c >= 88) & (c <= 117)))
refl = np.where(obj, 0.35, 1.0)                        # objects reflect 35%, background 100%
U, V, Fv, clean = u.ravel(), v.ravel(), F.ravel(), (F * refl).ravel()
bg = np.flatnonzero(~obj.ravel())                      # background pixels: the field itself
border = ((r < 16) | (r >= 112) | (c < 16) | (c >= 112)).ravel()   # 16-px band at the edges

SIGMAS, NS, SEEDS, ALPHA = (0.02, 0.04), (30, 100, 300), range(10), 4e-4
DMAX = {30: 6, 100: 10, 300: 10}                       # keep p < n: below interpolation

def npar(d):                                           # coefficients of total degree <= d
    return (d + 1) * (d + 2) // 2

# Monomials u^i v^j, i + j <= 10, ordered by total degree, so degree d uses the first npar(d).
PHI = np.stack([U ** i * V ** (t - i) for t in range(11) for i in range(t, -1, -1)], axis=1)

def draw(seed):
    """A random order of the background pixels (first 300: nested training pool; the rest:
    test pixels) and one standard-normal draw per pixel, scaled later by sigma."""
    rng = np.random.default_rng(seed)
    perm = rng.permutation(bg)
    return perm[:300], perm[300:], rng.standard_normal(H * W)

def ols(X, y):
    return np.linalg.lstsq(X, y, rcond=None)[0]

def ridge(X, y, alpha):                                # penalize every coefficient but the constant
    p = X.shape[1]
    A = np.vstack([X, np.sqrt(alpha) * np.eye(p)[1:]])
    return np.linalg.lstsq(A, np.concatenate([y, np.zeros(p - 1)]), rcond=None)[0]

# --- checks on the data and the code ---
dup = on_obj = 0
for s in SEEDS:
    pool, test, _ = draw(s)
    dup += pool.size - np.unique(pool).size
    on_obj += int(obj.ravel()[pool].sum())
print(f"field {F.min():.4f} to {F.max():.4f}; background {bg.size} px; test px per seed {test.size}; "
      f"pool px on objects {on_obj}, duplicates {dup}; max |u|, |v| {max(np.abs(U).max(), np.abs(V).max()):.4f}; "
      f"largest p/n {max(npar(DMAX[n]) / n for n in NS):.3f}")
g = (0.5 + 0.2 * U - 0.1 * V + 0.05 * U**2 + 0.03 * U * V - 0.04 * V**2 + 0.02 * U**3
     - 0.01 * U**2 * V + 0.015 * U * V**2 - 0.025 * V**3)            # an exact cubic surface
pool0, test0, z0 = draw(0)
tr = pool0[:100]
err = [np.abs(PHI[:, :npar(d)] @ ols(PHI[tr, :npar(d)], g[tr]) - g).max() for d in (3, 10)]
print(f"cubic surface, no noise, fitted at d=3 and d=10: max error {max(err):.1e} (tolerance 1e-11):",
      "held" if max(err) <= 1e-11 else "failed")
I0 = clean + 0.02 * z0
pf = PolynomialFeatures(degree=10, include_bias=False)
A, At = pf.fit_transform(np.column_stack([U[tr], V[tr]])), pf.transform(np.column_stack([U[test0], V[test0]]))
X10 = PHI[tr, :npar(10)]
d_ols = np.abs(PHI[test0, :66] @ ols(X10, I0[tr]) - LinearRegression().fit(A, I0[tr]).predict(At)).max()
d_rdg = np.abs(PHI[test0, :66] @ ridge(X10, I0[tr], ALPHA) - Ridge(alpha=ALPHA).fit(A, I0[tr]).predict(At)).max()
print(f"vs scikit-learn (n=100, d=10): OLS {d_ols:.1e} (tol 1e-10), ridge {d_rdg:.1e} (tol 1e-9):",
      "held" if d_ols <= 1e-10 and d_rdg <= 1e-9 else "failed")

# --- the sweep: degree x number of samples x noise, ten seeds ---
K = len(SEEDS)
res = {(sg, n): {k: np.zeros((K, DMAX[n] + 1)) for k in ("train", "test", "bord", "int", "cond")}
       for sg in SIGMAS for n in NS}                   # per (sigma, n): arrays over (seed, degree)
rdg = {(sg, n): {k: np.zeros(K) for k in ("train", "test", "bord", "int")}
       for sg in SIGMAS for n in NS}                   # ridge at the top degree: arrays over seed
for s in SEEDS:
    pool, test, z = draw(s)
    for sg in SIGMAS:
        I = clean + sg * z                             # same noise draw, scaled by sigma
        for n in NS:
            t = pool[:n]
            def score(w, p):
                e2 = (Fv[test] - PHI[test, :p] @ w) ** 2          # against the noise-free field
                return (np.mean((I[t] - PHI[t, :p] @ w) ** 2), e2.mean(),
                        e2[border[test]].mean(), e2[~border[test]].mean())
            o = res[sg, n]
            for d in range(DMAX[n] + 1):
                p = npar(d)
                o["train"][s, d], o["test"][s, d], o["bord"][s, d], o["int"][s, d] = score(ols(PHI[t, :p], I[t]), p)
                o["cond"][s, d] = np.linalg.cond(PHI[t, :p])
            q, p = rdg[sg, n], npar(DMAX[n])
            q["train"][s], q["test"][s], q["bord"][s], q["int"][s] = score(ridge(PHI[t, :p], I[t], ALPHA), p)

def gap(o, sg):                     # expected test MSE on fresh noisy pixels minus training MSE
    return o["test"] + sg ** 2 - o["train"]

worst = max(np.max(np.diff(res[sg, n]["train"], axis=1) / res[sg, n]["train"][:, :-1])
            for sg in SIGMAS for n in NS)
print(f"training MSE never rises with degree: largest relative increase {worst:.1e} (tolerance 1e-12)")

print("\nsigma = 0.02 (means over 10 seeds; test MSE against the noise-free field)")
for n in NS:
    o = res[0.02, n]
    print(f"n = {n}\n  d   p    cond      train       test   test min-max          gap   2s^2p/n")
    g_ = gap(o, 0.02)
    for d in range(DMAX[n] + 1):
        p = npar(d)
        print(f" {d:2d} {p:3d} {np.median(o['cond'][:, d]):7.1e}  {o['train'][:, d].mean():.3e}  "
              f"{o['test'][:, d].mean():.3e}  {o['test'][:, d].min():.2e}-{o['test'][:, d].max():.2e}  "
              f"{g_[:, d].mean():.3e}  {2 * 0.02**2 * p / n:.3e}")
print("approximation floor (degree-d fit to the noise-free field on all background px):")
fl = [np.mean((Fv[test0] - PHI[test0, :npar(d)] @ ols(PHI[bg, :npar(d)], Fv[bg])) ** 2) for d in range(11)]
print("  " + "  ".join(f"d={d}: {x:.3e}" for d, x in enumerate(fl)))

print("\nsigma = 0.04 (means over 10 seeds)")
for n in NS:
    o = res[0.04, n]
    g_ = gap(o, 0.04)
    print(f"n = {n}: d, train, test, gap")
    for d in range(DMAX[n] + 1):
        print(f" {d:2d}  {o['train'][:, d].mean():.3e}  {o['test'][:, d].mean():.3e}  {g_[:, d].mean():.3e}")

best = {(sg, n): int(np.argmin(res[sg, n]["test"].mean(0))) for sg in SIGMAS for n in NS}
print("\nbest degree (lowest mean test MSE), and per seed")
for sg in SIGMAS:
    for n in NS:
        print(f"  sigma {sg}  n {n:3d}: {best[sg, n]:2d}   per seed {[int(b) for b in np.argmin(res[sg, n]['test'], 1)]}")

print("\nridge (alpha = 4e-4) against least squares at the top degree")
print(" sigma    n   d   test OLS   test ridge     gap OLS   gap ridge   seeds ridge lower: test, gap")
for sg in SIGMAS:
    for n in NS:
        o, q, d = res[sg, n], rdg[sg, n], DMAX[n]
        go, gr = gap(o, sg)[:, d], gap(q, sg)
        print(f" {sg:.2f} {n:4d} {d:3d}  {o['test'][:, d].mean():.3e}  {q['test'].mean():.3e}   "
              f"{go.mean():.3e}  {gr.mean():.3e}   {int((q['test'] < o['test'][:, d]).sum())}/{K}, {int((gr < go).sum())}/{K}")

print("\ntest MSE on the 16-px border band and the interior (means over seeds)")
print(" sigma    n  d_best  border    interior   ratio |  d_top  border    interior   ratio")
for sg in SIGMAS:
    for n in NS:
        o, b, d = res[sg, n], best[sg, n], DMAX[n]
        rb = o["bord"][:, b].mean() / o["int"][:, b].mean()
        rt = o["bord"][:, d].mean() / o["int"][:, d].mean()
        print(f" {sg:.2f} {n:4d}  {b:4d}  {o['bord'][:, b].mean():.3e}  {o['int'][:, b].mean():.3e}  {rb:6.3f} |"
              f" {d:4d}  {o['bord'][:, d].mean():.3e}  {o['int'][:, d].mean():.3e}  {rt:6.3f}")

def hv(ok):
    return "held" if ok else "failed"
print("\nH1 training MSE non-increasing in degree:", hv(worst <= 1e-12))
for sg in SIGMAS:
    print(f"H2 sigma {sg}: n=30 test MSE U-shaped (interior minimum at d={best[sg, 30]}):",
          hv(0 < best[sg, 30] < DMAX[30]))
    print(f"H3 sigma {sg}: best degree non-decreasing in n:", hv(best[sg, 30] <= best[sg, 100] <= best[sg, 300]))
for sg in SIGMAS:
    for n in NS:
        o, q, d = res[sg, n], rdg[sg, n], DMAX[n]
        ok = q["test"].mean() < o["test"][:, d].mean() and gap(q, sg).mean() < gap(o, sg)[:, d].mean()
        print(f"H4 sigma {sg} n {n}: ridge lowers test MSE and gap:", hv(ok))
for n in NS:
    print(f"H5 n {n}: best degree lower at sigma 0.04 than at 0.02:", hv(best[0.04, n] < best[0.02, n]))
for sg in SIGMAS:
    for n in NS:
        o, d = res[sg, n], DMAX[n]
        print(f"H6 sigma {sg} n {n}: border above interior at the top degree:",
              hv(o["bord"][:, d].mean() > o["int"][:, d].mean()))

Output:

field 0.2204 to 0.8411; background 14162 px; test px per seed 13862; pool px on objects 0, duplicates 0; max |u|, |v| 0.9922; largest p/n 0.933
cubic surface, no noise, fitted at d=3 and d=10: max error 5.2e-14 (tolerance 1e-11): held
vs scikit-learn (n=100, d=10): OLS 5.9e-13 (tol 1e-10), ridge 2.1e-12 (tol 1e-9): held
training MSE never rises with degree: largest relative increase -5.4e-03 (tolerance 1e-12)

sigma = 0.02 (means over 10 seeds; test MSE against the noise-free field)
n = 30
  d   p    cond      train       test   test min-max          gap   2s^2p/n
  0   1 1.0e+00  2.149e-02  2.408e-02  2.36e-02-2.52e-02  2.997e-03  2.667e-05
  1   3 1.9e+00  1.122e-02  1.394e-02  1.30e-02-1.63e-02  3.122e-03  8.000e-05
  2   6 5.2e+00  9.144e-04  1.139e-03  8.65e-04-1.48e-03  6.245e-04  1.600e-04
  3  10 1.8e+01  4.945e-04  9.653e-04  5.85e-04-1.38e-03  8.708e-04  2.667e-04
  4  15 6.4e+01  2.199e-04  2.035e-03  6.09e-04-7.22e-03  2.215e-03  4.000e-04
  5  21 3.4e+02  1.280e-04  1.232e-02  1.11e-03-4.14e-02  1.259e-02  5.600e-04
  6  28 5.6e+03  3.612e-05  1.466e+01  5.46e-02-9.24e+01  1.466e+01  7.467e-04
n = 100
  d   p    cond      train       test   test min-max          gap   2s^2p/n
  0   1 1.0e+00  2.337e-02  2.379e-02  2.36e-02-2.42e-02  8.145e-04  8.000e-06
  1   3 1.8e+00  1.215e-02  1.315e-02  1.28e-02-1.35e-02  1.395e-03  2.400e-05
  2   6 4.7e+00  1.077e-03  9.067e-04  8.11e-04-1.01e-03  2.298e-04  4.800e-05
  3  10 1.2e+01  6.486e-04  4.530e-04  3.64e-04-6.91e-04  2.044e-04  8.000e-05
  4  15 2.9e+01  3.709e-04  1.432e-04  6.27e-05-3.76e-04  1.723e-04  1.200e-04
  5  21 7.7e+01  3.411e-04  2.046e-04  6.99e-05-6.00e-04  2.635e-04  1.680e-04
  6  28 2.2e+02  3.009e-04  4.591e-04  1.77e-04-8.72e-04  5.582e-04  2.240e-04
  7  36 6.9e+02  2.716e-04  1.905e-03  3.95e-04-7.03e-03  2.033e-03  2.880e-04
  8  45 2.4e+03  2.390e-04  1.032e-02  1.25e-03-3.16e-02  1.048e-02  3.600e-04
  9  55 7.9e+03  1.939e-04  3.154e-02  3.95e-03-1.36e-01  3.174e-02  4.400e-04
 10  66 3.8e+04  1.393e-04  4.339e-01  2.20e-02-2.70e+00  4.342e-01  5.280e-04
n = 300
  d   p    cond      train       test   test min-max          gap   2s^2p/n
  0   1 1.0e+00  2.393e-02  2.376e-02  2.36e-02-2.39e-02  2.316e-04  2.667e-06
  1   3 1.7e+00  1.283e-02  1.293e-02  1.28e-02-1.32e-02  5.009e-04  8.000e-06
  2   6 4.3e+00  1.167e-03  8.165e-04  8.00e-04-8.50e-04  4.936e-05  1.600e-05
  3  10 1.0e+01  6.932e-04  3.582e-04  3.46e-04-3.76e-04  6.499e-05  2.667e-05
  4  15 2.4e+01  3.932e-04  3.959e-05  3.17e-05-4.75e-05  4.636e-05  4.000e-05
  5  21 6.0e+01  3.746e-04  3.485e-05  2.57e-05-4.43e-05  6.021e-05  5.600e-05
  6  28 1.5e+02  3.602e-04  4.281e-05  2.59e-05-6.76e-05  8.262e-05  7.467e-05
  7  36 3.6e+02  3.496e-04  6.250e-05  4.65e-05-1.16e-04  1.129e-04  9.600e-05
  8  45 1.0e+03  3.378e-04  9.735e-05  7.25e-05-1.68e-04  1.595e-04  1.200e-04
  9  55 2.6e+03  3.257e-04  1.603e-04  7.89e-05-3.30e-04  2.346e-04  1.467e-04
 10  66 6.6e+03  3.113e-04  2.227e-04  1.11e-04-4.05e-04  3.114e-04  1.760e-04
approximation floor (degree-d fit to the noise-free field on all background px):
  d=0: 2.363e-02  d=1: 1.277e-02  d=2: 7.921e-04  d=3: 3.321e-04  d=4: 1.981e-05  d=5: 6.170e-06  d=6: 4.242e-07  d=7: 9.248e-08  d=8: 7.824e-09  d=9: 1.382e-09  d=10: 1.301e-10

sigma = 0.04 (means over 10 seeds)
n = 30: d, train, test, gap
  0  2.312e-02  2.415e-02  2.634e-03
  1  1.231e-02  1.409e-02  3.384e-03
  2  2.035e-03  1.419e-03  9.835e-04
  3  1.423e-03  1.627e-03  1.804e-03
  4  8.595e-04  7.639e-03  8.379e-03
  5  5.096e-04  5.070e-02  5.179e-02
  6  1.444e-04  5.862e+01  5.862e+01
n = 100: d, train, test, gap
  0  2.475e-02  2.381e-02  6.648e-04
  1  1.329e-02  1.315e-02  1.458e-03
  2  2.268e-03  9.678e-04  2.995e-04
  3  1.799e-03  5.795e-04  3.807e-04
  4  1.453e-03  4.339e-04  5.810e-04
  5  1.355e-03  7.344e-04  9.793e-04
  6  1.202e-03  1.857e-03  2.254e-03
  7  1.086e-03  7.665e-03  8.180e-03
  8  9.558e-04  4.121e-02  4.186e-02
  9  7.756e-04  1.261e-01  1.269e-01
 10  5.572e-04  1.736e+00  1.737e+00
n = 300: d, train, test, gap
  0  2.521e-02  2.377e-02  1.571e-04
  1  1.404e-02  1.294e-02  5.041e-04
  2  2.316e-03  8.316e-04  1.160e-04
  3  1.833e-03  3.917e-04  1.589e-04
  4  1.522e-03  9.339e-05  1.714e-04
  5  1.483e-03  1.156e-04  2.326e-04
  6  1.439e-03  1.705e-04  3.314e-04
  7  1.398e-03  2.491e-04  4.515e-04
  8  1.351e-03  3.885e-04  6.374e-04
  9  1.303e-03  6.409e-04  9.383e-04
 10  1.245e-03  8.903e-04  1.245e-03

best degree (lowest mean test MSE), and per seed
  sigma 0.02  n  30:  3   per seed [3, 3, 2, 3, 2, 4, 4, 4, 3, 4]
  sigma 0.02  n 100:  4   per seed [4, 5, 5, 3, 4, 4, 5, 4, 4, 4]
  sigma 0.02  n 300:  5   per seed [5, 5, 5, 5, 5, 6, 5, 5, 5, 5]
  sigma 0.04  n  30:  2   per seed [3, 3, 2, 3, 2, 2, 2, 2, 2, 3]
  sigma 0.04  n 100:  4   per seed [4, 4, 3, 3, 4, 4, 4, 4, 4, 4]
  sigma 0.04  n 300:  4   per seed [4, 4, 4, 4, 4, 4, 4, 4, 4, 5]

ridge (alpha = 4e-4) against least squares at the top degree
 sigma    n   d   test OLS   test ridge     gap OLS   gap ridge   seeds ridge lower: test, gap
 0.02   30   6  1.466e+01  5.550e-03   1.466e+01  5.850e-03   10/10, 10/10
 0.02  100  10  4.339e-01  3.265e-03   4.342e-01  3.432e-03   10/10, 10/10
 0.02  300  10  2.227e-04  1.022e-04   3.114e-04  1.760e-04   10/10, 10/10
 0.04   30   6  5.862e+01  2.232e-02   5.862e+01  2.353e-02   10/10, 10/10
 0.04  100  10  1.736e+00  1.334e-02   1.737e+00  1.400e-02   10/10, 10/10
 0.04  300  10  8.903e-04  4.101e-04   1.245e-03  7.058e-04   10/10, 10/10

test MSE on the 16-px border band and the interior (means over seeds)
 sigma    n  d_best  border    interior   ratio |  d_top  border    interior   ratio
 0.02   30     3  1.415e-03  5.200e-04   2.720 |    6  2.898e+01  4.830e-01  60.003
 0.02  100     4  2.286e-04  5.840e-05   3.915 |   10  8.700e-01  1.655e-03  525.713
 0.02  300     5  4.653e-05  2.328e-05   1.999 |   10  3.864e-04  6.057e-05   6.380
 0.04   30     2  1.824e-03  1.017e-03   1.794 |    6  1.159e+02  1.930e+00  60.029
 0.04  100     4  6.870e-04  1.829e-04   3.756 |   10  3.481e+00  6.620e-03  525.856
 0.04  300     4  1.173e-04  6.968e-05   1.683 |   10  1.545e-03  2.423e-04   6.375

H1 training MSE non-increasing in degree: held
H2 sigma 0.02: n=30 test MSE U-shaped (interior minimum at d=3): held
H3 sigma 0.02: best degree non-decreasing in n: held
H2 sigma 0.04: n=30 test MSE U-shaped (interior minimum at d=2): held
H3 sigma 0.04: best degree non-decreasing in n: held
H4 sigma 0.02 n 30: ridge lowers test MSE and gap: held
H4 sigma 0.02 n 100: ridge lowers test MSE and gap: held
H4 sigma 0.02 n 300: ridge lowers test MSE and gap: held
H4 sigma 0.04 n 30: ridge lowers test MSE and gap: held
H4 sigma 0.04 n 100: ridge lowers test MSE and gap: held
H4 sigma 0.04 n 300: ridge lowers test MSE and gap: held
H5 n 30: best degree lower at sigma 0.04 than at 0.02: held
H5 n 100: best degree lower at sigma 0.04 than at 0.02: failed
H5 n 300: best degree lower at sigma 0.04 than at 0.02: held
H6 sigma 0.02 n 30: border above interior at the top degree: held
H6 sigma 0.02 n 100: border above interior at the top degree: held
H6 sigma 0.02 n 300: border above interior at the top degree: held
H6 sigma 0.04 n 30: border above interior at the top degree: held
H6 sigma 0.04 n 100: border above interior at the top degree: held
H6 sigma 0.04 n 300: border above interior at the top degree: held

The first four lines are checks, and all held. The training pixels avoid the objects and do not repeat, the largest $p/n$ is 0.933, below the interpolation threshold. A noise-free cubic surface is recovered to rounding error at degree 3 and at degree 10, the hand fits agree with scikit-learn’s to about $10^{-12}$, and the training MSE fell with every added degree in every condition (hypothesis H1, which holds exactly for nested least squares, held).

Underfitting. Degrees 0 and 1 cannot follow the vignetting: with 300 samples their training MSE (2.393e-02 and 1.283e-02) is close to their test MSE and to the approximation floor (2.363e-02 and 1.277e-02), and far above the noise variance $\sigma^2 = 4 \times 10^{-4}$, the error a perfect model would make on noisy training pixels. That is the signature of the first row of the diagnosis table: both errors high and close together, and more data does not help, since the degree-0 and degree-1 test MSEs barely change from $n = 30$ to $n = 300$. The floor shows what does help: it falls by a factor of more than 600 between degree 1 and degree 4.

Overfitting. At the other end, the degree-6 surface fitted to 30 pixels has a training MSE of 3.612e-05, about a tenth of $\sigma^2$: it fits the noise in the samples, which no surface can do on new pixels. Its mean test MSE is 1.466e+01, about 38 times the square of the field’s whole range (0.2204 to 0.8411), with one seed as high as 92. Between the two, the $n = 30$ test MSE falls to a minimum at degree 3 and rises again, the U shape of hypothesis H2, which held at $\sigma = 0.02$ (minimum at degree 3) and at $\sigma = 0.04$ (minimum at degree 2). The condition number grows with degree to $5.6 \times 10^3$ at degree 6 for $n = 30$ and $3.8 \times 10^4$ at degree 10 for $n = 100$: in our reading, the coefficients are poorly determined by the samples well before the system becomes singular.

The gap against the formula. With 300 samples and degrees 4 and above, the printed gap stays within a factor of two of $2\sigma^2 p/n$, the in-sample prediction (for example 4.636e-05 against 4.000e-05 at degree 4, and 3.114e-04 against 1.760e-04 at degree 10). With 30 samples the gap at degree 6 is 1.466e+01 against 7.467e-04. The formula assumes test pixels at the training locations; here the test pixels are new locations, including corners far from any sample, and in our reading that is why the measured gap exceeds it most where the fit is least constrained, which is consistent with the border block below. At degrees 0 and 1 the gap also exceeds the formula (2.997e-03 against 2.667e-05 at $n = 30$, degree 0): in our reading, the bias terms cancel in the derivation only because the test points are the training points, and a model that underfits makes different errors at the 30 sampled pixels than across the rest of the image.

The data lever. More samples move the best degree up: 3, 4 and 5 for $n = 30$, 100 and 300 at $\sigma = 0.02$, and 2, 4 and 4 at $\sigma = 0.04$, so hypothesis H3 held at both noise levels. At a fixed degree, more data lowers the test MSE and raises the training MSE toward $\sigma^2$, the pattern of Murphy’s learning curves: at degree 4 the training MSE rises from 2.199e-04 to 3.709e-04 to 3.932e-04 as $n$ goes from 30 to 300, while the test MSE falls from 2.035e-03 to 3.959e-05. The degree-10 surface that is useless with 100 samples (test MSE 4.339e-01) is respectable with 300 (2.227e-04), and the gap shrinks with it.

Noise. Doubling $\sigma$ was expected to lower the best degree (hypothesis H5). It held at $n = 30$ (from 3 to 2) and at $n = 300$ (from 5 to 4), and failed at $n = 100$, where the best degree stayed at 4; seed by seed, the best degree at $\sigma = 0.04$ was lower than at $\sigma = 0.02$ on three of the ten seeds and the same on seven. In our reading, the printed floor explains the exception: it drops by a factor of about 17 between degrees 3 and 4 (3.321e-04 to 1.981e-05), a gain in approximation that the doubled noise did not outweigh at 100 samples (test MSE 5.795e-04 at degree 3 against 4.339e-04 at degree 4 for $\sigma = 0.04$).

Ridge. The fixed ridge penalty at the top degree lowered both the mean test MSE and the mean gap in all six conditions, and on every seed, so hypothesis H4 held. Its effect is largest where least squares overfits most: at $n = 30$ and degree 6 the test MSE falls from 1.466e+01 to 5.550e-03. But the penalty was not tuned, and it did not match choosing the degree: at $n = 30$ the ridge fit’s 5.550e-03 is still worse than the degree-3 fit’s 9.653e-04, and at $n = 300$ ridge at degree 10 (1.022e-04) is behind degree 5 without a penalty (3.485e-05). In practice the strength of the penalty is itself a capacity hyperparameter, chosen on validation data like the degree (Murphy, §4.5.4).

Where the overfitting is. At the top degree, the test MSE on the 16-pixel border band exceeds the interior’s in all six conditions, by factors of 6.4 to 526, so hypothesis H6, the image analogue of Caruana and colleagues’ regional overfitting, held. A flat-field correction from such a fit would be worst at the edges of the image, which a single average error hides and a map of the residuals shows. The ratio at the best degree, 1.7 to 3.9, was printed for comparison and decides nothing.

Process. The plan for this example was written before any code ran, with every hypothesis above, and its checksum was recorded before a pilot run. It departs from the batch plan’s description of the example in nine ways, each fixed before the pilot: (1) ten seeds rather than the minimum of five; (2) “below the interpolation threshold” taken as $p < n$, which limits $n = 30$ to degrees 0–6, so its top degree is 6; (3) the gap measured against fresh noisy pixels, while the test MSE is against the noise-free field; (4) the added printed values $2\sigma^2 p/n$ and the approximation floor, which decide nothing; (5) the border and interior measure with hypothesis H6, added because of Caruana and colleagues’ observation; (6) the ridge penalty fixed at $4 \times 10^{-4}$, from Murphy’s MAP reading, at both noise levels; (7) the field, object, noise and border-width parameters; (8) the choice of the scikit-learn check fit ($n = 100$, degree 10, $\sigma = 0.02$, seed 0); and (9) the same noise draw scaled for both noise levels. A planning script computed the check tolerances from the values a correct implementation reaches on this set-up, before the pilot, and the approximation floor printed by that script confirmed, before any hypothesis was tested, that the field is learnable by the polynomial family and not by its lowest degrees. After the pilot, two print-only changes were made, both approved before the first run: the per-seed best degrees were printed as plain integers instead of NumPy scalar representations, and the words held or failed were added to the two check lines from the comparisons they already printed. No value changed, and nothing was changed after the first run.

Common Misconceptions

• “Overfitting means the test error is rising.” Overfitting is a large gap between training and test error (Goodfellow et al., §5.2). In the example, with 300 samples the test MSE rises from degree 5 to degree 10, yet the degree-10 surface is still better than every degree up to 3; whether it overfits is read from its gap, not from that rise alone.
• “A model that underfits needs more data.” If the training error itself is too high, the model is not using the data it has (Goodfellow et al., §11.3); in the example, degrees 0 and 1 gain almost nothing from ten times as many samples.
• “Low training error is a good sign.” A training error below the noise level, like the degree-6 fit’s 3.612e-05 against $\sigma^2 = 4 \times 10^{-4}$, means the model is fitting noise.
• “More capacity always overfits.” Whether capacity is excessive depends on the data: the degree-10 surface overfits badly with 100 samples and much less with 300. With early stopping, Caruana and colleagues found nets that were too large generalizing worse than smaller ones on only three of their seven problems, and by a small margin.
• “Overfitting is one number.” It can be concentrated in parts of the input: in the example the top-degree fits’ test MSE on the image border is 6.4 to 526 times that of the interior.
• “Regularization replaces model selection.” A penalty strength chosen without validation can help, as the fixed ridge penalty did, and still fall short of the right capacity; the penalty is a hyperparameter to be selected like any other.

Where It Is Used

• Training workflows: neural network training reads training and validation curves for both failures, early stopping stops at the validation minimum, and hyperparameter tuning adjusts capacity by watching them.
• Evaluation: the training, validation and test sets separate fitting, selection and assessment; cross-validation estimates validation error with few labelled images; benchmark overfitting studies the effect of reusing test sets.
• Remedies: regularization, weight decay, dropout, data augmentation, transfer learning and fine-tuning reduce overfitting; the inductive bias of an architecture sets how much data a model needs before it stops overfitting.
• Classical estimation in vision: fitting smooth models to image measurements, such as shading and vignetting models in image formation, polynomial lens models in lens distortion and the least squares fits throughout geometry, faces the same choice of model complexity against the number and noise of the measurements.