Hyperparameter Tuning

Hyperparameter tuning is the search for the settings of a learning procedure that training itself does not learn: the learning rate and its schedule, the batch size, the weight-decay and dropout strengths, the augmentation magnitudes, the width and depth of the network. Each setting defines a different training run, and the run that scores best on held-out validation data is kept. Bergstra and Bengio (2012) write the problem as the minimization of a response function $\Psi(\lambda)$, the validation loss of the model trained with hyperparameters $\lambda$, over a search space $\Lambda$; in practice $\Psi$ is evaluated at a finite set of trials and the best one is returned. The workflow around it, from the data splits to the learning curves, is covered in Training a Neural Network, and the roles of the three data splits in Training, Validation, and Test Sets. This article covers the search: which hyperparameters matter, how search spaces are set, the strategies from grid search to Hyperband and population-based training, and what tuning costs and does to the numbers papers report.

Purpose

The best values depend on the data, the architecture and the other hyperparameters. Goodfellow et al. (§11.4.2) note that logistic regression and SVMs owe part of their popularity to needing only one or two tuned hyperparameters, while neural networks “often benefit significantly” from tuning forty or more. Every evaluation of $\Psi$ is a full training run, so the search is expensive, noisy (the same setting gives different scores with different seeds) and has no gradient for most hyperparameters. Tuning methods differ in how they choose the next trial and in how much training they give each one.

Tuning also decides which results can be compared. When one method is tuned with a large budget and its baseline with a small one, the comparison measures the budgets; several published re-evaluations, summarized under Evidence and Impact, found that differences between methods shrank once every method was tuned under the same protocol.

When to Use It

Tuning is needed whenever a training procedure is applied to a new dataset or architecture with no trustworthy defaults. It helps most when:

• a few hyperparameters dominate and their good region is narrow, as is often the case for the learning rate;
• the defaults come from a different setting: the DenseNet authors, for example, trained on ImageNet with hyperparameters tuned for ResNets and expected that more tuning could improve their results;
• a fair comparison between methods requires each to be tuned with the same budget.

It helps little, or misleads, when:

• the validation set is too small for the score differences between settings to exceed its noise, so the search chases noise (the statistical significance article gives the binomial arithmetic);
• the validation data come from a different distribution than deployment, so the tuned optimum is the wrong one;
• the test set is used to choose, which turns the reported score into an optimistic one (cross-validation shows the size of this optimism for a grid of 42 settings).

How It Works

What is tuned, and which hyperparameters matter

Goodfellow et al. (§11.4.1) organize manual tuning around effective capacity. Generalization error typically follows a U-shaped curve in a capacity hyperparameter: too little capacity underfits, too much overfits. Their Table 11.1 lists the direction of each effect: capacity rises as the number of hidden units or the kernel width increases, and as the weight-decay coefficient or the dropout rate decreases. The learning rate is the exception. Capacity is highest when it is correct for the optimization problem, not when it is especially large or small, and the book calls it perhaps the most important hyperparameter: if only one can be tuned, tune it. In the idealized quadratic case, gradient descent increases the training error once the learning rate is at least twice its optimal value (§11.4.1, citing LeCun et al.).

Which of the others matter depends on the problem. Bergstra and Bengio (§3) fitted a Gaussian process with one length scale per hyperparameter to the validation accuracies of 256 random trials of a one-hidden-layer network on each of eight image datasets (MNIST variants with rotations and backgrounds, and synthetic rectangle and convexity tasks). Of the seven hyperparameters, only one to four mattered on any dataset, and different ones mattered on different datasets: the learning rate always, but sometimes also its annealing, the L2 penalty or the number of hidden units. They call this a low effective dimension: $\Psi$ varies much more along some axes than others, and which axes those are is not known in advance.

Search spaces and scales

A search space gives each hyperparameter a range and a distribution. Positive hyperparameters whose effect is multiplicative, such as the learning rate, the weight-decay coefficient and often the number of units, are searched on a logarithmic scale. Goodfellow et al. (§11.4.4) give the example of sampling $\log_{10}$ of the learning rate uniformly between $-5$ and $-1$; their grid search example uses $\{0.1, 0.01, 10^{-3}, 10^{-4}, 10^{-5}\}$ (§11.4.3). Bergstra and Bengio drew the learning rate of their networks log-uniformly from 0.001 to 10 and the number of hidden units geometrically from 18 to 1,024, and made preprocessing, the activation function and the initialization scheme categorical choices. A conditional hyperparameter exists only when another takes a certain value, such as the fraction of variance kept when PCA preprocessing is chosen; their deep belief network space had 8 global hyperparameters and 8 per layer, 32 in all for three layers.

The bounds matter as much as the method. If the best value lies outside the range, no search finds it; Goodfellow et al. (§11.4.3) advise moving the grid when the best value lands on its edge and refining it when it lands inside.

Grid search

Grid search chooses a small set of values for each hyperparameter and trains a model for every combination; with $m$ hyperparameters of $n$ values each, the number of trials grows as $O(n^m)$ (§11.4.3). It is simple, reproducible and trivially parallel, and Goodfellow et al. describe it as common practice for three or fewer hyperparameters. Its weakness is the one low effective dimension exposes: a grid of $k^m$ points tests only $k$ distinct values of each hyperparameter. When only one hyperparameter matters, $k^{m-1}$ trials at each of its values are repetitions that differ only in settings that do not matter.

Random search

Random search draws each trial independently from the search space. Bergstra and Bengio’s Figure 1 shows the argument with nine trials on a function $f(x, y) = g(x) + h(y) \approx g(x)$: a $3 \times 3$ grid tests $g$ at three places, nine random points at nine. Random search has the efficiency in the relevant subspace that it would have had if only the relevant dimensions had been searched, without knowing which those are. The trials are independent, so a search can be extended, stopped or run on unreliable machines that drop jobs, which a grid does not allow.

Their experiments compared random search with the grid searches of Larochelle et al. (2007) for neural networks on the same datasets. Random searches of 8 trials matched or beat grids of on average 100 trials when both searched the same seven hyperparameters (their Figure 5). For deep belief networks, with up to 32 hyperparameters, random search with 128 trials was compared with Larochelle et al.’s combination of manual and grid search: the §5 text and the Figure 9 caption report a better model on one of eight datasets, an equally good one on four and a worse one on three, while the abstract speaks of “four of seven data sets” equal and one of seven superior. In a simulation (§4), low-discrepancy Sobol sequences found small target regions slightly more reliably than random points, and grids were the worst performers for elongated targets, which have low effective dimension.

The authors present random search as a baseline against which adaptive methods should be judged, and the Deep Learning book (§11.4.4) recommends it over grid search for the same reasons, with repeated runs that refine the space around good results.

Bayesian optimization

Model-based, or Bayesian, optimization fits a probabilistic model to the trials evaluated so far and chooses the next trial by maximizing an acquisition function that trades exploration of uncertain regions against exploitation of good ones (Goodfellow et al., §11.4.5). BOHB’s paper (Falkner et al., 2018, §3.1) writes the loop as: choose $x_{\text{new}} = \arg\max_x a(x)$, evaluate it, add it to the data and refit. A common acquisition function is the expected improvement over the best value seen so far, $\alpha$:

$$a(x) = \int \max(0,\ \alpha - f(x))\, dp(f \mid \mathcal{D}).$$

Gaussian processes are the most common model. Falkner et al. note that they scale cubically with the number of observations and do not apply to complex configuration spaces without special kernels. The Tree Parzen Estimator (TPE; Bergstra et al., 2011) instead models two densities over the search space, $l(x)$ for configurations whose loss is below a threshold and $g(x)$ for the rest, and chooses the candidate that maximizes $l(x)/g(x)$, which Falkner et al. state is equivalent to maximizing expected improvement; kernel density estimators handle mixed continuous and discrete spaces and scale linearly. Li et al. (2018, §2.1) name three established methods: SMAC, which models performance with random forests, TPE, and Spearmint (Snoek et al., 2012), which uses Gaussian processes; the CIFAR-10 article gives the error rates Snoek et al. reported after retuning the cuda-convnet network.

The Deep Learning book (§11.4.5), written in 2016, could not “unambiguously recommend” Bayesian hyperparameter optimization, which sometimes matched or beat human experts and sometimes failed catastrophically. Li et al. (§2.1) cite evidence that standard Bayesian optimization performs similarly to random search on high-dimensional problems.

Successive halving and Hyperband

Black-box methods, Bayesian or random, train every configuration to completion before learning anything from it, although a practitioner can usually tell early that a run is hopeless (Goodfellow et al., §11.4.5). Configuration evaluation methods exploit partial training. Jamieson and Talwalkar (2016) cast hyperparameter optimization as a non-stochastic best-arm identification problem, in which each arm is a configuration, a pull is a fixed number of training iterations and the observed loss is the validation loss of the partially trained model, and analysed the successive halving algorithm of Karnin et al. (2013) for it: give $n$ configurations an equal budget, evaluate them all, discard the worst half, and repeat with the survivors until one remains. In Hyperband’s version (Li et al., 2018) each round keeps a fraction $1/\eta$ instead of half.

Successive halving needs $n$ as an input, and for a fixed total budget $B$ the choice between many configurations with a little training each and few configurations trained longer depends on things not known in advance: how quickly the learning curves separate and how good a random configuration typically is. Li et al. call this the “$n$ versus $B/n$” problem. Hyperband (Li et al., 2018) hedges by running successive halving with several values of $n$. With $R$ the maximum resource for one configuration (in epochs, for example) and $\eta$ the elimination factor, $s_{\max} = \lfloor \log_\eta R \rfloor$, and each bracket $s = s_{\max}, \ldots, 0$ starts

$$n = \left\lceil \frac{s_{\max}+1}{s+1}\,\eta^{s} \right\rceil$$

configurations with resource $r = R\eta^{-s}$, then keeps the best $\lfloor n_i/\eta \rfloor$ at each rung $i$ and multiplies their resource by $\eta$ (their Algorithm 1). The most aggressive bracket, $s = s_{\max}$, starts with $R$ configurations at one unit of resource; the last, $s = 0$, trains $s_{\max}+1$ configurations fully and is plain random search. Algorithm 1 returns the configuration with the smallest intermediate loss seen. For $R = 81$ and $\eta = 3$, Algorithm 1 gives brackets starting with 81, 34, 15, 8 and 5 configurations; the paper’s Table 1, for the same $R$ and $\eta$, lists 81, 27, 9, 6 and 5, so the table and the algorithm disagree for the middle three brackets. The example below implements Algorithm 1 and prints both.

The resource need not be iterations: Li et al. (§3.4) also use training-set subsamples and numbers of random features. They recommend $\eta$ of 3 or 4 and write that results are not very sensitive to it (§3.6). Their analysis shows that Hyperband needs a budget only log factors larger than successive halving run with the best $n$, without knowing it (§3.7).

In their convolutional-network experiments on CIFAR-10, rotated MNIST with background images and SVHN (one unit of resource = 100 minibatch iterations, $R = 300$ or 600, $\eta = 4$, a total budget of $50R$ per searcher, ten trials each), Hyperband was more than an order of magnitude faster than SMAC, TPE and Spearmint on CIFAR-10. On SVHN, Bayesian methods were competitive and SMAC with early termination of learning curves did better (§4.1, Figure 4). On 117 OpenML datasets with dataset subsampling as the resource, the differences in test error from random search were nearly all within half a point, and random search with twice the budget ranked best on test error (§4.2.1, Figures 5 and 6).

The method’s weak point is the assumption that partial training ranks configurations correctly. Li et al. (§6) name configurations with different convergence rates, such as different learning-rate decays or network sizes, as an open problem: when the slow configurations end up best, early elimination discards them.

ASHA

In Hyperband as written, a rung of successive halving waits until all its configurations have been evaluated, and Li et al. (§6) left parallel versions to future work. ASHA, from a later paper by Li, Jamieson, Rostamizadeh and others, “Massively Parallel Hyperparameter Tuning” (2018, not read here), is described by Li and Talwalkar (2019, §2.1) as random search combined with early stopping, which they used as a hyperparameter-optimization baseline for neural architecture search. In their CIFAR-10 runs, ASHA started each architecture at 1 epoch, allowed at most 300, and promoted the top quarter of each rung to four times the resource (§4); in 9 GPU days it evaluated over 700 architectures, and its best architecture reached an average test error of 3.03 ± 0.13 (§4.1.1). Falkner et al. (§4.2) describe the parallel scheme of that work as filling the next free worker with the ready-to-run job of the largest budget.

BOHB

Random sampling gives Hyperband strong performance at small budgets, but with large budgets its advantage over random search typically diminishes, because it relies on randomly drawn configurations (Falkner et al., §3.2). BOHB (Falkner et al., 2018) keeps Hyperband’s brackets and budgets but replaces the random choice of new configurations with a TPE-like model: one multidimensional kernel density estimator over the configurations, fitted on the largest budget that has enough observations, with a fixed fraction $\rho$ of configurations still drawn at random so that Hyperband’s guarantees are kept (their §4.1, Algorithm 2). In their last experiment BOHB tuned the learning rate, momentum, weight decay and batch size of a Shake-Shake residual network with Cutout on CIFAR-10, with budgets of 22, 66, 200 and 600 epochs on 19 parallel workers; the run took 33 GPU days and reached a test error of 2.78% ± 0.09% (§5.5).

Population-based training

Hyperband and BOHB choose one configuration for a whole run. Population-based training (PBT; Jaderberg et al., 2017, not read here) adapts hyperparameters during training. As Ho et al. (2019, §3.2) describe it, it optimizes a network’s parameters jointly with its hyperparameters: a fixed population of models is trained in parallel; at intervals, the worse members copy the weights of a better member (“exploit”) and perturb its hyperparameters (“explore”). The output is a trained model and a schedule of hyperparameters rather than a single setting, and because weights are copied rather than reinitialized, the total cost is that of training one model times the population size. Ho et al. used it to learn schedules of augmentation probabilities and magnitudes (Population Based Augmentation): a trial was ready for exploit and explore every 3 epochs, a member in the bottom 25% copied one in the top 25%, and exploration resampled or perturbed each hyperparameter (§3.4).

Choosing on validation data

Every strategy above minimizes a validation estimate, so the protocol around the search decides what the final number means. The search runs on a validation split or inside cross-validation, the selected configuration is often retrained on training and validation data together, and the test set is used once. Bergstra and Bengio (§2.1) point out that test error is not monotone in validation error: when several trials have nearly equal validation scores, which test score to report is unclear, so they report a weighted average of the trials’ test scores, each weighted by the probability that the trial is the best given the validation noise. Reporting the selected configuration’s own validation score overstates it, since it is the maximum of many noisy estimates; nested cross-validation estimates the whole tune-then-train procedure without that selection optimism.

How to Apply It

1. Fix the protocol first. Choose the validation split (grouped by video, patient or scene where needed), the metric and the total compute budget, and keep the test set out of every decision.
2. Start from a known recipe and tune the learning rate first. Goodfellow et al. (§11.4.1) single it out as the one hyperparameter to tune when only one can be; the others are then adjusted by watching whether the model under- or overfits.
3. Write the search space down. Log scales for learning rates, weight decay and widths; bounds wide enough that the best value is not at an edge; conditional hyperparameters only where they apply.
4. Use random search as the baseline. It is as simple as a grid and does not waste trials on unimportant dimensions; add early stopping of poor runs (successive halving, Hyperband, ASHA) when partial training is informative, and model-based proposals (BOHB, Bayesian optimization) when the budget allows many sequential rounds.
5. Check the result. Look at where the best trials lie in the space, retrain the chosen setting with several seeds, and report the spread.

Example

The script compares search strategies on synthetic images with known structure, so that every evaluation is exact and cheap. The images are 16×16 grayscale pictures of a horizontal bar (+1) or a vertical bar (−1), 10 by 3 pixels, whose centre is shifted by up to two pixels in each direction, with contrast drawn from 0.5 to 1.0 and Gaussian noise of standard deviation 0.5; floating-point intensities on a zero background. There are 100 training images, 500 validation images and, for the second study, 1,000 test images.

Study 1 is Bergstra and Bengio’s comparison on a three-dimensional space. A setting blurs the images with a Gaussian of $\sigma \in [0, 3]$ pixels (edges replicated), keeps the principal components of the centred training pixels that explain a fraction $v \in [0.5, 1]$ of their variance, and fits ridge regression to the ±1 labels with strength $\lambda$, $\log_{10}\lambda \in [-4, 2]$, minimizing $\frac{1}{n}\lVert Z w - y\rVert^2 + \lambda \lVert w \rVert^2$ in closed form; $\Psi$ is the validation error of the sign of the prediction. Grids of $k = 2$ to 5 values per axis (8 to 125 trials) are placed two ways, since the placement is a choice the sources leave open: at the centres of $k$ equal cells, or including the endpoints. Random search is repeated 60 times with seeds 1000–1059; a search of $S$ trials uses the first $S$ draws. The script also measures how much of the variance of $\Psi$ each hyperparameter explains on its own (its main effect, from ten equal-width bins), as a check of the effective dimension.

Study 2 compares successive halving and Hyperband with random search on gradient descent from $w = 0$ on $\frac{1}{2n}\lVert X w - y_c \rVert^2 + \frac{\lambda}{2} \lVert w \rVert^2$ (pixels blurred with $\sigma = 1$ and centred; the intercept is the mean label), whose iterates are computed exactly through the eigendecomposition of $X^\top X / n$. The hyperparameters are the learning rate, $\log_{10}\alpha \in [-3, 0.5]$, and $\log_{10}\lambda \in [-4, 0]$; gradient descent diverges when $\alpha(\mu_{\max} + \lambda) > 2$, with $\mu_{\max}$ the largest eigenvalue, and a setting whose weights overflow is scored as chance (0.5). One unit of resource is 10 steps, $R = 81$ units and $\eta = 3$. To make ranks exact, a pool of 5,000 settings is drawn once and evaluated at every rung; each bracket in each repetition draws its settings from the pool without replacement. Over 200 repetitions (seeds 5000–5199), the script runs each Hyperband bracket alone, full Hyperband, and random search with Hyperband’s total budget. Two choices are printed both ways. The budget counts each rung either as retraining from scratch, as Algorithm 1 is written (each call trains a configuration for $r_i$ units), or as resumed from the previous rung. The return rule is either Algorithm 1’s, the smallest validation error seen at any rung, which can return a partially trained model, or the best of the settings trained to $R$. For each method the script prints the budget in multiples of $R$, the mean test error of the returned model and the mean percentile of the returned setting’s full-budget validation error within the pool (0 is best). It runs in about 10 seconds with NumPy 2.3.5 and SciPy 1.16.3, and two runs give identical output.

import os
os.environ["OMP_NUM_THREADS"] = "1"                 # identical sums on every run
import math
import numpy as np
from scipy.ndimage import gaussian_filter
from scipy.stats import spearmanr

REPS1, REPS2 = 60, 200          # repetitions of Study 1 and Study 2

# ---------- data: horizontal (+1) vs vertical (-1) bars in noise ----------
S = 16
yy, xx = np.mgrid[0:S, 0:S]

def make(n, rng):
    y = np.repeat([1.0, -1.0], n // 2)
    X = np.zeros((n, S, S)); info = []
    for i in range(n):
        dy, dx = rng.integers(-2, 3, size=2)
        c = rng.uniform(0.5, 1.0)
        a, b = (yy - 7.5 - dy, xx - 7.5 - dx) if y[i] > 0 else (xx - 7.5 - dx, yy - 7.5 - dy)
        X[i] = c * ((np.abs(a) <= 1.0) & (np.abs(b) <= 4.5))
        info.append((dy, dx, c))
    return X + 0.5 * rng.standard_normal(X.shape), y, np.array(info)

def blur(X, sig):                # separable Gaussian, edges replicated
    G = np.eye(S) if sig <= 0 else gaussian_filter(np.eye(S), (sig, 0), mode="nearest")
    return G @ X @ G.T

rng = np.random.default_rng(0)
Xtr, ytr, itr = make(100, rng); Xva, yva, iva = make(500, rng); Xte, yte, ite = make(1000, rng)
inf = np.vstack([itr, iva, ite])
print(f"data: class means {ytr.mean():.0f} {yva.mean():.0f} {yte.mean():.0f}; jitter {inf[:,:2].min():.0f}..{inf[:,:2].max():.0f}; "
      f"contrast {inf[:,2].min():.3f}..{inf[:,2].max():.3f}")

# ---------- Study 1: blur sigma, PCA variance kept v, ridge lambda ----------
LO, HI = np.array([-4.0, 0.0, 0.5]), np.array([2.0, 3.0, 1.0])   # log10 lam, sigma, v

def ridge_pca(A, y, lam, v):
    ev, U = np.linalg.eigh(A @ A.T); ev, U = ev[::-1], U[:, ::-1]
    keep = ev > 1e-10 * ev[0]; ev, U = ev[keep], U[:, keep]
    k = int(np.searchsorted(np.cumsum(ev) / ev.sum(), v - 1e-12) + 1)
    s = np.sqrt(ev[:k]); V = A.T @ U[:, :k] / s
    return V @ (s / (s ** 2 + len(A) * lam) * (U[:, :k].T @ y))

def psi(p):                      # validation error of one setting
    loglam, sig, v = p
    A = blur(Xtr, sig).reshape(100, -1); B = blur(Xva, sig).reshape(500, -1)
    mu = A.mean(0)
    w = ridge_pca(A - mu, ytr - ytr.mean(), 10.0 ** loglam, v)
    return np.mean(np.sign((B - mu) @ w + ytr.mean()) != yva)

A = blur(Xtr, 1.0).reshape(100, -1); A = A - A.mean(0)
w_direct = np.linalg.lstsq(np.vstack([A, np.sqrt(100 * 0.1) * np.eye(256)]),
                           np.r_[ytr - ytr.mean(), np.zeros(256)], rcond=None)[0]
print(f"check C1: PCA ridge vs direct solve, max |diff| {np.abs(ridge_pca(A, ytr - ytr.mean(), 0.1, 1.0) - w_direct).max():.1e}")

print("\nStudy 1: best validation error after S trials")
print("   S   grid(centred) grid(ends)   random median [10%-90%]   random<=grid  distinct blur grid/random")
draws = [np.random.default_rng(1000 + r).uniform(LO, HI, size=(125, 3)) for r in range(REPS1)]
vals = np.array([[psi(p) for p in d] for d in draws])
for k in (2, 3, 4, 5):
    n = k ** 3
    best = []
    for place in ("centred", "ends"):
        ax = [(LO[j] + (np.arange(k) + 0.5) * (HI[j] - LO[j]) / k) if place == "centred"
              else np.linspace(LO[j], HI[j], k) for j in range(3)]
        grid = np.array(np.meshgrid(*ax, indexing="ij")).reshape(3, -1).T
        assert len(np.unique(grid, axis=0)) == n and np.all((grid >= LO) & (grid <= HI))
        best.append(min(psi(p) for p in grid))
    rb = vals[:, :n].min(axis=1)
    q10, q50, q90 = np.quantile(rb, [0.1, 0.5, 0.9])
    print(f"{n:4d}     {best[0]:.3f}      {best[1]:.3f}      {q50:.3f} [{q10:.3f}-{q90:.3f}]"
          f"         {np.mean(rb <= best[0]):.2f}         {k} / {n}")
P = np.vstack(draws)[:, :]; V = vals.ravel()
assert np.all((P >= LO) & (P <= HI))
print(f"best value seen {V.min():.3f}; main-effect share of variance:", end="")
for j, name in enumerate(["log10 lambda", "blur", "variance kept"]):
    bins = np.minimum(((P[:, j] - LO[j]) / (HI[j] - LO[j]) * 10).astype(int), 9)
    m = np.array([V[bins == b].mean() if np.any(bins == b) else V.mean() for b in range(10)])
    c = np.array([np.sum(bins == b) for b in range(10)])
    print(f" {name} {np.sum(c * (m - V.mean()) ** 2) / len(V) / V.var():.2f}", end="")
print()

# ---------- Study 2: successive halving and Hyperband on gradient descent ----------
A = blur(Xtr, 1.0).reshape(100, -1); mu = A.mean(0); A = A - mu
b = A.T @ (ytr - ytr.mean()) / 100
ev, Q = np.linalg.eigh(A.T @ A / 100); ev = np.clip(ev, 0, None)
c = Q.T @ b
Pv = (blur(Xva, 1.0).reshape(500, -1) - mu) @ Q; Pt = (blur(Xte, 1.0).reshape(1000, -1) - mu) @ Q

def gd_weights(alpha, lam, t):   # closed-form iterate t of GD from w = 0 (eigenbasis)
    with np.errstate(over="ignore", invalid="ignore"):
        return (1 - (1 - alpha * (ev + lam)) ** t) / (ev + lam) * c

w = np.zeros(256)
for _ in range(810):
    w -= 0.05 * (A.T @ (A @ w) / 100 + 1e-3 * w - b)
print(f"\ncheck C2: closed-form GD vs loop, max rel diff {np.abs(Q @ gd_weights(0.05, 1e-3, 810) - w).max() / np.abs(w).max():.1e}")

UNIT, R, ETA, RUNGS = 10, 81, 3, [1, 3, 9, 27, 81]
N = 5000
pr = np.random.default_rng(777)
pool = np.column_stack([10.0 ** pr.uniform(-3, 0.5, N), 10.0 ** pr.uniform(-4, 0, N)])
ev_v, ev_t = np.zeros((N, 5)), np.zeros((N, 5))
for j, r in enumerate(RUNGS):
    W = gd_weights(pool[:, :1], pool[:, 1:], UNIT * r)
    ok = np.all(np.isfinite(W), axis=1); W = np.nan_to_num(W)
    with np.errstate(over="ignore", invalid="ignore"):   # rows with ok = False are scored 0.5
        ev_v[:, j] = np.where(ok, np.mean(np.sign(W @ Pv.T + ytr.mean()) != yva, axis=1), 0.5)
        ev_t[:, j] = np.where(ok, np.mean(np.sign(W @ Pt.T + ytr.mean()) != yte, axis=1), 0.5)
print("pool: unstable share", f"{np.mean(pool[:,0] * (ev.max() + pool[:,1]) > 2):.3f};",
      "min full-budget val error", f"{ev_v[:,4].min():.3f};",
      "Spearman with r=81:", " ".join(f"r={r}: {spearmanr(ev_v[:,j], ev_v[:,4]).correlation:.2f}" for j, r in enumerate(RUNGS[:4])))
col = {r: j for j, r in enumerate(RUNGS)}

def bracket(s, ids):             # Algorithm 1, lines 2-9, on pool indices; returns (seen, cost_scratch, cost_resume)
    n = math.ceil((4 + 1) * ETA ** s / (s + 1)); r = R // ETA ** s
    T = ids[:n]; seen = []; cs = cr = 0; prev = 0
    for i in range(s + 1):
        ri = r * ETA ** i
        L = ev_v[T, col[ri]]
        seen += [(L[q], T[q], ri) for q in range(len(T))]
        cs += len(T) * ri; cr += len(T) * (ri - prev); prev = ri
        T = T[np.argsort(L, kind="stable")[: len(T) // ETA]]
    return seen, cs, cr, n

def ret(seen, rule):             # returned (setting, rung)
    if rule == "any":            # line 11: smallest intermediate loss seen
        l, i, r = min(seen, key=lambda z: (z[0], -z[2]))
    else:                        # best among settings trained to R
        l, i, r = min([z for z in seen if z[2] == R], key=lambda z: z[0])
    return i, r

pct = lambda i: np.mean(ev_v[:, 4] < ev_v[i, 4])
res = {}
def add(name, i, r, cs, cr):
    res.setdefault(name, []).append((ev_t[i, col[r]], pct(i), cs / R, cr / R))

budget_s = budget_r = 0
for s in range(4, -1, -1):
    _, a_, b_, _ = bracket(s, np.arange(N)); budget_s += a_; budget_r += b_
print(f"Hyperband budget: {budget_s} units from scratch ({budget_s/R:.2f} R), {budget_r} resumed ({budget_r/R:.2f} R)")
print("bracket sizes n_0:", [bracket(s, np.arange(N))[3] for s in range(4, -1, -1)], "(Table 1 of the paper: 81, 27, 9, 6, 5)")

for rep in range(REPS2):
    rg = np.random.default_rng(5000 + rep)
    hb_seen, hs, hr = [], 0, 0
    for s in range(4, -1, -1):
        seen, cs, cr, _ = bracket(s, rg.choice(N, size=81, replace=False))
        hb_seen += seen; hs += cs; hr += cr
        for rule in ("any", "R"):
            add(f"bracket s={s} ({rule})", *ret(seen, rule), cs, cr)
    for rule in ("any", "R"):
        add(f"Hyperband ({rule})", *ret(hb_seen, rule), hs, hr)
    for acc, units in (("scratch", budget_s), ("resumed", budget_r)):
        ids = rg.choice(N, size=units // R, replace=False)
        i = ids[np.argmin(ev_v[ids, 4])]
        add(f"random, {acc} budget", i, R, len(ids) * R, len(ids) * R)

print(f"\nStudy 2 over {REPS2} repetitions (return rule: any = smallest loss at any rung, R = best trained to R)")
print("method                      budget(R) scratch/resumed   test error   percentile in pool")
for name, v in res.items():
    v = np.array(v)
    print(f"{name:28s}   {v[:,2].mean():5.2f} / {v[:,3].mean():5.2f}        {v[:,0].mean():.3f}        {100*v[:,1].mean():5.1f}")

data: class means 0 0 0; jitter -2..2; contrast 0.500..1.000
check C1: PCA ridge vs direct solve, max |diff| 1.0e-15

Study 1: best validation error after S trials
   S   grid(centred) grid(ends)   random median [10%-90%]   random<=grid  distinct blur grid/random
   8     0.158      0.184      0.152 [0.140-0.164]         0.73         2 / 8
  27     0.154      0.146      0.144 [0.138-0.154]         0.90         3 / 27
  64     0.152      0.144      0.140 [0.130-0.144]         1.00         4 / 64
 125     0.142      0.144      0.136 [0.130-0.142]         0.93         5 / 125
best value seen 0.124; main-effect share of variance: log10 lambda 0.00 blur 0.77 variance kept 0.05

check C2: closed-form GD vs loop, max rel diff 4.0e-14
pool: unstable share 0.071; min full-budget val error 0.138; Spearman with r=81: r=1: -0.11 r=3: -0.07 r=9: 0.34 r=27: 0.75
Hyperband budget: 1902 units from scratch (23.48 R), 1581 resumed (19.52 R)
bracket sizes n_0: [81, 34, 15, 8, 5] (Table 1 of the paper: 81, 27, 9, 6, 5)

Study 2 over 200 repetitions (return rule: any = smallest loss at any rung, R = best trained to R)
method                      budget(R) scratch/resumed   test error   percentile in pool
bracket s=4 (any)               5.00 /  3.67        0.137         62.5
bracket s=4 (R)                 5.00 /  3.67        0.135         11.5
bracket s=3 (any)               4.48 /  3.41        0.136         55.9
bracket s=3 (R)                 4.48 /  3.41        0.143         26.5
bracket s=2 (any)               4.33 /  3.44        0.135         41.3
bracket s=2 (R)                 4.33 /  3.44        0.146         32.1
bracket s=1 (any)               4.67 /  4.00        0.136         21.6
bracket s=1 (R)                 4.67 /  4.00        0.139         17.5
bracket s=0 (any)               5.00 /  5.00        0.138         14.1
bracket s=0 (R)                 5.00 /  5.00        0.138         14.1
Hyperband (any)                23.48 / 19.52        0.136         49.1
Hyperband (R)                  23.48 / 19.52        0.132          2.9
random, scratch budget         23.00 / 23.00        0.136          2.6
random, resumed budget         19.00 / 19.00        0.135          3.4

The checks hold: ridge through the principal components matches a direct least-squares solve, and the closed-form iterates match an explicit gradient-descent loop over 810 steps.

In Study 1 the blur explains about three quarters of the variance of the validation error on its own, while $\lambda$ and the variance kept explain almost nothing: a low effective dimension, with the blur as the axis that matters. The important axis is not the learning-rate-like one that experience would suggest; in our reading, truncating to the leading components already regularizes, so $\lambda$ has little left to do. Random search’s median beats the cell-centred grid at every budget and is at least as good in 73% to 100% of repetitions, because a grid of $k^3$ trials tries only $k$ blur values while random search tries one per trial. Where the grid’s few blur values fall decides much of its result: the endpoint grid with two values tests only $\sigma = 0$ and $\sigma = 3$ and is the worst search at 8 trials, but beats the centred grid at 27 and 64 trials: which few values a grid happens to contain decides its result.

Study 2’s pool line shows why early elimination is risky here: the rank of a setting after 10 or 30 steps is slightly anti-correlated with its rank after 810, and only after 270 steps do the two agree well. With the return rule that keeps only fully trained settings, the brackets still rank sensibly: the most aggressive bracket ($s = 4$, successive halving from 10 steps) returns settings at about the 12th percentile, against the 14th for $s = 0$, five fully trained random settings, at the same from-scratch budget. Under Algorithm 1’s own rule the aggressive brackets return settings near the middle of the pool (the 62nd percentile for $s = 4$), because the lowest validation error seen is often that of a model stopped after a few rungs; those partially trained models are hardly worse on the test set, since early-stopped gradient descent is itself regularized (our reading; the early stopping article covers the equivalence). Hyperband with the trained-to-$R$ rule lands at the 2.9th percentile, between random search given the from-scratch budget (2.6) and the resumed budget (3.4). The hypothesis written into the plan before the first run was that “the aggressive bracket s = 4 will return worse settings than equal-budget random search, and Hyperband, hedging across brackets, will land between them”. It held under Algorithm 1’s return rule only. Under the trained-to-$R$ rule it failed: the $s = 4$ bracket returned better settings than the $s = 0$ bracket, and Hyperband did not land between them. Across all methods the mean test errors differ by less than 1.5 points: the pool contains many settings of nearly equal quality, so the choice of search strategy changes the percentile far more than the error.

Variants

• Quasi-random search. Sobol and other low-discrepancy sequences, or Latin hypercube samples, in place of independent draws. Bergstra and Bengio (§4) found Sobol slightly better than random points in simulation and Latin hypercubes no better than random.
• Coordinate and manual search. Tuning one hyperparameter at a time, as Larochelle et al. did for their deep belief networks (Bergstra and Bengio, §5); it relies on experience and is hard to reproduce.
• Multi-fidelity Bayesian optimization. Methods that model performance across budgets (training-set subsets, epochs) to extrapolate, of which Li et al. (§2.1) and Falkner et al. (§2) list several; freeze–thaw optimization keeps several runs and can pause and resume them (Goodfellow et al., §11.4.5).
• Gradient-based tuning. When the validation error is differentiable in a continuous hyperparameter, it can be followed by its gradient; Goodfellow et al. (§11.4.5) note that the gradient is usually too costly or undefined for discrete settings.
• Searching architectures. Neural architecture search treats the architecture as a structured hyperparameter; Li and Talwalkar (2019) frame it as a specialized hyperparameter optimization problem.
• Searching augmentation policies. AutoAugment and RandAugment, described in data augmentation, search or fix augmentation hyperparameters; Population Based Augmentation learns a schedule of them with PBT.

Best Practices

• Search the learning rate on a log scale before anything else, and search regularization (weight decay, dropout, augmentation strength) together with it rather than one at a time; the AdamW paper found that decoupling weight decay from Adam’s gradient step made the two more separable in search than Adam’s L2 penalty.
• Prefer random search to grids beyond two or three hyperparameters, and use its independence: add trials until the best result stops changing.
• Use early stopping of trials when partial training predicts the final ranking; check this on a few full runs first, since a learning-rate decay or a slow-starting configuration can reverse early rankings, as in the example.
• Re-centre and narrow the space after a first search, and widen it if the best trials sit on an edge.
• Tune every compared method with the same budget and protocol, and report the space, the budget, the number of trials and the selection rule.
• Report several seeds of the chosen setting, not the best seed, and keep the test set for one final evaluation.

Pitfalls

• Tuning on the test set. Any choice informed by test scores, including the number of epochs, makes them optimistic; see Training, Validation, and Test Sets.
• Reporting the winner’s validation score. The best of many noisy validation scores is biased upward; nested cross-validation estimates the tuned procedure.
• Ranges that exclude the optimum, or linear scales for multiplicative hyperparameters, which spend most trials on the largest values.
• Grids on the wrong axes. A grid fixed in advance tests few values of the hyperparameter that turns out to matter; which one that is varies with the dataset (Bergstra and Bengio, §3).
• Trusting short runs blindly. Successive halving discards configurations that start slowly; Li et al. (§6) list differing convergence rates as an open problem.
• Unequal tuning budgets between a proposed method and its baselines, which turns a comparison of methods into a comparison of effort.
• Ignoring seed noise. Bouthillier et al. (2021), as the statistical significance article reports, found that the choice of hyperparameters by random search, a perturbed grid or Bayesian optimization contributed on average about as much variance to benchmark results as weight initialization; a difference smaller than that is not evidence that one setting is better.

Evidence and Impact

Bergstra and Bengio’s comparison made random search the standard baseline, and both the Deep Learning book (§11.4.4) and later configuration-evaluation methods build on it: Hyperband is explicitly a way of speeding up random search (Li et al., §1), and BOHB a way of guiding Hyperband. Speed-ups have been reported in budget rather than in accuracy. Li et al. report Hyperband 5 to 30 times faster than Bayesian optimization methods in their deep-learning and kernel experiments (§1), and over 20 times faster than random search on their three image datasets within the evaluation window (§4.1). Falkner et al.’s BOHB reached a 2.78% CIFAR-10 test error with 33 GPU days, better than the 3.65% and 3.41% of two architecture searches and short of the 2.4% and 2.1% of two arXiv results that used 60 to 95 times more compute (§5.5); they caution that the numbers come from different optimization and regularization set-ups and are not directly comparable.

The cost of tuning also shapes what is known about methods. Bergstra and Bengio found that on the four datasets with the lowest effective dimension two random trials often did reasonably well, while the three with the highest (convex, MNIST rotated and rectangles) consistently needed at least eight (§3). Re-evaluations under equal tuning have repeatedly narrowed reported gaps: in a large comparison of GANs, Lucic et al. (2018) found that most models could reach similar scores with enough hyperparameter optimization and random restarts, as the mode collapse article reports; Musgrave et al. (2020), as the metric learning article reports, tuned 13 losses with 50 iterations of Bayesian optimization each and found the 2006 contrastive loss within about three points of the best. Hyperband’s own experiments on 117 datasets showed nearly all searchers within half a point of random search’s test error (Li et al., §4.2.1, Figure 6). The search strategy matters most for the budget; the protocol (equal budgets, validation hygiene, several seeds) matters most for what the results mean.