Benchmark Overfitting

Benchmark overfitting, also called adaptive overfitting or test-set overuse, is the inflation of benchmark scores that occurs when a research community reuses one fixed test set to make its choices. Each paper that keeps an idea because it raised the ImageNet or CIFAR-10 score, tunes a hyperparameter against the leaderboard, or abandons an architecture that scored lower, makes the next generation of models depend a little on the test images. Summed over thousands of submissions and a decade, that dependence could in principle produce models that are good at the test set rather than at the task. The question has a theory, adaptive data analysis, which says how fast scores can become optimistic and how to slow it, and an empirical record, a series of replication studies that built fresh test sets for CIFAR-10, ImageNet and MNIST and compared public and private leaderboards on Kaggle. Their answer was unexpected: accuracies did drop on new data, but mostly for reasons other than test-set reuse. The training, validation and test sets article covers the split itself, leakage within one study and benchmarks with hidden test labels; this article covers reuse across a community.

Definition

A test-set score is an unbiased estimate of a model’s error on new data only if the model was chosen without looking at the test set. Adaptive overfitting is the bias that appears when that condition fails because the model, or the sequence of models that led to it, was selected using feedback from the same test set. It is called adaptive because each choice depends on the results of earlier evaluations: the test set is not used once for a fixed list of models but repeatedly, with each evaluation steering what is tried next.

Three things are easily confused with it:

• Single-study leakage: a training set that contains near-duplicates of test images, or a hyperparameter search scored on the test set within one paper. This is a failure of one experiment’s split and is covered in the training, validation and test sets article. Benchmark overfitting can happen even when every individual paper follows the rules, because the community as a whole selects among the published results.
• Distribution shift: a model scoring lower on new images because they differ from the test images in a way the model is sensitive to. A fresh test set collected later, by other annotators or with a different filtering threshold, measures shift and adaptivity together; separating the two is the central difficulty of the replication studies below.
• A benchmark that no longer measures what matters: a test set whose labels are wrong or ambiguous for a growing share of images, or whose remaining errors are no longer the ones that matter in use. Scores can then stop tracking real progress even without any adaptive overfitting. This is sometimes called benchmark saturation.

Intuition

Imagine a classroom that takes the same practice exam every week and is told only its total score. Nobody sees the answer key, yet by trying different answers and keeping whichever changes raise the score, the class can push its mark upward without learning anything. Each score is a few bits of information about the key, and enough scores reveal it. A fresh exam on the same material exposes the gap.

Two things limit how far this goes. First, the class may be learning the material at the same time, so most of the improvement is real. Second, if every student copies the same few strategies, the class explores only a few answer patterns and learns little about the key. Both turn out to matter for vision benchmarks.

Formal Definition

Let images and labels $(\mathbf{x}, y)$ be drawn from a distribution $\mathcal{D}$, and let $L_{\mathcal{D}}(f)$ be the population error of a classifier $f$ and $L_S(f)$ its error on a test set $S$ of $n$ images. For a fixed list of $k$ classifiers chosen without reference to $S$, Hoeffding’s inequality and a union bound give, for errors in $[0, 1]$,

$$\Pr\Big[\max_{1 \le t \le k} \big|L_S(f_t) - L_{\mathcal{D}}(f_t)\big| > \varepsilon\Big] \le 2k\, e^{-2\varepsilon^2 n},$$

so the worst-case error of the estimates grows only like $\sqrt{\log k / n}$: a test set of $n$ images supports exponentially many non-adaptive evaluations (Hardt and Blum, 2015, eq. 2; the authors are printed in this order on the paper and listed as Blum and Hardt in the proceedings metadata). The argument fails when $f_t$ is chosen after seeing $L_S(f_1), \dots, L_S(f_{t-1})$, because $f_t$ then depends on $S$.

How fast reuse can go wrong

Hardt and Blum (2015) give a simple worst case, the boosting attack. Submit $k$ random label vectors for a binary test set, keep those whose reported error is at most $1/2$, and output their coordinate-wise majority vote. Each kept vector is slightly correlated with the hidden labels by chance, and the majority vote adds these correlations up. Its reported error is $\tfrac{1}{2} - \Omega(\sqrt{k/n})$ while its true error stays at $\tfrac{1}{2}$, so a leaderboard that reports exact scores can be fooled by an amount that grows like $\sqrt{k}$ instead of $\sqrt{\log k}$. In their experiment with 4,000 public labels, the estimate of an exact-score leaderboard became more and more biased as the attack submitted more vectors, while the attacking submission’s error on the 8,000 held-back labels stayed near $\tfrac{1}{2}$; under the Ladder the bias stayed small.

Mechanisms that slow it down

Dwork et al. (2015) treat the test set as a database queried by an analyst and limit what each answer reveals about it. Their reusable holdout algorithm, Thresholdout, answers a query with the training-set value unless the training and holdout values differ by more than a noisy threshold, in which case it returns the holdout value plus noise. The algorithm is differentially private, and its guarantee follows from that: the analyst learns so little about any individual holdout example that the holdout estimates stay accurate for a number of adaptive queries exponential in $n$, as long as the number of times the analyst actually overfits the training set stays small (Science paper, Supplemental Material §2).

Their synthetic experiment, which the Science paper presents in its figures (the numbers here are from the NIPS 2015 version), is a warning about ordinary practice. An analyst with 10,000 training and 10,000 holdout examples of 10,000 independent Gaussian features and random labels took the features most correlated with the label on the training set, kept those whose correlation agreed in sign and was large on both sets, then built a linear classifier from them. No classifier can beat 50% on such data, yet with 500 selected features both the training and the holdout accuracy exceeded 63%, averaged over 100 runs; only a fresh data set revealed the truth. With Thresholdout in place of the plain holdout, the reported accuracy stayed valid.

Hardt and Blum’s Ladder applies the same idea to competition leaderboards. It reports a new score only when a submission beats the best previous one by a margin, and otherwise repeats the old score:

$$R_t = \begin{cases} [L_S(f_t)]_\eta & \text{if } L_S(f_t) < R_{t-1} - \eta, \\ R_{t-1} & \text{otherwise,} \end{cases}$$

where $[\cdot]_\eta$ rounds to a multiple of $\eta$. Because the sequence of released scores can be described in few bits, a union bound over all possible transcripts applies, and the error of the best score shown is $O\big((\log(kn)/n)^{1/3}\big)$ for any adaptive submitter, against the $\Omega(\sqrt{k/n})$ of exact reporting; they also prove that no estimator can do better than $\Omega(\sqrt{\log k / n})$. Their parameter-free version sets the margin with a paired test: a submission counts as better if its error is lower than the previous best’s by more than the standard deviation of the per-example loss differences divided by $\sqrt{n}$. They describe this test as heuristic, roughly a one-sided test at the 0.15 level, since adaptivity voids its assumptions.

Decomposing a replication gap

Recht et al. (2019) write the difference between a model’s error on the original test set $S$ and on a new test set $S'$, drawn from a distribution $\mathcal{D}'$ that imitates $\mathcal{D}$, as

$$L_S - L_{S'} = \underbrace{(L_S - L_{\mathcal{D}})}_{\text{adaptivity gap}} + \underbrace{(L_{\mathcal{D}} - L_{\mathcal{D}'})}_{\text{distribution gap}} + \underbrace{(L_{\mathcal{D}'} - L_{S'})}_{\text{generalization gap}}.$$

The generalization gap is sampling noise, small for a test set of 10,000 images. For one model, the adaptivity and distribution gaps cannot be told apart. Their key idea is to look across many models: if later models were increasingly adapted to $S$, their gains on $S$ should transfer less and less to $S'$, giving diminishing returns. If the gap comes from the distribution, it should affect models more uniformly.

Properties

The replication studies

CIFAR-10.1 and ImageNetV2. Recht et al. repeated the original data collection for both benchmarks: new CIFAR-10 images from the Tiny Images source labelled by two of the authors following the original instructions, and new ImageNet images from Flickr labelled on Mechanical Turk with the original task design, 2,000 and 10,000 images respectively. For ImageNet they replicated the validation set, because most papers develop and report on it, and treated “validation” and “test” as the same set. The ICML abstract reports accuracy drops of 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet. The tables differ slightly for ImageNet: the proceedings’ Table 1 lists a top-1 drop of 10.3 points for one model, and the full table of 67 models in the proceedings’ supplementary material (Table 14) shows drops of 10.3 to 13.6 points for the convolutional networks and 6.2 to 11.0 for the Fisher-vector baselines. (The 2018 preprint on CIFAR-10 alone, which evaluated 30 models, reported 4% to 10%.) Yet the models’ order was almost unchanged, and new accuracy was close to a linear function of original accuracy with slope above 1: $1.69\,a - 72.7\%$ on CIFAR-10 and $1.11\,a - 20.2\%$ for ImageNet top-1, where $a$ is the original accuracy, fitted on a linear accuracy scale to all CIFAR-10 models and to the ImageNet convolutional networks without the Fisher-vector baselines (Appendix D.2 of the proceedings’ supplement finds an even better fit on a probit scale). Each point gained on the old test set was worth more than a point on the new one, the opposite of diminishing returns, so the authors concluded that adaptivity was an unlikely explanation and attributed the drop to a distribution gap: their new images were slightly harder. In support, three ImageNet test sets drawn from the same candidate pool by different sampling rules based on the Mechanical Turk selection frequency (the fraction of workers who confirmed an image) changed average top-1 accuracy by $-11.8$, $-3.2$ and $+2.1$ points relative to the original.

The authors noted what their test cannot rule out: an adaptivity gap that lowers every model’s accuracy by a roughly constant amount would leave the linear trend intact. Detecting it would need a truly i.i.d. second test set rather than a new collection.

QMNIST. MNIST provided exactly that. Its test set had been subsampled to 10,000 images from 60,000 prepared by the original procedure, and the other 50,000 were never distributed. Yadav and Bottou (2019) reconstructed the MNIST preprocessing from the NIST source, reconstructed the full 60,000-digit test set, and evaluated nearest-neighbour classifiers, SVMs, multilayer perceptrons and convolutional networks on the 50,000 unseen ones. Error rates on the new digits were consistently slightly higher, mostly within the confidence intervals, and the ranking of models on the old test set predicted the ranking on the new one almost perfectly; the hyperparameters that were best on MNIST were best on the new digits for the nearest-neighbour classifiers and SVMs. They concluded that “testing set rot” exists but is far less severe than feared, and that a shifting distribution is the greater danger.

Kaggle. Roelofs et al. (2019) used the public/private split of Kaggle competitions as a natural experiment. Competitors see scores on a public part of the test set throughout and the private part only at the end, so the public-minus-private difference estimates the adaptivity gap without any distribution shift, since the two parts are normally drawn at random from one pool. Across 120 classification competitions with at least 1,000 submissions each, public and private scores agreed closely. Among the top 10% of submissions some competitions showed deviations of about one percentage point, and the clear outliers had non-i.i.d. splits or very small test sets. From the accuracy competitions they suggest at least 10,000 test examples as a reasonable minimum to protect against adaptive overfitting.

Statistical bias in replication. Engstrom et al. (2020) re-examined the ImageNetV2 gap. Matching selection frequencies between the new and original sets relied on estimates from a limited number of workers per image (at least ten, in Recht et al.’s design). In their analysis, because the Flickr candidate pool is likely to have lower selection frequencies than the ImageNet images, selecting candidates by a noisy estimate systematically picks images whose true selection frequency is lower than estimated, and those images are harder. After re-measuring selection frequencies and correcting for this bias, they estimate that only $3.6\% \pm 1.5\%$ of the original $11.7\% \pm 1.0\%$ accuracy drop remains unaccounted for.

Why reuse did so little damage

Recht et al. offer two explanations without choosing between them. The research community may follow something like the Ladder implicitly, keeping only clear improvements. And the set of models actually evaluated may be small in the relevant sense: SGD-trained networks on a fixed training set that fit their training data well. Mania et al. (2019) found that ImageNet models agree in their predictions far more than their accuracies alone imply, and proved a generalization bound that improves with this similarity: similar models extract little new information from the test set. Yadav and Bottou point to pairing: comparisons between models on the same test images are much more precise than the absolute error rates, so model selection stays reliable long after the absolute numbers become optimistic.

When the benchmark stops measuring the task

The replication studies addressed adaptivity; other work asks whether the labels themselves still support fine distinctions. Beyer et al. (2020) re-annotated the ImageNet validation set with multiple labels per image (ReaL) and found that recent models’ gains were substantially smaller under the new labels; recent high-accuracy models now predicted the new labels better than the original labels did, which they read as a sign that the original labels’ usefulness for evaluation may be nearing an end. They attribute part of the progress on the original labels to models fitting the idiosyncrasies of the original labelling pipeline. Northcutt et al. (2021) estimate an average of at least 3.3% label errors across ten widely used test sets, and at least 6% in the ImageNet validation set; with corrected labels, ResNet-18 overtakes ResNet-50 if the share of originally mislabelled test images rises by 6%. Shankar et al. (2020) had five trained human labellers classify images from ImageNet and ImageNetV2: the humans scored the same on both sets while every model dropped, and on 590 object classes the humans remained 4 and 11 points above the best model on the two sets. These are not adaptive overfitting, but they limit what a benchmark score can say near the top of the leaderboard (our framing).

Examples

The first script simulates a leaderboard for a two-class problem on synthetic 16×16 grayscale images: a faint horizontal or vertical bar under Gaussian noise, with floating-point intensities (background 0, bar 0.25, noise standard deviation 1). The classifier is a linear template, predicting a vertical bar when $\langle \mathbf{w}, \mathbf{x} - \mathbf{m} \rangle > 0$, with $\mathbf{m}$ the midpoint of the two class means. Because the noise is Gaussian and isotropic, the matched template $\mathbf{w}_0 = T_1 - T_0$ is the Bayes-optimal classifier, with accuracy $\Phi(1) \approx 0.841$; no classifier can do better, so whatever public score the analyst gains by moving away from $\mathbf{w}_0$ is overfitting. An analyst starts from $\mathbf{w}_0$, adds a random perturbation to the current best template for each query, submits it to a leaderboard that scores 500 public test images, and keeps the perturbation whenever the leaderboard shows an improvement. The leaderboard either releases the exact public accuracy or follows the parameter-free Ladder. Each chosen template is also scored on 20,000 fresh images. The script runs 20 seeds of 4,000 queries in about 17 seconds with NumPy 2.3.5.

import numpy as np

S = 16                                        # image size (pixels)
T0 = np.zeros((S, S)); T0[6:10, 2:14] = 1     # class 0: horizontal bar
T1 = np.zeros((S, S)); T1[2:14, 6:10] = 1     # class 1: vertical bar
CONTRAST, NOISE = 0.25, 1.0                   # bar intensity and Gaussian noise sd

def images(n, rng):
    """n grayscale 16x16 images: background 0, bar CONTRAST, plus noise; half of each class."""
    y = rng.permutation(np.arange(n) % 2)
    X = CONTRAST * np.where(y[:, None, None] == 1, T1, T0)
    X = X + rng.normal(scale=NOISE, size=(n, S, S))
    return X.reshape(n, -1), y

def predict(w, X):                            # linear template classifier, score s(x) = <w, x - m>
    m = CONTRAST * (T0 + T1).ravel() / 2
    return ((X - m) @ w > 0).astype(int)

def climb(Xp, yp, Xf, yf, w0, queries, rng, ladder, step=0.1, checkpoints=(10, 100, 1000, 4000)):
    """An analyst perturbs the current best template at random and keeps the
    perturbation whenever the leaderboard reports a better public score."""
    n = len(yp)
    best_w = w0
    best_loss = (predict(w0, Xp) != yp).astype(float)    # per-image 0/1 loss on the public set
    shown = 1 - best_loss.mean()                          # score the leaderboard displays
    out = []
    for q in range(1, queries + 1):
        w = best_w + step * rng.normal(size=w0.size)
        loss = (predict(w, Xp) != yp).astype(float)
        if ladder:   # parameter-free Ladder: release only a significant improvement
            better = loss.mean() < best_loss.mean() - np.std(loss - best_loss, ddof=1) / np.sqrt(n)
        else:        # standard leaderboard: release the exact score every time
            better = 1 - loss.mean() > shown
        if better:
            best_w, best_loss, shown = w, loss, 1 - loss.mean()
        if q in checkpoints:
            out.append((shown, np.mean(predict(best_w, Xf) == yf)))
    return np.array(out)

w0 = (T1 - T0).ravel()                        # the matched template: the best linear classifier
res = {False: [], True: []}
for seed in range(20):
    rng = np.random.default_rng(seed)
    Xp, yp = images(500, rng)                 # public test set, reused for every query
    Xf, yf = images(20000, rng)               # fresh images from the same distribution
    for ladder in (False, True):
        res[ladder].append(climb(Xp, yp, Xf, yf, w0, 4000, rng, ladder))

print("20 seeds; mean [min, max] of public score minus fresh accuracy, in points")
print("queries   exact scores released     Ladder")
for i, q in enumerate((10, 100, 1000, 4000)):
    cells = []
    for ladder in (False, True):
        r = np.array(res[ladder])[:, i]
        gap = 100 * (r[:, 0] - r[:, 1])
        cells.append(f"{gap.mean():5.1f} [{gap.min():4.1f}, {gap.max():4.1f}]")
    print(f"{q:7d}   {cells[0]:22s}  {cells[1]}")
r = np.array(res[False])[:, -1]; l = np.array(res[True])[:, -1]
print(f"after 4000 queries, fresh accuracy: exact {r[:, 1].mean():.3f}, Ladder {l[:, 1].mean():.3f}")

Output:

20 seeds; mean [min, max] of public score minus fresh accuracy, in points
queries   exact scores released     Ladder
     10     1.8 [-0.3,  3.9]        0.5 [-2.9,  2.8]
    100     3.7 [ 1.6,  5.8]        1.9 [-1.2,  4.4]
   1000     5.8 [ 3.2,  8.5]        3.3 [ 0.6,  5.3]
   4000     7.4 [ 5.0, 10.0]        4.1 [ 0.6,  6.1]
after 4000 queries, fresh accuracy: exact 0.806, Ladder 0.830

With exact scores, the public score pulls away from the fresh accuracy by about two points for every tenfold increase in queries, and after 4,000 queries it overstates the truth by 5 to 10 points in every seed. The leaderboard now shows about 88%, above the 84.1% that no classifier can exceed. The damage is not only to the reported number: the analyst was led away from the optimal template, and the chosen model’s fresh accuracy fell by 3.5 points on average, from 84.1% to 80.6%. The Ladder roughly halves the gap and cuts the loss of true accuracy to about a third, because perturbations that only fit a few public images rarely pass its paired test, so the analyst keeps fewer of them. It does not remove the gap here. In our reading, a test at roughly the 0.15 level, applied thousands of times without correction for multiple comparisons (a limitation Hardt and Blum point out), still lets through some lucky perturbations. The public set is deliberately small; the bounds above predict a smaller gap for a larger one, in line with the 10,000-example minimum that Roelofs et al. suggest.

The second script reproduces the signature that Recht et al. used to argue against adaptivity. It fits a line to the ImageNet and CIFAR-10 rows of their Table 1 (eight and seven key models, out of the 67 ImageNet and 34 CIFAR-10 models in their full tables), then simulates 25 classifiers of varying quality, the matched template corrupted by template noise of increasing strength, evaluated on an “original” test set and on a “replicated” one whose bars are fainter (contrast 0.40 instead of 0.50). Nothing in the simulation is adapted to either test set; the replicated images are simply harder.

import numpy as np

# (1) Printed numbers: Recht et al. (2019), Table 1, top-1 accuracy (%) of the key models.
imagenet = np.array([[82.9, 72.2], [82.5, 72.2], [78.3, 67.0], [78.0, 66.1],
                     [77.1, 65.3], [74.2, 61.9], [56.5, 44.0], [35.1, 24.1]])
cifar10 = np.array([[98.4, 95.5], [97.1, 93.0], [95.9, 89.7], [93.5, 85.2],
                    [93.0, 84.9], [88.5, 77.5], [85.6, 73.1]])
for name, t in (("ImageNet", imagenet), ("CIFAR-10", cifar10)):
    slope, offset = np.polyfit(t[:, 0], t[:, 1], 1)
    print(f"{name:9s} Table 1 rows: new = {slope:.2f} * orig {offset:+.1f}")

# (2) Simulation: the same classifiers on an original test set and on a replicated one
#     whose images are slightly harder (bar contrast 0.50 -> 0.40), nothing adapted to either.
S = 16
T0 = np.zeros((S, S)); T0[6:10, 2:14] = 1
T1 = np.zeros((S, S)); T1[2:14, 6:10] = 1

def images(n, contrast, rng):
    y = rng.permutation(np.arange(n) % 2)
    X = contrast * np.where(y[:, None, None] == 1, T1, T0) + rng.normal(size=(n, S, S))
    return X.reshape(n, -1), y

rng = np.random.default_rng(0)
Xo, yo = images(20000, 0.50, rng)             # original test set
Xn, yn = images(20000, 0.40, rng)             # replicated test set, harder images
# 25 "models": the matched template corrupted by template noise of growing strength
models = [(T1 - T0).ravel() + a * rng.normal(size=S * S) for a in np.linspace(0, 1.5, 25)]
acc = np.array([[np.mean(((X - c * (T0 + T1).ravel() / 2) @ w > 0) == y)
                 for X, y, c in ((Xo, yo, 0.50), (Xn, yn, 0.40))] for w in models]) * 100
slope, offset = np.polyfit(acc[:, 0], acc[:, 1], 1)
hi = acc[:, 0] > 90
slope_hi = np.polyfit(acc[hi, 0], acc[hi, 1], 1)[0]
rank_o, rank_n = np.argsort(np.argsort(-acc[:, 0])), np.argsort(np.argsort(-acc[:, 1]))
print(f"simulation: original {acc[:, 0].min():.1f}-{acc[:, 0].max():.1f}%, "
      f"drop {np.min(acc[:, 0] - acc[:, 1]):.1f}-{np.max(acc[:, 0] - acc[:, 1]):.1f} points")
print(f"  fit over all models: new = {slope:.2f} * orig {offset:+.1f}; "
      f"slope above 90%: {slope_hi:.2f}")
print(f"  largest rank change: {np.max(np.abs(rank_o - rank_n))} places of 25")

Output:

ImageNet  Table 1 rows: new = 1.01 * orig -12.0
CIFAR-10  Table 1 rows: new = 1.73 * orig -75.9
simulation: original 72.2-97.7%, drop 3.1-5.9 points
  fit over all models: new = 1.03 * orig -7.2; slope above 90%: 1.26
  largest rank change: 1 places of 25

The Table 1 subset gives a CIFAR-10 slope close to the published 1.69 and an ImageNet slope of 1.01 rather than 1.11, a reminder that a fit to a handful of models is fragile. The ImageNet subset also includes a Fisher-vector model, which the published fit leaves out; that fit covers the 64 convolutional networks of Table 14 in the proceedings’ supplement, the CIFAR-10 fit all 34 models. The simulation shows that a purely distributional change produces the same pattern as the real data: a drop for every model, a ranking that moves by at most one place, and a slope above 1 among the most accurate models. The reason is that, for this Gaussian model, a harder test set scales each classifier’s margin by a constant factor ($0.8$ here), so accuracies lie on a straight line on a probit scale, $\Phi^{-1}(a_{\text{new}}) = 0.8\,\Phi^{-1}(a_{\text{orig}})$. Converting back to accuracy stretches the top of the scale: the slope in accuracy units at probit value $z$ is $0.8\,\varphi(0.8z)/\varphi(z)$, with $\varphi$ the standard normal density, which exceeds 1 once $z$ passes about 1.11, an original accuracy of about 87% (our derivation for this model; Recht et al. give their own model, in which Gaussian-distributed image difficulty also makes accuracies linear on a probit scale, in Appendix B of the ICML version and Section 5.3 of the arXiv version). The first script showed what adaptive overfitting looks like instead: reported gains with no counterpart on fresh data.

Common Misconceptions

• “A drop on a new test set proves the community overfit the old one.” The drop on ImageNetV2 is mostly explained by the new images being harder, and by statistical bias in how they were selected; adaptive overfitting predicts diminishing returns, which the replication studies did not find.
• “A benchmark used for a decade is worthless.” On CIFAR-10, ImageNet, MNIST and Kaggle, the ranking of models survived years of reuse. Absolute numbers are optimistic for a different reason: they are measured on one distribution and do not transfer to others.
• “Hidden test labels prevent benchmark overfitting.” They prevent training on the test set, but a server that returns an exact score for every submission still leaks information; the boosting attack needs only the scores. Rate limits help by bounding $k$, and the Ladder by releasing fewer bits per submission.
• “Robustness benchmarks measure overfitting.” ImageNet-C or ObjectNet measure performance under a deliberate distribution shift, which is a different question from whether the original test score is biased.

Where It Is Used

The concern shapes how benchmarks are run. Many vision benchmarks withhold test labels and limit submissions to an evaluation server, as the training, validation and test sets article describes for ImageNet, COCO, Cityscapes and KITTI; Kaggle splits its test set into a public part for the live leaderboard and a private part for the final ranking, and Hardt and Blum note that it also limits the submission rate and the precision of released scores. Within a single project, the same logic argues for a validation set, or cross-validation, for every choice, and for reporting the test set once.

When a benchmark ages, the responses are new test sets collected by the original procedure (CIFAR-10.1, ImageNetV2, QMNIST), cleaned or re-annotated labels (ReaL, the corrected test sets of Northcutt et al.), and evaluation on several benchmarks at once. Robustness benchmarks test the same classes under controlled shifts instead: ImageNet-C applies 15 algorithmic corruption types from noise, blur, weather and digital categories at five severity levels to ImageNet validation images (Hendrycks and Dietterich, 2019); ImageNet-A keeps real photographs selected by adversarial filtration against existing models, on which a DenseNet-121 reaches about 2% accuracy (Hendrycks et al., 2021); and ObjectNet’s 50,000 images, crowdsourced with randomized backgrounds, rotations and viewpoints and released without a training set, produce a 40–45% drop in performance for the ImageNet-trained models the authors call object detectors, relative to other benchmarks (Barbu et al., 2019). The ImageNet article lists further test sets built on its classes. Small differences between models on any single benchmark still need the confidence intervals and paired tests that statistical significance testing in evaluation provides, as the accuracy article introduces.