Do ImageNet Classifiers Generalize to ImageNet?

“Do ImageNet Classifiers Generalize to ImageNet?” is a paper by Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt and Vaishaal Shankar of the University of California, Berkeley, published at the International Conference on Machine Learning (ICML) 2019 in Long Beach, California; a footnote says the authors are listed alphabetically. It built new test sets for CIFAR-10 and ImageNet by repeating each dataset’s original collection and labelling procedure, released them as CIFAR-10.1 and ImageNetV2, and evaluated a decade of image classification models on them. Every model lost accuracy, by roughly 3 to 15 points on CIFAR-10 and 6 to 14 points on ImageNet, yet the models kept almost the same order, and each point gained on the old test set was worth more than a point on the new one. The authors concluded that years of test-set reuse had not produced the diminishing returns that adaptive overfitting predicts, and that the drop came instead from the new images being slightly harder in ways that a careful replication had not controlled. The theory of test-set reuse and the other replication studies are covered in benchmark overfitting.

Problem

By 2018, CIFAR-10 and ImageNet had been the main image classification benchmarks for almost a decade, and progress on them was measured on test sets that many papers had used to choose architectures and hyperparameters, the kind of reuse that can make reported scores optimistic (see benchmark overfitting and training, validation and test sets). Nobody had measured whether this had happened to vision benchmarks. Doing so needs fresh test images from the original distribution, and neither dataset had a hidden second test set drawn at the same time as the first.

For ImageNet there was a further practical point. The ILSVRC 2012 test labels were never released, and most papers report accuracy on the 50,000-image validation set, which had no submission limits. The authors therefore replicated the validation set and use “validation set” and “test set” interchangeably for ImageNet throughout the paper; following this site’s convention, “original test set” below means the ImageNet validation set.

Contribution

• A replication method. Instead of borrowing images from another dataset, which would bring that dataset’s own biases, the authors repeated each benchmark’s documented collection process from the same image source, with the same class structure, annotation format and cleaning steps.
• Two new test sets. CIFAR-10.1 has about 2,000 images drawn from Tiny Images; ImageNetV2 has three test sets of 10,000 Flickr images each, ten per class, built from one candidate pool with three different sampling rules.
• A large evaluation. The full tables list 34 CIFAR-10 models and 67 ImageNet models, from random-feature and Fisher-vector baselines to the best networks of 2018, each scored on the original and new test sets with 95% Clopper–Pearson intervals.
• A test for adaptivity. The paper splits the gap between old and new accuracy into an adaptivity gap, a distribution gap and sampling error, and argues that the two systematic terms can be told apart by looking across many models rather than at one (the decomposition is given in the benchmark overfitting article).
• A measured sensitivity to labelling. By varying only how images were sampled from the annotated pool, the paper showed that the annotation step alone can move average accuracy by about 14 points.

Method

CIFAR-10.1

Every CIFAR-10 image has an exact duplicate in Tiny Images, so the authors could recover the Tiny Images keyword behind each one and match the new test set’s keyword distribution to the original’s, using the 25 most common keywords per class (more than 95% of CIFAR-10). Two of the authors re-enacted the original roles: one, like CIFAR-10’s paid student labellers, selected candidate images following the original labelling instructions; the other, like the original researchers, reviewed all of about 9,000 candidates and removed unclear ones. After sampling to match the keyword distribution, near-duplicates within the new set and of any CIFAR-10 training or test image were removed by manually checking the ten nearest neighbours of each new image. No classifier was run on the new data during collection. The ICML text gives the size as 2,000 images. The set behind the main results was not exactly class-balanced (8% ships instead of 10%), and the supplement adds an exactly balanced variant of 2,000 images, 200 per class (Table 12), on which the best model’s drop is unchanged. The repository’s README, written for the 2018 CIFAR-10 preprint, calls these versions v4 (2,021 images, used for the main results) and v6 (2,000 images, recommended for later use).

ImageNetV2: building the candidate pool

Flickr queries. The original validation set comes almost entirely from Flickr, so the new images did too. Each class was searched using each of its WordNet synonyms as a separate query. The authors chose a date window just after ImageNet’s: images taken and uploaded between 11 July 2012 and 11 July 2013, starting the day after the final ILSVRC 2012 data release, because the original window’s best images might already have been used and near-duplicates of ImageNet images would be hard to rule out. Results were ordered by upload date rather than by Flickr’s “relevance” or “interestingness”, which might rely on models trained on ImageNet. Classes short of images were expanded in steps, much as ImageNet’s queries had been: appending a word from the parent class, raising the per-class limit from 100 to 200 images, extending the window to July 2014, joining compound queries, and finally choosing query words by hand, including translations. The pool reached 208,145 candidates.

Mechanical Turk labelling. The task copied ImageNet’s design: a grid of 48 candidate images for one target class, a description derived from the original instructions with the class definition and a link to Wikipedia, and the instruction to select images of the class regardless of occlusion or clutter while excluding drawings and paintings. Each task contained at least six images from the original validation set, three from the target class and three from a class nearby in WordNet, at random positions, with obfuscated URLs and matched image sizes so that workers could not tell them apart. At least ten workers saw each task. The main output was each image’s selection frequency, the fraction of workers who selected it for its target class, a term Engstrom et al. (2020) credit this paper with coining. Because original validation images were embedded in the same tasks, the same statistic could be measured for at least 20 original images per class.

Near-duplicate removal. For each candidate the authors computed 30 nearest neighbours under three measures: Euclidean distance on raw pixels, Euclidean distance on VGG fc7 features, and SSIM, the last only against the candidates and the five WordNet-closest ImageNet classes because it is expensive. Pairs below per-metric thresholds were reviewed by hand. They note that, as far as they know, ImageNet itself used only $\ell_2$ nearest neighbours.

ImageNetV2: three sampling rules

The ImageNet publications do not give the per-class agreement thresholds that decided which images entered the dataset, and the 2010–2012 worker population cannot be recruited again, so the authors did not try to reproduce the original threshold. They sampled ten images per class from the annotated pool in three ways:

• MatchedFrequency divided each class’s annotated original validation images into five selection-frequency bins, $[0, 0.2)$, $[0.2, 0.4)$, $[0.4, 0.6)$, $[0.6, 0.8)$ and $[0.8, 1.0]$, and sampled candidates bin by bin to match that class’s histogram. When a bin ran out, the next higher bin was used, which affected 0.8% of the images and, as the authors note, makes the set easier.
• Threshold0.7 sampled ten images per class with selection frequency at least 0.7.
• TopImages took the ten images per class with the highest selection frequency.

MatchedFrequency is the main test set because its average selection frequency, 0.73, is closest to the 0.71 measured for the original validation images in the same tasks.

Final review

The authors then reviewed every image in each test set, side by side with ten random original validation images of the class, removed incorrect images and near-duplicates, and resampled from the pool, sometimes with new targeted Flickr searches. Their blacklisting criteria were images not representative of the class; cartoons, paintings, drawings or renderings; images clearly different in distribution from the validation set; and unclear, blurry, heavily occluded or edited images or ones with a small target object. For example, about half of one revision’s “great white shark” images showed museum models or statues, so some were replaced with live sharks. MatchedFrequency converged after 33 iterations. Every revision was kept under version control, and model accuracies on intermediate revisions were computed only after the final one was fixed, so that no classifier influenced the data.

Evaluation

For the deep networks the authors used published code and pretrained weights where available and otherwise ran the repositories’ training commands. As pre-deep-learning baselines they wrote their own random-features model for CIFAR-10 and Fisher-vector models for ImageNet; the paper warns that the Fisher-vector models reach top-5 accuracy 17 points below the published ILSVRC 2012 result and are a baseline inspired by that approach, not a reproduction. Accuracy is top-1 unless stated, reported with 95% Clopper–Pearson intervals; on 10,000 images such an interval is at most $\pm 1$ point, and $\pm 2$ points at 99.99% confidence, so sampling error alone could not produce drops of 10 points (see statistical significance in evaluation and accuracy for the intervals).

Results

Accuracy drops

The ICML abstract reports drops of 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet. The full tables in the supplementary material give these ranges in percentage points of top-1 accuracy:

Test set	Models	Original accuracy	Drop	Source
CIFAR-10.1	34	83.3–98.4	2.9–15.4	Table 11
ImageNetV2 MatchedFrequency, convolutional networks	64	56.5–82.9	10.3–13.6	Table 14
ImageNetV2 MatchedFrequency, Fisher vectors	3	21.2–35.1	6.2–11.0	Table 14

The CIFAR-10 range matches the abstract. For ImageNet the table is wider than the abstract’s 11 to 14: ten of the 64 networks, including all four NASNet and PNASNet models at the top of the table, lose less than 11 points (10.3 to 10.9), and the Fisher-vector baselines lose less still. Table 1 of the main text shows the pattern on key models. The best CIFAR-10 model, PyramidNet trained with AutoAugment, fell from 98.4% to 95.5%; a 110-layer ResNet from 93.5% to 85.2%. On ImageNet the best model, PNASNet-5-Large, fell from 82.9% to 72.2% top-1 and, by the text and Table 15, from 96.2% to 90.1% top-5. The authors put the top-5 loss in context: the best ILSVRC entries reached about 89% top-5 in 2013 and 93% in 2014, so a 6-point drop is roughly five years of progress.

Ranking and the linear fit

Models kept their order: in Table 14, no model’s top-1 rank among the 67 changed by more than five places (our count), and models with similar original accuracy lost similar amounts. New accuracy was close to a linear function of original accuracy $a$ (both in percent):

$$a_{\text{new}} \approx 1.69\,a - 72.7 \;\;\text{(CIFAR-10)}, \qquad a_{\text{new}} \approx 1.11\,a - 20.2 \;\;\text{(ImageNet top-1)}.$$

From 100,000 bootstrap samples the 95% intervals are $[1.63, 1.76]$ for the CIFAR-10 slope and $[1.07, 1.19]$ for the ImageNet slope. The ImageNet fit covers the convolutional networks only: the supplement says the Fisher-vector models are left out of the main-text plots, and a least-squares fit to the 64 convolutional networks of Table 14 reproduces the printed slope, and the offset to within 0.1 point (our check). Since both slopes exceed 1, more accurate models lost less, and the authors concluded that neither benchmark shows diminishing returns. The supplement adds that across the full range from 21% to 83%, including the Fisher vectors, a straight line fits better after a probit transform, $\Phi^{-1}(a_{\text{new}}) = u\,\Phi^{-1}(a_{\text{orig}}) + v$, with $\Phi$ the standard normal CDF (Appendix D.2 and Figure 6).

Sampling strategy

Table 2 is the paper’s central evidence about where the drop comes from. All three test sets come from the same pool, are reviewed for correct labels and have average selection frequency above 0.7, yet:

Sampling rule	Average selection frequency	Average top-1 change	Average top-5 change
MatchedFrequency	0.73	−11.8	−8.2
Threshold0.7	0.85	−3.2	−1.2
TopImages	0.93	+2.1	+1.8

These averages match the mean over all 67 models of Tables 14 to 19, Fisher vectors included (our check). The spread between TopImages and MatchedFrequency, about 14 points top-1 and 10 top-5, is by the paper’s comparison larger than the spread among all convolutional networks after AlexNet (and the SqueezeNets designed to match it), roughly 14 points top-1 and 8 points top-5. The paper reports that all three sets give a good linear fit but prints slopes only for MatchedFrequency. Fitting the convolutional networks of Tables 16 and 18 ourselves gives slopes of about 1.10 for Threshold0.7 and 0.96 for TopImages, so the slope above 1 is a property of the harder sets, not of every sampling rule. Our reading is that this is expected: on a set easier than the original, the most accurate models approach the ceiling and gain less per point.

The supplement’s Figures 15 to 17 split accuracy by selection-frequency bin. Images in the $[0.4, 0.6)$ bin are about 20 points harder than MatchedFrequency as a whole on the new set and about 10 to 15 points harder than the whole validation set on the original images. When both sets are binned, new images are still harder than original images in the same bin, by about 20 points in the middle bins and 5 to 10 points in the lowest and highest.

Hypotheses that did not explain the drop

The supplement reports follow-up experiments, none of which closed the gap:

• Near-duplicates. About 800 images (8%) of the original CIFAR-10 test set have near-duplicates in its training set by the authors’ stricter criterion, but by their calculation, even at 100% accuracy on them, they account for only about 0.6 points of a 93% model’s accuracy, at most 1 point of the drop.
• Re-tuning. A grid over learning rate, dropout and weight decay for a VGG-style CIFAR-10 model raised new-set accuracy at most from 85.3% to 85.8%.
• Training on new data. Adding half of CIFAR-10.1 to the training set gave 85.1% and 85.4% on the other half in two runs, against 84.9% without it.
• Cross-validation. Three CIFAR-10 models retrained on five reshuffled splits of the original data stayed within about one point of their original test accuracy (Table 4). For ImageNet, a ResNet-50 trained with each of five 50,000-image folds held out from the training set scored 92.55–92.62% top-5 on the held-out folds, against 92.5% on the validation set and 84.7% on MatchedFrequency (Table 13). The cross-validation article discusses why held-out folds of the same data cannot predict such a drop.
• Human accuracy. Nine graduate students labelled the hardest 5% of each CIFAR-10 test set (500 original and 115 new images, ranked by how many models got them wrong); on average they did not do worse on the new images, which the authors call preliminary.
• A discriminator. ResNets trained to tell original from new CIFAR-10 test images reached at most 53.1%.
• Final review. Reviewing raised a ResNet’s accuracy on MatchedFrequency by about 4 points (the text says ResNet-50, the caption of Figure 7 ResNet-152), and the first, unreviewed revision already showed the drop and the linear fit, so the review did not create them.

Worked Example: Matching on a Noisy Selection Frequency

The paper’s MatchedFrequency rule matches a histogram of observed selection frequencies, each estimated from about ten votes. The script below simulates the three sampling rules on synthetic images so that every image’s true selection frequency is known. Images are 16×16 grayscale with floating-point intensities: background 0, a horizontal (class 0) or vertical (class 1) bar of contrast $c \sim U(0, 0.6)$, and Gaussian noise of standard deviation 1.5. An annotator confirms an image’s label with probability $\mathrm{SF}(c) = \Phi\big((c - 0.3)/0.2\big)$, its true selection frequency, so faint bars are both harder for annotators and harder for the classifier, a fixed matched template that predicts a vertical bar when $\langle T_1 - T_0, x \rangle > 0$. The “original” test set keeps raw web images that most of ten annotators confirmed. The replication draws a new candidate pool from the same web distribution, has $k$ annotators label it together with 5,000 embedded original images, and samples 5,000 new test images by one of the paper’s three rules. The script runs ten seeds (0 to 9) per row in about 20 seconds (NumPy 2.3.5, SciPy 1.16.3).

import numpy as np
from scipy.stats import norm

S = 16                                        # image size (pixels)
T0 = np.zeros((S, S)); T0[6:10, 2:14] = 1     # class 0: horizontal bar
T1 = np.zeros((S, S)); T1[2:14, 6:10] = 1     # class 1: vertical bar
W = (T1 - T0).ravel()                         # fixed classifier: predict class 1 if <W, x> > 0

def web_images(n, rng):
    """Raw 'web' images: random class, bar contrast c ~ U(0, 0.6), Gaussian noise sd 1.5."""
    y = rng.integers(0, 2, n)
    c = rng.uniform(0, 0.6, n)
    X = c[:, None] * np.where(y[:, None] == 1, T1.ravel(), T0.ravel())
    return X + 1.5 * rng.normal(size=(n, S * S)), y, c

def true_sf(c):                               # chance that one annotator confirms the label
    return norm.cdf((c - 0.3) / 0.2)

def votes(c, k, rng):                         # observed selection frequency from k annotators
    return rng.binomial(k, true_sf(c)) / k

BINS = [0, 0.2, 0.4, 0.6, 0.8, 1.0001]        # the paper's five selection-frequency bins
N_NEW = 5000

def replicate(seed, k, strategy):
    rng = np.random.default_rng(seed)
    # Original test set: web images kept when most of 10 annotators confirm them.
    X, y, c = web_images(60000, rng)
    keep = votes(c, 10, rng) > 0.5
    Xo, yo, co = X[keep][:10000], y[keep][:10000], c[keep][:10000]
    # Replication: a new candidate pool from the same web source, annotated by k annotators
    # in the same tasks as 5,000 embedded original images.
    Xc, yc, cc = web_images(80000, rng)
    s_c, s_o = votes(cc, k, rng), votes(co[:5000], k, rng)
    if strategy == "MatchedFrequency":        # match the originals' histogram over the five bins
        target = np.histogram(s_o, BINS)[0] / len(s_o)
        bin_c = np.digitize(s_c, BINS) - 1
        pick = np.concatenate([rng.choice(np.flatnonzero(bin_c == b), int(round(N_NEW * t)),
                                          replace=False) for b, t in enumerate(target)])
    elif strategy == "Threshold0.7":          # any candidates with observed SF >= 0.7
        pick = rng.choice(np.flatnonzero(s_c >= 0.7), N_NEW, replace=False)
    else:                                     # TopImages: the highest observed SF
        pick = np.argsort(-s_c, kind="stable")[:N_NEW]
    acc = lambda X, y: 100 * np.mean((X @ W > 0) == y)
    return (s_o.mean(), s_c[pick].mean(), true_sf(co).mean(), true_sf(cc[pick]).mean(),
            acc(Xo, yo), acc(Xc[pick], yc[pick]))

print("strategy          annotators  observed SF  true SF     accuracy (%)  change in points:")
print("                              orig / new   orig / new  orig / new    mean [min, max], 10 seeds")
for strategy, k in (("MatchedFrequency", 10), ("MatchedFrequency", 40), ("MatchedFrequency", 200),
                    ("Threshold0.7", 10), ("TopImages", 10)):
    r = np.array([replicate(seed, k, strategy) for seed in range(10)])
    m, d = r.mean(axis=0), r[:, 5] - r[:, 4]
    print(f"{strategy:16s}  {k:10d}  {m[0]:.2f} / {m[1]:.2f}  {m[2]:.2f} / {m[3]:.2f}  "
          f"{m[4]:.1f} / {m[5]:.1f}   {d.mean():+.1f} [{d.min():+.1f}, {d.max():+.1f}]")

Output:

strategy          annotators  observed SF  true SF     accuracy (%)  change in points:
                              orig / new   orig / new  orig / new    mean [min, max], 10 seeds
MatchedFrequency          10  0.75 / 0.75  0.75 / 0.71  87.7 / 85.7   -2.0 [-3.0, -0.8]
MatchedFrequency          40  0.75 / 0.75  0.75 / 0.73  87.7 / 87.1   -0.6 [-1.6, +0.7]
MatchedFrequency         200  0.75 / 0.75  0.75 / 0.74  87.7 / 87.4   -0.3 [-0.8, +0.5]
Threshold0.7              10  0.75 / 0.84  0.75 / 0.79  87.7 / 89.1   +1.4 [+1.0, +1.9]
TopImages                 10  0.75 / 1.00  0.75 / 0.87  87.7 / 92.0   +4.3 [+3.6, +4.7]

Two things happen, and the paper documents the first. The sampling rule alone moves accuracy: from the same candidate pool, the three rules order themselves as in Table 2, from a drop on MatchedFrequency to a gain on TopImages, because they keep progressively clearer images. The smaller effects here (about 6 points between extremes, against the paper’s 14) reflect the simulation’s choices, not ImageNet’s. The second effect illustrates, in a toy setting, the mechanism Engstrom et al. (2020) later identified; it does not reproduce their measurements. With ten annotators, MatchedFrequency reproduces the originals’ observed selection frequency exactly, yet the new images’ true selection frequency is lower and every seed shows a drop, although the source distribution, the annotators and the classifier are the same. The originals were already filtered, so an original image observed at 0.7 is more likely truly above 0.7, while an unfiltered candidate observed at 0.7 is more likely below it; matching observed values therefore picks harder images. With more annotators per image the estimates sharpen, and the drop shrinks to within the seed-to-seed spread. TopImages shows the same regression to the mean: its images are observed at 1.00 but are truly at 0.87.

How the Paper Reads Its Results

The paper is firm about adaptivity and tentative about everything else. On adaptivity, the introduction calls it an unlikely explanation, and the discussion says the experiments rule out its most dangerous form, diminishing returns, on both datasets, while stating that a roughly constant adaptivity gap would go undetected. It offers two mechanisms for why reuse did so little damage and says its experiments offer little evidence to favour either; the benchmark overfitting article discusses both, and the limit of the test.

For the drop itself, the language is that of a hypothesis. The authors conjecture that the drops stem from small variations in the human annotation process, call the distribution gap their leading hypothesis, and name the subjective annotation step as the likely main difficulty, citing the annotator population, task format, compensation and classes without exact definitions, with three ambiguous ImageNet class pairs as examples (projectile and missile, tusker and Indian elephant, CRT screen and monitor). “Harder” appears in quotation marks in the abstract: the new images are correct, from Flickr and selected by most workers, and the binned analysis suggests one could build a correct test set with lower accuracies still. The introduction says the original accuracies can be recovered almost exactly by keeping only the easiest images; in Table 2 the easiest set, TopImages, overshoots by about 2 points on average.

The authors’ model for the linear fit, in Appendix B of the ICML supplement, gives each image a difficulty $\tau$ drawn from a normal distribution and each model $j$ a skill $s_j$, with probability $\Phi(s_j - \tau)$ of a correct answer. Averaging over the difficulty distribution makes probit-transformed accuracies on any two test sets linearly related, with a slope set by the two difficulty spreads and an offset by their means. The authors stress that the model “should not be seen as the true explanation”, only as a way the observed linearity could arise.

Versions

The paper exists as two arXiv versions and the ICML proceedings version. The dates below are from the arXiv submission history and the PMLR page’s publication date; the comparison of contents is ours.

Version	Date	Contents
arXiv v1	13 February 2019	Full paper in a one-column format: Sections 1–7, then Appendix A (overview), B (CIFAR-10) and C (ImageNet)
arXiv v2	12 June 2019	Same main text; the ImageNet discriminator experiment removed from the appendix and later tables renumbered
ICML 2019 (PMLR 97)	24 May 2019	Main text of Sections 1–6 in the two-column proceedings format; Appendices A–D as supplementary material

The main text of the two arXiv versions is the same apart from layout and the position of footnotes and tables. Version 1 had one more ImageNet experiment, Appendix C.3.3: ResNet-152 discriminators trained to distinguish original validation images from each of the three new test sets, with accuracies of 49.3% to 50.4%, all within their intervals of chance. Its table labels two rows “MatchedFrequency”; the second of them presumably stands for TopImages. Version 2 drops this section, and its full results tables move from Tables 15–20 to Tables 14–19. The CIFAR-10 discriminator experiment remains in every version.

The ICML version mostly reorganizes. arXiv Section 5.3, the model for the linear fit, becomes supplementary Appendix B, and arXiv Section 6, related work, becomes Appendix A, with the same text; the arXiv CIFAR-10 and ImageNet Appendices B and C become C and D. The supplement uses v2’s table numbers, has no ImageNet discriminator experiment, and contains the full 67-model tables, the probit analysis and the binned figures, so the numbers cited above from “the supplement” are in the proceedings version. The one substantive cut is the conclusion: arXiv Section 7 lists directions for future work and suggestions for future datasets (releasing the dataset-creation code, testing annotators on a common image set and reporting the results, keeping a “super hold-out” test set hidden for years, choosing tasks simpler for humans, and expert-annotated test sets), none of which is in the ICML paper or its supplement. The CIFAR-10 material first appeared on its own in June 2018 as “Do CIFAR-10 Classifiers Generalize to CIFAR-10?” (arXiv:1806.00451), which evaluated 30 models and reported drops of 4% to 10%.

Released Data and Code

The paper points to two repositories. The ImageNetV2 repository (README at commit 0b0c3fe, 13 May 2023, the latest as of October 2026) describes the three test sets, identified internally as imagenetv2-a-44 (Threshold0.7), imagenetv2-b-33 (MatchedFrequency) and imagenetv2-c-12 (TopImages), where the last number is the review revision; the 33 matches the paper’s 33 iterations. It also releases the code for each stage of the pipeline, from Flickr search and HIT generation to near-duplicate checking, sampling, the review server and evaluation, and states that the candidate pool is released with metadata such as the Flickr queries and the MTurk annotations. The README’s own description is stronger than the paper’s: it says the new data’s independence of existing models guarantees that the accuracy scores are unaffected by adaptive overfitting, whereas the paper says its test cannot exclude an adaptivity gap that lowers all models by a constant amount. The code is MIT-licensed; the README says the licence does not cover the images, which come from Flickr with Flickr’s licence information. As of that commit the test sets are hosted on Hugging Face.

The maintainers have documented three practical issues in the repository’s issue tracker:

• Class folders are indices. The released archives name class folders 0 to 999 by class index (folder 0 is tench, 1 goldfish) rather than by WordNet ID as the validation set does (issue 4, 2019). PyTorch’s ImageFolder sorts folder names as strings (0, 1, 10, 100, …), so it assigns wrong labels; a maintainer confirmed in 2024 that ImageFolder is not natively compatible (issue 10) and pointed to the separate ImageNetV2_pytorch loader, which takes each label from the integer folder name. The ImageNetV2 README at the pinned commit still shows ImageFolder as the way to load the data.
• File names changed. After the S3 bucket holding the data was accidentally emptied in October 2020, the data were re-uploaded with file names replaced by internal candidate IDs, which link each image to the released metadata; the maintainers stated that the old file names were lost (issue 6). Annotations made by others against the old names no longer match.
• Hosting moved. The data moved from S3 to Hugging Face, and older S3 download links stopped working during the migration (issue 4).

The CIFAR-10.1 repository (README at commit d9982ab, 16 April 2020) distributes versions v4 and v6 as NumPy arrays, as described above. Its images are a subset of Tiny Images, which its creators withdrew on 29 June 2020 after offensive categories and images were found in it, asking the community to stop using it and delete copies; CIFAR-10.1 inherits that provenance, as does CIFAR-10 itself.

Impact

Against the conventional reading of a drop on new data as adaptation to the old test set, the paper argued for a distribution gap. A year later, Engstrom et al. (2020) summarized the replication studies as having generally found little evidence of adaptive overfitting, alongside significant drops on the new test sets, and later work used the linear relationship between old and new accuracy as the baseline against which robustness is measured (see What Came After). ImageNetV2 has been reused as a shifted test set alongside the ImageNet validation set: it is among the natural distribution shifts in the CLIP paper’s robustness evaluation, and its smaller size makes scores on it vary more from one training seed to the next than validation scores do, as the statistical significance in evaluation article shows for one 100-seed study. Further reproductions followed, such as the CIFAR-10.2 test set used by Miller et al. (2021), and the selection-frequency statistic itself became an object of study.

Limitations

• Adaptivity is bounded, not measured. A constant adaptivity gap cannot be separated from the distribution gap; the i.i.d. test the authors asked for later came from reconstructed MNIST digits (see benchmark overfitting).
• The replication is not exact. The worker population, the date window, the stricter near-duplicate search and the authors’ own final review all differ from ImageNet’s, and some review choices were subjective by the authors’ own account. The paper measured the effect of the review but could not measure the others, and matching on ten-worker estimates of selection frequency introduced the bias described under What Came After.
• Single-label evaluation. All numbers are top-1 or top-5 accuracy against one label per image; many ImageNet images contain several valid classes, which changes the size of the measured gap (see Shankar et al. below).
• Internal inconsistencies. Besides the abstract’s narrower ImageNet range and the “almost exactly” recovery noted above, the text and caption disagree on which ResNet Figure 7 shows, and arXiv v1’s discriminator table repeats a row label.

What Came After

Statistical bias in the replication. Engstrom et al. (2020) re-measured selection frequencies for all of ImageNetV2 and a subset of the original validation set, using 40 workers per image and a closely matching task with some changes to worker screening. The averages were 85.2% for the original and 80.7% for ImageNetV2, with 95% bootstrap intervals of $\pm 0.1$, where Recht et al. had measured 0.71 and 0.73: the new set’s apparent 2-point advantage became a 5-point deficit. They attribute this to the mechanism the worked example above illustrates, matching on noisy estimates when the candidate pool has lower true selection frequencies than the already-filtered originals. Reweighting by the 40-worker estimates alone cut the gap to 5.7 points; with bias corrections they estimate that selection-frequency bias accounts for 8.1 of the 11.7-point drop, leaving $3.6 \pm 1.5$ points unexplained, and they name annotator heterogeneity and changes in annotators’ criteria over the years as unmodelled sources that could explain more.

Multi-label evaluation and human accuracy. Shankar et al. (2020), all four original authors with two colleagues, had three trained labellers give multiple labels to 20,000 original validation images and 20,683 images from the three ImageNetV2 sets, finding about 20% of images with more than one valid label. Under these multi-label annotations, all models in their testbed still dropped by 5 to 8 points from ImageNet to ImageNetV2 (ResNet-50 by 8.4 points; the two strongest models in their Table 1, an EfficientNet-B8 trained with AdvProp and a ResNeXt trained on a billion Instagram images, by 5.3 and 5.9), while five trained human labellers scored within about 1 point on both sets. They read this as evidence that the shift is real for models but not for people.

Distribution shift as a research problem. Taori et al. (2020) treated ImageNetV2 as one of several natural distribution shifts in a testbed of 204 ImageNet models and 213 test conditions. On ImageNetV2, standard models fell on a line after logit scaling with $r^2 = 1.00$, and robustness interventions gave little or no benefit beyond that line; the best case, EfficientNet-L2 trained with NoisyStudent on extra data, narrowed the drop only from 8.6 to 7.5 points. They called accuracy above the line “effective robustness”, the measure later used in the CLIP paper. Miller et al. (2021) generalized the paper’s linear fit: across architectures, hyperparameters, training set sizes and many distribution shifts, including CIFAR-10.1, a further CIFAR-10 reproduction and ImageNet-V2, out-of-distribution accuracy was a precise linear function of in-distribution accuracy on a probit scale, with exceptions they documented, and they gave a Gaussian data model in which such trends arise.