Learning Multiple Layers of Features from Tiny Images

“Learning Multiple Layers of Features from Tiny Images” is a 60-page document by Alex Krizhevsky, dated April 8, 2009, and distributed from the University of Toronto’s computer science server. Its subject is unsupervised learning: how to get restricted Boltzmann machines (RBMs), and a two-layer deep belief network built from them, to learn useful filters from about 2 million $32 \times 32$ colour images taken from the 80 Million Tiny Images collection, after others had failed to. To test whether the filters help recognition, the report introduced a hand-labelled subset of those images, CIFAR-10, and announced a second, CIFAR-100. On the CIFAR-10 test set a network initialized from the RBM’s features reached 64.84% accuracy, against 41.13% for logistic regression on the pixels. A final chapter describes an algorithm for training large RBMs across several networked machines. The generative models were soon superseded; the two datasets became standard benchmarks. This article covers the report; the datasets as they are used today, their duplicates and their later test sets are described in the CIFAR article.

The Document

The title page prints only the title, the author and the date; it names no institution, report number or series. The official CIFAR page calls it “the tech report”, asks users of the datasets to cite it, and says that its Chapter 3 describes the collection in detail; the file name ends in “TR”. Krizhevsky’s own home page on the same server lists the same file under the heading “MSc thesis”. Later papers follow one description or the other: Ranzato, Krizhevsky and Hinton (2010) and Ranzato and Hinton (2010) cite it as an MSc thesis of the Department of Computer Science, University of Toronto; Krizhevsky’s 2010 manuscript, Coates, Lee and Ng (2011) and the 2012 AlexNet paper as a master’s thesis; Barz and Denzler (2020) as a technical report. A search of the University of Toronto’s TSpace repository in October 2026 found no record of it, and Geoffrey Hinton’s publication and student pages do not list it. The official page credits Krizhevsky, Vinod Nair and Geoffrey Hinton with creating CIFAR-10 and CIFAR-100; the report thanks Nair in a footnote for a substantial contribution to the labelling.

This article follows the copy linked from the official page. Its embedded metadata give a creation date of 25 January 2010, nine months after the date on the title page; no earlier copy was available to compare, so whether the text changed in between is unknown. Chapter 1 covers background (natural images, whitening, RBMs, feed-forward networks), Chapter 2 the generative models, Chapter 3 the labelled subsets and the classification experiments, and Chapter 4 parallel training; the appendices cover whitening (A), backpropagation (B), the labellers’ instruction sheet (C) and the CIFAR-100 classes (D).

Problem

The tiny images had been collected by colleagues at MIT and NYU (Torralba, Fergus & Freeman, 2008) by sending every non-abstract English noun in WordNet to image search engines and keeping up to about 3,000 results per term at $32 \times 32$ pixels. The report describes the result as 80 million images over 79,000 search terms, collected over six months (§1.2.1); the TPAMI paper on the collection gives 8 months and 79,302,017 images from 75,062 nouns.

For the Toronto group such a collection was in principle ideal for training deep generative models without labels, but the report cites two failures (§2.2). Fergus (a personal communication) had trained this kind of model on the tiny images and obtained noisy global filters and point-like identity filters, and another group (Le Roux, also a personal communication) had obtained only uniform or global filters from $16 \times 16$ colour patches. The motivation in §2.1 is threefold: a good generative model of natural images, features such as the positions and orientations of contours that a classifier could use better than pixels, and some resemblance to the edge-detecting neurons of the visual cortex.

The second problem was evaluation. The search term attached to each image is a label the report calls extremely unreliable, so the collection could not measure whether learned features help recognition (abstract; §1.2.1).

Contribution

• A way to train Gaussian–Bernoulli RBMs on whole $32 \times 32$ colour images that learns localized edge filters, reached through whitening, deleting low-variance directions, training on patches and merging, and a faster parallel implementation (Chapter 2).
• A second layer of features, a binary RBM trained on the first layer’s 10,000 hidden probabilities, making a two-layer deep belief network (§2.8).
• CIFAR-10 and CIFAR-100, two hand-filtered labelled subsets of the tiny images, with the labelling criteria and the instruction sheet (§3.1, Appendices C and D).
• Classification experiments on CIFAR-10 showing that features learned without labels on 2 million images beat raw pixels and randomly initialized networks (§3.2, Figure 3.1).
• A distributed version of contrastive divergence for training large RBMs on several machines, with measured speedups and a communication analysis (Chapter 4).

Method

Covariance and whitening

The report starts from the covariance matrix of the tiny images (§1.2.2, Figures 1.1 and 1.2), with pixels indexed row by row and the 1,024 red values first, then green, then blue, the layout the CIFAR files still use. Nearby pixels are strongly correlated and distant ones weakly, more so within the same colour channel and within the same row or column, and faint anti-diagonals show the images to be roughly symmetric about the vertical and horizontal axes, which the report attributes to photographers levelling the ground plane and centring the main object.

Because these pairwise correlations are strong, the report argues that a model might spend its capacity learning that neighbouring pixels are similar, and proposes removing second-order structure first so that the model must find higher-order regularities (§1.3.1). Appendix A derives the symmetric whitening matrix $W = (XX^\top)^{-1/2}$ for zero-mean data in the columns of $X$, computed from the eigendecomposition $XX^\top = PDP^\top$ as $PD^{-1/2}P^\top$, and calls it the Zero Components Analysis (ZCA) solution; the dewhitening matrix is $PD^{1/2}P^\top$. Each row of $W$ is a filter the size of an image; the whitening filters are highly local (Figure 1.3), and a whitened image keeps its edges while regions of fairly uniform colour go to zero (§1.3.3, Figure 1.5). The construction itself and its relation to PCA are described in the principal component analysis article.

RBMs and deep belief networks

Section 1.4 gives the standard background. A binary RBM has an energy over visible and hidden units, with connections only between the two layers, so each layer’s units are conditionally independent given the other’s. Training follows the gradient of the log-likelihood, with the model expectation approximated by contrastive divergence; the report uses CD-1 almost exclusively. A deep belief network is trained greedily: each new RBM takes the hidden activities of the layer below as its data, and when a second layer of the same size is initialized from the first’s weights, training it raises a lower bound on the log-likelihood that starts out tight, and so raises the log-likelihood itself (§1.4.2, equation 1.7). The report notes that the bound is not tight for Gaussian–Bernoulli RBMs, the kind it uses on images.

A Gaussian–Bernoulli RBM replaces each binary visible unit with a real-valued one whose energy term is quadratic, $(v_i - b_i)^2 / 2\sigma_i^2$, and whose input to the hidden units is scaled by $1/\sigma_i$ (equation 1.8). Section 1.4.3.2 derives an update for learning the standard deviations $\sigma_i$ as well. Since the likelihood of a large RBM cannot be computed, the report tracks reconstruction error, and says itself that this is not a good measure of how well the model fits the data, because it depends heavily on how fast the Markov chain mixes (§1.4.4).

Getting an RBM to learn filters

Chapter 2 is written as a sequence of attempts.

1. Whitened images (§2.3). An RBM with 8,000 hidden units on 2 million whitened images, with visible variances fixed at 1, learned filters the report calls meaningless (Figure 2.1).
2. Deleting directions of variance (§2.4). Suspecting that high-frequency noise distracted the model, Krizhevsky removed the 1,000 least significant of the 3,072 principal components, keeping 2,072. The log-eigenspectrum shows those 1,000 directions to have variance several orders of magnitude below the leading ones (Figure 2.2), and images whitened and dewhitened with the truncated filter look the same as before, while the whitened images look smoother (Figure 2.3). The RBM still learned nothing useful, but the step was kept for all later experiments.
3. Patches (§2.5). Each image was cut into 25 overlapping $8 \times 8$ patches, plus a 26th patch made by subsampling the whole image to $8 \times 8$, and 26 independent RBMs with 300 hidden units each were trained. These learned edge detectors (Figures 2.5 and 2.6), with an affinity for low-frequency colour filters and high-frequency black-and-white ones; the report offers two speculative explanations, one being that precise edge positions and rough colour may suffice to model natural images.
4. Merging (§2.5.1). The 26 RBMs were combined into one RBM with 7,800 hidden units connected to every pixel, weights outside each unit’s patch initialized to zero and the subsampled patch’s weights spread over the full image (each weight copied to 16 pixels and divided by 16, Figure 2.7). Reconstruction error jumped at the merge and then fell below its earlier minimum (Figure 2.10).
5. Whole images (§2.6). With a new algorithm that parallelized training across a machine’s CPU cores (later extended across machines, Chapter 4), an RBM with 10,000 hidden units trained directly on whitened $32 \times 32$ images learned filters similar to the patch RBMs’ (Figure 2.11). On unwhitened data, with the 1,000 directions still deleted and the visible standard deviation set to the measured average pixel standard deviation of 69, every filter first passed through a long “point” stage, copying single pixels to hidden units, before turning into an edge filter (Figures 2.12 and 2.13). The report warns that this stage can make training look finished when it is not. The unwhitened filters were larger, which the report attributes to a single whitened pixel standing for a whole region of colour.
6. Learning the standard deviations (§2.7). Learning $\sigma_i$ needed, in the report’s experience, a learning rate about 100 to 1,000 times smaller than the weights’; otherwise many filters stayed point-like. The learned values fell from 69 to about 30, and stayed close to each other (their standard deviation was 0.42); reconstruction error improved greatly, but the filters became noisier and more of them stayed point-like (Figure 2.14, an RBM with 8,192 hidden units).
7. Second layer (§2.8). A binary RBM with 10,000 hidden units trained on the 10,000 hidden probabilities of the unwhitened RBM tended to connect strongly to similar first-layer filters (Figure 2.15).

Section 2.6 refers to earlier successes with “8×8 and 16×16 patches”, but only $8 \times 8$ patches are described in the report (our observation; the only other $16 \times 16$ patches mentioned are Le Roux’s in §2.2).

The report’s explanation of the whitened RBM’s weaker features in Chapter 3 is that whitening sets the variance in every direction to 1 and may exaggerate directions that are mostly noise. The example below repeats the deletion of §2.4 on synthetic images, where the noise-free version of every image is known. It renders 100,000 colour photographs of $32 \times 32$ pixels (a colour gradient and one disc or square, blurred by one pixel), adds sensor noise with a standard deviation of 2 grey levels and rounds to 8 bits, computes the $3{,}072 \times 3{,}072$ covariance and its eigendecomposition, and builds the full ZCA filter and the truncated one. It then whitens 1,000 new images and their noise-free versions.

import numpy as np
from scipy.ndimage import gaussian_filter

def scenes(rng, n, size=32):
    """n synthetic 32x32 colour photographs: a colour gradient and one disc or square, lens blur."""
    y, x = np.mgrid[0:size, 0:size] / size
    c0, c1 = rng.uniform(0, 255, (2, n, 1, 1, 3))
    img = c0 + (c1 - c0) * (x * rng.uniform(-1, 1, (n, 1, 1)) + y * rng.uniform(-1, 1, (n, 1, 1)))[..., None]
    (cy, cx), r = rng.uniform(8, 24, (2, n, 1, 1)), rng.uniform(3, 10, (n, 1, 1))
    disc = (y * size - cy) ** 2 + (x * size - cx) ** 2 < r ** 2
    square = (abs(y * size - cy) < r) & (abs(x * size - cx) < r)
    mask = np.where(rng.random((n, 1, 1)) < 0.5, disc, square)
    img = np.where(mask[..., None], rng.uniform(0, 255, (n, 1, 1, 3)), img)
    return np.clip(gaussian_filter(img, (0, 1, 1, 0)), 0, 255).reshape(n, 3072)

def camera(rng, clean):                                     # sensor noise (sd 2 grey levels), 8-bit storage
    return np.round(np.clip(clean + rng.normal(0, 2, clean.shape), 0, 255))

def corr(a, b):                                             # mean correlation between paired rows
    a, b = a - a.mean(1, keepdims=True), b - b.mean(1, keepdims=True)
    return np.mean((a * b).sum(1) / np.linalg.norm(a, axis=1) / np.linalg.norm(b, axis=1))

rms = lambda a, b: np.sqrt(np.mean((a - b) ** 2))
for seed in range(2):
    rng = np.random.default_rng(seed)
    XtX, total, N = np.zeros((3072, 3072)), np.zeros(3072), 0
    for _ in range(10):                                     # 100,000 unlabelled images, in chunks
        X = camera(rng, scenes(rng, 10000))
        XtX += X.T @ X; total += X.sum(0); N += len(X)
    mu = total / N
    D, P = np.linalg.eigh(XtX / N - np.outer(mu, mu))       # eigenvalues in ascending order
    k = np.arange(3072) >= 1000                             # keep 2,072 directions, delete 1,000
    W_full = (P / np.sqrt(D)) @ P.T                         # ZCA whitening matrix P D^-1/2 P^T (App. A)
    W_del = (P[:, k] / np.sqrt(D[k])) @ P[:, k].T           # the same with 1,000 directions deleted (2.4)
    deW = (P[:, k] * np.sqrt(D[k])) @ P[:, k].T             # dewhitening P D^1/2 P^T on 2,072 directions

    clean = scenes(rng, 1000); test = camera(rng, clean)    # new images, with noise-free versions
    back = (test - mu) @ W_del @ deW + mu                   # whiten, delete, dewhiten
    print(f"seed {seed}: variance in the 1,000 smallest directions {D[:1000].sum() / D.sum():.4%} of total; "
          f"mean eigenvalue, largest 1,000 / smallest 1,000: {D[-1000:].mean() / D[:1000].mean():,.0f}")
    print(f"  whiten-delete-dewhiten changes pixels by {rms(back, test):.2f} grey levels rms; "
          f"distance to noise-free image {rms(test, clean):.2f} before, {rms(back, clean):.2f} after")
    print(f"  whitened image vs whitened noise-free image, correlation: full ZCA "
          f"{corr((test - mu) @ W_full, (clean - mu) @ W_full):.2f}, "
          f"1,000 directions deleted {corr((test - mu) @ W_del, (clean - mu) @ W_del):.2f}")

Output:

seed 0: variance in the 1,000 smallest directions 0.0205% of total; mean eigenvalue, largest 1,000 / smallest 1,000: 4,881
  whiten-delete-dewhiten changes pixels by 1.15 grey levels rms; distance to noise-free image 1.96 before, 1.65 after
  whitened image vs whitened noise-free image, correlation: full ZCA 0.49, 1,000 directions deleted 0.60
seed 1: variance in the 1,000 smallest directions 0.0204% of total; mean eigenvalue, largest 1,000 / smallest 1,000: 4,899
  whiten-delete-dewhiten changes pixels by 1.15 grey levels rms; distance to noise-free image 1.97 before, 1.65 after
  whitened image vs whitened noise-free image, correlation: full ZCA 0.48, 1,000 directions deleted 0.59

The two seeds agree closely. The 1,000 smallest directions hold about 0.02% of the pixel variance, and their eigenvalues are more than three orders of magnitude below the largest thousand, the shape of the report’s Figure 2.2. Deleting them changes an image by about 1.15 grey levels, which is what isotropic noise of standard deviation 2 contributes to 1,000 of 3,072 directions ($2\sqrt{1000/3072} \approx 1.14$): the deleted part is almost all noise, the image moves closer to its noise-free version, and the change is invisible, as Figure 2.3 shows for real tiny images. After whitening the picture reverses. Every direction then has unit variance, so the deleted directions, worthless in pixel space, would have made up about a third of the whitened image (our reasoning from the construction). Even with them removed, the whitened noisy image correlates only about 0.6 with the whitened noise-free one, because many of the remaining 2,072 directions are also dominated by noise. Deleting 1,000 directions thus reduces the noise that whitening amplifies but leaves much of it, consistent with the report’s caveat; whether real tiny images behave the same way, and whether this explains its result, the example cannot show. The synthetic images are deliberately blurred, as downscaled photographs are; with sharp-edged shapes, part of the image content also falls in the smallest directions and is lost.

The labelled subsets

Section 3.1 describes the labelling briefly, and the CIFAR article gives the details. Students paid by the hour filtered the images retrieved by each class name and by its WordNet hyponyms, rejecting those that did not belong; the researchers checked every label; near-duplicates were removed by an L2 comparison whose threshold the report describes but does not state; and 5,000 random images per class went to training. The four criteria listed in §3.1 condense an eight-point instruction sheet reproduced in Appendix C, which adds, for instance, that false positives are worse than false negatives. CIFAR-100, “only very recently completed”, has 600 images in each of 100 classes grouped into 20 superclasses, and the report says its methodology was identical; Appendix D, which lists the classes, adds one documented change: for classes with plural names, labellers were told not to reject images with several instances. The name comes from the Canadian Institute for Advanced Research, which funded the project.

Classification experiments

Every classifier in §3.2 ends in multinomial logistic regression and predicts the most probable of the ten classes. The inputs compared were raw pixels, whitened or unwhitened; the 10,000 features of the first-layer RBM trained on whitened or unwhitened data; and the second-layer RBM’s 10,000 features. The best model follows Hinton and Salakhutdinov (2006): a network with one hidden layer of logistic units whose input weights come from the unwhitened RBM and whose output weights come from the logistic regression trained on its features. The network starts out computing exactly what the logistic regression computes and is then fine-tuned with backpropagation. Two hidden layers, initialized from both RBMs, were tried as well. For comparison, networks with one hidden layer of 1,000 or 10,000 units were trained on unwhitened pixels from random initialization. All experiments used CIFAR-10 only.

Parallel training

Chapter 4 distributes CD-1 over $K$ machines by splitting the hidden and visible units into $K$ chunks (§4.2). Each machine, holding the whole dataset, computes the hidden activities of its own $J/K$ hidden units, all machines exchange them, each computes $I/K$ of the reconstructed visible units, they exchange again, and each computes its share of the hidden activities for the reconstruction, with a synchronization point after each step. Because binary hidden states are one bit each, the exchanges are cheap; Gaussian visible units cost more. Most weights are known to two machines and updated twice, and for real-valued data the two copies drift apart through different floating-point orderings, so a weight synchronization step copies one machine’s version to the others. Biases are each owned by one machine.

The implementation is a C extension to a Python program, using BLAS for the matrix work, TCP sockets between machines and pthreads between cores, with one writer and one reader thread per remote machine so that sending never blocks computation (§4.3). Weights were synchronized after every batch of 8,000 images, which kept them within $10^{-9}$ of each other at single precision. Section 4.5 sketches alternatives, among them a variant for RBMs with fewer hidden than visible units in which each machine holds only $1/K$ of the input dimensions, and its mirror image for more hidden than visible units. The chapter’s footnote 4, attached to this mirror-image variant, says that the variant tested on the benchmark problem ran about 30% faster than the algorithm the chapter describes in detail, “even though” that problem has more hidden than visible units, and that it was the version used to train the report’s RBMs. That “even though” fits the first variant, designed for fewer hidden than visible units, better than the one the footnote is attached to, so which variant trained the RBMs is not certain (our reading).

Results

Filters. The report’s qualitative results are figures of learned filters: meaningless ones from the first whitened RBM (Figure 2.1), edge detectors from patches and from whole images (Figures 2.5–2.13), and second-layer units grouping similar first-layer edges (Figure 2.15). The models were judged by the look of their filters and by reconstruction error; no likelihoods are reported.

Classification. Figure 3.1 gives CIFAR-10 test accuracy for ten models; the values below are the bar labels.

Bar	Model (Figure 3.1 caption)	Test accuracy
a	Logistic regression on raw whitened pixels	37.32%
b	Logistic regression on raw unwhitened pixels	41.13%
c	Logistic regression on 10,000 features of an RBM trained on whitened data	59.61%
d	Logistic regression on 10,000 features of an RBM trained on unwhitened data	63.78%
e	One-hidden-layer network initialized from (d)’s RBM and logistic regression, fine-tuned	64.84%
f	One-hidden-layer network, 1,000 units, trained on unwhitened pixels	49.78%
g	One-hidden-layer network, 10,000 units, trained on unwhitened pixels	51.53%
h	Logistic regression on the second-layer RBM’s 10,000 features	56.62%
i	One-hidden-layer network initialized from the second-layer RBM, fine-tuned	62.20%
j	Two-hidden-layer network initialized from both RBMs, fine-tuned	60.34%

The caption pairs (a, b) with “raw, whitened and unwhitened data” and the legend lists whitened first, so (a) is whitened; the numbered list in §3.2 happens to name unwhitened pixels first (our reading). The report draws four conclusions. RBM features are far more useful than raw pixels. Features of the RBM trained on unwhitened data beat those trained on whitened data (63.78% against 59.61%), possibly because whitening exaggerates noise directions. Fine-tuning improves on the logistic regression only slightly (64.84% against 63.78%). And the RBM’s features beat those of randomly initialized networks of the same size, which also need many more epochs to train, although the RBM itself, a footnote concedes, took a long time to learn. A second layer of features lowered accuracy (56.62% with logistic regression), and the two-hidden-layer network fell short of the one-hidden-layer one; the chapter’s footnote 3 attributes this to its roughly $2 \times 10^8$ parameters, a strong tendency to overfit and too few hyperparameter settings tried. The report notes that two hidden layers giving no improvement agrees with Serre et al. (2007).

Figure 3.2, the confusion matrix of model (d), shows animals and non-animals forming two clusters, cats often taken for dogs and birds occasionally for planes. In our reading of the figure, ships and airplanes, and trucks and automobiles, are also confused more often than other pairs of vehicles.

Speed. Chapter 4’s benchmark is an RBM with 8,000 visible and 20,000 hidden units on 8,000 random examples, run on machines with two dual-core 3 GHz Intel Xeon processors over a network measured at 105 MB/s (§4.4). With binary visible units, doubling the number of machines from two to four or from four to eight very nearly doubled the speed (Figure 4.6); with Gaussian visible units it almost did (Figure 4.7). Read from the plots of Figure 4.2, eight machines with four threads each ran about 11 to 16 times faster than one thread on one machine, depending on precision and minibatch size, binary visible units throughout. The total data sent grows linearly with the number of machines, $48(K-1)$ MB per batch for binary visible units, while the amount per machine stays below 48 MB (§4.4.1, Table 4.1). Single precision was nearly three times faster than double at first and about twice as fast after heavy parallelization. On a small RBM for binarized MNIST digits (784 visible, 392 hidden units) the benefit vanished after the second machine, by which point a batch took only a few seconds.

Impact

In our reading, the report is cited today almost entirely for CIFAR. The official page asks users to cite it, and the CIFAR article traces how CIFAR-10 and CIFAR-100 became standard small-scale benchmarks for image classification, generative models and robustness. Because the report documented the labelling, the instruction sheet could be reused: Recht et al. (2018, §3) built the CIFAR-10.1 test set a decade later by having one author select new Tiny Images pictures under the Appendix C guidelines and another review them, as the students and the researchers had, with near-duplicates removed by L2 nearest neighbours. They also found that every CIFAR-10 image has an exact copy in Tiny Images. That study and its ImageNet counterpart are described in the article on ImageNetV2.

Its results became the first reference points on CIFAR-10. Ranzato, Krizhevsky and Hinton (2010, Table 2) and Ranzato and Hinton (2010, Table 2) reprinted the Gaussian RBM numbers of Figure 3.1 (59.6%, 63.8%, 64.8% and 56.6%) as the baselines their models beat, and Coates, Lee and Ng (2011, Table 1) listed the raw-pixel result, quoting the whitened 37.3% rather than the unwhitened 41.13%. The preprocessing travelled too: Ranzato et al. trained on “ZCA whitened images as processed by Krizhevsky (2009)”, and Krizhevsky’s 2010 convolutional model again deleted the 1,000 least significant principal components.

The report’s central idea, learning features from a large unlabelled image collection and fine-tuning them on a small labelled one, is the pattern that transfer learning and self-supervised learning later made routine with different models. In our reading, the report is an early demonstration of it on natural colour images at a scale of millions, not its origin: the method it applied came from Hinton and Salakhutdinov (2006), whose networks were pretrained as stacks of RBMs (see autoencoder), and the report itself notes that RBMs and deep belief networks had already learned meaningful features from handwritten digits and faces (§1.4.2).

Limitations

The report states several limitations itself: reconstruction error is a poor measure of a generative model (§1.4.4); the second layer of features and the two-hidden-layer network did not improve classification, and the largest model was too slow to tune (Chapter 3, footnote 3); learning the visible standard deviations improved reconstruction but made the filters worse (§2.7); and CIFAR-100 was not used in any experiment.

Other limitations follow from what the report does not give (our reading):

• Missing training details. Learning rates, epochs, momentum, weight decay and training times are not given for the 2-million-image RBMs or the classifiers, so the experiments cannot be repeated from the text. No validation set is mentioned, and each result in Figure 3.1 is a single number without a spread over runs.
• Overlap between unlabelled and labelled data. The report does not say whether the 2 million unlabelled images excluded the CIFAR-10 images. The papers that followed did specify it: Ranzato et al. (2010) and Ranzato and Hinton (2010) trained on two million tiny images that do not overlap with CIFAR-10, and Krizhevsky (2010) notes that 3.2% of the CIFAR-10 images are in his 1.6-million-image pretraining set.
• Duplicates. The duplicate-removal threshold is not given. Barz and Denzler (2020) later found duplicates, mostly near-duplicates, of 3.25% of the CIFAR-10 test images and 10% of the CIFAR-100 test images in the training sets or within the test sets; none in CIFAR-10 was an exact copy, while CIFAR-100 had some. They suggested that the duplicate removal the report describes may have been omitted for CIFAR-100, and that some of the labelling rules seem to have been relaxed for it (§3.3 and §4 of the journal version), so the report’s statement that CIFAR-100’s methodology was identical to CIFAR-10’s is in doubt. The plural-name rule of Appendix D, which they do not mention, documents one such relaxation; line drawings and cartoons in CIFAR-100, which they also report, are not covered by it.
• Inconsistencies in the text (our observations). Appendix A derives $W = \sqrt{n-1}\,(XX^\top)^{-1/2}$ and then states the result without the factor $\sqrt{n-1}$, while still saying it gives unit variance; the factor only rescales the whitened data. In §4.5, putting the stated sizes ($v = 8{,}000$, $h = 20{,}000$, $K = 8$) into the printed inequality $mhK/8 + 4mvK \geq 4vh$ gives a minibatch threshold of about 2,319, not the 1,538 the text states (our arithmetic). Table 4.1’s communication figures do follow from the chapter’s formulas.

The source collection is no longer available. On 29 June 2020 its creators withdrew 80 Million Tiny Images, citing derogatory terms among its categories and offensive images that the automated collection from WordNet nouns had brought in (Torralba, Fergus & Freeman, 2020). The CIFAR article describes what that does and does not imply for CIFAR.

What Came After

The Toronto group’s next steps addressed the weakness of the Gaussian–Bernoulli RBM that the report had worked around. Ranzato, Krizhevsky and Hinton (2010) argued that Gaussian–binary RBMs model the covariance structure of natural images poorly and proposed a factored three-way RBM whose hidden units represent the local covariance of an image; trained on $8 \times 8$ patches of two million tiny images and applied over a grid of positions, its three-layer features reached 65.3% on CIFAR-10 (Table 2). Ranzato and Hinton’s mean-covariance RBM (CVPR 2010) reached 71.0% with three layers, and, unlike the report’s model, improved when a second and a third layer were added (Tables 1 and 2, §4.2). Krizhevsky’s 2010 manuscript trained a two-layer convolutional deep belief network with capped rectified linear units on 1.6 million tiny images, pretraining for up to 45 hours on an Nvidia GTX 280 GPU, and reported 78.9% (Table 2).

Coates, Lee and Ng (2011) then questioned how much the learning algorithm mattered. With a single layer of features, whitening, a stride of one pixel and many features, k-means clustering reached 79.6% on CIFAR-10, ahead of every other result in their Table 1, including the 78.9% of Krizhevsky’s convolutional network. They found whitening crucial for the clustering methods and its effect ambiguous for sparse autoencoders and sparse RBMs; for sparse RBMs a significant benefit with 100 features disappeared with more (§4.2). In our reading, this suggests that whether whitening helps an RBM depends on the setup, and that the report’s comparison of one pair of models does not settle the question.

Supervised convolutional networks on GPUs overtook all of these. Krizhevsky’s cuda-convnet reported 18% error without augmentation and 11% with it, numbers the official CIFAR page still quotes, and the AlexNet paper (Krizhevsky, Sutskever & Hinton, 2012) used CIFAR-10 for its tests of rectified linear units and response normalization. It used no unsupervised pretraining, “to simplify our experiments”, while expecting that pretraining would help once networks outgrew the labelled data. It trained on two GPUs rather than on a cluster of CPUs like the one in Chapter 4. Unsupervised feature learning returned at much larger scale with other objectives, described under self-supervised learning, and CIFAR-10 and CIFAR-100 are among the downstream tests used to evaluate it.