ImageNet: A Large-Scale Hierarchical Image Database

“ImageNet: A Large-Scale Hierarchical Image Database” is a paper by Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei of Princeton University, presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2009. It introduced ImageNet, a collection of web images organized by the noun hierarchy of WordNet, described how the images were gathered with search engines and verified by workers on Amazon Mechanical Turk, and reported the database as it stood then: 3.2 million images in 5,247 categories. The paper received the PAMI Technical Committee’s Longuet-Higgins Prize in 2019. This article covers the 2009 publication; the dataset as it is used today, including the 1,000-class ILSVRC subset, is described in the ImageNet article.

Problem

The authors noted that the web held over 3 billion photos on Flickr alone, and argued that how to organize such data for recognition was an open problem. The well-labeled datasets that served as benchmarks, such as Caltech101 and Caltech256, MSRC and PASCAL VOC, were small, and the authors argued that the next generation of algorithms needed larger and harder ones. Torralba et al. (2008) had gathered 80 million images by sending every WordNet noun to search engines, but stored them at $32 \times 32$ pixels and without cleaning, so only 10–25% of each category’s images were likely correct. The ESP game had labeled many images, but players’ labels were ambiguous (“bank” as river or institution), clustered at the basic level (“bird” rather than “sparrow”), and were mostly not public. The paper’s goal was a resource that was large, clean, full-resolution and densely organized at once.

Contribution

• A hierarchy of images. Each ImageNet category is a WordNet synset, a set of synonyms naming one concept, and inherits WordNet’s IS-A links. The stated target was 500–1,000 clean, full-resolution images for most of WordNet’s roughly 80,000 noun synsets, which the authors put at around 50 million images in about 50,000 synsets, to be reached within two years.
• A snapshot of the first 12 subtrees: mammal, bird, fish, reptile, amphibian, vehicle, furniture, musical instrument, geological formation, tool, flower and fruit, holding 5,247 synsets and 3.2 million images, about 10% of WordNet’s synsets.
• A construction pipeline of search-engine crawling and crowdsourced verification, and three small demonstrations of what a clean hierarchy makes possible.

Method

Collecting candidates. Since only about 10% of image search results were correct, the authors gathered over 10,000 candidates per synset on average, after removing duplicates. Each synset’s queries were its WordNet synonyms. Search engines returned at most a few hundred to a thousand images per query, so the query set was expanded by appending a word from a parent synset when that word appears in the target’s WordNet definition: “whippet”, defined as a dog of greyhound type, also produced “whippet dog” and “whippet greyhound”. The queries were then translated into Chinese, Spanish, Dutch and Italian using WordNets in those languages.

Cleaning. Workers on Amazon Mechanical Turk saw candidates with the synset’s definition and a Wikipedia link, and marked the images containing the concept, regardless of occlusion or clutter. Since workers err and disagree, each image needed a convincing majority of several votes. The number of votes needed was set per synset: for an initial random sample, at least 10 workers voted on each image, giving a table of the probability that an image is good given its votes. In the paper’s example, a single “yes” gives a confidence of 0.85 for “cat” but 0.69 for “Burmese cat”; three “yes” votes give 0.99 and 0.90. The remaining candidates were then labeled until a preset confidence threshold was reached. The authors note that the table also measures how hard a synset is to illustrate, and some synsets never reached a majority for any image.

Measuring diversity. To quantify how varied a synset’s photos are, the authors averaged a synset’s images, downsampled the average to $32 \times 32$ and measured its size as a lossless JPEG: varied photos blur into a smooth, nearly gray average that compresses well, while similar photos leave a sharp, structured average. The sketch below applies the measure to synthetic $64 \times 64$ photos of one category, an elliptical object with fixed markings on random clutter, whose pose varies more as spread grows. Pillow’s JPEG writer offers no lossless mode, so the sketch uses PNG, another lossless format.

import io
import numpy as np
from PIL import Image

H = W = 64
yy, xx = np.mgrid[0:H, 0:W]
pattern = np.random.default_rng(42).uniform(0, 1, (8, 8))  # the category's markings

def photo(rng, spread):
    """A synthetic 64x64 grayscale photo of one category: an elliptical object
    with fixed markings on a cluttered background. spread in [0, 1] sets how
    much the object's position, size and orientation vary between photos."""
    cy, cx = 32 + spread * rng.uniform(-16, 16, 2)
    scale = 1 + spread * rng.uniform(-0.4, 0.4)
    t = spread * rng.uniform(-np.pi, np.pi)
    u = ((xx - cx) * np.cos(t) + (yy - cy) * np.sin(t)) / scale   # object coordinates
    v = (-(xx - cx) * np.sin(t) + (yy - cy) * np.cos(t)) / scale
    inside = (u / 18) ** 2 + (v / 12) ** 2 <= 1
    iu = np.clip(((u + 18) / 36 * 8).astype(int), 0, 7)
    iv = np.clip(((v + 12) / 24 * 8).astype(int), 0, 7)
    img = rng.uniform(0, 255, (H, W))                     # background clutter
    img[inside] = 40 + 200 * pattern[iv, iu][inside]
    return img

def png_size(img):                                        # lossless encoding
    buf = io.BytesIO()
    Image.fromarray(img.round().astype(np.uint8)).save(buf, "PNG", optimize=True)
    return len(buf.getvalue())

def average_image_size(spread, n=300, seed=0):
    rng = np.random.default_rng(seed)
    avg = np.mean([photo(rng, spread) for _ in range(n)], axis=0)
    small = avg.reshape(32, 2, 32, 2).mean(axis=(1, 3))   # downsample to 32x32
    return png_size(small)

for spread in [0.0, 0.25, 0.5, 1.0]:
    print(f"spread {spread:.2f}: average image {average_image_size(spread)} bytes")
print(f"uniform gray:  {png_size(np.full((32, 32), 128.0))} bytes")

spread 0.00: average image 725 bytes
spread 0.25: average image 607 bytes
spread 0.50: average image 571 bytes
spread 1.00: average image 563 bytes
uniform gray:  81 bytes

More varied poses give smaller files, as the paper intends; other seeds give the same trend, with the two largest spreads within a few bytes of each other. Even the most varied set stays far above a uniform gray image, because the averaged clutter leaves residual noise that shrinks with the number of images: in our runs, 50 images at the largest spread give 741 bytes, more than 1,200 images at no spread (631 bytes). In our reading, the measure is therefore fair only between equally sized sets, which is how the paper used it, comparing each Caltech101 category with an equal number of ImageNet images.

Results

Properties. Synsets averaged over 600 images; more than half had over 500, while about 20% had very few, because the web held few examples or the concept, such as “two-year-old horse”, is hard to depict. The mammal subtree alone had 1,170 synsets and 862,000 images, including 147 dog categories. Compared with Caltech101/256, MSRC and PASCAL, the authors counted 20 times as many categories and 100 times as many images. Images averaged about $400 \times 350$ pixels. An independent group checked 80 synsets sampled at every depth of the mammal and vehicle subtrees and found 99.7% precision on average, although deeper synsets, such as Siamese versus Burmese cat, are harder to label. Unlike the ESP game’s mammal labels, its images were spread much more evenly over the depths of the hierarchy. On the diversity measure, the authors present ImageNet as more diverse than Caltech101 in four randomly sampled mammal categories, and as comparable to Caltech256, which was designed to be diverse.

Applications. Three experiments, mostly on the mammal and vehicle subtrees, illustrated uses of the data:

• Non-parametric recognition. Test images were Caltech256 images from 16 categories shared with the mammal subtree, scored by ROC curves. Voting among the 100 nearest $32 \times 32$ neighbors, in the style of Torralba et al., worked better on cleaned images than on the raw candidates. The naive Bayes nearest-neighbor classifier (NBNN) of Boiman et al. (2008), which compares SIFT descriptors and so needs full-resolution images, performed substantially better. Limited to 100 images per category it did worse, but still beat voting over the whole dataset.
• Tree-based classification. A “tree-max” classifier scored a synset by the maximum response of the classifiers in its subtree, so that “dog” also uses the “German shepherd” classifier. At nearly all levels it gave a higher area under the ROC curve than independent classifiers, without extra training. Both did better near the leaves, which the authors attribute to stronger visual coherence there.
• Object localization. A non-parametric graphical model by Li et al., learned for each category against a background class, placed one bounding box around the most likely region of each image in 22 categories. Precision and recall were measured in pixel area against boxes the authors drew on “100 images in 22 different categories” (the paper does not say whether per category or in total). Precision was low, which the authors blame on highly variable and small objects; clustering the detected boxes recovered typical views and poses.

Impact

ImageNet became the backbone of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), held every year from 2010. Russakovsky et al. (2015) describe the full database as 14,197,122 images in 21,841 synsets as of August 2014, and report that ILSVRC took its classification training images directly from ImageNet and collected and verified its validation and test images with the same pipeline described here. Krizhevsky et al. (2012) then trained a convolutional network on 1.2 million ILSVRC images and won the 2012 challenge (see the AlexNet paper). The PAMI Technical Committee’s Longuet-Higgins Prize, given at CVPR to CVPR papers from ten years earlier with significant impact on computer vision, went to this paper in 2019.

Limitations

• A snapshot. The paper reports an unfinished dataset, analyzed mainly through the mammal and vehicle subtrees, and its demonstrations are reported only as plots (ROC curves, AUC and pixel precision and recall), without tables of numbers.
• One label per image. Images were verified for one synset each; boxes and segmentations, which LabelMe and the Lotus Hill dataset already offered, were left as future work.
• A proxy for diversity. In our reading, compressed size of an average image measures how much structure survives averaging, not diversity directly, and depends on how many images are averaged.
• Inherited vocabulary and search bias. Yang et al. (2020) examined the 2,832 synsets in the full ImageNet’s person subtree, judged 1,593 potentially offensive, and found only 158 of the remaining 1,239 to be visual. Among the causes they name are WordNet’s dated vocabulary, the aim of illustrating every synset, and search engines that surface stereotyped images even for concepts that cannot be seen. The adaptive vote count gave such synsets fewer images, but a synset stayed as long as some images were verified.

What Came After

ILSVRC turned a 1,000-class subset of ImageNet into the standard image classification benchmark and added single-object localization and, from 2013, object detection tasks with boxes collected by later crowdsourcing pipelines (Russakovsky et al., 2015).