Caltech-101

Caltech-101 is a collection of web images of 101 object categories, from accordions and airplanes to wrenches and yin-yang symbols, plus a background class, gathered in September 2003 by Fei-Fei Li, Marco Andreetto and Marc’Aurelio Ranzato in Pietro Perona’s group at the California Institute of Technology. Fei-Fei, Fergus and Perona introduced it at the 2004 CVPR Workshop on Generative-Model Based Vision, in a paper on learning object categories from a few examples. Earlier experiments on object categories had used one to six categories, and Caltech-101 soon became the de facto standard for multi-class image classification (Ponce et al., 2006). Its successor, Caltech-256 (Griffin, Holub and Perona), was released in 2006 and described in a 2007 technical report. Both are remembered as much for their protocol, a fixed number of training images per class with accuracy averaged over classes, and for the biases later work found in them, as for the images.

Purpose

Fei-Fei, Fergus and Perona (2004) wanted to test whether priors learned from a few unrelated categories let a generative model learn a new category from one to a few images. Earlier work on object categories had been tested on one or two categories, or at most four (Weber et al.) and six (Fergus et al.); the authors counted a dataset fifteen times larger among their paper’s main contributions. A journal version with the same title and abstract appeared in Computer Vision and Image Understanding (received in 2005, published in 2007). The authors’ TPAMI paper on one-shot learning (2006) also describes the dataset and its annotation, and the official page asks users to cite it when they use the annotations.

Caltech-256 had a different aim. By 2007, Griffin, Holub and Perona wrote, performance above 60% on Caltech-101 had become commonplace, and they wanted a set that would challenge the methods of the day: more categories, at least 80 images in each, no rotation artifacts, and a larger background class for testing whether an algorithm can tell objects from clutter.

Dataset Size

The counts in the papers and on the official pages differ slightly, and none of them states a total for the 101 object categories alone (subtracting the report’s 467 background images from its total gives 8,677):

Version	Categories	Images	Images per category (min / median / mean / max)	Source
Caltech-101	101 + background	9,144	31 / 59 / 90 / 800	Griffin et al. (2007), Fig. 2
Caltech-256	256 + clutter	30,607	80 / 100 / 119 / 827	Griffin et al. (2007), Fig. 2

The report’s totals and statistics include the background classes, which it counts separately as 467 images for Caltech-101 and 827 for Caltech-256 (its Fig. 4); the 827-image maximum of Caltech-256 is its clutter class. Zeiler and Fergus (2014) also counted 9,144 Caltech-101 images. The original Caltech-101 page describes “about 40 to 800 images per category”, most with about 50, while the TPAMI paper’s figure caption says each category contains between 45 and 400 images. The smallest category has 31 images; Pinto, Cox and DiCarlo (2008) name it as inline skate. The largest object categories are airplanes (800), motorbikes (798) and faces-easy (435) in the report’s count; the original page lists airplanes and motorcycles at 800+ and faces at 400+.

Caltech-101 images are roughly 300 × 200 pixels according to the official page, and were scaled to about 300 pixels wide during collection (Fei-Fei et al., 2004). Caltech-256 images vary more: their mean size, measured as $\sqrt{\text{width} \cdot \text{height}}$, is 351 pixels and their mean aspect ratio 1.17 (Griffin et al., 2007, Fig. 3). The CaltechDATA records (as of October 2026) distribute Caltech-101 as a 137 MB zip archive and Caltech-256 as a 1.2 GB tar archive.

Classes

Caltech-101. The 2004 paper says the category names were chosen by flipping through Webster’s Collegiate Dictionary and picking categories that came with a drawing; Google Image Search then supplied as many images as possible for each, and two graduate students not involved in the experiments removed irrelevant ones (a zebra-patterned shirt in the zebra category). The TPAMI paper adds that the dictionary supplied 97 new categories, which were combined with the four earlier Caltech categories (faces, motorbikes, airplanes and spotted cats) to make 101. The categories range over animals (okapi, gerenuk, dalmatian), vehicles, musical instruments, household objects, buildings (pagoda, minaret) and cartoon characters (Snoopy, Garfield).

The background class, a folder named “BACKGROUND_Google” in the archive as torchvision’s loader reads it, holds images returned by a Google search for the keyword “things” (TPAMI paper; Griffin et al., 2007). Such images can contain objects of the 101 categories, as Griffin et al. point out. Papers differ on whether it counts as a class: Pinto et al. (2008) describe a 102-category task, while torchvision’s loader (v0.20.0) drops the folder, which its source calls “not a real class”.

Caltech-256. Griffin et al. asked a small group of vision dataset users for the names of about 300 object categories and downloaded images for each from Google and PicSearch, keeping images at least 100 pixels on each side; this gave about 400–600 images per category. Duplicates were removed by matching SIFT descriptors. Four people rated each image as good (a clear example), bad (confusing, occluded, cluttered, artistic, or with the object occupying only a small part of the image) or not applicable, and only good images were kept. Of 304 candidate categories, 48 had fewer than 80 good images and were dropped. The largest 30 Caltech-101 categories were carried over, with images added where needed to reach 80. Categories 251 to 256 (airplanes-101, car-side-101, faces-easy-101, greyhound, tennis-shoe and toad) are deliberately close to others (fighter-jet, car-tire, people, dog, sneaker and frog), to test whether a classifier generalizes across near-synonyms.

Category 257 is clutter: 947 crops of photographs by Stephen Shore, cut to match the size distribution of the other images, from which crops that showed Caltech-256 objects such as people and cars were removed, leaving 827. Griffin et al. judged it an improvement on the Google “things” class and on Caltech-6’s background of photographs taken around Caltech, but noted that grass, brick and clouds seem over-represented.

Annotation Types

• One category label per image, given by the folder the image is in. The object of interest typically fills most of the image and is roughly centered (Ponce et al., 2006).
• Object outlines (Caltech-101 only). The official page says the authors “carefully clicked outlines” of each object, distributed as Annotations.tar with a MATLAB viewer. The TPAMI paper describes the annotation as the location and contour of each object, made by two people, and used it to measure how many detected features fell on the object. torchvision’s loader reads the contour from each annotation file, and Torralba and Efros (2011) used Caltech-101 among datasets that provide object bounding boxes.
• Caltech-256 has no object annotations, so its authors note that it offers no ground truth for segmentation.

Splits

Neither dataset has an official training/test split. Instead, users draw a fixed number of training images per class at random, test on images from the rest, and repeat the draw several times. The cross-validation article discusses this kind of repeated random subsampling. The protocol and its variants are described under Evaluation.

Tasks

• Multi-class object recognition: assign each image one of the 101 (or 256) categories. This became the main use.
• Learning from few examples: the original purpose, testing how performance grows from one or a few training images per class.
• Object-versus-background detection: the 2004 paper’s task, deciding for each category whether an image contains the object, and Caltech-256’s background-rejection benchmark.
• Transfer evaluation: measuring features learned on other data, such as ImageNet, by training a classifier on 15 or 30 Caltech images per class (transfer learning).

Griffin et al. (2007) warned that both sets are unsuited to localization experiments, because objects tend to be centered.

Evaluation

The protocol settled over the dataset’s first three years.

2004: detection with ROC areas. For each category, Fei-Fei et al. drew $N = 0, 1, 3, 6$ or $15$ training images at random, then 50 test images from the rest (all of the rest when fewer than 50 remained), and scored an object-present/absent decision against 50 background images by the area under the ROC curve, repeating each experiment 10 times. With one training image, the Bayesian method averaged 71% over the 101 categories.

2006: multi-class discrimination. In the TPAMI paper the authors learned a model for every category, assigned each test image to the model with the highest likelihood, repeated the draw 50 times, and averaged the diagonal of the resulting $101 \times 101$ confusion table. This gave 10.4%, 13.9% and 17.7% for 3, 6 and 15 training images, against about 1% for random guessing.

The official recommendations. The original page (last updated in April 2006) asked users to train and test on a fixed number of images per class, repeat with different random selections to obtain error bars, and keep track of which images were used and misclassified. It listed 1, 3, 5, 10, 15, 20 and 30 as popular training sizes and 20 and 30 as popular test sizes. A note of 17 March 2005 added the rule that defines the benchmark: the categories with the most images (airplanes, motorcycles and faces) are somewhat easier, so aggregate results must be normalized across categories, either by testing on all available images and averaging the error rate over categories, or by testing on the same number of images per category. An overall error rate over unequal numbers of test images per category, it warned, will tend to be too optimistic.

The standard protocol. The form most papers followed is the one Lazebnik, Schmid and Ponce (2006) describe, after Grauman and Darrell and J. Zhang et al.: train on 30 images per class (or 15), test on the rest, at most 50 per class, repeat ten times with different random draws, record the mean of the per-class recognition rates for each run, and report the mean and standard deviation over runs. With $K$ classes and $\text{acc}_c$ the fraction of class $c$’s test images classified correctly, the score of a run is

$$\text{mean per-class accuracy} = \frac{1}{K} \sum_{c=1}^{K} \text{acc}_c ,$$

the balanced accuracy, or equivalently the mean of the diagonal of the row-normalized confusion matrix. Because the smallest class has 31 images, 30 is the largest training size that leaves every class a test image, and with 30 training images that class is tested on a single image. Lazebnik et al. noted that the percentage of all test images classified correctly can be biased, because some of the easiest classes are disproportionately large. Papers still differ in the details, which makes numbers comparable only with care: the test cap (50, or 30 in Pinto et al., 2008), the number of repetitions (ten in Lazebnik et al., five folds in Zeiler and Fergus, 2014), and whether the background class is included.

Caltech-256. Griffin et al. (2007) proposed $N_\text{train} = 5, 10, 15, 20, 25, 30$ and $40$ with $N_\text{test} = 25$ per class, so that every class weighs equally, averaged over repeated draws (ideally at least ten). They suggested removing only the clutter class and reporting the mean of the diagonal of the $256 \times 256$ confusion matrix, which they found typically within one percent of the $250 \times 250$ result without the six closely related categories, and asked authors to publish the confusion matrix itself. Later work, such as Zeiler and Fergus (2014), used 15, 30, 45 and 60 training images per class. The report also defined two secondary benchmarks: a “generality” score from the confusion between the six near-synonym pairs, and an ROC curve for rejecting clutter images, where a false positive is a clutter image classified as an object.

Why the average matters

The script below builds a synthetic dataset with Caltech-101’s shape: 24 classes of rendered polygons, four large classes (800, 798, 435 and 435 images) and twenty small ones (31 to 120 images). As the 2005 note on the official page describes, the large classes are the easy ones: their polygons are centered at a fixed size and pose on a clean background, while the small classes vary in position, size and rotation and carry two clutter bars. A ridge-regression classifier on raw pixels is trained on 30 images per class and tested under both versions of the protocol, on all remaining images and on at most 50 per class, over 10 random splits.

import numpy as np
from sklearn.linear_model import RidgeClassifier

S = 32                                    # image side in pixels
SIZES = [800, 798, 435, 435] + [31, 31, 34, 38, 40, 42, 45, 47, 50, 52,
                                55, 59, 62, 65, 70, 78, 85, 98, 110, 120]   # images per class
K = len(SIZES)                            # 24 classes of rendered shapes
yy, xx = np.mgrid[:S, :S] / (S - 1) * 2 - 1   # pixel coordinates in [-1, 1]

def render(c, rng, aligned):
    """Class c is a regular polygon with 3 + c % 6 sides, drawn filled or as an outline,
    at one of two orientations. Aligned images are centred at a fixed scale and pose
    on a plain background; the others vary in position, scale and rotation, with clutter."""
    n = 3 + c % 6
    outline, turn = (c // 6) % 2 == 1, (c // 12) * np.pi / n
    if aligned:
        cx, cy, r, rot = 0.0, 0.0, 0.6, turn
    else:
        cx, cy = rng.uniform(-0.1, 0.1, 2)
        r, rot = rng.uniform(0.5, 0.7), turn + rng.normal(0, 0.1)
    ang = np.arctan2(yy - cy, xx - cx) - rot
    rho = np.hypot(xx - cx, yy - cy) * np.cos(np.pi / n) / np.cos((ang % (2 * np.pi / n)) - np.pi / n)
    img = (np.abs(rho - r * 0.85) < 0.08) if outline else (rho < r)
    img = img.astype(float)
    if not aligned:                       # clutter: two random bars
        for _ in range(2):
            x0, y0 = rng.integers(0, S - 8, 2)
            img[y0:y0 + rng.integers(2, 8), x0:x0 + rng.integers(2, 8)] += 0.8
    return img + 0.3 * rng.normal(size=(S, S))

def make_dataset(rng):
    X, y = [], []
    for c, n in enumerate(SIZES):
        aligned = n > 400                 # the large classes are the tidy, aligned ones
        X += [render(c, rng, aligned).ravel() for _ in range(n)]
        y += [c] * n
    return np.array(X), np.array(y)

def split(y, rng, n_train=30, max_test=None):
    """Caltech protocol: n_train random images per class for training, the rest
    (or at most max_test of the rest) for testing."""
    train, test = [], []
    for c in range(K):
        idx = rng.permutation(np.flatnonzero(y == c))
        train += list(idx[:n_train])
        test += list(idx[n_train:] if max_test is None else idx[n_train:n_train + max_test])
    return np.array(train), np.array(test)

def scores(y_true, y_pred):
    overall = np.mean(y_true == y_pred)                              # every image counts once
    per_class = np.array([np.mean(y_pred[y_true == c] == c) for c in range(K)])
    return overall, per_class.mean(), per_class                      # every class counts once

X, y = make_dataset(np.random.default_rng(0))
print(f"{K} classes, {len(y)} images, {min(SIZES)}-{max(SIZES)} per class; chance = {1 / K:.3f}")
for name, max_test in [("30 train, rest test   ", None), ("30 train, <=50 test   ", 50)]:
    runs, per_class_runs = [], []
    for seed in range(10):                                           # 10 random splits
        tr, te = split(y, np.random.default_rng(100 + seed), max_test=max_test)
        clf = RidgeClassifier(alpha=100.0).fit(X[tr], y[tr])
        overall, mean_pc, pc = scores(y[te], clf.predict(X[te]))
        runs.append((overall, mean_pc)); per_class_runs.append(pc)
    r, pc = np.array(runs), np.array(per_class_runs)
    big = np.isin(y[te], range(4)).mean()
    print(f"{name} test images {len(te):5d} ({big:.0%} from the 4 large classes)  "
          f"overall {r[:, 0].mean():.3f} +- {r[:, 0].std():.3f}   "
          f"mean per-class {r[:, 1].mean():.3f} +- {r[:, 1].std():.3f}")
    if max_test is None:
        print(f"    per-class accuracy, mean over splits: 4 large classes {pc[:, :4].mean():.3f}, "
              f"20 small classes {pc[:, 4:].mean():.3f}")
        print(f"    spread over splits of one 31-image class (1 test image): "
              f"{np.round(pc[:, 4], 2)}")

Output (NumPy 2.3.5, scikit-learn 1.7.2):

24 classes, 3680 images, 31-800 per class; chance = 0.042
30 train, rest test    test images  2960 (79% from the 4 large classes)  overall 0.871 +- 0.004   mean per-class 0.530 +- 0.030
    per-class accuracy, mean over splits: 4 large classes 1.000, 20 small classes 0.435
    spread over splits of one 31-image class (1 test image): [0. 1. 1. 0. 1. 0. 0. 0. 0. 0.]
30 train, <=50 test    test images   719 (28% from the 4 large classes)  overall 0.569 +- 0.015   mean per-class 0.528 +- 0.031

Testing on all remaining images, four of the 24 classes supply four fifths of the test set, and they are the ones the classifier gets right, so the overall accuracy (0.871) says little about the twenty classes that average 0.435. This is the optimism the 2005 note warned about. Capping the test set at 50 images per class shrinks the gap but does not close it, because most of the small classes still contribute fewer than 50 test images each; overall accuracy remains about four points above the mean per-class value. Mean per-class accuracy is the same under both versions (0.530 and 0.528), since each class’s accuracy is estimated without reference to the others’ sizes, which is why papers using different test caps could still be compared on it. The spread line shows the cost: a 31-image class is scored on one image per split, so its accuracy is 0 or 1, and only the average over repeated splits makes it meaningful. In our reading this is why the protocol asks for repeated draws and a standard deviation: across these 10 splits the mean per-class value itself has a standard deviation of 0.03.

License

As of October 2026, both CaltechDATA records, Caltech 101 and Caltech 256, published on 6 April 2022 as version 1.0, state the license Creative Commons Attribution 4.0 International (CC BY 4.0). The records say nothing about the copyright of the images themselves, which were downloaded from Google Image Search and, for Caltech-256, PicSearch, so the terms for reusing the images are unclear. The original Caltech pages stated no license and asked users to cite the 2004 paper (images) and the TPAMI paper (images and annotations) for Caltech-101, and the technical report for Caltech-256. The report’s own CaltechAUTHORS record carries a separate statement for the document: no commercial reproduction, distribution, display or performance rights are provided.

Limitations and Bias

The original page itself noted that most images have little or no clutter, objects tend to be centered, and most are shown in a stereotypical pose. Later work measured how much these regularities matter.

• Centered, aligned objects. Antonio Torralba averaged the images of each category, and many categories remain recognizable from the average alone (official page; Ponce et al., 2006, who resized the images to 150 × 128 before averaging 100 classes). If position and pose varied, the averages would approach a uniform field. The alignment was partly deliberate: in categories whose objects appeared in two mirror-image views, such as motorbikes, airplanes and cannons, images were flipped by hand so that all faced the same way (Fei-Fei et al., 2004; TPAMI paper). Griffin et al. (2007) note that all Caltech-101 airplanes face right, and did not align Caltech-256.
• Rotation artifacts. Categories with mostly vertical structure were rotated to an arbitrary angle during collection, because the constellation model ordered its parts by their horizontal position; the 2004 paper calls this a matter of programming simplicity. The rotation left black corners in the images, and some methods exploited them: Lazebnik et al. (2006) reported 97.6% on minaret, a class “dominated by rotation artifacts”, and Griffin et al. collected a new minaret category for Caltech-256 for this reason.
• Class size and difficulty. The largest classes (airplanes, motorbikes, faces, cars) are among the easiest (official page; Pinto et al., 2008), so overall accuracy rewards them, as the example above shows. Caltech-256 carried over the 30 largest Caltech-101 categories, and in the report’s per-category results for spatial pyramid matching, seven of the categories highlighted as best carry their -101 suffix (Fig. 8).
• Ancillary cues. A classifier that sees only each image’s width and height identified the correct Caltech-256 category 3.7 ± 0.6% of the time with 30 training images per class, where random guessing gives about 0.4% (1/256; the report says less than 0.25%); the authors call this a danger in many recognition datasets (Griffin et al., 2007, Section 4.1).
• A test that easy models pass. Pinto, Cox and DiCarlo (2008) built a deliberately simple “V1-like” model, normalized and thresholded Gabor filter outputs fed to a linear SVM. With a few cheap extra features (intensity histograms) it reached 61% and 67% mean per-class accuracy with 15 and 30 training images per class, and somewhat less without them (their Fig. 1), comparable to or better than published methods, while raw pixels gave 16.1% and 17.3%. The same model fell to chance on a two-class task of rendered cars and planes once position, scale and pose varied. The authors concluded that Caltech-101 did not engage the core difficulty of recognition, tolerance to such variation.
• Saturation and tuning. Ponce et al. (2006), with the dataset’s role as a de facto standard in view, observed that results had risen from 17% in 2004 to about 60% in 2006, that methods with very different models reached similar scores, and that long public availability lets researchers tune parameters to the dataset; Caltech-101 might, they wrote, be reaching the end of its useful shelf life. Torralba and Efros (2011) cited it as an example of a dataset that became a closed world of its own: a classifier could tell Caltech-101 images from those of other datasets, and a “car” classifier trained on Caltech-101 scored 96.9 average precision on its own test images but 26.7 on average over five other datasets (their Table 1).
• Overlap with web-scale training sets. Because the images come from web search, they can reappear in later web datasets. Zeiler and Fergus (2014) found 44 Caltech-101 and 243 Caltech-256 images in their ImageNet training set and removed them before retraining. Pinto et al. (2008) mention that an earlier version of Caltech-101 contained duplicates.
• Small and single-object. With 31 to 800 images per class and a large, centered object in most images, Caltech-101 does not test detection in cluttered scenes. PASCAL VOC chose images without biases such as centered, well-lit objects, and Torralba and Efros (2011) describe the scene datasets MSRC and LabelMe as a reaction against the Caltech-like single centered object.

Common Uses

• Benchmark for multi-class recognition in the period of bags of visual words, kernel methods and support vector machines. Methods were compared by curves of mean per-class accuracy against the number of training images per class.
• Few-shot learning, its original purpose, with 1 to 15 training images per class.
• Transfer evaluation of convolutional network features: a linear classifier trained on 15 or 30 images per class on top of frozen ImageNet features, as in DeCAF and the VGG networks’ generalization experiments (transfer learning, VGG paper).
• Caltech-256 for the same purposes at a harder level, in later work with up to 60 training images per class (Zeiler and Fergus, 2014).

Historical Milestones

• 2004–2006. The dataset and the generative one-shot model, whose multi-class version scored 17.7% with 15 training images per class (TPAMI paper).
• 2005–2006. Ponce et al. (2006) noted that methods using SVMs did well, including the two best in their comparison. Lazebnik et al. (2006) cite 43% for Grauman and Darrell’s pyramid match kernel with 30 training images, and 48% for Berg et al. with 15. Spatial pyramid matching reached 64.6 ± 0.8% with 30 training images (Lazebnik et al., 2006, Table 2) and 56.4% with 15 (their text), and in the same proceedings H. Zhang et al.’s SVM-KNN reported 66.2 ± 0.5% and 59.1 ± 0.6% (as Lazebnik et al. cite it). The bag of visual words article describes spatial pyramids.
• 2007. Caltech-256. Griffin et al.’s own spatial pyramid implementation scored 67.6% on Caltech-101 and 34.1% on Caltech-256 with 30 training images per class (Section 5 and Fig. 12; the sentence in Section 4.3 gives the two numbers in the opposite order).
• 2008. Pinto et al.’s V1-like baseline matched the state of the art, a result read as a warning about the benchmark rather than as progress.
• 2014. Zeiler and Fergus kept the layers of an ImageNet-trained network fixed and trained only a softmax classifier on Caltech images: 83.8 ± 0.5% with 15 and 86.5 ± 0.5% with 30 training images per class, against 46.5 ± 1.7% for the same network trained from scratch on 30 images per class (ECCV version, Table 3; Table 4 in arXiv v3), and 74.2% on Caltech-256 with 60 training images per class against 55.2% for the best earlier method (Table 4; Table 5 in arXiv v3). The text says the 30-image Caltech-101 result beats Bo et al. (2013) by 2.2%, but the table lists Bo et al. at 81.4, 5.1 points lower; the 2.2-point gap is to Yang et al. (2009), at 84.3. The layer-by-layer table (Table 6; Table 7 in arXiv v3) gives 85.4 ± 0.4% for a softmax on layer-7 features rather than 86.5. With the DeCAF features of Donahue et al., which gave a similar Caltech-101 result the same year (see transfer learning), this showed features learned on ImageNet overtaking hand-designed pipelines; the arXiv version adds that the result questions the usefulness of benchmarks with fewer than $10^4$ training images.

Related Datasets

Caltech-4 (faces, motorbikes, airplanes and spotted cats in the TPAMI paper; Ponce et al., 2006, list cars instead of spotted cats), used in the earlier constellation-model work, supplied four of Caltech-101’s categories; Caltech-6, as Griffin et al. (2007) describe it, has 3,738 images of cars, motorcycles, airplanes, faces and leaves. Caltech-256 extends Caltech-101 and shares its 30 largest categories. The Caltech vision group’s other datasets include Caltech Pedestrians and CUB-200-2011 birds. PASCAL VOC (2005–2012) moved recognition to cluttered scenes with several objects, and ImageNet, whose paper compared its scale and diversity directly with Caltech-101 and Caltech-256 (ImageNet paper), replaced both as the large-scale classification benchmark. Torralba and Efros (2011) compared Caltech-101 and Caltech-256 with ten other datasets in their study of dataset bias.

Resources

• Caltech 101 on CaltechDATA: images and annotations, license, and the related publication.
• Caltech 256 on CaltechDATA: images and a link to the technical report.
• Caltech Vision Lab datasets page: the lab’s current list, which links to the CaltechDATA records.
• Original Caltech 101 page (archived): the 2005 notes on reporting error rates and the list of early papers.
• Caltech-256 technical report on CaltechAUTHORS.