Caltech-UCSD Birds (CUB-200)

Caltech-UCSD Birds, usually shortened to CUB, is a collection of photographs of 200 bird species gathered from Flickr by groups at the California Institute of Technology and the University of California, San Diego. The first release, CUB-200 (Welinder, Branson, Mita, Wah, Schroff, Belongie and Perona, 2010), had 6,033 images with boxes, rough outlines and attribute labels. The second, CUB-200-2011 (Wah, Branson, Welinder, Perona and Belongie, 2011), roughly doubled the images per species to 11,788 and added 15 part locations per image, each the median of five crowd workers’ clicks. The 2011 report calls the dataset “Birds-200” in its text. CUB-200-2011 is the release used in the results that the articles linked here collect. It is a standard benchmark of fine-grained image classification, and its attributes and class-disjoint splits serve zero-shot recognition, metric learning and few-shot recognition. This article covers the two releases, their annotations and files, the licence statements, and the part-localization measures, which no task article describes in detail.

Purpose

The 2010 report set out to support subordinate categorization: telling apart categories within one basic-level class, which the popular datasets of the time, such as PASCAL VOC and Caltech-101, did not allow. Its authors argued that few people can name more than a handful of birds without a field guide, so a recognition system would be useful here even if it were not perfect. They pictured a user photographing a bird and querying a knowledge base such as Wikipedia (Welinder et al., 2010, §1).

The 2011 report broadened the aim. One domain was to be annotated in depth (boxes, parts and attributes) to build up the kind of research depth that pedestrians and faces had. It named four research uses (Wah et al., 2011, §3):

• subordinate category recognition, where bag-of-words representations popular on Caltech-101 are often less successful;
• multi-class detection with shared part models, since all 200 species are annotated with the same parts;
• attribute-based recognition with localized attributes, which earlier attribute datasets such as Animals with Attributes lacked;
• crowdsourcing research, using the released annotations of several workers per training image.

Dataset Size

Release	Images	Species	Annotations	Source
CUB-200 (2010)	6,033	200	box, rough outline, 25 multi-valued attributes	Welinder et al. (2010)
CUB-200-2011	11,788	200	box, 15 part locations with visibility, 312 binary attributes	Wah et al. (2011); dataset page

The 2011 report’s Fig. 6 gives the statistics as histograms only. Most classes have 60 images and most images are roughly 500 × 500 pixels, and the bird’s box covers a varying fraction of the image. Liu and Belhumeur (2013, §5.1) describe “about 60 images per species”. On CaltechDATA, as of November 2026, the images and annotations are one archive, CUB_200_2011.tgz, of 1,150,585,339 bytes. The segmentation masks are a separate archive of 39,272,883 bytes. The dataset page rounds these to 1.1 GB and 37 MB. This article was written without opening either archive, so it does not describe the files inside.

Classes

The 200 classes are bird species, “mostly North American” (Welinder et al., 2010). The 2010 report describes how they were chosen:

1. A list of 278 species was compiled from an online field guide.
2. Species without a Wikipedia article or without images on it were dropped.
3. The remaining names were sent to Flickr as queries, keeping up to 40 images per species.
4. Names that returned fewer than 20 images were removed, leaving 223 species.
5. After the images were filtered (below), species with fewer than 20 images were excluded, leaving 200.

The 2011 report adds that each species is linked to a Wikipedia article and placed in the scientific classification (order, family, genus, species). Some class pairs are hard even for experts: the 2011 report calls some pairs nearly indistinguishable even for expert bird watchers, and gives many sparrow species as an example of visually similar ones. The fine-grained image classification article gives an example of a confused pair and an estimate of label noise.

Annotation Types

All annotations were collected on Amazon Mechanical Turk (MTurk).

Filtering (2010). Two workers saw each Flickr image next to the species’ Wikipedia exemplar. Each drew a rough outline and rated the image as “same”, “similar”, “different” or “difficult” to compare. An image was kept only if both workers chose “same” or “similar” and the boxes around their outlines overlapped. The authors then checked the survivors, so three people reviewed each image (Welinder et al., 2010, §2).

Boxes and outlines. In 2010, workers drew a rough outline with a thick brush along the bird’s boundary, and the box was deduced from the outline. This was chosen over detailed segmentation because it is faster (§3). The caption of the report’s Fig. 2 differs, saying workers were asked for either a bounding box or an outline. In 2011, boxes were collected with a dedicated box interface (Wah et al., 2011, Fig. 4). Segmentation masks for CUB-200-2011, provided by Ryan Farrell, are distributed as a separate CaltechDATA record.

Attributes. The 2010 release asked for 25 multi-valued attributes from an online bird guide, such as bill shape, wing colour and size. Each answer carried a confidence of “definitely”, “probably” or “guessing”, and there were five annotations per image and attribute from 1,577 workers in total (Welinder et al., 2010, §3, Table 1). The report’s introduction calls them “binary attribute annotations”, against “multi-valued” in Table 1. The 2011 release has 312 binary attributes in 28 groups, selected on the basis of the same site, whatbird.com, that the 2010 report cites. For example, the group “belly color” has 15 colour choices. Most attributes describe the colour, pattern or shape of a part, and the report’s Fig. 2(b) associates each group with a part or with the head, body or whole bird. The 2011 report does not say how many workers answered each attribute question.

Parts (2011). Fifteen parts were marked by pixel location and visibility: back, beak, belly, breast, crown, forehead, left eye, left leg, left wing, nape, right eye, right leg, right wing, tail and throat (Fig. 2(a)). The interface asked for one part at a time, with a diagram of the part and a box to tick if “more than half of the part is not visible” (Fig. 3). Five workers labelled each image, and the ground-truth location is the median of their five locations (Wah et al., 2011, §2). The report does not say how their visibility answers were combined. The report’s Fig. 6(d) plots the average labelling time per part. Liu and Belhumeur (2013, §4) counted 736 different combinations of visible parts across the dataset, a measure of how much pose and occlusion vary.

Releasing annotations from several workers per training image was deliberate: the report hoped it would support research on combining crowd annotations. The image annotation article discusses consensus from several annotators.

Splits

CUB-200 (2010). Flickr results for a species often come from a few photographers, who may shoot one bird many times in quick succession. Near-identical images would then let a nearest-neighbour method perform “artificially well”. To prevent this, the authors chose a date for each species that split its images into roughly equal halves: images before the date are for training and images after it are for testing. They “strongly suggest” always using the dataset this way and published the split on the project website (Welinder et al., 2010, §4).

CUB-200-2011. The release ships its own train/test split. Liu and Belhumeur (2013, §5.1) “use the train/test split provided in the dataset”, with roughly 30 training images per species. The image counts of that split (5,994 training and 5,794 test images) are given, with their sources, in fine-grained image classification. The 2011 report does not say how its split was drawn.

Derived class splits. Other fields split the classes rather than the images. Zero-shot recognition describes the 150/50 class split and the later “proposed splits” of Xian et al. Metric learning describes the split that trains on the first 100 classes and tests on the other 100 with Recall@K.

Tasks

The 2011 report defines three benchmarks (Wah et al., 2011, §4):

1. Localized species categorization. Given the ground-truth part locations, assign one of the 200 classes.
2. Part localization. Given the full, uncropped image, predict the location and visibility of each part.
3. Species categorization/detection. Given only the uncropped image, assign the class by any method. The authors expected that “some degree of localization” would be needed.

The classification benchmarks are covered in fine-grained image classification, where results are reported with or without the box at test time and with or without parts in training. Part localization is a keypoint detection problem with a visibility label per part, and its measures are below.

Evaluation

Classification on CUB is scored by accuracy on the test split. The part-localization benchmark has two sets of measures in the literature:

• the 2011 report’s normalized error;
• the four measures of Liu and Belhumeur (2013), among them PCP.

As of November 2026, no official evaluation code for part localization is linked from the dataset page, from either CaltechDATA record or from either report. Both measures are therefore known only from their prose definitions, and several details are left open.

The 2011 report’s normalized error

The report measured “the distance between predicted part locations and ground truth, normalized on a per-part basis by the standard deviation over part click locations for multiple MTurk users”. It bounded each part’s error at 5 standard deviations, and the same 5 is charged when a part’s visibility is misclassified (§4). In symbols, for part $p$ of an image with predicted location $\hat{\mathbf{x}}_p$, ground truth $\mathbf{x}_p$, and visibilities $\hat v_p$ and $v_p$:

$$e_p = \begin{cases} \min\!\left( \lVert \hat{\mathbf{x}}_p - \mathbf{x}_p \rVert / \sigma_p,\; 5 \right) & \hat v_p = v_p = 1, \\ 5 & \hat v_p \ne v_p, \\ 0 \text{ or not counted} & \hat v_p = v_p = 0, \end{cases}$$

and the score is the mean of $e_p$. The report’s baseline, HOG part detectors in a mixture of tree-structured pictorial structures, had an average error of 2.47 standard deviations. The report adds that “an average MTurk user should be off by 1 standard deviation”.

Three details are not stated:

• how $\sigma_p$ is computed: per axis or as a distance, about the workers’ mean or about the median (the example below does not print a per-axis normalization, so its effect is not shown);
• whether parts invisible in both prediction and ground truth count as zero or are left out;
• how “an average MTurk user” was scored.

Liu and Belhumeur’s measures

Liu and Belhumeur (2013, §5.1) report four numbers:

• PCP (Percentage of Correctly estimated Parts). A part is correct if it is visible and within 1.5 standard deviations “of a MTurk user’s click”, or if both the estimate and the ground truth are invisible.
• AE (Average Error). The distance normalized per part by the standard deviation and bounded at 5, averaged over parts visible in both prediction and ground truth. Unlike the report’s error, a visibility mistake costs nothing here.
• FVR (False Visibility Rate). The percentage of parts incorrectly estimated as visible.
• FIR (False Invisibility Rate). The percentage of parts incorrectly estimated as invisible.

The authors write that AE best describes the accuracy of predicted locations. The open points are which click serves as the reference in PCP (the median ground truth, one worker, or any worker) and which parts form the denominator of FVR and FIR.

This PCP is not the limb-based PCP of human pose estimation, which tests a limb’s two endpoints against half the limb’s length. It is also not the PCK family of keypoint detection, whose tolerance is a fraction of the object’s size or of a body part rather than of the annotators’ spread.

Example: the measures on synthetic birds with simulated annotators

The example applies both sets of measures to synthetic data whose truth is known exactly. No image of a real bird and no dataset file is used, and nothing is trained.

Birds. Each bird is a body ellipsoid with semi-axes 0.50, 0.22 and 0.25 body units, plus a head sphere of radius 0.14. CUB’s 15 part names sit as points on the surfaces, or just outside them for the beak, tail and legs. The model frame has $x$ toward the beak, $y$ to the bird’s left and $z$ up. Each bird gets:

• a random yaw;
• a camera elevation between −15° and 45°;
• a scale of 40 to 80 pixels per body unit.

It is projected orthographically by a camera on the $+y$ side into a 160 × 160 frame with image $y$ pointing down. A part is visible exactly when no solid lies in front of it along the viewing ray.

Annotators. Five simulated workers each flip a part’s true visibility with probability 0.05. Each clicks every part it reports visible, at the true position plus Gaussian noise with a per-axis standard deviation of $s_p$ times the bird’s scale. $s_p$ is 0.02 for the beak and eyes and 0.08 for back, breast and belly. Following the report, the ground-truth location is the median of the clicks, taken coordinate-wise (our choice). Visibility is the majority of the five answers, which is our choice. The uniform visibility errors and Gaussian clicks are our simplifications of real workers. The image annotation article measures how such a consensus compares with the truth; here the consensus simply is the ground truth, as in CUB.

$\sigma_p$. Computed from the clicks in two ways:

• S_sd, the pooled per-axis sample standard deviation of the clicks about each image’s mean click, our literal reading of the report;
• S_md, the mean distance of a click from the ground-truth median.

Localizers. These are error models, not detectors:

• A adds Gaussian noise of 1.5 $s_p$ times the scale to the true positions and flips the true visibility with probability 0.10;
• B has A’s locations but predicts every part visible;
• C doubles A’s location noise (the same draws) and keeps A’s visibility.

The “Human” row scores each worker against the dataset’s ground truth, which includes that worker, and averages over the five workers.

Open choices, printed every way.

• The report’s error with both-invisible parts counted as zero (repAE/all) and left out (repAE/excl).
• PCP against three references:
  • the median (PCP/median);
  • the first worker who clicked the part (PCP/one; for humans, the first other worker);
  • any worker’s click (PCP/any; for humans, any other worker).
• FVR and FIR over all parts, and over the ground-truth-invisible (FVR/inv) or ground-truth-visible (FIR/vis) parts.

capped% is the share of both-visible scores bounded at 5. Lines starting with K are checks of the implementation.

The script runs in about half a second with NumPy 2.3.5, and two runs gave identical output.

import os
os.environ["OPENBLAS_NUM_THREADS"] = os.environ["OMP_NUM_THREADS"] = "1"
import sys
import warnings
import numpy as np
warnings.simplefilter("ignore", RuntimeWarning)   # NaN slices: parts no worker clicked

# CUB-200-2011's 15 parts. Model frame: x toward the beak, y the bird's left, z up (body units).
PARTS = ["back", "beak", "belly", "breast", "crown", "forehead", "left eye", "left leg",
         "left wing", "nape", "right eye", "right leg", "right wing", "tail", "throat"]
BODY = (np.zeros(3), np.array([0.50, 0.22, 0.25]))           # ellipsoid: centre, semi-axes
HEAD = (np.array([0.55, 0.0, 0.22]), np.full(3, 0.14))       # sphere as an ellipsoid
def on(solid, d):                                            # surface point in direction d
    c, a = solid
    d = np.asarray(d, float)
    return c + d / np.sqrt(((d / a) ** 2).sum())
PTS = np.array([on(BODY, [0, 0, 1]), [0.72, 0, 0.20], on(BODY, [0, 0, -1]), on(BODY, [0.8, 0, -0.2]),
                on(HEAD, [0, 0, 1]), on(HEAD, [0.6, 0, 0.8]), on(HEAD, [0.5, 0.75, 0.35]),
                [0.05, 0.08, -0.40], on(BODY, [-0.1, 1, 0.3]), on(HEAD, [-0.7, 0, 0.7]),
                on(HEAD, [0.5, -0.75, 0.35]), [0.05, -0.08, -0.40], on(BODY, [-0.1, -1, 0.3]),
                [-0.68, 0, 0.08], on(HEAD, [0.5, 0, -0.85])])
# Annotator click noise per part (per-axis SD in body units): eyes and beak precise, body regions diffuse.
S_P = np.array([.08, .02, .08, .08, .04, .04, .02, .05, .07, .04, .02, .05, .07, .05, .04])
P, W, Q_FLIP = 15, 5, 0.05

def rotation(yaw, elev):
    cy, sy, ce, se = np.cos(yaw), np.sin(yaw), np.cos(elev), np.sin(elev)
    Rz = np.array([[cy, -sy, 0], [sy, cy, 0], [0, 0, 1]])
    Rx = np.array([[1, 0, 0], [0, ce, se], [0, -se, ce]])    # tilts the back toward the camera (+y)
    return Rx @ Rz

def visible(R):
    """Exact ray cast along y (camera at +y): a part is visible if no solid lies in front of it."""
    X = PTS @ R.T
    front = np.full(P, -np.inf)
    for c, a in (BODY, HEAD):
        A = R @ np.diag(1 / a ** 2) @ R.T
        cc = R @ c
        w = X - cc
        w[:, 1] = 0
        qa, qb, qc = A[1, 1], w @ A[:, 1], np.einsum("pi,ij,pj->p", w, A, w) - 1
        disc = qb ** 2 - qa * qc
        hit = disc >= 0
        t = cc[1] + (-qb + np.sqrt(np.where(hit, disc, 0))) / qa   # nearer root
        front = np.where(hit, np.maximum(front, t), front)
    return X, X[:, 1] >= front - 1e-9

def make_birds(rng, n):
    xy, vis, scale = np.zeros((n, P, 2)), np.zeros((n, P), bool), rng.uniform(40, 80, n)
    for i in range(n):
        X, vis[i] = visible(rotation(rng.uniform(0, 2 * np.pi), np.deg2rad(rng.uniform(-15, 45))))
        xy[i] = 80 + scale[i] * np.c_[X[:, 0], -X[:, 2]]      # 160 x 160 image, y down
    return xy, vis, scale

def annotate(rng, xy, vis, scale):
    """5 workers: flip visibility w.p. 0.05, click reported-visible parts with Gaussian noise."""
    n = len(xy)
    rep = vis[:, None, :] ^ (rng.random((n, W, P)) < Q_FLIP)
    clk = xy[:, None] + rng.standard_normal((n, W, P, 2)) * (S_P * scale[:, None])[:, None, :, None]
    clk[~rep] = np.nan
    gt_vis = rep.sum(1) >= 3                                   # majority of 5 (our choice)
    gt = np.nanmedian(np.where(gt_vis[:, None, :, None], clk, np.nan), axis=1)   # coordinate-wise median
    return rep, clk, gt_vis, gt

def sigmas(clk, gt_vis, gt):
    c = np.where(gt_vis[:, None, :, None], clk, np.nan)
    k = (~np.isnan(c[..., 0])).sum(1)                          # clicks per image and part
    dev2 = np.nansum((c - np.nanmean(c, 1, keepdims=True)) ** 2, axis=(1, 3))
    s_sd = np.sqrt(dev2.sum(0) / (2 * (k - 1).clip(0).sum(0)))  # pooled per-axis SD
    s_md = np.nanmean(np.linalg.norm(c - gt[:, None], axis=-1), axis=(0, 1))   # mean distance to GT
    return s_sd, s_md

def scores(pred, pvis, gt, gt_vis, sig, refs):
    """Report AE (two denominators), Liu AE, PCP@1.5 sigma for each reference, capped share."""
    e = np.minimum(np.linalg.norm(pred - gt, axis=-1) / sig, 5)
    both, none = pvis & gt_vis, ~pvis & ~gt_vis
    rep = np.where(both, e, np.where(none, 0, 5.0))
    pcp = []
    for ref in refs:                                           # ref: (..., k, 2), NaN = no click
        d = np.linalg.norm(pred[..., None, :] - ref, axis=-1) / sig[..., None]
        ok = (d <= 1.5).any(-1)                                # NaN compares False
        pcp.append(100 * np.mean(both & ok | none))
    return (rep.mean(), rep[~none].mean(), e[both].mean(), *pcp,
            100 * np.mean(np.linalg.norm(pred - gt, axis=-1)[both] / np.broadcast_to(sig, e.shape)[both] > 5))

def vis_rates(pvis, gt_vis):
    fv, fi = pvis & ~gt_vis, ~pvis & gt_vis
    return 100 * fv.mean(), 100 * fv.sum() / (~gt_vis).sum(), 100 * fi.mean(), 100 * fi.sum() / gt_vis.sum()

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
rng = np.random.default_rng(SEED)
N = 2000
xy, vis, scale = make_birds(rng, N)
rep, clk, gt_vis, gt = annotate(rng, xy, vis, scale)
s_sd, s_md = sigmas(clk, gt_vis, gt)

# Simulated localizers (error models, not detectors). A and C share visibility flips; B predicts all visible.
flip = rng.random((N, P)) < 0.10
noise = rng.standard_normal((N, P, 2)) * (S_P * scale[:, None])[..., None]
LOC = {"A": (xy + 1.5 * noise, vis ^ flip), "B": (xy + 1.5 * noise, np.ones((N, P), bool)),
       "C": (xy + 3.0 * noise, vis ^ flip)}
first = np.argmax(~np.isnan(clk[..., 0]), axis=1)              # first worker who clicked each part
one = np.take_along_axis(clk, first[:, None, :, None], 1)[:, 0, :, None]   # (N, P, 1, 2)

print(f"Synthetic CUB-style birds: {N} images, {P} parts, {W} workers, seed {SEED}")
X0, v0 = visible(rotation(0.0, 0.0))
print("K2 side view, visible:", {p: int(v0[PARTS.index(p)]) for p in
                                 ("left eye", "right eye", "left wing", "right wing")})
k = (~np.isnan(clk[..., 0])).sum(1)
print(f"K6 fewest clicks on a GT-visible part: {k[gt_vis].min()}")
L, Rr = [PARTS.index("left " + s) for s in ("eye", "wing", "leg")], [PARTS.index("right " + s) for s in ("eye", "wing", "leg")]
mis = sum(int((visible(rotation(a, e))[1][L] != visible(rotation(np.pi - a, e))[1][Rr]).sum())
          for a in np.linspace(0, 2 * np.pi, 360, endpoint=False) for e in np.deg2rad(np.arange(-15, 46, 5)))
print(f"K3 mirror check, left part at yaw a vs right part at yaw pi - a, 4680 poses x 3 pairs: {mis} mismatches")
tv = vis.mean(0)
dz = [(vis[:, l].astype(float) - vis[:, r]) for l, r in zip(L, Rr)]
print("K3 true visible fraction left/right (z of difference): " + ", ".join(
    f"{s} {tv[l]:.3f}/{tv[r]:.3f} ({d.mean() / (d.std(ddof=1) / np.sqrt(N)):+.2f})"
    for s, l, r, d in zip(("eye", "wing", "leg"), L, Rr, dz)))
rms = np.sqrt(np.mean(scale ** 2))
print(f"{'part':10s} true-vis GT-vis  S_sd/(s_p*rms)  S_sd px  S_md px")
for j, p in enumerate(PARTS):
    print(f"{p:10s}  {tv[j]:.3f}  {gt_vis[:, j].mean():.3f}     {s_sd[j] / (S_P[j] * rms):.4f}     {s_sd[j]:6.2f}  {s_md[j]:6.2f}")

refs_loc = [gt[:, :, None], one, clk.transpose(0, 2, 1, 3)]
rows = {}
for name, (pred, pvis) in LOC.items():
    rows[name] = (pred, pvis, refs_loc)
print("\nVisibility       FVR/all  FVR/inv  FIR/all  FIR/vis   (%)")
hv = np.mean([vis_rates(rep[:, w], gt_vis) for w in range(W)], 0)
print("Human (mean of 5) " + "  ".join(f"{v:6.2f}" for v in hv))
for name, (pred, pvis, _) in rows.items():
    print(f"{name:17s} " + "  ".join(f"{v:6.2f}" for v in vis_rates(pvis, gt_vis)))

o = scores(gt, gt_vis, gt, gt_vis, s_sd, [gt[:, :, None]])
print(f"\nK1 oracle (GT as prediction): report AE {o[0]:.3f}, Liu AE {o[2]:.3f}, PCP {o[3]:.2f}%,"
      f" FVR/FIR {vis_rates(gt_vis, gt_vis)[0]:.2f}/{vis_rates(gt_vis, gt_vis)[2]:.2f}%")
for label, sig in (("S_sd (SD of clicks)", s_sd), ("S_md (mean click distance to GT)", s_md)):
    print(f"\nsigma = {label}")
    print("row        repAE/all repAE/excl  LiuAE  PCP/median PCP/one PCP/any  capped%")
    hs = []
    for w in range(W):
        others = np.delete(clk, w, axis=1)
        f2 = np.argmax(~np.isnan(others[..., 0]), axis=1)
        one_o = np.take_along_axis(others, f2[:, None, :, None], 1)[:, 0, :, None]
        hs.append(scores(clk[:, w], rep[:, w], gt, gt_vis, sig,
                         [gt[:, :, None], one_o, others.transpose(0, 2, 1, 3)]))
    out = {"Human": np.mean(hs, 0)} | {n: scores(p, v, gt, gt_vis, sig, r) for n, (p, v, r) in rows.items()}
    for n, s in out.items():
        print(f"{n:9s}   {s[0]:6.3f}    {s[1]:6.3f}    {s[2]:6.3f}   {s[3]:6.2f}   {s[4]:6.2f}  {s[5]:6.2f}   {s[6]:6.2f}")
    d = np.linalg.norm(clk - gt[:, None], axis=-1)[rep & gt_vis[:, None]]
    j = np.broadcast_to(np.arange(P), rep.shape)[rep & gt_vis[:, None]]
    u = np.mean(d / sig[j])
    print(f"Human Liu AE before the cap: {u:.6f} (|x - 1| = {abs(u - 1):.1e})")

Synthetic CUB-style birds: 2000 images, 15 parts, 5 workers, seed 0
K2 side view, visible: {'left eye': 1, 'right eye': 0, 'left wing': 1, 'right wing': 0}
K6 fewest clicks on a GT-visible part: 3
K3 mirror check, left part at yaw a vs right part at yaw pi - a, 4680 poses x 3 pairs: 0 mismatches
K3 true visible fraction left/right (z of difference): eye 0.536/0.531 (+0.31), wing 0.527/0.505 (+1.00), leg 0.934/0.932 (+0.21)
part       true-vis GT-vis  S_sd/(s_p*rms)  S_sd px  S_md px
back        0.753  0.753     1.0071       4.92    5.29
beak        0.703  0.703     1.0051       1.23    1.32
belly       0.239  0.239     1.0255       5.01    5.39
breast      0.395  0.394     0.9933       4.85    5.20
crown       0.761  0.759     0.9907       2.42    2.60
forehead    0.654  0.655     1.0163       2.48    2.66
left eye    0.536  0.537     1.0056       1.23    1.32
left leg    0.934  0.933     1.0031       3.06    3.30
left wing   0.527  0.526     0.9984       4.27    4.61
nape        0.600  0.600     0.9995       2.44    2.62
right eye   0.531  0.530     0.9956       1.22    1.30
right leg   0.932  0.931     0.9973       3.05    3.28
right wing  0.505  0.505     0.9907       4.24    4.55
tail        0.945  0.944     0.9973       3.05    3.27
throat      0.311  0.311     0.9887       2.42    2.58

Visibility       FVR/all  FVR/inv  FIR/all  FIR/vis   (%)
Human (mean of 5)   1.89    4.99    3.04    4.89
A                   3.67    9.69    6.27   10.08
B                  37.85  100.00    0.00    0.00
C                   3.67    9.69    6.27   10.08

K1 oracle (GT as prediction): report AE 0.000, Liu AE 0.000, PCP 100.00%, FVR/FIR 0.00/0.00%

sigma = S_sd (SD of clicks)
row        repAE/all repAE/excl  LiuAE  PCP/median PCP/one PCP/any  capped%
Human        0.881     1.376     1.074    80.01    63.26   84.56     0.02
A            1.587     2.411     1.951    56.20    52.30   72.12     1.40
B            3.105     3.105     1.951    24.48    20.05   42.15     1.40
C            2.357     3.581     3.329    41.38    41.14   50.37    25.02
Human Liu AE before the cap: 1.074307 (|x - 1| = 7.4e-02)

sigma = S_md (mean click distance to GT)
row        repAE/all repAE/excl  LiuAE  PCP/median PCP/one PCP/any  capped%
Human        0.838     1.308     1.000    82.22    66.02   86.44     0.01
A            1.513     2.300     1.819    58.45    54.45   74.16     0.84
B            3.024     3.024     1.820    26.96    22.47   44.40     0.83
C            2.272     3.452     3.177    42.37    42.14   51.73    20.38
Human Liu AE before the cap: 1.000000 (|x - 1| = 4.4e-16)

The example is an illustration, not a test. The hypotheses’ directions were written before any code ran. Every bar and tolerance was set after a planning run of the same script with seed 100 had printed its values. Localizer C’s location noise, twice A’s, was chosen at planning so that the ranking flip derived below would show. The mirror check and the z-scores of the left/right check were added to the planning script after its first version had printed the left/right fractions, before any bar was set. The output above is from seed 0, a different draw, and the script is unchanged from the plan.

Checks. Every check held:

• Oracle. A localizer that returns the ground truth itself scores zero error, 100% PCP and no visibility errors.
• Ray cast. In a side view the left eye and left wing are visible and the right ones hidden. Over 4,680 poses, a left part at yaw $a$ is visible exactly when the right part is at yaw $\pi - a$, with 0 mismatches. Across the 2,000 birds the left and right visible fractions differ by about one standard error or less (z at most +1.00).
• Annotators. Every ground-truth-visible part has at least three clicks. The printed majority-vote and true visible fractions differ by at most 0.002 per part. S_sd recovers the simulated noise to within 2.6% for every part (ratios 0.9887 to 1.0255).
• Localizer B. By construction it has no false invisibilities (FIR 0.00% over both denominators) and calls 100% of hidden parts visible.

What “1 standard deviation” means for a human. Under S_md, the human Liu AE before the cap is 1 to within rounding error. This holds by construction: S_md is the mean of exactly the normalized distances that the human AE averages. Liu and Belhumeur’s Table 2 lists “Human” at an AE of 1.00. In our reading, a $\sigma$ defined as the workers’ mean distance from the ground truth would explain that value, but their paper does not say how $\sigma$ or the human row was computed. Under the literal reading S_sd, the same workers score 1.074, still near the report’s 1 standard deviation for an average MTurk user. Both hypotheses held: the value under S_md equals 1 (deviation 4.4e-16) and its capped table value is 1.000, and the value under S_sd lies between 0.8 and 1.25.

The human row also depends on the reference. Against the median of five clicks that include their own, workers reach 80.01% PCP. Against one other worker’s click they reach 63.26%. Their report error is lower than their Liu AE when both-invisible parts count as zero (0.881) and higher when those parts are left out (1.376), because each visibility disagreement costs 5.

Visibility errors: Liu’s AE ignores them, the report’s error does not. A and B have the same locations, so their Liu AEs agree: 1.951 and 1.951 under S_sd, and 1.819 and 1.820 under S_md. B, however, calls every hidden part visible, which is 37.85% of all parts here. The report’s error charges 5 for each of them, so B’s error is $0.3785 \times 5 + 0.6215 \times 1.951 = 3.105$, against A’s 1.587. The hypothesis held under both $\sigma$: Liu AEs within 0.05 of each other, and B’s report error higher by more than 0.5 (1.518 and 1.511). Liu and Belhumeur saw the same pattern in their own model. Without visibility prediction their AE moved only from 1.80 to 1.85, while FVR rose from 28.48% to 60.03% (§5.3 and Table 2).

The two measures can rank methods in opposite orders. C localizes worse than B but predicts visibility. Liu’s AE ranks B ahead (1.951 against 3.329 under S_sd). The report’s error with both-invisible parts counted as zero ranks C ahead (2.357 against 3.105). This flip held under both $\sigma$. It depends on the denominator, however. With both-invisible parts left out, C’s report error is 3.581 and B’s 3.105, so B stays ahead. B never predicts a part invisible, so it has no both-invisible parts and its two denominators agree. The cap matters too: a quarter of C’s both-visible scores (25.02% under S_sd) are bounded at 5, so its large misses all count the same.

PCP depends on the reference click. For A and C, PCP against one worker’s click was lower than against the median, and PCP against any of the clicks was at least as high as against one. Both parts of this hypothesis held under both $\sigma$. In our reading, a single click is noisier than the median of five, which lowers PCP against it, and any of up to five clicks is easier to land near than one. For A the reference moves PCP between 52.30% and 72.12% under S_sd. For C the gap between the median and one worker is only about 0.2 points (41.38% against 41.14%, and 42.37% against 42.14%), because, in our reading, C’s own error dominates the reference’s noise.

FVR and FIR denominators. B’s FVR is 100% of hidden parts but 37.85% of all parts. Liu and Belhumeur’s percentages could be either, and comparisons between papers need to know which.

License

The sources disagree, as of November 2026.

• CaltechDATA. The records for the images and annotations and for the segmentations both list their rights as “cc-by”. Neither record names a version of the licence or says anything about the images’ copyright.
• Caltech dataset page. Next to the download link, the page states that the group does not own the copyrights to the images and that “Their use is restricted to non-commercial research and educational purposes.”

A Creative Commons Attribution licence does not restrict commercial use, so the two statements cannot both describe the images. The images come from Flickr (both reports), and neither source names the photographers’ licences. The terms for commercial use of the images are therefore unclear, and this article does not resolve them.

Separately, the CaltechAUTHORS records of the two reports state “No commercial reproduction, distribution, display or performance rights in this work are provided”. That statement applies to the report PDFs, not to the data.

Limitations and Bias

• Overlap with ImageNet. The dataset page warns that “Images in this dataset overlap with images in ImageNet”. It advises caution with networks pretrained on ImageNet or other Flickr images, because CUB’s test images may be in their training data. The page gives no count. Results from networks pretrained on such images can therefore overstate accuracy; the general problem is discussed in benchmark overfitting.
• Photographer clusters. The 2010 report found that Flickr results for one species often come from few photographers, with near-duplicate shots. It split CUB-200 by date to keep them out of the test set (§4). The 2011 report does not say whether its split does the same.
• Selection. Species were taken from an online field guide and limited to those with a Wikipedia article and at least 20 usable Flickr images, so the set is mostly North American birds that people photograph often. The two-worker “same or similar” filter keeps images that resemble a Wikipedia exemplar. The general consequences of such choices are discussed in dataset bias.
• Labels. As far as the reports describe, the species labels come from the Flickr query, filtered by crowd workers (and in 2010 checked by the authors), not from expert identification. Fine-grained image classification reports a published estimate of label noise.
• Part annotations. Each ground-truth location is the median of five workers. The visibility flag follows the interface’s rule that more than half of the part is hidden, which leaves borderline cases to judgement. The per-part $\sigma$ comes from the same workers, so the error unit varies with how precisely each part can be clicked.
• Undefined details. With no official code, papers that report “PCP” or “AE” on CUB may differ in the reference click, the $\sigma$ and the denominators, and the example shows that these choices change both the numbers and the ranking. Some methods also leave parts out. Liu and Belhumeur note that one compared detector predicts 13 parts, omitting the legs (§5.3).
• Two releases, one name. “CUB-200” names the 2010 release, which has no part locations and 6,033 images, but the short name is also used loosely. The 2011 report’s number is given as CNS-TR-2011-001 by the dataset page and by Liu and Belhumeur, while CaltechAUTHORS lists the report under series number 2010-001. The 2010 report’s conclusion calls its dataset “CUB-2007” once, against “CUB-200” everywhere else in that report. Checking the image count and the annotations used is the reliable way to tell which release a paper means.

Common Uses

• Fine-grained classification. CUB-200-2011 is the bird benchmark in the comparisons collected in fine-grained image classification, alongside cars, aircraft and dogs. It is also one of the FGVC datasets used to compare parameter-efficient fine-tuning methods.
• Zero-shot learning. The 312 attributes, averaged per class, serve as a class embedding; zero-shot recognition gives the standard and proposed splits.
• Retrieval and metric learning. The class-disjoint split is a standard benchmark of metric learning.
• Few-shot and cross-domain transfer. Few-shot recognition uses CUB for fine-grained and cross-domain tests.
• Part localization. The second benchmark of the 2011 report, with the measures above.
• Weak localization and interpretability. The boxes score weakly supervised localization, as in CutMix’s results reported in mixup and CutMix. Part-prototype models are evaluated on cropped CUB images, as described in neural network interpretability.
• Generation. Text-to-image models before DALL·E were evaluated on CUB, among other datasets (DALL·E).

Historical Milestones

• 2010: CUB-200. 6,033 images of 200 species. On the full 200 classes, nearest-neighbour baselines reached 0.6% accuracy with image-size features and 1.7% with colour histograms. The report also lists 19% for the one-against-all SVM with SIFT and a spatial pyramid of Branson et al. (2010), “the first paper to use the dataset” (Welinder et al., 2010, §4 and Fig. 6).
• 2011: CUB-200-2011 and three benchmarks. Colour histograms and vector-quantized SIFT with a linear SVM reached 17.3% given the ground-truth parts. With parts from the HOG and pictorial-structures detector the figure was 10.3% (both with 52 training images per class, Fig. 7(d) and 7(b)), and that detector’s part-localization error was 2.47 standard deviations (Wah et al., 2011, §4).
• 2013: part localization with exemplars. Liu and Belhumeur’s exemplar-based model with pose and subcategory consistency reached 59.74% PCP, 1.80 AE, 28.48% FVR and 4.52% FIR. The best compared method, a consensus-of-exemplars model they modified to handle visibility, reached 48.70% PCP and 2.13 AE. Their “Human” row reads 84.72%, 1.00, 20.72% and 6.03% (Table 2).
• 2014–2017: learned features. Classification accuracy rose from the report’s baselines to above 80% with part-based networks, bilinear pooling and attention. The steps and their conditions (box at test time or not, parts in training or not) are in fine-grained image classification.
• 2022: CaltechDATA. The images and annotations and the segmentations were published as CaltechDATA records on 11 April 2022, each version 1.0 with a DOI.

Related Datasets

• CUB-200 (2010) is the first release, with outlines and multi-valued attributes but no parts. It has been superseded by CUB-200-2011; the 2010 report remains its only documentation.
• Other fine-grained benchmarks. Stanford Dogs, Stanford Cars, FGVC-Aircraft and iNaturalist are covered with CUB in fine-grained image classification.
• Attribute datasets. The 2011 report contrasts CUB with Animals with Attributes, which has no localization. Zero-shot recognition covers both with SUN and aPY.
• Keypoint datasets. Animal and object keypoint benchmarks, with their PCK and OKS measures, are in keypoint detection.
• Caltech’s other datasets. The same group released Caltech-101 and Caltech-256, as well as the Caltech Pedestrian dataset.

Resources

• CUB-200-2011 dataset page (Caltech Vision Lab): description, ImageNet-overlap warning, licence statement, citation and download links.
• CaltechDATA record for the images and annotations and for the segmentations.
• The 2011 technical report and the 2010 CUB-200 report on CaltechAUTHORS.
• There is no official leaderboard or official evaluation code. Classification results are collected, with their conditions, in fine-grained image classification.