Fine-Grained Image Classification

Fine-grained image classification, also called fine-grained visual categorization (FGVC) or subordinate categorization, assigns an image to one of many closely related categories inside a single basic-level class: which of 200 bird species, which make, model and year of car, which variant of an aircraft. It is image classification with an unusual balance of difficulty. Lin, RoyChowdhury and Maji (2015) put it in one sentence: the visual differences between categories are small and easily overwhelmed by differences caused by pose, viewpoint and the object’s position in the image. Their example is two gull species that differ in the pattern on their beaks; two ladybird species pictured in the iNaturalist paper differ in the number of spots on their backs (Van Horn et al., 2018).

The task shaped several ideas in recognition. Its main bird benchmark came with part annotations at a time when, its authors wrote, datasets with comprehensive part locations were scarce, which made it a testing ground for part detectors and pose normalization; it motivated bilinear (second-order) pooling of convolutional features; and it is a main testing ground for learned attention that zooms into small regions. It is also a setting in which the benefit of ImageNet pretraining was shown to depend on how well the source data covers the target classes.

Problem Definition

Given $K$ subordinate classes of one basic-level category (birds, cars, dogs, aircraft, plants) and labelled training images, predict the class of a new image. Formally this is the multi-class problem of image classification, usually trained with a softmax and the cross-entropy loss. What makes it a separate task are three properties its sources stress:

• Small inter-class differences. Classes share the same parts in the same layout, and the evidence that separates them lies in a few local regions, such as the pattern on a beak or the number of windows on an airliner (Lin et al., 2015).
• Large intra-class variation. Pose, viewpoint, lighting, background, age and sex change appearance more than the class does. Wah et al. (2011) describe birds that are flying, swimming or perched and partly hidden by branches; Khosla et al. (2011) note dogs of one breed differing in age, pose, clothing and fur colour.
• Expert labels and few examples. Only domain experts can label many fine-grained classes, and the finer the categories, the fewer images each class tends to have (Van Horn et al., 2018). Cui et al. (2018) describe the commonly used FGVC datasets as typically containing around 10,000 labelled training images.

The standard assumptions are those of image classification: a fixed label set, one label per image, and test images from the training distribution. Some protocols also assume an object bounding box or part locations at test time; results are only comparable under the same assumption (see Evaluation Metrics).

Input and Output

The input is a photograph that contains one object of the basic-level category, usually not cropped tightly: in CUB-200-2011 most images are roughly 500 × 500 pixels, and the bird’s box covers a varying fraction of the image (Wah et al., 2011, Fig. 6). Training sets may add annotations beyond the class label:

• a bounding box per image (CUB-200-2011, Stanford Dogs, Stanford Cars);
• part locations: 15 named parts per bird in CUB-200-2011, each with pixel position and visibility, the median of five crowd workers’ clicks;
• attributes: 312 binary attributes in 28 groups in CUB-200-2011, most describing the colour, pattern or shape of a part (Wah et al., 2011).

The output is a score per class, from which the top-1 label is taken; some benchmarks score the top three or five labels (iNaturalist, below).

Variants

• Supervision levels. “Strongly supervised” methods use boxes and part annotations during training, and sometimes at test time; “weakly supervised” methods use only class labels. The papers below report results in separate columns for these settings.
• Large-scale, long-tailed classification. The iNaturalist datasets have thousands of species with a heavily imbalanced number of images per class (iNat2017: 5,089 classes, largest class over smallest about 435 times; Van Horn et al., 2018).
• Fine-grained retrieval and open-set recognition. When test classes are not seen in training, the task becomes retrieval with a learned embedding; CUB-200-2011 and Cars196 split by class are the standard metric learning benchmarks.
• Face identification can be seen as an extreme fine-grained problem, although face recognition relies on face-specific alignment and preprocessing (Van Horn et al., 2018, §2).
• Fine-grained detection. Finding and naming subordinate classes together, as in the iNaturalist detection task, or in aerial object detection with ship and vehicle subtypes.

Challenges

• Dilution by pooling. A standard convolutional classifier ends in global pooling over the whole feature map, so a cue occupying a few per cent of the image contributes a few per cent of the pooled vector, while background and pose contribute the rest. The example at the end of Approaches makes this concrete.
• Localization. Birds are small in the image compared with aircraft, and birds and cars appear in more clutter (Lin et al., 2015).
• Annotation cost. Part annotations are much harder to collect than labels, and manually defined parts may not be the ones that best separate classes (Lin et al., 2015; Fu et al., 2017).
• Labels. Some class pairs differ in features that are not visible: the most confused pair in Lin et al. (2015), American crow and common raven, is told apart by wingspan, habitat and voice. The same paper cites an estimated 4.4% label noise in CUB-200-2011.
• Data imbalance. In iNat2017 the most populated 1% of classes hold over 16% of the training images, and accuracy falls for classes with few training images (Van Horn et al., 2018).
• Domain gap from pretraining data. ImageNet contains about 120 dog breeds and about 60 bird species but only ten high-level car classes, so pretrained features fit some fine-grained domains better than others (Van Horn et al., 2018; Kornblith et al., 2019).

Approaches

Hand-crafted features and the first benchmarks

The CUB-200-2011 report set three benchmarks: species classification given ground-truth part locations, part localization, and species classification from the uncropped image. Its baselines used colour histograms and vector-quantized SIFT with a linear SVM, and HOG part detectors with a mixture of tree-structured pictorial structures. They reached 17.3% accuracy with ground-truth parts and 10.3% when the parts came from the detector, and the authors expected that the uncropped task could not be solved well without some localization (Wah et al., 2011). The report argues that bag-of-words representations, successful on datasets such as Caltech-101, are often less successful on subordinate categories because the categories look so similar.

Fisher vectors on dense SIFT did better on rigid objects than on birds. In the uniform re-implementation of Lin et al. (2015, Table 1), FV-SIFT reached 18.8% on CUB-200-2011 but 61.0% on FGVC-Aircraft and 59.2% on Stanford Cars, and a heavily engineered FV-SIFT ensemble was the best FGComp 2013 entry without boxes at test time (track 2) on both aircraft (80.7%) and cars (82.7%).

Part-based methods with part annotations

The early deep approach was a two-step pipeline: detect the object and its parts, then describe each part with a convolutional network and classify the concatenation. It conditions appearance on part location, which removes much of the pose variation.

Part-based R-CNN. Zhang, Donahue, Girshick and Darrell (2014) extended the R-CNN framework with part annotations (Fu et al., 2017): detectors for the object and its parts run on region proposals, a step that Lin et al. (2015) note needs thousands of network evaluations, and a CNN classifier then describes the detected parts. The paper itself could not be read for this article; the description and numbers here come from later papers. They report the method’s CUB-200-2011 accuracy in different settings and do not agree: 73.9% without annotations at test time, 76.4% with the box at test time and 82.0% with box and parts at test time (Krause et al., 2015, Table 4; Lin et al., 2015, Table 1, gives the first two); 81.6% in the table of Fu et al. (2017) and 76.4% in that of Zheng et al. (2017), both of which mark only that annotations were used in training. The comparison of Jaderberg et al. (2015) lists 74.9% for a row labelled “Zhang ’14”, but the reference that row points to in their list is a different paper, by X. Zhang et al. on approximating convolutional networks, so that row cannot be attributed with certainty.

Pose-normalized CNN. Branson et al. (2014), whose paper was not read here either, computed local features after estimating the bird’s pose from keypoints, which require keypoint annotations for training (Fu et al., 2017; Jaderberg et al., 2015); Lin et al. (2015) list it at 75.7% on CUB-200-2011 without annotations at test time and Krause et al. (2015) at 85.4% with boxes and parts at test time. Its reliance on keypoints connects the task to keypoint detection.

These methods showed that localization helps, but every new domain needed new part definitions and annotations. Krause et al. (2015) argued that this cost is prohibitive when scaling to hundreds or thousands of fine-grained domains.

Parts without part annotations

Two lines of work removed the part labels. Krause et al. (2015) generated parts by co-segmentation and alignment: they segmented the object in each training image with a co-segmentation model that uses the training boxes and the small appearance variation within a class, aligned pairs of images with similar poses through their segmentations, composed these into an alignment of all images, and sampled points across the aligned images to define parts, whose detectors they combined in a discriminative mixture. With boxes in training only and part features from a 19-layer VGGNet, they reported 82.0% on CUB-200-2011 and 92.6% on Stanford Cars (Tables 4 and 5), tied with Part-based R-CNN’s result with box and parts at test time, which used an AlexNet-style network (Lin et al., 2015).

Xiao et al. (2015) used two levels of attention with no box or part annotations at any stage. Bottom-up proposals from selective search are filtered by a network that judges whether a patch shows the basic-level object (object-level attention); the filters of a second, domain-specific network are grouped by spectral clustering into part detectors (part-level attention). They reported 69.7% on CUB-200-2011 with an AlexNet-style network and 77.9% with VGG features.

Bilinear pooling

Lin, RoyChowdhury and Maji (2015) replaced explicit parts with a pooled second-order feature. Two feature extractors $f_A$ and $f_B$, convolutional networks truncated at a convolutional layer, produce feature vectors $\mathbf{a}_l \in \mathbb{R}^M$ and $\mathbf{b}_l \in \mathbb{R}^N$ at every location $l$. Their outer product is summed over locations,

$$\boldsymbol{\Phi}(I) = \sum_{l} \mathbf{a}_l \, \mathbf{b}_l^{\top} \in \mathbb{R}^{M \times N},$$

reshaped to a vector $\mathbf{x}$ of length $MN$, passed through a signed square root $\mathbf{y} = \operatorname{sign}(\mathbf{x}) \sqrt{|\mathbf{x}|}$ and $\ell_2$ normalization $\mathbf{z} = \mathbf{y} / \lVert \mathbf{y} \rVert_2$, and classified with logistic regression or a linear SVM. In matrix form, with $\mathbf{A}$ ($L \times M$) and $\mathbf{B}$ ($L \times N$) holding the features at $L$ locations, $\mathbf{x} = \operatorname{vec}(\mathbf{A}^{\top}\mathbf{B})$, and the gradients are $\partial \ell / \partial \mathbf{A} = \mathbf{B} (\partial \ell / \partial \mathbf{x})^{\top}$ and $\partial \ell / \partial \mathbf{B} = \mathbf{A} (\partial \ell / \partial \mathbf{x})$, so both networks can be fine-tuned end to end from image labels.

The sum is orderless, like the global pooling of a classifier, but each entry records how strongly channel $i$ of one network and channel $j$ of the other fire at the same place. The authors’ motivation was that one network could act as a part detector and the other as a local descriptor, so their product describes a part’s appearance wherever the part is. They showed that bag-of-visual-words, VLAD, Fisher vectors and second-order pooling (O2P) can all be written as bilinear models with particular, fixed feature functions; the bilinear CNN learns both. Their visualizations, however, did not show a clean split into a “where” and a “what” network.

With images resized to 448 × 448, ImageNet-initialized networks and fine-tuning on image labels alone, the B-CNN that combines a VGG-16 (“D-Net”) and a VGG-M (“M-Net”) reached 84.1% on CUB-200-2011, against 70.4% for a fine-tuned VGG-16 with its fully connected layers and 74.7% for a Fisher vector on its convolutional features. The same configuration gave 83.9% on FGVC-Aircraft and 91.3% on Stanford Cars; two VGG-16 networks gave 84.1% on aircraft (Lin et al., 2015, Table 1). The authors write that the bounding box, given at training and test time, improved their most accurate B-CNN by less than 1%, and read this as greater invariance to the location of parts; in their table the D-Net/M-Net model, the most accurate, goes from 84.1% to 85.1% (a gain of 1.0 point) and the two-D-Net model from 84.0% to 84.8%.

The kernel view and compact versions. The full descriptor has $512 \times 512 \approx 262{,}000$ dimensions for VGG features. Gao, Beijbom, Zhang and Darrell (2016) observed that the inner product of two bilinear descriptors, built from local features $\mathbf{x}_s$ of one image and $\mathbf{y}_u$ of another, is

$$\langle \mathbf{B}(\mathcal{X}), \mathbf{B}(\mathcal{Y}) \rangle = \sum_{s} \sum_{u} \langle \mathbf{x}_s \mathbf{x}_s^{\top}, \mathbf{y}_u \mathbf{y}_u^{\top} \rangle = \sum_{s} \sum_{u} \langle \mathbf{x}_s, \mathbf{y}_u \rangle^2,$$

a sum of second-order polynomial kernels between all pairs of local features. Any low-dimensional map $\phi$ with $\langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle \approx \langle \mathbf{x}, \mathbf{y} \rangle^2$ therefore gives a compact descriptor $\sum_s \phi(\mathbf{x}_s)$. They used two such maps, Random Maclaurin and Tensor Sketch, both differentiable. With $d = 8{,}192$ dimensions and VGG-16, the text says that after fine-tuning compact and full bilinear pooling all reached 16.0% error on CUB-200-2011; their Table 4 gives 16.00% for full bilinear pooling and Tensor Sketch and 16.14% for Random Maclaurin. In their dimension sweep with VGG-M, fine-tuned Tensor Sketch with $d = 16$K dimensions reached 22.66% error against 22.44% for full bilinear pooling.

Kong and Fowlkes (2017) kept the bilinear feature as a matrix and constrained the classifier instead: a low-rank bilinear classifier is evaluated without forming the $c^2$-dimensional feature, and a shared projection across classes compresses it further. With VGG-16 their model reached 84.21% on CUB-200-2011, 90.92% on cars and 87.31% on aircraft, with 0.8 MB of parameters against 200 MB for the full bilinear classifier and 8 MB for Tensor Sketch (Table 2). The abstract calls the model three orders of magnitude smaller than the bilinear CNN and the introduction a hundred times smaller; the table’s ratio is 250.

Attention and learned part localization

A third family learns where to look from class labels and passes the selected region, at higher resolution, to a classifier. The attention mechanism article covers soft attention in general; the fine-grained methods mostly use hard crops.

Spatial transformers. Jaderberg et al. (2015) placed two or four spatial transformers in parallel before Inception sub-networks, each predicting a crop of the input image. On CUB-200-2011, trained from labels only, the Inception baseline at 224 pixels reached 82.3% and four transformers on 448-pixel inputs 84.1% (Table 3). In their visualizations one transformer of the two-transformer model became a head detector and the other fixated on the body, without any part labels.

RA-CNN. Fu, Zheng and Mei (2017) stacked three classification networks at three scales. At each scale an attention proposal network predicts a square $(t_x, t_y, t_l)$, centre and half side length; the region is cut out with a differentiable mask built from logistic functions,

$$M = \big[h(x - t_x + t_l) - h(x - t_x - t_l)\big] \cdot \big[h(y - t_y + t_l) - h(y - t_y - t_l)\big], \qquad h(x) = \frac{1}{1 + e^{-kx}},$$

and enlarged by bilinear interpolation for the next scale. Training alternates a softmax loss at each scale with a pairwise ranking loss between neighbouring scales, $\max\{0,\, p_t^{(s)} - p_t^{(s+1)} + \text{margin}\}$, which asks the finer scale to give the true class $t$ a higher probability than the coarser one ($k = 10$, margin 0.05). Inputs were 448 × 448 at the first scale and the attended regions 224 × 224, with VGG-19 for birds and cars and VGG-16 for dogs. The second scale alone reached 82.4% on CUB-200-2011 against 77.8% for VGG-19 on the full image, and the three scales combined 85.3%; on Stanford Dogs and Stanford Cars the combination reached 87.3% and 92.5% (Tables 3–5). The authors describe the attention moving from body to head and then beak for birds, and the attended regions falling mostly on dog heads and on the front or back of cars. Fu et al. list the B-CNN’s 85.1% as using annotations in training; in Lin et al. (2015) that figure uses the box at both training and test time.

MA-CNN. Zheng, Fu, Mei and Luo (2017) clustered spatially correlated channels of the last convolutional layer into groups, each of which generates one part by weighting and pooling its channels, and classified the image from each part separately, with a channel-grouping loss that makes the parts compact and diverse. Each part is then cropped around the peak of its attention map, enlarged and described by its own network, alongside the whole object. With VGG-19 (448-pixel inputs for part localization, 224-pixel part crops), four parts and the whole object, they reported 86.5% on CUB-200-2011, 89.9% on FGVC-Aircraft and 92.8% on Stanford Cars, all from labels only.

The trend across these papers is that the gap between methods using part annotations and those using labels alone closed: in the CUB-200-2011 tables of the two 2017 papers, the best label-only method came within 0.1 point of the best annotated one (RA-CNN, 85.3% against 85.4%) or exceeded it (MA-CNN, 86.5% against 85.4%).

Pretraining and resolution

Nearly every result above starts from ImageNet weights and uses inputs larger than ImageNet’s. The sources report several effects, each under its own set-up.

• Resolution. Lin et al. (2015) resized images to 448 × 448, twice the resolution the networks were trained on, and Cui et al. (2018, Table 8) compare methods at the same 448 × 448 size, which they call a convention. Jaderberg et al. (2015) gained 0.8 points by moving two spatial transformers from 224- to 448-pixel inputs (83.1% to 83.9%). On the iNat2017 minival set, Cui et al. (2018, Table 3) trained Inception-v3 at 299, 448 and 560 pixels, and top-1 error fell from 29.93% to 26.51% and 25.37%, top-5 error from 10.61% to 9.02% and 8.56% (the caption mentions only top-5 error; the table gives both). Their explanation is that higher-resolution inputs contain the subtle details fine-grained recognition needs, while most ImageNet images are about 500 × 375 with objects of similar scale, limiting the benefit there.
• Which pretraining data. Cui et al. (2018, Table 7) pretrained Inception-v3 from scratch on ImageNet, on iNat2017, and on both, then fine-tuned at 299 pixels on seven fine-grained datasets. iNat pretraining gave 89.26% on CUB-200-2011 against 82.84% from ImageNet, but 78.46% on Stanford Dogs against 84.19%, and lower accuracy on cars and aircraft; training on both datasets combined gave results between the two on birds, dogs and aircraft, though not on cars or Food-101. Two source subsets built from the ImageNet and iNat classes most similar to the seven targets, measured by an Earth Mover’s Distance between class-mean features, did well on all seven datasets.
• Whether pretraining helps at all. Kornblith, Shlens and Le (2019) found that fine-tuning beat training from random initialization in 189 of 192 dataset–model combinations, but by little on Stanford Cars and FGVC-Aircraft: for Inception v4 at 299 pixels, 93.3% against 92.7% on cars and 89.0% against 88.8% on aircraft. They note that ImageNet has more car images than Stanford Cars but only ten high-level car classes. The transfer learning and fine-tuning articles discuss these results and the practices behind them.
• Overlap. The Stanford Dogs images were drawn from ImageNet (Khosla et al., 2011), so an ImageNet-pretrained network may already have trained on images in its test set. This is our reasoning from the dataset’s construction, not a measurement reported by the sources read here, but it is one reason to read Dogs results with care.

Example: global pooled features against a part-localized feature

The example builds 64 × 64 synthetic “birds”: a striped elliptical body of random size, position, tilt and stripe orientation, with a plain disc as the head. The class (four classes) is set only by a short dark bar on the head, at 0°, 45°, 90° or 135° (5 pixels long when horizontal or vertical, 3 pixels along the diagonals). Features are the energies of eight fixed Gabor filters (four orientations, two frequencies), with no learning, so the comparison isolates where features are pooled. Five feature sets go to the same logistic regression: global average and global max pooling; global bilinear pooling with the signed square root and $\ell_2$ normalization of Lin et al. (2015); average pooling in a 13 × 13 window around the head, located either from the generator’s ground truth (a “part annotation”) or by normalized cross-correlation with a disc template (a simple part detector); and the annotated-head window after the image has been downsampled by 2 and by 4. Two label-free checks come first: classifying each head crop by correlation with the four ideal bar templates, without training, and the mark’s share of the total filter energy. The script loops over five data seeds and runs in under a minute (40 to 50 seconds in our runs; NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2, scikit-learn 1.7.2).

import numpy as np
from scipy.signal import fftconvolve
from skimage.draw import ellipse, disk, line_aa
from skimage.filters import gabor_kernel
from skimage.feature import match_template
from skimage.transform import rescale, resize
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

N, R = 64, 6                                   # image size; part window is (2R+1) x (2R+1)
ANGLES = np.deg2rad([0, 45, 90, 135])          # class k = orientation of the mark on the head

def bird(rng, k):
    """Striped body + plain head; the class is only the small bar at the head centre."""
    while True:
        cy, cx = rng.uniform(26, 38, 2)
        a, b = rng.uniform(12, 16), rng.uniform(7, 9)
        th = rng.uniform(-np.pi / 6, np.pi / 6) + np.pi * rng.integers(2)   # tilt, facing left/right
        r = rng.uniform(5, 6)
        hy = int(round(cy + (a + r - 2) * np.sin(th)))
        hx = int(round(cx + (a + r - 2) * np.cos(th)))
        if R <= hy < N - R and R <= hx < N - R:
            break
    yy, xx = np.mgrid[:N, :N]
    img = 0.3 + 0.03 * rng.standard_normal((N, N))
    phi, period = rng.uniform(0, np.pi), rng.uniform(4, 6)
    stripes = 0.7 + 0.25 * np.sin(2 * np.pi * (xx * np.cos(phi) + yy * np.sin(phi)) / period)
    rr, cc = ellipse(cy, cx, b, a, (N, N), rotation=-th)
    img[rr, cc] = stripes[rr, cc]
    rr, cc = disk((hy, hx), r, shape=(N, N))
    img[rr, cc] = 0.8
    clean = img.copy()
    dy, dx = 2 * np.sin(ANGLES[k]), 2 * np.cos(ANGLES[k])
    rr, cc, v = line_aa(int(round(hy - dy)), int(round(hx - dx)), int(round(hy + dy)), int(round(hx + dx)))
    img[rr, cc] -= 0.5 * v
    noise = 0.05 * rng.standard_normal((N, N))
    return img + noise, clean + noise, (hy, hx)

KERNELS = [gabor_kernel(f, theta=t) for f in (0.25, 0.125) for t in ANGLES]

def energy(stack):                              # (n, N, N) -> (n, 8, N, N) Gabor energy maps
    return np.stack([np.abs(fftconvolve(stack, k[None], mode="same", axes=(1, 2))) for k in KERNELS], 1)

def crop_pool(E, centres):                      # average of each channel in the window at each centre
    return np.array([e[:, y - R:y + R + 1, x - R:x + R + 1].mean((1, 2)) for e, (y, x) in zip(E, centres)])

def bilinear(E):                                # mean outer product over pixels, signed sqrt, L2 norm
    F = E.reshape(len(E), E.shape[1], -1)
    B = np.einsum("ncp,ndp->ncd", F, F) / F.shape[2]
    iu = np.triu_indices(E.shape[1])
    z = np.sqrt(B[:, iu[0], iu[1]])               # entries are non-negative (energies)
    return z / np.linalg.norm(z, axis=1, keepdims=True)

DISC = np.zeros((15, 15)); DISC[disk((7, 7), 5.5, shape=(15, 15))] = 1
def detect_head(img):                           # normalized cross-correlation with a disc template
    m = match_template(img, DISC, pad_input=True)
    y, x = np.unravel_index(np.argmax(m), m.shape)
    return int(np.clip(y, R, N - R - 1)), int(np.clip(x, R, N - R - 1))

def bar_templates():
    T = []
    for t in ANGLES:
        z = np.zeros((2 * R + 1, 2 * R + 1))
        rr, cc, v = line_aa(int(round(R - 2 * np.sin(t))), int(round(R - 2 * np.cos(t))),
                            int(round(R + 2 * np.sin(t))), int(round(R + 2 * np.cos(t))))
        z[rr, cc] = -v                          # the mark is darker than the head
        T.append((z - z.mean()).ravel())
    return np.array(T)

def accuracy(Xtr, ytr, Xte, yte):
    clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
    return clf.fit(Xtr, ytr).score(Xte, yte)

def run(seed, n_train=1000, n_test=600):
    rng = np.random.default_rng(seed)
    y = rng.integers(4, size=n_train + n_test)
    data = [bird(rng, k) for k in y]
    imgs = np.array([d[0] for d in data]); clean = np.array([d[1] for d in data])
    heads = [d[2] for d in data]
    E = energy(imgs)
    found = [detect_head(im) for im in imgs]
    feats = {
        "global average": E.mean((2, 3)),
        "global max": E.max((2, 3)),
        "global bilinear": bilinear(E),
        "part, annotated head": crop_pool(E, heads),
        "part, detected head": crop_pool(E, found),
    }
    for f in (2, 4):
        low = np.array([resize(rescale(im, 1 / f, anti_aliasing=True), (N, N), order=1) for im in imgs])
        feats[f"part, annotated, 1/{f} res"] = crop_pool(energy(low), heads)
    tr, te = slice(0, n_train), slice(n_train, None)
    acc = {name: accuracy(X[tr], y[tr], X[te], y[te]) for name, X in feats.items()}
    # label-free checks: no training, labels used only to score
    T = bar_templates()
    crops = np.array([im[h - R:h + R + 1, w - R:w + R + 1].ravel() for im, (h, w) in zip(imgs, heads)])
    tmpl_acc = np.mean(np.argmax((crops - crops.mean(1, keepdims=True)) @ T.T, 1) == y)
    E0 = energy(clean)
    share = (E.sum((1, 2, 3)) - E0.sum((1, 2, 3))) / E.sum((1, 2, 3))
    Ep, E0p = crop_pool(E, heads).sum(1), crop_pool(E0, heads).sum(1)
    share_part = (Ep - E0p) / Ep
    hit = np.mean([max(abs(a - c), abs(b - d)) <= 3 for (a, b), (c, d) in zip(heads, found)])
    return acc, tmpl_acc, np.median(share), np.median(share_part), hit

results = []
for seed in range(5):
    acc, tmpl_acc, share, share_part, hit = run(seed)
    results.append(acc)
    print(f"seed {seed}: template check {tmpl_acc:.3f}, mark share of energy: image {100 * share:.2f}%, "
          f"head window {100 * share_part:.1f}%; "
          f"head found within 3 px {hit:.3f}")
print(f"\n{'features':28s} seed 0   range over seeds 0-4   (chance 0.25)")
for name in results[0]:
    v = [r[name] for r in results]
    print(f"{name:28s} {v[0]:.3f}    {min(v):.3f}-{max(v):.3f}")

Output:

seed 0: template check 1.000, mark share of energy: image 1.02%, head window 10.4%; head found within 3 px 1.000
seed 1: template check 1.000, mark share of energy: image 0.98%, head window 9.9%; head found within 3 px 0.999
seed 2: template check 1.000, mark share of energy: image 1.00%, head window 10.1%; head found within 3 px 1.000
seed 3: template check 1.000, mark share of energy: image 1.00%, head window 10.2%; head found within 3 px 1.000
seed 4: template check 1.000, mark share of energy: image 1.01%, head window 10.1%; head found within 3 px 1.000

features                     seed 0   range over seeds 0-4   (chance 0.25)
global average               0.255    0.255-0.278
global max                   0.322    0.320-0.337
global bilinear              0.400    0.335-0.400
part, annotated head         0.547    0.523-0.552
part, detected head          0.543    0.523-0.553
part, annotated, 1/2 res     0.495    0.482-0.517
part, annotated, 1/4 res     0.262    0.262-0.287

The template check reads every mark correctly, so the cue is visible at full resolution and the task can be learned; yet the mark carries only about 1% of the image’s filter energy. Global average pooling is at chance (25.5–27.8% over the five seeds); in our reading, averaged over 4,096 pixels, the bar’s orientation is lost under the stripe texture, whose orientation varies at random from image to image. Max pooling does a little better, and bilinear pooling better again (33.5–40.0%). In our reading, the second-order entries record which orientation channels fire together at the same pixels, a statistic the thin dark bar on a plain disc changes more than it changes any single channel’s average, which is the kind of interaction Lin et al. built the bilinear model to capture.

Pooling only inside the head window doubles the accuracy over chance, to 52.3–55.2%, and the detected head works as well as the annotated one because the disc template finds the head within 3 pixels in 99.9–100% of images. Inside the window the mark’s share of the energy rises from about 1% to about 10%. By our reading, localization removed most of the nuisance, but not all of it, since the head’s outline and the edge of the striped body still fall in the window, and average pooling discards where in the window each orientation responds. The matched template, which keeps that layout, is perfect on the same crops. Downsampling the image by 2 costs a few points; by 4 the part feature falls to chance, presumably because the 1-pixel bar is averaged away, a small-scale version of why fine-grained models use larger inputs.

Two things changed after the first run. The label-free template check first printed 0.000 for every seed: the templates had been drawn as bright bars while the mark is dark, so the check picked the worst match. The template sign was corrected, which changed no classifier result. The plan predicted “the oracle part crop is above 90%”; the first run gave 52–55% instead, and the line measuring the mark’s share of energy inside the head window was then added to the printout to explain the gap, without changing any accuracy or setting.

Datasets and Benchmarks

• CUB-200-2011 (Caltech-UCSD Birds-200-2011; Wah et al., 2011): 11,788 Flickr images of 200 bird species, filtered by crowd workers; it extends the earlier CUB-200 dataset of Welinder et al. by roughly doubling the images per class. Every image has a box, 15 part locations and 312 binary attributes. The fixed split has 5,994 training and 5,794 test images, about 30 per class each (Gao et al., 2016, Table 5; Fu et al., 2017, Table 1). Results are reported with and without the box at test time, and methods are often grouped by whether boxes or parts are used in training. As of November 2026 the CaltechDATA record lists the licence as “cc-by”; the images themselves come from Flickr.
• Stanford Cars (also Cars196; Krause et al., 2013): 16,185 images of 196 classes at the level of make, model and year, such as “2012 Tesla Model S” (Lin et al., 2015), split into 8,144 training and 8,041 test images (Fu et al., 2017; Kornblith et al., 2019). Training boxes are available and were used by Krause et al. (2015). It was a track of the FGComp 2013 challenge. Accuracy is top-1 (Kornblith et al., 2019, Table 1).
• FGVC-Aircraft (Maji et al., 2013, a technical report that could not be read here): 10,000 images of 100 aircraft variants, such as Boeing 737-300 against 737-400, introduced with FGComp 2013 (Lin et al., 2015). The usual protocol trains on 6,667 images (training plus validation) and tests on 3,333 (Kong & Fowlkes, 2017; Cui et al., 2018); Van Horn et al. (2018) list 3,334 training images, which by our arithmetic is the training subset without the validation images. Kornblith et al. (2019) score it by mean per-class accuracy, whereas Lin et al. (2015) report per-image accuracy. Airplanes occupy more of the image and appear on cleaner backgrounds than birds, and the dataset has no part annotations (Lin et al., 2015).
• Stanford Dogs (Khosla et al., 2011): images of 120 dog breeds taken, with their boxes, from ImageNet leaf nodes under Canis familiaris, keeping images of at least 200 × 200 pixels and removing duplicates and degenerate images. The paper’s table gives 20,580 images while its text says over 22,000. The fixed split uses 100 images per class for training (12,000) and the rest, at least 50 per class, for testing (8,580 in the tables of Fu et al., 2017, and Cui et al., 2018); the paper also proposes training with 15, 30 or 60 images per class, averaged over 10 random draws.
• iNaturalist (Van Horn et al., 2018; inat_comp repository): species photographs from the iNaturalist citizen-science platform, labelled by community consensus (“research grade”). iNat2017 has 5,089 species in 13 super-classes, 579,184 training, 95,986 validation and 182,707 test images, and 561,767 boxes drawn at the super-class level for nine super-classes. Splits are made by observer: for each species the observers with the fewest observations were placed in the test split first, so that one person’s camera, location and backgrounds never appear in both training and test images of a species. Later editions grew: 8,142 species in 2018 and 10,000 species with about 2.7 million training images in 2021, plus a “mini” training set of 50 images per species. The competition metric was top-5 error in 2017, top-3 in 2018 and top-1 in 2021, with five labels still submitted; the 2017 README notes that several labels are allowed because some species are separated only by the observer’s location and date, or several species appear in one photo. The baseline Inception-ResNet-v2 SE, fine-tuned from ImageNet at 299 pixels, reached 67.7% top-1 on the private iNat2017 test set (Table 3), against 80.2% for the same model on the ILSVRC 2012 validation set. The terms in the repository’s READMEs (as of November 2026) allow use only for non-commercial research and education, require following iNaturalist’s terms of service, and forbid redistributing the images.

The terms of use of Stanford Cars, Stanford Dogs and FGVC-Aircraft could not be read from their official pages for this article, so none are stated here. Other fine-grained datasets named in these papers include Oxford Flowers-102, Oxford-IIIT Pets, NABirds (555 classes), Birdsnap and Food-101 (Van Horn et al., 2018, Table 1; Cui et al., 2018, Table 2).

Evaluation Metrics

Most fine-grained benchmarks report top-1 accuracy over test images, which Lin et al. (2015) call per-image accuracy. Mean per-class accuracy weights every class equally; the two differ when classes have unequal numbers of test images, and papers state which they use (FGVC-Aircraft by mean per-class accuracy in Kornblith et al., 2019). The iNaturalist competitions used top-$k$ error,

$$e_i = \min_{j} d(l_{ij}, g_i), \qquad d(x, y) = \begin{cases} 0 & x = y \\ 1 & \text{otherwise,} \end{cases}$$

averaged over test images, where $l_{i1}, \dots, l_{ik}$ are the $k$ submitted labels for image $i$ and $g_i$ its true label ($k = 5$ in 2017, $k = 3$ in 2018), and top-1 error in 2021. Van Horn et al. (2018) also report accuracy per super-class and as a function of training images per class, since an overall figure hides the tail.

Comparisons need the same annotation regime, as the tables of Krause et al. (2015) and Fu et al. (2017) show by sorting results by annotations used in training and testing; the same backbone (VGG-M, VGG-16, Inception); the same input resolution; and the same pretraining data. Retrieval versions of the datasets are scored with Recall@K on unseen classes, as the metric learning article describes.

Applications

The sources name species identification for biodiversity monitoring at a global scale, which motivated iNaturalist (Van Horn et al., 2018); recognition of car make and model; expert-level image recognition and rich image captioning (Fu et al., 2017); and image generation and machine teaching (Cui et al., 2018).

Open Problems

• Annotation-free localization that is reliable. Label-only attention methods match annotated ones in accuracy, but in the papers above whether they attend to the cues experts use is shown by visualizations, not measured.
• Long tails. Accuracy on classes with few training images remains far below that on common ones (Van Horn et al., 2018; Cui et al., 2018).
• Fair comparison. Reported gains mix changes of method with changes of backbone, resolution and pretraining data, and the same method’s number is quoted differently across papers, as the figures for Part-based R-CNN above show. In our reading, newer backbones such as the vision transformer and larger pretraining sets make this harder still, because each shifts the baseline that a fine-grained method is measured against.
• Labels that images cannot settle. Some classes are separated by size, sound, location or season rather than appearance; the iNaturalist metrics allow several guesses partly for this reason.