Zero-Shot Recognition

Zero-shot recognition is image classification of classes for which the classifier saw no training images. What makes it possible is a description of every class, seen or unseen, in a shared form: a list of attributes (“striped”, “has wings”, “is yellow”), a word vector of the class name, a sentence, or a text prompt. A model learns on the seen classes how images relate to these descriptions, and at test time recognizes an unseen class from its description alone. The problem is usually called zero-shot learning (ZSL); related names are zero-data learning, attribute-based classification and, for the version in which seen classes remain possible answers, generalized zero-shot learning (GZSL). The field moved from attribute classifiers (2009) to learned compatibility functions between image features and class embeddings (2013–2017), to generating features for the unseen classes (2018), and since 2021 to transfer from image–text models such as CLIP, whose authors use “zero-shot” in a broader sense. Its evaluation was reorganized in 2017, when Xian, Schiele and Akata showed that many published numbers were not comparable and some were flawed.

Problem Definition

The classes are split into seen classes $\mathcal{Y}^s$, which have labelled training images, and unseen classes $\mathcal{Y}^u$, which have none, with $\mathcal{Y}^s \cap \mathcal{Y}^u = \emptyset$. Every class $y$ in both sets has a class embedding $\phi(y)$, also called side information, auxiliary information or semantic embedding: in the attribute datasets a vector of per-class attribute values, elsewhere a word vector or a sentence embedding. Training data are pairs $(x_n, y_n)$ with $y_n \in \mathcal{Y}^s$, plus the embeddings of all classes. Most methods score an image against a class with a compatibility function $F(x, y)$ and predict

$$f(x) = \arg\max_{y \in \mathcal{Y}_{\text{test}}} F(x, y),$$

where the search space $\mathcal{Y}_{\text{test}}$ defines the setting (Xian et al., 2017, eq. 2):

• Standard (conventional) zero-shot learning: test images come only from unseen classes and $\mathcal{Y}_{\text{test}} = \mathcal{Y}^u$. The model is told, in effect, that every test image is novel.
• Generalized zero-shot learning: test images come from both seen and unseen classes and $\mathcal{Y}_{\text{test}} = \mathcal{Y}^s \cup \mathcal{Y}^u$. Chao et al. (2016) call the restriction of conventional ZSL to unseen test classes unrealistic, since a deployed classifier does not know in advance whether an image is new. Socher et al. (2013) had already built a model that predicts both seen and unseen classes, and DeViSE (Frome et al., 2013) evaluated its zero-shot model with and without the training labels in the search space.

The assumption that no image of an unseen class enters training extends, in Xian et al.’s argument, to the image features: a network pre-trained on a dataset that contains test classes has in effect been trained on them, since feature extraction is part of training (2017, §4.2).

Input and Output

The input is an image, or for aPY an object crop given by its bounding box, and the embeddings of the candidate classes. Image features are usually taken from a network pre-trained on ImageNet-1K and not fine-tuned: Xian et al. (2017) standardized on the 2,048-dimensional pooled features of ResNet-101. The output is one label from the search space, or a ranked list for top-$k$ metrics. Some methods also output intermediate attribute predictions, which can describe an object, including one of an unseen class, without naming it: Farhadi et al. (2009) framed recognition as describing, including reports of unusual attributes, such as a typical attribute that is absent or a “leaf” seen on a bird.

Variants

• Generalized zero-shot learning, defined above; the setting most papers from 2017 on report.
• Transductive zero-shot learning, in which unlabelled images of the unseen classes are available during training (Xian et al., 2018, §2). Inductive ZSL, the default, has none.
• Zero-shot detection and open-vocabulary detection, which localize as well as classify; open-vocabulary object detection covers this line, from word-vector detectors to detectors built on CLIP.
• Zero-shot transfer, CLIP’s usage: Radford et al. (2021) note that zero-shot learning in computer vision usually means generalizing to unseen object categories, and use the term instead for applying a pre-trained model to a dataset it was not trained on, with classes named in text (ICML paper, §3.2). Under this usage the test classes may well have appeared in the pre-training data, which the classic setting forbids.
• Few-shot recognition, which allows a handful of labelled images per new class instead of none; the few-shot recognition article covers it. Akata et al. (2013) designed ALE to move smoothly from zero-shot prediction to learning with many examples as labels arrive.
• Open-set recognition, which only has to reject images of unknown classes rather than name them; Xian et al. (2017) cite Scheirer et al.’s open-set recognition paper for the generalized setting.

Challenges

• Seen-class bias. A model fitted on seen classes scores them higher than unseen ones, so in the generalized setting unseen images are assigned to seen classes. DeViSE gives an early measurement: its top-1 hit rate on 1,589 ImageNet classes two hops from the training classes was 6.0% when only the zero-shot labels were candidates and 0.8% when the 1,000 training labels were added, which its authors read as a bias toward training classes (Frome et al., 2013, Table 2). In Xian et al.’s (2017) generalized evaluation, DAP reached 88.7% on the seen classes of AwA and 0.0% on the unseen ones (Table 5).
• Attribute quality and correlation. Attribute classifiers learn whatever co-occurs with the attribute. Farhadi et al. (2009) describe a “wheel” detector trained on cars, buses and trains that may learn “metallic” and then fail on wooden carriages; the ground-truth correlation of “wheel” and “metallic” was 0.71 in aPascal and 0.17 in aYahoo, and on aYahoo the predictions of the two classifiers trained on aPascal correlated at 0.56 with all features and 0.28 with their feature selection. Attributes chosen by people are also not always discriminative: a naive Bayes classifier on the ground-truth Pascal attributes told the 20 classes apart with only 74% accuracy, which led them to add learned discriminative attributes.
• Annotation cost. Fine-grained classes need many attributes to be told apart, which raises the cost of collecting them; Akata et al. (2015) turned to embeddings learned from unlabelled text for that reason.
• Leaky evaluation. Test classes contained in the pre-training data, hyperparameters tuned on test classes and test splits that differ between papers made results incomparable until the proposed splits (see Datasets and Benchmarks).
• What the embedding encodes. An unseen class can only be recognized through properties shared with seen classes; a class whose distinguishing property no seen class has cannot be reached. This follows from the formulation (our reasoning), and it is why the example below keeps every attribute value among the seen classes.

Approaches

Xian et al. (2017) group the methods they evaluate into intermediate attribute classifiers, linear and nonlinear compatibility learning, and hybrid models that express an image or a class as a mixture of seen classes. Generative methods, which synthesize training data for the unseen classes, came after their evaluation, and vision–language pre-training changed the problem again.

Attribute classifiers

Two CVPR 2009 papers made attributes the route to unseen classes. Farhadi, Endres, Hoiem and Forsyth (2009) annotated 64 attributes on the roughly 12,000 object instances of the PASCAL VOC 2008 trainval set through Amazon Mechanical Turk, in three kinds: shape (“is 3D boxy”, “is cylindrical”), parts (“has wheel”, “has wing”) and materials (“has wood”, “is furry”). They added 1,000 discriminative attributes, linear SVMs on random splits of classes or attributes, and selected base features within each class so that attribute classifiers would generalize across classes. Attribute classifiers trained on aPascal transferred to aYahoo’s twelve new classes, where a linear SVM on the predicted attributes reached 69.8% accuracy. The representation also supported learning from very few examples and from description alone: describing a new class by its positive attributes (“furry”, “four legged”, “has snout”, “has head”) and assigning a test image to the nearest description gave 32.5% accuracy, which the authors equate to about 100 visual examples in semantic attribute space, 8 with base features and 3 with semantic and discriminative attributes (the text does not name the test classes of this experiment).

Lampert, Nickisch and Harmeling (2009) introduced the Animals with Attributes dataset and two probabilistic models; their CVPR paper and its TPAMI version (2014) could not be read for this article (the authors’ copies are no longer served and the publisher’s site is not used), so the methods are described here as later papers state them. In direct attribute prediction (DAP), one probabilistic classifier per attribute is trained on the seen classes; a test image is assigned to the unseen class whose attribute signature $a^c$ best matches the predicted attribute probabilities. Xian et al. (2017, eq. 11) write the rule as

$$f(x) = \arg\max_{c} \prod_{m=1}^{M} \frac{p(a_m^c \mid x)}{p(a_m^c)},$$

with $M$ attributes and the attribute prior $p(a_m^c)$ in the denominator; Akata et al. (2013, eq. 12) write the same posterior without the prior. The companion model, indirect attribute prediction (IAP), is not described in the sources read here beyond its name; the AwA2 page tabulates it below DAP on both AwA versions. Palatucci, Pomerleau, Hinton and Mitchell (2009) gave the same idea a formal name in the same year, the semantic output code classifier, which maps the input to a space of semantic features and then to the nearest class in a knowledge base; they analysed it in a PAC framework and applied it to decoding words from fMRI images, so the approach was not specific to vision.

Akata et al. (2013) listed three shortcomings of DAP: attribute classifiers are learned independently of the end task, so they may be good at attributes but not at classes; DAP is hard to extend when labelled images of the new classes arrive; and other side information, such as a class hierarchy, is hard to add. Compatibility learning addressed the first point.

Compatibility functions and semantic embeddings

Instead of predicting attributes, these methods learn a function that scores an image against a class embedding, trained so that the correct class ranks above the others. Most use a bilinear form,

$$F(x, y; W) = \theta(x)^\top W \phi(y),$$

where $\theta(x)$ is the image feature and $\phi(y)$ the class embedding, both fixed, and $W$ is learned (Akata et al., 2015, eq. 1). Because $\phi$ is given for unseen classes too, the trained $W$ scores them without any of their images. The methods differ in their loss and their embeddings:

• ALE (attribute label embedding; Akata, Perronnin, Harchaoui and Schmid, 2013) embeds each class as its attribute vector and trains $W$ with the WSABIE weighted ranking loss, which weights violations by the rank of the correct label so that the top of the list counts most. With Fisher vectors, ALE beat DAP on the 10 AwA test classes (37.4% against 36.1% top-1) and on the 50 CUB test classes (18.0% against 10.5%), while DAP’s own attribute predictions stayed more accurate (mean AUC 71.9% against 65.7% on AwA), since DAP optimizes attributes directly (Table 3). The authors read the larger gain on CUB as a sign that the approach benefits more when attribute quality is higher.
• DeViSE (deep visual-semantic embedding; Frome et al., 2013) used word vectors instead of attributes, so that any word in a vocabulary becomes a class. A skip-gram model trained on 5.4 billion words of Wikipedia text gave 500- or 1,000-dimensional unit-norm label vectors; an ImageNet-trained convolutional network had its softmax replaced by a linear projection into that space, trained with a hinge rank loss of margin 0.1 and later fine-tuned end to end. An $\ell_2$ regression loss toward the label vector, as in Socher et al. (2013), gave about half the accuracy of the rank loss, which the authors attribute to classification being a ranking problem.
• SJE (structured joint embedding; Akata, Reed, Walter, Lee and Schiele, 2015) trains $W$ with an unregularized structured SVM, which penalizes only the highest-scoring wrong class, and compared output embeddings systematically. With GoogLeNet features, continuous attributes gave 66.7% on AwA and 50.1% on CUB, against 52.0% and 37.8% for binary ones; embeddings learned without supervision from Wikipedia reached 58.8% on AwA with GloVe and 28.4% on CUB with word2vec, above the previous supervised state of the art on CUB (26.9%), and combining embeddings helped further (Tables 2, 3 and 5).
• Cross-modal transfer (CMT) (Socher et al., 2013) maps images into a word-vector space with a two-layer network and adds novelty detection that decides whether an image belongs to a seen or an unseen class (below).

Xian et al. (2017) write DeViSE, ALE and SJE in one form that differs only in the ranking loss: DeViSE sums hinge violations over all wrong classes, ALE weights them by rank, SJE keeps only the worst violation (eqs. 4–7). ESZSL adds a regularization term to the objective (eq. 8), and LATEM a piecewise-linear compatibility with several matrices $W_i$. Hybrid models such as ConSE and SYNC instead express images or class embeddings as mixtures of seen classes; ConSE, for instance, averages the embeddings of the seen classes an ordinary classifier ranks highest, weighted by their probabilities.

Generalized zero-shot learning and calibration

Applying a zero-shot model to the generalized setting means comparing scores for seen classes, fitted on labelled data, with scores for unseen classes, extrapolated from their embeddings. Two remedies came before generative models:

• Novelty detection. Socher et al. (2013) first decided whether an image is novel, with thresholds on isometric Gaussians around the seen-class word vectors or with a local outlier probability, and then used a softmax classifier for seen images and a Gaussian model for novel ones. On CIFAR-10 with cat and truck held out, both strategies traded seen-class accuracy against unseen-class accuracy as their thresholds varied: at 70% seen-class accuracy with the Gaussian model, the unseen classes reached between 15% and 30%, depending on the class, against 10% chance.
• Calibrated stacking. Chao, Changpinyo, Gong and Sha (2016) found, as their abstract states, that applying the classifiers of existing ZSL methods directly to GZSL performs poorly, and proposed a calibration factor that balances recognizing seen and unseen classes. The paper itself could not be read for this article; later papers describe the method as subtracting a constant from every seen-class score (Atzmon & Chechik, 2019, §2) or setting different prediction thresholds for seen and unseen classes (Liu et al., 2018). Written out,

$$\hat{y} = \arg\max_{y \in \mathcal{Y}^s \cup \mathcal{Y}^u} \big( F(x, y) - \gamma\, \mathbb{1}[y \in \mathcal{Y}^s] \big),$$

with $\gamma$ a constant to be chosen (this formula is our transcription of those descriptions; the example below chooses $\gamma$ on validation classes). Sweeping $\gamma$ traces a curve of unseen-class accuracy against seen-class accuracy; its area, the AUSUC (area under the seen–unseen accuracy curve), is the metric Chao et al. proposed to summarize the trade-off independently of one operating point.

Generative feature methods

Xian, Lorenz, Schiele and Akata (2018) attacked the bias from the data side: if a classifier could be trained on examples of the unseen classes, the imbalance between seen and unseen classes would be alleviated. Their f-CLSWGAN is a conditional generative adversarial network that generates CNN features rather than images. The generator, a multilayer perceptron with one hidden layer of 4,096 units and a ReLU output to match ResNet-101’s pooled features, maps Gaussian noise $z$ and a class embedding $c(y)$ to a feature $\tilde{x} = G(z, c(y))$. It is trained on seen classes only with a conditional Wasserstein GAN loss with gradient penalty plus a classification loss,

$$\min_G \max_D \; \mathcal{L}_{\text{WGAN}} + \beta\, \mathcal{L}_{\text{CLS}}, \qquad \mathcal{L}_{\text{CLS}} = -\mathbb{E}_{\tilde{x}}\left[\log P(y \mid \tilde{x}; \theta)\right],$$

where $P(y \mid \tilde{x}; \theta)$ is a softmax classifier pre-trained on real seen-class features, so that generated features stay discriminative ($\lambda = 10$ for the gradient penalty and $\beta = 0.01$ on all datasets). Feeding the embedding of an unseen class then yields as many synthetic features of it as wanted, and an ordinary softmax classifier, or any compatibility model, is trained on real seen features plus synthetic unseen ones. The authors report that images generated by the GAN they compared against were not good enough to train a classifier, which is why they generate features instead.

On the proposed splits, a softmax classifier on f-CLSWGAN features raised the harmonic mean from the best previous 34.4% to 49.7% on CUB, from 26.3% to 39.4% on SUN and from 27.5% to 59.6% on AwA (Table 2). Without generated features, every compatibility model in their table had much higher seen than unseen accuracy, which the authors read as many images being assigned to seen classes; with them, in the authors’ description, unseen accuracy rose while seen accuracy was maintained. Atzmon and Chechik (2019, Fig. 3) later observed the opposite tendency across published models: generative models tended to be biased toward the unseen classes and non-generative ones toward the seen classes.

Vision–language models

CLIP and similar dual encoders learn an image encoder and a text encoder jointly from hundreds of millions of image–text pairs, so the class embedding $\phi(y)$ becomes the text embedding of a prompt such as “a photo of a {label}.” and the compatibility is a cosine similarity. In structure this is, by our reading, the DeViSE recipe with both towers trained together on far more data; the CLIP article covers its training, prompts and results. Two differences matter for zero-shot recognition as defined here. First, the classes are not held out: CLIP’s pre-training text was collected with queries built from Wikipedia words and WordNet synsets, and its authors describe their evaluation as zero-shot transfer to unseen datasets rather than unseen categories (Radford et al., 2021, ICML paper, §3.2). Second, there is no fixed side-information vocabulary to annotate. By Xian et al.‘s rule that feature pre-training must not include test classes, CLIP’s results are not comparable with numbers on the proposed splits (our reading; CLIP’s authors make no such comparison). CLIP’s own comparison is with Visual N-Grams, an earlier model trained on the YFCC100M photo collection: on aYahoo, the test classes of aPY, it reported 98.4% against Visual N-Grams’ 72.4%, while noting that many differences between the two systems were not controlled for (Table 1, §3.1). Vision–language models that generate text extend the open vocabulary to answers rather than a choice among given class names.

Datasets and Benchmarks

Four attribute datasets dominate the classic benchmark, with ImageNet’s 21K classes as a large-scale test using word vectors. Xian et al. (2017) re-examined all of them and proposed new splits (“proposed splits”, PS) to replace the “standard splits” (SS) used until then. Their rules: test classes must not be among the ImageNet-1K classes used to pre-train the feature extractor; hyperparameters are tuned on validation classes split off the training classes, never on test classes; and in the proposed splits, some images of the training classes are held out for testing, so that the generalized setting can be evaluated. In the standard splits, 6 of AwA’s 10 test classes (chimpanzee, giant panda, leopard, persian cat, pig, hippopotamus), 7 of aPY’s 12, 6 of SUN’s 72 and one of CUB’s 50 (Indigo Bunting) were ImageNet-1K classes, and all methods scored higher on those overlapping classes (§4.2).

Dataset	Images	Classes	Attributes	Grain	SS train / test classes	PS train + val / test classes
SUN	14,340	717	102	fine	645 / 72	580 + 65 / 72
CUB	11,788	200	312	fine	150 / 50	100 + 50 / 50
AwA	30,475	50	85	coarse	40 / 10	27 + 13 / 10
aPY	about 15K	32	64	coarse	20 / 12	15 + 5 / 12

Figures from Xian et al. (2017, Table 1 and §4.1); in the standard splits the train classes include the validation classes. At test time the proposed splits add held-out images of the training classes: for example 1,764 seen-class and 2,967 unseen-class test images on CUB, and 4,958 and 5,685 on AwA.

• Animals with Attributes (AwA, AwA2). Lampert et al. (2009) aligned 50 animal classes with a class–attribute matrix of 85 attributes created by Osherson et al. (1991) and Kemp et al. (2006), as the official AwA2 page credits it; the standard 40/10 split is theirs (Xian et al. cite the TPAMI version for it). AwA2, released on 9 June 2017 by Lampert’s group, is described on its page as a drop-in replacement with the same class structure and 85 attributes: 37,322 images collected from public sources such as Flickr in 2016, restricted to images licensed for free use and redistribution, with per-image licence files in the archive. The page asks users to cite the TPAMI version of Xian et al.’s evaluation (which was not read for this article; the page dates it 2018, volume 40(8), while Crossref gives 2019, volume 41(9)), and to use only the proposed class split with ImageNet-pretrained features, because the ILSVRC classes partly overlap the AwA2 classes (as of November 2026). It also tabulates results from that paper, with ZSL accuracies on both versions; DAP, for instance, scored 58.7% on AwA2 with the original split and 46.1% with the proposed one.
• CUB-200-2011 (Wah et al., 2011): 11,788 images of 200 bird species with 312 attributes, used in zero-shot work as continuous per-class values (Akata et al., 2015); the fine-grained image classification article describes its annotations and its classification protocol. Akata et al. (2013) introduced its first zero-shot split, 150 training and 50 test classes; Xian et al. (2017) attribute that split to ALE’s TPAMI version in one sentence and to the CVPR version in another. Sentence descriptions collected for CUB serve as an alternative class embedding (Xian et al., 2018).
• SUN Attribute (Patterson & Hays, 2012): 20 images for each of 717 SUN scene categories, 14,340 images in all, each labelled with 102 attributes of materials, surface properties, functions and spatial layout; the scene recognition article describes how the attributes were chosen and labelled. With 20 images per class it has the fewest images per class of the four datasets and by far the most classes.
• aPascal and aYahoo (aPY) (Farhadi et al., 2009): the 20 PASCAL VOC 2008 classes (the dataset page says its training set corresponds to the VOC 2008 train set and its test set to the VOC 2008 validation set), and 12 classes collected from Yahoo image search (wolf, zebra, goat, donkey, monkey, statue of people, centaur, bag, building, jet ski, carriage, mug), chosen to resemble Pascal classes but with different correlations between the attributes. Each object has 64 attributes, and features are extracted from the bounding box. Xian et al. (2017) argue that zero-shot results on this small, coarse-grained dataset are not conclusive.
• ImageNet. DeViSE tested on ImageNet 2011 21K classes outside ILSVRC 2012 1K, grouped into a “2-hop” set of 1,589 classes within two hops of the training classes in the hierarchy, a “3-hop” set of 7,860 and all 20,841 (Frome et al., 2013). Xian et al. (2017) count the 2-hop and 3-hop sets, following SYNC, as 1,509 and 7,678 classes, and add splits of the 500, 1K and 5K most and least populated classes and of all remaining classes with at least one image; their best method reached 0.96% top-1 on all roughly 20K classes (Table 4).

Oxford Flowers (FLO), with 102 classes and sentence descriptions but no attributes, is a fifth dataset in later papers such as f-CLSWGAN.

Evaluation Metrics

• Average per-class top-1 accuracy. Accuracy is computed for each test class and then averaged over classes, so that densely populated classes do not dominate (Xian et al., 2017, §4.2). Results before 2017 often reported per-image accuracy, and DeViSE reported “flat” hit@$k$, the fraction of images whose correct label is among the top $k$.
• Generalized setting. Per-class top-1 accuracy is computed separately over test images of seen classes, $s$ (written $acc_{\mathcal{Y}^{tr}}$ by Xian et al.), and of unseen classes, $u$, both with the full label space as the search space, and summarized by their harmonic mean

$$H = \frac{2\, s\, u}{s + u}.$$

Xian et al. chose the harmonic rather than the arithmetic mean because a high seen-class accuracy would otherwise dominate the result; $H$ is high only when both accuracies are. A method that never predicts an unseen class has $H = 0$ whatever its seen-class accuracy.

• AUSUC, the area under the seen–unseen accuracy curve traced by calibrated stacking (Chao et al., 2016), which compares methods across all operating points rather than at one.

Xian et al. (2017) also ranked methods with the non-parametric Friedman test over datasets and validation splits, and measured robustness by tuning on three different validation splits. Their results on the attribute datasets with ResNet-101 features show how the splits and the setting change the numbers (Tables 3 and 5; per-class top-1 accuracy in %):

Method	AwA ZSL SS	AwA ZSL PS	CUB ZSL PS	SUN ZSL PS	AwA GZSL u / s / H	CUB GZSL u / s / H	SUN GZSL u / s / H
DAP	57.1	44.1	40.0	39.9	0.0 / 88.7 / 0.0	1.7 / 67.9 / 3.3	4.2 / 25.1 / 7.2
ConSE	63.6	45.6	34.3	38.8	0.4 / 88.6 / 0.8	1.6 / 72.2 / 3.1	6.8 / 39.9 / 11.6
DeViSE	72.9	54.2	52.0	56.5	13.4 / 68.7 / 22.4	23.8 / 53.0 / 32.8	16.9 / 27.4 / 20.9
SJE	76.7	65.6	53.9	53.7	11.3 / 74.6 / 19.6	23.5 / 59.2 / 33.6	14.7 / 30.5 / 19.8
ALE	78.6	59.9	54.9	58.1	16.8 / 76.1 / 27.5	23.7 / 62.8 / 34.4	21.8 / 33.1 / 26.3
f-CLSWGAN + softmax			57.3	60.8	57.9 / 61.4 / 59.6	43.7 / 57.7 / 49.7	42.6 / 36.6 / 39.4

The last row is from Xian et al. (2018, Table 2), on the same proposed splits and features; its ZSL accuracy on AwA was 68.2%. AwA here is the original AwA; the CVPR evaluation predates AwA2. Three patterns stand out. Moving from the standard to the proposed split cut AwA accuracy by 9 to 20 points for every method, as expected when most standard test classes were in ImageNet-1K, while CUB and SUN changed little. In the generalized setting every method lost most of its unseen-class accuracy, and the methods that learn independent attribute or object classifiers (DAP, ConSE) did well on seen classes and almost never predicted unseen ones, while the compatibility methods ALE, DeViSE and SJE ranked highest by $H$. Feature generation more than doubled $H$ on AwA. Xian et al. also report problems with published numbers: they could not reproduce SSE’s AwA result (76.3% reported, 64.9% reproduced), two methods reported SUN results on a 10-class test split instead of the standard 72, and SYNC’s SUN features came from a network pre-trained on Places, a dataset that overlaps SUN’s classes (§5.1).

Example: attributes, held-out classes and the seen-class bias

The script builds 48 classes of synthetic $32 \times 32$ colour images from three attribute groups: four shapes (disk, square, triangle, cross), four colours (red, green, blue, yellow) and three textures (solid, stripes, dots), with random position, size, rotation, colour jitter, pattern phase and noise. Each class is defined by an 11-dimensional binary signature, one bit per attribute value. The 12 classes whose shape, colour and texture indices sum to a multiple of 4 are unseen; every attribute value still occurs among the 36 seen classes. Images are described by HOG features at two cell sizes and a colour histogram, standing in for the pre-trained CNN features of the benchmarks.

Two models are trained on the seen classes only. DAP fits one logistic regression per attribute and scores a class by $\sum_m \log p(a_m = a_m^c \mid x)$; since the sources write the rule with and without the attribute prior, both versions are printed. The compatibility model has the bilinear form of ALE and SJE, with $\ell_2$-normalized signatures as $\phi(y)$, trained with SJE’s stochastic update. Each is tested in the standard setting (unseen test images, 12 candidate labels) and the generalized setting (held-out images of the seen classes plus unseen images, 48 candidates), with Xian et al.’s per-class accuracies and harmonic mean. For calibrated stacking, $\gamma$ is chosen to maximize $H$ on validation classes: 9 of the seen classes play the unseen role while the models are fitted on the other 27, and the models are then refitted on all 36 seen classes. The validation images are a separate set of 20 held-out images per seen class, none of them a test image, scored by the models fitted on 27 classes. Keeping them apart matters: a $\gamma$ chosen on test images is fitted to the very images on which $H$ is then reported, which inflates $H$. Xian et al. (2017, §1) make the same point for hyperparameters in general: they are tuned on validation classes split off the training classes, because tuning them on test classes violates the zero-shot assumption. The script loops over seeds 0, 1 and 2, prints the table for seed 0 and the range over the three seeds, and takes about a minute single-threaded (NumPy 2.3.5, scikit-image 0.25.2, scikit-learn 1.7.2). Two things were added after the first run. A first attempt with default BLAS threading was stopped after about seven minutes without output; the unchanged script was then run with the three thread variables set to 1 from the shell, and the script below sets them itself at the top. The “oracle” line was then added as a separate question, with its hypothesis written in the plan before it was run (discussed below).

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                 # single-threaded BLAS: same result, much faster here
import numpy as np
from skimage.feature import hog
from sklearn.linear_model import LogisticRegression

SHAPES = ["disk", "square", "triangle", "cross"]
COLOURS = {"red": (0.85, 0.15, 0.15), "green": (0.15, 0.7, 0.2),
           "blue": (0.2, 0.3, 0.9), "yellow": (0.9, 0.85, 0.15)}
TEXTURES = ["solid", "stripes", "dots"]
CLASSES = [(s, c, t) for s in range(4) for c in range(4) for t in range(3)]   # 48 classes
N_ATT = 4 + 4 + 3

def signature(cls):
    s, c, t = cls
    a = np.zeros(N_ATT)
    a[s] = a[4 + c] = a[8 + t] = 1
    return a

A = np.stack([signature(k) for k in CLASSES])                  # 48 x 11 class-attribute matrix
UNSEEN = [i for i, (s, c, t) in enumerate(CLASSES) if (s + c + t) % 4 == 0]
SEEN = [i for i in range(len(CLASSES)) if i not in UNSEEN]

S = 32
yy, xx = np.mgrid[0:S, 0:S].astype(float)

def render(cls, rng):
    s, c, t = cls
    cx, cy = rng.uniform(11, 21, 2)
    r = rng.uniform(7, 12)
    th = rng.uniform(0, np.pi)
    u = (xx - cx) * np.cos(th) + (yy - cy) * np.sin(th)        # rotated coordinates
    v = -(xx - cx) * np.sin(th) + (yy - cy) * np.cos(th)
    if s == 0:
        mask = u**2 + v**2 < r**2
    elif s == 1:
        mask = (abs(u) < 0.8 * r) & (abs(v) < 0.8 * r)
    elif s == 2:
        mask = (v < 0.5 * r) & (v > -r + 2 * abs(u) * 0.9)
    else:
        mask = ((abs(u) < 0.3 * r) & (abs(v) < r)) | ((abs(v) < 0.3 * r) & (abs(u) < r))
    phi = rng.uniform(0, np.pi)
    period = rng.uniform(3, 5)
    if t == 0:
        pattern = np.zeros((S, S), bool)
    elif t == 1:
        pattern = np.sin(2 * np.pi * (xx * np.cos(phi) + yy * np.sin(phi)) / period) > 0
    else:
        ox, oy = rng.uniform(0, 5, 2)
        pattern = (((xx + ox) % 5 - 2.5) ** 2 + ((yy + oy) % 5 - 2.5) ** 2) < 2.2
    colour = np.clip(np.array(list(COLOURS.values())[c]) + rng.normal(0, 0.06, 3), 0, 1)
    img = 0.5 + 0.08 * rng.standard_normal((S, S, 1)) * np.ones((1, 1, 3))
    obj = np.where(pattern[..., None], colour * rng.uniform(0.3, 0.5), colour)
    img = np.where(mask[..., None], obj, img)
    return np.clip(img + 0.04 * rng.standard_normal((S, S, 3)), 0, 1)

def features(img):
    g = img.mean(axis=2)
    h4 = hog(g, orientations=9, pixels_per_cell=(4, 4), cells_per_block=(1, 1))
    h8 = hog(g, orientations=9, pixels_per_cell=(8, 8), cells_per_block=(2, 2))
    q = np.minimum((img * 4).astype(int), 3).reshape(-1, 3)
    col = np.bincount(q[:, 0] * 16 + q[:, 1] * 4 + q[:, 2], minlength=64) / (S * S)
    return np.concatenate([h4, h8, col])

def make_set(class_ids, n, rng):
    X, y = [], []
    for k in class_ids:
        for _ in range(n):
            X.append(features(render(CLASSES[k], rng)))
            y.append(k)
    return np.array(X), np.array(y)

def per_class_acc(y_true, y_pred):
    return np.mean([np.mean(y_pred[y_true == k] == k) for k in np.unique(y_true)])

def train_dap(X, y):
    """One logistic regression per attribute on seen-class images."""
    return [LogisticRegression(C=1.0, max_iter=2000).fit(X, A[y, m]) for m in range(N_ATT)]

def dap_scores(models, X, train_classes, use_prior):
    P = np.clip(np.stack([m.predict_proba(X)[:, 1] for m in models], axis=1), 1e-6, 1 - 1e-6)
    logp1, logp0 = np.log(P), np.log(1 - P)
    scores = logp1 @ A.T + logp0 @ (1 - A).T                   # sum_m log p(a_m = a_m^c | x)
    if use_prior:                                               # GBU eq. 11 divides by p(a_m^c)
        prior = np.clip(A[train_classes].mean(axis=0), 1e-3, 1 - 1e-3)
        scores -= np.log(prior) @ A.T + np.log(1 - prior) @ (1 - A).T
    return scores                                               # n x 48

def train_sje(X, y, train_classes, rng, epochs=20, lr=0.01):
    """Bilinear compatibility theta(x)^T W phi(y), SJE's SGD update with 0/1 Delta."""
    Phi = A / np.linalg.norm(A, axis=1, keepdims=True)
    W = np.zeros((X.shape[1], N_ATT))
    tc = np.array(train_classes)
    for _ in range(epochs):
        for i in rng.permutation(len(y)):
            f = X[i] @ W @ Phi[tc].T + (tc != y[i])             # Delta(y_n, y) + F(x_n, y)
            yhat = tc[np.argmax(f)]
            if yhat != y[i]:
                W += lr * np.outer(X[i], Phi[y[i]] - Phi[yhat])
    return W, Phi

def sje_scores(model, X):
    W, Phi = model
    return X @ W @ Phi.T

def evaluate(scores, y, seen, unseen, gamma=0.0):
    """GZSL with calibrated stacking: subtract gamma from every seen-class score."""
    sc = scores.copy()
    sc[:, seen] -= gamma
    allc = np.array(seen + unseen)
    pred = allc[np.argmax(sc[:, allc], axis=1)]
    m_s, m_u = np.isin(y, seen), np.isin(y, unseen)
    s = per_class_acc(y[m_s], pred[m_s])
    u = per_class_acc(y[m_u], pred[m_u])
    H = 2 * s * u / (s + u) if s + u > 0 else 0.0
    return s, u, H

def zsl_acc(scores, y, unseen):
    unseen = np.array(unseen)
    m = np.isin(y, unseen)
    pred = unseen[np.argmax(scores[m][:, unseen], axis=1)]
    return per_class_acc(y[m], pred)

def curve(scores, y, seen, unseen):
    span = np.abs(scores).max()
    pts = [evaluate(scores, y, seen, unseen, g)[:2] for g in np.linspace(-span, span, 401)]
    s, u = np.array(pts).T
    order = np.argsort(s)
    return np.trapezoid(u[order], s[order])                     # area under seen-unseen curve

def run(seed, verbose):
    rng = np.random.default_rng(seed)
    Xtr, ytr = make_set(SEEN, 60, rng)
    Xts, yts = make_set(SEEN, 20, rng)
    Xtu, ytu = make_set(UNSEEN, 40, rng)
    mu, sd = Xtr.mean(0), Xtr.std(0) + 1e-6
    Xtr, Xts, Xtu = (Xtr - mu) / sd, (Xts - mu) / sd, (Xtu - mu) / sd
    Xtest, ytest = np.vstack([Xts, Xtu]), np.concatenate([yts, ytu])

    # validation split of the seen classes, for choosing gamma
    val_cls = sorted(np.random.default_rng(100).choice(SEEN, 9, replace=False).tolist())
    fit_cls = [k for k in SEEN if k not in val_cls]
    tr = np.isin(ytr, fit_cls)
    Xval, yval = make_set(SEEN, 20, rng)   # held-out images, none in the test set:
    Xval = (Xval - mu) / sd                # 27 "seen" + 9 "unseen" validation classes

    results = {}
    methods = {
        "DAP (no prior)": lambda Xf, yf, cls, r: (lambda M: lambda Z: dap_scores(M, Z, cls, False))(train_dap(Xf, yf)),
        "DAP (prior)":    lambda Xf, yf, cls, r: (lambda M: lambda Z: dap_scores(M, Z, cls, True))(train_dap(Xf, yf)),
        "SJE":            lambda Xf, yf, cls, r: (lambda M: lambda Z: sje_scores(M, Z))(train_sje(Xf, yf, cls, r)),
    }
    for name, fit in methods.items():
        # 1) choose gamma on validation classes
        f_val = fit(Xtr[tr], ytr[tr], fit_cls, np.random.default_rng(seed + 10))
        sv = f_val(Xval)
        span = np.abs(sv).max()
        grid = np.linspace(-span, span, 401)
        Hs = [evaluate(sv, yval, fit_cls, val_cls, g)[2] for g in grid]
        gamma = grid[int(np.argmax(Hs))]
        # 2) retrain on all seen classes, evaluate on the test set
        f = fit(Xtr, ytr, SEEN, np.random.default_rng(seed + 20))
        st = f(Xtest)
        results[name] = dict(
            seen_acc=per_class_acc(yts, np.array(SEEN)[np.argmax(st[:len(yts)][:, SEEN], 1)]),
            zsl=zsl_acc(st, ytest, UNSEEN),
            raw=evaluate(st, ytest, SEEN, UNSEEN, 0.0),
            cs=evaluate(st, ytest, SEEN, UNSEEN, gamma),
            gamma=gamma, ausuc=curve(st, ytest, SEEN, UNSEEN),
            oracle=max(evaluate(st, ytest, SEEN, UNSEEN, g)[2]
                       for g in np.linspace(-np.abs(st).max(), np.abs(st).max(), 401)))
    if verbose:
        print(f"seed {seed}: {len(SEEN)} seen, {len(UNSEEN)} unseen classes; "
              f"chance (ZSL) = {1/len(UNSEEN):.3f}")
        print(f"{'method':16s} {'seen-only':>9s} {'ZSL':>6s} | {'s':>6s} {'u':>6s} {'H':>6s} |"
              f" {'gamma':>6s} {'s':>6s} {'u':>6s} {'H':>6s} | {'AUSUC':>6s}")
        for name, r in results.items():
            print(f"{name:16s} {r['seen_acc']:9.3f} {r['zsl']:6.3f} | "
                  + " ".join(f"{v:6.3f}" for v in r["raw"]) + f" | {r['gamma']:6.2f} "
                  + " ".join(f"{v:6.3f}" for v in r["cs"]) + f" | {r['ausuc']:6.3f}")
        print("best H over gamma chosen on the test set itself (oracle, not a valid protocol): "
              + ", ".join(f"{n} {r['oracle']:.3f}" for n, r in results.items()))
    return results

all_res = [run(seed, verbose=(seed == 0)) for seed in (0, 1, 2)]
print("\nrange over seeds 0, 1, 2 (min-max)")
for name in all_res[0]:
    get = lambda f: [f(r[name]) for r in all_res]
    rows = {"ZSL": get(lambda r: r["zsl"]), "GZSL u": get(lambda r: r["raw"][1]),
            "GZSL H": get(lambda r: r["raw"][2]), "CS u": get(lambda r: r["cs"][1]),
            "CS H": get(lambda r: r["cs"][2])}
    print(f"{name:16s} " + "  ".join(f"{k} {min(v):.3f}-{max(v):.3f}" for k, v in rows.items()))

seed 0: 36 seen, 12 unseen classes; chance (ZSL) = 0.083
method           seen-only    ZSL |      s      u      H |  gamma      s      u      H |  AUSUC
DAP (no prior)       0.501  0.742 |  0.457  0.285  0.351 |   5.83  0.346  0.467  0.397 |  0.268
DAP (prior)          0.501  0.742 |  0.457  0.285  0.351 |   5.44  0.357  0.448  0.397 |  0.268
SJE                  0.467  0.698 |  0.414  0.340  0.373 |   2.15  0.354  0.448  0.396 |  0.248
best H over gamma chosen on the test set itself (oracle, not a valid protocol): DAP (no prior) 0.406, DAP (prior) 0.405, SJE 0.398

range over seeds 0, 1, 2 (min-max)
DAP (no prior)   ZSL 0.742-0.756  GZSL u 0.285-0.298  GZSL H 0.346-0.363  CS u 0.467-0.567  CS H 0.347-0.397
DAP (prior)      ZSL 0.742-0.756  GZSL u 0.285-0.298  GZSL H 0.346-0.363  CS u 0.448-0.567  CS H 0.348-0.397
SJE              ZSL 0.698-0.725  GZSL u 0.340-0.381  GZSL H 0.373-0.404  CS u 0.448-0.488  CS H 0.386-0.438

The standard setting works: with no image of the 12 held-out classes, both models name them correctly 70–76% of the time, against 8.3% by chance, because each unseen class is a new combination of attribute values the models learned to recognize on other classes. The generalized setting shows the bias. Once the 36 seen classes are also candidates, unseen-class accuracy falls to about half its standard-setting value or less (under half for both models at seed 0, and for DAP on every seed), while at seed 0 seen-class accuracy stays near what the models score on the seen classes alone; the seen classes’ scores were fitted to labelled images, the unseen ones are extrapolated from their signatures. DAP is the more biased of the two on every seed, as it was among the most biased methods in Xian et al.’s evaluation.

Calibrated stacking does what it is meant to: at seed 0, subtracting the validated $\gamma$ moves unseen-class accuracy up by 0.11 (SJE) to 0.18 (DAP). The harmonic mean, however, rises only by about two (SJE) to five (DAP) points, against the plan’s expectation of “H at least 1.5x uncalibrated”, because seen-class accuracy falls by more than half as much as unseen-class accuracy rises; that hypothesis was not met. The oracle line, which picks $\gamma$ on the test labels and is not a valid protocol, asks whether the validated choice of $\gamma$ is to blame. The plan’s hypothesis for it was that “the oracle H is at most 0.03 above the validated H”; this held for all three models, whose validated $\gamma$ came within 0.01 of the oracle’s $H$. Even the oracle $\gamma$, though, raises $H$ by no more than about five and a half points over the uncalibrated value, far from a rise by half, so the trade-off itself, not the choice of $\gamma$, is what limits calibration here. Our reading is that the seen classes are themselves hard in this example (about half of their test images are right even with only seen candidates), so near the optimum each unseen image recovered costs about one seen image, and the seen–unseen curve, whose area is the AUSUC column, has no bulge for calibration to exploit. In Xian et al.’s Table 5, by contrast, DAP and ConSE on AwA and CUB have seen accuracies of 68–89% beside unseen accuracies near zero, which leaves far more room.

The two DAP variants give identical results before calibration. Here every value of an attribute group occurs in the same number of seen classes (9 of 36 for each shape and colour, 12 of 36 for each texture), so the prior term is the same for every class and cancels in the arg max; the variants differ only through the $\gamma$ chosen on the 27 fit classes, whose attribute frequencies are not uniform. On a dataset with unequal attribute frequencies, such as AwA, the prior changes the ranking of classes, so the choice between the two published forms is not neutral in general (our reasoning).

Applications

The benchmarks point to where the problem arises: categories too numerous or too rare to collect labelled images for. Akata et al. (2015) motivate zero-shot learning with fine-grained classification, where labelling is laborious and fine-grained collections have few images per class, and Xian et al. (2017) argue, because labels for rare fine-grained classes require expert opinion, that zero-shot methods should be evaluated mainly on least populated or rare classes. SUN’s attributes describe scenes by materials, functions and layout, so a new kind of place can be specified by its properties; the scene recognition article describes them. Palatucci et al. (2009) applied the same reasoning outside vision, to decoding which word a person is thinking of from fMRI data. In current systems the idea mostly survives inside open-vocabulary models: open-vocabulary object detection scores regions against text embeddings of category names, and CLIP-style classifiers take their class list as text at test time.

Open Problems

• Bias toward seen classes. Calibration and generated features reduce it, but the trade-off between seen and unseen accuracy remains the central difficulty of the generalized setting, and methods choose different operating points along it (Atzmon & Chechik, 2019).
• Comparable evaluation. Xian et al. (2017) showed how split choice, pre-training overlap and test-set tuning changed results; with features from models pre-trained on web-scale image–text data, which may contain nearly any named category, “unseen” can no longer be guaranteed by excluding ImageNet-1K classes (our reasoning).
• Better class descriptions. Attributes are expensive and incomplete, while word vectors and text are cheap but noisy; SJE’s results show that the embedding matters as much as the model, and combining embeddings helped.
• Large label spaces. On all roughly 20,000 remaining ImageNet classes, no method in Xian et al.’s evaluation exceeded 1% top-1 accuracy (Table 4), and the 5K least populated classes, the most fine-grained, were among the hardest splits.