Open-Vocabulary Segmentation

Open-vocabulary segmentation is semantic segmentation in which the list of classes is given as text at test time and may include classes for which the model saw no labelled pixels. The model receives an image and a vocabulary such as “wall, sofa, chandelier, potted plant” and labels every pixel with one of those names. What makes it possible is a shared embedding space: class names are encoded by a text encoder, pixels or image regions by an image encoder, and each pixel takes the name whose embedding scores highest, so a new name needs no new classifier weights. Zhao et al. (2017) posed it as open vocabulary scene parsing; in 2019 two groups, each presenting the task as new, posed it with seen and unseen classes split within one dataset and word vectors as class embeddings, as zero-shot semantic segmentation (ZS3, Bucher et al., 2019) and zero-label segmentation (Xian et al., 2019). Since 2022 most methods have been built on CLIP and other image–text encoders from vision–language pretraining, trained on one dataset (usually COCO) and evaluated on others whose names were never used in training. The same problem is called language-driven or text-driven segmentation, and with instance masks it becomes open-vocabulary panoptic or instance segmentation.

Problem Definition

Ding et al. (2022) describe semantic segmentation as a pair of mappings: a partition of the image into segments $R_1, \dots, R_N$, and a label for each segment from a category set. If $\mathcal{S}$ is the set of classes annotated in training and $\mathcal{E}$ the set that appears at test time, they distinguish three settings:

• fully supervised: $\mathcal{E} \subseteq \mathcal{S}$;
• zero-shot (ZS3): $\mathcal{S} \cap \mathcal{E} = \varnothing$, so only unseen classes are candidates;
• generalized zero-shot (GZS3): $\mathcal{S} \subset \mathcal{E}$, with unseen classes $\mathcal{U} = \mathcal{E} \setminus \mathcal{S}$ competing against the seen ones.

The generalized setting is the one most papers report, because a deployed model does not know which pixels belong to new classes; ZS3Net’s authors chose it because evaluating unseen classes alone hides the bias toward seen classes (Bucher et al., 2019, §3.2). Every class $c$ is represented by a fixed embedding $\mathbf{t}_c$ of its name. A model maps pixel $i$ (or segment $q$) to a vector $\mathbf{f}_i$ in the same space and scores

$$p(c \mid i) = \frac{\exp\big(\cos(\mathbf{f}_i, \mathbf{t}_c)/\tau\big)}{\sum_{c' \in \mathcal{C}} \exp\big(\cos(\mathbf{f}_i, \mathbf{t}_{c'})/\tau\big)},$$

with $\mathcal{C} = \mathcal{S}$ during training and $\mathcal{C} = \mathcal{S} \cup \mathcal{U}$ (or any list of names) at test time. Only the map from pixels to the space is learned; swapping the rows $\mathbf{t}_c$ changes the vocabulary. SPNet’s semantic projection layer is this construction with word vectors and an inner product (Xian et al., 2019, Eq. 1); ZegFormer applies it to segments with CLIP text embeddings and $\tau = 0.01$ (Ding et al., 2022, Eq. 1).

Three further distinctions decide what a reported number means:

• What happens to unseen pixels in training images. Seen and unseen classes co-occur, so removing every training image that contains an unseen class would remove most of the data. SPNet therefore keeps all images and ignores the labels of unseen-class pixels in every training stage, and its authors call the task “zero-label” because the model may see those pixels without their labels (Xian et al., 2019, §2, §4.1). ZegFormer likewise uses only the pixel labels of seen classes. ZS3Net’s graph-context variant excludes images with unseen classes from training its generator (Bucher et al., 2019, §2.2).
• Inductive or transductive. If unlabelled pixels of unseen classes are available during training and used, for example by self-training on pseudo-labels, the setting is transductive; otherwise it is inductive (Zhou et al., 2022, §4.2). ZS3Net’s self-training extension, ZS5Net, and MaskCLIP+ are transductive in this sense, although ZS3Net’s authors describe theirs as a semi-supervised relaxation rather than transductive learning.
• Whether the class embeddings saw the classes. Word vectors trained on text alone never saw an image. CLIP’s encoders were trained on hundreds of millions of image–caption pairs, which, by our reasoning, are bound to mention many “unseen” classes; ZegFormer’s authors note in a footnote that using CLIP text embeddings is a relaxed setting compared with pure GZS3 (Ding et al., 2022, §4.4). Open-vocabulary papers therefore usually measure transfer to other datasets rather than held-out classes, and call it zero-shot only in the sense that the evaluation dataset’s training split was not used (Ghiasi et al., 2022, §4.3).

Input and Output

The input is an image and a vocabulary of class names, usually wrapped in prompt templates. ZegFormer uses “A photo of a {class name} in the scene”; MaskCLIP averages the text embeddings of 85 templates per class (Zhou et al., 2022, §4); OpenSeg adds synonyms and subcategories and takes the maximum over them, and adds short context to vague names; together these raised its mIoU by 2.6 points on average over four datasets (Ghiasi et al., 2022, §4.3). Because the vocabulary enters only through the text embeddings, it is encoded once and costs little more than a matrix product per image, as in the late-fusion detectors of open-vocabulary object detection.

The output is a label map over the vocabulary, as in semantic segmentation, or a set of masks with labels and scores for the panoptic and instance variants. OpenSeg also defines a grounding evaluation in which the vocabulary for each image is only the classes present in its ground truth, which removes competition from absent classes (Ghiasi et al., 2022, §4.3).

Variants

• Zero-shot segmentation within one dataset, with a seen/unseen class split and the generalized metrics described below (SPNet, ZS3Net, ZegFormer).
• Cross-dataset open-vocabulary segmentation: train on COCO (panoptic or COCO-Stuff), test on ADE20K, PASCAL Context and PASCAL VOC with their own class names (OpenSeg, OVSeg, SAN, CAT-Seg). Xu, Zhang, Wei et al. (2022) describe this as harder than the within-dataset setting, because the model must handle new classes and a domain gap at once.
• Annotation-free segmentation: no segmentation labels at all, only a pretrained image–text model, as in MaskCLIP (Zhou et al., 2022).
• Open-vocabulary panoptic and instance segmentation: masks for individual objects as well as stuff, scored by panoptic quality and mask average precision (ODISE, FC-CLIP, X-Decoder).
• Hierarchical open-vocabulary parsing: Zhao et al. (2017) embedded pixels and WordNet concepts jointly so that a pixel of an unseen class can be labelled with a hypernym (“vehicle” for an unseen tricycle) and scored with hierarchical precision and recall.
• Referring segmentation segments the region a phrase describes rather than all instances of a class, a form of visual grounding; promptable segmentation such as SAM returns masks for points or boxes without names; few-shot segmentation gives a few annotated examples of the new class instead of its name (LSeg’s zero-shot experiments use the few-shot benchmarks PASCAL-5i and COCO-20i, according to its README).

Challenges

• Bias toward seen classes. Scores for seen classes are fitted to labelled pixels and those for unseen classes are extrapolated, so in the generalized setting unseen pixels are assigned to seen classes. Without calibration SPNet reached 0.2% mIoU on unseen COCO-Stuff classes and 0.01% on unseen PASCAL VOC classes (Xian et al., 2019, §4.1.2). SPNet borrowed the remedy from zero-shot recognition: subtract a constant $\gamma$ from every seen-class score before the arg max, with $\gamma$ tuned on held-out seen classes (Eq. 5). ZegFormer applies the same calibration to segment scores.
• The background class. In a dataset with a catch-all background class, such as PASCAL VOC, or PC-59, which assigns every class outside its 59 to background, objects outside the training vocabulary are labelled background in the ground truth, and a model learns to suppress them. The open-vocabulary detection article describes the same conflict for boxes; the example below shows it for pixels.
• Granularity. CLIP is trained to match whole images with captions, while segmentation labels pixels. Ding et al. (2022) argue that text describes objects and segments, not pixels, so pixel-level matching is unnatural; OpenSeg’s authors argue that image–text models miss a grouping step that organizes pixels before alignment (Ghiasi et al., 2022, abstract).
• Fine-tuning breaks the alignment. MaskCLIP’s authors initialized DeepLab with CLIP’s image encoder, mapped CLIP text embeddings to its classifier weights, and found that every mapper they tried segmented seen classes well and unseen classes poorly; they attribute it to breaking CLIP’s image–text association (Zhou et al., 2022, §3.2). FC-CLIP’s and CAT-Seg’s authors report the same overfitting to seen classes when CLIP is fine-tuned through feature decoders.
• Ambiguous and overlapping names. Class names in segmentation datasets are not exclusive: in OpenSeg’s example, “building”, “house” and “brick” all describe a region labelled building in ADE20K. Benchmarks also overlap with COCO to different degrees (see Datasets and Benchmarks).
• Vocabulary size. Every added name is a competitor for every pixel. ZegFormer’s authors found the pixel-level baseline “very unstable” when given the 847 ADE20K-Full names instead of COCO-Stuff’s 171, while their segment-level model was less affected (Ding et al., 2022, Fig. 6).

Approaches

The field moved from pixel classifiers in a word-vector space (2017–2019) to CLIP-based models, first per-pixel and then two-stage (mask proposals, then classification), and from 2023 to single-stage designs that keep CLIP frozen or adapt it carefully. Each transition responded to a measured failure of the previous design.

Pixel classifiers in a word-embedding space

Zhao et al. (2017) posed open vocabulary scene parsing on ADE20K: a network embedded every pixel into a 300-dimensional space shared with WordNet concepts, using an asymmetric score that treats an image region as a specialization of its label’s concept, so that hypernym relations are preserved. Trained on the 150 frequent classes, the joint model reached a hierarchical F-score of 0.617 on infrequent classes, against 0.551 for a pixel-level DeViSE-style regression to word2vec vectors (Table 2). Predictions were thresholded rather than arg-maxed, so a pixel could be labelled with several concepts along a hierarchy.

SPNet (Xian et al., 2019) made the classifier of a DeepLab-v2 network a fixed matrix of word embeddings (fastText, word2vec, or their 600-dimensional concatenation) and trained the network with ordinary per-pixel cross-entropy. The cross-entropy version outperformed a hinge-loss variant it built as a baseline, and concatenating the two word embeddings raised unseen-class mIoU on COCO-Stuff (non-generalized) from 33.1 (fastText) and 32.1 (word2vec) to 35.2 (Table 2). In the generalized setting with calibration, SPNet-C reached 8.33% unseen and 34.52% seen mIoU on COCO-Stuff (harmonic mean 13.42%) and 29.33% unseen mIoU on PASCAL VOC with a harmonic mean of 42.45% (§4.1.2).

ZS3Net (Bucher et al., 2019) took the generative route of zero-shot classification: a generative moment matching network, conditioned on a class’s word2vec embedding, learned to generate the 256-dimensional pixel features that a DeepLabv3+ feeds into its last $1 \times 1$ classification layer; features generated for unseen classes were mixed with real seen-class features to retrain that layer. A DeViSE-style baseline that regressed word2vec vectors from each pixel scored 3.2% unseen mIoU on its 2-unseen-class PASCAL VOC split against ZS3Net’s 35.4% (Table 1). Its graph-context variant conditions the generator on the adjacency graph of the segmentation’s connected components, which helped on PASCAL Context. The authors also retrained the ImageNet backbone without the unseen classes, to avoid supervision leakage.

Language-driven per-pixel models: LSeg

LSeg (Li et al., ICLR 2022) replaced word vectors with CLIP’s text encoder. Its paper could not be read for this article (OpenReview and arXiv were not used); according to the official README, a transformer image encoder computes dense per-pixel embeddings and is trained contrastively to align them with the text embedding of each pixel’s class, so that related labels (“cat” and “furry”) map to related regions. The released demo model uses a ViT-L/16 image encoder with CLIP ViT-B/32 as text encoder. OpenSeg’s authors re-implemented the design as LSeg+: an FPN on an ALIGN image encoder, fine-tuned on COCO with a per-pixel cross-entropy against ALIGN text embeddings of the class names, 134 of them in the authors’ count (Ghiasi et al., 2022, §4.3). LSeg+ beat ALIGN’s class-activation-map baseline by 39.5 mIoU on COCO and 28.0 on PC-59, whose classes COCO mostly covers, but fell 1.0 mIoU below it on A-847; the authors concluded that training on COCO’s limited categories hurts generalization.

Decoupling: mask proposals, then classification

ZegFormer (Ding et al., 2022) split the problem into class-agnostic grouping and segment-level zero-shot classification. It is a MaskFormer (Cheng et al., 2021) with 100 queries whose decoder outputs segment embeddings; one projection gives a mask, another a 512-dimensional embedding classified against CLIP ViT-B/16 text embeddings, with a learnable “no object” embedding. Grouping does not involve class names, so the authors expected it to transfer to unseen classes. With identical backbones and codebases, ZegFormer’s segment classification beat an FPN re-implementation of SPNet in the generalized setting on COCO-Stuff (unseen mIoU 21.4 against 11.0 with CLIP text embeddings, 10.8 against 6.9 with fastText plus word2vec), and its masks had better boundary F-scores (Tables 1–2). The gain from switching word vectors to CLIP was 10.6 unseen mIoU for segments and 4.1 for pixels, which the authors read as segment features aligning better with CLIP than pixel features. A second classifier passes each segment, cropped and masked to $224 \times 224$, through CLIP’s image encoder; the two scores are combined by a geometric mean for unseen classes. Crop and mask together gave 33.1 unseen mIoU, crop alone 19.7, below the segment classifier without CLIP’s image encoder (Table 3).

Mask classification models assemble pixel scores from masks. With mask probabilities $m_q(i)$ and class probabilities $p_q(c)$ for $N$ queries, ZegFormer predicts

$$\hat{y}_i = \arg\max_{c \in \mathcal{S} \cup \mathcal{U}} \; \sum_{q=1}^{N} p_q(c)\, m_q(i) \;-\; \gamma\, \mathbb{1}[c \in \mathcal{S}],$$

the semantic inference of Mask2Former with SPNet’s calibration added (Eq. 2). Xu, Zhang, Wei, Lin, Cao, Hu and Bai’s simple baseline (ECCV 2022) is the same two-stage idea with a frozen CLIP: a MaskFormer trained on seen classes produces proposals, each binarized proposal is masked, cropped, resized to $224^2$ and classified by CLIP’s image encoder, and this is ensembled with a MaskFormer whose classifier weights are CLIP text embeddings. Trained on COCO-Stuff and tested elsewhere, the two-stage model beat an FCN built from the same CLIP by 13.1 mIoU on Cityscapes, 19.5 on PC-59 (19.6 in its text), 5.6 on A-150 and 2.9 on A-847 (Table 1). Its text gives the larger ADE20K vocabulary as 857 classes, its table as 847.

OpenSeg (Ghiasi et al., 2022) learned grouping and naming from cheaper labels. A cross-attention module proposes masks (128 in its evaluation), trained with a Dice loss only against class-agnostic COCO panoptic masks; proposals left unmatched are not penalized and may segment unlabelled regions. Mask-pooled region features are aligned with the nouns and adjectives of image captions by a grounding loss in which each word attends to one or a few regions. Trained on COCO, OpenSeg’s proposals reached 84.5% recall at IoU 0.5 on PC-59 against 62.9% for the earlier COB proposals (Table 1). Adding Localized Narratives (652K images, with pseudo-masks from a COCO-trained teacher) raised mIoU by 2.5 on average over four benchmarks.

OVSeg (Liang et al., 2023) located the bottleneck of the two-stage design with two oracles on A-150: ground-truth masks classified by a pretrained CLIP reached only 20.1 mIoU, while COCO-trained MaskFormer proposals labelled by an oracle classifier reached 66.5. Masked crops look unlike the natural images CLIP was trained on, so OVSeg fine-tuned CLIP on 1.3M masked proposals paired with nouns mined from COCO Captions, matched by CLIP itself, and replaced the zero tokens of masked patches with learnable mask prompts. Its authors found these noisy but diverse pairs preserved CLIP’s generalization better than COCO-Stuff’s 171 manual classes.

Dense prediction from CLIP without training: MaskCLIP

Zhou, Loy and Dai (2022) asked how much CLIP knows about pixels. In CLIP’s ResNet encoder the last layer is attention pooling: the average-pooled feature is the query, each spatial position gives a key and a value, and the output is a weighted sum of linearly transformed values. Their MaskCLIP drops the query and key embeddings and applies the value embedding and final linear layer as $1 \times 1$ convolutions, so each position gets an embedding in CLIP’s joint space, scored against text embeddings with no training (for ViT they substitute the class token’s query and add the residual). Simply reading CLIP’s dense features without this change gave 8.3 mIoU on PC-59 with RN50 and 9.0 with ViT-B/16; MaskCLIP gave 18.5 and 21.7, and 25.5 with ViT-B/16 after key smoothing and prompt denoising, two training-free refinements (Table 1). MaskCLIP+ uses these maps as pseudo-labels to train a DeepLab: 31.1 mIoU on PC-59 without any annotation (no self-training in this setting), and, in the transductive zero-shot setting, where it then self-trains on its own predictions, 86.1 unseen mIoU on PASCAL VOC (Table 2). Another method of the same name, Ding, Wang and Tu’s MaskCLIP for open-vocabulary universal segmentation (ICML 2023, not read), appears in the tables of SAN, ODISE, FC-CLIP and X-Decoder; it is a different model.

Single-stage and adapter designs

Two-stage pipelines run CLIP once per proposal, on masked crops unlike its training images. Later models share computation and keep CLIP’s features intact:

• SAN (Xu, Zhang, Wei, Hu & Bai, 2023) attaches a light side network (8 transformer blocks of width 240, 100 queries) to a frozen CLIP ViT. The side network reuses CLIP’s intermediate features, predicts mask proposals, and predicts attention biases that steer extra [SLS] tokens in CLIP’s last layers so that each one pools the features of one mask. Trained end to end, the proposals become CLIP-aware. The authors report 8.4M trainable parameters and 64.3 GFLOPs at $640^2$ with ViT-B/16, against 147.2M and 1,916.7 GFLOPs for their re-implementation of OVSeg (Table 3).
• ODISE (Xu, Liu, Vahdat, Byeon, Wang & De Mello, 2023) extracts features for mask generation from the frozen UNet of Stable Diffusion, fed an “implicit caption” computed from the image, and classifies masks with category labels or caption nouns; at inference it fuses these scores with mask-pooled CLIP features by a geometric mean.
• FC-CLIP (Yu et al., 2023) builds a Mask2Former on a single frozen convolutional CLIP (ConvNeXt-Large from OpenCLIP, trained on LAION-2B). An in-vocabulary classifier mask-pools the trained pixel decoder’s features; an out-of-vocabulary classifier, used only at test time, mask-pools the frozen CLIP features; their scores are combined by a geometric mean in which the out-of-vocabulary score has exponent $\alpha = 0.4$ for training classes and $\beta = 0.8$ for others (Eq. 7). Fine-tuning the backbone in this single-stage design misaligned image and text features. The authors also observed that ViT CLIP features grow noisier as input resolution rises, while convolutional ones stay smooth (Fig. 1). They report 21M trainable parameters; the frozen backbone is 238M parameters in their introduction, 200M in Table 1 and 198M in their results text.
• X-Decoder (Zou et al., 2023) decodes pixel-level masks and language tokens from one decoder with generic and text-induced queries, pretrained on a limited amount of segmentation data and millions of image–text pairs, and transfers zero-shot to segmentation datasets; its authors caution that the comparisons in their table are not strictly fair in supervision and models.
• CAT-Seg (Cho et al., 2024) works on the cost volume, the cosine similarities between dense CLIP image embeddings (obtained with MaskCLIP’s last-layer change) and every class’s text embedding. Spatial aggregation (Swin transformer blocks applied to each class’s cost map) and class aggregation (attention across classes, permutation-invariant and indifferent to the number of classes) refine it, and an upsampling decoder produces the mask. Because the trainable layers see similarities rather than features, the authors argue they overfit less, which lets CLIP’s encoders be fine-tuned without losing unseen classes.

CLIP-based dense prediction and its limits

The papers above document five limits of using CLIP for pixels. CLIP’s pooled embedding loses location, and its dense features need surgery (MaskCLIP) before they mean anything; even then, MaskCLIP’s best training-free result on PC-59 is 25.5 mIoU. Its resolution is low (224 or 336 pixels in pretraining), and ViT features degrade at segmentation resolutions (FC-CLIP); SAN, for example, feeds CLIP $320^2$ or $448^2$ inputs and only its side network $640^2$. Masked or cropped regions are out of CLIP’s training distribution (ZegFormer’s crop ablation, OVSeg’s oracle). Fine-tuning on a segmentation dataset’s few hundred classes erodes the open vocabulary (MaskCLIP, LSeg+, FC-CLIP), so most methods freeze CLIP or adapt it through prompts, side networks or cost aggregation. And the benchmarks’ names were never chosen for text encoders: OpenSeg changed “glass” to “drinking glass” to avoid ambiguity.

Reported results

Cross-dataset results as reported by each paper (mIoU, single model, trained on COCO, evaluated on validation sets). Backbones and training data differ, so rows are milestones, not a controlled comparison.

Method	Backbone / VLM	Training data	A-847	PC-459	A-150	PC-59	PASCAL VOC	Source
LSeg+	ResNet-101 (ImageNet init.) / ALIGN text	COCO panoptic	2.5	5.2	13.0	36.0	59.0 (PAS-20b)	OpenSeg Table 2
OpenSeg	ResNet-101 (ImageNet init.) / ALIGN text	COCO	4.0	6.5	15.3	36.9	60.0 (PAS-20b)	OpenSeg Table 2
OpenSeg + prompt eng.	EfficientNet-B7 / ALIGN	COCO + Loc. Narr.	8.8	12.2	28.6	48.2	72.2 (PAS-20b)	OpenSeg Table 2
Simple baseline	ResNet-101 + CLIP ViT-B/16	COCO-Stuff	7.0		20.5	47.7		Xu, Zhang, Wei et al. Table 1
OVSeg	Swin-B + CLIP ViT-L/14	COCO-Stuff + captions	9.0	12.4	29.6	55.7	94.5 (PAS-20)	OVSeg Table 1
SAN	CLIP ViT-L/14	COCO-Stuff	12.4	15.7	32.1	57.7	94.6 (PAS-20)	SAN Table 2
ODISE (labels)	Stable Diffusion + CLIP	COCO panoptic	11.1	14.5	29.9	57.3	84.6 (PAS-21)	ODISE Table 2
FC-CLIP	ConvNeXt-L CLIP	COCO panoptic	14.8	18.2	34.1	58.4	95.4 (PAS-20)	FC-CLIP Table 3
CAT-Seg	CLIP ViT-L/14	COCO-Stuff	16.0	23.8	37.9	63.3	97.0 (PAS-20)	CAT-Seg Table 1

SAN’s numbers are means over five trials, printed with a spread of 0.26–0.42 that the paper calls variance. ODISE’s paper does not name its CLIP model; CAT-Seg’s table lists it as ViT-L/14. The LSeg+ and OpenSeg ResNet-101 rows start from an ImageNet backbone, while the EfficientNet-B7 row starts from ALIGN’s image encoder (Ghiasi et al., 2022, §4.3). The VOC column mixes three protocols, explained below. The papers disagree with each other in places. OVSeg’s table puts OpenSeg’s 60.0, and SAN’s table its 60.0 and 72.2, in the same column as 20-class VOC results, although OpenSeg assigned a background class (PAS-20b); CAT-Seg puts OpenSeg’s results in a separate PAS-20b column. FC-CLIP’s table lists SAN at 13.7 (A-847), 17.1 (PC-459), 33.3 (A-150), 60.2 (PC-59) and 95.5 (PAS-20), higher than the 12.4, 15.7, 32.1, 57.7 and 94.6 of SAN’s own table for its default model, and says FC-CLIP surpasses SAN by 1.1 on A-847 and PC-459. SAN’s CVPR paper mentions an ensembled, non-default variant in its text and table caption but prints no ensembled row, and FC-CLIP’s table does not say where its numbers come from. ODISE’s abstract gives 23.4 PQ and 30.0 mIoU on ADE20K, while its tables give 23.4 PQ with 28.7 mIoU (caption supervision) and 22.6 PQ with 29.9 mIoU (label supervision). OVSeg’s introduction gives A-847 as 846 categories, against 847 in its evaluation section.

Within-dataset zero-shot results in the generalized setting (mIoU of seen and unseen classes and their harmonic mean hIoU):

Method	Setting	VOC seen / unseen / hIoU	COCO-Stuff seen / unseen / hIoU	Source
SPNet-C	inductive	76.84 / 29.33 / 42.45	34.52 / 8.33 / 13.42	SPNet §4.1.2, Fig. 4
ZegFormer (R-101)	inductive	86.4 / 63.6 / 73.3	36.6 / 33.2 / 34.8	ZegFormer Table 4
Simple baseline	inductive	83.5 / 72.5 / 77.5	39.3 / 36.3 / 37.8	Xu, Zhang, Wei et al. Tables 2–3
Simple baseline + self-training	transductive	79.2 / 78.1 / 79.3	39.6 / 43.6 / 41.5	Xu, Zhang, Wei et al. Tables 2–3
MaskCLIP+	transductive	88.8 / 86.1 / 87.4	— / 54.7 / 45.0	MaskCLIP Tables 2–3

The harmonic mean of 79.2 and 78.1 is 78.6, not the 79.3 printed in the simple baseline’s Table 3. Later papers tabulate SPNet on VOC far lower than its own text: 15.6 unseen and 26.1 hIoU in ZegFormer’s and MaskCLIP’s tables, 15.0 and 25.1 in the simple baseline’s. For COCO-Stuff, ZegFormer’s table gives SPNet 35.2 seen, 8.7 unseen and 14.0 hIoU, against SPNet’s own 34.52, 8.33 and 13.42. ZegFormer’s caption says its non-self-training numbers come from the original papers, but these SPNet values do not appear in SPNet’s paper, and the five-unseen-class VOC result these tables give ZS3Net (17.7 unseen, 28.7 hIoU) does not appear in ZS3Net’s, whose VOC splits hold out 2, 4, 6, 8 or 10 classes, as we read them; the source of these numbers is not stated.

Example: matching pixels and masks against fixed class embeddings

The script compares the two families on synthetic scenes. Each $64 \times 64$ image holds three shapes on a grey textured background; a class is a colour (red, green, blue, yellow), a texture (flat, striped) and a shape (disk, rotated square, rotated triangle), 24 classes, of which 8 are unseen. Every colour, texture and shape also occurs among the 16 seen classes.

No pretrained CLIP is used. The “text embedding” of a class is a fixed vector built from the attributes in its name, $\mathbf{t}_c = [\text{onehot(colour)}, \text{onehot(texture)}, \text{onehot(shape)}, 0]/\sqrt{3}$, with background on a tenth axis. It stands in for a text encoder in one respect: names that share words share directions, so a map fitted on seen classes can score unseen ones, which is what SPNet and LSeg rely on. It cannot show whether real text embeddings encode visual attributes, the effect of prompts and synonyms, CLIP’s image-side alignment, the domain gap of masked crops, or vocabularies of hundreds of names.

All three models are linear maps into this fixed space, trained with the cross-entropy of the problem definition. The pixel model (SPNet/LSeg style) maps 12 box-filtered features per pixel (colour, saturation and stripe energy in $5 \times 5$ and $15 \times 15$ windows) into the space. The two-stage model (ZegFormer style) first finds class-agnostic proposals, the connected components of a logistic foreground classifier, then classifies each proposal from its mean pixel features plus two shape descriptors, extent and compactness ($4\pi \cdot \text{area}/\text{perimeter}^2$). Its segment classifier is fitted on ground-truth masks of seen objects; a variant fits it on the training images’ own proposals instead, as mask-classification models train their class heads on predicted masks. Oracle masks apply the ground-truth-fitted classifier to ground-truth masks, OVSeg’s oracle in miniature. The construction builds in its answer: by design, a small window sees colour and stripes but hardly any shape, while a whole mask’s compactness separates disks, squares and triangles almost linearly. Shape is nearly invisible to a pixel and nearly linear for a mask, a deliberately strong form of ZegFormer’s argument.

The sources leave one choice open, so both are printed: pixels of unseen objects in the training images are either ignored in every loss, as in SPNet, or labelled background, as a dataset with a catch-all background class labels objects outside its vocabulary. Scores are generalized: all 24 classes and background compete at test time, IoU is accumulated over the 200 test images of each seed, and hIoU excludes background. The script loops over seeds 0, 1 and 2 and takes about 15 seconds single-threaded with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2; two runs gave identical output.

import os
os.environ.setdefault("OMP_NUM_THREADS", "1")
import numpy as np
from scipy import ndimage
from scipy.optimize import minimize
from skimage.measure import regionprops
from skimage.draw import polygon

COLOURS = {"red": (0.85, 0.15, 0.15), "green": (0.15, 0.7, 0.2),
           "blue": (0.15, 0.25, 0.85), "yellow": (0.85, 0.8, 0.15)}
TEXTURES = ["flat", "striped"]
SHAPES = ["disk", "square", "triangle"]
CLASSES = [(c, t, s) for c in COLOURS for t in TEXTURES for s in SHAPES]      # 24 classes
UNSEEN = [("red", "flat", "disk"), ("red", "striped", "triangle"),
          ("green", "striped", "disk"), ("green", "flat", "square"),
          ("blue", "flat", "triangle"), ("blue", "striped", "square"),
          ("yellow", "striped", "disk"), ("yellow", "flat", "triangle")]
SEEN = [k for k in CLASSES if k not in UNSEEN]
ORDER = SEEN + UNSEEN                      # label 0..15 seen, 16..23 unseen; 24 = background
BG, NS, NC = 24, len(SEEN), len(ORDER)

def embedding(k):
    """Fixed 'text embedding' built from the attributes in the class name."""
    c, t, s = k
    e = np.zeros(10)
    e[list(COLOURS).index(c)] = e[4 + TEXTURES.index(t)] = e[6 + SHAPES.index(s)] = 1
    return e / np.sqrt(3)

E = np.array([embedding(k) for k in ORDER] + [np.eye(10)[9]])     # [25, 10]; row 24 = background

def shape_mask(kind, cy, cx, r, angle, H=64):
    yy, xx = np.mgrid[:H, :H]
    if kind == "disk":
        return (yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2
    n, rad = (4, r * 1.25) if kind == "square" else (3, r * 1.45)
    a = angle + np.arange(n) * 2 * np.pi / n
    m = np.zeros((H, H), bool)
    rr, cc = polygon(cy + rad * np.sin(a), cx + rad * np.cos(a), (H, H))
    m[rr, cc] = True
    lab, n = ndimage.label(m)                     # drop the isolated pixels polygon() can leave
    return lab == 1 + np.argmax(ndimage.sum(m, lab, range(1, n + 1)))

def scene(rng, H=64):
    img = 0.5 + 0.04 * rng.standard_normal((H, H, 1)) * np.ones(3)
    lab = np.full((H, H), BG)
    taken = np.zeros((H, H), bool)
    for _ in range(3):
        k = rng.integers(NC)
        c, t, s = ORDER[k]
        for _ in range(100):                                   # rejection sampling, no overlap
            r = rng.uniform(8, 12)
            cy, cx = rng.uniform(r + 2, H - r - 2, 2)
            m = shape_mask(s, cy, cx, r, rng.uniform(0, 2 * np.pi))
            if not (ndimage.binary_dilation(m, iterations=2) & taken).any():
                break
        col = np.clip(np.array(COLOURS[c]) + 0.05 * rng.standard_normal(3), 0, 1)
        pix = np.ones((H, H, 1)) * col
        if t == "striped":
            pix[:, (np.arange(H) // 2) % 2 == 1] *= 0.45          # vertical stripes, period 4
        img[m] = pix[m]
        lab[m], taken = k, taken | m
    img = np.clip(img + 0.03 * rng.standard_normal(img.shape), 0, 1)
    return img, lab

def pixel_features(img):
    g = img.mean(2)
    dx = np.abs(np.diff(g, axis=1, append=g[:, -1:]))
    dy = np.abs(np.diff(g, axis=0, append=g[-1:, :]))
    sat = img.max(2) - img.min(2)
    maps = [img[..., 0], img[..., 1], img[..., 2], sat, dx, dy]
    f = [ndimage.uniform_filter(m, w, mode="nearest") for w in (5, 15) for m in maps]
    return np.stack(f, -1)                                     # [H, W, 12]

def mask_features(F, m):
    p = regionprops(m.astype(np.uint8))[0]
    extent = p.area / p.area_bbox
    compact = 4 * np.pi * p.area / max(p.perimeter, 1) ** 2
    return np.r_[F[m].mean(0), extent, compact]

def add_bias(X):
    return np.c_[X, np.ones(len(X))]

def fit_embedding_map(X, y, Ecand):
    """Multinomial logistic regression whose class weights are fixed embeddings:
    logits = (X W) Ecand^T; only W is learned (as in SPNet's semantic projection)."""
    X = add_bias(X)
    mu, sd = X.mean(0), X.std(0) + 1e-8
    mu[-1], sd[-1] = 0, 1
    Xs = (X - mu) / sd
    Y = np.eye(len(Ecand))[y]
    d = Xs.shape[1]

    def loss(w):
        W = w.reshape(d, -1)
        Z = Xs @ W @ Ecand.T
        Z -= Z.max(1, keepdims=True)
        P = np.exp(Z); P /= P.sum(1, keepdims=True)
        L = -np.log(P[Y > 0] + 1e-12).mean() + 1e-4 * (w ** 2).sum()
        G = Xs.T @ ((P - Y) / len(X)) @ Ecand + 2e-4 * W
        return L, G.ravel()

    w = minimize(loss, np.zeros(d * Ecand.shape[1]), jac=True, method="L-BFGS-B",
                 options={"maxiter": 2000}).x
    return lambda Xn: ((add_bias(Xn) - mu) / sd) @ w.reshape(d, -1)

def fit_logistic(X, y):
    X = add_bias(X)
    mu, sd = X.mean(0), X.std(0) + 1e-8
    mu[-1], sd[-1] = 0, 1
    Xs = (X - mu) / sd

    def loss(w):
        z = Xs @ w
        p = 1 / (1 + np.exp(-z))
        L = -(y * np.log(p + 1e-12) + (1 - y) * np.log(1 - p + 1e-12)).mean() + 1e-4 * w @ w
        return L, Xs.T @ (p - y) / len(y) + 2e-4 * w

    w = minimize(loss, np.zeros(Xs.shape[1]), jac=True, method="L-BFGS-B").x
    return lambda Xn: ((add_bias(Xn) - mu) / sd) @ w > 0

def iou_table(preds, labs):
    I, U = np.zeros(NC + 1), np.zeros(NC + 1)
    for p, l in zip(preds, labs):
        for k in range(NC + 1):
            a, b = p == k, l == k
            I[k] += (a & b).sum(); U[k] += (a | b).sum()
    iou = I / np.maximum(U, 1)
    s, u = iou[:NS].mean(), iou[NS:NC].mean()
    return s, u, 2 * s * u / (s + u + 1e-12), iou[BG]

def proposals(fg_clf, F, min_px=30):
    comps, n = ndimage.label(fg_clf(F.reshape(-1, 12)).reshape(F.shape[:2]))
    return [comps == c for c in range(1, n + 1) if (comps == c).sum() >= min_px]

def objects(l):
    out = []
    for k in np.unique(l[l != BG]):
        lk, nk = ndimage.label(l == k)
        out += [(k, lk == c) for c in range(1, nk + 1)]
    return out

def paint(masks, F, clf, shape):
    p, names = np.full(shape, BG), []
    for m in masks:
        p[m] = (clf(mask_features(F, m)[None]) @ E[:NC].T).argmax()
        names.append(p[m][0])
    return p, names

def run(seed, n_train=300, n_test=200):
    rng = np.random.default_rng(seed)
    train = [scene(rng) for _ in range(n_train)]
    test = [scene(rng) for _ in range(n_test)]
    Ftr = [pixel_features(im) for im, _ in train]
    Fte = [pixel_features(im) for im, _ in test]
    Xall = np.concatenate([f.reshape(-1, 12) for f in Ftr])
    Lall = np.concatenate([l.ravel() for _, l in train])
    pick = rng.choice(len(Xall), 60000, replace=False)
    # segment classifier fitted on ground-truth masks of seen objects in the training images
    gt = [(mask_features(F, m), k) for (im, l), F in zip(train, Ftr) for k, m in objects(l) if k < NS]
    seg_gt = fit_embedding_map(np.array([f for f, _ in gt]), np.array([k for _, k in gt]), E[:NS])
    labs = [l for _, l in test]
    out = {}
    for policy in ("ignore", "background"):
        X, y = Xall[pick], Lall[pick].copy()
        unseen = (y >= NS) & (y < NC)
        if policy == "ignore":
            X, y = X[~unseen], y[~unseen]
        else:
            y[unseen] = BG
        y_pix = np.where(y == BG, NS, y)                      # index into [seen..., background]
        pix_clf = fit_embedding_map(X, y_pix, np.r_[E[:NS], E[BG:]])
        fg_clf = fit_logistic(X, (y != BG).astype(float))
        # segment classifier fitted on the training images' own proposals (majority label, seen only)
        tp = []
        for (im, l), F in zip(train, Ftr):
            for m in proposals(fg_clf, F):
                ks, cnt = np.unique(l[m], return_counts=True)
                if ks[cnt.argmax()] < NS:
                    tp.append((mask_features(F, m), ks[cnt.argmax()]))
        seg_pr = fit_embedding_map(np.array([f for f, _ in tp]), np.array([k for _, k in tp]), E[:NS])
        P = {m: [] for m in ("pixel", "two-stage", "two-stage, fit on proposals", "oracle masks")}
        shape_err, wrong, compact = [], {"two-stage": [], "two-stage, fit on proposals": []}, []
        for (im, l), F in zip(test, Fte):
            p = (pix_clf(F.reshape(-1, 12)) @ E.T).argmax(1).reshape(l.shape)   # 24 classes + bg
            P["pixel"].append(p)
            for k in range(NS, NC):                            # unseen-object pixels
                q = p[l == k]
                if q.size:
                    wrong_shape = [(j < NC and j != k and ORDER[j][:2] == ORDER[k][:2]) for j in q]
                    shape_err.append((np.sum(wrong_shape), q.size))
            props = proposals(fg_clf, F)
            for method, clf in (("two-stage", seg_gt), ("two-stage, fit on proposals", seg_pr)):
                p2, names = paint(props, F, clf, l.shape)
                P[method].append(p2)
                for m, j in zip(props, names):
                    ks, cnt = np.unique(l[m], return_counts=True)
                    k = ks[cnt.argmax()]
                    if k != BG:
                        wrong[method].append((k >= NS, ORDER[j][2] != ORDER[k][2]))
            P["oracle masks"].append(paint([m for _, m in objects(l)], F, seg_gt, l.shape)[0])
            for m in props:
                ks, cnt = np.unique(l[m], return_counts=True)
                k = ks[cnt.argmax()]
                if k != BG and ORDER[k][2] != "disk":
                    g = (l == k) & ndimage.binary_dilation(m, iterations=3)
                    compact.append((ORDER[k][2], mask_features(F, m)[-1], mask_features(F, g)[-1]))
        se = np.array(shape_err).sum(0)
        out[policy] = {m: iou_table(P[m], labs) for m in P}
        out[policy]["shape_err"] = se[0] / se[1]
        out[policy]["wrong"] = {m: [np.mean([w for u, w in v if u == flag]) for flag in (False, True)]
                                for m, v in wrong.items()}
        out[policy]["compact"] = {s: np.mean([(a, b) for t, a, b in compact if t == s], 0)
                                  for s in ("square", "triangle")}
    return out

if __name__ == "__main__":
    res = {s: run(s) for s in (0, 1, 2)}
    rng_ = lambda v: f"{np.mean(v):.2f} ({np.min(v):.2f}-{np.max(v):.2f})"
    print("generalized setting: 16 seen + 8 unseen classes + background; mean over seeds 0-2 (min-max)")
    print(f"{'unseen pixels':11s} {'method':28s} {'seen mIoU':17s} {'unseen mIoU':17s} {'hIoU':17s} bg IoU")
    for policy in ("ignore", "background"):
        for method in ("pixel", "two-stage", "two-stage, fit on proposals", "oracle masks"):
            cells = [rng_([res[s][policy][method][c] for s in res]) for c in range(4)]
            print(f"{policy:11s} {method:28s} " + " ".join(f"{x:17s}" for x in cells))
    for policy in ("ignore", "background"):
        print(f"\n{policy}: pixel model, unseen-object pixels given the right colour and texture "
              f"but the wrong shape: " + rng_([res[s][policy]["shape_err"] for s in res]))
        for method in ("two-stage", "two-stage, fit on proposals"):
            print(f"{policy}: {method}, proposals named with the wrong shape: seen objects "
                  + rng_([res[s][policy]["wrong"][method][0] for s in res]) + ", unseen objects "
                  + rng_([res[s][policy]["wrong"][method][1] for s in res]))
        for sh in ("square", "triangle"):
            v = np.array([res[s][policy]["compact"][sh] for s in res])
            print(f"{policy}: {sh} compactness, proposal {v[:, 0].mean():.2f} vs ground truth {v[:, 1].mean():.2f}")

generalized setting: 16 seen + 8 unseen classes + background; mean over seeds 0-2 (min-max)
unseen pixels method                       seen mIoU         unseen mIoU       hIoU              bg IoU
ignore      pixel                        0.17 (0.17-0.18)  0.22 (0.20-0.25)  0.19 (0.19-0.20)  0.99 (0.99-0.99)
ignore      two-stage                    0.69 (0.66-0.75)  0.77 (0.75-0.81)  0.73 (0.70-0.78)  0.99 (0.99-0.99)
ignore      two-stage, fit on proposals  0.76 (0.69-0.83)  0.66 (0.51-0.81)  0.70 (0.58-0.82)  0.99 (0.99-0.99)
ignore      oracle masks                 0.96 (0.96-0.97)  0.94 (0.93-0.96)  0.95 (0.94-0.97)  1.00 (1.00-1.00)
background  pixel                        0.12 (0.11-0.14)  0.18 (0.17-0.21)  0.15 (0.14-0.16)  0.90 (0.89-0.91)
background  two-stage                    0.49 (0.45-0.51)  0.51 (0.50-0.53)  0.50 (0.48-0.51)  0.92 (0.92-0.93)
background  two-stage, fit on proposals  0.36 (0.34-0.39)  0.29 (0.25-0.35)  0.32 (0.29-0.37)  0.92 (0.92-0.93)
background  oracle masks                 0.96 (0.96-0.97)  0.94 (0.93-0.96)  0.95 (0.94-0.97)  1.00 (1.00-1.00)

ignore: pixel model, unseen-object pixels given the right colour and texture but the wrong shape: 0.48 (0.36-0.54)
ignore: two-stage, proposals named with the wrong shape: seen objects 0.19 (0.14-0.22), unseen objects 0.12 (0.09-0.13)
ignore: two-stage, fit on proposals, proposals named with the wrong shape: seen objects 0.13 (0.07-0.16), unseen objects 0.17 (0.06-0.32)
ignore: square compactness, proposal 0.86 vs ground truth 0.80
ignore: triangle compactness, proposal 0.71 vs ground truth 0.64

background: pixel model, unseen-object pixels given the right colour and texture but the wrong shape: 0.26 (0.23-0.32)
background: two-stage, proposals named with the wrong shape: seen objects 0.23 (0.22-0.25), unseen objects 0.22 (0.20-0.23)
background: two-stage, fit on proposals, proposals named with the wrong shape: seen objects 0.35 (0.33-0.38), unseen objects 0.40 (0.34-0.44)
background: square compactness, proposal 0.70 vs ground truth 0.79
background: triangle compactness, proposal 0.59 vs ground truth 0.65

The fixed embeddings do transfer: classified on correct masks, the eight unseen classes reach 0.94 mIoU, almost the seen classes’ 0.96, because each unseen name is a new combination of directions the segment map has learned. The pixel model fails on seen and unseen classes alike (0.17 and 0.22): about half of the pixels of unseen objects go to a class with the right colour and texture but the wrong shape, the attribute a local window barely sees. This mirrors, in a form the construction exaggerates, the gap ZegFormer measured between segment-level classification and a pixel-level SPNet. In our reading, the pixel model’s unseen mIoU is not below its seen mIoU because its failure is shape blindness rather than a bias toward seen classes.

The two-stage model sits between them, at 0.77 unseen mIoU, which does not support the plan’s expectation of a result “within 0.1 of the oracle-mask result”. The printed diagnostics locate the gap in naming, not grouping: the box-filtered foreground rounds corners, raising the compactness of square proposals from 0.80 to 0.86 and of triangles from 0.64 to 0.71, so a classifier fitted on clean masks misreads the shape of 12–19% of proposals. This is OVSeg’s finding at small scale: a segment classifier meets proposals that differ from what it was trained on. Fitting the classifier on proposals instead moved the shape errors rather than removing them, from seen objects (0.19 to 0.13) to unseen ones (0.12 to 0.17, up to 0.32 for one seed), and unseen mIoU fell to 0.66. By our reading, a classifier fitted on noisy proposals of seen classes can lean on which colour–texture pairs occur with which shapes among the seen classes, a regularity that unseen classes break; this is the overfitting to seen classes that FC-CLIP and CAT-Seg describe when a model adapts to its training vocabulary, though the example does not isolate the mechanism.

Labelling unseen objects as background hurt both families, against the plan’s expectation that it would lower pixel-level unseen mIoU “by more than 0.1” and change the two-stage result “by less than 0.05”. The pixel model lost only 0.04, having already misnamed most of those pixels; the two-stage model lost 0.26. The printed numbers point to the foreground classifier, which was told that about a third of the coloured objects are background: background IoU falls from 0.99 to 0.92 as object pixels go unclaimed, and the proposals become less compact than the objects (squares 0.70 against 0.79), the reverse of the corner rounding seen under “ignore”. A class-agnostic proposal network generalizes to new classes only if its training labels do not call those classes background, which is why the ignore protocol matters for two-stage models in particular.

The example was planned before it was first run, and the script was changed three times after results had been seen. The first run printed three rows for each policy (“pixel”, “two-stage” and “oracle masks”) and the pixel model’s shape-error share; the two-stage model’s shortfall against the oracle was then traced in two diagnostic runs, after which the shape-error shares and the compactness of proposals were added to the printout (a change to what is printed only). Because those diagnostics pointed to a classifier fitted on clean masks meeting rounder proposals, the “fit on proposals” row was then added as a new case. Its hypothesis was written in the plan before it was run: “under ‘ignore’ its unseen mIoU comes within 0.1 of the oracle masks; under ‘background’ it recovers less than half of the gap to the oracle”. The first part was not met; the second holds only trivially, since the row fell below the ground-truth-fitted classifier. Finally, the wrong-shape shares were split into seen and unseen objects, also a print-only change. The rows of the first run are unchanged in the output above.

Datasets and Benchmarks

Within-dataset zero-shot splits.

• PASCAL VOC 2012: 15 seen and 5 unseen classes, the last five in VOC order (potted plant, sheep, sofa, train, tv monitor). SPNet’s rule of choosing unseen classes absent from ImageNet-1K could not be applied here (only 4 VOC classes are absent, “person” has no similar class, and all vehicles appear in ImageNet), so its authors simply took the last five (Xian et al., 2019, §4.1). Training uses the SBD-augmented set, reported as 10,582 images by ZegFormer and as 11,185 by SPNet and the simple baseline, and testing the 1,449 validation images, with the background class ignored (Zhou et al., 2022, §4.2). ZS3Net instead used nested random splits of 2, 4, 6, 8 and 10 unseen classes (2: cow, motorbike; 4: adds airplane, sofa; and so on). The PASCAL VOC article describes the dataset.
• COCO-Stuff: 15 unseen classes, as listed by Zhou et al. (2022, §4.2) (frisbee, skateboard, cardboard, carrot, scissors, suitcase, giraffe, cow, road, wall-concrete, tree, grass, river, clouds, playing field). SPNet picked unseen classes that are not in ImageNet-1K. Its table counts 155 training plus 12 validation classes as seen; ZegFormer and the simple baseline give 156 seen of 171 valid classes. Training uses the 118K training images and testing the 5,000 validation images.
• PASCAL Context (59 classes): ZS3Net’s nested splits of 2 to 10 unseen classes (cow, motorbike, sofa, cat, boat, fence, bird, tv monitor, keyboard, aeroplane), with background not ignored (Zhou et al., 2022, §4.2).
• ADE20K-Full: ZegFormer’s split of the 847 classes into 572 seen classes (present in more than 10 images) and 275 unseen (in fewer than 10); there even a fully supervised MaskFormer reached only 5.6 unseen mIoU (Table 5).

Cross-dataset benchmarks. The standard suite since OpenSeg and OVSeg trains on COCO and tests on five label sets, from the official code of CAT-Seg (commit 6d3a188) where the papers are silent:

• A-150: ADE20K’s SceneParse150, 2,000 validation images, as in supervised segmentation; the ADE20K article describes the dataset and its official scoring.
• A-847: the same images labelled with 847 ADE20K-Full classes. MaskFormer introduced the version as the classes present in both training and validation sets of the open-vocabulary annotation (Cheng et al., 2021, appendix); its dataset section says 874, its introduction, tables and appendix 847. OpenSeg adopted it from MaskFormer. In CAT-Seg’s code, labels come from the ADE20K_2021_17_01 release, images whose shorter side exceeds 512 pixels are resized to 512, and every object id outside the 847 is ignored. Class names are WordNet-style synonym lists such as “stove, kitchen stove, range, kitchen range, cooking stove”.
• PC-59: PASCAL Context’s 59 most frequent classes (Mottaghi et al., 2014), with the rest as background; in CAT-Seg’s code the background is ignored in evaluation. Mottaghi et al. labelled the 10,103 trainval images of PASCAL VOC 2010 with 540 categories; the validation half has 5,105 images according to ZS3Net and MaskCLIP and 5,005 according to the simple baseline, and CAT-Seg’s validation list holds 5,104 identifiers.
• PC-459: the full label set of PASCAL Context’s released annotations (labels.txt, 459 names). CAT-Seg’s conversion script asserts that no pixel is unlabelled, and its ignore label (459) lies outside the 459 class indices, so nothing is ignored, and the name list includes “unknown”, the label annotators gave regions they could not recognize, which therefore counts as a class. The 459 is the number of labels in the release; the paper describes 540 categories.
• PASCAL VOC: three conventions. PAS-20 scores the 20 object classes with background ignored (OVSeg, SAN, CAT-Seg); PAS-20b, from OpenSeg, keeps background and predicts it wherever a PC-59 class that is not a VOC class wins, because the word “background” is ambiguous (Ghiasi et al., 2022, §4.1); PAS-21 scores 20 classes plus a background class (ODISE, FC-CLIP). FC-CLIP reports 95.4 on PAS-20 and 81.8 on PAS-21 for the same model, so the conventions are not interchangeable.

SAN’s authors measured how close each label set is to COCO-Stuff’s, by the Hausdorff distance between label sets with CLIP text similarities: 0.91 for VOC, 0.86 for PC-59, 0.73 for A-150, 0.70 for PC-459 and 0.57 for A-847 (Table 1). They read the first two as measuring in-domain recognition and the others cross-domain. CAT-Seg also reports MESS, a suite of 22 domain-specific datasets (earth monitoring, medicine, engineering, agriculture and biology) on which open-vocabulary models remain far below supervised ones: CAT-Seg with ViT-L/14 averaged 34.70 against 70.99 for the best supervised models (Table 2).

Evaluation Metrics

Results are reported as mean intersection over union, accumulated over the dataset for each class and averaged over classes, with conventions that change the number:

• Seen, unseen and harmonic mean. For zero-shot splits, mIoU is computed over seen and unseen classes separately, with every class a candidate, and summarized by $\text{hIoU} = 2\,\text{mIoU}_s\,\text{mIoU}_u / (\text{mIoU}_s + \text{mIoU}_u)$ (Xian et al., 2019, Eq. 7). The harmonic mean is low whenever either term is, so a model that never predicts unseen classes scores 0 however good it is on seen ones. ZS3Net’s Table 2 shows what restricting the candidates to unseen classes hides: on its 10-unseen VOC split, the baseline’s unseen mIoU was 1.9 in the generalized evaluation and 41.7 when seen-class scores were ignored.
• Vocabulary per image. Standard mIoU uses the full vocabulary for every image; OpenSeg’s grounding mIoU uses only the classes present, and gave, for example, 32.1 against 9.0 for the same OpenSeg model on PC-459 (Ghiasi et al., 2022, Fig. 5).
• Background and ignore labels (PAS-20, PAS-20b, PAS-21; PC-59 with or without background; unknown in PC-459), as described above.
• Panoptic and instance variants use PQ and mask AP on ADE20K, Cityscapes and Mapillary Vistas, as in the panoptic segmentation article.

Applications

The papers emphasize uses where categories cannot be listed in advance. MaskCLIP segmented web images with fine-grained or novel names (“red car” against “yellow car”, Batman and Joker) without annotation, and suggests its maps as free labels for training other segmenters. X-Decoder’s authors combined it with Stable Diffusion for text-driven image editing, and SAN’s authors visualize ImageNet categories segmented with a COCO-trained model. The cross-domain MESS results suggest, by our reading, that applications in specialized domains such as medical or earth-observation imagery cannot yet rely on zero-shot transfer.

Open Problems

• Fair evaluation. No benchmark guarantees that its classes were never seen: CLIP, ALIGN and Stable Diffusion were trained on web captions, COCO’s classes overlap with VOC and PC-59, and SAN’s label-set similarities show how much of each benchmark is near the training vocabulary.
• Large vocabularies. The highest A-847 mIoU among the papers read here is 16.0, for CAT-Seg, and the 459- and 847-name lists contain synonyms, parts and labels such as “unknown”.
• Adapting CLIP without forgetting. Freezing, prompting, side networks and cost aggregation are different answers to the same trade-off between fitting the training vocabulary and keeping the open one.
• Domain shift. Beyond the gap MESS measures, the example shows that a proposal network’s training labels and a classifier’s training masks both decide what transfers.