Open-Vocabulary Object Detection

Open-vocabulary object detection asks a detector to find objects of categories that are named in text when the detector is used, including categories for which it never saw a box during training. A closed-set object detector ends in a classifier with one learned weight vector per training category; an open-vocabulary detector replaces those weights with embeddings of the category names, so that each region is scored by its similarity to the text of whatever names the user supplies. The idea was first studied as zero-shot object detection with word vectors; from 2021 it was reframed around image–text data and, soon after, around CLIP and grounding data. Related names are open-set detection, text-conditioned detection and, for the variant that must also flag and later learn unknown objects, open-world detection. Terminology differs between papers, and the differences change what a reported number means (see Problem Definition). This is an active research area; methods, results and repository states below are dated as of November 2026.

Problem Definition

A detector is trained on images with boxes for a set of base categories $\mathcal{C}_B$ and evaluated on a vocabulary $\mathcal{C}_T$ given at test time, which contains novel categories $\mathcal{C}_N$ with no box annotations in training. In Bansal et al.’s (2018) zero-shot detection the base classes are called seen and the novel ones unseen; Zareian et al. (2021) call them base and target; most later papers say base and novel. For a region with feature $\mathbf{f}$ (from a region proposal, an anchor or a decoder query), a learned map $g$ produces a region embedding, and each category $k$ is represented by a fixed text embedding $\mathbf{t}_k$ of its name. The class scores are

$$s_k(\mathbf{f}) = \frac{\langle g(\mathbf{f}), \mathbf{t}_k \rangle}{\tau}, \qquad k \in \mathcal{C}_T,$$

with a dot product or cosine similarity and a temperature $\tau$. Training fits $g$ (and usually the backbone and box head) on base categories only; at test time the rows $\mathbf{t}_k$ are swapped for the test vocabulary, without retraining. Localization is trained as in any detector and is assumed to generalize to new categories; Detic’s authors found region proposals to generalize well enough, as they say ViLD had also reported (Zhou et al., 2022, §2).

What may be used in training differs between formulations, and the papers say so:

• Zero-shot detection (Bansal et al., 2018) allows only boxes for seen classes and a word embedding of every class name. Bansal et al. removed every COCO training image that contained an unseen object.
• Open-vocabulary detection in Zareian et al.’s sense adds image–caption pairs, which may mention novel categories, and keeps the target classes unknown during training; in their words, target classes can be “any subset of the entire language vocabulary”.
• OWL-ViT’s authors use open-vocabulary for any detection from text queries, with categories that may overlap between training and testing, and reserve zero-shot for categories with no localized training instances (Minderer et al., 2022, §2).
• Grounding DINO’s authors treat open-set, open-world and open-vocabulary detection as one task and use zero-shot to mean that the training split of the test dataset is not used, a definition under which the test categories may well have been annotated in other datasets (Liu et al., 2024). GLIP’s authors likewise describe their COCO and LVIS results as zero-shot domain transfer and do not exclude any categories from training (Li et al., 2022, §2).

The task article’s numbers therefore come in two families: benchmark protocols with held-out novel categories (OV-COCO and OV-LVIS, below), and transfer to a whole dataset whose training split was unused, where only categories that no training source annotates are truly unseen.

Input and Output

The input is an image and a vocabulary: a list of category names, often wrapped in prompt templates such as “a photo of a {}”, or, for grounding-based detectors, a sentence or phrase. The output is the usual detection set, boxes with a label from the vocabulary and a score, and masks for detectors with a mask head, since the LVIS protocol scores masks. Two families differ in how the vocabulary enters the model. Detectors with late fusion (ViLD, Detic, OWL-ViT) embed each name once, independently of the image, so thousands of names cost little more than a matrix product; OWL-ViT used all 1,203 LVIS names as queries for every image. Detectors with early fusion (GLIP, Grounding DINO) feed the prompt into the image network, so the vocabulary changes the image features and a long vocabulary has to be split into several prompts and forward passes; OWL-ViT’s authors give this as the reason they avoided fusion (Minderer et al., 2022, §3.1).

Some models also accept an image as the query. Because OWL-ViT has no fusion, the embedding of a box in a reference image can replace a text embedding, giving one-shot, image-conditioned detection of things that are hard to name, such as specialized technical parts, in the authors’ example (Minderer et al., 2022, §2, §3.1).

Variants

• Referring and phrase-level detection. The text describes one object (“the left lion”) rather than a category; this is the subject of visual grounding, and grounding-based detectors handle both.
• Image-conditioned (one-shot) detection. The query is an example image; on COCO categories held out from training, OWL-ViT reached 41.8 one-shot AP50 against 24.3 for the best earlier method, AIT (means of four category splits; Minderer et al., 2022, Table 2). The paper’s text gives AIT’s figure as 26.0, its result on the first split.
• Open-vocabulary segmentation applies the same region–text scoring to masks; LVIS-based protocols already score masks.

Open-world detection

Open-world detection, defined by Joseph et al. (2021), sets a different goal: the detector has no names for new categories at all. It must label objects of classes it was not trained on as unknown, and later, when an oracle supplies labels for some of these unknowns, learn the new classes incrementally without forgetting the old ones. Joseph et al. motivated it with the observation, which they attribute to Dhamija et al. (2020), that standard detectors often assign unknown objects to known classes with high confidence, and added that unlabelled unknown objects in training images are learned as background. Their protocol groups the 20 Pascal VOC classes and 60 further COCO classes into four tasks introduced one after another, and measures confusion with wilderness impact (the relative drop in precision on known classes when unknown objects are added, at recall 0.8) and absolute open-set error (the number of unknown objects labelled as a known class), alongside mAP on known classes. Their method, ORE, used contrastive clustering and an energy-based unknown identifier. Open-vocabulary detection assumes the user can name what they want; open-world detection assumes they cannot and asks for a separate “unknown” output.

Challenges

• The background class. A closed-set detector learns a background class from every region that matches no annotated object. With novel categories held out, regions on novel objects in training images are labelled background, which teaches the detector to suppress exactly what it should later detect. Bansal et al. (2018, §3.2) discussed it at length: a single fixed background vector risks mapping unseen objects onto it, so they also spread background boxes over a large vocabulary by an EM-like latent assignment. Zareian et al. fixed the background embedding at zero and down-weighted the loss of background proposals by a factor $\alpha$ found by cross-validation ($\alpha = 0.2$ in most of their experiments), which they found improved target classes significantly at a small cost on base classes (§3.2, §4.2).
• Base-class bias. The region-to-text map is fitted on base classes only. Zareian et al. describe the earlier projection approach as overfitting the small set of base embeddings, to the point that the best zero-shot method’s target mAP was nine times lower than its base mAP; their method closed this ratio to about two (§3).
• Image-level versus region-level alignment. Image–text models such as CLIP align whole images with captions. RegionCLIP’s authors cropped ground-truth LVIS boxes and classified them with CLIP: accuracy fell from about 60% on ImageNet to 19% on LVIS regions with a similar number of classes, and CLIP’s scores on proposals did not reflect localization quality (Zhong et al., 2022, Fig. 1).
• Unseen does not mean absent. Captions, image-level labels and web pseudo-labels can name novel categories; OWLv2’s authors note that a model pseudo-labelled with a curated vocabulary including LVIS names is not fully open-vocabulary on LVIS, although no human box for an LVIS rare class was used (Minderer et al., 2023, §3.1). Pretraining corpora can also contain copies of test images, which OWL-ViT and OWLv2 removed by deduplication against COCO, LVIS and other datasets.
• Prompt and vocabulary dependence. Scores depend on how names are written. GLIP’s authors found that with a BERT text encoder the prompt “person. bicycle. car. … . toothbrush” worked better than a more natural sentence (Li et al., 2022, §3), and in YOLO-World’s ablation a frozen CLIP text encoder gave 14.5 $\mathrm{AP}_r$ on LVIS minival against 3.4 for a frozen BERT (Cheng et al., 2024, Table 5; the text gives the gain as 10.1).

Approaches

Zero-shot detection with word embeddings

Bansal et al. (2018) posed zero-shot object detection on COCO and Visual Genome; other groups proposed the problem in the same year, among them Demirel et al. with a hybrid region embedding. Their baseline was an R-CNN-style pipeline: EdgeBoxes proposals, Inception-ResNet-v2 features, and a linear projection of each box feature into a 300-dimensional word-embedding space (GloVe or fastText), trained with a max-margin loss so that a box scored higher with its class embedding than with any other seen class; at test time a box takes its nearest unseen class embedding. They added a static background vector (SB), a latent assignment of background boxes to a large vocabulary (LAB), and densely sampled extra classes from Open Images (DSES). On their COCO split, recall@100 at IoU 0.5 for the 17 unseen classes was 22.1% for the baseline, 24.4% for SB and 27.2% for DSES, while the mAP values printed alongside were below 1% (Table 1); they used recall as the main metric because Visual Genome is not exhaustively labelled. Later zero-shot methods, the polarity loss (PL) of Rahman et al. and the feature-generating DELO of Zhu et al., reached 10.0 and 7.6 mAP at IoU 0.5 on the same 17 classes, DELO with the target classes known in training, as tabulated by Zareian et al. (Table 1).

The lesson the next papers drew was that word vectors trained on text alone, and a projection fitted to 48 base embeddings, give a semantic space too sparse to generalize from (Bansal et al. already argued for “dense sampling” of the label space).

Region–text alignment

All later methods share the scoring rule of Problem Definition and differ in where the region embeddings and text embeddings come from and how they are aligned. Zareian et al.’s classification head (Eq. 6) is typical of the two-stage detectors:

$$p(i \text{ is } k) = \frac{\exp \langle \mathbf{e}_i, \mathbf{t}_k \rangle}{1 + \sum_{k' \in \mathcal{C}_B} \exp \langle \mathbf{e}_i, \mathbf{t}_{k'} \rangle},$$

where the 1 comes from a background class with a fixed all-zero embedding. GLIP writes the same thing for a whole prompt: with region features $O \in \mathbb{R}^{N \times d}$ from the image encoder and contextual token features $P \in \mathbb{R}^{M \times d}$ from the language encoder, the alignment scores $S_{\text{ground}} = O P^\top$ replace a detector’s classification logits, a category name’s score being the average over its tokens (Li et al., 2022, Eq. 3). Detection becomes phrase grounding with a prompt made of category names, and the classifier weights become a function of the text.

Alignment is learned in three main ways: by pretraining the visual backbone on image–text data so that its region features already live in the text space (OVR-CNN, RegionCLIP), by distilling a frozen image–text model’s region embeddings into the detector (ViLD), or by training on large grounding and pseudo-labelled data so that detection and grounding share one objective (GLIP, Grounding DINO, OWLv2’s self-training).

Caption pretraining: OVR-CNN

Zareian et al. (2021) proposed the open-vocabulary formulation and the method OVR-CNN. A ResNet-50 and a linear vision-to-language (V2L) layer were pretrained on COCO Captions with a weakly supervised grounding loss: each caption word attends over image regions, and the resulting image–caption score is trained contrastively against other images and captions in the batch, with masked language modeling and image–text matching as auxiliary losses through a multimodal transformer. The pretrained ResNet and V2L layer then initialized a Faster R-CNN whose classifier was the frozen V2L layer followed by fixed BERT input embeddings of the class names, trained on the 48 base classes. On COCO they reported 27.5 AP50 on the 17 target classes (target-only vocabulary) and 22.8 in the generalized setting, against 10.0 for the best zero-shot baseline (Table 1). Their ablations showed where the gain came from: without caption pretraining, target AP50 fell to 4.4; without transferring the V2L layer, to 4.9; with the V2L layer unfrozen during detection training, to 23.4; and with web Conceptual Captions instead of COCO Captions, to 16.7 (Table 2).

Distilling CLIP: ViLD

ViLD (Gu, Lin, Kuo and Cui; ICLR 2022 according to the reference lists of Detic and YOLO-World, while OWL-ViT and OWLv2 cite the arXiv preprint) brought CLIP into the task. We could not read the paper itself; the description here follows the authors’ official repository README and later papers that report ViLD. In the README’s description, a pretrained open-vocabulary image classifier (CLIP) acts as teacher: it encodes the category names and the image regions of object proposals, and a two-stage detector (a Mask R-CNN, according to Detic’s reproduction) is trained so that its region embeddings are aligned with both. Detic’s authors describe ViLD-text, the variant with CLIP text embeddings as classifier and no distillation, as identical to their own box-supervised baseline, and the full ViLD as adding distillation from CLIP’s visual backbone; ViLD-ensemble combines the two models (Zhou et al., 2022, §5.4). OWL-ViT’s authors add that the distillation targets came from CLIP or ALIGN applied to crops from a class-agnostic proposal network, which they argue limits generalization (Minderer et al., 2022, §2). RegionCLIP’s authors note that ViLD trained with large-scale jittering and a 16× schedule; Detic’s appendix gives ViLD’s schedule as 32× and says that half of it sufficed to match ViLD’s baselines.

ViLD also defined the LVIS protocol used since: all 337 rare categories of LVIS v1.0 are held out as novel, as later papers state when they adopt it (Zhou et al., 2022, §5; Zhong et al., 2022, §4). The numbers reported for it vary by source. Detic’s and OV-DETR’s tables give ViLD-text 10.1, ViLD 16.1 and ViLD-ensemble 16.6 mask $\mathrm{AP}_r$ with ResNet-50 (Zhou et al., 2022, Table 2; Zang et al., 2022, Table 3), and the README states 16.1 mask $\mathrm{AP}_r$ in its summary. The README’s table of released checkpoints lists 16.6 for ViLD and 18 for ViLD-ensemble with ResNet-50 (metric not stated), so the released checkpoints do not match the numbers other papers report for ViLD. RegionCLIP’s table gives 16.7 for ViLD with ResNet-50 without saying whether box or mask AP. On OV-COCO, OV-DETR’s table gives ViLD-text 5.9 and ViLD 27.6 novel AP50 (generalized setting). The README also states that ViLD transferred without fine-tuning to Pascal VOC (72.2 AP50), COCO (36.6 AP) and Objects365 (11.8 AP).

GLIP’s reference list cites the arXiv preprint under a different title, “Zero-shot detection via vision and language knowledge distillation”.

Region-level pretraining: RegionCLIP

RegionCLIP (Zhong et al., 2022) addressed the image–region gap directly. Starting from CLIP, it parsed object concepts from a caption corpus, filled them into prompt templates (“A photo of a kite”), and let the CLIP teacher assign each proposal from a proposal network (trained on LVIS base boxes without labels) its best-matching concept. The student visual encoder was then pretrained on these pseudo region–text pairs with a region-level contrastive loss, a distillation loss towards the teacher’s distribution over concepts, and an image-level contrastive loss on the original captions. Fine-tuned as a Faster R-CNN (ResNet-50-C4, standard 1× schedule) on base classes, it reached 31.4 novel AP50 on OV-COCO (generalized) from CC3M pretraining and 39.3 with a ResNet-50×4 backbone, and 22.0 $\mathrm{AP}_r$ on OV-LVIS with ResNet-50×4, against 19.8 for ViLD with ResNet-152 as RegionCLIP tabulates it (Tables 1 and 2). Initialized from plain CLIP instead of its region-pretrained encoder, the same detector reached 14.2 novel AP50 on OV-COCO.

Image-level labels: Detic

Detic (Zhou et al., 2022) widened the vocabulary with classification data instead of captions. It used CLIP text embeddings of the class names as the classifier of a two-stage detector (CenterNet2 in the main experiments) and trained it on detection data and image-labelled data in alternating batches. For an image with only image-level labels, it applied a binary cross-entropy loss for the labelled classes to the features of the largest proposal, with no attempt to find the right box; only the classifier was trained on such images. The authors argue that prediction-based assignment, as in YOLO9000, fails for novel classes because the detector cannot yet recognize them, and that the max-size proposal covered 92.8% of target objects on an annotated subset of their ImageNet data, against 69.0% for a prediction-based choice (§5.2). On OV-LVIS with the 997 ImageNet-21K classes that overlap LVIS, novel mask AP rose from 16.3 for the box-supervised baseline to 24.6, close to the 25.5 of a detector trained with boxes for all classes (Table 1, means of three runs); in ViLD’s training setting it reached 17.8 against ViLD-ensemble’s 16.6 (Table 2), and on OV-COCO with OVR-CNN’s architecture and COCO captions 27.8 novel AP50 (Table 3). A classifier ablation isolates the role of the text embeddings: the box-supervised baseline reached 16.4 novel mask AP with CLIP embeddings, 9.0 with fastText word vectors, and 0 with a trained classifier, which has no weights for unseen classes (Table 5). Trained with all 21K ImageNet classes and a Swin-B backbone, Detic was evaluated without fine-tuning on Objects365 (21.5 box AP) and Open Images (55.2 AP50).

Detic’s title promises twenty thousand classes; its benchmarks use image labels from classes that include the LVIS rare categories, so OWLv2’s authors place it under “human-curated vocabulary”, where the evaluation vocabulary may be accessed at training time (Minderer et al., 2023, Table 1).

Transformer detectors: OV-DETR and OWL-ViT

DETR’s Hungarian matching needs a classification cost for every class, which is unavailable for novel classes. OV-DETR (Zang et al., 2022) turned detection into conditional binary matching: the decoder queries are conditioned on a CLIP embedding (of a class name or, during training, of an image crop, including crops of proposals that may contain novel objects), and each prediction is matched by whether it belongs to the conditioning query. Built on Deformable DETR, it reported 17.4 novel mask AP on OV-LVIS and 29.4 novel AP50 on OV-COCO (Table 3). Its ablation is informative: adding proposals of possible novel objects to training with ordinary label assignment lowered novel mask AP from 9.5 to 6.3, while conditional matching raised it to 17.4 (Table 2).

OWL-ViT (Minderer et al., 2022) simplified further. It contrastively pretrains a Vision Transformer and a text transformer on image–text pairs (from scratch on 3.6 billion pairs in the LiT recipe, or from public CLIP checkpoints), removes the image encoder’s token pooling, and attaches to every output token a box MLP and a linear projection to a class embedding that is compared with text embeddings of the queries. There is no decoder: each token predicts one object, with its default box centred on the token’s patch. Detection fine-tuning uses DETR’s bipartite matching loss with a sigmoid focal loss, since federated datasets such as LVIS label some objects with several names, and uses each image’s positive and negative category annotations, plus sampled pseudo-negatives, as its queries. The authors removed every box whose label matched an LVIS rare category from their detection training data (Objects365 and Visual Genome), so that rare-category AP measures zero-shot performance. Their best model, a CLIP ViT-L/14, reached 34.6 AP and 31.2 $\mathrm{AP}_r$ on LVIS v1.0 val (Table 1, mean of three fine-tuning runs), and detection performance kept improving with model size and image-level pretraining beyond 20 billion image–text pairs seen. The paper’s Table 1 places ViLD-ensemble’s 16.6, which Detic gives as mask AP, beside OWL-ViT’s numbers; since OWL-ViT predicts boxes only, its own numbers are box AP by our inference, and the paper does not say so.

Self-training at web scale: OWLv2

OWLv2 (Minderer et al., 2023) attacked the remaining bottleneck, the small size of detection data, by self-training. The public OWL-ViT CLIP L/14 annotated the roughly 10 billion WebLI images with pseudo-boxes, queried with all word N-grams (up to length 10) of each image’s alt-text with minimal filtering, keeping every box above a weak confidence threshold. New detectors were trained on these pseudo-labels and optionally fine-tuned briefly on LVIS base. Efficiency changes (dropping the half of image patches with the lowest pixel variance, an objectness head that selects about 10% of tokens for the loss, and mosaics of up to 6 × 6 images) roughly halved training FLOPs and doubled throughput; the improved architecture is what the authors call OWLv2. With a CLIP L/14 backbone trained on 2.3 billion image–text pairs and fine-tuned on LVIS base, it reached 44.6 $\mathrm{AP}_r$ on LVIS val, and 47.2 with a SigLIP G/14 backbone, against 32.8 for F-VLM, a detector built on frozen CLIP features, as the prior best in their table (Table 1). The paper also documents the cost of fine-tuning: for the L/14 model, fine-tuning on LVIS base raised LVIS $\mathrm{AP}_r$ from 34.9 to 44.6 while lowering mean AP on the 13 ODinW datasets from 53.0 to 50.1 (Table 1), a trade-off for which the authors found weight ensembling gives strong performance on both. Self-training on a curated vocabulary helped the curated classes but generalized worse to unseen classes and datasets than N-gram queries (Fig. 2).

Grounding: GLIP, Grounding DINO and YOLO-World

GLIP (Li et al., 2022) unified detection and phrase grounding through the alignment scores above, added deep cross-attention fusion between image and text encoders, and pretrained on 27 million grounding examples: 3 million human-annotated and 24 million web image–text pairs boxed by a GLIP teacher. Without seeing COCO images in pretraining, GLIP-L reached 49.8 AP on COCO val2017 and 26.9 AP (17.1 $\mathrm{AP}_r$) on LVIS v1.0 val; on the 5,000-image LVIS minival split introduced by MDETR, the same model scored 37.3 AP and 28.2 $\mathrm{AP}_r$ (Table 3). Because GLIP does not hold out the rare categories, these are domain-transfer numbers, not OV-LVIS results. The ablation shows the role of grounding data: gold grounding data added 4.2 $\mathrm{AP}_r$ on minival, and pseudo-labelled image–text data another 3.1 (§4.2).

Grounding DINO (Liu et al., 2024) applied grounded pretraining to the DINO detection transformer, fusing language in the neck, the query selection and the decoder; the Grounding DINO article gives its architecture and results. YOLO-World (Cheng et al., 2024) brought the approach to a real-time one-stage detector: text embeddings from a CLIP text encoder are fused into a YOLO detector’s feature pyramid and trained with a region–text contrastive loss on detection, grounding and image–text data. For deployment the user’s vocabulary is encoded once (“prompt-then-detect”) and folded into the network’s weights. The authors reported 35.4 AP on LVIS minival (Fixed AP, zero-shot) at 52.0 FPS on a V100 GPU, pre-trained on Objects365 and GoldG (abstract and §4.2).

Reported results

The two benchmark protocols as reported by the methods’ papers (generalized novel AP50 on OV-COCO with box AP; mask $\mathrm{AP}_r$ on OV-LVIS unless marked). Backbones, schedules and extra data differ, so rows are milestones, not a controlled comparison.

Method	Backbone, extra data	OV-COCO novel AP50	OV-LVIS $\mathrm{AP}_r$	Source
SB (zero-shot)	Inception-ResNet-v2	0.31		OVR-CNN Table 1
PL (zero-shot)	not given by OVR-CNN	4.12		OVR-CNN Table 1
OVR-CNN	ResNet-50-C4, COCO Captions	22.8		OVR-CNN Table 1
ViLD-text	ResNet-50	5.9	10.1	OV-DETR Table 3
ViLD	ResNet-50, CLIP	27.6	16.1	OV-DETR Table 3
ViLD-ensemble	ResNet-50, CLIP		16.6	Detic Table 2
RegionCLIP	ResNet-50-C4, CC3M	31.4	17.1 (metric not stated)	RegionCLIP Tables 1–2
RegionCLIP	ResNet-50×4-C4, CC3M	39.3	22.0 (metric not stated)	RegionCLIP Tables 1–2
Detic	ResNet-50-C4 (COCO), Mask R-CNN ResNet-50 (LVIS), image labels	27.8	17.8	Detic Tables 2–3
OV-DETR	Deformable DETR, CLIP ViT-B/32	29.4	17.4	OV-DETR Table 3
OWL-ViT	CLIP ViT-L/14, O365 + VG		31.2 (box by our inference; LVIS rare removed from all data)	OWL-ViT Table 1
OWLv2	CLIP ViT-L/14, WebLI self-training + LVIS base		44.6 (box)	OWLv2 Table 1

OV-DETR’s Table 3 disagrees with its sources in two places: it gives OVR-CNN’s overall and base AP50 as 46.0 and 39.9, which OVR-CNN’s own table reports the other way round (39.9 overall, 46.0 base), and its caption gives the OV-LVIS split as 886 base and 317 novel classes, while its text, like the other sources, holds out the 337 rare classes. The OWL-ViT and OWLv2 rows train on Objects365 and Visual Genome, OWLv2’s also on web pseudo-labels.

Example: scoring regions against class-name embeddings

The script below reproduces the core mechanism on synthetic images. Each $96 \times 96$ image contains three shapes; a class is a colour and a shape (“red circle”). Eight classes are base and annotated in training; five are novel: four whose colour and shape each occur in some base class (red triangle, green circle, blue square, yellow triangle) and purple square, whose colour occurs in no base class. A region’s feature is computed from the pixels in its box: the mean colour of the pixels that differ from the grey background, the fraction of the box they fill and its square. Proposals are jittered object boxes plus random boxes, a stand-in for a class-agnostic proposal network. The “text embeddings” are fixed 16-dimensional vectors. In the compositional condition, each colour word and shape word gets a random unit vector and a class name embeds as the normalized sum of its two words, a toy version of an embedding space in which related names are related vectors. In the unstructured condition every class name gets its own random vector. Only the $16 \times 6$ map $W$ from region features into the embedding space is learned, on base classes with a zero background embedding, as in OVR-CNN’s head; then regions are scored against the embeddings of the test vocabulary and AP at IoU 0.5 is computed per class after non-maximum suppression.

The sources differ on two training choices, so the code prints both. Bansal et al. dropped every training image containing an unseen object; OVR-CNN, ViLD and Detic keep the images and drop only the novel boxes, so novel objects become background proposals. OVR-CNN also down-weighted background proposals by $\alpha$ ($\alpha = 0.2$ in most of its experiments). The table averages five seeds and gives their range.

import os
os.environ.setdefault("OMP_NUM_THREADS", "1")       # small arrays: threads only add overhead
import numpy as np
from scipy.optimize import minimize

COLORS = {"red": (0.85, 0.15, 0.15), "green": (0.15, 0.7, 0.2), "blue": (0.15, 0.25, 0.85),
          "yellow": (0.85, 0.8, 0.15), "purple": (0.6, 0.15, 0.75)}
SHAPES = ["circle", "square", "triangle"]
BASE = ["red circle", "red square", "green square", "green triangle",
        "blue circle", "blue triangle", "yellow circle", "yellow square"]
NOVEL = ["red triangle", "green circle", "blue square", "yellow triangle", "purple square"]
CLASSES = BASE + NOVEL
S, BG = 96, 0.5
yy, xx = np.mgrid[0:S, 0:S] + 0.5

def draw(rng, n_images):
    """Images with 3 non-overlapping objects each; returns images and (box, class index) lists."""
    images, objects = [], []
    for _ in range(n_images):
        img = np.full((S, S, 3), BG) + rng.normal(0, 0.03, (S, S, 3))
        objs = []
        while len(objs) < 3:
            r, cx, cy = rng.uniform(9, 14), rng.uniform(16, S - 16), rng.uniform(16, S - 16)
            if any(np.hypot(cx - (b[0] + b[2]) / 2, cy - (b[1] + b[3]) / 2) < r + 16 for b, _ in objs):
                continue
            k = rng.integers(len(CLASSES))
            color, shape = CLASSES[k].split()
            x, y = xx - cx, yy - cy
            m = {"circle": x**2 + y**2 < r**2, "square": (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r),
                 "triangle": (y < 0.8 * r) & (y > -r + 2 * abs(x))}[shape]
            img[m] = np.array(COLORS[color]) + rng.uniform(-0.08, 0.08, 3)
            ys, xs = np.nonzero(m)
            objs.append(((xs.min(), ys.min(), xs.max() + 1, ys.max() + 1), k))
        images.append(np.clip(img, 0, 1))
        objects.append(objs)
    return images, objects

def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    return w * h / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - w * h)

def proposals(rng, objs):
    """Stand-in for a class-agnostic proposal network: jittered object boxes plus random boxes."""
    props = []
    for (x0, y0, x1, y1), _ in objs:
        w, h = x1 - x0, y1 - y0
        for j in range(4):
            d = rng.uniform(-0.15, 0.15, 4) * (w, h, w, h) if j else np.zeros(4)
            props.append(np.clip(np.round((x0, y0, x1, y1) + d), 0, S).astype(int))
    for _ in range(20):
        w, h = rng.uniform(12, 36, 2); x0, y0 = rng.uniform(0, S - w), rng.uniform(0, S - h)
        props.append(np.array([x0, y0, x0 + w, y0 + h]).astype(int))
    return props

def features(img, box):
    """Region feature computed from the pixels in the box: foreground colour and fill."""
    crop = img[box[1]:box[3], box[0]:box[2]].reshape(-1, 3)
    fg = np.linalg.norm(crop - BG, axis=1) > 0.15
    rgb = crop[fg].mean(0) if fg.any() else np.full(3, BG)
    fill = fg.mean()
    return np.r_[rgb, fill, fill**2]

def embeddings(rng, kind, d=16):
    unit = lambda v: v / np.linalg.norm(v)
    if kind == "unstructured":
        return np.array([unit(rng.standard_normal(d)) for _ in CLASSES])
    word = {w: unit(rng.standard_normal(d)) for w in list(COLORS) + SHAPES}
    return np.array([unit(word[c.split()[0]] + word[c.split()[1]]) for c in CLASSES])

def dataset(rng, images, objects, train, drop_novel=False):
    X, y, boxes, img_id = [], [], [], []
    for i, (img, objs) in enumerate(zip(images, objects)):
        if train and drop_novel and any(k >= len(BASE) for _, k in objs):
            continue                                   # Bansal et al.: drop images with novel objects
        annotated = [(b, k) for b, k in objs if k < len(BASE)] if train else objs
        for p in proposals(rng, objs):
            ious = [iou(p, b) for b, _ in annotated]
            best = int(np.argmax(ious)) if ious else -1
            label = annotated[best][1] if best >= 0 and ious[best] >= 0.5 else -1   # -1 background
            if train and best >= 0 and 0.3 <= ious[best] < 0.5:
                continue                               # ambiguous proposals are ignored
            X.append(features(img, p)); y.append(label); boxes.append(p); img_id.append(i)
    return np.array(X), np.array(y), np.array(boxes), np.array(img_id)

def train(X, y, T, alpha):
    """Learn W so that <W f, t_k> scores base class k; background has a fixed zero embedding.
    Weighted softmax cross-entropy (background weight alpha), minimized by L-BFGS from W = 0."""
    F = np.c_[X, np.ones(len(X))]
    lab = np.where(y < 0, len(BASE), y)
    Y = np.eye(len(BASE) + 1)[lab]
    wts = np.where(y < 0, alpha, 1.0)
    Tb = np.r_[T[:len(BASE)], np.zeros((1, T.shape[1]))]
    shape = (T.shape[1], F.shape[1])
    def loss(w):
        Z = F @ w.reshape(shape).T @ Tb.T
        L = Z - Z.max(1, keepdims=True); L -= np.log(np.exp(L).sum(1, keepdims=True))
        G = wts[:, None] * (np.exp(L) - Y) / wts.sum()
        return -(wts * L[np.arange(len(L)), lab]).sum() / wts.sum(), (Tb.T @ G.T @ F).ravel()
    r = minimize(loss, np.zeros(np.prod(shape)), jac=True, method="L-BFGS-B",
                 options={"maxiter": 5000, "gtol": 1e-8})
    return r.x.reshape(shape)

def ious(a, B):
    w = np.clip(np.minimum(a[2], B[:, 2]) - np.maximum(a[0], B[:, 0]), 0, None)
    h = np.clip(np.minimum(a[3], B[:, 3]) - np.maximum(a[1], B[:, 1]), 0, None)
    area = lambda b: (b[..., 2] - b[..., 0]) * (b[..., 3] - b[..., 1])
    return w * h / (area(a) + area(B) - w * h)

def nms(M, scores, thr=0.5):
    """Greedy NMS given the pairwise IoU matrix M of the boxes."""
    keep = []
    for i in np.argsort(-scores):
        if not keep or M[i, keep].max() < thr:
            keep.append(i)
    return keep

def ap50(k, scores, boxes, groups, objects):
    """All-point interpolated AP at IoU 0.5 for class k over the test images."""
    dets, n_gt = [], sum(kk == k for objs in objects for _, kk in objs)
    for i, idx, M in groups:
        for j in nms(M, scores[idx]):
            dets.append((scores[idx][j], i, boxes[idx][j]))
    dets.sort(key=lambda t: -t[0])
    used, tp = set(), []
    for s, i, b in dets:
        gts = [(n, bb) for n, (bb, kk) in enumerate(objects[i]) if kk == k]
        hit = [(iou(b, bb), n) for n, bb in gts if (i, n) not in used and iou(b, bb) >= 0.5]
        tp.append(bool(hit))
        if hit:
            used.add((i, max(hit)[1]))
    tp = np.array(tp, float)
    rec, prec = np.cumsum(tp) / n_gt, np.cumsum(tp) / np.arange(1, len(tp) + 1)
    prec = np.maximum.accumulate(prec[::-1])[::-1]
    return float(np.sum(np.diff(np.r_[0, rec]) * prec))

def evaluate(W, T, X, boxes, img_id, objects, vocab):
    F = np.c_[X, np.ones(len(X))]
    Z = F @ W.T @ np.r_[T[vocab], np.zeros((1, T.shape[1]))].T
    P = np.exp(Z - Z.max(1, keepdims=True)); P /= P.sum(1, keepdims=True)
    groups = []
    for i in np.unique(img_id):
        idx = np.nonzero(img_id == i)[0]
        groups.append((i, idx, np.array([ious(b, boxes[idx]) for b in boxes[idx]])))
    return {k: ap50(k, P[:, n], boxes, groups, objects) for n, k in enumerate(vocab)}

def run(seed):
    rng = np.random.default_rng(seed)
    tr_img, tr_obj = draw(rng, 300)
    te_img, te_obj = draw(rng, 200)
    Xte, _, bte, ite = dataset(rng, te_img, te_obj, train=False)
    data = {drop: dataset(np.random.default_rng(seed + 100), tr_img, tr_obj, True, drop)
            for drop in (False, True)}
    mu, sd = data[False][0].mean(0), data[False][0].std(0)
    out = {}
    for kind in ("compositional", "unstructured"):
        T = embeddings(np.random.default_rng(seed + 200), kind)
        for drop, alpha in ((False, 1.0), (False, 0.2), (True, 1.0)):
            if kind == "unstructured" and (drop or alpha != 1.0):
                continue
            Xtr, ytr = data[drop][0], data[drop][1]
            W = train((Xtr - mu) / sd, ytr, T, alpha)
            nov = evaluate(W, T, (Xte - mu) / sd, bte, ite, te_obj, list(range(8, 13)))
            gen = evaluate(W, T, (Xte - mu) / sd, bte, ite, te_obj, list(range(13)))
            base = evaluate(W, T, (Xte - mu) / sd, bte, ite, te_obj, list(range(8)))
            out[(kind, drop, alpha)] = (np.mean([nov[k] for k in range(8, 12)]), nov[12],
                                        np.mean([gen[k] for k in range(8, 12)]), gen[12],
                                        np.mean([gen[k] for k in range(8)]), np.mean(list(base.values())))
    return out

results = [run(seed) for seed in range(5)]
print("mean AP50 over seeds 0-4 (min-max)")
print("                                       novel-only vocabulary         | generalized vocabulary (13 classes)          | base only")
print("embeddings     training data   alpha  4 novel          purple sq.    | 4 novel          purple sq.       8 base      | 8 base")
for key in results[0]:
    kind, drop, alpha = key
    cells = []
    for c in range(6):
        v = np.array([r[key][c] for r in results])
        cells.append(f"{v.mean():.2f} ({v.min():.2f}-{v.max():.2f})")
    name = "drop novel imgs" if drop else "keep, unlabeled"
    print(f"{kind:14s} {name:15s} {alpha:4.1f}   {cells[0]}  {cells[1]} | {cells[2]}  {cells[3]}  {cells[4]} | {cells[5]}")

mean AP50 over seeds 0-4 (min-max)
                                       novel-only vocabulary         | generalized vocabulary (13 classes)          | base only
embeddings     training data   alpha  4 novel          purple sq.    | 4 novel          purple sq.       8 base      | 8 base
compositional  keep, unlabeled  1.0   0.53 (0.50-0.57)  0.20 (0.16-0.27) | 0.65 (0.62-0.67)  0.58 (0.20-1.00)  0.66 (0.54-0.75) | 0.73 (0.72-0.74)
compositional  keep, unlabeled  0.2   0.54 (0.49-0.58)  0.21 (0.16-0.29) | 0.70 (0.66-0.76)  0.62 (0.24-1.00)  0.67 (0.57-0.73) | 0.71 (0.70-0.72)
compositional  drop novel imgs  1.0   0.55 (0.44-0.62)  0.16 (0.08-0.26) | 0.65 (0.54-0.73)  0.31 (0.08-0.55)  0.70 (0.64-0.73) | 0.68 (0.66-0.72)
unstructured   keep, unlabeled  1.0   0.10 (0.05-0.20)  0.19 (0.01-0.69) | 0.11 (0.05-0.22)  0.22 (0.01-0.88)  0.45 (0.28-0.64) | 0.73 (0.71-0.74)

The model and the detector are the same in every row; only the embeddings of the class names differ between the compositional and unstructured rows. With compositional embeddings, the four novel classes whose words occur among the base classes are detected at about 0.53 AP50 with the novel-only vocabulary, against 0.10 with unstructured embeddings, while base classes score the same 0.73 with a base-only vocabulary under both. A map fitted on base classes transfers exactly as far as the text space relates novel names to base ones, the toy counterpart of Detic’s classifier ablation (16.4 novel mask AP with CLIP embeddings, 9.0 with fastText, 0 with a trained classifier). Purple square, whose colour word no base class carries, is not transferred reliably: its AP depends on where the random “purple” vector happens to fall relative to the learned map, and its range over seeds runs from 0.20 to 1.00 in the generalized setting.

Two results went against the plan’s hypotheses, which expected that the “generalized vocabulary lowers novel AP50 relative to novel-only” and that keeping novel objects as unlabelled background “lowers novel AP50 relative to dropping those images”. First, the generalized vocabulary gave higher novel AP50 (0.65–0.70) than the novel-only vocabulary (0.53–0.55), the opposite of OVR-CNN’s COCO result (27.5 target-only against 22.8 generalized). In our reading, the reason is the test images: they also contain base objects, and with a novel-only vocabulary a red circle has no name of its own, so it goes to the novel name that shares a word with it (“green circle”, “red triangle”) and outranks true detections; with all 13 names, base names claim base objects. In our reading, the map trained here is not strongly biased towards base classes, a bias that in real detectors can make the generalized setting harder, as OVR-CNN’s numbers suggest. The unstructured row shows the same competition from the other side: its five random novel vectors take base objects away from base names, so base AP falls from 0.73 with base names only to 0.45 with all 13. Second, neither keeping novel objects as unlabelled background nor $\alpha = 0.2$ changed novel AP beyond the spread over seeds; the ranges overlap in every comparison. The background conflict that Bansal et al. and Zareian et al. describe did not show here, although 5 of the 13 classes are novel, so that about one training object in three becomes background in the “keep” rows; the example does not test why.

The example was planned before it was first run, and the script was changed three times after results had been seen. The first version trained $W$ with 400 steps of gradient descent; two diagnostic runs showed that it had not converged (training loss still falling, most training objects classified as background), so the shown code minimizes the same loss with L-BFGS to convergence. The speed-ups made with it and in the second change (a vectorized, per-image IoU matrix for NMS and a single BLAS thread) leave the printed values unchanged. The third change concerns only what is printed: the last column (base classes with a base-only vocabulary) was added after a diagnostic run, to show the cause of the unstructured row’s low generalized base AP. Run with NumPy 2.3.5 and SciPy 1.16.3, the script takes about 30 seconds; two runs gave identical output.

Datasets and Benchmarks

OV-COCO. Built from COCO on the seen/unseen split of Bansal et al. (2018): they clustered the word vectors of the COCO classes, assigned 80% of each cluster’s classes at random to the seen set and the rest to the unseen set, and kept only classes with a WordNet synset and a word vector, giving 48 seen and 17 unseen classes (§4); the 15 other COCO classes are not used. Detic’s appendix describes the 48 classes as manually selected, which differs from Bansal et al.’s account. Zareian et al. adopted this split with the 2017 train and validation sets, removed every box not labelled with a base class and every image left without boxes (107,761 training images with 665,387 base instances), and evaluated on 4,836 validation images with 28,538 base and 4,614 target instances (§4.1). The metric is box AP at IoU 0.5, reported for target classes alone, base classes alone, and the generalized setting with all 65 classes; later papers report mainly the generalized novel AP50. Novel objects remain in the training images without boxes.

OV-LVIS. Built from LVIS v1.0 by ViLD: the 337 rare categories (those in 1–10 training images) are novel and their annotations are removed from training; the 866 common and frequent categories are base (Zhou et al., 2022, §5; Zhong et al., 2022, §4). The metric is mask $\mathrm{AP}_r$ on the 337 rare categories of LVIS v1.0 val, with overall mask AP and $\mathrm{AP}_c$, $\mathrm{AP}_f$ alongside; box-only detectors report box AP instead, and some methods attach a class-agnostic mask head to report masks (Detic, OV-DETR). The LVIS article explains the federated annotation, the 300-detections-per-image limit and why $\mathrm{AP}_r$ varies across training runs much more than overall AP.

Zero-shot transfer benchmarks. Grounding-based detectors report COCO val2017 AP and LVIS AP on the full val set or on the 5,000-image minival split from MDETR, with no LVIS or COCO training data; YOLO-World reports Fixed AP on minival, which the LVIS article describes. ODinW (Object Detection in the Wild) averages AP over small detection datasets from diverse domains: the 13 public datasets that GLIP’s authors chose (Li et al., 2022, §5), or the larger collection, described by Grounding DINO’s authors as more than 35 datasets, on which they report mean AP; OWLv2’s authors cite the ELEVATER benchmark for the suite. OWLv2’s authors removed images resembling the train, validation and test splits of 68 vision datasets from WebLI, but not the ODinW datasets.

Evaluation Metrics

Results use average precision with intersection over union matching, with four conventions that change the number and must match before two results are compared:

• Box or mask. OV-COCO uses box AP50; OV-LVIS uses mask AP by convention, and tables often mix mask and box results without saying which (as above).
• Generalized or restricted vocabulary. Scoring with all base and novel names is the realistic setting; restricting to novel names removes the base classes’ competition and changed OVR-CNN’s target AP50 from 22.8 to 27.5.
• Split and metric variant. LVIS v1.0 val against minival, standard AP against Fixed AP, and the detections-per-image limit.
• What “zero-shot” excludes, which differs between papers (see Problem Definition and Challenges).

Open-world detection adds wilderness impact and absolute open-set error (see Variants).

Applications

The papers emphasize uses where categories cannot be fixed in advance. Detic’s authors evaluated one detector with a 21,000-class vocabulary on Objects365 and Open Images without fine-tuning. OWL-ViT’s image-conditioned queries target objects that are hard to describe in words. Grounding DINO’s authors combined it with Stable Diffusion for text-guided image editing, and its boxes prompt segmentation models in pipelines described in the Grounding DINO article. Open-vocabulary detectors also produce training data: GLIP and OWLv2 pseudo-labelled web images with an earlier detector to train the next one.

Open Problems

• Fairness of evaluation. With web-scale pretraining and pseudo-labels, no protocol guarantees that a novel category was never seen with a box, a caption or an image label (see Challenges).
• Specialized domains. Grounding detectors still score poorly on some ODinW datasets, as the Grounding DINO article records, and Bansal et al. found zero-shot performance to depend on a class’s relation to the seen classes as well as on its appearance (their “cat” and “zebra” cases); the purple square of the example fails for the same reason.
• Fine-tuning without forgetting. Fine-tuning on a target dataset raises its accuracy and lowers open-vocabulary transfer, as OWLv2’s LVIS and ODinW numbers show; its authors used weight ensembling to keep both.
• Names that do not exist. Detection by text assumes that the user can name the target; open-world detection’s unknown class and image-conditioned queries address cases where they cannot, and the two problem definitions have not yet merged.
• Cost. Early-fusion detectors re-encode the image for each prompt and limit the vocabulary per pass; YOLO-World’s offline vocabulary and OWL-ViT’s late fusion trade some of that flexibility for speed.