Open Images

Open Images (the Open Images Dataset, often abbreviated OID or written OpenImages) is a collection of about 9 million Flickr photographs released by Google with annotations for several tasks on the same images: image-level class labels, bounding boxes for object detection, instance segmentation masks, visual relationships, localized narratives and point labels. The first version appeared in 2016 with image-level labels only; six more versions followed, up to V7 in October 2022, each adding a kind of annotation. Unlike COCO, whose 80 categories are labelled in every image, Open Images verifies only some classes per image, and its classes form a hierarchy, so the Open Images Challenges of 2018 and 2019 scored detectors with a variant of average precision that uses verified negative labels, expands the hierarchy and treats boxes around groups of objects specially.

Purpose

Kuznetsova et al. (2020), the paper that describes V4 and that the website asks users of V5–V7 to cite as well, give three ways in which the images differ from earlier datasets. All images were listed on Flickr under a Creative Commons Attribution license, so that they can be shared and adapted; images that also appear elsewhere on the web were removed, which drops the simple images that image search engines return and keeps a high share of complex scenes; and the images were not gathered with a predefined list of class names or search terms, which the authors argue gives natural class statistics. The paper’s other aim is unification: annotations for classification, detection and relationship detection coexist on the same images, so that the tasks can be studied together.

The collection procedure (Kuznetsova et al., 2020, §2.1): identify all Flickr images with a CC BY license (done in November 2015), download the originals, record attribution metadata, remove inappropriate content with Flickr’s safety filters and Google SafeSearch, remove near-duplicates, remove images that appear elsewhere on the internet, recover the orientation intended by the photographer, and partition the result into train, validation and test.

Versions

Every version keeps the same images and adds or corrects annotations. Version 1 to 3 were documented in the GitHub repository openimages/dataset; from V4 the project moved to its own website, where V4–V7 each have a description page.

Version	Released	What it added	Source
V1	2016	9,011,219 training and 167,057 validation images with machine-generated labels; human verification on the validation set; 7,844 labels, about 6,000 of them “trainable”	READMEV1
V2	20 July 2017	V1 validation split into validation (41,620) and test (125,436); 2,070,219 boxes for 600 classes (1,240,316 in train); 9.38M human-verified labels in train; box attributes on validation and test	READMEV2, changelist
V3	16 November 2017	3,709,509 training boxes, 1.22M of them drawn by extreme clicking; 20.9M human-verified training labels	READMEV3
V4	30 April 2018	14,610,229 training boxes on 1,743,042 images, drawn exhaustively per label (15,440,132 boxes in all); attributes on all splits; 374,768 visual relationship annotations of 329 triplets in train; 30.1M human-verified labels on 19,794 classes; first Open Images Challenge	V4 page, news, Kuznetsova et al. (2020)
V5	8 May 2019	2,785,498 instance masks for 350 classes; denser validation and test boxes (15,851,536 in all); 36.5M human-verified labels	V5 page, news
V6	26 February 2020	675,155 localized narratives; relationships expanded to 3,284,280 annotations of 1,466 triplets, with human actions; 59.9M human-verified labels; the extreme points of training boxes	V6 page, news
V7	25 October 2022	66,391,027 point labels on 5,827 classes over about 1.4M images; 61.4M human-verified labels on 20,638 classes	V7 page, news

V4 and V5 both answer a weakness of the previous release. Until V3 the training set had about two boxes per image, because in general only one instance of a class was boxed per image (READMEV3); V4 redrew the training boxes so that every instance of every positively verified class is boxed. V4’s validation and test sets had about 5 boxes per image against 8.4 in train; V5 added more than 400,000 boxes there, raising them to 7.3 and 7.5 (Kuznetsova et al., 2020, Table 8; V5 announcement).

The description pages disagree with themselves in places, and both readings are reported here. The V4–V6 pages call V7 “released Sep 2021”, while the V7 page and its announcement give October 2022. The V5 page gives 19,969 classes with image-level labels in its overview and 19,949 in its text. The V3 README’s table lists 3,709,509 training boxes, its text 3.68M.

Dataset Size

The V7 description page gives the current counts (as of October 2026):

Annotation (V7)	Train	Validation	Test	Classes
Images	9,011,219	41,620	125,436	–
Human-verified image-level labels	58,783,034	618,184	2,003,748	20,638
Machine-generated image-level labels	164,809,380	681,049	2,060,717	15,382
Images with boxes	1,743,042	41,620	125,436	–
Boxes	14,610,229	303,980	937,327	600
Instance masks	2,686,666	24,730	74,102	350
Relationship annotations	3,174,291	27,243	82,746	1,466 triplets
Localized narratives	507,444	41,691	126,020	–
Point labels	55,258,859	1,483,322	9,648,846	5,827

The split sizes (9,178,275 images) are those of the V4 paper and every version since V2. The V7 page’s table of image-level labels lists 8,992,648 training images rather than 9,011,219, without explanation. The home page totals (15,851,536 boxes, 2,785,498 masks, 3,284,280 relationships, 675,155 narratives, 66,391,027 points, 61,404,966 human-verified labels) are the sums of the table’s rows. The densely annotated subset is the 1.9 million images with boxes; the V7 announcement gives these images an average of 6.7 image-level classes, 8.3 boxes, 1.7 relationships, 1.5 masks, 0.4 narratives and 34.8 point labels.

Classes

Classes are identified by machine-generated ids (MIDs) from Freebase and the Google Knowledge Graph, such as /m/01g317, with a separate file of names. The image-level vocabulary comes from JFT, an internal Google dataset: Kuznetsova et al. (2020) selected 19,794 classes for V4, covering coarse and fine-grained objects (animal, Pembroke Welsh corgi), scenes (sunset), events (birthday) and materials and attributes (leather, red). Later versions extended it to 20,638 classes (V7). A class counts as trainable if it has enough verified examples; the threshold has changed: at least 50 validation and 20 training images in V1 (about 6,000 classes), at least 30 verified training samples and 5 in validation or test in V2 and V3 (5,000), and at least 100 positive verifications in the training set from V4 (7,186 in V4, 9,668 in V7).

Of the image-level classes, 600 with a clear spatial extent are boxable: animals, clothing, vehicles, food, people, buildings, sports equipment, furniture and kitchenware, including objects absent from earlier detection challenges such as “fedora” and “snowman”. The boxable classes form a hierarchy, published as a JSON file with an interactive viewer: “Car” has the children “Limousine” and “Van” (READMEV3), “Helmet” has “Football helmet” and “Bicycle helmet” (challenge evaluation page). Annotators boxed each object at the most specific level, so the released ground truth holds leaf-most labels only; a limousine is annotated as limousine, not also as car.

The masks cover 350 of the 600 boxable classes, chosen for a coherent appearance, a definable annotation policy (which pixels belong to a nose?) and a shape that segmentation networks can capture (V5 page). The V4 relationships involve 57 object classes; the V6 ones 288 classes and 15 attributes. Point labels cover 5,827 classes, including stuff such as mountain and pavement as well as things.

Annotation Types

Image-level labels

Labelling every image for presence or absence of 19,794 classes is not feasible, so Kuznetsova et al. (2020, §2.3) used a computer-assisted protocol. An image classifier proposes candidate labels; humans verify them. For the validation and test sets, a Google-internal Inception-based classifier, of the kind offered by the Google Cloud Vision API, proposed every class scored above 0.5. For the training set, dozens of classifiers proposed candidates, sampled by stratified score ranges so that the verified set mixes easy and hard positives and negatives. Most verifications were done by Google’s in-house annotators and the rest through the Crowdsource app (about 20% of verified training labels in V4, under 1% in validation and test).

Each image therefore carries three kinds of label for each class:

• Verified positive: a human confirmed that the class is present.
• Verified negative: a human confirmed that the class is absent.
• Unannotated: nobody checked; the class may or may not be there.

The released files also include machine-generated labels with fractional confidences, which the website warns have a substantial false-positive rate. Verification practically eliminates false positives but not false negatives. Kuznetsova et al. (2020, §4.3) estimated the recall of the boxable image-level labels by having two experts list every missing boxable object in 100 random images: 43% of all objects, and 59% when objects that PASCAL would mark difficult are disregarded, against an estimated 83% for COCO. In V4 the training split had 2.4 positive labels per image and the validation and test splits 8.8, because candidates were generated more densely there.

Bounding boxes

Annotators were told that a perfect box is the smallest box that contains all visible parts of the object, and were shown two hand-drawn examples per class before each session. Two methods produced the boxes (Kuznetsova et al., 2020, §2.4):

• Extreme clicking (Papadopoulos et al., 2017): the annotator clicks the top, bottom, left- and right-most points of the object, which defines the box. Annotators qualified by drawing 800 boxes with an IoU above 0.84 to ground truth at 20 seconds or less per box. It took 7.4 seconds per box on average, and it produced about 90% of all boxes, including all validation and test boxes. Since V6 the four clicked points of the training boxes are released.
• Box verification series (Papadopoulos et al., 2016): a detector proposes a box, an annotator accepts or rejects it, rejected regions are excluded and the detector is retrained, with up to four attempts per class and image. Annotators were trained to accept boxes whose IoU with the perfect box exceeds 0.7. About 10% of the training boxes, the first 1.5 million drawn, came from this method.

Re-drawing 50,000 extreme-clicking boxes gave a mean IoU of 0.87 between annotators; re-drawing 1% of the verified boxes gave 0.77 against the originals (§4.2). The V2 and V3 READMEs and the V4 page quote “mean IoU ~0.82” for the semi-automatic boxes while the V5–V7 pages cite the paper’s 0.77. Training a Faster R-CNN on either kind of box for the same objects changed its test score by less than 0.001 (§4.2). An expert check of 100 images for each of the first 150 boxable classes in alphabetical order, more than 26,000 boxes, found box precision of 97.7% and recall, given an image-level label, 98.2%; the worst classes were bidet (86% wrong, confused with toilets), cello (55%, confused with violins) and coffee table (35%) (Table 12, Fig. 26).

Because a positive label at several levels of the hierarchy would lead to two boxes on one object (animal and zebra), the boxes were de-duplicated hierarchically. On the training set, parent labels were removed before boxing whenever a child label was present; on validation and test, annotators boxed all labels and a parent box was removed if it overlapped a child box with IoU above 0.8 (§2.4.4). The human body parts and “Mammal” were boxed on only 95,335 training images, which hold 1,327,596 such boxes; on validation and test they are boxed wherever a positive label exists.

Each box carries five binary attributes, defined on the download page:

• IsGroupOf: the box spans a group of objects, such as a bed of flowers or a crowd. Annotators used it for more than 5 instances that heavily occlude each other and physically touch. (The paper’s §2.4.1 says “five or more”; its §2.4.5 and every web page say more than 5.)
• IsOccluded: another object in the image occludes it.
• IsTruncated: it extends beyond the image boundary.
• IsDepiction: it is a depiction, such as a cartoon, drawing or statue, not a physical instance.
• IsInside: the picture is taken from inside the object, such as a car interior.

On the V4 training set, 66.06% of boxes are occluded, 25.09% truncated, 5.99% group-of, 5.45% depictions and 0.24% inside (Table 7); the text of §4.1 calls Inside 0.4% of all boxes. Checked by experts, the attributes had precision and recall in the 90s except IsInside, with 66.7% precision (Table 13). COCO’s crowd regions serve a similar purpose but are defined differently: after 10–15 instances had been segmented, the rest of a category became one crowd region.

Segmentation masks

V5 added masks for boxed instances of the 350 classes, each starting from its box. Training masks were made with interactive segmentation (Benenson, Popov & Ferrari, 2019), in which annotators correct a segmentation network’s output with clicks; the V5 page reports 84% mIoU for them, and the clicks are released with the masks. Validation and test masks were painted entirely by hand, with 90% mIoU self-consistency, capped at 600 instances per class and split. Only instances larger than 40×80 or 80×40 pixels, with a boundary the annotator could determine and a box without the IsGroupOf, IsDepiction or IsInside attribute, were segmented, and four classes were capped (“clothing” at 441k, “person” at 149k, “woman” at 117k and “man” at 114k instances). Instances without a mask keep their box. The CVPR paper reports 2.5M instances over 300 classes, against 2.8M over 350 in the V5 release; neither source reconciles the two.

Visual relationships

A relationship triplet such as ⟨woman, playing, guitar⟩ connects two boxes; attributes are triplets with the relationship “is”, such as ⟨table, is, wooden⟩. For V4, candidate pairs were boxes whose classes match a triplet and that overlap after enlarging them by 20%, and annotators verified each relationship; the acceptance rate over all triplets was 28.2% (Kuznetsova et al., 2020, §2.5). Annotation is exhaustive: for an image containing a woman and a guitar, every woman–guitar pair in the relation “playing” is listed and every other pair is a negative. The paper selected 326 candidate two-object triplets, of which 287 had training instances, and added 42 attribute triplets, all with instances, 329 in total; the V4 and V5 pages say 467 candidate triplets gave the same 329. V6 widened the vocabulary to 1,466 triplets, adding human–human relations (“two men shake hands”), human actions (“man is jumping”) and 61 zero-shot triplets that occur in validation and test but not in train.

Localized narratives

A localized narrative is a spoken description of an image recorded while the annotator moves the mouse over what they describe, transcribed by hand and aligned word by word to the automatic transcription, so that every word has a mouse-trace segment (Pont-Tuset et al., 2020). Annotation took 144.7 seconds per image on average, 72% of it manual transcription. The paper annotated 848,749 images: COCO train and val, all of Flickr30k, ADE20K train and val, plus 671k Open Images images, including the full validation and test sets and 504k training images that already had masks, boxes or relationships. V6 released the training part in February 2020 and the validation and test parts in July 2020 (V6 page).

Point labels

V7’s point labels give a class and a yes, no or unsure answer at single pixels. Three sources produced them (V7 page): yes/no questions generated by a model (“is this point on a pumpkin?”) and answered by several annotators, 22.6M labels over 4,171 classes (Benenson & Ferrari, 2022); free-form clicks with a typed class name, always positive; and corrective clicks from interactive mask annotation, the V5 clicks plus new ones, converted into labels. Of the 66.4M labels, 40% are yes, 45% no and 15% unsure. Benenson and Ferrari (2022) argue that sparse points suffice for evaluation as well as training: on DAVIS 2017, IoU computed from 50 points per frame ranked 15 segmentation methods with an average Kendall’s τ of 0.97 against dense IoU.

File format

Annotations are CSV files keyed by a 64-bit image id written in hexadecimal. A box row holds the source (xclick or activemil), the label MID, a confidence, XMin, XMax, YMin and YMax in normalized image coordinates from 0 to 1, the five attributes and the four extreme points. Verified image-level label rows hold the source (verification or crowdsource-verification), the MID and a confidence of 1 (present) or 0 (absent). Masks are binary PNG images linked to the box they started from; the image-id files carry each image’s Flickr URL, license, author, title and a rotation value, because 2.6% of the stored images are rotated relative to the photographer’s intent (rotation note, May 2018).

Splits

The 9,178,275 images are divided into train (9,011,219), validation (41,620) and test (125,436), with all annotations of the validation and test sets public. The challenges used a separate Challenge set of 100,000 images with hidden annotations, hosted by Kaggle; the 2018 and 2019 editions and all three tracks of 2019 used the same set (V5 announcement; 2019 challenge page). The 2019 page warns that the V5 test set is not this Challenge set. The V5–V7 pages recommend reporting detection results on the Challenge set: the validation and test splits have a simpler, more centred distribution of object positions than train, closer to PASCAL VOC 2012.

Tasks

• Image classification, as multi-label classification over thousands of classes, evaluated on verified labels only.
• Object detection over the 600 boxable classes, or the 500 of the challenge.
• Instance segmentation over 350 classes, or the 300 of the 2019 challenge.
• Visual relationship detection and phrase detection.
• Vision and language: grounded captioning from the localized narratives.
• Semantic segmentation trained and evaluated from sparse point labels.

Evaluation

The Open Images Challenge object detection metric (evaluation page; Kuznetsova et al., 2020, §5.2.1) is mean average precision at IoU 0.5 over the 500 challenge classes, with matching as in PASCAL VOC (the page names the 2010 protocol, the paper the 2012 one; both use the all-point AP described in the average precision article, and the official code computes that). Three properties of the annotations change which detections count as true positives (TP), false positives (FP) or neither.

Negative labels and non-exhaustive labelling

For each image, a class is evaluatable if it has a verified image-level label, positive or negative, or a ground-truth box. Detections of any other class are ignored: since nobody checked whether the class is present, a detection of it can be neither confirmed nor refuted. Detections of a verified-negative class are all false positives, because the class is known to be absent. Detections of a positive class are matched to its boxes as in PASCAL VOC, so a duplicate detection of an already matched box is a false positive. The scheme relies on the annotation rule that every instance of a positive class is boxed. In the official code, OpenImagesChallengeEvaluator stores, per image, the union of all labelled classes (any confidence) and box classes, and drops detections of every other class before matching.

Without this rule, a correct detection of an unannotated object would count as a false positive. Kuznetsova et al. (2020, Fig. 29) report that ignoring unannotated classes changes the score more than the other two components. Read from their figure for a Faster R-CNN with an Inception-ResNet-V2 backbone, the full metric is about 0.54, while a metric close to plain PASCAL VOC, which only ignores detections inside group-of boxes, gives about 0.40.

Class hierarchy

AP for a non-leaf class counts the instances of all its descendants: AP(Helmet) is computed over everything annotated as helmet, football helmet or bicycle helmet. A detector must therefore output one detection per relevant level, a “Football helmet” and a “Helmet” box for the same object, to reach full recall for both classes. The root “Entity” is not evaluated. In practice the ground truth is expanded before evaluation with the official script oid_hierarchical_labels_expansion.py: each box is copied to every ancestor class, a positive image-level label is copied to every ancestor and a negative label to every descendant (if no animal is present, no dog is). Detections are not expanded; the evaluation page asks participants to produce the parent detections themselves. (The evaluation page also lists the open-source tool FiftyOne, which performs the ground-truth expansion automatically.)

Group-of boxes

A group-of box contains more than five instances whose individual locations are unknown. A detection is inside a group-of box $g$ if its intersection over the detection’s own area exceeds 0.5:

$$\mathrm{IoA}(d, g) = \frac{|d \cap g|}{|d|} > 0.5 .$$

If at least one detection is inside, the group-of box yields a single true positive; otherwise it counts as a single false negative. Further detections inside the same group-of box are neither rewarded nor penalized. Kuznetsova et al. (2020, §5.2.1) describe this as a weight $w$ on group-of boxes: with $w = 0$ they are ignore regions, and the challenge metric uses $w = 1$. The official “Open Images V2” metric in the same code base uses $w = 0$, and the evaluation page warns that it is not the challenge metric.

The official code (per_image_evaluation.py, read at the pinned commit) fixes three details that the page leaves open. Detections are first matched by IoU to ordinary boxes, and only the detections left unmatched, including duplicates already counted as false positives, are then tested against group-of boxes. The single true positive of a group takes the highest score among the detections inside it, so it enters the precision–recall ranking at that score. And both comparisons use $\geq 0.5$, where the page and paper write “greater than 0.5”. The number of positives for a class is the number of ordinary boxes plus $w$ times the number of group-of boxes.

The other tracks

The instance segmentation track (2019) uses mask IoU above 0.5 over 300 classes, with the same treatment of image-level labels and the hierarchy. Because not every box has a mask, matching proceeds in three stages per image: detections are matched to ground-truth masks (true positives); the rest to boxes without a mask and not group-of, by box IoU, and matches are ignored; the rest to group-of boxes by IoA, and matches are ignored again; whatever remains is a false positive. The visual relationship track scores a weighted combination of mAP for relationship detection, Recall@50 for relationship detection and mAP for phrase detection, where a relationship is correct only if both boxes have IoU above 0.5 ($\geq$ in the code) with an unmatched ground-truth pair and all three labels match; group-of boxes and the hierarchy are not used there.

Example: the rules on synthetic boxes

The code below draws two synthetic images into label maps and reads the ground-truth boxes off the drawn pixels. Image A contains two cars, a man, a flowerbed of twelve touching flowers annotated as one group-of box, and a bicycle that is visible but carries no image-level label; its verified labels are Car, Man and Flower (present) and Animal (absent). Image B contains a dog and a bicycle, with Car verified absent. A small hierarchy places Car and Bicycle under Vehicle, Man under Person and Dog under Animal. The invented detections include a duplicate car, a single “Vehicle” detection, a correct “Person” detection on the man, four detections of single flowers in the bed and one where there is no flower, a dog detection in image A, a correct detection of the unlabelled bicycle, and a car detection on the bicycle in image B. Each class is scored twice: VOC-style, with leaf labels, no image-level labels and the group-of box treated as an ordinary box; and with the challenge rules.

import numpy as np

# Two synthetic 100x160 images. Objects are drawn into a label image and their
# ground-truth boxes are read off the drawn pixels; boxes are [x1, y1, x2, y2].
H, W = 100, 160
yy, xx = np.mgrid[0:H, 0:W]

def rect(x1, y1, x2, y2):
    return (xx >= x1) & (xx < x2) & (yy >= y1) & (yy < y2)

def disc(cx, cy, r):
    return (xx - cx) ** 2 + (yy - cy) ** 2 <= r ** 2

def box_of(mask):
    ys, xs = np.nonzero(mask)
    return [xs.min(), ys.min(), xs.max() + 1, ys.max() + 1]

# Image A: two cars, a man, a flowerbed of 12 touching flowers (one group-of
# box), and a bicycle that is visible but was never labelled.
flowers = np.zeros((H, W), bool)
for i in range(12):
    flowers |= disc(112 + 8 * (i % 6), 76 + 8 * (i // 6), 4.5)
A = dict(car1=rect(8, 60, 48, 84), car2=rect(56, 62, 96, 88),
         man=rect(122, 12, 136, 58), flowers=flowers, bicycle=rect(60, 20, 92, 44))
# Image B: a dog and a bicycle.
B = dict(dog=rect(20, 50, 70, 90), bicycle=rect(90, 40, 140, 80))

# Leaf-level ground truth as released: (class, box, IsGroupOf).
gt = {"A": [("Car", box_of(A["car1"]), 0), ("Car", box_of(A["car2"]), 0),
            ("Man", box_of(A["man"]), 0), ("Flower", box_of(A["flowers"]), 1)],
      "B": [("Dog", box_of(B["dog"]), 0), ("Bicycle", box_of(B["bicycle"]), 0)]}
# Verified image-level labels: 1 = present, 0 = verified absent. Bicycle is
# unannotated in A although one is visible (non-exhaustive labelling).
labels = {"A": {"Car": 1, "Man": 1, "Flower": 1, "Animal": 0},
          "B": {"Dog": 1, "Bicycle": 1, "Car": 0}}
parent = {"Car": "Vehicle", "Bicycle": "Vehicle", "Man": "Person",
          "Woman": "Person", "Dog": "Animal", "Cat": "Animal"}
children = {p: [c for c in parent if parent[c] == p] for p in set(parent.values())}

def jitter(b, d):
    return [b[0] + d[0], b[1] + d[1], b[2] + d[2], b[3] + d[3]]

c1, c2 = box_of(A["car1"]), box_of(A["car2"])
fl = [[108 + 8 * i, 72, 116 + 8 * i, 80] for i in range(4)]  # single flowers
det = {"A": [("Car", jitter(c1, (1, -1, 2, 1)), .95), ("Car", jitter(c2, (-2, 0, 0, 2)), .80),
             ("Car", jitter(c1, (8, 3, 6, 0)), .70),         # duplicate on car 1
             ("Vehicle", jitter(c1, (0, 1, 1, 0)), .60),     # parent class, once
             ("Man", jitter(box_of(A["man"]), (0, 2, 1, 0)), .90),
             ("Person", jitter(box_of(A["man"]), (-1, 0, 0, 1)), .85),
             ("Flower", fl[0], .90), ("Flower", fl[1], .80), ("Flower", fl[2], .75),
             ("Flower", fl[3], .50),
             ("Flower", [10, 30, 18, 38], .93),             # nothing there
             ("Dog", jitter(box_of(A["man"]), (-2, 10, 2, 0)), .80),   # Animal: absent
             ("Bicycle", jitter(box_of(A["bicycle"]), (1, 0, 0, 1)), .88)],  # unlabelled
       "B": [("Dog", jitter(box_of(B["dog"]), (2, 0, 0, -3)), .70),
             ("Bicycle", jitter(box_of(B["bicycle"]), (0, 2, -2, 0)), .60),
             ("Car", jitter(box_of(B["bicycle"]), (5, 5, 0, 0)), .65)]}  # Car: absent

def ancestors(c):
    out = []
    while c in parent:
        c = parent[c]
        out.append(c)
    return out

def descendants(c):
    out = list(children.get(c, []))
    for k in children.get(c, []):
        out += descendants(k)
    return out

def expand(img):
    """Hierarchy expansion of the ground truth, as in the official script:
    boxes and positive labels go up to every ancestor, negative labels down."""
    boxes = [(a, b, g) for c, b, g in gt[img] for a in [c] + ancestors(c)]
    lab = {}
    for c, v in labels[img].items():
        for k in [c] + (ancestors(c) if v == 1 else descendants(c)):
            lab[k] = max(lab.get(k, 0), v)
    return boxes, lab

def overlap(d, g):
    iw = max(0, min(d[2], g[2]) - max(d[0], g[0]))
    ih = max(0, min(d[3], g[3]) - max(d[1], g[1]))
    inter = iw * ih
    area = lambda b: (b[2] - b[0]) * (b[3] - b[1])
    return inter / (area(d) + area(g) - inter), inter / area(d)   # IoU, IoA

def evaluate_image(img, cls, oi):
    """Return (scores, labels, n_gt) for one class in one image. labels are
    1 = TP, 0 = FP; ignored detections are dropped."""
    if oi:
        boxes, lab = expand(img)
        if cls not in lab and not any(c == cls for c, _, _ in boxes):
            return [], [], 0                   # unannotated: ignore everything
    else:
        boxes = gt[img]                         # leaf labels only, no groups
    g = [(b, gof and oi) for c, b, gof in boxes if c == cls]
    d = sorted([(s, b) for c, b, s in det[img] if c == cls], key=lambda t: -t[0])
    used = [False] * len(g)
    tp = [0] * len(d)
    in_group = [False] * len(d)
    for i, (s, b) in enumerate(d):              # stage 1: ordinary boxes, IoU
        ious = [overlap(b, gb)[0] if not gof else -1 for gb, gof in g]
        if ious and max(ious) >= 0.5:
            j = int(np.argmax(ious))
            if not used[j]:
                used[j], tp[i] = True, 1
    group_score = {}
    for i, (s, b) in enumerate(d):              # stage 2: group-of boxes, IoA
        if tp[i]:
            continue
        ioas = [overlap(b, gb)[1] if gof else -1 for gb, gof in g]
        if ioas and max(ioas) >= 0.5:
            j = int(np.argmax(ioas))
            in_group[i] = True
            group_score[j] = max(group_score.get(j, 0), s)
    scores = [s for i, (s, _) in enumerate(d) if not in_group[i]]
    lbls = [tp[i] for i in range(len(d)) if not in_group[i]]
    scores += list(group_score.values())        # one TP per detected group
    lbls += [1] * len(group_score)
    return scores, lbls, len(g)

def average_precision(scores, lbls, n_gt):
    """Area under the precision-recall curve with precision made monotone
    (all points, as in PASCAL VOC 2010 and the official OID code)."""
    if n_gt == 0:
        return np.nan
    order = np.argsort(-np.asarray(scores, float), kind="stable")
    t = np.asarray(lbls, float)[order]
    tp, fp = np.cumsum(t), np.cumsum(1 - t)
    rec = np.concatenate([[0], tp / n_gt, [1]])
    prec = np.concatenate([[0], tp / np.maximum(tp + fp, 1e-12), [0]])
    prec = np.maximum.accumulate(prec[::-1])[::-1]
    k = np.nonzero(rec[1:] != rec[:-1])[0] + 1
    return float(np.sum((rec[k] - rec[k - 1]) * prec[k]))

classes = ["Car", "Vehicle", "Man", "Person", "Flower", "Dog", "Animal", "Bicycle"]
print(f"{'class':8s} | {'VOC-style':^21s} | {'Open Images Challenge':^21s}")
print(f"{'':8s} | {'GT':>3} {'TP':>3} {'FP':>3} {'AP':>6}    | {'GT':>3} {'TP':>3} {'FP':>3} {'AP':>6}")
res = {}
for c in classes:
    row = []
    for oi in (False, True):
        S, L, N = [], [], 0
        for img in ("A", "B"):
            s, l, n = evaluate_image(img, c, oi)
            S += s; L += l; N += n
        ap = average_precision(S, L, N)
        res.setdefault(oi, {})[c] = ap
        ap_txt = "   -  " if np.isnan(ap) else f"{ap:6.3f}"
        row.append(f"{N:>3} {sum(L):>3} {len(L) - sum(L):>3} {ap_txt}")
    print(f"{c:8s} | {row[0]}    | {row[1]}")
for oi, name in ((False, "VOC-style"), (True, "Open Images")):
    v = np.array(list(res[oi].values()))
    print(f"{name:12s} mAP over {np.sum(~np.isnan(v))} classes with ground truth: "
          f"{np.nanmean(v):.3f}")

Output:

class    |       VOC-style       | Open Images Challenge
         |  GT  TP  FP     AP    |  GT  TP  FP     AP
Car      |   2   2   2  1.000    |   2   2   2  1.000
Vehicle  |   0   0   1    -      |   3   1   0  0.333
Man      |   1   1   0  1.000    |   1   1   0  1.000
Person   |   0   0   1    -      |   1   1   0  1.000
Flower   |   1   0   5  0.000    |   1   1   1  0.500
Dog      |   1   1   1  0.500    |   1   1   1  0.500
Animal   |   0   0   0    -      |   1   0   0  0.000
Bicycle  |   1   1   1  0.500    |   1   1   0  1.000
VOC-style    mAP over 5 classes with ground truth: 0.600
Open Images  mAP over 8 classes with ground truth: 0.667

Each rule shows up in one row. Under VOC-style matching, every single-flower detection has an IoU of about 0.08 with the box around the whole bed (our arithmetic from the boxes: 64 pixels against 49 × 17), so all five flower detections are false positives and the bed is missed; under the challenge rules the four inside the bed become one true positive at score 0.90, and only the detection on empty ground, ranked first at 0.93, remains a false positive, giving AP 0.5. The correct but unlabelled bicycle in image A is a false positive under VOC-style matching and is ignored under the challenge rules, because no one verified whether image A contains a bicycle. The dog detection in image A stays a false positive under both, but for a different reason in the challenge: Animal is verified absent there, and the negative label propagates down to Dog; the car detection in image B is likewise a false positive because Car is verified absent. Hierarchy expansion creates three classes with ground truth, Vehicle, Person and Animal: one Vehicle detection out of three expanded instances gives Vehicle an AP of 1/3, and Animal scores 0 because the detector never output an “Animal” box for the dog. Duplicates are penalized identically under both protocols. The two means are not comparable class for class, since they average different sets of classes, which is also true of the real metric: its 500 classes include parents such as Vehicle.

The per-class APs above were cross-checked by running the official OpenImagesDetectionChallengeEvaluator and PascalDetectionEvaluator (tensorflow/models at the pinned commit; the ground truth expanded with the official OIDHierarchicalLabelsExpansion class) on the same boxes, with the TensorFlow and absl imports, which the evaluation itself does not use, replaced by stubs. All eight challenge APs and all five VOC-style APs agreed exactly, as did the two means.

License

As of October 2026, the V7 description page states that the annotations are licensed by Google LLC under CC BY 4.0. The images are a separate matter: the page says they “are listed as having” a CC BY 2.0 license, and adds that Google makes no representations or warranties about the license status of each image, which users should verify themselves. The image-id files record each image’s Flickr license URL and author for attribution; Kuznetsova et al. (2020) describe the license field as a subtype of CC BY. The repository openimages/dataset, with the downloader and other tools, is under the Apache License 2.0; the V1–V3 READMEs state the same annotation and image terms, with “Google Inc.” as licensor.

Limitations and Bias

• Incomplete labels. With a recall of 43% for boxable image-level labels in V4, most classes present in an image are unannotated. Any use that treats an unannotated class as absent, for training or for evaluation, inherits this; the challenge metric avoids it by ignoring such detections, and it follows that a false detection of an unannotated class is never penalized either.
• Training and evaluation annotations differ. Training images were labelled by sampling from classifier score strata and validation and test images from all candidates above 0.5 (2.4 against 8.8 positive labels per image in V4); parent labels were de-duplicated by different rules; and masks were made interactively in train but painted by hand in validation and test.
• Classifier-proposed vocabulary. Every verified label started as a model’s proposal, so a class the proposing models never suggested for an image cannot be verified there. The candidates came from models trained on JFT, an internal dataset; we infer that this ties the label distribution to those models, which the paper does not measure.
• Semantic and cultural errors. Kuznetsova et al. (2020) attribute the bidet and cello confusions to cultural differences, and found that some annotators drew a “human hand” box around the whole arm.
• Geography and people. Shankar et al. (2017) geolocated, at country level, about 2 million of the 9 million images (the paper describes the 2016 release) and found more than 32% of them in the United States; this and audits of consent in large web datasets are discussed in dataset bias. The MIAP extension (Schumann et al., 2021) adds exhaustive person boxes for 100,000 images, nearly 100,000 of them not annotated by the original pipeline, with perceived gender presentation and perceived age range for fairness research; the Extended page says that gender identity cannot be labelled from images and that Google does not condone building gender or age classifiers from these labels.
• Overlap with other datasets. A duplicate detector run by the Open Images team in 2022 found 67 COCO val2017 images (1.3%) and 617 COCO test2017 images (1.5%) among the Open Images training images with boxes, and 11 and 6 images of the Flickr30k validation and test sets; models trained on one and tested on the other see some test images in training.

Common Uses

• Large-scale detection training and benchmarking, with the challenge metric or with COCO-style AP on subsets; detectors pretrained on Open Images are also fine-tuned on smaller datasets.
• Multi-label classification over thousands of classes with verified negatives.
• Instance segmentation on the 350 mask classes; Benenson, Popov and Ferrari (2019) described their masks as the largest existing instance segmentation dataset at the time.
• Vision and language: nocaps (Agrawal et al., 2019) draws 15,100 validation and test images from Open Images and trains with COCO captions plus Open Images labels and boxes; the localized narratives support grounded image captioning.
• Annotation research: the released extreme clicks, corrective clicks, verification votes and point labels are data for studying efficient labelling.

Historical Milestones

• 2016. V1, with 9 million images and machine labels for about 6,000 trainable classes, and an Inception v3 classifier trained on it (READMEV1).
• 2017. V2 and V3 introduced boxes for 600 classes; a ResNet-101 classifier trained on V2 was released, and an Inception-ResNet-v2 detection model trained on V2 through the TensorFlow Object Detection API (READMEV2, READMEV3).
• 2018. V4 redrew the training boxes exhaustively, mostly by extreme clicking, giving 15.4 million boxes, and the first Open Images Challenge was held at ECCV 2018 on Kaggle, with 12.2 million training boxes for 500 classes and a visual relationship track. The object detection track was won by team kivajok (Baidu) and the relationship track by team Seiji (Rutgers University, NVIDIA) (2018 challenge page).
• 2019. V5’s masks and the second challenge at ICCV 2019, which added instance segmentation over 300 classes. MMFruit won the detection track, Layer6 AI the relationship track and MMFruit Seg the segmentation track (2019 challenge page).
• 2020. No separate challenge; the detection and segmentation tracks joined the Robust Vision Challenge 2020. V6 added localized narratives. In the IJCV paper on V4, a Faster R-CNN with Inception-ResNet-V2 kept improving up to all 14.6M training boxes, reaching about 0.54 mAP under the challenge metric on the V4 test set, while an SSD with MobileNetV2 levelled off at about 0.33 from 1M boxes (values read from Kuznetsova et al., 2020, Fig. 30).
• 2022. V7 added point labels, the first Open Images annotation to cover stuff classes.

Related Datasets

• COCO: 80 categories with exhaustive instance masks and crowd regions on fewer images; Kuznetsova et al. (2020, Table 6) count 886,284 boxes in COCO 2017 train and val against 15,440,132 in Open Images V4 train, validation and test.
• PASCAL VOC: the 20-class benchmark whose matching and AP the challenge metric extends.
• ImageNet: its detection set (ILSVRC-DET) is the third dataset of the V4 paper’s comparisons; the paper’s detectors were pretrained on ILSVRC-2012.
• LVIS (Gupta, Dollár & Girshick, 2019): COCO images with a large vocabulary and a federated design, in which each category has verified positive and negative image sets, an idea close to Open Images’ positive and negative image-level labels.
• Visual Genome: relationships and region descriptions collected free-form; Kuznetsova et al. (2020) contrast its open vocabulary with Open Images’ controlled, exhaustively verified triplets.

Resources

• Open Images website: description of each version, downloads, visualizers, news and the Extended sets.
• Challenge evaluation page and the challenge overview, which links the 2019 Kaggle evaluation servers for late submissions.
• TensorFlow Object Detection API: the official challenge evaluation code (oid_challenge_evaluation.py, oid_vrd_challenge_evaluation.py) and the hierarchy expansion script.
• openimages/dataset: the V1–V3 documentation and the image downloader.
• Localized Narratives: the narratives for Open Images, COCO, Flickr30k and ADE20K.