Visual Genome

Visual Genome is a dataset of about 108,000 photographs in which crowd workers described dozens of regions per image and then turned those descriptions into structure: boxed objects, attributes of those objects, relationships between pairs of them, and one scene graph per image that joins everything, together with question–answer pairs. Objects, attributes and relationships are mapped to WordNet synsets. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson and colleagues at Stanford University, with collaborators at TU Dresden, Yahoo, Snapchat and CWI Amsterdam, completed version 1.0 in December 2015, posted the paper on arXiv in February 2016 and published it in the International Journal of Computer Vision in 2017 (Krishna et al., 2017). The dataset has no official benchmark of its own. Its uses come from later work: a filtered split with 150 object and 50 predicate classes became the standard test of scene graph generation (Xu et al., 2017), its region descriptions defined dense captioning (Johnson et al., 2016), and its object and attribute labels trained the Faster R-CNN behind the “bottom-up” region features of vision–language pretraining (Anderson et al., 2018).

Purpose

The authors argued that datasets built for perception (classification and detection) do not support tasks that require reasoning about how objects interact, and that a COCO caption summarizes the most salient part of an image and ignores the rest (Krishna et al., 2017, §1, §3.2). They set out to add three things: a grounding of visual concepts to language, a dense set of region-level descriptions and questions per image, and a formal representation of each image as a scene graph in the style of the knowledge bases used in natural-language processing. Relationships and attributes were to be annotated as carefully as objects, not as an afterthought. The dataset was meant as a general-purpose representation of the visual world rather than a benchmark for one task (§3.1).

Dataset Size

The counts changed between versions, and the paper never names one. Its journal version describes version 1.2, judging by its counts, which match the v1.2 home page; the arXiv preprint of February 2016 describes version 1.0, judging by its image count. Each row gives its source:

Version	Completed	Source	Images	Region descriptions	Objects	Attributes	Relationships	QA pairs
1.0	10 Dec 2015	home page, archived March 2016	108,249	4.2M	2.1M	1.8M	1.8M	1.7M
1.0	10 Dec 2015	paper, arXiv v1 (text)	108,249	42 per image	21 per image	1.6M	1.8M	1,773,258
1.2	29 Aug 2016	home page, archived November 2016 and unchanged on the October 2026 copy	108,077	5.4M	3.8M	2.8M	2.3M	1.7M
1.2	29 Aug 2016	paper, IJCV (§2, §5, §6.3, Tables 1, 3, 4)	108,077	50 per image; 5,406,939	35 per image; 3,843,636	26 per image; 2.8M	21 per image; over 2.3M	1,773,258
1.4	12 Jul 2017	version 1.4 readme	108,077 (unchanged)	unchanged	re-released, cleaner	unchanged	re-released with new boxes	unchanged

Version dates are those of the official download page. The first v1.0 release offered only image metadata, region descriptions and question answers; objects followed by February 2016, while attributes and relationships were still marked “coming soon” (download page, archived December 2015 and February 2016). Version 1.4 changed only the objects and the relationships that refer to them, and added files of object and relationship aliases (lines of synonyms); files of synsets per object, attribute and relationship appear under v1.4 on later copies of the page (archived 2022). The readme states that images, image metadata, region descriptions, attributes and question answers are those of v1.2. It gives no new counts.

The journal version is far more consistent than the preprint, which gave, for example, 16, 18 and 21 attributes per image in different sections; the paper article compares the two versions. A few figures still disagree in the journal version, so it matters which number is quoted. Relationships per image are 21 in the abstract, §2.4, §5.6, §6.2 and Tables 1 and 4, but 19 in §5.5, which prints the total as “over 2,347,187 million”. Descriptions run from 1 to 16 words in §2.1 and in the caption of Figure 16, but to at most 12 in the text of §5.2. The text of §5.6 gives 21 relationships per region graph, where Table 4 gives 0.43. The training, validation and test descriptions of §6.3 add up to 5,406,937, two fewer than its total of 5,406,939. As a rule this article takes the journal version’s per-image figures of §2 and Tables 1 and 4, and the totals of the official pages.

Classes

Visual Genome has no fixed class list. Workers named objects, attributes and relationships in their own words, so the raw vocabulary is open: after lower-casing and stemming, the journal version counts 33,877 object categories, 68,111 distinct attributes and 42,374 distinct relationships (Table 1, §3.3, §5.4, §5.5). Two layers make it usable:

• WordNet synsets. Objects, attributes, relationships and the noun phrases of region descriptions and questions carry WordNet synset names such as man.n.03 or yellow.s.01 (§4.8). Prepositional relationships such as “on” get none, because WordNet has no prepositions. This connects the dataset to ImageNet, which is also indexed by WordNet, and lets users merge words such as “man”, “boy” and “person” through their common hypernym.
• Filtered vocabularies chosen by each user. The benchmarks built on the dataset keep a frequent subset and map the rest away. The scene graph split keeps the 150 most frequent object names and 50 most frequent predicates after merging aliases (Xu et al., 2017); the bottom-up detector kept 1,600 object and 400 attribute classes, starting from 2,000 object and 500 attribute classes and removing abstract classes that detected poorly, without merging near-duplicates such as “person”, “man” and “guy” or “tree” and “trees” (Anderson et al., 2018, §4.1.1).

The distribution is long-tailed and dominated by people, clothing, parts and spatial relations. The images themselves cover 972 ImageNet-style synsets, with “ski”, “ballplayer” and “racket” the most frequent, which the authors describe as a bias toward people and sports (Krishna et al., 2017, §5.1). Within the 150-object, 50-predicate split, Zellers et al. (2018, Table 1) grouped 50.0% of relation instances as geometric (above, behind, under; 50.9% in the caption of their Figure 2) and 40.9% as possessive (has, part of, wearing); semantic relations such as eating or carrying made up 8.7% of instances although they are nearly half of the predicate types.

Annotation Types

The paper names seven components (§2):

• Region descriptions. A box and a free-form phrase, 50 per image on average (§2.1, §5.2; 42 in the arXiv preprint’s v1.0 data). Regions may overlap heavily when their descriptions differ.
• Objects. Tight boxes for every object a description mentions, each with a name and a synset. Objects mentioned in several descriptions are merged into one, so a box can carry several names.
• Attributes. Words attached to one object: colours, sizes, materials, states such as “standing”.
• Relationships. Directed predicates from a subject object to an object, written predicate(subject, object), such as wearing(man, hat) or jumping over(man, fire hydrant): actions, spatial relations, verbs, prepositions, comparatives and prepositional phrases (§2.4).
• Region graphs. For each region description, the graph of its objects, attributes and relationships. Descriptions such as “it is a sunny day” have no objects and no graph.
• Scene graphs. The union of all region graphs of an image, one per image, with co-referring objects merged.
• Question–answer pairs. Freeform questions about the whole image and region-based questions written from a region description, each starting with one of the “six Ws” (what, where, when, who, why, how; §2.7, §4.6, §5.7). The arXiv preprint’s instructions in §4.6 listed “seven Ws”, adding which. Binary yes/no questions were excluded (§5.7).

The released files follow the same structure (data readme, v1.2): image metadata with the image’s width, height, URL, COCO identifier and Flickr identifier; region descriptions with boxes, phrases and the synsets found in each phrase; objects, attributes and relationships per image with object identifiers; region graphs and scene graphs; question answers with synsets in question and answer; and a mapping from region-based questions to their regions.

How the annotations were collected

Everything was written and verified by workers on Amazon Mechanical Turk: over 33,000 workers and about 800,000 tasks over six months, after 15 months of experimentation, paid so that continuous work earned $6–8 an hour (§4.1). The pipeline ran in stages (§4.2–§4.7):

1. Descriptions first. Asking workers for scene graphs directly produced easy, frequent relations such as wearing(man, shirt), so the authors started from language. Each worker drew three boxes and wrote a description for each, shown up to seven of the most similar earlier descriptions of the image; this was repeated until an image had 50. A new description was rejected if its BLEU similarity to any earlier description of the image, or to the 100 most common descriptions in the dataset, exceeded 0.7, which kept phrases such as “sky is blue” from repeating.
2. Objects. One worker per description extracted its objects and boxed them, offered the image’s earlier boxes as suggestions (so “man” and “person” could point to one box) and nouns found by a dependency parser. Boxes had to cover the whole object and be tight to within four pixels.
3. Attributes and relationships were extracted from each description and its objects, forming the region graph.
4. Scene graphs. Objects from different region graphs became candidates for merging when their boxes had an IoU of 0.9 (§4.5 and Figure 12; the arXiv preprint’s text said 0.8), and workers confirmed each merge.
5. Verification. Three workers voted on every annotation except objects, and it was kept if at least two judged it correct; objects were verified with a faster “rapid judgment” interface. Vague, subjective or opinionated descriptions and questions were removed at this stage.

Canonicalization (§4.8, §5.8) mapped each noun phrase to its most frequent WordNet sense by lexeme counts, corrected by hand-written rules for common failures (30 for objects, 15 for attributes, 20 for relationships, mostly to prefer the concrete or spatial sense: a “table” is furniture, “long” is about length). Relationships ignore prepositions, which WordNet lacks, and map verbs through WordNet’s sentence frames to a root hypernym. Workers then checked each mapping against four alternatives. On hand-labelled samples (200 attributes and 200 relationships among them), the journal version’s §5.8 gives a mapping accuracy of 88% for noun phrases, 85.7% for attributes and 92.9% for relationships, with recall of 98.5%, 95.9% and 88.5%; Table 5 lists the same accuracies as precision, although its caption also promises a column of mapping accuracies. By our reading, about one relationship synset in fourteen that the method assigns is wrong, and prepositional relationships, among them “on”, the most common relationship (§5.5), get no synset at all. The arXiv preprint printed the same table but gave lower mapping accuracies in its text: 86%, 83.5% and 77.6%.

Splits

Visual Genome itself defines no train, validation or test split; the paper split the data separately for each baseline experiment (§6), by image for question answering (§6.4). The splits in use come from the papers that built tasks on it:

Split	Defined by	Images	How
Dense captioning	Johnson et al., 2016	77,398 / 5,000 / 5,000 (our arithmetic from 87,398 total)	the authors’ version of the region descriptions (94,313 images); images with 20–50 descriptions after dropping descriptions over 10 words
Scene graph split (often called VG150)	Xu et al., 2017	70% train, 30% test	150 object and 50 predicate classes; later work takes a 5,000-image validation set from training
Bottom-up detector	Anderson et al., 2018	about 98K / 5K / 5K	objects and attributes only; images shared with COCO kept in the same split as in the COCO captioning splits

The scene graph split as its code builds it

Xu et al. (2017, §4) describe their dataset as a cleaned version of the 108,077 scene graphs, which had an average of 38 objects and 22 relationships per image: they manually corrected 22 boxes or names, deleted 7.4 boxes and merged 5.4 duplicate boxes per image on average, then kept the most frequent 150 object categories and 50 predicates. That leaves about 11.5 objects and 6.2 relationships per image. Their preprocessing script, vg_to_roidb.py (repository at commit 4c9e3c6, read line by line), does the following with its default settings:

1. Lowercases names and predicates, strips punctuation and replaces them through alias lists shipped with the script (VG/object_alias.txt and VG/predicate_alias.txt; the first entry of each line of synonyms).
2. Drops boxes smaller than 0.2% of the image area.
3. Merges duplicate boxes: two boxes count as the same object if they are identical or overlap by more than 0.9, if one lies inside the other with the same name, or if they overlap by more than 0.6 with the same name. The script’s overlap divides the intersection by the area of the rectangle enclosing both boxes, not by the union, so it is smaller than the IoU.
4. Keeps only the object names and predicates listed in two further files shipped with the script, VG/object_list.txt (150 names) and VG/predicate_list.txt (50 predicates); the frequency count picks the most frequent names only when no list is given. The paper describes the classes as the most frequent ones; whether the lists match a frequency count cannot be checked without the annotation files. A box with several names takes the listed one that is most frequent, boxes with none are dropped, and so are relationships whose boxes were dropped, whose predicate is not listed, or whose subject and object became the same box.
5. Assigns the first 70% of images, in the order of the input files, to training and the rest to test; shuffling is off by default and the validation fraction is empty.
6. Stores the boxes at two scales, with the longer image side at 512 and at 1,024 pixels; Tang’s loader reads the 1,024-pixel version and rescales to the original image.

The resulting file, VG-SGG.h5, is what later papers download rather than rebuild. Zellers et al. (2018, §5.3) used it with a development set of 5,000 images sampled from training, and later papers followed. In the widely used third-party benchmark code of Kaihua Tang (repository at commit ceb71fa), the loader takes the first 5,000 training images as validation and, for relation models, drops images without relationships from every split. Only in training does it sample one predicate when a pair has several and, for SGDet with the box-overlap option set, drop relationships between boxes that do not overlap. Test images keep every annotated relation, including several predicates on the same pair. That benchmark ships its own version of the file with attribute labels added, VG-SGG-with-attri.h5; its documentation says its default settings turn the attribute head off, following the conventional setting.

Tasks

• Scene graph generation: predict a set of (subject, predicate, object) triplets with boxes. Xu et al. (2017) defined three settings that differ in what the model is given: predicate classification (PredCls; boxes and object labels given), scene graph classification (SGCls; boxes given, labels predicted) and scene graph generation (SGGen, later also called SGDet; everything predicted from the image).
• Dense captioning: detect regions and describe each in a phrase (Johnson et al., 2016).
• Detector and attribute pretraining: training an object detector with an attribute head on the open vocabulary, as for bottom-up features (Anderson et al., 2018).
• Visual question answering: the 1.7M question–answer pairs serve as extra training data, as when the Up-Down model added about 485K Visual Genome questions whose answers were in its vocabulary to VQA v2 training (Anderson et al., 2018, §4.1.1).
• Grounding and vision–language pretraining: region descriptions are phrases with boxes, used as grounding data and as caption-like pairs in pretraining corpora.
• Attribute and relationship classification on given boxes, the paper’s own baselines (§6.1–§6.2).

Evaluation

Scene graph generation is scored by Recall@K over triplets, in the per-query form that article calls recall at cut-off $K$, with the image as the query. Xu et al. (2017, §4, §4.1) adopted image-wise Recall@50 and Recall@100 from Lu et al.’s visual relationship detection work and gave the reason for recall rather than average precision: with only 1.6% of object pairs labelled with a relationship, a precision-based measure would penalize correct predictions that the annotators did not write down. A triplet is correct when its subject, predicate and object labels match a ground-truth triplet and both boxes overlap the ground-truth boxes with an IoU of at least 0.5.

The Visual Genome pages offer no evaluation server or evaluation code. The protocol below follows a widely used implementation, Tang’s Scene-Graph-Benchmark.pytorch at commit ceb71fa (sgg_eval.py, vg_eval.py and the relation post-processor). Its recall classes state that they are based on the neural-motifs code of Zellers et al. It is a third-party implementation, not one the dataset’s authors host, and was read line by line rather than run.

Recall@K as the code computes it

1. Mode. Ground-truth boxes and labels given: PredCls. Boxes only: SGCls. Neither: SGDet. In PredCls every object score is set to 1.
2. Candidates. The model scores every ordered pair of its boxes (no self-pairs) over the 50 predicates and a background class. In SGDet a test option can restrict the pairs to overlapping boxes; it is on in the code’s defaults and off in the shipped relation configuration. The post-processor sorts the pairs by the product of subject score, object score and the best non-background predicate score.
3. With the graph constraint (the default “R@K”), each pair contributes one triplet, its highest-scoring predicate, and the first $K$ pairs are kept.
4. Without the graph constraint (“ng-R@K”), every (pair, predicate) entry competes: the scores are products of the two object scores and each predicate’s probability, the 100 highest are kept, and the first $K$ of those count. A pair can then contribute “on” and “riding” at once.
5. Matching. A ground-truth triplet is recalled if any kept triplet has the same three labels and both boxes at IoU ≥ 0.5, computed in single precision with the +1-pixel box convention of maskrcnn-benchmark. There is no one-to-one assignment: one prediction can recall two identical ground-truth triplets.
6. Averaging. Each image’s recall is the number of recalled ground-truth triplets divided by its number of ground-truth triplets; images without ground-truth relations are skipped, and R@K is the mean over images, for $K$ = 20, 50 and 100.
7. Mean recall (mR@K) computes, for each predicate, the recall of that predicate’s triplets in each image that has any, averages it over those images, and averages over all 50 predicates, with 0 for a predicate that never occurs. The code also reports a zero-shot recall restricted to (subject, predicate, object) combinations absent from training.

Some of these details are stated only in the code: the +1-pixel IoU, the cap of 100 entries on the unconstrained list, the absence of a one-to-one assignment and the order of averaging in mean recall. The rest has a paper trail. Xu et al. reported only R@50 and R@100; Zellers et al. (2018, Table 2) added R@20 and stated that, following earlier work, a system must not output several predicates for the same pair. Newell and Deng (2017, §5) had placed no such restriction on their graphs; Zellers et al. re-evaluated Newell and Deng’s saved predictions with the constraint for their comparison, and the documentation of Tang’s code credits Newell and Deng with the unconstrained setting and Zellers et al. with its name. Mean recall was proposed at CVPR 2019 by Tang et al. in VCTree and by Chen et al. in KERN, both to measure performance on the many rare predicates that the head predicates swamp in R@K.

The two recalls answer different questions, and a number of one kind is not comparable with a number of the other. The reimplementations reported in the METRICS.md of Tang’s code show the size of the gap on the real split: MOTIFS reaches a PredCls R@100 of 67.01 with the constraint and 88.24 without it, and an SGDet R@100 of 37.16 and 43.43. The same documentation describes a second confusion: some papers score PredCls and SGCls only on the ground-truth subject–object pairs, which removes the ranking among unrelated pairs and yields a top-K accuracy rather than a recall.

Example: the graph constraint on rendered scenes

The example renders each scene as a label image of rectangles on a 96 × 96 canvas, later objects occluding earlier ones, and reads every ground-truth box off the pixels. Scenes are built from plausible relation templates such as wearing(man, hat) or riding(man, horse); three templates put two predicates on one pair, as Visual Genome often does (man riding and on a horse, man wearing and has a hat). A simulated relation model gives every ordered pair a softmax over background and eight predicates, with noise, a bias toward frequent predicates, and extra evidence for each true predicate. In SGDet mode the boxes are jittered, 15% of labels are wrong and objects get confidences. The function implements steps 2–7.

import numpy as np

OBJ = ["man", "horse", "hat", "shirt", "grass", "dog", "frisbee", "tree"]
PRED = ["on", "has", "wearing", "near", "riding", "holding", "behind", "eating"]
P_PRIOR = np.array([30, 20, 15, 10, 6, 4, 3, 1.0])   # long-tailed predicate frequencies
P_PRIOR /= P_PRIOR.sum()

def render(rng, n_obj, size=96):
    """Paint objects as rectangles into a label image; read each box back off its pixels."""
    lab = np.zeros((size, size), int)
    for k in range(1, n_obj + 1):
        h, w = rng.integers(12, 48, 2)
        y, x = rng.integers(0, size - h), rng.integers(0, size - w)
        lab[y:y + h, x:x + w] = k                      # later objects occlude earlier ones
    boxes = []
    for k in range(1, n_obj + 1):
        ys, xs = np.nonzero(lab == k)
        if len(ys):
            boxes.append([xs.min(), ys.min(), xs.max(), ys.max()])
    return np.array(boxes, float)

# (subject, object, predicates): templates for plausible relations; two predicates on one pair occur in VG
TEMPLATES = [(0, 2, [2]), (0, 2, [2, 1]), (0, 3, [2]), (0, 1, [4, 0]), (0, 1, [4]), (1, 4, [0]),
             (5, 4, [0]), (0, 6, [5]), (5, 6, [5]), (7, 0, [6]), (0, 5, [3]), (1, 4, [7]), (0, 3, [1])]

def make_scene(rng, templates=None):
    if templates is None:
        templates = [TEMPLATES[t] for t in rng.choice(len(TEMPLATES), rng.integers(3, 7))]
    cls, rels = [], []
    for s, o, preds in templates:        # each template adds a new subject and object
        cls += [s, o]
        rels += [(len(cls) - 2, len(cls) - 1, p) for p in preds]
    boxes = render(rng, len(cls))
    if len(boxes) < len(cls):            # an object vanished under others: redraw
        return make_scene(rng, templates)
    return boxes, np.array(cls), np.array(rels)

def predict(rng, boxes, cls, rels, sgdet):
    """Simulated relation model: softmax over [background] + predicates for every ordered pair."""
    n = len(boxes)
    if sgdet:   # detector: jittered boxes, 15% wrong labels, confidence per box
        boxes = boxes + rng.normal(0, 2.5, boxes.shape)
        cls = np.where(rng.random(n) < 0.15, rng.integers(0, len(OBJ), n), cls)
        obj_sc = rng.uniform(0.5, 1.0, n)
    else:       # PredCls: ground-truth boxes and labels, object scores 1
        obj_sc = np.ones(n)
    pairs = np.array([(i, j) for i in range(n) for j in range(n) if i != j])
    logit = rng.normal(0, 1, (len(pairs), 1 + len(PRED)))
    logit[:, 1:] += np.log(P_PRIOR) + 1.0          # frequency bias toward head predicates
    logit[:, 0] += 2.0                              # most pairs are unrelated
    for i, j, p in rels:
        r = np.flatnonzero((pairs[:, 0] == i) & (pairs[:, 1] == j))[0]
        logit[r, 1 + p] += 2.5                      # evidence for each true predicate
        logit[r, 0] -= 3.0
    prob = np.exp(logit - logit.max(1, keepdims=True)); prob /= prob.sum(1, keepdims=True)
    return boxes, cls, obj_sc, pairs, prob

def iou(a, b):   # inclusive pixel boxes, the +1 convention of Tang's code
    lt, rb = np.maximum(a[:2], b[:, :2]), np.minimum(a[2:], b[:, 2:])
    wh = np.clip(rb - lt + 1, 0, None); inter = wh[:, 0] * wh[:, 1]
    area = lambda x: (x[..., 2] - x[..., 0] + 1) * (x[..., 3] - x[..., 1] + 1)
    return inter / (area(a) + area(b) - inter)

def recall_at_k(gt, pred, ks=(20, 50, 100), graph_constraint=True, thr=0.5):
    """Per-image recall of ground-truth triplets; returns {K: recall} and the matched GT per K."""
    g_box, g_cls, g_rel = gt
    p_box, p_cls, p_sc, pairs, prob = pred
    trip_sc = p_sc[pairs[:, 0]] * p_sc[pairs[:, 1]]
    if graph_constraint:   # one predicate per pair: the best non-background one
        lab = 1 + prob[:, 1:].argmax(1)
        order = np.argsort(-(trip_sc * prob[:, 1:].max(1)), kind="stable")
        cand = [(pairs[r, 0], pairs[r, 1], lab[r] - 1) for r in order]
    else:                  # every (pair, predicate) entry competes; keep the top 100
        s = trip_sc[:, None] * prob[:, 1:]
        flat = np.argsort(-s.ravel(), kind="stable")[:100]
        cand = [(pairs[r, 0], pairs[r, 1], p) for r, p in zip(*np.unravel_index(flat, s.shape))]
    hit = []               # for each candidate, the GT relations it matches
    for i, j, p in cand:
        m = [g for g, (a, b, q) in enumerate(g_rel)
             if q == p and g_cls[a] == p_cls[i] and g_cls[b] == p_cls[j]
             and iou(g_box[a], p_box[i][None])[0] >= thr and iou(g_box[b], p_box[j][None])[0] >= thr]
        hit.append(m)
    out = {}
    for k in ks:
        matched = set(g for m in hit[:k] for g in m)
        out[k] = (len(matched) / len(g_rel), matched)
    return out

def evaluate(seed, n_img=300, sgdet=False):
    rng = np.random.default_rng(seed)
    R = {gc: {k: [] for k in (20, 50, 100)} for gc in (True, False)}
    per_pred = {gc: [[] for _ in PRED] for gc in (True, False)}       # for mR@100
    ceiling = []   # best recall one predicate per pair allows: labelled pairs / relations
    for _ in range(n_img):
        gt = make_scene(rng)
        ceiling.append(len({(a, b) for a, b, _ in gt[2]}) / len(gt[2]))
        pred = predict(rng, *gt, sgdet)
        for gc in (True, False):
            res = recall_at_k(gt, pred, graph_constraint=gc)
            for k in res: R[gc][k].append(res[k][0])
            matched = res[100][1]
            for p in np.unique(gt[2][:, 2]):
                idx = np.flatnonzero(gt[2][:, 2] == p)
                per_pred[gc][p].append(np.mean([g in matched for g in idx]))
    row = {}
    for gc in (True, False):
        row[gc] = [100 * np.mean(R[gc][k]) for k in (20, 50, 100)]
        row[gc].append(100 * np.mean([np.mean(v) if v else 0.0 for v in per_pred[gc]]))
    row["ceiling"] = 100 * np.mean(ceiling)
    return row

# One rendered image: man riding/on horse, man wearing hat, horse on grass, dog on grass (PredCls)
rng = np.random.default_rng(0)
gt = make_scene(rng, [TEMPLATES[3], TEMPLATES[0], TEMPLATES[6], TEMPLATES[5]])
pred = predict(rng, *gt, sgdet=False)
name = lambda i, j, p: f"{OBJ[gt[1][i]]}{i}-{PRED[p]}-{OBJ[gt[1][j]]}{j}"
print("ground truth:", ", ".join(name(*r) for r in gt[2]))
for gc in (True, False):
    res = recall_at_k(gt, pred, ks=(3, 5, 8), graph_constraint=gc)
    print(f"graph constraint={gc!s:5}", "  ".join(f"R@{k}={res[k][0]:.2f}" for k in res),
          "  missed at R@8:", [name(*gt[2][g]) for g in range(len(gt[2])) if g not in res[8][1]])
print()
for mode in ("predcls", "sgdet"):      # 300 images per seed, seeds 0-4: min-max over seeds
    rows = [evaluate(s, sgdet=(mode == "sgdet")) for s in range(5)]
    for gc in (True, False):
        v = np.array([r[gc] for r in rows])
        print(f"{mode:7} {'with' if gc else 'no':4} constraint ", "  ".join(
            f"{m} {lo:4.1f}-{hi:4.1f}" for m, lo, hi in zip(("R@20", "R@50", "R@100", "mR@100"), v.min(0), v.max(0))))
    gap = [r[False][2] - r[True][2] for r in rows]
    ceil = [r["ceiling"] for r in rows]
    print(f"{mode:7} R@100 gain without the constraint: {min(gap):.1f}-{max(gap):.1f} points;"
          f" ceiling with it: {min(ceil):.1f}-{max(ceil):.1f}")

ground truth: man0-riding-horse1, man0-on-horse1, man2-wearing-hat3, dog4-on-grass5, horse6-on-grass7
graph constraint=True  R@3=0.40  R@5=0.60  R@8=0.80   missed at R@8: ['man0-on-horse1']
graph constraint=False R@3=0.40  R@5=0.60  R@8=1.00   missed at R@8: []

predcls with constraint  R@20 59.0-62.0  R@50 61.6-63.9  R@100 61.7-64.0  mR@100 54.0-56.6
predcls no   constraint  R@20 66.1-71.1  R@50 80.7-84.1  R@100 89.5-92.0  mR@100 84.9-86.7
predcls R@100 gain without the constraint: 25.5-29.1 points; ceiling with it: 88.1-89.3
sgdet   with constraint  R@20 33.0-34.6  R@50 34.5-36.5  R@100 34.6-36.5  mR@100 30.7-32.6
sgdet   no   constraint  R@20 38.3-39.8  R@50 46.7-47.2  R@100 51.1-52.0  mR@100 47.1-48.9
sgdet   R@100 gain without the constraint: 14.6-17.5 points; ceiling with it: 88.1-89.4

The single image shows the first reason the two recalls differ. Its man–horse pair has two correct predicates, and under the constraint the pair can contribute only its top one, “riding”, so “on” is lost at every $K$; without the constraint the second predicate enters the list as soon as $K$ is large enough. Across 300 scenes per seed, pairs with two predicates cap the constrained recall at about 88–89%, yet constrained R@100 stops near 62–64% in PredCls. The rest of the gap is the second reason: when the frequency bias or noise puts a wrong predicate first on a related pair, the correct one is the second or third choice, which only the unconstrained list can reach. Under the constraint, R@50 and R@100 hardly differ because these scenes have at most 132 ordered pairs, and the pairs that matter rank near the top. Mean recall falls further than recall under the constraint, since the rare predicates are the ones the biased argmax loses. In SGDet, wrong labels and jittered boxes remove matches that no ranking can recover, so both recalls fall and the gap narrows. The synthetic numbers are larger than real ones; the mechanisms are those behind the 67 versus 88 of MOTIFS above.

Dense captioning

Johnson et al. (2016, §4) scored dense captioning by mean average precision over all pairs of an IoU threshold (0.3 to 0.7 in steps of 0.1) and a METEOR threshold (0 to 0.25 in steps of 0.05) between predicted and ground-truth regions. Before evaluation, heavily overlapping ground-truth boxes (IoU 0.7) of each validation and test image were merged by averaging into one box with several reference captions.

License

As of October 2026, the copy of the about page on Ranjay Krishna’s University of Washington pages states that Visual Genome by Ranjay Krishna is “licensed under a Creative Commons Attribution 4.0 International License” (CC BY 4.0), based on a work at visualgenome.org; the archived visualgenome.org about page of September 2017 carries the same notice, and every page’s footer links the licence. The notice does not say whether it covers the images as well as the annotations, and the pages state no separate terms for the images. The images are Flickr photographs: the paper takes them from the intersection of YFCC100M and COCO (§5.1), and the YFCC100M paper describes its photos as published by their owners under Creative Commons licences, commercial or non-commercial, which vary from photo to photo (Thomee et al., 2016, §2). By our reading, the image terms are therefore those of each photo’s own licence, and the CC BY 4.0 notice is the licence of the annotations; the site does not say so explicitly. The image metadata file records each image’s Flickr identifier, through which, we assume, the photo’s licence can be traced.

Limitations and Bias

• Incomplete graphs. Annotators described what they noticed, so most true relations are missing: Xu et al. (2017, §4) found only 1.6% of object pairs labelled. A correct prediction of an unlabelled relation counts as a miss, which is why the benchmark uses recall and why absolute numbers stay low.
• Head predicates and language priors. Zellers et al. (2018, §3.1) found that, given the subject and object labels, the most frequent predicate for the pair is correct 70% of the time and the five most frequent contain it 97% of the time; their FREQ baseline, which never looks at the image beyond the detector’s labels, outperformed earlier models on the paper’s mean over modes and recalls (40.2 against 39.3 for their reimplementation of message passing; Table 2). Tang et al. (2020, §5.1) note that 92% of the raw predicate categories have no more than 10 instances. R@K therefore rewards predicting “on”, “has” and “wearing”; mean recall was introduced to expose this.
• Noisy and duplicate boxes, synonyms. The cleanup by Xu et al. (22 corrected, 7.4 deleted and 5.4 merged boxes per image) and the v1.4 re-release show how noisy the raw objects were. Even the 150-class vocabulary keeps near-synonyms, and Zellers et al. (2018, §7) list “wearing” against “wears” as a common failure.
• Canonicalization errors. The authors’ own samples put synset accuracy at 88%, 85.7% and 92.9% for noun phrases, attributes and relationships, with recall of 98.5%, 95.9% and 88.5% (§5.8, Table 5); prepositional relationships such as “on” carry no synset.
• Image content. The images favour people and sports scenes (§5.1, §5.4), and their content follows what Flickr users photographed.
• COCO overlap and contamination. The paper says every image is in both YFCC100M and COCO (§1, §5.1), while Anderson et al. (2018, §4.1.1) found about 51K of the images in the COCO captions dataset and Tang et al. (2020, §5.1) selected 41,859 images shared by Visual Genome and COCO Captions. Both later counts are against COCO’s captioned images, only part of the 328,000 images the paper names, so by our reading they do not contradict the paper outright; a count from the metadata’s COCO identifiers would settle it. Any model pretrained on Visual Genome may have seen COCO validation or test images, so users keep shared images in matching splits, as Anderson et al. did.
• The people in the images and the annotators. As of October 2026, neither the paper (journal version or arXiv preprint) nor the official pages say anything about the consent of the people photographed or about removing an image on request. The paper reports the annotators’ demographics: 93.02% of workers in the United States, ages 18 to 68 with most between 25 and 34, and 54.15% male (§4.1, Table 2, Figure 9).
• Availability. As of October 2026 (archived copy of 5 October 2026) the address printed in the paper and in later papers, visualgenome.org, serves unrelated online-casino content. The dataset pages survive in archived copies and in a copy on the first author’s University of Washington pages, whose data readme link returns an error.

Common Uses

• Bottom-up features. The Faster R-CNN with 1,600 object and 400 attribute classes trained on Visual Genome (Anderson et al., 2018) supplied the region features of many captioning, VQA and pretraining models; the image captioning, visual question answering and vision–language pretraining articles describe those models.
• Scene graph generation. The 150/50 split with PredCls, SGCls and SGDet is the standard benchmark, often run through Tang’s benchmark code.
• Paired image–text data. Region descriptions enter pretraining corpora next to COCO captions; the vision–language pretraining article gives UNITER’s counts.
• Question answering and reasoning. Its QA pairs augment VQA training, and GQA builds compositional questions from its scene graphs.
• Dense captioning and region-level grounding, and as a source of boxes and region descriptions for generated instruction data, such as the LRV-Instruction set described in multimodal hallucination.

Historical Milestones

Scene graph results are percentages on the test images of the Xu et al. split, with the graph constraint unless marked. Detectors, backbones and code differ between rows:

Year	Result	Data and protocol	Source
2016	Paper baselines: relationship classification top-1 8.74% from a VGG crop of the subject–object union; QA answered by the 100 most frequent training answers, 41.1%, against 96.6% for humans choosing among ten answers (journal version only)	the paper’s own splits; the same baseline numbers in the arXiv preprint (2016, v1.0 counts) and the journal version (v1.2 counts)	Krishna et al., 2017, Tables 7, 9
2016	DenseCap (FCLN): 5.39 mAP	the authors’ version of the regions, 5,000 test images, dense captioning AP with RPN proposals	Johnson et al., 2016, Table 1
2017	Iterative message passing: PredCls R@100 53.08, SGCls 24.38, SGGen 4.24	first results on the 150/50 split	Xu et al., 2017, Table 1
2017	Pixels to Graphs: PredCls R@100 75.2 (86.4 in the updated arXiv v2 table)	no graph constraint	Newell & Deng, 2017, Table 1 (NIPS); arXiv v2 Table 1
2018	MotifNet: PredCls R@100 67.1, SGCls 36.5, SGDet 30.3; the image-blind FREQ baseline 64.1, 34.0, 27.6; IMP reimplemented with their detector 61.3, 35.4, 24.5	adds R@20 and states the constraint	Zellers et al., 2018, Table 2
2019	VCTree: PredCls mR@100 19.4, against 15.3 for MOTIFS	mean recall over the 50 predicates	Tang et al., 2019, Table 2
2020	Unbiased inference (TDE) on MOTIFS: PredCls mR@100 29.1, against 15.8 for the same model without it	reimplemented in Tang’s code, Faster R-CNN with a ResNeXt-101-FPN backbone (§5.3)	Tang et al., 2020, Table 1

The rows trace the benchmark’s history. The first scene graph numbers were limited by detection: the message-passing model went from 4.24 to 24.5 SGDet R@100 when Zellers et al. reimplemented it with their own detector, edge model and NMS settings. The image-blind frequency baseline then showed how much of R@K the label statistics explain, and from 2019 papers reported mean recall beside it to show gains on rare predicates. The Up-Down detector (Anderson et al., 2018) mattered more outside this benchmark; its captioning and VQA results are in the articles linked under Common Uses.

Related Datasets

• COCO shares about 51K of the images, by Anderson et al.’s count, and supplies whole-image captions and 80 fixed categories with masks, where Visual Genome has open-vocabulary boxes and region phrases.
• Visual Relationship Detection (Lu et al., 2016), by Lu, Krishna, Bernstein and Fei-Fei, is the relationship dataset whose Recall@K protocol Xu et al. adopted.
• Open Images has visual relationship annotations from a controlled vocabulary, exhaustively verified, which its authors contrast with Visual Genome’s free-form ones (as the Open Images article reports).
• GQA generates balanced compositional questions from Visual Genome scene graphs; VQA asks free-form questions about COCO images with ten answers each.
• Flickr30k Entities, described in the Flickr30k article, grounds the phrases of whole-image captions in boxes, the other common source of phrase grounding data.
• LVIS also ties a large vocabulary to WordNet synsets, for instance segmentation of more than 1,000 categories on COCO images.

Resources

• Official pages: the dataset site at visualgenome.org survives only in archived copies, for example the download page listing v1.4, v1.2 and v1.0 and the v1.4 readme. As of October 2026 a copy of the home, download and about pages is on Ranjay Krishna’s University of Washington pages.
• Scene graph split: Danfei Xu’s preprocessing scripts and the preprocessed VG-SGG.h5 linked from their README (commit 4c9e3c6).
• Evaluation: Scene-Graph-Benchmark.pytorch (commit ceb71fa, third-party), whose METRICS.md defines R@K, ng-R@K, mR@K and zR@K and lists reimplemented baselines under both recalls.
• Paper: the open-access IJCV version (DOI 10.1007/s11263-016-0981-7; the publisher’s PDF is also in the CWI repository) and the earlier preprint, arXiv:1602.07332v1; the paper article describes both.