Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

“Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations” is a paper by Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein and Li Fei-Fei, from Stanford University with collaborators at Dresden University of Technology, Yahoo, Snapchat and CWI Amsterdam. It was posted on arXiv in February 2016 and published, open access, in the International Journal of Computer Vision in 2017 (received 23 February 2016, accepted 12 September 2016, online 6 February 2017). It introduced Visual Genome: about 108,000 photographs in which crowd workers wrote dozens of region descriptions per image and turned them into boxed objects, attributes, pairwise relationships and one scene graph per image, alongside question–answer pairs, with every concept mapped to a WordNet synset. The paper describes how the data were collected, analyses them at length and reports baselines for four tasks. This article covers the publication and its two versions; the dataset as it is used today, its versions and the scene graph benchmark built on it are described in the Visual Genome article.

Section, table and figure numbers below are those of the journal version, read in the publisher’s PDF as deposited in the CWI institutional repository. The arXiv preprint (v1, the only arXiv version) reports a different, earlier state of the data, and is cited as “arXiv v1” where it is used.

Problem

The paper opens with a distinction between perceptual tasks, such as classification and object detection, and cognitive ones, such as describing an image or answering questions about it (abstract, §1). Its example asks what vehicle a person is riding: answering needs the objects and also the relations riding(man, carriage) and pulling(horse, carriage). The authors argued that models for cognitive tasks were still trained on data designed for perception. COCO gave 80 object categories with masks and five captions per image, and VQA added free-form questions, but a COCO caption summarizes the most salient part of a scene: the paper quotes three COCO captions of one image, all about the elephant and none about anything else (§3.2). A captioning model’s “two men are standing next to an elephant” could not tell apart images that differ in what the people are doing.

The representation the paper adopted came from earlier work by several of the same authors: Johnson et al. (2015) had used the scene graph, objects with attributes as nodes and relationships as edges, for image retrieval, and the paper says it uses a similar representation that generalizes earlier ones (§1, §3.5). Relationship datasets were small: the paper’s comparison lists 17 relationships in Visual Phrases, 156 actions in COCO-a and 6,500 relationships in VisKE, the last in far fewer images (§5.5, Table 1). The paper proposed to add three things at scale (§1): a grounding of visual concepts in language, a dense set of region-level descriptions and questions per image, and a formal, knowledge-base-like representation of each image.

Contribution

• A seven-part annotation of each image (§2): region descriptions, objects, attributes, relationships, region graphs, scene graphs and question–answer pairs, with objects, attributes, relationships and the noun phrases of descriptions and questions mapped to WordNet synsets.
• A crowdsourcing pipeline that collects the structure through language: workers first describe regions in free text, and later stages extract objects, attributes and relationships from each description (§4).
• The dataset itself, described in the journal version as 108,077 images with an average of 50 region descriptions, 35 objects, 26 attributes, 21 relationships and 17 question–answer pairs per image (Table 1; §1 and §5.7 for the question–answer pairs; the abstract says “over 108K images”).
• An analysis of each component (§5) and baselines for attribute prediction, relationship prediction, region description generation and question answering (§6).

The paper deliberately does not define a benchmark. It presents the dataset as a general-purpose representation of the visual world “without bias toward a particular task” (§3.1), and the baselines use splits made for each experiment.

Method

The Visual Genome article describes the components and the pipeline step by step. This section records the design decisions as the paper states them.

Language first. The authors’ first attempt asked workers for scene graphs directly, and workers wrote easy, frequent relations such as wearing(man, shirt), as in earlier graph datasets (§4.2). Asked to describe an image in sentences, workers started with the most salient part and then moved to the others, so the pipeline collected region descriptions first, three boxes and three phrases per task, until an image had 50. To keep descriptions diverse, each new phrase was compared with the image’s earlier ones and with the 100 most common phrases in the dataset by a BLEU-like score: the geometric mean of $n$-gram precisions $p_n$ times a brevity penalty (Eqs. 1–2),

$$S(d_i, d_j) = b(d_i, d_j)\,\exp\!\Big(\frac{1}{N}\sum_{n=1}^{N} \log p_n(d_i, d_j)\Big), \qquad b(d_i, d_j) = \begin{cases} 1 & \text{if } |d_i| > |d_j| \\ e^{\,1 - |d_j|/|d_i|} & \text{otherwise,} \end{cases}$$

and the phrase was rejected above a threshold of 0.7. Workers saw only the seven most similar earlier phrases, so they did not have to read all of them.

Structure from each phrase. One worker per description extracted and boxed its objects; boxes had to cover the object and be tight to within four pixels, relaxed from one (§4.3). Suggestions from the image’s earlier boxes let a worker reuse the box of “man” for “person”, which built co-reference chains, and nouns from the Stanford dependency parser were offered as hints but not relied on, because the parser missed compound nouns. A second stage added attributes and relationships, giving one region graph per description (§4.4). The scene graph is the union of the region graphs, with objects from different graphs merged when their boxes overlap with an IoU of 0.9 and a worker confirms the merge (§4.5, Fig. 12).

Questions. Each question starts with one of six “W” words: what, where, when, who, why and how (§4.6). Freeform questions were written eight at a time about the whole image, with at least three different Ws; region-based questions were written from a single description, chosen among regions larger than 5,000 pixels with phrases longer than four words. The paper puts the cost of about 20 region-based pairs at that of eight freeform ones, and says the freeform ones are more diverse and the region-based ones more factual. Binary yes/no questions were excluded on purpose, because 38.37% of VQA questions are yes/no and a model can score reasonably by answering “yes” or “no” (§5.7).

Verification and canonicalization. Three workers voted on every annotation except objects, which were checked with the authors’ faster “rapid judgment” interface (§4.7). WordNet mapping took the most frequent sense by lexeme counts and corrected it with hand-written rules (30 for nouns, 15 for attributes, 20 for relationships), mostly to prefer concrete or spatial senses over abstract ones (§4.8). Prepositions were dropped, since WordNet has none, and verbs were matched through WordNet’s sentence frames and reduced to a root hypernym. Workers then checked each mapping against four alternatives, without being told which of the five was the proposed one, so that they would not simply accept it.

The operation was large: over 33,000 Amazon Mechanical Turk workers, about 800,000 tasks, six months of collection after 15 months of experimenting with the representation, and pay designed to come to $6–8 an hour of continuous work; 94.1% of workers who completed one task went on to do more (§4.1). A survey task gave the workers’ demographics: 93.02% were in the United States (Table 2), most were aged 25 to 34, and the split was 54.15% male and 45.85% female (Fig. 9).

Results

Two versions of the numbers

The journal version does not merely fix typographical errors in the preprint. It reports a later and larger state of the annotations. Its image count, 108,077, is the one the official home page showed while version 1.2 (August 2016) was current, and the arXiv v1 count, 108,249, the one it showed under version 1.0 (December 2015), so by our reading the two versions describe those two releases (see the Visual Genome article). The paper itself does not mention dataset versions. The main counts as each version prints them:

Quantity	arXiv v1 (February 2016)	IJCV (2017)
Images	108,249 (§1, §5.1); 108,000 (Table 1)	108,077 (§1, §5.1, Tables 1, 3)
Region descriptions per image	42 (§2.1); 50 (Table 1)	50 (§2.1, §5.2, Table 1)
Region descriptions in total	4,158,841 (§6.3)	5,406,939 (§6.3)
Objects per image	21 (§2.2; 21.26 in Table 4); 16 (Table 1)	35 (§2.2, Tables 1, 4)
Objects in total	4,102,818 (Table 1); 255,718 (Table 3)	3,843,636 (Tables 1, 3; §5.3)
Object categories	over 17,000 (§3.3); 18,136 (Table 3); 76,340 (Table 1)	33,877 (§3.3, Tables 1, 3)
Attributes per image	16 (§2.3, Table 1; 16.21 in Table 4); 18 (abstract, §3.1, §6.1); 21 (§5.4)	26 (§2.3, §5.4, Tables 1, 4)
Attributes in total; distinct	1.6M (§5.4) or 1.8M (§6.1); 13,041 (§5.4) or 15,626 (Table 1)	2.8M (§5.4, §6.1); 68,111 (§5.4, Table 1)
Relationships per image	18 (§2.4, Table 1; 18.67 in Table 4); 19 (§5.5)	21 (§2.4, §6.2, Tables 1, 4); 19 (§5.5)
Relationships in total; distinct	over 1.8M (§5.5); 13,894 (§5.5) or 47 (Table 1)	2.3M (§6.2); 42,374 (§5.5, Table 1)
Question–answer pairs	1,773,258	1,773,258
Canonicalization accuracy, nouns / attributes / relationships	88.0 / 85.7 / 92.9 (Table 5); 86% / 83.5% / 77.6% (text of §5.8)	88% / 85.7% / 92.9% (§5.8, Table 5)

The journal version resolved most of the preprint’s internal contradictions: the object totals of Tables 1 and 3 now agree, the merge threshold is 0.9 in both the text of §4.5 and Figure 12 (0.8 and 0.9 in the preprint), and §4.6 asks for “six Ws” where the preprint said seven, adding “which”. The canonicalization accuracies of Table 5 are the same in both versions; only the preprint’s text gave lower ones, and the journal text now matches the table. Table 1 of the journal version also states how its category counts were made, by lower-casing and stemming the names; that may be why they exceed the more than 18,000 WordNet synsets that §5.1 gives for the same concepts. The number of question–answer pairs is the same in both versions.

Analyses

The paper characterizes each component (§5). The images are the Creative Commons photos in both COCO and YFCC100M, between 72 and 1,280 pixels wide (500 on average); classified with ImageNet’s method, they fall into 972 synsets led by “ski”, “ballplayer” and “racket”, a bias toward people and sports that the authors acknowledge (§5.1). Region descriptions average five words (§5.2). Most regions span 10–15% of the image width and height, with a peak of full-width regions for “sky”, “ocean” or “snow” and no matching peak in height. Averaging word2vec vectors per description and clustering them gave 71 clusters. A single image’s descriptions span 17 clusters on average, and five random descriptions span four or five clusters against two for an image’s five COCO captions ($t = -240$, $p < 0.01$).

Objects are dense: 35 per image, more categories than any dataset compared, and about 2,239 objects per category over the top 200 categories, close to the 2,671.5 of ImageNet’s detection set (§5.3, Fig. 21). “Man” occurs about twice as often as “woman”, which the paper attributes to biases in the images. About 34% of regions have no attribute, because many descriptions are about relationships (§5.4), and co-occurrence graphs of attributes form cliques typical of one kind of object, such as athletic, young, skateboarding for skateboarders. Relationships follow a Zipf distribution headed by “on” (§5.5). The subjects and objects of “riding” separate cleanly into riders and things that can be ridden, which the authors present as affordances learnable from the data. Questions average 5.7 words and answers 1.8 words; 57.3%, 18.1% and 15.7% of answers have one, two and three words, against 89.32%, 6.91% and 2.74% in VQA (§5.7). Canonicalization was checked against hand labels on samples of 200 region descriptions, 200 attributes and 200 relationships, and the paper notes that its open-vocabulary setting is not comparable with word-sense-disambiguation benchmarks (§5.8).

Baselines

The experiments cover the components the authors considered least studied; object detection and scene-graph retrieval were left out as already established (§6).

Experiment	Setup	Result
Attribute prediction (§6.1, Table 6)	100 most common attributes, up to 500 samples each (about 50,000 instances); VGG-16 fine-tuned on the object crop, 100 SVMs; images split 80/10/10	top-1 18.97%, top-5 43.11%
Object–attribute pairs (same)	about 43,000 instances, object predicted too	top-1 43.17%, top-5 71.97%
Relationship prediction (§6.2, Table 7)	100 most frequent relationships, crop of the union of the subject and object boxes	top-1 8.74%, top-5 26.69%
Subject–relationship–object (same)	the triple predicted jointly	top-1 25.83%, top-5 65.57%
Region descriptions (§6.3, Table 8)	NeuralTalk on VGG-16 features of ground-truth regions, split by image	BLEU-1 0.17, BLEU-4 0.01, CIDEr 0.30, METEOR 0.09; judged correct by workers 43.03%, against 1.6% for the same model trained on full-image captions
Question answering (§6.4, Table 9)	60/40 split by image; predict the $k$ most frequent training answers, correct on an exact match	41.1%, 57.3% and 64.1% for $k$ = 100, 500, 1,000; humans 96.6%

The authors draw three readings. Relationships are much harder to recognize from appearance than attributes, because one predicate looks very different across scenes, and knowing the objects helps: “drive” only occurs with a person as subject. A captioning model trained on region phrases describes the region it is shown, while one trained on whole-image captions describes the whole scene and almost always fails on a region. And the answers have a long tail: VQA’s 100, 500 and 1,000 most frequent answers cover 63%, 75% and 80% of its test answers, against 41.1%, 57.3% and 64.1% here. By question type, “why” is hardest (3.4% at $k$ = 100) and “how”, mostly counting questions, the easiest (78.0%). The human figure is new in the journal version: people chose the correct answer among ten, nine of them random answers from the dataset, so it measures a multiple-choice task rather than the open-ended one the baseline faces.

The baseline numbers are identical in both versions, although the data counts changed. In §6.3, for instance, the journal version gives 5,406,939 descriptions split 3,784,857 / 811,040 / 811,040 where the preprint gave 4,158,841, and Table 8 is unchanged. The paper does not say whether the experiments were rerun on the later data. By our reading they were not, and the journal version’s setup counts describe data the results were not computed on.

Impact

The paper’s §7 lists applications it expected the data to enable, and several followed, often by the same group. Dense captioning, describing every region of an image in a phrase, came first. Johnson, Karpathy and Fei-Fei (2016) had already trained on their own version of the region descriptions (94,313 images) before the journal paper appeared. Scene graph generation became a benchmark when Xu et al. (2017) cleaned the scene graphs and kept the 150 most frequent object classes and 50 predicates, the split that most later work uses. The object and attribute annotations trained the detector behind the “bottom-up” region features of Anderson et al. (2018), which fed many captioning, VQA and pretraining models. GQA (Hudson & Manning, 2019) generated compositional questions from Visual Genome scene graphs over 113K images, adding 5,000 hidden scene graphs of its own. The Visual Genome article gives the details and results of each.

The paper also proposed scene graphs as a way to evaluate captions and answers, matched against an image’s ground-truth graph (§7). SPICE (Anderson et al., 2016) built a caption metric on scene graphs, a structure it calls consistent with Visual Genome and earlier datasets. It compares graphs parsed from the candidate and reference captions, not an image’s annotated graph, so it realizes the idea only in part; the image captioning article describes the metric. The paper’s main legacy, by our reading, is the representation: grounded relationship triples and attributes at a scale large enough to train models on, collected through language rather than drawn as graphs.

Limitations

The authors name several limitations. Annotations are incomplete: it is not feasible to collect every attribute and relationship of every object, so evaluation metrics are needed that do not penalize a model for correct predictions missing from the ground truth (§7). Their own attribute baseline is marked wrong for a “white stuffed animal” that is white (§6.1). The exact-match question-answering evaluation suits short answers and fails for long ones (§6.4). The images are biased toward people and some sports (§5.1, §5.4), and synsets were not standardized: “man” and “person” keep different synsets, which users must merge through WordNet themselves (§2.2).

Other problems are visible in the paper itself. The journal version still contradicts itself in places, and where text and tables disagree the table values are reported above:

• Relationships per image are 21 in §2.4, §6.2 and Tables 1 and 4, but 19 in §5.5, which also prints the total as “over 2,347,187 million”.
• The longest description has 16 words in §2.1 and in the caption of Figure 16, but 12 in the text of §5.2.
• §5.6 gives 0.71 objects, 0.52 attributes and “21 relationships” per region graph, where Table 4 gives 0.43 relationships.
• The relationship top-5 accuracy is 26.69% in Table 7 and 29.69% in the text of §6.2.
• Table 8 labels the comparison model “Flickr8K”, while §6.3 says throughout that it was trained on Flickr30K.
• Table 5’s caption promises precision, recall and mapping accuracy but the table has two columns; the text of §5.8 calls the first column “mapping accuracy” and points to it as “Fig. 5”.
• Table 1 lists 27,472 total objects for COCO, while Table 3 lists 2,500,000 objects and 27,472.50 objects per category.
• The three parts of the §6.3 split add up to 5,406,937, two fewer than the stated total of 5,406,939.

Several of these are carried over from arXiv v1: the 16 and 12 word maxima, the 26.69% and 29.69%, the “Flickr8K” label, Table 5’s caption and the two COCO object counts are already there, and its §6.3 split also falls one short of its total. The 19 relationships of §5.5 are in the preprint too, though there they sit beside 18 elsewhere. The “2,347,187 million”, the 21 relationships per region graph and the pointer to “Fig. 5” are new in the journal version.

One claim is hard to square with later work. The paper says that all 108,077 images come from the intersection of COCO and YFCC100M (§1, §5.1), but Anderson et al. (2018, §4.1.1) found about 51K Visual Genome images in the COCO captions dataset, and Tang et al. (2020, §5.1) selected 41,859 images shared by Visual Genome and COCO Captions. Both counts are against COCO’s captioned images, which are only part of the 328,000 the paper names, so by our reading they do not settle the question, and none of the papers explains the difference. The paper also says nothing about the consent of the people photographed.

What Came After

The dataset kept changing after the paper’s numbers were fixed. Version 1.4 (July 2017) re-released cleaner object annotations and the relationships with new subject and object boxes, and added alias files, so, by our reading, neither version of the paper describes the objects and relationships of the latest release. Users had to clean the scene graphs for themselves: Xu et al. (2017, §4) reported averages of 38 objects and 22 relationships per image in the raw graphs, then corrected names and boxes and deleted and merged duplicates before keeping 150 objects and 50 predicates. The long tail the paper plotted in §5.5 became the benchmark’s central difficulty: Tang et al. (2020, §5.1) note that 92% of the raw predicate categories have no more than 10 instances. The Visual Genome article covers this line of work, the frequency baselines and the Recall@K protocol it led to. The incompleteness the authors flagged in §7 is the reason the benchmark is scored by recall rather than precision.

The paper’s address for the data, visualgenome.org, no longer serves the dataset: as of October 2026 it shows unrelated content, and the official pages survive in archived copies and on the first author’s University of Washington pages.