Microsoft COCO: Common Objects in Context

“Microsoft COCO: Common Objects in Context” is a paper by Tsung-Yi Lin and Serge Belongie (Cornell), Michael Maire and Pietro Perona (Caltech), James Hays (Brown), Deva Ramanan (UC Irvine), and Piotr Dollár and C. Lawrence Zitnick (Microsoft Research), presented at the European Conference on Computer Vision (ECCV) in 2014. It introduced COCO, a collection of 328,000 photographs of everyday scenes in which every instance of 91 object categories was to be outlined with its own segmentation mask, and described how the images were found and annotated by crowd workers. The European Computer Vision Association lists it among the recipients of the Koenderink Prize at ECCV 2024, alongside the LSD-SLAM paper. This article covers the publication; the dataset as released and used today, its splits, evaluation protocol, later annotation types and milestones are described in the COCO article.

The paper exists in four versions whose numbers differ, so each detail below names its version. This article was written from the Springer copy of the ECCV paper (16 pages) and the three arXiv versions of 1405.0312: v1 (1 May 2014), v2 (5 July 2014), whose text matches the ECCV paper apart from layout and an appendix, and v3 (21 February 2015), an extended version that adds Lubomir Bourdev and Ross Girshick as authors, moves Dollár to last author, and reports a later state of the annotation.

Problem

The authors framed object recognition as one part of scene understanding, and asked which datasets would carry the field further. They grouped existing recognition datasets into three kinds (Fig. 1 and §2 of the ECCV paper):

• Classification datasets, from MNIST and COIL through Caltech 101 and 256, CIFAR and ImageNet, give image-level labels. ImageNet had used the WordNet hierarchy to reach both entry-level and fine-grained categories.
• Detection datasets give boxes. The paper counts 20 categories over 11,000 images in PASCAL VOC (Everingham et al., 2010), with over 27,000 boxes and almost 7,000 detailed segmentations, and 350,000 boxes over 400,000 images in the 200-category ImageNet detection challenge.
• Semantic scene labeling datasets label every pixel by class without separating instances. The SUN database (Xiao et al., 2010) came closest to COCO’s goals, with 908 scene categories and 3,819 object categories, but its instance counts followed a long tail: 20,213 walls and 16,080 windows against 179 airplanes and 349 boats.

Against this background the paper named three problems it wanted a dataset to address. Web image searches for a category tend to return iconic views: one large object near the center of a neatly composed photo, which the authors believed recognizers already handled fairly well, unlike objects in the background, partly occluded or in clutter. Many objects can only be identified from their context, which requires scenes with several objects rather than objects in isolation. And precise 2D localization requires every instance of every category to be labeled and segmented, since a box is a crude stand-in for an articulated object, most of whose box pixels may lie off the object (Fig. 8).

Contribution

• A category list of 91 entry-level “thing” categories in 11 supercategories, chosen for frequency, usefulness and diversity, with all 20 PASCAL VOC classes kept for backward compatibility.
• An image collection method that queried Flickr for pairs of categories and for scene–object pairs to obtain mostly non-iconic images, 328,000 in all.
• A crowdsourcing pipeline on Amazon Mechanical Turk (AMT) in three stages, category labeling, instance spotting and instance segmentation, with its costs and an analysis of worker quality against experts.
• Statistics comparing COCO with ImageNet, PASCAL VOC 2012 and SUN, and baselines for box detection with a deformable part model (DPM) and for segmentation obtained from detections.

Method

Choosing the categories

The authors wanted categories that were representative, relevant to applications and frequent enough for a large dataset (§3 of the ECCV paper). They kept only things, objects whose instances can be outlined individually, and left out stuff such as sky, street and grass, since the goal was localizing instances, though they expected stuff labels to help later as context. They limited the list to entry-level names, the words people use when describing objects (dog rather than mammal or German shepherd), so each category could gather many instances, and left out object parts such as faces, hands and wheels while noting that these would be useful.

Candidates came from the PASCAL VOC classes, a subset of the 1,200 most frequently used words that denote visually identifiable objects, and the objects named by several children aged 4 to 8 asked to list everything they saw indoors and outdoors; the appendix of all three arXiv versions also names WordNet, LabelMe and SUN as sources. The ECCV paper and v1–v2 put the candidate list at 271; v3 lists 272, in descending order of votes (Table 2 of v3). The co-authors scored each candidate from 1 to 5 for how common it is, how useful in practice and how different from the others. The final selection favored high scores while keeping the number of categories per supercategory balanced, dropped categories for which more than 5,000 instances would be hard to obtain, and added back any PASCAL VOC class. In v3 the authors report that the 2014 release segmented only 80 of the 91 categories (§6 of v3; the COCO article lists the other eleven and the reasons given).

Collecting non-iconic images

The paper sorts photos into iconic-object images, iconic-scene images (canonical views of places, usually without people) and non-iconic images, and aimed for a majority of the last kind, citing Torralba and Efros (2011) for the claim that datasets with more non-iconic images generalize better. Two strategies served this aim. Images came from Flickr, whose amateur photos are less often iconic than search-engine results, a source PASCAL VOC had popularized. And instead of single categories, the authors searched for pairs: “dog” returns large centered dogs, while “dog + car” returns scenes, which they found usually contained further categories besides the two searched. A second set of queries paired objects with 40 scene categories from SUN, half indoor and half outdoor (appendix of v1–v3). At most five photos were kept from one photographer within a short time window. For categories that rarely co-occurred with others, single-category searches were followed by explicit filtering of iconic images.

The appendix describes that filtering as an AMT task: workers removed invalid or iconic images from a grid of 128 candidates, guided by examples of both kinds, and the authors found the wording of the instructions crucial.

The annotation pipeline

All crowdsourcing used AMT (§4). In category labeling, a worker went through the 11 supercategories in turn, a hierarchical scheme following Deng et al. (2014) that avoids 91 yes-or-no questions per image, and for each category present dragged its icon onto one instance. In instance spotting, a worker placed a cross on each instance of one category, up to 10 per image, starting from the icon placed in the first stage and with a magnifying glass that doubles the resolution of the selected region. In instance segmentation, a worker outlined one spotted instance in an interface adapted from OpenSurfaces (Bell et al., 2013), seeing the masks already drawn in the image, and could report that no instance or no unsegmented instance remained. Because spots from different workers could refer to the same object, segmentation tasks were posted one after another, skipping spots already covered by a mask (appendix).

The number of workers per image in the first two stages, and the effort reported, grew from version to version:

Version	Workers per image, stages 1 and 2	Category labeling	Instance spotting	Total stated in the discussion
arXiv v1 (May 2014)	3	11,972 worker hours	6,715 worker hours	over 32,000 worker hours
ECCV 2014 and arXiv v2	5	17,751 worker hours	8,417 worker hours	around 60,000 worker hours
arXiv v3 (Feb 2015)	8	about 20,000 worker hours	about 10,000 worker hours	over 70,000 worker hours

Segmentation was the expensive stage, at over 22 worker hours per 1,000 masks in every version, so each instance was outlined by a single worker (the ECCV text says each image; v3 says each instance). Since most workers at first drew only coarse outlines, every worker had to pass a training task for each category, redrawing a mask until it matched the reference; only about 1 in 3 workers passed. The ECCV paper, like v1 and v2, adds that approved workers’ output was checked periodically. Groups of instances too tightly packed to separate, such as a crowd of people, were marked as one segment labeled “do not care”, so that detections in them would not affect a detector’s score.

Version 3 describes the further quality steps taken after the conference paper. Each mask went to a verification task: workers saw a grid of 64 masks, four of them known to be bad, and had to find at least three of those four. Each mask was first shown to 3 workers; if any flagged it, 2 more judged it, and a mask with fewer than 4 favorable votes out of 5 was discarded and its instance returned to the unsegmented pool. All work from approved workers who consistently produced poor masks was also discarded. Instances beyond the first 10 to 15 of a category in an image were labeled as a crowd by painting their pixels, an interface that v3 compares to semantic segmentation.

The first two arXiv versions also report progress: by 1 May 2014, 590,000 masks had been collected for 55 categories, workers were producing 25,000 masks in 480 hours each day, and the authors expected to finish in about three months (Appendix II of v1 and v2).

Checking worker quality

The paper measured the first stage against seven experts, co-authors of the paper, by comparing each expert’s precision and recall with those of the union of one to ten AMT workers (Fig. 4a). Recall mattered more than precision, since false positives could be removed in later stages but missed categories could not be recovered. The ECCV text reports that five workers, the number used, matched the recall of most experts, and that the experts’ own recall was only 65–80% because a category counted as present if any single expert labeled it, a liberal definition under which many presences are ambiguous; recall of 70–75% captured the unambiguous categories. Fig. 4b plotted each worker’s precision and recall against labels from a majority vote, colored by the number of jobs done: workers who did more jobs were generally more accurate, and all jobs from workers below a threshold curve were rejected. The ECCV caption of Fig. 4 still says that three workers match the experts and describes a majority vote of AMT workers, wording carried over from v1, where three workers were used; the text describes a union of five.

Version 3 restates this analysis with eight workers and a different ground truth, the majority vote of the experts. The union of eight workers then had higher recall than any expert, and recall leveled off at around 9 to 10 workers. The authors argue that an unambiguous category that each worker marks with probability above 0.5 is missed by all eight with probability at most $0.5^8 \approx 0.004$, and estimate from the growth of recall with the number of workers that over 99% of categories not later rejected as false positives were found. For Fig. 4b, each worker was now scored against the union of all other workers’ answers, which raised the precision shown. In our reading the expert and worker points plotted in Fig. 4a look the same in the ECCV paper and in v3, although the ground truth stated for them changed between the versions.

Results

Dataset statistics

Section 5 compares COCO with ImageNet (its detection validation set, since ImageNet’s training images had one labeled object each), PASCAL VOC 2012 and SUN, in Fig. 5. The averages printed in the figure’s legends are:

Dataset	Categories per image	Instances per image
COCO	3.5	7.7
ImageNet detection	1.7	3.0
PASCAL VOC	1.4	2.3
SUN	9.8	17.0

The text summarizes ImageNet and PASCAL VOC as having fewer than 2 categories and 3 instances per image, while the figure gives ImageNet 3.0 instances, as does the introduction. The text also says that only 10% of COCO images contain a single category, against over 60% for ImageNet and PASCAL VOC. Read from Fig. 5(b), the share of single-category images is about 18% for COCO and about 61% and 71% for ImageNet and PASCAL VOC; 10% is where COCO’s curve starts in Fig. 5(c), the share of images with a single instance. SUN, being scene-based, has the most context per image. Fig. 5(d) places each dataset by number of categories and instances per category on log–log axes: COCO has fewer categories than ImageNet or SUN but more instances per category, which the authors hypothesized would help learn models capable of precise localization, and more of both than PASCAL VOC. Fig. 5(e) shows instance size as a fraction of image size; reading the plot, COCO and SUN place about two thirds of their instances in the two smallest bins (4% and 6% of the image), against under half for ImageNet and PASCAL VOC. Version 1’s Section 5 text printed 7.6 instances per image, although its introduction and Fig. 5 give 7.7.

Counts moved with the annotation. The abstract of every version gives 91 categories and 2.5 million labeled instances in 328,000 images, and the introduction adds that 82 categories have more than 5,000 labeled instances. The ECCV paper planned splits of 164,000 training, 82,000 validation and 82,000 test images, divided by date and photographer after removing near-duplicates, and reported over 580,000 segmented people (v1: over 200,000). Version 3 instead reports a 2014 release of 82,783 training, 40,504 validation and 40,775 test images, with nearly 270,000 segmented people and 886,000 segmented instances in the 2014 training and validation images, and a planned cumulative 2015 release of 165,482 training, 81,208 validation and 81,434 test images.

Detection baseline

The detection experiment (§6 of the ECCV paper, §7 of v3) used a 55,000-image subset of the training and validation data, before the final split according to a footnote in v3, with tight boxes derived from the masks. Two models were compared on the 20 PASCAL categories: DPMv5-P, release 5 of the deformable part model of Felzenszwalb et al. (2010) trained on PASCAL VOC 2012, with the VOC results reported by its authors, and DPMv5-C, the same code with default settings trained on COCO with 5,000 positive and 10,000 negative examples (the text says images, the table caption training examples). Table 1 gives average precision per category; the paper does not restate the matching criterion for it. The averages are:

Model	Tested on PASCAL VOC 2012	Tested on COCO
DPMv5-P (trained on PASCAL VOC)	29.6	16.9
DPMv5-C (trained on COCO)	26.8	19.1

The authors read the drop for DPMv5-P, which they describe as nearly a factor of 2 (the averages give 1.75), as evidence that COCO’s images are harder: more occluded, cluttered and non-iconic. On PASCAL VOC the COCO-trained model beat the PASCAL-trained one in 6 of the 20 categories (bicycle, bird, bus, cow, horse and TV monitor) and did worse on others such as dog, cat and person; on COCO it was better in 14 of 20, by our count from the table. Citing earlier work, the authors note that hard, non-iconic training examples can act as noise for a model not rich enough to capture their variability. Borrowing the cross-dataset “performance drop” of Torralba and Efros (2011), they compared the gap between the two test sets, 12.7 AP for DPMv5-P and 7.7 AP for DPMv5-C, and took these observations as support for two hypotheses: that COCO is significantly harder than PASCAL VOC, and that models trained on COCO can generalize better to easier datasets given more training data.

In our reading, the comparison is less direct than it looks. Torralba and Efros measured the drop relative to a model’s score on its own dataset’s test data (see dataset bias); by that measure DPMv5-P loses 43% when moved to COCO, while DPMv5-C scores higher on PASCAL VOC than on COCO itself, a negative drop. The smaller gap for DPMv5-C therefore mixes generalization with the difference in difficulty between the two test sets. Version 1 printed the same numbers as two separate tables, one per test set.

Segmentation from detections

For masks, the authors averaged the aligned training masks of each DPM mixture component into a soft shape mask, thresholded it and pasted it into each detected box (Fig. 7), following earlier work that derived segmentations from detections. To separate segmentation quality from detection errors, they scored only correct detections, those whose box had an intersection over union (IoU) of at least 0.5 with the ground truth, and measured the IoU between predicted and ground-truth masks. For the person category (Fig. 9, center) most instances have a lower mask overlap than box overlap. Averaged per category on COCO (Fig. 9, right), the overlap ranges, as read from the bar chart, from about 0.3 for sofa to about 0.65 for bus, with most categories between 0.45 and 0.55, which the authors took as a measure of how hard COCO’s masks are to recover. Every version calls it a baseline; version 1 also introduced it as novel evaluation criteria for instance-level segmentation, a paragraph later versions dropped.

Impact

The dataset described here became the basis of the COCO challenges: a captioning challenge with human judges at the CVPR 2015 LSUN workshop, an object detection challenge at ICCV 2015 held jointly with ImageNet, with box and segmentation output tracks, and a keypoint challenge at ECCV 2016 (COCO challenge pages). Its per-instance masks and its evaluation server shaped the benchmarks for object detection and instance segmentation described in the COCO article, and later datasets were annotated on its images, among them COCO-Stuff (Caesar, Uijlings & Ferrari, 2018) and LVIS (Gupta, Dollár & Girshick, 2019). The European Computer Vision Association’s award list names it, with LSD-SLAM, as a Koenderink Prize recipient at ECCV 2024.

Limitations

• A snapshot of unfinished work. At publication segmentation was still under way, and every version reports different counts; the ECCV figures reflect the state of mid-2014, and the abstract’s 2.5 million instances exceed the 1.5 million the website lists for the release (the COCO article compares the two).
• No evaluation protocol. The paper defined splits and withheld test annotations, but v3 deferred the discussion of metrics until the evaluation server was finished. No later arXiv version followed, and the protocol, AP averaged over IoU thresholds from 0.5 to 0.95, was published on the dataset website instead.
• Preliminary baselines. The DPM experiments used a subset chosen before the final split and a detector that the authors expected stronger methods, such as convolutional networks, to surpass; they serve to compare datasets, not to set a reference score.
• Deliberately narrow labels. Things only, entry-level names only, no parts or fine-grained categories, one worker’s outline per instance, and crowds left unsegmented. Gupta, Dollár and Girshick (2019) later measured COCO’s masks as coarser than expert outlines.

What Came After

The ECCV discussion listed directions for future annotation, and v3 added captions and a 2015 release. Their fate, as of October 2026:

• Stuff. Stuff annotations were added for COCO 2017 images, first on a 40,000-image training subset and the 5,000 validation images in 2017, and for all 2017 images, with panoptic annotations, in 2018 (download page; Caesar, Uijlings & Ferrari, 2018).
• Keypoints. The 2016 keypoint task released keypoints for more than 250,000 people, covering the majority of COCO’s people at medium and large scales (keypoint challenge page).
• Sentence descriptions. Version 3 announced five captions per image and a separate publication, which appeared as Chen et al. (2015) with its own evaluation server.
• Occlusion, scene types and attributes. The download page lists no such annotations among the official files.
• The 2015 release and the missing categories. In our reading of the download page, the cumulative 2015 release appeared only in part. A 2015 test set of about 81,000 images matches the planned test split in size, but the labeled training and validation images were never extended: the 2017 splits redistributed the 123,287 labeled images of 2014 (82,783 + 40,504) as about 118,000 training and 5,000 validation images, using the same images, and added unlabeled images (120,000 in the page’s text, 123,000 in its file list). The eleven unsegmented categories were not added, and the dataset is still evaluated on 80 categories.