COCO

COCO (Microsoft Common Objects in Context) is a dataset of photographs of everyday scenes in which every instance of 80 object categories is outlined with a segmentation mask. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C. Lawrence Zitnick introduced it at ECCV 2014, and a longer arXiv version (2015) added Lubomir Bourdev and Ross Girshick as authors. Keypoints, captions, stuff, panoptic and DensePose labels were later added to the same images, and its evaluation protocol, average precision averaged over IoU thresholds from 0.50 to 0.95, became the standard score for object detectors.

Purpose

Lin et al. (2014) aimed at three problems in scene understanding: recognizing objects in non-iconic views, reasoning about objects in context, and localizing objects precisely in 2D. An iconic image shows one large, centered object in a canonical view, as a web search for “bike” tends to return. To find cluttered scenes instead, the authors searched Flickr for pairs of categories (“dog + car”) and scene–object pairs, and filtered iconic images out of the rare single-category searches. Precise localization meant a mask for every instance, not only a box. The paper reports 3.5 categories and 7.7 instances per image on average, against fewer than 2 and 3 for ImageNet and PASCAL VOC.

Dataset Size

The paper’s abstract describes 91 object types “easily recognizable by a 4 year old”, with 2.5 million labeled instances in 328,000 images (Lin et al., 2014); 82 of the 91 categories have more than 5,000 instances. The website’s home page lists 330K images (more than 200K labeled), 1.5 million object instances, 80 object categories, 91 stuff categories, 5 captions per image and 250,000 people with keypoints (as of October 2026). The paper describes the collection while segmentation was still under way; the website describes the release. Neither explains the gap between 2.5 and 1.5 million instances.

Classes

Candidates came from the PASCAL VOC classes, frequent words for visible objects, and objects named by children aged 4 to 8. The co-authors voted on them (the extended version lists 272), balancing supercategories and removing categories with too few instances, but kept all PASCAL VOC classes. The 91 categories are things, objects whose instances can be outlined individually (person, chair, car), grouped into 11 supercategories. The authors left out stuff, materials and regions without clear boundaries (sky, street, grass), and fine-grained and part categories.

Only 80 categories received masks. The extended version lists the other 11: hat, shoe and eyeglasses (too many instances), mirror, window, door and street sign (ambiguous), plate and desk (confused with bowl and dining table), and blender and hair brush (too few instances) (Lin et al., 2015). The 80 are the categories evaluated in detection, segmentation and panoptic tasks; their ids lie in the range 1–91, so they are not consecutive.

Annotation Types

All crowdsourcing used Amazon Mechanical Turk in three stages (Lin et al., 2014):

1. Category labeling. For each category present, workers dragged its icon onto one instance, going through the 11 supercategories in turn.
2. Instance spotting. Workers marked every instance of each found category with a cross, up to 10 per category and image.
3. Instance segmentation. One worker outlined each instance, after a per-category training task that about one in three workers passed.

The ECCV paper used five workers per image in the first two stages; the extended version used eight and added a stage in which 3 to 5 workers verified each mask. The annotations are:

• Object instances. A polygon mask (several if occlusion splits the object), an enclosing bounding box, and an area.
• Crowd regions. After 10 to 15 instances of a category in an image, the rest became one crowd region (Lin et al., 2015), stored as a run-length encoded (RLE) mask with iscrowd=1 and ignored in evaluation.
• Person keypoints. 17 keypoints for most medium and large non-crowd people, each flagged as not labeled, labeled but not visible, or visible.
• Captions. At least five per image from Mechanical Turk workers, and 40 for each of 5,000 test images (Chen et al., 2015).
• Stuff. COCO-Stuff (Caesar, Uijlings & Ferrari, 2018) labeled 91 stuff classes on all 164K images of COCO 2017, annotating superpixels to save time.
• Panoptic. Panoptic segmentation (Kirillov et al., 2019) gives every pixel a class and, for things, an instance id; COCO’s panoptic labels combine the 80 thing categories with a merged subset of the stuff classes, without overlaps.
• DensePose. Güler, Neverova and Kokkinos (2018) mapped the pixels of about 50,000 people to a 3D body surface model.

File format

Annotations are JSON files with lists of images, annotations and categories. Each annotation names its image and category and stores segmentation (polygons, or RLE for crowds), area, iscrowd, and bbox as [x, y, width, height] in pixels, measured from the top-left image corner and 0-indexed (COCO data format page). Uncompressed RLE lists alternating runs of 0s and 1s, starting with 0s, over the mask read in column-major order (comments in the COCO API’s mask.py).

The code below reads an invented annotation file in this format, not real COCO data: one $96 \times 64$ image with a person, skis and a crowd region. It converts boxes to corners, rasterizes polygons with Matplotlib by testing pixel centers, decodes the RLE, and assigns COCO’s size buckets (small below $32^2$ pixels, large above $96^2$) by mask area and, for comparison, by box area.

import numpy as np
from matplotlib.path import Path

# An invented COCO-format annotation file (not real COCO data): one 96x64 image
# with a person, a pair of skis, and a crowd region of distant people.
coco = {
    "images": [{"id": 1, "file_name": "invented_000001.jpg", "width": 96, "height": 64}],
    "categories": [{"id": 1, "name": "person", "supercategory": "person"},
                   {"id": 2, "name": "skis", "supercategory": "sports"}],
    "annotations": [
        {"id": 101, "image_id": 1, "category_id": 1, "iscrowd": 0,
         "segmentation": [[18, 4, 30, 4, 32, 14, 42, 22, 40, 32, 34, 27, 35, 60, 27, 60,
                           24, 40, 21, 60, 13, 60, 14, 27, 8, 32, 6, 22, 16, 14]],
         "bbox": [6, 4, 36, 56], "area": 1196},
        {"id": 102, "image_id": 1, "category_id": 2, "iscrowd": 0,
         "segmentation": [[44, 56, 82, 22, 85, 24, 47, 58]],
         "bbox": [44, 22, 41, 36], "area": 180},
        {"id": 103, "image_id": 1, "category_id": 1, "iscrowd": 1,
         "segmentation": {"size": [64, 96],   # uncompressed RLE, column-major
                          "counts": [3852, 8, 54, 10, 55, 9, 57, 7, 54, 10, 53, 11, 56, 8, 55, 9, 1836]},
         "bbox": [60, 9, 8, 11], "area": 72},
    ],
}

def xywh_to_xyxy(b):                     # COCO [x, y, width, height] -> corners
    x, y, w, h = b
    return [x, y, x + w, y + h]

def polygons_to_mask(polys, h, w):       # pixel (r, c) is inside if its center is
    yy, xx = np.mgrid[0:h, 0:w]
    centers = np.c_[xx.ravel() + 0.5, yy.ravel() + 0.5]
    mask = np.zeros(h * w, bool)
    for p in polys:                      # an object may have several polygons
        mask |= Path(np.reshape(p, (-1, 2))).contains_points(centers)
    return mask.reshape(h, w)

def rle_to_mask(rle):                    # runs alternate 0s and 1s, starting with 0s
    h, w = rle["size"]
    vals = np.arange(len(rle["counts"])) % 2
    return np.repeat(vals, rle["counts"]).astype(bool).reshape((h, w), order="F")

def shoelace(p):                         # area enclosed by a polygon
    x, y = np.reshape(p, (-1, 2)).T
    return 0.5 * abs(np.dot(x, np.roll(y, -1)) - np.dot(y, np.roll(x, -1)))

def size_bucket(area):                   # COCO thresholds, applied to the mask area
    return "small" if area < 32**2 else "medium" if area < 96**2 else "large"

img = coco["images"][0]
names = {c["id"]: c["name"] for c in coco["categories"]}
label = np.full((img["height"], img["width"]), ".")
print(" id  category  crowd  corners (x1, y1, x2, y2)  box area  polygon  mask px  "
      "by mask  by box")
for a in coco["annotations"]:
    seg = a["segmentation"]
    if a["iscrowd"]:
        mask, poly = rle_to_mask(seg), "-"
    else:
        mask = polygons_to_mask(seg, img["height"], img["width"])
        poly = f"{sum(shoelace(p) for p in seg):.1f}"
    label[mask] = "c" if a["iscrowd"] else names[a["category_id"]][0].upper()
    box_area = a["bbox"][2] * a["bbox"][3]
    print(f"{a['id']:>3}  {names[a['category_id']]:8s}  {a['iscrowd']:>5}  "
          f"{str(xywh_to_xyxy(a['bbox'])):24s}  {box_area:>8}  {poly:>7}  "
          f"{mask.sum():>7}  {size_bucket(a['area']):>7}  {size_bucket(box_area):>6}")
    assert mask.sum() == a["area"]

print()
for row in label[1::4]:                  # every 4th row and 2nd column of the image
    print("".join(row[1::2]))

Output:

 id  category  crowd  corners (x1, y1, x2, y2)  box area  polygon  mask px  by mask  by box
101  person        0  [6, 4, 42, 60]                2016   1191.0     1196   medium  medium
102  skis          0  [44, 22, 85, 58]              1476    178.0      180    small  medium
103  person        1  [60, 9, 68, 20]                 88        -       72    small   small

................................................
.........PPPPPP.................................
........PPPPPPP.................c...............
........PPPPPPPP..............cccc..............
......PPPPPPPPPPPP............cccc..............
...PPPPPPPPPPPPPPPPP............................
...PPPPPPPPPPPPPPPPP...................SS.......
...PP..PPPPPPPPPP.PP.................SS.........
.......PPPPPPPPPP.................SSS...........
.......PPPPPPPPPP...............SSS.............
.......PPPPPPPPPP.............SS................
......PPPPP.PPPPP...........SS..................
......PPPPP.PPPPP........SSS....................
......PPPPP..PPPP......SSS......................
......PPPP...PPPP......S........................
................................................

The person’s mask fills 1,196 of the 2,016 pixels in its box, close to the polygon’s exact area of 1,191. The diagonal skis show why the choice of area matters: their box covers 1,476 pixels, which would make them medium, but their mask only 180, so COCO evaluation counts them as small. The COCO API rasterizes polygons with its own C routine, which may differ from this pixel-center test along the boundary.

Splits

The 2014 release has 82,783 training, 40,504 validation and 40,775 test images (Lin et al., 2015); a 2015 test set of about 81K images followed. In 2017, following community feedback, the same images were reorganized into 118K training (train2017) and 5K validation (val2017) images with unchanged detection and keypoint annotations, plus about 41K test images split into test-dev and test-challenge of about 20K each, and 123K unlabeled images (download and guidelines pages). The authors removed near-duplicates across splits and grouped images by photographer and date (Lin et al., 2015).

Tasks

Each annotation type supports a task with its own challenge track: object detection with box output, instance segmentation with mask output, keypoint detection (people must be found too, as no boxes are given at test time), stuff segmentation (challenges in 2017–2019), panoptic segmentation, image captioning, and DensePose.

Evaluation

Test annotations are withheld. Test-dev results come from an evaluation server (five submissions per day), and the guidelines ask papers to report them for fair comparison. Detection is scored by twelve numbers: AP averaged over IoU thresholds 0.50:0.05:0.95 (the primary metric, often written AP@[.5:.95]), AP at IoU 0.50 and 0.75, AP for small, medium and large objects, and average recall at 1, 10 and 100 detections per image and at the three scales. Box and mask evaluation differ only in computing IoU on boxes or masks; the average precision article explains the computation. Keypoints replace IoU with an object keypoint similarity; panoptic segmentation uses panoptic quality (PQ), stuff segmentation mean IoU, and captions metrics such as BLEU and CIDEr, with human judges in the 2015 challenge.

License

According to the terms of use (as of October 2026), the annotations and website belong to the COCO Consortium and are licensed under a Creative Commons Attribution 4.0 License. The consortium does not own the images’ copyright: their use must abide by the Flickr Terms of Use, and users accept full responsibility for it. Each image record carries a license id and its Flickr URL. The COCO API code has a BSD-style license.

Limitations and Bias

• A small, frequent vocabulary. The 80 entry-level categories exclude parts, fine-grained classes and rare objects, and are unbalanced: person accounts for 30% of COCO, according to Gupta, Dollár and Girshick (2019), who built LVIS to study the long tail.
• Coarse masks. Against outlines by experts on 100 instances, Gupta et al. (2019) measured a mean mask IoU of 0.83–0.87 (95% interval) for COCO and 0.90–0.92 for LVIS, and argue that coarse masks limit how finely mask quality can be measured.
• Small objects. About 41% of objects are small, 34% medium and 24% large (detection evaluation page).
• Image source. The images come from Flickr searches and so, we infer (the COCO papers do not measure it), reflect what Flickr users photographed and tagged, not a balanced sample of the world.
• Test-set reuse. Papers have reported on the same test-dev images for years; the organizers state that they saw no evidence of overfitting to specific test splits.

Common Uses

• Benchmarking detectors and instance segmenters, reporting COCO AP on test-dev or val2017.
• Captioning and vision–language research, using the captions.
• Human pose and dense pose estimation, using the keypoint and DensePose annotations.

Historical Milestones

• 2014. The ECCV paper and the first release. A deformable part model scored nearly half as well on COCO as on PASCAL VOC, which the authors took as evidence of harder, non-iconic images.
• 2015. The captioning challenge (CVPR 2015) and the first detection challenge (ICCV 2015, jointly with ImageNet), won in detection and segmentation by MSRA with ResNet. He et al. (2016) reported that replacing VGG-16 with ResNet-101 in Faster R-CNN raised AP@[.5:.95] by 6.0 points, a 28% relative gain.
• 2016. The first keypoint challenge.
• 2017. The 2017 splits, unlabeled images, and the stuff challenge.
• 2018. Panoptic and DensePose challenges. From 2018 the detection challenge featured only mask output, to encourage instance segmentation.
• 2020. The most recent COCO challenges listed on the website, at ECCV 2020 jointly with LVIS; for 2021 the website pointed participants to the LVIS challenge instead.

Related Datasets

• PASCAL VOC: the earlier detection benchmark whose 20 classes COCO includes; COCO reports $\mathrm{AP}_{50}$ as its VOC-style metric.
• LVIS (Gupta et al., 2019): new instance masks on the same COCO 2017 images for over 1,000 categories, with COCO-style AP.
• Objects365 (Shao et al., 2019): 365 categories in over 600,000 training images, annotated with more than 10 million boxes.
• Open Images (Kuznetsova et al., 2020): in version 4, 15.4 million boxes for 600 classes on 1.9 million images.
• ImageNet: the classification benchmark whose trained networks commonly initialize detector backbones.

Resources

• cocodataset.org: downloads, data format, task and evaluation pages, and the terms of use.
• COCO API: Python, MATLAB and Lua tools for loading annotations, decoding masks and running the official evaluation.
• Detection leaderboard and the other tasks’ leaderboards, for current results.
• COCO-Stuff, LVIS and DensePose project pages.