ADE20K

ADE20K is a dataset of scene photographs in which every pixel belongs to a named object or stuff region, and many objects are further divided into parts and parts of parts. It was built at MIT CSAIL by Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler (University of Toronto), Adela Barriuso and Antonio Torralba. The first release, ADE20K_2016_07_26, was public by September 2016; the dataset was described at CVPR 2017 (Zhou et al., 2017) and in more detail in the International Journal of Computer Vision (Zhou et al., 2019). Its annotation was done by one expert annotator with an open vocabulary, which grew to thousands of classes. From it the authors derived SceneParse150, a semantic segmentation benchmark of 150 classes scored by pixel accuracy and mean intersection over union, first used in the Scene Parsing Challenge 2016. SceneParse150 is what most papers mean when they report “ADE20K mIoU”.

Purpose

Zhou et al. (2017) wanted images annotated densely, with every pixel labeled, over an unrestricted vocabulary and a wide range of scenes. They judged that existing datasets fell short in one way or another: PASCAL VOC and COCO label a fixed set of objects, often not the ones most common in everyday scenes; Cityscapes covers only street scenes; Pascal-Context labels whole images but still centers on the 20 VOC classes; and the SUN database has noisy object labels. The intended use was scene parsing, recognizing and segmenting both stuff (floor, wall, sky) and discrete objects, and going further to the parts a robot would need to act on, such as a chair’s seat or a cup’s handle.

Dataset Size

Numbers differ by release and by source:

• 2016 release, as published. The papers give 20,210 training, 2,000 validation and 3,000 test images (Zhou et al., 2017; 2019). Their dataset table counts the 22,210 training and validation images, with 434,826 object instances in 2,693 object and stuff classes and 175,961 part instances in 476 part classes. The IJCV version gives 3,169 class labels in total and describes the dataset as 25k images (Zhou et al., 2019).
• Per image. Every image contains at least 5 objects; the most crowded has 273 object instances, or 419 counting parts. The text of both papers gives 19.5 instances and 10.5 classes per image on average, while their dataset table gives 9.9 classes per image and the per-image histogram in both versions reads 19.6 instances and 9.9 classes.
• Image size. The archived official page gave a median of 307,200 pixels and an average of 1.3 megapixels for the training images, with a median aspect ratio of 4/3.
• Current release. As of October 2026, the official page lists 25,574 training and 2,000 validation images, and the test set as “to be released later”. The ADE20K repository describes the current version, stored as ADE20K_2021_17_01, as 27,574 images from 365 scenes with 707,868 objects from 3,688 categories and 193,238 object parts and parts of parts. Its README calls the 2,000 images “for testing”, where the official page calls them validation images.

The papers report how the vocabulary grew during annotation: the number of classes kept rising with the number of labeled instances, and at the time of writing a new object class appeared about once every 300 segmented instances (Zhou et al., 2017). Object frequencies follow a roughly Zipfian distribution, as is typical when every object in an image is labeled, rather than the more uniform distribution of datasets balanced by design.

Classes

The full dataset has no fixed class list. The annotator kept a dictionary of visual concepts and added classes as new ones appeared, to keep names consistent across images. Object names were mapped to WordNet synsets, and the official files give each class its WordNet hypernyms and definition (archived official page). The papers distinguish three kinds of concept: discrete objects with a well-defined shape (car, person), stuff regions without one (grass, sky), and parts, components of an object with a functional meaning (head, leg) (Zhou et al., 2019).

The same name can be an object in one image and a part in another: a door is an object indoors and a part of a car or a building. The official page offers a per-class proportion of how often a class behaves as a part, to decide which role dominates.

SceneParse150. The benchmark keeps the 150 classes with the largest total pixel ratio: 35 stuff classes and 115 discrete objects. Together they cover 92.75% of all annotated pixels, 60.92% for stuff and 31.83% for objects (Zhou et al., 2017). The devkit’s class table shows how skewed the pixel distribution is: wall alone covers 15.76% of pixels and wall, building, sky and floor together 41.47%, while the smallest classes (glass, clock, flag) cover 0.02% each. Every other pixel, objects outside the 150 and unlabeled regions, gets label 0, “other objects”, which the evaluation ignores.

Annotation Types

Polygons from one expert. All images were annotated by a single expert worker in the LabelMe interface, with three kinds of annotation: segmented objects with names, object parts, and attributes (Zhou et al., 2017). Each instance is segmented separately and carries flags for whether it is occluded or cropped, so the data also serve instance-level tasks. Images were drawn from LabelMe, the SUN database and Places and chosen to cover the 900 scene categories of SUN.

Parts of parts. Parts are attached to object instances, and parts can have parts: a rim is part of a wheel, which is part of a car; a knob is part of a door that may be part of a cabinet. The hierarchy has depth 3. The IJCV version reports that 76% of object instances have parts, with 3 parts per object on average, that building has the most part classes (79), and that parts cover about 10% of the pixels. Part annotation is exhaustive on the validation images but not on the training images (Zhou et al., 2017).

From polygons to masks. Objects are outlined in full, even where occluded, so polygons overlap. The IJCV version describes how they are flattened: background classes such as sky and wall form the farthest layer; a polygon fully inside another is placed in front of it; for partial overlaps, the object whose polygon has more vertices in the shared region is placed in front; and masks are drawn from far to near so that parts never hide whole objects and no object is hidden by its own parts.

Files. In the 2016 release, each image had a _seg.png whose red and green channels encode the object class and blue channel the instance, one _seg_parts_N.png per level $N$ of the part hierarchy in the same encoding, and a text file listing every instance with its part level, occlusion flag, WordNet-parsed class name, raw name and attributes (archived official page). The 2021 release adds the original polygons, annotation times, depth ordering and an amodal mask for each instance, showing its full extent behind occluders (ADE20K repository README).

Consistency. Because one person labeled everything, the authors measured how consistent that person was. They had the annotator relabel 61 randomly chosen validation images six months later; 82.4% of pixels received the same label. The other 17.6% split into segmentation quality (5.7% of pixels), naming differences such as car versus truck or palm tree versus tree (6.0%), and objects missing from one of the two annotations (5.9%; the IJCV text says “of the error pixels”, but the three shares add up to the 17.6%); the medians were 4.8%, 0.3% and 2.6%, so a few images dominate the means (Zhou et al., 2017). Two invited external annotators with labeling experience, given 20 validation images, disagreed with the expert on 58.5% and 75% of pixels. For the better of the two, 7.9% of pixels had inconsistent segmentation, 14.9% inconsistent naming and 35.8% belonged to objects the external annotator had not labeled; the external annotators drew 16 segments per image on average against the expert’s 29. The official page’s consistency set holds 64 images, not 61, and the archived page described a 64-image study with 82% agreement.

Splits

SceneParse150 uses the 2016 split: 20,210 training and 2,000 validation images with public annotations, and a test set whose annotations were never released. The papers and the PSPNet paper (Zhao et al., 2017) give 3,000 test images; the devkit README, written before the test images were released, gives 2,000. The test images themselves are linked from the benchmark page. The papers rescaled large benchmark images to a shorter side of 512 pixels; the devkit’s convertFromADE.m does this for label maps whose shorter side exceeds 512, with nearest-neighbor interpolation, and leaves square maps unchanged.

Tasks

• Scene parsing (SceneParse150). A class from the 150 for every pixel.
• Instance segmentation (InstSeg100). Introduced in the IJCV version for the 2017 challenge: masks for instances of 100 foreground object classes, with 218K instances in total, about 10 per image, and every class except ship having more than 100 instances; it is scored with COCO-style mask average precision, overall and for small, medium and large objects (Zhou et al., 2019).
• Part segmentation. The CVPR paper segmented 36 part classes of eight objects (person, building, car, chair, table, sofa, bed, lamp) and scored them by pixel accuracy rather than IoU, because part annotations are not exhaustive; the IJCV version trained objects and parts jointly over 59 of the 150 classes and 153 part classes.
• Panoptic segmentation. Later work uses ADE20K’s 150 classes split into 100 things and 50 stuff classes for panoptic segmentation (Cheng et al., 2022).

Evaluation

The scene parsing benchmark ranks entries by the average of two numbers, both computed over the test set:

$$\text{score} = \frac{\text{pixel accuracy} + \text{mIoU}}{2}.$$

The authors reasoned that pixel accuracy is dominated by a few large stuff classes such as wall and floor, while mean IoU reflects how well each class, including small discrete objects, is segmented (Zhou et al., 2017). Later papers, PSPNet and Mask2Former among them, mostly report mIoU on the validation set, single-scale or multi-scale, rather than the challenge score.

The definitions of pixel accuracy and IoU are standard; what decides the numbers is the protocol, which the official MATLAB and Python code in the development kit fixes (devkit at commit e202229, demoEvaluation.m and evaluationCode/):

• Label 0 is ignored on both sides. Before counting, the code sets the prediction to 0 wherever the ground truth is 0, then counts intersections and areas only for labels 1–150. A prediction on an “other” pixel is therefore neither a true positive nor a false positive of any class. Pixel accuracy divides the correctly labeled pixels by the pixels whose ground truth is nonzero.
• IoU is accumulated over the dataset. For each class, intersection and union areas are counted per image, summed over all images, and divided once; mIoU is the mean of these 150 ratios. Pixel accuracy is likewise pooled: correct pixels over labeled pixels, summed over the whole set. Neither is an average of per-image scores.
• Absent classes count as zero. The denominator carries a machine epsilon, so a class with no ground-truth and no predicted pixels anywhere scores 0 and still enters the mean over 150. On the full validation set this never happens, since the devkit’s class table lists every class as present there; it matters when someone scores a subset.
• Predictions at annotation size. Predictions must be single-channel label maps of the annotation’s size. The MATLAB demo resizes a mismatched prediction with imresize at its default interpolation, which in our reading of the code mixes label ids along boundaries, so resizing should be done before scoring, with nearest-neighbor interpolation.

The authors’ PyTorch code (semantic-segmentation-pytorch at commit 8f27c9b), which reimplements PSPNet and UPerNet, computes the same quantities in another indexing: labels are shifted down by one so that “other” becomes −1 and the classes 0–149; pixel accuracy counts only labels of at least 0, and the per-image accuracies are averaged with weights equal to their labeled-pixel counts, which reproduces the pooled ratio; intersections and unions are summed over images before dividing. MMSegmentation, a widely used segmentation toolbox, maps label 0 to its ignore value 255 and shifts the rest down by one (reduce_zero_label), and also accumulates over the dataset, but it averages the per-class IoUs with nanmean, so a class with zero union is skipped rather than counted as 0 (MMSegmentation at commit b040e14). The two conventions agree whenever every class occurs, as on the validation set.

The test set was scored by an evaluation server that limited each participant to two submissions per week (benchmark page). When checked in October 2026, the server page displayed its unexecuted PHP source instead of a working page.

Example: what the official counting does

The code below renders four synthetic scenes, two rooms and two streets, with a toy vocabulary of eight classes standing in for the 150 and label 0 for an object outside the vocabulary. Each prediction carries known mistakes, among them a floor boundary three rows too low, a missed painting, a trimmed car, a small distant car that is missed entirely; in the “other” regions the predictions say floor and building, as a SceneParse150 model must, since it never outputs 0. The function inter_union follows the devkit’s per-image counting, and official accumulates over images as demoEvaluation.m does.

import numpy as np

# A toy vocabulary standing in for SceneParse150's classes 1..150; 0 = "other" (not evaluated).
NAMES = ["other", "wall", "floor", "painting", "sky", "building", "tree", "road", "car"]
K = len(NAMES) - 1
OTHER, WALL, FLOOR, PAINTING, SKY, BUILDING, TREE, ROAD, CAR = range(len(NAMES))
H, W = 48, 64
yy, xx = np.mgrid[:H, :W]

def room(horizon, painting, clutter=None):
    lab = np.full((H, W), WALL, np.uint8)
    lab[horizon:] = FLOOR
    r0, r1, c0, c1 = painting
    lab[r0:r1, c0:c1] = PAINTING
    if clutter is not None:                       # an object outside the vocabulary -> 0
        r0, r1, c0, c1 = clutter
        lab[r0:r1, c0:c1] = OTHER
    return lab

def street(skyline, tree, cars, sign=None):
    lab = np.full((H, W), SKY, np.uint8)
    lab[skyline:30] = BUILDING
    lab[30:] = ROAD
    cy, cx, ry, rx = tree
    lab[((yy - cy) / ry) ** 2 + ((xx - cx) / rx) ** 2 <= 1] = TREE
    for r0, r1, c0, c1 in cars:
        lab[r0:r1, c0:c1] = CAR
    if sign is not None:
        r0, r1, c0, c1 = sign
        lab[r0:r1, c0:c1] = OTHER
    return lab

# Ground truth: two rooms and two streets, rendered with exactly known regions.
gts = [room(30, (8, 18, 20, 36), clutter=(34, 44, 40, 58)),
       room(32, (10, 16, 8, 16)),
       street(12, (22, 52, 9, 8), [(32, 42, 6, 28)], sign=(14, 22, 30, 34)),
       street(16, (20, 10, 10, 9), [(31, 34, 40, 45)])]
# Predictions with known mistakes. A model trained on SceneParse150 never outputs 0.
preds = [room(33, (8, 18, 20, 36)),                       # horizon 3 rows low; clutter called floor
         room(32, (0, 0, 0, 0)),                          # painting missed (labelled wall)
         street(12, (22, 50, 9, 9), [(33, 42, 6, 26)]),   # car slightly trimmed; sign called building
         street(15, (20, 10, 11, 9), [])]                 # small distant car missed
preds[2][14:22, 30:34] = BUILDING

def inter_union(pred, lab, k=K):
    """Official SceneParse150 counting for one image: classes 1..k, label 0 ignored."""
    pred = pred * (lab > 0)                       # predictions on 'other' pixels are zeroed
    inter = np.bincount((pred * (pred == lab)).ravel(), minlength=k + 1)[1:]
    area_p = np.bincount(pred.ravel(), minlength=k + 1)[1:]
    area_l = np.bincount(lab.ravel(), minlength=k + 1)[1:]
    return inter, area_p + area_l - inter

def official(preds, gts):
    """Accumulate over the dataset, then divide (sceneparsing devkit)."""
    I, U = map(sum, zip(*(inter_union(p, g) for p, g in zip(preds, gts))))
    iou = I / (U + np.spacing(1))                 # a class absent everywhere scores 0
    correct = sum(np.sum((p == g) & (g > 0)) for p, g in zip(preds, gts))
    labelled = sum(np.sum(g > 0) for g in gts)
    acc, miou = correct / labelled, iou.mean()
    return acc, miou, (acc + miou) / 2, iou

acc, miou, score, iou = official(preds, gts)
print("per-class IoU:", {n: round(float(v), 3) for n, v in zip(NAMES[1:], iou)})
print(f"official    pixel acc {acc:.4f}  mIoU {miou:.4f}  score {score:.4f}")

# 1. Per-image averaging: mIoU of each image over the classes it contains, then the mean.
per_img = []
for p, g in zip(preds, gts):
    i, u = inter_union(p, g)
    per_img.append(np.mean(i[u > 0] / u[u > 0]))
print("per-image mIoU:", np.round(per_img, 4), f" mean {np.mean(per_img):.4f}")

# 2. Not ignoring 0: treat 'other' as a ninth class that the model never predicts.
gts9 = [np.where(g == 0, K + 1, g) for g in gts]
I9, U9 = map(sum, zip(*(inter_union(p, g, K + 1) for p, g in zip(preds, gts9))))
acc9 = sum(np.sum(p == g) for p, g in zip(preds, gts9)) / sum(g.size for g in gts9)
print(f"0 as a class pixel acc {acc9:.4f}  mIoU over 9 {np.mean(I9 / U9):.4f}"
      f"  floor IoU {I9[FLOOR - 1] / U9[FLOOR - 1]:.3f} (ignored: {iou[FLOOR - 1]:.3f})")

# 3. Scoring a subset (the two rooms): absent classes count as 0 in the devkit's mean.
a, m, s, iou_rooms = official(preds[:2], gts[:2])
present = iou_rooms[[WALL - 1, FLOOR - 1, PAINTING - 1]]
print(f"rooms only  devkit mIoU {m:.4f} (8 classes)   mean over present classes {present.mean():.4f}")

Output:

per-class IoU: {'wall': 0.94, 'floor': 0.904, 'painting': 0.769, 'sky': 0.964, 'building': 0.927, 'tree': 0.825, 'road': 0.968, 'car': 0.766}
official    pixel acc 0.9632  mIoU 0.8828  score 0.9230
per-image mIoU: [0.9014 0.6589 0.8873 0.7487]  mean 0.7991
0 as a class pixel acc 0.9466  mIoU over 9 0.7745  floor IoU 0.829 (ignored: 0.904)
rooms only  devkit mIoU 0.3266 (8 classes)   mean over present classes 0.8710

We checked the reimplementation by running the official Python functions on the same maps: intersectionAndUnion and pixelAccuracy from the devkit’s utils_eval.py, accumulated as its comments prescribe, and accuracy and intersectionAndUnion from semantic-segmentation-pytorch with its shifted labels; both gave the same per-class IoUs, pixel accuracy 0.9632 and mIoU 0.8828.

Averaging per image costs more than eight points of mIoU. A missed painting or a missed distant car is a class with IoU 0 in an image that has only three to five classes, so it pulls that image’s mean down hard, and each image weighs the same; pooled over the dataset, the painting’s missed pixels are set against the correct painting in the other room, and the small car against the large one. The dataset-level number therefore says how well each class is segmented across the collection, and the per-image mean how well a typical image is parsed; ADE20K reports the former. Treating “other” as a class does two things: the model’s floor and building labels on the unlabeled clutter and sign become false positives (floor IoU falls from 0.904 to 0.829), and the never-predicted ninth class adds a zero to the mean. The last line shows the subset trap: scored on the two rooms alone, the devkit’s mean still divides by all eight classes, and the five street classes contribute zeros.

License

ADE20K’s Terms of Use (as of October 2026) treat images and annotations separately:

• Images. The page states that MIT CSAIL does not own the copyright of the images. Researchers and educators may request access for non-commercial research and educational use through a request on the site. Use is limited to non-commercial research and education; the researcher takes full responsibility for their use, including any copies of copyrighted images, and indemnifies MIT; associates may be given access only if they agree to the same terms; MIT may end access at any time; and a researcher employed by a commercial entity binds the employer to the terms. The images come from the SUN and Places databases (official page) and, according to the papers, LabelMe.
• Annotations and software. The website, the image annotations and the software are said to belong to MIT CSAIL and to be licensed under a “Creative Commons BSD-3 License Agreement”, with a link to the BSD 3-Clause license and its text reproduced below (copyright 2019 MIT CSAIL). BSD-3-Clause is not a Creative Commons license, so the label is unusual, but the reproduced text is the BSD 3-Clause license.

The scene parsing development kit is under the BSD 3-Clause license (devkit LICENSE). The 2016 release was a direct download from the official page; as of October 2026 the full dataset requires registration, while the benchmark page links the SceneParse150 training, validation and test images directly; its “License” section links only the development kit’s BSD 3-Clause license and does not say whether that covers the images.

Limitations and Bias

• One annotator. A single expert gives consistent names and detailed boundaries but builds one person’s labeling habits into the whole dataset, and even that person matched their own earlier labels on only 82.4% of pixels after six months (Zhou et al., 2017). The IJCV version treats this figure, as pixel accuracy, as an approximate upper bound on model performance (Zhou et al., 2019).
• A long tail inside the benchmark. Four stuff classes cover over 40% of all annotated pixels and the rarest of the 150 classes 0.02% each. The authors found the best-segmented classes to be stuff such as sky, building and road, and the worst small, rarely seen objects such as blanket, tray and glass (Zhou et al., 2019).
• Layout priors. The archived official page listed under “Dataset bias” that the per-pixel mode of the object segmentations is sky, wall, building and floor from top to bottom, and that labeling every validation image with this mode gets 20.3% of its pixels right; both papers give 20.9% per image on average, and the IJCV version 20.30% on the test set.
• Partial part labels. Parts are exhaustively labeled only on the validation images, which is why the CVPR paper scored part segmentation by accuracy rather than IoU.
• Validation and test differ. PSPNet’s challenge scores on the test set, 57.21% for its ensemble and 55.38% for a single model, were lower than on the validation set, which its authors attributed possibly to a difference between the data distributions of the validation and test sets (Zhao et al., 2017).
• People. The ADE20K repository states that the images in the current release are anonymized by blurring faces and license plates (README, as of October 2026); the papers and the official page say nothing further about consent or demographics. Dataset Bias covers these questions in general.

Common Uses

• Benchmarking semantic segmentation, by mIoU on the SceneParse150 validation set, often next to Cityscapes and Pascal-Context.
• Evaluating backbones and pretraining. Self-supervised and transformer backbones are compared by fine-tuning a segmentation head on ADE20K, for example masked autoencoders and Vision Transformers with UPerNet.
• Panoptic and instance segmentation, with the things and stuff split and the InstSeg100 classes.
• Semantic image synthesis, generating images from ADE20K label maps (image-to-image translation).
• Part and hierarchical recognition, using the part hierarchy and the WordNet tree the IJCV version builds over the 150 classes.

Historical Milestones

• 2016, baselines. On the validation set the CVPR paper reported mIoU 29.39% and pixel accuracy 71.32% for FCN-8s, 32.31% and 73.55% for DilatedNet, and 34.90% and 74.52% for its Cascade-DilatedNet, which segments stuff and objects in separate streams (Zhou et al., 2017).
• 2016, Scene Parsing Challenge. Held with ILSVRC 2016 in a provided-data track: only the challenge data could be used, with models pretrained on the ImageNet 1.2M images or Places365 allowed (challenge page). It drew 75 submissions from 22 teams. SenseCUSceneParsing, with PSPNet, won with a test score of 0.57205, ahead of Adelaide (0.56735) and 360+MCG-ICT-CAS_SP (0.55565); the DilatedNet and FCN-8s baselines scored 0.45665 and 0.44800 (2016 results page). PSPNet reached 44.94% mIoU and 81.69% pixel accuracy on the validation set with a ResNet-269 and multi-scale testing (Zhao et al., 2017).
• 2017, Places Challenge. Held jointly with the COCO challenge at ICCV 2017, with scene parsing and a new instance segmentation track. The scene parsing track received 27 submissions from 11 teams, by the IJCV paper’s count (the archived results page lists 26 entries from 10 teams); the winner, CASIA_IVA_JD, scored 0.5547, below the 2016 winner. In instance segmentation, Megvii (Face++) reached 0.2977 mask mAP against 0.2241 for the organizers’ Mask R-CNN baseline (Zhou et al., 2019).
• 2018, open reimplementations. The IJCV version released PyTorch reimplementations; UPerNet-101 reached 42.66% mIoU and 81.01% pixel accuracy on validation, with multi-scale testing. It also found that batch normalization statistics computed over too few images, two per GPU without synchronization, lowered single-scale validation mIoU from 41.26% to 34.03% (Zhou et al., 2019).
• 2022, mask transformers. Mask2Former with a Swin-L backbone pretrained on ImageNet-22K and a FaPN pixel decoder reported 56.4% mIoU single-scale and 57.7% multi-scale on the validation set (Cheng et al., 2022), nearly double the 2016 FCN baseline.

Related Datasets

• Pascal-Context (Mottaghi et al., 2014) labels every pixel of the 10,103 training and validation images of the PASCAL VOC 2010 detection challenge with 540 categories, by six in-house annotators who started from 80 chosen labels and added classes as needed; experiments use the 59 most frequent classes, with the rest as background.
• COCO-Stuff (Caesar, Uijlings & Ferrari, 2018) adds pixel-level labels for 91 stuff classes, curated by an expert annotator, to all 164K COCO 2017 images, alongside COCO’s 80 thing classes; stuff was labeled on superpixels to save time.
• Cityscapes covers urban street scenes from a car, with 19 evaluated classes and an instance-weighted IoU; its narrow domain was one of the gaps ADE20K was built to fill.
• The SUN database and LabelMe, which supplied ADE20K’s scene categories and many of its images, and the Places database, which supplied others.

Resources

• ADE20K official page: registration for the full dataset, terms of use and the consistency set.
• ADE20K repository: README, starter notebook and loading utilities for the current release.
• MIT Scene Parsing Benchmark: SceneParse150 data, the 2016 challenge pages and results, and the evaluation server with its leaderboard.
• Scene parsing development kit: class lists, conversion from ADE20K format and the official evaluation code in MATLAB and Python.
• semantic-segmentation-pytorch: the authors’ PyTorch training and evaluation code and pretrained models.