Cityscapes

Cityscapes is a dataset and benchmark suite of urban street scenes recorded from a car in 50 cities, introduced at CVPR 2016 by Marius Cordts and colleagues from Daimler AG R&D, TU Darmstadt, MPI Informatics and TU Dresden. Its 5,000 finely annotated frames, with 30 pixel-level classes and separate instances for people and vehicles, made it the standard benchmark for semantic segmentation of driving scenes; panoptic segmentation (2019) and 3D vehicle detection (2020) were added later.

Purpose

Driving datasets of the time, such as KITTI, CamVid and Daimler Urban Segmentation, each came from one city and had at most several hundred pixel-annotated frames. Cordts et al. (2016) argued that they were much smaller than general datasets like PASCAL VOC and COCO and missed the complexity of inner-city traffic.

Dataset Size

Several hundred thousand frames were recorded over several months, in spring, summer and fall, in 50 cities, “primarily in Germany but also in neighboring countries” (Cordts et al., 2016); the website’s terms of use name German, French and Swiss cities. The authors deliberately avoided heavy rain and snow; the website lists daytime, good or medium weather.

The camera was a stereo rig behind the windshield, with a 22 cm baseline and two 2-megapixel rolling-shutter CMOS sensors at 17 Hz. Images are 2048 × 1024 pixels (Chen et al., 2018; He et al., 2018), recorded as 16-bit linear high-dynamic-range images and also provided as 8-bit images after logarithmic tone mapping. Each annotated frame is the 20th of a provided 30-frame snippet (1.8 s). Extras include the right stereo views, precomputed disparity maps, camera calibration, vehicle odometry, GPS coordinates and outside temperature (cityscapesScripts README). The disparity maps were computed with semi-global matching (Gählert et al., 2020).

Classes

The 30 classes are grouped into eight categories: flat, construction, nature, vehicle, sky, object, human and void. Classes too rare for evaluation (parking, rail track, guard rail, bridge, tunnel, pole group, caravan, trailer) and the void classes (static, dynamic, ground) are ignored, which leaves 19 evaluated classes. Human and vehicle classes carry instance labels; 8 of them (person, rider, car, truck, bus, train, motorcycle, bicycle) are evaluated as instances.

Annotation files store a label id from 0 to 33 per pixel. The official labels.py also proposes a train id that numbers the 19 evaluated classes 0–18 and maps everything else to 255, the usual ignore value: road (id 7) becomes 0, car (id 26) becomes 13, and ego vehicle (id 1) and parking (id 9) become 255. The file calls train ids a suggestion and asks that results be submitted in the original ids.

Annotation Types

• Fine annotations. 5,000 frames, chosen by hand from 27 cities for varied objects and layouts, were labeled in-house with layered polygons. Annotators worked back to front, so each boundary is drawn once and the layers imply a depth order. Annotation and quality control took over 1.5 hours per image on average. Two annotators agreed on 96% of pixels in 30 doubly annotated images.
• Coarse annotations. 20,000 frames from the other 23 cities, one every 20 s or 20 m of driving, were labeled in under 7 minutes each with rough single-class polygons that leave boundaries unlabeled. The released train_extra set has 19,998 images.
• Instances. Each person and vehicle has its own mask, unless the boundary between instances cannot be seen; then the group is labeled together (for example, car group). Instance images encode a pixel as label id × 1000 plus an index.
• 3D boxes. Cityscapes 3D (Gählert et al., 2020) added 9-degree-of-freedom 3D boxes for vehicles, labeled from the stereo data alone and paired with the 2D instance masks.

Splits

The fine frames are split by city into 2,975 training, 500 validation and 1,525 test images. The splits balance city size, geographic position and time of year. Test annotations are withheld; coarse frames are extra training data only.

Tasks

• Pixel-level semantic labeling: a class for every pixel.
• Instance-level semantic labeling: a scored mask for each person and vehicle.
• Panoptic semantic labeling (added in 2019): a class for every pixel and an instance id for things, following Kirillov et al. (2019).
• 3D vehicle detection (added in 2020): 2D and 3D boxes for cars, trucks, buses, trains, motorcycles and bicycles.

The stereo data also serve stereo matching and depth estimation research, without an official benchmark.

Evaluation

All tasks are scored on the test set by an evaluation server. Pixel-level labeling uses the per-class intersection over union, $\mathrm{IoU} = \mathrm{TP} / (\mathrm{TP} + \mathrm{FP} + \mathrm{FN})$, with pixels counted over the whole test set and void pixels excluded. The main score, IoU class, is the mean over the 19 classes; IoU category averages over the 7 non-void categories, so confusing a bus with a truck costs nothing.

Because IoU counts pixels, one large nearby car outweighs many small distant ones, so Cordts et al. (2016) added an instance-level IoU,

$$\mathrm{iIoU} = \frac{\mathrm{iTP}}{\mathrm{iTP} + \mathrm{FP} + \mathrm{iFN}},$$

in which every true-positive and false-negative pixel of a ground-truth instance is weighted by the class’s average instance size divided by that instance’s size. False positives stay unweighted, since a per-pixel labeling assigns them to no instance. The official script fixes the averages (12,794 pixels for car) and, in our reading of its code, leaves pixels of group regions, which have no instance id, out of iTP and iFN. iIoU class and iIoU category average over the 8 instance classes and the 2 instance categories.

The instance-level task uses mask average precision, averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05 and over the 8 classes, following COCO; duplicate detections count as false positives. Secondary scores are AP at 50% overlap and AP within 100 m and 50 m. The panoptic task uses panoptic quality (PQ); the 3D task multiplies 2D AP by the mean of location, orientation and size scores.

Example: IoU and iIoU when distant cars are missed

The code builds an invented street scene in Cityscapes label ids (not Cityscapes data), with an ignored parking strip, a void hood, one near car of $30 \times 60$ pixels and four distant cars of 15 to 24 pixels, maps it to train ids, and scores two simulated predictions of the car class: A misses the distant cars, B misses as many pixels from the near car. The average instance size comes from this one image, standing in for the dataset-wide value.

import numpy as np

# A few rows of cityscapesScripts' labels.py: name -> (label id, train id). 255 = ignored.
LABELS = {"ego vehicle": (1, 255), "road": (7, 0), "sidewalk": (8, 1), "parking": (9, 255),
          "building": (11, 2), "sky": (23, 10), "car": (26, 13)}
lut = np.full(256, 255, np.uint8)                 # every id not listed maps to 255 (ignore)
for name, (lid, tid) in LABELS.items():
    lut[lid] = tid
ROAD, SIDEWALK, BUILDING, SKY, CAR = (LABELS[n][1] for n in ("road", "sidewalk", "building", "sky", "car"))

# Invented 96 x 192 ground truth in Cityscapes label ids (not real Cityscapes data).
H, W = 96, 192
ids = np.full((H, W), LABELS["sky"][0], np.uint8)
ids[20:50, :] = LABELS["building"][0]
ids[50:, :] = LABELS["road"][0]
ids[50:58, :] = LABELS["sidewalk"][0]
ids[58:64, 150:192] = LABELS["parking"][0]         # annotated, but ignored in evaluation
ids[88:, :] = LABELS["ego vehicle"][0]             # the car's own hood: void
inst = np.zeros((H, W), np.int32)                  # instance ids: label id * 1000 + index
cars = [(56, 86, 10, 70),                          # one near car, 30 x 60 px
        (48, 52, 100, 106), (48, 52, 120, 125), (49, 53, 140, 146), (48, 51, 160, 165)]  # 4 distant cars
for k, (r0, r1, c0, c1) in enumerate(cars):
    ids[r0:r1, c0:c1] = LABELS["car"][0]
    inst[r0:r1, c0:c1] = LABELS["car"][0] * 1000 + k
gt = lut[ids]                                      # train ids, as used for training

def scores(gt, inst, pred, cls, avg_size):
    """Class IoU and iIoU as defined by Cordts et al. (2016), for one class."""
    valid = gt != 255                              # void pixels never count
    tp = np.sum(valid & (gt == cls) & (pred == cls))
    fn = np.sum(valid & (gt == cls) & (pred != cls))
    fp = np.sum(valid & (gt != cls) & (pred == cls))
    itp = ifn = 0.0
    for i in np.unique(inst[inst > 0]):
        m = inst == i
        w = avg_size / m.sum()                     # small instances get large weights
        itp += w * np.sum(pred[m] == cls)
        ifn += w * np.sum(pred[m] != cls)
    return tp / (tp + fp + fn), itp / (itp + fp + ifn)

sizes = [int(np.sum(inst == i)) for i in np.unique(inst[inst > 0])]
avg = np.mean(sizes)    # stand-in for the class's average instance size over the dataset
print("car instance sizes (px):", sizes, " average:", round(avg, 1))
print("pixels per train id:", {int(k): int(v) for k, v in zip(*np.unique(gt, return_counts=True))})

# Prediction A: perfect, except that all four distant cars are labeled road or building.
a = gt.copy()
for r0, r1, c0, c1 in cars[1:]:
    a[r0:r1, c0:c1] = np.where(np.arange(r0, r1)[:, None] < 50, BUILDING, ROAD)
# Prediction B: the same number of car pixels missed, all from the near car's top rows.
missed = sum((r1 - r0) * (c1 - c0) for r0, r1, c0, c1 in cars[1:])
b = gt.copy()
r0, r1, c0, c1 = cars[0]
rows = missed // (c1 - c0)
b[r0:r0 + rows, c0:c1] = SIDEWALK
b[r0 + rows, c0:c0 + missed - rows * (c1 - c0)] = SIDEWALK
for name, p in [("perfect", gt), ("A: distant cars missed", a), ("B: near car trimmed", b)]:
    iou, iiou = scores(gt, inst, p, CAR, avg)
    print(f"{name:<24} car pixels missed {np.sum((gt == CAR) & (p != CAR)):3d}   IoU {iou:.3f}   iIoU {iiou:.3f}")

Output:

car instance sizes (px): [1800, 24, 20, 24, 15]  average: 376.6
pixels per train id: {0: 3828, 1: 1371, 2: 5722, 10: 3840, 13: 1883, 255: 1788}
perfect                  car pixels missed   0   IoU 1.000   iIoU 1.000
A: distant cars missed   car pixels missed  83   IoU 0.956   iIoU 0.200
B: near car trimmed      car pixels missed  83   IoU 0.956   iIoU 0.991

Both predictions miss 83 of 1,883 car pixels and score IoU 0.956. In iIoU every instance carries the same total weight, 376.6, so in A the four lost cars outweigh the found one four to one and iIoU falls to 1/5 = 0.200; in B the near car loses under 5% of its weight and iIoU stays at 0.991. Cordts et al. (2016) saw the same effect at scale: ground truth subsampled by 2 and resampled to full resolution kept 97.2% class IoU but only 92.6% iIoU, and most methods that regularize with a CRF dropped more from IoU to iIoU than the others.

License

Cityscapes has no open license. Its terms and conditions (checked October 2026) make it free for non-commercial purposes such as research, teaching and personal experimentation, after free registration limited to people affiliated with a body pursuing scientific interests. The dataset and modified versions may not be redistributed; derived works such as trained models may be shared only if the data cannot be recovered from them, and commercial use of the dataset or derived works is excluded. Users must cite the dataset. Images published outside the portal must be anonymized; the portal offers an anonymized substitute. The cityscapesScripts code is under the MIT license.

Limitations and Bias

• Narrow conditions. Night, rain, snow and other continents are absent.
• Small fine-labeled training set. The 2,975 training images hold about 200 to 500 instances each of truck, bus and train; He et al. (2017) found a large gap between validation and test AP for these classes, which they took to suggest domain shift.
• Sensitivity to conditions. An FCN trained on the validation set scored 3.8 points higher on the end-of-year test subset than on the whole test set, which Cordts et al. (2016) attributed to softer light on cloudy fall days.

Common Uses

• Benchmarking semantic segmentation networks, reporting class IoU on the validation set or the test server.
• Instance and panoptic segmentation, alongside COCO.
• Training from coarse or weak labels, using the coarse set.
• Domain adaptation, with Cityscapes as the unlabeled target and synthetic sources such as 24,966 labeled frames from the game GTA V (Richter et al., 2016) or SYNTHIA (Ros et al., 2016). Hoffman et al. (2016), who call theirs the first domain-adaptive semantic segmentation method, evaluated GTA5 → Cityscapes and SYNTHIA → Cityscapes on Cityscapes classes.
• Transfer to other driving datasets. Without fine-tuning, an FCN trained on Cityscapes beat the best reported results on CamVid and one KITTI label set (Cordts et al., 2016).

Historical Milestones

• 2016. The best baseline in the original paper reached 67.1% test class IoU (Dilated10), and the best instance-level baseline 4.6% AP (Cordts et al., 2016).
• 2017. On the test set, PSPNet (ResNet-101) reported 78.4% class IoU from fine annotations and 80.2% adding the coarse set (Zhao et al., 2017); DeepLab (ResNet-101, CRF) reported 70.4% (Chen et al., 2018). Mask R-CNN (ResNet-50-FPN) reported 26.2% test AP from fine annotations and 32.0% with COCO pretraining, in the appendix of the paper’s arXiv version (He et al., 2018).
• 2019. Panoptic task added; Kirillov et al. (2019) measured a human consistency of 69.7 PQ on 30 doubly annotated images.
• 2020. Cityscapes 3D and the 3D vehicle detection task added.

Related Datasets

KITTI covers stereo, flow, odometry and detection around Karlsruhe; its segmentation benchmark follows the Cityscapes label format. Mapillary Vistas (Neuhold et al., 2017) has 25,000 street-level images from every continent except Antarctica, taken with many devices in varied weather, seasons and times of day, with 66 classes, 37 with instances. BDD100K is another widely used driving dataset; ADE20K covers general scene parsing.

Resources

• Cityscapes website: downloads (registration required), class definitions and the benchmark suite with current leaderboards.
• cityscapesScripts: label definitions (labels.py), conversion tools and the official evaluation code.