Semantic Segmentation

Semantic segmentation assigns every pixel of an image one label from a fixed list of classes: this pixel is road, that one sky, those belong to a car. The result is a label map the size of the image. It is the dense version of image classification: instead of one prediction per image, the model makes one per pixel. The task is also called scene parsing. The image segmentation hub places it among the other segmentation variants; this article covers the task, how methods for it developed, and how it is measured.

Problem Definition

Given an image on a pixel grid $\Omega$ and a set of $K$ classes, predict a label map $L : \Omega \to \{1, \dots, K\}$, usually as the argmax of $K$ per-pixel class scores. Datasets often add a void label for ambiguous or unlabeled pixels, which training and evaluation ignore.

The task does not separate objects: two touching cars form one “car” region, and a class region may consist of disconnected pieces. Classes are of two kinds. Stuff is amorphous and uncountable: sky, road, grass, wall. Things are countable objects: people, cars, chairs. Semantic segmentation treats both the same way. Giving each thing its own mask is instance segmentation; combining instance ids for things with class labels for stuff is panoptic segmentation.

Input and Output

The input is an image, sometimes with depth or neighboring video frames; the output is an $H \times W$ map of class indices, or the $K \times H \times W$ score volume behind it. Ground truth has the same form: PASCAL VOC stores label maps as PNG images, with a void band along object boundaries.

Per-pixel prediction exposes what Long, Shelhamer and Darrell (2015) call a tension between semantics and location: global information decides what a pixel is, local information decides where the boundary runs. Recognizing a pixel as part of a car takes a receptive field that covers much of the car. Classification networks build such fields by downsampling with strided convolutions and pooling; in the VGG-16 network that FCN adapted, the last layer has a stride of 32 pixels, and placing a boundary to the nearest pixel then takes extra machinery. Most architectures below answer this tension.

Variants

• Scene parsing. Labeling whole scenes with large vocabularies that mix stuff and things. ADE20K’s SceneParse150 benchmark uses the 150 classes that cover the most pixels: 35 stuff and 115 discrete objects (Zhou et al., 2017).
• Human parsing. Segmenting a person into body parts and clothing. The Look into Person (LIP) dataset has more than 50,000 images annotated with 19 part labels and a background label (Gong et al., 2017).
• Open-vocabulary segmentation. Segmenting classes named in text at test time, beyond the training label set, usually with vision-language models.
• Video, 3D and weak supervision. Labeling frames consistently over time or labeling point clouds, and training from tags, boxes or scribbles instead of costly full masks.

Challenges

• Class imbalance. Road, sky and building pixels outnumber poles and signs by orders of magnitude, in losses and in pixel-level scores alike.
• Small and thin objects. Poles, wires and distant pedestrians are only a few pixels wide and disappear at coarse output strides, as the example below shows.
• Boundaries. Errors pile up along object outlines, where downsampled features and upsampled scores are least precise.
• Context and confusion. Classes that look alike up close, such as chair and sofa or road and sidewalk, need context to tell apart. PSPNet’s authors show an FCN labeling a boat on ADE20K as a car, a mistake the scene context, a boathouse, rules out (Zhao et al., 2017).
• Domain shift. Models trained in one city, season or sensor degrade in another.
• Cost. Driving and robotics need dense predictions on large images in real time.

Approaches

Before deep learning: unaries and CRFs

Early systems scored each pixel or superpixel with a classifier and then smoothed the labels with a conditional random field (CRF). TextonBoost (Shotton et al., 2006) combined a boosted classifier on texton-based shape and texture features with color, location and edge terms in a CRF on the 4-connected pixel grid. On their 21-class database of 591 photographs (later known as MSRC-21), the authors reported 72.2% overall pixel accuracy, against 69.6% for the shape–texture classifier alone. Krähenbühl and Koltun (2011) connected every pair of pixels, with pairwise terms built from Gaussian kernels on position and color. Mean-field inference with high-dimensional filtering made this dense CRF fast, about 0.2 seconds per image in their experiments, and it outlived the hand-made unaries: early deep networks used it to sharpen their coarse outputs.

Fully convolutional networks

Long, Shelhamer and Darrell (2015) recast classification CNNs such as VGG-16 as fully convolutional networks (FCNs): the fully connected layers become convolutions, so the network maps an image of any size to a coarse map of class scores, which is upsampled to full resolution and trained end to end with per-pixel loss. Because the stride of 32 limited detail, they added skips that fuse scores from finer layers (FCN-16s, FCN-8s). FCN-8s reached 62.2% mean IoU on the PASCAL VOC 2012 test set, a 20% relative improvement on the previous best, with inference taking about 175 ms against roughly 50 s for SDS, the previous best.

Encoder–decoders

Encoder–decoders recover resolution step by step. SegNet (Badrinarayanan, Kendall & Cipolla, 2017) uses VGG-16’s 13 convolutional layers as an encoder; its decoder upsamples with the max-pooling indices stored by the encoder, so no upsampling needs to be learned, and the sparse maps are then densified by trainable convolutions. U-Net (Ronneberger, Fischer & Brox, 2015) concatenates encoder features with the matching decoder stage through skip connections; it was designed for biomedical images. Learned upsampling layers in such decoders are often transposed convolutions.

Dilated convolutions and multi-scale context

The other answer is not to downsample so far. Dilated (atrous) convolutions space a filter’s taps apart, so stacking them with dilations 1, 2, 4, … expands the receptive field exponentially without losing resolution (Yu & Koltun, 2016). DeepLab removes the downsampling from the backbone’s last few max-pooling layers and dilates the filters of the layers that follow, so features are computed at a higher sampling rate. Its journal version (Chen et al., 2018) adds atrous spatial pyramid pooling (ASPP), parallel atrous convolutions at several rates, and refines its bilinearly upsampled output with the dense CRF of Krähenbühl and Koltun. DeepLabv3+ (Chen et al., 2018, ECCV) adds a light decoder to sharpen boundaries and reports its results without CRF post-processing.

PSPNet (Zhao et al., 2017) gathers global context with a pyramid pooling module: it average-pools the final feature map into $1 \times 1$, $2 \times 2$, $3 \times 3$ and $6 \times 6$ bins, upsamples the pooled features and concatenates them with the original map before classifying.

Transformers

Transformers give every token a global receptive field from the first layer. SETR (Zheng et al., 2021) encodes an image as a sequence of $16 \times 16$ patches with a pure transformer encoder, initialized from ViT or DeiT, with no convolutions or downsampling, and adds simple decoders. SegFormer (Xie et al., 2021) pairs a hierarchical, multi-scale transformer encoder without positional encodings, whose interpolation hurt accuracy at new test resolutions, with a lightweight all-MLP decoder.

Mask classification

MaskFormer (Cheng, Schwing & Kirillov, 2021) predicts a set of binary masks, each with one class label, instead of classifying each pixel, using a transformer decoder as in DETR. Its authors found the gain over per-pixel baselines larger when the number of classes is large. Mask2Former (Cheng et al., 2022) restricts cross-attention to the predicted mask regions and serves semantic, instance and panoptic segmentation with one architecture.

Losses

The basic loss is cross-entropy averaged over the labeled pixels, with void pixels excluded. Against class imbalance the loss can be weighted per class; SegNet’s authors used median frequency balancing, taken from Eigen and Fergus, weighting each class by the median class frequency divided by its own. Other losses target overlap directly: the Dice loss (see loss function) maximizes a differentiable Dice coefficient. The Lovász-softmax loss (Berman, Rannen Triki & Blaschko, 2018) is a surrogate of the Jaccard (IoU) loss built from the convex Lovász extension of submodular set functions; the authors reported higher IoU than cross-entropy on PASCAL VOC and Cityscapes.

Example: what a coarse output stride costs

The NumPy code builds the ground-truth label map of a $64 \times 64$ street scene (sky, ground, an $18 \times 24$ car, and a pole 2 pixels wide and 35 tall) and scores two simulated predictions with the four metrics of Long et al. (2015). The first is an idealized model that predicts at 1/8 resolution: its scores are the fraction of each class in every $8 \times 8$ cell, upsampled by nearest-neighbor or bilinear interpolation (pixel-center aligned, as PyTorch’s align_corners=False). The second predicts at full resolution, with scores equal to the one-hot ground truth plus Gaussian noise.

import numpy as np
from scipy import ndimage

# Ground-truth label map of a 64 x 64 street scene: 0 sky, 1 ground, 2 car, 3 pole.
H = W = 64
names = ["sky", "ground", "car", "pole"]
K = len(names)
gt = np.zeros((H, W), int)
gt[40:, :] = 1                      # ground below the horizon
gt[29:47, 7:31] = 2                 # car, 18 x 24 px, overlapping the horizon
gt[6:41, 45:47] = 3                 # pole, 2 px wide, 35 px tall

def confusion(gt, pred, k):
    """n[i, j] = number of pixels of class i predicted as class j."""
    return np.bincount(k * gt.ravel() + pred.ravel(), minlength=k * k).reshape(k, k)

def metrics(n):
    """The four scores defined by Long, Shelhamer and Darrell (2015)."""
    t = n.sum(1)                                 # pixels of each true class
    tp = np.diag(n)
    iou = tp / (t + n.sum(0) - tp)
    return dict(pixel_acc=tp.sum() / t.sum(), mean_acc=np.mean(tp / t),
                iou=iou, miou=iou.mean(), fw_iou=(t * iou).sum() / t.sum())

def report(title, pred):
    m = metrics(confusion(gt, pred, K))
    print(f"{title:<22} pixel acc {m['pixel_acc']:.3f}  mean acc {m['mean_acc']:.3f}  "
          f"mIoU {m['miou']:.3f}  fwIoU {m['fw_iou']:.3f}")
    print(" " * 23 + "IoU " + ", ".join(f"{a} {b:.3f}" for a, b in zip(names, m["iou"])))

onehot = np.eye(K)[gt].transpose(2, 0, 1)        # K x H x W

# (a) A model that predicts at 1/8 resolution. Its best possible score map is the
#     fraction of each class inside every 8 x 8 cell; upsample it, then take the argmax.
s = 8
coarse = onehot.reshape(K, H // s, s, W // s, s).mean(axis=(2, 4))   # K x 8 x 8
nearest = coarse.repeat(s, 1).repeat(s, 2)
c = (np.arange(H) + 0.5) / s - 0.5                # pixel centers in coarse coordinates
yy, xx = np.meshgrid(c, c, indexing="ij")
bilinear = np.stack([ndimage.map_coordinates(ch, [yy, xx], order=1, mode="nearest")
                     for ch in coarse])
report("1/8, nearest", nearest.argmax(0))
report("1/8, bilinear", bilinear.argmax(0))
print("largest pole fraction in any 8x8 cell:", coarse[3].max())

# (b) A full-resolution model whose scores are noisy: one-hot plus Gaussian noise.
rng = np.random.default_rng(0)
noisy = onehot + rng.normal(0, 0.3, onehot.shape)
report("full res, noisy", noisy.argmax(0))
print("pixels per class:", dict(zip(names, np.bincount(gt.ravel()).tolist())))

Output:

1/8, nearest           pixel acc 0.952  mean acc 0.693  mIoU 0.659  fwIoU 0.909
                       IoU sky 0.934, ground 0.971, car 0.732, pole 0.000
1/8, bilinear          pixel acc 0.967  mean acc 0.716  mIoU 0.697  fwIoU 0.936
                       IoU sky 0.950, ground 0.988, car 0.853, pole 0.000
largest pole fraction in any 8x8 cell: 0.25
full res, noisy        pixel acc 0.978  mean acc 0.975  mIoU 0.877  fwIoU 0.960
                       IoU sky 0.972, ground 0.970, car 0.911, pole 0.654
pixels per class: {'sky': 2228, 'ground': 1366, 'car': 432, 'pole': 70}

The pole vanishes from both coarse predictions: a 2-pixel pole fills at most a quarter of any $8 \times 8$ cell, so sky or ground wins the argmax everywhere, before and after interpolation. Its IoU drops to zero, yet it holds only 70 of 4,096 pixels, so pixel accuracy stays at 0.95–0.97. Bilinear upsampling improves the car’s outline (IoU 0.732 to 0.853) but cannot restore a class that never wins a cell. The noisy full-resolution model has about the same pixel accuracy (0.978) but keeps the pole (IoU 0.654), and its mIoU is 0.877 against 0.697. Over noise seeds 0–9 its pole IoU ranged from 0.58 to 0.73, and seed 0 is typical. Frequency-weighted IoU barely registers the pole; mean accuracy and mIoU, which weight classes equally, expose it. FCN’s own numbers on a 736-image subset of VOC 2011 val show the same pattern: going from stride 32 to stride 8 (FCN-32s to FCN-8s) raised pixel accuracy by 1.2 points (89.1% to 90.3%) and mean IoU by 3.3 (59.4% to 62.7%).

In PyTorch the classifier is a $1 \times 1$ convolution on the feature map, followed by upsampling and per-pixel cross-entropy that skips void pixels (run with PyTorch 2.14.1):

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)
K, IGNORE = 4, 255
feats = torch.randn(2, 64, 8, 8)                  # backbone features at stride 8 for 64 x 64 images
labels = torch.randint(0, K, (2, 64, 64))         # per-pixel class indices
labels[:, :, 30:34] = IGNORE                      # a void band, as along VOC object boundaries

head = nn.Conv2d(64, K, kernel_size=1)            # a 1 x 1 convolution: one linear classifier per position
logits = head(feats)                                              # 2 x 4 x 8 x 8
logits = F.interpolate(logits, size=(64, 64), mode="bilinear", align_corners=False)
loss = F.cross_entropy(logits, labels, ignore_index=IGNORE)       # mean over labeled pixels

# The same loss by hand: per-pixel negative log-softmax, averaged over non-void pixels.
logp = F.log_softmax(logits.detach(), dim=1)
valid = labels != IGNORE
manual = -logp.permute(0, 2, 3, 1)[valid].gather(1, labels[valid][:, None]).mean()
print(f"logits {tuple(logits.shape)}, labeled pixels {int(valid.sum())} of {labels.numel()}")
print(f"cross-entropy {loss.item():.4f}, by hand {manual.item():.4f}, ln K = {torch.log(torch.tensor(K)).item():.4f}")

Output:

logits (2, 4, 64, 64), labeled pixels 7680 of 8192
cross-entropy 1.4375, by hand 1.4375, ln K = 1.3863

With random inputs the loss sits near $\ln K$, the value for uniform predictions; the 512 void pixels contribute nothing. The NumPy bilinear interpolation above agrees with F.interpolate(..., align_corners=False) to within $10^{-15}$.

Datasets and Benchmarks

• PASCAL VOC 2012 (Everingham et al., 2010): 20 object classes plus background, scored by mean IoU on a withheld test set. FCN and many later methods also trained on the extra annotations of the Semantic Boundaries Dataset.
• Cityscapes (Cordts et al., 2016): street scenes from 50 cities with 5,000 finely annotated images; of 30 annotated classes, 19 are used in the benchmark, which also reports an instance-weighted iIoU.
• ADE20K (Zhou et al., 2017): diverse scenes with 20,210 training and 2,000 validation images; SceneParse150 is its benchmark of 150 classes.
• COCO-Stuff (Caesar, Uijlings & Ferrari, 2018): 91 stuff classes added to the thing classes of COCO, on 164K images.

Reported results show the progression; backbones, pretraining data and test-time augmentation differ, so rows compare only roughly.

Method	Year	Backbone and pretraining	Benchmark	mIoU (%)
FCN-8s	2015	VGG-16, ImageNet	VOC 2012 test	62.2
DeepLab (journal)	2018	ResNet-101, plus COCO; multi-scale, dense CRF	VOC 2012 test	79.7
PSPNet	2017	ResNet-101; VOC data only / plus COCO	VOC 2012 test	82.6 / 85.4
DeepLabv3+	2018	Xception, best configuration; without / with JFT-300M	VOC 2012 test	87.8 / 89.0
FCN-8s (ADE20K baseline)	2017	VGG-16	ADE20K val	29.4
PSPNet	2017	ResNet-269, multi-scale	ADE20K val	44.94
SETR-MLA	2021	ViT-Large, ImageNet-21K; multi-scale	ADE20K val	50.28
SegFormer-B5	2021	MiT-B5, ImageNet-1K; single / multi-scale	ADE20K	51.0 / 51.8
MaskFormer	2021	Swin-L, ImageNet-22K; multi-scale	ADE20K val	55.6
Mask2Former	2022	Swin-L with FaPN, ImageNet-22K; multi-scale	ADE20K val	57.7

The ADE20K FCN-8s figure is the baseline reported by Zhou et al. (2017). DeepLabv3+’s test entries use its best model; the paper states their pretraining only for JFT, though it pretrains on COCO for its state-of-the-art comparisons. The SegFormer paper does not name its ADE20K split; it compares against SETR’s validation-set numbers.

Evaluation Metrics

With $n_{ij}$ the number of pixels of true class $i$ predicted as class $j$, $t_i = \sum_j n_{ij}$ the pixels of class $i$, and $K$ classes, Long et al. (2015) collected four scores from earlier segmentation and scene-parsing evaluations:

$$\text{pixel accuracy} = \frac{\sum_i n_{ii}}{\sum_i t_i}, \qquad \text{mean accuracy} = \frac{1}{K} \sum_i \frac{n_{ii}}{t_i},$$ $$\text{mean IoU} = \frac{1}{K} \sum_i \frac{n_{ii}}{t_i + \sum_j n_{ji} - n_{ii}}, \qquad \text{frequency-weighted IoU} = \frac{1}{\sum_k t_k} \sum_i \frac{t_i \, n_{ii}}{t_i + \sum_j n_{ji} - n_{ii}}.$$

Pixel accuracy and frequency-weighted IoU are dominated by large classes. Mean IoU, the per-class intersection over union averaged over classes, is the standard score; that article covers its history at PASCAL VOC. The ADE20K scene parsing challenge used the average of pixel accuracy and mean IoU as its criterion (Zhou et al., 2017). Boundary metrics complement region scores: following Csurka et al., SegNet’s authors reported a boundary F1 score that matches predicted and true class boundaries within a distance tolerance (Badrinarayanan et al., 2017).

Applications

Driving and robotics segment road, sidewalks, vehicles and pedestrians, the setting Cityscapes was built for. Medical imaging delineates organs, tumors and tissue in CT, MRI and microscopy, often with U-Net-style networks. Remote sensing maps land cover from aerial and satellite images. Augmented reality and photo editing use sky, person and clothing segmentation for effects such as background replacement.

Open Problems

• Open vocabularies. Fixed label sets do not cover the classes users ask for.
• Robustness. Accuracy drops under changes of weather, lighting, sensor and geography.
• Thin structures and boundaries. They hold few pixels, barely move pixel-level scores, and remain error-prone.
• Labels. Pixel-accurate annotation is costly, which motivates weak, semi-supervised and synthetic-data training.
• Efficiency. Real-time, high-resolution segmentation on embedded hardware still trades accuracy for speed.