Panoptic Segmentation

Panoptic segmentation labels every pixel of an image with a class and, for countable objects, with the identity of the object it belongs to. A street scene comes out as one road, one sky and one sidewalk region, plus a separate mask for each car and each pedestrian, and no pixel belongs to two segments. Kirillov, He, Girshick, Rother and Dollár defined this form of image segmentation, with its metric, panoptic quality (PQ), in 2019 to join semantic segmentation, which labels pixels but cannot separate objects, and instance segmentation, which separates objects but ignores amorphous regions and lets masks overlap. “Panoptic” means seeing everything in one view.

Problem Definition

The task rests on the distinction between things, countable objects such as people, animals and tools, and stuff, amorphous regions of similar texture or material such as grass, sky and road; Kirillov et al. credit Adelson (2001) with arguing that vision should study stuff as well as things. The two had been studied as separate tasks with separate metrics.

Given $L$ classes, split into disjoint stuff and thing subsets $\mathcal{L}^{\mathrm{St}}$ and $\mathcal{L}^{\mathrm{Th}}$, a panoptic segmentation maps each pixel $i$ to a pair $(l_i, z_i)$: a class $l_i$ and an instance id $z_i$ (Kirillov et al., 2019). For stuff classes the id is ignored, so all sky pixels form one segment; for things, each object has its own pair. Ambiguous or out-of-class pixels may be labeled void. The dataset’s creator decides which classes are stuff.

Three properties separate the task from its parents:

• Non-overlapping. Each pixel has one label and one id, unlike instance-segmentation masks.
• Complete. Stuff and things are predicted together; without thing classes, the format is that of semantic segmentation.
• No confidence scores, so human annotations have the same form as machine output and can be scored like it.

Input and Output

The input is an image; the output is a class map and an instance-id map, or one map of segment ids. COCO stores a PNG of segment ids (void is 0) and a JSON list with each segment’s category and crowd flag (COCO Consortium, data format).

Variants

• Video panoptic segmentation (Kim, Woo, Lee & Kweon, 2020) adds a time axis: each object keeps its instance id across frames. Their Cityscapes-VPS labels every fifth frame, six per video, of the 500 Cityscapes validation videos. Video panoptic quality (VPQ) computes PQ on space-time tubes in windows of $k \in \{0, 5, 10, 15\}$ frames and averages over $k$; $k = 0$ is image PQ, and an identity switch inside a window lowers the tube’s IoU.
• Amodal panoptic segmentation, labeling occluded extents, was named by Kirillov et al. as an extension for future work.

Challenges

• Overlaps between things. Region-based instance segmenters such as Mask R-CNN segment each detection independently, so their overlapping masks must be resolved into one partition.
• Separating instances. Per-pixel classifiers label touching objects of one class identically.
• Recognition more than delineation. In the human-versus-machine comparison of Kirillov et al. (2019), machine segmentation quality was close to human but recognition quality far below it: on ADE20K, RQ 43.2 against 78.6 for humans.
• Small objects. Even between human annotators on Cityscapes, RQ was 51.5 for the smallest quarter of objects and 94.1 for the largest quarter, while SQ stayed reasonable (Kirillov et al., 2019).

Approaches

Heuristic combination

The first baseline combined separate systems (Kirillov et al., 2019). Instance predictions are sorted by confidence and low scores removed; going down the list, each instance loses the pixels already assigned to earlier ones and is kept only if enough of it remains. Stuff comes from a semantic segmentation, and overlaps between things and stuff go to the thing. The thresholds were chosen by grid search to maximize PQ. With PSPNet and a COCO-pretrained Mask R-CNN on Cityscapes val this gave 61.2 PQ, against a human consistency of 69.7 measured on 30 doubly annotated images.

Unified two-branch networks

Panoptic FPN (Kirillov, Girshick, He & Dollár, 2019) keeps Mask R-CNN on a feature pyramid network and adds a light semantic branch that upsamples every pyramid level to 1/4 resolution, sums them and predicts the stuff classes plus one “other” class for all thing pixels. The same heuristic merges the outputs, and also removes stuff regions labeled “other” or below an area threshold. One network matched the PQ of two separate ResNet-50-FPN networks with roughly half the compute.

UPSNet (Xiong et al., 2019) replaces the post-processing with a parameter-free panoptic head: one channel per stuff class and one per detected instance (the semantic head’s logits for its class inside its box plus its pasted mask logits), so a per-pixel softmax assigns each pixel to one stuff class or one instance. It also predicts an “unknown” class, scored as void, because a wrong label costs both a false negative and a false positive while an unknown costs only the false negative.

Bottom-up methods

DeeperLab (Yang et al., 2019) and Panoptic-DeepLab (Cheng et al., 2020) skip boxes. Panoptic-DeepLab has one branch for semantic segmentation and one class-agnostic branch that predicts a heatmap of instance centers and, at every thing pixel, a 2-D offset to its center. Centers are found by keypoint-style NMS, a $7 \times 7$ max-pooling, and each thing pixel joins the center nearest to its position plus offset. A majority vote of the semantic labels in each group gives the instance’s class. With an Xception-71 backbone the authors reported 65.5 PQ on the Cityscapes test set, with Mapillary Vistas as extra training data.

Mask classification

DETR (Carion et al., 2020) attaches a mask head to its set predictions and merges the masks by a pixel-wise argmax, so they cannot overlap and stuff and things are handled alike. MaskFormer (Cheng, Schwing & Kirillov, 2021) drops the boxes: a transformer decoder turns $N$ queries into pairs of class probabilities $p_i$ and masks $m_i$, and each pixel goes to $\arg\max_i p_i(c_i)\, m_i[h, w]$, where $c_i$ is the pair’s most likely class and “no object” pairs are excluded. Low-confidence pairs and mostly hidden segments are removed. The authors found the gain over DETR largest on stuff, which bounding boxes describe poorly. Mask2Former (Cheng et al., 2022) restricts cross-attention to each query’s predicted mask and uses one architecture for panoptic, instance and semantic segmentation.

Results

PQ on COCO panoptic as reported in each paper, without multi-scale or flip testing. Backbones, schedules and post-processing differ, so rows compare only roughly.

Method	Year	Backbone and setting	Split	PQ	PQTh	PQSt
Panoptic FPN	2019	ResNet-101-FPN	val2017	40.3	47.5	29.5
Panoptic FPN	2019	ResNet-101-FPN	test-dev	40.9	48.3	29.7
UPSNet	2019	ResNet-50-FPN	val2017	42.5	48.5	33.4
Panoptic-DeepLab	2020	Xception-71	val2017	39.7	43.9	33.2
DETR	2020	ResNet-50	val2017	43.4	48.2	36.3
MaskFormer	2021	Swin-L (ImageNet-22K)	val2017	52.7	58.5	44.0
Mask2Former	2022	Swin-L (ImageNet-22K), 200 queries	val2017	57.8	64.2	48.1

On Cityscapes val, Mask2Former with Swin-L reported 66.6 PQ.

Datasets and Benchmarks

The first benchmarks reused datasets with both kinds of labels (Kirillov et al., 2019):

• Cityscapes (Cordts et al., 2016): street scenes with 19 evaluated classes, 8 of them things; its benchmark added a panoptic task in 2019.
• ADE20K (Zhou et al., 2017): with the 2017 Places Challenge vocabulary of 100 thing and 50 stuff classes.
• Mapillary Vistas (Neuhold et al., 2017): 25,000 street-level images; the research edition has 37 thing and 28 stuff classes.
• COCO panoptic (Lin et al., 2014; COCO Consortium), the main benchmark: 80 thing and 53 stuff classes, 133 in all, with overlapping instance masks resolved.

Cityscapes-VPS and a re-formatted synthetic VIPER dataset serve the video task (Kim et al., 2020).

Evaluation Metrics

Panoptic quality

PQ (Kirillov et al., 2019) scores a panoptic segmentation in two steps.

Matching. A predicted and a ground-truth segment of the same class match only if their IoU is strictly greater than 0.5. Because predicted segments do not overlap, each ground-truth segment has at most one such partner, and vice versa. The proof: for disjoint predictions $p_1, p_2$ and a ground-truth segment $g$,

$$\mathrm{IoU}(p_1, g) + \mathrm{IoU}(p_2, g) \le \frac{|p_1 \cap g|}{|g|} + \frac{|p_2 \cap g|}{|g|} \le 1,$$

since each union is at least $|g|$ and the two intersections are disjoint parts of $g$. If one IoU exceeds 0.5, the other cannot. Matching therefore needs no assignment algorithm, and greedy and optimal matching give the same result.

Scoring. For each class, matched pairs are true positives (TP), unmatched predictions false positives (FP), and unmatched ground truth false negatives (FN). Then

$$\mathrm{PQ} = \frac{\sum_{(p, g) \in \mathit{TP}} \mathrm{IoU}(p, g)}{|\mathit{TP}| + \tfrac{1}{2}|\mathit{FP}| + \tfrac{1}{2}|\mathit{FN}|} = \underbrace{\frac{\sum_{(p, g) \in \mathit{TP}} \mathrm{IoU}(p, g)}{|\mathit{TP}|}}_{\text{segmentation quality (SQ)}} \times \underbrace{\frac{|\mathit{TP}|}{|\mathit{TP}| + \tfrac{1}{2}|\mathit{FP}| + \tfrac{1}{2}|\mathit{FN}|}}_{\text{recognition quality (RQ)}}.$$

SQ is the mean IoU of matched segments and RQ is the F1 score of the matching. The two are not independent, since SQ is measured only on matches. PQ is computed per class and averaged over classes, so the decomposition holds per class but not for the averages. PQ^Th and PQ^St average over thing and stuff classes. In the official COCO code, TP, FP, FN and summed IoU are accumulated over the whole dataset before each class’s ratio is taken, and classes with none of the three are skipped (panopticapi).

Void and crowd regions. Ground-truth void pixels are not evaluated: they are removed from predicted segments before IoU is computed, and an unmatched prediction that is more than half void is dropped rather than counted as an FP. Crowd regions, which label several adjacent instances of one class without ids, are excluded from matching, and an unmatched prediction lying more than half in a crowd region of its class is also not an FP (Kirillov et al., 2019). The official code adds a prediction’s void and same-class crowd pixels together before this one-half test (panopticapi).

In the arXiv version of their paper (not the CVPR version), Kirillov et al. found on doubly annotated images that optimal matching without a threshold gave fewer than 16% of matches below IoU 0.5, supporting the threshold. Human consistency was 69.7 PQ on Cityscapes, 67.1 on ADE20K and 57.5 on Vistas, similar for stuff and things; the authors call these agreement measures, not upper bounds.

Criticisms and alternatives

• Every segment counts equally. Kirillov et al. note that segments receive equal importance regardless of area; Yang et al. (2019) point out that a 10 × 10-pixel instance then counts as much as a 1000 × 1000 one, making PQ sensitive to small false positives. UPSNet and Panoptic-DeepLab both turn their predicted stuff segments below an area threshold (4,096 pixels on COCO, 2,048 on Cityscapes) into void before evaluation, citing this sensitivity.
• Stuff is one segment. PQ treats all regions of a stuff class as one instance (Yang et al., 2019). In our reading this makes stuff all-or-nothing per image: the class either matches, adding its IoU, or counts as an FN (plus an FP if it was predicted), with no credit for an IoU of 0.5 or less.
• Parsing Covering. Yang et al. (2019) proposed PC, an area-weighted per-class covering score without matching or threshold, for applications where large objects matter more.

Example: merging network outputs and computing PQ

The NumPy code builds a 128 × 192 scene with sky and ground (stuff), two touching cars and a small person (things), and a void “hood” at the bottom. It simulates scored instance masks and a semantic map whose horizon is 3 rows too low, (1) merges them with the heuristic of Kirillov et al., (2) computes per-class SQ, RQ and PQ, and (3) repeats with two scores swapped.

import numpy as np

# Classes: 0 = void, stuff 1 = sky, 2 = ground; things 3 = car, 4 = person.
NAMES = {1: "sky", 2: "ground", 3: "car", 4: "person"}
STUFF, THINGS = (1, 2), (3, 4)
H, W = 128, 192
yy, xx = np.mgrid[:H, :W]

def rect(r0, r1, c0, c1):
    return (yy >= r0) & (yy < r1) & (xx >= c0) & (xx < c1)

# Ground truth as two maps: class l and instance id z (z = 0 for stuff and void).
gt_cls = np.where(yy < 50, 1, 2)
gt_ins = np.zeros((H, W), int)
for z, (c, m) in enumerate([(3, rect(60, 100, 20, 90)),      # car 1, large
                            (3, rect(66, 100, 90, 124)),     # car 2, touches car 1
                            (4, rect(70, 86, 150, 157))], 1):  # small person
    gt_cls[m], gt_ins[m] = c, z
gt_cls[yy >= 118] = 0                                        # unlabeled hood: void

# (1) Heuristic merge (Kirillov et al., 2019): instance masks, sorted by score,
#     claim pixels first; the semantic map fills in only stuff pixels.
sem = np.where(yy < 53, 1, 2)                                # horizon 3 rows too low
sem[rect(60, 100, 20, 124)] = 3                              # semantic net sees "car"
sem[rect(71, 85, 150, 157)] = 4                              # ... and the person
instances = [  # (mask, class, score)
    (rect(61, 100, 21, 124), 3, 0.95),   # one mask over both cars (merged)
    (rect(66, 100, 91, 124), 3, 0.80),   # car 2 on its own, less confident
    (rect(16, 24, 120, 128), 4, 0.70),   # a "person" in the sky: false positive
    (rect(71, 86, 150, 157), 4, 0.30),   # the real person, low score
    (rect(119, 127, 140, 170), 3, 0.60), # a "car" on the void hood
]

def merge(instances, sem, score_thr=0.5, keep_frac=0.5, min_stuff=64):
    cls, ins = np.zeros((H, W), int), np.zeros((H, W), int)
    taken = np.zeros((H, W), bool)
    z = 0
    for m, c, s in sorted(instances, key=lambda t: -t[2]):
        if s < score_thr:
            continue                                         # low score: dropped
        free = m & ~taken
        if free.sum() < keep_frac * m.sum():
            continue                                         # mostly covered: dropped
        z += 1
        cls[free], ins[free], taken[free] = c, z, True
    for c in STUFF:                                          # stuff from the semantic map
        m = (sem == c) & ~taken
        if m.sum() >= min_stuff:
            cls[m] = c
    return cls, ins                                          # unfilled pixels stay void (0)

pr_cls, pr_ins = merge(instances, sem)

# (2) Panoptic quality, following the official COCO panopticapi rules.
def segments(cls, ins):
    """Dictionary {(class, id): mask} of the non-void segments."""
    return {(c, z): (cls == c) & (ins == z)
            for c, z in set(zip(cls[cls > 0].ravel(), ins[cls > 0].ravel()))}

def pq_stats(gt_cls, gt_ins, pr_cls, pr_ins):
    G, P = segments(gt_cls, gt_ins), segments(pr_cls, pr_ins)
    void = gt_cls == 0
    st = {c: dict(iou=0.0, tp=0, fp=0, fn=0) for c in NAMES}
    matched_p = set()
    for gk, g in G.items():
        for pk, p in P.items():
            if gk[0] != pk[0]:
                continue                                     # only the same class can match
            inter = (g & p).sum()
            union = g.sum() + p.sum() - inter - (p & void).sum()  # void pixels of p ignored
            if inter / union > 0.5:                          # unique by the theorem
                st[gk[0]]["iou"] += inter / union
                st[gk[0]]["tp"] += 1
                matched_p.add(pk)
                break
        else:
            st[gk[0]]["fn"] += 1
    for pk, p in P.items():
        if pk not in matched_p and (p & void).sum() / p.sum() <= 0.5:
            st[pk[0]]["fp"] += 1                             # mostly-void segments are ignored
    return st

def report(st):
    pq, sq, rq = {}, {}, {}
    for c, s in st.items():
        d = s["tp"] + 0.5 * s["fp"] + 0.5 * s["fn"]
        sq[c] = s["iou"] / s["tp"] if s["tp"] else 0.0
        rq[c] = s["tp"] / d
        pq[c] = s["iou"] / d
        print(f"  {NAMES[c]:7s} TP {s['tp']} FP {s['fp']} FN {s['fn']}"
              f"   SQ {sq[c]:.3f}  RQ {rq[c]:.3f}  PQ {pq[c]:.3f}")
    avg = lambda d, cs: np.mean([d[c] for c in cs])
    print(f"  PQ {avg(pq, NAMES):.3f}  SQ {avg(sq, NAMES):.3f}  RQ {avg(rq, NAMES):.3f}"
          f"   PQ_Th {avg(pq, THINGS):.3f}  PQ_St {avg(pq, STUFF):.3f}")

print("ground-truth segments:", {f"{NAMES[c]}/{z}": int(m.sum())
                                 for (c, z), m in sorted(segments(gt_cls, gt_ins).items())})
print("predicted segments:  ", {f"{NAMES[c]}/{z}": int(m.sum())
                                 for (c, z), m in sorted(segments(pr_cls, pr_ins).items())})
report(pq_stats(gt_cls, gt_ins, pr_cls, pr_ins))

# Pixel-level IoU of the same prediction, as semantic segmentation would score it.
valid = gt_cls > 0
print("  semantic IoU:", {NAMES[c]: round(float(((gt_cls == c) & (pr_cls == c) & valid).sum()
      / (((gt_cls == c) | (pr_cls == c)) & valid).sum()), 3) for c in NAMES})

# (3) Same outputs, but the network is more confident in the separate car 2.
swapped = [(m, c, {0.95: 0.80, 0.80: 0.95}.get(s, s)) for m, c, s in instances]
print("scores of the two car masks swapped:")
report(pq_stats(gt_cls, gt_ins, *merge(swapped, sem)))

Output:

ground-truth segments: {'sky/0': 9600, 'ground/0': 8988, 'car/1': 2800, 'car/2': 1156, 'person/3': 112}
predicted segments:   {'sky/0': 10112, 'ground/0': 9902, 'car/1': 4017, 'car/3': 240, 'person/2': 64}
  sky     TP 1 FP 0 FN 0   SQ 0.937  RQ 1.000  PQ 0.937
  ground  TP 1 FP 0 FN 0   SQ 0.912  RQ 1.000  PQ 0.912
  car     TP 1 FP 0 FN 1   SQ 0.652  RQ 0.667  PQ 0.435
  person  TP 0 FP 1 FN 1   SQ 0.000  RQ 0.000  PQ 0.000
  PQ 0.571  SQ 0.625  RQ 0.667   PQ_Th 0.217  PQ_St 0.924
  semantic IoU: {'sky': 0.937, 'ground': 0.912, 'car': 0.932, 'person': 0.0}
scores of the two car masks swapped:
  sky     TP 1 FP 0 FN 0   SQ 0.937  RQ 1.000  PQ 0.937
  ground  TP 1 FP 0 FN 0   SQ 0.912  RQ 1.000  PQ 0.912
  car     TP 2 FP 0 FN 0   SQ 0.933  RQ 1.000  PQ 0.933
  person  TP 0 FP 1 FN 1   SQ 0.000  RQ 0.000  PQ 0.000
  PQ 0.696  SQ 0.696  RQ 0.750   PQ_Th 0.467  PQ_St 0.924

The confident mask over both cars takes all of car 2’s pixels, so the separate car-2 mask is discarded. The merged mask still matches car 1 at IoU 0.652, and car 2 becomes an FN: car PQ is 0.652 / 1.5 = 0.435. The real person (score 0.30) is dropped and its pixels stay void, since only instances supply things: an FN without an FP. The sky “person” is an FP; the hood “car”, all void, is ignored. The low horizon costs the stuff classes only SQ. Mean SQ times mean RQ would give 0.417, not the PQ of 0.571, as the decomposition holds only per class. Pixel IoU for car, 0.932, does not see the merge at all. Swapping the two car scores recovers both cars (car PQ 0.933): the heuristic’s output depends on the score order.

Applications

Kirillov et al. (2019) named autonomous driving and augmented reality as motivating applications; in our reading, both need regions (road, walls) and individual objects (each vehicle or pedestrian) in one consistent map.

Open Problems

• Recognition and small objects, the main gap to human consistency on the original benchmarks.
• Metric design: PQ’s equal weighting of segments invites post-processing that removes small segments.
• Temporal consistency in video, which VPQ measures over short windows.
• Open vocabularies and amodal extents, beyond a fixed set of visible classes.