Panoptic Quality

Panoptic quality (PQ) scores a panoptic segmentation, a partition of an image into non-overlapping stuff regions and thing instances, against the ground truth. Predicted and ground-truth segments of the same class are paired when their intersection over union exceeds 0.5; each pair contributes its IoU, and each unpaired segment on either side counts as half an error. Kirillov, He, Girshick, Rother and Dollár (2019) introduced it with the task, as one metric that treats stuff and things alike and needs no confidence scores. It factors into segmentation quality (SQ), the mean IoU of the pairs, and recognition quality (RQ), an F1 score over segments. The panoptic segmentation article gives the definition, the proof that the pairing is unique, and the paper’s void and crowd rules; this article covers what the number means, how the official code computes it, where it misleads, and its variants.

Formula

For a class $c$, let $\mathit{TP}_c$ be the matched pairs $(p, g)$ with $\mathrm{IoU}(p, g) > 0.5$, $\mathit{FP}_c$ the unmatched predicted segments and $\mathit{FN}_c$ the unmatched ground-truth segments, pooled over all images of the evaluation set. Then

$$\mathrm{PQ}_c = \frac{\sum_{(p, g) \in \mathit{TP}_c} \mathrm{IoU}(p, g)}{|\mathit{TP}_c| + \tfrac{1}{2}|\mathit{FP}_c| + \tfrac{1}{2}|\mathit{FN}_c|} = \underbrace{\frac{\sum_{\mathit{TP}_c} \mathrm{IoU}(p, g)}{|\mathit{TP}_c|}}_{\mathrm{SQ}_c} \times \underbrace{\frac{|\mathit{TP}_c|}{|\mathit{TP}_c| + \tfrac{1}{2}|\mathit{FP}_c| + \tfrac{1}{2}|\mathit{FN}_c|}}_{\mathrm{RQ}_c}, \qquad \mathrm{PQ} = \frac{1}{|\mathcal{C}'|} \sum_{c \in \mathcal{C}'} \mathrm{PQ}_c ,$$

where, in the official code, $\mathcal{C}'$ is the set of classes with at least one TP, FP or FN in the whole evaluation set (panopticapi). PQ^Th and PQ^St average over the thing and the stuff classes of $\mathcal{C}'$; SQ and RQ are averaged the same way, so the mean PQ is not the product of the mean SQ and mean RQ.

Written with precision $P = |\mathit{TP}| / (|\mathit{TP}| + |\mathit{FP}|)$ and recall $R = |\mathit{TP}| / (|\mathit{TP}| + |\mathit{FN}|)$ over segments, RQ is $2|\mathit{TP}| / (2|\mathit{TP}| + |\mathit{FP}| + |\mathit{FN}|) = 2PR/(P + R)$, the F1 score. PQ replaces the count $|\mathit{TP}|$ in the numerator by the sum of the matched IoUs, so in our reading it is an F1 score in which each correct detection earns partial credit for how well it is delineated.

The uniqueness theorem relies on both partitions being non-overlapping. As far as we can see, it also needs the inequality to be strict: a ground-truth segment cut exactly in half by two predicted segments has two partners at IoU exactly 0.5, which “≥ 0.5” would admit. Removing ground-truth void pixels from predictions before computing IoU keeps them disjoint, so the theorem still holds.

Intuition

A perfect prediction has every segment matched at IoU 1, so PQ = SQ = RQ = 1. Each error adds one half to the denominator: a hallucinated segment is half an error, a missed one another half. A segment that is found but delineated with IoU just under 0.5 is the worst case: it is both a false negative and a false positive, adding a full unit to the denominator and nothing to the numerator, where at IoU 0.51 it would have added about half a unit to the numerator.

Kirillov et al. computed PQ per class and then averaged, so that frequent classes do not dominate. Within a class, every segment counts the same regardless of size: a 20-pixel false positive costs as much as a 20,000-pixel one.

In the arXiv version of the paper, the authors generalized RQ with a weight $\alpha$ on the errors, after the $F_\beta$ score:

$$\mathrm{RQ}^{\alpha} = \frac{|\mathit{TP}|}{|\mathit{TP}| + \alpha|\mathit{FP}| + \alpha|\mathit{FN}|}.$$

The default is $\alpha = 1/2$; $\alpha = 1$ gives the Jaccard index over segments, the form HOTA uses for its detection accuracy. On the human-consistency annotations the reported RQ was 72.5, 82.1 and 89.2 for $\alpha = 1, 1/2, 1/4$ on Cityscapes (SQ 84.2, unchanged), and 59.4, 71.4 and 81.4 on Mapillary Vistas (SQ 79.5) (Kirillov et al., arXiv v3, Fig. 8).

Interpretation

PQ lies in $[0, 1]$ and is usually reported as a percentage, as panopticapi prints it. As a reference point, Kirillov et al. measured the agreement between two annotations of the same images. On Cityscapes (30 images) SQ was 84.2 and RQ 82.1, on ADE20K (64 images) 85.8 and 78.0, and on Vistas (46 images) 79.5 and 71.4 (CVPR version, Table 1). The authors warned against comparing these numbers across datasets. PQ also falls steeply with object size: splitting objects into the smallest 25%, middle 50% and largest 25%, human PQ was 35.1, 62.3 and 84.8 on Cityscapes and 35.6, 47.7 and 69.4 on Vistas (Table 2).

SQ and RQ should be read together. SQ is computed only over matches above 0.5, so it can never fall below 0.5 for a class with any match. A method can raise SQ by dropping hard, poorly segmented objects, which lowers RQ instead.

Whether the 0.5 threshold matters was checked only in the arXiv version; the CVPR version refers readers there for further analysis of the metric. Besides the distribution of match IoUs (Fig. 6), described in the panoptic segmentation article, it reports human PQ at matching thresholds 0.25, 0.5 and 0.75: 71.3, 69.7 and 59.2 on Cityscapes, 68.1, 67.1 and 61.6 on ADE20K, and 60.2, 57.5 and 49.2 on Vistas (Fig. 7). Lowering the threshold to 0.25 therefore changed little; raising it to 0.75 cost much more.

The Official Implementations

COCO and Cityscapes compute PQ with panopticapi’s evaluation.py (commit 7bb4655 at the time of writing; Cityscapes’s evalPanopticSemanticLabeling.py copies the same logic). The details below come from the code and are not all stated in the paper:

• Format. Each image is a PNG whose RGB value encodes a segment id, $R + 256G + 256^2 B$, with 0 meaning void, plus a JSON list giving each id’s category_id and, for the ground truth, iscrowd. Every ground-truth image must have a prediction, and the ids in the PNG and the JSON must agree, or the script stops with an error.
• Intersections. One np.unique over $\text{gt id} \cdot 256^3 + \text{pred id}$ yields every pairwise intersection at once, so no IoU matrix is built.
• IoU. $\mathrm{IoU} = |p \cap g| / (|p| + |g| - |p \cap g| - |p \cap \text{void}|)$, which removes the prediction’s pixels on ground-truth void; the match tests (same class, no crowd, IoU strictly above 0.5) are the paper’s.
• Crowd and void. Crowd regions are never false negatives. For the void-plus-crowd test on unmatched predictions, the code keeps only one crowd segment per class and image (the last one listed). Cityscapes’s converter creates exactly one: thing pixels that carry a class but no instance id become a single crowd segment for that class, and classes ignored in evaluation become void.
• Accumulation. Counts and IoU sums are pooled over images and empty classes skipped, as in the formula above. The script reports PQ, SQ and RQ for all classes, things and stuff, with the number of classes averaged, $N$.

COCO’s panoptic task evaluates 80 thing and 53 stuff categories. Some original stuff classes are merged, and the pixels of 12 removed ones are set to void. PQ decides the challenge, and the other eight numbers (SQ, RQ and the thing and stuff versions of all three) are for analysis (COCO panoptic evaluation page, as of October 2026). Cityscapes test submissions use the dataset’s label ids, not its train ids (evalPanopticSemanticLabeling.py header).

When to Use It

PQ fits any output that is a partition of the image into labeled segments: panoptic segmentation, and also semantic segmentation (every class scored as stuff) or instance segmentation once overlaps are resolved. Kirillov et al. gave two reasons for a new metric:

• Not AP. Average precision needs a confidence score per segment and rewards overlapping hypotheses. The Cityscapes val output of a COCO-pretrained Mask R-CNN lost AP when its masks were made non-overlapping (36.4 to 33.1), and semantic segmentations have no scores at all. Between the two instance-segmentation entries on each dataset, AP and PQ^Th ranked the methods the same way (Table 3).
• Not mean IoU. Pixel IoU ignores instances. On a 1k-image subset of the ADE20K test images from the 2017 Places Challenge, the semantic-segmentation entry G-RMI had a mean IoU of 30.6 against CASIA IVA JD’s 32.3, but a PQ^St of 19.3 against 27.4. The authors traced the gap to many small hallucinated patches of absent classes, which barely affect IoU but each add a false positive (Table 4).

When objects should count in proportion to their area, or when a ranked list of hypotheses is the output, PQ is the wrong tool; AP and area-weighted scores such as parsing covering (below) answer those questions.

Failure Cases

• The threshold cliff. An IoU of 0.51 earns about half a unit of numerator; 0.49 earns nothing and costs a full unit of errors. Porzi et al. (2019) showed a Cityscapes prediction in which pole (IoU 0.49) and traffic light (0.46) fell below the threshold while a similar-looking sidewalk (0.62) passed.
• Stuff is all or nothing. All pixels of a stuff class in an image form one segment. Porzi et al. gave the example of a street with two sidewalks, the larger of which is labeled road: the sidewalk segment then falls below 0.5 and the class scores zero for that image.
• Small segments weigh as much as large ones, which rewards post-processing that deletes small regions (the criticism and the area thresholds methods use are in the panoptic segmentation article). Weber et al. (2021) note that most, if not all, panoptic methods remove small stuff regions or predict void for this reason. panopticapi’s own script for combining semantic and instance outputs drops stuff segments below 64 × 64 = 4,096 pixels by default.
• Void is free. Predicted void pixels do not affect evaluation (Kirillov et al., 2019), so abstaining pays, as the panoptic segmentation article explains. Porzi et al.’s summary of the metric also drops ground-truth segments more than half covered by predicted void from the FN count. Neither the CVPR paper nor panopticapi at the commit above does so, so numbers from implementations that follow that description need not match the official code.
• Splits and merges. Splitting an object into two parts that each cover less than half of it, or merging two equal objects, loses the match entirely; uneven splits and merges keep one match at a lower IoU. Pixel-level IoU sees none of this (see the example).
• Rare classes and empty classes. Because classes are averaged with equal weight, a class with one ground-truth segment in the whole set moves PQ as much as one with thousands. In our reading of the code, a single spurious prediction of a class absent from the ground truth adds that class to the average with PQ 0.
• Pooling. Statistics are pooled over the evaluation set, so PQ is not the mean of per-image PQ values, and per-image scores computed by other tools do not average to the benchmark number.

Variants

• RQ^α (Kirillov et al., arXiv version) changes the weight of unmatched segments; $\alpha = 1/2$ is the standard.
• PQ† (Porzi et al., 2019) keeps PQ for things but scores stuff class $c$ as $\frac{1}{|S_c|}\sum_{(s, \hat{s}) \in M_c} \mathrm{IoU}(s, \hat{s})$, where $S_c$ are the ground-truth segments of $c$ and $M_c$ the pairs with any overlap: the mean IoU over the images whose ground truth contains $c$, with no threshold.
• Parsing covering (PC; Yang et al., 2019) averages over classes the area-weighted best IoU of each ground-truth region, $\mathrm{Cov}_i = \frac{1}{N_i} \sum_{R \in S_i} |R| \max_{R' \in S'_i} \mathrm{IoU}(R, R')$, where $S_i$ and $S'_i$ are the ground-truth and predicted regions of class $i$ and $N_i$ the total ground-truth area of class $i$, with no matching and no threshold, for applications where large objects matter more.
• Boundary PQ (Cheng et al., 2021) replaces each pair’s mask IoU by $\min(\text{mask IoU}, \text{boundary IoU})$, where boundary IoU counts only pixels near the two contours. The authors found mask IoU insensitive to boundary errors on large objects; on synthetic COCO and Cityscapes predictions the new score was the more sensitive one (analysis in the paper’s supplement).
• PartPQ (de Geus et al., 2021) keeps PQ’s matching for part-aware panoptic segmentation but replaces the IoU of a matched pair, for classes with parts, by the mean IoU of the part classes inside the two segments.
• Video. Video panoptic quality (VPQ; Kim et al., 2020), described in the panoptic segmentation article, computes PQ on space-time tubes over short windows. Segmentation and tracking quality (STQ; Weber et al., 2021) abandons segment matching: it is $\sqrt{\mathrm{AQ} \times \mathrm{SQ}}$, where SQ here is the pixel mean IoU over the video and AQ an association score over thing pixels, chosen in part to avoid PQ’s threshold and its sensitivity to small segments. Weber et al. also discuss Hurtado et al.’s panoptic tracking quality (PTQ), which subtracts identity switches from PQ’s numerator.

Example

The NumPy code (NumPy 2.3.5) draws a 120 × 160 street scene with sky and road (stuff), two touching cars and a person (things), a crowd region of people without instance ids, and a void strip at the bottom. It computes PQ in two independent ways: a direct implementation from the paper’s rules, which tests every same-class pair and asserts that no segment ever has two partners above 0.5, and a re-implementation of panopticapi’s pq_compute_single_core at commit 7bb4655. The two are compared on 300 randomly corrupted predictions (shifted, split, mislabeled and hallucinated segments). The one difference from panopticapi is that ground-truth areas come from the map, not from the JSON file.

import numpy as np

# Classes: stuff 1 = sky, 2 = road; things 3 = car, 4 = person. Segment id 0 = void.
NAMES = {1: "sky", 2: "road", 3: "car", 4: "person"}
THINGS = {3, 4}
H, W = 120, 160
yy, xx = np.mgrid[:H, :W]

def rect(r0, r1, c0, c1):
    return (yy >= r0) & (yy < r1) & (xx >= c0) & (xx < c1)

def build(parts):
    """parts: list of (mask, class, iscrowd), painted in order.
    Returns a COCO-style pair: segment-id map and {id: (class, iscrowd)}."""
    ids, info = np.zeros((H, W), np.int64), {}
    for k, (m, c, crowd) in enumerate(parts, 1):
        ids[m] = k if c > 0 else 0                              # class 0 paints void
        info[k] = (c, crowd)
    present = set(np.unique(ids)) - {0}
    return ids, {k: v for k, v in info.items() if k in present}

# Ground truth: sky, road, two touching cars, a person, a crowd of people, void hood.
CAR_A, CAR_B, PERSON = rect(60, 90, 10, 60), rect(62, 90, 60, 95), rect(50, 90, 112, 124)
GT_PARTS = [(yy < 40, 1, 0), (yy >= 40, 2, 0), (CAR_A, 3, 0), (CAR_B, 3, 0),
            (PERSON, 4, 0), (rect(48, 88, 132, 156), 4, 1),     # crowd: no instance ids
            (yy >= 112, 0, 0)]                                  # hood: void
gt = build(GT_PARTS)

# --- 1. Direct implementation: test every same-class pair (Kirillov et al., 2019, Sec. 4).
def stats_direct(gt, pr):
    (g_ids, g_info), (p_ids, p_info) = gt, pr
    void = g_ids == 0
    st = {c: [0.0, 0, 0, 0] for c in NAMES}                     # iou_sum, TP, FP, FN
    match_p = {}
    for gk, (gc, crowd) in g_info.items():
        if crowd:
            continue                                            # crowd regions never match
        g = g_ids == gk
        partners = []
        for pk, (pc, _) in p_info.items():
            p = (p_ids == pk) & ~void                           # void pixels removed from p
            iou = (g & p).sum() / (g | p).sum()
            if pc == gc and iou > 0.5:
                partners.append((pk, iou))
        assert len(partners) <= 1                               # the uniqueness theorem
        if partners:
            pk, iou = partners[0]
            assert pk not in match_p
            match_p[pk] = gk
            st[gc][0] += iou; st[gc][1] += 1
        else:
            st[gc][3] += 1
    for pk, (pc, _) in p_info.items():
        if pk in match_p:
            continue
        p = p_ids == pk
        crowd_same = np.zeros_like(p)
        for gk, (gc, crowd) in g_info.items():
            if crowd and gc == pc:
                crowd_same |= g_ids == gk
        if ((p & void).sum() + (p & crowd_same).sum()) / p.sum() <= 0.5:
            st[pc][2] += 1                                      # else ignored, not an FP
    return st

# --- 2. Re-implementation of panopticapi pq_compute_single_core (evaluation.py, commit 7bb4655).
OFFSET = 256 * 256 * 256
def stats_panopticapi(gt, pr):
    (g_ids, g_info), (p_ids, p_info) = gt, pr
    g_area = dict(zip(*np.unique(g_ids, return_counts=True)))
    p_area = dict(zip(*np.unique(p_ids, return_counts=True)))
    joint = g_ids.astype(np.uint64) * OFFSET + p_ids.astype(np.uint64)
    inter = {(int(l // OFFSET), int(l % OFFSET)): int(n)
             for l, n in zip(*np.unique(joint, return_counts=True))}
    st = {c: [0.0, 0, 0, 0] for c in NAMES}
    g_m, p_m = set(), set()
    for (gk, pk), n in inter.items():
        if gk not in g_info or pk not in p_info or g_info[gk][1] == 1:
            continue
        if g_info[gk][0] != p_info[pk][0]:
            continue
        union = p_area[pk] + g_area[gk] - n - inter.get((0, pk), 0)
        if n / union > 0.5:
            st[g_info[gk][0]][0] += n / union; st[g_info[gk][0]][1] += 1
            g_m.add(gk); p_m.add(pk)
    crowd = {}
    for gk, (gc, iscrowd) in g_info.items():
        if gk in g_m:
            continue
        if iscrowd:
            crowd[gc] = gk                                      # one crowd id per class
            continue
        st[gc][3] += 1
    for pk, (pc, _) in p_info.items():
        if pk in p_m:
            continue
        n = inter.get((0, pk), 0)
        if pc in crowd:
            n += inter.get((crowd[pc], pk), 0)
        if n / p_area[pk] > 0.5:
            continue
        st[pc][2] += 1
    return st

def accumulate(total, st):
    for c in st:
        total[c] = [a + b for a, b in zip(total[c], st[c])]
    return total

def pq_average(st, classes=NAMES):
    """Per-class PQ/SQ/RQ, then the mean over classes with TP + FP + FN > 0."""
    per, used = {}, []
    for c in classes:
        iou, tp, fp, fn = st[c]
        if tp + fp + fn == 0:
            continue
        d = tp + 0.5 * fp + 0.5 * fn
        per[c] = (iou / d, iou / tp if tp else 0.0, tp / d)
        used.append(c)
    mean = [np.mean([per[c][i] for c in used]) for i in range(3)]
    return mean, per

# Cross-check the two implementations on 300 randomly corrupted predictions.
rng = np.random.default_rng(0)
def random_prediction():
    parts = [(yy < 40 + rng.integers(-6, 7), 1, 0), (yy >= 40, 2, 0)]
    for m, c in ((CAR_A, 3), (CAR_B, 3), (PERSON, 4), (rect(48, 88, 132, 156), 4)):
        m = np.roll(m, tuple(rng.integers(-8, 9, 2)), axis=(0, 1))
        if rng.random() < 0.3:                                  # split it in two
            cut = xx < np.median(xx[m])
            parts += [(m & cut, c, 0), (m & ~cut, c, 0)]
        else:
            parts.append((m, c if rng.random() > 0.1 else 7 - c, 0))
    for _ in range(rng.integers(0, 4)):                         # random small blobs
        r, q, s = rng.integers(0, H - 10), rng.integers(0, W - 10), rng.integers(2, 10)
        parts.append((rect(r, r + s, q, q + s), int(rng.integers(1, 5)), 0))
    return build(parts)

tot_d, tot_a = ({c: [0.0, 0, 0, 0] for c in NAMES} for _ in range(2))
for _ in range(300):
    pr = random_prediction()
    tot_d = accumulate(tot_d, stats_direct(gt, pr))
    tot_a = accumulate(tot_a, stats_panopticapi(gt, pr))
assert all(np.allclose(tot_d[c], tot_a[c]) for c in NAMES)    # every count and IoU sum
print("300 random predictions; uniqueness assertion never failed")
print("  direct      PQ %.6f  SQ %.6f  RQ %.6f" % tuple(pq_average(tot_d)[0]))
print("  panopticapi PQ %.6f  SQ %.6f  RQ %.6f" % tuple(pq_average(tot_a)[0]))

# Uniqueness needs non-overlapping predictions: two overlapping masks can both pass 0.5.
a, b = rect(60, 90, 10, 55), rect(60, 90, 14, 60)
print("overlapping masks: IoU %.2f and %.2f with the same car"
      % ((CAR_A & a).sum() / (CAR_A | a).sum(), (CAR_A & b).sum() / (CAR_A | b).sum()))

Output:

300 random predictions; uniqueness assertion never failed
  direct      PQ 0.463024  SQ 0.771804  RQ 0.565936
  panopticapi PQ 0.463024  SQ 0.771804  RQ 0.565936
overlapping masks: IoU 0.90 and 0.92 with the same car

The exhaustive search never found a second partner above 0.5, and the histogram-based code, which counts every pair above 0.5 without checking whether either segment already has a partner, produced the same counts and IoU sums for every class: the theorem is what makes panopticapi’s shortcut exact. The split and shifted crowd predictions exercise the crowd filter, and every road prediction overlaps the void strip, which exercises the void correction of the IoU; no prediction in this run is more than half void, so the void filter for false positives is not tested here. The last line shows why the theorem needs a partition: two overlapping masks, as an instance segmenter may output, can both exceed 0.5 with one car.

The second block, which continues from the first, starts from a good prediction and introduces one error at a time. It compares the car class, PQ, PQ^Th and PQ^St with the pixel-level mean IoU that semantic segmentation would report. It then slides one car’s prediction across the IoU threshold, and finally pools three images.

# Continues from the previous block.
def miou(gt, pr):
    """Semantic-segmentation mean IoU of the same maps (crowd pixels count as person)."""
    gc = np.vectorize(lambda k: gt[1][k][0] if k else 0)(gt[0])
    pc = np.vectorize(lambda k: pr[1][k][0] if k else 0)(pr[0])
    valid = gc > 0
    return np.mean([((gc == c) & (pc == c) & valid).sum() / (((gc == c) | (pc == c)) & valid).sum()
                    for c in NAMES])

# A good prediction: horizon 2 rows low, cars a little small, the crowd split into
# three "person" segments (ignored: they lie in the crowd region).
car_a, car_b = rect(61, 89, 11, 59), rect(63, 89, 61, 94)
BASE = [(yy < 42, 1, 0), (yy >= 42, 2, 0), (car_a, 3, 0), (car_b, 3, 0), (PERSON, 4, 0),
        (rect(50, 86, 133, 141), 4, 0), (rect(50, 86, 141, 148), 4, 0), (rect(50, 86, 148, 155), 4, 0)]
cut = lambda f: xx < 11 + f * (59 - 11)                         # vertical cut through car A
scenarios = {
    "base":                  BASE,
    "+ 5x5 FP 'car'":        BASE + [(rect(45, 50, 100, 105), 3, 0)],
    "+ 30x30 FP 'car'":      BASE + [(rect(10, 40, 100, 130), 3, 0)],
    "car A split 60/40":     BASE[:2] + [(car_a & cut(0.6), 3, 0), (car_a & ~cut(0.6), 3, 0)] + BASE[3:],
    "car A split 50/50":     BASE[:2] + [(car_a & cut(0.5), 3, 0), (car_a & ~cut(0.5), 3, 0)] + BASE[3:],
    "cars A+B merged":       BASE[:2] + [(car_a | car_b | rect(63, 89, 59, 61), 3, 0)] + BASE[4:],
    "person missed (road)":  BASE[:4] + BASE[5:],
}
print(f"{'prediction':22s} car TP/FP/FN   car SQ  RQ    PQ  |  PQ    PQ_Th PQ_St  mIoU")
for name, parts in scenarios.items():
    pr = build(parts)
    st = stats_panopticapi(gt, pr)
    (pq, sq, rq), per = pq_average(st)
    pq_th = np.mean([per[c][0] for c in THINGS]); pq_st = np.mean([per[c][0] for c in (1, 2)])
    print(f"{name:22s}   {st[3][1]}/{st[3][2]}/{st[3][3]}       {per[3][1]:.3f} {per[3][2]:.3f} "
          f"{per[3][0]:.3f} | {pq:.3f} {pq_th:.3f} {pq_st:.3f}  {miou(gt, build(parts)):.3f}")

# The threshold cliff: slide the car B prediction to the right.
print("\nshift of car B prediction (px):  IoU   car PQ   PQ     mIoU")
for s in (8, 9, 10, 11, 12):
    parts = BASE[:3] + [(np.roll(car_b, s, axis=1), 3, 0)] + BASE[4:]
    pr = build(parts)
    seg_b = pr[0] == 4
    iou = (seg_b & CAR_B).sum() / (seg_b | CAR_B).sum()
    (pq, _, _), per = pq_average(stats_panopticapi(gt, pr))
    print(f"{s:31d}  {iou:.3f}  {per[3][0]:.3f}   {pq:.3f}  {miou(gt, pr):.3f}")

# Dataset-level pooling (panopticapi) versus averaging per-image PQ.
images = [build(scenarios["base"]), build(scenarios["car A split 50/50"]),
          build(scenarios["person missed (road)"])]
pooled = {c: [0.0, 0, 0, 0] for c in NAMES}
per_image = []
for pr in images:
    st = stats_panopticapi(gt, pr)
    pooled = accumulate(pooled, st)
    per_image.append(pq_average(st)[0][0])
print(f"\nthree images: pooled PQ {pq_average(pooled)[0][0]:.3f}, "
      f"mean of per-image PQ {np.mean(per_image):.3f}")

Output:

prediction             car TP/FP/FN   car SQ  RQ    PQ  |  PQ    PQ_Th PQ_St  mIoU
base                     2/0/0       0.886 1.000 0.886 | 0.936 0.943 0.929  0.907
+ 5x5 FP 'car'           2/1/0       0.886 0.800 0.709 | 0.891 0.854 0.927  0.904
+ 30x30 FP 'car'         2/1/0       0.886 0.800 0.709 | 0.858 0.854 0.862  0.815
car A split 60/40        2/1/0       0.708 0.800 0.567 | 0.856 0.783 0.929  0.907
car A split 50/50        1/2/1       0.876 0.400 0.350 | 0.802 0.675 0.929  0.907
cars A+B merged          1/0/1       0.575 0.667 0.383 | 0.812 0.692 0.932  0.914
person missed (road)     2/0/0       0.886 1.000 0.886 | 0.673 0.443 0.903  0.811

shift of car B prediction (px):  IoU   car PQ   PQ     mIoU
                              8  0.582  0.739   0.888  0.864
                              9  0.547  0.722   0.883  0.859
                             10  0.514  0.705   0.877  0.853
                             11  0.482  0.448   0.811  0.847
                             12  0.452  0.448   0.810  0.842

three images: pooled PQ 0.830, mean of per-image PQ 0.804

The three “person” segments in the crowd region are ignored in every row, so the base prediction is penalized only for its boundaries. The two false positives behave as the counting rule predicts: the 25-pixel car and the 900-pixel car cost the car class exactly the same, an RQ of 0.8. The large one also eats into the sky, which lowers PQ^St and mean IoU, so in the mean IoU column the small error is nearly invisible and the large one is not. Splits and merges show the opposite blindness. Mean IoU does not change when car A is cut in two, and it even rises for the merge, because the merged mask also covers the gap between the cars. PQ distinguishes the 60/40 split, which keeps a weaker match plus an FP, from the 50/50 split, which loses the match and leaves an FN and two FPs.

In the sweep, the one-pixel step that takes car B’s IoU across 0.5 costs car PQ and overall PQ far more than any other step, while mean IoU falls by the same small amount at every step. The pooled PQ of the three images differs from the mean of their per-image PQs, so per-image numbers do not average to the benchmark score.

Related Metrics

PQ is built from intersection over union for each pair and the F1 score of precision and recall over segments. Average precision scores ranked, possibly overlapping instance predictions, and mean IoU scores semantic segmentation at the pixel level. The Dice coefficient is the same F1 computed over pixels instead of segments. In tracking, HOTA uses a Jaccard form of the same TP, FP and FN counts, and MOTA a difference of counts.