Instance Segmentation

Instance segmentation finds every object of a set of classes in an image and marks the pixels of each one. Where semantic segmentation labels a crowd as “person”, instance segmentation returns one mask per person, each with a class and a confidence score. It joins object detection, which separates objects but describes each by a box, with pixel-level segmentation; Hariharan et al. (2014) called it simultaneous detection and segmentation.

Problem Definition

Given an image and $K$ thing classes, predict a set of instances $\{(M_i, c_i, s_i)\}$: a binary mask $M_i$ over the pixel grid, a class $c_i \in \{1, \dots, K\}$ and a score $s_i$. The number of instances varies from image to image.

• Things only. Stuff classes such as sky and road are left to semantic and panoptic segmentation.
• Masks may overlap. Instance segmentation allows overlapping segments, whereas panoptic segmentation gives each pixel exactly one class and instance id (Kirillov et al., 2019). Predictions are a ranked list of hypotheses, like detections, not a partition of the image.
• Visible extent. Ground-truth masks usually cover the visible part of each object, so an occluded object’s mask has a gap where the occluder is.

Input and Output

The input is an image; the output is a list of masks with classes and scores. Region-based methods predict a fixed $m \times m$ grid of probabilities inside each detected box, $28 \times 28$ in Mask R-CNN’s FPN head, then resize it to the box and threshold it at 0.5 (He et al., 2017). One-stage and query-based methods predict each mask over the whole image at a fraction of its resolution.

Variants

Amodal instance segmentation predicts each object’s full extent, occluded parts included; video instance segmentation also links each object’s masks through time, as in tracking by detection; large- and open-vocabulary segmentation extends the labels to a long tail of rare classes or to classes named in text; promptable segmentation returns the mask a point or box indicates, without a class, as in SAM (Kirillov et al., 2023).

Challenges

• Touching objects of one class look like one region to a per-pixel classifier, as the example below shows.
• Occlusion and crowds fragment masks, put other objects inside each box, and invite missed or duplicate instances.
• Small objects cover few feature-map cells; every method in the results table below reports far lower mask AP on small COCO objects than on large ones.
• Mask resolution. A $28 \times 28$ grid over a 300-pixel box has cells about 11 pixels wide; SOLOv2’s authors attribute Mask R-CNN’s weaker boundaries to this limit (Wang et al., 2020).

Approaches

Detect then segment

Early deep methods followed R-CNN’s scheme. SDS (Hariharan et al., 2014) scored 2,000 bottom-up region candidates per image from MCG with SVMs on CNN features of each candidate’s box and masked foreground, then refined the survivors with a coarse $10 \times 10$ top-down figure–ground mask. DeepMask (Pinheiro, Collobert & Dollár, 2015) learned the candidates: a network outputs a class-agnostic mask and an objectness score per image patch, and SharpMask (Pinheiro et al., 2016) refines that mask top-down through successively higher-resolution layers.

Multi-task network cascades (MNC; Dai, He & Sun, 2016) put three stages in one network with shared features: propose boxes, regress a class-agnostic $28 \times 28$ mask per box from RoI-pooled features, and classify each masked instance. It ran in 360 ms per image with VGG-16, by the authors’ account two orders of magnitude faster than earlier systems, and underlay the winning COCO 2015 entry. FCIS (Li et al., 2017), the 2016 winner, made the region head fully convolutional: position-sensitive “inside” and “outside” score maps, assembled within each region, decide jointly whether a pixel belongs to the object and whether the region is an object. He et al. (2017) reported that FCIS makes systematic errors on overlapping instances and creates spurious edges.

Mask R-CNN (He et al., 2017) adds a small fully convolutional mask branch to the Faster R-CNN detector, in parallel with the box head. RoIAlign samples region features bilinearly without rounding, keeping the mask aligned with the image, and the branch predicts one binary mask per class with a per-pixel sigmoid, so classes do not compete for pixels; the box head’s class selects the mask. PANet (Liu et al., 2018) adds a bottom-up path to the feature pyramid and pools each proposal from all levels; it won the COCO 2017 instance segmentation challenge. Cascade Mask R-CNN adds a mask branch to the multi-stage Cascade R-CNN; Hybrid Task Cascade (HTC; Chen et al., 2019) interleaves the box and mask branches across stages and adds a semantic segmentation branch for context.

Bottom-up grouping

Bottom-up methods label pixels first and group them into instances, with no boxes. De Brabandere, Neven and Van Gool (2017) map each pixel to an embedding with a discriminative loss: a pull of each embedding toward its instance’s mean (inactive within a margin $\delta_v$), a push between instance means (inactive beyond $2\delta_d$), and a weak pull of the means toward the origin. At test time a variant of mean shift clusters the embeddings, and a separate semantic segmentation supplies the class. On the Cityscapes val set, replacing their predicted semantic masks and clustering with ground truth raised AP from 21.4 to 47.8; the authors found the semantic masks the larger factor.

The deep watershed transform (Bai & Urtasun, 2017) borrows the classical watershed. Given a PSPNet semantic segmentation, one network predicts at each pixel a unit vector pointing away from the nearest instance boundary, and a second turns these into an energy map with 16 discrete levels in which each instance is a basin; a cut at one level leaves the instances as connected components. The authors reported more than double the previous best AP on the Cityscapes instance task.

One-stage and dynamic masks

YOLACT (Bolya et al., 2019) targets real time. One branch predicts $k$ prototype masks $P$ ($h \times w \times k$) over the whole image ($138 \times 138$ for a $550 \times 550$ input); the detection head predicts $k$ coefficients per instance, $C$ ($n \times k$); and the masks $M = \sigma(P C^\top)$ are cropped to each predicted box. The authors reported 29.8 mask AP on COCO test-dev at 33.5 fps on a Titan Xp (ResNet-101-FPN), using Fast NMS, a quicker variant of non-maximum suppression with a small accuracy cost.

SOLO (Wang et al., 2020) drops boxes altogether. It divides the image into an $S \times S$ grid; the cell containing an object’s center predicts its class, and channel $k$ of an $S^2$-channel output predicts that object’s mask, with pyramid levels separating objects by size. SOLOv2 (Wang et al., 2020) generates the mask channels dynamically: each cell predicts a convolution kernel that is applied to a shared mask feature map. Its Matrix NMS decays the scores of overlapping masks in parallel. CondInst (Tian, Shen & Chen, 2020) builds on the FCOS detector: for each instance, a controller predicts the weights of a tiny mask head (three convolutions with 8 channels) that runs on a shared feature map with coordinates relative to the instance’s location.

Query-based models

Mask2Former (Cheng et al., 2022) predicts a set of masks, each with a class, from learned queries, following DETR; as an end-to-end model it needs no post-processing such as NMS. Its transformer decoder restricts cross-attention to the foreground of each query’s current mask, and one architecture handles semantic, instance and panoptic segmentation, trained separately for each. With a Swin-L backbone and 200 queries it reported 50.1 mask AP on COCO val2017 (single-scale inference), but the authors note that its small-object AP still lags: 29.9, against 31.0 for the HTC-based Swin-HTC++. Promptable models such as SAM, fed a detector’s boxes, also yield instance masks.

Results

Single-model mask AP on COCO as reported in each paper. Training schedules and augmentation differ, so rows compare only roughly.

Method	Year	Backbone and setting	Split	Mask AP
MNC	2016	ResNet-101	test-dev	24.6
FCIS	2017	ResNet-101-C5-dilated, OHEM	test-dev	29.2
Mask R-CNN	2017	ResNet-101-FPN	test-dev	35.7
PANet	2018	ResNet-50, single-scale training	test-dev	36.6
YOLACT-550	2019	ResNet-101-FPN, 33.5 fps	test-dev	29.8
SOLO	2020	ResNet-101-FPN, 72 epochs	test-dev	37.8
CondInst	2020	ResNet-101-FPN, 3× schedule	test-dev	39.1
SOLOv2	2020	ResNet-101-FPN, 36 epochs	test-dev	39.7
Mask2Former	2022	Swin-L (ImageNet-22K), 200 queries, 100 epochs	val2017	50.1

Example: merged components, mask resolution and mask AP

The NumPy and SciPy code draws two bright disks and two dim triangles with Gaussian noise; disk B lies in front of disk A, so A’s ground truth is its visible part. It (a) thresholds the image into a semantic map and takes connected components as instances; (b) resamples each ground-truth mask to $m \times m$ in its box RoIAlign-style, pastes it back, and scores it with mask IoU and Boundary IoU (defined below), then tries a crude thresholding “mask head”; (c) computes mask AP for six ranked predictions, including a duplicate and a false positive.

import numpy as np
from scipy import ndimage

# A 256 x 256 scene with four objects of two classes, listed back to front.
# Disk B lies in front of disk A, so B hides part of A; each ground-truth mask is
# the visible part of its object. Class 1 = disk (bright), class 2 = triangle (dim).
H = W = 256
yy, xx = np.mgrid[:H, :W] + 0.5                      # pixel centers

def disk(cy, cx, r):
    return (yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2

def triangle(top, left, size):                       # right-angled, hypotenuse on the right
    return (yy >= top) & (xx >= left) & (yy - top + xx - left <= size)

objects = [("A", 1, disk(84, 88, 52)), ("B", 1, disk(124, 136, 44)),
           ("C", 2, triangle(168, 12, 80)), ("D", 2, triangle(20, 196, 24))]
inst = np.zeros((H, W), int)                          # instance ids; later objects occlude
for k, (_, _, m) in enumerate(objects, 1):
    inst[m] = k
gt = [inst == k for k in range(1, 5)]
cls = np.array([c for _, c, _ in objects])
img = np.choose(inst, [0.2] + [0.8 if c == 1 else 0.5 for c in cls])
img = img + np.random.default_rng(0).normal(0, 0.05, img.shape)

def iou(a, b):
    return (a & b).sum() / (a | b).sum()

def boundary_iou(a, b, d):
    """Cheng et al. (2021): IoU of the mask pixels within d of each mask's contour."""
    band = lambda m: m & (ndimage.distance_transform_edt(m) <= d)
    return iou(band(a), band(b))

print("visible pixels:", {n: int(g.sum()) for (n, _, _), g in zip(objects, gt)},
      " A hidden by B:", int((objects[0][2] & objects[1][2]).sum()))

# (a) A perfect-looking semantic map, then connected components as "instances".
sem = np.digitize(img, [0.35, 0.65])                  # 0 background, 1 triangle, 2 disk
print("semantic pixel accuracy: %.4f" % (sem == np.choose(inst, [0] + [3 - c for c in cls])).mean())
comps = []
for s in (2, 1):
    lab, n = ndimage.label(sem == s)
    comps += [lab == i for i in range(1, n + 1) if (lab == i).sum() >= 10]
print("components:", len(comps), " best IoU per object:",
      {n: round(float(max(iou(g, c) for c in comps)), 3) for (n, _, _), g in zip(objects, gt)})

# (b) Detect then segment: for each box, an m x m mask, pasted back bilinearly.
def box_of(m):                                        # tight box (y0, x0, y1, x1), pixel edges
    r, c = np.nonzero(m)
    return r.min(), c.min(), r.max() + 1, c.max() + 1

def roi_sample(f, box, m, s=2):
    """RoIAlign-style: average s x s bilinear samples in each of the m x m bins."""
    y0, x0, y1, x1 = box
    t = (np.arange(m * s) + 0.5) / (m * s)
    ys, xs = np.meshgrid(y0 + t * (y1 - y0) - 0.5, x0 + t * (x1 - x0) - 0.5, indexing="ij")
    v = ndimage.map_coordinates(f.astype(float), [ys, xs], order=1, mode="nearest")
    return v.reshape(m, s, m, s).mean(axis=(1, 3))

def paste(mm, box):
    """Resize the m x m soft mask to the box (bilinear) and threshold at 0.5."""
    y0, x0, y1, x1 = box
    m = mm.shape[0]
    py, px = np.mgrid[y0:y1, x0:x1] + 0.5
    gy, gx = (py - y0) / (y1 - y0) * m - 0.5, (px - x0) / (x1 - x0) * m - 0.5
    out = np.zeros((H, W), bool)
    out[y0:y1, x0:x1] = ndimage.map_coordinates(mm, [gy, gx], order=1, mode="nearest") >= 0.5
    return out

boxes = [box_of(g) for g in gt]
d = round(0.02 * np.hypot(H, W))                     # 2% of the image diagonal: 7 px
print(f"\nbest m x m mask (ground truth resampled), pasted back: mask IoU / boundary IoU (d = {d})")
for m in (14, 28, 56):
    pm = [paste(roi_sample(g, b, m), b) for g, b in zip(gt, boxes)]
    print(f"  m = {m:2d}:", "  ".join(f"{n} {iou(p, g):.3f}/{boundary_iou(p, g, d):.3f}"
                                     for (n, _, _), g, p in zip(objects, gt, pm)))

# A crude per-box "mask head": threshold the image inside the box at 28 x 28.
lo_hi = {1: (0.65, np.inf), 2: (0.35, 0.65)}
pred = []
for (n, c, _), b in zip(objects, boxes):
    patch = roi_sample(img, b, 28)
    pred.append(paste(((patch > lo_hi[c][0]) & (patch < lo_hi[c][1])).astype(float), b))
print("threshold-in-box masks at m = 28, IoU:",
      {n: round(float(iou(p, g)), 3) for (n, _, _), g, p in zip(objects, gt, pred)})

# (c) Mask AP: score instance predictions, match by mask IoU, per class, then mean.
preds = [(pred[0], 1, 0.95), (pred[1], 1, 0.90), (pred[2], 2, 0.85), (pred[3], 2, 0.60),
         (paste(roi_sample(gt[1], boxes[1], 28), boxes[1]), 1, 0.70),   # duplicate of B
         (disk(220, 200, 14), 2, 0.75)]                                  # false positive

def ap(preds, c, t):
    ps = sorted([p for p in preds if p[1] == c], key=lambda p: -p[2])
    gts = [g for g, k in zip(gt, cls) if k == c]
    used, tp = set(), []
    for mask, _, _ in ps:                              # greedy, highest score first
        o, j = max(((iou(mask, g), j) for j, g in enumerate(gts) if j not in used),
                   default=(0, None))
        tp.append(o >= t)
        if o >= t:
            used.add(j)
    tp = np.array(tp)
    rec = np.cumsum(tp) / len(gts)
    prec = np.cumsum(tp) / np.arange(1, len(tp) + 1)
    prec = np.maximum.accumulate(prec[::-1])[::-1]     # precision envelope
    r = np.linspace(0, 1, 101)                          # COCO's 101 recall points
    i = np.searchsorted(rec, r, side="left")
    return np.where(i < len(prec), prec[np.minimum(i, len(prec) - 1)], 0).mean()

for t in (0.5, 0.75):
    a = [ap(preds, c, t) for c in (1, 2)]
    print(f"mask AP@{t}: disk {a[0]:.3f}, triangle {a[1]:.3f}, mean {np.mean(a):.3f}")

Output:

visible pixels: {'A': 6804, 'B': 6092, 'C': 3240, 'D': 300}  A hidden by B: 1688
semantic pixel accuracy: 0.9988
components: 3  best IoU per object: {'A': 0.527, 'B': 0.472, 'C': 0.998, 'D': 0.997}

best m x m mask (ground truth resampled), pasted back: mask IoU / boundary IoU (d = 7)
  m = 14: A 0.964/0.824  B 0.973/0.870  C 0.980/0.941  D 1.000/1.000
  m = 28: A 0.987/0.936  B 0.990/0.942  C 0.999/0.999  D 1.000/1.000
  m = 56: A 0.998/0.989  B 0.999/0.991  C 1.000/1.000  D 1.000/1.000
threshold-in-box masks at m = 28, IoU: {'A': 0.723, 'B': 0.919, 'C': 0.976, 'D': 1.0}
mask AP@0.5: disk 1.000, triangle 0.835, mean 0.917
mask AP@0.75: disk 0.252, triangle 0.835, mean 0.544

The semantic map is 99.9% correct, yet it gives three components for four objects: the two disks form one 12,896-pixel blob, so neither disk is matched with IoU above 0.53, while the isolated triangles come out almost exactly. In (b), the best $14 \times 14$ mask loses about 3% of IoU on the large disks, $28 \times 28$ about 1%, $56 \times 56$ almost nothing; Boundary IoU, computed on a 7-pixel band along the contours, shows the loss more clearly (0.936 against 0.987 for disk A at $m = 28$). The 24-pixel triangle D is exact at every size, since even $14 \times 14$ cells are under two pixels wide: a fixed grid limits large objects. The thresholding “mask head” fails differently: A’s box contains much of B, which is equally bright, so A’s mask takes in B’s pixels (IoU 0.723). A learned mask head must tell the box’s own object from a neighbor of the same class.

That mask still counts at IoU 0.5, so disk AP is 1.0; the duplicate of B, ranked below both true positives, costs nothing. At 0.75 the top-scoring disk prediction becomes a false positive and disk AP falls to 0.252. The triangle AP is 0.835 at both thresholds because its false positive outranks triangle D. Over noise seeds 0–9 every printed value changed by at most 0.007 and the AP values not at all.

Datasets and Benchmarks

• COCO (Lin et al., 2014): per-instance masks for 80 thing categories; its test-dev mask AP is the standard comparison.
• Cityscapes (Cordts et al., 2016): street scenes in which 8 classes of people and vehicles (person, rider, car, truck, bus, train, motorcycle, bicycle) are evaluated as instances, with COCO-style AP plus AP for objects within 100 m and 50 m.
• LVIS (Gupta, Dollár & Girshick, 2019): planned as 2.2 million masks for over 1,000 categories on the 164,000 COCO images, with a long tail of rare classes and masks closer to expert outlines than COCO’s.
• PASCAL VOC with the SBD annotations served early methods such as SDS and MNC.

Evaluation Metrics

Instance segmentation is scored like detection, with average precision, but matching uses mask IoU instead of box IoU; SDS called this $\text{AP}^r$ (Hariharan et al., 2014). COCO’s mask AP averages over IoU thresholds 0.50:0.05:0.95, with objects sized by mask area; the COCO article covers the protocol.

Mask IoU barely registers boundary errors on large objects, where most pixels are interior. Cheng et al. (2021) defined Boundary IoU, the IoU of the parts of the two masks within distance $d$ of their contours ($d$ = 2% of the image diagonal for COCO-sized images), and Boundary AP, which matches by $\min(\text{mask IoU}, \text{Boundary IoU})$ so that a ring cannot score perfectly against a disk. Mask2Former gained 2.1 points of Boundary AP over Swin-HTC++ but 0.6 points of mask AP.

Applications

Instance masks serve wherever objects must be counted, measured or handled one at a time: pedestrians and vehicles in driving scenes, items to grasp in robotics, cells and nuclei in microscopy, and cut-outs in photo and video editing.

Open Problems

• Small and crowded objects: small-object AP remains far below large-object AP, including for query-based models.
• Boundaries: low-resolution masks and coarse annotations limit boundary quality and its measurement.
• Occlusion and amodal extent.
• Rare and unseen classes, as in LVIS’s long tail.
• Speed: real-time models give up mask AP.