Segmentation

Mask R-CNN (2017 paper)

The 2017 ICCV paper by He, Gkioxari, Dollár and Girshick that turned Faster R-CNN into an instance segmentation system by adding a parallel mask branch and RoIAlign, won the ICCV 2017 Best Paper award (Marr Prize), and became a widely used baseline for instance-level recognition.

intermediate

“Mask R-CNN” by Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick of Facebook AI Research (FAIR) was posted to arXiv on 20 March 2017 and presented at the IEEE International Conference on Computer Vision (ICCV) in 2017. It extended the Faster R-CNN detector to instance segmentation by adding a small branch that predicts a binary mask for each region, in parallel with the existing class and box outputs, and by replacing RoI pooling with RoIAlign, a feature-extraction layer without coordinate rounding. A third arXiv version (January 2018) added appendices on Cityscapes and enhanced COCO results, and a journal version appeared in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2020. The paper received the ICCV 2017 Best Paper award (Marr Prize). The model as understood today, with its architecture, training and full results, is covered in the Mask R-CNN article.

Problem

The paper opens by observing that rapid progress in detection and semantic segmentation had been driven by strong, simple baselines: Fast/Faster R-CNN for detection and the fully convolutional network (FCN; Long, Shelhamer & Darrell, 2015) for segmentation. Its stated goal was a comparably enabling framework for instance segmentation, which must find every object and also delineate each one, and so combines the two tasks. Against the expectation that this needs a complex method, the authors set out to show that a “surprisingly simple, flexible, and fast system” could beat the state of the art.

The related-work section sorts the instance segmentation systems of the time into two families and states what it saw as wrong with each (the instance segmentation article describes the systems themselves):

  • Segment proposals first. Following R-CNN, earlier methods classified bottom-up segments, and DeepMask and its successors learned to propose segment candidates that Fast R-CNN then classified. In these methods segmentation precedes recognition, which the authors call slow and less accurate. Multi-task network cascades (MNC; Dai, He & Sun, 2016), the COCO 2015 winner, predicted segment proposals from box proposals and then classified them, a cascade the paper calls complex.
  • Position-sensitive outputs. FCIS (Li et al., 2017), the COCO 2016 winner, predicted a set of position-sensitive output channels fully convolutionally, addressing class, box and mask at once, which made it fast. The paper argues that it makes systematic errors on overlapping instances and creates spurious edges, and shows FCIS outputs beside its own (Figure 5).

In most recent systems, the paper notes, classification depended on the mask prediction; Mask R-CNN removes that dependency.

Contribution

The paper presents Mask R-CNN as a natural extension of Faster R-CNN and locates its contributions in getting that extension right:

  1. A mask branch in parallel. A small FCN applied to each region of interest (RoI) predicts a mask alongside classification and box regression, rather than before or after them. The authors present this as the same move Fast R-CNN made when it ran classification and box regression in parallel instead of in R-CNN’s stages.
  2. RoIAlign. The paper calls pixel-to-pixel alignment the main missing piece of Fast/Faster R-CNN, and identifies the coordinate rounding in RoI pooling as its most evident cause. RoIAlign samples features at exact, unrounded positions. In the paper’s comparison, RoIWarp from MNC, which also used bilinear sampling but still rounded the RoI, performed on par with RoI pooling, which the authors read as showing that alignment, not interpolation, was what mattered.
  3. Decoupling masks from classes. The mask branch predicts one binary mask per class with a per-pixel sigmoid, and only the mask of the class chosen by the box branch is used. The paper contrasts this with the per-pixel softmax of FCNs, where classes compete for each pixel, and reports that the coupled version worked poorly for instances.

A fourth claim is generality: modeling each human keypoint as a one-hot mask turns the same system into a pose estimator with minimal changes. The abstract summarizes the result as top results in all three tracks of the COCO challenges, beating every existing single-model entry, including the 2016 winners. The authors offered the system as a “solid baseline” for instance-level recognition.

Method

The method section is short: a review of Faster R-CNN, the multi-task loss, the case for a convolutional mask representation, RoIAlign, and network configurations named by backbone (ResNet-C4, ResNet-FPN, ResNeXt) whose box heads, taken from the ResNet and feature pyramid network papers, gain a mask head. The Mask R-CNN article walks through the heads, loss and training settings, and the two-stage object detectors article compares RoIAlign with RoI pooling.

A design choice that shaped the experiments is that, for convenient ablation, the region proposal network was trained separately from Mask R-CNN and did not share features unless stated. The timing experiment instead used a shared model, trained with Faster R-CNN’s four-step scheme, which the authors report reached statistically the same mask AP.

Results

The experiments are organized around COCO, training on trainval35k and reporting ablations on the 5k-image minival split and final numbers on test-dev:

  • Main results (Section 4.1). Every configuration beat the baseline variants of MNC and FCIS; the best, ResNeXt-101-FPN, reached 37.1 mask AP on test-dev. With ResNet-101-FPN and no extras, Mask R-CNN exceeded FCIS+++, which added multi-scale training and testing, flip testing and online hard example mining.
  • Ablations (Section 4.2). Five controlled comparisons, one per design decision: backbone depth and type, sigmoid against softmax masks, class-specific against class-agnostic masks, RoI layer (including stride-32 features, where alignment mattered most), and convolutional against fully connected mask heads.
  • Box detection (Section 4.3). Running the trained model with its mask output ignored, Mask R-CNN gave 39.8 box AP on test-dev with ResNeXt-101-FPN. A version trained without the mask branch isolated the effect of multi-task training (0.9 box AP). The authors read the small gap between this box AP and the 37.1 mask AP as evidence that the approach had largely closed the gap between detection and instance segmentation.
  • Timing (Section 4.4). About 200 ms per image on a GPU, and one to two days of training on one 8-GPU machine, which the authors hoped would ease research on the problem; they add that the design was not optimized for speed.
  • Human pose (Section 5). With ResNet-50-FPN, the keypoint variant scored 62.7 keypoint AP on test-dev, 0.9 above the multi-stage COCO 2016 winner. Joint training on the person category showed an asymmetry on minival: adding the mask branch improved the box-only and keypoint-only models, but adding the keypoint branch slightly lowered box and mask AP.

Appendices in arXiv v3

The third arXiv version, of 24 January 2018, adds two appendices that are not in the ICCV proceedings. Appendix A applies ResNet-50-FPN to Cityscapes: 26.2 AP on the test set with fine annotations only, against 20.0 for the previous best entry, and 32.0 with COCO pre-training. They trace a gap between val and test AP mainly to the rare truck, bus and train classes, which they attribute to domain shift. Appendix B stacks improvements (end-to-end training, ImageNet-5k pre-training, train-time augmentation, ResNeXt-152, non-local blocks, test-time augmentation) that raise COCO minival mask AP from 36.7 to 41.8; with data distillation on unlabeled images and test-time augmentation, minival keypoint AP reaches 70.4. According to this appendix, the result was the basis of the authors’ COCO 2017 submission, and the first three teams in the 2017 instance segmentation challenge were all reportedly based on extensions of Mask R-CNN.

Impact

The ICCV abstract promised code; the arXiv v3 abstract points to Detectron, FAIR’s detection platform written in Python on Caffe2 (Girshick et al., 2018), whose public GitHub repository begins with an initial commit dated 22 January 2018. Detectron was later deprecated in favor of Detectron2 (Wu et al., 2019), a rewrite in PyTorch that includes Mask R-CNN.

The PAMI Technical Committee’s list of ICCV paper awards gives Mask R-CNN as the 2017 Best Paper (Marr Prize).

Later papers treat Mask R-CNN as the reference system. Panoptic FPN (Kirillov et al., 2019) names it with FCN as the simple baselines that had aided progress, states that Mask R-CNN with FPN had been the foundation of all top entries in recent recognition challenges, and builds a panoptic segmentation model by adding a semantic segmentation branch to it. Cai and Vasconcelos (2021) call it arguably the most popular instance segmentation method before extending their Cascade R-CNN with a mask branch. Mask Scoring R-CNN (Huang et al., 2019) and PointRend (Kirillov et al., 2020) each build on it and target one of its parts, the mask score and the mask resolution respectively.

Limitations

The paper names some of its own limits: the design is not tuned for speed, the pose experiments use minimal domain knowledge, the keypoint branch does not help the other tasks, and, in arXiv v3, Cityscapes results are limited by little training data for some classes. Later work identified others:

  • Mask score. Huang et al. (2019) point out that the mask’s score is the box classification confidence, which does not measure mask quality.
  • Fixed mask grid. Kirillov et al. (2020) note that Mask R-CNN-style heads predict masks on a 28×2828 \times 28 grid whatever the object size, which gives over-smoothed masks for large objects.
  • Region-bound outputs. In our reading, because each mask is predicted inside its own box, masks can overlap; Panoptic FPN, which needs a non-overlapping labeling, resolves such overlaps with a separate post-processing step.

What Came After

Mask Scoring R-CNN learned a mask-IoU score, PointRend refined masks at uncertain points, and Cascade Mask R-CNN added a mask branch to a multi-stage detector; the Mask R-CNN article reports their results. Other work dropped the region-based design: the instance segmentation article covers one-stage alternatives such as YOLACT and SOLO.

Related

  • Mask R-CNN

    An instance segmentation model that adds a small fully convolutional mask branch and RoIAlign to the Faster R-CNN detector, predicting a box, a class and a binary mask for every detected object.

  • Instance Segmentation

    Detecting every object of a fixed set of "thing" classes and giving each one its own pixel mask, class label and confidence score, so that two touching cars come out as two cars rather than one car-colored region.

  • Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

    The 2015 NIPS paper by Ren, He, Girshick and Sun that replaced hand-designed region proposals with a region proposal network sharing the detector's convolutional features, making proposals nearly free and the whole detector one network.

  • Panoptic Segmentation

    Giving every pixel of an image a class label and, for countable objects, an instance id, so that stuff such as sky and road and things such as cars and people are segmented together without overlaps, and scoring the result with panoptic quality (PQ).

  • Feature Pyramid Network

    A network design that turns a convolutional backbone's feature maps at strides 4 to 32 into a pyramid with strong semantic features at every scale, through a top-down pathway and lateral connections, so a detector can find small and large objects from one image scale.

  • Two-Stage Object Detectors

    Object detectors that first propose a few thousand class-agnostic candidate regions and then classify and refine each one with features pooled from a shared CNN feature map, the design line of R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN.

References

  1. He, K., Gkioxari, G., Dollár, P. & Girshick, R. (2017). Mask R-CNN. IEEE International Conference on Computer Vision (ICCV), 2980–2988.
  2. He, K., Gkioxari, G., Dollár, P. & Girshick, R. (2018). Mask R-CNN. arXiv:1703.06870v3 (version with appendices on Cityscapes and enhanced COCO results).
  3. He, K., Gkioxari, G., Dollár, P. & Girshick, R. (2020). Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2), 386–397.
  4. Ren, S., He, K., Girshick, R. & Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28 (NIPS).
  5. Long, J., Shelhamer, E. & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3431–3440.
  6. Dai, J., He, K. & Sun, J. (2016). Instance-Aware Semantic Segmentation via Multi-task Network Cascades. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3150–3158.
  7. Li, Y., Qi, H., Dai, J., Ji, X. & Wei, Y. (2017). Fully Convolutional Instance-Aware Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4438–4446.
  8. Kirillov, A., Girshick, R., He, K. & Dollár, P. (2019). Panoptic Feature Pyramid Networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6392–6401.
  9. Huang, Z., Huang, L., Gong, Y., Huang, C. & Wang, X. (2019). Mask Scoring R-CNN. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6402–6411.
  10. Kirillov, A., Wu, Y., He, K. & Girshick, R. (2020). PointRend: Image Segmentation as Rendering. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9796–9805.
  11. Cai, Z. & Vasconcelos, N. (2021). Cascade R-CNN: High Quality Object Detection and Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5), 1483–1498.
  12. Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P. & He, K. (2018). Detectron. Source code.
  13. Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y. & Girshick, R. (2019). Detectron2. Source code.
  14. IEEE Computer Society Technical Committee on Pattern Analysis and Machine Intelligence. ICCV Paper Awards.