Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

“Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks” by Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun of Microsoft Research was posted to arXiv in June 2015 and published in Advances in Neural Information Processing Systems 28 (NIPS 2015). It introduced the region proposal network (RPN), a small fully convolutional network that predicts object proposals from the same feature map the detector uses, so that proposal generation, until then a separate CPU algorithm, became nearly free. An extended version appeared in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2017. The paper received the NeurIPS 2025 Test of Time Paper Award. The detector, with its architecture, training details and full results tables, is covered in the Faster R-CNN article.

Problem

In 2015 the strongest object detectors were region-based: an external algorithm proposed a few thousand candidate boxes, and a convolutional network classified and refined each one. SPPnet and Fast R-CNN (Girshick, 2015) had made the network part cheap by computing convolutional features once per image and pooling a feature for each region from the shared map. The authors’ point of departure was that this exposed the proposals as the bottleneck. Selective search, which greedily merges superpixels using hand-engineered low-level features, took about 2 seconds per image on a CPU (1.51 s on average in the paper’s own timing table), an order of magnitude more than the detection network, and EdgeBoxes (Zitnick & Dollár, 2014), the best speed–quality tradeoff they knew of, still took 0.2 seconds, about as long as detection.

The authors granted that this compared GPU detectors with CPU proposal methods, and that porting selective search to a GPU was an obvious fix, but argued that a reimplementation would still ignore the downstream detector and miss the chance to share computation with it.

Contribution

The paper’s central observation is that the feature maps a region-based detector already computes can also generate proposals. It contributes:

• The region proposal network. A few convolutional layers on the shared feature map predict, at every position, object/not-object scores and box offsets for several candidate boxes. Because the backbone is shared, the authors measured the marginal cost of proposals at about 10 ms per image.
• Anchors. Each position predicts boxes relative to $k$ reference boxes of different scales and aspect ratios, which the paper names anchors; it uses $k = 9$.
• A training scheme for shared features. Alternating optimization that gives the RPN and the Fast R-CNN detector one set of convolutional layers.
• Evidence that learned proposals beat hand-designed ones in accuracy as well as speed, on PASCAL VOC 2007 and 2012, with 300 proposals per image instead of about 2,000.

Method

How the paper presents anchors. The NIPS version introduces translation-invariant anchors and argues for them against MultiBox, whose 800 k-means anchors are not translation invariant, so its output layer needs $(4 + 1) \times 800$ outputs; the RPN applies the same nine anchors and the same weights at every position, so its output layer has $(4 + 2) \times 9$, and its proposal layers have 2.4 million parameters against MultiBox’s 27 million with GoogLeNet. The authors expected the smaller layer to overfit less on a dataset the size of PASCAL VOC.

The journal version adds a second framing, with a new Fig. 1 (Fig. 1 of the NIPS version shows the RPN head). It compares three ways to handle objects of many sizes: pyramids of images and feature maps, with the classifier run at every scale; pyramids of filters of several sizes on one feature map, as in DPM’s aspect-ratio-specific models; and the authors’ own pyramid of regression references, in which one image scale, one feature map and one filter size suffice because the range of scales and shapes is carried by the anchors. The authors present this as the key to sharing features at no extra cost for scale; the anchor boxes article covers anchors in general.

Training as published. The authors did not simply train the RPN and Fast R-CNN as one network, because Fast R-CNN training assumed fixed proposals and it was unclear whether training would converge with the proposals changing. The NIPS paper therefore uses a pragmatic four-step alternating optimization, training the RPN, then a detector on its proposals, then fine-tuning each network’s own layers on top of the detector’s frozen convolutional layers, and calls joint optimization an interesting question for future work. The journal version reports that an approximate joint training that ignores gradients with respect to the proposal coordinates gave close results with 25–50% less training time, although all experiments in the paper still used alternating training.

Results

The NIPS experiments use PASCAL VOC only, with the ZF network for ablations and VGG-16 for the main results; the paper reports detection mAP, treating proposal recall as a diagnostic proxy. The numbers below are from the NIPS version, on VOC 2007 test.

• Proposal methods compared. With ZF and Fast R-CNN trained and tested on the same proposals, selective search gave 58.7% mAP with 2,000 proposals, EdgeBoxes 58.6% with 2,000, and the shared RPN 59.9% with at most 300.
• Sharing. Stopping after the second training step, with unshared features, gave 58.7%; the authors attribute the gain from sharing to the RPN being fine-tuned on detector-tuned features in step 3.
• Ablations of the proposal step, with a fixed detector trained on selective search: the top 100 RPN proposals still gave 55.1%, and 6,000 proposals without non-maximum suppression 55.2%, against 56.8% for the top 300 after NMS, so NMS did not hurt. Without the classification scores, 100 random proposals gave 44.6%; without box regression, using the anchors themselves, 52.1%.
• Deeper networks. An RPN trained with VGG-16 raised the same fixed detector from 56.8% to 59.2%: unlike a fixed algorithm, learned proposals improve with the network.
• Main results and timing. With VGG-16 the system reached 69.9% mAP trained on VOC 2007 and 73.2% trained on VOC 2007 and 2012, and 70.4% on VOC 2012 test, all above selective search with Fast R-CNN. On a K40 GPU it took 198 ms per image (5 frames per second), against 1,830 ms with selective search; with ZF it reached 17 frames per second.
• One stage or two. An OverFeat-like one-stage variant, Fast R-CNN applied to dense sliding windows of the nine anchor shapes, reached 53.9% against 58.7% for the two-stage cascade.

The journal version adds MS COCO experiments with VGG-16 (42.7% mAP@0.5 and 21.9% mAP@[.5, .95] on test-dev), which the authors read as better localization at high IoU than their Fast R-CNN baseline, and COCO pretraining that raised VOC 2007 mAP to 78.8%; the Faster R-CNN article tabulates these results.

Impact

In the residual network paper (He et al., 2016), replacing VGG-16 with ResNet-101 in Faster R-CNN raised COCO validation mAP@[.5, .95] from 21.2% to 27.2%, and the resulting systems won ImageNet and COCO detection at the 2015 competitions (see the ResNet paper). The journal version of Faster R-CNN lists Faster R-CNN and the RPN as the basis of first-place entries in ImageNet detection, ImageNet localization, COCO detection and COCO segmentation, and mentions use in a commercial system at Pinterest.

The feature pyramid network (Lin et al., 2017) adapted the RPN and Fast R-CNN to a multi-level feature pyramid, and Mask R-CNN (He et al., 2017) describes itself as extending Faster R-CNN with a mask branch. Faster R-CNN and the RPN are among the algorithms implemented in FAIR’s Detectron, and TorchVision’s Faster R-CNN models are based on the paper. In tracking, SORT (Bewley et al., 2016) used Faster R-CNN detections to show how much detection quality matters for tracking-by-detection (see the SORT paper), and the MOT17 benchmark ships Faster R-CNN detections as one of its three public detection sets (see MOTChallenge).

In November 2025 the NeurIPS communications chairs announced that the paper had received the NeurIPS 2025 Test of Time Paper Award, which recognizes papers published at the conference ten years earlier. The announcement singles out its near-real-time speed and its replacement of selective search and other handcrafted proposals with a fully learnable two-stage pipeline.

Limitations

• Not real-time with VGG-16. The title says “towards”: 5 frames per second with VGG-16, and 17 with the smaller ZF network.
• Stage-wise training. The published scheme runs four training stages in sequence; the journal’s approximate joint training ignores the gradient through the proposal boxes.
• One coarse feature map. All proposals come from a single map with a stride of 16 pixels, about 10 pixels on a typical PASCAL image; the authors note that a smaller stride might improve accuracy.
• Hand-set components. Anchor scales and ratios, IoU thresholds for labels and the NMS threshold are design choices; the journal version’s ablation found three scales with one aspect ratio as good as the default nine anchors on VOC 2007.

What Came After

One-stage detectors such as YOLO and SSD dropped the second stage and predicted classes directly on dense positions, while two-stage successors such as R-FCN, Cascade R-CNN and Mask R-CNN refined or extended the head (see two-stage object detectors). Feature pyramids answered the single-map limitation, and DETR later removed both anchors and NMS.