Recognition & Detection

Faster R-CNN

A two-stage object detector in which a region proposal network, sharing the detector's convolutional features, replaces hand-designed region proposals, making proposal generation nearly free and the whole detector one network.

intermediate

Faster R-CNN is a two-stage object detector introduced by Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun at NIPS 2015, with an extended journal version in IEEE TPAMI (2017). It adds a region proposal network (RPN), a small fully convolutional head that predicts object proposals from the same convolutional feature map the detector uses, to the Fast R-CNN detector. Proposals, until then computed by a separate algorithm on the CPU, became a by-product of the network, about 10 ms per image. The two-stage object detectors article covers the R-CNN family’s design line, including RoI pooling and RoIAlign; this article covers what Faster R-CNN added.

Problem Addressed

By 2015, Fast R-CNN with VGG-16 classified and refined 2,000 proposals in about 0.3 s on a GPU, but the proposals came from selective search (Uijlings et al., 2013), which merges superpixels by low-level cues and took about 1.5 s per image on a CPU in Ren et al.’s measurement. Reimplementing selective search on a GPU, the authors argued, would still ignore the downstream detector and miss the chance to share its computation. Their alternative was to compute proposals with a network that shares the detector’s convolutional layers.

Architecture

The image, rescaled to a shorter side of 600 pixels, passes once through a convolutional backbone. The RPN slides over the last shared feature map and outputs scored proposals; Fast R-CNN pools a feature for each from the same map, then classifies and refines it.

Component Configuration in the paper Output
Backbone ImageNet-pretrained ZF (5 shareable conv layers) or VGG-16 (13), total stride 16 512 channels at about H/16×W/16H/16 \times W/16 positions (VGG-16)
RPN conv 3×33 \times 3 convolution with ReLU 256-d (ZF) or 512-d (VGG-16) per position
RPN cls 1×11 \times 1 convolution 2k2k scores per position, k=9k = 9 anchors
RPN reg 1×11 \times 1 convolution 4k4k box offsets per position
Proposal layer Decode, clip to image, NMS at IoU 0.7, keep top NN 2,000 proposals for training, 300 at test
Detection head Fast R-CNN: RoI pooling to 7×77 \times 7, two fully connected layers, two sibling outputs K+1K+1 class scores, 4K4K box offsets per proposal

Backbone

The paper uses the “fast” ZF net (Zeiler & Fergus, 2014) and VGG-16 (Simonyan & Zisserman, 2015), both with stride 16 at the last convolutional layer; the receptive field of the RPN’s 3×33 \times 3 window is 171 pixels for ZF and 228 for VGG-16.

Region Proposal Network

At every position of the feature map, the 3×33 \times 3 convolution maps the window to a 512-dimensional vector (256 for ZF), and two sibling 1×11 \times 1 convolutions predict, for each of kk reference boxes called anchors, two object/not-object scores and four box offsets. Three scales (1282128^2, 2562256^2, 5122512^2 pixels) and three aspect ratios give k=9k = 9; anchors are labeled by their IoU with the ground truth (positive above 0.7 or best for some ground-truth box, negative below 0.3). The anchor boxes article covers this design, the labeling rule and the box parameterization. The proposal layers are translation invariant and small, the authors note: 2.4 million parameters for VGG-16, against 27 million for MultiBox with GoogLeNet.

Detection Head

The detector is Fast R-CNN unchanged (Girshick, 2015): RoI pooling extracts a 7×77 \times 7 grid from the shared map for each proposal, VGG-16’s fully connected layers process it, and sibling layers output a softmax over KK classes plus background and class-specific box offsets.

Training

RPN loss. For a mini-batch of anchors indexed by ii, with predicted object probability pip_i, label pi∗∈{0,1}p_i^* \in \{0, 1\}, predicted offsets tit_i and target offsets ti∗t_i^*,

L({pi},{ti})=1Ncls∑iLcls(pi,pi∗)+λ1Nreg∑ipi∗ Lreg(ti,ti∗),L(\{p_i\}, \{t_i\}) = \frac{1}{N_{\text{cls}}} \sum_i L_{\text{cls}}(p_i, p_i^*) + \lambda \frac{1}{N_{\text{reg}}} \sum_i p_i^* \, L_{\text{reg}}(t_i, t_i^*),

where LclsL_{\text{cls}} is the two-class log loss and Lreg(ti,ti∗)=∑jsmooth⁡L1(ti,j−ti,j∗)L_{\text{reg}}(t_i, t_i^*) = \sum_j \operatorname{smooth}_{L_1}(t_{i,j} - t^*_{i,j}), the robust loss of Fast R-CNN; the factor pi∗p_i^* restricts regression to positive anchors. In the released code, Ncls=256N_{\text{cls}} = 256 (the mini-batch size), Nreg≈2,400N_{\text{reg}} \approx 2{,}400 (the number of anchor positions) and λ=10\lambda = 10, which weights the two terms roughly equally. The NIPS version gives these values in a footnote on its early implementation; the journal version adds that VOC 2007 mAP changed by about 1 point for λ\lambda from 1 to 100 and that the normalization is not required.

Sampling. Each RPN mini-batch comes from one image: 256 anchors sampled at random, up to half of them positive, padded with negatives. Anchors crossing the image border are ignored in training; the authors report that training did not converge otherwise. On PASCAL VOC, SGD ran at learning rate 0.001 for 60k mini-batches, then 0.0001 for 20k.

Sharing features. Trained separately, the RPN and Fast R-CNN would learn different convolutional layers. The NIPS paper uses 4-step alternating training: (1) train the RPN from an ImageNet-pretrained model; (2) train Fast R-CNN, also from an ImageNet model, on the step-1 proposals; (3) initialize the RPN from the detector, freeze the shared layers and fine-tune only the RPN’s own layers; (4) with the shared layers still fixed, fine-tune Fast R-CNN’s own layers (the fully connected layers in the NIPS wording). Joint optimization was left as future work, because Fast R-CNN training assumed fixed proposals.

The journal version describes three options and still uses alternating training for all its experiments. Approximate joint training merges both networks: each SGD step generates proposals in the forward pass, treats them as fixed inputs to Fast R-CNN, and combines the gradients of both losses in the shared layers, ignoring the gradient with respect to the proposal coordinates. The authors found it gave close results in 25–50% less training time. Non-approximate joint training would need RoI pooling that is differentiable in the box coordinates, which they leave to other work. py-faster-rcnn ships approximate joint training as its “end2end” script, about 1.5 times faster than alternating optimization for VGG-16 with similar accuracy, according to its README.

COCO. For MS COCO (journal version only), training used 8 GPUs, a fourth anchor scale of 64264^2 for small objects, and Fast R-CNN negatives from IoU [0,0.5)[0, 0.5) instead of [0.1,0.5)[0.1, 0.5).

Inference

Every anchor becomes a box through its predicted offsets; boxes are clipped to the image, and non-maximum suppression at IoU 0.7 on the objectness scores leaves about 2,000 per image. Fast R-CNN was trained on these 2,000 and tested on the top 300. py-faster-rcnn adds two details: proposals smaller than 16 pixels (at original image scale) are dropped, and at test time only the 6,000 highest-scoring boxes enter NMS. The detector’s class-specific boxes then go through a second, per-class NMS (0.3 in py-faster-rcnn).

On a K40 GPU with VGG-16, the authors measured 198 ms per image (convolutions 141 ms, proposals 10 ms, region-wise layers 47 ms), or 5 frames per second, against 1,830 ms for selective search with Fast R-CNN; with ZF, 59 ms, or 17 frames per second.

Example: the proposal stage on a synthetic image

The NumPy code runs the proposal layer on a 600×800600 \times 800 gray image holding three objects (ellipse, rectangle, triangle), with ground-truth boxes read from their masks. No RPN is trained: a stand-in gives plausible outputs, objectness rising with an anchor’s best IoU and offsets pointing toward the overlapped object, both noisy. The code follows py-faster-rcnn’s steps and measures recall of the ground truth by the top NN proposals over 40 such images, with ablations like the paper’s: no NMS, no offsets, no scores (random anchors).

import numpy as np

H, W, STRIDE = 600, 800, 16                      # image rescaled to shorter side 600; VGG-16 stride

def make_anchors(scales=(128, 256, 512), ratios=(0.5, 1, 2)):
    """k = 9 anchors centered at (0, 0) as (x1, y1, x2, y2); area s^2, ratio h / w."""
    a = [[-s / np.sqrt(r) / 2, -s * np.sqrt(r) / 2, s / np.sqrt(r) / 2, s * np.sqrt(r) / 2]
         for r in ratios for s in scales]
    return np.array(a)

def grid_anchors():
    cy, cx = np.mgrid[0:H // STRIDE, 0:W // STRIDE] * STRIDE + STRIDE / 2       # 37 x 50 positions
    c = np.stack([cx.ravel(), cy.ravel(), cx.ravel(), cy.ravel()], 1)
    return (c[:, None, :] + make_anchors()[None]).reshape(-1, 4)

def iou(A, B):
    lt, rb = np.maximum(A[:, None, :2], B[None, :, :2]), np.minimum(A[:, None, 2:], B[None, :, 2:])
    inter = np.prod(np.clip(rb - lt, 0, None), 2)
    area = lambda X: np.prod(X[:, 2:] - X[:, :2], 1)
    return inter / (area(A)[:, None] + area(B)[None] - inter)

def encode(g, a):                                  # targets t* of the anchor-relative parameterization
    gw, gh, aw, ah = g[:, 2] - g[:, 0], g[:, 3] - g[:, 1], a[:, 2] - a[:, 0], a[:, 3] - a[:, 1]
    return np.stack([((g[:, 0] + g[:, 2]) - (a[:, 0] + a[:, 2])) / 2 / aw,
                     ((g[:, 1] + g[:, 3]) - (a[:, 1] + a[:, 3])) / 2 / ah,
                     np.log(gw / aw), np.log(gh / ah)], 1)

def decode(t, a):
    aw, ah = a[:, 2] - a[:, 0], a[:, 3] - a[:, 1]
    cx, cy = (a[:, 0] + a[:, 2]) / 2 + t[:, 0] * aw, (a[:, 1] + a[:, 3]) / 2 + t[:, 1] * ah
    w, h = aw * np.exp(t[:, 2]), ah * np.exp(t[:, 3])
    return np.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], 1)

def nms(boxes, thr):
    order, keep = np.arange(len(boxes)), []
    while order.size:
        i = order[0]
        keep.append(i)
        order = order[1:][iou(boxes[i:i + 1], boxes[order[1:]])[0] <= thr]
    return np.array(keep, int)

def synthetic_image(rng):
    """Gray image with 3 objects (ellipse, rectangle, triangle); boxes are read from the masks."""
    yy, xx = np.mgrid[0:H, 0:W] + 0.5
    img = rng.normal(0.3, 0.05, (H, W))
    boxes = []
    for kind in range(3):
        w, h = rng.uniform(70, 330), rng.uniform(70, 330)
        cx, cy = rng.uniform(w / 2, W - w / 2), rng.uniform(h / 2, H - h / 2)
        u, v = (xx - cx) / (w / 2), (yy - cy) / (h / 2)
        m = [u ** 2 + v ** 2 <= 1, (np.abs(u) <= 1) & (np.abs(v) <= 1),
             (v <= 1) & (np.abs(u) <= (v + 1) / 2)][kind]
        img[m] = 0.6 + 0.15 * kind
        r, c = np.nonzero(m)
        boxes.append([c.min(), r.min(), c.max() + 1, r.max() + 1])
    return img, np.array(boxes, float)

def fake_rpn_outputs(anchors, gt, rng):
    """Stand-in for a trained RPN: objectness rises with an anchor's best IoU, deltas point
    toward that object for anchors that overlap it, all with noise."""
    ov = iou(anchors, gt)
    best, j = ov.max(1), ov.argmax(1)
    logit = 10 * (best - 0.45) + rng.normal(0, 1.5, len(anchors))
    score = 1 / (1 + np.exp(-logit))
    t = rng.normal(0, [0.08, 0.08, 0.15, 0.15], (len(anchors), 4))     # noise for every anchor
    near = best > 0.2
    t[near] += encode(gt[j[near]], anchors[near])                   # imperfect regression
    return score, t

def propose(anchors, score, t, pre_nms=6000, nms_thr=0.7, post_nms=300, min_size=16):
    boxes = decode(t, anchors)                                       # 1. apply deltas to anchors
    boxes[:, 0::2] = boxes[:, 0::2].clip(0, W)                       # 2. clip to the image
    boxes[:, 1::2] = boxes[:, 1::2].clip(0, H)
    ok = ((boxes[:, 2] - boxes[:, 0]) >= min_size) & ((boxes[:, 3] - boxes[:, 1]) >= min_size)
    boxes, score = boxes[ok], score[ok]                              # 3. drop tiny boxes
    order = np.argsort(-score)[:pre_nms]                             # 4.-5. sort, keep top 6000
    keep = order if nms_thr is None else order[nms(boxes[order], nms_thr)]  # 6. NMS 0.7
    keep = keep[:post_nms]                                           # 7. top N
    return boxes[keep], score[keep], ok.sum()

def recall(props, gt, thr):
    return (iou(gt, props).max(1) >= thr).mean() if len(props) else 0.0

anchors = grid_anchors()
print("anchors:", len(anchors), "=", H // STRIDE, "x", W // STRIDE, "x 9")

rng = np.random.default_rng(0)
img, gt = synthetic_image(rng)
score, t = fake_rpn_outputs(anchors, gt, rng)
props, s, n_ok = propose(anchors, score, t, post_nms=2000)
print("ground truth:\n", gt.astype(int))
print(f"after clip and size filter: {n_ok}; after NMS 0.7: {len(props)}")
print("top 5 proposals (score, box):")
for b, p in zip(props[:5], s[:5]):
    print(f"  {p:.3f}  {np.round(b).astype(int)}  best IoU with GT {iou(b[None], gt).max():.2f}")

# Recall of ground truth over many synthetic images, for top-N proposals.
Ns = (10, 50, 300)
rows = {k: [] for k in ("sort + NMS 0.7", "sort, no NMS", "no reg (anchors)", "random anchors")}
for _ in range(40):
    img, gt = synthetic_image(rng)
    score, t = fake_rpn_outputs(anchors, gt, rng)
    p, _, _ = propose(anchors, score, t)
    raw, _, _ = propose(anchors, score, t, nms_thr=None)               # NMS switched off
    noreg, _, _ = propose(anchors, score, np.zeros_like(t))             # deltas switched off
    rnd = anchors[rng.choice(len(anchors), 300, replace=False)]
    for k, P in zip(rows, (p, raw, noreg, rnd)):
        rows[k].append([[recall(P[:n], gt, thr) for n in Ns] for thr in (0.5, 0.7)])
print(f"\nrecall of 120 objects in 40 images      IoU 0.5: N=" + "/".join(map(str, Ns))
      + "     IoU 0.7: N=" + "/".join(map(str, Ns)))
for k, v in rows.items():
    r = np.mean(v, 0)
    print(f"{k:22s}  " + "  ".join(f"{x:.2f}" for x in r[0]) + "        "
          + "  ".join(f"{x:.2f}" for x in r[1]))
anchors: 16650 = 37 x 50 x 9
ground truth:
 [[195 342 381 461]
 [563 310 692 598]
 [ 14  68 136 327]]
after clip and size filter: 16650; after NMS 0.7: 1370
top 5 proposals (score, box):
  0.997  [182 352 370 466]  best IoU with GT 0.78
  0.991  [194 323 351 472]  best IoU with GT 0.68
  0.989  [196 333 397 448]  best IoU with GT 0.77
  0.988  [521 324 679 600]  best IoU with GT 0.65
  0.984  [574 302 706 570]  best IoU with GT 0.73

recall of 120 objects in 40 images      IoU 0.5: N=10/50/300     IoU 0.7: N=10/50/300
sort + NMS 0.7          0.94  1.00  1.00        0.76  1.00  1.00
sort, no NMS            0.87  0.99  1.00        0.69  0.95  1.00
no reg (anchors)        0.88  0.97  0.98        0.44  0.57  0.57
random anchors          0.03  0.22  0.66        0.01  0.05  0.12

NMS at 0.7 leaves 1,370 of the 6,000 top-scoring boxes, the order of the paper’s 2,000. Three of the top five proposals cover one object; without NMS such near-duplicates fill the top places, and recall at N=10N = 10 falls from 0.94 to 0.87. Without the offsets, recall at IoU 0.5 stays high but at IoU 0.7 stalls near 0.57, since the anchors’ fixed shapes rarely fit an object tightly; the paper’s no-reg ablation likewise lost VOC mAP (52.1% against 56.8%). Without scores, 300 random anchors reach a recall of only 0.66 at IoU 0.5 and 0.12 at 0.7; the paper’s no-cls ablation fell to 44.6% mAP with 100 random proposals. These numbers come from a simulated RPN, so they show how the pipeline behaves, not how accurate a trained RPN is; by our reasoning, an image with dozens of objects, unlike these three, needs a larger NN.

The RPN head itself is three layers. In PyTorch (run with version 2.14.1), applied to a random tensor shaped like the VGG-16 conv5_3 map of a 600×800600 \times 800 image (37×5037 \times 50 with PyTorch’s default floor rounding in pooling; Caffe, which rounds pooling outputs up, gives 38×5038 \times 50):

import torch
import torch.nn as nn

class RPNHead(nn.Module):
    """3x3 'sliding window' conv + ReLU, then sibling 1x1 convs: cls (2k) and reg (4k)."""
    def __init__(self, in_ch=512, mid_ch=512, k=9):
        super().__init__()
        self.conv = nn.Conv2d(in_ch, mid_ch, 3, padding=1)
        self.cls = nn.Conv2d(mid_ch, 2 * k, 1)       # object / not-object score per anchor
        self.reg = nn.Conv2d(mid_ch, 4 * k, 1)       # (t_x, t_y, t_w, t_h) per anchor
        for m in (self.conv, self.cls, self.reg):     # new layers: zero-mean Gaussian, std 0.01
            nn.init.normal_(m.weight, std=0.01)
            nn.init.zeros_(m.bias)

    def forward(self, f):
        h = torch.relu(self.conv(f))
        return self.cls(h), self.reg(h)

torch.manual_seed(0)
conv5 = torch.randn(1, 512, 37, 50)                  # VGG-16 conv5_3 map of a 600 x 800 image
head = RPNHead()
cls, reg = head(conv5)
k = 9
print("conv5_3 features:", tuple(conv5.shape))
print("cls logits:      ", tuple(cls.shape), "->", tuple(cls.view(1, 2, k, 37, 50).shape), "(bg/fg, anchor, y, x)")
print("reg deltas:      ", tuple(reg.shape), "->", tuple(reg.view(1, k, 4, 37, 50).shape), "(anchor, t, y, x)")
print("anchors:", 37 * 50 * k)
for name, ch in (("VGG-16 (512-d)", 512), ("ZF (256-d)", 256)):
    n = sum(p.numel() for p in RPNHead(ch, ch).parameters())
    print(f"RPN head parameters, {name}: {n:,}")
print("PyTorch", torch.__version__)
conv5_3 features: (1, 512, 37, 50)
cls logits:       (1, 18, 37, 50) -> (1, 2, 9, 37, 50) (bg/fg, anchor, y, x)
reg deltas:       (1, 36, 37, 50) -> (1, 9, 4, 37, 50) (anchor, t, y, x)
anchors: 16650
RPN head parameters, VGG-16 (512-d): 2,387,510
RPN head parameters, ZF (256-d): 603,958
PyTorch 2.14.1

The 2.39 million parameters of the VGG-16 head match the paper’s 2.4 million, which omits biases; the ZF count assumes the 256-channel conv5 of py-faster-rcnn’s ZF model. The channel layout follows py-faster-rcnn, which reads the first kk score channels as background; the paper notes that kk logistic outputs could replace its two-class softmax.

Performance

All results are from Ren et al. with VGG-16, 300 proposals at test time and single-scale testing. PASCAL VOC results are mAP at IoU 0.5; “07+12” is VOC 2007 trainval plus VOC 2012 trainval, and “07++12” adds VOC 2007 test (see PASCAL VOC).

Test set Training data Metric Faster R-CNN Selective search + Fast R-CNN Version
VOC 2007 test VOC 2007 trainval mAP 69.9 66.9 NIPS and journal
VOC 2007 test 07+12 mAP 73.2 70.0 NIPS and journal
VOC 2007 test COCO trainval only mAP 76.1 – Journal
VOC 2007 test COCO, then 07+12 mAP 78.8 – Journal
VOC 2012 test VOC 2012 trainval mAP 67.0 65.7 NIPS and journal
VOC 2012 test 07++12 mAP 70.4 68.4 NIPS and journal
VOC 2012 test COCO, then 07++12 mAP 75.9 – Journal
COCO val COCO train AP@0.5 / AP@[.5:.95] 41.5 / 21.2 38.6 / 18.9 Journal
COCO test-dev COCO train AP@0.5 / AP@[.5:.95] 42.1 / 21.5 39.3 / 19.3 Journal
COCO test-dev COCO trainval AP@0.5 / AP@[.5:.95] 42.7 / 21.9 – Journal

The VOC baselines are the selective-search Fast R-CNN results reported by Girshick (2015); the COCO baseline is the authors’ own Fast R-CNN implementation. On COCO test-dev, the authors read the gain over Fast R-CNN (2.8 points at AP@0.5, 2.2 at AP@[.5:.95]) as evidence that RPN proposals improve localization at higher IoU thresholds. Pretraining on COCO and fine-tuning on VOC added 5.6 points on VOC 2007.

In ZF ablations on VOC 2007 (both versions), sharing features gave 59.9% against 58.7% unshared, and a one-stage variant, Fast R-CNN on dense sliding windows of the nine anchor shapes instead of proposals, reached 53.9% against 58.7% for the two-stage system.

Strengths

  • Nearly free proposals. Sharing the backbone made the proposal step 10 ms with VGG-16, and the detector one network.
  • Fewer, better proposals. 300 learned proposals outperformed 2,000 from selective search, and recall declined little from 2,000 to 300 proposals, which the authors attribute to the cls scores.
  • Benefits from better backbones. Learned proposals improve with deeper networks, as the ResNet results below show.
  • Extensible head. Mask R-CNN later added masks and keypoints to the per-region head.

Limitations

  • Speed. At 5 frames per second with VGG-16 it was not real-time, and the per-region layers run hundreds of times per image; one-stage detectors such as YOLO and SSD avoid this cost.
  • Single-scale features. All anchors come from one stride-16 map, which suits small objects poorly.
  • Hand-tuned components. Anchor shapes, IoU thresholds, sampling ratios and two NMS stages are set by hand; DETR later removed anchors and NMS.
  • Complex training. The 4-step scheme runs four training stages in sequence; approximate joint training simplifies it but ignores gradients through the proposal boxes.

Historical Importance

In our reading, Faster R-CNN completed the shift R-CNN began in 2014: every stage of a region-based detector was now a network trained on detection data. Its anchors were taken up by SSD, YOLOv2 and RetinaNet, whose papers cite the RPN for them.

He et al. (2016) used Faster R-CNN as the detector in their ResNet experiments; replacing VGG-16 with ResNet-101 raised COCO val AP@[.5:.95] from 21.2% to 27.2% (see ResNet paper). Their Faster R-CNN-based systems won first place in ImageNet detection and COCO detection in the ILSVRC and COCO 2015 competitions, and the Faster R-CNN journal version adds that the RPN was part of the winning ImageNet localization and COCO segmentation entries. Faster R-CNN also supplied the detections for SORT’s tracking-by-detection (see SORT paper).

The paper received the NeurIPS 2025 Test of Time Paper Award, given to NeurIPS papers from ten years earlier.

Variants

  • ResNet backbones. Because ResNet has no hidden fully connected layers, He et al. (2016) shared conv1 to conv4 (stride 16) between RPN and detector and ran conv5 on each RoI after pooling, a configuration Detectron2’s model zoo calls C4.
  • Feature pyramid network. Lin et al. (2017) run the RPN on every level of a feature pyramid, with one anchor scale per level, and pool each RoI from a level chosen by its size. With ResNet-101, they reported 36.2 AP on COCO test-dev. All of TorchVision’s Faster R-CNN builders use FPN.
  • Mask R-CNN. He et al. (2017) added a mask branch and replaced RoI pooling with RoIAlign (see two-stage object detectors). Faster R-CNN with RoIAlign alone reached 37.3 box AP on COCO test-dev with ResNet-101-FPN, against the 36.2 that Lin et al. reported with RoI pooling, a gain the Mask R-CNN authors attribute to RoIAlign.
  • R-FCN and Cascade R-CNN change the per-region head and its training; the two-stage object detectors article covers both.

Implementations

The official code is in MATLAB (ShaoqingRen/faster_rcnn); py-faster-rcnn reimplements it in Python with Caffe, with similar but not identical mAP, according to its README. Detectron (Girshick et al., 2018), written for Caffe2 and now deprecated, implements Faster R-CNN with VGG16, ResNet, ResNeXt and FPN backbones; its PyTorch successor Detectron2 (Wu et al., 2019) has C4, dilated-C5 and FPN variants in its model zoo. TorchVision provides builders such as fasterrcnn_resnet50_fpn and fasterrcnn_mobilenet_v3_large_fpn; check the documentation of the installed version, as the API changes between releases.

Related

  • Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

    The 2015 NIPS paper by Ren, He, Girshick and Sun that replaced hand-designed region proposals with a region proposal network sharing the detector's convolutional features, making proposals nearly free and the whole detector one network.

  • Anchor Boxes

    Reference boxes of fixed sizes and aspect ratios, tiled at every feature-map position, that a detector classifies and refines, and that decide which of its predictions are trained on which objects.

  • Non-Maximum Suppression

    The post-processing step that removes duplicate detections by repeatedly keeping the highest-scoring box and discarding the boxes that overlap it beyond an IoU threshold, with its soft, distance-aware and learned variants.

  • Feature Pyramid Network

    A network design that turns a convolutional backbone's feature maps at strides 4 to 32 into a pyramid with strong semantic features at every scale, through a top-down pathway and lateral connections, so a detector can find small and large objects from one image scale.

  • YOLO

    A family of real-time one-stage object detectors, begun by Redmon et al. in 2016, that predict every box and class probability of an image in a single network pass; later versions from several groups added anchors, multi-scale heads, anchor-free heads and NMS-free training.

  • COCO

    Microsoft's Common Objects in Context, a dataset of everyday scenes with per-instance segmentation masks for 80 object categories, plus keypoints, captions, stuff and panoptic labels, whose evaluation protocol became the standard benchmark for object detection.

References

  1. Ren, S., He, K., Girshick, R. & Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28 (NIPS).
  2. Ren, S., He, K., Girshick, R. & Sun, J. (2017). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149.
  3. Ren, S., He, K., Girshick, R. & Sun, J. (2016). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Extended tech report, arXiv:1506.01497v3.
  4. Girshick, R. (2015). Fast R-CNN. IEEE International Conference on Computer Vision (ICCV), 1440–1448.
  5. Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T. & Smeulders, A. W. M. (2013). Selective Search for Object Recognition. International Journal of Computer Vision, 104(2), 154–171.
  6. Zeiler, M. D. & Fergus, R. (2014). Visualizing and Understanding Convolutional Networks. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 818–833.
  7. Simonyan, K. & Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference on Learning Representations (ICLR). arXiv:1409.1556.
  8. He, K., Zhang, X., Ren, S. & Sun, J. (2016). Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.
  9. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B. & Belongie, S. (2017). Feature Pyramid Networks for Object Detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 936–944.
  10. He, K., Gkioxari, G., Dollár, P. & Girshick, R. (2017). Mask R-CNN. IEEE International Conference on Computer Vision (ICCV), 2980–2988.
  11. Girshick, R. & Bell, S. (2015). py-faster-rcnn (Python reimplementation of Faster R-CNN). Source code: README, lib/fast_rcnn/config.py, lib/rpn/proposal_layer.py and models/pascal_voc/*/faster_rcnn_end2end/test.prototxt.
  12. Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P. & He, K. (2018). Detectron. Source code.
  13. Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y. & Girshick, R. (2019). Detectron2. Source code.
  14. TorchVision documentation: Faster R-CNN models.
  15. NeurIPS Blog (2025). Announcing the Test of Time Paper Award for NeurIPS 2025. 26 November 2025.