Recognition & Detection
Faster R-CNN
A two-stage object detector in which a region proposal network, sharing the detector's convolutional features, replaces hand-designed region proposals, making proposal generation nearly free and the whole detector one network.
intermediate
Faster R-CNN is a two-stage object detector introduced by Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun at NIPS 2015, with an extended journal version in IEEE TPAMI (2017). It adds a region proposal network (RPN), a small fully convolutional head that predicts object proposals from the same convolutional feature map the detector uses, to the Fast R-CNN detector. Proposals, until then computed by a separate algorithm on the CPU, became a by-product of the network, about 10 ms per image. The two-stage object detectors article covers the R-CNN family’s design line, including RoI pooling and RoIAlign; this article covers what Faster R-CNN added.
Problem Addressed
By 2015, Fast R-CNN with VGG-16 classified and refined 2,000 proposals in about 0.3 s on a GPU, but the proposals came from selective search (Uijlings et al., 2013), which merges superpixels by low-level cues and took about 1.5 s per image on a CPU in Ren et al.’s measurement. Reimplementing selective search on a GPU, the authors argued, would still ignore the downstream detector and miss the chance to share its computation. Their alternative was to compute proposals with a network that shares the detector’s convolutional layers.
Architecture
The image, rescaled to a shorter side of 600 pixels, passes once through a convolutional backbone. The RPN slides over the last shared feature map and outputs scored proposals; Fast R-CNN pools a feature for each from the same map, then classifies and refines it.
| Component | Configuration in the paper | Output |
|---|---|---|
| Backbone | ImageNet-pretrained ZF (5 shareable conv layers) or VGG-16 (13), total stride 16 | 512 channels at about positions (VGG-16) |
| RPN conv | convolution with ReLU | 256-d (ZF) or 512-d (VGG-16) per position |
| RPN cls | convolution | scores per position, anchors |
| RPN reg | convolution | box offsets per position |
| Proposal layer | Decode, clip to image, NMS at IoU 0.7, keep top | 2,000 proposals for training, 300 at test |
| Detection head | Fast R-CNN: RoI pooling to , two fully connected layers, two sibling outputs | class scores, box offsets per proposal |
Backbone
The paper uses the “fast” ZF net (Zeiler & Fergus, 2014) and VGG-16 (Simonyan & Zisserman, 2015), both with stride 16 at the last convolutional layer; the receptive field of the RPN’s window is 171 pixels for ZF and 228 for VGG-16.
Region Proposal Network
At every position of the feature map, the convolution maps the window to a 512-dimensional vector (256 for ZF), and two sibling convolutions predict, for each of reference boxes called anchors, two object/not-object scores and four box offsets. Three scales (, , pixels) and three aspect ratios give ; anchors are labeled by their IoU with the ground truth (positive above 0.7 or best for some ground-truth box, negative below 0.3). The anchor boxes article covers this design, the labeling rule and the box parameterization. The proposal layers are translation invariant and small, the authors note: 2.4 million parameters for VGG-16, against 27 million for MultiBox with GoogLeNet.
Detection Head
The detector is Fast R-CNN unchanged (Girshick, 2015): RoI pooling extracts a grid from the shared map for each proposal, VGG-16’s fully connected layers process it, and sibling layers output a softmax over classes plus background and class-specific box offsets.
Training
RPN loss. For a mini-batch of anchors indexed by , with predicted object probability , label , predicted offsets and target offsets ,
where is the two-class log loss and , the robust loss of Fast R-CNN; the factor restricts regression to positive anchors. In the released code, (the mini-batch size), (the number of anchor positions) and , which weights the two terms roughly equally. The NIPS version gives these values in a footnote on its early implementation; the journal version adds that VOC 2007 mAP changed by about 1 point for from 1 to 100 and that the normalization is not required.
Sampling. Each RPN mini-batch comes from one image: 256 anchors sampled at random, up to half of them positive, padded with negatives. Anchors crossing the image border are ignored in training; the authors report that training did not converge otherwise. On PASCAL VOC, SGD ran at learning rate 0.001 for 60k mini-batches, then 0.0001 for 20k.
Sharing features. Trained separately, the RPN and Fast R-CNN would learn different convolutional layers. The NIPS paper uses 4-step alternating training: (1) train the RPN from an ImageNet-pretrained model; (2) train Fast R-CNN, also from an ImageNet model, on the step-1 proposals; (3) initialize the RPN from the detector, freeze the shared layers and fine-tune only the RPN’s own layers; (4) with the shared layers still fixed, fine-tune Fast R-CNN’s own layers (the fully connected layers in the NIPS wording). Joint optimization was left as future work, because Fast R-CNN training assumed fixed proposals.
The journal version describes three options and still uses alternating training for all its experiments. Approximate joint training merges both networks: each SGD step generates proposals in the forward pass, treats them as fixed inputs to Fast R-CNN, and combines the gradients of both losses in the shared layers, ignoring the gradient with respect to the proposal coordinates. The authors found it gave close results in 25–50% less training time. Non-approximate joint training would need RoI pooling that is differentiable in the box coordinates, which they leave to other work. py-faster-rcnn ships approximate joint training as its “end2end” script, about 1.5 times faster than alternating optimization for VGG-16 with similar accuracy, according to its README.
COCO. For MS COCO (journal version only), training used 8 GPUs, a fourth anchor scale of for small objects, and Fast R-CNN negatives from IoU instead of .
Inference
Every anchor becomes a box through its predicted offsets; boxes are clipped to the image, and non-maximum suppression at IoU 0.7 on the objectness scores leaves about 2,000 per image. Fast R-CNN was trained on these 2,000 and tested on the top 300. py-faster-rcnn adds two details: proposals smaller than 16 pixels (at original image scale) are dropped, and at test time only the 6,000 highest-scoring boxes enter NMS. The detector’s class-specific boxes then go through a second, per-class NMS (0.3 in py-faster-rcnn).
On a K40 GPU with VGG-16, the authors measured 198 ms per image (convolutions 141 ms, proposals 10 ms, region-wise layers 47 ms), or 5 frames per second, against 1,830 ms for selective search with Fast R-CNN; with ZF, 59 ms, or 17 frames per second.
Example: the proposal stage on a synthetic image
The NumPy code runs the proposal layer on a gray image holding three objects (ellipse, rectangle, triangle), with ground-truth boxes read from their masks. No RPN is trained: a stand-in gives plausible outputs, objectness rising with an anchor’s best IoU and offsets pointing toward the overlapped object, both noisy. The code follows py-faster-rcnn’s steps and measures recall of the ground truth by the top proposals over 40 such images, with ablations like the paper’s: no NMS, no offsets, no scores (random anchors).
import numpy as np
H, W, STRIDE = 600, 800, 16 # image rescaled to shorter side 600; VGG-16 stride
def make_anchors(scales=(128, 256, 512), ratios=(0.5, 1, 2)):
"""k = 9 anchors centered at (0, 0) as (x1, y1, x2, y2); area s^2, ratio h / w."""
a = [[-s / np.sqrt(r) / 2, -s * np.sqrt(r) / 2, s / np.sqrt(r) / 2, s * np.sqrt(r) / 2]
for r in ratios for s in scales]
return np.array(a)
def grid_anchors():
cy, cx = np.mgrid[0:H // STRIDE, 0:W // STRIDE] * STRIDE + STRIDE / 2 # 37 x 50 positions
c = np.stack([cx.ravel(), cy.ravel(), cx.ravel(), cy.ravel()], 1)
return (c[:, None, :] + make_anchors()[None]).reshape(-1, 4)
def iou(A, B):
lt, rb = np.maximum(A[:, None, :2], B[None, :, :2]), np.minimum(A[:, None, 2:], B[None, :, 2:])
inter = np.prod(np.clip(rb - lt, 0, None), 2)
area = lambda X: np.prod(X[:, 2:] - X[:, :2], 1)
return inter / (area(A)[:, None] + area(B)[None] - inter)
def encode(g, a): # targets t* of the anchor-relative parameterization
gw, gh, aw, ah = g[:, 2] - g[:, 0], g[:, 3] - g[:, 1], a[:, 2] - a[:, 0], a[:, 3] - a[:, 1]
return np.stack([((g[:, 0] + g[:, 2]) - (a[:, 0] + a[:, 2])) / 2 / aw,
((g[:, 1] + g[:, 3]) - (a[:, 1] + a[:, 3])) / 2 / ah,
np.log(gw / aw), np.log(gh / ah)], 1)
def decode(t, a):
aw, ah = a[:, 2] - a[:, 0], a[:, 3] - a[:, 1]
cx, cy = (a[:, 0] + a[:, 2]) / 2 + t[:, 0] * aw, (a[:, 1] + a[:, 3]) / 2 + t[:, 1] * ah
w, h = aw * np.exp(t[:, 2]), ah * np.exp(t[:, 3])
return np.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], 1)
def nms(boxes, thr):
order, keep = np.arange(len(boxes)), []
while order.size:
i = order[0]
keep.append(i)
order = order[1:][iou(boxes[i:i + 1], boxes[order[1:]])[0] <= thr]
return np.array(keep, int)
def synthetic_image(rng):
"""Gray image with 3 objects (ellipse, rectangle, triangle); boxes are read from the masks."""
yy, xx = np.mgrid[0:H, 0:W] + 0.5
img = rng.normal(0.3, 0.05, (H, W))
boxes = []
for kind in range(3):
w, h = rng.uniform(70, 330), rng.uniform(70, 330)
cx, cy = rng.uniform(w / 2, W - w / 2), rng.uniform(h / 2, H - h / 2)
u, v = (xx - cx) / (w / 2), (yy - cy) / (h / 2)
m = [u ** 2 + v ** 2 <= 1, (np.abs(u) <= 1) & (np.abs(v) <= 1),
(v <= 1) & (np.abs(u) <= (v + 1) / 2)][kind]
img[m] = 0.6 + 0.15 * kind
r, c = np.nonzero(m)
boxes.append([c.min(), r.min(), c.max() + 1, r.max() + 1])
return img, np.array(boxes, float)
def fake_rpn_outputs(anchors, gt, rng):
"""Stand-in for a trained RPN: objectness rises with an anchor's best IoU, deltas point
toward that object for anchors that overlap it, all with noise."""
ov = iou(anchors, gt)
best, j = ov.max(1), ov.argmax(1)
logit = 10 * (best - 0.45) + rng.normal(0, 1.5, len(anchors))
score = 1 / (1 + np.exp(-logit))
t = rng.normal(0, [0.08, 0.08, 0.15, 0.15], (len(anchors), 4)) # noise for every anchor
near = best > 0.2
t[near] += encode(gt[j[near]], anchors[near]) # imperfect regression
return score, t
def propose(anchors, score, t, pre_nms=6000, nms_thr=0.7, post_nms=300, min_size=16):
boxes = decode(t, anchors) # 1. apply deltas to anchors
boxes[:, 0::2] = boxes[:, 0::2].clip(0, W) # 2. clip to the image
boxes[:, 1::2] = boxes[:, 1::2].clip(0, H)
ok = ((boxes[:, 2] - boxes[:, 0]) >= min_size) & ((boxes[:, 3] - boxes[:, 1]) >= min_size)
boxes, score = boxes[ok], score[ok] # 3. drop tiny boxes
order = np.argsort(-score)[:pre_nms] # 4.-5. sort, keep top 6000
keep = order if nms_thr is None else order[nms(boxes[order], nms_thr)] # 6. NMS 0.7
keep = keep[:post_nms] # 7. top N
return boxes[keep], score[keep], ok.sum()
def recall(props, gt, thr):
return (iou(gt, props).max(1) >= thr).mean() if len(props) else 0.0
anchors = grid_anchors()
print("anchors:", len(anchors), "=", H // STRIDE, "x", W // STRIDE, "x 9")
rng = np.random.default_rng(0)
img, gt = synthetic_image(rng)
score, t = fake_rpn_outputs(anchors, gt, rng)
props, s, n_ok = propose(anchors, score, t, post_nms=2000)
print("ground truth:\n", gt.astype(int))
print(f"after clip and size filter: {n_ok}; after NMS 0.7: {len(props)}")
print("top 5 proposals (score, box):")
for b, p in zip(props[:5], s[:5]):
print(f" {p:.3f} {np.round(b).astype(int)} best IoU with GT {iou(b[None], gt).max():.2f}")
# Recall of ground truth over many synthetic images, for top-N proposals.
Ns = (10, 50, 300)
rows = {k: [] for k in ("sort + NMS 0.7", "sort, no NMS", "no reg (anchors)", "random anchors")}
for _ in range(40):
img, gt = synthetic_image(rng)
score, t = fake_rpn_outputs(anchors, gt, rng)
p, _, _ = propose(anchors, score, t)
raw, _, _ = propose(anchors, score, t, nms_thr=None) # NMS switched off
noreg, _, _ = propose(anchors, score, np.zeros_like(t)) # deltas switched off
rnd = anchors[rng.choice(len(anchors), 300, replace=False)]
for k, P in zip(rows, (p, raw, noreg, rnd)):
rows[k].append([[recall(P[:n], gt, thr) for n in Ns] for thr in (0.5, 0.7)])
print(f"\nrecall of 120 objects in 40 images IoU 0.5: N=" + "/".join(map(str, Ns))
+ " IoU 0.7: N=" + "/".join(map(str, Ns)))
for k, v in rows.items():
r = np.mean(v, 0)
print(f"{k:22s} " + " ".join(f"{x:.2f}" for x in r[0]) + " "
+ " ".join(f"{x:.2f}" for x in r[1]))
anchors: 16650 = 37 x 50 x 9
ground truth:
[[195 342 381 461]
[563 310 692 598]
[ 14 68 136 327]]
after clip and size filter: 16650; after NMS 0.7: 1370
top 5 proposals (score, box):
0.997 [182 352 370 466] best IoU with GT 0.78
0.991 [194 323 351 472] best IoU with GT 0.68
0.989 [196 333 397 448] best IoU with GT 0.77
0.988 [521 324 679 600] best IoU with GT 0.65
0.984 [574 302 706 570] best IoU with GT 0.73
recall of 120 objects in 40 images IoU 0.5: N=10/50/300 IoU 0.7: N=10/50/300
sort + NMS 0.7 0.94 1.00 1.00 0.76 1.00 1.00
sort, no NMS 0.87 0.99 1.00 0.69 0.95 1.00
no reg (anchors) 0.88 0.97 0.98 0.44 0.57 0.57
random anchors 0.03 0.22 0.66 0.01 0.05 0.12
NMS at 0.7 leaves 1,370 of the 6,000 top-scoring boxes, the order of the paper’s 2,000. Three of the top five proposals cover one object; without NMS such near-duplicates fill the top places, and recall at falls from 0.94 to 0.87. Without the offsets, recall at IoU 0.5 stays high but at IoU 0.7 stalls near 0.57, since the anchors’ fixed shapes rarely fit an object tightly; the paper’s no-reg ablation likewise lost VOC mAP (52.1% against 56.8%). Without scores, 300 random anchors reach a recall of only 0.66 at IoU 0.5 and 0.12 at 0.7; the paper’s no-cls ablation fell to 44.6% mAP with 100 random proposals. These numbers come from a simulated RPN, so they show how the pipeline behaves, not how accurate a trained RPN is; by our reasoning, an image with dozens of objects, unlike these three, needs a larger .
The RPN head itself is three layers. In PyTorch (run with version 2.14.1), applied to a random tensor shaped like the VGG-16 conv5_3 map of a image ( with PyTorch’s default floor rounding in pooling; Caffe, which rounds pooling outputs up, gives ):
import torch
import torch.nn as nn
class RPNHead(nn.Module):
"""3x3 'sliding window' conv + ReLU, then sibling 1x1 convs: cls (2k) and reg (4k)."""
def __init__(self, in_ch=512, mid_ch=512, k=9):
super().__init__()
self.conv = nn.Conv2d(in_ch, mid_ch, 3, padding=1)
self.cls = nn.Conv2d(mid_ch, 2 * k, 1) # object / not-object score per anchor
self.reg = nn.Conv2d(mid_ch, 4 * k, 1) # (t_x, t_y, t_w, t_h) per anchor
for m in (self.conv, self.cls, self.reg): # new layers: zero-mean Gaussian, std 0.01
nn.init.normal_(m.weight, std=0.01)
nn.init.zeros_(m.bias)
def forward(self, f):
h = torch.relu(self.conv(f))
return self.cls(h), self.reg(h)
torch.manual_seed(0)
conv5 = torch.randn(1, 512, 37, 50) # VGG-16 conv5_3 map of a 600 x 800 image
head = RPNHead()
cls, reg = head(conv5)
k = 9
print("conv5_3 features:", tuple(conv5.shape))
print("cls logits: ", tuple(cls.shape), "->", tuple(cls.view(1, 2, k, 37, 50).shape), "(bg/fg, anchor, y, x)")
print("reg deltas: ", tuple(reg.shape), "->", tuple(reg.view(1, k, 4, 37, 50).shape), "(anchor, t, y, x)")
print("anchors:", 37 * 50 * k)
for name, ch in (("VGG-16 (512-d)", 512), ("ZF (256-d)", 256)):
n = sum(p.numel() for p in RPNHead(ch, ch).parameters())
print(f"RPN head parameters, {name}: {n:,}")
print("PyTorch", torch.__version__)
conv5_3 features: (1, 512, 37, 50)
cls logits: (1, 18, 37, 50) -> (1, 2, 9, 37, 50) (bg/fg, anchor, y, x)
reg deltas: (1, 36, 37, 50) -> (1, 9, 4, 37, 50) (anchor, t, y, x)
anchors: 16650
RPN head parameters, VGG-16 (512-d): 2,387,510
RPN head parameters, ZF (256-d): 603,958
PyTorch 2.14.1
The 2.39 million parameters of the VGG-16 head match the paper’s 2.4 million, which omits biases; the ZF count assumes the 256-channel conv5 of py-faster-rcnn’s ZF model. The channel layout follows py-faster-rcnn, which reads the first score channels as background; the paper notes that logistic outputs could replace its two-class softmax.
Performance
All results are from Ren et al. with VGG-16, 300 proposals at test time and single-scale testing. PASCAL VOC results are mAP at IoU 0.5; “07+12” is VOC 2007 trainval plus VOC 2012 trainval, and “07++12” adds VOC 2007 test (see PASCAL VOC).
| Test set | Training data | Metric | Faster R-CNN | Selective search + Fast R-CNN | Version |
|---|---|---|---|---|---|
| VOC 2007 test | VOC 2007 trainval | mAP | 69.9 | 66.9 | NIPS and journal |
| VOC 2007 test | 07+12 | mAP | 73.2 | 70.0 | NIPS and journal |
| VOC 2007 test | COCO trainval only | mAP | 76.1 | – | Journal |
| VOC 2007 test | COCO, then 07+12 | mAP | 78.8 | – | Journal |
| VOC 2012 test | VOC 2012 trainval | mAP | 67.0 | 65.7 | NIPS and journal |
| VOC 2012 test | 07++12 | mAP | 70.4 | 68.4 | NIPS and journal |
| VOC 2012 test | COCO, then 07++12 | mAP | 75.9 | – | Journal |
| COCO val | COCO train | AP@0.5 / AP@[.5:.95] | 41.5 / 21.2 | 38.6 / 18.9 | Journal |
| COCO test-dev | COCO train | AP@0.5 / AP@[.5:.95] | 42.1 / 21.5 | 39.3 / 19.3 | Journal |
| COCO test-dev | COCO trainval | AP@0.5 / AP@[.5:.95] | 42.7 / 21.9 | – | Journal |
The VOC baselines are the selective-search Fast R-CNN results reported by Girshick (2015); the COCO baseline is the authors’ own Fast R-CNN implementation. On COCO test-dev, the authors read the gain over Fast R-CNN (2.8 points at AP@0.5, 2.2 at AP@[.5:.95]) as evidence that RPN proposals improve localization at higher IoU thresholds. Pretraining on COCO and fine-tuning on VOC added 5.6 points on VOC 2007.
In ZF ablations on VOC 2007 (both versions), sharing features gave 59.9% against 58.7% unshared, and a one-stage variant, Fast R-CNN on dense sliding windows of the nine anchor shapes instead of proposals, reached 53.9% against 58.7% for the two-stage system.
Strengths
- Nearly free proposals. Sharing the backbone made the proposal step 10 ms with VGG-16, and the detector one network.
- Fewer, better proposals. 300 learned proposals outperformed 2,000 from selective search, and recall declined little from 2,000 to 300 proposals, which the authors attribute to the cls scores.
- Benefits from better backbones. Learned proposals improve with deeper networks, as the ResNet results below show.
- Extensible head. Mask R-CNN later added masks and keypoints to the per-region head.
Limitations
- Speed. At 5 frames per second with VGG-16 it was not real-time, and the per-region layers run hundreds of times per image; one-stage detectors such as YOLO and SSD avoid this cost.
- Single-scale features. All anchors come from one stride-16 map, which suits small objects poorly.
- Hand-tuned components. Anchor shapes, IoU thresholds, sampling ratios and two NMS stages are set by hand; DETR later removed anchors and NMS.
- Complex training. The 4-step scheme runs four training stages in sequence; approximate joint training simplifies it but ignores gradients through the proposal boxes.
Historical Importance
In our reading, Faster R-CNN completed the shift R-CNN began in 2014: every stage of a region-based detector was now a network trained on detection data. Its anchors were taken up by SSD, YOLOv2 and RetinaNet, whose papers cite the RPN for them.
He et al. (2016) used Faster R-CNN as the detector in their ResNet experiments; replacing VGG-16 with ResNet-101 raised COCO val AP@[.5:.95] from 21.2% to 27.2% (see ResNet paper). Their Faster R-CNN-based systems won first place in ImageNet detection and COCO detection in the ILSVRC and COCO 2015 competitions, and the Faster R-CNN journal version adds that the RPN was part of the winning ImageNet localization and COCO segmentation entries. Faster R-CNN also supplied the detections for SORT’s tracking-by-detection (see SORT paper).
The paper received the NeurIPS 2025 Test of Time Paper Award, given to NeurIPS papers from ten years earlier.
Variants
- ResNet backbones. Because ResNet has no hidden fully connected layers, He et al. (2016) shared conv1 to conv4 (stride 16) between RPN and detector and ran conv5 on each RoI after pooling, a configuration Detectron2’s model zoo calls C4.
- Feature pyramid network. Lin et al. (2017) run the RPN on every level of a feature pyramid, with one anchor scale per level, and pool each RoI from a level chosen by its size. With ResNet-101, they reported 36.2 AP on COCO test-dev. All of TorchVision’s Faster R-CNN builders use FPN.
- Mask R-CNN. He et al. (2017) added a mask branch and replaced RoI pooling with RoIAlign (see two-stage object detectors). Faster R-CNN with RoIAlign alone reached 37.3 box AP on COCO test-dev with ResNet-101-FPN, against the 36.2 that Lin et al. reported with RoI pooling, a gain the Mask R-CNN authors attribute to RoIAlign.
- R-FCN and Cascade R-CNN change the per-region head and its training; the two-stage object detectors article covers both.
Implementations
The official code is in MATLAB (ShaoqingRen/faster_rcnn); py-faster-rcnn reimplements it in Python with Caffe, with similar but not identical mAP, according to its README. Detectron (Girshick et al., 2018), written for Caffe2 and now deprecated, implements Faster R-CNN with VGG16, ResNet, ResNeXt and FPN backbones; its PyTorch successor Detectron2 (Wu et al., 2019) has C4, dilated-C5 and FPN variants in its model zoo. TorchVision provides builders such as fasterrcnn_resnet50_fpn and fasterrcnn_mobilenet_v3_large_fpn; check the documentation of the installed version, as the API changes between releases.
Related
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
The 2015 NIPS paper by Ren, He, Girshick and Sun that replaced hand-designed region proposals with a region proposal network sharing the detector's convolutional features, making proposals nearly free and the whole detector one network.
- Anchor Boxes
Reference boxes of fixed sizes and aspect ratios, tiled at every feature-map position, that a detector classifies and refines, and that decide which of its predictions are trained on which objects.
- Non-Maximum Suppression
The post-processing step that removes duplicate detections by repeatedly keeping the highest-scoring box and discarding the boxes that overlap it beyond an IoU threshold, with its soft, distance-aware and learned variants.
- Feature Pyramid Network
A network design that turns a convolutional backbone's feature maps at strides 4 to 32 into a pyramid with strong semantic features at every scale, through a top-down pathway and lateral connections, so a detector can find small and large objects from one image scale.
- YOLO
A family of real-time one-stage object detectors, begun by Redmon et al. in 2016, that predict every box and class probability of an image in a single network pass; later versions from several groups added anchors, multi-scale heads, anchor-free heads and NMS-free training.
- COCO
Microsoft's Common Objects in Context, a dataset of everyday scenes with per-instance segmentation masks for 80 object categories, plus keypoints, captions, stuff and panoptic labels, whose evaluation protocol became the standard benchmark for object detection.
References
- Ren, S., He, K., Girshick, R. & Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28 (NIPS).
- Ren, S., He, K., Girshick, R. & Sun, J. (2017). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149.
- Ren, S., He, K., Girshick, R. & Sun, J. (2016). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Extended tech report, arXiv:1506.01497v3.
- Girshick, R. (2015). Fast R-CNN. IEEE International Conference on Computer Vision (ICCV), 1440–1448.
- Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T. & Smeulders, A. W. M. (2013). Selective Search for Object Recognition. International Journal of Computer Vision, 104(2), 154–171.
- Zeiler, M. D. & Fergus, R. (2014). Visualizing and Understanding Convolutional Networks. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 818–833.
- Simonyan, K. & Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference on Learning Representations (ICLR). arXiv:1409.1556.
- He, K., Zhang, X., Ren, S. & Sun, J. (2016). Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.
- Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B. & Belongie, S. (2017). Feature Pyramid Networks for Object Detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 936–944.
- He, K., Gkioxari, G., Dollár, P. & Girshick, R. (2017). Mask R-CNN. IEEE International Conference on Computer Vision (ICCV), 2980–2988.
- Girshick, R. & Bell, S. (2015). py-faster-rcnn (Python reimplementation of Faster R-CNN). Source code: README, lib/fast_rcnn/config.py, lib/rpn/proposal_layer.py and models/pascal_voc/*/faster_rcnn_end2end/test.prototxt.
- Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P. & He, K. (2018). Detectron. Source code.
- Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y. & Girshick, R. (2019). Detectron2. Source code.
- TorchVision documentation: Faster R-CNN models.
- NeurIPS Blog (2025). Announcing the Test of Time Paper Award for NeurIPS 2025. 26 November 2025.