Two-Stage Object Detectors

A two-stage object detector splits object detection into two steps: it first proposes candidate regions that might contain an object of any class, then pools features for each region from a shared convolutional feature map, classifies it as one of the object classes or background, and refines its box. The design is also called region-based detection or, after its first deep-learning member, the R-CNN family.

Core Idea

Most windows in an image contain no object. A two-stage detector spends cheap computation to discard most of them and expensive computation only on the survivors:

• Stage 1, proposals. A class-agnostic generator scores locations for “objectness” and outputs candidate boxes, typically a few hundred to about 2,000 per image.
• Stage 2, per-region recognition. For each proposal, a head predicts $K+1$ class scores ($K$ object classes plus background) and a class-specific correction to the box.

Girshick (2015) found that this cascade helps accuracy, not only speed: trained and tested on about 45,000 dense boxes per image, his mid-sized Fast R-CNN model reached 52.9% PASCAL VOC 2007 test mAP, against 59.2% with selective-search proposals in the paper’s main comparison.

Building Blocks

Backbone. A convolutional neural network pre-trained on ImageNet classification computes a feature map of the image once, typically at stride 16 (VGG16 conv5, ResNet conv4).

Proposal generator. Early detectors used selective search, bottom-up grouping outside the network. Faster R-CNN replaced it with a region proposal network (RPN) that scores anchor boxes as object or not and regresses their offsets; non-maximum suppression at IoU 0.7 leaves about 2,000 proposals per image.

RoI feature extraction. The head needs a fixed-size input, but proposals differ in size and shape. R-CNN warped image crops; later detectors pool a fixed grid, such as $7 \times 7$, from the region of interest (RoI) on the shared feature map, with spatial pyramid pooling, RoI pooling or RoIAlign. The pooling article covers the first two.

Per-region head. Fully connected or convolutional layers map each pooled feature to $K+1$ class scores and $4K$ box offsets.

How Data Flows

In a Faster R-CNN-style detector, the image passes through the backbone once. The RPN decodes its best anchors into boxes, applies NMS and keeps the top $N$ (300 at test time). RoI pooling or RoIAlign extracts a $7 \times 7 \times C$ tensor per proposal from the same feature map; the head scores each, its offsets are decoded relative to the proposal, and per-class NMS gives the detections. Only the pooling and the head run per region.

Mathematical Formulation

Box regression. A proposal $P = (P_x, P_y, P_w, P_h)$ (center, width, height) is regressed toward its ground-truth box $G$ through the targets

$$t_x = \frac{G_x - P_x}{P_w}, \qquad t_y = \frac{G_y - P_y}{P_h}, \qquad t_w = \log\frac{G_w}{P_w}, \qquad t_h = \log\frac{G_h}{P_h},$$

introduced by R-CNN; the anchor boxes article covers decoding, with the anchor in the proposal’s role.

Multi-task loss. Fast R-CNN trains classification and regression jointly. For an RoI with true class $u$ (0 for background), predicted class distribution $p$, predicted offsets $t^u$ for class $u$ and targets $v$,

$$L(p, u, t^u, v) = -\log p_u + \lambda\,[u \ge 1] \sum_{i \in \{x, y, w, h\}} \operatorname{smooth}_{L_1}(t^u_i - v_i), \qquad \operatorname{smooth}_{L_1}(z) = \begin{cases} 0.5\,z^2 & |z| < 1 \\ |z| - 0.5 & \text{otherwise.} \end{cases}$$

The indicator $[u \ge 1]$ switches the box term off for background RoIs. Girshick (2015) chose smooth $L_1$ because it is less sensitive to outliers than the $L_2$ loss of R-CNN and SPPnet, which can need careful learning-rate tuning; targets are normalized to zero mean and unit variance, and $\lambda = 1$. The RPN loss has the same structure with two classes.

RoI pooling. With stride $\kappa$, RoI pooling rounds the box to the feature grid, $\hat{x}_1 = [x_1/\kappa]$ and likewise for the other coordinates, splits the rounded box into $k \times k$ bins whose edges are again rounded, and max-pools each bin. He et al. (2017) point out that both roundings misalign the pooled features with the region: harmless for classification, which tolerates small shifts, but harmful for pixel-accurate masks.

RoIAlign. RoIAlign keeps $x_1/\kappa$ unrounded. Bin $(i, j)$, of size $b_w \times b_h$ feature cells, averages $s \times s$ regularly spaced samples,

$$r_{ij} = \frac{1}{s^2} \sum_{a=0}^{s-1} \sum_{b=0}^{s-1} f\!\left(\frac{y_1}{\kappa} + \Big(i + \frac{a + \frac{1}{2}}{s}\Big) b_h,\; \frac{x_1}{\kappa} + \Big(j + \frac{b + \frac{1}{2}}{s}\Big) b_w\right),$$

where $f$ is the feature map evaluated by bilinear interpolation (see image warping). Mask R-CNN uses four samples per bin ($s = 2$) with max or average; the authors note that one sample at the bin center is nearly as effective. Whether a cell’s value sits at the cell’s corner or its center is a convention, and the two choices differ by half a cell; the example below uses the center.

Example: RoI pooling and RoIAlign on a synthetic feature map

The NumPy code implements both operators with a $4 \times 4$ output on a $16 \times 16$ feature map at stride 16 and asks two questions. Which image columns does each bin read for a box from $x = 100$ to $172$ pixels (6.25 to 10.75 cells)? And when the map holds one sharp feature, a Gaussian peak one cell wide, and the object and its $72 \times 72$ box shift together by less than a stride, do the pooled features stay the same, as they should?

import numpy as np

STRIDE = 16                      # feature-map stride: cell (i, j) covers image pixels [16j, 16j + 16)

def roi_pool(F, box, out=4):
    """RoI max pooling: quantize the box to cells, then split it into out x out bins with floor/ceil."""
    x1, y1, x2, y2 = [int(np.floor(v / STRIDE + 0.5)) for v in box]   # [x / 16], rounded
    h, w = max(y2 - y1, 1), max(x2 - x1, 1)
    Y, cells = np.zeros((out, out)), []
    for i in range(out):
        r0, r1 = y1 + int(np.floor(i * h / out)), y1 + int(np.ceil((i + 1) * h / out))
        for j in range(out):
            c0, c1 = x1 + int(np.floor(j * w / out)), x1 + int(np.ceil((j + 1) * w / out))
            Y[i, j] = F[r0:r1, c0:c1].max()
            if i == 0:
                cells.append((c0, c1))
    return Y, cells

def bilinear(F, y, x):
    """Value at continuous index (y, x); cell (i, j) has its value at its center, index (i, j)."""
    H, W = F.shape
    y, x = np.clip(y, 0, H - 1), np.clip(x, 0, W - 1)
    y0, x0 = min(int(y), H - 2), min(int(x), W - 2)
    dy, dx = y - y0, x - x0
    return ((1 - dy) * (1 - dx) * F[y0, x0] + (1 - dy) * dx * F[y0, x0 + 1]
            + dy * (1 - dx) * F[y0 + 1, x0] + dy * dx * F[y0 + 1, x0 + 1])

def roi_align(F, box, out=4, s=2):
    """RoIAlign: no rounding; average of s x s bilinear samples per bin."""
    x1, y1, x2, y2 = [v / STRIDE for v in box]                         # x / 16, not rounded
    bh, bw = (y2 - y1) / out, (x2 - x1) / out
    Y = np.zeros((out, out))
    for i in range(out):
        for j in range(out):
            ys = y1 + (i + (np.arange(s) + 0.5) / s) * bh - 0.5          # -0.5: cell centers
            xs = x1 + (j + (np.arange(s) + 0.5) / s) * bw - 0.5
            Y[i, j] = np.mean([bilinear(F, a, b) for a in ys for b in xs])
    return Y

# 1. Where each bin's features come from, for a box with fractional cell coordinates.
box = (100, 100, 172, 172)       # x1, y1, x2, y2 in image pixels: 6.25 to 10.75 cells
_, cells = roi_pool(np.zeros((16, 16)), box)
edges = np.linspace(box[0], box[2], 5)
print("bin  ideal x-range (px)  RoIPool reads (px)  RoIAlign samples x (px)")
for j, (c0, c1) in enumerate(cells):
    xs = box[0] + (j + np.array([0.25, 0.75])) * (box[2] - box[0]) / 4
    print(f" {j}   [{edges[j]:5.1f}, {edges[j + 1]:5.1f})     [{c0 * STRIDE:3d}, {c1 * STRIDE:3d})"
          f"           {xs[0]:.1f}, {xs[1]:.1f}")

# 2. A sharp feature: one object's response peak on a 16 x 16 map. Shift object and box together
#    by sub-stride amounts; ideally the RoI features would not change.
def feature_map(cx, cy, sigma=16.0):
    c = (np.arange(16) + 0.5) * STRIDE                                  # cell centers in pixels
    return np.exp(-((c[None, :] - cx) ** 2 + (c[:, None] - cy) ** 2) / (2 * sigma ** 2))

np.set_printoptions(precision=2, suppress=True)
ref_pool = ref_align = None
print("\nshift  RoIPool columns  max|change| RoIPool  max|change| RoIAlign")
for shift in range(0, 17, 4):
    cx = cy = 136 + shift                                               # object center
    F = feature_map(cx, cy)
    b = (cx - 36, cy - 36, cx + 36, cy + 36)                            # 72 x 72 box on the object
    P, cells = roi_pool(F, b)
    A = roi_align(F, b)
    if shift == 0:
        ref_pool, ref_align = P, A
        print("RoIPool at shift 0:\n", P, "\nRoIAlign at shift 0:\n", A)
    q = [int(np.floor(v / STRIDE + 0.5)) for v in b]
    print(f"{shift:4d}px   {q[0]:2d}..{q[2] - 1:2d}         {np.abs(P - ref_pool).max():.3f}"
          f"              {np.abs(A - ref_align).max():.3f}")

bin  ideal x-range (px)  RoIPool reads (px)  RoIAlign samples x (px)
 0   [100.0, 118.0)     [ 96, 128)           104.5, 113.5
 1   [118.0, 136.0)     [112, 144)           122.5, 131.5
 2   [136.0, 154.0)     [128, 160)           140.5, 149.5
 3   [154.0, 172.0)     [144, 176)           158.5, 167.5

shift  RoIPool columns  max|change| RoIPool  max|change| RoIAlign
RoIPool at shift 0:
 [[0.37 0.61 0.61 0.37]
 [0.61 1.   1.   0.61]
 [0.61 1.   1.   0.61]
 [0.37 0.61 0.61 0.37]] 
RoIAlign at shift 0:
 [[0.08 0.22 0.22 0.08]
 [0.22 0.61 0.61 0.22]
 [0.22 0.61 0.61 0.22]
 [0.08 0.22 0.22 0.08]]
   0px    6..10         0.000              0.000
   4px    7..10         0.443              0.049
   8px    7..10         0.320              0.012
  12px    7..11         0.202              0.049
  16px    7..11         0.000              0.000

RoI pooling rounds the box to columns 6 to 10 (96 to 176 pixels), and rounding the 1.25-cell bins outward gives every bin two cells: 32 pixels against the ideal 18, overlapping its neighbors. RoIAlign samples inside each ideal bin. In the shift test, a 4-pixel shift moves the left edge from 6.25 to 6.5 cells, which rounds to 7, so RoI pooling drops a column and its output changes by up to 0.443 on a peak of 1; RoIAlign changes by at most 0.049, which we attribute to bilinear interpolation of this one-cell-wide peak. A full-stride shift leaves both unchanged. In our reading, this is the misalignment He et al. (2017) describe.

The PyTorch version below, run with PyTorch 2.14.1 after the NumPy code, uses grid_sample, whose align_corners=False convention also puts values at cell centers. It reproduces the NumPy result and, because bilinear sampling is differentiable in the sample positions, returns gradients with respect to the box coordinates; by our reasoning, RoI pooling’s rounding makes those gradients zero almost everywhere.

import torch
import torch.nn.functional as Fn

def roi_align_torch(F, box, out=4, s=2, stride=16):
    """RoIAlign via grid_sample; F is (C, H, W), box (x1, y1, x2, y2) in image pixels."""
    C, H, W = F.shape
    t = (torch.arange(out * s, dtype=F.dtype) + 0.5) / (out * s)            # sample positions in [0, 1]
    xs = box[0] + t * (box[2] - box[0])                                    # image pixels
    ys = box[1] + t * (box[3] - box[1])
    # align_corners=False puts -1 and +1 at the outer edges of the map, i.e. image x = 0 and 16 W.
    gx, gy = 2 * xs / (stride * W) - 1, 2 * ys / (stride * H) - 1
    grid = torch.stack(torch.meshgrid(gy, gx, indexing="ij")[::-1], dim=-1)  # (out*s, out*s, 2) as (x, y)
    samples = Fn.grid_sample(F[None], grid[None], mode="bilinear",
                             padding_mode="border", align_corners=False)[0]
    return Fn.avg_pool2d(samples, s)                                        # average s x s samples per bin

F = torch.tensor(feature_map(136, 136))[None]                               # map from the NumPy code
box = torch.tensor([100.0, 100.0, 172.0, 172.0], dtype=F.dtype, requires_grad=True)
A = roi_align_torch(F, box)
ref = torch.tensor(roi_align(feature_map(136, 136), (100, 100, 172, 172)))
print(f"max |torch - NumPy| = {(A[0] - ref).abs().max().item():.1e}")
A.sum().backward()
print("d(sum of RoI features)/d(x1, y1, x2, y2):", [f"{g:.4f}" for g in box.grad.tolist()])
print("PyTorch", torch.__version__)

max |torch - NumPy| = 0.0e+00
d(sum of RoI features)/d(x1, y1, x2, y2): ['0.0599', '0.0599', '-0.0599', '-0.0599']
PyTorch 2.14.1

Why It Works

• Imbalance handled by construction. Lin et al. (2017, focal loss) describe two-stage detectors as handling the object–background imbalance with the proposal cascade and sampling heuristics, such as a fixed one-to-three ratio of foreground to background RoIs; the one-stage object detectors article explains why dense detectors find this harder.
• Shared computation. From SPPnet on, the backbone runs once per image; in Faster R-CNN the RPN added about 10 ms per image.

Training Considerations

Sampling. Fast R-CNN builds each minibatch from 2 images and 128 RoIs: 25% foreground, from proposals with IoU at least 0.5 with a ground-truth box, and the rest background, from proposals with maximum IoU in $[0.1, 0.5)$; the lower bound, Girshick notes, acts as a heuristic for hard-example mining. Sharing the backbone’s computation among RoIs of one image is about 64 times faster than taking one RoI from each of 128 images, as R-CNN and SPPnet did.

Variants

• R-FCN (Dai et al., 2016) removes the per-region layers, which with ResNet-101 had become a 10-layer subnetwork run on every RoI. It instead predicts $k^2(C+1)$ position-sensitive score maps, $k \times k$ per class, each for one relative position in the RoI; bin $(i, j)$ of the RoI pools only from map $(i, j)$, and the bins vote by averaging. With ResNet-101 on VOC 2007 test (trained on 07+12, with online hard example mining), the authors reported 79.5% mAP at 0.17 s per image, against 79.3% at 0.42 s for Faster R-CNN trained the same way.
• Feature pyramid network (Lin et al., 2017) runs the RPN on every pyramid level and pools each RoI from a level chosen by its size, small boxes from finer maps, with a lighter head of two fully connected layers.
• Mask R-CNN (He et al., 2017) adds a branch that predicts a binary mask per class for each RoI and replaces RoI pooling with RoIAlign. On COCO minival with a ResNet-50-C4 backbone, RoIAlign raised mask AP from 26.9 to 30.3, and by 7.3 points with stride-32 features. RoIWarp, which interpolates bilinearly but still rounds the box, performed like RoI pooling, so the authors credit alignment, not interpolation.
• Cascade R-CNN (Cai & Vasconcelos, 2018) targets localization quality. Raising a head’s training IoU threshold from 0.5 to 0.7 can degrade results, the authors explain: positives become exponentially scarce, so the head overfits, and at test time it meets lower-quality proposals than it was trained on. The cascade chains three heads trained at IoU 0.5, 0.6 and 0.7, each trained on the previous head’s refined boxes; R-CNN’s tech report had found that reapplying a single regressor did not help.

Use in Computer Vision

The per-region head extends to other per-object outputs: Mask R-CNN predicts instance masks and, in the same paper, human keypoints. Two-stage detectors also supplied the detections for early deep tracking-by-detection; SORT ran on Faster R-CNN boxes.

Limitations

• Speed. The head runs hundreds of times per image; Faster R-CNN’s 5 frames per second was below the real-time rates one-stage detectors targeted.
• Recall is capped by the proposals. An object that receives no proposal cannot be detected.
• Hand-designed parts. Anchors, sampling thresholds and ratios, and two rounds of NMS all need tuning; DETR later predicted the set of objects directly, without anchors or NMS.
• No guaranteed accuracy lead. With the focal loss, the one-stage RetinaNet surpassed the accuracy of the two-stage detectors of its time on COCO (Lin et al., 2017).

Important Models

R-CNN (2014)

R-CNN (Girshick et al., 2014), “Regions with CNN features”, showed that features learned for ImageNet classification transfer to detection. At test time it ran selective search in its fast mode for about 2,000 category-independent proposals; enlarged each box so that 16 pixels of context surround it after warping, and warped the crop to the $227 \times 227$ input of an AlexNet-style CNN; took a 4,096-dimensional feature from the second fully connected layer (fc7); scored it with one linear SVM per class; and applied greedy NMS per class.

Training had three stages. First, fine-tune the ImageNet CNN on warped VOC proposals with a 21-way output (20 classes plus background), proposals with IoU at least 0.5 counting as positives, in minibatches of 32 positive and 96 background windows; this raised VOC 2007 test mAP by 8.0 points to 54.2%. Second, train the SVMs with hard negative mining, using ground-truth boxes as positives and proposals below IoU 0.3 as negatives; the threshold came from a grid search, and 0.5 cost 5 mAP points. Third, fit class-specific bounding-box regressors, ridge regressions on the pool5 features of proposals with IoU above 0.6 with a ground-truth box (tech report, Appendix C), which added 3 to 4 mAP points. The authors reported 53.7% mAP on VOC 2010 test, against 35.1% for the same proposals with spatial-pyramid bag-of-words features and 33.4% for deformable part models, and 53.3% on VOC 2011/12 test. Every proposal went through the whole CNN, so proposals and features took 13 s per image on a GPU (53 s on a CPU), and, as Girshick (2015) notes, SVM and regressor training required features for every proposal to be written to disk.

SPPnet (2014)

SPPnet (He et al., ECCV 2014; details here from the 2015 journal version) ran the convolutional layers once per image and pooled each proposal’s window on the conv5 map with a four-level spatial pyramid (50 bins), testing 24 to 102 times faster than R-CNN with comparable accuracy (59.2% against 58.5% mAP on VOC 2007 test with five scales). Its fine-tuning could update only the fully connected layers.

Fast R-CNN (2015)

Fast R-CNN (Girshick, 2015) replaced the pyramid with single-level RoI pooling and trained in one stage with the multi-task loss, updating convolutional layers too (from conv3_1 up in VGG16); its softmax scored 0.1 to 0.8 mAP points above post-hoc SVMs. With VGG16 it trained 9 times and tested 213 times faster than R-CNN (with truncated SVD of the fully connected layers), at 66.9% mAP on VOC 2007 test (70.0% trained on 07+12). Selective search, at 2 s per image in a CPU implementation according to Ren et al. (2015), now dominated the run time.

Faster R-CNN (2015)

Faster R-CNN (Ren et al., 2015) generated the proposals with the RPN on the shared feature map, making the detector one network that ran at 5 frames per second with VGG-16, against 0.5 for selective search with Fast R-CNN. The Faster R-CNN article covers the model.

Summary

Method	Year	Proposals	Per-region computation	Key reported result
R-CNN	2014	Selective search, ~2,000	Whole CNN on each warped $227 \times 227$ crop; per-class SVMs	VOC 2010 test mAP 53.7%, VOC 2007 test 58.5% (AlexNet, with box regression); 13 s/image on a GPU
SPPnet	2014	Selective search, ~2,000	Pyramid pooling on shared conv5 map; fc layers; SVMs	VOC 2007 test mAP 59.2% (ZF-5, 5 scales), 0.38 s/image on a GPU without proposals (journal version)
Fast R-CNN	2015	Selective search, ~2,000	RoI pooling ($7 \times 7$); fc layers; softmax and box regression	VOC 2007 test mAP 66.9% (VGG16, 07 trainval), 0.32 s/image without proposals
Faster R-CNN	2015	RPN, 300 at test	As Fast R-CNN	VOC 2007 test mAP 73.2% (VGG-16, 07+12), 5 fps including proposals
R-FCN	2016	RPN, 300	Position-sensitive RoI pooling and voting; no per-region layers	VOC 2007 test mAP 79.5% (ResNet-101, 07+12, with OHEM), 0.17 s/image
Faster R-CNN + FPN	2017	RPN on each pyramid level	RoI pooling from a size-dependent level; 2 fc layers	COCO test-dev AP 36.2 (ResNet-101-FPN)
Mask R-CNN	2017	RPN with FPN	RoIAlign; box head and mask FCN	COCO test-dev box AP 38.2, mask AP 35.7 (ResNet-101-FPN)
Cascade R-CNN	2018	RPN with FPN	Three heads at IoU 0.5, 0.6, 0.7	COCO test-dev AP 42.8 against 38.8 for its FPN+ baseline (ResNet-101)

All results are as reported by the authors, with different backbones, training sets and hardware, so they show the progression, not a controlled comparison. PASCAL VOC numbers are mAP at IoU 0.5; COCO AP averages over IoU thresholds from 0.5 to 0.95.