R-CNN

R-CNN, short for “Regions with CNN features”, is an object detection model introduced by Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik in an arXiv tech report in November 2013 and at CVPR 2014. It takes about 2,000 class-agnostic candidate boxes per image from a bottom-up proposal method, selective search, warps the pixels of each box to the fixed input size of a convolutional neural network pre-trained on ImageNet, scores the resulting feature vector with one linear SVM per class, and refines the box with a class-specific regressor. Its two direct successors kept the proposals and changed everything after them. SPPnet (He et al., 2014) ran the convolutional layers once per image and pooled a feature for each proposal from the shared map. Fast R-CNN (Girshick, 2015) replaced R-CNN’s three training stages with one network trained by a multi-task loss. This article follows that line from R-CNN to Fast R-CNN, as a model family built on CNNs. The 2014 publication itself, with its ablations and results, is the subject of the Rich Feature Hierarchies paper article, and the two-stage object detectors article places the line among the region-based detectors that followed.

Problem Addressed

In 2012, a deep CNN won ImageNet classification by a wide margin (AlexNet), while PASCAL VOC detection was led by systems built on SIFT- and HOG-like features, such as deformable part models. The authors set aside a sliding-window CNN because its top units had $195 \times 195$ pixel receptive fields and a 32-pixel stride, too coarse for precise localization, and VOC had too few labeled boxes to train a large network from scratch. R-CNN instead classified a few thousand proposals per image, within what the authors call the “recognition using regions” paradigm of earlier work, and transferred the ImageNet network to detection (transfer learning) by fine-tuning it on VOC.

Architecture

R-CNN is a pipeline of separately built modules rather than one network:

Stage	R-CNN (2014)	Output per image
Proposals	Selective search, “fast mode”	About 2,000 boxes
Crop	Dilate each box for 16 pixels of context, warp to $227 \times 227$	One image per box
Features	Caffe reimplementation of Krizhevsky et al.’s network: 5 convolutional, 2 fully connected layers	4,096-d fc7 vector per box
Classification	One linear SVM per class	$N$ scores per box
Localization	One ridge regressor per class on pool5 features	4 box offsets per box and class
Post-processing	Greedy non-maximum suppression, per class	Detections

Region Proposals

R-CNN is agnostic to how proposals are made; the authors chose selective search (Uijlings et al., 2013) to compare directly with earlier detectors that used the same boxes. Selective search is hierarchical grouping. It starts from an over-segmentation into small regions by Felzenszwalb and Huttenlocher’s graph-based method (see superpixels), then repeatedly merges the most similar pair of neighboring regions until one region covers the image. Every region formed along the way contributes its bounding box, so the boxes span all scales. The similarity adds up to four terms, each in $[0, 1]$: colour-histogram intersection, intersection of histograms of Gaussian derivatives (texture), a size term that merges small regions first, and a fill term that prefers pairs whose joint bounding box they cover well. The search is diversified by running it in several colour spaces, with several combinations of similarity terms and several starting segmentations, and pooling the boxes.

The fast mode used by R-CNN combines two colour spaces (HSV and Lab), two similarity combinations and two segmentation thresholds, eight runs in all. On VOC 2007, Uijlings et al. report 2,147 boxes in 3.79 s per image on trainval, and on test a recall of 98.0% at the PASCAL overlap criterion (IoU of 0.5) with 2,134 boxes per image. Proposals therefore cap what the detector can find, but on VOC the cap was high.

Warped Region Features

Each proposal is dilated so that 16 pixels of context surround it after warping, then warped anisotropically to the network’s fixed $227 \times 227$ input; the paper article implements this warp and the alternatives the tech report compared. The crop passes through the five convolutional and two fully connected layers of the Caffe reimplementation of Krizhevsky et al.’s network, and the 4,096-dimensional output of the second fully connected layer, fc7, is the region’s feature. The max-pooled output of the fifth convolutional layer, pool5, is a $6 \times 6 \times 256$ map (9,216 numbers) that the box regressor uses instead.

Class-Specific SVMs

Each class has a linear support vector machine on fc7 features, so all scores of an image are one $2000 \times 4096$ by $4096 \times N$ matrix product for $N$ classes. The CNN computation is shared by all classes, and adding a class costs one column of weights and one round of non-maximum suppression (NMS).

Bounding-Box Regressor

R-CNN’s error analysis showed many detections that were close to an object but poorly localized, and the authors added a regression stage, inspired by the box regression of deformable part models but driven by CNN features (tech report, Appendix C). A proposal $P = (P_x, P_y, P_w, P_h)$, given by its center, width and height in pixels, is mapped to a predicted box $\hat{G}$ by

$$\hat{G}_x = P_w\, d_x(P) + P_x, \qquad \hat{G}_y = P_h\, d_y(P) + P_y, \qquad \hat{G}_w = P_w \exp\big(d_w(P)\big), \qquad \hat{G}_h = P_h \exp\big(d_h(P)\big),$$

where each $d_\star(P) = \mathbf{w}_\star^\top \phi_5(P)$ is a linear function of the proposal’s pool5 features $\phi_5(P)$. Inverting these equations for a ground-truth box $G$ gives the training targets

$$t_x = \frac{G_x - P_x}{P_w}, \qquad t_y = \frac{G_y - P_y}{P_h}, \qquad t_w = \log\frac{G_w}{P_w}, \qquad t_h = \log\frac{G_h}{P_h},$$

and each weight vector solves a ridge regression,

$$\mathbf{w}_\star = \arg\min_{\hat{\mathbf{w}}} \sum_{i} \big(t^i_\star - \hat{\mathbf{w}}^\top \phi_5(P^i)\big)^2 + \lambda \lVert \hat{\mathbf{w}} \rVert^2,$$

which has a closed-form solution. The center offsets are measured in units of the proposal’s size and the size changes in log space. A shift of a tenth of the box width is therefore the same target for a 30-pixel and a 300-pixel proposal, and a decoded width can never be negative. The authors report two details that mattered: regularization, with $\lambda = 1000$ chosen on a validation set, and training only on proposals that overlap a ground-truth box by IoU above 0.6, since learning to map a distant proposal onto an object would be, as the authors put it, “a hopeless learning problem”. Regressors are class-specific, and the regression is applied once: re-scoring and regressing the refined box again did not improve results. The same parameterization later became the target encoding of anchor boxes, with the anchor in place of the proposal.

Training

R-CNN is trained in four separate steps.

1. Supervised pre-training. The network was trained on ILSVRC 2012 classification, with image labels only.
2. Domain-specific fine-tuning. The 1,000-way output layer is replaced by a randomly initialized $(N+1)$-way layer, 21 outputs for VOC’s 20 classes plus background, and stochastic gradient descent continues on warped proposals at a learning rate of 0.001. A proposal is positive for the class of the ground-truth box it overlaps most if that IoU is at least 0.5; all others are background. Each mini-batch of 128 samples 32 positive and 96 background windows, a bias towards the rare positives.
3. SVMs with hard negative mining. Features of all proposals are computed with the fine-tuned network and cached. For each class, the positives are only the ground-truth boxes, the negatives are proposals with IoU below 0.3 to every instance of that class, and proposals in between are ignored. The threshold of 0.3 came from a grid search over $\{0, 0.1, \ldots, 0.5\}$; 0.5 lowered mAP by 5 points and 0 by 4. Because the negatives do not fit in memory, training alternates between fitting the SVM and adding the negatives it scores highest, standard hard negative mining; mAP stopped improving after one pass over the images.
4. Box regressors, fitted as above on cached pool5 features.

Why fine-tuning and the SVMs use different thresholds. The tech report (Appendix B) explains the mismatch historically. The SVM definitions came first, chosen as the best of several options when the SVMs ran on features of the network without fine-tuning; fine-tuning with the same definitions gave much worse results. The authors conjecture that fine-tuning the whole network needs the roughly 30 times more positives that the “jittered” proposals with IoU between 0.5 and 1 provide, at the price of not training for precise localization. That, and a softmax trained on random rather than hard negatives, is their explanation for why the fine-tuned network’s own classifier scored below the SVMs (numbers in the paper article). Fast R-CNN later closed that gap.

Cost. Steps 3 and 4 need features for every proposal of every training image on disk. Girshick (2015) reports that with VGG16 this took 2.5 GPU-days for the 5,000 images of VOC 2007 trainval and hundreds of gigabytes of storage, and that training the whole pipeline took 84 hours, against 22 hours with the AlexNet-sized network.

Example: learning R-CNN’s box regressor

The NumPy code below follows Appendix C on synthetic data. Each $128 \times 128$ image holds one bright rectangle of random size and position on noisy background, and the proposal is the true box with its center and size randomly perturbed. As in R-CNN, the regressor is trained only on proposals with IoU above 0.6, and its features are computed from the warped proposal, with context padding. A $32 \times 32$ crop with 4 pixels of padding stands in for the $227 \times 227$ crop with 16, and the row and column intensity profiles of the crop, 64 numbers plus a bias, stand in for pool5. The code checks that decoding inverts encoding, fits ridge regression for several values of $\lambda$, and then regresses the same features onto raw pixel offsets instead of $(t_x, t_y, t_w, t_h)$.

import numpy as np

def encode(P, G):
    """R-CNN regression targets for proposals P and ground truth G, boxes as (cx, cy, w, h)."""
    return np.stack([(G[:, 0] - P[:, 0]) / P[:, 2], (G[:, 1] - P[:, 1]) / P[:, 3],
                     np.log(G[:, 2] / P[:, 2]), np.log(G[:, 3] / P[:, 3])], axis=1)

def decode(P, d):
    """Inverse of encode: apply predicted offsets d to proposals P."""
    return np.stack([P[:, 0] + P[:, 2] * d[:, 0], P[:, 1] + P[:, 3] * d[:, 1],
                     P[:, 2] * np.exp(d[:, 2]), P[:, 3] * np.exp(d[:, 3])], axis=1)

def iou(A, B):
    """IoU of matching rows of A and B, boxes as (cx, cy, w, h)."""
    ix = np.clip(np.minimum(A[:, 0] + A[:, 2] / 2, B[:, 0] + B[:, 2] / 2)
                 - np.maximum(A[:, 0] - A[:, 2] / 2, B[:, 0] - B[:, 2] / 2), 0, None)
    iy = np.clip(np.minimum(A[:, 1] + A[:, 3] / 2, B[:, 1] + B[:, 3] / 2)
                 - np.maximum(A[:, 1] - A[:, 3] / 2, B[:, 1] - B[:, 3] / 2), 0, None)
    inter = ix * iy
    return inter / (A[:, 2] * A[:, 3] + B[:, 2] * B[:, 3] - inter)

def warp(img, box, size=32, pad=4):
    """Warp box (cx, cy, w, h) to size x size with pad pixels of context (nearest neighbour)."""
    cx, cy, w, h = box
    sx, sy = w / (size - 2 * pad), h / (size - 2 * pad)
    u = np.arange(size) + 0.5 - pad
    xs = np.floor(cx - w / 2 + u * sx).astype(int)
    ys = np.floor(cy - h / 2 + u * sy).astype(int)
    H, W = img.shape
    out = img[np.clip(ys, 0, H - 1)[:, None], np.clip(xs, 0, W - 1)[None, :]]
    outside = (ys[:, None] < 0) | (ys[:, None] >= H) | (xs[None, :] < 0) | (xs[None, :] >= W)
    return np.where(outside, img.mean(), out)

def features(crop):
    """Stand-in for pool5 features: row and column intensity profiles of the warped crop, plus a bias."""
    return np.concatenate([crop.mean(axis=0), crop.mean(axis=1), [1.0]])

def make_set(n, rng, min_iou):
    """n synthetic 128 x 128 images, each with one bright rectangle, and a jittered proposal per image."""
    X, P, G = [], [], []
    while len(X) < n:
        w, h = rng.uniform(24, 80, size=2)
        cx, cy = rng.uniform(w / 2, 128 - w / 2), rng.uniform(h / 2, 128 - h / 2)
        g = np.array([cx, cy, w, h])
        img = rng.normal(0.2, 0.1, size=(128, 128))
        img[int(cy - h / 2):int(cy + h / 2), int(cx - w / 2):int(cx + w / 2)] += 0.6
        p = g * np.r_[1, 1, np.exp(rng.normal(0, 0.15, 2))] + np.r_[rng.normal(0, 0.12, 2) * g[2:], 0, 0]
        if iou(p[None], g[None])[0] < min_iou:          # R-CNN trains only on proposals near a box
            continue
        X.append(features(warp(img, p))); P.append(p); G.append(g)
    return np.array(X), np.array(P), np.array(G)

rng = np.random.default_rng(0)
Xtr, Ptr, Gtr = make_set(600, rng, min_iou=0.6)
Xte, Pte, Gte = make_set(300, rng, min_iou=0.5)

T = encode(Ptr, Gtr)
print("round trip max |decode(encode(P, G)) - G| =", f"{np.abs(decode(Ptr, T) - Gtr).max():.1e}")
print("target std (tx, ty, tw, th):", np.round(T.std(axis=0), 3))

base = iou(Pte, Gte)
print(f"\ntest proposals: mean IoU {base.mean():.3f}, IoU >= 0.7: {np.mean(base >= 0.7):.0%}")
print("lambda   mean IoU after regression   IoU >= 0.7")
for lam in [0.0, 0.1, 1.0, 10.0, 100.0]:
    A = Xtr.T @ Xtr + lam * np.eye(Xtr.shape[1])
    Wr = np.linalg.solve(A, Xtr.T @ T)                   # ridge regression, one column per target
    after = iou(decode(Pte, Xte @ Wr), Gte)
    print(f"{lam:6.1f}   {after.mean():.3f}                       {np.mean(after >= 0.7):.0%}")

# The same features regressed onto raw pixel offsets (Gx - Px, ..., Gh - Ph) instead of (tx, ty, tw, th).
A = Xtr.T @ Xtr + 1.0 * np.eye(Xtr.shape[1])
Wp = np.linalg.solve(A, Xtr.T @ (Gtr - Ptr))
print(f"\nraw pixel offsets, lambda 1: mean IoU {iou(Pte + Xte @ Wp, Gte).mean():.3f}")

round trip max |decode(encode(P, G)) - G| = 1.4e-14
target std (tx, ty, tw, th): [0.087 0.091 0.134 0.134]

test proposals: mean IoU 0.675, IoU >= 0.7: 40%
lambda   mean IoU after regression   IoU >= 0.7
   0.0   0.899                       100%
   0.1   0.900                       100%
   1.0   0.900                       100%
  10.0   0.885                       98%
 100.0   0.793                       86%

raw pixel offsets, lambda 1: mean IoU 0.846

Decoding inverts encoding up to rounding error, so a regressor that predicted the targets exactly would recover every box. The learned linear regressor lifts the test proposals from a mean IoU of 0.675 to about 0.90, and every test box then reaches IoU 0.7. Light regularization changes little here, where 600 examples fit 65 weights per target; strong regularization pulls the predictions towards zero offsets and gives up much of the gain. R-CNN’s $\lambda = 1000$ was set for 9,216-dimensional pool5 features with a different scale, so the values are not comparable. The last line is the more instructive one. Regressing the same features onto pixel offsets is worse (0.846), and by our reasoning this is because the features are computed in the warped proposal’s frame, where an edge one tenth of the way in from the crop’s border means a shift of one tenth of the proposal’s width, whatever its size in pixels. The scale-normalized targets make that relation linear; pixel targets would need it multiplied by the proposal’s size.

Inference

At test time R-CNN runs selective search, warps each of the roughly 2,000 proposals, computes its fc7 and pool5 features with one forward pass per proposal, scores every proposal with every class’s SVM, applies each class’s box regressor, and runs greedy NMS for each class separately. The tech report gives 13 s per image on a GPU and 53 s on a CPU for proposals and features together. Girshick (2015) measured 9.8 s per image on an Nvidia K40 GPU for the AlexNet-sized network and 47 s with VGG16. Almost all of this is the 2,000 separate CNN passes over overlapping crops; the per-class work is small.

Performance

All results below are as reported by the authors, as mAP at IoU 0.5 on PASCAL VOC (see average precision). With the AlexNet-sized network and box regression, R-CNN reached 53.7% on VOC 2010 test, 53.3% on VOC 2011/12 test and 58.5% on VOC 2007 test, where DPM v5 scored 33.7%; box regression added 3 to 4 points. With VGG16 in place of the smaller network, the tech report’s version 5 gives 66.0% on VOC 2007 test, and Fast R-CNN’s tables list 62.4% on VOC 2012 test for that configuration, citing the journal version (Girshick et al., 2016). The paper article has the ablations and the ILSVRC 2013 results.

Strengths

• Transfer from classification. Features learned on ImageNet, fine-tuned on a few thousand detection images, outperformed hand-designed features by a wide margin, which set the pattern of pre-training and fine-tuning for detection.
• Scales with the number of classes. Proposals and features are shared by all classes; each class adds only a linear classifier and a regressor.
• Modular. Any proposal method or network can be swapped in; the VGG16 version raised VOC 2007 mAP by 7.5 points with the same fine-tuning protocol, apart from smaller mini-batches.

Limitations

• Slow at test time. A full CNN pass for each of about 2,000 overlapping crops recomputes the same convolutions many times.
• Multi-stage, disk-bound training. Fine-tuning, SVMs and regressors are trained separately, the last two on cached features, and none of the later stages updates the network.
• Inconsistent objectives. The network is fine-tuned with one definition of positives and the SVMs are trained with another; the softmax learned during fine-tuning is discarded.
• Fixed proposals. Recall is capped by selective search, which also runs on the CPU and cannot be learned.
• Distorted crops. Anisotropic warping stretches every object to a square, so the network sees the same class at many aspect ratios.

Historical Importance

The authors present R-CNN as the first demonstration that a CNN gives much higher detection accuracy on PASCAL VOC than systems built on HOG-like features. In our reading, it changed the question in object detection from which hand-designed features to use to how to compute CNN features for many regions efficiently. Its box parameterization and the idea of classifying a sparse set of proposals outlived the pipeline and run through the whole two-stage line. Each successor in this article removed one limitation: SPPnet the repeated convolutions, Fast R-CNN the multi-stage training, and Faster R-CNN (Ren et al., 2015) the external proposals. The CVPR 2014 paper received the PAMI Longuet-Higgins Prize in 2024 (see the paper article).

Variants

SPPnet (2014)

SPPnet (He et al., arXiv June 2014, ECCV 2014; details here from the TPAMI journal version, read as arXiv v4) computes the convolutional feature maps of the whole image once and, for each proposal window, max-pools the part of the conv5 map inside it into a fixed number of bins. A spatial pyramid of $1 \times 1$, $2 \times 2$, $3 \times 3$ and $6 \times 6$ bins gives 50 bins and a 12,800-dimensional vector ($256 \times 50$) for any window size, which then passes through the fully connected layers (see pooling for the layer itself). A window is projected onto the feature map through its corners: with $S$ the product of all strides before conv5 (16 for the ZF-5 network used), the journal version’s appendix maps the left or top edge $x$ to $\lfloor x/S \rfloor + 1$ and the right or bottom edge to $\lceil x/S \rceil - 1$.

For scale, SPPnet resizes the image so that its shorter side is $s$; the five-scale version uses $s \in \{480, 576, 688, 864, 1200\}$ and pools each window from the single scale at which it is closest to $224 \times 224$ pixels. The authors estimate the convolutional cost as $O(n \cdot 227^2)$ for R-CNN with $n \approx 2000$ windows and $O(r \cdot s^2)$ for SPPnet with aspect ratio $r \approx 4/3$, about 1/160 of R-CNN’s at $s = 688$. Training otherwise followed R-CNN, with SVMs, box regression on the pooled conv5 features and, as the main difference, fine-tuning of the fully connected layers only.

With the ZF-5 network and box regression, SPPnet reported 59.2% mAP on VOC 2007 test with five scales and 58.0% with one, against 58.5% for R-CNN with AlexNet (journal version, Table 9). In the journal version’s same-network comparison (Table 10: ZF-5 for both, timed on a GPU), feature computation took 0.142 s per image with one scale and 0.382 s with five, against 14.46 s for R-CNN: 102 and 38 times faster, at 58.0% and 59.2% mAP against R-CNN’s 59.2%. The versions differ here: arXiv v1 states speedups of 24 to 64 times overall, measured against R-CNN with AlexNet as in Table 9, and the journal abstract’s 24 to 102 times spans both tables, adding the same-network comparison.

Fast R-CNN (2015)

Fast R-CNN (Girshick, arXiv April 2015, ICCV 2015) keeps SPPnet’s shared feature map and makes everything after it one network, trained in one stage. The network takes an image and a list of proposals. Its convolutional layers produce one feature map, an RoI pooling layer extracts a fixed-size feature for each proposal, and fully connected layers end in two sibling outputs: a softmax over $K + 1$ classes and four box offsets for each of the $K$ object classes.

RoI pooling. An RoI is a rectangle on the feature map. RoI max pooling divides an $h \times w$ RoI into an $H \times W$ grid of sub-windows of about $h/H \times w/W$ cells and max-pools each channel in each sub-window; $H = W = 7$ for VGG16, to match its first fully connected layer. It is SPPnet’s pooling with a single pyramid level, with the sub-window computation taken from SPPnet. To build the network from an ImageNet model, the last max-pooling layer becomes the RoI pooling layer, the final classifier becomes the two sibling layers, and the network is given two inputs, images and RoIs.

Why SPPnet could not fine-tune its convolutions. Girshick (2015) locates the cause in how R-CNN and SPPnet sampled training data: each mini-batch held RoIs from many different images. An RoI’s receptive field often covers the whole image, so back-propagating through the SPP layer for 128 RoIs from 128 images would need a forward and backward pass over 128 full images. Fast R-CNN samples hierarchically instead, 64 RoIs from each of 2 images, so the RoIs of one image share all convolutional computation; the author saw no convergence problem from the correlated RoIs. The two-stage object detectors article gives the foreground and background sampling rules, which keep R-CNN’s IoU threshold of 0.5 for foreground.

Back-propagation through RoI pooling. With $x_i$ the inputs to the layer and $y_{rj}$ the $j$-th output of RoI $r$, the forward pass sets $y_{rj} = x_{i^*(r,j)}$, where $i^*(r, j)$ is the argmax over the sub-window. The gradient follows the argmax switches and accumulates over every RoI and output that selected an input:

$$\frac{\partial L}{\partial x_i} = \sum_r \sum_j \big[i = i^*(r, j)\big] \frac{\partial L}{\partial y_{rj}}.$$

Overlapping RoIs therefore add their gradients in the shared feature map.

Multi-task loss. Each RoI’s loss is the log loss of its true class plus, for foreground RoIs only, a smooth $L_1$ loss between the predicted offsets for that class and targets in R-CNN’s parameterization; smooth $L_1$ grows linearly for large errors, avoiding the learning-rate tuning that the $L_2$ loss of R-CNN and SPPnet could need. The two-stage object detectors article gives the formula. The softmax replaces the SVMs and the regression layer replaces the ridge regressors, so one round of SGD trains everything, and no features are written to disk.

Which layers to fine-tune. Freezing all 13 convolutional layers of VGG16, which emulates SPPnet’s training at one scale, lowered VOC 2007 mAP from 66.9% to 61.4%. Training from conv2_1 instead of conv3_1 gained only 0.3 points at 1.3 times the training time, and including conv1 exceeded GPU memory, so the VGG16 models fine-tune from conv3_1 up.

Truncated SVD. In detection the fully connected layers run once per RoI, and before compression fc6 and fc7 took about 45% of VGG16’s forward pass. Fast R-CNN factorizes a $u \times v$ weight matrix with its top $t$ singular values,

$$W \approx U \Sigma_t V^\top,$$

where $U$ is $u \times t$, $\Sigma_t$ is $t \times t$ and $V$ is $v \times t$ (see singular value decomposition). The layer becomes two fully connected layers with no nonlinearity between them, the first with weights $\Sigma_t V^\top$ and no bias, the second with $U$ and the original bias, which reduces the parameters from $uv$ to $t(u + v)$. Keeping 1,024 singular values of the $25088 \times 4096$ fc6 matrix and 256 of the $4096 \times 4096$ fc7 matrix cut detection time by more than 30%, from 320 ms to 223 ms per image, for a 0.3-point mAP drop (66.9% to 66.6%) without re-training.

Design experiments. Fast R-CNN made training fast enough to test design questions directly on VOC 2007. In the author’s experiments:

• Training with the multi-task loss improved classification alone by 0.8 to 1.1 mAP points over training with the classification loss only, for all three networks; adding a box regressor afterwards, with the rest frozen, did worse than joint training.
• Image pyramids with five scales gained 1.3 and 1.5 points for the two smaller networks at about four times the test time; single-scale processing at $s = 600$ pixels (longest side capped at 1,000) gave the best trade-off. Since PASCAL images average $384 \times 473$ pixels, the effective stride at the RoI pooling layer is about 10 pixels.
• Adding VOC 2012 trainval to VOC 2007 trainval, about 16,500 images, raised VOC 2007 test mAP from 66.9% to 70.0%.
• Post-hoc SVMs trained with hard negative mining were 0.1 to 0.8 points worse than the network’s own softmax, which, unlike one-vs-rest SVMs, makes the classes compete for each RoI.
• More proposals did not help. Sweeping selective search’s quality mode from 1,000 to 10,000 boxes per image, mAP rose and then fell slightly while average recall kept rising, so the author cautions against judging proposals by recall alone; about 45,000 dense boxes instead of proposals did worse still.

Results. With VGG16, the author reported 66.9% mAP on VOC 2007 test, 66.1% on VOC 2010 test and 65.7% on VOC 2012 test, where R-CNN with the same network scored 66.0%, 62.9% and 62.4%. Against R-CNN with VGG16, training was 9 times faster (9.5 against 84 hours) and testing 146 times faster, or 213 times with truncated SVD (0.22 s per image, excluding proposals). Against SPPnet with VGG16, training was 2.7 times faster and testing 7 or 10 times faster, and the mAP was higher (66.9% against 63.1%), which the author attributes to fine-tuning the convolutional layers, despite SPPnet’s five scales. A preliminary COCO baseline gave 35.9% PASCAL-style mAP and 19.7% COCO-style AP on test-dev. With selective search taking 1 to 2 s per image on a CPU, by the SPPnet journal version’s estimate, the proposals now took longer than the detector itself.

Example: warping against RoI pooling

The code compares the two ways of producing a fixed-size feature for a proposal, using a toy network, $16 \times 16$ max pooling with stride 16, on a synthetic $240 \times 320$ image. R-CNN’s route warps the proposal’s pixels to a $64 \times 64$ crop and runs the network on it, giving a $4 \times 4$ output. Fast R-CNN’s route runs the network once on the whole image and RoI-pools each proposal from that shared map into $4 \times 4$ bins, after rounding the box to the nearest cells. The proposals are a $128 \times 128$ box on a disc whose brightness falls from its center and a $24 \times 24$ box on a small square whose brightness ramps from left to right. The last line repeats SPPnet’s estimate of the convolutional work.

import numpy as np

STRIDE, OUT = 16, 4                         # feature-map stride; pooled output is OUT x OUT

# Synthetic 240 x 320 image on black: a disc whose brightness falls from 1 at its center to 0 at its
# rim, and a 24 x 24 square whose brightness ramps from 0.2 (left) to 1.0 (right).
H, W = 240, 320
yy, xx = np.mgrid[0:H, 0:W] + 0.5
img = np.clip(1 - np.hypot(xx - 100, yy - 120) / 64, 0, None)
img[146:170, 266:290] = np.linspace(0.2, 1.0, 24)[None, :]

def block_max(a, s=STRIDE):
    """Toy 'network': max over s x s blocks, a feature map with stride s."""
    h, w = a.shape[0] // s, a.shape[1] // s
    return a[:h * s, :w * s].reshape(h, s, w, s).max(axis=(1, 3))

def warp_then_features(img, box):
    """R-CNN route: warp the box's pixels to OUT*STRIDE pixels square, then run the network on the crop."""
    x1, y1, x2, y2 = box
    n = OUT * STRIDE
    u = np.arange(n) + 0.5
    xs = np.floor(x1 + u * (x2 - x1) / n).astype(int)    # nearest-neighbour anisotropic warp
    ys = np.floor(y1 + u * (y2 - y1) / n).astype(int)
    return block_max(img[ys[:, None], xs[None, :]])

def roi_pool(fmap, box):
    """Fast R-CNN route: project the box onto the shared map, split into OUT x OUT bins, max-pool."""
    c1, r1, c2, r2 = [int(np.floor(v / STRIDE + 0.5)) for v in box]    # round to the nearest cell
    c2, r2 = max(c2, c1 + 1), max(r2, r1 + 1)                          # at least one cell
    Y = np.zeros((OUT, OUT))
    for i in range(OUT):
        a = r1 + int(np.floor(i * (r2 - r1) / OUT)); b = r1 + int(np.ceil((i + 1) * (r2 - r1) / OUT))
        for j in range(OUT):
            c = c1 + int(np.floor(j * (c2 - c1) / OUT)); d = c1 + int(np.ceil((j + 1) * (c2 - c1) / OUT))
            Y[i, j] = fmap[a:b, c:d].max()
    return Y, (c1 * STRIDE, r1 * STRIDE, c2 * STRIDE, r2 * STRIDE)

fmap = block_max(img)                       # computed once for the whole image
print("shared feature map:", fmap.shape, "cells")
np.set_printoptions(precision=2, suppress=True)
proposals = {"disc, 128 x 128": (36, 56, 164, 184), "small square, 24 x 24": (266, 146, 290, 170)}
for name, box in proposals.items():
    A = warp_then_features(img, box)
    B, region = roi_pool(fmap, box)
    print(f"\n{name} px, box {box}; RoI pooling reads pixels {region}")
    print("warp then network:\n", A, "\nRoI pooling on the shared map:\n", B)
    print("max |difference| =", f"{np.abs(A - B).max():.2f}")

# Convolutional work, counted in input pixels the network must process (SPPnet's comparison).
n, s, r = 2000, 688, 4 / 3
print(f"\nR-CNN: {n} x 227^2 = {n * 227**2 / 1e6:.0f} M pixels;  shared map: (4/3) x {s}^2 = "
      f"{r * s**2 / 1e6:.2f} M pixels;  ratio {n * 227**2 / (r * s**2):.0f}")

shared feature map: (15, 20) cells

disc, 128 x 128 px, box (36, 56, 164, 184); RoI pooling reads pixels (32, 64, 160, 192)
warp then network:
 [[0.28 0.49 0.49 0.27]
 [0.49 0.99 0.98 0.48]
 [0.49 0.98 0.97 0.48]
 [0.27 0.48 0.48 0.26]] 
RoI pooling on the shared map:
 [[0.31 0.61 0.62 0.41]
 [0.43 0.93 0.99 0.55]
 [0.41 0.85 0.87 0.54]
 [0.15 0.36 0.37 0.23]]
max |difference| = 0.14

small square, 24 x 24 px, box (266, 146, 290, 170); RoI pooling reads pixels (272, 144, 288, 176)
warp then network:
 [[0.37 0.58 0.79 1.  ]
 [0.37 0.58 0.79 1.  ]
 [0.37 0.58 0.79 1.  ]
 [0.37 0.58 0.79 1.  ]] 
RoI pooling on the shared map:
 [[0.93 0.93 0.93 0.93]
 [0.93 0.93 0.93 0.93]
 [0.93 0.93 0.93 0.93]
 [0.93 0.93 0.93 0.93]]
max |difference| = 0.56

R-CNN: 2000 x 227^2 = 103 M pixels;  shared map: (4/3) x 688^2 = 0.63 M pixels;  ratio 163

For the large proposal, both routes see the same bright center and dark corners. The warped crop gives a symmetric pattern because it is aligned with the box; the RoI-pooled features are shifted down, because rounding moved the box’s top and bottom edges 8 pixels down and its sides 4 pixels left. For the $24 \times 24$ square, rounding leaves one column of cells. Every bin reads from that column, whose two cells hold the same maximum, and the left-to-right ramp, which the warped crop resolves into four levels, disappears. This is the price of sharing: the shared map is computed once, but a small proposal gets features no finer than the map’s stride. In a real network the cells have large receptive fields and many channels, so the loss is softer than in this one-channel toy, but the rounding is the same, and it is what RoIAlign later removed for Mask R-CNN. The work count shows what sharing buys: about 163 times fewer input pixels pass through the convolutional layers, in line with SPPnet’s estimate of 1/160.

Faster R-CNN and Mask R-CNN

Faster R-CNN (Ren et al., 2015) replaced selective search with a region proposal network on Fast R-CNN’s shared features, and Mask R-CNN (He et al., 2017) added a mask branch and replaced RoI pooling with RoIAlign. The two-stage object detectors article compares RoI pooling with RoIAlign and follows the line further.

Implementations

The official R-CNN code (rbgirshick/rcnn) uses MATLAB and Caffe and is released under the Simplified BSD License; its README describes it as a historical artifact that is no longer maintained. The official Fast R-CNN code (rbgirshick/fast-rcnn) is written in Python and C++ with Caffe under the MIT License and is marked deprecated in favour of Detectron. As of October 2026, TorchVision provides the RoI pooling operator as torchvision.ops.roi_pool, documented as the operator described in Fast R-CNN, which takes the feature maps, the boxes in $(x_1, y_1, x_2, y_2)$ format, the output size and a spatial_scale that maps box coordinates to feature-map coordinates (1/16 for a stride of 16).