Fast R-CNN

Fast R-CNN is a region-based object detector introduced by Ross Girshick of Microsoft Research, first as an arXiv report in April 2015 and then at ICCV 2015. Like R-CNN, it classifies a few thousand class-agnostic region proposals per image and refines their boxes. Unlike R-CNN, it runs the convolutional neural network once over the whole image, takes each proposal’s feature from the shared feature map with an RoI pooling layer, and trains the classifier and the box regressor together, in one stage, with a single loss. The name, its author wrote, refers to its being comparatively fast to train and to test. It sits between R-CNN (2014) and SPP-net (2014) before it and Faster R-CNN (2015) after it, which kept Fast R-CNN as its detection head and replaced the external proposals. The R-CNN article follows the whole line in outline; this article covers Fast R-CNN itself: what it took from SPP-net, its architecture and training, truncated SVD, and its results and ablations as reported in the ICCV paper.

Problem Addressed

Girshick (2015) lists three drawbacks of R-CNN (Girshick et al., 2014). Training was a pipeline of three stages: fine-tune the network on proposals with log loss, fit one linear SVM per class on its features, then fit box regressors. Training was costly in time and space: features of every proposal of every image were written to disk, which for VGG16 took 2.5 GPU-days on the 5,000 images of VOC 2007 trainval and hundreds of gigabytes. And detection was slow, 47 s per image with VGG16 on a GPU, because the network ran once per proposal and shared nothing.

SPP-net, the predecessor

Spatial pyramid pooling networks (He, Zhang, Ren and Sun, ECCV 2014, with a journal version in TPAMI in 2015; read here in arXiv versions v1 and v4) removed the repeated computation. SPP-net computes the conv5 feature maps of the whole image once and, for each candidate window, max-pools the part of the map inside it with a four-level spatial pyramid of $1 \times 1$ , $2 \times 2$ , $3 \times 3$ and $6 \times 6$ bins, 50 bins and a 12,800-dimensional vector for the ZF-5 network’s 256 channels, whatever the window’s size. A five-scale version resizes the image so that its shorter side is one of $\{480, 576, 688, 864, 1200\}$ and pools each window at the scale where it is closest to $224 \times 224$ pixels. With ZF-5 and box regression, SPP-net reached 59.2% mAP on VOC 2007 test with five scales and 58.0% with one, against 58.5% for R-CNN with AlexNet, at 24 and 64 times R-CNN’s speed (arXiv v4, Table 9).

The rest of the pipeline stayed R-CNN’s. Pooled features were cached; fully connected layers fc6 and fc7 and a new 21-way fc8 were fine-tuned on them, with positives overlapping a ground-truth box by IoU in $[0.5, 1]$ , negatives by $[0.1, 0.5)$ and 25% positives per mini-batch; linear SVMs were then trained with hard negative mining, and box regressors on the pooled conv5 features. The convolutional layers were not fine-tuned at all. The SPP-net papers give the reason in two words, “for simplicity” (He et al.), since the cached features came from windows of any size. Girshick (2015) gives a different, structural reason, discussed under Training below, and argues that the frozen convolutions limit the accuracy of very deep networks.

Fast R-CNN set out to fix both methods’ shortcomings at once. Its paper claims four advantages: higher mAP than R-CNN and SPP-net, single-stage training with a multi-task loss, training that can update all layers, and no disk storage for cached features.

Architecture

A Fast R-CNN network takes two inputs, an image and a list of regions of interest (RoIs), normally the image’s precomputed proposals. Convolutional and max-pooling layers turn the image into a feature map; for each RoI, an RoI pooling layer extracts a fixed-size feature from that map; fully connected layers turn it into a feature vector, which feeds two sibling output layers. One gives softmax probabilities over $K$ object classes plus a catch-all background class; the other gives four box offsets for each of the $K$ classes.

Backbone

The paper experiments with three networks pretrained on ImageNet, each with five max-pooling layers and five to thirteen convolutional layers: CaffeNet, essentially AlexNet, called model S (small); VGG_CNN_M_1024, as deep as S but wider, called model M (medium); and VGG16, called model L (large). A pretrained network becomes a Fast R-CNN network through three changes:

The last max-pooling layer is replaced by an RoI pooling layer whose output size matches the first fully connected layer: $H = W = 7$ for VGG16.
The last fully connected layer and the 1,000-way softmax are replaced by the two sibling layers: a fully connected layer with softmax over $K + 1$ categories and the class-specific box regressors.
The network takes a list of images and a list of RoIs in those images.

RoI pooling layer

An RoI is a rectangle on the feature map, written $(r, c, h, w)$ for its top-left corner and its height and width. RoI max pooling divides the $h \times w$ window into an $H \times W$ grid of sub-windows of about $h/H \times w/W$ cells and max-pools each channel in each sub-window. Girshick describes it as the special case of SPP-net’s pyramid with a single level, using SPP-net’s sub-window computation. In the released Caffe layer, the box corners are scaled by the map’s spatial scale ( $1/16$ for VGG16) and rounded to the nearest cell, and each bin runs from the floor of its start to the ceiling of its end. The RoI pooling article covers the operator, its rounding and its successors, RoIAlign among them.

Because pooling takes a maximum, the layer’s backward pass (the paper’s Eq. 4, written out in the RoI pooling article) routes each output’s gradient to the input cell that won, and a cell that is the maximum for several bins, in one RoI or in overlapping RoIs, accumulates all their gradients. This is what lets the loss reach the convolutional layers, which SPP-net never trained.

Sibling output layers

For each RoI the classification layer outputs $p = (p_0, \dots, p_K)$ , a softmax over $K + 1$ outputs, with class 0 the background. The regression layer outputs $t^k = (t^k_x, t^k_y, t^k_w, t^k_h)$ for each object class $k$ , in R-CNN’s parameterization: a translation of the box centre relative to the proposal’s width and height, and log-space changes of width and height (see bounding box regression). The regression is class-specific: every class gets its own refined box for the same RoI.

Training

Fine-tuning trains every layer from conv3_1 up in VGG16 (from conv2 up in models S and M) with one SGD procedure; nothing is cached and no second classifier is fitted.

Hierarchical mini-batch sampling

Girshick traces SPP-net’s frozen convolutions to how R-CNN and SPP-net sampled training data: each mini-batch took its RoIs from many different images. An RoI’s receptive field can cover the whole image, so back-propagating through the pooling layer for 128 RoIs from 128 images would mean a forward and backward pass over 128 large inputs. Fast R-CNN samples hierarchically: $N$ images first, then $R/N$ RoIs from each. RoIs from one image share the convolutional computation and memory in both passes, so with $N = 2$ and $R = 128$ training is roughly 64 times faster than taking one RoI from each of 128 images. The author expected that correlated RoIs from the same image might slow convergence and reports that they did not; the method needed fewer SGD iterations than R-CNN.

Each mini-batch has two images, drawn by iterating over permutations of the dataset, and 64 RoIs from each. A quarter of the RoIs are foreground: proposals whose IoU with a ground-truth box is at least 0.5, labelled with that box’s class $u \ge 1$ . The rest are background, $u = 0$ , from proposals whose maximum IoU lies in $[0.1, 0.5)$ , a range the paper takes from SPP-net. The lower bound, the paper says, appears to act as a heuristic for hard example mining; the label assignment and hard negative mining articles discuss both rules and what later detectors replaced them with. Images are flipped horizontally with probability 0.5, and no other data augmentation is used. Three details appear in the released code rather than the paper: the ground-truth boxes are added to each training image’s proposals, so they can be sampled as foreground RoIs; an image with too few foreground or background proposals contributes a smaller sample rather than one drawn with replacement; and only RoIs with IoU at least 0.5 receive regression targets.

Multi-task loss

Each labelled RoI contributes

L(p, u, t^u, v) = L_{\text{cls}}(p, u) + \lambda [u \ge 1] L_{\text{loc}}(t^u, v),

where $L_{\text{cls}}(p, u) = -\log p_u$ is the log loss of the true class, $v = (v_x, v_y, v_w, v_h)$ is the regression target and $t^u$ the predicted offsets for the true class. The Iverson bracket $[u \ge 1]$ switches the box term off for background RoIs, which have no box. The box term is

L_{\text{loc}}(t^u, v) = \sum_{i \in \{x, y, w, h\}} \operatorname{smooth}_{L_1}(t^u_i - v_i), \qquad \operatorname{smooth}_{L_1}(x) = \begin{cases} 0.5 x^2 & \text{if } |x| < 1, \\ |x| - 0.5 & \text{otherwise.} \end{cases}

Girshick chose this robust loss because it is less sensitive to outliers than the $L_2$ loss of R-CNN and SPP-net: with unbounded targets, $L_2$ training can need careful tuning of learning rates to avoid exploding gradients. Only the regression outputs of the true class receive gradient. The targets are normalized to zero mean and unit variance and all experiments use $\lambda = 1$ ; the released code computes the means and standard deviations separately for each class and, when it saves a trained model, folds them back into the regression layer’s weights and biases, so that the saved network outputs unnormalized offsets. The bounding box regression article compares smooth $L_1$ with the losses that came after it.

The paper sets its joint training against stage-wise schemes, OverFeat’s, R-CNN’s and SPP-net’s, and against Erhan et al.’s scalable detection network, which used a related loss to train a class-agnostic proposal network separate from the classifier.

Optimization

The classification and regression layers are initialized from zero-mean Gaussians with standard deviations 0.01 and 0.001, biases at zero. Every layer uses a per-layer learning rate of 1 for weights and 2 for biases, with a global rate of 0.001, momentum 0.9 and weight decay 0.0005 on weights and biases. On VOC 2007 or VOC 2012 trainval, SGD runs 30,000 mini-batch iterations, then 10,000 more at a rate of 0.0001. On the 16,500 images of VOC 2007 and 2012 trainval combined it ran 60,000 iterations, and on the 21,500-image union of VOC 2007 trainval and test with VOC 2012 trainval, 100,000 iterations with the rate lowered tenfold every 40,000.

Which layers to update was a memory and time decision for VGG16. Updating from conv2_1 instead of conv3_1 gained 0.3 points of mAP for 1.3 times the training time (12.5 against 9.5 hours), and updating conv1_1 too ran out of GPU memory; in models S and M, letting conv1 learn made no meaningful difference, which the paper attributes to conv1 being generic.

Scale

The paper compares two ways of handling object scale, both from SPP-net. In “brute force” single-scale training and testing, every image is resized so that its shorter side $s$ is 600 pixels, with the longer side capped at 1,000, and the network must learn scale invariance itself. These values were chosen so that VGG16 fits in GPU memory during fine-tuning. PASCAL images average $384 \times 473$ pixels, so this usually upsamples them by about 1.6, and the effective stride at the RoI pooling layer is about 10 pixels. In multi-scale training, each sampled image gets a random scale from SPP-net’s five, as a form of data augmentation, with the longer side capped at 2,000; this was possible for models S and M only, because of GPU memory.

Inference

Detection is one forward pass over the image, or an image pyramid given as a list of images, and its list of proposals, typically about 2,000; the paper also tests about 45,000. In a pyramid each RoI is pooled from the scale at which its area is closest to $224^2$ pixels, as in SPP-net. For each RoI the network outputs class probabilities and one refined box per class, and the confidence for class $k$ is simply $p_k$ . Non-maximum suppression then runs separately for each class with R-CNN’s algorithm and settings; the released code suppresses boxes with IoU of 0.3 or more (see non-maximum suppression). On a K40 GPU the VGG16 network took 0.32 s per image, excluding the proposals.

Truncated SVD

In image classification the fully connected layers cost little next to the convolutions. In detection they run once per RoI, and before compression fc6 and fc7 took 45% of VGG16’s forward pass in Fast R-CNN (Fig. 2 of the paper: 122 ms and 20 ms of about 320 ms). Fast R-CNN compresses them with truncated singular value decomposition, citing earlier uses in convolutional networks and in speech recognition. A $u \times v$ weight matrix is approximated by

W \approx U \Sigma_t V^\top,

with $U$ the first $t$ left-singular vectors ( $u \times t$ ), $\Sigma_t$ the top $t$ singular values and $V$ the first $t$ right-singular vectors ( $v \times t$ ). The layer becomes two fully connected layers with no nonlinearity between them, the first with weights $\Sigma_t V^\top$ and no bias, the second with $U$ and the original bias, and the parameter count falls from $uv$ to $t(u + v)$ . Keeping 1,024 singular values of the $25088 \times 4096$ fc6 matrix and 256 of the $4096 \times 4096$ fc7 matrix reduces those layers from about 102.8 million and 16.8 million weights to 29.9 million and 2.1 million (our arithmetic). The forward pass fell from 320 ms to 223 ms, fc6 from 122 ms to 37 ms, and VOC 2007 mAP from 66.9% to 66.6%, without fine-tuning after compression. The paper adds that fine-tuning again after compression would allow further speed-ups with smaller drops. The released code applies the factorization to a trained model with a separate script.

Example: shared features against cropping each proposal

The example trains two small detectors on synthetic images of shapes and changes only how a proposal gets its features. The shared route is Fast R-CNN’s: the backbone runs once per image, at stride 4, and each proposal is RoI-max-pooled to $4 \times 4$ from the shared map with the Caffe layer’s rounding and bin rule (except at the right and bottom edges, where the code clips the rounded box to the map before dividing it into bins, while Caffe clips each bin). The crop route takes R-CNN’s input: each proposal, enlarged by R-CNN’s context margin (16 pixels at $227 \times 227$ , the same fraction here), is warped by bilinear sampling to $16 \times 16$ pixels and passed through the same backbone, which leaves a $4 \times 4$ map. Both routes use the same head, the multi-task loss with $\lambda = 1$ and normalized targets, Fast R-CNN’s initialization of the output layers, and its mini-batches scaled down: 2 images, up to 32 RoIs each, a quarter foreground, background from $[0.1, 0.5)$ , flips, and ground-truth boxes added to the training RoIs as in the released code. The crop route is therefore not R-CNN, which trained SVMs on cached features; it isolates the one thing Fast R-CNN changed in how features are computed.

The images are $64 \times 64$ pixels with one to three filled squares, discs and triangles of 8 to 28 pixels a side, of random brightness, on a sloped background with Gaussian noise. Selective search is not run; a stand-in generator gives 40 proposals per image, six jittered copies of each object’s box plus random boxes. Before training, the script checks that the task is learnable but not trivial: the shapes stand out from the noise, mean brightness alone does not separate the classes, and most proposals that count as foreground need refining. It then counts the convolutional work per image, trains both routes with seeds 0, 1 and 2, and scores mAP on 300 test images in the VOC style (per-class NMS at 0.3, AP from the precision–recall curve, see average precision) at IoU 0.5 and 0.7, with and without the predicted box offsets. Finally, it applies truncated SVD to fc6 of one trained shared model. The script uses PyTorch 2.14.1 and NumPy 2.5.4 on the CPU with four threads, and runs in about 100 seconds on the machine it was drafted on, almost all of it training the six models.

import math
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(4)
SIZE, STRIDE, POOL, K = 64, 4, 4, 3          # image side, backbone stride, RoI grid, classes
NAMES = ["square", "disc", "triangle"]
N_PROP, CROP = 40, 16                        # proposals per image, warp size for the crop route


# ---------- synthetic data: shapes with known boxes, and proposals ----------
def render(rng):
    yy, xx = np.mgrid[0:SIZE, 0:SIZE] + 0.5
    g = rng.uniform(-0.15, 0.15, 2)
    bg0 = rng.uniform(0.05, 0.25)
    img = bg0 + g[0] * (xx - 32) / 64 + g[1] * (yy - 32) / 64
    boxes, labels, contrast, inside = [], [], [], []
    for _ in range(rng.integers(1, 4)):
        for _try in range(20):
            w, h = rng.uniform(8, 28, 2)
            x1, y1 = rng.uniform(0, SIZE - w), rng.uniform(0, SIZE - h)
            b = np.array([x1, y1, x1 + w, y1 + h])
            if all(iou(b, o) < 0.1 for o in boxes):
                break
        else:
            continue
        c = rng.integers(K)
        cx, cy = x1 + w / 2, y1 + h / 2
        if c == 0:
            m = (xx >= x1) & (xx < x1 + w) & (yy >= y1) & (yy < y1 + h)
        elif c == 1:
            m = ((xx - cx) / (w / 2)) ** 2 + ((yy - cy) / (h / 2)) ** 2 <= 1
        else:  # apex at top centre, base at the bottom
            t = (yy - y1) / h
            m = (yy >= y1) & (yy < y1 + h) & (np.abs(xx - cx) <= t * w / 2)
        bg_c = bg0 + g[0] * (cx - 32) / 64 + g[1] * (cy - 32) / 64
        val = bg_c + rng.uniform(0.25, 0.6)
        img[m] = val
        boxes.append(b); labels.append(c); contrast.append((val - bg_c) / 0.08)
    img = img + rng.normal(0, 0.08, img.shape)
    for b in boxes:
        x1, y1, x2, y2 = [int(round(v)) for v in b]
        inside.append(img[y1:y2, x1:x2].mean())
    return img.astype(np.float32), np.array(boxes), np.array(labels), contrast, inside


def iou(a, b):
    ix = max(0.0, min(a[2], b[2]) - max(a[0], b[0])); iy = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = ix * iy
    return inter / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter)


def iou_matrix(A, B):
    ix = np.clip(np.minimum(A[:, None, 2], B[None, :, 2]) - np.maximum(A[:, None, 0], B[None, :, 0]), 0, None)
    iy = np.clip(np.minimum(A[:, None, 3], B[None, :, 3]) - np.maximum(A[:, None, 1], B[None, :, 1]), 0, None)
    inter = ix * iy
    area = lambda X: (X[:, 2] - X[:, 0]) * (X[:, 3] - X[:, 1])
    return inter / (area(A)[:, None] + area(B)[None, :] - inter)


def proposals(rng, gt):
    """Stand-in for selective search: 6 jittered copies of each object plus random boxes."""
    out = []
    for x1, y1, x2, y2 in gt:
        w, h = x2 - x1, y2 - y1
        for _ in range(6):
            cx = (x1 + x2) / 2 + rng.normal(0, 0.15 * w); cy = (y1 + y2) / 2 + rng.normal(0, 0.15 * h)
            ww, hh = w * math.exp(rng.normal(0, 0.2)), h * math.exp(rng.normal(0, 0.2))
            out.append([cx - ww / 2, cy - hh / 2, cx + ww / 2, cy + hh / 2])
    while len(out) < N_PROP:
        ww, hh = rng.uniform(6, 36, 2); cx, cy = rng.uniform(0, SIZE, 2)
        out.append([cx - ww / 2, cy - hh / 2, cx + ww / 2, cy + hh / 2])
    P = np.clip(np.array(out[:N_PROP]), 0, SIZE)
    P[:, 2] = np.maximum(P[:, 2], P[:, 0] + 4); P[:, 3] = np.maximum(P[:, 3], P[:, 1] + 4)
    return np.clip(P, 0, SIZE)


def make_set(n, seed):
    rng = np.random.default_rng(seed)
    data = []
    for _ in range(n):
        img, gt, lab, con, ins = render(rng)
        data.append(dict(img=img, gt=gt, lab=lab, con=con, ins=ins, props=proposals(rng, gt)))
    return data


# ---------- box parameterization (R-CNN), used by Fast R-CNN ----------
def encode(P, G):
    pw, ph = P[:, 2] - P[:, 0], P[:, 3] - P[:, 1]; gw, gh = G[:, 2] - G[:, 0], G[:, 3] - G[:, 1]
    return np.stack([((G[:, 0] + G[:, 2]) - (P[:, 0] + P[:, 2])) / 2 / pw,
                     ((G[:, 1] + G[:, 3]) - (P[:, 1] + P[:, 3])) / 2 / ph,
                     np.log(gw / pw), np.log(gh / ph)], 1)


def decode(P, t):
    pw, ph = P[:, 2] - P[:, 0], P[:, 3] - P[:, 1]
    cx, cy = (P[:, 0] + P[:, 2]) / 2 + t[:, 0] * pw, (P[:, 1] + P[:, 3]) / 2 + t[:, 1] * ph
    w, h = pw * np.exp(t[:, 2]), ph * np.exp(t[:, 3])
    return np.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], 1)


# ---------- the two detectors: same backbone, same head ----------
class Detector(nn.Module):
    def __init__(self, route):
        super().__init__()
        self.route = route
        self.backbone = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1), nn.ReLU())           # stride 4
        self.fc6 = nn.Linear(64 * POOL * POOL, 256); self.fc7 = nn.Linear(256, 256)
        self.cls = nn.Linear(256, K + 1); self.bbox = nn.Linear(256, 4 * K)
        nn.init.normal_(self.cls.weight, 0, 0.01); nn.init.normal_(self.bbox.weight, 0, 0.001)
        nn.init.zeros_(self.cls.bias); nn.init.zeros_(self.bbox.bias)

    def roi_features(self, imgs, rois, bidx):
        if self.route == "shared":   # one backbone pass per image, RoI max pooling (Caffe rounding)
            fmap = self.backbone(imgs)
            feats = []
            for (x1, y1, x2, y2), b in zip(rois.tolist(), bidx.tolist()):
                c1, r1 = math.floor(x1 / STRIDE + 0.5), math.floor(y1 / STRIDE + 0.5)
                c2, r2 = math.floor((x2 - 1) / STRIDE + 0.5), math.floor((y2 - 1) / STRIDE + 0.5)
                c1, r1 = min(c1, SIZE // STRIDE - 1), min(r1, SIZE // STRIDE - 1)
                c2, r2 = max(c2, c1), max(r2, r1)
                feats.append(F.adaptive_max_pool2d(fmap[b, :, r1:r2 + 1, c1:c2 + 1], POOL))
            return torch.stack(feats)
        # crop route: dilate for R-CNN's context (16 px at 227), warp each box to CROP x CROP
        r = torch.as_tensor(rois, dtype=torch.float32)
        cx, cy = (r[:, 0] + r[:, 2]) / 2, (r[:, 1] + r[:, 3]) / 2
        s = 227 / 195
        w, h = (r[:, 2] - r[:, 0]) * s, (r[:, 3] - r[:, 1]) * s
        theta = torch.zeros(len(r), 2, 3)
        theta[:, 0, 0], theta[:, 0, 2] = w / SIZE, 2 * cx / SIZE - 1
        theta[:, 1, 1], theta[:, 1, 2] = h / SIZE, 2 * cy / SIZE - 1
        grid = F.affine_grid(theta, (len(r), 1, CROP, CROP), align_corners=False)
        crops = F.grid_sample(imgs[torch.as_tensor(bidx)], grid, align_corners=False)
        return self.backbone(crops)

    def forward(self, imgs, rois, bidx, fc6=None):
        x = self.roi_features(imgs, rois, bidx).flatten(1)
        x = F.relu(self.fc6(x) if fc6 is None else fc6(x))
        x = F.relu(self.fc7(x))
        return self.cls(x), self.bbox(x)


def smooth_l1(x):
    a = x.abs()
    return torch.where(a < 1, 0.5 * x ** 2, a - 0.5)


def label_rois(P, gt, lab):
    ov = iou_matrix(P, gt); arg = ov.argmax(1); mx = ov.max(1)
    return mx, lab[arg] + 1, gt[arg]


def train(route, train_set, seed, mean, std, iters=2000):
    torch.manual_seed(seed); rng = np.random.default_rng(seed)
    net = Detector(route)
    opt = torch.optim.SGD(net.parameters(), lr=0.01, momentum=0.9, weight_decay=5e-4)
    order = []
    for it in range(iters):
        if it == 1500:
            for gpar in opt.param_groups: gpar["lr"] = 0.001
        if len(order) < 2: order += list(rng.permutation(len(train_set)))
        imgs, rois, bidx, labels, targets = [], [], [], [], []
        for b in range(2):                               # N = 2 images
            d = train_set[order.pop(0)]
            img, gt, P = d["img"], d["gt"].copy(), np.vstack([d["props"], d["gt"]])  # GT boxes join the RoIs
            if rng.random() < 0.5:                       # horizontal flip
                img = img[:, ::-1].copy(); gt[:, [0, 2]] = SIZE - gt[:, [2, 0]]; P[:, [0, 2]] = SIZE - P[:, [2, 0]]
            mx, u, g = label_rois(P, gt, d["lab"])
            fg = np.where(mx >= 0.5)[0]; bg = np.where((mx >= 0.1) & (mx < 0.5))[0]
            fg = rng.choice(fg, min(8, len(fg)), replace=False)             # 25% of 32 RoIs per image
            bg = rng.choice(bg, min(32 - len(fg), len(bg)), replace=False)
            keep = np.concatenate([fg, bg]).astype(int)
            lab_k = u[keep].copy(); lab_k[len(fg):] = 0
            imgs.append(img); rois.append(P[keep]); bidx += [b] * len(keep); labels.append(lab_k)
            targets.append((encode(P[keep], g[keep]) - mean) / std)
        imgs = torch.as_tensor(np.stack(imgs))[:, None]
        rois = np.vstack(rois); labels = torch.as_tensor(np.concatenate(labels)).long()
        targets = torch.as_tensor(np.vstack(targets), dtype=torch.float32)
        scores, deltas = net(imgs, rois, np.array(bidx))
        l_cls = F.cross_entropy(scores, labels)
        fgm = labels > 0
        d_u = deltas.view(-1, K, 4)[fgm, labels[fgm] - 1]               # offsets of the true class
        l_loc = smooth_l1(d_u - targets[fgm]).sum() / len(labels)
        loss = l_cls + 1.0 * l_loc                                       # lambda = 1
        opt.zero_grad(); loss.backward(); opt.step()
    return net


def nms(B, s, thr=0.3):
    order, keep = np.argsort(-s), []
    while len(order):
        i = order[0]; keep.append(i)
        order = order[1:][iou_matrix(B[[i]], B[order[1:]])[0] < thr]
    return keep


@torch.no_grad()
def detect(net, test_set, mean, std, regress=True, fc6=None):
    dets = [[] for _ in range(K)]
    for i, d in enumerate(test_set):
        P = d["props"]
        scores, deltas = net(torch.as_tensor(d["img"])[None, None], P, np.zeros(len(P), int), fc6)
        prob = F.softmax(scores, 1).numpy(); deltas = deltas.numpy().reshape(-1, K, 4)
        for k in range(K):
            B = decode(P, deltas[:, k] * std + mean) if regress else P
            for j in nms(B, prob[:, k + 1]):
                dets[k].append((prob[j, k + 1], i, B[j]))
    return dets


def mean_ap(dets, test_set, thr):
    aps = []
    for k in range(K):
        gts = {i: d["gt"][d["lab"] == k] for i, d in enumerate(test_set)}
        used = {i: np.zeros(len(g), bool) for i, g in gts.items()}
        npos = sum(len(g) for g in gts.values())
        tp = []
        for s, i, b in sorted(dets[k], key=lambda x: -x[0]):
            g = gts[i]
            if len(g) == 0: tp.append(0); continue
            ov = iou_matrix(b[None], g)[0]; j = ov.argmax()
            if ov[j] >= thr and not used[i][j]: used[i][j] = True; tp.append(1)
            else: tp.append(0)
        tp = np.array(tp); ctp = np.cumsum(tp); rec = ctp / npos; prec = ctp / np.arange(1, len(tp) + 1)
        mrec = np.concatenate([[0], rec, [1]]); mpre = np.concatenate([[0], prec, [0]])
        for j in range(len(mpre) - 2, -1, -1): mpre[j] = max(mpre[j], mpre[j + 1])
        idx = np.where(mrec[1:] != mrec[:-1])[0]
        aps.append(np.sum((mrec[idx + 1] - mrec[idx]) * mpre[idx + 1]))
    return 100 * np.mean(aps)


if __name__ == "__main__":
    train_set, test_set = make_set(1500, 1000), make_set(300, 2000)

    # --- check before training: visible, not separable by brightness, proposals need refining ---
    con = np.concatenate([d["con"] for d in train_set])
    print(f"objects: train {sum(len(d['gt']) for d in train_set)}, test {sum(len(d['gt']) for d in test_set)}")
    print(f"contrast / noise sigma: min {con.min():.1f}, median {np.median(con):.1f}")
    for k in range(K):
        ins = np.concatenate([np.array(d["ins"])[d["lab"] == k] for d in train_set])
        print(f"  mean intensity in box, {NAMES[k]:8s}: 5-95% range {np.percentile(ins, 5):.2f}-{np.percentile(ins, 95):.2f}")
    mxs = np.concatenate([label_rois(d["props"], d["gt"], d["lab"])[0] for d in train_set])
    fgp = mxs[mxs >= 0.5]
    print(f"proposals per image {N_PROP}: foreground {np.mean(mxs >= 0.5) * 100:.0f}%, "
          f"background [0.1,0.5) {np.mean((mxs >= 0.1) & (mxs < 0.5)) * 100:.0f}%, ignored {np.mean(mxs < 0.1) * 100:.0f}%")
    print(f"foreground proposals: median IoU {np.median(fgp):.2f}, with IoU >= 0.7: {np.mean(fgp >= 0.7) * 100:.0f}%")
    print(f"chance accuracy over {K} classes + background: {100 / (K + 1):.0f}%")

    # regression-target normalization (zero mean, unit variance) from the training proposals
    T = []
    for d in train_set:
        mx, u, g = label_rois(d["props"], d["gt"], d["lab"])
        T.append(encode(d["props"][mx >= 0.5], g[mx >= 0.5]))
    T = np.vstack(T); mean, std = T.mean(0), T.std(0)

    # --- convolutional work per test image ---
    def conv_macs(side):
        return side * side * 16 * 9 + (side // 2) ** 2 * 32 * 16 * 9 + (side // 4) ** 2 * 64 * 32 * 9
    print("\nbackbone work per image   input pixels   conv MACs")
    print(f"  shared (any # of RoIs)   {SIZE * SIZE:12,d} {conv_macs(SIZE):11,d}")
    for n in (N_PROP, 2000):
        print(f"  crop, {n:5d} proposals    {n * CROP * CROP:12,d} {n * conv_macs(CROP):11,d}")

    # --- train both routes over three seeds ---
    print("\nVOC-style mAP (%) on 300 test images")
    print("route   seed  @0.5   @0.7   @0.5 no reg  @0.7 no reg")
    res, nets = {}, {}
    for route in ("shared", "crop"):
        for seed in (0, 1, 2):
            net = train(route, train_set, seed, mean, std)
            nets[(route, seed)] = net
            r = [mean_ap(detect(net, test_set, mean, std, reg), test_set, t)
                 for reg in (True, False) for t in (0.5, 0.7)]
            res[(route, seed)] = r
            print(f"{route:6s}  {seed:4d}  {r[0]:5.1f}  {r[1]:5.1f}  {r[2]:10.1f}  {r[3]:11.1f}")
    for route in ("shared", "crop"):
        a = np.array([res[(route, s)] for s in (0, 1, 2)])
        print(f"{route:6s} range over seeds: @0.5 {a[:, 0].min():.1f}-{a[:, 0].max():.1f}, "
              f"@0.7 {a[:, 1].min():.1f}-{a[:, 1].max():.1f}")

    # --- truncated SVD of fc6 in the shared model (seed 0), no re-training ---
    net = nets[("shared", 0)]
    W, bias = net.fc6.weight.data, net.fc6.bias.data          # 256 x 1024
    U, S, Vh = torch.linalg.svd(W, full_matrices=False)
    print("\ntruncated SVD of fc6 (1024 -> 256), shared model, seed 0")
    print("   t   fc6 params   mAP@0.5")
    for t in (256, 64, 32, 16, 8):
        first = nn.Linear(1024, t, bias=False); second = nn.Linear(t, 256)
        first.weight.data = torch.diag(S[:t]) @ Vh[:t]          # Sigma_t V^T, no bias
        second.weight.data = U[:, :t].clone(); second.bias.data = bias.clone()   # U, original bias
        fc6 = nn.Sequential(first, second)
        m = mean_ap(detect(net, test_set, mean, std, True, fc6), test_set, 0.5)
        print(f"{t:4d}  {t * (1024 + 256):11,d}  {m:8.1f}")
    print(f" full {1024 * 256:11,d}  {res[('shared', 0)][0]:8.1f}")

objects: train 2964, test 593
contrast / noise sigma: min 3.1, median 5.3
  mean intensity in box, square  : 5-95% range 0.38-0.77
  mean intensity in box, disc    : 5-95% range 0.32-0.66
  mean intensity in box, triangle: 5-95% range 0.23-0.50
proposals per image 40: foreground 24%, background [0.1,0.5) 27%, ignored 49%
foreground proposals: median IoU 0.63, with IoU >= 0.7: 26%
chance accuracy over 3 classes + background: 25%

backbone work per image   input pixels   conv MACs
  shared (any # of RoIs)          4,096  10,027,008
  crop,    40 proposals          10,240  25,067,520
  crop,  2000 proposals         512,000 1,253,376,000

VOC-style mAP (%) on 300 test images
route   seed  @0.5   @0.7   @0.5 no reg  @0.7 no reg
shared     0   99.6   73.9        98.6         31.8
shared     1   99.4   75.0        98.1         33.9
shared     2   99.2   70.1        96.1         31.4
crop       0   99.3   95.1        98.4         49.8
crop       1   99.2   94.6        98.3         49.5
crop       2   99.2   93.7        98.5         51.1
shared range over seeds: @0.5 99.2-99.6, @0.7 70.1-75.0
crop   range over seeds: @0.5 99.2-99.3, @0.7 93.7-95.1

truncated SVD of fc6 (1024 -> 256), shared model, seed 0
   t   fc6 params   mAP@0.5
 256      327,680      99.6
  64       81,920      99.6
  32       40,960      99.6
  16       20,480      99.6
   8       10,240      99.1
 full     262,144      99.6

The check before training behaves as intended: objects stand at least 3.1 noise standard deviations above their background, the brightness ranges of the three classes overlap, and only about a quarter of the foreground proposals already reach IoU 0.7. At IoU 0.5 both detectors score about 99% for every seed, so this simple shape vocabulary is easy to classify and the 0.5 threshold, PASCAL’s, cannot tell the routes apart. Sharing costs nothing there, while the convolutional work no longer grows with the number of proposals: the crop route already does 2.5 times as much at 40 proposals, and would do 125 times as much at Fast R-CNN’s usual 2,000.

At IoU 0.7 the routes separate: 93.7–95.1% for crop against 70.1–75.0% for shared over the three seeds. Box regression adds 39 to 45 points at 0.7 to both, so it is doing its job in both, but the shared route regresses and scores from coarser evidence. Our reading is that this is RoI pooling’s quantization. On a stride-4 map, rounding moves each box edge by up to 2 pixels before pooling, a large fraction of an 8-to-28-pixel object, and two proposals that differ by less than a cell can receive identical features. The columns without regression point the same way: there the output boxes are the proposals themselves, and the crop route, which samples every proposal at its exact coordinates, ranks the well-fitting ones higher (about 50% at 0.7 against about 32%). The experiment does not separate quantization from the crop route’s other differences, which are resampling every proposal to the same size, enlarging small objects, and its context margin. The imprecision it shows is the one that RoIAlign later removed in Mask R-CNN (He et al., 2017), as the RoI pooling article measures directly. In the paper’s VGG16 setting the effective stride is about 10 pixels on objects that are typically far larger than these shapes, and all its VOC results are at IoU 0.5.

Truncated SVD behaves as the paper reports at much larger scale: keeping 16 of fc6’s 256 singular values, under a tenth of the weights, leaves mAP unchanged without re-training, and 8 values cost half a point. The factorization saves parameters only when $t < uv/(u + v)$ , about 205 here, so at $t = 256$ it is exact but larger than the original layer.

Performance

All results below are from Girshick (2015), with VGG16 unless stated, as mAP at IoU 0.5 on PASCAL VOC test sets (see PASCAL VOC). Training-set keys follow the paper: 07 is VOC 2007 trainval, 07∖diff the same without examples marked “difficult”, 07+12 the union with VOC 2012 trainval, and 07++12 the union of VOC 2007 trainval and test with VOC 2012 trainval.

Test set	Fast R-CNN	Training data	Comparison (same table)
VOC 2007 test (Table 1)	66.9%	07	R-CNN with box regression 66.0% (07); SPP-net with box regression 63.1% (07∖diff, five scales)
VOC 2007 test (Table 1)	68.1%	07∖diff	SPP-net 63.1% (07∖diff)
VOC 2007 test (Table 1)	70.0%	07+12	—
VOC 2010 test (Table 2)	66.1%	12	R-CNN 62.9% (12); SegDeepM 67.2% (12 with segmentation labels)
VOC 2010 test (Table 2)	68.8%	07++12	—
VOC 2012 test (Table 3)	65.7%	12	R-CNN 62.4% (12); NUS NIN c2000 63.8%; BabyLearning 63.2%
VOC 2012 test (Table 3)	68.4%	07++12	—

The VGG16 R-CNN numbers are those of Girshick, Donahue, Darrell and Malik’s journal paper “Region-Based Convolutional Networks for Accurate Object Detection and Segmentation”, not the CVPR 2014 paper, whose network was AlexNet-sized; the SPP-net VGG16 numbers were computed for the paper by SPP-net’s authors. On VOC 2012 the paper reports the top result on the comp4 (outside data) leaderboard track as of April 2015. On VOC 2010, SegDeepM was higher with Fast R-CNN’s smaller training set; its R-CNN backbone could be swapped for Fast R-CNN, the paper notes, and with 07++12 Fast R-CNN surpassed it. The BabyLearning and NUS NIN c2000 entries used networks derived from Network in Network, with no publication at the time.

A preliminary run on COCO trained on the 80,000-image training set for 240,000 iterations gave 35.9% PASCAL-style mAP and 19.7% COCO-style AP, averaged over IoU thresholds, on test-dev (see COCO).

Speed

Table 4 compares the same networks in the three systems on one K40 GPU, with Fast R-CNN at a single scale and SPP-net at five:

	Fast R-CNN S	M	L	R-CNN S	M	L	SPP-net L
Training time (h)	1.2	2.0	9.5	22	28	84	25
Test time (s/image)	0.10	0.15	0.32	9.8	12.1	47.0	2.3
With truncated SVD	0.06	0.08	0.22	—	—	—	—
VOC 2007 mAP (%)	57.1	59.2	66.9	58.5	60.2	66.0	63.1
With truncated SVD	56.5	58.7	66.6	—	—	—	—

For VGG16 this is 9 times faster training than R-CNN (84 against 9.5 hours) and 146 times faster testing, 213 times with truncated SVD; against SPP-net, 7 or 10 times faster testing. The paper disagrees with itself slightly on SPP-net’s training time: the table gives 25 hours, the text 25.5 hours and a 2.7-fold speed-up, and the abstract and introduction give the speed-up as 3 times. The SPP-net timings came from its authors. The paper states that its own detection times exclude computing the proposals.

Ablations

Fast R-CNN made training cheap enough to answer design questions on VOC 2007 directly:

Which layers to fine-tune (Table 5, VGG16). Freezing all thirteen convolutional layers, which emulates SPP-net’s training at a single scale, gave 61.4%; training from conv3_1 up gave 66.9% and from conv2_1 up 67.2%. Five-scale SPP-net reached 63.1% at seven times the test cost. The paper takes this as confirming that training through the RoI pooling layer matters for very deep networks.
Multi-task training (Table 6). Training with the classification loss only gave 52.2%, 54.7% and 62.6% for S, M and L. Training with the multi-task loss but testing without box regression gave 53.3%, 55.5% and 63.4%, so the box loss improved classification alone by 0.8 to 1.1 points. Adding a regressor afterwards to the classification-only network, with everything else frozen, gave 54.6%, 56.6% and 64.0%, against 57.1%, 59.2% and 66.9% for full multi-task training.
Scales (Table 7). Five scales raised S from 57.1% to 58.4% and M from 59.2% to 60.7%, at roughly four times the test time; VGG16 at one scale, 66.9%, was slightly above R-CNN’s 66.0%, which warps every proposal to a canonical size. Single-scale processing was used everywhere else.
Softmax or SVMs (Table 8). Post-hoc SVMs trained with hard negative mining, with R-CNN’s procedure, scored 56.3%, 58.7% and 66.8% against the softmax’s 57.1%, 59.2% and 66.9%. The paper notes that the softmax, unlike one-vs-rest SVMs, makes the classes compete for each RoI.
More training data (Section 5.3). VOC 2007 and 2012 trainval together, about three times VOC 2007’s images, raised VOC 2007 test mAP from 66.9% to 70.0%.

Proposals: sparse or dense

The paper’s last ablation (Section 5.5, Figure 3) asks whether more candidates help, retraining and retesting model M for every point. Its numbers come from a plot and from the text, and they are not all on the same baseline as the tables:

Selective search, quality mode, 1,000 to 10,000 proposals per image. mAP rose and then fell slightly while the proposals’ average recall kept rising, so the paper warns that higher recall from more proposals does not imply higher mAP. Read from the plot, the curve stays between about 58% and 59%.
Selective-search boxes replaced by their nearest dense boxes. Of a dense set of about 45,000 boxes per image over position, scale and aspect ratio, using for each selective-search box the dense box with the highest IoU dropped mAP by “only 1 point” to 57.7%. The text thus implies a baseline of about 58.7%. The plot places this point at 2,000 proposals and the selective-search curve there at about 58.7% (our reading of the figure: its mAP axis labels appear to be evenly spaced ticks rounded down, so the gridline labelled 58 lies at about 58.7); Table 4’s 59.2% for model M comes from the paper’s main runs, and the paper does not reconcile the two.
Random dense boxes added to 2,000 selective-search boxes. mAP fell more than when adding selective-search boxes, reaching 53.0%.
Dense boxes only, about 45,000 per image. 52.9% with the softmax and 49.3% with SVMs trained with hard negative mining.

Read like for like, within Figure 3, dense boxes alone cost about 6 points against 2,000 selective-search proposals (52.9% against about 58.7%). The paper’s interpretation is that sparse proposals act as a cascade that rejects most candidates before the classifier sees them, and that this improves accuracy, not only speed; it adds that undiscovered techniques might one day let dense boxes do as well. The region proposals and selective search articles discuss the proposal methods themselves.

Strengths

One network, one training stage. The softmax and the regressors are trained with the features, in one SGD run, without SVMs, separate regressors or a feature cache on disk.
The whole network learns. Sampling RoIs from few images makes back-propagation through RoI pooling affordable; the paper attributes its gain over SPP-net (66.9% against 63.1%) to fine-tuning the convolutional layers, and freezing them cost 5.5 points in its ablation.
Speed. The convolutional cost is paid once per image; with truncated SVD the per-RoI layers became cheap enough that the convolutions dominated again.
A reusable head. RoI pooling, sibling outputs and the multi-task loss with smooth $L_1$ form a detection head that later detectors took over largely unchanged.
Cheap experiments. Model M trained and tested in under 2.5 hours, which let the paper test design choices, such as the number of proposals, by training rather than by proxy metrics.

Limitations

Proposals stay outside. Fast R-CNN classifies precomputed proposals, and its 0.32 s per image excludes them. Ren et al. (2015) put selective search at about 2 s per image on a CPU (1.51 s on average in their timing), against Fast R-CNN’s 320 ms with VGG16, so the proposal step had become the bottleneck.
Quantized pooling. RoI pooling rounds boxes to feature-map cells and bins, so features are no finer than the map’s stride and shift when a box moves less than a cell; the example above shows the cost at strict IoU thresholds.
Heuristic sampling. The IoU thresholds, the 25% foreground fraction and the $[0.1, 0.5)$ background range are fixed rules, and the background floor ignores easy negatives by design rather than by their loss.
Single-scale features. VGG16 could be trained only at one scale for memory reasons, and the paper handles scale by resizing the image, not with features at several resolutions.
Per-RoI fully connected layers. Their cost grows with the number of RoIs, which is why truncated SVD was needed.
Memory. Fine-tuning VGG16 from conv1_1 did not fit on the GPU, and the image size was chosen to fit.

Historical Importance

Fast R-CNN completed the move from R-CNN’s pipeline to a single trainable detection network. SPP-net had shared the convolutional computation; Fast R-CNN made the shared features trainable and replaced R-CNN’s three training stages with one loss. Faster R-CNN (Ren et al., 2015), by Ren, He, Girshick and Sun, then generated the proposals with a region proposal network on the same feature map and kept Fast R-CNN as its detector, so the external proposals, the last separate stage, disappeared. In our reading, the paper’s lasting contributions are the detection head, RoI pooling followed by fully connected layers with a softmax and class-specific box regressors, the multi-task loss with smooth $L_1$ , and image-centric sampling of training RoIs, all of which appear, sometimes renamed, in Faster R-CNN and Mask R-CNN. Its proposal experiments also showed how a fast detector could evaluate proposals by training on them rather than by recall.

The PAMI Technical Committee lists “Fast R-CNN” among the 2025 winners of the Helmholtz Prize, which recognizes ICCV papers from ten years earlier with significant impact on computer vision research.

Variants

Faster R-CNN (Ren et al., 2015) adds a region proposal network that shares the backbone with Fast R-CNN; the Faster R-CNN article covers it, including how its training alternates between the two networks.
Online hard example mining (Shrivastava, Gupta and Girshick, 2016) keeps Fast R-CNN but selects each image’s training RoIs by their current loss, removing the background floor of 0.1 and the 1:3 foreground-to-background ratio; its authors report that removing or raising that ratio in plain Fast R-CNN lowers mAP by about 3 points. See hard negative mining.
RoIAlign in Mask R-CNN (He et al., 2017) removes the rounding of RoI pooling, and other region feature extractors followed; see RoI pooling.
The two-stage object detectors article follows the design line after Faster R-CNN.

Implementations

The official implementation, rbgirshick/fast-rcnn, is written in Python and C++ on a fork of Caffe with an RoI pooling layer, and is released under the MIT License. As of November 2026 its README marks it as deprecated, describes it as a historical artifact kept to supplement the ICCV paper, and points to Detectron and to py-faster-rcnn, which can also train Fast R-CNN; its last commit dates from January 2018. Training is Python only, and test-time detection also exists in MATLAB. Its defaults match the paper: 2 images and 128 RoIs per mini-batch, 25% foreground, background IoU in $[0.1, 0.5)$ , flipped images, a shorter side of 600 pixels capped at 1,000, test-time NMS at 0.3, and a script that compresses a trained network by truncated SVD. The demo uses precomputed selective-search proposals. The RoI pooling operator itself is available in modern libraries, as the RoI pooling article describes.