Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

“Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation” by Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik of UC Berkeley (Donahue and Darrell also at ICSI) was presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2014, after a first tech report on arXiv in November 2013. It introduced R-CNN, “Regions with CNN features”: a convolutional neural network pre-trained on ImageNet classification, fine-tuned on detection data, and applied to about 2,000 bottom-up region proposals per image. On PASCAL VOC it raised detection mean average precision by more than 30% relative to the previous best result. The arXiv tech report kept growing after the conference: version 3 added ILSVRC2013 results and version 5 (October 2014) a 16-layer network, and its appendices cover material the CVPR paper left to supplementary material. Details taken from the tech report are marked below.

Problem

The authors open from the observation that progress on VOC object detection had been slow from 2010 to 2012. The best systems were ensembles built on SIFT- and HOG-like features plus context. On VOC 2010 test, the HOG-based deformable part model (DPM v5) scored 33.4% mAP, and SegDPM, then at the top of the VOC leaderboard, 40.4%.

Krizhevsky et al.’s 2012 ImageNet result had revived CNNs for classification. The paper frames the open question from the ILSVRC 2012 workshop debate as whether that result carried over to detection on VOC. Two obstacles stood in the way. One was localization: the network’s top units have $195 \times 195$ pixel receptive fields and a stride of 32 pixels, which the authors considered too coarse for precise sliding-window detection. The other was data: VOC had too few labeled boxes to train a large CNN.

Contribution

The abstract names two insights:

1. CNNs on bottom-up region proposals. Instead of regressing boxes or sliding a window, classify a few thousand category-independent candidate regions, in what the authors call the “recognition using regions” paradigm.
2. Supervised pre-training, then domain-specific fine-tuning. When labeled data is scarce, pre-train on a large auxiliary task (ILSVRC 2012 classification, image labels only), then continue training on the small target set. The paper describes unsupervised pre-training as the conventional solution at the time. Fine-tuning added 8 mAP points on VOC 2007.

Method

R-CNN has three modules: proposals, a feature extractor and class-specific linear SVMs. At test time, selective search (Uijlings et al., 2013) in its “fast mode” gives about 2,000 proposals. Each proposal’s box is dilated so that, after warping, 16 pixels of context surround it. All pixels in the dilated box are warped anisotropically to $227 \times 227$, ignoring aspect ratio, and passed through a Caffe reimplementation of Krizhevsky et al.’s network to give a 4,096-dimensional feature. Each class’s SVM scores every feature, and greedy non-maximum suppression runs per class. Appendix A of the tech report compares warping with isotropic “tightest square” crops, with and without context. Warping with $p = 16$ won by 3 to 5 mAP points in pilot experiments, and pixels outside the image were filled with the image mean.

The NumPy sketch below is our own implementation of that warp, with nearest-neighbour sampling, on a synthetic image whose object touches the left border:

import numpy as np

S, P = 227, 16                     # CNN input size and context padding, as in the paper

def warp_with_context(img, box, size=S, pad=P):
    """Anisotropically warp box (x1, y1, x2, y2) to size x size, with pad pixels of context
    around it in the warped frame. Pixels outside the image get the image mean."""
    x1, y1, x2, y2 = box
    sx = (x2 - x1) / (size - 2 * pad)          # source pixels per output pixel, x
    sy = (y2 - y1) / (size - 2 * pad)          # and y: the two scales differ (anisotropic)
    u = np.arange(size) + 0.5                  # output pixel centers
    xs = x1 + (u - pad) * sx                   # source coordinates of output pixel centers
    ys = y1 + (u - pad) * sy
    xi, yi = np.floor(xs).astype(int), np.floor(ys).astype(int)   # nearest-neighbour sampling
    H, W = img.shape
    inside = (yi[:, None] >= 0) & (yi[:, None] < H) & (xi[None, :] >= 0) & (xi[None, :] < W)
    out = np.full((size, size), img.mean())
    out[inside] = img[np.clip(yi, 0, H - 1)[:, None], np.clip(xi, 0, W - 1)[None, :]][inside]
    src = (x1 - pad * sx, y1 - pad * sy, x2 + pad * sx, y2 + pad * sy)
    return out, src

# Synthetic 200 x 300 image: background 0, a 60 x 120 (h x w) "object" of value 1 touching the left edge.
img = np.zeros((200, 300))
img[50:110, 0:120] = 1.0
box = (0, 50, 120, 110)                         # tight box on the object: x1, y1, x2, y2

crop, src = warp_with_context(img, box)
print("source rectangle read:", tuple(round(v, 2) for v in src))
rows = np.where(crop.mean(axis=1) > 0.5)[0]
cols = np.where(crop.mean(axis=0) > 0.5)[0]
print(f"object in warped crop: rows {rows[0]}..{rows[-1]}, cols {cols[0]}..{cols[-1]} of 0..{S-1}")
pad_cols = np.where(np.all(crop == img.mean(), axis=0))[0]
print(f"columns filled with the image mean (left of the image): {pad_cols[0]}..{pad_cols[-1]}")

source rectangle read: (-9.85, 45.08, 129.85, 114.92)
object in warped crop: rows 16..210, cols 16..210 of 0..226
columns filled with the image mean (left of the image): 0..15

The box, 120 pixels wide and 60 tall, fills the central $195 \times 195$ pixels of the crop: it is stretched 1.625 times horizontally and 3.25 times vertically. The context border needs about 10 source pixels to the left of the image, and these become the mean.

Run-time analysis. The paper argues that the design scales with the number of classes. Proposals and features (13 s per image on a GPU, 53 s on a CPU) are shared by all classes. The per-class work is one $2000 \times 4096$ by $4096 \times N$ matrix product plus NMS. For 100,000 classes, the authors estimate this product at about 10 s on a multi-core CPU. With 360,000-dimensional features, the UvA selective-search system would need 134 GB for the same predictors, against 1.5 GB.

Training. The network was pre-trained on ILSVRC 2012 with Caffe; its top-1 validation error was 2.2 points higher than Krizhevsky et al.’s. It was then fine-tuned with SGD on warped VOC proposals at a learning rate of 0.001, one tenth of the pre-training rate, with a new 21-way output layer. The two-stage object detectors article describes the SVM and box-regressor stages, with their thresholds. Appendix B of the tech report explains why fine-tuning and the SVMs define positives differently: counting proposals with IoU of 0.5 or more as positives gives about 30 times more positive examples, which the authors conjecture fine-tuning needs to avoid overfitting. It also reports, in a figure corrected in version 4, that using the fine-tuned softmax directly instead of SVMs lowered VOC 2007 mAP from 54.2% to 50.9%.

Results

All numbers below are as reported by the authors, in mAP with an IoU threshold of 0.5.

VOC 2010–2012. Hyperparameters were validated on VOC 2007. For the final runs, the CNN was fine-tuned on VOC 2012 train and the SVMs trained on VOC 2012 trainval, with one server submission for each variant. On VOC 2010 test (Table 1), R-CNN scored 50.2% without box regression and 53.7% with it. UvA, with the same proposals but spatial-pyramid bag-of-words features over SIFT variants, scored 35.1%, and Regionlets 39.7%. On VOC 2011/12 test it scored 53.3%.

VOC 2007 ablations (Table 2):

Features	Without fine-tuning	Fine-tuned on VOC 2007 trainval
pool5	44.2	47.3
fc6	46.2	53.1
fc7	44.7	54.2
fc7 + box regression	—	58.5

For reference, DPM v5 scored 33.7%, and two DPM variants with learned features scored 29.1% and 34.3%. Without fine-tuning, fc7 was worse than fc6, so 29% of the parameters could be dropped without loss, and pool5, with 6% of the parameters, still did well. Fine-tuning helped fc6 and fc7 much more than pool5, which the authors read as evidence that the pool5 features learned on ImageNet are general and that most of the gain comes from domain-specific non-linear classifiers on top of them.

Visualization and errors. The authors ranked about 10 million held-out proposals by one pool5 unit’s activation, applied NMS and displayed the top regions. Some units responded to people, dog faces or text; others to dot arrays, red blobs or specular reflections. Hoiem et al.’s analysis tool showed that, compared with DPM, more of R-CNN’s false positives were poor localizations rather than confusions with background or other classes. The authors blame the bottom-up proposals and the positional invariance learned from whole-image classification. Fine-tuning improved both the best- and worst-performing subsets for nearly all object characteristics analyzed, including occlusion and truncation, so it did more than fix small objects or unusual aspect ratios. Bounding-box regression, motivated by this analysis, added 3 to 4 mAP points.

ILSVRC2013 detection (tech report, from v3). With the VOC hyperparameters unchanged, R-CNN reached 31.4% mAP on the 200-class test set. The previous best was OverFeat’s post-competition 24.3%. Selective search covered only 91.6% of ground-truth boxes on val at IoU 0.5, against about 98% on VOC.

Deeper network (tech report, v5). Replacing Krizhevsky et al.’s network (“T-Net”) with Simonyan and Zisserman’s 16-layer network (“O-Net”) raised VOC 2007 test mAP from 58.5% to 66.0%, with a forward pass about 7 times slower.

Semantic segmentation. The authors classified CPMC regions inside the open-source O2P framework, without fine-tuning the CNN. Concatenating fc6 features of the full region box and of the masked foreground (“full+fg”) gave 47.9% mean accuracy on VOC 2011 validation, against 46.4% for O2P (Carreira et al., 2012). On VOC 2011 test it scored 47.9%, against 47.6% for O2P, which the authors call a likely tie.

Impact

The authors present the paper as the first to show that a CNN gives much higher detection accuracy on PASCAL VOC than systems built on HOG-like features. They end by conjecturing that the “supervised pre-training/domain-specific fine-tuning” recipe would work for other vision problems with little data. The pipeline started the region-based line described below. The PAMI Technical Committee lists the paper as the 2024 recipient of the Longuet-Higgins Prize, which is awarded at CVPR to CVPR papers from ten years earlier that have had significant impact on computer vision research. A journal version, “Region-Based Convolutional Networks for Accurate Object Detection and Segmentation”, appeared in IEEE TPAMI in 2016.

Limitations

• Speed. Every proposal went through the whole CNN (13 s per image). Fast R-CNN reported 47 s per image on a GPU with VGG16. The tech report notes that OverFeat, which shares convolutional computation between windows, was about 9 times faster.
• Multi-stage training. Fine-tuning, SVMs and box regressors were trained separately. Girshick (2015) adds that, with VGG16, the cached features took 2.5 GPU-days to compute for VOC 2007 trainval and hundreds of gigabytes of disk.
• Proposal ceiling. Recall is bounded by selective search, which was noticeably weaker on ILSVRC than on VOC.
• Data overlap. Pre-training on ImageNet risks seeing test images. Appendix F of the tech report found near-duplicates of 38 of the 4,952 VOC 2007 test images (under 1%) in ILSVRC 2012 trainval, and of about 1.5% of VOC 2012 test images.

What Came After

SPPnet (He et al., 2014) ran the convolutional layers once per image and pooled each proposal’s features from the shared map. Fast R-CNN (Girshick, 2015) trained one network with a multi-task loss in place of the SVMs and regressors; on VOC 2012 test it reported 65.7% mAP against 62.4% for R-CNN, both with VGG16. Faster R-CNN (Ren et al., 2015) generated the proposals with a network sharing the same features (see the Faster R-CNN paper). The two-stage object detectors article traces this line, and the object detection article places it beside one-stage detectors and DETR.