SSD: Single Shot MultiBox Detector

“SSD: Single Shot MultiBox Detector” by Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu and Alexander C. Berg, of the University of North Carolina at Chapel Hill, Zoox, Google and the University of Michigan, was presented at the European Conference on Computer Vision (ECCV) in Amsterdam in 2016; its proceedings chapter appeared online on 17 September 2016. It introduced SSD, a one-stage detector that predicts class scores and box offsets for a fixed set of default boxes on several feature maps of one network, with no proposal stage and no resampling of pixels or features for each box. The acknowledgment says the work began as an internship project at Google and continued at UNC. The paper went through five arXiv versions between December 2015 and December 2016: the first, by five authors, was a different paper built on an Inception network; the ECCV paper used VGG-16; and the last three added a new augmentation and higher numbers. This article covers the publication and its versions; the SSD article describes the model, including default boxes, matching, the loss and hard negative mining with a worked example.

Problem

The ECCV introduction describes the detectors of the time as one pipeline: hypothesize boxes, resample pixels or features for each, and apply a strong classifier. From Selective Search to the leading results on PASCAL VOC, COCO and ILSVRC, all based on Faster R-CNN, the authors call this approach accurate but too slow for real time, giving 7 frames per second for Faster R-CNN, which they call the fastest high-accuracy detector. Attempts to speed up a stage of the pipeline had, in their account, bought speed only at a large cost in accuracy. YOLO (Redmon et al., 2016) had dropped proposals and reached 45 frames per second, but at 63.4% mAP on VOC 2007 test against Faster R-CNN’s 73.2%.

SSD’s direct ancestor was MultiBox, from the Google co-authors. Erhan et al. (2014) regressed a fixed number of class-agnostic boxes with a network, and Szegedy et al. (2014) made the priors convolutional and multi-scale; in both, a second network classified the proposals. The SSD article traces this line in more detail.

Contribution

The ECCV paper presents SSD as the first deep detector that does not resample pixels or features for box hypotheses and is as accurate as detectors that do. The authors do not claim to be the first to drop resampling, citing OverFeat and YOLO; they attribute the accuracy to three changes, each small on its own: small convolutional filters that predict categories and box offsets, separate filters for different aspect ratios, and predictions from several feature maps of the later layers, so that relatively low-resolution input suffices. They describe the rise from YOLO’s 63.4% to SSD’s 74.3% on real-time VOC detection as a larger relative improvement than that of residual networks.

The contribution list adds that SSD predicts category scores and offsets for a fixed set of default boxes, trains end to end, and is evaluated for speed and accuracy on PASCAL VOC, COCO and ILSVRC. In the related-work section the authors place SSD next to the region proposal network: both use fixed reference boxes (“anchor boxes” in Faster R-CNN; see anchor boxes), but SSD scores every category in each box instead of pooling features for a second classifier. With one box per position on the top map SSD would resemble OverFeat, and with a fully connected predictor over the whole top map and no explicit aspect ratios it would approximately reproduce YOLO.

The first arXiv version framed its claims differently, as several firsts: the first to combine convolutional priors with prediction of multiple categories, and the first to attach priors to feature maps of different resolutions. Its conclusion argued that carefully designed sliding-window methods could match methods built on sparse proposals, and that more windows need not mean worse results. These claims are not in the later versions.

Method

The published method is the one the SSD article describes: a VGG-16 base modified as in DeepLab-LargeFOV, extra convolutional layers, $3 \times 3$ predictors on six maps for SSD300 (seven for SSD512), default boxes with scales spaced between 0.2 and 0.9 and aspect ratios 1, 2, 3, 1/2 and 1/3, matching to the best-overlapping box and then any box with IoU above 0.5, a softmax and smooth $L_1$ loss with weight $\alpha = 1$, hard negatives kept at 3:1, and crop sampling with photometric distortion. Three points about how the method was published matter for reading it:

• Equations by version. The ECCV version gives the overall objective (its Eq. 1) and the scale rule (Eq. 2) but describes the localization and confidence terms only in words. The arXiv versions from v3 on write both terms out as Eqs. 2 and 3, renumber the scale rule as Eq. 4, and add that the loss is zero when nothing is matched.
• Settings per dataset. The tiling changes with the data. On VOC, conv4_3 gets a scale of 0.1; on COCO, whose objects are smaller, the smallest scale becomes 0.15 and conv4_3’s 0.07. Only v3 and later give SSD512’s own values (0.15 and 0.07 on VOC, 0.1 and 0.04 on COCO) in footnotes. Training schedules were also set per dataset, all with SGD, batch 32 and an initial learning rate of $10^{-3}$: 40k, 10k and 10k iterations at successive tenfold drops on VOC 2007; 60k and 20k on VOC 2012; 160k, 40k and 40k on COCO; and 320k, 80k and 40k on ILSVRC.
• Inference. From v2 on, detections are thresholded at a confidence of 0.01, suppressed per class at IoU 0.45 (non-maximum suppression) and capped at 200 per image. Version 2 describes a GPU implementation with sorting by the Thrust library and puts the step at about 2.2 ms per image for SSD300; the ECCV and later versions give about 1.7 ms, close to the 2.4 ms of all the added layers.

The training and test code was released in Caffe at the address the abstract gives.

Versions

The arXiv record lists five versions; the ECCV chapter sits between v2 and v3. The dates are from the arXiv submission history and Crossref; the descriptions come from our comparison of the texts.

Version	Date	Authors	Base network and models	Main changes
v1	8 Dec 2015	5	Inception variant; 299 and 443 pixels	First version; experiments mainly on ILSVRC detection
v2	30 Mar 2016	7	VGG-16; SSD300 and SSD500	New paper structure; VOC, COCO and ILSVRC results; speed table
ECCV	17 Sep 2016 (online)	7	VGG-16; SSD300 and SSD512	Revised tiling and augmentation; higher numbers
v3	8 Nov 2016	7	As ECCV, plus SSD300* and SSD512*	Expansion augmentation (Sec. 3.6, Table 6); loss equations; new COCO and speed tables
v4	30 Nov 2016	7	As v3	Fig. 6 on object size; “SSD500” and “SSD500*” in Table 6 corrected to SSD512 and SSD512*
v5	29 Dec 2016	7	As v3	Reference list only

Version 1 has five authors: Liu at UNC, Anguelov, Erhan and Szegedy at Google, and Reed at Michigan. It fine-tuned an earlier Inception variant with 76.4% top-1 accuracy on ILSVRC, using batch normalization and Adagrad, and called the default boxes “priors”. The predictors were $1 \times 1$ convolutions, which the authors describe as a simplification of the RPN’s $3 \times 3$ kernels; the localization loss was $L_2$ on corner coordinates, the confidence loss a multi-class logistic loss, and $\alpha$ was 0.06. The scale rule ran from 0.1 to 0.7. Its experiments were mainly on ILSVRC 2014 detection, against a two-stage MultiBox system built on the same network. A design study on the 20 PASCAL classes of ILSVRC (Table 4 of v1) is where the matching rule of later versions comes from: matching the priors, first by best overlap and then by a threshold (“per-prediction” matching), gave 39.0 mAP, against 31.7 for one-to-one bipartite matching as in MultiBox and 20.8 for threshold matching on the network’s own predictions (all three with 0/1 confidence targets). Version 1 gave no timing measurements, only the claim of being 100 to 1000 times faster than proposal-based methods, and reported that training took 500k steps on VOC.

Version 2 is a new paper in the structure that ECCV kept, with Fu and Berg added. It moved to VGG-16, named its models SSD300 and SSD500, and added COCO and a speed table. Several of its settings differ from ECCV’s: $s_{\max} = 0.95$, three default boxes on conv4_3, a pooling layer (pool11) as the last prediction map, 7,308 boxes for SSD300 and 20,097 for SSD500, no photometric distortion (the text notes that YOLO used it), and a VOC 2007 schedule ending with 20k iterations at $10^{-4}$ instead of 10k each at $10^{-4}$ and $10^{-5}$. Its ablation also tested dropping conv4_3, which cost 4.0 points.

The ECCV version renamed SSD500 to SSD512, adding a conv12_2 map, and replaced pool11 with conv11_2. It set $s_{\max}$ to 0.9, used four boxes on conv4_3, conv10_2 and conv11_2, added photometric distortions, and added training on VOC 2007 alone and COCO pre-training to the VOC tables. The paper does not say why the input size changed, and does not separate how much of the gain over v2 came from each change.

Versions 3 to 5 were posted after the conference. Version 3 added Section 3.6, which places each training image on a mean-filled canvas of “16× of the original image size” before cropping, a “zoom out” that creates small training objects, and doubles the training iterations. An abstract footnote reports 77.2% (300) and 79.8% (512) on VOC 2007 with it. The same version rewrote the COCO and speed tables (below) and the corresponding text, and changed other numbers, such as the 74.1 in the ECCV version’s VOC 2007 table, which became 74.3 with different per-class values. Version 4 added Fig. 6, which shows the gain on small objects, and a reference for the “à trous” algorithm; v5 dropped a reference.

Results

All VOC results are mAP at IoU 0.5 on the test sets (see average precision). “07+12” is VOC 2007 and 2012 trainval, “07++12” adds VOC 2007 test, and “+COCO” means pre-training on COCO trainval35k first.

PASCAL VOC by Version

Version	Models	VOC 2007 test, 07+12	VOC 2012 test
v1	SSD 299, SSD 443 (Inception)	61.2, 69.2	54.4, 63.3 (trained on 07+12)
v2	SSD300, SSD500	72.1, 75.1	70.3, 73.1 (07++12)
ECCV	SSD300, SSD512	74.1 in Table 1 (74.3 elsewhere), 76.8	72.4, 74.9 (07++12)
v3–v5	SSD300, SSD512	74.3, 76.8	72.4, 74.9 (07++12)
v3–v5	SSD300*, SSD512*	77.2, 79.8	75.8, 78.5 (07++12)

Against these, the baselines were Faster R-CNN at 73.2% on VOC 2007 and 70.4% on VOC 2012, and YOLO at 57.9% on VOC 2012. In v1, Fast and Faster R-CNN were still ahead, which the authors suggested was due to their larger input images on VOC 2007 and to the extra VOC 2007 test images in their training data on VOC 2012. In v2 only SSD500 beat Faster R-CNN; from the ECCV version on, both SSD models did, on both test sets. With COCO pre-training, the ECCV version reports 81.5% for SSD512 on VOC 2007 and 80.0% on VOC 2012; v3 and later give 81.6% on VOC 2007. The ECCV and later abstracts give SSD512 76.9% on VOC 2007, while every table prints 76.8.

COCO and the Column Labels

The COCO test-dev results were printed under different column orders:

Version	Columns as printed	SSD300	SSD512 (SSD500 in v2)
v2	AP@0.5, AP@0.75, AP@[0.5:0.95]	38.0, 20.5, 20.8	43.7, 24.7, 24.4
ECCV	AP@0.5, AP@0.75, AP@[0.5:0.95]	41.2, 23.2, 23.4	46.4, 26.7, 27.7
v3–v5	AP@[0.5:0.95], AP@0.5, AP@0.75	23.2, 41.2, 23.4	26.8, 46.5, 27.8

So SSD300’s COCO AP over IoU 0.5 to 0.95 is 23.4 in the ECCV version and 23.2 in the final one, and the two numbers trade places as AP at 0.75. Each version’s text matches its own table: the ECCV text says SSD300 has an AP over 0.5 to 0.95 similar to Faster R-CNN’s, and the v3 text says it has an AP at 0.75 similar to ION’s and Faster R-CNN’s. One baseline row can be checked against its source. The ION paper (Bell et al., 2016, Table 3) gives its Fast R-CNN baseline 20.5 AP over 0.5 to 0.95 and 19.4 at 0.75; the ECCV table prints 20.5 under 0.75 and 19.4 under 0.5 to 0.95, while v3 and later print them as ION does. In our reading this favors the later labelling, but it does not prove what SSD’s own columns were. The ION rows also differ between the versions because each quotes a different ION model. Version 3 adds AP and recall by object size: SSD300 has an AP of 5.3 on small objects and 39.6 on large ones, SSD512 9.0 and 41.9. Its text says SSD512 is 5.3 points better than Faster R-CNN at IoU 0.75, while its table gives 27.8 against 23.5, a 4.3-point gap. With the expansion augmentation, SSD300* and SSD512* reach 25.1 and 28.8 AP.

ILSVRC Detection

On the ILSVRC 2014 detection val2 set, SSD300 reached 41.1 mAP in v2, 43.2 in the ECCV version and 43.4 in v3 and later, in a section the authors call preliminary. Version 1 had made this benchmark its main test: its full Inception model reached 31.0 mAP, 34.7 with two lower feature maps added and 39.6 at $443 \times 443$, against 44.7 for the two-stage MultiBox baseline on the same network.

Speed

All speed tables are for VOC 2007 test, with SSD timed on an Nvidia Titan X:

• v1: no measurements.
• v2 (Table 5): SSD300 58 and SSD500 23 frames per second, measured with batches of 8.
• ECCV (Table 6): SSD300 46 frames per second with batch 1 and 59 with batch 8, SSD512 19 and 22. The paper names only the GPU.
• v3–v5 (Table 7): the same SSD rows, with the hardware stated: cuDNN v4 and an Intel Xeon E5-2667v3 at 3.20 GHz. The table adds a column of input resolutions.

The 59 frames per second of the abstract is the batch-8 figure; the authors call SSD300 the first real-time method above 70% mAP. The paper does not say who timed the comparison rows. Faster R-CNN with VGG-16 appears at 7 frames per second in every version from v2 on. That figure, and those for Fast YOLO (52.7%, 155 frames per second) and YOLO with VGG-16 (66.4%, 21), match Table 1 of the YOLO paper, whose authors report 7 frames per second for Faster R-CNN “in our tests” without naming the hardware; the Faster R-CNN paper itself reports 5 frames per second for VGG-16 on a K40 GPU. The Faster R-CNN with ZF row of v2 and ECCV (17 frames per second) matches the Faster R-CNN paper’s own figure rather than YOLO’s 18. From v3 the table drops the ZF row and the original YOLO (63.4%, 45), adds YOLO with VGG-16, and changes Faster R-CNN’s box count from 300 to about 6,000.

Baselines as the Paper Ran Them

The VOC and COCO baseline rows cite other papers, and their values match those papers’ tables. The Faster R-CNN rows with COCO pre-training (78.8% on VOC 2007, 75.9% on VOC 2012) and on COCO test-dev (42.7 AP at 0.5, 21.9 over 0.5 to 0.95) appear in the January 2016 arXiv version of Faster R-CNN, not in the NIPS paper that SSD cites. The strongest Faster R-CNN row on COCO (45.3 AP at 0.5) is attributed to the COCO leaderboard, accessed on 25 July 2016. SSD512’s lead over ION (Bell et al., 2016) is against ION’s single models trained on COCO train with Selective Search proposals; the ION paper also reports a competition entry with other proposals and training data at 31.2 AP, which SSD’s table does not list. The Fast and Faster R-CNN baselines use VGG-16, like SSD, but input images with a shorter side of 600 pixels, against SSD’s 300 or 512.

Analyses

The ablations ran on SSD300 and VOC 2007; the SSD article reports what they found. Between versions they changed in ways that alter their message. In v2, the crop sampling was worth 6.7 points and a network without dilated convolution was 0.7 points worse and about 50% slower; from the ECCV version on, the sampling was worth 8.8 points (65.5% to 74.3%) and the network without dilation was “about the same” and about 20% slower, at 74.4% in the ECCV table and 74.2% from v3. In the layer ablation (Table 3), which removed prediction maps while keeping the number of boxes near the original 8,732, the text says mAP dropped “monotonically” from 74.3% to 62.4%, but the second row (74.6%, without conv11_2) is above the first. The error analysis with the tool of Hoiem et al. (2012), in which SSD localized better than R-CNN but confused similar categories more and was very sensitive to object size, appears in every version from v1 on.

Impact

The Caffe implementation accompanied the paper from v2 on. Its training scripts fix details the paper rounds or omits, such as the exact box sizes and the scaling of the offsets, which the SSD article lists.

The authors’ own follow-up came first. DSSD (Fu et al., January 2017), by three of SSD’s authors, called SSD the best detector of the time with respect to the trade-off between speed and accuracy, then replaced VGG-16 with ResNet-101 and added deconvolution layers to bring context to the fine maps, especially for small objects. Its tables give SSD at 513 pixels with ResNet-101 31.2 AP on COCO test-dev.

Other groups adopted SSD as a reference point and as a template:

• YOLOv2. Redmon and Farhadi (2017) compared YOLOv2 with SSD300 and SSD512 on VOC and copied the v3 COCO table, crediting it to the SSD paper. They name SSD and Faster R-CNN as running their predictors on several feature maps and chose a passthrough layer instead, and describe their augmentation as similar to YOLO’s and SSD’s. Their VOC 2007 table lists SSD512’s 76.8% and 19 frames per second under the v2 name “SSD500”.
• A meta-architecture. Huang et al. (2017), comparing detectors in one TensorFlow implementation, used “SSD” broadly for any single feed-forward network that predicts classes and anchor offsets without a second, per-proposal stage, a definition under which MultiBox and the RPN are precursors. Their SSD with VGG-16 reproduced the result reported in the ECCV version. They found SSD models very poor on small objects but competitive on large ones, and less dependent than Faster R-CNN or R-FCN on the classification accuracy of their feature extractor; SSD with MobileNet or Inception V2 formed the fast end of their accuracy–speed frontier.
• Feature pyramids. Lin et al. (2017), introducing the feature pyramid network, call SSD one of the first attempts to use a network’s feature hierarchy as if it were an image pyramid, and argue that by starting at conv4_3 it misses the higher-resolution maps they found important for small objects.
• Focal loss. Lin, Goyal et al. (2017) credit SSD and YOLO with renewing interest in one-stage detectors, put SSD’s AP 10 to 20% below that of two-stage methods, and name SSD among the detectors that face a large class imbalance and answer it with hard negative mining. Their RetinaNet kept all anchors and down-weighted the easy ones instead. In their ablation on COCO minival, a mining variant with SSD’s 1:3 ratio reached at most 31.1 AP, against 36.0 for the focal loss with the same ResNet-101-FPN detector; in our reading, that variant, which applies NMS to the examples before sampling, is RetinaNet’s adaptation rather than SSD’s exact procedure.
• Mobile detection. MobileNetV2 (Sandler et al., 2018) introduced SSDLite, SSD with separable convolutions in the prediction layers, and compared it with SSD300 and SSD512 at 23.2 and 26.8 COCO AP, numbers taken from the YOLOv2 paper and so from SSD’s later versions (see MobileNet).
• Feature fusion. FSSD (Li & Zhou, 2017) calls SSD one of the best detectors for both accuracy and speed, and argues that its pyramid makes it hard to fuse features across scales.

Limitations

The paper states several limitations of the model: much worse accuracy on small objects, which may carry no information in the top layers; more confusion between similar categories, partly because box locations are shared across classes; most of the time spent in the base network, so that SSD512 was close to but not at real time; and, from v3, an open question of how to design the tiling, with alignment of default boxes to each position’s receptive field left for future work. The expansion augmentation was introduced specifically for small objects.

As a publication, the paper is harder to cite than most:

• Numbers move between versions. The same model names carry different results in v2, ECCV and v3 and later; SSD300* and SSD512* exist only in the arXiv versions; and the COCO columns are labelled differently in the ECCV and later versions. Huang et al. note that the later arXiv version has higher numbers than the ECCV version. Later papers quote the starred models inconsistently: DSSD lists SSD300* at 77.5% and SSD512* at 79.5% on VOC 2007 (SSD’s Table 6: 77.2% and 79.8%), and FSSD lists SSD512 at 78.5%, SSD512*’s VOC 2012 value.
• Internal inconsistencies. The abstract’s 76.9% against the tables’ 76.8%; 74.1% against 74.3% for SSD300 in the ECCV version; the 5.3-point gap in the v3 text against 4.3 in its table; and the “monotonic” drop in the layer ablation.
• Mixed baselines. Speed comparisons combine SSD’s own timings with figures from YOLO’s and Faster R-CNN’s papers, and accuracy baselines come from several versions of other papers and a leaderboard.

What Came After

In our reading, the paper’s design became the usual layout of one-stage detectors: predictions from several maps of one network, with fixed reference boxes and a single loss. Several successors targeted weaknesses that the paper named or that later papers found in it. DSSD and FSSD brought deeper features to SSD’s fine maps, feature pyramid networks did so for both one-stage and two-stage detectors, and RetinaNet replaced hard negative mining with the focal loss; SSDLite carried the design to phones. The SSD article covers these variants and their results, and the one-stage object detectors article covers the family.