SSD (Single Shot MultiBox Detector)

SSD, the Single Shot MultiBox Detector, is a one-stage object detector introduced by Liu, Anguelov, Erhan, Szegedy, Reed, Fu and Berg at ECCV 2016. A convolutional neural network computes feature maps of decreasing resolution, and on each of them a $3 \times 3$ convolution predicts, at every position, class scores and box offsets for a small set of fixed reference boxes, which the paper calls default boxes. One forward pass, score thresholding and non-maximum suppression give the detections. With a 300-pixel input it was the first detector, in its authors’ words, to run in real time above 70% mAP on PASCAL VOC 2007. The one-stage object detectors article covers the family, and anchor boxes the reference-box idea in general.

Problem Addressed

In 2015 the accurate detectors resampled the image for every candidate box. Faster R-CNN (Ren et al., 2015) proposed regions with a network and then pooled features from each proposal for a second classifier, and the SSD paper puts its speed at 7 frames per second. YOLO (Redmon et al., 2016) dropped the second stage and ran at 45 frames per second, but at 63.4% mAP on VOC 2007 test against Faster R-CNN’s 73.2%. The SSD authors set out to keep the single pass and recover the accuracy. They attribute the gain to three choices: small convolutional predictors instead of YOLO’s fully connected layer, separate predictors for boxes of different aspect ratios, and predictions from several feature maps, so that each map handles a range of object sizes. They note that a single default box per position on the topmost map would make SSD resemble OverFeat, and that a fully connected predictor over the whole top map without explicit aspect ratios would approximately reproduce YOLO.

Architecture

Base Network

SSD300 is built on VGG-16 pre-trained on ImageNet classification, which the paper calls the base network (a backbone in later terms). Following DeepLab-LargeFOV (see DeepLab), the fully connected layers fc6 and fc7 become convolutions with subsampled weights, pool5 changes from $2 \times 2$ with stride 2 to $3 \times 3$ with stride 1, and fc6 uses dilated (“à trous”) convolution; dropout and fc8 are removed. The first prediction layer, conv4_3, has a larger feature norm than the deeper layers, so SSD applies the L2 normalization of ParseNet, an earlier paper by the first and last authors: each position’s feature vector is normalized and multiplied by a learned scale initialized to 20.

Extra Feature Layers

Four blocks are added after conv7 (the converted fc7), each a $1 \times 1$ convolution followed by a $3 \times 3$ convolution. With a $300 \times 300$ input, SSD300 predicts from six maps (paper, Fig. 2):

Layer	Map size	Channels	Default boxes per position	Boxes
conv4_3 (VGG)	$38 \times 38$	512	4	5,776
conv7 (fc7)	$19 \times 19$	1024	6	2,166
conv8_2	$10 \times 10$	512	6	600
conv9_2	$5 \times 5$	256	6	150
conv10_2	$3 \times 3$	256	4	36
conv11_2	$1 \times 1$	256	4	4

That gives 8,732 default boxes per image. SSD512 adds a seventh map, conv12_2, and has 24,564 boxes. Unlike a feature pyramid network, the maps are used as the network computes them, with no top-down path.

Convolutional Predictors

On an $m \times n$ map with $p$ channels and $k$ default boxes per position, a $3 \times 3 \times p$ convolution with $(c + 4)k$ filters outputs $c$ class scores (including background) and 4 offsets for every box, $(c + 4)kmn$ numbers per map. Each map has its own predictor weights. The four offsets are shared by all classes, a choice the authors link to SSD’s confusions between similar categories.

Default Boxes

For $m$ prediction maps, the scale of the boxes on map $k$, as a fraction of the image side, is

$$s_k = s_{\min} + \frac{s_{\max} - s_{\min}}{m - 1}\,(k - 1), \qquad k \in [1, m],$$

with $s_{\min} = 0.2$ and $s_{\max} = 0.9$. Each scale is combined with aspect ratios $a_r \in \{1, 2, 3, \tfrac{1}{2}, \tfrac{1}{3}\}$, giving width $w = s_k \sqrt{a_r}$ and height $h = s_k / \sqrt{a_r}$, so $a_r$ is width over height (the anchor boxes article uses height over width). A sixth, square box has the intermediate scale $s'_k = \sqrt{s_k s_{k+1}}$. Boxes are centered at $\left(\frac{i + 0.5}{|f_k|}, \frac{j + 0.5}{|f_k|}\right)$ on a $|f_k| \times |f_k|$ map. The scales are set so that each map responds to a band of object sizes; the arXiv versions (not the ECCV text) add that they need not match the layers’ actual receptive fields, and from v3 call the optimal tiling an open question.

For VOC, SSD300 gives conv4_3 a separate scale of 0.1 and leaves out the ratios 3 and 1/3 on conv4_3, conv10_2 and conv11_2. For COCO, whose objects are smaller, $s_{\min}$ becomes 0.15 and the conv4_3 scale 0.07, about 21 pixels; the revised arXiv version adds that SSD512 uses 0.1 and 0.04. The released Caffe script rounds the spacing: for VOC it sets the box sizes to 10%, 20%, 37%, 54%, 71% and 88% of the image side, so the top scale is 0.88 rather than 0.9.

Training

Matching Strategy

Each ground-truth box is first matched to the default box with the highest IoU (“Jaccard overlap” in the paper), as in MultiBox, and then every default box whose IoU with some ground-truth box exceeds 0.5 is matched to it as well. The second rule lets several overlapping boxes be positives for one object, which the authors say simplifies learning, since the network need not pick the single best box. All unmatched boxes are negatives. In the paper’s Fig. 1, a dog matches a box on the $4 \times 4$ map but none on the $8 \times 8$ map, whose boxes are too small.

Training Objective

With $x_{ij}^p \in \{0, 1\}$ indicating that default box $i$ is matched to ground-truth box $j$ of class $p$, the loss is

$$L(x, c, l, g) = \frac{1}{N}\left( L_{\text{conf}}(x, c) + \alpha\, L_{\text{loc}}(x, l, g) \right),$$

where $N$ is the number of matched default boxes and $\alpha = 1$, a value the authors set by cross-validation. The confidence loss is the softmax cross-entropy over the classes:

$$L_{\text{conf}}(x, c) = -\sum_{i \in \text{Pos}} x_{ij}^p \log \hat{c}_i^{\,p} \;-\; \sum_{i \in \text{Neg}} \log \hat{c}_i^{\,0}, \qquad \hat{c}_i^{\,p} = \frac{\exp(c_i^p)}{\sum_q \exp(c_i^q)},$$

with class 0 the background. The localization loss is the smooth $L_1$ loss of Fast R-CNN, $\operatorname{smooth}_{L_1}(z) = 0.5 z^2$ for $|z| < 1$ and $|z| - 0.5$ otherwise, summed over the positives and the four offsets $(cx, cy, w, h)$; the offsets are the center shift divided by the default box’s size and the log ratio of sizes, the encoding described in the anchor boxes article. The ECCV version describes both terms in words; the revised arXiv version (v3 onward) writes out the equations and sets the loss to 0 when $N = 0$. The released code also divides the offsets by “variances” of 0.1 for the center and 0.2 for the size, a scaling that the paper does not mention.

Hard Negative Mining

After matching, nearly all default boxes are negatives. SSD sorts the negatives by their confidence loss and keeps only the highest, so that there are at most three negatives per positive; the authors report faster optimization and more stable training than with all negatives. The other negatives contribute nothing to that step. (The earliest arXiv versions describe the sort key as the highest confidence rather than the confidence loss.)

Data Augmentation

Each training image is replaced by one of: the whole image; a patch whose minimum IoU with the objects is 0.1, 0.3, 0.5, 0.7 or 0.9; or a random patch. Patches cover between 0.1 and 1 of the image with aspect ratios between 1/2 and 2, a ground-truth box is kept when its center lies in the patch, and the patch is resized, flipped with probability 0.5 and photometrically distorted. In the ablation (Table 2), this sampling raised SSD300’s VOC 2007 test mAP from 65.5% to 74.3%, the largest effect in that table; the authors expect two-stage detectors to gain less, because pooling features from each proposal already gives some robustness to translation. See data augmentation for the technique in general.

The revised arXiv version (v3 onward, November 2016) adds a “zoom out” step for small objects: before cropping, the image is placed at a random position on a larger canvas filled with the mean color (16 times the image size in the paper’s wording; the released code expands half of the images, by a side factor drawn between 1 and 4), and training runs twice as long. The authors describe a consistent gain of 2 to 3 points of mAP for the models trained this way, SSD300* and SSD512*; their Table 6 shows gains from 1.6 to 3.6 points (results below).

Training used SGD with momentum 0.9, weight decay 0.0005, batch size 32 and an initial learning rate of $10^{-3}$; on VOC 2007 SSD300 trained 40k iterations at that rate, then 10k each at $10^{-4}$ and $10^{-5}$. New layers were initialized with Xavier initialization.

Example: Default Boxes, Matching and Hard Negative Mining

The NumPy code builds SSD300’s 8,732 default boxes with the scale formula (conv4_3 at 0.1, the formula over the other five maps, and $s_{m+1} = 1$ for the last extra square box, our choice: the first arXiv version set it to 1, the later versions leave it undefined, and the released code uses 1.05). It then matches a rendered $300 \times 300$ scene of five objects, with and without conv4_3, and square objects of eight sizes at 300 random positions each.

import numpy as np

# SSD300 tiling: six feature maps, the paper's scale rule, aspect ratios per layer.
fmap = [38, 19, 10, 5, 3, 1]                       # conv4_3, conv7, conv8_2, conv9_2, conv10_2, conv11_2
m = len(fmap) - 1                                   # formula over the 5 layers after conv4_3
s = [0.1] + [0.2 + (0.9 - 0.2) / (m - 1) * (k - 1) for k in range(1, m + 1)]
s_next = s[1:] + [1.0]                              # scale used for the extra square box
ratios = [[1, 2, 1/2], [1, 2, 3, 1/2, 1/3], [1, 2, 3, 1/2, 1/3],
          [1, 2, 3, 1/2, 1/3], [1, 2, 1/2], [1, 2, 1/2]]

def default_boxes(fmap, s, s_next, ratios, img=300):
    boxes, layer = [], []
    for k, f in enumerate(fmap):
        shapes = [(s[k] * np.sqrt(a), s[k] / np.sqrt(a)) for a in ratios[k]]   # w = s*sqrt(a), h = s/sqrt(a)
        shapes.append((np.sqrt(s[k] * s_next[k]),) * 2)                      # extra square, s'_k
        c = (np.arange(f) + 0.5) / f
        cy, cx = np.meshgrid(c, c, indexing="ij")
        for (w, h) in shapes:                                                # (cx, cy, w, h) -> corners
            b = np.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], -1).reshape(-1, 4)
            boxes.append(b * img)
            layer += [k] * len(b)
    return np.concatenate(boxes), np.array(layer)

D, layer = default_boxes(fmap, s, s_next, ratios)
print("scales:", [round(x, 3) for x in s], "| boxes per layer:",
      np.bincount(layer).tolist(), "| total:", len(D))

# A rendered 300 x 300 scene: ground-truth boxes are read from the object masks.
H = W = 300
yy, xx = np.mgrid[0:H, 0:W] + 0.5
masks = {"bus":    (np.abs(xx - 110) <= 85) & (np.abs(yy - 200) <= 55),
         "person": ((xx - 245) / 22) ** 2 + ((yy - 150) / 70) ** 2 <= 1,
         "dog":    ((xx - 230) / 30) ** 2 + ((yy - 40) / 20) ** 2 <= 1,
         "bottle": (np.abs(xx - 60) <= 7) & (np.abs(yy - 50) <= 15),
         "bird":   ((xx - 150) / 6) ** 2 + ((yy - 60) / 5) ** 2 <= 1}
def box_of(mk):
    r, c = np.nonzero(mk)
    return np.array([c.min(), r.min(), c.max() + 1, r.max() + 1], float)
G = np.stack([box_of(mk) for mk in masks.values()])

def iou(A, B):
    lt, rb = np.maximum(A[:, None, :2], B[None, :, :2]), np.minimum(A[:, None, 2:], B[None, :, 2:])
    inter = np.prod(np.clip(rb - lt, 0, None), 2)
    area = lambda X: np.prod(X[:, 2:] - X[:, :2], 1)
    return inter / (area(A)[:, None] + area(B)[None] - inter)

def match(D, G, thr=0.5):
    """SSD matching: each object takes its best default box, then every box with IoU > thr."""
    J = iou(D, G)
    gt = J.argmax(1)
    gt[J.max(1) <= thr] = -1                          # background
    for j in range(len(G)):                           # best-overlap rule overrides
        gt[J[:, j].argmax()] = j
    return gt, J

gt, J = match(D, G)
names = list(masks)
print(f"{'object':>7} {'w x h':>9} {'best IoU':>8} {'matched':>7}  per layer (38,19,10,5,3,1)")
for j, n in enumerate(names):
    w, h = G[j, 2:] - G[j, :2]
    per = [int(((gt == j) & (layer == k)).sum()) for k in range(len(fmap))]
    print(f"{n:>7} {w:4.0f}x{h:<4.0f} {J[:, j].max():8.2f} {(gt == j).sum():7d}  {per}")
print("positives:", (gt >= 0).sum(), "of", len(D))

# Same scene without conv4_3: the coarsest tiling the small objects can use is 19 x 19.
D2, l2 = D[layer > 0], layer[layer > 0]
gt2, J2 = match(D2, G)
print("without conv4_3:", {n: (int((gt2 == j).sum()), round(float(J2[:, j].max()), 2))
                           for j, n in enumerate(names)})

# Square objects of several sizes at 300 random positions each: matches above 0.5.
rng = np.random.default_rng(0)
print(" side   mean matches  share with none > 0.5   best layer")
for side in [10, 20, 30, 45, 60, 100, 150, 220]:
    c = rng.uniform(side / 2, 300 - side / 2, (300, 2))
    B = np.hstack([c - side / 2, c + side / 2])
    Jb = iou(D, B)
    n_above = (Jb > 0.5).sum(0)
    best_layer = np.bincount(layer[Jb.argmax(0)], minlength=6).argmax()
    print(f"{side:5d}   {n_above.mean():12.1f}   {np.mean(n_above == 0):20.0%}   {fmap[best_layer]:>4} x {fmap[best_layer]}")

Output:

scales: [0.1, 0.2, 0.375, 0.55, 0.725, 0.9] | boxes per layer: [5776, 2166, 600, 150, 36, 4] | total: 8732
 object     w x h best IoU matched  per layer (38,19,10,5,3,1)
    bus  170x110      0.68      18  [0, 0, 13, 5, 0, 0]
 person   44x140      0.58       8  [0, 8, 0, 0, 0, 0]
    dog   60x40       0.67      16  [8, 8, 0, 0, 0, 0]
 bottle   14x30       0.47       1  [1, 0, 0, 0, 0, 0]
   bird   12x10       0.13       1  [1, 0, 0, 0, 0, 0]
positives: 44 of 8732
without conv4_3: {'bus': (18, 0.68), 'person': (8, 0.58), 'dog': (8, 0.67), 'bottle': (1, 0.12), 'bird': (1, 0.03)}
 side   mean matches  share with none > 0.5   best layer
   10            0.0                   100%     38 x 38
   20            0.0                   100%     38 x 38
   30           10.6                     0%     38 x 38
   45           10.9                     0%     38 x 38
   60           14.7                     0%     19 x 19
  100           16.6                     0%     10 x 10
  150           16.7                     0%     10 x 10
  220           13.1                     0%      3 x 3

The tiling reproduces the paper’s 8,732 boxes. Objects of medium and large size are matched by 8 to 18 default boxes, all on the one or two maps whose scale fits them: the bus on the $10 \times 10$ and $5 \times 5$ maps, the tall person only on the $19 \times 19$ map, through its tall 1:2 and 1:3 boxes. The bottle and the bird reach no default box above 0.5 and get a single positive only through the best-overlap rule, the bird with an IoU of 0.13, which asks the network to turn a 30-pixel box into a 12-pixel one. Without conv4_3 the forced matches fall to IoUs of 0.12 and 0.03. The random placements show the threshold behind this: the smallest default boxes have a scale of 30 pixels, and by our arithmetic a square inside the $30 \times 30$ box reaches IoU 0.5 only from a side of $30/\sqrt{2} \approx 21$ pixels, so every 10- and 20-pixel object went unmatched by the threshold rule while every object from 30 pixels up had about 11 to 17 matches.

The second block adds a bright poster, which is not an object, and stands in for a partly trained detector over the 21 VOC classes: each box’s background logit falls with the mean brightness inside it, while the overlapped object’s class and, on the poster, “tvmonitor” rise. It keeps the hardest negatives at 3:1 and evaluates the full loss with offsets equal to the targets plus noise.

# Continues from the first block: D, layer, G, names, masks, gt, J, H, W, xx, yy.
rng = np.random.default_rng(1)
cls = {"bus": 6, "person": 15, "dog": 12, "bottle": 5, "bird": 3}   # PASCAL VOC indices, 0 = background
labels = np.array([cls[n] for n in names])

# Image: the objects plus a bright poster that is not an object, with noise.
img = np.zeros((H, W))
for mk in masks.values():
    img[mk] = 1.0
poster = (np.abs(xx - 40) <= 25) & (np.abs(yy - 105) <= 18)
img[poster] = 1.0
img += rng.normal(0, 0.1, img.shape)

# A partly trained detector: the background logit falls with the mean brightness inside the box.
ii = np.pad(img.cumsum(0).cumsum(1), ((1, 0), (1, 0)))              # integral image
x1, y1, x2, y2 = np.clip(np.round(D), 0, 300).astype(int).T
area = np.maximum((x2 - x1) * (y2 - y1), 1)
bright = (ii[y2, x2] - ii[y1, x2] - ii[y2, x1] + ii[y1, x1]) / area
logits = rng.normal(0, 1, (len(D), 21))
logits[:, 0] += 8 - 9 * bright
near = J.argmax(1)                                                  # class of the most-overlapped object
logits[np.arange(len(D)), labels[near]] += 3 * bright * (J.max(1) > 0.1)
on_poster = poster[np.clip(((D[:, 1] + D[:, 3]) / 2).astype(int), 0, 299),
                   np.clip(((D[:, 0] + D[:, 2]) / 2).astype(int), 0, 299)]
logits[on_poster, 20] += 4 * bright[on_poster]                      # "tvmonitor" for the poster

target = np.where(gt >= 0, labels[np.maximum(gt, 0)], 0)
logp = logits - logits.max(1, keepdims=True)
logp -= np.log(np.exp(logp).sum(1, keepdims=True))
conf_loss = -logp[np.arange(len(D)), target]                         # softmax loss per default box

pos, neg = gt >= 0, gt < 0
n_pos = pos.sum()
k = min(3 * n_pos, neg.sum())
order = np.argsort(-np.where(neg, conf_loss, -np.inf), kind="stable")
mined = np.zeros(len(D), bool)
mined[order[:k]] = True                                             # hardest negatives, at most 3:1
print(f"positives {n_pos}, negatives {neg.sum()}, kept {mined.sum()} "
      f"({mined.sum() / neg.sum():.1%} of the negatives)")
print(f"mean loss: positives {conf_loss[pos].mean():.2f}, all negatives {conf_loss[neg].mean():.3f}, "
      f"kept negatives {conf_loss[mined].mean():.2f}")
print(f"negatives with loss below 0.05: {np.mean(conf_loss[neg] < 0.05):.0%}")
touch = J.max(1) > 0
print("kept negatives: on an object", (mined & touch & ~on_poster).sum(),
      "| centred on the poster", (mined & on_poster).sum(),
      "| elsewhere", (mined & ~touch & ~on_poster).sum())
print("kept negatives per layer (38,19,10,5,3,1):", np.bincount(layer[mined], minlength=6).tolist())

# The loss L = (L_conf + alpha L_loc) / N, with offsets that are the targets plus noise.
def encode(g, d):
    gc, dc = (g[:, :2] + g[:, 2:]) / 2, (d[:, :2] + d[:, 2:]) / 2
    gw, dw = g[:, 2:] - g[:, :2], d[:, 2:] - d[:, :2]
    return np.hstack([(gc - dc) / dw, np.log(gw / dw)])
def smooth_l1(x):
    return np.where(np.abs(x) < 1, 0.5 * x ** 2, np.abs(x) - 0.5)
t = encode(G[gt[pos]], D[pos])
l = t + rng.normal(0, 0.2, t.shape)
L_loc = smooth_l1(l - t).sum()
L_mined = conf_loss[pos].sum() + conf_loss[mined].sum()
L_all = conf_loss[pos].sum() + conf_loss[neg].sum()
print(f"N = {n_pos}: L_loc/N {L_loc / n_pos:.3f}, L_conf/N {L_mined / n_pos:.3f} with mining, "
      f"{L_all / n_pos:.3f} with every negative")

Output:

positives 44, negatives 8688, kept 132 (1.5% of the negatives)
mean loss: positives 1.80, all negatives 0.912, kept negatives 6.26
negatives with loss below 0.05: 47%
kept negatives: on an object 128 | centred on the poster 4 | elsewhere 0
kept negatives per layer (38,19,10,5,3,1): [101, 31, 0, 0, 0, 0]
N = 44: L_loc/N 0.079, L_conf/N 20.569 with mining, 181.839 with every negative

Mining keeps 1.5% of the negatives, and they are the confusing ones: small boxes lying on part of an object, which by the matching rule are background, and boxes on the poster; none comes from the plain background, and nearly half of all negatives have a loss below 0.05. With every negative included, the confidence term would be about nine times larger and would be driven by the many partly overlapping boxes rather than by the positives. Here the kept negatives come from the two finest maps, which hold most of the boxes that fit inside objects; in real training this depends on the images and on the stage of training.

Inference

SSD resizes the image to $300 \times 300$ or $512 \times 512$, runs the network once, and decodes the 8,732 or 24,564 boxes. Boxes with a class confidence below 0.01 are dropped, NMS at IoU 0.45 is applied per class, and the top 200 detections per image are kept. The authors report that this post-processing takes about 1.7 ms per image for SSD300 with the 20 VOC classes, close to the 2.4 ms of all the added layers, and that about 80% of the forward time is spent in VGG-16.

Speeds on VOC 2007 test (final arXiv version, Table 7), measured on an Nvidia Titan X with cuDNN v4 and an Intel Xeon E5-2667v3 at 3.20 GHz:

Model	mAP (%)	Frames per second, batch 1	Frames per second, batch 8	Boxes	Input
Faster R-CNN (VGG-16)	73.2	7	—	~6,000	~1000 × 600
YOLO (VGG-16)	66.4	21	—	98	448 × 448
Fast YOLO	52.7	155	—	98	448 × 448
SSD300	74.3	46	59	8,732	300 × 300
SSD512	76.8	19	22	24,564	512 × 512

The 59 frames per second of the abstract is the batch-8 figure. The ECCV version’s speed table (Table 6) has the same SSD rows but lists the original YOLO (63.4%, 45 frames per second) and a Faster R-CNN with the ZF network instead of YOLO with VGG-16. The baseline speeds are not the SSD authors’ own measurements: the Faster R-CNN and YOLO rows match the YOLO paper’s tests, and Faster R-CNN’s own paper reports 5 frames per second on a K40. The SSD paper article traces each row to its source.

Performance

All results are as reported by the authors, with mAP at IoU 0.5 for VOC (see average precision). “07+12” is the union of VOC 2007 and 2012 trainval, “07++12” adds VOC 2007 test, and “+COCO” means pre-training on COCO trainval35k before fine-tuning. Values are from the final arXiv version (v5); the ECCV version differs where noted.

Model	VOC 2007 test, 07+12	VOC 2007 test, +COCO	VOC 2012 test, 07++12	VOC 2012 test, +COCO
Faster R-CNN (VGG-16)	73.2	78.8	70.4	75.9
SSD300	74.3 (ECCV Table 1: 74.1)	79.6	72.4	77.5
SSD512	76.8	81.6 (ECCV: 81.5)	74.9	80.0
SSD300*	77.2	81.2	75.8	79.3
SSD512*	79.8	83.2	78.5	82.2

The abstract gives 76.9% for SSD512 on VOC 2007, while the tables print 76.8. On VOC 2007, training on 2007 trainval alone gave 68.0% (SSD300) and 71.6% (SSD512). On the ILSVRC 2014 detection val2 set, a preliminary SSD300 reached 43.4 mAP (43.2 in the ECCV version).

On COCO test-dev 2015, the final version reports AP averaged over IoU 0.5 to 0.95, AP at 0.5 and AP at 0.75 of 23.2, 41.2 and 23.4 for SSD300, 26.8, 46.5 and 27.8 for SSD512, 25.1, 43.1 and 25.8 for SSD300*, and 28.8, 48.5 and 30.3 for SSD512*. The ECCV table labels its columns differently, so that SSD300’s 23.4 appears there as the AP over 0.5 to 0.95 and 23.2 as the AP at 0.75; the SSD paper article compares the two labellings. The final version adds results by object size: SSD300 has an AP of 5.3 on small objects against 39.6 on large ones, SSD512 9.0 against 41.9. SSD512’s advantage over Faster R-CNN is larger at IoU 0.75 than at 0.5 (4.3 against 1.2 points in the table; the text says 5.3 at 0.75). The authors conjecture that Faster R-CNN stays more competitive on small objects because it refines each box twice.

In the multi-layer ablation (Table 3), the authors removed prediction maps while keeping the number of boxes near 8,732 by stacking scales on the remaining maps. SSD300 fell from 74.3% with all six maps to 62.4% with conv7 alone, and the trend depended on whether boxes crossing the image boundary were kept. Removing the ratios 3 and 1/3 cost 0.6 points, and removing 2 and 1/2 as well another 2.1 (Table 2).

Strengths

• Speed with accuracy. It matched or exceeded Faster R-CNN with the same VGG-16 base at several times the speed, at a time when single-pass detectors were far less accurate.
• Simple training and deployment. One network, one loss, no proposal stage, no per-region layers.
• Aspect ratios. The authors’ error analysis found SSD robust to object aspect ratio, which they attribute to its default boxes of several shapes per position.
• Localization. Compared with R-CNN in the same analysis tool, SSD had fewer localization errors, which the authors attribute to learning box regression and classification together.

Limitations

• Small objects. The authors found SSD very sensitive to object size, with much worse results on small objects, and note that small objects may carry no information in the top layers. A larger input helps but, they write, leaves much room to improve. The example above shows the assignment side of the problem: objects smaller than the finest default boxes get only weak forced matches.
• Weak features on the finest map. The finest prediction map, conv4_3, is a shallow VGG layer. Lin et al. (2017), introducing the feature pyramid network, describe SSD as avoiding the low-level high-resolution maps and starting its pyramid at conv4_3, which leaves out the finer maps they found important for small objects. In our reading, FPN, DSSD and FSSD all answer this by bringing deeper features to the fine maps.
• Class confusion. SSD confused similar categories, especially animals, more than R-CNN did; the authors relate this to sharing box locations across classes.
• Hand-set tiling. Scales, ratios, the 0.5 threshold and the 3:1 ratio are fixed by hand, and the published scales differ between datasets and input sizes.
• Dependence on augmentation. Without the crop sampling SSD300 lost 8.8 points, and the zoom-out step added 1.6 to 3.6 more, so the reported accuracy rests on the augmentation recipe as much as on the architecture.

Historical Importance

SSD grew out of MultiBox, developed by several of its authors at Google. Erhan et al. (2014) trained a network to output a fixed number of class-agnostic boxes (100 or 200) from fully connected layers, with an $L_2$ loss on normalized corner coordinates and a logistic objectness score, and assigned predictions to ground truth by bipartite matching to priors found by k-means clustering; a second network then classified the boxes. Szegedy et al. (2014) made the priors convolutional and multi-scale: 11 priors per position on grids from $8 \times 8$ down to $2 \times 2$ plus a single largest prior, 1,420 in all; in their comparison, convolutional priors raised class-agnostic AP at IoU 0.5 on the ILSVRC detection data from 0.417 to 0.529. SSD kept the matching and the form of the objective but predicted all classes directly from each default box, so that one network replaced proposal and classification.

The paper itself changed considerably before publication. The first arXiv version (December 2015) used an Inception network, called the default boxes “priors”, used an $L_2$ localization loss and a multi-class logistic loss with $\alpha = 0.06$, and reported 61.2% on VOC 2007 test at $299 \times 299$; the March 2016 version moved to VGG-16 with SSD300 and SSD500 (72.1% and 75.1%), and the ECCV version to SSD512.

Together with YOLO, SSD showed that a single dense pass could compete with two-stage detectors; in our reading, its design of predicting from several maps of one network became the usual layout of one-stage detectors. The next steps addressed its weaknesses: feature pyramids added top-down context to the fine maps, and the focal loss of RetinaNet (Lin et al., 2017) kept all negatives but down-weighted the easy ones instead of mining the hard ones (see one-stage object detectors). Its design was also carried to other domains: the face detector S³FD (Zhang et al., 2017) cites SSD’s multi-scale prediction as an inspiration, and the ACT action detector (Kalogeiton et al., 2017) builds on SSD with anchor cuboids, default boxes extended over time (see face detection and spatio-temporal action detection).

Variants

DSSD (Fu, Liu, Ranga, Tyagi and Berg, arXiv, January 2017), from three of SSD’s authors, replaced VGG-16 with ResNet-101 and added deconvolution layers after the SSD layers, an asymmetric encoder–decoder whose upsampled maps are combined with the SSD maps of the same size. Its deconvolution module upsamples with a learned $2 \times 2$ transposed convolution, passes both inputs through convolutions with batch normalization, and multiplies them element-wise, which worked better than summing (78.6% against 78.4% on VOC 2007 test). ResNet-101 SSD at $321 \times 321$ alone reached 76.4%, below VGG-16 SSD300*, and a residual prediction module before each output raised it to 77.1%. The authors also added an aspect ratio of 1.6 after clustering the training boxes, and trained the deconvolution side first with the SSD weights frozen; fine-tuning everything afterwards lowered accuracy (77.9%). DSSD513 reported 81.5% on VOC 2007 test (07+12), 80.0% on VOC 2012 test (07++12) and 33.2 AP on COCO test-dev, against 31.2 for SSD513 with the same backbone; on COCO’s small objects DSSD321 raised AP from 6.2 to 7.4. It ran at 5.5 frames per second (batch 1, Titan X) against 6.8 for SSD513.

FSSD (Li and Zhou, arXiv, December 2017) fuses features once instead of layer by layer. Conv4_3, fc7 and a $10 \times 10$ extra layer are reduced to 256 channels by $1 \times 1$ convolutions, resized to $38 \times 38$ by bilinear interpolation, concatenated and batch-normalized; a chain of stride-2 $3 \times 3$ convolutions then builds a new pyramid for the SSD heads. In the ablation, concatenation beat element-wise summation (78.6% against 76.3%). FSSD300 reported 78.8% on VOC 2007 test (07+12) and 82.7% with COCO pre-training, at 65.8 frames per second on a GTX 1080 Ti where the authors measured SSD300 at 85, and 27.1 COCO AP against 25.1 for SSD300*; FSSD512 reached 31.8 AP against 28.8, with a small-object AP (14.2) above DSSD513’s (13.0).

SSDLite (Sandler et al., 2018) was introduced with MobileNetV2 for mobile devices: every regular convolution in the SSD prediction layers becomes a depthwise separable one. With MobileNetV2 and 80 classes, this cut the head from 14.8M parameters and 1.25B multiply-adds to 2.1M and 0.35B; the MobileNet article gives its COCO results.

Implementations

The authors’ implementation is in Caffe (the ssd branch of weiliu89/caffe). Its training scripts fix the details noted above that the paper leaves out or rounds, and their crop sampler draws the patch’s side length between 0.3 and 1 of the image’s.

TorchVision 0.29 (as of October 2026) provides ssd300_vgg16 and ssdlite320_mobilenet_v3_large, each with COCO weights that its documentation says were “produced by following a similar training recipe” to the paper’s. The documentation lists 25.1 box mAP on COCO val2017, 35.6M parameters and 34.86 GFLOPs for ssd300_vgg16, and 21.3 mAP, 3.4M parameters and 0.58 GFLOPs for ssdlite320_mobilenet_v3_large. In the source, ssd300_vgg16 uses the COCO scales (0.07, 0.15, 0.33, 0.51, 0.69, 0.87 and 1.05 for the extra box), offsets scaled by 10, 10, 5 and 5 (the variances above), the matching rule with threshold 0.5, three negatives per positive, and the paper’s 0.01 score threshold, NMS at 0.45 and 200 detections. The SSDLite builder uses $s_{\min} = 0.2$, $s_{\max} = 0.95$, ratios 2 and 3 on all six maps, a score threshold of 0.001, NMS at 0.55 and 300 detections. Both models resize every input to their fixed size. The detection module is marked beta, so check the documentation of the installed version.