One-Stage Object Detectors

A one-stage object detector turns object detection into dense prediction. A convolutional network computes a feature map, or a pyramid of them, and a small head predicts, at every position, class scores and box coordinates, usually relative to a few anchor boxes per position; score thresholding and non-maximum suppression then reduce the tens of thousands of candidates to a short list of detections. There is no proposal stage and no per-region computation. YOLO and SSD made the design fast, RetinaNet made it as accurate as two-stage detectors, and anchor-free detectors such as FCOS and CenterNet replaced the anchors with points.

Core Idea

A two-stage detector first proposes a sparse set of candidate regions, then pools features from each one and classifies and refines it. A one-stage detector skips the first stage: it treats every position of a feature map, at every scale and anchor shape, as a candidate and classifies all of them at once with shared convolutional weights. In our reading, Faster R-CNN’s region proposal network (Ren et al., 2015) already works this way, as a class-agnostic dense detector; one-stage detectors give such a head the full set of classes and use its output as the final answer.

Two consequences follow. The cost no longer grows with the number of candidates, since one convolution scores all positions together. And almost all of the candidates are background: Lin et al. (2017, focal loss) put the count at about 100,000 locations per image, against the 1,000–2,000 proposals that the first stage of a two-stage detector passes on. Much of the history of one-stage detectors is about coping with that imbalance.

Building Blocks

• Backbone. A convolutional neural network pretrained on ImageNet classification: VGG-16 in SSD, ResNet in RetinaNet and FCOS, EfficientNet in EfficientDet.
• Multi-scale features. Objects of different sizes are predicted from feature maps of different strides: extra convolutional layers of decreasing resolution in SSD, a feature pyramid network (FPN; Lin et al., 2017) in RetinaNet and FCOS.
• Dense head. Small convolutional subnetworks applied at every position of every level, one producing $K$ class scores per anchor and one producing 4 box offsets per anchor. In RetinaNet the weights are shared across pyramid levels.
• Label assignment. A rule that decides which outputs are trained as positives for which object, by IoU with anchors or by a point’s position inside the object (see the anchor boxes article).
• Loss. A classification loss over all outputs plus a box-regression loss, typically smooth $L_1$, over the positives.
• Post-processing. Decode the boxes, drop low scores, keep the top candidates, and run NMS.

How Data Flows

RetinaNet is the reference design (Lin et al., 2017, focal loss). A ResNet produces stages $C_3$ to $C_5$; an FPN turns them into pyramid levels $P_3$ to $P_7$, where level $P_l$ has stride $2^l$ and 256 channels. On each level, a classification subnet applies four $3 \times 3$ convolutions with 256 filters and ReLU, then a $3 \times 3$ convolution with $KA$ filters and a sigmoid, giving $K$ independent binary probabilities per anchor with no background class; a box subnet of the same shape but separate weights ends in $4A$ linear outputs. With $A = 9$ anchors and the $K = 80$ COCO classes, every position emits 720 class probabilities and 36 offsets.

For a $640 \times 640$ input, the levels are $80 \times 80$, $40 \times 40$, $20 \times 20$, $10 \times 10$ and $5 \times 5$, so the head scores $8{,}525 \times 9 = 76{,}725$ anchors (our arithmetic). At inference RetinaNet keeps at most 1,000 top-scoring predictions per level above a confidence of 0.05, merges the levels, and applies NMS at IoU 0.5.

Mathematical Formulation

For one anchor and one class, let $y \in \{0, 1\}$ be the label and $p = \sigma(z)$ the predicted probability from logit $z$. Write $p_t$ for the probability of the true label: $p_t = p$ if $y = 1$ and $p_t = 1 - p$ otherwise. Binary cross-entropy is $-\log p_t$. The focal loss (Lin et al., 2017) multiplies it by a modulating factor:

$$\mathrm{FL}(p_t) = -\alpha_t \,(1 - p_t)^{\gamma} \log p_t ,$$

where $\alpha_t = \alpha$ for positives and $1 - \alpha$ for negatives. With $\gamma = 0$ and $\alpha_t = 1$ it is cross-entropy. For $\gamma = 2$ the authors note that an example with $p_t = 0.9$ gets 100 times less loss than under cross-entropy, one with $p_t \approx 0.968$ 1,000 times less, while an example with $p_t \le 0.5$ loses at most a factor of 4. RetinaNet uses $\gamma = 2$ and $\alpha = 0.25$, applies the loss to every class output of every anchor, and divides the image’s total by the number of anchors assigned to objects rather than by the number of anchors. The training loss adds a smooth $L_1$ box loss over the assigned anchors.

SSD (Liu et al., 2016) uses a softmax over $K + 1$ classes instead, and keeps the imbalance in check by sampling:

$$L = \frac{1}{N}\left(L_{\text{conf}} + \alpha L_{\text{loc}}\right), \qquad \alpha = 1,$$

with $N$ the number of matched default boxes, $L_{\text{conf}}$ the softmax loss and $L_{\text{loc}}$ a smooth $L_1$ loss on box offsets. Instead of all negatives, SSD keeps the highest-loss ones, at most three per positive (hard negative mining).

Why It Works

Each easy negative contributes little loss, but together they dominate; Lin et al. warn that they can overwhelm training and lead to degenerate models. Two-stage detectors avoid this with their proposal stage and fixed-ratio sampling, such as one positive to three negatives. Hard negative mining discards most negatives; the focal loss keeps them all and shrinks the loss of those already classified well. In RetinaNet’s COCO minival ablation with ResNet-50-FPN, the best $\alpha$-balanced cross-entropy reached 31.1 AP and the focal loss with $\gamma = 2$, $\alpha = 0.25$ reached 34.0; with ResNet-101-FPN, the focal loss beat the best online hard example mining setting by 3.2 AP (36.0 against 32.8).

The NumPy code makes the imbalance concrete on a $32 \times 32$ objectness grid. A synthetic image holds one bright $4 \times 4$ object (16 positive cells) and a dimmer patch that is not an object; a stand-in for a partly trained detector predicts objectness from local brightness. Negatives with $p_t > 0.9$ count as easy. The code compares the share of the total loss from each group as $\gamma$ grows.

import numpy as np

# A 32 x 32 image with one 4 x 4 object and one bright distractor; one "objectness" output per cell.
rng = np.random.default_rng(4)
img = rng.normal(0.0, 0.15, (32, 32))
img[10:14, 18:22] += 1.0                         # the object: 16 positive cells
img[24:26, 4:7] += 0.7                           # a bright patch that is not an object
y = np.zeros((32, 32))
y[10:14, 18:22] = 1

# A partly trained detector: the logit rises with local brightness (3 x 3 mean).
pad = np.pad(img, 1, mode="edge")
local = sum(pad[i:i + 32, j:j + 32] for i in range(3) for j in range(3)) / 9
z = 9.0 * local - 4.0                            # logits
p = 1 / (1 + np.exp(-z))                         # predicted probability of "object"

p_t = np.where(y == 1, p, 1 - p)                 # probability of the true label
pos, neg = y == 1, y == 0
easy = neg & (p_t > 0.9)                         # well-classified background
hard = neg & ~easy
print(f"cells: {pos.sum()} positive, {easy.sum()} easy negatives, {hard.sum()} hard negatives")
print(f"mean p_t: positives {p_t[pos].mean():.2f}, easy negatives {p_t[easy].mean():.3f}, "
      f"hard negatives {p_t[hard].mean():.2f}")

def focal(p_t, gamma, alpha_t=1.0):
    return -alpha_t * (1 - p_t) ** gamma * np.log(p_t)

print("gamma   total   positives   easy neg.   hard neg.   (share of the total loss)")
for gamma in [0, 0.5, 1, 2, 5]:
    L = focal(p_t, gamma)
    tot = L.sum()
    print(f"{gamma:5}  {tot:7.3f}   {L[pos].sum() / tot:8.1%}   {L[easy].sum() / tot:8.1%}"
          f"   {L[hard].sum() / tot:8.1%}")

alpha_t = np.where(y == 1, 0.25, 0.75)           # RetinaNet's alpha = 0.25 for the object class
L = focal(p_t, 2, alpha_t)
print(f"gamma 2, alpha 0.25: positives {L[pos].sum() / L.sum():.1%}, "
      f"easy neg. {L[easy].sum() / L.sum():.1%}, hard neg. {L[hard].sum() / L.sum():.1%}")
print("FL / CE for one example, gamma 2:",
      {q: round(float(focal(q, 2) / focal(q, 0)), 4) for q in [0.5, 0.9, 0.968]})

Output:

cells: 16 positive, 985 easy negatives, 23 hard negatives
mean p_t: positives 0.80, easy negatives 0.980, hard negatives 0.78
gamma   total   positives   easy neg.   hard neg.   (share of the total loss)
    0   30.513      13.8%      66.4%      19.9%
  0.5    8.986      29.3%      35.2%      35.6%
    1    4.099      43.2%      12.8%      44.0%
    2    1.602      56.0%       1.1%      42.9%
    5    0.234      64.3%       0.0%      35.7%
gamma 2, alpha 0.25: positives 29.8%, easy neg. 1.8%, hard neg. 68.5%
FL / CE for one example, gamma 2: {0.5: 0.25, 0.9: 0.01, 0.968: 0.001}

Under cross-entropy ($\gamma = 0$), the 985 easy negatives, each with a loss of about 0.02, supply two thirds of the total, almost five times the 16 positives’ share. At $\gamma = 2$ their share falls to about 1%, and the loss is split between the object and the hard negatives: the distractor and the cells next to the object. The last line reproduces the factors quoted above. With $\alpha = 0.25$ the positives’ share drops further; the authors found that lower $\alpha$ suits higher $\gamma$ because, once easy negatives are down-weighted, the positives need less emphasis. Over noise seeds 0–9, the easy negatives’ share was 64–67% at $\gamma = 0$ and 0.7–1.3% at $\gamma = 2$, while the split of the rest varied with the distractor’s brightness. Real detectors are far more imbalanced than this grid’s 1:63 (the paper’s example is 1:1000).

In PyTorch, binary_cross_entropy_with_logits returns $-\log p_t$ computed stably from the logits. The snippet continues from the arrays above, confirms the totals, and uses autograd to measure the easy negatives’ share of the gradient with respect to the logits. It was run with PyTorch 2.14.1.

import torch
import torch.nn.functional as F

def focal_loss(logits, targets, gamma=2.0, alpha=None):
    ce = F.binary_cross_entropy_with_logits(logits, targets, reduction="none")  # -log p_t
    p_t = torch.exp(-ce)
    loss = (1 - p_t) ** gamma * ce
    if alpha is not None:
        loss = (alpha * targets + (1 - alpha) * (1 - targets)) * loss
    return loss

logits = torch.tensor(z, dtype=torch.float64, requires_grad=True)   # from the NumPy block
targets = torch.tensor(y, dtype=torch.float64)
print("torch", torch.__version__)
for gamma in [0, 2]:
    logits.grad = None
    loss = focal_loss(logits, targets, gamma)
    loss.sum().backward()
    g = logits.grad.abs()
    print(f"gamma {gamma}: total {loss.sum().item():.3f}, "
          f"share of |gradient| from easy negatives {g[easy].sum().item() / g.sum().item():.1%}")
fl = focal_loss(logits, targets, 2.0, alpha=0.25).sum() / targets.sum()   # normalized by positives
print(f"gamma 2, alpha 0.25, normalized by the {int(targets.sum())} positives: {fl.item():.4f}")

Output:

torch 2.14.1
gamma 0: total 30.513, share of |gradient| from easy negatives 70.7%
gamma 2: total 1.602, share of |gradient| from easy negatives 1.9%
gamma 2, alpha 0.25, normalized by the 16 positives: 0.0471

Easy negatives supply 70.7% of the gradient magnitude under cross-entropy and 1.9% under the focal loss, so the updates are driven by the object and the confusing patch.

Training Considerations

• Initialization. With the usual initialization, every output starts near $p = 0.5$, and the loss of the background outputs swamps everything. RetinaNet sets the bias of the last classification layer to $b = -\log((1 - \pi)/\pi)$ with $\pi = 0.01$, so that every anchor starts as background with confidence about 0.99. The authors report that training with plain cross-entropy diverged without this and reached 30.2 AP with it (ResNet-50, COCO minival).
• Data augmentation. SSD’s random crops raised its VOC 2007 mAP by 8.8 points in the authors’ ablation.

Variants

Anchor-based, multi-scale. SSD and RetinaNet (below); YOLOv2 (Redmon and Farhadi, 2017) added anchors to YOLO.

Anchor-free. FCOS (Tian et al., 2019) predicts, at each location inside an object, the distances to the four sides of its box, assigns objects to pyramid levels by size, and adds a one-layer center-ness branch, a target of 1 at the object’s center that decays toward its edges. Multiplied with the class score at test time, it down-weights boxes predicted far from the center, which NMS then removes; in the authors’ ablation it raised COCO minival AP from 33.5 to 37.1. Zhang et al. (2020) traced the remaining gap between FCOS and an anchor-based RetinaNet mainly to how positives and negatives are defined. CornerNet (Law and Deng, 2018) detects a box as a pair of keypoints, its top-left and bottom-right corners, grouped by learned embeddings. CenterNet (Zhou et al., 2019) predicts a heatmap of object centers and regresses the box size at each peak, replacing IoU-based NMS with a $3 \times 3$ max-pooling over the heatmap. Both keypoint detectors train their heatmaps with a variant of the focal loss.

Efficiency-oriented. EfficientDet (Tan et al., 2020) follows the one-stage design with anchors and the focal loss, adds a weighted bi-directional feature pyramid (BiFPN), and scales backbone, pyramid, head and input resolution together with one coefficient.

NMS-free. YOLOv10 (Wang et al., 2024) trains a second head with one-to-one assignment and uses only that head at inference, so it needs no NMS.

Use in Computer Vision

One-stage detectors are the usual choice when speed matters: video, robotics, driving, embedded devices, and the per-frame detector in tracking-by-detection. Compared with two-stage designs:

• Speed. With the same VGG-16 backbone on PASCAL VOC 2007 test, the SSD authors measured Faster R-CNN at 7 frames per second and 73.2% mAP, and SSD300 at 46 frames per second and 74.3% mAP.
• Simplicity and imbalance. One network with no proposal stage or RoI pooling, but trained on all of the background.
• Localization. A two-stage detector refines each box twice and pools features aligned to the proposal; a one-stage head predicts from features at a fixed grid position. The evidence is mixed: Redmon et al. (2016) found localization errors to be YOLO’s largest error source, more than in Fast R-CNN, while the SSD authors found SSD to have less localization error than R-CNN; the two analyses use different baselines.

Limitations

• Small objects cover few positions of coarse maps; the SSD authors report much worse results on them than on large objects, improved but not solved by a larger input.
• Duplicates. Dense heads fire on many neighboring positions, so most designs depend on NMS and its threshold, which fails when real objects overlap heavily.
• Hyperparameters. Anchor shapes, IoU thresholds or per-level size ranges, and the focal loss’s $\gamma$ and $\alpha$, which interact, must be tuned, sometimes again for a new dataset.

Important Models

OverFeat (Sermanet et al., 2014), which Lin et al. (2017) count among the first modern one-stage detectors, applied one ConvNet as a sliding window over several image scales, computed efficiently by running the convolutions over the whole image, with a regression network predicting a box at each position. Instead of suppressing overlapping boxes it merged them, accumulating their evidence. It placed third in the ILSVRC 2013 detection task with 19.4% mAP and reached 24.3% in post-competition work.

YOLO (Redmon et al., 2016) divides the image into an $S \times S$ grid; the cell containing an object’s center is responsible for it and predicts $B$ boxes with confidences and one set of class probabilities, an $S \times S \times (5B + C)$ output ($7 \times 7 \times 30$ on VOC) from fully connected layers. The YOLO article covers the family.

SSD (Liu et al., 2016), unlike YOLO, predicts with $3 \times 3$ convolutions on six feature maps of decreasing resolution. Trained on VOC 2007 and 2012 trainval, SSD300 reached 74.3% mAP on VOC 2007 test at 59 frames per second on a Titan X with batches of 8 images, as given in the abstract and speed table; the per-class table of the ECCV version lists 74.1 for the same training data.

RetinaNet (Lin et al., 2017) combined an FPN, dense anchors and the focal loss. The authors report that it matched the speed of earlier one-stage detectors while surpassing the accuracy of the two-stage detectors of the time: their best model, ResNet-101-FPN with 800-pixel images, trained with scale jitter and 1.5 times longer than the models in their speed–accuracy table, reached 39.1 AP on COCO test-dev at about 5 frames per second, against 36.2 for Faster R-CNN with FPN on the same backbone.

Method	Year	Backbone	Feature maps	Assignment	Classification loss	Reported result
OverFeat	2014	Own ConvNet, layers 1–5 similar to AlexNet	Sliding window over several image scales	—	Softmax classifier with a background class; $\ell_2$ box regression	ILSVRC 2013 detection test 24.3% mAP (post-competition)
YOLO	2016	24 conv + 2 FC layers	One $7 \times 7$ grid	Cell containing the object center	Sum-squared error	VOC 2007 test 63.4% mAP, 45 fps
SSD300	2016	VGG-16	6 maps, $38 \times 38$ to $1 \times 1$	Default boxes, IoU > 0.5	Softmax + hard negatives 3:1	VOC 2007 test 74.3% mAP (74.1 in its Table 1), 59 fps (batch 8); COCO test-dev 23.4 AP
RetinaNet	2017	ResNet-101-FPN	$P_3$–$P_7$	Anchors, IoU ≥ 0.5 / < 0.4	Focal loss	COCO test-dev 39.1 AP (800 px, scale jitter, 1.5× schedule)
CornerNet	2018	Hourglass-104	One map, stride 4	Corner keypoints	Focal-loss variant	COCO test-dev 40.5 AP single-scale, 42.1 multi-scale
FCOS	2019	ResNet-101-FPN	$P_3$–$P_7$	Points inside the box, size range per level	Focal loss + center-ness	COCO test-dev 41.5 AP
CenterNet	2019	Hourglass-104	One map, stride 4	Object center	Focal-loss variant	COCO test-dev 42.1 AP single-scale, 45.1 multi-scale
EfficientDet-D7	2020	EfficientNet-B6 + BiFPN	$P_3$–$P_7$	Anchors	Focal loss ($\gamma = 1.5$)	COCO test-dev 52.2 AP, single-scale

All results are as reported by the authors, with different input sizes, training schedules and hardware, so they are not directly comparable. COCO AP is averaged over IoU thresholds 0.50–0.95; VOC mAP uses IoU 0.5 (see average precision).