RetinaNet

RetinaNet is a one-stage object detector introduced by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He and Piotr Dollár of Facebook AI Research in the paper “Focal Loss for Dense Object Detection” (ICCV 2017). It is a fully convolutional network: a ResNet backbone with a feature pyramid network (FPN), and two small subnetworks that classify and regress nine anchor boxes at every position of every pyramid level. The architecture was deliberately ordinary; the contribution was the focal loss, a reshaped cross-entropy loss that lets a dense detector train on all of its roughly 100,000 anchors per image without being swamped by the easy background ones. With it, the authors reported, a one-stage detector matched and then exceeded the accuracy of the two-stage detectors of the time on COCO. The paper received the ICCV 2017 Best Student Paper award (PAMI TC award list), and RetinaNet became one of the standard baselines of dense detection.

The one-stage object detectors article places RetinaNet in its family and works through the focal loss on fixed predictions. This article covers the model itself: the loss and its derivation, its two parameters, the initialization of the classification prior, the head, the ablations as reported, and RetinaNet’s later life as a baseline.

Problem Addressed

In 2017 the most accurate detectors were two-stage: a proposal stage such as selective search or a region proposal network cut the candidate locations to one or two thousand, and a second stage classified them with sampled minibatches, for instance one positive to three negatives, or with online hard example mining (OHEM; Shrivastava et al., 2016). One-stage detectors such as YOLO and SSD were faster but less accurate; Lin et al. put them within 10–40% of the two-stage state of the art.

The authors asked why, and answered: the extreme foreground–background imbalance during training, which they give as about 1:1000 in their example. A dense detector scores $10^4$ to $10^5$ candidate locations per image, almost all of them background, and they describe two consequences: training is inefficient, because most locations are easy negatives that carry no useful signal, and the easy negatives together “can overwhelm training and lead to degenerate models”. The standard remedy, hard negative mining, selects a small set of examples; the paper argues that in a one-stage detector this remains inefficient, because training is still dominated by easily classified background. Their alternative changes the loss so that every example is kept and the easy ones contribute little.

The paper insists on the division of credit: the detector achieves top results “not based on innovations in network design but due to our novel loss.”

Architecture

RetinaNet is one network with three parts: a ResNet-FPN backbone that turns the image into a pyramid of feature maps, a classification subnet and a box regression subnet. The two subnets are applied to every pyramid level with weights shared across levels.

Backbone

The backbone is an FPN built on a ResNet-50 or ResNet-101 pretrained on ImageNet-1k. The pyramid has levels $P_3$ to $P_7$, where $P_l$ has a resolution $2^l$ times lower than the input, and every level has $C = 256$ channels. $P_3$ to $P_5$ come from the ResNet stages $C_3$ to $C_5$ through FPN’s top-down pathway and lateral connections; $P_6$ is a $3 \times 3$ stride-2 convolution on $C_5$, and $P_7$ a ReLU followed by another $3 \times 3$ stride-2 convolution on $P_6$. Compared with the original FPN, RetinaNet drops the high-resolution level $P_2$ for computational reasons, computes $P_6$ by a strided convolution instead of subsampling, and adds $P_7$ for large objects; the authors report that these changes improve speed while keeping accuracy. They stress that the pyramid itself is essential: preliminary experiments with features from the final ResNet layer alone gave low AP.

Anchors

Each level carries anchors of base area $32^2$ on $P_3$ up to $512^2$ on $P_7$, at aspect ratios 1:2, 1:1 and 2:1 and at three sub-octave scales $2^0$, $2^{1/3}$ and $2^{2/3}$ of the base size: $A = 9$ anchors per position, covering sizes from 32 to 813 pixels on the input. An anchor is assigned to a ground-truth box if their IoU is at least 0.5, to background if its IoU with every box is below 0.4, and ignored in between; the label assignment article covers this rule and its successors. A positive anchor receives a one-hot vector of $K$ class targets and four box regression targets; a background anchor receives a vector of zeros.

Classification Subnet

The classification subnet is a small fully convolutional network: four $3 \times 3$ convolutions with $C = 256$ filters, each followed by ReLU, then a $3 \times 3$ convolution with $KA$ filters and a sigmoid. Each output is an independent binary probability that an anchor contains an object of one class; there is no background class and no softmax. The authors contrast this with the region proposal network of Faster R-CNN: the subnet is deeper, uses only $3 \times 3$ convolutions and shares no parameters with box regression, and they found these choices more important than the exact hyperparameter values.

Box Regression Subnet

The box subnet has the same structure with separate weights and ends in $4A$ linear outputs per position: for each anchor, the offsets of the R-CNN parameterization (see bounding box regression). The regressor is class-agnostic, which the authors note uses fewer parameters and was equally effective, unlike most detectors of the time.

For $K = 80$ COCO classes and $A = 9$, every position emits 720 class probabilities and 36 offsets. The paper counts about 100,000 anchors per image, which means about eight million binary classification outputs per image (our arithmetic).

Training

Focal Loss

For one binary output with label $y \in \{\pm 1\}$ and predicted probability $p = \sigma(x)$ from logit $x$, the paper writes the cross-entropy as $\mathrm{CE}(p_t) = -\log p_t$, where $p_t = p$ for $y = 1$ and $p_t = 1 - p$ otherwise. Even a well-classified example, with $p_t \gg 0.5$, has a loss of noticeable size; the authors’ point is that, summed over a vast number of easy examples, these small losses outweigh the rare class.

The first fix they consider is a class weight, the α-balanced cross-entropy $-\alpha_t \log p_t$, with $\alpha_t = \alpha$ for the foreground and $1 - \alpha$ for the background. It balances the two classes but, they note, does not differentiate between easy and hard examples. The focal loss adds a modulating factor with a focusing parameter $\gamma \ge 0$:

$$\mathrm{FL}(p_t) = -(1 - p_t)^{\gamma} \log p_t ,$$

and in practice its α-balanced form, which the authors report as slightly more accurate:

$$\mathrm{FL}(p_t) = -\alpha_t \, (1 - p_t)^{\gamma} \log p_t .$$

Two properties follow. When an example is misclassified, $p_t$ is small and the factor is close to 1, so the loss is nearly unchanged; as $p_t \to 1$ the factor goes to 0. And $\gamma$ sets how fast easy examples are down-weighted: $\gamma = 0$ gives back cross-entropy. The ratio of the two losses is just the modulating factor,

$$\frac{\mathrm{FL}(p_t)}{\mathrm{CE}(p_t)} = (1 - p_t)^{\gamma},$$

so at $\gamma = 2$ an example with $p_t = 0.9$ gets $1/100$ of its cross-entropy and one with $p_t \approx 0.968$ about $1/1000$, while an example with $p_t \le 0.5$ loses at most a factor of 4. The loss layer combines the sigmoid with the loss for numerical stability.

The gradient shows where the training signal goes. Appendix B of the arXiv version (v2) gives the derivatives with respect to the logit:

$$\frac{\partial\, \mathrm{CE}}{\partial x} = y\,(p_t - 1), \qquad \frac{\partial\, \mathrm{FL}}{\partial x} = y\,(1 - p_t)^{\gamma}\,\bigl(\gamma\, p_t \log p_t + p_t - 1\bigr),$$

for the loss without $\alpha$. In our reading of the two expressions, the cross-entropy gradient of an easy negative shrinks only linearly, as $1 - p_t$, while the focal gradient shrinks as $(1 - p_t)^{\gamma + 1}$ near $p_t = 1$, since $\gamma p_t \log p_t + p_t - 1 \approx -(\gamma + 1)(1 - p_t)$ there; the paper’s own remark is that the focal derivative becomes small as soon as an example is classified correctly ($x_t = yx > 0$).

The authors also say the exact form is not crucial. Appendix A of v2 defines $\mathrm{FL}^* = -\log \sigma(\gamma x_t + \beta) / \gamma$, a shifted and steepened logistic loss in $x_t = yx$; RetinaNet-50 trained with it at 600 pixels reached 33.8 AP with $\gamma = 2, \beta = 1$ and 33.9 with $\gamma = 4, \beta = 0$, against 34.0 for FL and 31.1 for the best cross-entropy, all on COCO minival. Settings that reduce the loss of correctly classified examples worked across a range of $\gamma$ and $\beta$; the hinge loss on $p_t$, tried early, was unstable. The authors describe the focal loss as the opposite of a robust loss such as the Huber loss: a robust loss down-weights examples with large errors (outliers), while the focal loss down-weights the inliers, the easy examples, so that their total stays small however many there are.

The paper treats the binary case and remarks in a footnote that the multi-class extension is straightforward and works well. RetinaNet’s $K$ sigmoid outputs make every class a separate binary problem, so the loss is applied to all $K \times A$ outputs at every position.

The Parameters α and γ

The two parameters interact. Across $\gamma$ from 0 to 5 the best $\alpha$ fell from 0.75 to 0.25, and the authors explain the trend: once easy negatives are down-weighted by the modulating factor, the positives need less extra weight. Note the direction: $\alpha$ weights the rare foreground class, yet the best setting for $\gamma = 2$, $\alpha = 0.25$, gives the foreground a smaller weight than the background (0.25 against 0.75). The authors tested $\alpha$ from 0.01 to 0.999 and found the best values within 0.25–0.75. They used $\gamma = 2$, $\alpha = 0.25$ in all further experiments, report RetinaNet as relatively robust for $\gamma \in [0.5, 5]$, and write that $\alpha = 0.5$ works nearly as well (0.4 AP lower), a run that is not in their tables.

The authors also analysed a converged model, the ResNet-101 model at 600 pixels (36.0 AP), sampling about $10^7$ negative and $10^5$ positive windows and sorting each group’s normalized focal loss. For positives, the cumulative loss curves changed little with $\gamma$; about 20% of the hardest positives carried roughly half of the positive loss. For negatives, increasing $\gamma$ concentrated the loss sharply: at $\gamma = 2$ the vast majority of the loss came from a small fraction of the samples.

Initialization of the Classification Prior

A freshly initialized binary classifier outputs $p \approx 0.5$ for every example. Under imbalance, the authors note, the loss of the frequent class then dominates the total and can make early training unstable. They introduce a prior $\pi$: the probability the model assigns to the rare class at the start of training, set low. With the weights of the last classification layer near zero, setting its bias to

$$b = -\log \frac{1 - \pi}{\pi}$$

gives $\sigma(b) = \pi$ for every anchor. With $\pi = 0.01$, $b \approx -4.6$: every anchor starts as background with confidence about 0.99. This is a change of initialization, not of the loss, and the authors found that it stabilized training for both cross-entropy and the focal loss under heavy imbalance. In their first experiment, plain cross-entropy with the standard initialization diverged; with the prior, the same loss trained RetinaNet-50 to 30.2 AP. They report the results to be insensitive to the exact value of $\pi$ and use 0.01 throughout. The other new convolutions in the subnets have zero bias and Gaussian weights with $\sigma = 0.01$.

The arithmetic of the first iteration shows the size of the effect (our arithmetic, per binary output). Under cross-entropy with $b = 0$ each background output has a loss of $\log 2 \approx 0.69$; with the prior it has $-\log 0.99 \approx 0.010$, about 69 times less, and under the focal loss with $\gamma = 2$ and $\alpha = 0.25$ it has $0.75 \times 0.01^2 \times 0.010 \approx 7.5 \times 10^{-7}$. A positive output starts at $-\log 0.01 \approx 4.6$ under cross-entropy and at $0.25 \times 0.99^2 \times 4.6 \approx 1.1$ under the focal loss. With millions of background outputs per image against far fewer positive ones, the initial cross-entropy without the prior is dominated by the background by several orders of magnitude, and the same holds for the gradient on the shared bias.

Normalization and Procedure

The focal loss is applied to all of about 100,000 anchors of each image, with no sampling, and the image’s total is divided by the number of anchors assigned to a ground-truth box. The authors normalize by the assigned anchors rather than all anchors because most anchors are easy negatives with negligible loss under the focal loss. The training loss is the focal loss plus the smooth $L_1$ box loss of Fast R-CNN on the assigned anchors.

Training uses synchronized stochastic gradient descent over 8 GPUs with 16 images per minibatch, 90,000 iterations, an initial learning rate of 0.01 divided by 10 at 60,000 and 80,000 iterations, weight decay 0.0001 and momentum 0.9. Horizontal flipping is the only augmentation unless noted. The models of the speed–accuracy table took 10 to 35 hours to train. Training uses COCO trainval35k (the 80,000 train images and a 35,000-image subset of val); ablations are evaluated on minival (the remaining 5,000 val images), and main results on test-dev through the evaluation server.

Example: Training a Dense Predictor with Cross-Entropy and Focal Loss

The one-stage object detectors article measures the loss shares on fixed predictions. Here a small dense predictor is trained from scratch with each loss and each initialization, to see the two training effects the paper reports: divergence without the prior, and the gap between cross-entropy and the focal loss.

Each synthetic $128 \times 128$ scene holds two thin rings of radius 3 to 4.2 pixels, the objects, and distractors: two filled discs, two “C” shapes (rings with a gap of about 100°), four line segments, a brightness ramp and Gaussian noise. There is one output per pixel, and the positives are the $3 \times 3$ pixels at each ring centre, so one location in 910 is positive, close to the paper’s example ratio of 1:1000. The discs, C shapes and segments are meant to make the task harder than finding bright pixels. The predictor is a stand-in for RetinaNet’s classification subnet: four $3 \times 3$ convolutions with 16 channels and ReLU, then a $3 \times 3$ convolution to one logit, with the final weights drawn from $\mathcal{N}(0, 0.01^2)$ as in the authors’ Detectron code (the paper states only that layer’s bias) and the final bias set to 0 or to the prior $\pi = 0.01$. It is trained on 64 scenes with SGD (learning rate 0.01, momentum 0.9, 8 scenes per batch) for 300 steps, and scored by per-location average precision on 32 test scenes.

The paper normalizes the focal loss by the number of assigned anchors but does not say how its plain cross-entropy runs were normalized, so every variant runs under both choices: the sum divided by the number of positives, and the mean over all locations. The script loops over three model seeds (0, 1, 2) with the same data and the same batch order; it uses PyTorch 2.14.1 with NumPy 2.5.4 and one thread. It is slow, since it trains 30 networks: the run below took 1,139 s of wall-clock time (about 12 minutes of CPU time) on a shared, heavily loaded machine, and a second run took 758 s and printed identical output apart from the runtime. The script is shown as first run.

# Cross-entropy against focal loss when training a dense predictor under strong imbalance.
# Synthetic 128 x 128 scenes: 2 rings (targets) + discs, "C"s and line segments (distractors).
# Positives: the 3 x 3 pixels at each ring centre, about 1 location in 900.
import time
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)
H = W = 128
yy, xx = np.mgrid[0:H, 0:W]

def ring(img, cy, cx, r, gap=False):
    d = np.hypot(yy - cy, xx - cx)
    m = np.abs(d - r) < 0.8
    if gap:
        m &= np.abs(np.arctan2(yy - cy, xx - cx)) > 0.9    # a "C": ring with a ~100 degree gap
    img[m] += 1.0

def scene(rng):
    img = rng.uniform(-0.3, 0.3) * (xx / W) + rng.uniform(-0.3, 0.3) * (yy / H)
    lab = np.zeros((H, W))
    centres = []
    def place():
        while True:
            cy, cx = rng.integers(8, H - 8, 2)
            if all(np.hypot(cy - a, cx - b) > 14 for a, b in centres):
                centres.append((cy, cx))
                return cy, cx
    for _ in range(2):                                      # targets: thin rings
        cy, cx = place(); ring(img, cy, cx, rng.uniform(3, 4.2))
        lab[cy - 1:cy + 2, cx - 1:cx + 2] = 1
    for _ in range(2):                                      # filled discs
        cy, cx = place(); img[np.hypot(yy - cy, xx - cx) < rng.uniform(3, 4.5)] += 1.0
    for _ in range(2):                                      # "C" shapes
        cy, cx = place(); ring(img, cy, cx, rng.uniform(3, 4.2), gap=True)
    for _ in range(4):                                      # line segments
        cy, cx = rng.integers(4, H - 4, 2); a = rng.uniform(0, np.pi); L = rng.uniform(6, 14)
        t = np.linspace(-L / 2, L / 2, 40)
        img[np.clip(np.round(cy + t * np.sin(a)).astype(int), 0, H - 1),
            np.clip(np.round(cx + t * np.cos(a)).astype(int), 0, W - 1)] += 1.0
    img += rng.normal(0, 0.25, img.shape)
    return img, lab

def make(seed, n):
    rng = np.random.default_rng(seed)
    s = [scene(rng) for _ in range(n)]
    return (torch.tensor(np.stack([a for a, _ in s])[:, None], dtype=torch.float32),
            torch.tensor(np.stack([b for _, b in s])[:, None], dtype=torch.float32))

Xtr, Ytr = make(0, 64)
Xte, Yte = make(1000, 32)
print(f"locations per positive: train {Ytr.numel() / Ytr.sum():.0f}, test {Yte.numel() / Yte.sum():.0f}")

def head(seed, prior):
    torch.manual_seed(seed)
    layers, c = [], 1
    for _ in range(4):
        layers += [nn.Conv2d(c, 16, 3, padding=1), nn.ReLU()]
        c = 16
    last = nn.Conv2d(16, 1, 3, padding=1)
    nn.init.normal_(last.weight, std=0.01)                  # as in Detectron
    nn.init.constant_(last.bias, 0.0 if prior is None else -np.log((1 - prior) / prior))
    return nn.Sequential(*layers, last)

def loss_fn(z, y, kind, alpha, gamma, norm):
    ce = F.binary_cross_entropy_with_logits(z, y, reduction="none")   # -log p_t
    p_t = torch.exp(-ce)
    l = ce if gamma == 0 else (1 - p_t) ** gamma * ce
    if alpha is not None:
        l = (alpha * y + (1 - alpha) * (1 - y)) * l
    return l.sum() / y.sum().clamp(min=1) if norm == "pos" else l.mean()

def ap(score, lab):                                         # per-location average precision
    o = np.argsort(-score.ravel(), kind="stable"); y = lab.ravel()[o]
    prec = np.cumsum(y) / np.arange(1, len(y) + 1)
    return float((prec * y).sum() / y.sum())

def train(seed, prior, kind, alpha, gamma, norm, steps=300, bs=8):
    net = head(seed, prior)
    opt = torch.optim.SGD(net.parameters(), lr=0.01, momentum=0.9)
    g = torch.Generator().manual_seed(123)                  # same batches for every variant
    order = torch.cat([torch.randperm(64, generator=g) for _ in range(steps * bs // 64 + 1)])
    hist, nan = [], False
    for it in range(steps):
        idx = order[it * bs:(it + 1) * bs]
        loss = loss_fn(net(Xtr[idx]), Ytr[idx], kind, alpha, gamma, norm)
        if not torch.isfinite(loss):
            nan = True
            break
        hist.append(loss.item())
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        zte = net(Xte).numpy()
        ztr = net(Xtr)
        ce = F.binary_cross_entropy_with_logits(ztr, Ytr, reduction="none")
        p_t = torch.exp(-ce)
        l = ce if gamma == 0 else (1 - p_t) ** gamma * ce
        if alpha is not None:
            l = (alpha * Ytr + (1 - alpha) * (1 - Ytr)) * l
        easy = (Ytr == 0) & (p_t > 0.9)
        share = (l[easy].sum() / l.sum()).item()
    return hist, nan, (float("nan") if nan else ap(zte, Yte.numpy())), share

variants = [("CE", None, 0, None), ("CE", 0.01, 0, None), ("a-CE a=0.75", 0.01, 0, 0.75),
            ("FL g=2 a=0.25", None, 2, 0.25), ("FL g=2 a=0.25", 0.01, 2, 0.25)]
t0 = time.time()
for norm in ["pos", "mean"]:
    print(f"\nnormalization: {'sum / number of positives' if norm == 'pos' else 'mean over all locations'}")
    print("loss            bias init   loss@0   max loss 0-19   NaN   test AP (seeds 0, 1, 2)   easy-neg. share")
    for name, prior, gamma, alpha in variants:
        aps = []
        for seed in [0, 1, 2]:
            hist, nan, a, share = train(seed, prior, name, alpha, gamma, norm)
            aps.append(a)
            if seed == 0:
                h0, m20, n0, s0 = hist[0], max(hist[:20]), nan, share
        print(f"{name:15} {'pi=0.01' if prior else 'b=0':9} {h0:8.3f} {m20:13.3f}   {str(n0):5}"
              f"   {aps[0]:.3f}  {aps[1]:.3f}  {aps[2]:.3f}          {s0:6.1%}")
print(f"\nruntime {time.time() - t0:.0f} s, torch {torch.__version__}")

Output:

locations per positive: train 910, test 910

normalization: sum / number of positives
loss            bias init   loss@0   max loss 0-19   NaN   test AP (seeds 0, 1, 2)   easy-neg. share
CE              b=0        632.078       632.078   True    nan  0.001  nan            nan%
CE              pi=0.01     13.764        13.764   False   0.001  0.001  0.001           12.8%
a-CE a=0.75     pi=0.01      5.742         5.742   False   0.994  0.001  0.919            5.6%
FL g=2 a=0.25   b=0        118.730       118.730   False   0.001  nan  0.001            0.0%
FL g=2 a=0.25   pi=0.01      1.128         1.128   False   0.971  0.968  0.980            6.8%

normalization: mean over all locations
loss            bias init   loss@0   max loss 0-19   NaN   test AP (seeds 0, 1, 2)   easy-neg. share
CE              b=0          0.694         0.694   False   0.002  0.002  0.003           15.6%
CE              pi=0.01      0.015         0.015   False   0.001  0.001  0.001           58.4%
a-CE a=0.75     pi=0.01      0.006         0.006   False   0.001  0.002  0.007           38.3%
FL g=2 a=0.25   b=0          0.130         0.130   False   0.001  0.001  0.001           14.3%
FL g=2 a=0.25   pi=0.01      0.001         0.001   False   0.004  0.003  0.008            0.1%

runtime 1139 s, torch 2.14.1

Under normalization by positives, the step-0 losses show the effect of the prior on a small scale: with the standard bias, every one of the 131,072 locations in a batch starts near $p = 0.5$, and the cross-entropy, divided by the 144 positives, starts at 632, against 13.8 with the prior. Plain cross-entropy with bias 0 produced non-finite losses on two of the three seeds, the divergence the paper reports for its first attempt. With the prior, cross-entropy no longer diverged, but it still ended at chance level (AP 0.001 on all seeds); α-balanced cross-entropy with $\alpha = 0.75$ learned on two seeds (0.994 and 0.919) and failed on the third. The focal loss with the prior reached 0.968 to 0.980 on all three seeds, and the same loss without the prior failed on all three, once with a non-finite loss. In this small setting, neither the loss nor the initialization was enough alone, and the combination the paper uses was the only one that trained on every seed. That is more extreme than the paper’s result, where cross-entropy with the prior reached 30.2 AP; in our reading, a deep pretrained backbone and 90,000 iterations give cross-entropy more room to recover than this 300-step run does. The number of steps, the learning rate and the network were fixed before the run and not tuned.

Under the mean over all locations, nothing learned in 300 steps (AP at most 0.008): dividing by 16,384 locations per image instead of 18 positives makes the positives’ gradient about 900 times smaller at the same learning rate. Training does not diverge, but it does not progress within the budget either, which is why the normalization and the learning rate have to be chosen together. The last column is the share of the final training loss carried by easy negatives ($p_t > 0.9$): it is lower under the focal loss than under cross-entropy in both normalizations (6.8% against 12.8%, and 0.1% against 58.4%); in our reading the first contrast is muted because the cross-entropy network did not learn. The columns other than test AP are printed for seed 0 only. The column “max loss 0-19” equals the step-0 loss in every row, since the losses fell from the first step.

Inference

An image is passed once through the network. To save time, boxes are decoded only for at most the 1,000 top-scoring predictions per pyramid level, after discarding scores below 0.05; the predictions of all levels are merged and non-maximum suppression at IoU 0.5 gives the final detections.

Speed depends on the backbone and on the image scale (the shorter side). The authors report that RetinaNet-101 at 600 pixels took 122 ms per image against 172 ms for Faster R-CNN with ResNet-101-FPN at similar accuracy, both measured on an Nvidia M40 GPU; in their timing table RetinaNet-101 at 800 pixels takes 198 ms, and the introduction gives the best model’s speed as 5 frames per second.

Performance

The ablations (Table 1 of the ICCV paper) use COCO trainval35k for training and minival for testing, a 600-pixel train and test scale, ResNet-50-FPN unless noted, and the 90,000-iteration schedule.

Setting (COCO minival, 600 px)	AP	AP$_{50}$	AP$_{75}$
CE with prior, $\alpha = 0.5$ (unweighted CE scaled by 0.5)	30.2	46.7	32.8
α-balanced CE, best $\alpha = 0.75$	31.1	49.4	33.0
FL, $\gamma = 0.5$, $\alpha = 0.5$	32.9	51.7	35.2
FL, $\gamma = 1$, $\alpha = 0.25$	33.7	52.0	36.2
FL, $\gamma = 2$, $\alpha = 0.25$	34.0	52.5	36.5
FL, $\gamma = 5$, $\alpha = 0.25$	32.2	49.6	34.8
One square anchor per position (FL)	30.3	49.0	31.8
2 scales × 3 aspect ratios (FL)	34.2	53.1	36.5
ResNet-101-FPN, best OHEM (batch 128, NMS 0.5)	32.8	50.3	35.1
ResNet-101-FPN, best OHEM 1:3 (batch 128, NMS 0.5)	31.1	47.2	33.2
ResNet-101-FPN, FL	36.0	54.9	38.7

Three findings stand out. First, the loss accounts for 2.9 AP over the best α-balanced cross-entropy with the same network. Second, at the CE end of Table 1a, $\alpha = 0.1$ gave 0.0 AP and $\alpha = 0.25$ 10.8 AP: weighting the background too heavily under cross-entropy is ruinous, while the same $\alpha = 0.25$ is the best choice once $\gamma = 2$. Third, OHEM in its original form and with SSD’s 1:3 ratio fell more than 3 AP short of the focal loss with ResNet-101; the authors note that other OHEM settings and variants did no better. The hard negative mining article compares mining with reweighting more broadly. Anchor density mattered too: three scales and three aspect ratios beat one square anchor by nearly 4 AP, and beyond six to nine anchors there was no further gain.

The speed–accuracy table (Table 1e) is on test-dev, with the same 90,000-iteration schedule:

Backbone	Scale	AP (test-dev)	AP$_S$	AP$_M$	AP$_L$	Time (ms)
ResNet-50-FPN	400	30.5	11.2	33.8	46.1	64
ResNet-50-FPN	600	34.3	16.2	37.4	47.4	98
ResNet-50-FPN	800	35.7	18.9	38.9	46.3	153
ResNet-101-FPN	400	31.9	11.6	35.8	48.5	81
ResNet-101-FPN	600	36.0	17.4	39.6	49.7	122
ResNet-101-FPN	800	37.8	20.2	41.1	49.2	198

For the comparison with the state of the art (Table 2, COCO test-dev, single model), RetinaNet-101-800 was trained with scale jitter and for 1.5 times longer than in Table 1e, which the authors credit with 1.3 AP. It reached 39.1 AP (59.1 AP$_{50}$, 21.8 AP$_S$), against 33.2 for DSSD513, the best earlier one-stage model, 36.2 for Faster R-CNN with ResNet-101-FPN and 36.8 for Faster R-CNN with an Inception-ResNet-v2-TDM backbone, the best two-stage model listed. The baseline rows cite each method’s own paper, so their training set-ups are not RetinaNet’s. The arXiv v2 version adds a ResNeXt-101-FPN (32×8d) backbone at 40.8 AP on test-dev, and notes that after publication a variant of Faster R-CNN in Detectron gave faster and more accurate results.

Strengths

• Simplicity. One fully convolutional network, no proposal stage, no per-region computation, no sampling of anchors; the head is four convolutions per subnet.
• A loss that transfers. The focal loss addresses a property of dense prediction rather than of RetinaNet, and later dense detectors kept it as their classification loss (Variants).
• Robust settings. The ablations show a broad optimum for $\gamma$ (with $\alpha$ chosen alongside), the prior and the anchor set, so the defaults $\gamma = 2$, $\alpha = 0.25$ and $\pi = 0.01$ needed little tuning in the paper’s experiments.
• Speed–accuracy range. By changing backbone and image scale, the same design spans 64 to 198 ms per image in the paper’s timing table.

Limitations

• Hyperparameters that interact. $\alpha$ and $\gamma$ must be chosen together (Table 1a against 1b), and the anchor set and the IoU thresholds of 0.5 and 0.4 are further choices; FCOS (Tian et al., 2019) cites RetinaNet’s own anchor ablation, where changing the anchors’ sizes, aspect ratios and number moved AP by up to about 4 points, as a drawback of anchor-based detection.
• Binary labels. The focal loss assumes 0/1 targets, and the classification score carries no information on how well the box is localized; later losses replaced the target with a localization quality (Variants).
• Small objects. AP$_S$ stays far below AP$_L$ (20.2 against 49.2 for RetinaNet-101-800 on test-dev), even with $P_3$ at stride 8.
• Speed at the low end. The authors note that the high-frame-rate regime would need a special network design, as in YOLO, rather than an off-the-shelf backbone; at 400 pixels RetinaNet-50 still took 64 ms.
• Dense outputs and NMS. Nine anchors per position and per-class sigmoids produce many overlapping candidates, so inference depends on score thresholds, top-$k$ selection and NMS.

Historical Importance

Lin et al. located the accuracy gap between one-stage and two-stage detectors in training rather than in network design. By showing that a plain dense detector with a different loss and initialization reached the top of the COCO table, they changed what one-stage design work focused on: in our reading, after 2017 dense detectors dropped sampling and hard negative mining for the focal loss on all anchors, and research moved on to which locations count as positives (FCOS, ATSS) and what the classification target should be (GFL). Its initialization also carried over: FCOS and ATSS, for instance, state that they initialize their newly added layers as in RetinaNet.

RetinaNet appeared in 2017, the same year as FPN and Mask R-CNN from overlapping authors at Facebook AI Research, and its code was released in the Detectron framework. The journal version appeared in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2020 (volume 42, issue 2); it was not read for this article, and the details above come from the ICCV paper and the arXiv v2 version.

Variants

RetinaNet as a baseline. Later papers report retrained RetinaNet baselines, whose numbers rose with longer schedules, larger inputs and better implementations; only numbers from the same split, backbone and schedule can be compared. FCOS was presented as an anchor-free counterpart of RetinaNet trained with its hyperparameters: on COCO minival with ResNet-50-FPN and RetinaNet’s own settings, RetinaNet reached 35.9 AP and FCOS 36.3. ATSS (Zhang et al., 2020) used RetinaNet with one square anchor per position to show that the difference between anchor-based and anchor-free detectors lies mainly in how positives and negatives are selected, and its own reimplementation of the nine-anchor RetinaNet (ResNet-50, 800 pixels, 90,000 iterations) gave 36.3 AP on minival; the label assignment and anchor-free detection articles cover that analysis. NAS-FPN (Ghiasi et al., 2019) searched for a new feature pyramid inside the RetinaNet framework and reported 48.3 AP on test-dev with an AmoebaNet backbone, 1280-pixel inputs and DropBlock. EfficientDet (Tan et al., 2020) started its ablations from “the standard RetinaNet” with ResNet-50 and FPN, kept the focal loss with $\alpha = 0.25$ and $\gamma = 1.5$, and listed RetinaNet-R50 at 37.0 AP on test-dev at 640 pixels.

Successors of the focal loss. FCOS uses RetinaNet’s head layout and focal loss with its own assignment and a centre-ness branch. CornerNet (Law and Deng, 2018) and CenterNet in the “Objects as Points” form of Zhou et al. (2019) train their keypoint heatmaps with a penalty-reduced variant of the focal loss. Generalized Focal Loss (Li et al., 2020) replaces the binary target with an IoU-based quality $y \in [0, 1]$ and generalizes the modulating factor to $|y - \sigma|^{\beta}$, giving the quality focal loss

$$\mathrm{QFL}(\sigma) = -\,|y - \sigma|^{\beta}\,\bigl((1 - y)\log(1 - \sigma) + y \log \sigma\bigr),$$

with $\beta = 2$ best in their experiments; with $y \in \{0, 1\}$ and $\beta = \gamma$ it reduces to the focal loss without $\alpha$, a special case the GFL paper itself lists.

Implementations

The authors’ code is in Detectron, built on Caffe2; the repository was archived and read-only as of November 2026, under the Apache 2.0 licence. Its RetinaNet configuration (retinanet_R-50-FPN_1x.yaml, at commit 04155a0) trains at 800 pixels (maximum side 1,333) on coco_2014_train plus coco_2014_valminusminival, equivalent to COCO train2017, with a base learning rate of 0.01 for 90,000 iterations (the model zoo’s general description of the 1x schedule gives 0.02 for 16 images per minibatch, with a 500-iteration warm-up), $\gamma = 2$, $\alpha = 0.25$, three scales per octave and three aspect ratios. Defaults in its configuration file match the paper’s prior ($\pi = 0.01$), IoU thresholds (0.5 and 0.4), inference threshold (0.05) and 1,000 candidates per level, and set the smooth $L_1$ transition point to 0.11, a value the paper does not give. The code also contains a softmax option with a background class, whose bias is initialized to $\log\bigl(K(1 - \pi)/\pi\bigr)$ for $K$ foreground classes (the other biases are 0), so that the foreground classes together start at probability $\pi$ (our arithmetic). Its model zoo lists, on coco_2014_minival (equivalent to val2017), 35.7 AP for ResNet-50-FPN with the 1x schedule, 37.8 for ResNet-101-FPN with 2x, and 39.8 for ResNeXt-101-64x4d-FPN with 1x.

Detectron2, the PyTorch successor, lists RetinaNet on COCO val2017 (trained on train2017) at 37.4 AP for ResNet-50 with the 1x schedule, 38.7 with 3x (about 37 epochs), and 40.4 for ResNet-101 with 3x (as of November 2026). TorchVision 0.29 (as of November 2026) provides retinanet_resnet50_fpn, with COCO weights “produced by following a similar training recipe” to the paper’s and 36.4 box AP on val2017, and retinanet_resnet50_fpn_v2, an improved model that cites the ATSS paper and lists 41.5 AP; the detection module is marked beta.