Anchor Boxes

An anchor box is a reference box of fixed size and aspect ratio placed at a position of a convolutional feature map. A detector tiles a few of them at every position, predicts for each whether it covers an object and how to shift and stretch it to fit, and trains each prediction against the object its anchor overlaps. Introduced with the region proposal network of Faster R-CNN (Ren et al., 2015), anchors became the standard way for dense object detectors to cover many object sizes and shapes, until anchor-free and set-prediction detectors showed alternatives.

Definition

An anchor is a box $a = (x_a, y_a, w_a, h_a)$, with center $(x_a, y_a)$, width $w_a$ and height $h_a$, attached to one output position of a network. Its center is fixed by the position and the network’s stride, its size and shape before training. With $k$ anchors per position, the detector outputs at each position $k$ scores and $k$ four-number box offsets. During training, every anchor is labeled positive, negative or ignored by its intersection over union (IoU) with the ground-truth bounding boxes, and positive anchors receive regression targets that encode their object’s box relative to the anchor.

Intuition

A convolutional head sees the same window of features at every position; anchors give it $k$ output slots there, and slot $i$ is responsible only for objects whose box resembles anchor $i$: a tall slot learns pedestrians, a wide slot cars, a large slot objects that fill much of the image, each predicting small corrections to a box that is already roughly right. The anchors also decide which outputs are trained on which object.

Formal Definition

Anchor set. For a feature map with stride $s$, a set of scales $\{\sigma_m\}$ (square-root areas, in pixels) and a set of aspect ratios $\{r_n\}$ with $r = h/w$, the anchors at feature position $(i, j)$ are

$$x_a = (j + \tfrac{1}{2})\,s, \quad y_a = (i + \tfrac{1}{2})\,s, \qquad w_a = \frac{\sigma_m}{\sqrt{r_n}}, \quad h_a = \sigma_m \sqrt{r_n},$$

so each anchor has area $\sigma_m^2$, and an $H' \times W'$ map carries $H' W' k$ anchors, $k$ being the number of scale–ratio pairs (implementations differ by half a pixel in the centers).

Faster R-CNN used 3 scales with areas $128^2$, $256^2$ and $512^2$ pixels and 3 aspect ratios, 1:1, 1:2 and 2:1, giving $k = 9$, on images resized so that the shorter side is 600 pixels, where the last convolutional layer of VGG-16 (and of the smaller ZF network) has a total stride of 16 (Ren et al., 2015). The py-faster-rcnn code (Girshick and Bell, 2015), which reproduces the official MATLAB anchors, also takes the ratio as height over width but rounds shapes to integers ($184 \times 96$ rather than $181 \times 91$).

Label assignment. The region proposal network (RPN) labels an anchor positive if (i) it has the highest IoU with some ground-truth box, or (ii) its IoU with any ground-truth box is above 0.7; negative if it is not positive and its IoU is below 0.3 for every ground-truth box; and the rest contribute nothing to the loss (Ren et al., 2015). Condition (i) is there, the journal version explains, because in rare cases (ii) finds no positive sample. Anchors crossing the image border were also ignored in training, leaving about 6,000 of roughly 20,000 in a $1000 \times 600$ image.

Box parameterization. A positive anchor is regressed toward its ground-truth box $(x^*, y^*, w^*, h^*)$ through the targets

$$t^*_x = \frac{x^* - x_a}{w_a}, \qquad t^*_y = \frac{y^* - y_a}{h_a}, \qquad t^*_w = \log\frac{w^*}{w_a}, \qquad t^*_h = \log\frac{h^*}{h_a},$$

and a prediction $(t_x, t_y, t_w, t_h)$ is decoded as $x = x_a + t_x w_a$, $y = y_a + t_y h_a$, $w = w_a e^{t_w}$, $h = h_a e^{t_h}$. The parameterization comes from R-CNN’s bounding-box regression (Girshick et al., 2014), where the reference was a region proposal. Normalizing by the anchor size makes the targets scale-independent, and the logarithm keeps decoded sizes positive. The RPN loss adds a two-class log loss over sampled anchors to a smooth $L_1$ loss on $t - t^*$ for positives only, and the $k$ anchor shapes have separate regressors that share no weights.

Properties

Translation invariance. The same anchors and convolutional weights are used at every position, so a shifted object produces the same predictions, shifted. Ren et al. (2015, 2017) contrast MultiBox, whose 800 k-means anchors are not translation invariant and need a $(4 + 1) \times 800$ output layer, with their convolutional $(4 + 2) \times 9$. The invariance holds only up to the total stride, the journal version notes; a shift by a fraction of the stride changes which anchors match.

Multiple scales from one feature map. Instead of an image pyramid or filters of several sizes, the journal version describes a “pyramid of anchors”: one image scale, one feature map and one filter size, with the scale range carried by the reference boxes. The head’s receptive field is 228 pixels for VGG-16, yet it can predict larger boxes, the authors note, inferring an object’s extent from its visible middle.

Imbalance. Nearly all anchors are negatives. The RPN samples 256 anchors per image, up to half of them positive; SSD keeps the hardest negatives, at most three per positive; RetinaNet keeps all of them and down-weights easy ones with the focal loss (see loss function).

Sensitivity to design. Scales, ratios, their number and the IoU thresholds are all hyperparameters. In the journal version of Faster R-CNN (Ren et al., 2017), one anchor per position cost 3–4 points of PASCAL VOC 2007 mAP against the default nine, while 3 scales with one ratio matched 3 scales with 3 ratios (69.8% against 69.9%). Small objects are the hardest case: it follows from the IoU rule that one lying between anchor centers may have no anchor above 0.7, as in the example below.

Examples

The NumPy code prints Faster R-CNN’s nine anchor shapes, then reads three ground-truth boxes (a tall ellipse, a wide rectangle, a small square) from a $256 \times 256$ label image. For this small image the scales are reduced to 32, 64 and 128 pixels at stride 16, a $16 \times 16 \times 9$ grid. It assigns labels with the RPN rule, ignoring cross-border anchors, encodes and decodes the targets, and shifts every object by one stride.

import numpy as np

def make_anchors(scales, ratios):
    """Anchors centered at (0, 0) as (x1, y1, x2, y2); area s^2, ratio r = h / w."""
    out = []
    for r in ratios:
        for s in scales:
            w, h = s / np.sqrt(r), s * np.sqrt(r)
            out.append([-w / 2, -h / 2, w / 2, h / 2])
    return np.array(out)

np.set_printoptions(suppress=True)
base = make_anchors([128, 256, 512], [0.5, 1, 2])   # Faster R-CNN's 9 anchors (VGG-16, s = 600)
wh = base[:, 2:] - base[:, :2]
print("Faster R-CNN anchors (w x h):", [f"{w:.0f}x{h:.0f}" for w, h in wh])

# A synthetic 256 x 256 label image with three objects; ground-truth boxes come from it.
H = W = 256
yy, xx = np.mgrid[0:H, 0:W] + 0.5
img = np.zeros((H, W), int)
shapes = [((xx - 70) / 18) ** 2 + ((yy - 120) / 50) ** 2 <= 1,          # tall, person-like
         (np.abs(xx - 175) <= 55) & (np.abs(yy - 200) <= 22),            # wide, car-like
         (np.abs(xx - 190) <= 14) & (np.abs(yy - 60) <= 14)]             # small square
for i, m in enumerate(shapes):
    img[m] = i + 1
def box_of(m):
    r, c = np.nonzero(m)
    return np.array([c.min(), r.min(), c.max() + 1, r.max() + 1], float)
gt = np.stack([box_of(img == i + 1) for i in range(len(shapes))])
print("ground truth (x1, y1, x2, y2):\n", gt)

# Anchor grid: stride 16 gives a 16 x 16 feature map; anchors scaled down for the small image.
stride = 16
cell = make_anchors([32, 64, 128], [0.5, 1, 2])
cx = (np.arange(W // stride) + 0.5) * stride
CX, CY = np.meshgrid(cx, cx)
shifts = np.stack([CX.ravel(), CY.ravel(), CX.ravel(), CY.ravel()], 1)
anchors = (shifts[:, None, :] + cell[None]).reshape(-1, 4)              # (16*16*9, 4)
print("anchors:", len(anchors))

def iou_matrix(A, B):
    lt = np.maximum(A[:, None, :2], B[None, :, :2])
    rb = np.minimum(A[:, None, 2:], B[None, :, 2:])
    inter = np.prod(np.clip(rb - lt, 0, None), axis=2)
    area = lambda X: np.prod(X[:, 2:] - X[:, :2], axis=1)
    return inter / (area(A)[:, None] + area(B)[None] - inter)

def assign(anchors, gt, pos_thr=0.7, neg_thr=0.3):
    """RPN rule: 1 positive, 0 negative, -1 ignored; cross-boundary anchors ignored."""
    inside = (anchors[:, 0] >= 0) & (anchors[:, 1] >= 0) & (anchors[:, 2] <= W) & (anchors[:, 3] <= H)
    iou = iou_matrix(anchors, gt)
    iou[~inside] = -1
    best_gt, best = iou.argmax(1), iou.max(1)
    labels = np.full(len(anchors), -1)
    labels[inside & (best < neg_thr)] = 0
    labels[best > pos_thr] = 1
    for j in range(len(gt)):                       # (i) highest-IoU anchor(s) for each object
        top = iou[:, j] == iou[:, j].max()
        labels[top], best_gt[top] = 1, j
    return labels, best_gt, iou, inside

labels, best_gt, iou, inside = assign(anchors, gt)
print(f"positive {(labels == 1).sum()}, "
      f"negative {(labels == 0).sum()}, ignored {(labels == -1).sum()} "
      f"({(~inside).sum()} cross the border, {((labels == -1) & inside).sum()} have IoU in [0.3, 0.7])")
for j in range(len(gt)):
    p = (labels == 1) & (best_gt == j)
    print(f"object {j}: best anchor IoU {iou[:, j].max():.2f}, anchors above 0.7: "
          f"{(iou[:, j] > 0.7).sum()}, positives {p.sum()}")

def to_cxcywh(b):
    return np.stack([(b[:, 0] + b[:, 2]) / 2, (b[:, 1] + b[:, 3]) / 2,
                     b[:, 2] - b[:, 0], b[:, 3] - b[:, 1]], 1)

def encode(gt_boxes, anc):
    g, a = to_cxcywh(gt_boxes), to_cxcywh(anc)
    return np.stack([(g[:, 0] - a[:, 0]) / a[:, 2], (g[:, 1] - a[:, 1]) / a[:, 3],
                     np.log(g[:, 2] / a[:, 2]), np.log(g[:, 3] / a[:, 3])], 1)

def decode(t, anc):
    a = to_cxcywh(anc)
    cx, cy = a[:, 0] + t[:, 0] * a[:, 2], a[:, 1] + t[:, 1] * a[:, 3]
    w, h = a[:, 2] * np.exp(t[:, 2]), a[:, 3] * np.exp(t[:, 3])
    return np.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2], 1)

pos = np.flatnonzero(labels == 1)
t = encode(gt[best_gt[pos]], anchors[pos])
print("positive anchors and their targets (t_x, t_y, t_w, t_h):")
for k, tk in zip(pos, t):
    print(f"  object {best_gt[k]}  anchor {np.round(anchors[k]).astype(int)}  "
          f"IoU {iou[k, best_gt[k]]:.2f}  t = {np.round(tk, 3)}")
print("max |decode(t) - gt| over all positives:", np.abs(decode(t, anchors[pos]) - gt[best_gt[pos]]).max())

# Translation: move every object by one stride; the positives move by one cell, targets unchanged.
labels2, best_gt2, _, _ = assign(anchors, gt + [stride, 0, stride, 0])
pos2 = np.flatnonzero(labels2 == 1)
t2 = encode(gt[best_gt2[pos2]] + [stride, 0, stride, 0], anchors[pos2])
print("shifted by 16 px: same positive count", len(pos2) == len(pos),
      "| indices shifted by 9 anchors", np.array_equal(pos2, pos + 9),
      "| same targets", np.allclose(t2, t))

Output:

Faster R-CNN anchors (w x h): ['181x91', '362x181', '724x362', '128x128', '256x256', '512x512', '91x181', '181x362', '362x724']
ground truth (x1, y1, x2, y2):
 [[ 52.  70.  88. 170.]
 [120. 178. 230. 222.]
 [176.  46. 204.  74.]]
anchors: 2304
positive 4, negative 1098, ignored 1202 (1148 cross the border, 54 have IoU in [0.3, 0.7])
object 0: best anchor IoU 0.73, anchors above 0.7: 1, positives 1
object 1: best anchor IoU 0.80, anchors above 0.7: 2, positives 2
object 2: best anchor IoU 0.53, anchors above 0.7: 0, positives 1
positive anchors and their targets (t_x, t_y, t_w, t_h):
  object 2  anchor [168  40 200  72]  IoU 0.53  t = [ 0.188  0.125 -0.134 -0.134]
  object 0  anchor [ 49  75  95 165]  IoU 0.73  t = [-0.044  0.    -0.229  0.1  ]
  object 1  anchor [123 177 213 223]  IoU 0.80  t = [ 0.077  0.     0.195 -0.028]
  object 1  anchor [139 177 229 223]  IoU 0.80  t = [-0.099  0.     0.195 -0.028]
max |decode(t) - gt| over all positives: 0.0
shifted by 16 px: same positive count True | indices shifted by 9 anchors True | same targets True

Only 4 of 2,304 anchors are positive, and most ignored ones cross the border. The $28 \times 28$ square sits between anchor centers and reaches only 0.53, so only rule (i) makes it a positive. Decoding recovers every box exactly, and a one-stride shift moves each positive one cell over with identical targets.

YOLOv2 (Redmon and Farhadi, 2017) chose anchor shapes by clustering the training boxes. Because Euclidean k-means on widths and heights lets large boxes dominate, they used $d = 1 - \mathrm{IoU}$ between a box and a centroid; since only widths and heights are clustered, implementations compute this IoU with the two boxes placed at a common center. The second block clusters 3,000 synthetic box shapes of tall, wide and square objects and reports the mean IoU of each box with its best anchor.

import numpy as np

def wh_iou(wh, C):
    """IoU of box shapes aligned at a common center: (N, 2) against (K, 2)."""
    inter = np.minimum(wh[:, None, 0], C[None, :, 0]) * np.minimum(wh[:, None, 1], C[None, :, 1])
    return inter / (wh[:, None].prod(2) + C[None].prod(2) - inter)

def kmeans_iou(wh, k, rng, iters=100):
    C = wh[rng.choice(len(wh), k, replace=False)]
    for _ in range(iters):
        a = np.argmax(wh_iou(wh, C), 1)              # nearest centroid under d = 1 - IoU
        newC = np.array([wh[a == j].mean(0) if np.any(a == j) else C[j] for j in range(k)])
        if np.allclose(newC, C):
            break
        C = newC
    return C[np.argsort(C.prod(1))]

# Box shapes (w, h) of objects annotated in 256 x 256 images: three kinds of objects.
rng = np.random.default_rng(0)
n = 3000
kind = rng.choice(3, n, p=[0.5, 0.3, 0.2])
size = np.exp(rng.normal(np.log([60, 70, 30])[kind], 0.35))       # sqrt(area) in pixels
ratio = np.exp(rng.normal(np.log([2.5, 0.45, 1.0])[kind], 0.15))  # h / w: people, cars, balls
wh = np.stack([size / np.sqrt(ratio), size * np.sqrt(ratio)], 1)

hand = np.array([[s / np.sqrt(r), s * np.sqrt(r)] for r in (0.5, 1, 2) for s in (32, 64, 128)])
print(f"hand-picked 9 anchors: mean best IoU {wh_iou(wh, hand).max(1).mean():.3f}")
for k in (5, 9):
    C = kmeans_iou(wh, k, np.random.default_rng(1))
    print(f"k-means, k = {k}: mean best IoU {wh_iou(wh, C).max(1).mean():.3f}")
    print("   centroids (w x h):", " ".join(f"{w:.0f}x{h:.0f}" for w, h in C))

# Euclidean k-means on (w, h) for comparison, and the spread over initializations.
def kmeans_l2(wh, k, rng, iters=100):
    C = wh[rng.choice(len(wh), k, replace=False)]
    for _ in range(iters):
        a = np.argmin(((wh[:, None] - C[None]) ** 2).sum(2), 1)
        C = np.array([wh[a == j].mean(0) if np.any(a == j) else C[j] for j in range(k)])
    return C
r_iou = [wh_iou(wh, kmeans_iou(wh, 5, np.random.default_rng(s))).max(1).mean() for s in range(10)]
r_l2 = [wh_iou(wh, kmeans_l2(wh, 5, np.random.default_rng(s))).max(1).mean() for s in range(10)]
print(f"k = 5 over 10 initializations: 1 - IoU distance {min(r_iou):.3f}-{max(r_iou):.3f}, "
      f"Euclidean {min(r_l2):.3f}-{max(r_l2):.3f}")

Output:

hand-picked 9 anchors: mean best IoU 0.680
k-means, k = 5: mean best IoU 0.702
   centroids (w x h): 29x31 34x82 86x42 56x141 154x70
k-means, k = 9: mean best IoU 0.783
   centroids (w x h): 23x24 42x36 26x65 37x91 85x40 50x124 126x57 71x181 190x86
k = 5 over 10 initializations: 1 - IoU distance 0.701-0.702, Euclidean 0.684-0.685

Five clustered anchors beat the nine hand-picked ones (0.702 against 0.680), nine clusters reach 0.783, and the centroids follow the shapes in the data. Redmon and Farhadi reported the same pattern on PASCAL VOC 2007: average IoU 61.0 with 5 clusters, 60.9 for Faster R-CNN’s 9 anchors, 67.2 with 9 clusters, and 58.7 for 5 Euclidean clusters. With other data seeds, an occasional initialization stopped in a poorer local minimum (about 0.66), so restarts are worthwhile.

Common Misconceptions

• “Anchors are proposals.” An anchor is a fixed reference; the proposal is the anchor after the predicted offsets are applied.
• “More anchors are always better.” RetinaNet saw no gain beyond 6–9 anchors per position, and under ATSS one anchor did as well as nine.
• “Anchor-free detectors have no assignment hyperparameters.” They replace IoU thresholds with other rules, such as FCOS’s per-level ranges of object size.

Where It Is Used

Region proposal networks. Faster R-CNN’s RPN slides a $3 \times 3$ window over the shared feature map, scoring and refining 9 anchors per position; the two-stage object detectors and Faster R-CNN articles cover the system.

SSD default boxes. SSD (Liu et al., 2016) places its default boxes on several feature maps of decreasing resolution, with scales spaced regularly from 0.2 of the image size on the lowest map to 0.9 on the highest (SSD300 uses 0.1 on its first map), and 6 boxes per position (4 on some layers): aspect ratios 1, 2, 3, 1/2 and 1/3 plus an extra square of intermediate scale. Each ground-truth box is matched to its best default box and to every one with IoU above 0.5; SSD300 has 8,732 in all.

YOLOv2. Redmon and Farhadi replaced the original YOLO’s fully connected output with anchors on a $13 \times 13$ map, choosing $k = 5$ clustered shapes as a trade-off between recall and model complexity. Finding the RPN’s unconstrained offsets unstable early in training, they predict the center as the cell’s corner plus a sigmoid of $t_x$, $t_y$, so it stays in the cell, and the size as $p_w e^{t_w}$, $p_h e^{t_h}$ for a prior $p_w \times p_h$.

Anchors in feature pyramids. In a feature pyramid each level handles one range of sizes, so one anchor scale per level suffices. The feature pyramid network (Lin et al., 2017, FPN) uses areas $32^2$ to $512^2$ on levels $P_2$ to $P_6$, with 3 aspect ratios each and the RPN’s IoU rule. RetinaNet (Lin et al., 2017, focal loss) uses three sub-octave scales, $2^0$, $2^{1/3}$, $2^{2/3}$, times 3 ratios, 9 anchors per level on $P_3$ to $P_7$ covering 32 to 813 pixels, and labels anchors with IoU of at least 0.5 as foreground, below 0.4 as background, and ignores the rest. In its COCO ablations, one square anchor per position reached 30.3 AP and 9 anchors 34.0.

Anchor-free alternatives. Tian et al. (2019) list the drawbacks of anchors: sensitivity to their sizes, ratios and number, re-design for new datasets, and over 180,000 anchors, mostly negatives, in an FPN for an image with an 800-pixel shorter side. Their FCOS instead regresses, from each feature-map location inside an object, the distances to the four sides of its box. CornerNet (Law and Deng, 2018) detects a box as a pair of keypoints, its top-left and bottom-right corners, with a corner pooling layer. CenterNet (Zhou et al., 2019) predicts a heatmap of object centers and regresses the size at each peak, with one positive per object chosen by location rather than box overlap, which removes IoU-based non-maximum suppression.

Zhang et al. (2020) aligned the implementation details of FCOS and a one-anchor RetinaNet and found, on COCO minival, that the essential difference is how positive and negative samples are defined: with the same definition, regressing from a box or a point made no clear difference (37.0 against 36.9 AP with IoU-based selection, 37.8 against 37.8 with FCOS’s rule). Their adaptive training sample selection (ATSS) takes, per object, the 9 anchors closest to its center on each pyramid level and keeps those whose IoU reaches the mean plus the standard deviation of the candidates’ IoUs and whose centers lie inside the object. With it, nine anchors per position performed about the same as one.

Set prediction. DETR (Carion et al., 2020) removes anchors and hand-designed assignment. A transformer decoder turns a fixed number of learned object queries into predictions, and the Hungarian algorithm matches them one-to-one to the ground truth during training. In the paper’s arXiv version, the authors observed each query slot specializing in certain areas and box sizes, in our reading a learned counterpart of hand-set anchors.