Anchor-Free Object Detection

An anchor-free object detector predicts boxes without a set of preset reference boxes. Instead of tiling anchor boxes of several sizes and aspect ratios at every position of a feature map and regressing each one toward an object, it describes an object by a few keypoints (its corners or its centre), by the points of the feature map that fall inside it, or by an element of a predicted set that is matched to it during training. CornerNet (Law and Deng, 2018), CenterNet in the form of “Objects as Points” (Zhou et al., 2019), FCOS (Tian et al., 2019) and FoveaBox (Kong et al., 2020) made the design competitive on COCO, and DETR (Carion et al., 2020) removed anchors along with the hand-written assignment rules. Removing the anchors does not remove the question of which outputs learn which object: that question, label assignment, comes back in a new form, and Zhang et al. (2020) found that, once implementation details were aligned, it, rather than regressing from a box or from a point, was the essential difference between a centre-based anchor-free detector and an anchor-based one.

Definition

A dense object detector attaches every prediction to a reference: a place in the image, and in anchor-based designs also a box shape, from which the prediction is decoded and against which it is trained. A detector is anchor-free when its references carry no preset box shape. The reference is then

• a keypoint to be found in a heatmap: a box corner (CornerNet), the box centre (Objects as Points), or several points;
• a location of the feature map, which Zhang et al. (2020) call an anchor point, from which the network regresses the distances to the box sides (DenseBox, FCOS, FoveaBox);
• a learned query, one element of a fixed-size set of predictions, which is matched to an object only during training (DETR).

Zhang et al. (2020) group the first two kinds, the ones that are still dense, as keypoint-based and centre-based anchor-free detectors; this article treats set-based detectors such as DETR as a third group. The term “anchor-free” refers to the box prior only. Every one of these detectors still has to decide which outputs are positives for which object, and most still regress a box.

Why Anchors Were Dropped

The authors of the first competitive anchor-free detectors gave overlapping reasons. Law and Deng (2018) named two drawbacks: a detector needs a very large set of anchors (they cite more than 40,000 in DSSD and more than 100,000 in RetinaNet) so that most objects overlap some anchor well, which leaves a tiny fraction of positives and a heavy imbalance with negatives; and anchors add hyperparameters (how many, what sizes, what aspect ratios) chosen largely by ad-hoc heuristics, more so when each pyramid level has its own set. Tian et al. (2019) gave four: performance is sensitive to the anchors’ sizes, aspect ratios and number, by up to 4% AP on COCO for RetinaNet; fixed shapes struggle with objects whose shapes vary widely, particularly small ones, and must be redesigned for each new dataset; dense tiling produces more than 180,000 anchors in a feature pyramid network (FPN) for an image with an 800-pixel shorter side, most of them negatives; and assigning anchors requires computing the IoU of every anchor with every object. Tian et al. also argued from design: fully convolutional networks had succeeded at per-pixel prediction tasks such as semantic segmentation, depth estimation and keypoint detection, and detection deviated from that pattern mainly because of anchors.

Zhou et al. (2019) framed the change as one of assignment rather than shape: a centre point can be seen as a single shape-agnostic anchor, assigned by location alone and not by box overlap, with one positive per object, so that no IoU thresholds and no non-maximum suppression (NMS) are needed. Carion et al. (2020) went further and described anchors, proposals and window centres alike as surrogate problems whose results depend on hand-designed anchor sets, assignment heuristics and duplicate removal; their set-prediction detector drops anchors and duplicate removal and replaces the assignment heuristics with a one-to-one matching between predictions and objects.

Formal Definition

Let a dense detector predict, at location $\mathbf{p} = (x, y)$ of a feature map of stride $s$, class scores and a box. With an anchor of centre $\mathbf{p}$ and size $(w_a, h_a)$, the box is decoded from offsets relative to the anchor, for example as the centre $(x + \Delta x\, w_a,\; y + \Delta y\, h_a)$ and size $(w_a e^{\Delta w},\; h_a e^{\Delta h})$, as described in bounding box regression. A point-based detector regresses from the location itself: FCOS predicts the four distances

$$(l, t, r, b) = (x - x_0,\; y - y_0,\; x_1 - x,\; y_1 - y)$$

from $\mathbf{p}$ to the sides of the box $(x_0, y_0, x_1, y_1)$, defined only where $\mathbf{p}$ lies inside the box. A keypoint-based detector predicts a heatmap $\hat Y \in [0,1]^{W/R \times H/R \times C}$ per keypoint type at output stride $R$, trained toward Gaussian bumps at the keypoints as in heatmap regression, and reads each object off the peaks: Objects as Points regresses the box size $(x_1 - x_0, y_1 - y_0)$ and a sub-cell offset at the centre peak, CornerNet pairs a top-left peak with a bottom-right peak.

The training targets follow from a positive set $\mathcal{P}_g$ of locations for each object $g$. The rules differ in how they choose it:

Detector	Positive locations for an object	Level or scale rule	Contested location goes to
YOLO (2016)	the grid cell containing the centre	one $7 \times 7$ grid	two box predictors and one class per cell
CornerNet (2018)	one location per corner, negatives near it down-weighted	one map, stride 4	separate heatmap per class
Objects as Points (2019)	one location at the centre	one map, stride 4	separate heatmap per class; one shared size head
FCOS (2019)	every location inside the box	$\max(l, t, r, b)$ in a range per level	the box of smallest area
FoveaBox (code)	a central region of the box	$\sqrt{wh}$ in overlapping ranges per level	the smaller box (written last)
DETR (2020)	one query, by Hungarian matching	none	each query matched to at most one object

Two degrees of freedom stand out. The spatial rule says which locations of a level may represent an object, from the whole box down to a single cell. The scale rule says which level, if there are several. An anchor-based detector folds both into the IoU between anchor and object, Zhang et al. (2020) describe RetinaNet’s IoU as selecting positives in the spatial and scale dimensions simultaneously, and FCOS as applying a spatial constraint first and a scale constraint second.

Approaches

Keypoint-based

CornerNet (Law and Deng, 2018) detects a box as a pair of keypoints, its top-left and bottom-right corners. A stacked hourglass network predicts, for each class, a heatmap of top-left corners and one of bottom-right corners, together with an embedding for each corner and an offset that recovers the precision lost by downsampling. Following the associative embedding method of Newell et al. for multi-person pose, the network is trained to give the two corners of one object similar embeddings, and corners are grouped by embedding distance. Since a corner often lies outside the object, the authors add corner pooling, which max-pools features to the right of and below a location to find the object’s top and left boundaries. They argued that corners are easier to localize than centres, because a corner depends on two sides of the box and a centre on all four, and that $O(wh)$ corner positions can represent $O(w^2h^2)$ boxes.

Each corner has exactly one positive location. Training it against all other locations equally would punish near misses that still give a good box, so CornerNet lowers the penalty for negatives within a radius of each corner, with the radius set by object size so that a pair of corners inside it still yields a box of IoU at least 0.7 with the ground truth, and the reduction given by an unnormalized Gaussian with $\sigma$ one third of the radius. On their 5,000-image COCO validation split, removing the penalty reduction lowered AP from 38.5 to 32.9, and a fixed radius of 2.5 gave 35.6. On COCO test-dev, CornerNet with an Hourglass-104 backbone reported 40.5 AP single-scale and 42.1 multi-scale.

Objects as Points (Zhou et al., 2019) is one of two 2019 detectors named CenterNet. It finds each object as the peak of a per-class heatmap of box centres at stride 4 and regresses the box size and a sub-cell offset at that peak; at test time, the peaks are local maxima kept by a $3 \times 3$ max-pooling, and no IoU-based NMS is applied in single-scale testing. The other CenterNet, “CenterNet: Keypoint Triplets for Object Detection” by Duan, Bai, Xie, Qi, Huang and Tian (ICCV 2019), detects each object as a triplet of keypoints, adding a centre keypoint to CornerNet’s two corners and using information from the central region to cut the number of incorrect boxes, as its abstract describes; Zhang et al. (2020) list it among the keypoint-based methods. On COCO test-dev, Objects as Points with Hourglass-104 reported 42.1 AP single-scale and 45.1 multi-scale, and DLA-34 39.2 and 41.6. The same paper applies the formulation to 3D boxes on KITTI and to human pose on COCO. For assignment, the point that matters is that it uses one positive per object, chosen by location alone.

Other keypoint-based detectors that Zhang et al. (2020) list include ExtremeNet, which detects the four extreme points and the centre of each object, Grid R-CNN, whose second stage locates grid points, and RepPoints, which learns a set of sample points per object.

Centre- and point-based

Point-based detection predates the anchor-free label. Tian et al. (2019) call YOLO (Redmon et al., 2016) perhaps the most popular anchor-free detector: the grid cell that contains an object’s centre is responsible for it, and among the cell’s two box predictors, the one whose current prediction has the highest IoU with the object is trained on it. Redmon et al. noted that each cell predicts only two boxes and one class, which limits how many nearby objects the model can detect; Tian et al. add that using only points near the centre gave YOLO low recall, which YOLOv2 addressed by adopting anchors. DenseBox (Huang et al., 2015), as described by Tian et al. and by Zhang et al., predicts a class and the four distances to the box sides at every location of a feature map, with positives in a filled circle at the object’s centre, and crops and rescales training images to a fixed scale, so it detects on an image pyramid. Tian et al. note that this family was used mainly for faces and scene text and was considered unsuitable for generic detection because of overlapping boxes and low recall.

FCOS (Tian et al., 2019) brought the per-pixel formulation to generic detection with an FPN. Every location that falls inside a ground-truth box is a positive for it and regresses $(l, t, r, b)$. Each pyramid level $P_3$ to $P_7$ (strides 8 to 128) handles objects whose largest distance $\max(l, t, r, b)$ falls in its range, 0–64, 64–128, 128–256, 256–512 and above 512 pixels, so that the level ranges play the part of per-level anchor sizes. A one-layer centre-ness branch predicts how close a location is to the centre of its box and down-weights boxes predicted from near the edges; its design and ablation are in FCOS.

FoveaBox (Kong et al., 2020) takes positives only in the middle part of each object, in the description of Zhang et al. (2020). In the official code, each object goes to the pyramid levels whose range contains $\sqrt{wh}$, and the ranges overlap by a factor of two, so an object can train two levels; its positives are the locations within $\sigma = 0.4$ of the half-width and half-height of the box centre, the central 40% of each side; and its regression targets are the distances to the sides divided by a base length per level, in log space. Objects are written onto each level in order of decreasing size, so where two central regions overlap, the smaller object’s label is the one kept.

Other centre-based variants listed by Zhang et al. (2020) include the guided-anchoring RPN, which takes a central region as positives to predict proposals, FSAF, which adds an anchor-free branch with online feature selection to RetinaNet, and CSP, which uses only the centre point of each pedestrian.

Set-based

DETR (Carion et al., 2020) has neither anchors nor anchor points. A transformer decoder turns a fixed set of learned object queries into predictions, and during training the Hungarian algorithm finds the one-to-one matching between predictions and objects of lowest cost, built from class probability and box losses. The authors note that the matching plays the same role as the heuristic rules that assign ground truth to proposals or anchors, with the difference that it is one-to-one, so the network can learn not to produce duplicates. The DETR article covers the model, and label assignment covers one-to-one matching and its hybrids.

Resolving Ambiguity

A location inside two boxes is an ambiguous sample: if positives are defined by position, one location may qualify for several objects, and it can regress only one box. In our reading, an anchor-based detector meets the same problem more rarely, because two anchors at one position differ in shape and an anchor with high IoU to one object usually has low IoU to its neighbour. Tian et al. (2019) considered ambiguity, with recall, the main reason earlier per-pixel detectors were thought unsuited to generic object detection. The anchor-free designs resolve it in four ways.

• Separate levels. Objects of very different sizes are assigned to different pyramid levels, so their overlap is no longer shared on any one level. FCOS counted, on COCO minival, the share of positives that lie in two or more boxes: 23.16% with $P_4$ alone and 7.14% with five levels, and 17.84% against 3.75% when only overlaps of different classes are counted. They argued that same-class overlaps do not matter, since either object gives a correct class, and that most overlap occurs between objects of considerably different sizes. Of the boxes FCOS detected, 2.3% came from ambiguous locations, 1.5% when only different classes are counted.
• Pick the smaller object. FCOS gives a location still inside two boxes to the one of minimal area, and the FoveaBox code does the same by writing smaller objects last. The larger object keeps the locations that belong to it alone; Tian et al. note that such locations only risk missing some larger objects.
• Shrink the positive region. Positives near the centre (DenseBox’s circle, FoveaBox’s central region, the centre sampling later added to FCOS, YOLO’s centre cell) overlap less than whole boxes. One positive per object, as in Objects as Points, removes ambiguity except when two centres fall on the same cell. Zhou et al. counted 614 pairs of objects of the same category with the same centre cell at stride 4 among the 860,001 objects of COCO train2017, under 0.1%. They set this against 715 same-category pairs whose boxes overlap with IoU above 0.7, which they count as collisions for an assignment by anchors.
• Separate outputs. Per-class heatmaps (CornerNet, Objects as Points) let two objects of different classes peak at the same cell. The size and offset heads of Objects as Points are shared by all classes, however, so two objects whose centres coincide still share one size target; Zhou et al. count collisions per category only (their Appendix C).

Set-based detectors resolve ambiguity by construction: one-to-one matching gives each object exactly one prediction and each prediction at most one object. Prediction-aware assigners such as OTA treat ambiguous anchors jointly across objects; they are described in label assignment.

The price of resolving ambiguity is recall: a rule strict enough to give every location one object can leave an object with no positive at all. FCOS measured this as the best possible recall, the share of objects assigned at least one sample in training. With $P_4$ alone, FCOS reached 95.55%, above the 90.92% of RetinaNet in its Detectron implementation, which accepts low-quality matches only at IoU at least 0.4; with five levels it reached 98.40%, close to the 99.23% RetinaNet reaches when all low-quality matches are kept.

Comparisons That Hold Assignment Fixed

Early comparisons of anchor-free and anchor-based detectors changed several things at once. FCOS reported 37.1 AP against 35.9 for RetinaNet on COCO minival with ResNet-50-FPN, but with group normalization in its heads, $P_6$ and $P_7$ computed from $P_5$, an NMS threshold of 0.6, and its own assignment and centre-ness branch; without group normalization, with $C_5$ and an NMS threshold of 0.5, as in RetinaNet, it reported 36.3 against 35.9 with RetinaNet’s hyperparameters.

Zhang et al. (2020) separated the factors. They compared FCOS with a RetinaNet that tiles one square anchor per location (written RetinaNet #A=1), which differs from FCOS only in how positives are defined and in whether regression starts from a box or a point, once implementation details are aligned. Their setting was ResNet-50 with a five-level pyramid, 90,000 iterations at batch size 16, evaluated on COCO minival. Applying FCOS’s general improvements to RetinaNet #A=1 one by one, each on top of the previous ones, raised it from 32.5 AP to 37.0, against 37.8 for FCOS:

Added to RetinaNet (#A=1), cumulatively	AP (%)
none	32.5
group normalization in the heads	33.4
GIoU loss	34.9
positives restricted to inside the box	35.3
centre-ness	36.8
trainable scalar per level	37.0
FCOS, with all of these	37.8

The remaining 0.8 points were then split by swapping the two essential differences:

Positive and negative definition	Regress from a box	Regress from a point
IoU thresholds (RetinaNet)	37.0	36.9
Spatial and scale constraints (FCOS)	37.8	37.8

With the same definition of positives, starting from a box or a point made no clear difference; with the same starting point, FCOS’s definition gave 0.8 to 0.9 more AP. The authors concluded that the essential difference between one-stage anchor-based detectors and centre-based anchor-free ones is how positive and negative training samples are defined. Their adaptive training sample selection (ATSS) then set a threshold per object from the statistics of candidate IoUs and gave nearly the same accuracy with one anchor per location as with nine; the rule and its results are in label assignment and anchor boxes.

Two details of the paper qualify the comparison. The 37.8 for FCOS includes the improvements made after FCOS’s submission (centre-ness on the regression branch, GIoU loss and stride-normalized targets) but not centre sampling, which the ATSS authors describe as their own contribution later merged into FCOS. And the paper attributes the 32.5 AP of RetinaNet #A=1 to the FCOS paper, while the FCOS proceedings version consulted here compares against a standard RetinaNet at 35.9 and gives no one-anchor figure. The study covers centre-based detectors only; Zhang et al. did not run the same analysis for keypoint-based detectors, which, they note, follow a keypoint estimation pipeline different from anchor-based detectors. In our reading, the result reframes “anchor-free” as a statement about the assignment rule and the output parameterization, with assignment the part that still changed accuracy once the other differences were aligned.

Examples

The example counts ambiguous training locations under the target rules above on synthetic images of overlapping objects, the quantity FCOS reports in its Table 2 and Objects as Points in its collision count. Each image is $768 \times 768$ and holds filled ellipses drawn with scikit-image; each object’s ground-truth box is the bounding box of its own mask. Two scene types are drawn, 200 images each, each type from its own random stream seeded with 0: mixed scenes with 12 objects of sizes spread log-uniformly from about 16 to 400 pixels, and crowd scenes with 20 tall objects of similar size (60 to 140 pixels high) packed into a horizontal band, like a row of pedestrians. No network is trained; the counts are properties of the rules and the scenes.

The rules are FCOS’s inside-the-box positives on $P_4$ alone and on $P_3$–$P_7$ with its size ranges, the same with centre sampling of 1.5 strides as in the official FCOS code, FoveaBox’s level ranges and central region as in its official code, a RetinaNet-like baseline of nine anchors per location with positives at IoU $\geq 0.5$, and one centre cell per object at stride 4 for Objects as Points. A location is ambiguous when it qualifies for two or more objects (“amb. all”) or for objects of two or more classes (“amb. diff”, FCOS’s second column); both are shares of positive locations (of positive anchors for the anchor baseline, where the “pos/obj” and last columns also count anchors). Contested locations go to the smaller box (FCOS, FoveaBox) or the higher IoU (anchors), and the last column gives the share of objects left with no positive location after that. For the centre rule, the code prints both the same-class collisions that Zhou et al. count and the collisions of any class, which share the class-agnostic size head.

import numpy as np
from skimage.draw import ellipse

H = W = 768
LEVELS = [(8, 0, 64), (16, 64, 128), (32, 128, 256), (64, 256, 512), (128, 512, np.inf)]
FOVEA = [(8, 1, 64), (16, 32, 128), (32, 64, 256), (64, 128, 512), (128, 256, 2048)]


def scene(rng, kind):
    """Draw filled ellipses on a label image; boxes come from each object's own mask."""
    n = 12 if kind == "mixed" else 20
    label = np.zeros((H, W), np.int32)            # top-most object id per pixel (for display)
    boxes, classes = [], []
    for k in range(n):
        if kind == "mixed":
            ry, rx = np.exp(rng.uniform(np.log(8), np.log(200), 2))
            cy, cx = rng.uniform(0, H), rng.uniform(0, W)
        else:                                      # tall, similar-size objects packed in a band
            ry = rng.uniform(30, 70); rx = ry * rng.uniform(0.35, 0.6)
            cy, cx = rng.uniform(300, 470), rng.uniform(60, W - 60)
        rr, cc = ellipse(cy, cx, ry, rx, shape=(H, W))
        if len(rr) < 4:
            continue
        label[rr, cc] = k + 1
        boxes.append([cc.min(), rr.min(), cc.max() + 1, rr.max() + 1])
        classes.append(rng.integers(0, 3))
    return label, np.array(boxes, float), np.array(classes)


def grid(s):
    c = np.arange(0, W // s) * s + s // 2          # FCOS: floor(s/2) + i*s
    return np.meshgrid(c, c)                       # xs, ys


def ltrb(xs, ys, b):
    return np.stack([xs[..., None] - b[:, 0], ys[..., None] - b[:, 1],
                     b[:, 2] - xs[..., None], b[:, 3] - ys[..., None]], -1)


def fcos_masks(boxes, levels, centre=None):
    """Per level: boolean (h, w, n) of locations that qualify for each object."""
    out = []
    for s, lo, hi in levels:
        xs, ys = grid(s)
        d = ltrb(xs, ys, boxes)
        m = d.min(-1) > 0
        if lo is not None:
            mx = d.max(-1)
            m &= (mx >= lo) & (mx <= hi)
        if centre is not None:                     # official-code centre sampling, clipped to box
            cx = (boxes[:, 0] + boxes[:, 2]) / 2; cy = (boxes[:, 1] + boxes[:, 3]) / 2
            r = centre * s
            m &= (np.abs(xs[..., None] - cx) < r) & (np.abs(ys[..., None] - cy) < r)
        out.append(m)
    return out


def fovea_masks(boxes, sigma=0.4):
    out = []
    side = np.sqrt((boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1]))
    for s, lo, hi in FOVEA:
        n = W // s
        m = np.zeros((n, n, len(boxes)), bool)
        b = boxes / s
        hw = 0.5 * (b[:, 2] - b[:, 0]); hh = 0.5 * (b[:, 3] - b[:, 1])
        L = np.clip(np.ceil(b[:, 0] + (1 - sigma) * hw - 0.5), 0, n - 1).astype(int)
        R = np.clip(np.floor(b[:, 0] + (1 + sigma) * hw - 0.5), 0, n - 1).astype(int)
        T = np.clip(np.ceil(b[:, 1] + (1 - sigma) * hh - 0.5), 0, n - 1).astype(int)
        D = np.clip(np.floor(b[:, 1] + (1 + sigma) * hh - 0.5), 0, n - 1).astype(int)
        for k in np.nonzero((side >= lo) & (side <= hi))[0]:
            m[T[k]:D[k] + 1, L[k]:R[k] + 1, k] = True
        out.append(m)
    return out


def anchor_masks(boxes, thr=0.5):
    out = []
    for s, _, _ in LEVELS:
        xs, ys = grid(s)
        a = []
        for size in 4 * s * 2.0 ** (np.arange(3) / 3):
            for ratio in (0.5, 1.0, 2.0):          # ratio = h / w
                w = size / np.sqrt(ratio); h = size * np.sqrt(ratio)
                a.append(np.stack([xs - w / 2, ys - h / 2, xs + w / 2, ys + h / 2], -1))
        a = np.stack(a, 2)[..., None, :]           # (h, w, 9, 1, 4)
        iw = np.clip(np.minimum(a[..., 2], boxes[:, 2]) - np.maximum(a[..., 0], boxes[:, 0]), 0, None)
        ih = np.clip(np.minimum(a[..., 3], boxes[:, 3]) - np.maximum(a[..., 1], boxes[:, 1]), 0, None)
        inter = iw * ih
        area_a = (a[..., 2] - a[..., 0]) * (a[..., 3] - a[..., 1])
        area_b = (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])
        iou = inter / (area_a + area_b - inter)
        out.append((iou >= thr, iou))
    return out


def tally(masks, classes, prefer, stats):
    """masks: list of (..., n) bool; prefer: list of (..., n) priority (higher wins)."""
    n = len(classes)
    won = np.zeros(n, int)
    for m, p in zip(masks, prefer):
        m = m.reshape(-1, n); p = p.reshape(-1, n)
        q = m.sum(1)
        pos = q >= 1
        stats["pairs"] += m.sum()                  # (location, object) positives
        stats["pos"] += pos.sum()                  # positive locations, as in FCOS Table 2
        stats["amb"] += (q >= 2).sum()
        # different-class ambiguity: location qualifies for objects of >= 2 classes
        ncls = np.zeros(len(m), int)
        for c in np.unique(classes):
            ncls += m[:, classes == c].any(1)
        stats["amb_diff"] += (ncls >= 2).sum()
        winner = np.where(m, p, -np.inf).argmax(1)[pos]
        won += np.bincount(winner, minlength=n)
    stats["obj"] += n
    stats["none"] += (won == 0).sum()


def run(kind, n_img=200, seed=0):
    rng = np.random.default_rng(seed)
    names = ["FCOS, P4 only", "FCOS, P3-P7", "FCOS, P3-P7 + centre", "FoveaBox code", "Anchors IoU>=0.5"]
    st = {k: dict(pairs=0, pos=0, amb=0, amb_diff=0, obj=0, none=0) for k in names}
    cn = dict(obj=0, same=0, any=0)
    for _ in range(n_img):
        _, boxes, cls = scene(rng, kind)
        area = (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])
        small_first = lambda ms: [np.broadcast_to(-area, m.shape) for m in ms]
        for name, ms in [(names[0], fcos_masks(boxes, [(16, None, None)])),
                         (names[1], fcos_masks(boxes, LEVELS)),
                         (names[2], fcos_masks(boxes, LEVELS, centre=1.5)),
                         (names[3], fovea_masks(boxes))]:
            tally(ms, cls, small_first(ms), st[name])
        am = anchor_masks(boxes)
        tally([m for m, _ in am], cls, [i for _, i in am], st[names[4]])
        # CenterNet (Objects as Points): one cell per object at stride 4
        c = np.floor(np.stack([(boxes[:, 0] + boxes[:, 2]) / 2, (boxes[:, 1] + boxes[:, 3]) / 2], 1) / 4)
        key = c[:, 0] * 1000 + c[:, 1]
        cn["obj"] += len(cls)
        for i in range(len(cls)):
            for j in range(i + 1, len(cls)):
                if key[i] == key[j]:
                    cn["any"] += 1
                    cn["same"] += cls[i] == cls[j]
    print(f"\n{kind} scenes ({n_img} images, {cn['obj']} objects)")
    print(f"{'rule':22s} {'pos/obj':>8s} {'amb. all':>9s} {'amb. diff':>10s} {'no positive':>12s}")
    for k in names:
        s = st[k]
        print(f"{k:22s} {s['pairs'] / s['obj']:8.1f} {100 * s['amb'] / s['pos']:8.2f}% "
              f"{100 * s['amb_diff'] / s['pos']:9.2f}% {100 * s['none'] / s['obj']:11.2f}%")
    print(f"CenterNet, stride 4: centre collisions, same class {cn['same']} pairs, "
          f"any class {cn['any']} pairs ({100 * cn['any'] / cn['obj']:.2f}% of objects)")


for kind in ("mixed", "crowd"):
    run(kind)

mixed scenes (200 images, 2400 objects)
rule                    pos/obj  amb. all  amb. diff  no positive
FCOS, P4 only              46.5     9.70%      6.81%        0.12%
FCOS, P3-P7                24.1     2.69%      1.77%        1.00%
FCOS, P3-P7 + centre        8.0     0.63%      0.39%        1.12%
FoveaBox code               6.3     0.53%      0.38%        3.25%
Anchors IoU>=0.5           15.1     0.27%      0.22%       30.17%
CenterNet, stride 4: centre collisions, same class 0 pairs, any class 0 pairs (0.00% of objects)

crowd scenes (200 images, 4000 objects)
rule                    pos/obj  amb. all  amb. diff  no positive
FCOS, P4 only              19.1    27.94%     19.82%        0.75%
FCOS, P3-P7                32.3    13.81%      9.67%        0.07%
FCOS, P3-P7 + centre       10.1     4.82%      3.31%        0.17%
FoveaBox code               6.7     3.99%      2.66%        0.42%
Anchors IoU>=0.5           24.0     3.92%      2.66%        0.00%
CenterNet, stride 4: centre collisions, same class 2 pairs, any class 5 pairs (0.12% of objects)

The script, run with NumPy 2.3.5 and scikit-image 0.25.2, takes about 20 to 30 seconds. One change was made after the first run: the first version computed both ambiguous shares over (location, object) pairs, and they were changed to count positive locations, as FCOS’s Table 2 does; the scenes, rules and other columns were not changed.

In the mixed scenes the pattern follows FCOS’s COCO analysis. On one level, almost a tenth of positive locations lie in two boxes; spreading objects over five levels by size removes about three quarters of them, in our reading because most overlaps there are between a small object and a large one, which now train different levels. Restricting positives to the centre (FCOS with centre sampling, FoveaBox) leaves well under 1%, and no two centres fell on the same stride-4 cell.

The crowd scenes break FCOS’s premise that overlapping objects differ in size. Objects of similar size land on the same level, so the five-level rule removes only about half of the ambiguity, leaving nearly one positive location in seven contested, about five times the share the same rule leaves in the mixed scenes. Central regions help most there, but several percent of their positives stay ambiguous, about as many as the anchor baseline’s. Centre collisions appear, and the any-class count is larger than the same-class count: under the paper’s count, two of the five colliding pairs are a problem, while for the shared size head all five are.

The last column shows the cost of each rule. The anchor baseline has no best-anchor clause, and in the mixed scenes it leaves almost a third of objects with no anchor at IoU 0.5; in our reading these are mainly objects more elongated than the anchors’ aspect ratios of at most 2:1 (the mixed scenes draw the two semi-axes of each ellipse independently) or smaller than its smallest anchors. Such objects are the reason RetinaNet and the RPN add forced matches for the best anchor and the case in which FCOS measured a best possible recall of 86.82% for RetinaNet without low-quality matches. The point rules leave at most a few percent, the most for FoveaBox’s narrow central region.

Common Misconceptions

• “Anchor-free means no assignment hyperparameters.” The IoU thresholds are replaced by other choices: FCOS’s size ranges per level, FoveaBox’s $\sigma$ and overlapping ranges, CornerNet’s IoU of 0.7 that sets the penalty radius, centre-sampling radii.
• “Anchor-free detectors are more accurate because they regress from points.” With the definition of positives held fixed, regressing from a point and from a one-anchor box gave the same AP (Zhang et al., 2020).
• “Anchor-free detectors need no NMS.” FCOS and FoveaBox predict many boxes per object and use NMS. Objects as Points replaces IoU-based NMS with a $3 \times 3$ max-pooling over the heatmap, though its multi-scale testing merges results with NMS; Zhou et al. report that adding NMS changed AP from 39.2 to 39.7 for DLA-34 with flip testing and left Hourglass-104 at 42.2. Only one-to-one set prediction is designed to make duplicate removal unnecessary.
• “An anchor point is not an anchor.” Zhang et al. call FCOS’s location an anchor point precisely because it plays the same role as the centre of a RetinaNet anchor: it is the place a prediction is attached to and trained from. Anchor-free detectors drop the preset box shape, not the reference.
• “Per-pixel targets are hopelessly ambiguous.” On COCO, multi-level assignment left 3.75% of FCOS’s positives in overlaps of different classes. How much ambiguity remains depends on the scene: the example shows that crowds of similar-size objects behave differently from scenes of mixed sizes.

Where It Is Used

Anchor-free heads became common in one-stage object detectors after 2019; the YOLO article describes the anchor-free heads of YOLOX and YOLOv8. Objects as Points applied its centre heatmaps to 3D detection and pose estimation in the same paper. Keypoint-based detection shares its machinery with human pose estimation through heatmap regression, and CornerNet borrowed its grouping from multi-person pose. Set prediction led to the DETR family. In every form, the assignment rules described in label assignment, from ATSS to Hungarian matching, apply to anchor points and queries as they do to anchor boxes.