Label Assignment in Object Detection

Label assignment is the step in training an object detector that decides, for every prediction the network makes, whether it should learn to detect a particular ground-truth object, learn to say “background”, or be left out of the loss. A dense detector produces tens of thousands of candidate outputs per image, one or more per feature-map position and pyramid level, while an image holds a handful of objects; before any loss can be computed, each object must be given its share of those outputs. The same step is also called sample assignment, training sample selection, or matching predictions to ground truth. The rules range from fixed thresholds on the intersection over union (IoU) between anchor boxes and objects, through statistical and centre-based rules, to assignments computed from the network’s own predictions and to the one-to-one Hungarian matching of DETR. Zhang et al. (2020) showed that the gap between an anchor-based and an anchor-free one-stage detector came mainly from this step, which made assignment a research subject in its own right.

Definition

Given $N$ predictions $\{(p_j, \hat b_j)\}$, each with class scores $p_j$ and a box $\hat b_j$ attached to a reference (an anchor box, an anchor point, or a learned query), and $M$ ground-truth objects $\{(c_i, g_i)\}$, a label assignment maps each prediction $j$ to an object index $a(j) \in \{1, \dots, M\}$, to background ($a(j) = \varnothing$), or to “ignored”. The training loss then uses these labels as targets: positives get their object’s class and box, negatives the background class and no box target, and ignored predictions contribute nothing. An assignment is one-to-many when an object may receive several positives and one-to-one when it receives exactly one. It is fixed (static) when it depends only on the references and the ground truth, and dynamic (prediction-aware) when it depends on the current predictions, and therefore changes during training.

Intuition

The network sees the same features at many neighbouring positions and levels, and each of them could be taught to fire on a given object. Assignment decides which of them are told “this is yours”. Teach too few and the object receives little gradient and may not be recalled; teach too many and the detector learns from outputs whose features barely cover the object, while neighbouring objects compete for the same outputs. Three questions recur in every rule: which candidates qualify (overlap, distance to the centre, the pyramid level), how many each object gets, and who wins when two objects claim the same output.

Assignment also fixes what a high score means at test time. If several outputs per object are trained as positives, several will fire, and duplicates must be removed with non-maximum suppression (NMS). If exactly one is, the detector can learn to suppress duplicates itself, which is how DETR removes NMS.

Formal Definition

Write the assignment as a binary matrix $\pi \in \{0, 1\}^{(M+1) \times N}$ whose last row stands for background, with $\sum_{i=1}^{M+1} \pi_{ij} = 1$ for every prediction that is not ignored. For a dense detector the loss is typically

$$\mathcal{L} = \frac{1}{N_{\text{pos}}} \Big[ \sum_{j \notin \mathcal{I}} \mathcal{L}_{\text{cls}}\big(p_j, c_{a(j)}\big) + \lambda \sum_{j:\, a(j) \neq \varnothing} \mathcal{L}_{\text{box}}\big(\hat b_j, g_{a(j)}\big) \Big],$$

where $\mathcal{I}$ is the ignored set, $c_\varnothing$ is the background label, and $N_{\text{pos}}$ the number of positives. RetinaNet, for instance, sums its focal loss over all anchors and divides by the number of anchors assigned to a ground-truth box, since most anchors are easy negatives with negligible loss (Lin et al., 2017).

Many rules can be written as minimizing a cost $\sum_{i,j} C_{ij} \pi_{ij}$ under constraints on the row sums (how many positives each object supplies) and the column sums (each prediction gets one label). A fixed rule takes $C_{ij}$ from the geometry of the reference alone, for example $C_{ij} = -\mathrm{IoU}(\text{anchor}_j, g_i)$ with a threshold. A dynamic rule takes it from the predictions, typically

$$C_{ij} = \mathcal{L}_{\text{cls}}\big(p_j, c_i\big) + \alpha\, \mathcal{L}_{\text{reg}}\big(\hat b_j, g_i\big) + C^{\text{prior}}_{ij},$$

so that an output already good at predicting object $i$ is cheap to assign to it, with a geometric prior term that keeps candidates near the object. One-to-one matching adds the constraint $\sum_j \pi_{ij} = 1$ for each object. In the official code of the dynamic assigners consulted here (YOLOX and TOOD), the assignment is computed from detached predictions and treated as a constant target; gradients flow only through the loss that uses it.

Fixed Rules

IoU thresholds and the ignore band

The region proposal network of Faster R-CNN labels an anchor positive if its IoU with some object exceeds 0.7, or if it is the anchor with the highest IoU for an object, and negative if its IoU is below 0.3 for every object; the anchors in between are ignored (Ren et al., 2015). RetinaNet uses 0.5 and 0.4 instead (Lin et al., 2017), and SSD matches each object to its best default box and to every default box above 0.5. The anchor boxes article works through the RPN rule. The ignore band between the two thresholds removes the outputs whose label is least certain: an anchor that half-covers an object is neither a good example of it nor a clean example of background, and training it either way would teach a confusing target. The “best anchor” clause exists because a small or oddly shaped object may have no anchor above the positive threshold.

Two-stage detectors use a second, region-level assignment for their proposals. Fast R-CNN labels a proposal as foreground if its IoU with an object is at least 0.5 and as background if its maximum IoU lies in $[0.1, 0.5)$; Girshick notes that the lower limit of 0.1 appears to act as a heuristic for hard example mining (Girshick, 2015).

The positive threshold trades the quantity of positives against their quality. Cai and Vasconcelos (2018) observed that a detector trained with a threshold of 0.5 produces many “close but not correct” boxes, but that raising the threshold does not simply help: among their first-stage training samples (RPN proposals on COCO), 16.7% had IoU above 0.5, 8.0% above 0.6 and only 2.9% above 0.7, so a high threshold leaves few positives and the detector overfits; in their Figure 1(d) a detector trained at 0.7 had lower AP than ones trained at 0.5 or 0.6. Their Cascade R-CNN instead trains a sequence of heads at thresholds 0.5, 0.6 and 0.7, each on the refined boxes of the previous stage, whose IoU distribution has shifted upward so that every stage keeps enough positives. In their ablation (FPN with RoIAlign and ResNet-50, COCO val), AP rose from 34.9 for the single-stage baseline to 38.9, with most of the gain at high IoU (AP$_{90}$ 7.1 against 15.8).

Centre and scale rules

Anchor-free detectors replace overlap with position. The original YOLO makes the grid cell containing an object’s centre responsible for it. FCOS treats every feature-map location that falls inside a ground-truth box as a positive for that object, then removes it if the largest of its four regression distances lies outside the range assigned to its pyramid level (0–64, 64–128, 128–256, 256–512 and above 512 pixels for $P_3$ to $P_7$); a location still inside two boxes goes to the smaller one (Tian et al., 2019). In effect, the level ranges play the role that per-level anchor sizes play in a feature pyramid network. With them, the share of positives inside the overlap of two boxes of different classes fell from 17.84% (only $P_4$) to 3.75% on COCO minival.

Treating the whole box as positive includes locations near its edges whose features mostly show background. Centre sampling keeps only locations close to the centre. In the official FCOS code it is a square of 1.5 strides on either side of the box centre, clipped to the box, and is switched on in the improved configurations; the README links the trick to, and the code’s docstring says the function is copied from, the FCOS_PLUS repository of the GitHub user yqyao, while the ATSS authors describe centre sampling as their own contribution, the “lite” version of their method merged into the FCOS code, selecting the 9 candidates per level nearest the centre. In our reading the two rules are close but not identical: a $\pm 1.5$-stride square holds the $3 \times 3$ locations around the centre, which are the 9 nearest when the centre lies near a location but not always when it lies near a cell corner, and the code also clips the square to the box. On COCO minival with ResNet-50, it raised FCOS from 37.8 to 38.6 AP (Zhang et al., 2020).

ATSS

Adaptive training sample selection (ATSS; Zhang et al., 2020) replaces the fixed IoU threshold with one computed per object. For each object $g$ it takes, on each of the $L$ pyramid levels, the $k$ anchors whose centres are closest to the centre of $g$, giving $kL$ candidates $\mathcal{C}_g$. With $D_g$ the IoUs of these candidates with $g$, the threshold is

$$t_g = \operatorname{mean}(D_g) + \operatorname{std}(D_g),$$

and a candidate becomes positive if its IoU is at least $t_g$ and its centre lies inside $g$; an anchor claimed by several objects goes to the one with the highest IoU, and all other anchors are negatives. There is no ignore band. The mean measures how well the preset anchors suit the object, the standard deviation how concentrated the good candidates are on one level: a high standard deviation raises the threshold so that only the best level contributes. The authors report that each object then receives about $0.2\,kL$ positives regardless of its size, aspect ratio or position, where IoU and FCOS rules give large objects many more.

The method has one hyperparameter, $k = 9$. On COCO minival with ResNet-50, AP stayed between 39.0 and 39.3 for $k$ from 7 to 17 and dropped to 38.0 at $k = 3$. ATSS raised the improved one-anchor RetinaNet from 37.0 to 39.3 AP and FCOS from 37.8 to 39.2. It also changed the case for multiple anchors per location: with IoU thresholds, nine anchors per location beat one (38.4 against 37.0), but with ATSS they gave 39.2 against 39.3. The finding that the definition of positives, not regression from a box or a point, separated the two detector families is summarized in the anchor boxes article.

Prediction-Aware Assignment

Fixed rules decide from geometry before the network has seen the image. A family of methods from 2019 onward lets the predictions take part, on the argument that an output which already classifies and localizes an object well is the right one to train on it.

FreeAnchor (Zhang et al., 2019) builds for each object a bag of the 50 anchors with the highest IoU and trains with a likelihood that rewards the bag’s best anchor for classifying and localizing the object (recall) and pushes poorly localized anchors toward background (precision). Because the best anchor is unreliable early in training, it replaces the maximum over the bag by a “mean-max” function that weights all anchors when confidences are low and approaches the maximum as some anchors become confident. On COCO val with ResNet-50, it raised RetinaNet from 35.7 to 38.7 AP at nearly the same training time.

PAA (Kim and Lee, 2020) scores each candidate anchor of an object by the product of its classification score and a power of the IoU of its predicted box, whose negative logarithm is the sum of its classification and IoU losses, fits a two-component Gaussian mixture to the pooled scores of the object’s top $K$ candidates from each level by expectation–maximization, and splits the candidates into positives and negatives at a boundary derived from the fitted mixture, so that the number of positives adapts to the model.

OTA

Optimal transport assignment (OTA; Ge et al., 2021) argues that assigning each object independently fails on ambiguous anchors, those that qualify for two overlapping objects, and treats all objects of an image jointly. Each ground-truth object is a supplier holding $k$ units of positive label, a background supplier holds the remaining $n - mk$ units of negative label, and each of the $n$ anchors demands one unit. The cost of moving a positive unit from object $i$ to anchor $j$ is $\mathcal{L}_{\text{cls}} + \alpha \mathcal{L}_{\text{reg}}$ computed from the anchor’s predictions, the cost of a background unit is the anchor’s classification loss against the background class, and the transport plan

$$\pi^* = \arg\min_{\pi \ge 0} \sum_{i=1}^{m+1} \sum_{j=1}^{n} C_{ij}\, \pi_{ij} \quad \text{s.t.} \quad \sum_{i} \pi_{ij} = 1, \quad \sum_{j} \pi_{ij} = s_i ,$$

with $s_i = k$ for objects and $s_{m+1} = n - mk$, is found approximately with Sinkhorn–Knopp iterations. Each anchor goes to the supplier that sends it the most. Two additions matter in practice: a centre prior, an extra constant cost for anchors outside the $r^2$ nearest to an object’s centre on each level, and dynamic $k$ estimation, which sets each object’s supply to the sum of the top $q = 20$ IoUs between its predicted boxes and the object, so that objects that are already well regressed by many outputs receive more positives. The authors report under 20% extra training time and no test-time cost.

On COCO val with a ResNet-50 FCOS trained for 90k iterations, OTA reached 40.7 AP against 39.5 for FCOS with centre sampling and the same IoU branch; a constant $k = 10$ gave 40.3, and $k = 1$, a one-to-one assignment, only 36.5. Its main argument is the ambiguity table: as the centre region grew from $r = 3$ to $r = 7$, the average number of ambiguous anchors per image under ATSS rose from 2.1 to 36.3 and AP fell from 39.4 to 37.2, while OTA kept 0.2–0.3 ambiguous anchors and 40.4–40.7 AP. Resolving ambiguous anchors by a hand-written rule (smallest area, largest IoU or lowest loss) before running OTA cost 0.4–0.7 AP. On CrowdHuman val, a crowded pedestrian benchmark (ResNet-50, 30 epochs), OTA reached a log-average miss rate of 46.6% against 49.5% for ATSS and 48.7% for Faster R-CNN with FPN (lower is better); the paper’s text calls the Faster R-CNN figure “48.7% AP”, but its Table 6 lists it as the miss rate.

SimOTA

YOLOX (Ge, Liu, Wang, Li and Sun, 2021) simplified OTA into SimOTA, which replaces the Sinkhorn iterations by a per-object top-$k$ selection; the YOLOX paper (arXiv) was not read for this article, so the description follows the official code. In the code (commit of June 2025), candidates are the locations within 1.5 strides of an object’s centre. The cost is the binary cross-entropy of the class scores (the geometric mean of class and objectness probabilities) against the object’s one-hot class plus three times $-\log \mathrm{IoU}$ of the predicted box, plus $10^6$ for locations outside that object’s centre region. Each object’s $k$ is the integer part of the sum of the 10 highest predicted-box IoUs, at least 1, and the object takes its $k$ lowest-cost locations; a location chosen by two objects goes to the one with the lower cost. The earliest release (0.1.0) used a radius of 2.5 strides and admitted locations inside the box or the centre region, penalizing those not in both. The YOLO article places YOLOX among the family’s versions.

Task-aligned assignment

TOOD (Feng et al., 2021) starts from a different complaint: the location with the best classification score and the one with the best box are often not the same, so NMS may keep a confident but poorly localized box. Its task alignment learning (TAL) scores each prediction for an object by

$$t = s^{\alpha}\, u^{\beta},$$

with $s$ the predicted score for the object’s class and $u$ the IoU of the predicted box, and takes the $m$ predictions with the largest $t$ as positives. The paper chose $\alpha = 1$, $\beta = 6$ by a coarse search and $m = 13$, reporting 42.0–42.5 AP for $m$ from 5 to 21. The official code also requires a positive’s location to lie inside the object’s box, gives a location claimed by two objects to the one with the higher IoU, and uses ATSS for the first 4 training epochs, when predictions are still poor.

The same metric reweights the loss. Positives are trained toward a soft label $\hat t$, which is $t$ rescaled per object so that its maximum equals the object’s largest IoU, under a focal-style loss, and the GIoU box loss of each positive (see bounding box regression) is weighted by $\hat t$. In TOOD’s ablation on COCO minival (ResNet-50, 12 epochs, conventional head), IoU-based assignment gave 36.5 AP, centre sampling 37.3, ATSS 39.2, FreeAnchor 39.1, and TAL 40.3; with 18 epochs, PAA reached 39.9 (40.9 with its IoU prediction branch) and TAL 40.9. YOLOv10 cites YOLOv8 for the task-aligned metric with $\alpha = 0.5$ and $\beta = 6$ in its one-to-many head (Wang et al., 2024).

One-to-One Matching

DETR (Carion et al., 2020) matches its fixed set of predictions to the objects one-to-one with the Hungarian algorithm, using a cost that combines the predicted probability of the object’s class with an $\ell_1$ and a generalized-IoU box cost; every unmatched prediction is trained toward “no object”. The DETR article gives the matching cost and loss in full. In terms of the formulation above, it is the transport problem with every object supplying exactly one unit, and the DETR authors present it as the counterpart of the hand-designed rules that assign anchors or proposals to ground truth. Since exactly one output per object is trained to fire, the network must learn to suppress its own duplicates, and NMS becomes unnecessary.

The price is sparse supervision. Jia et al. (2023) note that Deformable DETR matches fewer than 30 of its 300 queries per image on COCO, since nearly 99% of COCO images have fewer than 30 annotated boxes, so most queries never receive a box loss. OTA’s $k = 1$ result on a convolutional detector points the same way. Two remedies add one-to-many supervision only during training:

• Hybrid matching (H-DETR; Jia et al., 2023) adds a second group of 1,500 queries matched by the Hungarian algorithm to the ground truth repeated $K = 6$ times, so each object has six positives in that group; masked self-attention keeps the groups apart, and only the original one-to-one queries are used at inference. On COCO with ResNet-50 and a 12-epoch schedule, this raised their Deformable DETR baseline from 47.0 to 48.7 AP.
• Consistent dual assignments (YOLOv10; Wang et al., 2024) give a convolutional detector two heads with the same structure, one trained with one-to-many task-aligned assignment and one with one-to-one matching, and keep only the second at inference. The one-to-one head simply takes each object’s top-scoring prediction under the metric $s \cdot p^{\alpha} \cdot \mathrm{IoU}^{\beta}$, with $s$ a spatial prior, which the authors found to perform as well as Hungarian matching at less training time. Using the same $\alpha$ and $\beta$ in both heads (up to a common factor) makes the best one-to-many positive also the one-to-one positive. Built on YOLOv8-S (COCO val), the one-to-many head alone reached 44.9 AP with 7.07 ms end-to-end latency including NMS, the one-to-one head alone 43.4 AP, and the dual scheme 44.3 AP at 2.44 ms without NMS.

Assignment, Sampling and Loss Weighting

Assignment decides labels; sampling and weighting decide how much each labelled output counts, and the three interact.

Sampling. Faster R-CNN’s RPN computes its loss on 256 anchors per image, sampled with a ratio of positives to negatives of up to 1:1 and padded with negatives when an image has fewer than 128 positives (Ren et al., 2015). Fast R-CNN takes 25% of its 64 regions per image from foreground proposals (Girshick, 2015). SSD keeps the highest-loss negatives, at most three per positive, a form of hard negative mining. Each sampler presumes a fixed assignment: it reweights the classes but cannot repair a rule that gives an object no good positive.

Weighting. RetinaNet keeps every anchor and lets the focal loss down-weight easy negatives, normalizing by the number of assigned anchors (Lin et al., 2017); the one-stage detectors article derives the loss. DETR down-weights the no-object class by a factor of 10. Later methods weight positives by quality: GFL’s quality focal loss trains the class score of a positive toward the IoU of its predicted box rather than 1 (Li et al., 2020), and TAL weights both the classification target and the box loss by $\hat t$.

The boundary between assigning and weighting is not sharp. TOOD’s ablation classifies methods by whether the positive/negative split is fixed or adaptive and whether the positive weights are: PISA and NoisyAnchor keep the split and adapt the weights, FreeAnchor and PAA adapt the split, and TAL adapts both. A soft target $\hat t$ near zero is close to not assigning the output at all, and an ignore band is a weight of zero. Normalization adds a further coupling, in our reading: a rule that produces more positives also enlarges $N_{\text{pos}}$, so the per-positive loss scale changes with the assignment.

Examples

The NumPy and SciPy code below runs six assignment rules on the same synthetic image and the same predictions, and counts positives per object and per pyramid level. A $512 \times 512$ label image holds six objects of three classes: a small disk, a medium square, a large ellipse, a slender pole, and two overlapping upright ellipses (“crowd A” and “crowd B”); each box is read from its object’s own mask, so the occluded crowd A keeps its full box. Four pyramid levels with strides 8, 16, 32 and 64 have one location per cell and one square anchor of side $8 \times$ stride, the one-anchor setting of the ATSS experiments, 5,440 locations in all. IoU thresholds (RetinaNet’s 0.5/0.4, plus the RPN’s clause that makes each object’s best anchor positive), FCOS centre sampling and ATSS use only the geometry. SimOTA, task-aligned assignment and Hungarian matching need predictions, so a stand-in for a partly trained detector (our construction) predicts at each location inside a box a noisy copy of that object’s box and a class score, both worse away from the centre and on levels that do not suit the object’s size; locations outside every box predict a jittered anchor with low scores. Because the stand-in is a modelling choice, the prediction-aware rules are run with “early” (noisy boxes, flat scores) and “late” predictions, and over five seeds. Each rule follows its paper or the official code cited above, reduced to its assignment step. It was run with NumPy 2.3.5 and SciPy 1.16.3 and takes under a second.

import numpy as np
from scipy.optimize import linear_sum_assignment

# A 512 x 512 label image with six objects of three classes; boxes come from each object's mask.
H = W = 512
yy, xx = np.mgrid[0:H, 0:W] + 0.5
def ellipse(cx, cy, a, b): return ((xx - cx) / a) ** 2 + ((yy - cy) / b) ** 2 <= 1
def rect(x1, y1, x2, y2): return (xx >= x1) & (xx < x2) & (yy >= y1) & (yy < y2)
objects = [("small disk", 0, ellipse(120, 60, 10, 10)), ("square", 1, rect(380, 40, 440, 100)),
           ("large ellipse", 2, ellipse(200, 340, 130, 100)), ("pole", 1, rect(40, 70, 54, 230)),
           ("crowd A", 2, ellipse(390, 330, 25, 65)), ("crowd B", 2, ellipse(420, 345, 25, 65))]
label = np.zeros((H, W), int)
for i, (_, _, m) in enumerate(objects):
    label[m] = i + 1                       # crowd B is drawn over crowd A
def box_of(m):
    r, c = np.nonzero(m)
    return np.array([c.min(), r.min(), c.max() + 1, r.max() + 1], float)
gt = np.stack([box_of(m) for _, _, m in objects])        # full (amodal) boxes from each mask
gcls = np.array([c for _, c, _ in objects])
names = [n for n, _, _ in objects]
M, C = len(gt), 3

# Pyramid levels P3-P6: one location per cell, one square anchor of side 8 * stride (ATSS's #A=1 setting).
strides = [8, 16, 32, 64]
pts, lvl, st = [], [], []
for l, s in enumerate(strides):
    c = (np.arange(W // s) + 0.5) * s
    X, Y = np.meshgrid(c, c)
    pts.append(np.stack([X.ravel(), Y.ravel()], 1)); lvl += [l] * X.size; st += [s] * X.size
pts, lvl, st = np.concatenate(pts), np.array(lvl), np.array(st, float)
anchors = np.concatenate([pts - 4 * st[:, None], pts + 4 * st[:, None]], 1)
N = len(pts)

def iou(A, B):                                           # (N, 4) x (M, 4) corner boxes
    lt = np.maximum(A[:, None, :2], B[None, :, :2]); rb = np.minimum(A[:, None, 2:], B[None, :, 2:])
    inter = np.prod(np.clip(rb - lt, 0, None), 2)
    area = lambda X: np.prod(X[:, 2:] - X[:, :2], 1)
    return inter / (area(A)[:, None] + area(B)[None] - inter)

def giou(A, B):
    lt = np.minimum(A[:, None, :2], B[None, :, :2]); rb = np.maximum(A[:, None, 2:], B[None, :, 2:])
    hull = np.prod(rb - lt, 2)
    i = iou(A, B)
    area = lambda X: np.prod(X[:, 2:] - X[:, :2], 1)
    union = (area(A)[:, None] + area(B)[None]) / (1 + i)
    return i - (hull - union) / hull

inside = ((pts[:, None, 0] > gt[None, :, 0]) & (pts[:, None, 0] < gt[None, :, 2]) &
          (pts[:, None, 1] > gt[None, :, 1]) & (pts[:, None, 1] < gt[None, :, 3]))   # (N, M)
gc = (gt[:, :2] + gt[:, 2:]) / 2
gsize = np.sqrt(np.prod(gt[:, 2:] - gt[:, :2], 1))
A_iou = iou(anchors, gt)

def resolve(claim, score):
    """claim: (N, M) bool; each location goes to the claiming object with the highest score."""
    s = np.where(claim, score, -np.inf)
    a = np.where(claim.any(1), s.argmax(1), -1)
    return a, int((claim.sum(1) > 1).sum())

# Rule 1: IoU thresholds (RetinaNet: positive >= 0.5, negative < 0.4) plus each object's best anchor.
def iou_threshold():
    best = A_iou.max(1)
    claim = A_iou >= 0.5
    claim[A_iou.argmax(0), np.arange(M)] = True
    a, amb = resolve(claim, A_iou)
    ignored = int(((best >= 0.4) & (a < 0)).sum())
    return a, amb, ignored

# Rule 2: FCOS centre sampling (radius 1.5 strides, clipped to the box) with per-level size ranges.
def centre_sampling(radius=1.5, ranges=((0, 64), (64, 128), (128, 256), (256, np.inf))):
    r = radius * st[:, None]
    lo = np.maximum(gc[None] - r[..., None], gt[None, :, :2]); hi = np.minimum(gc[None] + r[..., None], gt[None, :, 2:])
    in_c = np.all((pts[:, None] > lo) & (pts[:, None] < hi), 2)
    ltrb = np.concatenate([pts[:, None] - gt[None, :, :2], gt[None, :, 2:] - pts[:, None]], 2).max(2)
    rng = np.array(ranges)[lvl]
    claim = in_c & (ltrb >= rng[:, :1]) & (ltrb <= rng[:, 1:])
    return resolve(claim, -np.prod(gt[:, 2:] - gt[:, :2], 1)[None].repeat(N, 0)) + (0,)

# Rule 3: ATSS (k nearest anchors per level, IoU threshold = mean + std, centre inside the box).
def atss(k=9):
    claim = np.zeros((N, M), bool)
    d = np.linalg.norm(pts[:, None] - gc[None], axis=2)
    for j in range(M):
        cand = np.concatenate([np.flatnonzero(lvl == l)[np.argsort(d[lvl == l, j])[:k]] for l in range(len(strides))])
        t = A_iou[cand, j].mean() + A_iou[cand, j].std(ddof=1)
        keep = cand[(A_iou[cand, j] >= t) & inside[cand, j]]
        claim[keep, j] = True
    return resolve(claim, A_iou) + (0,)

def predictions(rng, sigma, sharp):
    """Stand-in for a partly trained detector: boxes and class probabilities at every location."""
    dn = np.abs(pts[:, None] - gc[None]) / ((gt[None, :, 2:] - gt[None, :, :2]) / 2)
    dn = np.where(inside, dn.max(2), np.inf)
    obj = np.where(inside.any(1), dn.argmin(1), -1)
    boxes = anchors + rng.normal(0, 0.1, (N, 4)) * 8 * st[:, None]
    prob = 1 / (1 + np.exp(4 - rng.normal(0, 0.5, (N, C))))
    for n in np.flatnonzero(obj >= 0):
        j = obj[n]; d = dn[n, j]; mis = abs(np.log2(gsize[j] / (8 * st[n])))
        cx, cy = gc[j]; w, h = gt[j, 2:] - gt[j, :2]
        e = rng.normal(0, sigma * (1 + 2 * d + mis), 4)
        cx, cy, w, h = cx + e[0] * w, cy + e[1] * h, w * np.exp(e[2]), h * np.exp(e[3])
        boxes[n] = [cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2]
        prob[n, gcls[j]] = 1 / (1 + np.exp(-(sharp * (1 - d) - 1.5 * mis - 1 + rng.normal(0, 0.5))))
    return boxes, prob

# Rule 4: SimOTA (YOLOX): centre-region candidates, cost = class BCE + 3 * IoU cost, dynamic k.
def simota(boxes, prob):
    P_iou = iou(boxes, gt)
    in_c = np.all(np.abs(pts[:, None] - gc[None]) < 1.5 * st[:, None, None], 2)
    fg = in_c.any(1)
    onehot = np.eye(C)[gcls]
    p = np.clip(prob, 1e-7, 1 - 1e-7)
    bce = -(onehot[None] * np.log(p[:, None]) + (1 - onehot[None]) * np.log(1 - p[:, None])).sum(2)
    cost = bce + 3 * -np.log(P_iou + 1e-8) + 1e6 * ~in_c
    claim = np.zeros((N, M), bool)
    for j in range(M):
        top = np.sort(P_iou[fg, j])[::-1][:10]
        k = max(int(top.sum()), 1)
        claim[np.flatnonzero(fg)[np.argsort(cost[fg, j])[:k]], j] = True
    return resolve(claim, -cost) + (0,)

# Rule 5: task-aligned assignment (TOOD): t = s * IoU^6, top 13 per object, centre inside the box.
def task_aligned(boxes, prob, m=13, alpha=1, beta=6):
    P_iou = iou(boxes, gt)
    t = prob[:, gcls] ** alpha * P_iou ** beta
    claim = np.zeros((N, M), bool)
    for j in range(M):
        top = np.argsort(-t[:, j])[:m]
        claim[top[(t[top, j] > 0) & inside[top, j]], j] = True
    return resolve(claim, P_iou) + (0,)

# Rule 6: Hungarian matching (DETR cost: -p(class) + 5 * L1 on normalized cx, cy, w, h - 2 * GIoU).
def hungarian(boxes, prob):
    cxcywh = lambda b: np.concatenate([(b[:, :2] + b[:, 2:]) / 2, b[:, 2:] - b[:, :2]], 1) / W
    l1 = np.abs(cxcywh(boxes)[:, None] - cxcywh(gt)[None]).sum(2)
    cost = -prob[:, gcls] + 5 * l1 - 2 * giou(boxes, gt)
    rows, cols = linear_sum_assignment(cost)
    a = np.full(N, -1); a[rows] = cols
    return a, 0, 0

def report(name, out):
    a, amb, ign = out
    per_obj = [int((a == j).sum()) for j in range(M)]
    per_lvl = [int(((a >= 0) & (lvl == l)).sum()) for l in range(len(strides))]
    print(f"{name:<16}" + "".join(f"{c:>8}" for c in per_obj) + f"{sum(per_obj):>7}{ign:>8}{amb:>6}   "
          + " ".join(f"{c:>3}" for c in per_lvl))
    return per_obj

print("boxes:", {n: tuple(int(v) for v in b) for n, b in zip(names, gt)})
print(f"{N} locations; best anchor IoU per object:", np.round(A_iou.max(0), 2))
head = f"{'rule':<16}" + "".join(f"{n.split()[-1][:7]:>8}" for n in names) + "  total ignored  amb.   P3  P4  P5  P6"
print(head)
for name, f in [("IoU 0.5/0.4", iou_threshold), ("centre sampling", centre_sampling), ("ATSS k=9", atss)]:
    report(name, f())
for quality, sigma, sharp in [("early", 0.15, 2.0), ("late", 0.05, 5.0)]:
    boxes, prob = predictions(np.random.default_rng(0), sigma, sharp)
    print(f"-- predictions: {quality} (seed 0); mean IoU of in-box predictions "
          f"{iou(boxes, gt)[inside].mean():.2f}")
    for name, f in [("SimOTA", simota), ("task-aligned", task_aligned), ("Hungarian", hungarian)]:
        report(name, f(boxes, prob))
print("-- per-object positives, min-max over seeds 0-4")
for quality, sigma, sharp in [("early", 0.15, 2.0), ("late", 0.05, 5.0)]:
    runs = {"SimOTA": [], "task-aligned": []}
    for seed in range(5):
        boxes, prob = predictions(np.random.default_rng(seed), sigma, sharp)
        for name, f in [("SimOTA", simota), ("task-aligned", task_aligned)]:
            a = f(boxes, prob)[0]
            runs[name].append([(a == j).sum() for j in range(M)])
    for name, r in runs.items():
        r = np.array(r)
        print(f"{quality:<6}{name:<13}" + "".join(f"{f'{lo}-{hi}':>8}" for lo, hi in zip(r.min(0), r.max(0))))

Output:

boxes: {'small disk': (110, 50, 130, 70), 'square': (380, 40, 440, 100), 'large ellipse': (70, 240, 330, 440), 'pole': (40, 70, 54, 230), 'crowd A': (365, 265, 415, 395), 'crowd B': (395, 280, 445, 410)}
5440 locations; best anchor IoU per object: [0.1  0.88 0.74 0.16 0.43 0.43]
rule                disk  square ellipse    pole       A       B  total ignored  amb.   P3  P4  P5  P6
IoU 0.5/0.4            1      15      14       1       1       1     33      65     0    19   0  14   0
centre sampling        6       9       9       0       9       7     40       0     2    15  16   9   0
ATSS k=9               6       9       9       6       9       9     48       0     0    39   0   9   0
-- predictions: early (seed 0); mean IoU of in-box predictions 0.20
SimOTA                 1       4       5       2       4       5     21       0     0    13   5   3   0
task-aligned           5      13      13      13      11      10     65       0     0    46  12   6   1
Hungarian              1       1       1       1       1       1      6       0     0     5   1   0   0
-- predictions: late (seed 0); mean IoU of in-box predictions 0.54
SimOTA                 4       7       8       5       7       7     38       0     0    25   9   3   1
task-aligned           7      13      13      13      13      13     72       0     0    56  11   4   1
Hungarian              1       1       1       1       1       1      6       0     0     4   2   0   0
-- per-object positives, min-max over seeds 0-4
early SimOTA            1-1     4-5     4-5     2-2     4-5     5-5
early task-aligned      3-5   13-13   13-13   13-13   11-13    9-12
late  SimOTA            3-4     7-8     7-8     4-5     7-7     7-8
late  task-aligned      7-7   13-13   13-13   13-13   13-13   13-13

The IoU rule gives the square and the large ellipse 14–15 positives each, because their sizes match an anchor scale, while the small disk, the pole and both crowd members never reach 0.5 with a square anchor and survive only through the forced best-anchor clause, one positive each, with 65 anchors in the ignore band. ATSS evens this out: every object gets 6–9 positives, close to the paper’s estimate of $0.2\,kL \approx 7$, and the small and slender objects gain the most. Centre sampling also gives 6–9 to most objects but none to the pole: from locations near its centre the largest regression distance is about 80 pixels, which assigns it to $P_4$; with $P_4$ locations every 16 pixels, at most one column of them can fall inside its 14-pixel width, and here the nearest, at $x = 40$, lies exactly on its left edge and fails the strict inside test, so it follows that a pole placed a pixel or two further left would have received positives from that column; the two locations claimed by both crowd members go to crowd A by the smaller-area rule (the boxes are equal, so the tie falls to the first). The level counts show the scale side of the same story: ATSS puts the large ellipse on $P_5$ and everything else on $P_3$, while centre sampling’s size ranges send the disk and the square to $P_3$, the crowd to $P_4$ and the ellipse to $P_5$.

The prediction-aware rules respond to prediction quality, as their designs intend. SimOTA’s dynamic $k$ is the sum of the top 10 IoUs, so with early predictions (mean in-box IoU 0.20) it gives 1–5 positives per object, and with late ones (0.54) 3–8; over seeds 0–4 the counts move by at most one. Task-aligned assignment gives 13 whenever 13 high-scoring locations lie inside the box, and fewer only for the disk, which has just 7 locations inside its box (6 on $P_3$, 1 on $P_4$), and for the crowd members with early predictions, when 2–3 of their 13 highest-scoring locations lie outside their own box. Hungarian matching gives exactly one positive per object regardless; which location wins depends on the predictions. None of the dynamic rules produced an ambiguous location here, since their per-object choices did not overlap in this scene; the ambiguity OTA measures arises with larger candidate regions and denser crowds.

Common Misconceptions

• “Anchor-free detectors need no assignment rule.” They replace IoU thresholds with other rules: inside-box tests, centre regions, scale ranges per level, and tie-breaking by area. In the example, FCOS’s scale range sent the thin pole to a level whose grid can place at most one column of locations inside it, and in that placement none.
• “A higher positive IoU threshold gives a more precise detector.” It gives fewer positives; Cascade R-CNN found a single detector trained at 0.7 worse than one trained at 0.5 or 0.6, and recovered precision only by resampling across stages.
• “Dynamic assignment is learned.” OTA, SimOTA and TAL solve a fixed optimization on the current predictions at each step, with no trainable parameters of their own; the assignment changes because the predictions do.
• “One-to-one matching is simply better.” It removes NMS, but on its own it trains few outputs per object; OTA’s $k = 1$ run lost about 4 AP, and H-DETR and YOLOv10 add one-to-many supervision during training.

Where It Is Used

Every trained detector has an assignment step. Two-stage detectors use IoU rules twice: in the RPN for anchors and in the second stage for proposals (two-stage object detectors, Faster R-CNN). SSD and RetinaNet use IoU rules with hard negative mining or the focal loss. FCOS and other anchor-free detectors use centre and scale rules, ATSS-style selection is used by ATSS and, for its first epochs, by TOOD, SimOTA is the assigner of YOLOX, and task-aligned assignment that of TOOD and, according to the YOLOv10 paper, of YOLOv8’s one-to-many head. DETR and its successors use Hungarian matching, often combined with an auxiliary one-to-many branch during training. Crowded scenes, where many outputs qualify for several objects at once, are where the choice of rule matters most, as OTA’s CrowdHuman results show.