Non-Maximum Suppression

Non-maximum suppression (NMS) removes duplicate detections. An object detector typically produces many overlapping candidate boxes around each object; NMS keeps the highest-scoring box, discards the boxes that overlap it by more than an intersection over union (IoU) threshold, and repeats on what remains. Most detectors run it at test time; its failures in crowded scenes motivated soft and learned variants and detectors trained to need no NMS.

The name is older than box detection. In the Canny edge detector it keeps pixels whose gradient magnitude is a maximum along the gradient direction, and in the Harris corner detector pixels whose response is a maximum in their neighborhood. Box NMS likewise reduces a dense response to its peaks, with IoU defining the neighborhood.

Problem

A detector’s score changes smoothly with a box’s position and size: a box shifted by a few pixels still covers most of the object and scores almost as high. Sliding-window detectors fire on many neighboring windows, region-based detectors regress several proposals onto the same object, and dense one-stage detectors predict a box at many feature-map locations on each object, so the raw output holds a cluster of boxes per object.

When detections are matched to ground truth for average precision, each object can be claimed by only one box, and every further box on it counts as a false positive. NMS reduces each cluster to one representative.

Inputs and Outputs

The input is a set of $n$ candidate boxes $b_1, \dots, b_n$ in corner form (bounding box conventions), their scores $s_1, \dots, s_n$, optionally their class labels, and an IoU threshold $N_t$. The output is the subset of indices that survive, usually ordered by decreasing score. Soft variants output all boxes with revised scores instead.

Intuition

The highest-scoring box is the detector’s best guess for some object, and a box that overlaps it strongly is probably another guess for the same object. The next surviving box is taken as a guess for a different object, and so on. This assumes that boxes on the same object overlap more than boxes on different objects, and that the highest-scoring box is also the best-localized one; the failure cases below break one of these assumptions.

Algorithm

Greedy NMS, as used in most detectors:

input:  boxes B = {b_1..b_n}, scores S, IoU threshold N_t
keep ← []
order ← indices of B sorted by decreasing score
while order is not empty:
    i ← first element of order          # highest remaining score
    append i to keep
    remove i from order
    remove from order every j with IoU(b_i, b_j) > N_t
return keep

Class-aware and class-agnostic NMS. In a multi-class detector, NMS usually runs separately for each class, so a person box never suppresses an overlapping bicycle box. R-CNN (Girshick et al., 2014) applied greedy NMS to each class independently, as did its successors. Class-agnostic NMS runs once over all boxes regardless of label, as for region proposals, which have only an objectness score. TorchVision’s batched_nms shifts every box by its class index times one more than the largest coordinate, so that boxes of different classes can never overlap, and then runs ordinary NMS once; for large inputs it runs NMS class by class instead.

The overlap measure has varied. Felzenszwalb et al. (2010) skipped a detection when at least 50% of its box was covered by a previously selected box, an overlap measured relative to the candidate’s own area rather than IoU. R-CNN and its successors use IoU.

Mathematical Formulation

Bodla et al. (2017) wrote the suppression step as a rescoring of every remaining box $b_i$ after the box $M$ with the highest score has been selected:

$$s_i \leftarrow s_i \, f\big(\mathrm{IoU}(M, b_i)\big), \qquad f_{\text{hard}}(o) = \begin{cases} 1, & o < N_t \\ 0, & o \ge N_t \end{cases}$$

Greedy NMS is the binary $f_{\text{hard}}$. Soft-NMS replaces it with a decay: linear,

$$f_{\text{lin}}(o) = \begin{cases} 1, & o < N_t \\ 1 - o, & o \ge N_t \end{cases}$$

or Gaussian, applied to all remaining boxes,

$$f_{\text{gauss}}(o) = \exp\!\left(-\frac{o^2}{\sigma}\right).$$

Decayed boxes stay in the ranked list, the selection repeats on the rescored set, and boxes whose score falls below a small threshold are pruned.

DIoU-NMS (Zheng et al., 2020) keeps the hard rule but changes the overlap measure. With $\rho(M, b_i)$ the distance between the two box centers and $c$ the diagonal of the smallest box enclosing both, box $b_i$ is removed when

$$\mathrm{IoU}(M, b_i) - \frac{\rho^2(M, b_i)}{c^2} \ge \varepsilon.$$

Two boxes with the same IoU are less likely to be suppressed when their centers are far apart, on the authors’ argument that distant centers probably belong to different objects.

Parameters

• IoU threshold $N_t$. Low values remove true neighbors along with duplicates; high values let duplicates through. Defaults differ by detector and stage. R-CNN describes its threshold as learned, and the evaluation function in its released code defaults to 0.3; the py-faster-rcnn code uses 0.3 at test time. Faster R-CNN (Ren et al., 2015) applies NMS at 0.7 to its region proposals, which leaves about 2,000 proposals per image. SSD (Liu et al., 2016) uses 0.45 per class and RetinaNet (Lin et al., 2017) uses 0.5.
• Score threshold. Boxes below a minimum score are dropped before NMS, which shrinks $n$: SSD uses 0.01 and RetinaNet 0.05.
• Top-$k$ caps limit the boxes entering and leaving NMS. RetinaNet decodes at most 1,000 top-scoring predictions per pyramid level; SSD keeps the top 200 detections per image; py-faster-rcnn keeps at most 100 detections per image over all classes.
• $\sigma$ for Gaussian Soft-NMS. Bodla et al. used $\sigma = 0.5$ and, for the linear version, $N_t = 0.3$.

Complexity

Sorting costs $O(n \log n)$. Each selected box is compared with all remaining boxes, so the worst case, when few boxes are suppressed, is $O(n^2)$ IoU computations; when each selection removes a large cluster, far fewer remain. Soft-NMS has the same $O(n^2)$ worst case, as Bodla et al. note; score thresholds and top-$k$ caps keep $n$ small. The selection is sequential, since whether a box survives depends on the boxes kept before it; Wang et al. (2020) cite this as a source of latency.

Implementation

The example places three people in a $96 \times 144$ image, two of them side by side with ground-truth IoU 0.37. To control the overlap exactly, it simulates a detector’s raw output instead of running one (the object detection article runs a sliding-window detector): twelve jittered candidates per person, scored by how well they fit, with the occluded person B scored lower, plus six background boxes. It applies greedy NMS at three thresholds and Gaussian Soft-NMS, matches the results to ground truth at IoU $\ge 0.5$, each object at most once, and reports true positives (TP), false positives (FP), false negatives (FN) and AP at IoU 0.5.

import numpy as np

def iou(b, B):                                       # one box against many, corner form (x1, y1, x2, y2)
    w = np.clip(np.minimum(b[2], B[:, 2]) - np.maximum(b[0], B[:, 0]), 0, None)
    h = np.clip(np.minimum(b[3], B[:, 3]) - np.maximum(b[1], B[:, 1]), 0, None)
    inter = w * h
    return inter / ((b[2] - b[0]) * (b[3] - b[1]) + (B[:, 2] - B[:, 0]) * (B[:, 3] - B[:, 1]) - inter)

def nms(boxes, scores, t):                           # greedy NMS: indices kept, best first
    order, keep = np.argsort(-scores, kind="stable"), []
    while len(order):
        i, order = order[0], order[1:]
        keep.append(i)
        order = order[iou(boxes[i], boxes[order]) <= t]   # discard IoU > t
    return np.array(keep)

def soft_nms(boxes, scores, sigma=0.5, s_min=0.001):  # Gaussian Soft-NMS (Bodla et al., 2017)
    s, rest, keep = scores.copy(), np.arange(len(scores)), []
    while len(rest):
        i = rest[np.argmax(s[rest])]
        keep.append(i)
        rest = rest[rest != i]
        s[rest] *= np.exp(-iou(boxes[i], boxes[rest]) ** 2 / sigma)
        rest = rest[s[rest] > s_min]                 # prune boxes whose score has decayed away
    return np.array(keep), s

def evaluate(boxes, scores, gt, t=0.5):              # greedy matching by score, AP as area under P-R
    used, tp = np.zeros(len(gt), bool), []
    for i in np.argsort(-scores, kind="stable"):
        o = iou(boxes[i], gt); o[used] = 0
        tp.append(o.max() >= t)
        if tp[-1]: used[o.argmax()] = True
    tp = np.array(tp); ctp = np.cumsum(tp)
    recall, precision = ctp / len(gt), ctp / np.arange(1, len(tp) + 1)
    precision = np.maximum.accumulate(precision[::-1])[::-1]       # interpolated precision
    ap = np.sum(np.diff(np.concatenate([[0], recall])) * precision)
    return f"TP {tp.sum()}, FP {len(tp) - tp.sum()}, FN {len(gt) - tp.sum()}, AP {ap:.2f}"

# Ground truth in a 96 x 144 image: two people side by side (B partly occludes A), one apart.
gt = np.array([[20, 16, 52, 72], [34, 18, 66, 74], [100, 30, 124, 70]], float)
print("IoU between ground-truth A and B:", round(iou(gt[0], gt[1:2])[0], 2))

# Simulated raw detector output: 12 jittered candidates per object, scored by how well they
# fit (the occluded person B scores lower), plus 6 low-scoring background boxes.
rng = np.random.default_rng(0)
boxes, scores = [], []
for g, q in zip(gt, [0.95, 0.75, 0.90]):
    size = np.array([g[2] - g[0], g[3] - g[1]] * 2)
    cand = g + rng.normal(0, 0.08, (12, 4)) * size
    boxes.append(cand)
    scores.append(np.clip(q * iou(g, cand) + rng.normal(0, 0.03, 12), 0.01, 1))
xy = rng.uniform([0, 0], [110, 60], (6, 2)); wh = rng.uniform(16, 32, (6, 2))
boxes.append(np.hstack([xy, xy + wh])); scores.append(rng.uniform(0.05, 0.3, 6))
boxes, scores = np.vstack(boxes), np.concatenate(scores)

print(f"raw          {len(boxes):2d} boxes:", evaluate(boxes, scores, gt))
for t in (0.3, 0.5, 0.7):
    keep = nms(boxes, scores, t)
    print(f"NMS  t={t}   {len(keep):2d} boxes:", evaluate(boxes[keep], scores[keep], gt))
keep, new = soft_nms(boxes, scores)
print(f"Soft-NMS     {len(keep):2d} boxes:", evaluate(boxes[keep], new[keep], gt))

def show(i, s_old, s_new=None):
    o = iou(boxes[i], gt)
    obj = "ABC"[o.argmax()] if o.max() >= 0.5 else "-"
    s = f"{s_old:.2f}" + (f" -> {s_new:.2f}" if s_new is not None else "")
    print(f"  {str(boxes[i].round().astype(int)):18s} score {s:12s} best IoU {o.max():.2f} ({obj})")

print("\nNMS t=0.3 keeps:")
for i in nms(boxes, scores, 0.3):
    show(i, scores[i])
b = np.argmax(np.where(iou(gt[1], boxes) >= 0.5, scores, 0))      # best candidate on person B
a = nms(boxes, scores, 0.3)[0]
print(f"best box on B: score {scores[b]:.2f}, IoU {iou(boxes[a], boxes[b:b+1])[0]:.2f} with the kept box on A")
print("Soft-NMS, top 5 after rescoring:")
for i in keep[:5]:
    show(i, scores[i], new[i])

Output:

IoU between ground-truth A and B: 0.37
raw          42 boxes: TP 3, FP 39, FN 0, AP 0.55
NMS  t=0.3    6 boxes: TP 2, FP 4, FN 1, AP 0.67
NMS  t=0.5    8 boxes: TP 3, FP 5, FN 0, AP 1.00
NMS  t=0.7   14 boxes: TP 3, FP 11, FN 0, AP 0.92
Soft-NMS     24 boxes: TP 3, FP 21, FN 0, AP 1.00

NMS t=0.3 keeps:
  [20 15 54 72]      score 0.93         best IoU 0.92 (A)
  [101  31 125  71]  score 0.81         best IoU 0.89 (C)
  [58 31 77 57]      score 0.15         best IoU 0.10 (-)
  [64 12 83 38]      score 0.09         best IoU 0.02 (-)
  [74 43 99 59]      score 0.07         best IoU 0.00 (-)
  [10 59 35 90]      score 0.07         best IoU 0.08 (-)
best box on B: score 0.67, IoU 0.41 with the kept box on A
Soft-NMS, top 5 after rescoring:
  [20 15 54 72]      score 0.93 -> 0.93 best IoU 0.92 (A)
  [101  31 125  71]  score 0.81 -> 0.81 best IoU 0.89 (C)
  [35 22 66 69]      score 0.62 -> 0.48 best IoU 0.79 (B)
  [101  30 122  62]  score 0.67 -> 0.28 best IoU 0.71 (C)
  [14 15 49 69]      score 0.64 -> 0.23 best IoU 0.71 (A)

The raw output finds all three people with 39 false positives, mostly duplicates, and AP 0.55. At threshold 0.3, NMS deletes person B: its best box overlaps the kept box on A with IoU 0.41, so the true neighbor is treated as a duplicate. At 0.7, more duplicates survive, and the AP of 0.92 means one of them now ranks above B’s box. Threshold 0.5 happens to suit this scene. Soft-NMS keeps 24 boxes, but rescoring puts the three true detections first: the box on B drops only from 0.62 to 0.48, while the duplicates of A and C fall to 0.28 and below. Over ten random seeds, NMS at 0.3 lost person B in four, NMS at 0.7 had AP below 1 in eight, and Soft-NMS and NMS at 0.5 reached AP 1.00 in all ten.

TorchVision’s torchvision.ops.nms implements the same greedy rule, discarding boxes whose IoU exceeds the threshold.

Properties and Behavior

• Training-free. NMS has no learned parameters and can be replaced without retraining the detector, which is how Soft-NMS was evaluated.
• Interaction with AP. A surviving duplicate costs AP only if it ranks above remaining true positives, which is why the example’s 21 low-scoring Soft-NMS leftovers cost nothing; removing a true neighbor caps recall. Redmon et al. (2016) reported that NMS added 2–3% mAP to YOLO and was less critical than for R-CNN or DPM, because its grid usually assigns each object to one cell.

Limitations

• Crowds. When two real objects overlap by more than $N_t$, the weaker one is removed. Hosang et al. (2017) describe the resulting trade-off between recall and precision as built into greedy clustering with a fixed threshold.
• One threshold for everything. The right value depends on how crowded a scene is, and Bodla et al. argue that under COCO-style AP, averaged over IoU thresholds from 0.5 to 0.95, a low $N_t$ suppresses boxes needed at strict evaluation thresholds while a high $N_t$ adds false positives at loose ones.
• Scores do not measure localization. The most confident box is kept even when a discarded neighbor fits better.
• Not part of training. The detector is trained without regard to the non-differentiable step that prunes its output.

Variants

• Soft-NMS (Bodla et al., 2017) decays scores instead of deleting boxes. On COCO test-dev the authors reported AP rising from 31.1 to 32.4 for R-FCN and from 24.4 to 25.5 for Faster R-CNN with the Gaussian decay, and, for R-FCN, per-class gains of 3–6% for animals that appear in herds, such as zebras and giraffes.
• DIoU-NMS (Zheng et al., 2020) adds the center-distance penalty above. Combined with detectors trained with their loss, the authors reported it better than standard NMS at every threshold they tried between 0.43 and 0.48 for YOLOv3 and SSD.
• Box voting (Gidaris & Komodakis, 2015) runs NMS at IoU 0.3 and then replaces each kept box by the score-weighted average of all boxes overlapping it with IoU above 0.5, so suppressed boxes still improve localization.
• Learned NMS. Hosang et al. (2017) trained a network, GossipNet, that rescores all detections from their boxes and scores alone so as to leave one high-scoring detection per object, and tested it on crowded pedestrian scenes and COCO.
• Matrix NMS (Wang et al., 2020), introduced for instance masks in SOLOv2, decays all scores at once with parallel matrix operations, avoiding both hard removal and the sequential loop.
• Peak extraction. CenterNet (Zhou et al., 2019) detects objects as peaks of a center heatmap and replaces IoU-based NMS with a $3 \times 3$ max-pooling, close to the older pixel-wise meaning of the term.
• NMS-free detectors. DETR (Carion et al., 2020) trains a transformer to output a set of predictions matched one-to-one to ground-truth objects with the Hungarian algorithm, so each object has a single positive prediction. Its authors found that NMS helped the first decoder layer’s predictions but lowered AP after the last layers by removing true positives. YOLOv10 (Wang et al., 2024) trains two heads, one with one-to-many label assignment for rich supervision and one with one-to-one assignment, and uses only the one-to-one head at inference, without NMS.