Object Detection

Object detection finds the objects of a given set of categories in an image and says, for each one, where it is and what it is. Where image classification returns one label for the whole image, a detector returns a list of bounding boxes, each with a class label and a confidence score, and the list can hold any number of entries, including none. This hub defines the task, outlines how the methods evolved and why, and points to the articles on metrics, datasets and detector designs.

Problem Definition

Given an image $I$ and a fixed set of $K$ classes, a detector outputs a set

$$\mathcal{D}(I) = \{(\mathbf{b}_i, c_i, s_i)\}_{i=1}^{N}, \qquad \mathbf{b}_i \in \mathbb{R}^4,\quad c_i \in \{1, \dots, K\},\quad s_i \in [0, 1],$$

where $\mathbf{b}_i$ is a box, $c_i$ its class and $s_i$ a confidence score. The number $N$ varies from image to image, and the output is a set: order carries no meaning, and each real object should appear exactly once.

A detection is correct when its class matches a ground-truth object and its box overlaps that object’s box with intersection over union (IoU) above a threshold such as 0.5. Each object can justify only one detection; a second box on it is a false positive.

Input and Output

The input is an image, with ground-truth boxes and classes for training. The raw output of most detectors is not $\mathcal{D}$ itself but many more candidate boxes, one or more per position of a feature map, each with class scores and box coordinates. Post-processing drops candidates below a score threshold and removes overlapping duplicates with non-maximum suppression (NMS). The bounding box article compares the coordinate conventions in which boxes are written.

Variants

• 3D detection predicts metric cuboids with a position, size and heading, from images, stereo pairs or point clouds.
• Oriented detection predicts rotated boxes for elongated objects at arbitrary angles, such as ships in aerial images or lines of text.
• Instance segmentation replaces or complements each box with a pixel mask. Mask R-CNN (He et al., 2017) adds a mask branch to the Faster R-CNN detector.
• Open-vocabulary detection detects categories that had no box annotations during training, specified by name at test time. Zareian et al. (2021) proposed the formulation, training with boxes for a few classes plus image–caption pairs that cover many more.
• Video object detection detects objects in every frame and can use neighboring frames against motion blur and occlusion; multi-object tracking must also keep identities over time.

Challenges

• Scale variation and small objects. Objects range from a few pixels to the whole image. Small ones cover few positions of a downsampled feature map, and a one-pixel error costs them far more IoU. About 41% of COCO’s annotated objects are small (area under $32^2$ pixels), according to its evaluation documentation.
• Occlusion and truncation. Objects are partly hidden or cut by the image border, and datasets differ on whether boxes cover the visible part or the full extent.
• Crowding and duplicates. A detector that fires on an object also fires on slightly shifted and rescaled boxes around it, so raw outputs contain many near-duplicates. Removing them by overlap fails in crowds, where two real objects can overlap as much as two duplicates.
• Object–background imbalance. Almost all candidate locations are background. Lin et al. (2017, focal loss) describe two-stage detectors narrowing the candidates to about 1,000–2,000 proposals, while one-stage detectors evaluate about 100,000 locations per image, most of them easy negatives.
• Localization precision. Benchmarks that average over strict IoU thresholds reward boxes that fit tightly, not just roughly.
• Real-time constraints. Driving, robotics and video often need tens of frames per second on embedded hardware.

Approaches

The field moved from scanning hand-designed classifiers over the image, to classifying region proposals with CNN features, to predicting boxes densely in one network, to predicting the set of objects directly.

Sliding windows with hand-crafted features

The classical recipe turned detection into classification: slide a window over the image at many positions and scales (an image pyramid), classify each window as object or background, and merge overlapping positives.

Viola and Jones (2001) made this fast enough for real-time face detection. They computed rectangle features in constant time with an integral image, selected a few of them with AdaBoost, and chained classifiers in a cascade whose cheap early stages discard most background windows; it processed $384 \times 288$ images at 15 frames per second on a 700 MHz Pentium III.

Dalal and Triggs (2005) introduced histograms of oriented gradients (HOG) for pedestrian detection: a $64 \times 128$ window is described by locally normalized gradient-orientation histograms on a dense grid (see feature descriptors), classified by a linear SVM, and scanned over all positions and scales. Felzenszwalb et al. (2010) extended the single HOG template to deformable part models (DPM): a root filter plus part filters that may shift relative to it, mixtures of such models for different viewpoints, and a latent SVM trained with mined hard negatives; they reported state-of-the-art results on PASCAL VOC. All of these rested on hand-designed features.

Region proposals and CNN features: two-stage detectors

R-CNN (Girshick et al., 2014) replaced them with the features of a convolutional neural network pre-trained on ImageNet classification and fine-tuned for detection. It classified about 2,000 region proposals per image from selective search, each warped to the network’s input size and scored by class-specific linear SVMs. On PASCAL VOC 2012 it reached 53.3% mean average precision, which the authors reported as over 30% better, in relative terms, than the previous best result.

Later work removed redundant computation. Fast R-CNN (Girshick, 2015) computed the convolutional features once per image and pooled a fixed-size feature for each proposal with RoI pooling; with VGG16 it trained 9 times and tested 213 times faster than R-CNN. Faster R-CNN (Ren et al., 2015) replaced selective search with a region proposal network that shares the detector’s features and predicts objectness and box offsets relative to reference boxes called anchors at every feature-map position; the full system ran at 5 frames per second on a GPU with VGG-16. The two-stage object detectors and Faster R-CNN articles cover these designs.

One-stage detectors

One-stage detectors drop the proposal stage and predict classes and boxes densely in one pass. YOLO (Redmon et al., 2016) divides the image into a grid and regresses boxes and class probabilities from the whole image in a single network evaluation, at 45 frames per second; the authors noted more localization errors than in two-stage systems and difficulty with small objects in groups, such as flocks of birds. SSD (Liu et al., 2016) predicts class scores and offsets for default boxes of several aspect ratios on feature maps of several resolutions, so that each map handles a range of object sizes. The one-stage object detectors and YOLO articles cover this family.

One-stage detectors trailed two-stage ones in accuracy until Lin et al. (2017) identified the cause as the object–background imbalance. Their focal loss reshapes the cross-entropy loss to down-weight well-classified examples; with it, they reported, the one-stage RetinaNet surpassed the accuracy of existing two-stage detectors.

Multi-scale features and anchor-free detection

Deep CNN layers carry strong semantics at coarse resolution, which suits large objects but not small ones. The feature pyramid network (FPN; Lin et al., 2017) adds a top-down pathway with lateral connections, so that every scale gets high-level features at its own resolution. Its authors evaluated it inside Faster R-CNN, and RetinaNet is built on it.

Anchors bring hyperparameters (sizes, aspect ratios, IoU rules for assigning them to objects) to which, Tian et al. (2019) note, detection performance is often sensitive. Anchor-free detectors predict boxes from points instead. FCOS (Tian et al., 2019) regresses, at each feature-map location inside an object, the distances to the four sides of its box, and still uses NMS. CenterNet (Zhou et al., 2019) detects each object as the peak of a center-point heatmap and regresses its size there; a $3 \times 3$ max-pooling over the heatmap takes the place of IoU-based NMS.

Set prediction with transformers

DETR (Carion et al., 2020) treats detection as direct set prediction. A transformer encoder–decoder turns CNN features and a fixed number of learned object queries into predictions, each a class and a box or “no object”, and training matches them one-to-one to ground-truth objects with the Hungarian algorithm, which removes the need for anchors and NMS. On COCO the authors reported DETR equal in AP to a tuned Faster R-CNN with FPN, better on large objects and worse on small ones.

Shared components

Two pieces recur across these families and have their own articles. Non-maximum suppression greedily keeps the highest-scoring box and removes boxes that overlap it too much, usually measured by IoU; R-CNN applied it to each class independently, DPM used a greedy variant that measures overlap as the fraction of a box covered by a kept box, and the HOG detector also ran non-maximum suppression. Anchor boxes are the reference boxes against which many detectors predict offsets, and they decide which predictions are trained as positives.

Family	Representative methods	Year	Key idea	Article
Sliding window, hand-crafted	Viola–Jones; HOG + linear SVM; DPM	2001–2010	Classify every window; cascades, gradient histograms, deformable parts	—
Two-stage	R-CNN, Fast R-CNN, Faster R-CNN	2014–2015	Propose regions, then classify and refine each with CNN features	Two-stage object detectors
One-stage	YOLO, SSD, RetinaNet	2016–2017	Dense predictions in one pass; focal loss against imbalance	One-stage object detectors
Multi-scale features	FPN	2017	Top-down pathway gives every scale strong features	Feature pyramid network
Anchor-free	FCOS, CenterNet	2019	Predict boxes from points or centers, without anchors	—
Set prediction	DETR	2020	Transformer outputs a set; Hungarian matching; no NMS	DETR

Example: a sliding-window detector on a synthetic image

The NumPy code builds a noisy $64 \times 96$ image with four squares of 8, 14, 22 and 12 pixels, the last one dim, slides square windows of five sizes over it with stride 2, and keeps windows scoring above 0.6. It compares two scores: the mean intensity in the window, and that mean minus the mean of a surrounding ring of width $s/4$ (center–surround contrast), both from an integral image as in Viola–Jones. It then applies greedy NMS at IoU 0.5 and matches detections to ground truth at IoU $\ge 0.5$, each object at most once, to count true positives (TP), false positives (FP) and false negatives (FN).

import numpy as np

# A 64 x 96 noisy image with four square objects: three bright, one dim.
rng = np.random.default_rng(3)
H, W = 64, 96
img = rng.normal(0.0, 0.25, (H, W))
gt = np.array([[10, 12, 18, 20], [40, 30, 54, 44],         # ground-truth boxes, xyxy:
               [64, 8, 86, 30], [14, 40, 26, 52]], float)  # 8, 14, 22 and 12 pixels wide
for (x1, y1, x2, y2), c in zip(gt.astype(int), [1.0, 1.0, 1.0, 0.5]):
    img[y1:y2, x1:x2] += c

S = np.zeros((H + 1, W + 1))                                 # integral image: O(1) box sums
S[1:, 1:] = img.cumsum(0).cumsum(1)
def box_mean(x1, y1, x2, y2):
    return (S[y2, x2] - S[y1, x2] - S[y2, x1] + S[y1, x1]) / ((x2 - x1) * (y2 - y1))

def mean_score(x1, y1, s):                                   # brightness inside the window
    return box_mean(x1, y1, x1 + s, y1 + s)

def contrast_score(x1, y1, s):                               # inside minus a ring of width s/4
    m = s // 4
    a, b, c, d = x1 - m, y1 - m, x1 + s + m, y1 + s + m
    if a < 0 or b < 0 or c > W or d > H:
        return -np.inf
    outer, inner = box_mean(a, b, c, d) * (s + 2*m)**2, mean_score(x1, y1, s) * s**2
    return mean_score(x1, y1, s) - (outer - inner) / ((s + 2*m)**2 - s**2)

def sliding_window(score, thr=0.6, sizes=(8, 12, 16, 20, 24), stride=2):
    dets = [(x, y, x + s, y + s, score(x, y, s)) for s in sizes
            for y in range(0, H - s + 1, stride) for x in range(0, W - s + 1, stride)]
    dets = np.array([d for d in dets if d[4] > thr])
    return dets[:, :4], dets[:, 4]

def iou(b, B):                                               # one box against many
    w = np.clip(np.minimum(b[2], B[:, 2]) - np.maximum(b[0], B[:, 0]), 0, None)
    h = np.clip(np.minimum(b[3], B[:, 3]) - np.maximum(b[1], B[:, 1]), 0, None)
    inter = w * h
    return inter / ((b[2] - b[0]) * (b[3] - b[1]) + (B[:, 2] - B[:, 0]) * (B[:, 3] - B[:, 1]) - inter)

def nms(boxes, scores, t=0.5):                               # greedy non-maximum suppression
    order, keep = np.argsort(-scores), []
    while len(order):
        i, order = order[0], order[1:]
        keep.append(i)
        order = order[iou(boxes[i], boxes[order]) < t]
    return np.array(keep)

def match(boxes, scores, t=0.5):                             # each ground truth matched at most once
    used, tp = np.zeros(len(gt), bool), 0
    for i in np.argsort(-scores):
        o = iou(boxes[i], gt)
        o[used] = 0
        if o.max() >= t:
            used[o.argmax()] = True
            tp += 1
    return f"TP {tp}, FP {len(boxes) - tp}, FN {len(gt) - tp}"

for name, score in [("mean", mean_score), ("contrast", contrast_score)]:
    boxes, scores = sliding_window(score)
    hits = [int((iou(g, boxes) >= 0.5).sum()) for g in gt]
    keep = nms(boxes, scores)
    print(f"{name:8s} raw: {len(boxes):3d} windows, IoU >= 0.5 with each object {hits}, {match(boxes, scores)}")
    print(f"{'':8s} NMS: {len(keep):3d} windows, {match(boxes[keep], scores[keep])}")

for i in keep:                                               # what survives, contrast score
    print(boxes[i].astype(int), f"score {scores[i]:.2f}", "best IoU", np.round(iou(boxes[i], gt).max(), 2))

boxes, scores = sliding_window(contrast_score, thr=0.45)     # a lower score threshold
keep = nms(boxes, scores)
print(f"contrast, threshold 0.45, NMS: {len(keep)} windows, {match(boxes[keep], scores[keep])}")

Output:

mean     raw: 428 windows, IoU >= 0.5 with each object [5, 24, 72, 0], TP 3, FP 425, FN 1
         NMS:  91 windows, TP 3, FP 88, FN 1
contrast raw:  32 windows, IoU >= 0.5 with each object [1, 8, 22, 0], TP 3, FP 29, FN 1
         NMS:   4 windows, TP 3, FP 1, FN 1
[10 12 18 20] score 1.01 best IoU 1.0
[64 10 84 30] score 0.84 best IoU 0.83
[38 30 54 46] score 0.76 best IoU 0.77
[40 36 48 44] score 0.68 best IoU 0.33
contrast, threshold 0.45, NMS: 19 windows, TP 4, FP 15, FN 0

The raw output shows the duplicate problem: with the contrast score, 32 windows pass, 22 of them on the largest object at IoU 0.5 or more, and since only one per object can be a true positive, 29 are false positives. NMS leaves 4: the three bright objects, at IoU 0.77–1.0 because the window sizes do not match every object size, and an 8-pixel window inside the 14-pixel object. That window overlaps the kept 16-pixel box with IoU $64/256 = 0.25$, below the suppression threshold: NMS removes duplicates, not partial detections.

The mean score leaves 88 false positives after NMS. In our reading, windows inside a bright object score as high as the tight one and overlap it too little to be suppressed; subtracting the surround makes the tight window win. The dim object scores about 0.5 and is missed; lowering the threshold to 0.45 finds it at the cost of 15 false positives, the trade-off that the precision–recall curve and average precision summarize. Over seeds 0–7 the contrast detector always found the three bright objects and missed the dim one, with 0–2 false positives after NMS.

Datasets and Benchmarks

• PASCAL VOC (Everingham et al., 2010): an annual challenge from 2005 that started with four classes and used 20 from VOC2007, matching detections at IoU above 0.5.
• COCO (Lin et al., 2014): 91 object categories and 2.5 million labeled instances in 328,000 images, with per-instance masks; the 2014 release has masks for 80 of the categories, the 80 used in its detection evaluation.
• ImageNet: the ILSVRC detection task had 200 classes.
• KITTI: 2D and 3D detection from a driving car, with an IoU threshold of 0.7 for cars and 0.5 for pedestrians and cyclists.
• Open Images V4 (Kuznetsova et al., 2020): 15.4 million boxes for 600 classes on 1.9 million images.
• Objects365 (Shao et al., 2019): 365 categories, over 600,000 training images and over 10 million boxes.
• LVIS (Gupta et al., 2019): large-vocabulary instance segmentation on COCO images, with a long tail of categories that have few examples; the paper describes a planned dataset of over 1,000 categories in 164,000 images, and the released versions report their own counts.

Evaluation Metrics

Detections are matched to ground truth by IoU, giving the counts from which precision and recall follow. Sweeping the score threshold traces a precision–recall curve per class; average precision (AP) summarizes it, and the mean over classes is mAP. VOC uses one IoU threshold of 0.5. COCO’s primary metric averages AP over ten IoU thresholds from 0.50 to 0.95 in steps of 0.05, which rewards precise localization, and it also reports AP for small, medium and large objects. KITTI uses class-dependent thresholds and difficulty levels. Plain accuracy is not used: a detector has no fixed set of negatives, since any box that is not an object would count as one.

Applications

In tracking-by-detection, a detector runs on every frame and a tracker such as SORT links its boxes into trajectories, and the SORT article shows how strongly tracking accuracy depends on the detector. Driving systems detect vehicles, pedestrians and cyclists in 2D and 3D. Other uses include face detection in cameras, counting and inspection in industry and retail, cell and lesion detection in medical images, and aerial imagery, where oriented boxes are common.

Open Problems

• Small and crowded objects. Reported AP on small objects is far below AP on large ones (for DETR on COCO, 20.5 against 61.1), and heavily overlapping objects are still lost to duplicate removal or confused with each other.
• Long-tailed and open vocabularies. Rare classes have few examples, and fixed label sets do not cover what deployed systems meet; open-vocabulary detection addresses this with language supervision, and how far it generalizes to classes never annotated with boxes is an open question.
• Robustness and domain shift. Detectors trained on one dataset or sensor degrade under other weather, lighting, viewpoints or sensors.
• Speed–accuracy trade-off on embedded hardware.
• Evaluation. AP depends on the IoU threshold, the matching rule and annotation conventions, and annotated boxes themselves vary by a few pixels, which matters most for small objects.