You Only Look Once: Unified, Real-Time Object Detection

“You Only Look Once: Unified, Real-Time Object Detection” by Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi, of the University of Washington, the Allen Institute for AI and Facebook AI Research, was posted to arXiv on 8 June 2015 and presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Las Vegas in June 2016. It introduced YOLO, which treats object detection as a single regression problem: one convolutional network looks at the whole image once and outputs every bounding box and class probability together. This article covers the paper as a publication; the YOLO article describes the model, its loss and training in detail, and the later versions.

Problem

The paper opens with the observation that detectors of the time repurposed classifiers. Deformable part models (DPM; Felzenszwalb et al., 2010) ran one at evenly spaced positions across the image; R-CNN (Girshick et al., 2014) classified about 2,000 Selective Search proposals per image, then refined the boxes, removed duplicates and rescored them, a two-stage design. The authors argued that such pipelines are slow and hard to optimize because each component is trained separately, citing more than 40 seconds per image for R-CNN. Faster variants still fell short of real time, and the only detector they knew of that reached 30 frames per second was a GPU implementation of DPM, whose accuracy was low.

A second problem was context: a classifier that sees one window or proposal cannot use the rest of the image. The authors give Fast R-CNN (Girshick, 2015) as an example that mistakes background patches for objects because it cannot see the larger scene.

Contribution

The paper drops the classifier-based pipeline. Detection becomes a regression from pixels to box coordinates and class probabilities, computed by one network in one evaluation and trained end to end. This is the “unified” of the title: the authors describe feature extraction, box prediction, duplicate suppression and contextual reasoning as done concurrently by one network rather than by separate stages.

The introduction makes three claims for this design, and the experiments are organized to support them:

1. Speed. With no pipeline, YOLO only needs a forward pass. The authors report 45 frames per second without batching on a Titan X GPU, under 25 ms of latency for streaming video, and more than twice the mean average precision of earlier real-time systems.
2. Global reasoning. Every prediction uses features of the entire image, so the network implicitly encodes context about classes as well as their appearance. The predicted consequence is fewer false detections on background.
3. Generalizable representations. Trained on natural images and tested on artwork, YOLO is claimed to outperform DPM and R-CNN by a wide margin, and so to be less likely to break on new domains.

The authors were explicit about the cost: YOLO lagged the best detectors in accuracy and struggled to localize some objects precisely, especially small ones.

Method

The image is resized to $448 \times 448$ and divided into an $S \times S$ grid. Each cell predicts $B$ boxes with confidences and one set of $C$ class probabilities, giving an $S \times S \times (5B + C)$ output; for PASCAL VOC the paper uses $S = 7$, $B = 2$ and $C = 20$, so 98 boxes per image instead of the roughly 2,000 proposals of Selective Search. The network has 24 convolutional layers, inspired by GoogLeNet, followed by two fully connected layers; Fast YOLO has 9 convolutional layers.

Section 3 positions YOLO against other systems. Unlike DPM, it learns its features for the task instead of chaining static features, region classification and box prediction. With R-CNN it shares the idea that each grid cell proposes and scores boxes with convolutional features, but its proposals are spatially constrained, far fewer and jointly optimized. MultiBox still needs a separate classifier, and OverFeat, a sliding-window localizer, sees only local information. The grid itself, the authors say, comes from Redmon’s earlier MultiGrasp system for robotic grasp detection.

Results

The experiments use PASCAL VOC unless stated otherwise, with mAP measured at an IoU of 0.5 (see average precision).

Real-time systems (Table 1, VOC 2007 test). Fast YOLO reached 52.7% mAP at 155 frames per second and YOLO 63.4% at 45, against 26.1% for the 30 Hz DPM. Faster R-CNN with VGG-16 was 10 mAP higher than YOLO but, in the authors’ tests, six times slower. A VGG-16 version of YOLO (66.4%, 21 frames per second) was set aside as slower than real time.

Error analysis (VOC 2007). Using the methodology and tools of Hoiem et al. (2012), the authors classified each of the top detections as correct, a localization error, a confusion with a similar or other class, or background. Fast R-CNN made far fewer localization errors (8.6% against 19.0%) but far more background errors (13.6% against 4.75%). The lower background rate supports the second claim; YOLO’s localization errors, which outnumbered all its other errors combined, are its main weakness.

Combining with Fast R-CNN (Table 2). Fast R-CNN detections that YOLO also predicted received a boost depending on YOLO’s probability and the overlap. On VOC 2007 test, the best Fast R-CNN model rose from 71.8% to 75.0% mAP, while combining it with other Fast R-CNN variants gained only 0.3 to 0.6 points. The authors read this as evidence that YOLO’s mistakes are of a different kind, not a generic ensembling effect.

VOC 2012 leaderboard (Table 3). YOLO scored 57.9% mAP on VOC 2012 test, below the state of the art and close to R-CNN with VGG-16; the authors point to small-object classes such as bottle, sheep and tv/monitor. The Fast R-CNN + YOLO combination scored 70.7%, which the paper places fourth on the public leaderboard as of 6 November 2015, five places above Fast R-CNN alone.

Artwork (Figure 5). For the third claim, person detectors trained on natural images were tested on the Picasso dataset and the People-Art dataset. R-CNN’s person AP fell from 54.2 on VOC 2007 to 10.4 on Picasso, DPM degraded less but started lower, and YOLO went from 59.2 to 53.3 on Picasso and scored 45 on People-Art. The authors attribute this to YOLO modeling object size, shape and layout, which artwork and photographs share even though their pixels differ.

In the wild. Connected to a webcam, YOLO stayed real-time including capture and display, and the authors note that it behaves like a tracker although it processes frames independently.

Impact

The paper states that all training and testing code is open source, with pretrained models to download and a webcam demo on the project page. Redmon distributed YOLO as part of Darknet, his open-source neural-network framework in C and CUDA.

The paper also established the template of the one-stage detector: no proposal stage, dense prediction from a shared feature map, and a single loss. SSD (Liu et al., 2016) presents itself as faster and significantly more accurate than YOLO, the previous state of the art among single-shot detectors; its authors note that using the whole topmost feature map with fully connected predictors, and no explicit aspect ratios, SSD would approximately reproduce YOLO.

The PAMI Technical Committee lists the paper, with “Deep Residual Learning for Image Recognition”, as a 2026 winner of the Longuet-Higgins Prize, which recognizes CVPR papers from ten years earlier that had significant impact on computer vision research.

Limitations

Section 2.4 of the paper lists the model’s limitations:

• Spatial constraints. Each cell predicts only two boxes and one class, limiting how many nearby objects can be detected; small objects in groups, such as flocks of birds, are the stated failure case.
• Unusual shapes. Because boxes are learned from data, objects in new or unusual aspect ratios or configurations generalize poorly.
• Coarse features. Repeated downsampling leaves relatively coarse features for predicting boxes.
• The loss. Errors in small and large boxes are treated alike, although the same error costs much more IoU on a small box. The authors name incorrect localization as the main source of error.

Two qualifications follow from the experiments: YOLO was almost 10 mAP below Faster R-CNN with VGG-16 on VOC 2007, and the 75.0% and 70.7% results come from an ensemble that runs both models and so gives up YOLO’s speed.

What Came After

Redmon and Farhadi’s YOLO9000 paper (CVPR 2017) starts from this paper’s error analysis: it cites YOLO’s many localization errors and relatively low recall compared with region-proposal methods, and responds with batch normalization, anchor boxes, a new backbone and multi-scale training, plus joint training on detection and classification data to cover more than 9,000 classes. YOLOv3 (Redmon & Farhadi, 2018), an informal arXiv report, added predictions at three scales and a residual backbone. The versions from other groups since YOLOv4 (2020) are surveyed in the YOLO article. In our reading, they keep this paper’s single-pass, proposal-free design and its focus on real-time speed, but little of its model: the fully connected head and the $7 \times 7$ grid were already gone in YOLOv2.