Rapid Object Detection using a Boosted Cascade of Simple Features

“Rapid Object Detection using a Boosted Cascade of Simple Features” by Paul Viola of Mitsubishi Electric Research Labs (MERL) and Michael Jones of Compaq’s Cambridge Research Laboratory was published at the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) in 2001, held in Kauai, Hawaii. It describes a frontal face detector that processed $384 \times 288$ greyscale images at 15 frames per second on a 700 MHz Pentium III, with detection rates the authors called comparable to the best published systems, and it presents the detector as the product of three ideas: the integral image, AdaBoost used to select a few rectangle features from a very large set, and a cascade of increasingly complex classifiers that discards most background windows after a few feature evaluations. A longer journal version, “Robust Real-Time Face Detection”, appeared in the International Journal of Computer Vision (IJCV) in 2004. The method as it is understood today, including the boosting procedure, the cascade’s rate and cost formulas, its variants and OpenCV’s implementation, is covered in the Viola–Jones article; this article covers the papers as publications.

Problem

The paper claims detection and false-positive rates equivalent to the best published face detectors of the late 1990s and distinguishes itself from them by speed. The neural-network detector of Rowley, Baluja and Kanade (1998), which classified every window of an image pyramid, is the main reference point: in a footnote the authors say it is widely considered the fastest detection system and has been heavily tested on real-world problems, and Rowley supplied them with implementations for direct comparison. The other systems named are Sung and Poggio’s (1998) mixture-of-Gaussians model, Osuna and colleagues’ support vector machine, Schneiderman and Kanade’s (2000) statistical detector, and the SNoW detector of Yang, Roth and Ahuja (1999), which used the Winnow learning rule over a very large binary feature set.

Two earlier lines of work supplied the ingredients. Papageorgiou, Oren and Poggio (1998) had classified windows with an overcomplete set of Haar wavelets; Viola and Jones describe their own features as reminiscent of those Haar basis functions, and the journal version calls Papageorgiou et al.’s selection of 37 of 1,734 features, based on feature variance, a significant reduction that still leaves many features to evaluate per window. Focus-of-attention models, from Tsotsos and colleagues and Itti, Koch and Niebur to the coarse-to-fine methods of Amit and Geman and of Fleuret and Geman (2001), rest on the notion, in the paper’s words, that it is often possible to rapidly determine where in an image an object might occur, so that expensive processing can be reserved for those regions. The paper credits Amit and Geman with the basic insight, but argues that their approach must first evaluate and group feature detections at every location, and that Fleuret and Geman’s detector, which learns from density estimation rather than discriminatively, reported no quantitative detection and false-positive results; Viola and Jones note that its example images each showed 2 to 10 false positives.

The authors also point out that fast systems of the time relied on extra cues, frame differences in video or skin colour in colour images. Their goal was speed from a single greyscale image, with those cues left available as an addition.

Contribution

The abstract and introduction of both versions name three contributions:

1. The integral image, an image representation from which any rectangle sum takes four array references, so that rectangle features can be evaluated at any position and scale in constant time. The paper notes the close relation to summed-area tables in computer graphics and keeps a different name to stress the use for image analysis. The Haar-like features article covers the representation and the features.
2. AdaBoost as feature selection. Following Tieu and Viola’s work on image retrieval, each weak classifier is restricted to a single rectangle feature, so each boosting round selects one feature. The paper argues that this aggressive selection matters because the set of rectangle features in a $24 \times 24$ window is far larger than the number of pixels.
3. The attentional cascade, a sequence of boosted classifiers with thresholds lowered so that almost no faces are rejected, which the conference abstract describes as an object-specific focus-of-attention mechanism. Each stage is trained on the windows that pass all earlier stages.

The conference version frames these as a framework for visual object detection, demonstrated on and partly motivated by faces. The journal version’s abstract calls it a face detection framework.

Method

The method in the papers matches the one described in the Viola–Jones article: three kinds of rectangle features (two-, three- and four-rectangle) in a $24 \times 24$ base window; discrete AdaBoost over single-feature threshold classifiers (Table 1 in both versions); stages whose AdaBoost threshold is lowered to keep nearly all faces; and stages added, each trained on the false positives of the cascade so far, until the overall targets are met. Detection scales the detector rather than the image, by factors of 1.25, and shifts it by $[s\Delta]$ pixels at scale $s$; windows are variance-normalized using a second integral image of the squared pixels; and overlapping detections are merged by averaging the corners of each group of overlapping boxes, as described in sliding window detection.

Both versions report that the first two features AdaBoost selected for faces are easy to read. The first compares a horizontal band across the eyes with a band across the upper cheeks, using the observation that the eye region is usually darker. The second compares the eyes with the brighter bridge of the nose. The features selected in early rounds had error rates between 0.1 and 0.3, and those in later rounds between 0.4 and 0.5.

As a preliminary result, a single boosted classifier with 200 features detected 95% of faces at a false-positive rate of 1 in 14,084 and scanned a $384 \times 288$ image in 0.7 s. The journal version adds the ROC curve of this classifier and the observation that a practical detector needs a false-positive rate closer to 1 in 1,000,000. Adding features to one classifier makes every window more expensive; the cascade was the authors’ answer.

Results

Training data and the trained cascade

The face training set had 4,916 hand-labelled faces from images collected during a random crawl of the web, scaled and aligned to $24 \times 24$ pixels. The conference paper adds the vertical mirror image of each face, for 9,832 training faces. The journal paper describes the cropping: a person drew a box from just above the eyebrows to about halfway between the mouth and the chin, the box was enlarged by 50%, and the result was cropped and scaled without aligning the eyes, so that the examples contain more of the head than Rowley et al.’s or Sung and Poggio’s. More tightly cropped $16 \times 16$ examples had given slightly worse results.

Negatives came from several thousand images manually checked to contain no faces, with about 350 million sub-windows between them; the first stage was trained on random windows from these images, and every later stage on the false positives of the partial cascade. The final detector had 38 layers. In the conference paper’s results section its first five layers have 1, 10, 25, 25 and 50 features, with more in later layers and 6,061 in total. The journal paper lists 2, 10, 25, 25, 50, 50 and 50 features, then layers of varying size, 6,060 in total; it says the number of features per layer for the first seven layers was chosen with human intervention to reduce training time, and that in later layers 25 features were added at a time before testing on the validation set. The journal paper also reports that training the 38-layer detector took on the order of weeks on a single 466 MHz AlphaStation XP900, and that the authors had since parallelized it to train a complete cascade in about a day.

Speed

On the MIT+CMU test set, the detector evaluated on average 10 of its 6,061 features per window in the conference paper and 8 of 6,060 in the journal paper, because most windows are rejected by the first or second layer. Both versions report about 0.067 s for a $384 \times 288$ image on a 700 MHz Pentium III, with a starting scale of 1.25 and $\Delta = 1.5$, and call this roughly 15 times faster than the Rowley–Baluja–Kanade detector and about 600 times faster than the Schneiderman–Kanade detector. The 15 frames per second of the abstract matches $1/0.067 \approx 15$ (our arithmetic). Both versions also report two frames per second on a low-power 200 MIPS StrongARM processor without floating-point hardware; the conference paper names the device, a Compaq iPaq handheld.

The paper does not say which Schneiderman–Kanade detector or timing the factor of 600 refers to. Schneiderman and Kanade’s own paper gives about one minute to search a $320 \times 240$ image over four octaves of size with its wavelet-based detector, and about 5 s with an optimized version of their earlier frontal detector; by our arithmetic, 600 times 0.067 s is about 40 s.

The journal version adds an argument for the integral image: computing a 12-level pyramid of a $384 \times 288$ image with bilinear interpolation took about 0.05 s on the same processor, about as long as the whole detection, so any detector that needs such a pyramid must be slower.

Accuracy on MIT+CMU

The test set is Rowley et al.’s frontal face set, called MIT+CMU in the paper: 130 images with 507 labelled frontal faces, which the face detection article describes. The authors plot the number of false detections rather than the false-positive rate, to compare with other systems, and obtain the ROC curve by sweeping the threshold of the last layer and then removing layers one at a time. The conference paper states that its results are for $\Delta = 1.0$ and a starting scale of 1.0, which scans 75,081,800 sub-windows; the journal paper plots ROC curves for two settings and does not say which one its table uses. Table 2 of the conference paper and Table 3 of the journal paper give these detection rates:

Detector	10	31	50	65	78	95	167	422
Viola–Jones	76.1%	88.4%	91.4%	92.0%	92.1%	92.9%	93.9%	94.1%
Viola–Jones (voting)	81.1%	89.7%	92.1%	93.1%	93.1%	93.2%	93.7%	–
Rowley–Baluja–Kanade	83.2%	86.0%	–	–	–	89.2%	90.1%	89.9%
Schneiderman–Kanade	–	–	–	94.4%	–	–	–	–
Roth–Yang–Ahuja	–	–	–	–	(94.8%)	–	–	–

The column for 422 false detections appears only in the journal table. Dividing by the number of windows scanned (our arithmetic), 10 false detections correspond to a false-positive rate of about $1.3 \times 10^{-7}$ per window and 167 to about $2.2 \times 10^{-6}$. The “voting” row combines the 38-layer detector with two similarly trained detectors by majority vote. The journal paper says the three were trained with different initial negatives, slightly different weights on negative and positive errors, and slightly different criteria for trading false positives against classifier size, and that because their errors are correlated the gain is measurable but modest.

The comparison rows need care, and the papers say so in part:

• Different operating points. Most earlier papers reported a single operating point, so the table lists Viola and Jones’s detection rate at each other system’s number of false detections. Both versions note that the Rowley–Baluja–Kanade entries come from different versions of that detector, and the journal paper adds that they are not points on one ROC curve, which, by our reading, is why the rate at 422 false detections is lower than at 167.
• Where the Rowley row comes from. None of the five Rowley–Baluja–Kanade entries appears among the Test Set 1 operating points in Table 1 of Rowley et al. (1998), or elsewhere in the text of the authors’ copy we read, which include, for example, 83.6% at 10 false detections (System 15) and 86.2% at 23 (System 11). The entry of 86.0% at 31 does match the Rowley et al. entry in Schneiderman and Kanade’s comparison table. The paper does not say whether the row was quoted or measured with the implementations Rowley supplied.
• A smaller test set for two rows. The journal paper explains that the Roth–Yang–Ahuja result, in parentheses, was reported on a subset of 125 images with 483 faces, after removing 5 images of line-drawn faces. Schneiderman and Kanade’s 94.4% at 65 false detections is from the same subset: their Table 2 marks it with the asterisk for the 125-image, 483-face set, and it is the result of an improved version of their earlier eigenvector-based detector, not of the wavelet-based detector the cited paper introduces, which scored 90.2% at 110 on the same subset. Both Viola–Jones tables print the 94.4% without parentheses under a caption for 130 images and 507 faces.
• Names. The SNoW paper lists its authors as Yang, Roth and Ahuja; the tables call the detector Roth–Yang–Ahuja.

The journal paper adds two results. On the 23-image MIT subset with 149 faces, on which Sung and Poggio had reported 79.9% at 5 false positives, the detector reached 77.8% at 5. And its ROC figure adds a second curve for the faster setting, $\Delta = 1.5$ and a starting scale of 1.25, which scans 18,901,947 sub-windows; it reports that a step of more than one pixel slightly lowers the detection rate and also the number of false positives.

Cascade against a monolithic classifier

An experiment in the journal version only, summarized in the Viola–Jones article, compares a cascade of ten 20-feature classifiers with a single 200-feature classifier trained on the union of the cascade’s examples: similar ROC curves, with the cascade nearly 10 times faster. Its first stage was trained on 5,000 faces and 10,000 random non-face windows, the second on the same faces and 5,000 false positives of the first, and so on. The authors add that without the cascade it would be hard to choose the negatives for the monolithic classifier.

Failure modes

The journal version lists failure modes observed informally: rotations beyond about ±15° in the image plane and ±45° out of plane, harsh backlighting and occluded eyes, which the Viola–Jones article also notes. It adds that a nonlinear variance normalization based on robust statistics improved detection under backlighting but was expensive in the integral image framework, and that a face with a covered mouth was usually still detected.

Versions

The work appeared in several forms:

• CVPR 2001, the cited paper, in the IEEE proceedings (pages I-511–I-518). We could not read the IEEE text, which IEEE Xplore did not serve to automated access; the details in this article come from the authors’ accepted copy, headed “Accepted Conference on Computer Vision and Pattern Recognition 2001”, and from MERL Technical Report TR2004-043, which reproduces the same text. Both still list the self-citation to Tieu and Viola as an anonymized reference (“Anonymous, 2000”), so they may differ from the printed proceedings.
• Workshop and ICCV abstracts, 2001. The NIPS 2001 paper cites a version titled “Robust Real-time Object Detection” from the IEEE Workshop on Statistical and Computational Theories of Vision, 2001, and the ICCV 2001 proceedings contain an entry, “Robust real-time face detection”, that Crossref’s metadata lists as the single page 747. We did not read either.
• IJCV 2004, “Robust Real-Time Face Detection”, received in September 2001 and accepted in July 2003, with Viola then at Microsoft Research and Jones at MERL. We read a copy of the typeset Kluwer article; Springer’s own copy is paywalled.

The journal paper is the more complete account: it adds the cascade training algorithm (its Table 2), the formulas for the cascade’s detection and false-positive rates and expected number of features, the monolithic-versus-cascade experiment, the training-set cropping, training time, the second ROC curve, the column for 422 false detections, the Sung–Poggio comparison, the failure modes, a derivation of the integral image as a double integral (connected to the “boxlets” of Simard and colleagues), and an analysis of AdaBoost’s cost against wrapper-style feature selection. The two versions also differ in numbers:

Item	CVPR 2001 (accepted copy)	IJCV 2004
Rectangle features in a $24 \times 24$ window	over 180,000	160,000
First stage	two features, 100% detection at 40% false positives (§4)	two features, 100% detection at 50% false positives (§4)
First layers of the 38-layer detector	1, 10, 25, 25, 50	2, 10, 25, 25, 50, 50, 50
Total features	6,061	6,060
Features evaluated per window on MIT+CMU	10	8
Face-free images	9,544	9,500
Negatives per layer	at most 10,000	at most 6,000 (§5.2)
Training faces	4,916 plus mirror images (9,832)	4,916 (mirroring not mentioned)
False-detection columns in the table	10 to 167	10 to 422
Handheld demonstration	Compaq iPaq	200 MIPS StrongARM processor, device not named

The conference abstract claims that the cascade, unlike earlier attention mechanisms, gives “statistical guarantees” that discarded regions are unlikely to contain the object. The journal abstract drops the claim, and its cascade section says it is not clear whether lowering a stage’s threshold preserves AdaBoost’s training and generalization guarantees.

Each version also disagrees with itself. In the conference paper, the cascade section describes a two-feature first stage, and the introduction gives it fewer than 1% false negatives at 40% false positives, but the results section lists a first layer of one feature. In the journal paper, the first stage filters out “over 50%” of the image while keeping 99% of faces in the introduction, detects 100% at 50% false positives in §4, and rejects about 50% while detecting close to 100% in §5.2; §4.3, discussing the benefit of cascades in general, speaks of the final layers searching for a set of 10,000 hard negatives, the conference paper’s figure, while §5.2 collected at most 6,000 per layer for the published detector. The journal paper calls the feature set about 400 times overcomplete, although its own 160,000 divided by the 576 pixels is about 280, and its formula for the expected number of features adds a term $n_0$ to a sum over stages $1$ to $K$, one term too many for $K$ classifiers (the Viola–Jones article gives the corrected form). Both versions print the window variance with its sign reversed, as $\sigma^2 = m^2 - \frac{1}{N}\sum x^2$ (see Haar-like features). Both cite Schneiderman and Kanade’s paper as appearing at the International Conference on Computer Vision, although it was published at CVPR 2000, and give Rowley et al.’s pages as 22–38 rather than 23–38.

The NIPS 2001 Companion Paper

“Fast and Robust Classification using Asymmetric AdaBoost and a Detector Cascade”, by the same authors at NIPS 2001, addresses a weakness of the cascade papers: AdaBoost selects features to minimize a bound on classification error, while a cascade stage needs few false negatives. It shows that raising the initial weight of the positives is only somewhat effective, because the first weak classifier absorbs the asymmetry, and proposes spreading the asymmetric weighting over all rounds; the Viola–Jones article describes the update. It reports that 4-feature classifiers cut the false-positive rate by about 20% at 99% detection, and that complete cascades trained with the asymmetric version had about half the false positives at a 91% detection rate on MIT+CMU. Its cascade is described differently again: over 6,000,000 binary features, stages of 1, 5, 20 and 50 features at the start, on average fewer than 2 stages evaluated per window, and layers added until fewer than 100 false positives remained on a validation set, which took 34 layers with conventional boosting and 38 with the asymmetric version.

Impact

The PAMI Technical Committee lists the paper as the 2011 recipient of the Longuet-Higgins Prize, which is awarded at CVPR to CVPR papers from ten years earlier that have had significant impact on computer vision research.

The detector made real-time face detection practical on ordinary hardware. Zhang and Zhang’s 2010 survey of face detection credits it with making face detection feasible in applications such as digital cameras and photo organization software, and describes boosting-based detectors as the de facto standard in real-world applications since the paper; it notes that most digital cameras of the time had built-in face detectors for auto-focus and auto-exposure. Li et al. (2015) describe the boosted cascade with simple features as the most popular and effective design for practical face detection since the paper. OpenCV distributes boosted cascades of this kind with trained frontal face models; as of October 2026, OpenCV 4.14.0 still ships haarcascade_frontalface_default.xml, a $24 \times 24$ stump-based cascade created by Rainer Lienhart, and the Viola–Jones article covers the implementation.

The journal paper’s conclusion expected the approach to carry over to pedestrians and cars. Viola, Jones and Snow (2003) built a pedestrian detector on it that combined appearance with motion between two consecutive video frames, detected people as small as $20 \times 15$ pixels and ran at about 4 frames per second.

Limitations

• One view, one class. The detector handles upright frontal faces within the tolerances above; other views and objects need other cascades.
• Weak features in uncontrolled conditions. Li et al. (2015) describe the Haar features as relatively weak in uncontrolled environments, and the journal paper’s own failure modes include backlighting and occluded eyes.
• Hand-tuned training. The journal paper concedes that the layer sizes of the published detector were chosen partly by hand and by trial and error, and training took weeks.
• Comparisons. As described under Results, two comparison rows in the tables come from a smaller test set, and the source of the Rowley–Baluja–Kanade row is not stated.

What Came After

Rotated rectangle features, Gentle AdaBoost stages, soft cascades, WaldBoost and other refinements of the training procedure are covered in the Viola–Jones and Haar-like features articles. For people and other articulated classes, detection moved to gradient features (HOG) and part models (deformable part models).

Harder benchmarks exposed the limits of the 2001 detector. On WIDER FACE, whose authors tested detectors trained on external data on its test partition, a Viola–Jones detector obtained from the authors or from OpenCV reached an average precision of 0.412, 0.333 and 0.137 on the Easy, Medium and Hard subsets, the lowest of the four baselines, and the authors summarize that it performs poorly on all settings (Yang et al., 2016). Detectors based on convolutional neural networks replaced it, some keeping the cascade structure: Li et al. (2015) built a cascade whose stages are small CNNs, with negatives mined by the cascade so far. The face detection article follows this transition to cascaded CNNs and single-stage detectors.