Face Detection

Face detection finds every face in an image and returns, for each one, a box, a confidence score and, in most modern detectors, a few facial landmarks such as the eye centres and mouth corners. It is object detection with a single class, and the WIDER FACE authors call it a critical step for all facial analysis, including face alignment, recognition, verification and parsing (Yang et al., 2016). A single class does not make it easy. Faces in photographs range from a few pixels to the whole frame, turn away from the camera, hide behind hands, hair and masks, and blur with motion, while the background contains countless face-like patterns.

The field has gone through the same stages as generic detection: neural networks scanning windows (Rowley, Baluja and Kanade, 1998), the boosted cascade of Viola and Jones (2001), part-based models, cascades of small convolutional networks (2015–2016), and single-stage detectors with anchors on feature pyramids (2017 onward), with compact versions of the last for phones. Two benchmarks shaped how results are reported: FDDB, scored with ROC curves of detection rate against the number of false positives, and WIDER FACE, scored with average precision on three nested difficulty settings. This article covers the task, the families of methods, both benchmarks’ protocols as their official code computes them, and deployment. Recognizing who a detected face belongs to, with alignment, embeddings and verification, is face recognition and is not covered here.

Problem Definition

Given an image, a face detector outputs a set of detections $\{(\mathbf{b}_i, s_i, \mathbf{l}_i)\}_{i=1}^{N}$, where $\mathbf{b}_i$ is a box (or, on FDDB, possibly an ellipse), $s_i$ a confidence score and $\mathbf{l}_i$ an optional vector of landmark coordinates. As in generic detection, $N$ varies from image to image and each face should be reported once; a detection counts as correct when its overlap with an annotated face, measured by intersection over union (IoU), passes a threshold of 0.5 in both major benchmarks (FDDB requires more than 0.5, WIDER FACE at least 0.5).

What counts as a face is a matter of annotation policy, and benchmarks differ in it. FDDB’s annotators were told to reject a region in which neither eye (nor a pair of glasses) was visible, faces looking away from the camera were labelled as non-faces, and regions under 20 pixels in height or width were excluded (Jain and Learned-Miller, 2010). WIDER FACE labels every “recognizable” face, including occluded ones, and gives an ignore flag to faces 10 pixels or smaller that are too hard to recognize (Yang et al., 2016). MAFA, a benchmark of masked faces, ignores faces whose box side is under 32 pixels, and detections of ignored faces count neither as true nor as false positives (Ge et al., 2017). A detector is therefore always judged against a definition, and the same detector can look better or worse depending on whose definition is used.

Input and Output

The input is a still image or a video frame, colour or greyscale. Training data are images with boxes and, for many detectors, landmarks.

The box itself is a convention. WIDER FACE requires a box that tightly contains the forehead, chin and cheeks (Yang et al., 2016). FDDB annotates an ellipse whose major axis runs from the chin to the top of the head and whose sides follow the boundary between face and ears, so that ears and hair are excluded (Jain and Learned-Miller, 2010). AFW, built by Zhu and Ramanan (2012), gives each face a box, six landmarks and a discretized viewpoint. Mathias et al. (2014) list how datasets differ: whether a statue’s head is a face, whether a head rotated more than 90° is a face, whether the box spans the facial landmarks or the whole head, where a profile box is centred, and the minimum and maximum annotated size. In their preliminary experiments, adjusting a detector’s output to the annotation style of the test set was key to competitive results.

Landmarks are now a common second output. MTCNN predicts five: the two eyes, the nose and the two mouth corners (Zhang et al., 2016). RetinaFace predicts the same five points and, in its CVPR version, the image-plane projections of 1,000 vertices of a 3D face mesh (Deng et al., 2020). BlazeFace predicts six: the eye centres, the ear tragions, the mouth centre and the nose tip (Bazarevsky et al., 2019). OpenCV’s FaceDetectorYN returns 15 numbers per face: the box’s top-left corner, width and height, five landmarks and a score (OpenCV 4.14.0). Downstream, the landmarks are used to rotate and scale the face to a canonical position before recognition or analysis, which is why detection and landmark localization are often learned together.

Variants

• Pose coverage. Jain and Learned-Miller (2010) sorted detectors into single-pose (upright frontal or profile), rotation-invariant (in-plane rotation allowed), multi-view (out-of-plane rotations binned into a predetermined set of views) and pose-invariant ones (no restriction on head orientation). Rowley et al.’s and Viola and Jones’s detectors were single-pose, for upright frontal faces.
• Joint detection and alignment. The detector also outputs landmarks, as MTCNN, RetinaFace and BlazeFace do; Zhu and Ramanan (2012) went further and estimated head pose in the same model. Facial landmark detection as a task of its own, with dense landmark sets, starts from a detected face.
• Occluded and masked faces. MAFA concentrates on faces covered by masks, hands or other objects (Ge et al., 2017).
• Tiny faces. Crowd scenes contain faces of a few pixels, which Hu and Ramanan (2017) treat as a problem of scale, resolution and context.
• Real-time and on-device detection. Detectors for phone cameras and video, such as BlazeFace and YuNet, trade accuracy for millisecond latency.

Challenges

• Scale. In WIDER FACE’s comparison of datasets, 50% of its faces are 10 to 50 pixels tall, against 8% in FDDB and 12% in AFW (Yang et al., 2016, Table 1). Small faces have little detail, and for an anchor-based detector they occupy few positions of a strided feature map; S³FD’s authors name a lowest detection stride of 8 or 16 pixels as one reason that anchor-based detectors miss small faces (Zhang et al., 2017).
• Pose. WIDER FACE labels a face atypical when its roll or pitch exceeds 30° or its yaw exceeds 90°; in the dataset paper’s baselines, the best detector recalled under 20% of atypical faces (Yang et al., 2016).
• Occlusion. On the MAFA test set, the authors’ LLE-CNNs, the best of the seven detectors compared, reached 76.4% AP over all masked faces but 22.5% on heavily occluded ones, where all four facial regions (eyes, nose, mouth, chin) are covered (Ge et al., 2017, Table 1).
• Blur, illumination and low resolution. FDDB’s annotators met regions where low resolution, occlusion and pose made it ambiguous whether a face was present at all (Jain and Learned-Miller, 2010).
• Background imbalance. A detector must reject far more candidate windows than it accepts. In S³FD, over 99.8% of the preset anchors are negatives, and about 75% of the 34,125 anchors of a $640 \times 640$ image come from the layer with the smallest ($16 \times 16$) anchors, the one that detects small faces (Zhang et al., 2017).
• Annotation conventions. Box styles, ellipses versus rectangles and missing labels on hard faces all change the measured accuracy, as the Evaluation Metrics section shows.
• Speed. Cameras and phones need many frames per second on limited hardware, which shaped the cascade designs and the mobile detectors.

Approaches

Neural networks on sliding windows

Rowley, Baluja and Kanade (1998) built an upright frontal face detector from small neural networks applied to every $20 \times 20$ window of a greyscale image, at every pixel position and on an image pyramid with a scale factor of 1.2 (the scan itself is covered in sliding window detection). Each window was corrected for lighting by subtracting a best-fit linear function inside an oval mask and then histogram-equalized. The networks had retinal connections: hidden units looked at $10 \times 10$ and $5 \times 5$ subregions and at overlapping $20 \times 5$ horizontal stripes, the stripes meant to catch features such as a pair of eyes or a mouth. Nearly 1,050 faces were aligned by their labelled eyes, nose and mouth, and each produced fifteen training examples by random rotations of up to 10°, scalings between 90% and 110%, sub-pixel translations and mirroring. Negatives came from a bootstrap procedure, credited to Sung’s thesis: the network was retrained repeatedly on the false detections it produced on 120 face-free scenery images, so that about 8,000 non-face windows were chosen from 146 million.

Two networks trained from different random initial weights, initial non-face sets and orders of scenery images made different mistakes, and arbitrating between them removed many false detections. On Test Set 1 of the paper, 130 images with 507 frontal faces requiring 83,099,211 windows, a single network with merging heuristics detected 90.5% of the faces with 570 false detections; requiring two networks to agree (System 10) gave 77.9% with 2 false detections, and letting either network detect (System 12) gave 90.3% with 185 (Rowley et al., 1998, Table 1).

Boosted cascades

Viola and Jones (2001) replaced the network by a cascade of boosted classifiers on Haar-like features that rejects most windows after a few feature evaluations, and processed $384 \times 288$ images at 15 frames per second; the Viola–Jones article covers the algorithm, its face results and its successors. WIDER FACE’s authors group the detectors that followed into cascade-based, part-based, channel-feature-based and neural-network-based methods, and among their four baselines is aggregated channel features (ACF), a boosted cascade on gradient-histogram, integral-histogram and colour channels that Yang et al. applied to faces (Yang et al., 2016). FDDB’s report used OpenCV’s Viola–Jones implementation as a baseline, with a scale factor of 1.2 and a minimum of 0 neighbours (Jain and Learned-Miller, 2010).

Part models and rigid templates

Deformable part models brought gradient features and flexible geometry to faces. Zhu and Ramanan (2012) modelled every facial landmark as a part, connected the parts in a tree, and used a mixture of trees with a shared pool of part templates to cover viewpoint, with 13 viewpoints and 5 frontal expressions giving 18 mixtures, so that one model detected the face, estimated its pose and located its landmarks. On their AFW test set (205 images, 468 faces in the paper; later papers give 473) and on faces larger than $150 \times 150$ pixels, AP rose from 77.4% for rigid multiview HOG templates to 87.8% with latent star-structured parts, 88.6% with supervised star parts and 92.9% with the tree model with independent parts (92.2% with all parts shared; their Figure 6b).

Mathias et al. (2014) then reported two surprises. A properly trained vanilla DPM obtained top performance across the datasets they considered, and HeadHunter, a detector of rigid templates similar in structure to Viola and Jones’s, reached similar performance. They argued that evaluation on the existing face datasets was biased by their different annotation guidelines, re-annotated PASCAL Faces to a uniform box policy (AFW’s boxes needed no major edits), added overlooked and hard faces to both, mostly with ignore flags, and proposed a global rigid adjustment of each method’s boxes to the dataset’s style before scoring. Li et al. (2015) later quote APs on AFW of 97.21% for this DPM and 97.14% for HeadHunter (their Figure 7).

Cascades of CNNs

Li et al. (2015) kept the cascade but made its stages convolutional networks. A small 12-net scans the image densely for faces of a chosen minimum size; a 12-calibration-net then shifts and rescales each surviving window by classifying it into one of 45 discrete scale-and-offset patterns; a 24-net and 24-calibration-net and a 48-net and 48-calibration-net follow, with non-maximum suppression between stages. Each stage’s negatives are mined from background images with the cascade so far, and the thresholds after the first two and the first four networks are set to keep 99% and 97% recall. On FDDB the average number of windows per image fell from 5,341.8 for the sliding window to 3.6 at the end, while recall fell from 95.9% to 85.1% (their Table 1). The detector ran at 14 frames per second on one CPU core for VGA images and 100 on a GPU.

MTCNN (Zhang et al., 2016; the details here are from the arXiv version submitted to IEEE Signal Processing Letters) made the stages multi-task. A fully convolutional Proposal Network (P-Net) scans an image pyramid and regresses box offsets, a Refine Network (R-Net) rejects most candidates and refines the boxes, and an Output Network (O-Net) refines them again and predicts the five landmarks; non-maximum suppression follows each stage. Each network is trained with a face/non-face cross-entropy loss, a Euclidean box-regression loss and a Euclidean landmark loss, with crops labelled negative below IoU 0.3 with every face, positive above 0.65 and part faces between 0.4 and 0.65, the part faces serving box regression only. Online hard sample mining back-propagates only the 70% of samples with the highest loss in each mini-batch. In the arXiv version’s Figure 4, MTCNN’s AP on the WIDER FACE Easy, Medium and Hard subsets was 0.851, 0.820 and 0.607, without a statement of the split. Hu and Ramanan (2017) list the same 0.607 for the Hard test set and 0.848, 0.825 and 0.598 on validation, so in our reading the figure shows test-set results. MTCNN ran at 16 frames per second on a 2.60 GHz CPU and 99 on a GPU in unoptimized MATLAB code.

Detectors adapted from generic object detection

S³FD’s authors describe a second route alongside CNN features in classical pipelines: treating face detection as a special case of generic object detection and inheriting its techniques. The detectors below take that route (see one-stage object detectors), redesigned for small faces. Hu and Ramanan (2017) studied tiny faces directly. They trained separate templates for different face sizes on a shared deep feature hierarchy, found that faces under 40 pixels tall were best found with templates applied to the image upsampled twofold and faces over 140 pixels with templates at half resolution, and gave small-face templates receptive fields far larger than the face, about $300 \times 300$ pixels, after a human experiment in which a fixed 300-pixel context window cut people’s error in classifying small faces by 20%. Their hybrid-resolution detector reached 0.819 AP on the WIDER FACE Hard test set, where earlier methods had 0.29 to 0.64.

S³FD (Zhang et al., 2017) is an SSD-style detector built on VGG16. It tiles square anchors of 16 to 512 pixels on six layers with strides from 4 to 128 pixels, each anchor four times its layer’s stride so that anchors of every scale have the same density on the image (the “equal-proportion interval” principle), and sized to the layer’s effective rather than theoretical receptive field. Because anchor scales are discrete while face sizes are continuous, faces between anchor sizes matched few anchors; a two-stage scale compensation matching lowers the IoU threshold from 0.5 to 0.35 and then gives each still-unmatched face its top-$N$ anchors with IoU above 0.1. A max-out background label predicts several background scores for each of the smallest anchors and keeps the highest, to suppress the false positives they produce. In the ablation on the WIDER FACE validation set, the framework alone raised Hard AP from 73.7% (an RPN-style face detector) and 71.6% (SSD-style) to 82.3%, and the two other components to 85.2% (their Table 3).

DSFD (Li et al., 2019) added a second “shot” of enhanced feature maps on top of the first, computed by a feature enhance module, with a progressive anchor loss that uses smaller anchors for the first shot than for the second and an improved anchor matching coupled to data augmentation. RetinaFace (Deng et al., 2020) builds on a feature pyramid from P2 to P6 with context modules made of deformable convolutions and two cascaded regression heads, the second refining the first one’s boxes. Anchors are square, at a scale step of $2^{1/3}$, covering 16 to 406 pixels for a $640 \times 640$ input: 102,300 anchors, 75% of them on P2. Its distinguishing idea is extra supervision. The authors annotated five landmarks for 84.6k training faces and 18.5k validation faces of WIDER FACE, generated 3D mesh annotations semi-automatically, and trained box, landmark and mesh regression together. In their ablation (ResNet-50), adding landmark and mesh regression raised WIDER FACE validation Hard AP from 90.642% to 91.447%, and the AP averaged over IoU thresholds 0.5:0.05:0.95 on the Hard test set from 54.20% to 55.02%; the authors show that the extra losses lower the scores of high-confidence false positives. With ResNet-152, the CVPR version reports 91.7% AP on the Hard test set; the earlier arXiv version, which pairs the landmarks with a self-supervised mesh decoder instead of 3D vertex regression, reports 91.4%.

TinaFace (Zhu et al., 2020) argued that face detection needs nothing beyond generic detection. It combines a ResNet-50 with a six-level feature pyramid, an Inception-style module, focal loss, a DIoU box-regression loss, an IoU-aware prediction branch and deformable convolutions, all existing modules, and reported 92.1% AP on the WIDER FACE Hard test set with a single model at a single scale, and 92.4% with test-time augmentation (arXiv v3, Table 2).

Detectors for phones and embedded devices

BlazeFace (Bazarevsky et al., 2019) targets the front and rear cameras of phones. Its $128 \times 128$ front-camera model uses a MobileNet-like feature extractor, stops at an $8 \times 8$ feature map with 6 anchors per cell instead of going down to $2 \times 2$, and replaces non-maximum suppression by blending: overlapping predictions are averaged with weights instead of discarded, which the authors report raised accuracy by 10% and cut a jitter metric by 40% on their front-camera data. On a private test set of 2,000 images, with only faces covering more than 20% of the image counted, it reached 98.61% AP at IoU 0.5 in 0.6 ms on an iPhone XS GPU, against 97.95% and 2.1 ms for a MobileNetV2-SSD with the same anchors (their Table 1). The numbers come from the authors’ own data and are not comparable with WIDER FACE.

YuNet (Wu, Peng and Yu, 2023) is the compact detector that OpenCV’s FaceDetectorYN runs; the class’s documentation links to the YuNet model in the OpenCV Zoo. The OpenCV Zoo model card notes that, because of its training scheme, it detects faces of roughly $10 \times 10$ to $300 \times 300$ pixels, and its evaluation table gives 0.8844, 0.8656 and 0.7503 AP on the WIDER FACE validation Easy, Medium and Hard settings, while the card’s opening sentence quotes 0.834, 0.824 and 0.708 (as of October 2026). The Deployment section describes how OpenCV runs it.

Reported results

The table collects WIDER FACE results as their papers print them. Test-set results come from the organizers’ evaluation, validation results from the official code; most top entries use multi-scale testing, so differences of a point or less mix model and test-time choices.

Detector	Year	Family	Validation Easy / Medium / Hard (AP %)	Test Easy / Medium / Hard (AP %)
MTCNN (Zhang et al.)	2016	CNN cascade with landmarks	—	85.1 / 82.0 / 60.7 (arXiv Fig. 4; split not stated)
HR (Hu and Ramanan)	2017	Scale-specific templates with context	91.9 / 90.8 / 82.3 (ResNet-101)	— / — / 81.9
S³FD (Zhang et al.)	2017	Single-stage, anchors on six layers	93.7 / 92.4 / 85.2	92.8 / 91.3 / 84.0
DSFD (Li et al.)	2019	Dual-shot single-stage	96.6 / 95.7 / 90.4	96.0 / 95.3 / 90.0
RetinaFace (Deng et al.)	2020	FPN, landmark and mesh supervision	96.7 / 96.1 / 91.4 (ResNet-50)	— / — / 91.7 (ResNet-152; arXiv version 91.4)
TinaFace (Zhu et al.)	2020	Generic detector components	96.3 / 95.7 / 93.0 (97.0 / 96.3 / 93.4 with TTA)	95.2 / 94.7 / 92.1 (95.8 / 95.3 / 92.4 with TTA)
YuNet (OpenCV Zoo)	2023	Compact, anchor-free decoding	88.4 / 86.6 / 75.0	—

Example: a template detector scored by both protocols

The example builds the simplest face detector, one template, and scores it the way the two benchmarks would. Images have intensities in $[0, 1]$. Faces are synthetic: an elliptical head of width $0.8h$ and height $h$ with two dark eye bars and a dark mouth bar, random skin brightness and contrast, rendered with $4 \times 4$ supersampling so that the parts of a 10-pixel face blur together. Each $160 \times 160$ scene holds six non-overlapping faces with heights drawn log-uniformly from 10 to 64 pixels, on clutter of rectangles, blobs, lines and face-like decoys (a bright ellipse with a single dark band), plus Gaussian noise with $\sigma = 0.03$. The ground truth box is the head’s bounding box, and the head ellipse serves as the FDDB-style annotation.

The detector is the mean of 300 synthetic faces drawn at the window size, a $w \times h = 20 \times 24$ window, scored by normalized cross-correlation at stride 1 on a 12-level pyramid that scales the image by $2.4 / 1.2^k$, so that faces from about 10 to 75 pixels tall fill the window at some level. Local maxima above 0.3 are kept, followed by greedy non-maximum suppression at IoU 0.3 and at most 100 detections per image. Evaluation follows the official code of each benchmark: WIDER FACE’s matching, min–max score normalization, 1,000 thresholds and all-point AP, with faces outside a height range ignored as WIDER’s settings ignore faces; and FDDB’s Hungarian matching on pixel-mask overlaps with discrete and continuous scores, read at 100 false positives. Two simplifications keep the script short: the IoU omits the one-pixel convention of WIDER FACE’s code, and the FDDB curve is swept over 150 score quantiles, where the official code uses every distinct detection score as a threshold. FDDB detections are submitted either as the rectangles or as the ellipses inscribed in them. Five seeds each draw a template and 40 test scenes; the script runs in about 10 s.

import numpy as np
from scipy import ndimage, signal
from scipy.optimize import linear_sum_assignment

TH, TW = 24, 20                     # detection window: 20 x 24 pixels (w x h)

# ---------- synthetic scenes with intensities in [0, 1] ----------
def clutter(rng, H, W):
    """Gradient background with rectangles, blobs, lines and face-like decoys
    (a bright ellipse with one dark band)."""
    y, x = np.mgrid[0:H, 0:W]
    img = rng.uniform(0.2, 0.6) + rng.uniform(-0.3, 0.3) * x / W + rng.uniform(-0.3, 0.3) * y / H
    for _ in range(H * W // 250):
        kind, cx, cy, v = rng.integers(4), rng.uniform(0, W), rng.uniform(0, H), rng.uniform()
        if kind == 0:
            w, h = rng.uniform(2, 24, 2); img[(abs(x - cx) < w / 2) & (abs(y - cy) < h / 2)] = v
        elif kind == 1:
            a, b = rng.uniform(2, 16, 2); img[((x - cx) / a) ** 2 + ((y - cy) / b) ** 2 <= 1] = v
        elif kind == 2:
            t = rng.uniform(0, np.pi)
            img[abs((x - cx) * np.sin(t) - (y - cy) * np.cos(t)) < rng.uniform(0.5, 2)] = v
        else:
            b = rng.uniform(5, 30); a = 0.8 * b
            head = ((x - cx) / a) ** 2 + ((y - cy) / b) ** 2 <= 1
            img[head] = rng.uniform(0.55, 0.9)
            img[head & (abs(y - cy - rng.uniform(-b, b) / 2) < rng.uniform(0.05, 0.15) * b)] = rng.uniform(0, 0.4)
    return img

def draw_face(img, cx, cy, h, rng, ss=4):
    """Face of height h centred at (cx, cy): ellipse head (width 0.8 h), two dark eyes
    and a dark mouth, rendered with ss x ss supersampling so that small faces blur."""
    x0, y0 = max(int(cx - 0.45 * h), 0), max(int(cy - 0.55 * h), 0)
    x1 = min(int(np.ceil(cx + 0.45 * h)), img.shape[1])
    y1 = min(int(np.ceil(cy + 0.55 * h)), img.shape[0])
    ys, xs = np.meshgrid(y0 + (np.arange((y1 - y0) * ss) + 0.5) / ss,   # pixel i covers [i, i + 1)
                         x0 + (np.arange((x1 - x0) * ss) + 0.5) / ss, indexing="ij")
    u, v = (xs - cx) / h, (ys - cy) / h                 # face coordinates, height 1
    skin = rng.uniform(0.5, 0.9)
    f = np.full(u.shape, skin)
    ey, dark = -0.12 + rng.uniform(-0.02, 0.02), skin - rng.uniform(0.3, 0.45)
    for ex in (-0.18, 0.18):
        f[(abs(u - ex) < 0.09) & (abs(v - ey) < 0.05)] = dark
    f[(abs(u) < 0.15) & (abs(v - 0.24 - rng.uniform(-0.02, 0.02)) < 0.035)] = skin - rng.uniform(0.2, 0.35)
    a = ((u / 0.4) ** 2 + (v / 0.5) ** 2 <= 1).astype(float)
    def pool(z):
        return z.reshape(z.shape[0] // ss, ss, z.shape[1] // ss, ss).mean(axis=(1, 3))
    A, F = pool(a), pool(a * f)
    patch = img[y0:y1, x0:x1]
    patch[:] = (1 - A) * patch + F
    return np.array([cx - 0.4 * h, cy - 0.5 * h, 0.8 * h, h])   # box x, y, w, h

def scene(rng, n_faces=6, H=160, W=160, hmin=10, hmax=64):
    img, boxes = clutter(rng, H, W), []
    while len(boxes) < n_faces:          # face heights log-uniform, no overlaps
        h = np.exp(rng.uniform(np.log(hmin), np.log(hmax)))
        cx, cy = rng.uniform(0.45 * h, W - 0.45 * h), rng.uniform(0.55 * h, H - 0.55 * h)
        b = np.array([cx - 0.45 * h, cy - 0.55 * h, 0.9 * h, 1.1 * h])
        if all(iou(c, b)[0] == 0 for c in boxes):
            boxes.append(draw_face(img, cx, cy, h, rng))
    return np.clip(img + rng.normal(0, 0.03, img.shape), 0, 1), np.array(boxes)

def iou(a, b):
    """IoU of boxes (x, y, w, h); a may be an array of boxes."""
    a = np.atleast_2d(a)
    ix = np.clip(np.minimum(a[:, 0] + a[:, 2], b[0] + b[2]) - np.maximum(a[:, 0], b[0]), 0, None)
    iy = np.clip(np.minimum(a[:, 1] + a[:, 3], b[1] + b[3]) - np.maximum(a[:, 1], b[1]), 0, None)
    inter = ix * iy
    return inter / (a[:, 2] * a[:, 3] + b[2] * b[3] - inter)

# ---------- detector: one mean-face template, normalized cross-correlation ----------
def make_template(rng, n=300):
    acc = np.zeros((TH, TW))
    for _ in range(n):                   # faces drawn at the window size on clutter
        img = clutter(rng, 48, 44)
        draw_face(img, 22 + rng.uniform(-0.5, 0.5), 24 + rng.uniform(-0.5, 0.5), 24.0, rng)
        acc += img[12:36, 12:32]
    t = acc - acc.mean()
    return t / np.linalg.norm(t)

def ncc(level, T):
    """Normalized cross-correlation of the zero-mean, unit-norm template T with every
    window of `level`, using integral images for the window means and variances."""
    num = signal.fftconvolve(level, T[::-1, ::-1], mode="valid")
    ii = lambda z: np.pad(z.cumsum(0).cumsum(1), ((1, 0), (1, 0)))
    s1, s2 = ii(level), ii(level ** 2)
    box = lambda s: s[TH:, TW:] - s[:-TH, TW:] - s[TH:, :-TW] + s[:-TH, :-TW]
    var = box(s2) - box(s1) ** 2 / (TH * TW)
    return num / np.sqrt(np.maximum(var, 1e-6))

def detect(img, T, s0=2.4, step=1.2, n_levels=12, thr=0.3, nms_iou=0.3, top_k=100):
    dets = []
    for k in range(n_levels):            # pyramid: image scaled by s0 / step^k
        s = s0 / step ** k
        src = ndimage.gaussian_filter(img, 0.5 / s) if s < 1 else img
        lev = ndimage.zoom(src, s, order=1)
        if min(lev.shape) < TH:
            break
        r = ncc(lev, T)
        peak = (r == ndimage.maximum_filter(r, 3)) & (r > thr)
        yy, xx = np.nonzero(peak)
        for y, x, sc in zip(yy, xx, r[yy, xx]):
            dets.append([x / s, y / s, TW / s, TH / s, sc])
    dets = np.array(dets).reshape(-1, 5)
    dets = dets[np.argsort(-dets[:, 4])]
    keep = []                            # greedy non-maximum suppression
    for i in range(len(dets)):
        if not keep or iou(dets[keep, :4], dets[i, :4]).max() <= nms_iou:
            keep.append(i)
            if len(keep) == top_k:
                break
    return dets[keep]

# ---------- WIDER FACE protocol (as in the official eval_tools/evaluation.m) ----------
def wider_ap(all_dets, all_gts, keep_fn, n_thr=1000, iou_thr=0.5):
    """keep_fn(gt boxes) -> boolean mask of faces that count; the rest are ignored."""
    scores = np.concatenate([d[:, 4] for d in all_dets])
    lo, hi = scores.min(), scores.max()
    props, recs, n_faces = np.zeros(n_thr), np.zeros(n_thr), 0
    thr = 1 - np.arange(1, n_thr + 1) / n_thr
    for dets, gts in zip(all_dets, all_gts):
        keep = keep_fn(gts); n_faces += keep.sum()
        s = (dets[:, 4] - lo) / (hi - lo)                    # global min-max normalization
        proposal = np.ones(len(dets), bool); recalled = np.zeros(len(gts), bool)
        pred_recall = np.zeros(len(dets))
        for h, d in enumerate(dets):                         # dets sorted by score
            o = iou(gts, d[:4]); j = o.argmax()
            if o[j] >= iou_thr:
                if not keep[j]:
                    proposal[h] = False                     # hit an ignored face
                else:
                    recalled[j] = True                      # duplicates stay proposals
            pred_recall[h] = recalled.sum()
        last = np.searchsorted(-s, -thr, side="right") - 1   # last det with s >= thr
        ok = last >= 0
        props[ok] += np.cumsum(proposal)[last[ok]]
        recs[ok] += pred_recall[last[ok]]
    prec = np.divide(recs, props, out=np.zeros(n_thr), where=props > 0)
    rec = recs / n_faces
    mrec, mpre = np.r_[0, rec, 1], np.r_[0, prec, 0]         # VOCap: all-point AP
    for i in range(len(mpre) - 2, -1, -1):
        mpre[i] = max(mpre[i], mpre[i + 1])
    i = np.nonzero(mrec[1:] != mrec[:-1])[0] + 1
    return np.sum((mrec[i] - mrec[i - 1]) * mpre[i]), rec[-1]   # AP, final recall

# ---------- FDDB protocol (as in the official evaluation code) ----------
def masks_iou(dets, gts, shape, ellipse_dets):
    """Pixel-mask IoU between detections (rectangles, or the ellipses inscribed in them)
    and elliptical annotations, as the FDDB code rasterizes regions."""
    y, x = np.mgrid[0:shape[0], 0:shape[1]] + 0.5
    def ell(b):
        return ((x - b[0] - b[2] / 2) / (b[2] / 2)) ** 2 + ((y - b[1] - b[3] / 2) / (b[3] / 2)) ** 2 <= 1
    G = [ell(g) for g in gts]
    M = np.zeros((len(dets), len(gts)))
    for i, d in enumerate(dets):
        if iou(gts, d[:4]).max() == 0:
            continue
        D = ell(d) if ellipse_dets else (x >= d[0]) & (x < d[0] + d[2]) & (y >= d[1]) & (y < d[1] + d[3])
        for j, g in enumerate(G):
            inter = (D & g).sum()
            M[i, j] = inter / ((D | g).sum()) if inter else 0
    return M

def fddb_roc(all_dets, all_gts, shape, ellipse_dets, n_thr=150):
    """Discrete and continuous TPR against the number of false positives."""
    Ms = [masks_iou(d, g, shape, ellipse_dets) for d, g in zip(all_dets, all_gts)]
    scores = np.concatenate([d[:, 4] for d in all_dets])
    N = sum(len(g) for g in all_gts)
    out = []
    for t in np.quantile(scores, np.linspace(1, 0, n_thr)):
        tp_d = tp_c = fp = 0
        for d, M in zip(all_dets, Ms):
            sel = d[:, 4] >= t
            fp += sel.sum()
            if sel.any():
                r, c = linear_sum_assignment(M[sel], maximize=True)   # Hungarian matching
                m = M[sel][r, c]
                tp_c += m.sum(); tp_d += (m > 0.5).sum(); fp -= (m > 0.5).sum()
        out.append((fp, tp_d / N, tp_c / N))
    return np.array(out)

def tpr_at(roc, n_fp, col):
    ok = roc[:, 0] <= n_fp
    return roc[ok, col].max() if ok.any() else 0.0

bins = [(10, 16), (16, 24), (24, 40), (40, 64)]
rows = []
for seed in range(5):
    rng = np.random.default_rng(seed)
    T = make_template(rng)
    data = [scene(rng) for _ in range(40)]
    gts = [b for _, b in data]
    dets = [detect(img, T) for img, _ in data]
    res = [wider_ap(dets, gts, lambda g, lo=lo, hi=hi: (g[:, 3] >= lo) & (g[:, 3] < hi))
           for lo, hi in bins]                               # one height bin kept, rest ignored
    ap_bins, rec_bins = [r[0] for r in res], [r[1] for r in res]
    ap_large = wider_ap(dets, gts, lambda g: g[:, 3] >= 24)[0]   # small faces ignored
    ap_all = wider_ap(dets, gts, lambda g: g[:, 3] >= 10)[0]     # every face counts
    rect = fddb_roc(dets, gts, (160, 160), ellipse_dets=False)
    ell = fddb_roc(dets, gts, (160, 160), ellipse_dets=True)
    rows.append(rec_bins + ap_bins + [ap_large, ap_all, tpr_at(rect, 100, 1), tpr_at(rect, 100, 2),
                           tpr_at(ell, 100, 1), tpr_at(ell, 100, 2)])
R = np.array(rows)
names = [f"recall, height {lo}-{hi} px" for lo, hi in bins] + [
    f"AP, height {lo}-{hi} px" for lo, hi in bins] + [
    "AP, faces >= 24 px (rest ignored)", "AP, all faces",
    "FDDB discrete TPR @100 FP, rectangles", "FDDB continuous TPR @100 FP, rectangles",
    "FDDB discrete TPR @100 FP, ellipses", "FDDB continuous TPR @100 FP, ellipses"]
print(f"{'over 5 seeds':42s} mean   min   max")
for n, c in zip(names, R.T):
    print(f"{n:42s} {c.mean():.3f} {c.min():.3f} {c.max():.3f}")

over 5 seeds                               mean   min   max
recall, height 10-16 px                    0.799 0.747 0.877
recall, height 16-24 px                    0.856 0.789 0.903
recall, height 24-40 px                    0.905 0.852 0.970
recall, height 40-64 px                    0.932 0.886 1.000
AP, height 10-16 px                        0.549 0.518 0.584
AP, height 16-24 px                        0.651 0.597 0.748
AP, height 24-40 px                        0.689 0.641 0.820
AP, height 40-64 px                        0.695 0.596 0.822
AP, faces >= 24 px (rest ignored)          0.733 0.688 0.856
AP, all faces                              0.719 0.669 0.802
FDDB discrete TPR @100 FP, rectangles      0.694 0.654 0.775
FDDB continuous TPR @100 FP, rectangles    0.547 0.512 0.616
FDDB discrete TPR @100 FP, ellipses        0.694 0.654 0.775
FDDB continuous TPR @100 FP, ellipses      0.616 0.578 0.685

Recall rises with face size, from 0.75–0.88 for 10–16-pixel faces to 0.89–1.00 for faces of 40 pixels or more, which in our reading is because the eyes and mouth survive the downsampling of large faces but are already blurred together in small ones, so upsampling has no detail to restore. The AP column shows a second effect of the protocol. Each height bin has only about a quarter of the faces, but background false positives count against every bin in full, since only detections that land on ignored faces are excused; so each bin’s mean AP sits below the mean AP with all faces counted, and the bins of large faces gain little AP from their higher recall. By our reading of the code the same holds for WIDER FACE’s settings: Easy AP still charges every false positive on the background, so it does not measure accuracy on easy faces in isolation. The spread across seeds (up to about 0.23 AP within one bin) comes from different templates and scenes, with only 240 faces per seed.

The FDDB rows separate the two scores. Discrete TPR is identical for rectangles and ellipses, since both shapes clear the 0.5 overlap on the same faces; continuous TPR is about 0.07 lower for rectangles, because even a perfect rectangle around an elliptical annotation has IoU $\pi/4 \approx 0.785$ with it, by our geometry. S³FD’s authors note that this mismatch has a great impact on the continuous score, which is why FDDB submissions convert boxes to ellipses (see Evaluation Metrics).

Datasets and Benchmarks

• MIT+CMU. Rowley et al.’s Test Set 1 (130 images, 507 frontal faces, including 23 images used by Sung) is the frontal test set on which the Viola–Jones papers report results under the name MIT+CMU. Jain and Learned-Miller (2010) note that the MIT+CMU collection contains only greyscale images.
• FDDB (Jain and Learned-Miller, 2010): 2,845 images with 5,171 faces, taken from the Faces in the Wild collection of news photographs, after removing near-duplicate images; faces annotated as ellipses, split into ten folds. Its report proposes two protocols: EXP-1, ten-fold cross-validation on FDDB, and EXP-2, unrestricted training that may include data outside FDDB, with ten ROC curves (one per fold) averaged in both cases. The WIDER FACE authors noted that its source, a news website, is biased towards celebrity faces (Yang et al., 2016). The official FDDB website could not be reached when checked in October 2026; archived copies of its pages and code are cited here.
• AFW (Zhu and Ramanan, 2012): 205 Flickr images, each with at least one large face, annotated with boxes, six landmarks and discretized pose.
• PASCAL Faces: 851 images with 1,335 faces from the PASCAL person layout test set, as S³FD’s authors describe it (Zhang et al., 2017; the WIDER FACE paper gives 1,341 faces); Mathias et al. (2014) re-annotated its boxes to a uniform policy.
• WIDER FACE (Yang et al., 2016): 32,203 images with 393,703 labelled faces, chosen from the WIDER event-recognition dataset and split 40%, 10% and 50% into training, validation and test sets within each event class. The paper counts 60 event classes; the official page and the evaluation code use 61. Test annotations are withheld, and the organizers evaluate submitted results. Faces carry occlusion labels (partial for 1–30% of the face area, heavy above 30%) and pose labels, and the validation ground-truth files also hold blur, expression and illumination labels. Two scenarios are defined: Scenario-Int, training on WIDER FACE only, and Scenario-Ext, training on any external data.
• MAFA (Ge et al., 2017): 30,811 images with 35,806 masked faces of at least $32 \times 32$ pixels, annotated with eyes, masks, five orientations, three occlusion degrees and four mask types, and split into 25,876 training images (29,452 faces) and 4,935 test images (6,354 faces).
• The WIDER FACE page also lists IJB-A, which serves detection and recognition (24,327 images, 49,759 faces), and MALF (5,250 images, 11,931 faces), which the page describes as the first face detection dataset to support fine-grained evaluation.

Evaluation Metrics

Both main benchmarks match detections to faces by IoU at 0.5 but summarize the result differently, and both depend on the details of their official code.

FDDB. Each detection is matched to at most one face, and each face to at most one detection, by the Hungarian algorithm, which maximizes the total overlap in the image; overlaps are computed on rasterized masks, so rectangles and ellipses can be compared. The discrete score counts a matched detection with overlap above 0.5 as a true positive; the continuous score credits each matched detection with its overlap. The report recommends ROC curves for both (Jain and Learned-Miller, 2010), but the curve differs from the textbook ROC curve: the official code writes, for each score threshold, the true positive rate against the number of false positives over the whole dataset, not a false positive rate, because the set of negatives (all non-face regions) has no fixed size. In the code (Results.cpp), a matched detection with overlap of 0.5 or less still adds its overlap to the continuous true-positive sum but is counted as a false positive in both curves. Results are usually read at a given number of false positives; DSFD, for example, reports 99.1% (discrete) and 86.2% (continuous) at 1,000 false positives (Li et al., 2019).

Because FDDB’s annotations are ellipses and most detectors output boxes, authors convert. Li et al. (2015) extended their square boxes vertically by 20% for the discrete score and fitted upright ellipses with axes $1.18h$ and $1.13w$ for the continuous one; Hu and Ramanan (2017), S³FD and DSFD trained regressors from boxes to ellipses. FDDB also has unlabelled faces: S³FD’s authors manually added 238 of them before evaluating, and DSFD reports that adding such annotations further reduces its false positives. Results with and without these adjustments are not comparable.

WIDER FACE adopts the PASCAL VOC average precision, and its evaluation code (evaluation.m) computes it as follows. Scores are min–max normalized over all detections of the evaluated set. Within each image, detections are processed in order of decreasing score, and each is matched to the face with which it has the highest IoU; if that IoU is at least 0.5 and the face is ignored in the current setting, the detection is dropped from the count of proposals, and otherwise the face is marked as recalled. A second detection on a recalled face remains a proposal, so it lowers precision as a false positive would. For each of 1,000 thresholds $\tau = 1 - t/1000$,

$$P(\tau) = \frac{\sum_i R_i(\tau)}{\sum_i Q_i(\tau)}, \qquad \mathrm{Rec}(\tau) = \frac{\sum_i R_i(\tau)}{N},$$

where $R_i(\tau)$ is the number of faces of image $i$ recalled by detections scoring at least $\tau$, $Q_i(\tau)$ the number of such detections not dropped, and $N$ the number of non-ignored faces. AP is the all-point interpolated area under this curve (VOCap.m), and the IoU uses the VOC development kit’s convention of adding one pixel to widths and heights (boxoverlap.m, which is taken from the DPM code).

The Easy, Medium and Hard settings do not partition the data: each is a list of faces that count, with all others ignored. The paper defines the three levels by how well EdgeBox object proposals recall the faces, 92%, 76% and 34% on average with 8,000 proposals per image (Yang et al., 2016). In the ground-truth files of the official tools, the validation set has 3,226 images and 39,708 faces, of which Easy keeps 7,211, Medium 13,319 and Hard 31,958; each list contains the previous one, and 7,750 faces are ignored in all three settings (our count). Hu and Ramanan (2017) describe Hard as containing all faces taller than 10 pixels; the smallest face kept in Hard in these files is 9 pixels tall (our count). Because of the nesting, Hard covers the most faces, and RetinaFace and TinaFace headline their Hard test results.

Other datasets use AP at IoU 0.5 (AFW, PASCAL Faces, MAFA). RetinaFace’s authors also report AP averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05, as COCO does, to reward precise boxes (Deng et al., 2020).

Applications

Face detection is the first stage of face alignment, face recognition and verification, and other facial analysis (Yang et al., 2016). Its techniques reached commercial products such as digital cameras (Jain and Learned-Miller, 2010), and the WIDER FACE authors name digital cameras and electronic photo albums as uses of near-frontal detectors. Augmented-reality pipelines, the target of BlazeFace, crop the detected face for keypoint, geometry, expression and segmentation models (Bazarevsky et al., 2019). FDDB’s report notes that the useful operating point depends on the application: access control tolerates low recall but needs few false detections, while surveillance needs every face (Jain and Learned-Miller, 2010).

Deployment notes

OpenCV offers both generations. The Haar cascades of cv::CascadeClassifier are covered in the Viola–Jones article. cv::FaceDetectorYN (OpenCV 4.14.0, as of October 2026) runs a YuNet model through the DNN module. FaceDetectorYN::create takes the model path, a configuration path (not needed for ONNX models) and an input size, with defaults of a score threshold of 0.9, an NMS IoU threshold of 0.3 and 5,000 boxes kept before NMS. The image passed to detect must have exactly the configured input size (the code checks this; setInputSize changes it), and it is padded to a multiple of 32. Decoding in face_detect.cpp is anchor-free: at every cell of the output maps at strides 8, 16 and 32, the score is $\sqrt{p_{\text{cls}}\, p_{\text{obj}}}$ of a classification and an objectness output, the box centre is the cell position plus a predicted offset times the stride, and the width and height are the exponentials of predicted values times the stride; the five landmarks are decoded like the centre. With strides of 8 to 32 and a model card stating a range of about 10 to 300 pixels, faces outside that range at the chosen input size are likely to be missed, so in our reading an application that resizes large images down for speed loses its smallest faces first, and one that expects very large faces should downscale.

Open Problems

• Small and heavily occluded faces. Hard-set AP is still several points below Easy-set AP for every detector in the table that reports both, and heavy occlusion remains far harder than light occlusion, as MAFA’s results show.
• Evaluation conventions. Box styles, ignore rules and unlabelled faces change reported numbers; Mathias et al. (2014) found adjusting a detector to the annotation style key to competitive results, and the WIDER FACE Hard test results of DSFD, RetinaFace and TinaFace in the table lie within about two points of each other.
• Speed against accuracy on devices. Compact detectors such as YuNet trail the large models by about 16 points of Hard AP on WIDER FACE validation (75.0 against 91.4 for RetinaFace with ResNet-50, which was tested at five scales with flipping).
• Generalization beyond the benchmarks. Detectors trained on web photographs meet different cameras, viewpoints and populations in deployment. None of the detection papers cited here reports accuracy broken down by demographic group; BlazeFace’s authors describe their private test set only as geographically diverse.