Facial Landmark Detection

Facial landmark detection locates a fixed set of named points on a face, such as the corners of the eyes, the tip of the nose, the corners of the mouth and points along the eyebrows, lips and jaw line. Given a face found by a face detector, it returns the image coordinates of each point, and the set of points, called the face’s shape, describes where the facial features are and how they are deformed by pose and expression. The task is also called face alignment, facial keypoint detection or facial feature (fiducial point) localization; the papers cited here use “face alignment” and “facial landmark localisation” interchangeably for it. It is a category-specific case of keypoint detection with its own landmark definitions, benchmarks and error measures, and it is a step in many face pipelines: face recognition aligns crops to a canonical geometry from a few landmarks, and expression analysis, face tracking, animation and 3D face modelling use the dense outline.

The methods went through four stages. Statistical shape and appearance models of the 1990s fitted a model of the whole face by iterative search. Cascaded regression, from about 2010, learned to move an initial shape towards the true one in a few steps of regression on shape-indexed features. Convolutional networks then either regressed the coordinates directly or predicted one map per landmark (heatmap regression), and 3D methods fitted a deformable 3D face model to handle large head rotations. Each step was driven by harder data: the in-the-wild benchmarks 300-W, COFW, AFLW and WFLW added pose, occlusion and expression that the earlier methods did not handle.

Problem Definition

A face shape with $N$ landmarks is the vector $S = (x_1, y_1, \ldots, x_N, y_N)^\top$. Given an image $I$ and a face region, usually a box from a face detector, the task is to estimate $S$ so that it is close to the annotated shape $\hat{S}$; Cao et al. (2012) write the goal as minimizing $\lVert S - \hat{S} \rVert^2$ and note that this alignment error is used both to guide training and to evaluate results, although it cannot be computed at test time because $\hat{S}$ is unknown.

Three properties set the task apart from detecting generic keypoints:

• One instance, a fixed point list. The face is given, and every landmark of the scheme is predicted, whether it is visible or not, unless the dataset marks occlusion explicitly.
• Semantic correspondence. Landmark $i$ must mean the same anatomical place on every face. Some points have a distinctive appearance (eye corners, mouth corners); others, such as points along the jaw or the eyebrows, are defined by their position along a contour. Cao et al. (2012) observe that only a few salient landmarks can be located from their appearance alone and that the others need the shape constraint, the correlation between landmarks.
• Dependence on the face box. Most methods start from the box: classical methods place a mean shape inside it, and networks crop and rescale the face by it. Benchmarks therefore fix the boxes. In the 300-W challenge every submitted binary was run on the same box from the organisers’ own detector (Sagonas et al., 2013).

Input and Output

The input is an image and a face box; most methods crop the face from the box and resample it to a fixed size, for example $256 \times 256$ pixels in Wu et al. (2018) and in Wang, Bo and Li (2019). The output is an $N \times 2$ array of coordinates, sometimes with a visibility or occlusion flag per landmark (COFW annotates both, Burgos-Artizzu, Perona and Dollár, 2013) and, for 3D methods, a depth or the parameters of a 3D face model. File conventions matter in practice: the 300-W annotations are stored in .pts files in the MATLAB convention, where the top-left pixel has coordinates $(1, 1)$ (300-W challenge page, as of November 2026), so code that assumes zero-based pixel indices is off by one pixel in each direction.

Landmark Schemes

There is no single definition of the face’s landmarks. Each dataset defines its own list, and a model trained on one list cannot predict another.

Scheme	Points	Where it is used	Notes
Multi-PIE / iBUG 68	68	300-W and its re-annotations of LFPW, HELEN, AFW, XM2VTS, IBUG; Menpo (frontal faces)	300-W also scores the 51 points without the face’s border (Sagonas et al., 2013)
WFLW 98	98	WFLW	Fully manual annotation (Wu et al., 2018)
LFPW / COFW 29	29	LFPW (35 points provided, 29 used), COFW	COFW annotates each point’s occlusion state (Burgos-Artizzu, Perona and Dollár, 2013)
AFLW 21	21 (19 in the common protocol)	AFLW	Points annotated only when visible (AFLW page, archived 2022)
HELEN 194	194	HELEN	Very dense outlines on high-resolution images (Sagonas et al., 2013)
Five or six points	5–6	Face detectors (MTCNN, RetinaFace, BlazeFace), AFW	Five: eye centres, nose tip, mouth corners; BlazeFace’s six: eye centres, ear tragions, mouth centre, nose tip; used for coarse alignment

The 68-point mark-up comes from the Multi-PIE database of Gross, Matthews, Cohn, Kanade and Baker (2010), whose configuration the 300-W organisers adopted when they re-annotated four existing datasets so that they could be compared under one scheme (Sagonas et al., 2013). The 300-W page shows the 68-point and 51-point mark-ups; the 51 points are the 68 without the 17 on the face’s border. AFLW’s markup has 21 points, including the ear lobes, and a point is left unannotated when it is not visible (AFLW page, archived 2022); the AFLW-Full protocol used in later papers keeps 19 of them. The two papers cited here for it attribute the protocol to different works by the same authors, Shizhan Zhu, Li, Loy and Tang: Wu et al. (2018) to their 2015 paper on coarse-to-fine shape searching, Feng et al. (2018) to their 2016 paper on cascaded compositional learning. The few points that face detectors predict are a separate, coarse scheme.

Converting between schemes is approximate. Milborrow, Bishop and Nicolls (2013) trained their models on the MUCT database and converted its landmarks to the 68 iBUG points by copying points such as the eye corners and interpolating others such as the jaw; they note that even copied points are not defined identically and that the conversion adds error. Large head rotations break the scheme itself. Zhu et al. (2016) note that in medium poses the contour landmarks are moved to the visible silhouette (“landmark marching”), while in large poses, with half of the face self-occluded, some landmarks have no image data at all. Bulat and Tzimiropoulos (2017) distinguish 2D annotations, where contour points follow the visible outline and lose correspondence across pose, from “3D” annotations, the 2D projections of 3D landmarks, which keep it.

Variants

• Sparse 2D alignment on near-frontal faces, the setting of LFPW, HELEN and the 300-W training data.
• Large-pose alignment, with yaw up to ±90°, where landmarks are self-occluded; AFLW and AFLW2000-3D test it (Zhu et al., 2016).
• Occlusion-aware alignment, which predicts each landmark’s visibility as well as its position, as Robust Cascaded Pose Regression (RCPR) does on COFW (Burgos-Artizzu, Perona and Dollár, 2013).
• 3D face alignment, which predicts landmarks consistent with a 3D face, or fits a dense 3D face model (Zhu et al., 2016; Bulat and Tzimiropoulos, 2017).
• Video alignment and tracking, evaluated on 300-VW, a set of 114 videos with 218,595 annotated frames (Bulat and Tzimiropoulos, 2017).

Challenges

• Pose. Out-of-plane rotation hides landmarks and changes the face’s appearance completely. On WFLW’s pose subset, the 2019 Adaptive Wing method had a normalized mean error of 7.38% against 4.36% on the whole test set, and failed on 13.50% of the faces against 2.84% overall (Wang, Bo and Li, 2019, Table 1).
• Occlusion. Hands, hair, glasses, food and microphones cover parts of the face. COFW was collected to have occlusion: its faces are on average more than 23% occluded, against about 2% for the LFPW training faces (Burgos-Artizzu, Perona and Dollár, 2013).
• Expression. Open mouths and squints move points far from the mean shape. 300-W’s organisers chose test images with spontaneous expressions; 14.16% of the test faces were labelled “Scream”, against at most 1.78% in LFPW, HELEN and AFW, and all six participants did worse on the outdoor images, which the organisers attribute mainly to illumination and to expressions such as surprise and screaming (Sagonas et al., 2013).
• Ambiguous definitions. A point on the jaw or the brow is defined only along a contour, so annotators disagree along it. Wu et al. (2018) motivate their boundary-aware method by this ambiguity, and Bulat and Tzimiropoulos (2017) judged that on the highest-error fittings they show from their 2D test sets (failures excluded) and from AFLW2000-3D, their networks’ predictions were in most cases more accurate than the ground truth.
• Initialization. Cascaded methods start from a mean shape placed in the face box, and Bulat and Tzimiropoulos (2017) note that their accuracy drops with inaccurate initialization and large poses. Simulating the detector’s errors in training is part of the method: SDM samples perturbed initial shapes from a normal distribution that matches the detector’s statistics (Xiong and De la Torre, 2013).
• Imbalanced data. Large rotations are rare in training sets. Feng et al. (2018) counter it with pose-based data balancing, which duplicates and perturbs the under-represented poses.

Approaches

Active shape models

Active shape models (ASMs), by Cootes, Taylor and colleagues, combine a statistical model of shape with local searches; the papers read for this article cite different original works for them, Cootes, Taylor, Cooper and Graham (1995) in Milborrow, Bishop and Nicolls (2013) and Cootes and Taylor (1992) in Cao et al. (2012). Their shape constraint is a principal component analysis model of the training shapes (Cao et al., 2012): a plausible shape is the mean shape plus a linear combination of a few modes of variation. Milborrow, Bishop and Nicolls (2013) summarize the search: an ASM starts from the mean shape placed in the box returned by a face detector and repeats two steps until convergence, (i) proposing a new position for each point by template matching around it, in the classical method by matching one-dimensional gradient profiles along a line orthogonal to the shape boundary, and (ii) conforming the proposed shape to the global shape model, which pools the unreliable local matches. The search is repeated on an image pyramid, typically four levels from coarse to fine. Later ASMs replaced the profiles by 2D descriptors; Milborrow et al. used a modified SIFT descriptor, which they call the Histogram Array Transform (HAT), on $15 \times 15$ patches and three ASMs for frontal and three-quarter views, chosen by an estimate of the face’s yaw.

Active appearance models

Active appearance models (AAMs) model the face’s texture as well as its shape. Matthews and Baker (2004) date the first proposal to Edwards, Taylor and Cootes (1998) and cite the formulation of Cootes, Edwards and Taylor (1998, 2001). In Matthews and Baker’s notation, the shape is a mesh $s = s_0 + \sum_{i=1}^{n} p_i s_i$, obtained by PCA of Procrustes-aligned training meshes, and the appearance is an image defined inside the mean mesh, $A(\mathbf{x}) = A_0(\mathbf{x}) + \sum_{i=1}^{m} \lambda_i A_i(\mathbf{x})$, obtained by PCA of training faces warped to the mean shape with a piecewise affine warp $W(\mathbf{x}; \mathbf{p})$. Fitting minimizes the difference between model and image over the shape and appearance parameters,

$$\sum_{\mathbf{x} \in s_0} \left[ A_0(\mathbf{x}) + \sum_{i=1}^{m} \lambda_i A_i(\mathbf{x}) - I(W(\mathbf{x}; \mathbf{p})) \right]^2 .$$

The original algorithms assumed a constant linear relation between the error image and the parameter update, learned by linear regression. Matthews and Baker showed that this assumption is in general incorrect and proposed an analytical fitting algorithm based on inverse compositional image alignment, in which the appearance variation is “projected out” in a precomputation. Such a project-out inverse compositional AAM, with edge-structure features, was the baseline of the 300-W challenge, and every submission beat it (Sagonas et al., 2013). Cao et al. (2012) summarize the weaknesses that motivated the next stage: a linear appearance model has limited power to represent unseen faces under varied pose, expression and illumination, and gradient-based fitting is sensitive to initialization. Constrained local models replace the holistic appearance model by a set of local landmark detectors or regressors constrained by a shape model (Ren et al., 2014).

Cascaded shape regression

Cascaded regression drops the generative model and learns the update directly. Starting from an initial shape $S^0$, usually the mean shape placed in the face box, a sequence of regressors refines it,

$$S^t = S^{t-1} + R^t\!\left(I, S^{t-1}\right), \qquad t = 1, \ldots, T,$$

where each $R^t$ reads features indexed relative to the current shape, so that the same feature refers to the same part of the face whatever the face’s position, scale and rotation. Cao et al. (2012) note that the cascaded pose regression of Dollár, Welinder and Perona (2010) used the same idea, which Xiong and De la Torre (2013) call pose-indexed features.

Explicit shape regression (ESR). Cao et al. (2012) regress the whole shape jointly and minimize the alignment error directly, rather than the parameters of a shape model. Each stage $R^t$ is itself a boosted sequence of ferns, small regressors that split the training samples into $2^F$ bins with $F = 5$ thresholded features, each bin storing the mean shape residual of its samples with shrinkage. The features are differences of two pixel intensities, each pixel indexed by its offset from its nearest landmark in a frame normalized by a similarity transform to the mean shape, and they are chosen by their correlation with random projections of the residual. Because every update is a mean of training-shape differences, the result is the initial shape plus a linear combination of training shapes: the shape constraint is non-parametric and learned, and the paper shows it moving from coarse changes such as yaw, roll and scale in the first stage to fine changes of the mouth, eyes and contour in the last. The settings were $T = 10$ stages of $K = 500$ ferns and $P = 400$ sampled pixels per stage; training augmented each image with 20 initial shapes taken from other training images, and testing took the median of five runs from different initial shapes. Errors were normalized by the inter-pupil distance; the method took 15 ms for 87 landmarks with five initializations.

Supervised descent method (SDM). Xiong and De la Torre (2013) derive cascaded regression from Newton’s method. With $h$ a feature extractor (SIFT in their experiments) and $\phi_* = h(d(\mathbf{x}_*))$ the features at the true landmarks, alignment minimizes $f(\mathbf{x}_0 + \Delta\mathbf{x}) = \lVert h(d(\mathbf{x}_0 + \Delta\mathbf{x})) - \phi_* \rVert_2^2$. A Newton step would need the Jacobian and Hessian of a non-differentiable feature, and $\phi_*$ is unknown at test time; SDM instead learns a sequence of generic descent directions $R_k$ and biases $b_k$ from training data,

$$\mathbf{x}_k = \mathbf{x}_{k-1} + R_{k-1}\,\phi_{k-1} + b_{k-1},$$

each by linear least squares on the residuals left by the previous steps, with initial shapes sampled to imitate the detector’s errors. In their experiments the cascade converged in four or five steps. On LFPW’s 29 landmarks they reported a mean error of 3.47, in units of $10^{-2}$ of the distance between the eyes, against 3.43 reported for ESR; SDM’s paper calls the normalizer the inter-ocular distance and ESR’s the distance between the pupils. LFPW was distributed as image URLs, many of them dead, and the two papers used different subsets (884 training and 245 test images for SDM, 812 and 249 for ESR), so in our reading the two numbers are only roughly comparable.

Local binary features (LBF). Ren et al. (2014) learn the features stage by stage under a “locality principle”: for each landmark, a random forest is trained to regress that landmark’s offset from pixel-difference features sampled only in a local region around it, whose radius shrinks from stage to stage. The forests’ leaves are then used as a sparse binary feature, one bit per leaf, all landmarks’ features are concatenated, and a global linear regression with $\ell_2$ regularization maps them to the shape update, which restores the shape constraint. On their 300-W split, with errors as percentages of the inter-pupil distance, LBF scored 6.32 on the full set, 4.95 on the common subset and 11.98 on the challenging subset, against 7.58, 5.28 and 17.00 for ESR and 7.52, 5.60 and 15.40 for SDM in the authors’ own implementations; its fast version ran at 3,100 frames per second on one CPU core (Ren et al., 2014, Table 1).

Ensemble of regression trees (ERT). Kazemi and Sullivan (2014) fit each stage by gradient boosting of regression trees on shape-indexed pixel differences, with a prior on the distance between the two pixels. On HELEN’s 194 landmarks they reported an inter-ocular-normalized error of 0.049, against 0.111 and 0.091 for two ASM variants, at about one millisecond per image after about an hour of training on one CPU. The FFHQ paper cites this method for the alignment of its faces, which its released code performs with 68 landmarks from the dlib library.

Cascaded regression was fast and accurate on near-frontal faces, but Bulat and Tzimiropoulos (2017) note its accuracy drops with poor initialization and with large poses or in-plane rotations, where many landmarks are self-occluded.

Convolutional networks: coordinates and heatmaps

Networks replaced hand-designed features and boosted regressors. Two output representations are used, and the heatmap regression article covers their mechanics, the quantization error of heatmaps and the coordinate alternatives in detail; what follows is the face-specific evidence.

• Coordinate regression. The network outputs the $2N$ coordinates. Zhou et al., the industry winner of the 300-W challenge, used a four-level cascade of convolutional networks, each refining the output of the previous one (Sagonas et al., 2013). Feng et al. (2018) studied the loss. With the same small network (CNN-6, $64 \times 64$ inputs) on AFLW-Full, an L2 loss gave a face-size-normalized error of $2.41 \times 10^{-2}$, L1 $2.00 \times 10^{-2}$ and smooth L1 $2.02 \times 10^{-2}$ (Table 1). They argue that L2 lets large errors dominate the gradient and proposed the Wing loss, logarithmic for small errors and linear for large ones, $\mathrm{wing}(x) = w \ln(1 + |x|/\epsilon)$ for $|x| < w$ and $|x| - C$ otherwise, with $C$ making the two parts meet; it reached $1.88 \times 10^{-2}$ with CNN-6, and $1.47 \times 10^{-2}$ with a ResNet-50 trained with pose-based data balancing (Tables 1 and 5).
• Heatmap regression. The network outputs one map per landmark, trained against a Gaussian at the landmark and decoded at its peak. Bulat and Tzimiropoulos (2017) built the Face Alignment Network (FAN) from four stacked hourglass networks whose bottleneck blocks were replaced by a hierarchical, parallel and multi-scale block, trained it on 300W-LP and found 2D-FAN close to saturating the 2D benchmarks they tested. Wu et al. (2018) predicted heatmaps of facial boundary lines and used them to guide landmark regression, arguing that boundaries remove the ambiguity of landmark definitions. Wang, Bo and Li (2019) replaced the mean squared error on heatmaps with the Adaptive Wing loss, described in the heatmap regression article.

Neither representation wins everywhere. On WFLW, as tabulated by Wang, Bo and Li (2019, Table 1), the coordinate-regression Wing method with a ResNet-50 backbone had a lower test-set error (5.11%) than the heatmap-based boundary method of Wu et al. (5.27%), and the heatmap-based Adaptive Wing method was lower still (4.36%). The three use different backbones, inputs and training, so the comparison is between systems, not representations. The example below holds everything but the output head fixed.

3D face alignment

3D methods fit a deformable 3D face to the image. Zhu et al. (2016) use the 3D morphable model of Blanz and Vetter, a PCA model of 3D face shape $S = \bar{S} + A_{id}\,\alpha_{id} + A_{exp}\,\alpha_{exp}$ with identity bases from the Basel Face Model and expression bases from FaceWarehouse, projected to the image by a weak perspective camera,

$$V(\mathbf{p}) = f \cdot \mathrm{Pr} \cdot R \cdot \left( \bar{S} + A_{id}\,\alpha_{id} + A_{exp}\,\alpha_{exp} \right) + \mathbf{t}_{2d},$$

with scale $f$, orthographic projection $\mathrm{Pr}$, rotation $R$ from pitch, yaw and roll, and translation $\mathbf{t}_{2d}$. Their 3D Dense Face Alignment (3DDFA) estimates the 234 parameters (6 for pose, 199 for identity, 29 for expression) with a cascade of convolutional networks, each taking the $100 \times 100$ image together with a rendering of the current fit (the Projected Normalized Coordinate Code) and predicting a parameter update, trained with a weighted parameter distance cost. Landmarks then follow from the fitted model, visible or not. To get training data in large poses they synthesized profile views of the 300-W faces, 61,225 images (122,450 with flipping), called 300W-LP, and they built AFLW2000-3D, the first 2,000 AFLW images with fitted 3D faces and 68 landmarks, for testing.

Bulat and Tzimiropoulos (2017) trained the same FAN architecture on 300W-LP’s 3D landmarks (3D-FAN), and a 2D-to-3D FAN that converts 2D annotations to 3D ones, which they used to create LS3D-W, about 230,000 images with 3D landmarks. On those test sets 3D-FAN largely outperformed 3DDFA trained on the same data. Their “3D” landmarks are the 2D projections of 3D points.

Datasets and Benchmarks

Benchmark	Landmarks	Split (as used in the cited papers)	Normalizer	What it tests
300-W challenge (private test)	68 (and 51 inner)	600 test faces: 300 indoor, 300 outdoor	Outer eye corners	Unseen in-the-wild faces, fixed detector boxes
300-W common/challenging/full split	68	Train 3,148; test 689 = 554 common + 135 challenging (IBUG)	Inter-pupil in Ren et al. (2014); both in later papers	Near-frontal faces, hard poses and expressions in IBUG
COFW	29	Train 1,345 (845 LFPW + 500 COFW); test 507	“Interocular” (Burgos-Artizzu et al.); inter-pupil or outer corners in later papers	Occlusion, with per-landmark occlusion labels
AFLW (AFLW-Full)	21 annotated, 19 used	Train 20,000; test 4,386 (AFLW-Frontal: 1,314 of them)	Face box size	Large pose, yaw up to ±90°
AFLW2000-3D	68 (3D-consistent)	2,000 test faces	Box size	3D alignment in large poses
WFLW	98	Train 7,500; test 2,500, with six attribute subsets	Outer eye corners	Pose, expression, illumination, make-up, occlusion, blur

300-W. The 300 Faces in-the-Wild challenge, held with ICCV 2013, was the first benchmark built for one shared protocol (Sagonas et al., 2013). Its organisers re-annotated LFPW, AFW, HELEN and the lab dataset XM2VTS with the 68-point mark-up, using a semi-automatic method with manual correction, added 135 highly expressive training images (IBUG), and collected a new test set of 600 faces by searching for terms such as “party”, “conference”, “protests” and “football”. Participants sent binaries, which the organisers ran with their own detector’s boxes and scored by the mean point-to-point error normalized by the outer-eye-corner distance, on 68 and 51 points, reporting cumulative error curves. Six teams took part; the winners were Yan et al. (academia) and Zhou et al. (industry). The organisers compared the best submissions with the error of reconstructing the test shapes from 25 principal components of the training shapes (98% of the shape variance) and concluded that the benchmark was not saturated. As of November 2026, the 300-W page states that the data are provided for research purposes only and that commercial use, including training commercial algorithms, is not allowed.

Most later papers use a different split of the same data, introduced by Ren et al. (2014) because the challenge provided only training data: training on AFW and the LFPW and HELEN training sets (3,148 images) and testing on the LFPW and HELEN test sets (the common subset, 554 images) and IBUG (the challenging subset, 135), together the full set of 689. “300-W” in a results table can therefore mean either the 600-image private test set or this 689-image split, and the normalizer differs between papers.

COFW. The Caltech Occluded Faces in the Wild dataset has 1,007 faces collected by four people, each asked to gather 250 faces representative of real-world images that would challenge vision methods, annotated with the 29 LFPW landmarks and their occlusion state (Burgos-Artizzu, Perona and Dollár, 2013). 150 images were annotated twice to measure human performance. Training uses the 845 available LFPW training faces plus 500 COFW faces, and testing the remaining 507; a re-annotation of the test set with 68 points (COFW-68) is used for cross-dataset tests (Wu et al., 2018).

AFLW. Annotated Facial Landmarks in the Wild contains about 25,000 faces in Flickr images with up to 21 landmarks each, about 380,000 landmarks in total, annotated only where visible; it also provides face rectangles, FDDB-compatible ellipses and a coarse head pose from fitting a mean 3D face (AFLW page, archived 2022). Its page reports that 59% of the faces are tagged female and 41% male, the only demographic information in the benchmark sources read for this article, and that the database is available for non-commercial research only (archived copy of October 2022; the live page returned 404 in November 2026). The face counts in papers differ: 25,000 images of 24,686 subjects in Sagonas et al. (2013), 24,386 faces in Wu et al. (2018), and 21,080 faces in the evaluation of Zhu et al. (2016). The AFLW-Full protocol, with 20,000 training and 4,386 test images and 19 landmarks, normalizes by the face box size because the inter-ocular distance collapses on profile faces (Wu et al., 2018; Feng et al., 2018).

WFLW. Wider Facial Landmarks in-the-wild, built by Wu et al. (2018) from WIDER FACE images, has 10,000 faces (7,500 for training, 2,500 for testing) with 98 manually annotated landmarks and attribute labels for occlusion, pose, make-up, illumination, blur and expression, from which six test subsets are formed. Wu et al. proposed it to replace switching between several datasets’ protocols with one dataset that tests all of these factors. Results are reported as NME normalized by the outer eye corners, failure rate at 10% and AUC up to 10% (Wang, Bo and Li, 2019).

None of these papers reports the age, sex or skin type composition of 300-W, COFW or WFLW, or landmark accuracy broken down by demographic group.

Evaluation Metrics

Normalized mean error. For one face with $N$ landmarks, predicted positions $\mathbf{p}_i$ and ground truth $\hat{\mathbf{p}}_i$,

$$\mathrm{NME} = \frac{1}{N} \sum_{i=1}^{N} \frac{\lVert \mathbf{p}_i - \hat{\mathbf{p}}_i \rVert_2}{d},$$

averaged over the test faces and usually reported as a percentage. The normalizer $d$ makes the error independent of the face’s size, and its definition changes the number:

• Inter-ocular distance, the distance between the outer eye corners, as in the 300-W challenge (Sagonas et al., 2013) and WFLW.
• Inter-pupil distance, between the eye centres, used by ESR, LBF and the papers that compare with them on the 689-image split (Cao et al., 2012; Ren et al., 2014; Feng et al., 2018). The pupils are closer together than the outer corners, so the same predictions give a larger NME: Wu et al. (2018) report 4.12% on the 300-W full set with inter-pupil normalization and 3.49% with inter-ocular normalization (Table 1).
• Face size, the width or height of the face box for AFLW-Full (Feng et al., 2018), or $\sqrt{w_{bbox}\, h_{bbox}}$ of the box around the ground-truth landmarks for FAN’s 2D and 3D tests (Bulat and Tzimiropoulos, 2017). Eye-based normalizers are biased on profile faces, where the eyes are close together in the image; Milborrow, Bishop and Nicolls (2013) point out that they inflate the measured error of yawed faces and took the use of eye-corner normalization in 300-W as a hint that its test set has no side views.

Which landmarks are averaged also matters (68 or the 51 inner points in 300-W; only visible points in AFLW, which Zhu et al., 2016, show can make a poor fit look good).

Failure rate. The share of test faces whose NME exceeds a threshold. COFW counts errors above 10% as failures (Burgos-Artizzu, Perona and Dollár, 2013), WFLW uses 10%, results on the 300-W private set are given at 8% and 10% (Wang, Bo and Li, 2019), and FAN used 7% (Bulat and Tzimiropoulos, 2017).

Cumulative error distribution and AUC. The cumulative error distribution (CED) curve plots the share of faces with NME below each threshold, and the area under it up to a cut-off, divided by the cut-off, summarizes accuracy and robustness in one number between 0 and 1. The cut-off is part of the metric: WFLW and the 300-W private set use 10% or 8% (Wang, Bo and Li, 2019), FAN’s comparisons 7% (Bulat and Tzimiropoulos, 2017). Some papers print AUC as a fraction and others as a percentage.

Numbers copied between papers drift. The SDM result on the 300-W common subset is 5.60 in Ren et al. (2014), whose authors computed it with their own implementation, and in Feng et al. (2018), but 5.57 in the tables of Wu et al. (2018) and Wang, Bo and Li (2019), all with inter-pupil normalization; its full-set value moves the same way, from 7.52 to 7.50. Comparisons are meaningful only when the split, the box initialization, the landmark set and the normalizer are all the same.

Example: heatmaps against coordinate regression

The example trains two networks that share a backbone, data and training budget and differ only in the output head, and scores them as the face benchmarks do. The images are synthetic $64 \times 64$ grey faces: an elliptical face with eyes (and darker pupils), eyebrows, a nose tip and a mouth whose opening varies, under a random similarity transform (rotation up to ±30°, scale 0.85–1.15, translation up to ±5 pixels), a lighting gradient, blur and noise. Ten landmarks are known exactly: the outer and inner eye corners, the pupil centres, the nose tip, the mouth corners and the chin. The heatmap network outputs ten $32 \times 32$ maps (stride 2) trained with mean squared error against Gaussians of $\sigma = 1$ cell, decoded by the argmax and by the quarter-cell shift; the regression network flattens the same features and outputs 20 coordinates. Each model is trained for 10 epochs on 3,000 images and tested on 500.

Three choices that the sources leave open are printed both ways: the normalizer (outer eye corners and pupils), the regression loss (L2 and L1, after Feng et al., 2018) and the heatmap decoder. Two rows need no learning: the training mean shape placed at the image centre, and the training mean shape placed by the image’s intensity moments, the analogue of starting a cascade from a face box. Two further rows decode the ideal target maps of the test faces, which gives the error floor of a perfect stride-2 heatmap network.

import numpy as np
import torch
import torch.nn as nn
from scipy.ndimage import gaussian_filter

torch.manual_seed(0)
torch.set_num_threads(4)
SIZE, STRIDE, K = 64, 2, 10


def make_face(rng):
    """One synthetic face: image and the 10 landmarks (x, y) in pixels."""
    a = rng.uniform(15, 18)                     # face half-width in the canonical frame
    b = 1.25 * a                                # half-height
    ex, ey = 0.42 * a, -0.18 * b                # eye centre offset
    ew, eh = 0.20 * a, 0.08 * b                 # eye half-width, half-height
    mw, mh = 0.30 * a, rng.uniform(0.02, 0.12) * b   # mouth half-width, opening
    my, ny = 0.45 * b, 0.15 * b
    canon = np.array([[-ex - ew, ey], [-ex + ew, ey], [ex - ew, ey], [ex + ew, ey],
                      [-ex, ey], [ex, ey], [0, ny], [-mw, my], [mw, my], [0, b]])
    th = np.deg2rad(rng.uniform(-30, 30))
    s = rng.uniform(0.85, 1.15)
    t = SIZE / 2 + rng.uniform(-5, 5, 2)
    R = s * np.array([[np.cos(th), -np.sin(th)], [np.sin(th), np.cos(th)]])
    pts = canon @ R.T + t
    # canonical coordinates of every pixel centre
    yy, xx = np.mgrid[0:SIZE, 0:SIZE].astype(float)
    P = np.stack([xx - t[0], yy - t[1]], -1) @ np.linalg.inv(R).T
    u, v = P[..., 0], P[..., 1]
    face, bg = rng.uniform(0.55, 0.8), rng.uniform(0.1, 0.35)
    img = np.full((SIZE, SIZE), bg)
    img[(u / a) ** 2 + (v / b) ** 2 <= 1] = face
    dark = face - rng.uniform(0.3, 0.45)
    for sx in (-1, 1):
        img[((u - sx * ex) / ew) ** 2 + ((v - ey) / eh) ** 2 <= 1] = dark + 0.1
        img[((u - sx * ex) / (0.4 * eh)) ** 2 + ((v - ey) / (0.9 * eh)) ** 2 <= 1] = dark - 0.1
        img[(np.abs(u - sx * ex) <= 1.1 * ew) & (np.abs(v - (ey - 0.12 * b)) <= 0.03 * b)] = dark
    img[((u / (0.08 * a)) ** 2 + ((v - ny) / (0.05 * b)) ** 2) <= 1] = dark + 0.05
    img[(u / mw) ** 2 + ((v - my) / mh) ** 2 <= 1] = dark
    g = rng.normal(size=2)
    g /= np.linalg.norm(g)
    light = 1 + rng.uniform(0, 0.5) * ((xx - SIZE / 2) * g[0] + (yy - SIZE / 2) * g[1]) / (SIZE / 2)
    img = gaussian_filter(img * light, 0.7) + rng.normal(0, 0.04, img.shape)
    return img.astype(np.float32), pts.astype(np.float32)


def dataset(n, rng):
    imgs, pts = zip(*(make_face(rng) for _ in range(n)))
    return np.stack(imgs), np.stack(pts)


def nme(pred, gt, norm):
    """Per-image mean point-to-point error divided by a normalizing distance."""
    i, j = {"inter-ocular": (0, 3), "inter-pupil": (4, 5)}[norm]
    d = np.linalg.norm(gt[:, i] - gt[:, j], axis=1)
    return np.linalg.norm(pred - gt, axis=2).mean(1) / d


def report(name, pred, gt):
    row = f"{name:28s}"
    for norm in ("inter-ocular", "inter-pupil"):
        e = nme(pred, gt, norm)
        thr = np.linspace(0, 0.10, 1001)
        ced = (e[None, :] <= thr[:, None]).mean(1)
        auc = np.trapezoid(ced, thr) / 0.10
        row += f"  {100 * e.mean():6.2f}  {100 * (e > 0.10).mean():5.1f}  {auc:5.3f}"
    print(row)


rng = np.random.default_rng(0)
Xtr, Ytr = dataset(3000, rng)
Xte, Yte = dataset(500, rng)
d_oc = np.linalg.norm(Yte[:, 0] - Yte[:, 3], axis=1)
d_pu = np.linalg.norm(Yte[:, 4] - Yte[:, 5], axis=1)
print(f"test inter-ocular distance {d_oc.mean():.1f} px, inter-pupil {d_pu.mean():.1f} px")
print(f"{'':28s}  {'inter-ocular':^20s}  {'inter-pupil':^20s}")
print(f"{'method':28s}" + "  {:>6s}  {:>5s}  {:>5s}".format("NME%", "FR%", "AUC") * 2)

# label-free baselines: mean shape at the image centre, and fitted to intensity moments
mean_shape = Ytr.mean(0) - Ytr.mean((0, 1))
report("mean shape, image centre", np.broadcast_to(mean_shape + SIZE / 2, Yte.shape), Yte)
def moments(img):
    w = np.clip(img - np.median(img), 0, None)
    yy, xx = np.mgrid[0:SIZE, 0:SIZE]
    m = w.sum()
    cx, cy = (w * xx).sum() / m, (w * yy).sum() / m
    C = np.cov(np.stack([xx.ravel() - cx, yy.ravel() - cy]), aweights=w.ravel())
    evals, evecs = np.linalg.eigh(C)
    major = evecs[:, 1] * np.sign(evecs[1, 1])          # face's long axis, pointing down
    ang = np.arctan2(-major[0], major[1])
    return np.array([cx, cy]), ang, np.sqrt(evals[1])
def fit_moments(X, Y=None, template=None):
    out = []
    for n, img in enumerate(X):
        c, ang, sc = moments(img)
        Rm = np.array([[np.cos(ang), -np.sin(ang)], [np.sin(ang), np.cos(ang)]]) * sc
        if template is None:
            out.append((Y[n] - c) @ np.linalg.inv(Rm).T)
        else:
            out.append(template @ Rm.T + c)
    return np.stack(out)
canon_mean = fit_moments(Xtr, Ytr).mean(0)
report("mean shape, moment fit", fit_moments(Xte, template=canon_mean), Yte)


class Backbone(nn.Module):
    def __init__(self):
        super().__init__()
        self.f = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(),
            nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.ReLU(),
            nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(),
            nn.Conv2d(32, 64, 3, stride=2, padding=1), nn.ReLU(),
            nn.Conv2d(64, 64, 3, padding=1), nn.ReLU())          # 16 x 16 x 64

    def forward(self, x):
        return self.f(x)


class HeatmapNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.b = Backbone()
        self.h = nn.Sequential(nn.ConvTranspose2d(64, 32, 4, stride=2, padding=1), nn.ReLU(),
                               nn.Conv2d(32, K, 3, padding=1))  # 32 x 32 x K

    def forward(self, x):
        return self.h(self.b(x))


class RegressionNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.b = Backbone()
        self.r = nn.Sequential(nn.Flatten(), nn.Linear(16 * 16 * 64, 256), nn.ReLU(),
                               nn.Linear(256, 2 * K))

    def forward(self, x):
        return self.r(self.b(x)).view(-1, K, 2)


def heatmap_targets(Y, sigma=1.0):
    n = SIZE // STRIDE
    yy, xx = np.mgrid[0:n, 0:n]
    mu = (Y - (STRIDE - 1) / 2) / STRIDE
    return np.exp(-((xx - mu[..., 0, None, None]) ** 2 + (yy - mu[..., 1, None, None]) ** 2)
                  / (2 * sigma ** 2)).astype(np.float32)


def train(model, X, T, loss_fn, epochs=10, bs=64):
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    X, T = torch.from_numpy(X[:, None]), torch.from_numpy(T)
    g = torch.Generator().manual_seed(0)
    for ep in range(epochs):
        perm = torch.randperm(len(X), generator=g)
        for k in range(0, len(X), bs):
            idx = perm[k:k + bs]
            opt.zero_grad()
            loss = loss_fn(model(X[idx]), T[idx])
            loss.backward()
            opt.step()
    return model


def decode(H, quarter):
    n = H.shape[-1]
    flat = H.reshape(*H.shape[:2], -1).argmax(-1)
    py, px = flat // n, flat % n
    out = np.stack([px, py], -1).astype(float)
    if quarter:
        for i in range(H.shape[0]):
            for k in range(K):
                x, y = px[i, k], py[i, k]
                if 0 < x < n - 1 and 0 < y < n - 1:
                    h = H[i, k]
                    out[i, k] += 0.25 * np.sign([h[y, x + 1] - h[y, x - 1], h[y + 1, x] - h[y - 1, x]])
    return out * STRIDE + (STRIDE - 1) / 2


Xte_t = torch.from_numpy(Xte[:, None])
T = heatmap_targets(Yte)                       # ideal maps: the decoding floor, no network
report("targets, argmax (floor)", decode(T, False), Yte)
report("targets, quarter (floor)", decode(T, True), Yte)
torch.manual_seed(0)
hm = train(HeatmapNet(), Xtr, heatmap_targets(Ytr), nn.MSELoss())
with torch.no_grad():
    H = hm(Xte_t).numpy()
report("heatmap, argmax", decode(H, False), Yte)
report("heatmap, quarter shift", decode(H, True), Yte)

scale = lambda Y: (Y - SIZE / 2) / (SIZE / 2)
for name, lf in [("regression, L2 loss", nn.MSELoss()), ("regression, L1 loss", nn.L1Loss())]:
    torch.manual_seed(0)
    rg = train(RegressionNet(), Xtr, scale(Ytr).astype(np.float32), lf)
    with torch.no_grad():
        P = rg(Xte_t).numpy() * (SIZE / 2) + SIZE / 2
    report(name, P, Yte)

test inter-ocular distance 20.6 px, inter-pupil 13.9 px
                                  inter-ocular          inter-pupil     
method                          NME%    FR%    AUC    NME%    FR%    AUC
mean shape, image centre       23.55   96.8  0.008   34.77   99.0  0.002
mean shape, moment fit         15.86   60.6  0.143   23.41   77.8  0.062
targets, argmax (floor)         3.74    0.0  0.626    5.52    0.0  0.448
targets, quarter (floor)        1.88    0.0  0.812    2.78    0.0  0.722
heatmap, argmax                 4.41    0.0  0.559    6.51    1.2  0.349
heatmap, quarter shift          2.94    0.0  0.706    4.34    0.0  0.566
regression, L2 loss             1.97    0.2  0.804    2.91    0.4  0.712
regression, L1 loss             1.81    0.2  0.820    2.67    0.8  0.735

The output came from PyTorch 2.14.1, NumPy 2.5.4 and SciPy 1.18.1 on a CPU with four threads, with seed 0 for the data and for each model; one run took between about 2.5 and 12 minutes depending on the other load on the machine.

The two baselines show that the task is learnable but not trivial: a mean shape at the centre misses by nearly a quarter of the inter-ocular distance, and placing it by the face’s intensity moments, which recovers position, scale and roll roughly, still fails on most faces at the 10% threshold. Both trained networks are far better.

The result went against our plan, which predicted that the heatmap model with the quarter shift would give “the lowest NME, below direct regression with either loss”. Both regression models are more accurate than the heatmap model with either decoder. The target rows explain why: decoding the perfect maps by argmax already costs 3.74% of the inter-ocular distance, because a map cell here is 2 pixels and the inter-ocular distance only 20.6 pixels, so the expected argmax error of about 0.38 of a cell is near 4% of the normalizer. The trained heatmap model is within about one percentage point of that floor with either decoder, while the regression models, which have no grid, reach about the floor of the quarter-shifted perfect maps. In our reading this is the low-resolution regime described in the heatmap regression article, where quantization dominates; benchmark crops are larger (Wang, Bo and Li, 2019, use $256 \times 256$ inputs and $64 \times 64$ maps on faces whose eye distances are many cells), and the synthetic faces vary only by a similarity transform and a small change of shape, which a fully connected head can learn from 3,000 images. The example shows that a heatmap’s resolution has to be judged against the size of the normalizer, not that regression is better in general. As Feng et al. (2018) found, L1 beat L2 for regression, here by a small margin.

The normalizer changes every number but not the ranking. The inter-pupil distance is 0.67 of the outer-corner distance here, so the same predictions have inter-pupil errors about 1.5 times larger, and the failure rate at the same 10% threshold changes too: the argmax heatmap model fails on no face with one normalizer and on 1.2% with the other. This is the effect that makes numbers normalized in different ways incomparable on 300-W and COFW.

The script went through three versions. The first run printed the same table without the two target rows, plus a line with the runtime and the PyTorch version. We then added the two target rows to measure the decoding floor, as a diagnostic, and deleted unused code (a list of landmark names and two dead assignments); the second version was run twice, and both runs printed the same values as each other and as the first run in every row the first run had. Because the printed runtime varied with the load on the machine (219 seconds in the first run, 737 and 193 seconds in the two runs of the second version), we removed the runtime line before the third version, whose output is shown above; no data, model or training setting was changed after the first run.

Applications

• Face recognition. Recognition systems align each face before computing an embedding, from five points in DeepID, or from six fiducial points and then 67 landmarks fitted to a 3D model in DeepFace; the face recognition article describes the alignment step.
• Expression analysis. Zhu et al. (2016) name face alignment as a preprocessing step for face analysis tasks including expression recognition, where landmark geometry and its changes over time describe the face’s deformation.
• Face tracking, animation and editing. Cao et al. (2012) list face tracking, face animation and 3D face modelling among the uses of face alignment, and Kazemi and Sullivan (2014) and Ren et al. (2014) designed their methods for real-time use, including on mobile phones.
• Dataset construction. Face datasets for generative models are cropped and aligned by landmarks, as FFHQ was with dlib’s 68 points.
• 3D face reconstruction. Landmarks constrain the fitting of 3D face models, and 3D alignment methods such as 3DDFA fit a dense model directly (Zhu et al., 2016).

Open Problems

• Large poses and heavy occlusion. Failure rates on WFLW’s pose and occlusion subsets remain about five and two times those on the full test set (Wang, Bo and Li, 2019, Table 1), and 2D landmark definitions themselves break down for profile faces.
• Annotation quality and consistency. When the best methods are judged more accurate than the ground truth on their highest-error fittings (Bulat and Tzimiropoulos, 2017), benchmark errors measure annotation noise as much as the method, and contour points have no unique position along the contour.
• Incompatible schemes and protocols. Different landmark lists, splits, box initializations and normalizers make published numbers hard to compare, a problem the 300-W organisers set out to solve in 2013 and which later protocols reintroduced.
• Demographic reporting. None of the benchmark papers cited here reports landmark accuracy by age, sex or skin tone, and only AFLW reports a sex composition of its faces, so differences in accuracy between groups, which affect the recognition and analysis systems built on landmarks, are not measured by the standard benchmarks.