Face Recognition

Face recognition decides who a face belongs to. In verification (one-to-one, 1:1) a system compares a face with one claimed identity and answers “same person or not”, as a phone does when it unlocks for its owner. In identification (one-to-many, 1:N) it searches a gallery of enrolled people and returns the best-matching candidates, or none. Both run the same pipeline: detect the face, align it using facial landmarks, map it to an embedding vector that describes identity, and compare embeddings with a similarity score and a threshold.

The field moved from holistic subspaces of pixel values (eigenfaces, Fisherfaces) and local texture histograms to deep networks trained on millions of labelled web images, and its benchmarks moved with it: from pair matching on Labeled Faces in the Wild (LFW) to template-based protocols reporting the true accept rate at very low false accept rates (IJB-B, IJB-C), million-distractor identification (MegaFace), and the evaluations of the U.S. National Institute of Standards and Technology (NIST), run on tens of millions of government photographs. The same NIST evaluations measured how error rates differ between demographic groups. This article covers the task definitions and their metrics, the pipeline, the history of methods, the benchmarks as their official documents define them, threshold choice, demographic differentials and presentation attacks. Losses for learning the embedding are covered in metric learning and triplet loss; the margin-based softmax losses (SphereFace, CosFace, ArcFace) are only named here.

Problem Definition

A recognizer maps an aligned face image $x$ to an embedding $f(x) \in \mathbb{R}^d$, here $\ell_2$-normalized, and scores a pair by cosine similarity $s(x, y) = f(x)^\top f(y)$; for unit vectors $\lVert f(x) - f(y) \rVert^2 = 2 - 2 s(x, y)$, so a distance threshold and a similarity threshold are interchangeable (similarity and distance measures). NIST describes recognition the same way: a feature extractor turns images into a template, two templates are compared to give a similarity score, and the score is compared with a threshold (Grother, Ngan & Hanaoka, 2019b). Comparisons of two images of the same person are genuine or mated; comparisons of different people are impostor or non-mated.

Verification. Accept the pair when $s \ge \tau$. Two error rates follow, both functions of $\tau$:

$$\mathrm{FMR}(\tau) = \frac{\#\{\text{impostor pairs with } s \ge \tau\}}{\#\{\text{impostor pairs}\}}, \qquad \mathrm{FNMR}(\tau) = \frac{\#\{\text{genuine pairs with } s < \tau\}}{\#\{\text{genuine pairs}\}}.$$

NIST calls them the false match rate and false non-match rate; benchmark papers usually speak of the false accept rate (FAR) and the true accept rate, $\mathrm{TAR} = 1 - \mathrm{FNMR}$, which are the false and true positive rates of the ROC curve. FaceNet, for example, defines its validation rate VAL$(d)$ and FAR$(d)$ as the fractions of same-identity and different-identity pairs whose distance is at most $d$ (Schroff, Kalenichenko & Philbin, 2015). The standard summary is TAR at a stated FAR, with the threshold set on impostor scores to give that FAR.

Identification. A probe is searched against a gallery of $N$ enrolled people, and the system returns a ranked candidate list, either of fixed length or of the candidates above a threshold. NIST’s FRVT Part 2 report defines the error rates per search (Grother, Ngan & Hanaoka, 2019a):

• the false positive identification rate, $\mathrm{FPIR}(N, T)$, is the fraction of non-mated searches (the probe’s person is not enrolled) that return one or more candidates with a score at or above the threshold $T$;
• the false negative identification rate, $\mathrm{FNIR}(N, R, T)$, is the fraction of mated searches in which the enrolled mate is outside the top $R$ ranks or scores below $T$; the true positive identification rate is $\mathrm{TPIR} = 1 - \mathrm{FNIR}$;
• the cumulative match characteristic ignores the threshold, $\mathrm{CMC}(N, R) = 1 - \mathrm{FNIR}(N, R, 0)$, and the rank-one hit rate is $\mathrm{CMC}(N, 1)$.

Closed-set identification assumes every probe is enrolled and is scored by the CMC, usually rank-1. Open-set identification also searches people who are not enrolled, so it needs a threshold and is scored by FNIR at a fixed FPIR. The report points out that the rank-one hit rate does not depend on scores, so it cannot distinguish a strong match from a weak one (Grother, Ngan & Hanaoka, 2019a). The two settings answer different questions: a watch-list or de-duplication search is open-set, because most people searched are not on the list.

Input and Output

The input is one or more face images or video frames per subject, usually cropped and aligned around a detected face. LFW pairs single $250 \times 250$ images (Huang et al., 2007); the YouTube Faces benchmark pairs whole videos (Wolf, Hassner & Maoz, 2011); IJB-B and IJB-C compare templates, sets of still images and video frames of one subject that an algorithm may fuse into a single representation (Maze et al., 2018). The output of verification is a score and an accept/reject decision at the operating threshold; the output of identification is a candidate list of gallery identities with scores.

Variants

• Verification against one enrolled template (phone unlock, physical access) and identification, which NIST divides by use (Grother, Ngan & Hanaoka, 2019a): positive identification (access without an identity claim, threshold high), negative identification (watch lists, duplicate applications, threshold high), and investigation, where the threshold is zero, a fixed number of candidates (50 in FRTE’s tables) is returned, and a human reviews every list.
• Image, video and template matching. YouTube Faces compares videos; IJB-C’s end-to-end mixed protocol requires the algorithm to detect faces in stills and videos, cluster them, build templates and search, with misses counted when the face was not detected (Maze et al., 2018).
• Cross-condition recognition: cross-pose (NIST reports a few algorithms matching side-view photographs against frontal galleries; Grother, Ngan & Hanaoka, 2019a), cross-age (the FGNet probe set of MegaFace), low-quality and masked faces.
• Face clustering, grouping unlabelled faces by identity, which FaceNet and IJB-B/IJB-C also evaluate.
• Related face analysis tasks that are not recognition: facial landmark detection, facial expression recognition, and attribute classification such as estimating sex or age. NIST stresses that classification algorithms are built around the classes they output, whereas a recognizer has no built-in notion of any particular person (Grother, Ngan & Hanaoka, 2019b).

Challenges

• Intra-personal variation. Belhumeur, Hespanha and Kriegman (1997) cite Moses et al.’s observation that differences between images of one face due to illumination and viewing direction are almost always larger than differences due to identity, and designed Fisherfaces for the lighting and expression case.
• Ageing and image quality. NIST attributes the remaining identification misses of the best 2019 algorithms in large part to long-run ageing, facial injury and poor image quality (Grother, Ngan & Hanaoka, 2019a).
• Look-alikes at scale. As a gallery grows, more enrolled people resemble any given probe. NIST explains the rapid rise of miss rates at very low FPIR by low mate scores (poor quality, ageing) mixing with high non-mate scores from look-alikes, including twins (Grother, Ngan & Hanaoka, 2019a), and MegaFace showed rank-1 rates falling as distractors were added to a fixed probe set (Kemelmacher-Shlizerman et al., 2016).
• Open-set thresholds. The threshold that limits false positives also turns some correct rank-1 matches into misses (Grother, Ngan & Hanaoka, 2019a).
• Training data. Deep recognizers need many images of many identities. Public sets were collected from the web, with label noise (WebFace260M had to be cleaned automatically) and concerns about how the data were collected; several are no longer distributed (see Training data).

Approaches

The pipeline

1. Detect each face, as covered in face detection. LFW’s images are the faces a Viola–Jones detector found (Huang et al., 2007), and IJB-B’s motivation was precisely that datasets filtered by a commodity detector limit pose and occlusion (Whitelam et al., 2017).
2. Align the face to a canonical geometry using landmarks. DeepID detects five landmarks (eye centres, nose tip, mouth corners) and applies a similarity transform defined by the two eye centres and the midpoint of the mouth corners (Sun, Wang & Tang, 2014). DeepFace fits a 2D similarity transform to six fiducial points, then localizes 67 points, fits an affine camera to a generic 3D face shape and warps the face piecewise-affinely to a frontal view, because a similarity transform cannot undo out-of-plane rotation (Taigman et al., 2014). FaceNet used no 2D or 3D alignment beyond a tight crop (scale and translation), and the crop still mattered: 98.87% LFW accuracy on a fixed centre crop of the LFW thumbnail and 99.63% when its own face detector cropped the face first, a step the paper calls extra face alignment (Schroff, Kalenichenko & Philbin, 2015).
3. Embed the aligned crop with a network, which today is trained once on identities that will never be enrolled, so that it acts as a generic extractor of identity information for people it has not seen (Grother, Ngan & Hanaoka, 2019b).
4. Compare embeddings by cosine similarity or Euclidean distance and threshold the score (verification) or rank the gallery (identification). NIST notes that some identification algorithms use fast-search techniques that avoid comparing a probe with all $N$ enrolments (Grother, Ngan & Hanaoka, 2019b), the problem addressed by approximate nearest neighbor search.

Holistic subspaces

Turk and Pentland (1991) projected face images onto the principal components of a training set of faces, which they called eigenfaces, and recognized a face by comparing its projection weights with those of known individuals. Belhumeur, Hespanha and Kriegman (1997) replaced principal components by Fisher’s linear discriminant (Fisherfaces), which seeks directions that separate identities, whereas principal components maximize the total scatter of all images and so retain variation due to lighting and expression. They observed that images of one face under varying illumination and fixed pose lie in a three-dimensional linear subspace if the face is a Lambertian surface without shadowing, and only near it for real faces. In their tests on the Harvard and Yale databases Fisherfaces had lower error rates than eigenfaces. Both treat each pixel of the aligned image as one coordinate of a vector; by our reasoning, a shift of a few pixels or an occluding hand then changes the vector as much as a change of identity can.

Local descriptors

Ahonen, Hadid and Pietikäinen (2006) divided an aligned face into a grid of regions ($7 \times 7$ in their experiments), computed a histogram of local binary patterns in each, concatenated the histograms into one descriptor and compared descriptors with a chi-square distance, optionally weighted per region, with a nearest-neighbour classifier on the FERET protocol (read in the authors’ accepted manuscript). Ahonen and colleagues motivate local regions by robustness to pose and illumination changes. On LFW, the results page lists a high-dimensional LBP method at 95.17% in the category that allows labelled outside data, against 97.35% for the DeepFace ensemble.

Deep embeddings from identity classification

DeepFace (Taigman et al., 2014) trained a nine-layer network with more than 120 million parameters, most in locally connected layers without weight sharing, to classify the 4,030 identities of an internal dataset of 4.4 million labelled faces (SFC) with a softmax output and the cross-entropy loss, and used the $\ell_2$-normalized output of its penultimate layer as the face representation. DeepID (Sun, Wang & Tang, 2014) likewise trained convolutional networks to classify thousands of celebrities (8,700 for its final result), took a 160-dimensional feature from the last hidden layer of a network for each of many face patches (60 in most experiments, 100 for the final result), reduced their concatenation by PCA and verified pairs with a Joint Bayesian model; its abstract argues that features learned by classifying many identities generalize to identities unseen in training, and more so as more classes are predicted. Both reached about 97.4% on LFW (see the table below).

In our reading, what changed was the training target, not the comparison rule: classification over thousands of identities forces the penultimate layer to separate identities, and the softmax classifier is discarded at test time, because the people a deployed system must recognize are not among its classes, which is the framing of metric learning.

Metric losses and margins

FaceNet (Schroff, Kalenichenko & Philbin, 2015) trained the embedding directly for comparison: a 128-dimensional, unit-length output, a triplet loss with online semi-hard mining, and 100 to 200 million training face thumbnails of about 8 million identities. It reached 99.63% on LFW and 95.12% on YouTube Faces. Later systems returned to classification over identities but changed the loss: normalizing the class weights (SphereFace) or both the weights and the features (CosFace, ArcFace), so that logits depend on angles, and adding an angular or cosine margin to the target class, which gave the margin-based softmax family of SphereFace, CosFace and ArcFace. Contrastive and triplet losses, proxies and the normalized softmax are compared in metric learning.

Training data

Deep face recognition is limited by labelled identities. The largest early training sets were private (SFC for DeepFace; FaceNet’s 8 million identities), and public sets were built to close that gap:

• CASIA-WebFace (Yi et al., 2014): 10,575 subjects and 494,414 images after cleaning, collected semi-automatically from the web and released because the best LFW results relied on private data.
• MS-Celeb-1M (Guo et al., 2016): a benchmark for recognizing one million celebrities linked to knowledge-base entities, with a version 1 training set of about 10 million images of the top 100,000 celebrities. Peng, Mathur and Narayanan (2021) record that it was retracted in April 2019, that Microsoft told the press it had been taken down because “the research challenge is over”, that the original remained available through Academic Torrents and archive.org, and that in their 20% sample of papers it or its derivatives were used 54 times in papers published in 2020; at the time of their study they found 20 GitHub repositories still offering models pre-trained on it.
• VGGFace2 (Cao et al., 2018): 3.31 million images of 9,131 subjects from Google Image Search, chosen for many images per identity (362.6 on average) and a wide range of pose, age and ethnicity. Peng, Mathur and Narayanan (2021) list it among datasets retracted after an investigation by Harvey and Laplace raised concerns about how the data were collected and used, and as of October 2026 its official page states that the download links are no longer available from the website.
• WebFace260M (Zhu et al., 2021): a noisy set of 260 million faces of 4 million identities collected from the Internet, cleaned automatically into WebFace42M (42 million faces of 2 million identities), together with a test protocol under inference-time constraints (FRUITS).

Reported results

Results on LFW as each paper prints them (mean accuracy over the ten View 2 folds unless stated):

Method	Year	Training data	LFW accuracy	YouTube Faces accuracy
DeepFace, single network (Taigman et al.)	2014	SFC, 4.4M images of 4,030 people	97.00 ± 0.28% (paper’s “restricted”)	91.4 ± 1.1%
DeepFace, ensemble	2014	SFC	97.35 ± 0.25% (paper’s “unrestricted”)	—
DeepID (Sun, Wang & Tang)	2014	CelebFaces+, 202,599 images of 10,177 celebrities	97.20% (no adaptation); 97.45% (Joint Bayesian adapted on the nine LFW training folds)	—
FaceNet (Schroff et al.)	2015	100M–200M images, about 8M identities	98.87 ± 0.15% (centre crop); 99.63 ± 0.09% (detector crop, “extra face alignment”)	95.12 ± 0.39% (first 100 frames)
Humans, Mechanical Turk (LFW results page)	—	—	99.20% (funneled images); 97.53% (cropped faces)	—

DeepFace labels its protocols “restricted” and “unrestricted” (its Table 3), but both rows use SFC, so under the 2014 LFW reporting rules they fall in the category that allows labelled outside data, where the LFW results page lists the ensemble; FaceNet states that it follows that protocol. The LFW page notes that the human experiment did not control for prior familiarity with the many famous people in the test set. On the template protocols of IJB-C, the arXiv version of the VGGFace2 paper (Table VIII) reports, for its best model (SENet pre-trained on MS-Celeb-1M and fine-tuned on VGGFace2), a TAR of 0.862 at FAR $= 10^{-4}$ and 0.768 at FAR $= 10^{-5}$ for 1:1 verification, a TPIR of 0.763 at FPIR $= 0.01$ and a rank-1 rate of 0.914 for 1:N identification; the same table reads the IJB-C authors’ baseline from their curves as a TAR of 0.75 at $10^{-4}$.

Datasets and Benchmarks

• Labeled Faces in the Wild (Huang et al., 2007): 13,233 images of 5,749 people, 1,680 of them with two or more images, from Viola–Jones detections on images gathered from news articles on the web, released as $250 \times 250$ JPEGs. View 1 is for development; View 2, for reporting, is ten subject-disjoint subsets, each with 300 matched and 300 mismatched pairs, used in ten-fold cross-validation (official README). Each fold’s classifier, threshold included, must be set on the other nine folds, and authors report the mean accuracy $\hat{\mu}$ and the standard error $\hat{\sigma}/\sqrt{10}$ over folds (Huang et al., 2007). The original report defined two training paradigms, image-restricted (only the given pair labels) and unrestricted (identities may be used to form new pairs); the 2014 update replaced them with six protocols, from “unsupervised” through “image-restricted, no outside data” to “unrestricted, labeled outside data”, the most permissive, which allows outside images, identities and annotations of anyone not in the test fold (Huang & Learned-Miller, 2014). The unsupervised protocol is scored by the area under the ROC curve, because without labels there is no principled threshold. The LFW page carries a disclaimer that no LFW result shows an algorithm is suitable for any commercial purpose.
• YouTube Faces (Wolf, Hassner & Maoz, 2011): 3,425 videos of 1,595 people; the benchmark is 5,000 video pairs, half same-person, in ten subject-disjoint splits, each tested after training on the other nine as in LFW. The official page publishes a list of labelling errors and corrected splits (YouTube Faces page, accessed October 2026).
• IJB-B (Whitelam et al., 2017): 1,845 subjects, a superset of IJB-A, with 21,798 still images (11,754 with faces, 10,044 without) and 55,026 frames from 7,011 videos, all under Creative Commons licences and annotated by hand rather than filtered by a detector. Its 1:1 protocol has 10,270 genuine and 8,000,000 impostor comparisons, chosen so that low-FAR operating points can be measured; its 1:N protocol has two disjoint galleries (S1, S2) of one template per subject, so that probes whose mate is in the other gallery test open-set identification.
• IJB-C (Maze et al., 2018): 3,531 subjects (it adds 1,661 to IJB-B; the IJB-C abstract counts IJB-B’s subjects as 1,870, where the IJB-B paper gives 1,845), 31,334 still images (21,294 with faces, 10,040 without) and 117,542 frames from 11,779 videos. Templates are subject-specific; galleries G1 (1,772 subjects) and G2 (1,759) are disjoint; the 1:1 mixed verification protocol provides 19,557 genuine and 15,638,932 impostor comparisons; ROC curves are reported for verification and CMC and identification error trade-off curves for identification. The end-to-end protocols count false positives as a raw number of non-mated searches with a rank-one candidate above threshold rather than a rate, so that a detector producing many false detections cannot dilute its FPIR. The VGGFace2 paper notes that the IJB authors evaluated verification pairs from each gallery separately and averaged them, unlike its own evaluation, a reminder that “TAR at FAR on IJB-C” depends on such details.
• MegaFace (Kemelmacher-Shlizerman et al., 2016): a gallery of distractors, 1,027,060 photos of 690,572 people from Flickr, added to probe sets from FaceScrub (about 100,000 photos of 530 celebrities) and FGNet (large age variation). Identification is scored with CMC curves as the number of distractors grows from 10 to one million; verification pairs every probe with every distractor, about 4 billion negative pairs, scored with ROC curves. A later training set, MF2, has 4.7 million photos of 672,057 identities. As of the archived site of September 2026, MegaFace states that the challenge has concluded, having reached benchmark performance above 99%, that it is being decommissioned, and that its data are no longer distributed.
• NIST evaluations. The Face Recognition Vendor Test (FRVT) evaluates algorithms submitted as compiled libraries, mostly on sequestered photographs from U.S. government applications; in August 2023 it was split and renamed the Face Recognition Technology Evaluation (FRTE) and the Face Analysis Technology Evaluation (FATE) (FRTE pages). The 2019 identification report covered 203 algorithms from 51 commercial developers and one university, on more than 30.2 million photographs of 14.4 million people (Grother, Ngan & Hanaoka, 2019a). As of October 2026 the FRTE 1:1 page tabulates FNMR at thresholds set for stated FMR values on several datasets, ordered by FNMR at FMR $= 10^{-6}$ on its visa-to-border dataset, and the 1:N page tabulates FNIR at a threshold set for FPIR $= 0.003$, separately for each algorithm and dataset, alongside investigation-mode FNIR at rank 1; the 1:1 page counts 1,483 algorithms from 455 developers submitted since 2017. NIST’s view of the academic benchmarks is blunt: it describes LFW with its near-equal-error-rate metric as an easy task, verification at very high false match rates, and MegaFace’s rank-one metric as unlike the high-threshold discrimination that large-population applications require (Grother, Ngan & Hanaoka, 2019a).

Benchmarks saturate and are reused for years, so differences of a few tenths of a percent on LFW are within the risks discussed in benchmark overfitting; LFW’s report itself warns that its standard error over ten folds may be biased downward, because the training sets of the folds overlap (Huang et al., 2007).

Evaluation Metrics

Verification: ROC, DET and TAR at FAR. Besides the ROC curve of TAR against FAR (ROC curve and AUC), NIST plots FNMR against FMR on logarithmic axes, the detection error trade-off (DET) plot; LFW reports accuracy at a threshold chosen on training folds. A low-FAR operating point rests on the few highest-scoring impostor pairs, which is why IJB-B and IJB-C were built with millions of impostor comparisons: at FAR $= 10^{-5}$, IJB-C’s 15.6 million impostor pairs leave about 156 above threshold.

Identification: CMC and FNIR at FPIR. Closed-set results are rank-$k$ rates from the CMC, open-set results FNIR (or TPIR) at fixed FPIR, traced as an identification error trade-off curve (Grother, Ngan & Hanaoka, 2019a). Both depend on the gallery size $N$; NIST’s 2019 identification report used galleries of up to 12 million people.

Choosing the threshold. A threshold is a property of a deployed system, not of an algorithm. NIST sets it from impostor scores to reach a target FMR or FPIR, and stresses that the vast majority of biometric systems use one fixed threshold for everyone, not one tailored to cameras, conditions or demographic groups (Grother, Ngan & Hanaoka, 2019b). It follows that a threshold estimated on one population’s impostors yields a different FMR on another population’s, which the example below shows. LFW’s protocol forbids choosing the threshold on the test fold; the unsupervised protocol forbids choosing it with labels at all (Huang & Learned-Miller, 2014).

Statistical care. Low-FAR operating points rest on few impostor pairs, and, by our reasoning, pairs that share an image or a person are not independent, so binomial confidence intervals that treat every pair as an independent trial are too narrow; the general issues are in statistical significance in evaluation.

Example: Verification, Identification and One Threshold for Two Groups

The example trains a small embedding network on synthetic faces and evaluates it as the benchmarks do. Each identity is a vector of eight geometric parameters: head width and height, eye spacing, height and size, nose length, mouth width and mouth height. Each image is a $32 \times 32$ floating-point rendering of that face under random in-plane rotation, shift and scale, a random smile width and eye opening, a lighting ramp, a gain and sensor noise, standardized to zero mean and unit variance. Two synthetic populations differ in how their identities are distributed: group A identities fill the whole parameter range, while group B identities are drawn from the upper half of every range, so they resemble each other more. The training set has 300 group A identities and 30 group B identities, so B is also rare in training, and the two causes are not separated here.

The network is trained to classify the training identities with a normalized softmax (cosine logits times 16, no margin), then evaluated on 150 new identities from each group with ten images each. Verification uses all pairs. Separately, one threshold is set for FMR $= 10^{-3}$ on group A impostor pairs only and applied to everyone, as NIST’s mugshot analysis did with white male impostors (Grother, Ngan & Hanaoka, 2019b); for comparison, group B’s FNMR is also computed at a threshold set on B’s own impostors. Identification enrols the first image of 100 identities of each group; the other 50 of each group are not enrolled, and their images form the non-mated searches. PyTorch 2.14.1, four threads, three seeds, about 30 s on a laptop CPU.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(4)
H = 32
yy, xx = torch.meshgrid(torch.arange(H) - 15.5, torch.arange(H) - 15.5, indexing="ij")

def identities(n, group, gen):
    """Identity = 8 geometric parameters in [0, 1]. Group A spreads over the whole
    cube; group B is concentrated in a sub-cube (a population of similar faces)."""
    u = torch.rand(n, 8, generator=gen)
    return u if group == "A" else 0.5 + 0.5 * u

def render(p, gen):
    """One 32x32 image per row of p: ellipse head, two eyes, nose, mouth, under
    random in-plane rotation, shift, scale, expression, lighting ramp and noise."""
    n = len(p)
    U = lambda lo, hi, *s: lo + (hi - lo) * torch.rand(n, *s, generator=gen)
    L = lambda k, lo, hi: (lo + (hi - lo) * p[:, k])[:, None, None]
    th, s = torch.deg2rad(U(-8, 8))[:, None, None], U(0.94, 1.06)[:, None, None]
    sx, sy = U(-1.5, 1.5)[:, None, None], U(-1.5, 1.5)[:, None, None]
    x = ((xx - sx) * th.cos() + (yy - sy) * th.sin()) / s     # face coordinates
    y = (-(xx - sx) * th.sin() + (yy - sy) * th.cos()) / s
    a, b = L(0, 8, 13), L(1, 10.5, 15)                         # head half-width, half-height
    ex, ey, er = L(2, 3, 7), L(3, -7, -2), L(4, 1, 2.6)
    nl, mw, my = L(5, 1, 5.5), L(6, 2.5, 7.5), L(7, 3.5, 8.5)
    mw = mw * U(0.85, 1.15)[:, None, None]                     # expression: smile width
    lid = U(0.5, 1.0)[:, None, None]                           # expression: eye opening
    head = (x / a) ** 2 + (y / b) ** 2 < 1
    eyes = ((x.abs() - ex) / er) ** 2 + ((y - ey) / (er * lid)) ** 2 < 1
    nose = (x.abs() < 0.8) & (y > ey + 1) & (y < ey + 1 + nl)
    mouth = (x.abs() < mw) & ((y - my).abs() < 1.0)
    g = U(-0.03, 0.03, 2)[:, :, None, None]
    img = (U(0.1, 0.4)[:, None, None]                                       # background
           + head * U(0.4, 0.6)[:, None, None]                              # skin
           - (eyes | nose | mouth) * head * U(0.35, 0.5)[:, None, None]     # dark parts
           + g[:, 0] * xx + g[:, 1] * yy) * U(0.7, 1.3)[:, None, None]      # light ramp, gain
    img = img + 0.06 * torch.randn(n, H, H, generator=gen)                  # sensor noise
    img = (img - img.mean((1, 2), keepdim=True)) / img.std((1, 2), keepdim=True)
    return img[:, None]

def embedder(d=64):
    return nn.Sequential(
        nn.Conv2d(1, 16, 3, padding=1), nn.BatchNorm2d(16), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Linear(64 * 4 * 4, d))      # keep layout: where parts are matters

def train(n_a, n_b, per_id, epochs, gen, lr=1e-2):
    """Identity classification with a normalized (cosine) softmax, scale 16, no margin."""
    P = torch.cat([identities(n_a, "A", gen), identities(n_b, "B", gen)])
    y = torch.arange(len(P)).repeat_interleave(per_id)
    f, W = embedder(), nn.Parameter(0.1 * torch.randn(len(P), 64, generator=gen))
    opt = torch.optim.Adam(list(f.parameters()) + [W], lr=lr)
    steps = epochs * -(-len(y) // 128)
    sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=lr, total_steps=steps)
    for _ in range(epochs):
        X = render(P[y], gen)                                   # fresh variations every epoch
        for idx in torch.randperm(len(y), generator=gen).split(128):
            cos = F.normalize(f(X[idx])) @ F.normalize(W).T
            loss = F.cross_entropy(16 * cos, y[idx])
            opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    return f

def rate_threshold(scores, rate):
    """Lowest threshold t with fraction(scores >= t) <= rate (scores are 1-D)."""
    k = int(rate * len(scores))                                 # allowed false accepts
    return scores.topk(k + 1).values[-1].item() + 1e-7

def evaluate(f, gen, n_id=150, per_id=10):
    PA, PB = identities(n_id, "A", gen), identities(n_id, "B", gen)
    P = torch.cat([PA, PB])                                     # unseen identities
    ids = torch.arange(2 * n_id).repeat_interleave(per_id)
    grp = (ids >= n_id)                                         # False = A, True = B
    with torch.no_grad():
        E = F.normalize(f(render(P[ids], gen)))
    S = E @ E.T
    iu = torch.triu_indices(len(ids), len(ids), 1)
    s, same = S[iu[0], iu[1]], ids[iu[0]] == ids[iu[1]]
    gA, gB = ~grp[iu[0]] & ~grp[iu[1]], grp[iu[0]] & grp[iu[1]]
    out = {}
    t = rate_threshold(s[~same], 1e-3)                          # 1:1 over all pairs
    out["TAR@1e-3"] = (s[same] >= t).float().mean().item()
    tA = rate_threshold(s[~same & gA], 1e-3)                    # set on group A impostors...
    out["FNMR_A"] = (s[same & gA] < tA).float().mean().item()   # ...applied to everyone
    out["FMR_B"] = (s[~same & gB] >= tA).float().mean().item()
    out["FNMR_B"] = (s[same & gB] < tA).float().mean().item()
    tB = rate_threshold(s[~same & gB], 1e-3)                    # B's own threshold
    out["FNMR_B@own"] = (s[same & gB] < tB).float().mean().item()
    # 1:N: gallery = first image of 100 A and 100 B identities; the other 50 + 50 not enrolled
    perm = torch.cat([torch.randperm(n_id, generator=gen), n_id + torch.randperm(n_id, generator=gen)])
    enrolled = torch.cat([perm[:100], perm[n_id:n_id + 100]])
    gal = enrolled * per_id                                     # index of each gallery image
    is_gal = torch.zeros(len(ids), dtype=torch.bool); is_gal[gal] = True
    mated = torch.isin(ids, enrolled) & ~is_gal
    nonmated = ~torch.isin(ids, enrolled)
    top, arg = S[:, gal].max(1)
    hit = ids[gal][arg] == ids
    out["rank1"] = hit[mated].float().mean().item()             # closed set
    T = rate_threshold(top[nonmated], 0.01)                     # FPIR = 0.01
    out["FNIR@.01"] = 1 - (hit & (top >= T))[mated].float().mean().item()   # open set, R = 1
    return out

print("PyTorch", torch.__version__)
keys = ["TAR@1e-3", "FNMR_A", "FMR_B", "FNMR_B", "FNMR_B@own", "rank1", "FNIR@.01"]
print("seed" + "".join(f"{k:>11}" for k in keys))
runs = []
for seed in range(3):
    gen = torch.Generator().manual_seed(seed)
    torch.manual_seed(seed)
    f = train(n_a=300, n_b=30, per_id=12, epochs=8, gen=gen).eval()
    runs.append(evaluate(f, gen))
    print(f"{seed:4d}" + "".join(f"{runs[-1][k]:11.4f}" for k in keys))
print("min " + "".join(f"{min(r[k] for r in runs):11.4f}" for k in keys))
print("max " + "".join(f"{max(r[k] for r in runs):11.4f}" for k in keys))

PyTorch 2.14.1
seed   TAR@1e-3     FNMR_A      FMR_B     FNMR_B FNMR_B@own      rank1   FNIR@.01
   0     0.4247     0.2844     0.0100     0.4726     0.8236     0.6839     0.8889
   1     0.3793     0.3459     0.0113     0.4957     0.8436     0.6661     0.9261
   2     0.3710     0.3757     0.0069     0.6311     0.8680     0.6300     0.8922
min      0.3710     0.2844     0.0069     0.4726     0.8236     0.6300     0.8889
max      0.4247     0.3757     0.0113     0.6311     0.8680     0.6839     0.9261

Running the script twice gives identical output. Three things stand out.

One threshold, two false match rates. The threshold that gives group A an FMR of $10^{-3}$ gives group B an FMR between 0.0069 and 0.0113 across the seeds, seven to eleven times the target, and at the same threshold B’s genuine pairs are also rejected more often (FNMR 0.47 to 0.63 against 0.28 to 0.38 for A). Our reading is that the network learned to spread group A’s faces over the embedding space while group B’s faces, more alike and rare in training, stay packed closer together, so B’s impostor scores run higher. The setup is synthetic and models no real population, but the mechanism, a score distribution that differs by group meeting a threshold that does not, is the one NIST describes.

Per-group thresholds hide the excursion. Setting a separate threshold for B at its own FMR of $10^{-3}$ brings its false matches back to target, and the cost moves to the other error: FNMR rises to 0.82–0.87. Reporting FNMR at a fixed FMR per group, which NIST says most academic studies do, would show only this second number, while a deployed system with one threshold would see the first pair. NIST makes the same objection (see below).

Rank-1 is not open-set accuracy. The closed-set rank-1 rate is 0.63 to 0.68, but once the threshold must keep false positives to 1% of searches by people who are not enrolled, the mate is returned above threshold at rank 1 in only 7% to 11% of mated searches (FNIR 0.89 to 0.93). Presumably the non-mated searches by group B faces, which find look-alikes among the enrolled B identities, push the threshold up; the script does not print scores by group to confirm this. The overall TAR at FAR $= 10^{-3}$ (0.37 to 0.42) is also below group A’s TAR at the same FMR ($1 - \mathrm{FNMR}_A$, 0.62 to 0.72), which by our reasoning reflects B’s high-scoring impostor pairs filling much of the top $10^{-3}$ of the pooled impostor scores. A single benchmark number therefore depends on the mix of people in the test set.

Demographic Differentials

The most extensive measurement is NIST’s FRVT Part 3 report (Grother, Ngan & Hanaoka, 2019b, published December 2019). It ran 189 mostly commercial algorithms from 99 developers on 18.27 million images of 8.49 million people from four U.S. government collections: domestic mugshots, applications for immigration benefits, visa photographs and border-crossing photographs. The mugshots carry race labels; the other sets carry country of birth, which the report restricts to 24 countries in 7 regions. The findings it states, each for a stated metric and threshold:

• False positives differ far more than false negatives. Across demographic groups, false positive rates often varied by factors of 10 to beyond 100; false negative rates varied by factors usually below 3, and those effects were more algorithm-specific.
• Application photographs (verification, thresholds set on mugshot impostors for FMR $= 3 \times 10^{-5}$, impostors matched in sex and age): false match rates were highest for people from West and East Africa and East Asia and lowest for Eastern Europeans, generally by a factor of 100 between countries; a number of algorithms developed in China reversed the effect, with low false match rates on East Asian faces.
• Mugshots (126 verification algorithms, 3 million impostor comparisons per sex–race group, threshold fixed so that white male impostors give FMR $= 10^{-4}$): the highest false match rates were for American Indians, with elevated rates for African American and Asian people. For one algorithm shown in detail (Imperial-002), the FMR for American Indian women was 0.0068, 68 times the white male rate; for American Indian men the multiple was 47.
• Sex and age. For verification algorithms, false positives were 2 to 5 times higher for women than for men, for most algorithms and datasets, and elevated for the elderly and for children.
• False negatives depended on image quality: with mugshots they were higher for Asian and American Indian people (a result the report says might instead reflect the time elapsed between photographs), while with the lower-quality border-crossing images they were generally higher for people born in Africa and the Caribbean, more so for older people.
• Identification. Differentials in verification were usually, but not always, present in one-to-many search; some developers supplied identification algorithms whose false positive differentials were undetectable.

The report also sets out what it did not do: it did not train or adapt algorithms, did not analyse causes (it did not relate errors to skin tone or other phenotypes), and did not use Internet or surveillance images. It argues that because systems run at one fixed threshold, both false negative and false positive rates should be reported per group at that threshold, and that reporting false negative rates at a fixed false positive rate per group hides the false positive excursions, the effect the example above reproduces. NIST has continued the work in later FRTE reports, including a 2022 report summarizing demographic differentials (Part 8, NISTIR 8429), listed in Ngan, Grother and Hom (2023).

Buolamwini and Gebru (2018) is often cited in this context, but it evaluated gender classification, not recognition: three commercial systems that label a face as male or female, on a benchmark of 1,270 parliamentarians balanced by sex and Fitzpatrick skin type, with error rates of up to 34.7% for darker-skinned women against at most 0.8% for lighter-skinned men. NIST’s report makes the distinction itself, noting that the gender classification studies did not evaluate face recognition algorithms although their results have been widely cited about recognition (Grother, Ngan & Hanaoka, 2019b).

Applications

Beyond the uses listed under Variants, NIST’s examples (Grother, Ngan & Hanaoka, 2019b) include verification to authorize a transaction such as dispensing a prescription drug, and negative identification to check whether a visa or driving-licence applicant is already enrolled under another name. The consequences of the two error types depend on the use: in access control a false negative inconveniences a legitimate user and can be retried, while in identification for fraud detection or surveillance the report notes that a false positive could lead to a false accusation, detention or deportation. Face clustering in photo collections, which FaceNet evaluated, groups faces without a gallery. Retrieval of faces from large collections shares its indexing with image retrieval.

Presentation attacks

A recognizer compares whatever face reaches the camera. A presentation attack, as ISO/IEC 30107 defines it and NIST quotes it, presents an artefact or human characteristics to the capture device in order to interfere with the system’s policy: a printed photo, a replayed video or a silicone mask held to the camera, with the intent of impersonation (being accepted as someone else) or evasion (not being recognized) (Ngan, Grother & Hom, 2023). Presentation attack detection, also called anti-spoofing, is scored by the attack presentation classification error rate (APCER, attacks accepted as genuine) and the bona fide presentation classification error rate (BPCER, genuine presentations rejected). NIST’s FATE evaluation of passive, software-only detectors covered physical attacks; digital injection attacks, which feed images to the software directly, were out of scope.

Open Problems

• Very low error rates at scale. NIST’s 2019 identification report found that requiring only one non-mated search in a thousand to produce a false positive made even the most accurate algorithm miss 7.9% of mated searches in a 12-million-person gallery, and that miss rates rose quickly at stricter thresholds (Grother, Ngan & Hanaoka, 2019a).
• Demographic differentials. NIST lists measures that, with adequate research and development, may prove effective against false positive differentials: threshold elevation, refined training, more diverse training data, features that distinguish twins, and combining face with iris; it discounts user-specific thresholds (Grother, Ngan & Hanaoka, 2019b). The report notes that the degree of differential that could be tolerated has never been formally specified for any biometric application.
• Data provenance. The largest public training sets were collected from the web, and the retraction of MS-Celeb-1M did not remove copies, derived datasets and pre-trained models from circulation; Peng, Mathur and Narayanan (2021) conclude that retraction alone has a limited effect and argue for ongoing stewardship of datasets.
• Benchmarks that match deployment. Academic benchmarks and NIST’s evaluations differ in data (web versus government photographs) and operating points, so, in our reading, results on one need not predict results on the other.
• Attacks. Presentation attacks come from an open-ended set of instruments, limited by the attacker’s effort and ingenuity, and digital injection attacks fell outside NIST’s 2023 detection evaluation (Ngan, Grother & Hom, 2023).