DeepFace

DeepFace is a face verification system published by Yaniv Taigman, Ming Yang and Marc’Aurelio Ranzato of Facebook AI Research and Lior Wolf of Tel Aviv University at CVPR 2014. It decides whether two face images show the same person, the one-to-one setting of face recognition, in three steps: it aligns each detected face to a frontal view with an explicit 3D face model (“frontalization”), maps the aligned image to a 4,096-dimensional feature vector with a nine-layer convolutional neural network whose middle layers are locally connected rather than convolutional, and compares two feature vectors with a simple similarity. The network was trained to classify 4,030 identities in an internal set of 4.4 million labelled faces, which the paper calls the Social Face Classification (SFC) dataset. An ensemble of such networks reached 97.35% mean accuracy on Labeled Faces in the Wild (LFW), against 97.53% for humans judging cropped faces, the gap of the paper’s title, and a single network reached 91.4% on YouTube Faces (Taigman et al., 2014).

DeepFace belongs to the first generation of deep face recognizers of 2014, with the DeepID networks of Sun, Wang and Tang, which learned a representation by classifying identities and then discarded the classifier. A year later FaceNet trained the embedding directly with a triplet loss and needed no 3D alignment, and the margin-based softmax losses that followed made identity classification the dominant training method again. This article covers the model as the CVPR paper describes it, its results with their protocols, the examples of its alignment idea that followed, and its place in that sequence.

Problem Addressed

The paper describes the conventional face recognition pipeline of 2014 as four stages, detect, align, represent and classify, and it revisits two of them: alignment and representation (Taigman et al., 2014). Methods that led LFW at the time combined tens of thousands of hand-engineered appearance descriptors, such as high-dimensional local binary patterns, with learned metrics; the strongest method listed in its comparison, a transfer-learned Joint Bayesian model, reached 96.33% on LFW. DeepFace’s authors argued for the opposite emphasis: learn the representation from raw RGB pixels with a large network and a large labelled set, keep the representation compact, and keep the comparison step “trivial”, so that the accuracy could be credited to the features.

Two observations shaped the design. First, alignment by a 2D similarity transform, as in the LFW-a aligned version of LFW, corrects scale, in-plane rotation and translation but cannot compensate for out-of-plane rotation, which is common in unconstrained photographs. The paper argues that faces are 3D objects and that 3D modelling, then out of favour for unconstrained images, is the right way to align them. Second, once faces are aligned well, each facial region sits at a nearly fixed pixel position, so the spatial stationarity that justifies weight sharing in convolution no longer holds: the region between the eyes and eyebrows differs in appearance and discriminative value from the region between nose and mouth. That motivated layers that learn different filters at every position.

Architecture

DeepFace is a pipeline of a geometric front end and a learned network. A face detection crop goes through 2D alignment and 3D frontalization to a $152 \times 152$ RGB image, which a nine-layer network (C1, M2, C3, L4, L5, L6, F7, F8 in the paper’s notation) maps to the face representation. Figure 1 of the paper shows the alignment stages and Figure 2 the network with the size of every layer.

2D alignment

Six fiducial points are detected inside the detection crop, at the centres of the eyes, the tip of the nose and mouth locations. The detector is a support vector regressor that predicts point configurations from a histogram of local binary patterns, applied in several iterations: after each one the image is transformed by the induced similarity and the detector is run again. Each iteration $i$ fits a similarity transform $T^i_{2d} = (s_i, R_i, t_i)$ that maps the six detected points to six anchor locations,

$$x^j_{\text{anchor}} = s_i \, [R_i \mid t_i] \, x^j_{\text{source}}, \qquad j = 1, \dots, 6,$$

and the iterations stop when the warped image no longer changes substantially. The composition $T_{2d} = T^1_{2d} \cdots T^k_{2d}$ gives the 2D-aligned crop (Taigman et al., 2014). The paper notes that this is similar to the alignment used for LFW-a.

3D alignment

A second regressor localizes 67 fiducial points $x_{2d}$ in the 2D-aligned crop. The generic 3D shape is the average of the 3D scans of the USF Human-ID database, with 67 anchor points placed on it by hand, which gives a full correspondence between the detected points and their 3D references $x_{3d}$. An affine 3D-to-2D camera $P$ of size $2 \times 4$, eight unknowns stacked in the vector $\vec{P}$, is fitted by generalized least squares:

$$\operatorname{loss}(\vec{P}) = r^\top \Sigma^{-1} r, \qquad r = x_{2d} - X_{3d} \vec{P},$$

where $X_{3d}$ is the $(67 \cdot 2) \times 8$ matrix that stacks, for each reference point, the $2 \times 8$ block $\left[x_{3d}(i)^\top, 1, \vec{0};\ \vec{0}, x_{3d}(i)^\top, 1\right]$ with $\vec{0}$ a row of four zeros. $\Sigma$ is a $(67 \cdot 2) \times (67 \cdot 2)$ covariance matrix of the landmark errors, estimated beforehand, so that noisy points, such as those on the face contour whose detected position depends strongly on depth and viewing angle, weigh less. A Cholesky factorization of $\Sigma$ turns the problem into ordinary least squares. An affine camera is the pinhole camera model without perspective division, which the paper acknowledges by calling the fitted $P$ an approximation.

Frontalization

Because the affine camera models neither perspective nor non-rigid deformation, the authors do not warp the face onto the generic shape exactly. They add the fitting residuals $r$ to the $x$–$y$ components of the 3D reference points, giving relaxed targets $\tilde{x}_{3d}$; without this relaxation, they write, all faces would be warped to the same 3D shape and lose discriminative information. The frontalized image is then produced by a piecewise affine transformation from $x_{2d}$ to $\tilde{x}_{3d}$ over the Delaunay triangulation of the 67 points, with extra triangles on the contour to avoid discontinuities, a form of image warping. Triangles that are invisible with respect to the fitted camera “can be replaced” by blending with their symmetric counterparts; the paper does not say whether its experiments did so. A footnote adds that $T_{2d}$ can be composed into the final warp, so the image need not be resampled twice. Figure 1(h) of the paper shows a new view rendered from the 3D model, which the paper states it did not use.

Front-end layers: C1, M2, C3

The first convolutional layer C1 applies 32 filters of size $11 \times 11 \times 3$ to the $152 \times 152$ input, giving $142 \times 142$ maps; M2 max-pools over $3 \times 3$ neighbourhoods with stride 2 ($71 \times 71$); C3 applies 16 filters of size $9 \times 9$ ($63 \times 63$). The text gives C3’s filters as $9 \times 9 \times 16$, but Figure 2 gives $16 \times 9 \times 9 \times 32$, which matches the 32 maps that M2 outputs. Pooling is applied only once, because, the authors argue, repeated pooling would discard the precise position of facial structures and micro-textures that aligned faces make available. They describe these layers as an adaptive pre-processing stage that does most of the computation with very few parameters.

Locally connected layers: L4, L5, L6

L4, L5 and L6 are locally connected: like convolutional layers they apply a filter bank over local neighbourhoods, but every output location has its own filters. By Figure 2, L4 has 16 filters of $9 \times 9 \times 16$ at each of $55 \times 55$ positions, L5 16 filters of $7 \times 7 \times 16$ at $25 \times 25$ positions (which implies a stride of 2, by our arithmetic) and L6 16 filters of $5 \times 5 \times 16$ at $21 \times 21$ positions. Removing weight sharing leaves the computation unchanged but multiplies the number of parameters, which, the authors write, only a large labelled set made affordable. A second argument is the receptive field: each L6 output depends on a $74 \times 74 \times 3$ patch of the input, and in aligned faces patches that large share hardly any statistics.

Fully connected layers and the representation

F7 (4,096 units) and F8 are fully connected, so they can combine features from distant parts of the face, such as the position and shape of the eyes and of the mouth. F8 has one output per training identity (4,030 in Figure 2) and feeds a softmax. The output of F7 is the face representation $G(I)$. A ReLU follows every convolutional, locally connected and fully connected layer except the last, and dropout is applied only to F7. On average 75% of the components in the topmost layers are exactly zero, which the paper attributes mainly to the ReLU.

The paper states that the network has more than 120 million parameters, more than 95% of them in the locally connected and fully connected layers. By our arithmetic from the layer sizes in Figure 2, with one bias per output unit, L4 has about 62.8 million parameters, F7 about 28.9 million, F8 about 16.5 million, L5 7.9 million and L6 2.8 million, and C1 and C3 together about 53,000, about 119 million in all, of which more than 99.9% are outside C1 and C3. The small gap to “more than 120 million” may come from details that the paper does not state. The follow-up paper by the same authors describes the baseline DeepFace system as having “about 100 million parameters” (Taigman et al., 2015).

Before comparison the features are normalized. Each component is divided by its largest value over the training set, then the vector is scaled to unit length:

$$\bar{G}(I)_i = \frac{G(I)_i}{\max(G_i, \epsilon)}, \qquad f(I) = \frac{\bar{G}(I)}{\lVert \bar{G}(I) \rVert_2},$$

with $\epsilon = 0.05$. The paper motivates the first step as reducing sensitivity to illumination; because the network has biases and ReLUs, it notes, the representation is not invariant to a rescaling of the image intensities.

Training

Data. The paper describes SFC as a large collection of photos from Facebook: 4.4 million labelled faces of 4,030 people, each with 800 to 1,200 faces. The identities were labelled by humans, with typically about 3% errors. For each identity the most recent 5% of images, by time stamp, were held out for testing, to simulate continuous identification through ageing. The authors state that they checked by name labels, with several automatic methods, that no SFC identity occurs in LFW or YouTube Faces, and that social photos vary more in quality, lighting and expression than the professional photographs of celebrities in those benchmarks. The paper refers to its supplementary material for more details of SFC; that material is not on the CVF open-access page and was not read for this article. The paper reports no demographic composition of SFC. Its abstract rounds the set to “four million facial images belonging to more than 4,000 identities”, and the follow-up paper describes it as four million images of 4,030 anonymized identities (Taigman et al., 2015).

Objective. The network is trained as a $K$-way identity classifier. With $o_k$ the $k$-th output of F8,

$$p_k = \frac{\exp(o_k)}{\sum_h \exp(o_h)}, \qquad L = -\log p_k \quad \text{for the true identity } k,$$

the cross-entropy loss of the softmax, minimized by stochastic gradient descent with backpropagation.

Procedure. The authors used a GPU-based engine, momentum 0.9, mini-batches of 128 and one learning rate of 0.01 for all layers, divided by ten by hand each time the validation error stopped decreasing, down to 0.0001. Weights were initialized from a zero-mean Gaussian with $\sigma = 0.01$ and biases set to 0.5. Training took about 15 epochs and 3 days. The authors report no significant overfitting during training, which they attribute to the size of the training set.

Scale and depth. The paper validates design choices by the classification error on the held-out 5% of SFC (its Table 1):

Network	SFC test error	Network	SFC test error	Network	SFC test error
DF-1.5K	7.00%	DF-10%	20.7%	DF-sub1	11.2%
DF-3.3K	7.22%	DF-20%	15.1%	DF-sub2	12.6%
DF-4.4K	8.74%	DF-50%	10.9%	DF-sub3	13.5%

The first column trains on subsets of 1.5K, 3K and 4K people (1.5, 3.3 and 4.4 million faces). The labels DF-3.3K and DF-4.4K carry the face counts in millions rather than the person counts (3K and 4K), while for DF-1.5K both counts are 1.5; the text reads the column by persons (7.0% on 1.5K persons, 7.2% on 3K). The error grows only modestly as the number of classes rises, which the authors read as the network accommodating the scale. The middle column keeps all identities but uses 10%, 20% or 50% of the images; the error rises to 20.7% because of overfitting, and since performance had not saturated at 4 million images, the authors conclude that more data would help. The right column removes C3 (sub1), L4 and L5 (sub2) or all three (sub3); the shallower networks stop improving after a few epochs at higher error.

Inference

At test time a face is detected, aligned, frontalized and passed through C1 to F7; the normalized F7 vector $f$ is the descriptor, and two descriptors are compared. On a single 2.2 GHz Intel core, the authors’ CPU implementation extracted features in 0.18 seconds and aligned in about 0.05 seconds, and the whole system ran at 0.33 seconds per image, including decoding, detection, alignment, the network and classification (Taigman et al., 2014).

Verification metrics

The paper’s stated aim is an unsupervised metric that generalizes across datasets, because fitting a model to a small target dataset reduces generalization; it gives as an example of domain bias that LFW is about 75% male and consists of celebrities photographed mostly by professionals. It used three comparison rules.

• Inner product. The unsupervised similarity is $f_1^\top f_2$, the cosine similarity of the normalized vectors (similarity and distance measures).
• Weighted $\chi^2$. The normalized feature is non-negative, sparse and bounded in $[0, 1]$, like a histogram of local binary patterns, so the authors borrowed the weighted $\chi^2$ similarity used for those histograms (Ahonen, Hadid & Pietikäinen, 2006):

$$\chi^2(f_1, f_2) = \sum_i w_i \frac{(f_1[i] - f_2[i])^2}{f_1[i] + f_2[i]},$$

with the weights $w_i$ learned by a linear support vector machine on the vectors of per-component terms $(f_1[i] - f_2[i])^2 / (f_1[i] + f_2[i])$. In the LFW experiments the paper instead trains a kernel SVM with $C = 1$ on the $\chi^2$-distance vector.

• Siamese network. The trained network without its top layer is replicated for the two images; the absolute difference of the two feature vectors feeds a fully connected layer with a single logistic unit (same or not same), trained with cross-entropy. The induced distance is $d(f_1, f_2) = \sum_i \alpha_i \lvert f_1[i] - f_2[i] \rvert$ with trainable $\alpha_i$. The paper credits the Siamese architecture to Chopra, Hadsell and LeCun (CVPR 2005). Only the two topmost layers are trained, to prevent overfitting; the shared weights keep the parameter count roughly that of one network, at twice the computation. Learning a comparison from pairs is the subject of metric learning.

Example: frontalization before verification

The script below builds synthetic 3D faces, renders them at known yaw angles and verifies pairs by cosine similarity after four kinds of alignment: none, a 2D similarity transform from six fiducials, and 3D frontalization with and without DeepFace’s residual relaxation. Each identity is a height field over a frontal coordinate frame (an ellipsoid with a nose, varied per identity in depth and nose size) carrying a texture of eyes, brows, mouth and smooth blobs at identity-specific positions. Faces are rendered by orthographic projection after a yaw rotation, with a depth buffer, Lambertian shading under a random light, a random shift, scale and in-plane roll standing in for detector jitter, and noise. A stand-in for the paper’s landmark regressors returns the true projected positions of 67 landmarks and six fiducials (eye centres, nose tip, both mouth corners and the mouth centre, our reading of the paper’s “mouth locations”) plus Gaussian noise of 0.7 pixels.

The 3D pipeline follows the paper’s equations with two simplifications: the affine camera is fitted by ordinary least squares ($\Sigma = I$) directly in the input image, which composes $T_{2d}$ into one warp as the paper’s footnote allows, and invisible triangles are not filled by symmetry. The embedding is an open choice, so the script prints both: masked raw pixels, mean-removed and unit-normalized, and a 40-component PCA fitted separately for each pipeline on images of 120 training identities at random yaw. Neither is a deep network; the example isolates the alignment step. Each of 120 test identities has a near-frontal gallery image and one probe at each yaw; genuine pairs share an identity, impostor pairs are all other gallery–probe combinations, and the score is the area under the ROC curve. The script uses NumPy 2.3.5, scikit-image 0.25.2 and scikit-learn 1.7.2, runs in about 35 seconds on our machine, and two runs gave identical output.

import numpy as np
from skimage.transform import SimilarityTransform, PiecewiseAffineTransform, warp
from sklearn.decomposition import PCA
from sklearn.metrics import roc_auc_score

SIZE, C, A, B = 64, 32.0, 20.0, 26.0          # image size, centre, face half-width/height (px)
rng = np.random.default_rng(0)

def new_identity(generic=False):
    d = (lambda s: 0.0) if generic else (lambda s: rng.normal(0, s))
    p = dict(eye_dx=8 + d(1.2), eye_y=-6 + d(1.0), brow_up=4 + d(0.8), brow_tilt=d(0.15),
             nose_y=3 + d(1.0), nose_amp=7 * (1 + d(0.25)), depth=18 * (1 + d(0.08)),
             mouth_y=12 + d(1.0), mouth_w=6 + d(0.8))
    if not generic:
        p["skin"] = 0.55 + rng.normal(0, 0.05)
        p["blobs"] = [(rng.uniform(-14, 14), rng.uniform(-18, 18), rng.normal(0, 0.12),
                       rng.uniform(2.5, 5)) for _ in range(6)]
    return p

def depth(p, u, v):
    base = p["depth"] * np.sqrt(np.clip(1 - (u / A) ** 2 - (v / B) ** 2, 0, None))
    return base + p["nose_amp"] * np.exp(-u ** 2 / 18 - (v - p["nose_y"]) ** 2 / 72)

def texture(p, u, v):
    t = np.full_like(u, p["skin"])
    for bu, bv, amp, s in p["blobs"]:
        t += amp * np.exp(-((u - bu) ** 2 + (v - bv) ** 2) / (2 * s * s))
    for sx in (-1, 1):
        ex, ey = sx * p["eye_dx"], p["eye_y"]
        t -= 0.4 * (((u - ex) / 2.5) ** 2 + ((v - ey) / 1.5) ** 2 < 1)
        by = ey - p["brow_up"] + sx * p["brow_tilt"] * (u - ex)
        t -= 0.3 * ((np.abs(u - ex) < 3.5) & (np.abs(v - by) < 0.8))
    t -= 0.35 * ((np.abs(u) < p["mouth_w"]) & (np.abs(v - p["mouth_y"]) < 0.8))
    return t

def landmarks_uv(p):
    """67 landmarks and 6 fiducials in the canonical frontal frame (u, v)."""
    pts = []
    phi = np.linspace(-0.2, np.pi + 0.2, 17)                        # jaw and cheek contour
    pts += list(zip(0.92 * A * np.cos(phi), 0.92 * B * np.sin(phi)))
    for sx in (-1, 1):
        ex, ey = sx * p["eye_dx"], p["eye_y"]
        for du in np.linspace(-3.5, 3.5, 5):                       # brow, 5 points
            pts.append((ex + du, ey - p["brow_up"] + sx * p["brow_tilt"] * du))
        for a in np.linspace(0, 2 * np.pi, 6, endpoint=False):     # eye, 6 points
            pts.append((ex + 2.5 * np.cos(a), ey + 1.5 * np.sin(a)))
    for s in np.linspace(0, 1, 4):                                  # nose bridge
        pts.append((0.0, p["eye_y"] + s * (p["nose_y"] - p["eye_y"])))
    for du in np.linspace(-4, 4, 5):                                # nose base
        pts.append((du, p["nose_y"] + 3))
    for a in np.linspace(0, 2 * np.pi, 12, endpoint=False):         # outer lip
        pts.append((p["mouth_w"] * np.cos(a), p["mouth_y"] + 2 * np.sin(a)))
    for du in np.linspace(-0.8, 0.8, 7):                            # inner lip
        pts.append((du * p["mouth_w"], p["mouth_y"]))
    lm = np.array(pts)
    fid = np.array([(-p["eye_dx"], p["eye_y"]), (p["eye_dx"], p["eye_y"]), (0.0, p["nose_y"]),
                    (-p["mouth_w"], p["mouth_y"]), (p["mouth_w"], p["mouth_y"]), (0.0, p["mouth_y"])])
    return lm, fid

def project(p, uv, pose):
    yaw, roll, s, t = pose
    u, v = uv[:, 0], uv[:, 1]
    z = depth(p, u, v)
    x = u * np.cos(yaw) + z * np.sin(yaw)
    d = -u * np.sin(yaw) + z * np.cos(yaw)
    cr, sr = np.cos(roll), np.sin(roll)
    col = s * (cr * x - sr * v) + C + t[0]
    row = s * (sr * x + cr * v) + C + t[1]
    return np.stack([col, row], 1), d

g = np.arange(-A, A + 1e-9, 0.25)
UU, VV = np.meshgrid(g, np.arange(-B, B + 1e-9, 0.25))
inside = (UU / A) ** 2 + (VV / B) ** 2 < 1
GRID = np.stack([UU[inside], VV[inside]], 1)

def render(p, pose):
    yaw = pose[0]
    xy, d = project(p, GRID, pose)
    u, v = GRID[:, 0], GRID[:, 1]
    eps = 0.25                                                     # surface normal, rotated by yaw
    zu = (depth(p, u + eps, v) - depth(p, u - eps, v)) / (2 * eps)
    zv = (depth(p, u, v + eps) - depth(p, u, v - eps)) / (2 * eps)
    n = np.stack([-zu, -zv, np.ones_like(zu)], 1)
    n /= np.linalg.norm(n, axis=1, keepdims=True)
    nx = n[:, 0] * np.cos(yaw) + n[:, 2] * np.sin(yaw)
    nz = -n[:, 0] * np.sin(yaw) + n[:, 2] * np.cos(yaw)
    L = np.array([rng.uniform(-0.3, 0.3), rng.uniform(-0.3, 0.3), 1.0])
    L /= np.linalg.norm(L)
    val = texture(p, u, v) * (0.6 + 0.4 * np.clip(nx * L[0] + n[:, 1] * L[1] + nz * L[2], 0, None))
    c, r = np.round(xy[:, 0]).astype(int), np.round(xy[:, 1]).astype(int)
    ok = (c >= 0) & (c < SIZE) & (r >= 0) & (r < SIZE)
    idx, d, val = r[ok] * SIZE + c[ok], d[ok], val[ok]
    order = np.lexsort((-d, idx))                                  # z-buffer: nearest sample wins
    first = np.unique(idx[order], return_index=True)[1]
    img = np.full(SIZE * SIZE, np.nan)
    img[idx[order][first]] = val[order][first]
    img = img.reshape(SIZE, SIZE)
    pad = np.pad(img, 1, constant_values=np.nan)                   # fill isolated holes
    nb = np.stack([pad[1 + dr:SIZE + 1 + dr, 1 + dc:SIZE + 1 + dc]
                   for dr in (-1, 0, 1) for dc in (-1, 0, 1) if dr or dc])
    hole = np.isnan(img) & (np.sum(~np.isnan(nb), 0) >= 5)
    img[hole] = np.nanmean(nb[:, hole], 0)
    img[np.isnan(img)] = 0.2                                       # background
    return img + rng.normal(0, 0.02, img.shape)

def sample_pose(yaw_deg):
    return (np.deg2rad(yaw_deg), np.deg2rad(rng.uniform(-8, 8)), rng.uniform(0.92, 1.08),
            rng.uniform(-3, 3, 2))

GEN = new_identity(generic=True)
GEN_LM, GEN_FID = landmarks_uv(GEN)
GEN_X3D = np.c_[GEN_LM, depth(GEN, GEN_LM[:, 0], GEN_LM[:, 1]), np.ones(len(GEN_LM))]
FRONT_LM, FRONT_FID = GEN_LM + C, GEN_FID + C                     # frontal targets (col, row)

def detect(p, pose):
    """Stand-in for the paper's SVR landmark detectors: true positions plus noise."""
    lm, fid = landmarks_uv(p)
    return (project(p, lm, pose)[0] + rng.normal(0, 0.7, (len(lm), 2)),
            project(p, fid, pose)[0] + rng.normal(0, 0.7, (len(fid), 2)))

def align(img, lm, fid, method):
    if method == "none":
        return img
    if method == "2D":                    # similarity transform from 6 fiducials to anchors
        T = SimilarityTransform()
        T.estimate(FRONT_FID, fid)        # maps output (frontal) -> input coordinates
        return warp(img, T, output_shape=img.shape, order=1, cval=0.2)
    P, *_ = np.linalg.lstsq(GEN_X3D, lm, rcond=None)   # affine 3D-to-2D camera, 8 unknowns
    resid = lm - GEN_X3D @ P
    target = FRONT_LM + (resid if method == "3D" else 0.0)
    T = PiecewiseAffineTransform()
    T.estimate(target, lm)                # piecewise affine over a Delaunay triangulation
    return warp(img, T, output_shape=img.shape, order=1, cval=0.2)

MASK = (((np.arange(SIZE)[None, :] - C) / A) ** 2 + ((np.arange(SIZE)[:, None] - C) / B) ** 2
        < 0.85 ** 2) & (np.arange(SIZE)[:, None] > C - 11)
METHODS = ["none", "2D", "3D no residual", "3D"]

def vec(img):
    x = img[MASK] - img[MASK].mean()
    return x / np.linalg.norm(x)

def sample(p, yaw_deg):
    pose = sample_pose(yaw_deg)
    img = render(p, pose)
    lm, fid = detect(p, pose)
    return {m: vec(align(img, lm, fid, m)) for m in METHODS}

N_TRAIN, N_TEST, YAWS = 120, 120, [0, 15, 30, 45]
train = [sample(new_identity(), rng.uniform(-45, 45)) for _ in range(N_TRAIN) for _ in range(4)]
test_ids = [new_identity() for _ in range(N_TEST)]
gallery = [sample(p, rng.uniform(-5, 5)) for p in test_ids]
probes = {y: [sample(p, y * rng.choice([-1, 1])) for p in test_ids] for y in YAWS}

def auc(G, Pr):
    S = G @ Pr.T                                   # cosine similarities, gallery x probe
    same = np.eye(len(G), dtype=bool)
    return roc_auc_score(same.ravel(), S.ravel())

for emb in ["pixels", "PCA-40"]:
    print(f"embedding: {emb}   ROC AUC by probe yaw (gallery near frontal)")
    print(f"{'alignment':>15s} " + " ".join(f"{y:>5d}°" for y in YAWS))
    for m in METHODS:
        G = np.array([s[m] for s in gallery])
        Ps = {y: np.array([s[m] for s in probes[y]]) for y in YAWS}
        if emb == "PCA-40":
            pca = PCA(n_components=40, svd_solver="full").fit(np.array([s[m] for s in train]))
            f = lambda X: (lambda Z: Z / np.linalg.norm(Z, axis=1, keepdims=True))(pca.transform(X))
            G, Ps = f(G), {y: f(Ps[y]) for y in YAWS}
        print(f"{m:>15s} " + " ".join(f"{auc(G, Ps[y]):6.3f}" for y in YAWS))
    print()

embedding: pixels   ROC AUC by probe yaw (gallery near frontal)
      alignment     0°    15°    30°    45°
           none  0.803  0.704  0.594  0.470
             2D  0.985  0.917  0.735  0.605
 3D no residual  0.981  0.917  0.820  0.765
             3D  0.998  0.975  0.911  0.853

embedding: PCA-40   ROC AUC by probe yaw (gallery near frontal)
      alignment     0°    15°    30°    45°
           none  0.793  0.698  0.581  0.464
             2D  0.981  0.906  0.714  0.592
 3D no residual  0.966  0.897  0.780  0.741
             3D  0.996  0.966  0.904  0.846

Without alignment, verification is already imperfect for frontal probes, because the jitter moves the features between images, and it falls to chance by 45°, where the pixels that a frontal gallery image and a turned probe share at the same position no longer belong to the same parts of the face. The 2D similarity transform removes the jitter, so at 0° it is almost as good as the best pipeline, but its advantage shrinks quickly with yaw: a similarity transform can scale and rotate the image plane, not turn the head, which is the paper’s argument for 3D alignment. Frontalization degrades much more slowly: by 45° the 2D pipeline has lost most of its margin above chance (an AUC of 0.5), while frontalization keeps most of its own. Even at 45°, part of the face is self-occluded and stretched by the warp, and the symmetry filling that the paper mentions was not used, so the 3D scores still fall.

The two 3D variants test the residual relaxation. Warping every face onto the generic landmark positions removes the identity information carried by where the eyes, brows and mouth are, and at 0° this costs more than the 2D transform, which keeps that geometry; with residuals added, the warp keeps it and is the best pipeline at every yaw. That is the effect the paper predicts, under conditions that favour it: the synthetic landmarks are unbiased and only slightly noisy, so the residual is mostly identity and little detection error. Hassner et al. (2015), discussed below, argued that real landmark detections are not guaranteed to fall at the same positions in different views of a face, in which case the residuals would add noise instead. The ranking of the pipelines is the same for both embeddings (with raw pixels, 2D and 3D without residuals tie at 15°); the PCA embedding, fitted on training identities, scores slightly lower than raw pixels in every row, which we did not expect. The script was run once as planned and not changed afterwards.

Performance

Labeled Faces in the Wild

The paper evaluates on the ten-fold View 2 pair-matching protocol of LFW: 6,000 pairs in ten splits, reported as mean accuracy with its standard error (it gives the dataset as 13,323 photographs of 5,749 people; the LFW documents give 13,233 images). It names three settings: restricted (only same/not-same labels of the training pairs), unrestricted (identities in the training splits may be used to form more pairs) and unsupervised (no training on LFW images). Table 3 of the paper:

Method	Accuracy ± SE	Protocol (paper’s label)
Joint Bayesian	0.9242 ± 0.0108	restricted
Tom-vs-Pete	0.9330 ± 0.0128	restricted
High-dim LBP	0.9517 ± 0.0113	restricted
TL Joint Bayesian	0.9633 ± 0.0108	restricted
DeepFace-single	0.9592 ± 0.0029	unsupervised
DeepFace-single	0.9700 ± 0.0028	restricted
DeepFace-ensemble	0.9715 ± 0.0027	restricted
DeepFace-ensemble	0.9735 ± 0.0025	unrestricted
Human, cropped	0.9753

DeepFace-single with the inner product of normalized features gives 95.92%; the paper does not state how the threshold for that accuracy was chosen. A kernel SVM on the $\chi^2$-distance vectors, trained on the 5,400 pair labels of each training split, gives 97.00%. By our arithmetic, the final 97.35% has 27.8% less error than the 96.33% of transfer-learned Joint Bayesian, which matches the abstract’s “more than 27%”. All DeepFace rows rely on a network trained on SFC, so the paper’s “restricted” and “unrestricted” labels refer only to how LFW’s own training splits are used. Under the reporting categories that the LFW curators introduced in 2014 they fall into “unrestricted, labeled outside data” (Huang & Learned-Miller, 2014), and the published face recognition and LFW articles list the ensemble there. The paper cites Kumar et al. (2009) for human performance; the LFW article describes that Mechanical Turk experiment and the caveats the LFW results page attaches to it.

Alignment ablations. On LFW the paper reports 94.3% with only the 2D alignment, 87.9% with no alignment (the centre of the detection crop, where parts of the face may fall outside) and 91.4% for frontalized images with a plain LBP and SVM combination instead of the network. Table 2 lists a network on 2D-aligned input (DeepFace-align2D) with an SFC error of 9.5% and LFW accuracy 0.9430 ± 0.0043, which matches the 94.3%; the paper does not say explicitly that they are the same network.

Ensembles. The restricted ensemble combines three networks trained on different inputs: DeepFace-single on 3D-aligned RGB, a network on grey levels plus gradient magnitude and orientation (SFC error 8.9%, LFW 0.9582 ± 0.0037 in Table 2), and the 2D-aligned network. Their distances are combined by a non-linear SVM ($C = 1$) with a sum of power conditionally positive definite kernels,

$$K_{\text{Combined}} = K_{\text{single}} + K_{\text{gradient}} + K_{\text{align2d}}, \qquad K(x, y) = -\lVert x - y \rVert_2,$$

giving 97.15%. For the unrestricted protocol the authors fine-tuned a Siamese network. Training it on pairs from LFW’s training splits overfitted badly, since the pairs come from roughly 9,000 photos and cannot constrain more than 120 million parameters. They therefore collected an additional dataset “following the same procedure as with the SFC”, with 100,000 new identities of 30 samples each, trained the Siamese network on it and then for two epochs on the LFW unrestricted training splits to correct dataset-dependent biases. Alone it reaches 0.9617 ± 0.0038 (Table 2); added to the ensemble it gives 97.25%, and adding four more DeepFace-single networks trained from different random seeds gives 97.35%.

YouTube Faces

YouTube Faces (Wolf, Hassner & Maoz, 2011) has 3,425 videos of 1,595 people, a subset of the LFW celebrities, with 5,000 video pairs in ten splits for video-level verification. DeepFace used the single network unchanged. For each training video pair it sampled 50 frame pairs, one frame from each video, labelled them with the pair’s label and learned a weighted $\chi^2$ model; for a test pair it averaged the learned similarity over 100 random frame pairs. Table 4 of the paper reports DeepFace-single at 91.4 ± 1.1% accuracy, an AUC of 96.3 and an equal error rate of 8.6, against 79.7 ± 1.8% for the best previous method listed (VSOF+OSS), which by our arithmetic is 58% less error and matches the paper’s “more than 50%”. The authors add that about 100 video pairs were mislabelled, as the YouTube Faces page had recently reported, and that after correcting them DeepFace-single reaches 92.5%.

Strengths

• Alignment that addresses pose. The 3D step corrects out-of-plane rotation that a similarity transform cannot; the paper reports 94.3% on LFW with 2D alignment only, against 95.92% (unsupervised) and 97.00% (restricted) for the frontalized single network, without stating the protocol of the 2D figure.
• A transferable representation. A single network trained on SFC, with a parameter-free inner product, reached 95.92% on LFW, close to the best previous result, which used supervised transfer learning; on YouTube Faces the same features worked without retraining the network.
• Compactness and speed. One 4,096-dimensional, sparse vector replaced the tens of thousands of descriptors of competing systems, at 0.33 seconds per image on one CPU core.
• Evidence for scale. Table 1 is an early controlled measurement in face recognition of how error depends on training-set size and depth.

Limitations

• Private training data. SFC is internal to the authors’ organization, and the paper describes no release of it or of the trained networks, so its results cannot be reproduced. The follow-up paper reports that training the same initial network on 500,000 images of CASIA-WebFace, the largest public set at the time, gave only 93.5% on LFW, largely from overfitting (Taigman et al., 2015).
• Unreported composition. The paper reports no demographic composition of SFC and no results broken down by group; for LFW it gives only that about 75% of the people are male. Measured demographic differentials of face recognition systems are described in face recognition.
• A complex, landmark-dependent front end. Accuracy depends on two landmark regressors, a 3D template and a warp. FaceNet’s authors list minimal alignment (a tight crop) as a strength of their system against DeepFace’s “complex 3D alignment” (Schroff, Kalenichenko & Philbin, 2015), and Hassner et al. (2015) argued that per-face 3D adjustments can misalign features when landmarks are not detected consistently.
• Parameters tied to the alignment. Locally connected layers assume that each region of the face is at a fixed position, so they depend on accurate alignment and multiply the number of parameters (about 63 million in L4 alone, by our arithmetic).
• A classification objective. The network is trained to separate the 4,030 training identities, not to make distances between new faces meaningful; the paper adds a separate metric (the $\chi^2$ SVM or Siamese network) for its best results, and ensembles for the last few tenths of a point.
• LFW near saturation. By our reading, the differences between the three supervised DeepFace rows of Table 3 (97.00% to 97.35%) are a few tenths of a point, about the size of one standard error, and the LFW results page notes that the human experiment did not control whether workers already knew the celebrities (benchmark overfitting).

Historical Importance

DeepFace was one of the first face verification systems built on a large deep network trained from raw pixels, and with DeepID (Sun, Wang & Tang, 2014), presented at the same conference, it established that a representation learned by classifying thousands of identities transfers to people outside the training set. Its LFW accuracy brought machine verification to within 0.18 points of the human figure on cropped faces, and its paper made the argument for 3D frontalization and for locally connected layers on aligned faces.

The sequence that followed moved away from both of its distinctive components. The same authors’ follow-up kept the architecture but studied the F7 bottleneck, drew its training sets from a pool of 10 million identities and replaced random subsampling by “semantic bootstrapping”; a single network reached 98.00% on LFW verification and 82.1% closed-set rank-1 accuracy on an LFW identification benchmark (Taigman et al., 2015). DeepID2+, at 99.47% as FaceNet cites it, and then FaceNet passed 99% on LFW; FaceNet, described in the FaceNet paper article, trained a 128-dimensional embedding end-to-end with a triplet loss on tight crops without 2D or 3D alignment, and reported 99.63% on LFW with its own detector’s crop (“extra face alignment”), which its authors describe as reducing DeepFace’s error by more than a factor of seven, and 95.12% on YouTube Faces (Schroff, Kalenichenko & Philbin, 2015). Later systems returned to classification over identities, as DeepFace had trained, but with normalized features and an angular or cosine margin in the softmax: the margin-based softmax losses of SphereFace, CosFace and ArcFace. The FaceNet model article covers the triplet-trained embedding as it is used today.

Variants

• DeepFace ensembles and Siamese fine-tuning, as described under Performance: networks on 3D-aligned RGB, on grey level and gradients, and on 2D-aligned input, a Siamese network, and seeds.
• Web-scale training (Taigman et al., 2015): the same layer sequence with a compressed F7 bottleneck (trainable from scratch down to 1,024 dimensions and, by pre-loading the earlier network’s weights, to much smaller sizes, of which 256 dimensions gave the best accuracy on the target domain), expanded locally connected layers in its final version, training on up to 7.5 million faces chosen from a set of 10 million identities that the paper says was randomly sampled from a social network, and evaluation on closed- and open-set LFW identification protocols as well as verification.
• Frontalization without per-face shape. Hassner, Harel, Paz and Enbar (2015) frontalized faces with a single unmodified 3D reference surface, a face geometry from the USF Human-ID database whose average scan DeepFace used, with “soft symmetry” to fill self-occluded regions, arguing that the per-face 3D surfaces of earlier methods, DeepFace’s among them, are rough approximations that preserve little identifying shape.

Implementations

The paper describes the authors’ CPU implementation but does not mention a release of code or trained models. The most visible open implementation that uses the name, the Python library deepface by Sefik Serengil, is a framework that wraps several face recognition models, one of them a DeepFace-style network. As of its source at commit 65697d2 (31 October 2026), that network has TensorFlow, PyTorch and ONNX backends. The TensorFlow backend follows the layer sizes of Figure 2 with Keras locally connected layers (L5 with stride 2) and an 8,631-way output, and loads weights from a third-party repository whose file name indicates training on VGGFace2 with 2D alignment; the other two load weight files of the same name in ONNX and PyTorch formats. All three warn that it scores 69.0% on LFW in its best configuration (Serengil, deepface repository). It is a reimplementation, not the model of the paper.