Metric Learning

Metric learning learns a distance function from examples of what should count as similar. Instead of choosing a fixed measure such as the Euclidean distance between pixel vectors, it fits the parameters of a distance $d_\theta(x, y)$, or of an embedding $f_\theta$ compared by a fixed measure, so that images of the same object, person or category come out close and other images far apart. The classical form, often called distance metric learning, learns a Mahalanobis distance on fixed feature vectors; deep metric learning trains a network end to end, usually as a siamese network that applies the same weights to every image of a pair or triplet. The measures themselves are covered under similarity and distance measures; this article covers learning them.

Purpose

A fixed measure weighs every difference between two feature vectors by a rule chosen in advance, and the differences that dominate are often the ones a task should ignore. In a pixel vector, a change of illumination across the whole image can move a point farther than a change of object; in a descriptor, the most variable coordinates need not be the most informative. Metric learning takes the notion of similarity from supervision instead: pairs labeled similar or dissimilar, triplets stating that one item is closer to an anchor than another, or class labels from which such pairs follow.

The second purpose is recognition of classes never seen in training. Chopra, Hadsell and LeCun (2005) framed face verification this way: the number of identities is large, each has few images, and only some are known when the system is trained, so a classifier with one output per class does not fit. A learned similarity can compare a new face with a stored one without retraining. Image retrieval, person re-identification and few-shot recognition have the same structure, and the standard benchmarks test on classes disjoint from the training classes.

When to Use It

Metric learning fits when the decision is a comparison: ranking a gallery by similarity to a query, thresholding the distance between two images (verification), clustering, or classifying with nearest neighbors in a space where new classes can be added by storing examples. Musgrave, Belongie and Lim (2020) give the same reason for using embeddings rather than a classifier’s logits at test time, even when a classification loss trains them: retrieval, and open-set classification, in which the test classes are disjoint from the training classes.

It is not needed when the classes are fixed and well represented, where an ordinary image classifier is simpler. When no labels exist but positive pairs can be made by augmentation, the same losses become contrastive learning, a branch of self-supervised learning. And a learned metric only knows the notion of similarity its supervision encodes: a metric trained to match products ignores what a metric trained to match identities would need.

How It Works

Learning a Mahalanobis distance

The classical methods learn a matrix $M \succeq 0$ that defines

$$d_M(x, y) = \sqrt{(x - y)^\top M (x - y)} = \lVert L x - L y \rVert_2, \qquad M = L^\top L,$$

so learning $M$ is the same as learning a linear map $L$ followed by the Euclidean distance; with $M = I$ the Euclidean distance is recovered, and a diagonal $M$ only reweights coordinates (the measure itself and its metric properties are in similarity and distance measures). A rectangular $L$ with $k < d$ rows gives a low-rank metric and a $k$-dimensional embedding. Four methods set most of the vocabulary that deep metric learning later reused.

Pairwise constraints (Xing et al., 2002). Given a set $\mathcal{S}$ of pairs known to be similar and a set $\mathcal{D}$ of dissimilar ones, Xing, Ng, Jordan and Russell minimized the squared distances of the similar pairs while keeping the dissimilar ones apart:

$$\min_{M} \sum_{(x_i, x_j) \in \mathcal{S}} \lVert x_i - x_j \rVert_M^2 \quad \text{s.t.} \quad \sum_{(x_i, x_j) \in \mathcal{D}} \lVert x_i - x_j \rVert_M \ge 1, \quad M \succeq 0 .$$

Without the constraint, $M = 0$ would win by collapsing every point onto one, the same failure that the deep losses below guard against. The problem is convex, so it has no local optima. The constraint uses unsquared distances on purpose: with squared ones, objective and constraint are both linear in $M$, the problem reduces to a generalized eigenvalue problem, and the authors showed that the optimal $M$ then always has rank one and projects the data onto a line (their footnote 3). They used the learned metric to improve clustering.

Neighbourhood Components Analysis (Goldberger et al., 2004). NCA targets $k$-nearest-neighbor classification directly. Each point $x_i$ picks a neighbor $x_j$ at random with a softmax over distances in the transformed space,

$$p_{ij} = \frac{\exp(-\lVert A x_i - A x_j \rVert^2)}{\sum_{k \neq i} \exp(-\lVert A x_i - A x_k \rVert^2)}, \qquad p_{ii} = 0,$$

and NCA maximizes $\sum_i \sum_{j \in C_i} p_{ij}$, the expected number of points whose randomly chosen neighbor has their class ($C_i$). This smooths the leave-one-out nearest-neighbor accuracy into a differentiable objective. It is not convex. A non-square $A$ learns a low-dimensional projection, which the authors used for visualization and to cut the storage and search cost of nearest-neighbor classification. The softmax over distances to the other points is the template of later deep losses, including Proxy-NCA below.

Large margin nearest neighbor (Weinberger et al., 2005; Weinberger & Saul, 2009). LMNN fixes for each input $k$ target neighbors, same-class points chosen by Euclidean distance before learning, and penalizes differently labeled points, impostors, that come within a unit margin of the target-neighbor radius. In the journal version’s notation, with $j \rightsquigarrow i$ meaning that $j$ is a target neighbor of $i$, $y_{il} = 1$ when $i$ and $l$ share a label, and $[z]_+ = \max(z, 0)$,

$$\varepsilon(L) = (1 - \mu) \sum_{j \rightsquigarrow i} \lVert L(x_i - x_j) \rVert^2 + \mu \sum_{j \rightsquigarrow i} \sum_{l} (1 - y_{il}) \big[ 1 + \lVert L(x_i - x_j) \rVert^2 - \lVert L(x_i - x_l) \rVert^2 \big]_+ .$$

The first term pulls target neighbors in; the second, a hinge on triples, pushes impostors out. Unlike Xing et al.’s objective it does not ask all same-class points to cluster, only each point’s own neighbors, which is all that nearest-neighbor classification needs. Written in $M = L^\top L$ it becomes a semidefinite program, convex in $M$; the authors solved it with a special-purpose subgradient solver that exploits the fact that few constraints are active. The weight $\mu$ “did not depend sensitively” on its value, and $\mu = 0.5$ worked well (Weinberger & Saul, 2009). The NIPS version reported a 1.3% test error on MNIST digits (deskewed, reduced to 164 principal components) with an energy-based variant of the classifier, cutting the Euclidean $k$-NN error by more than a third, and lower $k$-NN error on face images of 40 people (AT&T faces) than with Euclidean distances.

Information-theoretic metric learning (Davis et al., 2007). ITML keeps $M$ as close as possible to a prior $M_0$ (the identity, or an inverse covariance) while satisfying $d_M(x_i, x_j) \le u$ for similar pairs and $\ge \ell$ for dissimilar ones, where $d_M$ is here the squared distance. Closeness is the relative entropy between the Gaussians that the two metrics define, which reduces to the LogDet divergence

$$D_{\ell d}(M, M_0) = \operatorname{tr}(M M_0^{-1}) - \log\det(M M_0^{-1}) - d .$$

The authors solved it by Bregman projections, at a cost of $O(c\,d^2)$ per iteration for $c$ constraints, without eigenvalue decompositions or semidefinite programming, and the formulation can be kernelized. The divergence is invariant to any invertible linear change of coordinates applied to both matrices.

These methods learn a linear map on features someone else designed. They cannot add invariance that the features lack: a Mahalanobis metric on pixel vectors cannot make two shifted copies of an object close, because a shift is not a linear projection of the pixels. Learning the features as well is what the deep methods add.

Siamese networks and pair losses

Bromley et al. (1993) verified signatures with a “siamese” network: two identical subnetworks with shared weights, one per input, joined at their outputs, so that the comparison is symmetric and the same map embeds every input. Chopra, Hadsell and LeCun (2005) used the architecture to learn a similarity metric for faces with convolutional subnetworks. Their energy was the $\ell_1$ distance $E_W = \lVert G_W(X_1) - G_W(X_2) \rVert_1$ between the two outputs, and the loss for a pair with label $Y$ (0 for the same person, 1 for different people) was

$$L = (1 - Y)\,\frac{2}{Q}\,E_W^2 + Y\, 2Q\, e^{-\frac{2.77}{Q} E_W},$$

with $Q$ the upper bound of $E_W$. They argued that a loss which only lowered the energy of matching pairs could be minimized by a constant map, so a contrastive term that raises the energy of non-matching pairs is required (their Section 2.3 is titled “Contrastive Loss Function used for Training”), and that a squared-norm energy would be inappropriate because its gradient vanishes as the energy approaches zero. They trained on images from the AR and FERET face sets and tested verification on 40 AR subjects absent from training: at false accept rates of 10%, 7.5% and 5%, the false reject rates were 11%, 14.6% and 19% (their Table 1).

Hadsell, Chopra and LeCun (2006) gave the loss its now standard form, with the Euclidean distance $D_W = \lVert G_W(X_1) - G_W(X_2) \rVert_2$ and a margin $m > 0$:

$$L(W, Y, X_1, X_2) = (1 - Y)\,\tfrac{1}{2} D_W^2 + Y\, \tfrac{1}{2} \big\{ \max(0, m - D_W) \big\}^2 .$$

Dissimilar pairs contribute only while they are closer than $m$. The similar pairs could come from prior knowledge instead of distances in input space: MNIST images of 4s and 9s paired with horizontally shifted copies of themselves and of their neighbors, or images of a NORB airplane paired when taken from adjacent camera positions under any lighting, so that the learned mapping became invariant to the shift or to the lighting. The loss function and contrastive learning articles use the same formula.

A fixed margin treats every dissimilar pair alike. The triplet loss of FaceNet (Schroff, Kalenichenko & Philbin, 2015) instead compares an anchor’s distance to a positive with its distance to a negative, asking only for a relative margin; sampling its triplets well is the main difficulty of training it (see Sampling and mining). Later losses compare many pairs at once: the lifted structured loss of Song et al. (2016) lifts a batch to its full matrix of pairwise distances and, for each positive pair, applies a smooth maximum over all negatives of both its members; the multi-class N-pair loss identifies the positive among the batch’s other examples with a softmax, as described in contrastive learning.

Multi-similarity loss. Wang et al. (2019) analyzed pair losses through the weights their gradients give each pair, and identified three similarities that can set those weights: a pair’s own cosine similarity $S_{ij}$, its similarity relative to the other negative pairs of the same anchor, and relative to the positive pairs. Their loss first keeps a negative pair only if $S_{ij} > \min_{y_k = y_i} S_{ik} - \epsilon$ (it is more similar than the anchor’s hardest positive, up to a margin $\epsilon$) and a positive pair only if $S_{ij} < \max_{y_k \neq y_i} S_{ik} + \epsilon$, then weights the selected sets $\mathcal{P}_i$ and $\mathcal{N}_i$ with

$$\mathcal{L}_{MS} = \frac{1}{m} \sum_{i=1}^{m} \left\{ \frac{1}{\alpha} \log\Big[1 + \sum_{k \in \mathcal{P}_i} e^{-\alpha (S_{ik} - \lambda)}\Big] + \frac{1}{\beta} \log\Big[1 + \sum_{k \in \mathcal{N}_i} e^{\beta (S_{ik} - \lambda)}\Big] \right\},$$

where $\alpha$, $\beta$ and $\lambda$ are hyperparameters. The log-sum-exp terms weight the hardest pairs most, softly.

Proxy and classification-based losses

Pair and triplet losses see only the relations inside a batch, a tiny fraction of the $O(n^2)$ pairs or $O(n^3)$ triplets in a training set of $n$ images. Movshovitz-Attias et al. (2017) replaced the other data points by learned proxies, one per class in the simplest case, trained with the network. With an embedding $x$, its class proxy $p(y)$ and the proxies of the other classes $p(Z)$, Proxy-NCA is NCA’s softmax with proxies in place of neighbors,

$$\ell = -\log \frac{\exp\big(-d(x, p(y))\big)}{\sum_{p(z) \in p(Z)} \exp\big(-d(x, p(z))\big)} ,$$

with $d$ the squared Euclidean distance. Each anchor is compared with every class at every step, so no triplets are sampled. The authors showed that, for embeddings and proxies of fixed norm, the proxy losses bound the corresponding losses over the data, with a slack that grows with the largest distance from a point to its proxy.

Weight vectors of a classifier’s last layer play the same role. Zhai and Wu (2019) trained retrieval embeddings with a normalized softmax loss: remove the bias of the last linear layer, $\ell_2$-normalize both the embedding $x$ and the class weights $p_z$, and apply cross-entropy to cosine similarities divided by a temperature $\sigma$,

$$L_{\text{norm}} = -\log \frac{\exp(x^\top p_y / \sigma)}{\sum_{z \in Z} \exp(x^\top p_z / \sigma)} ,$$

with $\sigma = 0.05$ in their experiments. They view the class weights as proxies under the cosine distance. To scale to many classes they computed the softmax over a random subset of classes in each step: on Stanford Online Products, using 10% of the training classes per iteration cost less than 1% Recall@1, and 1% of the classes still reached 75.7 R@1. In face recognition the same construction appeared two years earlier, in NormFace (2017), and margin-based softmax losses that add an angular or cosine margin to the target logit (SphereFace in 2017, then CosFace and ArcFace) are covered in ArcFace.

Sampling and mining

Which pairs or triplets a loss sees matters as much as the loss. Most randomly drawn pairs are already far apart and contribute zero loss, especially late in training. Wu et al. (2017) made this precise for $\ell_2$-normalized embeddings in $n \ge 128$ dimensions: if points were spread uniformly over the unit sphere, pairwise distances would follow

$$q(d) \propto d^{\,n-2} \Big[1 - \tfrac{1}{4} d^2\Big]^{\frac{n-3}{2}},$$

which approaches a normal distribution around $\sqrt{2}$ with variance $1/(2n)$, so uniformly sampled negatives sit near $\sqrt{2}$ and, with a margin below that, carry no signal. The opposite strategy, always choosing the hardest negative, gives gradients whose direction is dominated by noise when the negative is very close, and with the triplet loss it can collapse the model to a single point. Mining strategies sit between these extremes:

• Semi-hard negatives, FaceNet’s mining rule for the triplet loss.
• Distance-weighted sampling (Wu et al., 2017): draw a negative with probability proportional to $\min(\lambda, q(d)^{-1})$, which spreads samples over all distances while clipping the noisiest close ones. They paired it with a margin loss, $(\alpha + y_{ij}(D_{ij} - \beta))_+$ with $y_{ij} = 1$ for a positive pair and $-1$ for a negative one, where $\beta$ is a learned boundary between positive and negative distances.
• Relative mining inside the loss, as in the multi-similarity loss above.
• Class-balanced batches: sample $C$ classes and $S$ images of each, so that every batch contains positives. In Zhai and Wu’s ablation on CUB-200-2011 with batches of 75, $S = 25$ images of each of 3 classes gave the best Recall@1 (61.3), no class balancing gave 59.5, and all 75 images from one class gave 40.9 (their Table 4).

Proxy and classification losses avoid most of this, since every class is present at every step.

How to Apply It

Protocol

The standard deep metric learning benchmarks split the classes, not the images. Song et al. (2016) trained on the first half of the classes of CUB-200-2011 (11,788 images of 200 bird species; Wah et al., 2011), Cars196 (Krause et al., 2013; 16,185 images of 196 car models as Movshovitz-Attias et al. describe it) and their own Stanford Online Products, and tested on the rest (the Recall@K article gives the Stanford Online Products split). Each test image then queries all other test images, and Recall@K, in this literature the fraction of queries with a same-class image among the first $K$, is reported. That article explains why Musgrave et al. (2020) call it Precision@1 at $K = 1$ and why they proposed MAP@R, a form of average precision over the first $R$ results, where $R$ is the number of images of the query’s class.

Model selection needs a validation set whose classes are disjoint from both training and test classes. Musgrave et al. split the first half of the classes into four partitions and ran four-fold cross-validation with class-disjoint folds, which most earlier papers had not done (see Pitfalls).

Example: a learned Mahalanobis distance on unseen classes

The NumPy script below renders $32 \times 32$ grayscale images of 36 classes, each a pair (shape, stripe angle): six shapes filled with stripes at one of six angles, centered within one pixel. Every image gets a random background level, a random linear illumination ramp, a random object contrast and Gaussian noise; intensities are floating-point values, nominally in $[0, 1]$ but not clipped, so that the lighting stays a linear nuisance. Eighteen classes, chosen in a checkerboard over the (shape, angle) grid, are used to learn the metric; the other 18 are new combinations and are used only for testing. The pixels are reduced to 48 dimensions by PCA fitted on the training classes, and four Mahalanobis distances are compared by leave-one-out Recall@1 among 540 images of each class set:

• $M = I$, the Euclidean distance;
• $M = \Sigma^{-1}$, whitening by the covariance of the training images (no labels);
• $M = S_w^{-1}$, whitening by the pooled within-class covariance (labels used only to estimate how images of one class vary), the transformation that Goldberger et al. (2004) used as their RCA baseline;
• NCA, full rank, with the gradient of Goldberger et al.’s Eq. 5, optimized by L-BFGS from a small random start.

It uses NumPy 2.3 and SciPy 1.16, repeats the comparison for ten seeds and takes about 13 seconds on a multi-core CPU; two runs printed identical output.

import numpy as np
from scipy.optimize import minimize

H = 32
yy, xx = np.mgrid[:H, :H] - 15.5
N_SHAPE, N_ANGLE = 6, 6                    # 36 classes: (shape, stripe angle)

def render(shape, angle, rng):
    """A centred striped shape (+-1 px jitter) under random lighting, float, unclipped."""
    r = rng.uniform(9, 12)
    y, x = yy - rng.uniform(-1, 1), xx - rng.uniform(-1, 1)
    mask = [x**2 + y**2 < r**2,                                          # disk
            (abs(x) < 0.8 * r) & (abs(y) < 0.8 * r),                     # square
            (y < 0.7 * r) & (y > -r + 2 * abs(x)),                       # triangle
            ((abs(x) < 0.35 * r) | (abs(y) < 0.35 * r)) & (abs(x) < r) & (abs(y) < r),  # cross
            (x**2 + y**2 < r**2) & (x**2 + y**2 > (0.6 * r) ** 2),       # ring
            (abs(x) < r) & (abs(y) < 0.45 * r)][shape]                   # bar
    t = np.deg2rad(30 * angle)
    stripes = 0.5 + 0.5 * np.sin(2 * np.pi * (x * np.cos(t) + y * np.sin(t)) / 5)
    gx, gy = rng.uniform(-0.025, 0.025, 2)                    # illumination ramp per pixel
    return (rng.uniform(0.2, 0.6) + gx * xx + gy * yy          # background level + ramp
            + rng.uniform(0.3, 0.6) * mask * stripes           # object contrast
            + 0.15 * rng.standard_normal((H, H)))              # sensor noise

def make_set(classes, n_per, rng):
    X = np.array([render(c // N_ANGLE, c % N_ANGLE, rng).ravel()
                  for c in classes for _ in range(n_per)])
    return X, np.repeat(classes, n_per)

def recall_at_1(Z, y):
    """Leave-one-out: does each image's nearest other image share its class?"""
    d = ((Z[:, None] - Z[None]) ** 2).sum(-1)
    np.fill_diagonal(d, np.inf)
    return (y[d.argmin(1)] == y).mean()

def nca(X, y, d_out, rng, iters=200):
    """Neighbourhood Components Analysis (Goldberger et al., 2004): find A maximizing
    sum_i p_i, the expected number of points whose soft nearest neighbour shares their label."""
    same = y[:, None] == y[None]
    def neg_f(a):
        A = a.reshape(d_out, -1)
        Z = X @ A.T
        D = ((Z[:, None] - Z[None]) ** 2).sum(-1)
        np.fill_diagonal(D, np.inf)                            # p_ii = 0
        P = np.exp(-(D - D.min(1, keepdims=True)))
        P /= P.sum(1, keepdims=True)                           # p_ij, their Eq. 1
        p = (P * same).sum(1)                                  # p_i,  their Eq. 2
        W = p[:, None] * P - P * same                          # weights in their Eq. 5
        L = np.diag(W.sum(0) + W.sum(1)) - W - W.T             # sum_ij W_ij x_ij x_ij^T = X^T L X
        return -p.sum(), -(2 * A @ (X.T @ L @ X)).ravel()
    a0 = 0.1 * rng.standard_normal(d_out * X.shape[1])
    return minimize(neg_f, a0, jac=True, method="L-BFGS-B", options={"maxiter": iters}).x.reshape(d_out, -1)

def run(seed, verbose=False):
    rng = np.random.default_rng(seed)
    c = np.arange(N_SHAPE * N_ANGLE)
    seen = c[(c // N_ANGLE + c % N_ANGLE) % 2 == 0]       # 18 classes for learning the metric
    unseen = c[(c // N_ANGLE + c % N_ANGLE) % 2 == 1]     # 18 new (shape, angle) pairs for testing
    Xtr, ytr = make_set(seen, 30, rng)
    Xte, yte = make_set(unseen, 30, rng)
    mu = Xtr.mean(0)                                        # PCA to 48-D, fitted on seen classes
    U = np.linalg.svd(Xtr - mu, full_matrices=False)[2][:48]
    Ztr, Zte = (Xtr - mu) @ U.T, (Xte - mu) @ U.T
    Sw = sum(np.cov(Ztr[ytr == k].T) for k in seen) / len(seen)   # pooled within-class covariance
    metrics = {"Euclidean (M = I)": np.eye(48),
               "whitened (M = Sigma^-1)": np.diag(1 / Ztr.std(0)),
               "within-class (M = Sw^-1)": np.linalg.inv(np.linalg.cholesky(Sw)),
               "NCA (M = A^T A)": nca(Ztr, ytr, 48, rng)}
    out = {}
    for name, L in metrics.items():                        # d_M(x, y) = ||L x - L y||, M = L^T L
        out[name] = recall_at_1(Ztr @ L.T, ytr), recall_at_1(Zte @ L.T, yte)
        if verbose:
            print(f"{name:26s} seen {out[name][0]:.3f}   unseen {out[name][1]:.3f}")
    return out

print("seed 0, R@1 (leave-one-out, 540 images per side)")
res = [run(0, verbose=True)] + [run(s) for s in range(1, 10)]
print("\nunseen-class R@1 over seeds 0-9: min / mean / max")
for name in res[0]:
    v = np.array([r[name][1] for r in res])
    print(f"{name:26s} {v.min():.3f} / {v.mean():.3f} / {v.max():.3f}")

seed 0, R@1 (leave-one-out, 540 images per side)
Euclidean (M = I)          seen 0.493   unseen 0.396
whitened (M = Sigma^-1)    seen 0.920   unseen 0.922
within-class (M = Sw^-1)   seen 0.987   unseen 0.981
NCA (M = A^T A)            seen 1.000   unseen 0.970

unseen-class R@1 over seeds 0-9: min / mean / max
Euclidean (M = I)          0.361 / 0.387 / 0.424
whitened (M = Sigma^-1)    0.896 / 0.914 / 0.933
within-class (M = Sw^-1)   0.980 / 0.987 / 0.994
NCA (M = A^T A)            0.911 / 0.958 / 0.974

Euclidean distance in pixel space finds a same-class neighbor for only about four test images in ten. The background level and the illumination ramp change every pixel and are the same kind of change for every class, so, in our reading, nearest neighbors tend to share a lighting pattern rather than a shape. Every learned metric helps on the unseen classes, and by our reading for one reason: what transfers to new classes is a description of the nuisance, which is shared, rather than of the training classes themselves. Even unsupervised whitening, which shrinks the high-variance lighting directions, more than doubles Recall@1; the within-class covariance describes the nuisance directly, and the simple $S_w^{-1}$ scored highest in every seed. NCA separated the training classes perfectly in seed 0 but did less well on the new ones, and it varied more across seeds than either whitening; we read this as overfitting: with 540 training images and $48 \times 48$ parameters, it partly fits what distinguishes the training classes rather than the variation they share. A learned metric must be judged on classes it has not seen.

Example: a deep embedding with a pair loss and a proxy loss

A linear metric cannot remove shifts. The PyTorch script below uses the same 36 classes but moves each object by up to five pixels and randomizes the stripe phase and scale, with the same kind of lighting and slightly less noise; each image is then standardized to zero mean and unit variance. It trains a small convolutional network $f_\theta$ ($32$-dimensional, $\ell_2$-normalized output) on the 18 training classes with either the contrastive loss of Hadsell et al. (all pairs in the batch, margin 1) or a normalized softmax loss with one proxy per training class (temperature 0.1). Batches are class-balanced, 8 classes by 8 images. It reports Recall@1 and MAP@R on 540 images of the 18 unseen classes for raw pixels, the untrained network, and the two trained networks, for three seeds. It uses PyTorch 2.14 on one CPU thread and takes about 30 seconds; two runs printed identical output.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)
H = 32
yy, xx = torch.meshgrid(torch.arange(H) - 15.5, torch.arange(H) - 15.5, indexing="ij")

def make_images(classes, n_per, gen):
    """Striped shapes as before (class = shape * 6 + stripe angle), now with nuisances no
    linear metric can undo: +-5 px shifts, random stripe phase, scale and lighting."""
    labels = classes.repeat_interleave(n_per)
    n = len(labels)
    U = lambda lo, hi, *s: lo + (hi - lo) * torch.rand(n, *s, generator=gen)
    r = U(7, 11)[:, None, None]
    c = U(-5, 5, 2)[:, :, None, None]
    y, x = yy - c[:, 0], xx - c[:, 1]
    shapes = torch.stack([
        x**2 + y**2 < r**2,
        (x.abs() < 0.8 * r) & (y.abs() < 0.8 * r),
        (y < 0.7 * r) & (y > -r + 2 * x.abs()),
        ((x.abs() < 0.35 * r) | (y.abs() < 0.35 * r)) & (x.abs() < r) & (y.abs() < r),
        (x**2 + y**2 < r**2) & (x**2 + y**2 > (0.6 * r) ** 2),
        (x.abs() < r) & (y.abs() < 0.45 * r)], 1)
    mask = shapes[torch.arange(n), labels // 6].float()
    t = torch.deg2rad(30.0 * (labels % 6))[:, None, None]
    stripes = 0.5 + 0.5 * torch.sin(2 * torch.pi * (x * t.cos() + y * t.sin()) / 5
                                    + U(0, 2 * torch.pi)[:, None, None])
    g = U(-0.025, 0.025, 2)[:, :, None, None]
    img = (U(0.2, 0.6)[:, None, None] + g[:, 0] * xx + g[:, 1] * yy
           + U(0.3, 0.6)[:, None, None] * mask * stripes
           + 0.1 * torch.randn(n, H, H, generator=gen))
    img = (img - img.mean((1, 2), keepdim=True)) / img.std((1, 2), keepdim=True)  # per-image standardization
    return img[:, None], labels

def embedder(d=32):
    return nn.Sequential(
        nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
        nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(64, d))

def contrastive_loss(z, y, m=1.0):
    """Hadsell, Chopra & LeCun (2006), Eq. 4, over all pairs in the batch (Y = 0: same class)."""
    D = torch.cdist(z, z)
    i, j = torch.triu_indices(len(z), len(z), 1)
    d, same = D[i, j], (y[i] == y[j]).float()
    return (same * 0.5 * d**2 + (1 - same) * 0.5 * F.relu(m - d) ** 2).mean()

def recall1_mapr(z, y):
    """R@1 and MAP@R (Musgrave et al., 2020): each test image queries all others."""
    S = z @ z.T
    S.fill_diagonal_(-float("inf"))
    order = S.argsort(1, descending=True)
    rel = (y[order] == y[:, None]).float()
    R = int((y == y[0]).sum()) - 1                       # same number per class here
    prec = rel[:, :R].cumsum(1) / torch.arange(1, R + 1)
    return rel[:, 0].mean().item(), ((prec * rel[:, :R]).sum(1) / R).mean().item()

def run(seed, loss_name, steps=300):
    torch.manual_seed(seed)
    gen = torch.Generator().manual_seed(seed)
    cls = torch.arange(36)
    seen, unseen = cls[(cls // 6 + cls % 6) % 2 == 0], cls[(cls // 6 + cls % 6) % 2 == 1]
    xtr, ytr = make_images(seen, 60, gen)
    xte, yte = make_images(unseen, 30, gen)
    if loss_name == "pixels":
        return recall1_mapr(F.normalize(xte.flatten(1), dim=1), yte)
    net = embedder()
    proxies = nn.Parameter(torch.randn(18, 32))        # one learned proxy per training class
    params = list(net.parameters()) + ([proxies] if loss_name == "norm-softmax" else [])
    opt = torch.optim.Adam(params, lr=3e-3)
    if loss_name in ("contrastive", "norm-softmax"):
        for step in range(steps):
            # class-balanced batch: 8 seen classes x 8 images
            pick = seen[torch.randperm(18, generator=gen)[:8]]
            idx = torch.cat([torch.nonzero(ytr == k).flatten()[torch.randperm(60, generator=gen)[:8]]
                             for k in pick])
            z = F.normalize(net(xtr[idx]), dim=1)
            if loss_name == "contrastive":
                loss = contrastive_loss(z, ytr[idx])
            else:                                       # normalized softmax, temperature 0.1
                logits = z @ F.normalize(proxies, dim=1).T / 0.1
                loss = F.cross_entropy(logits, torch.searchsorted(seen, ytr[idx]))
            opt.zero_grad()
            loss.backward()
            opt.step()
    with torch.no_grad():
        return recall1_mapr(F.normalize(net(xte), dim=1), yte)

if __name__ == "__main__":
    print("PyTorch", torch.__version__)
    print("unseen classes, 540 test images; R@1 / MAP@R")
    for name in ["pixels", "untrained", "contrastive", "norm-softmax"]:
        res = torch.tensor([run(s, name) for s in range(3)])
        print(f"{name:13s} " + "  ".join(f"seed {s}: {a:.3f} / {b:.3f}" for s, (a, b) in enumerate(res.tolist())))

PyTorch 2.14.1
unseen classes, 540 test images; R@1 / MAP@R
pixels        seed 0: 0.239 / 0.019  seed 1: 0.198 / 0.021  seed 2: 0.224 / 0.021
untrained     seed 0: 0.230 / 0.030  seed 1: 0.124 / 0.020  seed 2: 0.157 / 0.025
contrastive   seed 0: 0.600 / 0.357  seed 1: 0.537 / 0.262  seed 2: 0.617 / 0.338
norm-softmax  seed 0: 0.496 / 0.226  seed 1: 0.437 / 0.205  seed 2: 0.461 / 0.180

Raw pixels and the untrained network find a same-class neighbor for between one test image in eight and one in four (picking one of the other 539 images at random would succeed about once in 19), and their MAP@R of 0.02 to 0.03 shows that the rest of each class is scattered. Training on 18 classes for 300 steps more than doubled Recall@1 on the 18 new ones (0.44 to 0.62 against at most 0.24) and raised MAP@R at least sixfold (0.18 to 0.36), in every seed and for both losses: the network learned to discount position, stripe phase and lighting, and that invariance carries over to new combinations of shape and angle. MAP@R stays far below Recall@1, the gap that Musgrave et al. designed it to expose: a correct nearest neighbor does not mean that the class is gathered. The contrastive loss scored higher than the normalized softmax in all three seeds, but with a small network, 300 steps and an untuned margin and temperature this says nothing general about the two losses; equalizing such settings is what the comparison under Evidence and Impact did.

Libraries

Deep metric learning losses are short to write, but details such as the margin form, the reduction over pairs and the normalization change results. The pytorch-metric-learning library, maintained by Kevin Musgrave, implements the losses named here (ContrastiveLoss, TripletMarginLoss, NCALoss, ProxyNCALoss, NormalizedSoftmaxLoss, MultiSimilarityLoss, CosFaceLoss, ArcFaceLoss and others), miners such as DistanceWeightedMiner and MultiSimilarityMiner, and an accuracy calculator that reports Precision@1, R-precision and MAP@R (version 2.9.0, the latest release as of October 2026). Its ContrastiveLoss, for example, defaults to unsquared hinges $[d_p - m_{\text{pos}}]_+ + [m_{\text{neg}} - d_n]_+$ with $m_{\text{pos}} = 0$ and $m_{\text{neg}} = 1$, averaged over the pairs with non-zero loss, which is not Hadsell et al.’s squared form.

Variants

Family	Supervision per update	Examples	Distance
Linear (Mahalanobis)	pairs, triples or labels on fixed features	Xing et al. (2002), NCA (2004), LMNN (2005), ITML (2007)	$\lVert Lx - Ly \rVert$
Siamese pair losses	one pair	Chopra et al. (2005), contrastive loss (Hadsell et al., 2006)	learned $f_\theta$, Euclidean or $\ell_1$
Ranking losses	triplets or a batch	triplet loss, lifted structure, N-pair, margin loss, multi-similarity	learned $f_\theta$
Proxy and classification losses	one image against all class proxies	Proxy-NCA, normalized softmax, CosFace, ArcFace	learned $f_\theta$, usually cosine
Self-supervised contrastive	augmented views, no labels	InfoNCE, NT-Xent (contrastive learning)	learned $f_\theta$, cosine

Best Practices

• Normalize and compare consistently. Train and evaluate on $\ell_2$-normalized embeddings, as Musgrave et al. (2020) did for every loss, and use the same comparison (cosine or Euclidean, equivalent after normalization) at test time.
• Validate on held-out classes. Choose hyperparameters and the stopping point on classes disjoint from the training and test classes, and report the mean and spread over several runs.
• Hold the rest fixed when comparing losses. Backbone, embedding dimension, augmentation, batch construction and optimizer all move the numbers (see Evidence).
• Use class-balanced batches for pair and ranking losses, and tune the number of images per class.
• Try a classification baseline. A normalized softmax or Proxy-NCA loss needs no mining and was competitive in every comparison cited below.
• Report more than Recall@1. MAP@R or R-precision also measure how well a whole class is gathered.

Pitfalls

• Collapse. A loss with no repulsive term is minimized by a constant map (Chopra et al., 2005; Hadsell et al., 2006; Xing et al.’s constraint exists for the same reason).
• Training on the test set. Musgrave et al. (2020) found that most papers split each dataset into a training half and a test half with no validation set, checked test accuracy during training and reported the best value, which tunes hyperparameters to the test set (benchmark overfitting).
• Weak baselines. Low contrastive and triplet baselines were copied from paper to paper; Musgrave et al. suggest that some originate in the lifted structure paper, which sampled only $N/2$ pairs and $N/3$ triplets per batch of $N$ images and used a triplet margin of 1, where they found about 0.1 to work best.
• Unequal comparisons. Changing the pretrained network alone shifts results by several points: with ImageNet-pretrained features reduced to 512 dimensions and no metric training, Recall@1 on CUB-200-2011 was 41.1 for GoogleNet, 51.1 for BN-Inception and 48.7 for ResNet-50 (Musgrave et al., 2020, Table 1).
• Overfitting the training classes. A metric can separate the training classes perfectly and still generalize less well to new ones, as NCA did in the first example.
• Same name, different loss. “Contrastive loss” denotes Hadsell et al.’s squared form, unsquared variants with two margins, and in self-supervised learning the InfoNCE softmax; check the formula behind any implementation or reported number.

Evidence and Impact

Classical metric learning gave clear gains for nearest-neighbor classification on fixed features: LMNN’s MNIST result above, and NCA, whose test performance was the same as or better than that of Euclidean, whitened and RCA metrics on its six datasets (Goldberger et al., 2004). Deep metric learning then reported rapid progress on CUB-200-2011, Cars196 and Stanford Online Products. Proxy-NCA, for example, reported 73.22 Recall@1 on Cars196 against 58.11 for the best previous method in its Table 1, and normalized softmax reached 84.2 on Cars196 with a ResNet-50 and 512 dimensions (Zhai & Wu, 2019, Table 3).

Musgrave, Belongie and Lim (2020) retrained 13 losses and one loss-and-miner combination, from the contrastive loss of 2006 to losses of 2019, under one protocol: an ImageNet-pretrained BN-Inception trunk with frozen batch normalization, 128-dimensional normalized embeddings, batches of 32, the same augmentation and optimizer, hyperparameters chosen by 50 iterations of Bayesian optimization with four-fold class-disjoint cross-validation, and ten training runs per loss. They report each loss both with its four 128-dimensional fold models averaged (“separated”) and with their embeddings concatenated to 512 dimensions. In the concatenated setting, MAP@R spanned 23.1 to 26.7 on CUB-200-2011, 23.0 to 28.1 on Cars196 and 41.8 to 47.4 on Stanford Online Products, and the contrastive loss of 2006 scored 26.5, 24.9 and 44.4, within about three points of the best loss on each dataset (their Tables 4–6). Papers had often reported relative improvements of more than 100% over the contrastive loss; in this comparison the trend of accuracy against publication year was nearly flat. Their conclusion was that the improvements were marginal at best. Proxy-NCA’s Cars196 Precision@1 in this protocol was 83.56 (concatenated), and the contrastive loss’s was 81.78.

In our reading, the reality check does not show that the losses are interchangeable in every setting, only that on these benchmarks, with tuned hyperparameters and a fixed trunk, their differences were small next to the effects of the protocol. What the field established is its framing, learning a comparison that transfers to unseen classes, and its tools: siamese training, proxies and classification losses (which, Zhai and Wu noted, had already given state-of-the-art results in face verification), the sampling analysis, and the class-disjoint evaluation protocol.