Contrastive Loss

The contrastive loss trains an embedding from pairs of images labeled similar or dissimilar. It pulls the embeddings of a similar pair together and pushes those of a dissimilar pair apart, but only until they are a margin $m$ away from each other; beyond that, a dissimilar pair costs nothing. In the form of Hadsell, Chopra and LeCun (2006), the one defined below, the pull is half the squared distance and the push is half a squared hinge. A year earlier, Chopra, Hadsell and LeCun (2005) had titled a section of their face verification paper “Contrastive Loss Function used for Training”: their loss needed a contrastive term that raises the energy of non-matching pairs, because a loss that only lowered the energy of matching pairs could be minimized by mapping every image to the same point. Both papers trained it with a siamese network, two copies of one network with shared weights, and it is one of the pair and ranking losses of metric learning, which gives both papers’ formulas in its overview. This article covers the loss itself: its two original forms and their gradients, why the push term is needed, what the margin does, and the variants that share the name.

The name is overloaded. In self-supervised learning, “contrastive loss” usually means the InfoNCE or NT-Xent softmax over one positive and many negatives, covered in contrastive learning. This article is about the margin loss on pairs.

Definition

Given an embedding function $G_W$ with parameters $W$, a pair of inputs $(X_1, X_2)$ with a label $Y$ ($Y = 0$ for a similar pair, $Y = 1$ for a dissimilar one in Hadsell et al.’s convention), and the Euclidean distance $D_W = \lVert G_W(X_1) - G_W(X_2) \rVert_2$ between the two embeddings, the contrastive loss of the pair is

$$L(W, Y, X_1, X_2) = (1 - Y)\,\tfrac{1}{2}\, D_W^2 + Y\, \tfrac{1}{2} \big\{ \max(0,\, m - D_W) \big\}^2 ,$$

with a margin $m > 0$, and training minimizes its sum over all labeled pairs (Hadsell et al., 2006, Eqs. 1–4). Similar pairs are penalized for any distance; dissimilar pairs only while they are closer than $m$.

Intuition

Hadsell et al. described the loss as a system of springs between the embedded points (their Section 2.2). Each point is tied to every similar point by an attract-only spring with rest length zero: any separation produces a pull, proportional to the separation. Each point is tied to every dissimilar point by an m-repulse-only spring with rest length $m$: it pushes the two apart while they are closer than $m$, hardest when they coincide, and goes slack once they are farther than $m$, with no force pulling them back. Minimizing the loss lets the system settle into an equilibrium in which pulls and pushes balance. Hadsell et al. point out that the attract-only springs alone need not all reach zero length: if one point is tied to two others, shortening one spring can lengthen the other.

The springs explain the two design choices. Without the repulsive springs, the cheapest configuration puts every point in one place. Without the slack beyond $m$, dissimilar points would be pushed apart forever; in our reading, the loss would then reward spreading the embedding without limit rather than organizing it.

Formal Definition

Hadsell, Chopra and LeCun (2006)

Hadsell et al. wrote the loss over $P$ labeled pairs in a general form, with a partial loss $L_S$ for similar pairs and $L_D$ for dissimilar pairs (their Eqs. 2–3, with $L_S$ and $L_D$ as given in their Eqs. 6 and 8),

$$\mathcal{L}(W) = \sum_{i=1}^{P} \Big[ (1 - Y^i)\, L_S\big(D_W^i\big) + Y^i\, L_D\big(D_W^i\big) \Big], \qquad L_S(D) = \tfrac{1}{2} D^2, \quad L_D(D) = \tfrac{1}{2} \big(\max\{0, m - D\}\big)^2 ,$$

where $P$, the number of training pairs, may be as large as the square of the number of samples, since every pair of training samples can be labeled. They required only that minimizing the loss give low distances to similar pairs and high distances to dissimilar ones. Their gradients make the spring picture exact (Eqs. 7 and 9; equation and section numbers here are those of the authors’ copy dated November 2005, which was read):

$$\frac{\partial L_S}{\partial W} = D_W \frac{\partial D_W}{\partial W}, \qquad \frac{\partial L_D}{\partial W} = \begin{cases} -(m - D_W)\, \dfrac{\partial D_W}{\partial W}, & D_W < m, \\[4pt] 0, & D_W \ge m . \end{cases}$$

The attractive force grows with the distance $D_W$, the displacement of an attract-only spring; the repulsive force is largest, $m\,\partial D_W / \partial W$, when the two embeddings coincide, and vanishes at $D_W = m$. The total gradient of a siamese network is the sum of the contributions from its two branches (their Section 3.1).

The labels need not come from classes. Hadsell et al. built them from prior knowledge: MNIST digits paired with their own horizontal translations and those of their nearest neighbors, so that the learned two-dimensional map ignored horizontal shifts, and images of one airplane from the NORB set paired when taken from adjacent camera positions, whatever the lighting, so that the learned three-dimensional map was a cylinder organized by azimuth and elevation and invariant to lighting (their Sections 3.3–3.4). They called the method DrLIM, dimensionality reduction by learning an invariant mapping, and compared it with LLE: on the shifted digits with a Euclidean neighborhood graph, LLE separated differently registered samples, and on the airplane images with the same lighting-invariant neighborhood relations, LLE organized its embedding by lighting.

The energy-based form of Chopra, Hadsell and LeCun (2005)

Chopra et al. framed the problem with an energy-based model: a scalar energy $E_W(X_1, X_2) = \lVert G_W(X_1) - G_W(X_2) \rVert$ measures the compatibility between two inputs, low for a matching pair, and the loss depends on the inputs and parameters only through it (their Sections 2.2–2.3; section, equation and theorem numbers here are those of the authors’ copy, which was read). Their Eq. 1 writes the energy as a norm without a subscript, and a later remark contrasts it with the square norm as “the $L1$ norm”. A genuine pair (same person) should have low energy and an impostor pair high energy, with a margin between them: their Condition 1 asks for an $m > 0$ with $E_W(X_1, X_2) + m < E_W(X_1, X_2')$ for a genuine pair $(X_1, X_2)$ and an impostor pair $(X_1, X_2')$. The loss of a pair is $(1 - Y) L_G(E_W) + Y L_I(E_W)$, with partial losses $L_G$ for genuine and $L_I$ for impostor pairs.

They then asked which partial losses lead gradient descent to a solution that satisfies the margin condition. Write $H(E_W^G, E_W^I) = L_G(E_W^G) + L_I(E_W^I)$ for the total loss of one genuine and one impostor pair. If $H$ is convex in these two energies (not in $W$, which they state explicitly), has its minimum at infinity, and the negative of its gradient on the margin line $E_W^G + m = E_W^I$ has a positive dot product with the direction $[-1, 1]$ (their Condition 3), then minimizing $H$ over $W$ finds a $W$ that satisfies the margin condition, provided one exists (their Theorem 1). They note that Condition 3 holds whenever the genuine loss increases and the impostor loss decreases monotonically (the paper writes these partial losses $L_0$ and $L_1$ at that point). Their own choice, with $Q$ the upper bound of the energy, was

$$L(W, Y, X_1, X_2) = (1 - Y)\, \frac{2}{Q}\, (E_W)^2 + Y\, 2Q\, e^{-\frac{2.77}{Q} E_W}$$

(their Eq. 9; the metric learning article gives the same formula and their face verification results). The exponential impostor term has no margin parameter: it decreases for every energy, so every impostor pair is always pushed, more weakly the farther apart it is. The constants were chosen so that the negative gradient on the margin line points into the region of achievable energies, so that a gradient method does not stop at its boundary.

Their second remark concerns the energy itself. Using the square of the norm instead of the norm would be inappropriate, they argued, because the gradient of a squared-norm energy with respect to the parameters vanishes as the energy approaches zero, which creates a plateau in which the machine can fail to learn from an impostor pair whose energy is near zero. Hadsell et al.‘s loss follows the same choice: its distance $D_W$ is the norm, and only the similar pairs’ partial loss squares it.

Properties

Collapse without the contrastive term. Both papers state that the similar-pair term alone is minimized by a constant mapping, which gives zero distance and zero loss on every pair (Chopra et al., 2005, Section 2; Hadsell et al., 2006, Section 2.1). The dissimilar-pair term is what prevents it; Chopra et al. note that properly normalized probabilistic models do not have this problem, because raising the probability of one pair lowers that of the others. For a Mahalanobis metric the collapse is exact, by our derivation: the similar-pair term averaged over pairs is $\operatorname{tr}(CM)$ with $C = \tfrac{1}{2n}\sum \delta\delta^\top$ over the similar pairs ($n$ the total number of pairs), and if $C$ is positive definite, $\operatorname{tr}(CM) > 0$ for every nonzero $M \succeq 0$, so $M = 0$ is the only minimizer. The example below reaches it exactly.

Only nearby dissimilar pairs contribute. A dissimilar pair farther than $m$ has zero loss and zero gradient. As an embedding trains, more and more dissimilar pairs leave the margin, and pairs drawn at random stop contributing. Wu et al. (2017) noted that once a network converges very few negative margins are violated, and that hard negative mining usually speeds convergence for the contrastive loss. Which pairs are sampled therefore matters as much as the loss; the triplet loss and metric learning articles cover sampling and mining.

An absolute threshold. The loss asks every similar pair to come together and every dissimilar pair to be at least $m$ apart, with one $m$ for all of them. Wu et al. named this as its drawback against the triplet loss: a constant margin embeds visually diverse classes in the same small space as visually similar ones, and the loss spends effort gathering all positives as close as possible, while retrieval, clustering and verification only need the right relative order. The triplet loss’s own article sets out that comparison.

What the margin sets. The margin has units of distance, so its effect depends on whether the scale of the embedding is free. By our derivation, scaling every distance and the margin by the same $c > 0$ multiplies every pair’s loss by $c^2$. For a Mahalanobis metric this reads $f(M; m) = m^2 f(M/m^2; 1)$, so if $M^\ast$ minimizes the loss with margin 1, then $m^2 M^\ast$ minimizes it with margin $m$: every distance is multiplied by $m$, the same dissimilar pairs fall inside the margin, and the margin only sets the scale of the embedding. In our reading the same holds for any family of embeddings closed under positive scaling, such as a network whose last layer is linear and unnormalized. When the scale is fixed, for example by $\ell_2$-normalizing the embeddings, every distance lies between 0 and 2 and the margin becomes a real choice: with $m > 2$ every dissimilar pair is always inside it. The example checks a free and a fixed scale.

Convexity. The loss is not convex in a network’s weights. On a Mahalanobis metric it is, by our derivation: with $D_M^2 = \delta^\top M \delta$ for $\delta = X_1 - X_2$ and $M \succeq 0$, the similar-pair term $\tfrac{1}{2}\delta^\top M \delta$ is linear in $M$, and the dissimilar-pair term is $g(s) = \tfrac{1}{2}\max(0, m - \sqrt{s})^2$ of $s = \delta^\top M \delta$. On $0 \le s < m^2$, $g(s) = \tfrac{1}{2}m^2 - m\sqrt{s} + \tfrac{1}{2}s$ is convex because $\sqrt{s}$ is concave; it is zero beyond $m^2$, with a derivative that reaches zero continuously there. A convex function of a linear function of $M$ is convex, so fitting a Mahalanobis metric under the contrastive loss is a convex problem over the cone of positive semidefinite matrices. Written in terms of $L$ with $M = L^\top L$, the parameterization a network’s last layer has, it is not. This is the same split as in Chopra et al.’s Section 2.3, which assumes for Theorem 1 that the loss is convex in the energies and notes that convexity with respect to $W$ is not assumed. The convexity is not strict, so the minimizing $M$ need not be unique, but the minimum value is.

Which gradients vanish. The force on a dissimilar pair near zero distance depends on how the distance is computed and on which parameters the gradient is taken with respect to. With the norm as the distance and a linear embedding $G(x) = Lx$, $\partial D / \partial L$ has the same size $\lVert \delta \rVert$ at every distance, so Hadsell et al.’s push tends to $m \lVert \delta \rVert$ as the pair closes; with the squared norm, $\partial D^2 / \partial L = 2 (L\delta)\delta^\top$ shrinks with the distance and the push vanishes, which is Chopra et al.’s plateau. In terms of $M$, the opposite holds: $\partial D / \partial M = \delta\delta^\top / (2D)$ grows without bound as $D \to 0$, while $\partial D^2 / \partial M = \delta\delta^\top$ is constant. These are our derivations for the linear case; Chopra et al.’s argument concerns the parameters of the network, which play the role of $L$. The example computes these gradients.

Examples

Example: collapse, the margin and the vanishing push on a learned Mahalanobis metric

A network is not needed to see these properties: by the convexity shown above, a Mahalanobis metric $D_M(x_1, x_2) = \sqrt{\delta^\top M \delta}$, $\delta = x_1 - x_2$, fitted under Hadsell et al.’s loss is a convex problem, so its optimum can be found exactly and checked. The NumPy script below takes $7 \times 7$ patches, 49-dimensional vectors of raw gray values, from scikit-image’s bundled chelsea, rocket and coins images. A similar pair ($Y = 0$) is a patch and its copy shifted by one pixel in a random direction; a dissimilar pair ($Y = 1$) is two unrelated, non-overlapping patches; every patch gets independent Gaussian noise with standard deviation 0.01. There are 1,500 pairs of each kind, and the labels are known by construction. The loss is Eq. 4 averaged over the 3,000 pairs as a function of $M \succeq 0$, minimized by projected gradient descent with backtracking for 2,000 iterations from $M = I$; the projection clips negative eigenvalues. The script runs three parts:

1. Collapse. The fit with the pull term only.
2. The margin. Fits for margins $m = c\,\bar d$, where $\bar d$ is the median Euclidean distance of the dissimilar pairs and $c \in \{0.25, 0.5, 1, 2\}$, once with the scale free (any $M \succeq 0$) and once with it fixed ($\operatorname{tr} M = 49$, the trace of the Euclidean metric $M = I$, by projecting the eigenvalues onto a simplex). For each it prints the share of dissimilar pairs inside the margin (active, $D < m$), the share of similar pairs within $m$, how far $M(c)/c^2$ is from $M(1)$, and the optimality residual $\lVert M - \Pi(M - \nabla f) \rVert_F / \lVert M \rVert_F$ of the projected gradient $\Pi$. At $c = 1$ it also refits from a random positive semidefinite start.
3. The push near zero distance. For the closest dissimilar pair under the fitted metric ($c = 1$), it writes the embedding as $G(x) = Lx$ with $L$ the symmetric square root of $M$, shrinks it to $tL$, and prints the size of the gradient of the push term for Hadsell et al.’s squared hinge and Chopra et al.’s exponential, each with the norm $D$ or the squared norm $D^2$ as the energy, with respect to $L$ and with respect to $M$. Chopra et al.’s $Q$ is set to the largest energy over all pairs under the fitted metric (their $Q$ is the upper bound of the energy), and to its square for the squared energy. The analytic gradients with respect to $L$ are checked against central differences.

It uses NumPy 2.3.5 and scikit-image 0.25.2 on one BLAS thread, trains no network, and takes about 40 seconds; two runs printed identical output.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                                  # one BLAS thread, set before NumPy loads
import numpy as np
from skimage import data
from skimage.color import rgb2gray

K, P, N, SIGMA = 7, 1500, 1500, 0.01                     # 7x7 patches, pairs per label, pixel noise
imgs = [rgb2gray(data.chelsea()), rgb2gray(data.rocket()), data.coins() / 255.0]
rng = np.random.default_rng(0)

def patch(i, r, c):
    return imgs[i][r:r + K, c:c + K].ravel() + SIGMA * rng.standard_normal(K * K)

def make_pairs():
    """Rows are delta = x1 - x2. Y = 0: a patch and its copy shifted by one pixel;
    Y = 1: two unrelated patches (Hadsell et al.'s label convention)."""
    pos, neg = [], []
    while len(pos) < P:
        i = rng.integers(3); H, W = imgs[i].shape
        r, c = rng.integers(1, H - K - 1), rng.integers(1, W - K - 1)
        dy, dx = rng.integers(-1, 2, 2)
        if dy == 0 and dx == 0:
            continue
        pos.append(patch(i, r, c) - patch(i, r + dy, c + dx))
    while len(neg) < N:
        i, j = rng.integers(3, size=2)
        r1, c1 = rng.integers(0, imgs[i].shape[0] - K), rng.integers(0, imgs[i].shape[1] - K)
        r2, c2 = rng.integers(0, imgs[j].shape[0] - K), rng.integers(0, imgs[j].shape[1] - K)
        if i == j and abs(r1 - r2) < K and abs(c1 - c2) < K:
            continue                                      # overlapping patches are not dissimilar
        neg.append(patch(i, r1, c1) - patch(j, r2, c2))
    return np.array(pos + neg), np.r_[np.zeros(P), np.ones(N)]

def loss_grad(M, X, y, m, attract_only=False):
    """Hadsell et al. (2006), Eq. 4, averaged over pairs, as a function of M (D^2 = delta^T M delta)."""
    s = np.maximum(((X @ M) * X).sum(1), 0.0)
    D = np.sqrt(s)
    hinge = np.maximum(m - D, 0.0)
    push = 0.0 if attract_only else 1.0
    l = np.where(y == 0, 0.5 * s, push * 0.5 * hinge ** 2)
    w = np.where(y == 0, 0.5, push * -hinge / (2 * np.maximum(D, 1e-12)))   # dl/ds
    return l.mean(), (X * w[:, None]).T @ X / len(X)

def proj_psd(A):                                          # nearest PSD matrix
    lam, V = np.linalg.eigh((A + A.T) / 2)
    return (V * np.maximum(lam, 0.0)) @ V.T

def proj_trace(A, T=49.0):                                # nearest PSD matrix with trace T
    lam, V = np.linalg.eigh((A + A.T) / 2)
    u = np.sort(lam)[::-1]; css = np.cumsum(u) - T
    k = np.nonzero(u - css / np.arange(1, len(u) + 1) > 0)[0][-1]
    return (V * np.maximum(lam - css[k] / (k + 1), 0.0)) @ V.T

def fit(X, y, m, proj, M0=None, iters=2000, attract_only=False):
    """Projected gradient descent with backtracking; the problem is convex in M."""
    M, t = (np.eye(X.shape[1]) if M0 is None else M0.copy()), 1.0
    f, G = loss_grad(M, X, y, m, attract_only)
    zero_at = None
    for it in range(iters):
        t = min(2 * t, 1e6)                                 # capped step growth
        while True:
            Mn = proj(M - t * G)
            fn, Gn = loss_grad(Mn, X, y, m, attract_only)
            dM = Mn - M
            if fn <= f + (G * dM).sum() + (dM ** 2).sum() / (2 * t) + 1e-15:
                break
            t /= 2
        M, f, G = Mn, fn, Gn
        if zero_at is None and not M.any():
            zero_at = it + 1
    kkt = np.linalg.norm(M - proj(M - G)) / max(np.linalg.norm(M), 1e-300)
    return M, f, kkt, zero_at

X, y = make_pairs()
DI = np.linalg.norm(X, axis=1)
dbar = np.median(DI[y == 1])
C = X[y == 0].T @ X[y == 0] / (2 * len(X))                # attract-only loss = trace(C M)
print(f"{P} similar + {N} dissimilar pairs of {K}x{K} patches; Euclidean median D: "
      f"similar {np.median(DI[y == 0]):.4f}, dissimilar {dbar:.4f}")
print(f"smallest eigenvalue of C (attract-only loss = tr(C M)): {np.linalg.eigvalsh(C).min():.3e}")

# 1. Collapse: the pull term alone, from M = I
M0, f0, _, z = fit(X, y, dbar, proj_psd, attract_only=True)
print(f"\nattract-only: M == 0 exactly from iteration {z}; loss {f0}; max|M| {np.abs(M0).max()}")

# 2. The margin: free scale (any PSD M) against fixed scale (trace M = 49, the trace of I)
print("\n m/dbar | free scale: active dissim  similar<m  |M/c^2 - M1|/|M1|  KKT | "
      "trace 49: active dissim  similar<m  KKT")
fits = {}
for c in (0.25, 0.5, 1.0, 2.0):
    m = c * dbar
    fits[c] = [fit(X, y, m, proj_psd), fit(X, y, m, proj_trace)]
for c in (0.25, 0.5, 1.0, 2.0):
    m = c * dbar
    row = f"  {c:4.2f}  |"
    for k, (M, f, kkt, _) in enumerate(fits[c]):
        D = np.sqrt(np.maximum(((X @ M) * X).sum(1), 0))
        row += f"   {(D[y == 1] < m).mean():.4f}       {(D[y == 0] < m).mean():.4f}"
        if k == 0:
            dev = np.linalg.norm(M / c**2 - fits[1.0][0][0]) / np.linalg.norm(fits[1.0][0][0])
            row += f"       {dev:.1e}       {kkt:.0e} |"
        else:
            row += f"     {kkt:.0e}"
    print(row)
A = np.random.default_rng(1).standard_normal((49, 49)); R = A @ A.T; R *= 49 / np.trace(R)
for k, (name, proj) in enumerate((("free scale", proj_psd), ("trace 49", proj_trace))):
    f2 = fit(X, y, dbar, proj, M0=R)[1]
    print(f"m = dbar, {name}: loss {fits[1.0][k][1]:.8f}; from a random start {f2:.8f}; "
          f"relative difference {abs(f2 - fits[1.0][k][1]) / fits[1.0][k][1]:.1e}")

# 3. The push on the closest dissimilar pair as its distance shrinks (G(x) = t L x, M = L^T L)
M1 = fits[1.0][0][0]
lam, V = np.linalg.eigh(M1)
L = (V * np.sqrt(np.maximum(lam, 0))) @ V.T                # symmetric square root of M
Dall = np.linalg.norm(X @ L, axis=1)
delta = X[y == 1][np.argmin(Dall[y == 1])]
m, Q, Q2 = dbar, Dall.max(), Dall.max() ** 2               # Chopra et al.: Q = upper bound of the energy
nd = np.linalg.norm(delta)
dL = {"hadsell": lambda E, Qe: -max(m - E, 0.0), "chopra": lambda E, Qe: -2.77 * 2 * np.exp(-2.77 * E / Qe)}

def grads(t, loss, squared):
    """Analytic gradient of the impostor term w.r.t. L and w.r.t. M."""
    Lt = t * L; u = Lt @ delta; D = np.linalg.norm(u)
    E, Qe = (D ** 2, Q2) if squared else (D, Q)
    g = dL[loss](E, Qe)
    gL = g * (2 * np.outer(u, delta) if squared else np.outer(u, delta) / D)
    gM = g * (np.outer(delta, delta) if squared else np.outer(delta, delta) / (2 * D))
    return D, gL, gM

print(f"\nclosest dissimilar pair under M(m = dbar): ||delta|| = {nd:.5f}; m||delta|| = {m * nd:.5f}, "
      f"5.54||delta|| = {5.54 * nd:.5f}")
print("gradient norms of the push term; energy E = D or E = D^2; Hadsell: (1/2)max(0, m - E)^2, "
      "Chopra: 2Q exp(-2.77 E/Q)")
cols = " ".join(f"{c:>10s}" for c in ("Hadsell D", "Hadsell D^2", "Chopra D", "Chopra D^2"))
print(f"{'':14s}| {'w.r.t. L (G(x) = L x)':^43s} | {'w.r.t. M = L^T L':^43s}")
print(f"{'t':>6s} {'D':>7s} | {cols} | {cols}")
for t in (1.0, 0.1, 0.01, 0.001):
    out = [grads(t, l, s) for l, s in (("hadsell", False), ("hadsell", True), ("chopra", False), ("chopra", True))]
    print(f"{t:6.3f} {out[0][0]:.1e} | " + " ".join(f"{np.linalg.norm(o[1]):.4e}" for o in out)
          + " | " + " ".join(f"{np.linalg.norm(o[2]):.4e}" for o in out))

worst = 0.0                                                # central differences w.r.t. L
for t in (1.0, 0.1):
    for l, s in (("hadsell", False), ("hadsell", True), ("chopra", False), ("chopra", True)):
        def F(Lv):
            E = np.linalg.norm(Lv @ delta)
            return (0.5 * max(m - (E * E if s else E), 0.0) ** 2 if l == "hadsell"
                    else 2 * (Q2 if s else Q) * np.exp(-2.77 * (E * E if s else E) / (Q2 if s else Q)))
        gL = grads(t, l, s)[1]
        h = 1e-5
        num = np.zeros_like(L)
        for a in range(49):
            for b in range(49):
                Eab = np.zeros_like(L); Eab[a, b] = h
                num[a, b] = (F(t * L + Eab) - F(t * L - Eab)) / (2 * h)
        worst = max(worst, np.linalg.norm(gL - num) / np.linalg.norm(gL))
print(f"worst relative error, analytic vs central-difference gradients w.r.t. L: {worst:.1e}")

1500 similar + 1500 dissimilar pairs of 7x7 patches; Euclidean median D: similar 0.1751, dissimilar 1.3915
smallest eigenvalue of C (attract-only loss = tr(C M)): 1.462e-04

attract-only: M == 0 exactly from iteration 12; loss 0.0; max|M| 0.0

 m/dbar | free scale: active dissim  similar<m  |M/c^2 - M1|/|M1|  KKT | trace 49: active dissim  similar<m  KKT
  0.25  |   0.2533       0.9953       5.1e-07       7e-09 |   0.2413       0.9533     9e-07
  0.50  |   0.2533       0.9953       7.8e-08       7e-10 |   0.2460       0.9920     7e-08
  1.00  |   0.2533       0.9953       0.0e+00       9e-11 |   0.2513       0.9960     1e-10
  2.00  |   0.2533       0.9953       4.4e-08       1e-11 |   0.2593       0.9960     1e-11
m = dbar, free scale: loss 0.05312037; from a random start 0.05312037; relative difference 3.3e-15
m = dbar, trace 49: loss 0.05547804; from a random start 0.05547804; relative difference 4.4e-12

closest dissimilar pair under M(m = dbar): ||delta|| = 0.10699; m||delta|| = 0.14887, 5.54||delta|| = 0.59273
gradient norms of the push term; energy E = D or E = D^2; Hadsell: (1/2)max(0, m - E)^2, Chopra: 2Q exp(-2.77 E/Q)
              |            w.r.t. L (G(x) = L x)            |              w.r.t. M = L^T L              
     t       D |  Hadsell D Hadsell D^2   Chopra D Chopra D^2 |  Hadsell D Hadsell D^2   Chopra D Chopra D^2
 1.000 3.6e-02 | 1.4502e-01 1.0721e-02 5.8746e-01 4.2723e-02 | 2.1525e-01 1.5913e-02 8.7198e-01 6.3415e-02
 0.100 3.6e-03 | 1.4849e-01 1.0731e-03 5.9220e-01 4.2725e-03 | 2.2041e+00 1.5928e-02 8.7902e+00 6.3417e-02
 0.010 3.6e-04 | 1.4884e-01 1.0731e-04 5.9268e-01 4.2725e-04 | 2.2092e+01 1.5928e-02 8.7973e+01 6.3417e-02
 0.001 3.6e-05 | 1.4887e-01 1.0731e-05 5.9273e-01 4.2725e-05 | 2.2097e+02 1.5928e-02 8.7980e+02 6.3417e-02
worst relative error, analytic vs central-difference gradients w.r.t. L: 9.5e-06

Collapse. With the pull term alone, the fit reached $M = 0$ exactly, every entry and the loss equal to zero, at iteration 12. By our derivation this is the only possible outcome: the printed smallest eigenvalue of $C$ is positive, so $\operatorname{tr}(CM)$ is positive for every other $M$. The projection makes it exact rather than approximate, since once a gradient step leaves no positive eigenvalue, the nearest positive semidefinite matrix is zero. With the push term added, every fit has an active share of about a quarter rather than 1, which it would be if $M$ were zero and all distances vanished.

The margin. With the scale free, all four margins gave the same shares, 0.2533 of the dissimilar pairs active and 0.9953 of the similar pairs within the margin, and $M(c)/c^2$ matched $M(1)$ to within $5.1 \times 10^{-7}$ of its norm, as the scaling $f(M; m) = m^2 f(M/m^2; 1)$ requires: an eightfold change of margin changed the metric’s scale and nothing else. With the trace fixed, the margin matters. The active share increased with the margin at every step, from 0.2413 at $c = 0.25$ to 0.2593 at $c = 2$, and at the smallest margin fewer similar pairs fell within it (0.9533). We had also expected few dissimilar pairs to be active at the smallest margin and most at the largest; that expectation failed: the share moved only from 0.2413 to 0.2593 over an eightfold range of margins. In our reading, with its overall scale fixed the metric still chooses how to spread that scale over directions, and it uses this freedom to keep roughly the same share of pairs inside the margin; the run does not test this. The optimality residuals are at most $9 \times 10^{-7}$, and the random starts reached the same loss to a relative $3.3 \times 10^{-15}$ (free scale) and $4.4 \times 10^{-12}$ (fixed trace), as convexity predicts for the minimum value.

The push. With the norm as the energy and the gradient taken with respect to $L$, both pushes approach a nonzero limit as the pair closes: Hadsell et al.’s tends to $m\lVert\delta\rVert = 0.14887$ and Chopra et al.’s to $5.54\lVert\delta\rVert = 0.59273$, the values printed at $t = 0.001$. With the squared norm as the energy, both fall tenfold for each tenfold drop of $t$ and $D$: this is the plateau that Chopra et al. warned against, in which an impostor pair at near-zero energy produces almost no gradient. With respect to $M$, the picture reverses: the norm-energy gradients grow about tenfold per decade, without bound, and the squared-energy ones stay nearly constant. Whether a gradient vanishes is therefore a property of the parameterization as well as the loss; for a network, the parameters play the role of $L$, so Chopra et al.’s argument applies. The analytic gradients agreed with central differences to a relative $9.5 \times 10^{-6}$.

Metric learning’s examples measure retrieval with learned metrics, including a network trained with this loss; this example measures none of that.

Variants

Chopra et al.’s exponential form (2005). A squared genuine term and a decaying exponential impostor term on a norm energy, with constants tied to the energy’s upper bound $Q$, as above. It has no hinge, so no impostor pair ever stops contributing.

Hadsell et al.’s squared hinge (2006). The form defined above, with one margin for dissimilar pairs and none for similar ones.

Unsquared hinges and two margins. Some implementations drop the squares and add a margin for similar pairs. In pytorch-metric-learning (version 2.9.0, the version read here), ContrastiveLoss applies $[d_p - m_{\text{pos}}]_+$ to positive pairs and $[m_{\text{neg}} - d_n]_+$ to negative pairs, with defaults $m_{\text{pos}} = 0$ and $m_{\text{neg}} = 1$; its default distance, LpDistance(p=2), $\ell_2$-normalizes the embeddings first; its default reduction averages the non-zero losses of the positive pairs and of the negative pairs separately and adds the two averages; and with a similarity such as the cosine instead of a distance, it swaps the two arguments of each hinge. A positive margin lets similar pairs stop at a distance $m_{\text{pos}}$ instead of collapsing onto each other, which, in our reading, answers part of Wu et al.’s criticism that the loss gathers all positives as close as possible.

Margin-based loss with a learned boundary. Wu et al. (2017) replaced the two fixed targets by $(\alpha + y_{ij}(D_{ij} - \beta))_+$, with $y_{ij} = 1$ for a positive pair and $-1$ for a negative one, where $\beta$ is a learned boundary between positive and negative distances and $\alpha$ the margin of separation. Positive pairs then only need to be within $\beta - \alpha$, not at zero distance. On Stanford Online Products, with networks following the ResNet-50 architecture trained from scratch and 128-dimensional embeddings, their Table 1 (read from the rendered page of the CVF open-access copy) gives Recall@1 of 30.1 for the contrastive loss with random pairs, 49.4 with semi-hard sampling and 39.2 with their distance-weighted sampling, against 37.5, 61.0 and 61.7 for the margin-based loss. They reported that the contrastive loss was very sensitive to its hyperparameters and that they had not found good ones for distance-weighted sampling. The triplet loss article discusses the semi-hard result.

Softmax over many negatives. The N-pair, InfoNCE and NT-Xent losses replace the hinge on one pair by a softmax over one positive and many negatives, which weights hard negatives without a fixed margin; contrastive learning covers them.

Common Misconceptions

• “Contrastive loss” names one formula. It names Chopra et al.’s exponential form, Hadsell et al.’s squared hinge, unsquared and two-margin variants, and in self-supervised learning the InfoNCE softmax. Read the formula behind an implementation or a reported number, including its label convention: Hadsell et al. use $Y = 0$ for similar pairs, and Wu et al. write $y_{ij} = 1$ for positive pairs.
• Dissimilar pairs are pushed as far apart as possible. They are pushed only to the margin; beyond it the loss and its gradient are zero.
• A larger margin always separates classes better. If the embedding’s scale is free, the margin only rescales the solution; it matters when the scale is fixed, for example by normalization.
• The loss is convex, so training is. The loss is convex in the energies (Chopra et al.’s assumption) and, by our derivation, in a Mahalanobis matrix $M$, but not in $L$ or in the weights of a network.
• Squaring the distance is a harmless simplification. For dissimilar pairs it makes the push with respect to a network’s parameters vanish near zero distance, the case Chopra et al. warned about.

Where It Is Used

• Face verification. Chopra et al. (2005) trained convolutional siamese networks on face images and verified identities never seen in training by thresholding the energy; the metric learning article gives their false accept and false reject rates.
• Dimensionality reduction and invariance. DrLIM (Hadsell et al., 2006) learned low-dimensional maps from neighborhood relations alone, invariant to shift or lighting by the choice of pairs.
• Image retrieval. Radenović, Tolias and Chum fine-tuned retrieval networks with a contrastive loss on matching and non-matching pairs mined from 3D reconstructions of unlabeled photo collections, as the image retrieval article describes.
• Benchmarks of metric learning losses. In the comparison of Musgrave, Belongie and Lim (2020) under one protocol, which the metric learning article summarizes, the contrastive loss of 2006 came within about three points of MAP@R of the best loss on each dataset; the same article’s second example trains it on synthetic shapes and reports retrieval scores.
• Self-supervised learning. Contrastive self-supervised methods use softmax losses over many negatives instead; the contrastive learning article cites Hadsell et al.’s collapse argument for why negatives are needed.