Loss Function

A loss function measures how wrong a model’s output is on one example: it takes the prediction and the desired output and returns a single number, usually non-negative and zero for a perfect prediction. Training a neural network means adjusting its weights to make the average loss over the training images small, so the loss is where a task’s goal is written down.

Definition

For a model $f(\mathbf{x}; \theta)$ with parameters $\theta$, a per-example loss $\ell(f(\mathbf{x}; \theta), \mathbf{y})$ compares the output for input $\mathbf{x}$ (an image, a pair of frames, a video clip) with its target $\mathbf{y}$ (a class label, a mask, a depth map, a flow field). The training objective averages the loss over the $N$ training examples and usually adds a regularization term that prefers simpler models. Goodfellow, Bengio and Courville (2016, §4.3) treat “objective”, “cost”, “loss” and “error function” as interchangeable; this article uses “loss” for one example and “objective” for the average.

Intuition

A classifier that puts probability 0.9 on “cat” for a cat photograph should be penalized a little; one that puts 0.01 on it, a lot. A loss turns that judgment into a number, and its gradient tells each weight which way to move. It also ranks kinds of error: a depth map off by 10 cm everywhere and one perfect except for a few wildly wrong pixels can tie under one loss and differ greatly under another.

Formal Definition

Training minimizes the regularized empirical risk

$$J(\theta) = \frac{1}{N} \sum_{i=1}^{N} \ell\big(f(\mathbf{x}_i; \theta), \mathbf{y}_i\big) + \lambda\, \Omega(\theta).$$

What we really want to minimize is the risk, the expected loss over the distribution of images the model will meet; the empirical risk replaces that unknown distribution with the training set (Goodfellow et al., 2016, §8.1.1). The penalty $\Omega(\theta)$, weighted by $\lambda \ge 0$, is a regularizer such as the squared norm of the weights (§7.1). Gradient descent and its stochastic variants minimize $J$ using gradients computed by backpropagation.

Losses as negative log-likelihoods

Many losses are not arbitrary: they are the negative log-likelihood of the target under a probabilistic model of the output, $\ell = -\log p(\mathbf{y} \mid \mathbf{x}; \theta)$, so minimizing the empirical risk is maximum-likelihood estimation (Goodfellow et al., 2016, §5.5 and §6.2.1.1).

• Squared error and Gaussian noise. If the target is the network’s output plus Gaussian noise of fixed variance $\sigma^2$, then $-\log p(\mathbf{y} \mid \mathbf{x}) = \lVert \mathbf{y} - f(\mathbf{x}; \theta) \rVert^2 / (2\sigma^2) + \text{const}$, the squared error (§5.5.1), the same argument that makes least squares the maximum-likelihood estimate.
• Absolute error and Laplacian noise. Repeating the argument with heavier-tailed Laplace noise, density proportional to $\exp(-\lvert r \rvert / b)$, gives $\lvert y - f \rvert / b + \text{const}$, the $L^1$ loss (our derivation).
• Cross-entropy and categorical labels. If the network’s softmax output $p_k$ is the probability of class $k$, the negative log-likelihood of the true class $c$ is $-\log p_c$, the cross-entropy loss.

The minimizers differ too: with a flexible enough model, squared error predicts the conditional mean of the target and absolute error the conditional median (Goodfellow et al., 2016, §6.2.1.2). The example below shows what that difference does to an image.

Losses by Task

Task	Typical loss	Network output	Article
Image classification	Cross-entropy	Class probabilities	Cross-Entropy Loss
Box regression in detection	Smooth $L^1$	Box offsets	—
Dense detection	Focal loss	Per-anchor class probabilities	—
Semantic segmentation	Per-pixel cross-entropy, Dice	Per-pixel class probabilities	—
Optical flow	Endpoint error, $L^1$	Flow field	Endpoint Error, RAFT
Self-supervised depth	Photometric reconstruction	Disparity or depth map	Depth Estimation
Metric learning	Contrastive, triplet	Embedding vector	—
Restoration and synthesis	$L^2$, $L^1$, perceptual, adversarial	Image	Image Denoising

Classification. Cross-entropy on softmax probabilities is the default; Fast R-CNN’s classification branch, for example, uses the log loss $-\log p_u$ for the true class $u$ (Girshick, 2015).

Regression and dense prediction. Real-valued outputs use the squared ($L^2$) error, the absolute ($L^1$) error, or the Huber loss, quadratic for small residuals and linear for large ones (nonlinear least squares). Fast R-CNN regresses bounding-box offsets with the smooth $L^1$ loss, $0.5x^2$ for $\lvert x \rvert < 1$ and $\lvert x \rvert - 0.5$ otherwise, which is the Huber loss with threshold 1. Girshick (2015) chose it as less sensitive to outliers than the $L^2$ loss of earlier detectors, which with unbounded targets can need careful learning-rate tuning to avoid exploding gradients. FlowNet was trained on the endpoint error, the Euclidean distance between predicted and true flow vectors averaged over pixels (Dosovitskiy et al., 2015). RAFT instead sums the $L^1$ distance to the ground truth over its iterative predictions, weighted toward the final one (Teed & Deng, 2020).

Segmentation. Segmentation classifies every pixel, so the basic loss is cross-entropy summed or averaged over pixels; the fully convolutional networks of Long, Shelhamer and Darrell (2015) summed it. When the object is small, background pixels dominate. Milletari, Navab and Ahmadi (2016) proposed maximizing a differentiable Dice coefficient between the predicted foreground probabilities $p_i$ and the binary ground truth $g_i$ over the voxels of a medical volume,

$$D = \frac{2 \sum_i p_i g_i}{\sum_i p_i^2 + \sum_i g_i^2},$$

which they found needs no class weights to balance foreground and background. Maximizing $D$ is the same as minimizing the loss $1 - D$.

Detection. A one-stage detector scores $10^4$ to $10^5$ candidate locations per image, almost all of them easy background. Lin et al. (2017) multiply the cross-entropy by a factor that shrinks for well-classified examples, giving the focal loss $-(1 - p_t)^\gamma \log p_t$, where $p_t$ is the probability assigned to the true class; $\gamma = 0$ recovers cross-entropy, and the authors found $\gamma = 2$ worked best.

Metric learning. Face recognition and learned feature descriptors need embeddings in which distance means dissimilarity. The contrastive loss of Hadsell, Chopra and LeCun (2006) pulls similar pairs together with $\tfrac{1}{2}d^2$, where $d$ is the distance between their embeddings, and pushes dissimilar pairs apart with $\tfrac{1}{2}\max(0, m - d)^2$ until they are a margin $m$ apart; without the second term, mapping every image to one point would minimize the loss. FaceNet’s triplet loss (Schroff, Kalenichenko & Philbin, 2015) asks that an anchor face be closer, in squared distance between unit-length embeddings, to another image of the same person than to an image of anyone else by a margin $\alpha$, penalizing $[\lVert f(\mathbf{x}^a) - f(\mathbf{x}^p) \rVert^2 - \lVert f(\mathbf{x}^a) - f(\mathbf{x}^n) \rVert^2 + \alpha]_+$.

Image restoration and synthesis. $L^2$ was long the default for restoration networks, but in the joint denoising and demosaicking experiments of Zhao et al. (2017) it left splotchy artifacts in flat regions, and a network trained with $L^1$ even reached a lower $L^2$ error. Johnson, Alahi and Fei-Fei (2016) noted that two identical images offset by one pixel differ greatly under any per-pixel loss; their perceptual loss compares instead the activations $\phi_j$ of a layer of a fixed VGG-16 network pretrained on ImageNet, by squared Euclidean distance. Generative adversarial networks (Goodfellow et al., 2014) go further and learn the loss: a second network is trained to tell generated images from real ones, and the generator to fool it.

Self-supervised reconstruction. When no ground truth exists, the loss can come from the images themselves. Monodepth (Godard, Mac Aodha & Brostow, 2017) predicts a disparity map from the left image of a stereo pair, warps the right image with it to reconstruct the left, and scores the reconstruction with a photometric loss that mixes an SSIM term and an $L^1$ term, weighted 0.85 and 0.15, plus edge-aware smoothness and left-right consistency terms.

Properties

Differentiability. Accuracy changes only when a prediction flips class, so its derivative is zero or undefined everywhere (Goodfellow et al., 2016, §8.1.1); the negative log-likelihood of the correct class is the usual surrogate (§8.1.2). The kink of $L^1$ at zero is handled by subgradients and small steps. Goodfellow et al. also note that $L^2$ and $L^1$ combined with saturating output units can give very small gradients, one reason cross-entropy is preferred even when a full distribution over outputs is not needed (§6.2.1.2).

Robustness to outliers. The gradient of $\tfrac{1}{2}r^2$ is $r$, so a residual ten times larger pulls ten times harder; the gradient of $\lvert r \rvert$ is $\pm 1$ whatever its size, and Huber caps it at its threshold. Corrupted pixels, occlusions and mislabeled boxes therefore dominate an $L^2$ fit.

Label noise. Ghosh, Kumar and Sastry (2017) gave a sufficient condition, that the loss summed over all possible labels be constant, under which the minimizer of the risk is unchanged by uniform label noise with noise rate below $(k-1)/k$ for $k$ classes. The mean absolute error between the softmax output and the one-hot label satisfies it; cross-entropy and squared error do not.

Scale and balance. When losses are added, as classification and box regression are in Fast R-CNN, their scales and weights matter: Girshick normalized the regression targets to zero mean and unit variance and weighted the two terms equally.

Examples

Denoising a burst with $L^2$, $L^1$ and Huber losses. A camera captures seven frames of a static scene. Each frame has Gaussian noise on every pixel, and 20% of its pixels are hit by salt-and-pepper noise, set to pure black or white. We estimate the clean $32 \times 32$ image $\mathbf{u}$ by gradient descent on the loss between $\mathbf{u}$ and all frames, summed over pixels and averaged over frames, with each of three losses. The analytic gradients are checked against finite differences first.

import numpy as np

rng = np.random.default_rng(10)

# A 32x32 grayscale image in [0, 1]: a bright disk on a horizontal ramp.
yy, xx = np.mgrid[0:32, 0:32]
clean = 0.2 + 0.4 * xx / 31
clean[(yy - 16) ** 2 + (xx - 12) ** 2 < 8 ** 2] = 0.9

# A burst of K frames of the static scene: Gaussian noise (sigma 0.05) on every
# pixel, plus salt-and-pepper noise that sets 20% of each frame's pixels to 0 or 1.
K = 7
frames = clean + rng.normal(0, 0.05, (K, 32, 32))
hit = rng.random((K, 32, 32)) < 0.2
frames[hit] = rng.integers(0, 2, hit.sum()).astype(float)

# Each loss compares the estimate u with every frame, sums over pixels and
# averages over frames; it returns the loss and its gradient with respect to u.
def l2(u):
    r = u - frames
    return 0.5 * np.sum(r ** 2) / K, r.mean(axis=0)

def l1(u):
    r = u - frames
    return np.sum(np.abs(r)) / K, np.sign(r).mean(axis=0)

def huber(u, d=0.05):
    r = u - frames
    a = np.abs(r)
    rho = np.where(a <= d, 0.5 * r ** 2, d * (a - 0.5 * d))
    return np.sum(rho) / K, np.clip(r, -d, d).mean(axis=0)

losses = {"L2": l2, "L1": l1, "Huber": huber}

# Check each gradient against central finite differences at 8 pixels of a random image.
u0 = rng.random((32, 32))
for name, f in losses.items():
    g = f(u0)[1]
    rel = []
    for i in rng.choice(u0.size, 8, replace=False):
        e = np.zeros(u0.size)
        e[i] = 1e-6
        e = e.reshape(u0.shape)
        num = (f(u0 + e)[0] - f(u0 - e)[0]) / 2e-6
        rel.append(abs(num - g.flat[i]) / max(abs(num), 1e-12))
    print(f"{name:5s} gradient check: max relative error {max(rel):.1e}")

# Gradient descent from a uniform gray image, with a slowly shrinking step
# so that the L1 iterates settle instead of oscillating around the minimum.
def fit(f, steps=2000):
    u = np.full((32, 32), 0.5)
    for t in range(steps):
        u -= 0.5 / (1 + t / 100) * f(u)[1]
    return u

def psnr(u):  # peak signal-to-noise ratio against the clean image, in dB
    return 10 * np.log10(1.0 / np.mean((u - clean) ** 2))

print(f"one noisy frame:   PSNR {psnr(frames[0]):5.1f} dB")
for name, f in losses.items():
    print(f"{name:5s} estimate:    PSNR {psnr(fit(f)):5.1f} dB")
print(f"per-pixel mean:    PSNR {psnr(frames.mean(axis=0)):5.1f} dB")
print(f"per-pixel median:  PSNR {psnr(np.median(frames, axis=0)):5.1f} dB")

Output:

L2    gradient check: max relative error 6.6e-08
L1    gradient check: max relative error 4.0e-08
Huber gradient check: max relative error 2.4e-08
one noisy frame:   PSNR  11.9 dB
L2    estimate:    PSNR  19.7 dB
L1    estimate:    PSNR  25.7 dB
Huber estimate:    PSNR  26.1 dB
per-pixel mean:    PSNR  19.7 dB
per-pixel median:  PSNR  25.7 dB

The $L^2$ estimate converges to the per-pixel mean of the frames and the $L^1$ estimate to the per-pixel median, as the minimizers predict. Every corrupted value pulls the mean by its full residual: 19.7 dB. The median ignores a minority of extreme values, gaining 6 dB; it can fail only where four or more of the seven frames are corrupted, about 3% of pixels by the binomial distribution. The Huber loss, with its threshold set to the Gaussian noise level, averages the small residuals like $L^2$ and caps the large ones like $L^1$, and scores slightly higher. Over 16 random seeds $L^2$ gave 19.1–20.2 dB, $L^1$ 23.7–29.2 dB, and Huber 0.2–0.5 dB more than $L^1$.

Common Misconceptions

• “The loss is the evaluation metric.” Often it is a differentiable stand-in. Classifiers train on cross-entropy and are judged by accuracy; Long et al. (2015) trained on per-pixel cross-entropy and reported mean intersection over union; RAFT trains on $L^1$ and is judged by endpoint error and outlier rates. A lower loss usually means a better metric, but validation should track the metric itself.
• “A loss that ignores easy examples ignores outliers.” The focal loss down-weights easy, correct examples (inliers); a robust loss down-weights large errors (outliers). Lin et al. (2017) say the focal loss plays the opposite role of a robust loss.

Where It Is Used

Every trained vision model has a loss, from the flow networks of optical flow estimation to self-supervised depth estimation. The robust costs of classical estimation, used to minimize reprojection error, reappear as smooth $L^1$ and Huber losses in deep networks. Training a neural network minimizes the loss with gradient descent and backpropagation.