Cross-Entropy Loss

Cross-entropy loss scores a classifier by the probability it assigns to the correct label: the loss is $-\log p_y$, small when the true class gets most of the probability and unbounded as that probability goes to zero. It is the default loss function for image classification, and averaged over pixels it is the basic loss for semantic segmentation.

Definition

The cross-entropy of a predicted distribution $q$ relative to a target distribution $p$ over the same $K$ classes is the expected negative log-probability that $q$ assigns to outcomes drawn from $p$. As a loss, $p$ is the label (usually one-hot, all mass on the true class $y$) and $q$ is the network’s predicted distribution, so the loss reduces to $-\log q_y$. This is the negative log-likelihood (NLL) of the label under the model, so for classifiers the two names mean the same loss.

Intuition

Suppose a digit classifier sees an image of a 7. If it gives “7” probability 0.9, the loss is $-\ln 0.9 = 0.11$; at 0.01 it is 4.6; at $10^{-6}$, 13.8. Being confidently wrong costs far more than being unsure, and there is no ceiling. A classifier that predicts $1/K$ for every class pays $\ln K$, 2.30 for ten digits, a useful check on the loss at the start of training. In information-theoretic terms, $-\log q_y$ is the surprise of the true label under the model’s beliefs.

Formal Definition

Entropy, cross-entropy and KL divergence

Shannon (1948) defined the entropy of a set of probabilities $p_1, \dots, p_n$ as $-\sum_i p_i \log p_i$, up to a positive constant that only sets the unit; with base-2 logarithms the unit is the bit. Machine learning usually uses the natural logarithm and measures in nats (Goodfellow, Bengio & Courville, 2016, §3.13). For distributions $p$ and $q$ over $K$ classes,

$$H(p) = -\sum_{k=1}^{K} p_k \log p_k, \qquad H(p, q) = -\sum_{k=1}^{K} p_k \log q_k, \qquad \mathrm{KL}(p \,\Vert\, q) = \sum_{k=1}^{K} p_k \log \frac{p_k}{q_k},$$

and expanding the logarithm in the last expression gives

$$H(p, q) = H(p) + \mathrm{KL}(p \,\Vert\, q).$$

The Kullback–Leibler divergence is non-negative and zero only when $q = p$; it is the extra information needed to code samples from $p$ with a code designed for $q$, and it is not symmetric (Goodfellow et al., 2016, §3.13). Since $H(p)$ does not depend on the model, minimizing the cross-entropy over $q$ is the same as minimizing the KL divergence. With a one-hot label $H(p) = 0$, so the loss equals the KL divergence.

Categorical cross-entropy with softmax

A classifier maps an image $\mathbf{x}$ to a vector of logits $\mathbf{z} = f(\mathbf{x}; \theta) \in \mathbb{R}^K$, unnormalized log-probabilities, and the softmax function turns them into probabilities, $p_k = e^{z_k} / \sum_j e^{z_j}$ (see neural network). For a training image with label $y$,

$$\ell(\mathbf{z}, y) = -\log p_y = -z_y + \log \sum_{j=1}^{K} e^{z_j}.$$

The logarithm undoes the exponential in the softmax. Goodfellow et al. (2016, §6.2.2.3) point out that the first term, $-z_y$, always contributes directly and cannot saturate, while the log-sum-exp term behaves roughly like $\max_j z_j$, so the loss strongly penalizes the most confident wrong class and contributes little once the true class already has the largest logit.

The gradient with respect to the logits

Differentiate each term. $\partial(-z_y)/\partial z_k$ is $-1$ for $k = y$ and 0 otherwise, and

$$\frac{\partial}{\partial z_k} \log \sum_j e^{z_j} = \frac{e^{z_k}}{\sum_j e^{z_j}} = p_k.$$

Together,

$$\frac{\partial \ell}{\partial \mathbf{z}} = \mathbf{p} - \mathbf{e}_y,$$

the predicted probabilities minus the one-hot label. For a soft target $\mathbf{q}$ the same steps give $\mathbf{p} - \mathbf{q}$, which Szegedy et al. (2016) note is bounded between $-1$ and $1$. The components sum to zero; the true class is pushed up with strength $1 - p_y$ and each wrong class pushed down with strength $p_k$. Backpropagation starts from this vector.

Binary cross-entropy for multi-label images

A photograph can contain both a dog and a cat. A softmax makes the classes compete for one unit of probability. Multi-label classification instead gives each label $c$ its own logit and a sigmoid, $\sigma(z_c) = 1/(1 + e^{-z_c})$, and sums a binary cross-entropy over the labels:

$$\ell = -\sum_{c} \big[ t_c \log \sigma(z_c) + (1 - t_c) \log\big(1 - \sigma(z_c)\big) \big],$$

with $t_c \in \{0, 1\}$. The gradient for each label is $\sigma(z_c) - t_c$. Goodfellow et al. (2016, §6.2.2.2) show that this loss saturates only when the answer is already right; for a badly wrong logit its derivative tends to $\pm 1$.

Per-pixel cross-entropy for segmentation

Semantic segmentation predicts a class for every pixel: the network outputs $K \times H \times W$ logits and the loss averages the categorical cross-entropy over the labeled pixels. PyTorch’s CrossEntropyLoss accepts logits of shape $(N, C, H, W)$ with an $(N, H, W)$ map of class indices for this, and an ignore_index for unlabeled pixels (PyTorch documentation). The loss function article describes the Dice loss, proposed for when background pixels dominate that average.

Properties

Maximum likelihood. Averaging $-\log p_{y_i}$ over the training set is the negative log-likelihood of the labels, so minimizing cross-entropy is maximum-likelihood estimation; equivalently it minimizes the KL divergence from the empirical label distribution to the model (Goodfellow et al., 2016, §5.5 and §6.2.1.1). In the same sense the squared error is the cross-entropy of a Gaussian output model.

Minimum value. With one-hot labels the loss approaches zero only as $z_y - z_k \to \infty$ for every wrong class $k$, a limit no finite weights reach (Szegedy et al., 2016). With a soft target $\mathbf{q}$ it follows from the decomposition above that the minimum is $H(\mathbf{q}) > 0$, reached at $\mathbf{p} = \mathbf{q}$.

Why not squared error. The gradient of $\tfrac{1}{2}\lVert \mathbf{p} - \mathbf{e}_y \rVert^2$ with respect to the logits passes through the softmax Jacobian $\operatorname{diag}(\mathbf{p}) - \mathbf{p}\mathbf{p}^\top$, every entry of which contains a factor $p_k$. When the network is confidently wrong, $p_y \approx 0$ and the wrong class has $p \approx 1$, so the gradient nearly vanishes exactly when it is most needed. Goodfellow et al. (2016, §6.2.2.2–§6.2.2.3) describe this saturation and call squared error a poor loss for softmax units. The cross-entropy gradient $\mathbf{p} - \mathbf{e}_y$ has no such factor: it is largest when the prediction is most wrong.

Numerical stability. Computing $\log(\operatorname{softmax}(\mathbf{z})_y)$ literally overflows when a logit is large (in 32-bit floating point $e^{z}$ is infinite for $z$ above about 88.7) and gives $\log 0 = -\infty$ when a probability underflows. The softmax is unchanged by subtracting a constant from every logit (Goodfellow et al., 2016, §6.2.2.3), so with $m = \max_j z_j$

$$\log \sum_j e^{z_j} = m + \log \sum_j e^{z_j - m},$$

where every exponent is at most zero. Frameworks therefore compute the loss from logits: PyTorch’s CrossEntropyLoss expects unnormalized logits and is equivalent to LogSoftmax followed by NLLLoss; BCEWithLogitsLoss fuses the sigmoid into the loss and is described as more numerically stable than a sigmoid followed by BCELoss (PyTorch documentation).

Overconfidence and calibration. A classifier is calibrated when, of the images it labels with 80% confidence, about 80% are right. Guo, Pleiss, Sun and Weinberger (2017) found modern deep networks miscalibrated: on CIFAR-100 a 110-layer ResNet’s average confidence was well above its accuracy, while a 5-layer LeNet’s matched. The NLL can overfit while the error keeps falling; in one of their CIFAR-100 runs, test error dropped from 29% to 27% while the test NLL rose. Their remedy, temperature scaling, divides the logits by one scalar $T > 0$ fitted by minimizing the NLL on a validation set; it leaves the argmax, and so the accuracy, unchanged, and the authors found it often the most effective of the calibration methods they compared.

Label smoothing. Szegedy et al. (2016) replaced the one-hot target $\mathbf{q}$ with $q'_k = (1 - \varepsilon)\,\delta_{k,y} + \varepsilon / K$, so the loss becomes $(1 - \varepsilon) H(\mathbf{q}, \mathbf{p}) + \varepsilon H(\mathbf{u}, \mathbf{p})$ for the uniform distribution $\mathbf{u}$. Because every target class keeps some mass, the loss grows if the largest logit runs away from the others, which they argued limits overfitting and overconfidence. With $K = 1000$ and $\varepsilon = 0.1$ on ImageNet (ILSVRC 2012), the authors reported a consistent improvement of about 0.2% absolute in both top-1 and top-5 error. PyTorch exposes this as the label_smoothing argument.

Class imbalance. A weight $w_c$ per class multiplies each image’s loss; PyTorch’s weight argument does this, and its mean reduction then divides by the sum of the weights of the true classes rather than by the number of images (PyTorch documentation). The focal loss, which instead down-weights well-classified examples, is described in the loss function article.

Examples

Softmax classifier on digit images. A softmax classifier with no hidden layer (multinomial logistic regression) maps the 64 pixels of each $8 \times 8$ scikit-learn digit image to ten logits. The code checks the gradients against central finite differences, then trains the model by full-batch gradient descent with cross-entropy and with squared error on the softmax probabilities, each at its best step size from a small grid, from zero weights and from large random weights (logit standard deviation about 6) that saturate the softmax.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()                       # 1,797 grayscale 8x8 images, labels 0..9
X = digits.images.reshape(-1, 64) / 16.0     # one row of 64 pixel values in [0, 1] per image
y = digits.target
rng = np.random.default_rng(0)
idx = rng.permutation(len(y))
tr, te = idx[:1200], idx[1200:]              # 1,200 training images, 597 test images
Y = np.eye(10)[y]                            # one-hot labels

def log_softmax(Z):                          # stable: subtract the row maximum first
    Z = Z - Z.max(axis=1, keepdims=True)
    return Z - np.log(np.exp(Z).sum(axis=1, keepdims=True))

def cross_entropy(W, b, X, Y):
    """Mean cross-entropy over the images and its gradients."""
    logP = log_softmax(X @ W + b)
    dZ = (np.exp(logP) - Y) / len(X)         # gradient w.r.t. logits: softmax - one-hot
    return -(Y * logP).sum() / len(X), X.T @ dZ, dZ.sum(axis=0)

def squared_error(W, b, X, Y):
    """Mean squared error between softmax probabilities and one-hot labels."""
    P = np.exp(log_softmax(X @ W + b))
    G = (P - Y) / len(X)                     # gradient w.r.t. the probabilities
    dZ = P * (G - (G * P).sum(axis=1, keepdims=True))   # through the softmax Jacobian
    return 0.5 * ((P - Y) ** 2).sum() / len(X), X.T @ dZ, dZ.sum(axis=0)

# 1. Check the logit gradient of one image against central finite differences.
z, c = rng.normal(0, 2, 10), y[tr[0]]
loss = lambda z: -log_softmax(z[None])[0, c]
analytic = np.exp(log_softmax(z[None]))[0] - np.eye(10)[c]
numeric = np.array([(loss(z + e) - loss(z - e)) / 2e-6 for e in 1e-6 * np.eye(10)])
print(f"logit gradient check: max abs error {np.abs(analytic - numeric).max():.1e}")
# ... and the weight gradients of both losses at random weights.
W0, b0 = rng.normal(0, 0.1, (64, 10)), np.zeros(10)
for f in (cross_entropy, squared_error):
    g = f(W0, b0, X[tr], Y[tr])[1]
    errs = []
    for i, j in zip(rng.integers(0, 64, 5), rng.integers(0, 10, 5)):
        E = np.zeros_like(W0); E[i, j] = 1e-6
        num = (f(W0 + E, b0, X[tr], Y[tr])[0] - f(W0 - E, b0, X[tr], Y[tr])[0]) / 2e-6
        errs.append(abs(num - g[i, j]) / abs(num))
    print(f"{f.__name__:13s} weight gradient check: max relative error {max(errs):.1e}")

# 2. Train the same linear softmax classifier with each loss by full-batch gradient
#    descent, from zero weights and from large random weights (logit std about 6).
def train(f, W, lr, steps=300):
    W, b = W.copy(), np.zeros(10)
    for _ in range(steps):
        _, gW, gb = f(W, b, X[tr], Y[tr])
        W -= lr * gW; b -= lr * gb
    return W, b

def report(W, b, s):
    logP = log_softmax(X[s] @ W + b)
    acc = (logP.argmax(1) == y[s]).mean()
    wrong = (logP[np.arange(len(s)), y[s]] < np.log(0.01)).mean()  # true-class prob < 1%
    return acc, wrong

inits = {"zero init": np.zeros((64, 10)),
         "random init": np.random.default_rng(4).normal(0, 2, (64, 10))}
for name, W0 in inits.items():
    print(f"{name}: true class below 1% on {report(W0, 0, tr)[1]:.0%} of training images")
    for f in (cross_entropy, squared_error):
        # step size from a small grid, chosen by final training accuracy
        lr, (W, b) = max(((lr, train(f, W0, lr)) for lr in (3, 10, 30, 100)),
                         key=lambda r: report(*r[1], tr)[0])
        (a_tr, w_tr), (a_te, _) = report(W, b, tr), report(W, b, te)
        print(f"  {f.__name__:13s} (step size {lr:2d}): train acc {a_tr:.3f}, test acc {a_te:.3f}, "
              f"below 1% {w_tr:.0%}")

# 3. A confidently wrong prediction: the true class has logit 0, a wrong class logit 10.
z = np.zeros(10); z[3] = 10.0
for f in (cross_entropy, squared_error):
    # zero weights and bias z: the bias gradient is the gradient w.r.t. the logits
    gz = f(np.zeros((1, 10)), z, np.ones((1, 1)), np.eye(10)[[0]])[2]
    print(f"{f.__name__:13s} gradient on the true-class logit: {gz[0]:+.1e}")

Output:

logit gradient check: max abs error 5.1e-10
cross_entropy weight gradient check: max relative error 7.4e-08
squared_error weight gradient check: max relative error 6.3e-06
zero init: true class below 1% on 0% of training images
  cross_entropy (step size 30): train acc 0.998, test acc 0.960, below 1% 0%
  squared_error (step size 10): train acc 0.984, test acc 0.975, below 1% 0%
random init: true class below 1% on 77% of training images
  cross_entropy (step size 30): train acc 0.998, test acc 0.963, below 1% 0%
  squared_error (step size 30): train acc 0.779, test acc 0.750, below 1% 20%
cross_entropy gradient on the true-class logit: -1.0e+00
squared_error gradient on the true-class logit: -9.1e-05

From zero weights both losses train well; squared error even scores slightly higher on the test set, while unregularized cross-entropy fits the training images almost perfectly. From a saturated start, cross-entropy recovers the same accuracy, but squared error leaves 20% of the training images with less than 1% probability on the true class, nearly all of them 0s and 4s. The last two lines show why: for one confidently wrong prediction, the squared-error gradient on the true logit is about 10,000 times smaller than the cross-entropy gradient of $-1.0$. Over 20 random initializations of the same scale, cross-entropy reached 95.6–97.0% test accuracy and squared error 66.5–95.3% (median 78.3%).

Stable log-softmax. The same log_softmax, applied to the 32-bit logits of a confident classifier, next to the literal formula:

import numpy as np

def log_softmax(z):
    z = z - z.max(axis=-1, keepdims=True)    # shift so the largest logit is 0
    return z - np.log(np.exp(z).sum(axis=-1, keepdims=True))

z = np.array([100.0, 90.0, 0.0], dtype=np.float32)   # logits of a confident classifier
with np.errstate(all="ignore"):
    naive = np.log(np.exp(z) / np.exp(z).sum())
print("naive:", naive)
print("stable:", [f"{v:.5f}" for v in log_softmax(z)])

Output:

naive: [ nan  nan -inf]
stable: ['-0.00005', '-10.00005', '-100.00005']

$e^{100}$ overflows in 32-bit floats, so the literal formula divides infinity by infinity and takes the log of zero; the shifted version is exact to the printed precision.

The same losses in PyTorch. This snippet, run with PyTorch 2.14.1, passes raw logits for eight digit images to CrossEntropyLoss, with and without label smoothing, and the logits of a photograph labeled dog and cat to BCEWithLogitsLoss, and compares each with NumPy.

import numpy as np
import torch
from sklearn.datasets import load_digits

digits = load_digits()
X = digits.images[:8].reshape(8, 64) / 16.0          # eight digit images
y = digits.target[:8]
W = np.random.default_rng(0).normal(0, 0.5, (64, 10))
Z = X @ W                                              # raw logits of an untrained classifier

def log_softmax(Z):
    Z = Z - Z.max(axis=1, keepdims=True)
    return Z - np.log(np.exp(Z).sum(axis=1, keepdims=True))

logP = log_softmax(Z)
nll = -logP[np.arange(8), y]                           # -log p_y for each image
eps = 0.1                                              # label smoothing
ce_np = nll.mean()
ls_np = ((1 - eps) * nll + eps * -logP.mean(axis=1)).mean()

Zt, yt = torch.tensor(Z), torch.tensor(y)
ce_pt = torch.nn.CrossEntropyLoss()(Zt, yt).item()     # takes raw logits, not probabilities
ls_pt = torch.nn.CrossEntropyLoss(label_smoothing=eps)(Zt, yt).item()
print(f"cross-entropy    NumPy {ce_np:.6f}  PyTorch {ce_pt:.6f}")
print(f"label smoothing  NumPy {ls_np:.6f}  PyTorch {ls_pt:.6f}")

# Multi-label: independent labels (dog, cat, car) for one photo showing a dog and a cat.
z, t = np.array([2.0, 0.5, -3.0]), np.array([1.0, 1.0, 0.0])
# -log sigmoid(z) = log(1 + e^-z) and -log(1 - sigmoid(z)) = log(1 + e^z)
bce_np = np.mean(t * np.logaddexp(0, -z) + (1 - t) * np.logaddexp(0, z))
bce_pt = torch.nn.BCEWithLogitsLoss()(torch.tensor(z), torch.tensor(t)).item()
print(f"binary CE        NumPy {bce_np:.6f}  PyTorch {bce_pt:.6f}")
print("label probabilities:", torch.sigmoid(torch.tensor(z)).numpy().round(3))

Output:

cross-entropy    NumPy 4.355670  PyTorch 4.355670
label smoothing  NumPy 4.351303  PyTorch 4.351303
binary CE        NumPy 0.216531  PyTorch 0.216531
label probabilities: [0.881 0.622 0.047]

The values agree: PyTorch applies the log-softmax itself, and its label smoothing is the uniform mixture above. The multi-label probabilities need not sum to one.

Common Misconceptions

• “The loss takes probabilities.” CrossEntropyLoss and BCEWithLogitsLoss take logits. Applying a softmax first feeds values in $[0, 1]$ to the loss as logits; by our reasoning, its internal softmax can then put at most $e/(e + K - 1)$ on any class, 0.23 for ten classes, so the loss never falls below 1.46.
• “Low loss means trustworthy probabilities.” Networks trained with cross-entropy can be accurate and overconfident at once (Guo et al., 2017); calibration has to be measured, and temperature scaling is a cheap fix.

Where It Is Used

Cross-entropy trains image classifiers, from the softmax regression above to deep neural networks with nonlinear activation functions. Averaged over pixels it trains semantic segmentation networks, and it scores the classification branches of object detectors, alone or reweighted as the focal loss (loss function).