Label Smoothing

Label smoothing trains a classifier on a softened target: instead of putting all the probability on the labeled class, the target keeps $1 - \varepsilon$ there and spreads $\varepsilon$ uniformly over all $K$ classes. The cross-entropy loss against such a target has a finite minimum, so the network is no longer rewarded for pushing the true-class logit ever further above the others. Szegedy et al. (2016) introduced it with Inception-v3 as a regularizer for ImageNet classification, and it became common in image classification and transformer training recipes. Later work found that it also changes calibration, robustness to label noise, the transferability of features, and the usefulness of a network as a distillation teacher, not always for the better.

Purpose

With a one-hot target, the cross-entropy $-\log p_y$ reaches its infimum of zero only as the true-class logit $z_y$ grows infinitely larger than every other logit, a limit that no finite weights reach. Szegedy et al. (2016, §7) named two problems with chasing that limit: the model may overfit, since assigning full probability to every training label does not guarantee generalization, and the growing differences between the largest logit and the others, combined with the bounded gradient, reduce the model’s ability to adapt, which they put down to the model becoming too confident about its predictions. They proposed label smoothing as “a mechanism for encouraging the model to be less confident”, accepting that it lowers the log-likelihood of the training labels.

Their framing was a label-dropout argument: the smoothed target is the distribution of a label obtained by keeping the ground-truth label and, with probability $\varepsilon$, replacing it with a draw from a fixed distribution $u$. Training on the smoothed target estimates the marginalized effect of that random replacement without sampling it.

When to Use It

• Classifiers trained to convergence on single labels. Label smoothing is a cheap change to the loss of a softmax classifier and has a long record in ImageNet recipes (Evidence and Impact).
• Labels with noise. Lukasik et al. (2020) found that smoothing improved test accuracy when training labels were corrupted with symmetric noise, on CIFAR and, for $\varepsilon$ up to 0.4, on ImageNet.
• When calibrated confidence matters. Moderate smoothing often lowers the expected calibration error, but larger values can raise it again (Pitfalls).

It is a poor fit when:

• The penultimate features will be reused as fixed features for other tasks; Kornblith et al. (2019) found that label smoothing during ImageNet training made such features worse for transfer learning.
• The network will teach a student by distillation at a high temperature (Chandrasegaran et al., 2022).
• The targets are already soft, as in distillation from a teacher’s predictions; DeiT smooths true labels but not the pseudo-labels from its teacher (Touvron et al., 2021).

How It Works

Definition

For an image with label $y$, the network outputs logits $\mathbf{z}$ and probabilities $p_k = e^{z_k} / \sum_j e^{z_j}$. Szegedy et al. (2016) replace the one-hot target $q(k) = \delta_{k,y}$ by

$$q'(k) = (1 - \varepsilon)\,\delta_{k,y} + \varepsilon\, u(k),$$

a mixture of the label and a fixed distribution $u$. They proposed the prior distribution over labels as $u$ and used the uniform distribution $u(k) = 1/K$, which gives

$$q'(k) = (1 - \varepsilon)\,\delta_{k,y} + \frac{\varepsilon}{K},$$

and called the change label-smoothing regularization (LSR). The true class gets $1 - \varepsilon + \varepsilon/K$ and every other class $\varepsilon/K$. On ImageNet they used $K = 1000$ and $\varepsilon = 0.1$. PyTorch’s label_smoothing argument of CrossEntropyLoss computes this same uniform mixture, as the cross-entropy article checks numerically.

Other papers parameterize it differently. He, T. et al. (2019) give the true class $1 - \varepsilon$ and each other class $\varepsilon/(K - 1)$, and DeiT describes its smoothing the same way: the true label has probability $1 - \varepsilon$ and $\varepsilon$ is shared across the remaining classes (Touvron et al., 2021). For the same $\varepsilon$ the two forms differ slightly (for $K = 1000$ and $\varepsilon = 0.1$, the true-class target is 0.9001 in one and 0.9 in the other), so a reported $\varepsilon$ should be read with its form.

The equivalent penalty against a uniform prior

Because the cross-entropy is linear in the target,

$$H(q', p) = -\sum_{k} q'(k) \log p_k = (1 - \varepsilon)\, H(q, p) + \varepsilon\, H(u, p).$$

Szegedy et al. (2016) read this as replacing one loss by a pair: the usual cross-entropy with the label, and a second loss that penalizes the deviation of the prediction $p$ from the prior $u$, with relative weight $\varepsilon / (1 - \varepsilon)$. Since $H(u, p) = D_{\mathrm{KL}}(u \Vert p) + H(u)$ and $H(u)$ is fixed, the second term is equivalently a KL divergence from the uniform distribution to the prediction. They noted that the dissimilarity of $p$ from uniform could also be measured, though not equivalently, by the negative entropy $-H(p)$, which they did not try. Liu, Ben Ayed, Galdran and Dolz (2022) write the same decomposition as $\mathcal{L}_{\mathrm{CE}} + \tfrac{\varepsilon}{1 - \varepsilon} D_{\mathrm{KL}}(u \Vert s)$ up to constants (their Eq. 4, with $\alpha$ for $\varepsilon$ and $s$ for the softmax output).

For uniform $u$, the extra term is $H(u, p) = \log \sum_j e^{z_j} - \frac{1}{K} \sum_j z_j$: the log-sum-exp minus the mean logit. Liu et al. (2022, Proposition 1) show that this term is bounded above and below, up to constants, by the average distance between the largest logit and the others, $\frac{1}{K}\sum_k (\max_j z_j - z_k)$. Label smoothing is thus approximately a linear penalty on logit distances that pulls all logits toward each other, balanced against the cross-entropy term that pushes the true logit up.

Gradient

The gradient of the cross-entropy with respect to the logits is the prediction minus the target (cross-entropy article):

$$\frac{\partial H(q', p)}{\partial z_k} = p_k - q'(k).$$

With a one-hot target this is zero only in the limit $p_y \to 1$; with smoothing it is zero at $p = q'$, a point finite logits reach. Once the true-class probability exceeds $1 - \varepsilon + \varepsilon/K$, the gradient on its logit changes sign and pushes it back down.

The bounded logit gap

At the minimum, $p = q'$, and the ratio of two softmax probabilities is the exponential of the difference of their logits. So every wrong class has the same logit, and the true-class logit exceeds each of them by

$$g^*(\varepsilon) = \log \frac{1 - \varepsilon + \varepsilon/K}{\varepsilon/K} = \log\!\left(1 + \frac{K (1 - \varepsilon)}{\varepsilon}\right).$$

The gap is determined only up to a constant added to all logits, which the softmax ignores. It is infinite at $\varepsilon = 0$, shrinks as $\varepsilon$ grows, and is zero at $\varepsilon = 1$, where the target is uniform. In the $\varepsilon/(K-1)$ form, He, T. et al. (2019, Eq. 5) give the gap as $\log\big((K - 1)(1 - \varepsilon)/\varepsilon\big)$, zero at $\varepsilon = (K-1)/K$; for $K = 1000$ and $\varepsilon = 0.1$ both forms give about 9.1. They compared two ResNet-50-D networks trained with and without smoothing, computing for each ImageNet validation image the gap between the maximum logit and the average of the rest. With smoothing, the distribution of the gap centered on the theoretical value and had fewer extreme values (their Figure 4b).

Geometry of the penultimate layer

Müller, Kornblith and Hinton (2019), which could not be read for this article, studied what smoothing does to the features before the classifier. As Chandrasegaran et al. (2022) summarize their analysis: a logit can be approximately measured by the squared distance between the penultimate activations and a template for the class, so smoothing encourages each training example’s activations to be close to the template of its class and equidistant from the templates of all other classes. The activations of each class form tight clusters, and information about how an example resembles other classes is erased from the logits. Lukasik et al. (2020), using the same visualization on CIFAR-100 with label noise, found that smoothing made the mislabeled examples’ activations more uniform, so the model was less confident in their wrong labels.

Lukasik et al. (2020) write the smoothed loss as the ordinary risk plus a data-dependent regularizer that does not depend on the labels. For a linear softmax classifier, that regularizer is minimized by all-zero weights (their Theorem 1), so it shrinks the weights toward zero, much as an $\ell_2$ penalty such as weight decay does; for least squares on centered data they show that smoothing simply shrinks the solution.

How to Apply It

In practice it is one argument: PyTorch’s CrossEntropyLoss(label_smoothing=0.1) replaces the one-hot target by the uniform mixture above. The value 0.1 from Inception-v3 is the one used in the recipes under Best Practices.

Example: the logit gap of a smoothed classifier

The example trains a small stand-in for a network: a softmax classifier with no hidden layer (logistic regression) on the 256 pixels of $16 \times 16$ images of four classes (a horizontal bar, a vertical bar, a diagonal and an anti-diagonal band, with Gaussian pixel noise of standard deviation 1.75), with each of the 200 training labels moved to another class at random with probability 0.15 (11% were moved in this draw, as the first output line shows). With 257 weights per class (256 pixels and a bias) and 200 images, the model can fit any logits on the training set, as an over-parameterized network can. It is trained by full-batch gradient descent from zero weights with one-hot targets and with smoothed targets at $\varepsilon = 0.1$, 0.2 and 0.4, and the code records the training gap (the logit of each image’s label minus its largest other logit) as training runs. It also computes the exact minimizer of the smoothed loss and compares both with $g^*(\varepsilon)$.

The exact minimizer follows from two properties of gradient descent from zero here, which hold by construction: every gradient row $\mathbf{p} - \mathbf{q}'$ sums to zero over the classes, so the logits of any image stay centered, and every update is a combination of training images, so the weights stay in their span. The smoothed loss is minimized when each training image’s centered logits equal the centered $\log \mathbf{q}'$, and the minimum-norm weights that fit those logits are given by the pseudo-inverse of the design matrix.

import numpy as np

# Four classes of 16x16 images: horizontal bar, vertical bar, diagonal, anti-diagonal.
S, K = 16, 4
r, c = np.mgrid[0:S, 0:S]
box = (r >= 2) & (r <= 13) & (c >= 2) & (c <= 13)
templates = np.stack([(np.abs(r - 7.5) <= 1) & box, (np.abs(c - 7.5) <= 1) & box,
                      (np.abs(r - c) <= 1) & box, (np.abs(r + c - 15) <= 1) & box]).astype(float)

rng = np.random.default_rng(0)
n, sigma, rho = 200, 1.75, 0.15
y_clean = rng.integers(0, K, n)
X = templates[y_clean] + rng.normal(0, sigma, (n, S, S))     # Gaussian pixel noise
flip = rng.random(n) < rho                                    # 15% of labels moved to
y = np.where(flip, (y_clean + rng.integers(1, K, n)) % K, y_clean)  # another class
A = np.c_[X.reshape(n, -1), np.ones(n)]                       # 256 pixels + bias per image
print(f"{n} images, {A.shape[1]} weights per class, rank {np.linalg.matrix_rank(A)}, "
      f"{flip.mean():.0%} of labels flipped")

def softmax(Z):
    Z = Z - Z.max(axis=1, keepdims=True)
    E = np.exp(Z)
    return E / E.sum(axis=1, keepdims=True)

def gaps(W):
    """Logit of the training label minus each other logit, per image (n x K-1)."""
    Z = A @ W
    D = Z[np.arange(n), y][:, None] - Z
    return D[np.arange(K)[None] != y[:, None]].reshape(n, K - 1)

lr = 1 / (0.5 * np.linalg.eigvalsh(A.T @ A / n).max())      # 1 / smoothness bound
checkpoints = [10, 100, 1_000, 10_000, 100_000]
W_star = {}
for eps in (0.0, 0.1, 0.2, 0.4):
    Q = (1 - eps) * np.eye(K)[y] + eps / K                    # smoothed targets
    W, t = np.zeros((A.shape[1], K)), 0
    row = []
    for T in checkpoints:
        while t < T:                                          # full-batch gradient descent
            W -= lr * A.T @ (softmax(A @ W) - Q) / n
            t += 1
        row.append(np.median(gaps(W).min(axis=1)))
    print(f"eps {eps:.1f}: median gap at steps {checkpoints}: " + " ".join(f"{g:6.2f}" for g in row))
    if eps > 0:
        g_star = np.log((1 - eps + eps / K) / (eps / K))
        Zc = np.log(Q) - np.log(Q).mean(axis=1, keepdims=True)  # centred target logits
        W_star[eps] = np.linalg.pinv(A) @ Zc                    # minimum-norm exact fit
        print(f"         g* = {g_star:.4f}; max |gap - g*|: exact {np.abs(gaps(W_star[eps]) - g_star).max():.0e},"
              f" gradient descent {np.abs(gaps(W) - g_star).max():.0e};"
              f" ||W - W*|| / ||W*|| = {np.linalg.norm(W - W_star[eps]) / np.linalg.norm(W_star[eps]):.0e}")

ref = W_star[0.1] / np.linalg.norm(W_star[0.1])
for eps, Ws in W_star.items():
    g_star = np.log((1 - eps + eps / K) / (eps / K))
    print(f"W*({eps}): 1 - cos to W*(0.1) = {1 - (Ws * ref).sum() / np.linalg.norm(Ws):.0e}, "
          f"||W*|| / g* = {np.linalg.norm(Ws) / g_star:.6f}")

Output (NumPy 2.3.5; about 20 seconds):

200 images, 257 weights per class, rank 200, 11% of labels flipped
eps 0.0: median gap at steps [10, 100, 1000, 10000, 100000]:   2.14   4.20   6.38   8.64  10.90
eps 0.1: median gap at steps [10, 100, 1000, 10000, 100000]:   1.89   2.93   3.35   3.59   3.61
         g* = 3.6109; max |gap - g*|: exact 2e-14, gradient descent 3e-09; ||W - W*|| / ||W*|| = 1e-09
eps 0.2: median gap at steps [10, 100, 1000, 10000, 100000]:   1.65   2.35   2.70   2.83   2.83
         g* = 2.8332; max |gap - g*|: exact 2e-14, gradient descent 6e-14; ||W - W*|| / ||W*|| = 8e-13
eps 0.4: median gap at steps [10, 100, 1000, 10000, 100000]:   1.21   1.68   1.90   1.95   1.95
         g* = 1.9459; max |gap - g*|: exact 1e-14, gradient descent 2e-14; ||W - W*|| / ||W*|| = 2e-13
W*(0.1): 1 - cos to W*(0.1) = 2e-16, ||W*|| / g* = 0.715278
W*(0.2): 1 - cos to W*(0.1) = 0e+00, ||W*|| / g* = 0.715278
W*(0.4): 1 - cos to W*(0.1) = 0e+00, ||W*|| / g* = 0.715278

With one-hot targets the gap never settles: it grows by a little over 2 per tenfold increase in steps, mislabeled images included. This is the behavior Soudry et al. (2018, Theorem 7) proved for cross-entropy with a softmax on linearly separable data: gradient descent’s weights grow like $\hat{\mathbf{w}} \log t$, where $\hat{\mathbf{w}}$ is the multi-class hard-margin support vector machine solution, so the loss keeps falling only by inflating the margins, and in our reading every training gap eventually grows by at least $\ln 10 \approx 2.30$ per decade, close to what the output shows by the last decade. With smoothing, every training image’s gap converges to $g^*(\varepsilon)$, to rounding error at the pseudo-inverse solution and to within $3 \times 10^{-9}$ by gradient descent, including the images whose labels are wrong: the smoothed loss fits the wrong labels too, only with bounded confidence. Larger $\varepsilon$ converges faster.

The last three lines show a consequence of the construction. The centered log-target of each image is $g^*(\varepsilon)\,(\mathbf{e}_y - \tfrac{1}{K}\mathbf{1})$, so the exact solutions for different $\varepsilon$ are one weight matrix scaled by $g^*(\varepsilon)$ (the printed cosines and the ratio $\lVert W^* \rVert / g^*$, equal to the six decimals shown, agree with this): the same predictor, with $\varepsilon$ acting only as an inverse temperature on its logits. In a model that fits its training set exactly, smoothing chooses a scale rather than a direction (our reasoning; in a network the features themselves change, as described above).

The plan for this example also included measures on held-out images: accuracy, mean confidence, expected calibration error and negative log-likelihood, with the exact Bayes posterior as a reference. Their planned base check, that the exact smoothed solution classify held-out images nearly as well as the Bayes classifier, failed in the pilot: the small linear stand-in could not learn the base task at this scale, an error in the plan. By editor decision before the first run, the held-out measures were dropped, and the proportionality check in the last three lines was added, with its threshold fixed from a saved computation before its single pilot. Calibration is discussed from the published studies below.

Variants

• A non-uniform prior. Szegedy et al. (2016) proposed the prior distribution over labels as $u$; uniform $u$ is the special case they used. The term $H(u, p)$ then pulls predictions toward the class frequencies.
• Explicit confidence penalty. Adding the negative entropy $-H(p)$ of the prediction to the loss penalizes confident outputs directly; Szegedy et al. mentioned it as an alternative they did not try, and Liu et al. (2022) include it as a baseline (ECP), which they attribute to Pereyra et al. (2017).
• Margin-based label smoothing (MbLS). Liu et al. (2022) argue that smoothing, the confidence penalty and the focal loss all impose the same equality constraint, that every logit distance from the maximum be zero, and that the resulting gradient constantly pushes toward a non-informative solution. They replace it by a hinge penalty $\lambda \sum_k \max(0, \max_j z_j - z_k - m)$ that acts only on distances above a margin $m$, with $\lambda = 0.1$ and $m$ chosen on validation sets (6 or 10 in their experiments).
• Label smearing. Lukasik et al. (2020) write label smoothing, standard training and the backward correction for label noise as one family: the loss vector is multiplied by a matrix $M$, which is the identity for standard training and $(1-\varepsilon) I + \tfrac{\varepsilon}{K} J$ for smoothing ($J$ the all-ones matrix). That smoothing matrix is exactly the noise matrix of symmetric label noise (in our reading of their Table 1, where backward correction for such noise is its inverse) that flips a label with probability $\rho = \varepsilon (K-1)/K$, and backward correction multiplies by its inverse. Forward correction instead applies the noise matrix to the predicted probabilities.
• Soft targets from a model. In knowledge distillation the student is trained toward a teacher’s softened predictions. Yuan et al. (2020) prove that distillation is a learned form of label smoothing and that smoothing provides a virtual teacher; in their teacher-free framework the student learns from itself or from a manually designed distribution, which improved ImageNet accuracy by up to 0.65% over their baselines. Mixup, which mixes two images and their one-hot labels with the same weight (He, T. et al., 2019, Eqs. 7 and 8), and CutMix also train on soft targets.

Best Practices

• Start at $\varepsilon = 0.1$. Szegedy et al. (2016) used it on ImageNet, He, T. et al. (2019) followed them, and DeiT fixed it for all its experiments with true labels (Touvron et al., 2021). Liu et al. (2022) used 0.05 for CIFAR-10 and Tiny-ImageNet, following earlier calibration studies.
• Report the form. State whether $\varepsilon$ is spread over all $K$ classes or the $K - 1$ wrong ones.
• Measure calibration rather than assume it, with expected calibration error and the negative log-likelihood, since the effect depends on $\varepsilon$ and the data.
• Decide how the features will be used. For a backbone whose fixed features will be probed, Kornblith et al.’s (2019) results argue against smoothing; with fine-tuning they found training settings mattered much less.
• Smooth hard labels only. DeiT does not smooth the teacher’s pseudo-labels in hard distillation (Touvron et al., 2021).

Pitfalls

• Worse fixed features for transfer. Kornblith, Shlens and Le (2019) identified four choices in the Inception training procedure, adopted by several other models, that hurt transfer accuracy: no batch-normalization scale parameter, label smoothing, dropout and an auxiliary classifier head. These changed ImageNet top-1 accuracy by under 1%, yet for logistic regression on penultimate-layer features over 12 datasets, the difference between the default and the best settings was about as large as that between the worst and best ImageNet architectures. With fine-tuning the effects were smaller, and disabling label smoothing again improved performance (their Figure 5). In our reading, this is the cluster tightening described above: features shaped to be equidistant from every wrong class carry less information about similarities between classes.
• Dense prediction from a smoothed backbone. He, T. et al. (2019, Table 9) found that adding label smoothing to ImageNet pretraining lowered the ADE20K results of an FCN built on the backbone, from 39.33 to 38.75 mIoU, although the same backbone improved Faster R-CNN on PASCAL VOC 2007 test from 79.23 to 80.71 mAP (Table 8). They suggest that soft labels blur pixel-level information.
• Weaker distillation teachers. Müller et al. (2019) reported that a teacher trained with label smoothing distilled worse into a student, which they attributed to the information erased from the logits (as reported by Chandrasegaran et al., 2022). Lukasik et al. (2020) found the opposite under label noise: smoothing the teacher improved the student on CIFAR-10 and CIFAR-100. Shen et al. (2021), as Chandrasegaran et al. report, argued that smoothing is compatible with distillation, and Chandrasegaran et al. (2022) reconciled the two positions: with a smoothed teacher, distillation at higher temperatures diffuses the student’s features toward semantically similar classes, so they recommend a smoothed teacher with a low temperature ($T = 1$).
• Worse calibration at larger $\varepsilon$. In Lukasik et al.’s (2020) label-noise setting (ResNet-32 on CIFAR-100, ECE on 100 bins, their Table 5), smoothing lowered the expected calibration error slightly at $\varepsilon = 0.1$, from 0.111 to 0.108, an effect they note Müller et al. also observed, but raised it to 0.156 at $\varepsilon = 0.2$, the largest value in the table. Liu et al. (2022, Table 4) saw the same trend without label noise, for a ResNet-50 on Tiny-ImageNet: ECE 3.73% without smoothing, 3.17% at $\varepsilon = 0.05$, then 6.53%, 12.05% and 18.04% at 0.1, 0.2 and 0.3.
• Smoothing does not cure the cause. The example shows that smoothing fits wrong labels with bounded confidence rather than ignoring them, and that it only rescales the solution of a model that interpolates its training set.

Evidence and Impact

Inception-v3. In Szegedy et al.’s (2016) cumulative Table 3 (single crop, 48,238 ImageNet validation images), adding label smoothing to the network trained with RMSProp lowered top-1 error from 23.1% to 22.8% and top-5 error from 6.3% to 6.1%. The text of §7 describes this as a consistent improvement of about 0.2% absolute in both top-1 and top-5 error; the table’s top-1 difference is 0.3%. The Inception article places the change among the paper’s others.

ImageNet recipes. He, T. et al. (2019, Table 6) stacked refinements on ImageNet. Adding label smoothing after the cosine learning-rate schedule raised top-1 accuracy from 77.73% to 78.37% for ResNet-50-D (means over four runs, standard deviations 0.13 and 0.09), from 78.19% to 78.40% for Inception-V3, and from 72.83% to 72.93% for MobileNet. In Kornblith et al.’s (2019) supplementary experiments, an Inception-ResNet v2 retrained with label smoothing, dropout and the auxiliary head reached 81.4% top-1, 1.1% above the same model without them. DeiT used $\varepsilon = 0.1$ for its vision transformers (Touvron et al., 2021), the original transformer for translation also used 0.1, as the transformer article records, and the Vision Transformer authors tuned label smoothing with weight decay and dropout when pretraining on ImageNet (ViT paper). In person re-identification, Luo et al.’s strong baseline, as the person re-identification article reports, gained 0.1 points of rank-1 accuracy and 1.0 point of mAP on Market-1501 from label smoothing at $\varepsilon = 0.1$.

Label noise. Lukasik et al. (2020) injected 20% symmetric noise into the training labels (not the test labels). On ImageNet with ResNet-v2-50, clean test accuracy was 70.86% without smoothing and 71.12%, 71.55%, 70.95% and 70.59% at $\varepsilon = 0.1$, 0.2, 0.4 and 0.6 (their Table 2): a gain up to 0.4 and a small loss at 0.6, although the table’s caption says that smoothing improves over the baseline. Forward correction did better at every value. On CIFAR-10 and CIFAR-100 with a ResNet-32, smoothing improved significantly over the baseline, and values of $\varepsilon$ well above the noise rate improved results further (their Figure 2).

Calibration. Lukasik et al. (2020) and Liu et al. (2022) both cite Müller et al. (2019) for the finding that label smoothing improves calibration. In Liu et al.’s experiments with 15-bin ECE, smoothing at $\varepsilon = 0.05$ lowered the ECE of a ResNet-50 on CIFAR-10 from 5.85% to 2.79%, with accuracy 94.87% against 93.20%, and of a ResNet-101 on Tiny-ImageNet from 4.97% to 2.20% (their Tables 1 and 2). Their margin-based variant was lower still in nearly all their settings. The expected calibration error article sets smoothing beside mixup among the training-time changes that affect calibration, and compares it with post-hoc temperature scaling.

In sum, label smoothing is a small change to the targets with a well-understood effect on the training objective, a finite gap between the true-class logit and the others, and reported effects that depend on what the network is used for: modest accuracy gains on ImageNet classification, robustness to noisy labels and often better calibration, against poorer fixed features and, at high temperatures, weaker distillation.