Knowledge Distillation

Knowledge distillation trains one network, the student, to reproduce the outputs of another, the teacher, instead of, or as well as, learning from the training labels. In its common form the teacher is a large, accurate classifier or an ensemble, the student is smaller, and the student’s cross-entropy loss is computed against the teacher’s softmax probabilities softened by a temperature. The student can be any architecture, so distillation is one of the main routes of model compression, and it is also used to train students as large as their teachers, to move a convolutional network’s behaviour into a vision transformer, and, in a different form, to train self-supervised models such as DINO.

Training a small model to mimic a large one is older than the name. Bucilă, Caruana and Niculescu-Mizil (2006) compressed an ensemble into a single neural network, as Furlanello et al. (2018) and Park et al. (2019) recount. The name knowledge distillation goes with Hinton, Vinyals and Dean (2015), the framework that Cho and Hariharan and Beyer et al. (2022) say they follow. Neither the 2006 nor the 2015 paper was read for this article; the method is described here from the readable papers that restate and extend it.

Purpose

A network trained on one-hot labels learns from one bit of class identity per image, although the trained network itself outputs a full distribution over classes. The papers that restate distillation give two reasons why that distribution is a better training target. Cho and Hariharan (2019) state the classic one: the teacher’s soft probabilities contain more information about an image than its label, for instance that an image given high probability for several classes lies near the boundary between them, and a student made to mimic them should absorb some of what the teacher learned beyond the labels. Furlanello et al. (2018) call the information in the probabilities of the wrong classes dark knowledge.

The practical aim is usually cheaper inference. Beyer et al. (2022) point out that large models set the benchmark results while practitioners deploy models such as ResNet-50 or MobileNet that are orders of magnitude cheaper to run, and argue that distillation, unlike pruning, allows the model family to change during compression, from a ResNet to a MobileNet for instance.

When to Use It

• A strong model exists and a cheaper one must be deployed. This is the compression setting of the model compression article, where distillation sits beside pruning, quantization and factorization and can be combined with them, since it changes only the training targets.
• Few labelled images, but a teacher and more images. Beyer et al. (2022) distilled on datasets as small as Flowers102 (1,020 training images) for very long schedules with aggressive augmentation, and their students substantially outperformed a ResNet-50 trained from scratch on the labels; Menon et al. (2021) show that targets equal to the true class probabilities lower the variance of the training objective, which matters most when samples are few.
• A student of another architecture should inherit a teacher’s bias. DeiT trained a vision transformer with a convolutional teacher on ImageNet alone and found convolutional teachers better than a transformer teacher of similar accuracy, which the authors attribute probably to the inductive bias inherited through distillation (Touvron et al., 2021).
• The same architecture, retrained. Born-again networks (Furlanello et al., 2018) and self-distillation (Zhang et al., 2019) improved students that were no smaller than their teachers.

It is a poor fit, or needs care, when:

• The teacher is much larger than the student. Cho and Hariharan (2019) found that more accurate, larger teachers often made worse students (Pitfalls).
• The teacher was trained with label smoothing, which can make it a weaker teacher at high temperature, as the label smoothing article reports from Chandrasegaran et al. (2022).
• The teacher’s predictions are computed once and stored. Beyer et al. (2022) found this “fixed teacher” setting overfits on small datasets and plateaus below consistent teaching.

How It Works

Soft targets and temperature

For an image $x$, let the teacher produce logits $z_k(x)$ and the student logits $s_k(x)$ over $K$ classes. Cho and Hariharan (2019) soften the teacher’s probabilities with a temperature $\tau > 1$, noting that trained networks produce peaky distributions that may be less informative:

$$\tilde p^{t}_k(x) = \frac{e^{z_k(x)/\tau}}{\sum_j e^{z_j(x)/\tau}}, \qquad \tilde p^{s}_k(x) = \frac{e^{s_k(x)/\tau}}{\sum_j e^{s_j(x)/\tau}}.$$

The student’s loss is a weighted sum of the ordinary cross-entropy $\mathcal{L}_{\mathrm{cls}}$ with the labels at $\tau = 1$ and a distillation loss between the two softened distributions (their Section 3):

$$\mathcal{L} = \alpha\, \mathcal{L}_{\mathrm{cls}} + (1 - \alpha)\, \mathcal{L}_{\mathrm{KD}}, \qquad \mathcal{L}_{\mathrm{KD}} = -\tau^2 \sum_k \tilde p^{t}_k(x) \log \tilde p^{s}_k(x).$$

They name $\tau \in \{3, 4, 5\}$ and $\alpha = 0.9$ as popular choices. DeiT writes the same objective with the Kullback–Leibler divergence in place of the cross-entropy and its weight $\lambda$ on the distillation term (Touvron et al., 2021, Eq. 2); the two differ by the teacher’s entropy, which does not depend on the student. Beyer et al. (2022) use only the KL divergence to the tempered teacher, with no term for the labels. After training, the student is used at $\tau = 1$, as in the example below.

The temperature does not change which class the teacher ranks first, only how much probability the other classes receive. At $\tau = 1$ a confident teacher gives the wrong classes probabilities close to zero, and the cross-entropy barely depends on them; raising $\tau$ lifts them so that the student is also trained on how the teacher ranks the classes it does not choose. The softmax article shows how a distribution flattens as the temperature grows.

The $T^2$ factor

The gradient of the distillation term with respect to a student logit follows from the gradient of the cross-entropy, prediction minus target, and the chain rule through $s/\tau$:

$$\frac{\partial}{\partial s_k}\Big(-\sum_j \tilde p^{t}_j \log \tilde p^{s}_j\Big) = \frac{1}{\tau}\big(\tilde p^{s}_k - \tilde p^{t}_k\big).$$

As $\tau$ grows, both softened distributions approach uniform, and to first order in $1/\tau$, $\tilde p_k \approx \big(1 + (s_k - \bar s)/\tau\big)/K$, where $\bar s$ is the mean logit. Substituting gives

$$\frac{1}{\tau}\big(\tilde p^{s}_k - \tilde p^{t}_k\big) \approx \frac{1}{\tau^2}\,\frac{(s_k - \bar s) - (z_k - \bar z)}{K}.$$

So the gradient of the soft term shrinks as $1/\tau^2$, and multiplying the term by $\tau^2$, as Cho and Hariharan’s and DeiT’s losses do, keeps its size roughly constant as the temperature changes, and keeps its balance against the label term and any weight decay (this derivation is ours; the model compression article gives the same reason). The approximation also shows what distillation becomes at high temperature: the gradient of matching the teacher’s mean-centred logits by squared error, close to the logit matching of Ba and Caruana, which Furlanello et al. (2018) describe as an L2 penalty on the difference between student and teacher logits. The example below checks both statements numerically.

What the soft targets carry

Three accounts of why soft targets help are in the readable literature, and they are compatible.

• Dark knowledge. Furlanello et al. (2018, Eqs. 5–8) split the gradient of the distillation loss into a term from the wrong classes and a term from the true class, which equals the one-hot gradient with each image weighted by the teacher’s confidence in the true class. To separate the two, they trained students with only the confidence weighting (CWTM) and with the teacher’s wrong-class probabilities randomly permuted (DKPP). On CIFAR-100, DKPP students improved over their teachers in all four DenseNets of their Table 2 and CWTM students in two of the four (the text says three of four), both less than full distillation, so they conclude that the gain is not explained only by the information on each wrong class.
• Lower variance. Menon et al. (2021) note that the population risk already averages the loss over labels with the true class probabilities $p^*(y \mid x)$. A “Bayes teacher” that supplies $p^*$ gives an empirical risk with the same expectation as the one-hot empirical risk but no larger variance (their Lemma 1), which yields a tighter generalization bound. For a teacher that only approximates $p^*$, they prove a bias–variance trade-off (Proposition 3): what makes a teacher good is the quality of its probability estimates, measured for instance by log-loss or calibration, for which accuracy alone does not suffice.
• Regularization. Yuan et al. (2020) prove that distillation is a learned form of label smoothing, and that smoothing provides a virtual teacher. They found that a poorly trained teacher, and even a student teaching its teacher, still improved the trained model, and argue that the regularizing effect of soft targets matters as much as, or more than, the similarity information between categories.

How to Apply It

The usual recipe has four choices: the teacher, the loss (soft, hard-label or both, with $\alpha$ and $\tau$), the images the two networks see, and the training length. The papers below give working values: $\tau = 4$ and $\alpha = 0.9$ for ResNet, Wide ResNet and DenseNet students on CIFAR-10 and ResNet students on ImageNet (Cho and Hariharan, 2019), $\tau$ swept over $\{1, 2, 5, 10\}$ with no label term (Beyer et al., 2022), and DeiT’s hard-label variant with equal weights on the label and the teacher’s prediction (Touvron et al., 2021).

Example: distilling an exact teacher into small students

The example makes the statistical account concrete with a teacher whose probabilities are exactly the class probabilities, the setting Menon et al. (2021, §3.2) illustrate with two Gaussian classes in ten dimensions. Here the data are images: ten classes of $8 \times 8$ grayscale images, each class’s template being the mean image of one digit in scikit-learn’s bundled digits dataset (scaled to $[0, 1]$), plus Gaussian pixel noise of standard deviation 0.4, with all classes equally likely. Because the noise is Gaussian with the same covariance for every class, the exact Bayes posterior is a softmax of logits that are linear in the pixels, $z_k(x) = (\mu_k^\top x - \lVert \mu_k \rVert^2/2)/\sigma^2$; it is the teacher.

The students are small stand-ins for networks, logistic regression models (a convex fit):

• Student A sees all 64 pixels, so it can represent the teacher exactly.
• Student B sees the image averaged over $2 \times 2$ blocks, $4 \times 4$ values, as Beyer et al. (2022, §3.3) distil into a student that sees a lower-resolution view of the same crop. It cannot represent the teacher. The best it can do is the Bayes posterior given the pooled image, which is also linear-softmax and is computed exactly: the student’s ceiling.

Each student is trained on $n$ images drawn afresh for each of five seeds, either from the hard labels or from the teacher’s soft targets at $T \in \{1, 2, 5, 10\}$, the temperatures Beyer et al. swept, with the student’s softmax at the same $T$ and the loss multiplied by $T^2$. There is no label term. The $L^2$ penalty $\lambda$ is chosen the same way for every condition, from five values by log-loss on a separate set of 10,000 validation images, and every model is scored at $T = 1$ by accuracy and log-loss (the mean negative log-probability of the true label) on 20,000 test images. The last part of the script checks the $T^2$ factor on one test image, using the teacher’s logits and the logits of student B’s ceiling.

import os, sys
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                       # one thread: identical output on every run
import numpy as np
from scipy.optimize import minimize
from scipy.stats import multivariate_normal
from sklearn.datasets import load_digits

# Ten classes of 8x8 images: the mean of each bundled digit class, plus Gaussian pixel noise.
d = load_digits()
mu = np.stack([d.data[d.target == k].mean(0) for k in range(10)]) / 16.0   # templates in [0, 1]
K, sigma = 10, 0.4
P = np.zeros((16, 64))                        # 2x2 average pooling: 8x8 -> 4x4
for i in range(4):
    for j in range(4):
        for a in range(2):
            for b in range(2):
                P[4 * i + j, 8 * (2 * i + a) + 2 * j + b] = 0.25

def draw(n, rng):
    y = rng.integers(0, K, n)
    return mu[y] + rng.normal(0, sigma, (n, 64)), y

def log_softmax(Z):
    Z = Z - Z.max(1, keepdims=True)
    return Z - np.log(np.exp(Z).sum(1, keepdims=True))

def teacher_logits(X):                        # exact Bayes posterior: equal priors, shared covariance
    return (X @ mu.T - 0.5 * (mu ** 2).sum(1)) / sigma ** 2

def ceiling_logits(X):                        # Bayes posterior given only the pooled 4x4 image
    m = mu @ P.T
    return ((X @ P.T) @ m.T - 0.5 * (m ** 2).sum(1)) / (sigma ** 2 / 4)

def features(X, student):                     # A: all 64 pixels; B: the 4x4 pooled image
    F = X if student == "A" else X @ P.T
    return np.c_[F, np.ones(len(F))]

def fit(F, Q, T, lam):
    """Logistic regression on targets Q: mean T^2 * CE(Q, softmax(F W / T)) + lam/2 |W|^2."""
    n, D = F.shape
    def objective(w):
        W = w.reshape(D, K)
        L = log_softmax(F @ W / T)
        loss = -T * T * (Q * L).sum() / n + 0.5 * lam * (W[:-1] ** 2).sum()
        G = T * F.T @ (np.exp(L) - Q) / n     # T^2 * (p_T - q_T) / T, back through F
        G[:-1] += lam * W[:-1]
        return loss, G.ravel()
    r = minimize(objective, np.zeros(D * K), jac=True, method="L-BFGS-B",
                 options={"maxiter": 5000, "gtol": 1e-8, "ftol": 1e-12})
    return r.x.reshape(D, K)

def score(Z, y):                              # accuracy and log-loss at T = 1
    L = log_softmax(Z)
    return (L.argmax(1) == y).mean(), -L[np.arange(len(y)), y].mean()

Xv, yv = draw(10_000, np.random.default_rng(1001))   # validation: chooses lam
Xt, yt = draw(20_000, np.random.default_rng(1002))   # test
print("test class frequencies %.3f-%.3f, noise std %.4f, PP^T = I/4: %s" % (
    np.bincount(yt).min() / len(yt), np.bincount(yt).max() / len(yt), (Xt - mu[yt]).std(),
    np.allclose(P @ P.T, np.eye(16) / 4)))
logpdf = np.stack([multivariate_normal(mu[k], sigma ** 2 * np.eye(64)).logpdf(Xt[:100]) for k in range(K)], 1)
print("teacher vs Gaussian densities, max abs posterior difference: %.1e" % np.abs(
    np.exp(log_softmax(teacher_logits(Xt[:100]))) - np.exp(log_softmax(logpdf))).max())
print("teacher (full Bayes):        acc %.4f  log-loss %.4f" % score(teacher_logits(Xt), yt))
print("student B ceiling (4x4 Bayes): acc %.4f  log-loss %.4f" % score(ceiling_logits(Xt), yt))

seeds = [int(s) for s in sys.argv[1:]] or [0, 1, 2, 3, 4]
conditions = [("hard", 1), ("T=1", 1), ("T=2", 2), ("T=5", 5), ("T=10", 10)]
lams = [1e-4, 1e-3, 1e-2, 1e-1, 1.0]
print(f"seeds {seeds}; mean test accuracy | mean test log-loss")
print("student     n " + "".join(f"{c:>7s}" for c, _ in conditions) + "  |" + "".join(f"{c:>7s}" for c, _ in conditions))
spread = np.zeros((2, len(conditions)))
for student in ["A", "B"]:
    Fv, Ft = features(Xv, student), features(Xt, student)
    for n in [20, 50, 100, 200, 500, 1000, 2000, 5000]:
        res = np.zeros((len(seeds), len(conditions), 2))
        for s, seed in enumerate(seeds):
            X, y = draw(n, np.random.default_rng(seed))
            F, Z = features(X, student), teacher_logits(X)
            for c, (name, T) in enumerate(conditions):
                Q = np.eye(K)[y] if name == "hard" else np.exp(log_softmax(Z / T))
                fits = [fit(F, Q, T, lam) for lam in lams]
                W = min(fits, key=lambda W: score(Fv @ W, yv)[1])   # lam by validation log-loss
                res[s, c] = score(Ft @ W, yt)
        spread = np.maximum(spread, (res.max(0) - res.min(0)).T)
        m = res.mean(0)
        print(f"{student:>7s} {n:5d} " + "".join(f"{a:7.4f}" for a in m[:, 0]) + "  |" + "".join(f"{l:7.4f}" for l in m[:, 1]))
print("largest seed range per condition, accuracy:", " ".join(f"{v:.4f}" for v in spread[0]),
      "| log-loss:", " ".join(f"{v:.4f}" for v in spread[1]))

# The T^2 factor: gradient of T^2 * CE(softmax(z/T), softmax(s/T)) with respect to the student logits s.
z, s = teacher_logits(Xt[:1])[0], ceiling_logits(Xt[:1])[0]
def softmax(v):
    e = np.exp(v - v.max()); return e / e.sum()
def kd_loss(s, T):
    return -T * T * (softmax(z / T) * np.log(softmax(s / T))).sum()
limit = ((s - s.mean()) - (z - z.mean())) / K           # gradient of centred logit matching
print("   T  |grad|    |T^2 grad|  finite-diff err  T^2 grad vs logit matching")
for T in [1, 2, 5, 10, 20, 50, 100]:
    g = T * (softmax(s / T) - softmax(z / T))           # = T^2 * (p_T - q_T) / T
    h = 1e-5 * T
    fd = np.array([(kd_loss(s + h * e, T) - kd_loss(s - h * e, T)) / (2 * h) for e in np.eye(K)])
    print(f"{T:4d}  {np.linalg.norm(g) / T**2:8.2e}  {np.linalg.norm(g):8.4f}    {np.linalg.norm(g - fd) / np.linalg.norm(g):8.1e}"
          f"        {np.linalg.norm(g - limit) / np.linalg.norm(limit):.4f}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-learn 1.7.2; about 50 seconds on one thread; two runs gave identical output):

test class frequencies 0.098-0.103, noise std 0.4000, PP^T = I/4: True
teacher vs Gaussian densities, max abs posterior difference: 7.7e-15
teacher (full Bayes):        acc 0.9382  log-loss 0.1740
student B ceiling (4x4 Bayes): acc 0.7464  log-loss 0.6963
seeds [0, 1, 2, 3, 4]; mean test accuracy | mean test log-loss
student     n    hard    T=1    T=2    T=5   T=10  |   hard    T=1    T=2    T=5   T=10
      A    20  0.4077 0.4753 0.5748 0.6506 0.6536  | 2.2794 1.5798 1.3099 1.0464 1.0395
      A    50  0.6290 0.7393 0.8503 0.9104 0.9137  | 1.0873 0.7327 0.4120 0.2486 0.2397
      A   100  0.7401 0.8483 0.9190 0.9377 0.9379  | 0.7700 0.4413 0.2272 0.1751 0.1742
      A   200  0.8246 0.9031 0.9348 0.9379 0.9380  | 0.5304 0.2888 0.1825 0.1741 0.1741
      A   500  0.8766 0.9271 0.9373 0.9380 0.9380  | 0.3771 0.2164 0.1753 0.1740 0.1740
      A  1000  0.9021 0.9337 0.9377 0.9382 0.9381  | 0.3057 0.1934 0.1744 0.1740 0.1740
      A  2000  0.9158 0.9360 0.9377 0.9381 0.9381  | 0.2724 0.1856 0.1742 0.1740 0.1740
      A  5000  0.9220 0.9372 0.9382 0.9382 0.9382  | 0.2199 0.1812 0.1741 0.1740 0.1740
      B    20  0.3602 0.3883 0.4030 0.4299 0.4376  | 2.7157 1.8970 2.0574 2.1260 2.0715
      B    50  0.4867 0.5238 0.5356 0.5729 0.6006  | 1.4766 1.3352 1.3614 1.3824 1.3558
      B   100  0.5971 0.6291 0.6521 0.6767 0.6775  | 1.1342 1.0646 1.0200 1.0743 1.0862
      B   200  0.6676 0.6843 0.6960 0.6964 0.7061  | 0.9605 0.9329 0.8571 0.9131 0.9464
      B   500  0.6985 0.7112 0.7229 0.7094 0.7193  | 0.8472 0.7999 0.7742 0.8378 0.8932
      B  1000  0.7262 0.7293 0.7332 0.7281 0.7246  | 0.7564 0.7456 0.7419 0.8174 0.8636
      B  2000  0.7351 0.7376 0.7380 0.7321 0.7275  | 0.7330 0.7256 0.7279 0.8037 0.8519
      B  5000  0.7418 0.7425 0.7420 0.7338 0.7306  | 0.7164 0.7141 0.7185 0.7869 0.8395
largest seed range per condition, accuracy: 0.1141 0.1081 0.0963 0.1406 0.1552 | log-loss: 2.3976 0.4651 0.5251 0.3969 0.4374
   T  |grad|    |T^2 grad|  finite-diff err  T^2 grad vs logit matching
   1  1.15e+00    1.1493     1.9e-10        0.5933
   2  4.61e-01    1.8458     8.1e-11        0.7618
   5  7.30e-02    1.8249     1.8e-10        0.5720
  10  1.56e-02    1.5573     3.6e-10        0.2929
  20  3.54e-03    1.4160     6.8e-10        0.1432
  50  5.35e-04    1.3365     1.3e-09        0.0561
 100  1.31e-04    1.3112     4.5e-09        0.0278

The first four lines check the construction: the classes are balanced, the noise has the planned spread, and the closed-form teacher equals the posterior computed from the Gaussian densities. The teacher reaches 93.8% accuracy, and no model that sees only the pooled image can do better than 74.6% in expectation, so student B faces a capacity gap of about 19 points that no amount of data closes.

Soft targets against labels. At every training size, both students score higher accuracy and lower log-loss, on average over the five seeds, with the teacher’s targets at $T = 1$ than with hard labels, and the difference is largest with few images and small at 5,000: for student A it is 8 points of accuracy at 200 images and 1.5 points at 5,000, for student B under 0.1 point at 5,000. At $T = 1$ the two objectives have the same expectation, because the teacher’s targets are the class probabilities from which the labels are drawn, so with unlimited data both lead to the same model; with few images the soft targets give the less noisy estimate, which is Menon et al.’s (2021) argument and the pattern of their Gaussian illustration, where distilling from the true probabilities helped most at small sample sizes. Hard labels catch up slowly: student A trained on 5,000 labelled images is still below student A trained on the teacher’s targets for 200 images at $T = 2$. The seed spread is large: the summary line shows ranges of up to 0.16 in accuracy for a single condition and training size, so small differences between conditions are less certain than the means suggest.

Function matching. At $T = 5$ and 10, student A matches the teacher from 100 images on, to within 0.001 in accuracy and 0.002 in log-loss. It has 65 weights per class, and the soft targets fix each training image’s logits up to an added constant, nine equations per image, so a hundred images determine the teacher’s weights; this is Beyer et al.’s (2022) view of distillation as matching the teacher’s function on enough points, in its simplest form (our reading). At $T = 1$ the same student, given the same images and the same information, stays well below the teacher: at 200 images it reaches 90.3% against 93.8%. In our reading, the objective’s form explains this. At $T = 1$ the teacher is confident on most training images, so the student’s loss hardly changes along directions that would sharpen its predictions further, while the $L^2$ penalty, small as it is, grows with the large weights an exact match needs; at higher temperature the targets are flatter and the loss constrains those directions. In this example the temperature matters through how strongly the targets bind the fit, not through what they contain, since at every $T$ they are a fixed transformation of the same logits.

A student that cannot match its teacher. For student B the picture changes with $n$. With up to 200 images, $T = 10$ gives the highest accuracy, up to about 8 points above $T = 1$ (at 50 images), but not the lowest log-loss, which comes at $T = 1$ or 2. From 1,000 images on, $T = 1$ or 2 gives the lowest log-loss, and $T = 5$ and 10 are clearly worse in log-loss (0.84 at $T = 10$ against 0.71 at $T = 1$ with 5,000 images) and slightly worse in accuracy. Our reading: at $T = 1$ the teacher’s targets are the true class probabilities, so the best student B can do on them is its Bayes ceiling, which $T = 1$ approaches (0.7425 against 0.7464). At $T > 1$ the student is asked to match a flattened version of a teacher it cannot represent, and the best match to that, read back at $T = 1$, is a different and worse-calibrated model. Temperature is a real hyperparameter for a student smaller than its teacher, and the best value depended here on how much data the student had.

The $T^2$ factor. Without the factor, the gradient on the student logits falls by four orders of magnitude between $T = 1$ and $T = 100$, close to $1/T^2$ at the high end; with it, the gradient’s norm stays between 1.1 and 1.9. The analytic gradient agrees with finite differences, and at high temperature the scaled gradient approaches the gradient of centred logit matching, within 3% at $T = 100$, as the derivation above predicts.

Variants

Park et al. (2019) group the methods before theirs as individual knowledge distillation: each matches some output of the teacher for each example separately. They differ in which output is matched.

Logit and response distillation

• Soft targets, the form above, with or without the label term.
• Logit matching. Ba and Caruana trained shallow students to mimic a deep teacher by penalizing the difference between their logits, as Park et al. (2019) describe it (Furlanello et al., 2018, specify the L2 norm of the difference; Ba and Caruana’s paper was not read for this article); the $T^2$ derivation shows soft-target distillation approaching it at high temperature.
• Hard-label distillation. DeiT replaces the teacher’s distribution by its top class $y_t = \arg\max_c Z_t(c)$ and averages the cross-entropy with the true label and with $y_t$ (Touvron et al., 2021, Eq. 3). Because the teacher sees the augmented image, its label can change with the augmentation. For DeiT-B at $224 \times 224$ it gave 83.0% top-1 accuracy on ImageNet against 81.8% for soft distillation, which had not improved on training without distillation (Table 3).

Feature distillation

The student matches the teacher’s intermediate representations. Park et al. (2019, Eq. 3) describe FitNets (Romero et al.) as matching a hidden layer of the teacher by squared Euclidean distance through a learned linear map $\beta$ that bridges the student’s smaller width, and list attention transfer (Zagoruyko and Komodakis), which matches spatial attention maps, among the individual methods. Cho and Hariharan (2019) combined attention transfer with their early-stopped distillation on ImageNet. Neither FitNets nor attention transfer was read for this article. Feature losses need a choice of layers and of a map between widths, which the logit form avoids.

Relational distillation

Relational knowledge distillation (RKD; Park et al., 2019) transfers relations between examples rather than each example’s output. Over a mini-batch, the distance-wise loss matches the distances between pairs of embeddings, each normalized by the mean pairwise distance in the batch, and the angle-wise loss matches the angles formed by triplets, both through a Huber loss (their Eqs. 5–10). Because only relations are matched, the student’s embedding can have a different dimension and scale. The authors note that relational losses alone are not adequate where individual outputs matter, such as a softmax classifier, and combine them with other losses there. RKD suits metric learning, where the embedding’s geometry is the product: on Cars 196, a ResNet-18 student with a 128-dimensional embedding reached 82.50 recall@1 against 77.17 for its ResNet-50 teacher (their Table 1). For classification on CIFAR-100, a ResNet-50 teacher (77.76%) and a VGG-11 student (71.26% alone) gave 74.26% with soft-target distillation and 74.66% with soft targets plus RKD (their Table 4).

Born-again networks and self-distillation

• Born-again networks (Furlanello et al., 2018) train a student with the teacher’s exact architecture on the teacher’s outputs, without softening the logits, and repeat the procedure over generations. On CIFAR-100 every DenseNet student improved over its teacher, and students trained only on the teacher’s outputs did better than students trained on labels and outputs together, for instance 16.95% against 17.68% test error for DenseNet-112-33, whose teacher had 18.25% (their Table 2). The third generation of DenseNet-80-80 reached 15.5%, and an ensemble of the DenseNet-80-120 sequence, which for that network includes the teacher, gave 14.9%. The abstract reports these as validation error; the tables label them test error. Cho and Hariharan (2019) later found that the last generation of such a sequence can underperform a model trained from scratch, and that an ensemble of the generations did not beat an ensemble of the same number of independently trained models (their Table 4).
• Self-distillation within one network. Zhang et al. (2019) attach classifiers after each stage of a network and train the shallow ones with the labels, a KL divergence to the deepest classifier’s output and an $L^2$ hint loss on features; the extra branches can be removed after training. The deepest classifier of a ResNet-50 improved from 77.68% to 80.56% on CIFAR-100 and from 73.56% to 75.24% top-1 on ImageNet (their Tables 1 and 2). Park et al. (2019) also report self-distillation with RKD over generations in metric learning, where gains stopped after the first generation (their Table 2).
• Teacher-free distillation. Yuan et al. (2020) replace the teacher by the student itself or by a hand-designed distribution, which the label smoothing article covers with the rest of their analysis.

Distillation tokens

DeiT adds a distillation token to a vision transformer’s input, beside the class token; its output head is trained on the teacher’s prediction while the class head is trained on the label, and at test time the two heads’ softmax outputs are added (Touvron et al., 2021, §4). For DeiT-B at $224 \times 224$ the two-head classifier reached 83.4% on ImageNet, above hard distillation with the class token alone (83.0%) and above its RegNetY-16GF teacher (82.9%) (Tables 2 and 3). The patch embedding article describes how the token sits among the input tokens and what the trained tokens look like.

Distillation without a fixed teacher

DINO keeps the soft-target loss but has no pretrained teacher: as that article describes, the teacher is an exponential moving average of the student, its outputs are centred and sharpened, and the student matches them across different crops of the same image, with no labels. That use is self-supervised learning with a distillation loss, a separate subject from compressing a trained classifier. Diffusion distillation applies the teacher–student idea to a diffusion model’s iterative sampler, and fine-tuning methods such as Learning without Forgetting use a distillation loss to keep a model’s old outputs while it learns a new task.

Best Practices

• Give teacher and student the same view. Beyer et al. (2022) compared a teacher whose predictions are precomputed on fixed images, independent augmentations for the two networks, and identical augmented views. Only identical views reached peak student accuracy on all their datasets. Fixed-teacher settings reached much lower distillation loss on the training set but gave students that generalized poorly (their Figure 3).
• Distil on many points, and train long. Seeing distillation as function matching, Beyer et al. add an aggressive mixup, with mixing coefficients drawn uniformly from $[0, 1]$, to create inputs off the image manifold, and train for far longer than supervised schedules: their ResNet-50 student on Flowers102 showed no overfitting even at a million epochs and eventually matched its teacher on all four small and mid-sized datasets (their Figure 4), and on ImageNet reached 82.31% top-1 after 9,600 epochs from a BiT-ResNet-152x2 teacher with 83.0% (their Table 1 and §3.2). On the small datasets, a common 100-epoch schedule left the student much worse than a transferred ResNet-50.
• Choose the teacher by its probabilities, not its accuracy. Menon et al. (2021) argue that a good teacher has good probability estimates, and show on CIFAR-100 that among ResNet teachers of depth 8 to 56 distilled into a depth-8 student, the teacher with the best test log-loss (depth 14), not the most accurate one, gave the most accurate student (their Figure 1). They read temperature scaling of the teacher as one way of improving its calibration; the expected calibration error article reports what temperature scaling does to that measure.
• Sweep the temperature. Beyer et al. sweep $\{1, 2, 5, 10\}$, and Cho and Hariharan report $\tau \in \{3, 4, 5\}$ as popular; the example shows the best value moving with the student’s capacity and data.
• Check whether the student can follow the teacher. Cho and Hariharan (2019) measured the fraction of examples on which student and teacher disagree (on CIFAR-10) and the distillation loss (on ImageNet), and write that both are much higher for the largest teacher. Their tables only partly agree: the test disagreement rises with teacher width for both students (Table 2), but in Table 3 the distillation loss with the ResNet-50 teacher (1.553 train, 1.721 test) is below that with the ResNet-18 teacher (2.916, 3.358), and only the early-stopped rows rise with teacher size. In our reading, the disagreement rate is still a cheap diagnostic of a capacity gap.

Pitfalls

• Bigger teachers can make worse students. For a ResNet-18 student on ImageNet, Cho and Hariharan (2019, Table 1) report top-1 error 30.57%, 30.79% and 30.95% with ResNet-18, ResNet-34 and ResNet-50 teachers, against 30.24% for the student trained from scratch (§5.3 calls this figure an accuracy; the tables give it as error). They attribute the trend to mismatched capacity: the student cannot mimic a much larger teacher. Distilling through an intermediate-size teacher did not help (their Table 6, on CIFAR-10), but dropping the distillation term before the end of the student’s training did: with this early-stopped distillation the three teachers gave 29.01–29.35% error (Table 3), and on CIFAR-10 teachers whose own training was stopped early made better students than fully trained ones (their Figure 4).
• Distillation can hurt late in training. On ImageNet, Cho and Hariharan found that the distillation loss improved validation accuracy at first and then lowered it, which they attribute to an underfitting student trading the label loss for the distillation loss; their fix is to drop the distillation term for the end of training.
• High temperature with a smoothed teacher. A teacher trained with label smoothing and distilled at high temperature can diffuse the student’s features; the label smoothing article summarizes the debate and Chandrasegaran et al.’s (2022) advice to use $T = 1$ with such a teacher.
• Out-of-domain transfer images. Beyer et al. (2022) distilled a pets classifier using only food images and reached roughly 30% accuracy; distillation works to some extent on unrelated images, but in-domain images worked best, and related images such as ImageNet could come close, though they may need extra-long schedules.
• Temperature with a student that cannot match. In the example, temperatures that helped student B’s accuracy with few images made its probabilities worse with many; a temperature chosen on accuracy alone can cost calibration.

Evidence and Impact

Compression on ImageNet. Beyer et al. (2022, Table 2) compare ResNet-50 models on the ImageNet validation set at $224 \times 224$: 77.2% for the original ResNet-50, 78.4% for BiT-M-R50 pretrained on ImageNet-21k, and 82.8% for their distilled student whose teacher averages the logits of two BiT-ResNet-152x2 models at 224 and 384 pixels. At 160 pixels, a student that runs twice as fast reached 80.49%, against 78.8% for the best published model at that resolution, and a MobileNet v3 student reached 76.31% after 1,200 epochs, without the training tricks of the original MobileNet v3 paper. Training the same ResNet-50 with labels instead of a teacher, under their long schedules and mixup, gave clearly worse results and overfitted (their Figure 7).

Data-efficient transformers. DeiT’s distilled ViT-B, fine-tuned at $384 \times 384$ and trained for 1,000 epochs, reached 85.2% top-1 on ImageNet without extra training data (Touvron et al., 2021); without distillation its performance saturated after about 400 epochs, while the distilled model kept improving with longer training (their Figure 3). The vision transformer article places this result among the other ViT variants.

Students as large as teachers. Born-again DenseNets on CIFAR-100 and self-distilled ResNets on CIFAR-100 and ImageNet improved over identical networks trained on labels (Furlanello et al., 2018; Zhang et al., 2019), and RKD students in metric learning exceeded their teachers (Park et al., 2019). In our reading, these results fit the accounts of Menon et al. (2021) and Yuan et al. (2020): soft targets act on how a network learns and not only on how much it can hold.

Limits. Cho and Hariharan’s (2019) ImageNet experiments are the clearest evidence that distillation is not automatic: with the standard recipe no teacher improved a ResNet-18 student, and in their experiments gains appeared once distillation was stopped early. In our reading, Beyer et al.’s (2022) results suggest that part of the difference between such reports lies in the recipe, consistent views and very long schedules, rather than in the method.

In sum, knowledge distillation replaces one-hot labels by a teacher’s predictions, softened by a temperature whose gradient effect the $T^2$ factor cancels. Soft targets carry more information per image than labels and give a less noisy training objective, which is why a student can learn from fewer images; how much of the teacher it can absorb depends on its capacity, the views it is shown and the length of training. Feature, relational and self-distillation change what is matched, and the same loss, with a teacher built from the student itself, underlies self-supervised methods such as DINO.