Diffusion Distillation

Diffusion distillation trains a fast generator, the student, to reproduce what a pretrained diffusion model, the teacher, produces with many network evaluations, so that images can be generated in one to a few evaluations instead of tens to hundreds. It is knowledge distillation applied to an iterative sampler: the student usually has the teacher’s architecture and starts from its weights, and what it learns to imitate is either the teacher’s mapping from noise to image or the distribution of the teacher’s images. The main methods are progressive distillation (Salimans & Ho, 2022), which halves the number of sampling steps stage by stage; guided distillation (Meng et al., 2023), which also folds classifier-free guidance into the student; consistency distillation (Song et al., 2023), which teaches a consistency model to jump along the probability-flow ODE; adversarial diffusion distillation (Sauer et al., 2024), the method behind SDXL-Turbo; and distribution matching distillation (Yin et al., 2024). The progressive-distillation paper is an ICLR paper that was not read for this article; its method is described here through the readable papers that build on it and report it.

Purpose

A diffusion model generates by integrating its reverse SDE or probability-flow ODE in many small steps, each a pass through a large network. Yin et al. (2024, §1) put the cost at typically tens to hundreds of network evaluations, and Song et al. (2023, §1) at 10 to 2,000 times the cost of a single-step generator such as a GAN. Classifier-free guidance doubles it again: in Meng et al.’s words (abstract, §1) it needs two diffusion models, a conditional and an unconditional one, evaluated tens to hundreds of times per sample.

Better solvers, surveyed in Diffusion Sampling Methods, attack the step count without retraining. Yin et al. (§2) summarize their reach as cutting a pretrained model’s steps from a thousand to 20 to 50, with performance collapsing when steps are cut further. The reason, in our reading, is that a solver can only follow the ODE’s trajectories more cleverly; with very few steps it must extrapolate curved trajectories from a handful of slope evaluations, and one step of DDIM lands on the denoiser’s posterior mean, a blur of every image consistent with the noise. A distilled student is not bound by that: it is trained to output where the trajectory ends, or simply a realistic image, from wherever it starts.

The cost moves from inference to training. The teacher must exist first, and distillation itself is a substantial training run: Meng et al. trained 50,000 to 100,000 updates per halving stage on pixel-space models (supplement B.3), and Yin et al. trained their text-to-image generator for about 36 hours on 72 A100 GPUs (§4.3).

When to Use It

Distillation pays when a model will be sampled many times and latency matters: interactive tools, previews, real-time generation. Yin et al. report 20 frames per second at $512 \times 512$ with FP16 inference for their one-step distillation of Stable Diffusion 1.5 (§1, §4.3); Sauer et al. describe ADD as the first method to unlock single-step real-time synthesis with foundation models (abstract).

It is less attractive when:

• 20 to 50 solver steps are affordable. A good sampler on the original model needs no training and keeps the model’s full quality and diversity.
• The model changes often. Each fine-tuned or retrained teacher needs its own distillation. LCM-LoRA, which Sauer et al. (§2) describe as low-rank modules that can be plugged into different Stable Diffusion and SDXL checkpoints, eases part of this (see Consistency Model).
• Diversity matters more than fidelity. The adversarial and distribution-matching methods trade some sample diversity for sharpness (see Pitfalls).
• The teacher’s noise-to-image map matters. Uses that rely on the teacher’s deterministic map, such as DDIM inversion for editing, fit best, in our reading, with methods trained to imitate that map (progressive and consistency distillation); methods that match only distributions are not asked to keep it, although DMD’s regression loss keeps its generator aligned with the teacher on the same noise (Yin et al., §1, Figure 6).

How It Works

The methods differ in what the student is asked to match. Yin et al. (§1–§2) sort earlier work into methods that distill the noise-to-image mapping of the teacher’s sampler and contrast them with their own, which matches the teacher at the level of distributions. Adversarial diffusion distillation mixes the two ideas. The sections below follow that order.

Distilling the Sampler’s Mapping

The direct approach runs the teacher’s deterministic sampler to completion from many noise images and trains a one-step student by regression on the resulting noise–image pairs. Song et al. (§2) and Yin et al. (§2) attribute this to Luhman and Luhman (2021), whose paper was not read here. Its drawback, as both describe it, is cost: every training pair needs a full run of the teacher’s sampler, and fitting a high-dimensional, complex map in one shot is demanding (Yin et al., §1). The methods that followed shorten the distance the student has to learn at a time.

Progressive Distillation

Progressive distillation (Salimans & Ho, 2022) trains a series of students, each of which halves the number of sampling steps of the previous one (Yin et al., §2). The account here follows Meng et al., who describe it as the approach their second stage applies (§3.2) and give the training loop in their supplement (Algorithm 2).

Write $z_t = \alpha_t x + \sigma_t \epsilon$ for the noisy image at continuous time $t \in [0, 1]$, with log signal-to-noise ratio $\lambda_t = \log(\alpha_t^2 / \sigma_t^2)$, and $\hat x_\eta(z_t)$ for a network’s prediction of the clean image. One deterministic DDIM step from $t$ to $s < t$ is

$$z_s = \alpha_s\, \hat x_\eta(z_t) + \sigma_s\, \frac{z_t - \alpha_t\, \hat x_\eta(z_t)}{\sigma_t}$$

(Meng et al., Eq. 2), the update DDIM uses with $\eta = 0$. A teacher that samples in $2N$ steps on the grid $t = 1, \tfrac{2N - 1}{2N}, \dots, 0$ is distilled into a student that samples in $N$ steps as follows:

1. Draw a training image $x$, a time $t = i/N$ with $i$ uniform on $\{1, \dots, N\}$, and noise; form $z_t$.
2. Run two DDIM steps of the teacher, from $t$ to $t' = t - 0.5/N$ and from $t'$ to $t'' = t - 1/N$, reaching $\tilde z_{t''}$.
3. Compute the clean-image target $\tilde x$ that would carry $z_t$ to $\tilde z_{t''}$ in one DDIM step:

$$\tilde x = \frac{\tilde z_{t''} - (\sigma_{t''}/\sigma_t)\, z_t}{\alpha_{t''} - (\sigma_{t''}/\sigma_t)\, \alpha_t}$$

(supplement, Algorithm 2). Substituting $\tilde x$ into the DDIM update with $s = t''$ returns $\tilde z_{t''}$ exactly, which is how the formula is obtained. 4. Minimize $\omega(\lambda_t)\, \lVert \tilde x - \hat x_{\eta_2}(z_t) \rVert_2^2$ over the student’s parameters $\eta_2$. 5. When the student has converged, it becomes the new teacher, $N$ is halved, and a new student is initialized from it.

Each stage therefore learns a short, local task, two teacher steps compressed into one, and never needs the full trajectory; at the last stage, $N = 1$, the target is the endpoint of the previous student’s two-step sampler. This is why Song et al. (§2, §6.2) single out progressive distillation as the only earlier distillation method that does not first build a large dataset by running the diffusion model. The cost is that errors are inherited: each student imitates a teacher that is itself an approximation, and the training is a sequence of runs rather than one, which Sauer et al. (§2) list as a drawback.

Initialization matters. Meng et al. (§3.1) note, citing Salimans and Ho, that initialization plays a key role, and each of their students starts from its teacher’s weights. Following Salimans and Ho, Meng et al. use $v$-prediction models (§4.1; the Diffusion Model article defines $v$) and train the progressive stages with an “SNR truncation” weighting $\omega$ (stage one with what they call an “SNR loss”), which they cite to Kingma et al. and Salimans and Ho without restating it (supplement B.2–B.3). Their pixel-space schedule started from a 1,024-step DDIM teacher, trained each student for 50,000 updates (100,000 for two steps and one) with the learning rate annealed linearly from $10^{-4}$ to 0 and no EMA, which they say follows Salimans and Ho closely (supplement B.3).

Guided Distillation

Meng et al. (§2) state that progressive distillation could not, until their work, be applied directly to classifier-free guided models or studied with samplers other than deterministic DDIM, and their CVPR 2023 paper addresses both. Their method has two stages (§3):

• Stage one trains a single student $\hat x_{\eta_1}(z_t, w)$ to match the guided combination of the teacher’s conditional and unconditional predictions,

$$\min_{\eta_1}\ \mathbb{E}_{w \sim U[w_{\min}, w_{\max}],\, t,\, x}\ \omega(\lambda_t)\, \big\lVert \hat x_{\eta_1}(z_t, w) - \big[(1 + w)\, \hat x_{c,\theta}(z_t) - w\, \hat x_\theta(z_t)\big] \big\rVert_2^2$$

(Eq. 3), in the guidance convention of the classifier-free guidance article. The guidance weight $w$ is an input to the student, passed through a Fourier embedding and incorporated in the same way as the time step, so one student covers a range of guidance strengths; it is initialized from the teacher’s conditional model. This stage alone halves the evaluations per step, since one network replaces two.

• Stage two progressively distills the stage-one model, halving the steps each time as above, with $w$ drawn from the same range at every training step.

They also give an $N$-step stochastic sampler: one deterministic step of twice the usual length, followed by one step back up the noise scale with fresh noise (Eqs. 4–7, Figure 5), for which the distillation target is changed accordingly (supplement, Algorithm 3). The same model serves image-to-image translation in the manner of SDEdit and inpainting, in 2 to 4 steps (§4.2.3–4.2.4), and the encoder (the reverse DDIM process) can be distilled too (§4.3).

The Classifier-Free Guidance article summarizes this method among guidance variants; its training details and results are here.

Consistency Distillation

Consistency models (Song et al., 2023) take a different route to the same goal. Instead of halving steps stage by stage, a single network $f_\theta(x_t, t)$ is trained to map every point of a probability-flow ODE trajectory to the trajectory’s origin, by making its outputs agree at two adjacent noise levels, where the lower point comes from one solver step of the teacher. One training run then gives a one-step generator that can still spend more steps for better samples. Yin et al. (§2) group consistency distillation with TRACT and BOOT as methods that train a student to match its own output at a different time step on the ODE flow. The Consistency Model article covers the loss, the boundary condition and its results; Song et al. report that consistency distillation beat progressive distillation of the same teachers on every dataset and number of sampling steps they tried, except one-step generation on LSUN Bedroom, where CD with the squared $\ell_2$ metric slightly underperformed PD with $\ell_2$ (§6.2, Figure 4).

Adversarial Diffusion Distillation

Adversarial diffusion distillation (ADD; Sauer et al., ECCV 2024) gives up imitating the teacher’s trajectories. The student, initialized from a pretrained diffusion U-Net, is trained as a denoiser: from a real image $x_0$ noised to $x_s = \alpha_s x_0 + \sigma_s \epsilon$ at one of $N = 4$ student time steps, with the largest at $\tau_n = 1000$ and a zero-terminal-SNR schedule so that inference can start from pure noise (§3.1), it outputs a sample $\hat x_\theta(x_s, s)$. Two losses act on that sample (Eq. 1):

$$\mathcal{L} = \mathcal{L}^{G}_{\text{adv}}\big(\hat x_\theta(x_s, s), \phi\big) + \lambda\, \mathcal{L}_{\text{distill}}\big(\hat x_\theta(x_s, s), \psi\big), \qquad \lambda = 2.5 .$$

• Adversarial loss. A discriminator, following the design and training procedure of StyleGAN-T, applies trainable lightweight heads to the features of a frozen pretrained vision transformer at several layers, and is trained with the hinge loss and an R1 gradient penalty computed at the inputs of the heads (Eqs. 2–3). It is conditioned on a text embedding and, because for $s < 1000$ the student sees part of the real image, on an image embedding of $x_0$ as well (§3.2). DINOv2 ViT-S features worked best, better than ViT-L, as the discriminator’s feature network (Table 1a, §4.1); the text embedding comes from a CLIP ViT-g-14 encoder and the image embedding from the CLS token of DINOv2 ViT-L (§4).
• Distillation loss. A frozen teacher diffusion model with weights $\psi$ is applied not to the student’s sample directly, which would be out of distribution for it, but to the sample re-noised to a random time $t$, $\hat x_{\theta, t} = \alpha_t \hat x_\theta + \sigma_t \epsilon'$; its denoised prediction is the target:

$$\mathcal{L}_{\text{distill}} = \mathbb{E}_{t, \epsilon'}\Big[c(t)\, d\big(\hat x_\theta,\ \hat x_\psi(\operatorname{sg}(\hat x_{\theta, t}); t)\big)\Big], \qquad d(x, y) = \lVert x - y \rVert_2^2$$

(Eq. 4), with $\operatorname{sg}$ a stop-gradient. With a particular weighting $c(t)$ the loss is equivalent to the score distillation sampling (SDS) objective of DreamFusion (Poole et al., 2023, not read here), as Sauer et al. show in their supplementary material, which was not read; their ablation uses an exponential weighting $c(t) = \alpha_t$, and the final models use the noise-free score distillation (NFSD) weighting (§3.3, §4.1). For latent diffusion models the loss is computed in pixel space, which gave more stable gradients (§3.1).

In ablations on a Stable Diffusion 2.1 student with one sampling step, scored by zero-shot COCO FID on 5,000 captions and CLIP score (Table 1d), the adversarial loss alone gave FID 20.8 and CLIP score 0.315, the distillation loss alone 315.6 and 0.076, unable to improve on the undistilled student, and the two combined 20.6 and 0.319 with exponential weighting. A randomly initialized student collapsed (FID 293.6, Table 1c), which the authors present as an advantage of starting from a diffusion model over training a GAN from scratch. One teacher step was enough as the target (Table 1f), and a student distilled from a larger teacher took on that teacher’s characteristics (Table 1e). The discriminator’s access to real images means ADD, unlike the methods above, uses training data directly, while the teacher contributes, in the authors’ words, its structural understanding (§5). Sampling needs no classifier-free guidance (§1), and the diffusion framework keeps iterative refinement: with the seed fixed, samples with 1, 2 and 4 steps share their general layout while more steps improve consistency and detail (Figure 8).

ADD-M (860 million parameters) is distilled from Stable Diffusion 2.1 for ablations and from SD 1.5 for comparisons; ADD-XL (3.1 billion) from SDXL (§4). Stability AI released SDXL-Turbo on 28 November 2023 together with the ADD technical report (generative-models README, as of November 2026); the Stable Diffusion article describes the released Turbo models and their licences.

Distribution Matching Distillation

Distribution matching distillation (DMD; Yin et al., CVPR 2024) also abandons a pointwise correspondence between noise and teacher images, but replaces the discriminator with two diffusion models. The one-step generator $G_\theta$ has the base model’s architecture without time conditioning and is initialized as $G_\theta(z) = \mu_{\text{base}}(z, T - 1)$ (§3.1). Following ProlificDreamer’s variational score distillation, it minimizes the KL divergence from the generator’s (“fake”) distribution to the real one, whose gradient is

$$\nabla_\theta D_{\text{KL}}(p_{\text{fake}} \,\Vert\, p_{\text{real}}) = \mathbb{E}_{z}\Big[-\big(s_{\text{real}}(x) - s_{\text{fake}}(x)\big)\, \frac{dG}{d\theta}\Big], \qquad x = G_\theta(z)$$

(Eq. 2), where $s = \nabla_x \log p$ are scores. Intuitively, the real score pulls samples toward the modes of the data and the fake score pushes them apart (§3.2). Neither score is defined everywhere for the clean distributions, which may not overlap, so both are taken on noised versions, where they overlap and diffusion models estimate them (Figure 4):

• the real score comes from the frozen base model, $s_{\text{real}}(x_t, t) = -(x_t - \alpha_t \mu_{\text{base}}(x_t, t))/\sigma_t^2$ (Eq. 4);
• the fake score comes from a second diffusion model $\mu^\phi_{\text{fake}}$, initialized from the base model and trained throughout with the ordinary denoising loss on the generator’s current outputs, so that it tracks the changing fake distribution (Eqs. 5–6).

The generator’s gradient is the difference of the two, averaged over noise levels $t \sim U(0.02T, 0.98T)$ with a weight $w_t$ that normalizes the gradient’s magnitude by the mean absolute difference between the noised sample’s denoised version and the sample (Eqs. 7–8). On its own this objective is prone to mode dropping, since the score does not see how much probability mass a mode carries (§3.3; Figure 3 shows it on a two-mode toy problem). DMD therefore adds a regression loss: an LPIPS distance between the generator’s output and the teacher’s deterministic sample from the same noise, on a modest precomputed set of noise–image pairs (EDM’s Heun solver with 18 steps for CIFAR-10 and 256 for ImageNet, PNDM with 50 steps for LAION), weighted $\lambda_{\text{reg}} = 0.25$ (Eq. 9). For guided text-to-image models the pairs come from the guided teacher, the real score uses the guided prediction, and the guidance scale is fixed (§3.4).

The two ablations of Table 2 show what each part does: without distribution matching, FID rose from 2.66 to 3.82 on CIFAR-10 and from 2.62 to 9.21 on ImageNet $64 \times 64$, and the images lacked realism and structure; without the regression loss, to 5.58 and 5.61, with unstable training and mode collapse (Figure 5). The weighting of Eq. 8 beat the weightings of DreamFusion and ProlificDreamer on CIFAR-10 (2.66 against 3.60 and 3.71).

Score distillation sampling, developed to distill text-to-image diffusion models into 3D synthesis (Sauer et al., §2; the 3D Generation article describes DreamFusion), is the common ancestor here: ADD’s distillation loss is a form of it, and, in our reading, DMD’s gradient adds to its real-score term a learned fake-score term; Yin et al. describe their method as refining and extending variational score distillation to train a generator (§2).

How to Apply It

The papers above share a recipe, here summarized as practice rather than as a single source’s prescription:

1. Start from the teacher’s weights. Every method above initializes the student, and DMD also its fake-score model, from the pretrained diffusion model; ADD’s ablation shows a random start failing outright (Table 1c).
2. Choose what to match. Trajectory-matching methods (progressive and consistency distillation) imitate the teacher’s deterministic map, using noised training images as inputs; ADD also needs real images for its discriminator; DMD needs a set of teacher samples for its regression loss and otherwise only the teacher’s predictions.
3. Decide on guidance in advance. Distilled models usually bake guidance in. Meng et al. keep a range of $w$ as an input; DMD fixes one scale per model (3 or 8 for Stable Diffusion 1.5, §4.3); ADD samples without guidance.
4. Match the parameterization to the step count. For one to four steps the network predicts a clean image (or $v$) rather than noise, and the schedule must reach zero signal-to-noise at the start of sampling, which ADD enforces explicitly (§3.1).
5. Evaluate against the teacher and against a sampler at the same cost. The baselines that matter are the teacher at its normal step count and the best training-free sampler at the student’s step count, as in Meng et al.’s Table 1 (DDIM at matched steps) and Sauer et al.’s Table 2 (DPM-Solver).

Variants

Latent consistency models (LCM). Consistency distillation applied to Stable Diffusion’s latent space, with guidance distilled into the model; LCM-LoRA packages the result as a low-rank adapter that plugs into different checkpoints. The LCM paper was not read here; the Consistency Model article describes LCM from its official repository. Readable papers report it as a baseline: in Yin et al.’s evaluation on COCO-30k at guidance scale 8 (Table 4), LCM-LoRA (trained with guidance 7.5) scored FID 77.90 in one step, 24.28 in two and 23.62 in four, against 14.93 for one step of DMD; Sauer et al. found one step of ADD-XL preferred over four steps of LCM-XL in their user study (§4.2).

Rectified flow and InstaFlow. Yin et al. (§2) describe InstaFlow as progressively learning straighter flows on which a one-step prediction stays accurate over a larger distance; Sauer et al. (§2) describe it as using rectified flows to facilitate a better distillation process.

TRACT and BOOT match a student’s output at one time to its own output at another along the ODE, like consistency distillation (Yin et al., §2).

Distillation with and without a realism loss. ADD’s discriminator compares the student’s samples with real images; progressive and consistency distillation use training images only as inputs to be noised, and DMD’s generator learns from the teacher’s scores and samples. Sauer et al. credit the adversarial loss for the boosted realism and textures of ADD-XL’s samples (Figure 4); in our reading, a loss that compares samples with real images is also what lets ADD-XL be preferred over its teacher, whereas a trajectory-matching student is trained only to reproduce its teacher.

Best Practices

• Report steps and evaluations together. A guided teacher with $N$ steps costs $2N$ evaluations; Meng et al. write their baselines as “DDIM $16 \times 2$-step” for that reason, and their speed-up table counts both (supplement Table 3).
• Compare at equal base model and protocol. DMD’s ImageNet table states that its baseline numbers are taken from Song et al. (Table 1), and its COCO table that its numbers come from the GigaGAN paper or were evaluated by the authors with released models (Table 3). Sauer et al. compare distillation methods with one base model, SD 1.5 (Table 2).
• Keep a regression or teacher term with distribution-level losses. Both DMD’s ablation and ADD’s (distillation loss improving CLIP score when combined with the adversarial loss, Table 1d) show the teacher term carrying information that the realism term alone does not.
• Check several guidance values when guidance is an input. Meng et al.’s single student performed comparably to separately trained fixed-$w$ students across $w \in [0, 4]$ (Table 1 and supplement Table 4), but in Table 1 the step count at which its FID first fell below the teacher’s varies with $w$: eight steps at $w = 0$, one step at $w = 1$.

Pitfalls

• Error accumulation across stages. In progressive distillation each student copies an imperfect teacher. Yin et al. (§1) note that the performance of distilled models still lagged the original multi-step models, and Song et al.’s Tables 1 and 2 show progressive distillation (with the squared $\ell_2$ loss of Salimans and Ho) at one step far behind EDM: FID 8.34 on CIFAR-10 and 15.39 on ImageNet $64 \times 64$, against 2.04 for EDM at 35 evaluations and 2.44 at 79.
• Lost diversity. Sauer et al. report that ADD-XL’s sample diversity tends to be lower than SDXL’s (Figure 4) and that its realism comes at the cost of slightly decreased diversity (§4.3). DMD without its regression loss collapsed onto fewer modes (Table 2, Figure 5). A real-score-only objective, as Yin et al.’s toy example shows, sends every sample to the nearest mode (Figure 3a).
• Guidance fixed at training time. A student distilled at one guidance scale samples at that scale; changing the quality–diversity trade-off means distilling again, unless $w$ is an input as in Meng et al.
• Metric leakage. Several methods train with LPIPS, a network trained on ImageNet, and are scored by FID, which uses another ImageNet network; the Consistency Model article quotes the consistency-models model card’s warning that this can overstate quality. DMD’s regression loss is LPIPS as well (§3.3).
• Training memory and compute. DMD fine-tunes both the generator and the fake-score model, which Yin et al. list as a limitation (§5); ADD needs, besides the teacher, a discriminator built on a frozen feature network and frozen text and image encoders for its conditioning (§3.2, §4).
• The teacher’s traits carry over. In ADD’s ablation, a student distilled from SDXL took on SDXL’s higher FID and higher CLIP score (Table 1e).

Evidence and Impact

Results are reported on different protocols and are comparable only within one table.

Pixel-space guided distillation (Meng et al., Table 1, ImageNet $64 \times 64$, FID and Inception Score). Without guidance ($w = 0$), the teacher, DDIM with $1024 \times 2$ steps, scored FID 2.92; the distilled model scored 22.74 in one step, 4.14 in four, 2.79 in eight and 2.44 in sixteen with the deterministic sampler, while DDIM on the teacher scored 7.68 with $16 \times 2$ steps. Supplement Table 4 extends this to CIFAR-10, where the distilled model scored 3.18 in four deterministic steps against 2.73 for the $1024 \times 2$-step teacher. The authors summarize the pixel-space result as matching the teacher with 4 to 16 steps, a speed-up of up to 256 times (§4.1).

Latent guided distillation (Meng et al., Table 2, captioned “LAION 512X512” after the distilled $512 \times 512$ Stable Diffusion model, with $w = 8$; the table does not restate its protocol, and the evaluation setting the text gives for this model is FID and CLIP score on 5,000 random COCO 2017 validation captions). The distilled model scored FID 37.3 in two steps and 26.0 in four, against 98.8 and 34.1 for DPM-Solver++ at the same number of steps, each of which evaluates two networks; at eight steps DPM-Solver++ was better on FID (25.6 against 26.9). Against eight DDIM steps of the original model, FID and CLIP score showed no significant difference, although the distilled samples were visibly sharper and more coherent; the authors hypothesize that the two metrics do not fully capture such differences (§4.2.2, Figures 7 and 10).

Adversarial distillation (Sauer et al., Table 2, all on SD 1.5, zero-shot COCO FID on 5,000 samples and CLIP score). One step of ADD-M scored FID 19.7 and CLIP 0.326, against 20.1 and 0.318 for 25 steps of DPM-Solver, 37.2 and 0.275 for one step of Meng et al.’s distillation (which the table labels progressive distillation and cites to Meng et al.), and 23.4 and 0.304 for InstaFlow-0.9B. For their main comparison the authors used human preference studies, which they consider more reliable than automated metrics: one step of ADD-XL was preferred over all compared models except SDXL, and four steps of ADD-XL over SDXL with 50 steps in the majority of comparisons (§4.2, Figures 6–7).

Distribution matching (Yin et al.). On ImageNet $64 \times 64$ one step of DMD reached FID 2.62 against 2.32 for its EDM teacher at 512 evaluations, 6.20 for consistency distillation and 15.39 for progressive distillation (Table 1). On zero-shot COCO-30k at guidance 3 it scored 11.49 in 0.09 s, against 8.78 in 2.59 s for its teacher, Stable Diffusion 1.5 (Table 3); at guidance 8, FID 14.93 and CLIP score 0.320 against 13.45 and 0.322 (Table 4).

In our reading, the papers taken together show a move from imitating the sampler, which keeps the teacher’s map but loses quality at one or two steps, to matching distributions with adversarial or score-based losses, which brought one-step quality close to the teacher’s in their tables at some cost in diversity.