Denoising Diffusion Probabilistic Models

“Denoising Diffusion Probabilistic Models” by Jonathan Ho, Ajay Jain and Pieter Abbeel of UC Berkeley was posted to arXiv on 19 June 2020 and published at the 34th Conference on Neural Information Processing Systems (NeurIPS 2020). It showed that diffusion probabilistic models (Sohl-Dickstein et al., 2015) could generate images that rivaled GANs, with an unconditional CIFAR-10 Fréchet Inception Distance of 3.17 and $256 \times 256$ LSUN and CelebA-HQ samples. It also tied the variational bound to denoising score matching and the sampler to annealed Langevin dynamics. The procedure as understood today, with its derivation, is covered in the DDPM article; formulas use the notation of the diffusion model article.

Tables 3–4, Appendices A–D and Figures 9–19 are in the NeurIPS supplement; arXiv v2 (16 December 2020) combines paper and supplement with the same numbering.

Problem

In Sohl-Dickstein et al.’s diffusion models, a fixed Markov chain gradually noises the data and a learned Gaussian reverse chain is fitted with a variational bound. Ho et al. call them straightforward to define and efficient to train, but note that nobody, to their knowledge, had shown they could generate high-quality samples.

Meanwhile, noise-conditional score networks (NCSN; Song & Ermon, 2019) had approached GAN quality by a different route: one network estimated the score of images perturbed by Gaussian noise at several levels, trained by denoising score matching (Vincent, 2011), and generated with annealed Langevin dynamics (score matching covers the method).

Contribution

1. Sample quality. Diffusion models can produce high-quality samples, in some cases better than published results for other model families (§4).
2. A connection to score matching. With each reverse mean written through a noise predictor $\epsilon_\theta(x_t, t)$, each term of the bound becomes a weighted noise regression resembling denoising score matching at noise scales indexed by $t$ (Eq. 12), and the sampling step resembles Langevin dynamics with $\epsilon_\theta$ as a learned gradient of the data density. Training with a score-matching-like objective is thus variational inference for a Langevin-like sampler, an equivalence the introduction calls “one of our primary contributions”. Unlike NCSN, Appendix C notes, the sampler’s coefficients follow from the noise schedule instead of being set by hand after training.
3. A weighted bound. $L_{\text{simple}}$, which drops the weights of Eq. 12 and so reweights the bound, gave the best samples.
4. A compression reading. The sampler is a progressive lossy decoder that the authors present as a generalization of autoregressive decoding.

Method

Section 3 makes one choice per term of the bound: fixed forward variances rising linearly from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$, with $T = 1000$ to match the network evaluations of earlier work; fixed reverse variances $\beta_t$ or $\tilde\beta_t$, which gave similar results; the mean parameterized through $\epsilon_\theta$; and a discrete decoder that makes the bound a lossless codelength. The DDPM article derives these and gives Algorithms 1 and 2.

The network is a U-Net following the PixelCNN++ backbone, with group normalization, a sinusoidal embedding of $t$ added into each residual block, and self-attention at $16 \times 16$ (Appendix B). The CIFAR-10 model has 35.7 million parameters, the LSUN and CelebA-HQ models 114 million, and a larger LSUN Bedroom variant about 256 million.

All experiments ran on a TPU v3-8, which the authors liken to eight V100 GPUs. The CIFAR-10 model trained for 800,000 steps at batch 128 in 10.6 hours. The $256 \times 256$ models trained at 2.2 steps per second with batch 64, for 0.5 million steps on CelebA-HQ and 1.2 to 2.4 million on the LSUN categories; by our arithmetic, the 2.4-million-step Bedroom run took about 12.6 days. Each final model was trained once and reported at its lowest-FID checkpoint.

Results

CIFAR-10

Selected rows of Table 1 (unconditional unless marked; NLL in bits/dim, test with train in parentheses):

Model	IS	FID	NLL
Diffusion (original), Sohl-Dickstein et al.			≤ 5.40
Sparse Transformer			2.80
NCSN	8.87 ± 0.12	25.32
StyleGAN2 + ADA (v1)	9.74 ± 0.05	3.26
StyleGAN2 + ADA (v1), conditional	10.06	2.67
Ours ($L$, fixed isotropic $\Sigma$)	7.67 ± 0.13	13.51	≤ 3.70 (3.69)
Ours ($L_{\text{simple}}$)	9.46 ± 0.11	3.17	≤ 3.75 (3.72)

Scores use 50,000 samples, and FID is computed against the training set; against the test set it was 5.24. The abstract calls 3.17 state of the art, which holds among the unconditional models of Table 1; the conditional StyleGAN2 + ADA entry is lower, and the Inception Score is below StyleGAN2 + ADA’s. The true bound gave better codelengths but much worse samples, and the likelihoods, with a train–test gap of at most 0.03 bits/dim, were not competitive with other likelihood-based models.

The ablation (Table 2; a dash marks runs that were unstable and gave poor samples):

Mean parameterization	Objective	IS	FID
$\tilde\mu$ prediction	$L$, learned diagonal $\Sigma$	7.28 ± 0.10	23.69
$\tilde\mu$ prediction	$L$, fixed isotropic $\Sigma$	8.06 ± 0.09	13.22
$\tilde\mu$ prediction	unweighted $\lVert \tilde\mu - \tilde\mu_\theta \rVert^2$	–	–
$\epsilon$ prediction	$L$, learned diagonal $\Sigma$	–	–
$\epsilon$ prediction	$L$, fixed isotropic $\Sigma$	7.67 ± 0.13	13.51
$\epsilon$ prediction	$L_{\text{simple}}$	9.46 ± 0.11	3.17

Learning the variances made training unstable, and $\epsilon$ prediction beat $\tilde\mu$ prediction only with $L_{\text{simple}}$ (§4.2). Predicting $x_0$ directly had given worse samples in early experiments and is not in the table (§3.2).

LSUN and CelebA-HQ

Table 3 (supplement) gives FIDs at $256 \times 256$; the authors computed theirs with the StyleGAN2 code on 50,000 samples, and baselines are as published (StyleGAN’s Church and Cat via StyleGAN2):

Model	Bedroom	Church	Cat
ProgressiveGAN	8.34	6.42	37.52
StyleGAN	2.65	4.21	8.53
StyleGAN2	–	3.86	6.93
Ours ($L_{\text{simple}}$)	6.36	7.89	19.75
Ours ($L_{\text{simple}}$, large)	4.90	–	–

Against the abstract’s “similar to ProgressiveGAN”, the model beats ProgressiveGAN on Bedroom and Cat, trails it on Church, and trails StyleGAN and StyleGAN2 everywhere. For CelebA-HQ the paper shows samples and nearest training neighbors (Figures 1, 11, 12) but reports no FID.

Rate, distortion and progressive generation

Good samples with mediocre likelihoods led the authors to call diffusion models excellent lossy compressors (§4.3). Treating $L_1 + \dots + L_T$ as rate and $L_0$ as distortion, the best-sample CIFAR-10 model has a rate of 1.78 bits/dim and a distortion of 1.97 bits/dim, an RMSE of 0.95 on the 0–255 scale: more than half the lossless codelength describes imperceptible distortions.

In Algorithms 3 and 4 the sender transmits $x_T, \dots, x_0$ in turn, each coded with the model’s reverse step by a procedure, such as minimal random coding, that sends a sample of $q$ in about $\mathrm{KL}(q \Vert p)$ bits; the expected total is the bound (Eq. 5). At any step the receiver can estimate $\hat x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\, \epsilon_\theta(x_t, t))/\sqrt{\bar\alpha_t}$ (Eq. 15). On the CIFAR-10 test set (Figure 5, Table 4), at $t = 101$ the receiver has 0.120 bits/dim and an RMSE of 12.0; at $t = 1$, 1.776 bits/dim and 0.95. By our arithmetic, the last 100 steps carry 93% of the bits. The supplement calls the scheme a proof of concept, since minimal random coding is intractable for high-dimensional data.

Run during sampling, the same estimate shows large-scale features appearing first and details last (Figures 6 and 10). Splitting one CelebA-HQ reverse chain at $t = 1000$ gave very different faces; splitting at $t = 750$ or later gave samples sharing gender, hair color, eyewear, pose and expression, so intermediate latents encode these attributes (Figure 7, Appendix D). A forward process that masks one pixel coordinate per step would make the model autoregressive (Eq. 16); the authors read Gaussian diffusion as autoregression along a bit ordering no reordering of pixels can express, and speculate that it adds a useful inductive bias.

Interpolation

Two CelebA-HQ images are diffused to step $t$, interpolated linearly, and decoded by the reverse process, with the noise fixed across interpolation weights (§4.4; the printed formula interpolates $x_0$ and $x'_0$, evidently a slip for $x_t$ and $x'_t$). At $t = 500$ interpolations varied pose, skin tone, hairstyle, expression and background smoothly, but not eyewear (Figure 8); larger $t$ gave coarser ones, and $t = 1000$ novel samples (Figure 9).

Worked Example: Progressive Coding on a Texture

For Gaussian images the exact denoiser $\hat x_0 = \mathbb{E}[x_0 \mid x_t]$ is known, so Figure 5’s analysis needs no trained model. On $32 \times 32$ textures whose power falls with spatial frequency, under the paper’s schedule, the script prints the rate $I(x_0; x_t)$ per pixel, the RMSE of $\hat x_0$ (0–255 scale) and the variance left in a coarse and a fine frequency band. In our derivation, with an exact reverse model the expected bits sent by Algorithm 3 down to step $t$ telescope to $I(x_0; x_t)$, plus the negligible mismatch between $q(x_T)$ and $\mathcal{N}(0, I)$.

import numpy as np

T = 1000
beta = np.linspace(1e-4, 0.02, T)          # linear schedule of Ho et al. (2020)
ab = np.cumprod(1 - beta)                  # ab[t-1] = alpha_bar_t

# Image distribution: a stationary Gaussian texture on 32x32 images, mean 0, pixel
# std 0.5 on the [-1, 1] scale, with power falling as 1 / (f^2 + f0^2) in spatial
# frequency f (cycles per pixel). Its covariance is diagonal in the 2-D Fourier
# basis, with eigenvalues lam (one per frequency, D = 1024 in all).
n = 32
f = np.fft.fftfreq(n)
fr = np.sqrt(f[:, None] ** 2 + f[None, :] ** 2)        # radial frequency per coefficient
lam = 1.0 / (fr ** 2 + (1.0 / n) ** 2)
lam *= 0.25 / lam.mean()                               # mean pixel variance 0.25
coarse = fr < 1 / 8                                    # low-frequency band
print(f"D = {lam.size}, coarse band: {coarse.sum()} coefficients holding "
      f"{lam[coarse].sum() / lam.sum():.1%} of the variance")

# Exact denoiser: x0_hat = E[x0 | x_t] shrinks each Fourier coefficient of x_t by
# sqrt(ab) lam / (ab lam + 1 - ab). Per coefficient, the mean squared error of x0_hat
# is lam (1 - ab) / (ab lam + 1 - ab), and x_t carries I(x0; x_t) =
# 0.5 log2(1 + ab lam / (1 - ab)) bits about it.
def mmse(s):
    return lam * (1 - s) / (s * lam + 1 - s)

def info_bits(s):
    return 0.5 * np.log2(1 + s * lam / (1 - s))

# Monte Carlo check of the closed form on 200 sampled textures (one fixed seed).
rng = np.random.default_rng(0)
W = np.fft.fft2(rng.standard_normal((200, n, n)), norm="ortho")
x0 = np.fft.ifft2(np.sqrt(lam) * W, norm="ortho").real
for t in [100, 500, 900]:
    s = ab[t - 1]
    xt = np.sqrt(s) * x0 + np.sqrt(1 - s) * rng.standard_normal(x0.shape)
    x0_hat = np.fft.ifft2(np.sqrt(s) * lam / (s * lam + 1 - s) * np.fft.fft2(xt, norm="ortho"),
                          norm="ortho").real
    mc = np.sqrt(((x0 - x0_hat) ** 2).mean()) * 127.5
    print(f"t={t}: RMSE of x0_hat, sampled {mc:.2f}, closed form {np.sqrt(mmse(s).mean()) * 127.5:.2f}")

# Rate-distortion over the reverse process, read off at the step t the receiver has reached.
# rate = I(x0; x_t) / D in bits per pixel; distortion = RMSE of x0_hat on the 0-255 scale.
# The last two columns give the share of each band's variance still unresolved.
total = info_bits(ab[0]).mean()
print(f"\n{'t':>5s}{'steps done':>11s}{'rate':>8s}{'share':>7s}{'RMSE':>7s}{'coarse left':>13s}{'fine left':>11s}")
for t in [1000, 900, 800, 700, 600, 500, 400, 300, 200, 100, 50, 10, 1]:
    s = ab[t - 1]
    m = mmse(s)
    rate = info_bits(s).mean()
    print(f"{t:5d}{T - t:11d}{rate:8.4f}{rate / total:7.1%}{np.sqrt(m.mean()) * 127.5:7.2f}"
          f"{m[coarse].sum() / lam[coarse].sum():13.1%}{m[~coarse].sum() / lam[~coarse].sum():11.1%}")

Output:

D = 1024, coarse band: 45 coefficients holding 47.5% of the variance
t=100: RMSE of x0_hat, sampled 30.01, closed form 29.98
t=500: RMSE of x0_hat, sampled 59.48, closed form 59.21
t=900: RMSE of x0_hat, sampled 63.79, closed form 63.73

    t steps done    rate  share   RMSE  coarse left  fine left
 1000          0  0.0000   0.0%  63.75       100.0%     100.0%
  900        100  0.0000   0.0%  63.73        99.9%     100.0%
  800        200  0.0003   0.0%  63.63        99.2%     100.0%
  700        300  0.0013   0.0%  63.22        96.7%      99.8%
  600        400  0.0046   0.1%  61.94        89.1%      99.2%
  500        500  0.0142   0.3%  59.21        73.6%      97.7%
  400        600  0.0367   0.7%  54.96        52.8%      93.8%
  300        700  0.0862   1.7%  49.40        31.9%      85.4%
  200        800  0.2019   4.0%  42.02        15.0%      69.2%
  100        900  0.5665  11.2%  29.98         4.0%      38.5%
   50        950  1.1715  23.1%  19.12         1.1%      16.1%
   10        990  2.9615  58.4%   5.49         0.1%       1.3%
    1        999  5.0692 100.0%   1.27         0.0%       0.1%

Sampled errors agree with the closed form to within 0.5%. The curve has Figure 5’s shape: after 900 of 1,000 steps the receiver holds 11% of the bits and the RMSE has fallen from 63.75 to 29.98, and the last 100 steps spend 89% of the bits on the remaining, mostly fine, error. The coarse band resolves first: at $t = 300$, 68% of its variance is resolved against 15% of the fine band’s. In our reading, a coefficient of variance $\lambda$ is determined once $\bar\alpha_t \lambda / (1 - \bar\alpha_t)$ passes about 1, which low-frequency coefficients reach earlier. Absolute rates are not comparable with the paper’s: the texture is continuous, so its final rate is set by $\beta_1$, not by 8-bit quantization.

Impact

The authors released a TensorFlow implementation; as of October 2026 its README links to trained models and samples.

Song et al. (2021) grouped Song and Ermon’s models and DDPM as score-based generative models, noting that the DDPM objective implicitly computes scores, and showed DDPM’s sampler to be one discretization of a reverse-time variance-preserving SDE. Dhariwal and Nichol (2021) credit Ho et al. with introducing the U-Net architecture for diffusion models. The rate–distortion analysis has a direct descendant: the latent diffusion paper (Rombach et al., 2022, Figure 2 in both the CVPR and arXiv v2 versions) re-plots data from Ho et al. to argue that most bits of an image are imperceptible detail, the motivation for training diffusion models in an autoencoder’s latent space.

Broader impact statement

The statement warns of fake images and videos of high-profile figures made for political purposes, notes that detecting generated images may get harder as models improve, and warns that models trained on internet data reflect its biases, which proliferating samples would reinforce. On the positive side it lists data compression, representation learning and creative uses in art, photography and music.

Limitations

The paper concedes uncompetitive likelihoods, a proof-of-concept compression scheme and unstable training with learned variances. Hyperparameters tuned for CIFAR-10 were transferred to the other datasets; in our reading, single runs say nothing of run-to-run variation. Sampling takes 1,000 sequential network evaluations, 300 seconds per 128 images at $256 \times 256$ (Appendix B). Later work found more: Nichol and Dhariwal (2021, §3.2) observed that the linear schedule destroys information faster than necessary, and that models trained with $L_{\text{simple}}$ and fixed variances lost much more sample quality than learned-variance models when sampled with fewer steps (§4, Figure 8). Song, Meng and Ermon (2021) call the many iterations a critical drawback.

What Came After

Improved DDPM (Nichol & Dhariwal, 2021) learned the reverse variances with a hybrid objective and introduced a cosine schedule; its Table 3 lists CIFAR-10 likelihoods of 3.70 bits/dim for DDPM and 2.94 for itself, and it sampled with an order of magnitude fewer steps at little cost in quality (DDPM covers these changes). DDIM (Song, Meng & Ermon, 2021) kept DDPM’s training procedure and added deterministic samplers 10 to 50 times faster in wall-clock time (diffusion sampling). Dhariwal and Nichol (2021) improved the architecture and added classifier guidance (see classifier-free guidance); re-evaluated in their codebase, DDPM scored FIDs of 4.89 on LSUN Bedroom and 17.1 on LSUN Cat, against 1.90 and 5.57 for their model (Table 5, arXiv v4). Latent diffusion moved the diffusion process into a learned latent space and became the basis of Stable Diffusion.