Denoising Diffusion Probabilistic Models (DDPM)

Denoising diffusion probabilistic models (DDPM) are a training and sampling procedure for diffusion models. Ho, Jain and Abbeel (2020) took the diffusion framework of Sohl-Dickstein et al. (2015), rewrote its variational bound as one Gaussian KL divergence per step, and reparameterized each step so that a network predicts the noise in a noisy image. Training then reduces to a regression on randomly noised images (Algorithm 1), and generation to a loop of $T$ denoising steps from pure noise (Algorithm 2). This article derives both and covers the refinements of Nichol and Dhariwal (2021).

Problem

A diffusion model defines $p_\theta(x_0)$ through a reverse Markov chain with Gaussian steps $p_\theta(x_{t-1} \mid x_t) = \mathcal{N}(\mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$. DDPM specifies the objective for fitting $\mu_\theta$ and $\Sigma_\theta$ to images, their parameterization, and the sampler.

Inputs and Outputs

• Training input: images scaled to $[-1, 1]$, a step count $T$ and a schedule $\beta_1, \dots, \beta_T$.
• Training output: a noise predictor $\epsilon_\theta(x_t, t)$, in Ho et al. a U-Net given the step through a sinusoidal embedding.
• Sampling input: the trained $\epsilon_\theta$, the schedule, a choice of reverse variances $\sigma_t^2$, and Gaussian noise.
• Sampling output: an image $x_0$.

Intuition

Each training example is a clean image, a random step and a random noise draw; the network sees the noisy image and must say what noise was added. At generation time the noise estimate gives a slightly cleaner image, and a little fresh noise is added back, so each step is a small stochastic denoising move.

Algorithm

Algorithm 1 (training), repeated until converged:

1. Draw an image $x_0 \sim q(x_0)$, a step $t \sim \mathrm{Uniform}\{1, \dots, T\}$ and noise $\epsilon \sim \mathcal{N}(0, I)$.
2. Form $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$.
3. Take a gradient step on $\lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2$.

Algorithm 2 (sampling):

1. Draw $x_T \sim \mathcal{N}(0, I)$.
2. For $t = T, \dots, 1$: draw $z \sim \mathcal{N}(0, I)$ if $t > 1$, else set $z = 0$, and compute $x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\big(x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \epsilon_\theta(x_t, t)\big) + \sigma_t z$.
3. Return $x_0$.

Ho et al. write the coefficient in step 2 as $(1 - \alpha_t)/\sqrt{1 - \bar\alpha_t}$, the same value. The last step adds no noise: the sample is the mean $\mu_\theta(x_1, 1)$ (§3.3).

Mathematical Formulation

The bound as per-step KL terms

The negative ELBO of the diffusion model, with the forward process $q(x_{1:T} \mid x_0)$ as a fixed approximate posterior, is $L = \mathbb{E}_q\big[-\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\big] \ge -\log p_\theta(x_0)$ (Ho et al., Eq. 3). Using Bayes’ rule to reverse each forward factor given $x_0$, the terms telescope into (Eq. 5)

$$L = \mathbb{E}_q\Big[\underbrace{\mathrm{KL}\big(q(x_T \mid x_0) \,\Vert\, p(x_T)\big)}_{L_T} + \sum_{t > 1} \underbrace{\mathrm{KL}\big(q(x_{t-1} \mid x_t, x_0) \,\Vert\, p_\theta(x_{t-1} \mid x_t)\big)}_{L_{t-1}} \underbrace{-\log p_\theta(x_0 \mid x_1)}_{L_0}\Big].$$

Ho et al. take the derivation from Sohl-Dickstein et al. (2015), whose bound (§2.4, Eq. 14) has the same per-step KL terms with entropy terms at the ends, and repeat it in Appendix A (arXiv v2). Each KL term is between Gaussians, in closed form. With a fixed schedule, $L_T$ has no parameters; Ho et al. report it as about $10^{-5}$ bits per dimension for their schedule (§4).

The forward-process posterior

Conditioned on the clean image, the reverse of a forward step is Gaussian. By Bayes’ rule, $q(x_{t-1} \mid x_t, x_0) \propto q(x_t \mid x_{t-1})\, q(x_{t-1} \mid x_0)$, a product of two Gaussians in $x_{t-1}$ (Gaussian distribution covers the product rule). Adding the precisions $\alpha_t/\beta_t$ and $1/(1 - \bar\alpha_{t-1})$, and using $\alpha_t(1 - \bar\alpha_{t-1}) + \beta_t = 1 - \bar\alpha_t$, gives (Eqs. 6–7)

$$q(x_{t-1} \mid x_t, x_0) = \mathcal{N}\big(\tilde\mu_t(x_t, x_0),\ \tilde\beta_t I\big), \quad \tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\, \beta_t}{1 - \bar\alpha_t}\, x_0 + \frac{\sqrt{\alpha_t}\,(1 - \bar\alpha_{t-1})}{1 - \bar\alpha_t}\, x_t, \quad \tilde\beta_t = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\, \beta_t .$$

The mean is a weighted average of the clean and noisy images; $\tilde\beta_t < \beta_t$, and $\tilde\beta_1 = 0$ since $\bar\alpha_0 = 1$.

From mean matching to noise prediction

With $p_\theta(x_{t-1} \mid x_t) = \mathcal{N}(\mu_\theta, \sigma_t^2 I)$, the KL between the two Gaussians is

$$L_{t-1} = \mathbb{E}_q\Big[\frac{1}{2\sigma_t^2} \lVert \tilde\mu_t(x_t, x_0) - \mu_\theta(x_t, t) \rVert^2\Big] + C,$$

with $C$ independent of $\theta$ (Eq. 8). The obvious choice is a network that outputs $\tilde\mu_t$. Ho et al. instead substitute $x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\, \epsilon)/\sqrt{\bar\alpha_t}$ into $\tilde\mu_t$, which turns the target into $\frac{1}{\sqrt{\alpha_t}}(x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}} \epsilon)$. Since $x_t$ is an input, only $\epsilon$ is unknown, and the mean is parameterized through a noise predictor (Eq. 11), the update in Algorithm 2. The term becomes a weighted noise regression (Eq. 12):

$$L_{t-1} - C = \mathbb{E}_{x_0, \epsilon}\Big[\frac{\beta_t^2}{2\sigma_t^2\, \alpha_t (1 - \bar\alpha_t)} \big\lVert \epsilon - \epsilon_\theta\big(\sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon,\ t\big) \big\rVert^2\Big].$$

The decoder term and the simple loss

$L_0$ is a discrete decoder: each pixel’s probability is the mass that $\mathcal{N}(\mu_\theta(x_1, 1), \sigma_1^2)$ puts on that pixel’s bin of width $2/255$, with the end bins extended to infinity, which makes the bound a lossless codelength for 8-bit images (Eq. 13). Ho et al. trained on a further simplification (Eq. 14),

$$L_{\text{simple}}(\theta) = \mathbb{E}_{t, x_0, \epsilon}\big[\lVert \epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon,\ t) \rVert^2\big],$$

which drops the weight in Eq. 12 and approximates $L_0$ by the density times the bin width. They describe it as a weighted variational bound that down-weights the small-$t$ terms, so the network spends its capacity on the harder denoising tasks at larger $t$ (§3.4).

Reverse variances

Ho et al. fixed $\Sigma_\theta = \sigma_t^2 I$ and found $\sigma_t^2 = \beta_t$ and $\sigma_t^2 = \tilde\beta_t$ gave similar results. The first is optimal when $x_0 \sim \mathcal{N}(0, I)$, the second when $x_0$ is a single fixed point; following Sohl-Dickstein et al., they are the two extremes of the reverse-step entropy for data with unit variance per coordinate (§3.2).

Parameters

• Steps $T$. Ho et al. set $T = 1000$ without a sweep, to match the evaluation count of earlier work (§4; Appendix B, arXiv v2).
• Schedule. A linear schedule from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$, chosen from constant, linear and quadratic schedules constrained to $L_T \approx 0$ (Appendix B). The cosine alternative is given in the diffusion model article.
• Reverse variance. $\sigma_t^2 = \beta_t$ or $\tilde\beta_t$, fixed.
• Optimization. Adam at a learning rate of $2 \times 10^{-4}$ ($2 \times 10^{-5}$ for $256 \times 256$ images), batch 128 on CIFAR-10, and an exponential moving average of the weights with decay 0.9999 (Appendix B).

Complexity

A training step costs one forward and backward pass, whatever $T$ is; sampling costs $T$ sequential forward passes. Ho et al. report that sampling a batch of 256 CIFAR-10 images took 17 seconds on a TPU v3-8, and a batch of 128 images at $256 \times 256$ took 300 seconds (Appendix B, arXiv v2).

Implementation

The posterior and the loss weights. The NumPy script checks Eq. 7 for one pixel of a clean image, $x_0 = 0.6$, against Bayes’ rule normalized on a fine grid, checks the $\epsilon$ form of the mean, and prints the Eq. 12 weights at several steps.

import numpy as np

T = 1000
beta = np.linspace(1e-4, 0.02, T)              # linear schedule of Ho et al. (2020)
alpha = 1 - beta
ab = np.cumprod(alpha)                         # ab[t-1] = alpha_bar_t
ab_prev = np.concatenate([[1.0], ab[:-1]])     # alpha_bar_{t-1}, with alpha_bar_0 = 1

def posterior_closed_form(xt, x0, t):
    """Mean and variance of q(x_{t-1} | x_t, x_0) (Ho et al., Eqs. 6-7)."""
    i = t - 1
    mean = (np.sqrt(ab_prev[i]) * beta[i] * x0 + np.sqrt(alpha[i]) * (1 - ab_prev[i]) * xt) / (1 - ab[i])
    var = (1 - ab_prev[i]) / (1 - ab[i]) * beta[i]
    return mean, var

def gauss(x, m, v):
    return np.exp(-(x - m) ** 2 / (2 * v)) / np.sqrt(2 * np.pi * v)

# One pixel of a clean image, x0 = 0.6, and one observed noisy value x_t.
# Bayes: q(x_{t-1} | x_t, x_0) is proportional to q(x_t | x_{t-1}) q(x_{t-1} | x_0);
# normalize the product numerically on a fine grid and take its moments.
rng = np.random.default_rng(0)
x0 = 0.6
grid = np.linspace(-8, 8, 400001)
err_m = err_v = 0.0
print(f"{'t':>5s}{'x_t':>8s}{'mean (Bayes)':>14s}{'mean (Eq. 7)':>14s}{'var (Bayes)':>13s}{'var (Eq. 7)':>13s}")
for t in [2, 10, 100, 500, 1000]:
    i = t - 1
    xt = np.sqrt(ab[i]) * x0 + np.sqrt(1 - ab[i]) * rng.standard_normal()
    dens = gauss(xt, np.sqrt(alpha[i]) * grid, beta[i]) * gauss(grid, np.sqrt(ab_prev[i]) * x0, 1 - ab_prev[i])
    dens /= np.trapezoid(dens, grid)
    m_b = np.trapezoid(grid * dens, grid)
    v_b = np.trapezoid((grid - m_b) ** 2 * dens, grid)
    m_c, v_c = posterior_closed_form(xt, x0, t)
    err_m, err_v = max(err_m, abs(m_b - m_c)), max(err_v, abs(v_b / v_c - 1))
    print(f"{t:5d}{xt:8.3f}{m_b:14.6f}{m_c:14.6f}{v_b:13.3e}{v_c:13.3e}")
print(f"largest mean difference {err_m:.1e}, largest relative variance difference {err_v:.1e}")

# The epsilon form of the mean (Eq. 11): substitute x0 = (x_t - sqrt(1 - ab_t) eps) / sqrt(ab_t).
t, i = 300, 299
eps = rng.standard_normal(5)
xt = np.sqrt(ab[i]) * x0 + np.sqrt(1 - ab[i]) * eps
m_x0, _ = posterior_closed_form(xt, x0, t)
m_eps = (xt - beta[i] / np.sqrt(1 - ab[i]) * eps) / np.sqrt(alpha[i])
print(f"\nt={t}: max |mu_tilde(x_t, x_0) - eps form| = {np.abs(m_x0 - m_eps).max():.1e}")

# Weight of ||eps - eps_theta||^2 in L_{t-1} (Eq. 12): beta_t^2 / (2 sigma_t^2 alpha_t (1 - ab_t)).
# L_simple replaces these weights by 1 for every t.
bt = (1 - ab_prev) / (1 - ab) * beta
w_beta = beta ** 2 / (2 * beta * alpha * (1 - ab))
w_tilde = beta ** 2 / (2 * np.where(bt > 0, bt, np.nan) * alpha * (1 - ab))   # beta~_1 = 0
print("weight of the noise error in L_{t-1}, per dimension")
print(f"{'t':>5s}{'sigma^2 = beta':>16s}{'sigma^2 = beta~':>17s}{'beta~/beta':>12s}")
for t in [2, 10, 100, 500, 1000]:
    print(f"{t:5d}{w_beta[t-1]:16.4f}{w_tilde[t-1]:17.4f}{bt[t-1] / beta[t-1]:12.4f}")

Output:

    t     x_t  mean (Bayes)  mean (Eq. 7)  var (Bayes)  var (Eq. 7)
    2   0.602      0.600818      0.600818    5.453e-05    5.453e-05
   10   0.594      0.594611      0.594611    2.382e-04    2.382e-04
  100   0.774      0.770446      0.770446    2.035e-03    2.035e-03
  500   0.269      0.269152      0.269152    1.003e-02    1.003e-02
 1000  -0.532     -0.526424     -0.526424    2.000e-02    2.000e-02
largest mean difference 9.2e-15, largest relative variance difference 1.5e-14

t=300: max |mu_tilde(x_t, x_0) - eps form| = 4.4e-16
weight of the noise error in L_{t-1}, per dimension
    t  sigma^2 = beta  sigma^2 = beta~  beta~/beta
    2          0.2727           0.5997      0.4547
   10          0.0737           0.0864      0.8528
  100          0.0101           0.0103      0.9819
  500          0.0055           0.0055      0.9991
 1000          0.0102           0.0102      1.0000

The closed form matches the numerical posterior to rounding error, and the two forms of the mean agree. The weight table makes Ho et al.’s description concrete: with $\sigma_t^2 = \beta_t$, the true bound weights the noise error at $t = 2$ about 50 times more than at $t = 500$ (0.2727 against 0.0055), and $L_{\text{simple}}$ weights them equally, which relatively down-weights the low-noise steps. The last column shows that $\tilde\beta_t$ and $\beta_t$ differ only for small $t$, the observation Nichol and Dhariwal (2021, Figure 1) used to explain why the two fixed variances give similar samples.

Algorithms 1 and 2 on shape images. The PyTorch script (PyTorch 2.14.1) trains a small MLP noise predictor with Algorithm 1 on $8 \times 8$ images of four shapes, drawn with probabilities 0.4, 0.3, 0.2 and 0.1, with $T = 1000$ and the linear schedule; a learning rate of $2 \times 10^{-3}$ with cosine decay (Ho et al. used a constant $2 \times 10^{-4}$ and weight averaging) lets 6,000 steps suffice. It samples 4,000 images with Algorithm 2 for each reverse variance, and with $K = 100$ and $K = 10$ respaced steps (see Variants). The table gives the share of samples nearest each shape, the RMS distance to the nearest shape, the fraction whose pixel signs reproduce a shape exactly, and three points of the pixel histogram.

import math
import sys
import torch
import torch.nn as nn

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.manual_seed(seed)
torch.set_num_threads(4)

# Data: four 8x8 shape images in [-1, 1], drawn with probabilities 0.4, 0.3, 0.2, 0.1.
r, c = torch.meshgrid(torch.arange(8), torch.arange(8), indexing="ij")
masks = [((r == 1) | (r == 6) | (c == 1) | (c == 6)) & (r >= 1) & (r <= 6) & (c >= 1) & (c <= 6),  # square
         (r == 3) | (r == 4) | (c == 3) | (c == 4),                                               # plus
         (r == c) | (r + c == 7),                                                                 # cross
         (r == 3) | (r == 4)]                                                                     # bar
templates = torch.stack([torch.where(m, 1.0, -1.0).flatten() for m in masks])   # (4, 64)
weights = torch.tensor([0.4, 0.3, 0.2, 0.1])
on_share = (weights * (templates > 0).float().mean(1)).sum()   # expected share of +1 pixels

def sample_data(n):
    return templates[torch.multinomial(weights, n, replacement=True)]

# Forward process: T = 1000, linear schedule from 1e-4 to 0.02 (Ho et al., 2020).
T = 1000
beta = torch.linspace(1e-4, 0.02, T, dtype=torch.float64)
ab = torch.cumprod(1 - beta, 0)

# A small MLP denoiser eps_theta(x_t, t) with a sinusoidal embedding of t.
def t_embed(t, dim=32):
    freqs = torch.exp(-math.log(1000.0) * torch.arange(dim // 2) / (dim // 2))
    a = t.float()[:, None] * freqs[None]
    return torch.cat([a.sin(), a.cos()], 1)

net = nn.Sequential(nn.Linear(64 + 32, 256), nn.SiLU(), nn.Linear(256, 256), nn.SiLU(),
                    nn.Linear(256, 256), nn.SiLU(), nn.Linear(256, 64))
def eps_theta(x, t):
    return net(torch.cat([x, t_embed(t)], 1))

# Algorithm 1 (training), on minibatches of 256 images.
steps = 6000
opt = torch.optim.Adam(net.parameters(), lr=2e-3)
lr_decay = torch.optim.lr_scheduler.CosineAnnealingLR(opt, steps)
sab, s1ab = ab.sqrt().float(), (1 - ab).sqrt().float()
for step in range(steps):
    x0 = sample_data(256)                                   # x_0 ~ q(x_0)
    t = torch.randint(1, T + 1, (256,))                     # t ~ Uniform({1, ..., T})
    eps = torch.randn_like(x0)                              # eps ~ N(0, I)
    xt = sab[t - 1, None] * x0 + s1ab[t - 1, None] * eps
    loss = ((eps - eps_theta(xt, t)) ** 2).sum(1).mean()    # ||eps - eps_theta(x_t, t)||^2
    opt.zero_grad(); loss.backward(); opt.step(); lr_decay.step()
    if (step + 1) % 2000 == 0:
        print(f"step {step + 1}: L_simple = {loss.item():.3f}")

# Algorithm 2 (ancestral sampling). With K < T steps the chain visits K evenly
# spaced steps S and uses the betas implied by alpha_bar on S (Nichol & Dhariwal,
# 2021, Sec. 4); with K = T, S = (1, ..., T) and this is Algorithm 2 exactly.
@torch.no_grad()
def sample(n, K, var, gen):
    S = torch.round(torch.linspace(1, T, K, dtype=torch.float64)).long()
    ab_S = ab[S - 1]
    ab_prev = torch.cat([torch.ones(1, dtype=torch.float64), ab_S[:-1]])
    beta_S = 1 - ab_S / ab_prev
    sigma2 = beta_S if var == "beta" else (1 - ab_prev) / (1 - ab_S) * beta_S
    x = torch.randn(n, 64, generator=gen)                   # x_T ~ N(0, I)
    for i in range(K - 1, -1, -1):
        z = torch.randn(n, 64, generator=gen) if i > 0 else torch.zeros(n, 64)
        e = eps_theta(x, torch.full((n,), int(S[i])))
        x = (x - float(beta_S[i] / (1 - ab_S[i]).sqrt()) * e) / float((1 - beta_S[i]).sqrt()) \
            + float(sigma2[i].sqrt()) * z
    return x

def report(x, name):
    d2 = torch.cdist(x, templates) ** 2 / 64                # mean squared error to each shape
    share = torch.bincount(d2.argmin(1), minlength=4).float() / len(x)
    exact = (torch.sign(x)[:, None, :] == templates[None]).all(-1).any(1).float().mean()
    near = ((x.abs() - 1).abs() < 0.1).float().mean()       # pixels within 0.1 of -1 or +1
    mid = (x.abs() < 0.5).float().mean()                    # pixels far from both values
    print(f"{name:21s}" + "".join(f"{s:7.3f}" for s in share)
          + f"{d2.min(1).values.sqrt().mean():7.3f}{exact:7.3f}{near:7.3f}{mid:7.3f}{(x > 0).float().mean():7.3f}")

print(f"{'':21s}{'square':>7s}{'plus':>7s}{'cross':>7s}{'bar':>7s}"
      f"{'RMS':>7s}{'exact':>7s}{'~+-1':>7s}{'|x|<.5':>7s}{'x>0':>7s}")
print(f"{'data (expected)':21s}" + "".join(f"{w:7.3f}" for w in weights)
      + f"{0:7.3f}{1:7.3f}{1:7.3f}{0:7.3f}{on_share:7.3f}")
for K in [1000, 100, 10]:
    for var in ["beta", "beta_tilde"]:
        x = sample(4000, K, var, torch.Generator().manual_seed(seed))
        report(x, f"K={K:4d}, {var}")

Output:

step 2000: L_simple = 3.727
step 4000: L_simple = 2.355
step 6000: L_simple = 2.485
                      square   plus  cross    bar    RMS  exact   ~+-1 |x|<.5    x>0
data (expected)        0.400  0.300  0.200  0.100  0.000  1.000  1.000  0.000  0.331
K=1000, beta           0.427  0.278  0.178  0.116  0.170  0.960  0.625  0.042  0.328
K=1000, beta_tilde     0.427  0.278  0.178  0.116  0.168  0.961  0.639  0.042  0.328
K= 100, beta           0.437  0.297  0.169  0.097  0.193  0.945  0.546  0.051  0.331
K= 100, beta_tilde     0.437  0.297  0.168  0.097  0.172  0.955  0.651  0.050  0.331
K=  10, beta           0.426  0.265  0.182  0.127  0.481  0.548  0.184  0.180  0.325
K=  10, beta_tilde     0.433  0.270  0.177  0.119  0.282  0.906  0.557  0.117  0.324

With 1,000 steps, 96% of the samples reproduce one of the four shapes in every pixel’s sign, the shape shares are within 0.03 of the data’s probabilities, and the share of positive pixels, 0.328, matches the expected 0.331. Residual noise: 63–64% of pixels lie within 0.1 of $\pm 1$ and 4% between $-0.5$ and $0.5$. The two reverse variances give nearly the same samples at $K = 1000$, as Ho et al. found. With 10 respaced steps they part: $\sigma_t^2 = \beta_t$ drops the exact-shape fraction to 0.548, while $\tilde\beta_t$ keeps 0.906. In our reading, once $x_t$ identifies its shape, $x_0$ is effectively a fixed point, where $\tilde\beta_t$ is optimal; with 10 steps the variances differ over much of the chain, and $\beta_t$ adds noise the remaining steps cannot remove. The script runs in about 20 seconds; two runs gave identical output. Run with seeds 0 to 4 as its command-line argument, it gave exact-shape fractions of 0.955 to 0.988 at $K = 1000$ and, at $K = 10$, 0.536 to 0.621 with $\beta_t$ against 0.906 to 0.947 with $\tilde\beta_t$.

Properties and Behavior

• Sample quality versus likelihood. On unconditional CIFAR-10, Ho et al. report an FID of 3.17 and a test NLL bound of 3.75 bits/dim with $L_{\text{simple}}$, against an FID of 13.51 and 3.70 bits/dim when training on the true bound $L$ (Table 1). The simple loss trades a little likelihood for much better samples (Fréchet Inception Distance covers the metric).
• Parameterization matters with the simple loss. In their ablation (Table 2), predicting $\tilde\mu$ worked only with the true bound, and learning a diagonal $\Sigma_\theta$ gave worse FID or unstable training; $\epsilon$-prediction with fixed variances did about as well as $\tilde\mu$-prediction on the bound and was much better with $L_{\text{simple}}$.

Limitations

• Slow sampling. $T = 1000$ sequential network evaluations per image, and naive step skipping with fixed variances degrades samples, as the example shows.
• Weak likelihoods. Ho et al. note their log-likelihoods are not competitive with other likelihood-based models, and that more than half of the codelength describes imperceptible distortions (§4.3).

Variants

Improved DDPM. Nichol and Dhariwal (2021) kept Algorithm 1’s noise prediction but changed the variances, the loss and the schedule (equation numbers from the PMLR version; arXiv v1 numbers them one higher). They learn the reverse variance as a per-dimension interpolation in the log domain, $\Sigma_\theta = \exp(v \log \beta_t + (1 - v) \log \tilde\beta_t)$ with $v$ a network output (Eq. 14). Because $L_{\text{simple}}$ ignores $\Sigma_\theta$, they train on the hybrid loss $L_{\text{hybrid}} = L_{\text{simple}} + \lambda L_{\text{vlb}}$ with $\lambda = 0.001$, and stop the gradient of $L_{\text{vlb}}$ into $\mu_\theta$ so that the bound trains only the variance (Eq. 15). They introduced the cosine schedule (§3.2); to train on the much noisier $L_{\text{vlb}}$ alone, they importance-sample $t$ (Eq. 17). On CIFAR-10 in their runs (Table 2), going from the original setup ($T = 1000$, linear, $L_{\text{simple}}$: NLL 3.73, FID 3.29) to $T = 4000$, cosine and $L_{\text{hybrid}}$ gave an NLL of 3.17 bits/dim at FID 3.19; $L_{\text{vlb}}$ reached 2.94 bits/dim but an FID of 11.47.

Respaced sampling. A model trained with $T$ steps can sample on a subsequence $S$ of steps by recomputing $\beta^S_t = 1 - \bar\alpha_{S_t}/\bar\alpha_{S_{t-1}}$ and $\tilde\beta^S_t$ from the training $\bar\alpha$ (Eq. 18). Learned variances rescale automatically, and $L_{\text{hybrid}}$ models kept near-optimal FID with 100 steps, while fixed-variance $L_{\text{simple}}$ models degraded more (§4, Figure 8). In the same figure, DDIM gave better samples below 50 steps and worse ones at 50 or more; the diffusion sampling article covers DDIM and other fast samplers.

$x_0$- and $v$-prediction. The network can output $\hat x_0$ instead of $\epsilon$, with $\tilde\mu_t(x_t, \hat x_0)$ as the mean; Ho et al. found it worse in early experiments (§3.2). Salimans and Ho (2022, §4) needed a target that stays stable as the signal-to-noise ratio approaches zero and proposed predicting $x_0$, a mix of $x_0$ and $\epsilon$, or $v$, which the diffusion model article defines (their $\alpha_t$ and $\sigma_t$ are this site’s $\sqrt{\bar\alpha_t}$ and $\sqrt{1 - \bar\alpha_t}$). In their CIFAR-10 ablation without distillation (Table 1), all combinations reached stochastic-sampler FIDs of 2.45 to 2.65 except $\epsilon$-prediction with truncated-SNR weighting, which diverged; their text calls $x_0$-prediction slightly better, though the table’s lowest stochastic-sampler FID is $v$-prediction’s.