Latent Diffusion Model

A latent diffusion model (LDM) is a diffusion model trained on the latent representation of a pretrained autoencoder instead of on pixels. The encoder compresses an image $x$ into a smaller spatial grid $z = E(x)$; the diffusion model learns to generate such grids; and the decoder turns a generated grid back into an image in one pass. Every denoising step then runs on a much smaller grid, so training and sampling cost far less. Rombach et al. (2022) introduced the design together with cross-attention conditioning, and it underlies Stable Diffusion and its successors. The notation is that of the diffusion model article with $z$ in place of $x$, as in Rombach et al.: $z_t = \sqrt{\bar\alpha_t}\, z_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$, with noise predictor $\epsilon_\theta(z_t, t, \cdot)$.

Core Idea

Rombach et al. distinguish perceptual compression, which removes high-frequency detail that viewers barely notice but that makes up most of an image’s bits, from semantic compression, the generative modeling of what the image shows. Pixel-space diffusion can suppress imperceptible detail by minimizing the responsible loss terms, but its network and gradients are still evaluated on every pixel. The LDM hands perceptual compression to an autoencoder trained once, and lets the diffusion model work in a space the authors call perceptually equivalent to image space but computationally cheaper (arXiv v2, §1, §3). Unlike VQGAN (Esser, Rombach & Ommer, 2021), which modeled a heavily compressed discrete latent autoregressively, the LDM keeps the latent two-dimensional for a convolutional U-Net, which the authors argue permits milder compression.

Building Blocks

The first stage. The autoencoder follows VQGAN, with a perceptual and a patch-based adversarial loss (§3.1). Its latent is regularized either by KL-reg., a very weak KL penalty toward a standard normal described in the variational autoencoder article, or by VQ-reg., a vector quantization layer treated as the decoder’s first layer, so that diffusion runs on the latent before quantization (Appendix G). LDMs on VQ-regularized latents sometimes gave better samples, although those first stages reconstructed slightly worse (§4). The first stage is then frozen and can serve many diffusion models.

Downsampling factor. For $x \in \mathbb{R}^{H \times W \times 3}$, the encoder outputs $z \in \mathbb{R}^{h \times w \times c}$ with $f = H/h = W/w$ a power of 2. For $256 \times 256$ images, the models compared in §4.1 had latents of $128 \times 128 \times 2$ ($f = 2$), $64 \times 64 \times 3$, $32 \times 32 \times 4$, $16 \times 16 \times 8$ and $8 \times 8 \times 32$ ($f = 32$) (Table 13). Stable Diffusion v1 uses an $f = 8$ autoencoder (CompVis README, as of October 2026) with $64 \times 64 \times 4$ latents for $512 \times 512$ images.

The latent scaling factor. A diffusion schedule assumes data of roughly unit variance, which a weakly regularized latent need not have. For KL latents, Rombach et al. estimate a single standard deviation $\hat\sigma$ over all components of the first training batch and divide every latent by it (Appendix G); VQ latents already had variance close to 1 (Appendix D.1). The released code computes the factor as 1 / z.flatten().std(); the Stable Diffusion v1 configuration fixes it at 0.18215 and Stability AI’s SDXL base configuration at 0.13025 (as of October 2026). Stable Diffusion 3 subtracts a mean as well (Esser et al., 2024, Appendix B.2).

The denoiser. $\epsilon_\theta$ is a time-conditional U-Net built mainly from 2D convolutions (§3.2). For conditioning, each self-attention layer of Dhariwal and Nichol’s U-Net is replaced by a shallow transformer alternating self-attention, an MLP and cross-attention to the condition (Appendix E.2.1, Table 16).

The condition encoder. A domain-specific encoder $\tau_\theta$ maps the condition $y$ to $M$ tokens, $\tau_\theta(y) \in \mathbb{R}^{M \times d_\tau}$ (§3.3): a 32-layer transformer over up to 77 BERT-tokenized text tokens, a 16-layer transformer over bounding boxes encoded as discretized corners and a class, or a single class embedding (Appendix E.2.1, Table 17). The paper trains $\tau_\theta$ jointly with the denoiser; Stable Diffusion v1 instead uses a frozen CLIP ViT-L/14 text encoder (CompVis README).

Concatenation conditioning. A condition aligned with the output image is concatenated to $z_t$ along the channels (§4.3.2): downsampled semantic maps, the low-resolution image for super-resolution (with $\tau_\theta$ the identity, §4.4), and the masked image for inpainting (Table 15). Such convolutional models, trained on $256 \times 256$ crops, ran on inputs up to about a megapixel.

Mathematical Formulation

With $z_0 = E(x)/\hat\sigma$, the conditional objective is DDPM’s noise-prediction loss on latents (Rombach et al., Eq. 3, with the rescaling of Appendix G):

$$L_{\text{LDM}} = \mathbb{E}_{x,\, y,\, \epsilon \sim \mathcal{N}(0, I),\, t}\Big[\big\lVert \epsilon - \epsilon_\theta\big(\sqrt{\bar\alpha_t}\, z_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon,\ t,\ \tau_\theta(y)\big) \big\rVert_2^2\Big], \qquad \hat\sigma^2 = \frac{1}{bchw} \sum_{b,c,h,w} \big(z_{b,c,h,w} - \hat\mu\big)^2,$$

where the sum runs over a batch of $b$ unscaled latents with mean $\hat\mu$. Cross-attention layer $i$ uses the flattened U-Net features $\varphi_i(z_t) \in \mathbb{R}^{N \times d_\epsilon^i}$ as $N$ queries over the $M$ condition tokens:

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d}}\right) V, \qquad Q = \varphi_i(z_t)\, W_Q^{(i)}, \quad K = \tau_\theta(y)\, W_K^{(i)}, \quad V = \tau_\theta(y)\, W_V^{(i)},$$

with $W_Q^{(i)} \in \mathbb{R}^{d_\epsilon^i \times d}$ and $W_K^{(i)}, W_V^{(i)} \in \mathbb{R}^{d_\tau \times d}$. The paper writes the projections on the left and prints the shapes of $W_Q$ and $W_V$ the other way round; the shapes above are the ones for which the products are defined (our reading).

Rescaling matters because the signal-to-noise ratio of $z_t$ is $\mathrm{Var}(z_0)\, \bar\alpha_t / (1 - \bar\alpha_t)$. Rombach et al. report that unscaled KL latents gave a very high ratio, so the model fixed much semantic detail early in the reverse process, which affected convolutional sampling at large sizes (Appendix D.1).

How Data Flows

Training. Encode and rescale, $z_0 = E(x)/\hat\sigma$; draw $t$ and $\epsilon$; form $z_t$; encode or concatenate the condition; and step on the loss. The decoder is unused.

Sampling. Draw $z_T \sim \mathcal{N}(0, I)$ at latent size, run a diffusion sampler (Rombach et al. mostly used DDIM, with classifier-free guidance for their best conditional results), and decode once, $\tilde x = D(\hat\sigma z_0)$.

Why It Works

A grid with $f^2$ times fewer positions cuts the cost of each convolutional denoising step roughly in proportion (our reasoning), while the decoder restores texture. With equal compute (one A100, equal steps and parameters), class-conditional ImageNet models with $f = 1$ or 2 trained slowly, $f = 32$ stopped improving early, and $f = 4$ to 16 balanced the two, with a Fréchet Inception Distance (FID) gap of 38 between $f = 1$ and $f = 8$ after 2 million steps (§4.1, Fig. 6). Against sampling speed, $f = 4$ and 8 did best; complex datasets such as ImageNet needed lower compression (Fig. 7).

For class-conditional ImageNet $256 \times 256$, Table 18 lists 271 V100-days of training for LDM-4 against 916 for ADM, and 0.4 against 0.07 samples per second for the guided models at 250 steps; Peebles and Xie (2023, §5.1, arXiv v2) count 103.6 against 1,120 Gflops per forward pass. For inpainting, the text (§4.5) claims at least a 2.7-fold speed-up and a 1.6-fold FID improvement over pixel-space diffusion, but in Table 6 the FID falls from 24.74 to between 14.99 and 15.95, a factor of 1.55 for the model without first-stage attention.

Training Considerations

• Separate stages. Unlike LSGM, which trains the autoencoder and a score-based prior jointly, separate training needs no delicate balance of reconstruction against generation (Rombach et al., §1).
• Schedule naming. The paper’s tables list a “linear” schedule, but in the released code “linear” interpolates $\sqrt{\beta_t}$, not $\beta_t$ as in DDPM; Stable Diffusion v1’s configuration uses endpoints 0.00085 and 0.012 (as of October 2026). Esser et al. (2024, §3) call this LDM-Linear.

Variants

SDXL (Podell et al., 2024) keeps a latent U-Net but triples it to 2.6 billion parameters, against 860 million for Stable Diffusion 1.x, by moving transformer blocks to lower resolutions and conditioning on two text encoders, CLIP ViT-L and OpenCLIP ViT-bigG (arXiv v1, Table 1). An optional refiner, a second LDM in the same latent space specialized on the first 200 noise levels, re-noises the base model’s latents and denoises them in the manner of SDEdit (§2.5).

Latent transformers. Peebles and Xie’s Diffusion Transformer replaces the U-Net with a Vision Transformer-style network over $p \times p$ patches of Stable Diffusion’s $f = 8$ latents; halving $p$ quadruples the tokens and at least quadruples the compute (§3.2, §4).

MM-DiT with rectified flow. Stable Diffusion 3 (Esser et al., 2024) joins text and latent-patch tokens in one sequence and runs a DiT-like transformer with separate weights per modality, joined in attention, instead of cross-attending to fixed text features (§4). It trains a rectified flow, $z_t = (1 - t)\, z_0 + t\, \epsilon$ for $t \in [0, 1]$ with the network predicting the velocity and training steps drawn from a logit-normal distribution, which ranked consistently well among the formulations compared (§3.1, §5.1); uses an $f = 8$ autoencoder with 16 channels instead of 4, which reconstructed better (reconstruction FID 1.06 against 2.41, Table 3); and scales to 8 billion parameters.

Use in Computer Vision

Rombach et al. applied LDMs to unconditional, class-, text-, layout- and semantic-map-conditioned generation, super-resolution and inpainting (§4), including a 1.45-billion-parameter text-to-image model trained on LAION-400M. Stable Diffusion, text-to-image generation and image inpainting have their own articles.

Limitations

• Sequential sampling. Cheaper than pixel-space diffusion, but still slower than GANs (Rombach et al., §5).
• First-stage bottleneck. A sample has no detail the decoder cannot render; reconstruction error limits tasks that need fine pixel accuracy, including, the authors expect, their super-resolution (§5). SDXL and Stable Diffusion 3 both improved their autoencoders.
• Minimum image size. Podell et al. (2024, arXiv v1, §2.2) note that the two-stage design needs a minimum training image size; Stable Diffusion 1.4 and 1.5 discarded images with a side below 512 pixels.

Example

A linear first stage and an exact denoiser. The data are $32 \times 32$ RGB shape images: eight templates (disk, square, triangle and ring, red or blue) plus independent pixel noise of standard deviation 0.1 standing in for texture, an equal-weight Gaussian mixture. The “first stage” maps each $f \times f \times 3$ patch to 4 numbers by patch PCA, an $f$-strided linear encoder whose transpose is the decoder, and rescales by one standard deviation from the first batch (Appendix G). The script reports the template reconstruction for each $f$, then samples 200 images for each of seeds 0 to 2 with Ho et al.’s Algorithm 2 and the exact noise predictor (a linear map of a Gaussian mixture is again one), in pixel space and in the $f = 4$ and $f = 8$ latents, and prints the shares of the nearest templates (0.125 expected), the RMS distance to them, and the “texture” left after subtracting their first-stage reconstructions.

import numpy as np

# Toy data: 32x32 RGB shape images in [-1, 1]. Eight templates (disk, square,
# triangle, ring; red or blue, at slightly different places) are the modes of a
# Gaussian mixture; i.i.d. pixel noise of std 0.1 stands in for fine texture.
H = W = 32
yy, xx = np.mgrid[:H, :W] + 0.5
def shape(kind, cy, cx, r):
    if kind == "disk":     return (yy - cy) ** 2 + (xx - cx) ** 2 < r ** 2
    if kind == "square":   return (abs(yy - cy) < r) & (abs(xx - cx) < r)
    if kind == "triangle": return (yy - cy < r) & (abs(xx - cx) < (yy - cy + r) / 2)
    if kind == "ring":     return abs(np.hypot(yy - cy, xx - cx) - r) < 2
mu = np.array([np.where(shape(kind, 14 + 4 * j, 13 + 2 * i, 8 + j)[..., None], color, -1.0).ravel()
               for i, kind in enumerate(["disk", "square", "triangle", "ring"])
               for j, color in enumerate([(1, -1, -1), (-1, -1, 1)])])   # (8, 3072): red or blue on black
K, D = mu.shape
s = 0.1                                                          # texture std
rng = np.random.default_rng(0)
X = mu[rng.integers(K, size=2000)] + s * rng.standard_normal((2000, D))   # training images

# First stage: a linear encoder that maps each f x f x 3 patch to c = 4 numbers by
# patch PCA (an f-strided convolution with orthonormal filters); the decoder is its
# transpose. Latent shape (32/f) x (32/f) x 4.
def patches(x, f):          # (n, 3072) -> (n, (32/f)^2, f*f*3)
    p = x.reshape(-1, H // f, f, W // f, f, 3).transpose(0, 1, 3, 2, 4, 5)
    return p.reshape(len(x), (H // f) * (W // f), f * f * 3)
def unpatches(p, f):
    x = p.reshape(-1, H // f, W // f, f, f, 3).transpose(0, 1, 3, 2, 4, 5)
    return x.reshape(len(p), D)
def first_stage(f, c=4):
    P = patches(X, f).reshape(-1, f * f * 3)
    m = P.mean(0)
    V = np.linalg.svd(P - m, full_matrices=False)[2][:c]        # orthonormal filters
    sigma = ((patches(X[:256], f) - m) @ V.T).std()              # one scale from the first batch
    enc = lambda x: ((patches(x, f) - m) @ V.T).reshape(len(x), -1) / sigma
    dec = lambda z: unpatches((z * sigma).reshape(len(z), -1, c) @ V + m, f)
    return enc, dec, sigma

print("first stage    latent   dim  compression  scale 1/sigma  template PSNR (dB)")
for f in [2, 4, 8, 16]:
    enc, dec, sigma = first_stage(f)
    psnr = 10 * np.log10(4 / ((dec(enc(mu)) - mu) ** 2).mean())  # pixel range 2 -> peak^2 = 4
    print(f"LDM-{f:<2d}  {f'{H//f}x{W//f}x4':>12s}{(H//f)**2*4:6d}{D/((H//f)**2*4):10.0f}x"
          f"{1/sigma:13.3f}{psnr:13.1f}")

T = 1000
beta = np.linspace(1e-4, 0.02, T)                               # linear schedule (Ho et al.)
alpha, ab = 1 - beta, np.cumprod(1 - beta)

def sample(means, v0, n, rng):
    """Ancestral sampling (Ho et al., Algorithm 2) with the exact noise predictor of
    an equal-weight isotropic Gaussian mixture: modes `means`, within-mode variance v0."""
    means = means.astype(np.float32)
    mm = (means ** 2).sum(1)
    x = rng.standard_normal((n, means.shape[1]), dtype=np.float32)
    for t in range(T, 0, -1):
        a = ab[t - 1]
        v = a * v0 + 1 - a
        logits = -((x ** 2).sum(1)[:, None] - 2 * np.sqrt(a) * np.einsum("nd,kd->nk", x, means) + a * mm) / (2 * v)
        r = np.exp(logits - logits.max(1, keepdims=True)); r /= r.sum(1, keepdims=True)
        eps = np.sqrt(1 - a) * (x - np.sqrt(a) * np.einsum("nk,kd->nd", r, means)) / v
        x = (x - beta[t - 1] / np.sqrt(1 - a) * eps) / np.sqrt(alpha[t - 1])
        if t > 1:
            x += np.sqrt(beta[t - 1]) * rng.standard_normal(x.shape, dtype=np.float32)
    return x

print("\nmodel   dim  seed  share per template (min-max)  RMS to template  texture std")
n = 200
for f in [1, 4, 8]:
    if f == 1:   # pixel-space diffusion, no first stage
        enc, dec, sigma = (lambda x: x), (lambda z: z), 1.0
    else:
        enc, dec, sigma = first_stage(f)
    rec = dec(enc(mu))                       # what the first stage keeps of each template
    for seed in range(3):
        xs = dec(sample(enc(mu), (s / sigma) ** 2, n, np.random.default_rng(seed)))
        d2 = ((xs[:, None, :] - mu[None]) ** 2).mean(-1)
        k = d2.argmin(1)                     # nearest template
        share = np.bincount(k, minlength=K) / n
        texture = np.sqrt(((xs - rec[k]) ** 2).mean())
        name = "pixel" if f == 1 else f"LDM-{f}"
        print(f"{name:7s}{enc(mu).shape[1]:5d}{seed:5d}{share.min():16.3f} - {share.max():.3f}"
              f"{np.sqrt(d2.min(1).mean()):18.3f}{texture:13.3f}")

# Why rescale: the SNR of the latent at step t is Var(z) * ab_t / (1 - ab_t).
enc, dec, sigma = first_stage(4)
for name, var in [("unscaled", sigma ** 2), ("scaled", 1.0)]:
    print(f"LDM-4 {name}: Var(z) = {var:.2f}, SNR falls below 1 at t = {np.argmax(var * ab / (1 - ab) < 1) + 1}")

Output:

first stage    latent   dim  compression  scale 1/sigma  template PSNR (dB)
LDM-2        16x16x4  1024         3x        1.159         26.3
LDM-4          8x8x4   256        12x        0.620         20.3
LDM-8          4x4x4    64        48x        0.337         17.7
LDM-16         2x2x4    16       192x        0.197         15.3

model   dim  seed  share per template (min-max)  RMS to template  texture std
pixel   3072    0           0.095 - 0.160             0.101        0.101
pixel   3072    1           0.085 - 0.160             0.101        0.101
pixel   3072    2           0.100 - 0.145             0.101        0.101
LDM-4    256    0           0.095 - 0.150             0.195        0.029
LDM-4    256    1           0.105 - 0.160             0.195        0.029
LDM-4    256    2           0.100 - 0.140             0.195        0.029
LDM-8     64    0           0.080 - 0.160             0.264        0.014
LDM-8     64    1           0.100 - 0.140             0.261        0.014
LDM-8     64    2           0.075 - 0.170             0.259        0.015
LDM-4 unscaled: Var(z) = 2.60, SNR falls below 1 at t = 354
LDM-4 scaled: Var(z) = 1.00, SNR falls below 1 at t = 260

The first table is the $f$ trade-off in miniature: each doubling of $f$ divides the latent by 4 and lowers the template PSNR, from 26.3 dB at $f = 2$ to 15.3 dB at $f = 16$, as edges finer than a patch are lost. Every model generates all eight shapes, with shares between 0.075 and 0.170 over seeds 0 to 2, the latent models with 12 and 48 times fewer numbers. But they cannot exceed their first stage: the RMS distance to the nearest template, 0.101 in pixel space (the data’s 0.1), grows to 0.195 for $f = 4$ and 0.259 to 0.264 for $f = 8$, almost all of it template reconstruction error, and the texture shrinks to 0.029 and about 0.014. A real LDM differs here: its adversarially trained decoder renders plausible texture, which a linear decoder cannot. The last lines show why latents are rescaled: the unscaled $f = 4$ latent has variance 2.60, which moves the step where the SNR falls below 1 from 260 to 354.