High-Resolution Image Synthesis with Latent Diffusion Models

“High-Resolution Image Synthesis with Latent Diffusion Models” by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer was posted to arXiv in December 2021 and presented at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) in 2022. The authors were at LMU Munich and Heidelberg University, Esser at Runway ML; Rombach and Blattmann are marked as equal contributors. The paper trained a diffusion model on the latent grid of a pretrained autoencoder instead of on pixels, added a cross-attention mechanism for general conditioning, and reported results on seven image synthesis tasks at a fraction of the compute of pixel-space diffusion. The design as used today is covered in the latent diffusion model article.

Versions. The CVPR paper, the primary version, has eight pages of text; its appendices appear in the CVPR supplementary material and in arXiv v2 (13 April 2022). This article uses arXiv v2’s figure, table and appendix numbers. The CVPR text numbers the two figures of the $f$ study 5 and 6 (v2: Figs. 6 and 7) and swaps Tables 4 and 5; the other tables share numbers. The supplement’s appendix letters run one behind v2’s, which adds a changelog as Appendix A: v2’s Appendix G is the supplement’s F, and the supplement’s G holds the limitations that v2 puts in §5. Everything the changelog lists as new since v1 (the 1.45-billion-parameter text-to-image model, a retrained ImageNet model, guidance, user studies) is in the CVPR text.

Problem

By 2021 pixel-space diffusion models led class-conditional synthesis and super-resolution, at a cost: the paper cites 150 to 1,000 V100-days of training for Dhariwal and Nichol’s models and about five A100-days to draw 50,000 samples, which the authors argue only a small part of the field can afford (§1). Two-stage alternatives such as VQGAN (Esser, Rombach & Ommer, 2021) and DALL-E compressed images into discrete codes for autoregressive transformers, but needed high compression, which capped quality: on ImageNet validation, VQGAN’s $f = 16$ autoencoder reconstructs at a PSNR of 19.9 dB and DALL-E’s $f = 8$ one at 22.8 dB, against 27.4 dB for the authors’ $f = 4$ model (Fig. 1, Table 8).

The motivation is Fig. 2, a rate–distortion curve of a pixel-space DDPM from Ho et al. (2020). The authors read it as two stages: a perceptual compression stage that removes high-frequency detail while learning little semantic variation, then semantic compression, in which the model learns the conceptual composition of the data. A diffusion model can de-emphasize the first through its reweighted loss, but still evaluates its network and gradients on every pixel. Hence the proposal: learn a perceptually equivalent, cheaper space once and train the diffusion model there.

Contribution

The authors list six contributions (§1):

1. Unlike transformer-based two-stage methods, it scales more gracefully to higher-dimensional data, allowing milder compression and megapixel synthesis.
2. Competitive results on several tasks at much lower cost than pixel-based diffusion.
3. Unlike LSGM, which trains autoencoder and score-based prior jointly, no delicate weighting of reconstruction against generation.
4. For densely conditioned tasks, convolutional application to consistent images of about $1024^2$ pixels.
5. A general-purpose cross-attention conditioning mechanism, used for class-, text- and layout-conditioned models.
6. A release of pretrained latent diffusion and autoencoding models.

Method

The first stage is a VQGAN-style autoencoder trained with a perceptual and a patch-based adversarial loss, regularized either by a KL penalty weighted about $10^{-6}$ or by vector quantization absorbed into the decoder (§3.1, Appendix G). The second stage is a time-conditional U-Net trained with DDPM’s noise-prediction loss on latents (Eq. 2). For conditioning, a domain-specific encoder $\tau_\theta$, trained jointly with the U-Net, maps the condition to tokens that U-Net layers attend to (Eq. 3); spatially aligned conditions are instead concatenated to the input (§3.3, §4.3.2). Equations, shapes and the latent rescaling are in the latent diffusion model article.

Two details changed between versions. The CVPR text says VQ-regularized LDMs give better samples and are used throughout unless stated otherwise; arXiv v2 says “sometimes” and drops the second sentence, and specifies that the text-to-image model is KL-regularized (§4, §4.3.1). Two errors noted in the latent diffusion model article appear in both versions: the printed shapes of $W_Q$ and $W_V$ are swapped (§3.3), and the noise schedule the hyperparameter tables call “linear” (Tables 12–15) is, in the released code, linear in $\sqrt{\beta_t}$.

Results

The $f$ study (§4.1). Class-conditional ImageNet models with $f \in \{1, 2, 4, 8, 16, 32\}$, where LDM-1 is pixel-space diffusion, were each trained for 2 million steps on one A100 with about 391–396 million parameters (Table 13). LDM-1 and LDM-2 trained slowly, LDM-32 stagnated early, and the FID gap between LDM-1 and LDM-8 after 2 million steps was 38 (Fig. 6). Against sampling throughput, LDM-4 and LDM-8 did best on CelebA-HQ and ImageNet (Fig. 7). These FIDs use 5,000 samples (Appendix E.3.5). The comparison fixes steps and parameters, not data seen: batch sizes ran from 7 for LDM-1 to 112 for LDM-16 and LDM-32 (Table 13), so the larger-$f$ models also saw many more images (our reading).

Unconditional synthesis (Table 1, 256 × 256). FID 5.11 on CelebA-HQ with 500 DDIM steps, which the authors call a new state of the art; 4.98 on FFHQ (StyleGAN 4.16, ProjectedGAN 3.08); 4.02 on LSUN-Churches with a KL-regularized LDM-8 (StyleGAN2 3.86); and 2.95 on LSUN-Bedrooms (ADM 1.90). The authors claim consistently better precision and recall than GANs; recall ties StyleGAN on LSUN-Bedrooms (0.48) and beats the other GAN entries.

Class-conditional ImageNet (Table 3). The 400-million-parameter LDM-4 scored FID 10.56 and IS 103.49 with 250 DDIM steps, against 10.94 for the 554-million-parameter ADM. With classifier-free guidance (Ho & Salimans) at scale 1.5, LDM-4-G reached FID 3.60, IS 247.67, precision 0.87 and recall 0.48, against 4.59 for classifier-guided ADM-G.

Text-to-image (Table 2). A 1.45-billion-parameter KL-regularized LDM-8 with a 32-layer transformer $\tau_\theta$ over BERT tokens was trained on LAION-400M and evaluated against 30,000 MS-COCO validation images (Appendix E.3.2). The CVPR table reports FID 23.35 unguided and 12.61 with guidance scale 1.5 (IS 26.62), and compares only with DALL-E, CogView and LAFITE (FID 27.50 to 26.94). arXiv v2 prints 23.31 and 12.63 (IS 30.29) for the same model names, drops DALL-E and adds GLIDE (12.24, 6 billion parameters; Nichol et al., 2022) and Make-A-Scene (11.84, 4 billion), with which the authors call the guided model on par.

Layout-to-image (Table 9). On COCO, an LDM-4 pretrained on OpenImages and fine-tuned scored FID 40.91, against 41.11 for SPADE; on OpenImages it scored 32.02 at $256 \times 256$ against 45.33 for the transformer of Jahn et al. Appendix D.3 calls this a margin of nearly 11 FID, but the table’s gaps are 13.3 at $256 \times 256$ and 12.3 at $512 \times 512$.

Semantic synthesis (§4.3.2). Trained on $256 \times 256$ crops of landscape images with concatenated semantic maps, the model generated images up to the megapixel range when run convolutionally; no metrics are reported. The text describes an $f = 4$ first stage, but the landscapes column of Table 15 lists $f = 8$ with $32 \times 32 \times 4$ latents.

Super-resolution (Table 5). Following SR3 (Saharia et al.), a 169-million-parameter LDM-4 upscaled bicubically downsampled ImageNet images from 64 to 256 pixels. It scored FID 2.8 against validation features (4.8 against training features) and IS 166.3, against SR3’s 5.2 and 180.1; PSNR was lower (24.4 against 26.4 dB), and a regression model scored highest on PSNR (27.9 dB). In a user study, raters preferred LDM-4 over a pixel-space diffusion baseline in 70.6% of comparisons (Table 4).

Inpainting (Tables 6–7). On Places with LaMa’s masks and protocol (Suvorov et al., 2022), the text claims at least a 2.7-fold speed-up over a pixel-space model of equal size, which Table 6 supports, and at least a 1.6-fold FID improvement, which it does not (a factor of 1.55 for the model without first-stage attention). A larger 387-million-parameter model, fine-tuned for half an epoch at $512 \times 512$, reached FID 1.50 on all 30,000 test crops and 9.39 on masks covering 40–50% of the image, against 2.21 and 12.0 printed for LaMa (Table 7). Raters preferred LDM-4 to LaMa in 68.1% of comparisons (Table 4); see image inpainting.

Compute (Table 18). Training cost is given in V100-days, converting A100 time with a factor of 2.2: 271 for the ImageNet LDM-4 against 916 for ADM, 18 against 64 for StyleGAN2 on LSUN-Churches, and 26 against 32.13 on FFHQ. The table lists the first-stage autoencoders separately (29 V100-days for $f = 4$, 66 for $f = 8$), outside the LDM totals. Its LSUN-Bedrooms row prints a generator compute of 60 but an overall compute of 55.

Impact

The authors’ repository distributes the autoencoders and LDMs; its news section dates the release of the 1.45-billion-parameter LAION-400M model and the guided ImageNet model to April 2022 (CompVis README, as of October 2026). Its model zoo lists the CelebA-HQ model with 200 DDIM steps where Table 1 says 500.

The paper’s largest influence came through Stable Diffusion. Its first weights (v1.1 to v1.4) were released for download through the CompVis GitHub and Hugging Face pages on or around 22 August 2022 (Getty Images v Stability AI, ¶34). The CompVis README describes it as building on this paper, trained on a subset of LAION-5B with a frozen CLIP ViT-L/14 text encoder in place of the paper’s jointly trained transformer.

Limitations

The arXiv v2 text (§5) and the CVPR supplement (Appendix G) state the same limitations. Sampling, though cheaper than in pixel space, is still sequential and slower than in GANs. The autoencoder’s reconstruction ability can become a bottleneck for tasks that need pixel-level accuracy, and the authors expect their super-resolution models to be somewhat limited by it.

The societal-impact paragraphs note that cheaper generative models widen access and also make it easier to create manipulated images, misinformation and spam; that deep fakes disproportionately affect women; that generative models can reveal training data collected without consent, to an extent not yet understood for image diffusion models; and that models can reproduce or amplify dataset biases, and how far the paper’s own mix of adversarial and likelihood training misrepresents the data is open.

What Came After

Later latent models kept the two-stage split and changed its parts. SDXL (Podell et al., 2024) enlarged the U-Net, added a second text encoder and retrained the autoencoder. The Diffusion Transformer of Peebles and Xie (2023) replaced the U-Net with a transformer over patches of Stable Diffusion’s latents. Stable Diffusion 3 (Esser et al., 2024) combined a multimodal transformer with rectified flow and a 16-channel autoencoder. The latent diffusion model article summarizes these architectures, the Stable Diffusion article the released models, and text-to-image generation the wider field.