Image Generation

Image generation, also called image synthesis, is the task of producing new images that look as if they were drawn from the distribution of a training collection. In the unconditional form the generator receives only random numbers; in the class-conditional form it also receives a label, such as an ImageNet class. Image generative models are developed and compared on it; that article explains the model families, and this one the task: what is asked, how it is measured, and how it fails. Text-to-image generation, image-to-image translation and inpainting build on it with richer inputs.

Problem Definition

Training images $x^{(1)}, \dots, x^{(N)}$, optionally with labels $y^{(i)} \in \{1, \dots, K\}$, are samples from an unknown distribution $p_{\text{data}}(x)$ or $p_{\text{data}}(x \mid y)$. The task is to build a sampler whose outputs follow that distribution, which asks for three things:

• Fidelity: each sample looks like a real image of the domain and of its class.
• Diversity, or coverage: the samples cover the whole distribution in proportion, not a few typical images.
• Novelty: the samples are not copies of training images.

In our reading, novelty separates the task from replaying the training set, which reproduces the empirical distribution exactly. No input has a single correct output, so evaluation compares sets of generated and real images.

Input and Output

A trained generator is a function of noise and, for the conditional task, a label:

$$x = G_\theta(z, y), \qquad z \sim \mathcal{N}(0, I).$$

A GAN maps a short vector to an image in one pass: 120 to 160 dimensions in BigGAN, depending on resolution (Brock, Donahue & Simonyan, 2019, Appendix B, arXiv v2), and 512 in StyleGAN (Karras, Laine & Aila, 2019, §2). A diffusion model starts from noise shaped like its output, the image or, in a latent diffusion model, the latent ($32 \times 32 \times 4$ for a $256 \times 256 \times 3$ image in DiT; Peebles & Xie, 2023, §3.2, arXiv v2), and stochastic samplers add fresh noise at every step.

Variants

• Unconditional generation is the usual protocol on single-domain datasets, such as faces or bedrooms, and on CIFAR-10.
• Class-conditional generation is the usual protocol on ImageNet. The label also enables sampling-time knobs, truncation and guidance, that push samples toward the class.
• Resolution scale. Each resolution, from $32$ to $1024$ pixels, is a separate benchmark: BigGAN-deep’s FID without truncation was 5.7 at $128$, 6.9 at $256$ and 7.5 at $512$ pixels (Brock et al., Table 2, arXiv v2).
• Richer conditions. Text-to-image generation conditions on a caption, image-to-image translation on an input image, and inpainting on the visible part of an image; each adds agreement with the condition as a requirement.

Challenges

Fidelity against diversity. BigGAN’s truncation of $z$ and classifier or classifier-free guidance trade one for the other at sampling time. Raising the classifier scale traded recall for precision, so FID, which depends on both, was best at an intermediate scale (Dhariwal & Nichol, 2021, §4.3, arXiv v4). Unguided, DiT-XL/2 scored FID 9.62 with precision 0.67 and recall 0.67 on ImageNet $256 \times 256$; at guidance scale 1.5 it scored FID 2.27, precision 0.83 and recall 0.57 (Peebles & Xie, Table 2, arXiv v2).

Memorization. Carlini et al. (2023) count a training image as extracted when a generation lies within a small normalized pixel distance of it (§4.1). From 175 million Stable Diffusion generations, 500 for each of the 350,000 most-duplicated training captions, they extracted 94 images, or 107 counting 13 more near-copies judged by eye (§4.2.2; arXiv v1 prints 109). From 16 diffusion models trained on CIFAR-10, sampled $2^{20}$ times in all, they extracted 1,280 training images, about 2.5% of the set (§5.1). The best DDPM model memorized twice as many images as a StyleGAN-ADA of similar FID, and memorization grew as FID improved (§6). Somepalli et al. (2023, CVPR version) found blatant copying in diffusion models trained on 300 or 3,000 CelebA images, but no exact copies from a class-conditional latent diffusion model trained on ImageNet (§5–§6). In Stable Diffusion, about 1.88% of random generations exceeded a similarity of 0.5, above which they saw substantial copying, in a search of less than 0.6% of its training data (§7–§8). In our reading, the nearest-neighbour spot checks of a few samples in BigGAN (Appendix A) and ADM (Appendix C) could not detect copying at these rates.

High resolution. Karras et al. (2018, §6.3) note that earlier GAN datasets ranged from $32^2$ to $480^2$ pixels, and built CelebA-HQ and progressive growing to reach $1024^2$. BigGAN’s authors found classes needing global structure, such as crowds, harder than textural ones such as dogs (§5.1 and Fig. 7, arXiv v2). Cascades generate small images and upsample them: a guided $64$-pixel ADM followed by an upsampler reached FID 3.94 at $256$ pixels, against 4.59 for the single-stage guided model (Dhariwal & Nichol, Table 6, arXiv v4).

Compute. Peebles and Xie (§5.1) count 1,120 Gflops per forward pass for ADM at $256 \times 256$, against 103.6 for LDM-4 and 118.6 for DiT-XL/2, and the diffusion samples in the table below take 250 denoising steps each. Diffusion sampling covers samplers that need fewer steps.

Approaches

The hub compares the families. Likelihood models, among them variational autoencoders, lagged in sample quality, while GANs led through progressive growing, BigGAN and StyleGAN. Diffusion models reached strong sample quality on CIFAR-10 without adversarial training with DDPM (2020) and passed BigGAN-deep’s ImageNet FID, with better recall, in 2021 (Dhariwal & Nichol). Latent diffusion and transformer denoisers (DiT) then reduced their cost.

Datasets and Benchmarks

Dataset	Images	Resolution used	Labels	Notes
CIFAR-10	60,000 (50,000 train, 10,000 test)	$32 \times 32$	10 classes	Generated unconditionally in DDPM and the hub’s comparison
ImageNet (ILSVRC-2012)	1,281,167 train, 50,000 validation (Russakovsky et al., 2015, Table 2)	$64$, $128$, $256$, $512$	1,000 classes	Class-conditional; images resized from varying sizes
CelebA-HQ	30,000	$1024 \times 1024$	none used	Selected from CelebA’s 202,599 images
FFHQ	70,000	$1024 \times 1024$	none	Flickr faces, PNG, non-commercial licence
LSUN	about 10 million (10 scene categories), 59 million (20 object categories)	$256 \times 256$	one category per model	Labels from a partly automated pipeline

CIFAR-10 (Krizhevsky, 2009) contains near-duplicates: deduplicating at an image similarity of 0.85 removed 5,275 of its 50,000 training images (Carlini et al., §7.1). Prepared $64 \times 64$ ImageNet versions exist without labels (van den Oord, Kalchbrenner & Kavukcuoglu, 2016) and with them (Chrabaszcz, Loshchilov & Hutter, 2017). CelebA-HQ keeps the best 30,000 of all 202,599 processed CelebA images, ranked by a frequency-based quality measure that penalizes blur (Karras et al., 2018, Appendix C). FFHQ’s faces were crawled from Flickr, aligned and cropped (Karras, Laine & Aila, 2019, Appendix A). As of October 2026 its README states that the images carry permissive licences, the dataset is released under CC BY-NC-SA 4.0, it inherits Flickr’s biases, and it is not intended for developing facial recognition. LSUN (Yu et al., 2015) was labelled by alternating human annotation and classification, with label precision around 90% on 11 sampled object categories (arXiv v3); generative models train on one category, such as bedrooms or churches.

Evaluation Metrics

• Fréchet Inception Distance (Heusel et al., 2017) is the default, comparable only under one protocol: Dhariwal and Nichol (2021, §2.2) recompute other models’ scores with one codebase against the whole training set where samples or models are public, and DiT uses that code on 50,000 samples (§4).
• Inception Score (Salimans et al., 2016) scores generated images alone. Printed values for one model can disagree: for BigGAN-deep at $256 \times 256$ with FID 6.95, DiT’s Table 2 prints IS 171.4 and LDM’s Table 3 prints 203.6.
• Precision and recall (Kynkäänniemi et al., 2019) measure fidelity and coverage separately.
• Memorization checks search the training set for each sample’s nearest neighbour, in pixels (Carlini et al.) or in copy-detection features (Somepalli et al.); benchmark tables do not include them.
• Human evaluation. In Stein et al.’s (2023) study of 41 models, no existing metric correlated strongly with human judgment.

State of the Art on ImageNet 256×256

FIDs as printed by each paper (lower is better): representative steps, not a leaderboard.

Model	Year	Source (table, version)	FID unguided	FID as reported best	Precision	Recall
BigGAN-deep	2019	Brock et al., Table 2, arXiv v2	6.9 (no truncation)	6.9	—	—
BigGAN-deep, re-evaluated	2021	Dhariwal & Nichol, Table 5, arXiv v4	6.95	6.95	0.87	0.28
ADM-G (classifier guidance)	2021	Dhariwal & Nichol, Table 5, arXiv v4	10.94	4.59	0.82	0.52
LDM-4-G (classifier-free, scale 1.5)	2022	Rombach et al., Table 3, arXiv v2	10.56	3.60	0.87	0.48
DiT-XL/2-G (classifier-free, scale 1.5)	2023	Peebles & Xie, Table 2, arXiv v2	9.62	2.27	0.83	0.57

The diffusion models use 250 sampling steps. In our reading, BigGAN’s Table 2 is inconsistent: its column of scores at the best FID prints 7.0 ± 0.1, above the untruncated 6.9 ± 0.2. Without guidance the three diffusion models score about 10; the headline numbers are guided operating points with lower recall. DiT’s Table 2 also lists a GAN, StyleGAN-XL, at 2.30.

Example: No Single Number Catches Every Failure

The data are $16 \times 16$ images of four shape classes (disk, square, ring, cross) with random position, size, brightness and noise. Five stand-in generators each behave in one way: an ideal sampler draws fresh images; a memorizer replays training images; a 10% memorizer replays them one time in ten; a mode dropper draws only disks and squares; a blurry averager returns the mean of four same-class training images. Each is scored on 1,000 samples against 1,000 held-out real images by a Fréchet distance (FD) and $k$-nearest-neighbour precision and recall ($k = 3$), both in a 32-dimensional PCA space of the training images standing in for a pretrained embedding, and by a memorization check that flags a sample as a copy when it is closer to its nearest training image, in per-pixel RMS distance, than 99% of held-out real images are to theirs.

import sys
import numpy as np
from scipy import linalg

rng = np.random.default_rng(int(sys.argv[1]) if len(sys.argv) > 1 else 0)
H = 16
yy, xx = np.mgrid[:H, :H]

def draw(kind):
    """One 16x16 image of a shape class: 0 disk, 1 square, 2 ring, 3 cross."""
    cy, cx = 7.5 + rng.uniform(-3, 3, 2)
    r = rng.uniform(2.5, 4.5)
    dy, dx = yy - cy, xx - cx
    if kind == 0:   m = dy**2 + dx**2 <= r**2
    elif kind == 1: m = (abs(dy) <= r * 0.8) & (abs(dx) <= r * 0.8)
    elif kind == 2: m = abs(np.hypot(dy, dx) - r) <= 0.9
    else:           m = ((abs(dy) <= 0.9) | (abs(dx) <= 0.9)) & (abs(dy) <= r) & (abs(dx) <= r)
    img = rng.uniform(0.6, 1.0) * m + rng.normal(0, 0.05, (H, H))
    return np.clip(img, 0, 1).ravel()

def real(n, classes=(0, 1, 2, 3)):
    labels = rng.choice(classes, n)
    return np.stack([draw(k) for k in labels]), labels

train, ytr = real(2000)          # what the "generators" were trained on
ref, _ = real(1000)              # held-out real images used for evaluation

# Five generators (no networks: each is a stand-in for one behaviour)
def g_ideal(n):     return real(n)[0]                        # fresh draws from p_data
def g_memorize(n):  return train[rng.integers(0, len(train), n)]  # replays training images
def g_partial(n):                                             # 90% fresh, 10% replayed
    x = g_ideal(n); k = n // 10
    x[:k] = train[rng.integers(0, len(train), k)]; return x
def g_dropmodes(n): return real(n, classes=(0, 1))[0]          # only disks and squares
def g_average(n):                                             # mean of 4 same-class training images
    out = []
    for _ in range(n):
        c = rng.integers(4); idx = rng.choice(np.flatnonzero(ytr == c), 4, replace=False)
        out.append(train[idx].mean(0))
    return np.stack(out)

# Feature space: PCA of the training images (a stand-in for a pretrained embedding)
mu = train.mean(0)
_, _, Vt = np.linalg.svd(train - mu, full_matrices=False)
P = Vt[:32].T
feat = lambda x: (x - mu) @ P

def sqdist(a, b):
    return np.maximum((a**2).sum(1)[:, None] + (b**2).sum(1)[None] - 2 * a @ b.T, 0)

def frechet(fa, fb):
    m1, m2 = fa.mean(0), fb.mean(0)
    s1, s2 = np.cov(fa, rowvar=False), np.cov(fb, rowvar=False)
    return ((m1 - m2)**2).sum() + np.trace(s1 + s2 - 2 * linalg.sqrtm(s1 @ s2).real)

def knn_radius(f, k=3):
    d = sqdist(f, f); np.fill_diagonal(d, np.inf)
    return np.sort(d, 1)[:, k - 1]

def precision_recall(fr, fg, k=3):
    rr, rg = knn_radius(fr, k), knn_radius(fg, k)
    prec = (sqdist(fg, fr) <= rr[None]).any(1).mean()   # generated inside real manifold
    rec = (sqdist(fr, fg) <= rg[None]).any(1).mean()    # real inside generated manifold
    return prec, rec

def nn_train(x):  # per-pixel RMS distance to the nearest training image (Carlini-style l2)
    return np.sqrt(sqdist(x, train).min(1) / x.shape[1])

base = nn_train(ref)                 # how close genuinely new real images come to training
tau = np.quantile(base, 0.01)        # flag a sample closer than 99% of new real images
fref = feat(ref)
print(f"held-out real images: median NN distance to training {np.median(base):.3f}, "
      f"1% threshold {tau:.3f}")
print(f"{'generator':22s}{'FD':>7s}{'precision':>11s}{'recall':>8s}{'copies':>8s}{'median NN':>11s}")
for name, g in [("ideal sampler", g_ideal), ("memorizer", g_memorize),
                ("10% memorizer", g_partial), ("mode dropper", g_dropmodes),
                ("blurry averager", g_average)]:
    x = g(1000)
    fg = feat(x)
    p, r = precision_recall(fref, fg)
    d = nn_train(x)
    print(f"{name:22s}{frechet(fref, fg):7.2f}{p:11.2f}{r:8.2f}{(d < tau).mean():8.1%}{np.median(d):11.3f}")

Output (seed 0; NumPy 2.3.5, SciPy 1.16.3, about 1 second):

held-out real images: median NN distance to training 0.102, 1% threshold 0.042
generator                  FD  precision  recall  copies  median NN
ideal sampler            0.14       0.94    0.94    0.7%      0.105
memorizer                0.19       0.94    0.90  100.0%      0.000
10% memorizer            0.15       0.93    0.94   10.7%      0.102
mode dropper             1.05       0.95    0.50    1.4%      0.098
blurry averager          2.67       0.50    0.12    0.0%      0.132

Each failure shows up in a different column. The mode dropper keeps precision but halves recall; the blurry averager loses both, and FD ranks it worst. FD, precision and recall barely separate the memorizer from the ideal sampler, because replayed training images are real images; only the nearest-neighbour check exposes it. The 10% memorizer shows only in its flagged fraction, 10.7% against 0.7%. The script takes the seed as its argument. Over seeds 0 to 4, the ideal sampler’s FD ranged from 0.13 to 0.21, the memorizer’s from 0.16 to 0.23 and the 10% memorizer’s from 0.15 to 0.19 (0.15 against the ideal sampler’s 0.21 with seed 2); the non-copying generators had 0.0% to 2.3% flagged and the 10% memorizer 10.7% to 11.5%.

Applications

The task is mainly a research benchmark, where models and samplers are compared before conditional use. A trained unconditional model is also an image prior: Song et al. (2021) performed inpainting and colorization with one unconditional score-based model by conditioning its sampler. Somepalli et al. note the use of diffusion models in commercial art and graphic design. Synthetic images have been proposed as a privacy-preserving substitute for real data, a benefit that Carlini et al. note would be moot if the models reproduce their training data.

Open Problems

• Evaluation. FID mixes fidelity and diversity, depends on the protocol and tracks human judgment poorly (Stein et al., 2023), and the benchmark tables above report no measure of novelty.
• Memorization and privacy. Deduplicating CIFAR-10 reduced the images Carlini et al. extracted from 1,280 to 986, and differentially private training made their diffusion model diverge even at privacy budgets $\varepsilon \ge 50$ (§7.1–§7.2); they conclude that new advances in privacy-preserving training may be needed.
• Sampling cost. The diffusion results in the table use 250 denoising steps per image, against one network pass for a GAN.
• Data. Benchmarks inherit the biases and licences of their sources, such as Flickr for FFHQ, and LSUN’s labels are noisy.