Generative Model

A generative model learns the probability distribution that a collection of images was drawn from, so that it can produce new images that look as if they came from the same collection, and in some cases say how probable a given image is. Where a classifier learns $p(y \mid x)$, the probability of a label given an image, a generative model learns $p(x)$ itself, or $p(x \mid y)$ when the image must match a label, a sentence or another image. Image generation and its conditional variants are built on generative models, and models with an explicit density are also used for compression and denoising (Theis et al., 2016).

This article orients the reader across the main families of image generative models and the distinctions that cut across them. Each family has its own article: the variational autoencoder, the generative adversarial network (GAN) and the diffusion model, with the autoencoder and the evidence lower bound behind the first.

Definition

A generative model of images is a probability distribution $p_\theta$ over images, with parameters $\theta$ fitted so that $p_\theta$ is close to the unknown distribution $p_{\text{data}}$ of the training images. It is given either by a function that evaluates the probability (or density) of any image, an explicit model, or only by a procedure that draws samples, an implicit model. A conditional generative model $p_\theta(x \mid y)$ takes an extra input $y$, such as a class label or a caption, and generates images consistent with it.

Intuition

Think of all possible $64 \times 64$ color images. Almost all of them are noise; the photographs of faces, rooms or digits occupy a vanishingly small, highly structured region of that space. A generative model has to put its probability on that region and nowhere else. A model that spreads probability too widely produces blurry or noisy samples; one that concentrates it too narrowly produces convincing images but only a few kinds of them, a failure called mode collapse.

The families differ in how they represent the distribution: as a product of simple pieces that scores any image exactly, as a random code transformed into an image, or as the undoing of a gradual corruption by noise. Each choice makes some operations cheap (training, sampling, scoring) and others hard.

Formal Definition

Training images $x^{(1)}, \dots, x^{(N)}$ are samples from $p_{\text{data}}$. An explicit model is fitted by maximum likelihood, maximizing the average log-probability of the training images:

$$\theta^\ast = \arg\max_\theta \; \mathbb{E}_{x \sim p_{\text{data}}} \big[ \log p_\theta(x) \big] = \arg\min_\theta \; \mathrm{KL}\big(p_{\text{data}} \,\Vert\, p_\theta\big).$$

The two forms are equal because the entropy of $p_{\text{data}}$ does not depend on $\theta$, the identity that links the cross-entropy loss to the KL divergence; the negative log-likelihood is a loss function on images instead of labels. Likelihoods are reported in bits per dimension, the negative log-likelihood in bits divided by the number of values in the image, $D = H \times W \times C$, so that images of different sizes can be compared. Pixel values are discrete, so a continuous density is evaluated after adding uniform noise to the integer values (dequantization); Theis, van den Oord and Bethge (2016, §3.1) show that the result bounds the log-likelihood of a discrete model.

An implicit model draws a latent code $z \sim p_z$, often a standard Gaussian, and returns $x = G_\theta(z)$ for a neural network $G_\theta$. The induced distribution of $x$ has no tractable density when $z$ has fewer dimensions than $x$ or $G_\theta$ is not invertible, so it must be trained by comparing samples with data instead of by likelihood.

The Main Families

Goodfellow (2016, §2.2, Figure 9) arranges deep generative models by how they represent the likelihood: a tractable explicit density, an explicit density that must be approximated, or an implicit density reached only through samples, the branch where GANs sit. Diffusion models, which matured later, belong in our reading to the approximate-density branch, since they are trained on a variational bound.

Family	Density	Trained by	Network passes per sample	Examples
Autoregressive	Explicit, exact	Maximum likelihood via the chain rule	One per pixel or token	PixelRNN, PixelCNN, DALL·E
Variational autoencoder	Explicit, lower bound	Maximizing the ELBO	One decoder pass	VAE, NVAE
Normalizing flow	Explicit, exact	Maximum likelihood via change of variables	One inverse pass	Real NVP, Glow
Generative adversarial network	Implicit	Adversarial game with a discriminator	One generator pass	GAN, StyleGAN, BigGAN
Diffusion and score-based	Explicit, bound (exact with an ODE sampler)	Denoising, a reweighted ELBO	Tens to thousands of denoising steps	DDPM, score SDE, latent diffusion

Autoregressive models factor the image into a product of conditionals over pixels in a fixed order, usually row by row:

$$p_\theta(x) = \prod_{i=1}^{D} p_\theta(x_i \mid x_1, \dots, x_{i-1}).$$

PixelRNN and PixelCNN (van den Oord, Kalchbrenner & Kavukcuoglu, 2016) predict each color value with a 256-way softmax given the pixels above and to the left; the best PixelRNN reached 3.00 bits per dimension on the CIFAR-10 test set (their Table 5). The likelihood is exact, but sampling generates one value at a time, and those steps cannot be parallelized (Goodfellow, 2016, §2.3.1). Later models apply the same factorization to discrete image tokens produced by an autoencoder, with a transformer predicting the next token.

Variational autoencoders (Kingma & Welling, 2014) introduce a latent code $z$ with prior $p(z) = \mathcal{N}(0, I)$, a decoder $p_\theta(x \mid z)$ and an encoder $q_\phi(z \mid x)$, the encoder–decoder shape of an autoencoder made probabilistic. The likelihood $p_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dz$ is intractable, so training maximizes the evidence lower bound:

$$\mathbb{E}_{q_\phi(z \mid x)}\big[\log p_\theta(x \mid z)\big] - \mathrm{KL}\big(q_\phi(z \mid x) \,\Vert\, p(z)\big) \le \log p_\theta(x).$$

Sampling decodes one draw of $z$. Goodfellow (2016, §2.3.2) notes that VAEs are regarded as producing lower-quality samples, a subjective opinion rather than a measured fact.

Normalizing flows transform the image through an invertible network $f_\theta$ to a latent $z = f_\theta(x)$ with a simple density, and score the image by the change-of-variables formula $\log p_\theta(x) = \log p(z) + \log \lvert \det \partial f_\theta / \partial x \rvert$. Rezende and Mohamed (2015) named the approach, for approximate posteriors in variational inference; Real NVP (Dinh, Sohl-Dickstein & Bengio, 2017) and Glow (Kingma & Dhariwal, 2018) built image models with exact likelihood and one-pass sampling, at 3.49 and 3.35 bits per dimension on CIFAR-10 (Kingma & Dhariwal, 2018, Table 2). The price is architectural: every layer must be invertible with a cheap Jacobian determinant, and $z$ must have as many dimensions as $x$ (Goodfellow, 2016, §2.3.1).

Generative adversarial networks (Goodfellow et al., 2014) train a generator $G$ against a discriminator $D$ that tries to tell real images from generated ones, with the minimax value

$$V(D, G) = \mathbb{E}_{x}\big[\log D(x)\big] + \mathbb{E}_{z \sim p_z}\big[\log\big(1 - D(G(z))\big)\big].$$

With the optimal discriminator, the generator minimizes a Jensen–Shannon divergence between data and model, and in practice the authors train $G$ to maximize $\log D(G(z))$ instead, for stronger early gradients. A GAN is implicit: it samples in one forward pass but assigns no likelihood.

Diffusion and score-based models corrupt an image $x_0$ with Gaussian noise over $T$ steps, so that at step $t$

$$x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I),$$

and train a network $\epsilon_\theta(x_t, t)$, usually a U-Net, to predict the noise; sampling starts from pure noise and removes it step by step, each step a learned form of image denoising. Sohl-Dickstein et al. (2015) introduced the forward and learned reverse diffusion; Ho, Jain and Abbeel (2020) made it competitive in sample quality with 1,000 steps and a simple noise-prediction loss. Song and Ermon (2019) instead estimated the score, the gradient of the log-density of noise-perturbed data, and Song et al. (2021) unified the two views with stochastic differential equations, including a deterministic ordinary differential equation (ODE), the probability flow, that gives exact likelihoods.

Properties

Explicit and implicit density

An explicit density allows likelihood training, comparison on held-out data and uses that need probabilities, such as lossless compression, at an architectural price: autoregressive models fix an order, flows require invertibility, VAEs and diffusion models settle for a bound. Implicit models put very few restrictions on the generator (Goodfellow, 2016, §2.5) but reach their distribution only through samples (§2.4), so they are trained and evaluated through samples.

Likelihood and sample quality

Theis, van den Oord and Bethge (2016) showed that average log-likelihood and the visual quality of samples are largely independent for high-dimensional images. A lookup table that replays training images produces perfect samples but a poor likelihood on unseen test images. Conversely, in an argument they credit to van den Oord and Dambre (2015), if $p$ is an excellent model and $q$ is white noise, the mixture $0.01\,p + 0.99\,q$ produces noise 99% of the time, yet

$$\log\big[0.01\,p(x) + 0.99\,q(x)\big] \ge \log p(x) - \log 100,$$

a loss of at most $\log 100 \approx 4.61$ nats per image, while log-likelihoods of different models on $32 \times 32$ CIFAR-10 images can differ by thousands of nats (Theis et al., 2016, §3.2). A good likelihood therefore neither requires nor guarantees good samples, and the authors concluded that a model should be evaluated on the application it is meant for. Their Figure 1 shows the training-objective side of the same trade-off: fitting a single Gaussian to a mixture by maximum likelihood (KL from data to model) covers every mode and puts mass between them, while the Jensen–Shannon divergence associated with GANs locks onto one mode.

Published results show the effect. In the CIFAR-10 results of Ho et al. (2020, Table 1, arXiv version), Gated PixelCNN (van den Oord et al., 2016) has a test negative log-likelihood of 3.03 bits per dimension and a Fréchet Inception Distance of 65.93, while DDPM trained with its simplified loss has a weaker likelihood bound, at most 3.75 bits per dimension, and an FID of 3.17. DDPM trained on the true variational bound improves the bound to 3.70 but worsens the FID to 13.51. The authors found that more than half of the lossless codelength of their best-sample model describes imperceptible details.

Quality, diversity and speed

Xiao, Kreis and Vahdat (2022) call the difficulty of combining high sample quality, mode coverage and fast sampling the generative learning trilemma: in their account GANs lack coverage, VAEs and flows sample quality, and diffusion models speed. Their Table 1 (arXiv v2) collects unconditional CIFAR-10 results (FID against the training set; recall measures coverage; time is for a batch of 100 images on one V100 GPU); a selection:

Family	Model	FID ↓	Recall ↑	Network evaluations	Time (s)
GAN	StyleGAN2 with ADA	2.92	0.49	1	0.04
Diffusion	DDPM	3.21	0.57	1000	80.5
Diffusion, fast sampler	DDIM, 50 steps	4.67	0.53	50	4.01
Diffusion + GAN	Denoising diffusion GAN, 4 steps	3.75	0.57	4	0.21
VAE	NVAE	23.5	0.51	1	0.36
Flow	Glow	48.9	—	1	—
Autoregressive	PixelCNN	65.9	—	1024	—

Xiao et al. print DDPM’s FID as 3.21, while the DDPM paper’s own Table 1 prints 3.17; Xiao et al. do not explain the difference. Much later work attacks a family’s weak corner, such as DDIM, whose authors report high-quality samples 10 to 50 times faster in wall-clock time than DDPM (Song, Meng & Ermon, 2021), and latent diffusion, which runs diffusion in the smaller latent space of a pretrained autoencoder (Rombach et al., 2022).

Many models also expose a quality–diversity knob at sampling time: BigGAN’s truncation of the latent, low-temperature sampling in likelihood models, and guidance in diffusion models (Dhariwal & Nichol, 2021; Ho & Salimans, 2022). On ImageNet $256 \times 256$, Dhariwal and Nichol (2021, Table 5, arXiv v4) report BigGAN-deep at FID 6.95, precision 0.87 and recall 0.28, and their classifier-guided diffusion model, sampled with 250 steps, at FID 4.59, precision 0.82 and recall 0.52: in our reading, slightly less faithful samples but much better coverage.

Conditional and unconditional generation

Every family has a conditional form $p_\theta(x \mid y)$. Mirza and Osindero (2014) built conditional GANs by feeding the condition $y$, for example an MNIST class label, to both the generator and the discriminator; Gated PixelCNN conditioned an autoregressive model on ImageNet class labels or on a face embedding; DALL·E (Ramesh et al., 2021) modeled caption tokens and $32 \times 32 = 1{,}024$ image tokens as one autoregressive sequence; latent diffusion added cross-attention layers so that text, bounding boxes or other inputs could condition the denoiser (Rombach et al., 2022). The condition can itself be an image, as in inpainting and image-to-image translation. Guidance pushes samples further toward the condition at the cost of diversity; classifier-free guidance trains one network both with the condition and with a null token in its place, and combines the two predictions at sampling time (Ho & Salimans, 2022).

Evaluation

Likelihood in bits per dimension is the standard measure for explicit models, but it says little about samples, and implicit models have none. Sample-based metrics compare generated and real images in the feature space of a pretrained classifier. A widely used one is the Fréchet Inception Distance (Heusel et al., 2017), which fits Gaussians to Inception-v3 features of real and generated images and measures the distance between them; lower is better. FID folds quality and diversity into one number. Precision and recall for distributions separate them: precision is the fraction of generated images that fall within the support of the real images, recall the fraction of real images within the support of the generated ones (Kynkäänniemi et al., 2019). Theis et al. (2016) recommended against Parzen-window estimates of likelihood from samples, which in their MNIST experiment ranked a $k$-means lookup model above samples from the true distribution.

History

Before deep learning. Boltzmann machines, which Goodfellow (2016, §2.3.2) traces to work from 1983 to 1986, rely on Markov chains both for training and for sampling, and such chains become less efficient in high dimensions. In computer vision, Efros and Leung (1999) grew textures one pixel at a time, sampling each pixel from a conditional distribution estimated by finding similar neighborhoods in an example image: in our reading, a non-parametric autoregressive sampler. Deep belief networks (Hinton, Osindero & Teh, 2006), trained greedily one layer at a time, gave a good generative model of handwritten digit images and their labels.

2014: backpropagation-trained generators. The VAE and the GAN both dispensed with Markov chains and trained deep generators end to end with backpropagation, one by a variational bound and one by an adversarial game. GANs came to be subjectively regarded as producing better samples than other methods (Goodfellow, 2016, §2.5), and StyleGAN (Karras, Laine & Aila, 2019) generated high-resolution faces with scale-specific control; their weakness was a game that may not converge, with mode collapse, in Goodfellow’s words (§5.1), probably its most common harmful form.

2015–2018: exact likelihoods. Autoregressive models (PixelRNN, PixelCNN) and flows (Real NVP, Glow) gave exact likelihoods, but autoregressive sampling was slow and the samples of both lagged GANs in FID. Sohl-Dickstein et al. (2015) introduced diffusion probabilistic models in the same period.

2019–2021: diffusion matures. Score-based models (Song & Ermon, 2019), whose authors reported samples comparable to GANs, and DDPM (Ho et al., 2020) reached strong sample quality on CIFAR-10 without adversarial training, and Dhariwal and Nichol (2021) beat BigGAN-deep on ImageNet with an improved architecture and classifier guidance.

2021–2022: generation from text. DALL·E generated images from text autoregressively, and latent diffusion added cross-attention conditioning in an autoencoder’s latent space. As of October 2026, the CompVis repository describes Stable Diffusion as a latent text-to-image diffusion model that builds on that work, and states that its v1.3 and v1.4 checkpoints were trained with 10% of text conditioning dropped to improve classifier-free guidance sampling.

Examples

Likelihood versus sample quality on binary bar images. The images are $6 \times 6$ binary “bars”: each is horizontal or vertical, each of its six lines is on with probability 1/2, and then every pixel is flipped with probability 0.05. Because $p_{\text{data}}$ is known exactly (written p* in the code), we can score both directions: the test NLL, $-\log_2 q(x)$ averaged over test images, measures how well a model $q$ explains real images, and bits under p*, $-\log_2 p_{\text{data}}(x)$ averaged over the model’s samples, measures how typical its samples are. A sample is counted as plausible if it is within three flipped pixels of a clean bar image, and coverage counts how many of the 126 distinct clean images its plausible samples land near; these two play the roles of precision and recall. Two explicit models are fitted by maximum likelihood, independent pixels and a mixture of 128 independent-pixel images (a latent-variable model fitted by expectation-maximization), and compared with Theis et al.’s noise mixture and two implicit samplers that have no density.

import sys
import numpy as np

rng = np.random.default_rng(int(sys.argv[1]) if len(sys.argv) > 1 else 0)
S, EPS = 6, 0.05  # 6x6 binary images; each pixel is flipped with probability 0.05
D = S * S

# The 128 clean "bars" images (126 distinct): horizontal or vertical, each of the
# 6 lines switched on with probability 1/2.
lines = (np.arange(2**S)[:, None] >> np.arange(S)) & 1
horiz = np.repeat(lines[:, :, None], S, axis=2).reshape(-1, D)
vert = np.repeat(lines[:, None, :], S, axis=1).reshape(-1, D)
clean = np.vstack([horiz, vert]).astype(float)
distinct = np.unique(clean, axis=0)

def hamming(x, y):  # pairwise number of differing pixels between binary images
    return x @ (1 - y).T + (1 - x) @ y.T

def sample_true(n):
    x = clean[rng.integers(0, len(clean), n)]
    return np.abs(x - (rng.random((n, D)) < EPS))

def log2_true(x):  # exact log2 p*(x): the noise likelihood averaged over the clean images
    d = hamming(x, clean)
    ll = d * np.log(EPS) + (D - d) * np.log(1 - EPS)
    return (np.logaddexp.reduce(ll, axis=1) - np.log(len(clean))) / np.log(2)

train, test = sample_true(5000), sample_true(5000)

# Explicit model 1: independent pixels. Maximum likelihood sets each pixel's mean.
m = train.mean(0).clip(1e-3, 1 - 1e-3)
def log2_indep(x): return x @ np.log2(m) + (1 - x) @ np.log2(1 - m)
def samp_indep(n): return (rng.random((n, D)) < m).astype(float)

# Explicit model 2: a mixture of K independent-pixel images, a latent-variable
# model; maximum likelihood by expectation-maximization (EM).
K = 128
pi, mu = np.full(K, 1 / K), rng.uniform(0.3, 0.7, (K, D))
def comp_ll(x): return x @ np.log(mu).T + (1 - x) @ np.log(1 - mu).T
for _ in range(100):
    lr = comp_ll(train) + np.log(pi)
    r = np.exp(lr - np.logaddexp.reduce(lr, axis=1, keepdims=True))
    nk = r.sum(0) + 1e-9
    pi, mu = nk / nk.sum(), (r.T @ train / nk[:, None]).clip(1e-3, 1 - 1e-3)
def log2_mix(x): return np.logaddexp.reduce(comp_ll(x) + np.log(pi), axis=1) / np.log(2)
def samp_mix(n): return (rng.random((n, D)) < mu[rng.choice(K, n, p=pi)]).astype(float)

# Theis et al.'s construction: the mixture 1% of the time, white noise 99%.
def log2_theis(x): return np.logaddexp2(np.log2(0.01) + log2_mix(x), np.log2(0.99) - D)
def samp_theis(n):
    noise = (rng.random((n, D)) < 0.5).astype(float)
    return np.where(rng.random((n, 1)) < 0.01, samp_mix(n), noise)

# Implicit samplers, with no density: a lookup table that replays training images,
# and a collapsed generator that only ever draws 4 clean horizontal-bar images.
def samp_lookup(n): return train[rng.integers(0, len(train), n)]
def samp_collapsed(n): return horiz[rng.choice([3, 12, 33, 63], n)].astype(float)

def sample_stats(x):
    bits = -log2_true(x).mean()           # how probable the samples are under p*
    d = hamming(x, distinct)
    near = d.min(1) <= 3                  # within 3 flipped pixels of a bars image
    covered = len(np.unique(d.argmin(1)[near]))
    return bits, 100 * near.mean(), covered

print(f"entropy of p*: {-log2_true(test).mean():.1f} bits per image (test-set estimate)")
print(f"{'model':24s}{'test NLL':>9s}{'bits under p*':>15s}{'plausible':>11s}{'coverage':>10s}")
models = [("true distribution p*", log2_true, sample_true),
          ("independent pixels", log2_indep, samp_indep),
          (f"mixture, K={K}", log2_mix, samp_mix),
          ("1% mixture + 99% noise", log2_theis, samp_theis),
          ("lookup table", None, samp_lookup),
          ("collapsed generator", None, samp_collapsed)]
for name, log2q, sampler in models:
    nll = f"{-log2q(test).mean():9.1f}" if log2q else f"{'-':>9s}"
    bits, plaus, cov = sample_stats(sampler(10000))
    print(f"{name:24s}{nll}{bits:15.1f}{plaus:10.1f}%{cov:6d}/126")

seen = {img.tobytes() for img in train}
print(f"test images missing from the lookup table: {np.mean([t.tobytes() not in seen for t in test]):.0%}")

Output:

entropy of p*: 17.2 bits per image (test-set estimate)
model                    test NLL  bits under p*  plausible  coverage
true distribution p*         17.2           17.2      89.6%   126/126
independent pixels           36.0           56.2       0.0%     0/126
mixture, K=128               19.5           19.3      78.7%   126/126
1% mixture + 99% noise       25.7           55.7       0.8%    58/126
lookup table                    -           17.2      90.3%   126/126
collapsed generator             -            9.4     100.0%     4/126
test images missing from the lookup table: 75%

The true distribution sets the reference: 17.2 bits per image in both directions, and 89.6% plausible, because about a tenth of real images have four or more flipped pixels. The independent-pixel model cannot represent “a whole row is on”, so it has the worst likelihood and its samples are speckle. The mixture comes within 2.3 bits of the entropy and its samples cover every bar image. Mixing it with 99% noise costs 6.2 bits, inside the bound of $\log_2 100 = 6.64$ bits, and leaves its likelihood 10 bits better than the independent-pixel model’s, while 99% of its samples are noise.

The implicit samplers fail in the other direction. The lookup table’s samples are as good as real ones, but read as a density it gives probability zero to the 75% of test images it never stored, an infinite test NLL. The collapsed generator scores best of all on bits under $p_{\text{data}}$, 9.4 bits against the true distribution’s 17.2, because it emits only the most probable images; only coverage, 4 of 126, exposes it. In our reading, the two columns are the two directions of cross-entropy: test NLL estimates $H(p_{\text{data}}) + \mathrm{KL}(p_{\text{data}} \Vert q)$ and punishes missing modes, while bits under $p_{\text{data}}$ estimate $H(q) + \mathrm{KL}(q \Vert p_{\text{data}})$ and reward a low-entropy, collapsed $q$. The script takes a seed as its argument; over seeds 0 to 15, the mixture’s test NLL ranged from 19.2 to 19.8 bits, its bits under $p_{\text{data}}$ from 18.5 to 19.8 and its plausible rate from 75.5% to 82.7%; the noise mixture’s NLL ranged from 25.4 to 26.0 bits and its coverage from 50 to 65; and every other entry moved by at most 0.4 bits, 2.1 percentage points or one covered image. It runs in about 5 seconds.

Common Misconceptions

• “Better samples mean a better model of the data.” Samples show what a model generates, not what it misses; a lookup table or a collapsed generator produces excellent samples. Evaluate coverage as well as quality, and likelihood where it matters.
• “A model with good likelihood generates good images.” The noise-mixture construction loses at most $\log 100$ nats while generating noise 99% of the time (Theis et al., 2016).
• “Diffusion models have no likelihood.” DDPM reports variational bounds in bits per dimension, and the probability-flow ODE of Song et al. (2021) gives exact likelihoods, 2.99 bits per dimension on uniformly dequantized CIFAR-10. GANs are the family without one.
• “A lower FID means more diverse samples.” FID mixes fidelity and diversity: classifier guidance cut Dhariwal and Nichol’s ImageNet $256 \times 256$ FID from 10.94 to 4.59 while recall fell from 0.63 to 0.52 (their Table 5), a trade-off precision and recall expose.

Where It Is Used

Generative models underlie image generation, text-to-image generation, inpainting and image-to-image translation. Their components reappear elsewhere: denoising networks in image denoising, autoencoders in masked autoencoders and self-supervised learning, and text conditioning with CLIP text encoders in multimodal systems.