Generative Adversarial Nets

“Generative Adversarial Nets” by Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio at the Université de Montréal was posted to arXiv on 10 June 2014 and published in Advances in Neural Information Processing Systems 27 (NIPS 2014). It trained a generator against a discriminator in a minimax game, proved that the game’s optimum over all distributions is the data distribution, and sampled digits, faces and small natural images in one forward pass. arXiv lists the preprint as “Generative Adversarial Networks”, but both versions print “Generative Adversarial Nets”. The paper received the NeurIPS 2024 Test of Time Award. The method as understood today is in the generative adversarial network article; this one covers what the 2014 paper did and reported.

The arXiv preprint (v1, the only version) differs from the NIPS paper only in its related work and a few sentences noted below; this article follows the NIPS version.

Problem

Deep learning’s successes by 2014 were discriminative models trained with backpropagation and dropout using piecewise linear units. The authors blamed the smaller impact of deep generative models on intractable maximum-likelihood computations and on the difficulty of using piecewise linear units generatively. Deep Boltzmann machines needed many approximations to the likelihood gradient; generative stochastic networks trained by exact backpropagation but still sampled with a Markov chain. The paper set out to remove the Markov chain.

Contribution

• An adversarial criterion. The generator learns only through a competing classifier: in the paper’s analogy, counterfeiters against the police, with competition improving both until the fakes are indistinguishable from genuine currency.
• A non-parametric analysis showing that the game’s global optimum is the data distribution and that idealized training converges to it.
• Adversarial nets, the case where both players are multilayer perceptrons, trained with backpropagation and dropout and sampled by forward propagation alone.

The NIPS version rewrote the preprint’s related work. It notes that the “discriminator” of noise-contrastive estimation is a ratio of two densities that must both be evaluated. Only the NIPS version discusses Schmidhuber’s predictability minimization, a regularizer over scalar outputs posed as an optimization rather than a game, and adversarial examples, an analysis tool rather than a training method. The preprint lists the stochastic backpropagation of Kingma and Welling and of Rezende et al. as related work; the NIPS version says the authors were unaware of it while developing GANs, and contrasts GANs, which cannot model discrete data, with VAEs, which cannot have discrete latent variables.

Method

A noise prior $p_z(z)$ is mapped to data space by a generator $G(z; \theta_g)$, whose samples define $p_g$, and a discriminator $D(x; \theta_d)$ outputs the probability that $x$ is real. The two play

$$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}(x)}\big[\log D(x)\big] + \mathbb{E}_{z \sim p_z(z)}\big[\log\big(1 - D(G(z))\big)\big].$$

Algorithm 1. Optimizing $D$ to completion at every step would be too expensive and would overfit, so training alternates $k$ minibatch gradient steps on $D$ with one on $G$, which keeps $D$ near its optimum as long as $G$ changes slowly; the preprint, not the NIPS version, likens this to SML/PCD, which carries Markov chain samples between learning steps. The experiments used $k = 1$, the cheapest choice, with momentum. The printed algorithm samples noise from “$p_g(z)$”, a typo for $p_z(z)$, and its generator step descends $\log(1 - D(G(z)))$, the minimax form, whereas §3 warns that this term saturates early in training and recommends maximizing $\log D(G(z))$. The released code follows the text: in its AdversaryCost2 class, used by the MNIST and CIFAR-10 configurations, the generator minimizes the discriminator’s sigmoid cross-entropy against the “real” label, $-\log D(G(z))$, and one SGD step updates both networks from the same minibatch.

Formal results. Section 4 treats $p_g$ as a free density. Proposition 1 gives the optimal discriminator for fixed $G$, $D^\ast_G(x) = p_{\text{data}}(x) / (p_{\text{data}}(x) + p_g(x))$. Substituting it gives the “virtual training criterion” $C(G) = \max_D V(D, G)$; Theorem 1 states that its global minimum, $-\log 4$, is reached if and only if $p_g = p_{\text{data}}$, since

$$C(G) = -\log 4 + 2 \cdot \mathrm{JSD}\big(p_{\text{data}} \,\Vert\, p_g\big).$$

Proposition 2 states that if $G$ and $D$ have enough capacity, $D$ reaches its optimum at each step, and $p_g$ is updated to improve the criterion, then $p_g$ converges to $p_{\text{data}}$; the proof uses the convexity of $\sup_D V$ in $p_g$ and sufficiently small updates. The authors flag the gap: training optimizes $\theta_g$ within a limited family, not $p_g$, so the proofs do not apply, though the practical performance of multilayer perceptrons makes them a reasonable choice. The preprint instead notes that an MLP generator introduces multiple critical points in parameter space. Figure 1 sketches the dynamics in one dimension.

Results

Adversarial nets were trained on MNIST, the Toronto Face Database (TFD) and CIFAR-10. The paper says only that generators mixed rectified linear and sigmoid units, discriminators used maxout units with dropout, and noise entered only at the generator’s bottom layer. The released mnist.yaml fills in the rest: 100 uniform noise variables pass through two rectified linear layers of 1,200 units to 784 sigmoid outputs; the discriminator has two maxout layers of 240 units with 5 pieces and a sigmoid output, with dropout (retention 0.8 at the input, 0.5 elsewhere); SGD uses batches of 100, an initial learning rate of 0.1 and momentum rising from 0.5 to 0.7.

Likelihood estimates. Lacking a tractable likelihood, the authors fitted a Gaussian Parzen window to generated samples $x_i = G(z_i)$ and reported the test log-likelihood under it, a procedure they credit to Breuleux, Bengio and Vincent (2011):

$$\log \hat p_\sigma(x) = \log \frac{1}{n} \sum_{i=1}^{n} \mathcal{N}\big(x;\, x_i,\, \sigma^2 I\big),$$

with $\sigma$ cross-validated. Their Table 1:

Model	MNIST	TFD
DBN	138 ± 2	1909 ± 66
Stacked CAE	121 ± 1.6	2110 ± 50
Deep GSN	214 ± 1.1	1890 ± 29
Adversarial nets	225 ± 2	2057 ± 26

MNIST errors are standard errors across test examples, TFD errors across folds. Adversarial nets led on MNIST but trailed the stacked contractive autoencoder on TFD. The authors conceded that the estimator has somewhat high variance and does poorly in high dimensions, but called it the best available and asked for better evaluation of models that sample without a likelihood. The paper gives neither the sample count nor the bandwidth grid; the released parzen_ll.py defaults to 10,000 samples and ten bandwidths from 0.1 to 1, chosen on 1,000 validation images for MNIST.

Samples. Figure 2 shows MNIST, TFD and CIFAR-10 samples, the last from a fully connected model and from one with a convolutional discriminator and “deconvolutional” generator: random draws, not cherry-picked, with nearest training examples alongside to argue against memorization. Figure 3 interpolates digits linearly in $z$. The authors claimed only that they were competitive with the better generative models of the time.

Table 2 compares four model families on training, inference, sampling, evaluating $p(x)$ and model design; for adversarial models it lists synchronizing $D$ with $G$ (“Helvetica”) as the training difficulty, no sampling difficulties, and $p(x)$ approximable only by Parzen estimation.

Worked Example: Parzen Estimates on Synthetic Images

The estimator’s error can be measured where the true likelihood is known. The images are $8 \times 8$ shapes in $[-1, 1]$ plus Gaussian pixel noise of standard deviation 0.2, a six-component Gaussian mixture with an exact log-likelihood. Parzen windows are fitted to $n$ true samples, with $\sigma$ chosen on 1,000 validation images from the script’s default grid, and scored on 1,000 test images, for seeds 1 to 5. Two flawed “generators” are also scored: one that outputs only the six clean templates, and one that has dropped three shapes (NumPy 2.3.5, SciPy 1.16.3; about 15 s; two runs give identical output).

import numpy as np
from scipy.special import logsumexp

# Six 8x8 shape templates in [-1, 1]; an image is a random template plus
# Gaussian pixel noise (s = 0.2), so its true log-density is known exactly.
T = -np.ones((6, 8, 8))
T[0, 3:5, :] = 1                                    # horizontal bar
T[1, :, 3:5] = 1                                    # vertical bar
T[2, 1:7, 1:7] = 1; T[2, 2:6, 2:6] = -1             # square outline
i = np.arange(8); T[3, i, i] = 1; T[3, i, 7 - i] = 1    # X
T[4, 2:6, 2:6] = 1                                  # filled block
T[5, ::2, ::2] = 1; T[5, 1::2, 1::2] = 1            # checkerboard
T = T.reshape(6, 64)
S, D = 0.2, 64

def sample(n, rng):
    return T[rng.integers(0, 6, n)] + S * rng.standard_normal((n, D))

def parzen_ll(x, centers, sigma):
    """Per-image log-likelihood under a Gaussian Parzen window on `centers`."""
    d2 = (x**2).sum(1)[:, None] - 2 * x @ centers.T + (centers**2).sum(1)[None, :]
    return (logsumexp(-0.5 * d2 / sigma**2, axis=1) - np.log(len(centers))
            - D * np.log(sigma * np.sqrt(2 * np.pi)))

def true_ll(x):
    return parzen_ll(x, T, S)          # the true density is a Parzen window on the templates

SIGMAS = np.logspace(-1, 0, 10)        # 0.10 ... 1.00, the default grid of the authors' parzen_ll.py
rng = np.random.default_rng(0)
valid, test = sample(1000, rng), sample(1000, rng)

def evaluate(centers):
    """Choose sigma on the validation images, then score the test images (mean and SE)."""
    sigma = SIGMAS[np.argmax([parzen_ll(valid, centers, s).mean() for s in SIGMAS])]
    ll = parzen_ll(test, centers, sigma)
    return ll.mean(), ll.std() / np.sqrt(len(ll)), sigma

print(f"true mean log-likelihood of the test images: {true_ll(test).mean():.1f} nats")
print("Parzen estimates from n samples of the TRUE distribution (seeds 1-5):")
print("     n   estimate (seed 1)  SE   sigma   range over seeds 1-5")
for n in [100, 1000, 10000]:
    runs = [evaluate(sample(n, np.random.default_rng(seed))) for seed in range(1, 6)]
    m, se, sig = runs[0]
    lo, hi = min(r[0] for r in runs), max(r[0] for r in runs)
    print(f"{n:6d}   {m:8.1f}        {se:5.1f}  {sig:5.3f}   {lo:7.1f} to {hi:6.1f}")

print("Fixed n = 10000 (seed 1), each sigma on the grid:")
x = sample(10000, np.random.default_rng(1))
print("  " + "  ".join(f"{s:5.3f}" for s in SIGMAS))
print("  " + "  ".join(f"{parzen_ll(test, x, s).mean():5.0f}" for s in SIGMAS))

# A degenerate "generator" that only ever outputs the six clean templates:
# all its mass sits on six points, so its true log-likelihood of any test image is -inf.
m, se, sig = evaluate(T[np.random.default_rng(1).integers(0, 6, 10000)])
print(f"degenerate model (6 distinct images, n = 10000): {m:.1f} (SE {se:.1f}, sigma {sig:.3f})")
# A generator that has dropped three of the six shapes, otherwise perfect.
half = T[:3][np.random.default_rng(1).integers(0, 3, 10000)] + S * np.random.default_rng(2).standard_normal((10000, D))
m, se, sig = evaluate(half)
print(f"mode-dropping model (3 of 6 shapes, n = 10000): {m:.1f} (SE {se:.1f}, sigma {sig:.3f})")

Output:

true mean log-likelihood of the test images: 10.2 nats
Parzen estimates from n samples of the TRUE distribution (seeds 1-5):
     n   estimate (seed 1)  SE   sigma   range over seeds 1-5
   100       -5.9          0.1  0.278      -6.1 to   -5.7
  1000       -2.1          0.1  0.215      -2.2 to   -2.0
 10000        0.1          0.1  0.215      -0.0 to    0.1
Fixed n = 10000 (seed 1), each sigma on the grid:
  0.100  0.129  0.167  0.215  0.278  0.359  0.464  0.599  0.774  1.000
    -63    -22     -4      0     -3    -11    -22    -34    -48    -63
degenerate model (6 distinct images, n = 10000): 9.9 (SE 0.2, sigma 0.215)
mode-dropping model (3 of 6 shapes, n = 10000): -85.7 (SE 0.8, sigma 1.000)

Even with samples from the true distribution, the estimate falls 16 nats below the true 10.2 at 100 samples and still 10 nats below at the script’s default of 10,000, in only 64 dimensions. The bias dwarfs the spread over seeds (at most 0.4 nats) and the standard error across test images (0.1), the error bar Table 1 reports for MNIST. One grid step from the selected $\sigma$ costs another 3 to 4 nats. The degenerate model has no density at all, yet scores 9.9 nats, far above the true distribution’s own samples, while dropping shapes is punished heavily and drives $\sigma$ to the edge of the grid. Theis, van den Oord and Bethge (2016, §3.5) found both effects on real data: a large bias even for $6 \times 6$ CIFAR-10 patches, and on MNIST a $k$-means lookup model (true log-likelihood $-\infty$) scoring 313 nats, above real images (243).

Impact

On 27 November 2024 the NeurIPS blog announced that “Generative Adversarial Nets” and Sutskever, Vinyals and Le’s “Sequence to Sequence Learning with Neural Networks” had both received the NeurIPS 2024 Test of Time Paper Award, given to NeurIPS 2014 papers that shaped the field; awarding two was called an exception. The post calls it one of the foundational pieces for generative modeling.

GANs went on to produce the sharpest generated images for several years, through DCGAN, BigGAN and StyleGAN, and the adversarial loss spread to tasks such as image-to-image translation. The 2014 images were modest: Radford, Metz and Chintala (2016) called GAN images of natural scenes “noisy and incomprehensible” and earlier attempts to scale GANs up with CNNs unsuccessful.

Stated Advantages and Disadvantages

Section 6 names two disadvantages: there is no explicit $p_g(x)$, and $D$ must stay synchronized with $G$: trained too much without updating $D$, $G$ falls into “the Helvetica scenario”, collapsing too many values of $z$ to the same $x$. The paper does not explain the name; later work calls this mode collapse.

The advantages are mainly computational: no Markov chains, gradients by backpropagation only, no inference during learning, and a wide variety of model functions. Statistically, the generator may gain from learning only through the discriminator’s gradients, so that inputs are not copied into its parameters, and it can represent very sharp, even degenerate distributions, whereas Markov-chain methods need somewhat blurry ones to mix between modes.

Limitations

• Evaluation. The quantitative evidence is a Parzen table whose weaknesses the authors acknowledged; Theis et al. (2016, §4) argued that such estimates should be avoided.
• Scale. The models were small multilayer perceptrons, plus one convolutional variant, on small images.

What Came After

The paper closes with five extensions. The later work paired with them below is our mapping, not the paper’s:

1. Conditional models $p(x \mid c)$, by giving $c$ to both networks: Mirza and Osindero (2014) generated MNIST digits conditioned on class labels this way.
2. Learned approximate inference, by training a network to predict $z$ from $x$ after the generator is trained. BiGAN (Donahue, Krähenbühl & Darrell, 2017) and adversarially learned inference (Dumoulin et al., 2017) instead learned an encoder inside the game, with a discriminator on image–code pairs; Donahue et al. noted that a “latent regressor” like the paper’s, trained on generated images, never sees real ones.
3. All conditionals $p(x_S \mid x_{\not S})$ through parameter-sharing conditional models.
4. Semi-supervised learning with discriminator features: Salimans et al. (2016) added a “generated” class to a $K$-class classifier and reported state-of-the-art semi-supervised results on permutation-invariant MNIST, CIFAR-10 and SVHN.
5. Efficiency, through better coordination of $G$ and $D$ or better distributions for $z$.

In our reading, item 5’s coordination problem became the long effort to stabilize GAN training, and the call for better evaluation led to sample-based metrics such as the Fréchet Inception Distance.