Generative Adversarial Network

A generative adversarial network (GAN) learns to generate images by playing a game between two networks: a generator that turns random noise into an image, and a discriminator that is trained to tell generated images from real ones. The generator never sees a training image; it learns only through the discriminator’s gradients. Introduced by Goodfellow et al. (2014), GANs were for several years the generative models with the sharpest samples, and they generate an image in a single forward pass. Their weaknesses, unstable training and mode collapse, in which the generator covers only part of the data distribution, are one reason diffusion models overtook them for most image synthesis tasks.

This article covers the architecture as it is understood today; the hub compares GANs with the other families, and the 2014 paper has its own article.

Core Idea

A GAN defines an implicit model: there is no formula for the probability of an image, only a procedure for drawing one. A noise vector $z$ is drawn from a fixed prior $p_z$, usually a standard Gaussian or uniform distribution, and the generator maps it to an image $x = G(z)$; the distribution of $G(z)$, written $p_g$, is the model. Because $p_g$ cannot be evaluated, it cannot be fitted by maximum likelihood. Instead a second network, the discriminator $D$, is trained as an ordinary binary classifier to output the probability that its input is a real image, and the generator is trained to make that classifier wrong.

When both networks have enough capacity and the game reaches its equilibrium, $p_g$ equals the data distribution and the discriminator outputs $1/2$ everywhere (Goodfellow et al., 2014).

Building Blocks

• Generator $G(z; \theta_g)$: a differentiable network from the noise space to image space. Since DCGAN (Radford, Metz & Chintala, 2016) it is usually a convolutional network that upsamples a small feature map with transposed convolutions, ending in a $\tanh$ so that pixel values lie in $[-1, 1]$.
• Discriminator $D(x; \theta_d)$: a downsampling CNN that outputs the probability (or, before a sigmoid, the logit) that $x$ is real.
• Noise prior $p_z$: a simple distribution, the generator’s only source of randomness.
• The adversarial loss: the cross-entropy of the discriminator’s real-versus-fake classification, which the discriminator minimizes and the generator, in one form or another, maximizes; the discriminator acts as a learned loss function.

How Data Flows

Each training step draws a minibatch of real images and one of noise vectors, which the generator maps to fake images. The discriminator scores both, and one optimizer step updates $\theta_d$ with $\theta_g$ fixed. Then a fresh noise batch is passed through $G$ and $D$, and the generator’s loss is backpropagated through the discriminator to update only $\theta_g$. Goodfellow et al. (2014, Algorithm 1) allow $k$ discriminator steps per generator step and used $k = 1$. At sampling time the discriminator is discarded.

Mathematical Formulation

The minimax game

Goodfellow et al. (2014) define the value function

$$V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}\big[\log D(x)\big] + \mathbb{E}_{z \sim p_z}\big[\log\big(1 - D(G(z))\big)\big],$$

and train $D$ to maximize it and $G$ to minimize it, $\min_G \max_D V(D, G)$. $V$ is the negative of the binary cross-entropy of a classifier that labels real images 1 and generated images 0, so the inner maximization fits the best such classifier.

The optimal discriminator

For a fixed generator, writing the second expectation over $x \sim p_g$ turns $V$ into $\int p_{\text{data}}(x) \log D(x) + p_g(x) \log(1 - D(x))\, dx$. For each $x$, the function $y \mapsto a \log y + b \log(1 - y)$ with $a, b \ge 0$, not both zero, is maximized on $[0, 1]$ at $y = a / (a + b)$, so (Goodfellow et al., 2014, Proposition 1)

$$D^\ast_G(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}.$$

The optimal discriminator therefore encodes the density ratio $p_{\text{data}} / p_g$, which Goodfellow (2016, §3.2.1) identifies as the key approximation that sets GANs apart: they estimate a ratio of densities by supervised learning, instead of bounding a likelihood or running a Markov chain.

The Jensen–Shannon connection

Substituting $D^\ast_G$ gives the generator’s criterion $C(G) = \max_D V(D, G)$. With the mixture $m = \tfrac{1}{2}(p_{\text{data}} + p_g)$ and the Jensen–Shannon divergence $\mathrm{JSD}(p \,\Vert\, q) = \tfrac{1}{2}\mathrm{KL}(p \,\Vert\, m) + \tfrac{1}{2}\mathrm{KL}(q \,\Vert\, m)$, Goodfellow et al. (2014, Theorem 1) show

$$C(G) = -\log 4 + \mathrm{KL}\big(p_{\text{data}} \,\Vert\, m\big) + \mathrm{KL}\big(p_g \,\Vert\, m\big) = -\log 4 + 2\,\mathrm{JSD}\big(p_{\text{data}} \,\Vert\, p_g\big).$$

Since the JSD is zero only for equal arguments and positive otherwise, the global minimum $-\log 4$ is reached if and only if $p_g = p_{\text{data}}$, where $D^\ast_G = 1/2$. Their Proposition 2 adds that if $D$ reaches its optimum at every step and $p_g$ itself is updated by sufficiently small steps that improve the criterion, $p_g$ converges to $p_{\text{data}}$, using the convexity of $\sup_D V$ in $p_g$. The authors note that in practice the parameters of $G$ are updated, not $p_g$, so the guarantee does not carry over (see also Goodfellow, 2016, §3.2.2).

The JSD is bounded by $\log 2$. Reasoning from the formula, when $p_g$ and $p_{\text{data}}$ have disjoint supports the criterion sits at its maximum, $0$, whatever the generator does, so it gives no direction of improvement. Arjovsky and Bottou (2017, Theorems 2.1–2.4) made this precise for distributions concentrated on low-dimensional manifolds, as images and generator outputs are: a perfect discriminator exists, and as $D$ approaches it, the gradient of the minimax generator loss vanishes.

The non-saturating loss

Goodfellow et al. (2014, §3) observed the practical form of this problem: early in training, when $G$ is poor, $D$ rejects its samples with high confidence and $\log(1 - D(G(z)))$ saturates. They proposed training $G$ to maximize $\log D(G(z))$ instead, that is, to minimize

$$L_G = -\mathbb{E}_{z \sim p_z}\big[\log D(G(z))\big],$$

which has the same fixed point but much stronger gradients early on. This non-saturating loss is the form used in practice; the discriminator keeps its cross-entropy loss. With $D(G(z)) = \sigma(\ell)$ for a logit $\ell$ and the sigmoid $\sigma$, the two generator losses differ in their derivative with respect to the logit:

$$\frac{\partial}{\partial \ell} \log\big(1 - \sigma(\ell)\big) = -\sigma(\ell), \qquad \frac{\partial}{\partial \ell} \big[-\log \sigma(\ell)\big] = -\big(1 - \sigma(\ell)\big).$$

When the discriminator is confident that a sample is fake, $\sigma(\ell) \to 0$: the minimax gradient vanishes and the non-saturating gradient approaches its maximum magnitude. Goodfellow (2016, §3.2.3) describes it as flipping the target of the cross-entropy instead of its sign, a heuristic that makes the game no longer zero-sum. Arjovsky and Bottou (2017, Theorem 2.5) identified what it optimizes: with the optimal discriminator, its expected gradient equals the gradient of $\mathrm{KL}(p_g \,\Vert\, p_{\text{data}}) - 2\,\mathrm{JSD}(p_g \,\Vert\, p_{\text{data}})$. The reverse KL term charges heavily for implausible samples and little for missing modes, which those authors connect to the mode dropping seen in practice.

Why It Works

The discriminator is a learned, adaptive measure of whether an image looks real. In our reading, a per-pixel loss rewards the blurry average of plausible images, while the discriminator penalizes any difference from real images it can detect, including in sharpness and texture. Goodfellow et al. (2014, §6) add that no Markov chains or inference are needed and that GANs can represent very sharp, even degenerate distributions. The divergence is not the explanation: GANs trained toward the maximum-likelihood KL are sharp too, and Goodfellow (2016, §3.2.5) concludes that the approximation strategy matters more than the loss.

Training Considerations

DCGAN guidelines

Radford, Metz and Chintala (2016, §3) found that standard supervised CNN architectures trained GANs poorly and reported a family of convolutional architectures that trained stably across datasets, summarized as guidelines:

• replace pooling with strided convolutions in the discriminator and fractionally strided (transposed) convolutions in the generator;
• use batch normalization in both networks, except at the generator’s output and the discriminator’s input, where applying it caused sample oscillation and instability;
• remove fully connected hidden layers in deeper architectures;
• use ReLU in the generator, with $\tanh$ at the output, and LeakyReLU in the discriminator.

Their training settings (§4) were: images scaled to $[-1, 1]$, weights initialized with standard deviation 0.02, a LeakyReLU slope of 0.2, and Adam with learning rate 0.0002 and $\beta_1 = 0.5$, because the default $\beta_1 = 0.9$ caused oscillation.

Training instability

A GAN is a two-player game, not an optimization problem, and gradient descent on each player’s loss need not reach an equilibrium: each player’s step can undo the other’s progress, and training often oscillates from one kind of sample to another (Goodfellow, 2016, §5.1). The discriminator must be neither too weak nor so strong that the minimax loss saturates (Arjovsky & Bottou, 2017). At scale, Brock, Donahue and Simonyan (2019, §4) found that their BigGAN models eventually collapsed, so they stopped early. The discriminator’s accuracy was above 98% on ImageNet training images but 50–55% on validation images, which they took as evidence that it memorizes the training set, and constraining it strongly enough to prevent collapse cost substantial performance.

A standard GAN’s loss curves say little about sample quality, while Arjovsky, Chintala and Bottou (2017, §4.3, ICML version) found their Wasserstein critic’s estimate to correlate with it.

Mode collapse

Mode collapse (the “Helvetica scenario” of the 2014 paper) occurs when the generator maps many values of $z$ to the same output, so its samples cover only some modes of the data; complete collapse is rare, partial collapse common. Goodfellow (2016, §5.1.1) explains it through the order of the game: the minimax solution $\min_G \max_D V$ recovers the data distribution, but in the maximin order $\max_D \min_G V$ the generator maps every $z$ to the single point the discriminator rates most real. Simultaneous gradient descent often behaves like the second, and the generator cycles from mode to mode as the discriminator learns to reject each one. He argues that the JSD is not the cause, since GANs trained toward the maximum-likelihood KL collapse too.

Remedies attack the discriminator’s view of diversity. Minibatch discrimination lets the discriminator compare a sample with the others in its minibatch, so that a batch of near-identical images is easy to detect (Salimans et al., 2016, §3.2). Progressive GANs append the average standard deviation of features across the minibatch as an extra discriminator feature (Karras et al., 2018, §3).

Remedies for instability

• Wasserstein GAN. Arjovsky, Chintala and Bottou (2017) replaced the JSD with the Earth Mover’s (Wasserstein-1) distance, which changes continuously as one distribution slides toward another even when their supports do not overlap; in their Example 1 the JSD stays at $\log 2$ until the distributions coincide. By the Kantorovich–Rubinstein duality, $W(p_{\text{data}}, p_g) = \sup_{\lVert f \rVert_L \le 1} \mathbb{E}_{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{z \sim p_z}[f(G(z))]$, so a critic $f$ with an unbounded output is trained to maximize the difference and the generator to minimize it. To keep $f$ Lipschitz they clipped every weight to $[-0.01, 0.01]$ after each update, and ran five critic steps, with RMSProp, per generator step. They saw no mode collapse with WGAN in their experiments, and it trained generators known to perform poorly as standard GANs, such as one without batch normalization.
• WGAN-GP. Gulrajani et al. (2017, §3) showed that weight clipping biases the critic toward overly simple functions and makes gradients vanish or explode without careful tuning, and replaced it with a penalty $\lambda\, \mathbb{E}_{\hat x}\big[(\lVert \nabla_{\hat x} D(\hat x) \rVert_2 - 1)^2\big]$ on random interpolates $\hat x = \epsilon x + (1 - \epsilon) G(z)$ between real and generated images, with $\epsilon \sim U[0, 1]$ and $\lambda = 10$ throughout, no batch normalization in the critic (layer normalization is their suggested substitute), and default settings of five critic steps per generator step and Adam with learning rate 0.0001, $\beta_1 = 0$ and $\beta_2 = 0.9$.
• Spectral normalization. Miyato et al. (2018) bound the Lipschitz constant of the discriminator directly, by dividing each layer’s weight matrix by its largest singular value, $\bar W = W / \sigma(W)$, estimated cheaply with one step of power iteration per update. With 1-Lipschitz activations such as ReLU or LeakyReLU, the Lipschitz constant of the whole discriminator is then bounded by 1.
• Two time-scale update rule (TTUR). Heusel et al. (2017) gave the two networks separate learning rates and proved convergence to a stationary local Nash equilibrium under their assumptions. The faster player is not always the discriminator: their best DCGAN rates on CelebA were $10^{-5}$ for the discriminator and $5 \times 10^{-4}$ for the generator (Table 1).

Example: The Game and Mode Collapse on Synthetic Shapes

The data are $8 \times 8$ images of six shapes, each a template in $[-1, 1]$ (horizontal bar, vertical bar, square outline, X, filled block, checkerboard) plus Gaussian pixel noise with standard deviation 0.2, so the true distribution has six known, equally likely modes. Each generated image is assigned to its nearest template and counted as on-mode if it is at least as close as 99% of real images are; a shape is covered when it receives at least 5% of the on-mode samples (a perfect generator gives each about one sixth). The same small fully connected networks are trained with the minimax loss, the non-saturating loss and WGAN-GP, with a deliberately strong discriminator whose learning rate is five times the generator’s. The script also reports the mean $D(G(z))$ and the norm of the generator’s gradient over the last 100 steps (PyTorch 2.14.1, CPU, one thread, about 26 s for five seeds; two runs give identical output).

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)

# Six 8x8 shape templates in [-1, 1]: the modes of the data distribution.
T = -torch.ones(6, 8, 8)
T[0, 3:5, :] = 1                                   # horizontal bar
T[1, :, 3:5] = 1                                   # vertical bar
T[2, 1:7, 1:7] = 1; T[2, 2:6, 2:6] = -1            # square outline
i = torch.arange(8); T[3, i, i] = 1; T[3, i, 7 - i] = 1   # X
T[4, 2:6, 2:6] = 1                                 # filled block
T[5, ::2, ::2] = 1; T[5, 1::2, 1::2] = 1           # checkerboard
T = T.reshape(6, 64)
NAMES = ["hbar", "vbar", "square", "X", "block", "checker"]

def sample_real(n, g):  # a random template plus Gaussian pixel noise
    return (T[torch.randint(0, 6, (n,), generator=g)] + 0.2 * torch.randn(n, 64, generator=g)).clamp(-1, 1)

def nearest(x):  # distance to, and index of, the nearest template
    d = torch.cdist(x, T)
    return d.min(1).values, d.argmin(1)

# A generated image is "on-mode" if it is as close to a template as 99% of real images are.
THR = torch.quantile(nearest(sample_real(20000, torch.Generator().manual_seed(0)))[0], 0.99)

def mlp(n_in, n_out):
    return nn.Sequential(nn.Linear(n_in, 64), nn.LeakyReLU(0.2), nn.Linear(64, 64),
                         nn.LeakyReLU(0.2), nn.Linear(64, n_out))

def train(loss, seed, steps=3000, batch=64, zdim=8):
    torch.manual_seed(seed)
    g = torch.Generator().manual_seed(seed + 1000)
    G, D = nn.Sequential(mlp(zdim, 64), nn.Tanh()), mlp(64, 1)   # D outputs a logit
    opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
    opt_d = torch.optim.Adam(D.parameters(), lr=1e-3, betas=(0.5, 0.999))  # a fast discriminator
    d_fake, g_norm = [], []
    for step in range(steps):
        x, fake = sample_real(batch, g), G(torch.randn(batch, zdim, generator=g)).detach()
        if loss == "WGAN-GP":   # critic loss plus gradient penalty on interpolates (Gulrajani et al.)
            eps = torch.rand(batch, 1, generator=g)
            xh = (eps * x + (1 - eps) * fake).requires_grad_(True)
            grad, = torch.autograd.grad(D(xh).sum(), xh, create_graph=True)
            loss_d = D(fake).mean() - D(x).mean() + 10 * ((grad.norm(dim=1) - 1) ** 2).mean()
        else:                   # binary cross-entropy: -log D(x) - log(1 - D(G(z)))
            loss_d = F.softplus(-D(x)).mean() + F.softplus(D(fake)).mean()
        opt_d.zero_grad(); loss_d.backward(); opt_d.step()

        logit = D(G(torch.randn(batch, zdim, generator=g)))
        if loss == "minimax":            # minimize log(1 - D(G(z)))
            loss_g = -F.softplus(logit).mean()
        elif loss == "non-saturating":   # minimize -log D(G(z))
            loss_g = F.softplus(-logit).mean()
        else:                            # WGAN generator: maximize the critic's score
            loss_g = -logit.mean()
        opt_g.zero_grad(); loss_g.backward()
        if step >= steps - 100:          # monitor the last 100 steps
            d_fake.append(torch.sigmoid(logit).mean().item())
            g_norm.append(torch.cat([p.grad.flatten() for p in G.parameters()]).norm().item())
        opt_g.step()
    with torch.no_grad():
        dist, cls = nearest(G(torch.randn(5000, zdim, generator=g)))
    on = dist < THR
    counts = torch.bincount(cls[on], minlength=6)
    covered = int((counts >= 0.05 * max(int(on.sum()), 1)).sum())   # >= 5% of on-mode samples
    return on.float().mean().item(), counts.tolist(), covered, sum(d_fake) / 100, sum(g_norm) / 100

print(f"PyTorch {torch.__version__}; on-mode threshold {THR:.2f} (templates are at least {torch.pdist(T).min():.1f} apart)")
print(f"seed  {'loss':15s}{'on-mode':>7s}  {' '.join(f'{n:>7s}' for n in NAMES)}  covered  D(G(z))  |grad G|")
for seed in range(5):
    for loss in ["minimax", "non-saturating", "WGAN-GP"]:
        on, counts, covered, dg, gn = train(loss, seed)
        dg = f"{dg:8.3f}" if loss != "WGAN-GP" else f"{'-':>8s}"
        print(f"{seed:4d}  {loss:15s}{on:6.0%}   {' '.join(f'{c:7d}' for c in counts)}  {covered:4d}/6 {dg}  {gn:8.4f}")

Output:

PyTorch 2.14.1; on-mode threshold 1.49 (templates are at least 8.0 apart)
seed  loss           on-mode     hbar    vbar  square       X   block checker  covered  D(G(z))  |grad G|
   0  minimax           20%         0       0       0       0    1007       0     1/6    0.079    0.8875
   0  non-saturating    91%         0    2000       0       0    2527       0     2/6    0.225    1.3777
   0  WGAN-GP           60%       532     543     551     499     432     435     6/6        -    1.3257
   1  minimax           91%         0       0       0       0    4529       0     1/6    0.141    0.0603
   1  non-saturating    69%         0       0       0       0    3475       0     1/6    0.140    1.5879
   1  WGAN-GP           65%       545     515     588     564     456     559     6/6        -    1.3375
   2  minimax          100%         0    5000       0       0       0       0     1/6    0.136    0.0159
   2  non-saturating    79%         0       0    1801    2154       0       0     2/6    0.221    1.3250
   2  WGAN-GP           61%       551     509     603     610     335     419     6/6        -    1.3482
   3  minimax           71%         0       0      33       0    3531       0     1/6    0.157    0.3179
   3  non-saturating    74%         0    2472       0       0    1209       0     2/6    0.219    2.1939
   3  WGAN-GP           63%       459     543     602     545     456     540     6/6        -    1.3908
   4  minimax            0%         0       0       0       0       0       0     0/6    0.000    0.0001
   4  non-saturating    94%         0       0       0    4712       0       0     1/6    0.145    0.7962
   4  WGAN-GP           60%       405     488     513     632     501     470     6/6        -    1.3800

With seed 4 the minimax generator never gets started: the discriminator rejects every sample, $D(G(z))$ rounds to 0.000, and the generator’s gradient norm is $10^{-4}$, the saturation that the derivative $-\sigma(\ell)$ predicts. In seeds 0 to 3 it learns only one shape, and in three of them its gradient norm is 0.016 to 0.32, against 1.3 to 2.2 for the non-saturating generator. The non-saturating generator always learns and most of its samples are clean shapes, but it covers only one or two of the six, and which ones survive changes with the seed. WGAN-GP covers all six shapes in roughly equal numbers, but 35% to 40% of its images fall between templates, which suggests that after 3,000 steps it spreads mass across the gaps instead of committing to modes.

Changing range(5) to range(16) in the final loop (about 80 s) extends the run to seeds 0 to 15. Over those 16 seeds the minimax generator failed entirely (0% on-mode, generator gradient norm at most 0.0011) in 9 runs and otherwise covered one or two shapes; the non-saturating generator covered one or two shapes in all 16 runs, with 69% to 96% of samples on-mode; WGAN-GP covered all six in all 16 runs, with 47% to 65% on-mode. These numbers illustrate the mechanisms in this toy, not the relative merit of the losses on real images.

Variants

Conditional GANs

Mirza and Osindero (2014) built the conditional extension that Goodfellow et al. (2014, §7) had suggested: the condition $y$, such as a one-hot class label, is fed to both networks, and the game becomes

$$\min_G \max_D \; \mathbb{E}_{x \sim p_{\text{data}}}\big[\log D(x \mid y)\big] + \mathbb{E}_{z \sim p_z}\big[\log\big(1 - D(G(z \mid y) \mid y)\big)\big].$$

Their Equation 2 omits $y$ from the discriminator in the second term, although their text feeds $y$ to the discriminator; the form above follows the text. The condition can also be an image. pix2pix (Isola et al., 2017) trains a conditional GAN on paired images, such as label maps and photographs, with an added $L^1$ loss, a U-Net generator and a PatchGAN discriminator that classifies image patches as real or fake. CycleGAN (Zhu et al., 2017) translates between two unpaired collections, such as horses and zebras, with generators $G: X \to Y$ and $F: Y \to X$, adversarial losses in both domains and a cycle-consistency loss that encourages $F(G(x)) \approx x$ and $G(F(y)) \approx y$.

Scaling up

• Progressive growing (Karras et al., 2018) starts both networks at $4 \times 4$ pixels and adds layers for higher resolutions as training proceeds, fading each new layer in smoothly. The authors report faster, more stable training and $1024 \times 1024$ faces from CelebA-HQ, a high-quality CelebA version they built.
• BigGAN (Brock, Donahue & Simonyan, 2019) scaled class-conditional ImageNet GANs to two to four times the parameters and eight times the batch size of earlier work (batches of 2,048), on the SA-GAN baseline with its hinge loss and spectral normalization in the discriminator and, following SA-GAN, in the generator (§3 and Appendix H, arXiv v2). Orthogonal regularization of the generator made it amenable to the truncation trick, sampling $z$ from a truncated normal to trade variety for sample quality. Their Table 2 (arXiv v2) lists BigGAN-deep at $128 \times 128$ with FID 5.7 and IS 124.5 without truncation, and FID 7.4 at IS 166.5 when the truncation is set to match the IS of real validation images; the abstract’s “IS of 166.5 and FID of 7.4” is this truncated operating point, not the lowest FID in the table.
• StyleGAN (Karras, Laine & Aila, 2019) started from the Progressive GAN setup and redesigned the generator: an 8-layer mapping network turns $z$ into an intermediate latent $w$, which controls the synthesis network through adaptive instance normalization after each convolution, and per-layer noise inputs add stochastic detail, separating attributes such as pose from fine variation. StyleGAN has its own article.

Evaluation

GANs assign no likelihood, so they are evaluated through their samples, usually with the features of a pretrained Inception classifier. The Inception Score (Salimans et al., 2016, §4) applies the classifier to each generated image to get a label distribution $p(y \mid x)$ and computes

$$\mathrm{IS} = \exp\Big(\mathbb{E}_{x \sim p_g}\big[\mathrm{KL}\big(p(y \mid x) \,\Vert\, p(y)\big)\big]\Big),$$

which is high when each image is confidently classified and the marginal $p(y)$ is spread out; the authors found it to correlate well with human judgment and recommended about 50,000 samples. Because IS never compares generated images with real ones, Heusel et al. (2017) proposed the Fréchet Inception Distance (FID) between Gaussians fitted to Inception-v3 pool features of real and generated images:

$$\mathrm{FID} = \lVert \mu_r - \mu_g \rVert^2 + \mathrm{Tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\big);$$

lower is better. Neither score separates quality from coverage, and IS does not penalize lack of variety in class-conditional models: lowering the truncation threshold raises it directly (Brock, Donahue & Simonyan, 2019, §3.1).

Use in Computer Vision

GANs were used for unconditional and class-conditional image synthesis, paired and unpaired image-to-image translation, and as an extra loss in restoration and synthesis networks, where an adversarial term sharpens outputs that a pixel loss would blur. The adversarial loss has outlived GANs’ lead in generation: the autoencoder of latent diffusion is trained with a perceptual loss and a patch-based adversarial objective so that its reconstructions stay sharp (Rombach et al., 2022). Radford, Metz and Chintala (2016) also used DCGAN discriminator features for classification.

Limitations

• No likelihood. A GAN has no explicit $p_g(x)$ (Goodfellow et al., 2014, §6), so it cannot score an image or be compared on held-out likelihood.
• Unstable training and incomplete coverage. Results depend on architecture, optimizer settings and the balance between the networks; large models can collapse late in training, and mode collapse leaves parts of the distribution unrepresented.
• Discrete outputs. The generator is trained by differentiating through its output, so the basic method cannot produce discrete data such as one-hot text tokens (Goodfellow, 2016, §5.3).

Important Models

Model	Year	Contribution
GAN	2014	The adversarial game, the optimal discriminator and the non-saturating loss (Goodfellow et al.)
Conditional GAN	2014	Condition fed to both networks (Mirza & Osindero)
DCGAN	2016	Convolutional architecture guidelines and training defaults (Radford et al.)
Improved techniques	2016	Minibatch discrimination, feature matching, the Inception Score (Salimans et al.)
pix2pix, CycleGAN	2017	Paired and unpaired image-to-image translation (Isola et al.; Zhu et al.)
WGAN, WGAN-GP	2017	Wasserstein critic with weight clipping, then gradient penalty (Arjovsky et al.; Gulrajani et al.)
TTUR and FID	2017	Separate learning rates with a convergence proof; the Fréchet Inception Distance (Heusel et al.)
Progressive GAN	2018	Growing resolution during training; $1024 \times 1024$ faces (Karras et al.)
SN-GAN	2018	Spectral normalization of the discriminator (Miyato et al.)
BigGAN	2019	Large-scale class-conditional ImageNet GANs, truncation trick (Brock et al.)
StyleGAN	2019	Style-based generator with a mapping network (Karras et al.)

History and Decline

After DCGAN, three years of stabilizers and scaling methods led to BigGAN and StyleGAN. Dhariwal and Nichol (2021) hypothesized that GANs’ lead stemmed from at least two factors, heavily refined architectures and the ability to trade diversity for fidelity, and noted the drawbacks that made GANs hard to scale and apply to new domains: less diversity than likelihood-based models, and training that often collapses without carefully chosen hyperparameters and regularizers. In their account, diffusion models offered distribution coverage, a stationary training objective and easy scaling. After improving the architecture and adding classifier guidance, they reported an FID of 2.97 on ImageNet $128 \times 128$ against 6.02 for BigGAN-deep, with recall 0.59 against 0.35 (Table 5, arXiv v4); the hub article gives their $256 \times 256$ results and traces the move of text-to-image generation to autoregressive and diffusion models.

The cost of diffusion is sampling speed, which the same authors list as a limitation, and it is where GANs remain strong. In the unconditional CIFAR-10 comparison of Xiao, Kreis and Vahdat (2022, Table 1, arXiv v2), which the hub reproduces, StyleGAN2 with adaptive discriminator augmentation sampled about 2,000 times faster than DDPM with 1,000 steps at a similar FID, but with lower recall. Their denoising diffusion GAN models each large denoising step with a conditional GAN, combining the two families.