Mode Collapse

Mode collapse is the failure in which the generator of a generative adversarial network produces only part of the variety in its training data: many noise vectors map to the same or nearly the same image, so whole kinds of images (whole digits, whole object classes, a pose or a background) are never generated even though they are common in the data. When some modes are missing while others are still produced, the failure is also called mode dropping. Collapse is closely tied to a second problem of adversarial training, non-convergence: the generator and discriminator are trained by gradient steps on a game, not on a single loss, and those steps can circle an equilibrium, or jump from mode to mode, instead of settling on it.

The GAN article introduces both problems and summarizes the main remedies, and its example measures which of six synthetic shapes the minimax, non-saturating and WGAN-GP losses cover at the end of training. This article goes deeper: how collapse is defined, the analyses that explain it and the instability around it, the remedies designed against it, and how it is measured. The Wasserstein GAN article covers the Wasserstein loss and the gradient penalty of WGAN-GP; here they appear only as remedies.

Definition

A data distribution $p_{\text{data}}$ over images typically has many modes: regions of high probability separated by regions of low probability, such as the ten digits of MNIST. A generator $G$ maps noise $z \sim p_z$ to images and defines a distribution $p_g$. Mode collapse means that $p_g$ puts little or no probability on regions where $p_{\text{data}}$ has a lot.

The papers describe it at several strengths:

• Complete collapse. Salimans et al. (2016, §3.2) describe the generator collapsing to a parameter setting where it always emits the same point. Goodfellow et al. (2014, §6) had named the risk the “Helvetica scenario”: if the generator is trained too much without updating the discriminator, it maps too many values of $z$ to the same $x$ to have enough diversity to model the data.
• Missing modes. PacGAN’s authors (Lin et al., 2018, §1) list two types observed in earlier work: entire modes of the data are never generated, or the generator creates images only within a subset of a particular mode.
• Mode dropping. Arjovsky and Bottou (2017) use “mode dropping” for large regions where $p_{\text{data}}$ is high and $p_g$ is small or zero.

Lin et al. (2018, Definition 1) give a formal version that ignores the geometry of the modes: $p_{\text{data}}$ and $p_g$ exhibit $(\varepsilon, \delta)$-mode collapse, for $0 \le \varepsilon < \delta \le 1$, if there is a set $S$ of images with

$$p_{\text{data}}(S) \ge \delta \quad\text{and}\quad p_g(S) \le \varepsilon .$$

Large $\delta$ and small $\varepsilon$ mean severe collapse: a set that holds at least a fraction $\delta$ of the data receives at most $\varepsilon$ of the samples. Swapping the roles, $p_{\text{data}}(S) \le \varepsilon$ and $p_g(S) \ge \delta$, they call mode augmentation, generated mass where there is little data.

Training instability and non-convergence are separate failures that often accompany collapse. In non-convergence the parameters never settle at an equilibrium of the game; the generator’s samples keep changing. When the samples change from one mode to another, the generator covers different parts of the data at different times but not all of them at once.

Intuition

The generator never sees the training images. It sees only the discriminator’s verdict on each of its own images, one image at a time, and moves each image toward whatever the current discriminator rates most real. Salimans et al. (2016, §3.2) spell out what follows. When collapse to a single mode is imminent, the discriminator’s gradient points in similar directions for many similar generated images; because the discriminator processes each image independently, nothing tells the generator’s outputs to become different from one another, and all of them race toward the single point the discriminator currently believes is most realistic. Once they arrive, the discriminator learns that this point is fake, and its gradients push the whole cluster to another place. In their words, gradient descent cannot separate the identical outputs, and the single point is pushed around forever.

The original WGAN paper (Arjovsky, Chintala & Bottou, 2017, §3) states the same mechanism: for a fixed discriminator, the optimal generator is a sum of point masses at the images the discriminator rates highest. The paper describes the standard GAN on a mixture of Gaussians as rotating between the modes of the mixture without capturing the whole distribution (§4.1).

In our reading, two families of remedies follow from this picture. One gives the discriminator a view of several samples at once, so that a batch of near-identical images becomes easy to reject; minibatch discrimination and PacGAN take this route. The other changes the game’s dynamics so that the generator does not chase the current discriminator’s maximum; unrolled GANs, gradient penalties and Wasserstein losses take this route.

Formal Analyses

Which divergence the generator follows

With the optimal discriminator, the minimax game minimizes the Jensen–Shannon divergence between $p_{\text{data}}$ and $p_g$ (Goodfellow et al., 2014, Theorem 1). Arjovsky and Bottou (2017, Theorem 2.5) showed that the non-saturating generator loss used in practice instead follows the gradient of $\mathrm{KL}(p_g \,\Vert\, p_{\text{data}}) - 2\,\mathrm{JSD}(p_g \,\Vert\, p_{\text{data}})$. The reverse KL assigns a very high cost to generating fake-looking samples and a very low cost to mode dropping, and the JSD is symmetric, so in their account this explains why GANs, when stabilized, make good-looking samples and why they are commonly thought to drop many modes. The GAN article derives the Jensen–Shannon connection and presents the non-saturating loss.

What a limited discriminator can detect

Arora, Ge, Liang, Ma and Zhang (ICML 2017) asked what the game guarantees when the discriminator is a network with $p$ parameters trained on finitely many samples. They show that the Jensen–Shannon divergence and the Wasserstein distance do not generalize from samples (Lemma 1), and define the neural net distance, the largest difference in expected output that a discriminator of the given class can achieve, which does generalize with a number of samples polynomial in $p$ (Theorem 3.1). The price is a statement about diversity, their Corollary 3.1: a discriminator with $p$ parameters cannot tell a distribution apart from an empirical distribution of only about $p / \epsilon^2$ samples (up to logarithmic factors), and the authors note that letting it see many more real samples does not help, because its capacity is limited. In this analysis, a lack of diversity, a form of collapse, can be invisible to a discriminator of bounded capacity.

The same paper shows that if the generator may be a mixture of about $p \log(p/\epsilon)/\epsilon^2$ generator networks (Theorem 4.2), an approximate equilibrium exists in which the discriminator can do no better than random guessing, and, for the Wasserstein objective, an approximate pure equilibrium of a single larger network (Theorem 4.3). This motivated MIX+GAN, a mixture of a few generators and discriminators with learned mixture weights and an entropy regularizer on those weights, because components of the mixture tended to collapse and their weights to diminish (§5). On CIFAR-10 they report an Inception Score of 7.72 for a 5-component MIX+DCGAN against 7.16 for the DCGAN baseline and 7.34 for a DCGAN with five times the parameters (Table 1).

Games do not behave like losses

Salimans et al. (2016, §3) stress that GAN training seeks a Nash equilibrium of a non-convex game with continuous, high-dimensional parameters, using gradient methods built to minimize a cost. A step that lowers one player’s cost can raise the other’s. Their example: if one player minimizes $xy$ over $x$ and the other minimizes $-xy$ over $y$, gradient descent enters a stable orbit instead of converging to $x = y = 0$.

Mescheder, Geiger and Nowozin (2018, §1) credit earlier work of Mescheder et al. (2017) and of Nagarajan and Kolter (2017) with analyzing this through the Jacobian of the gradient vector field at an equilibrium. Write the two players’ parameters as $\theta$ (generator) and $\psi$ (discriminator) and the game objective as $L(\theta, \psi)$, which the discriminator maximizes and the generator minimizes. Simultaneous gradient descent with step $h$ is the iteration $(\theta, \psi) \leftarrow (\theta, \psi) + h\, v(\theta, \psi)$ with

$$v(\theta, \psi) = \begin{pmatrix} -\nabla_\theta L(\theta, \psi) \\ \nabla_\psi L(\theta, \psi) \end{pmatrix}.$$

As Mescheder, Geiger and Nowozin (2018, §1 and §2.1) summarize, if the Jacobian $v'$ at the equilibrium has only eigenvalues with negative real part, training converges locally for small enough learning rates; eigenvalues with positive real part rule out local convergence; and eigenvalues on the imaginary axis leave the continuous system neither guaranteed to converge nor to diverge, and if it converges it does so sublinearly. Eigenvalues close to the imaginary axis can require intractably small learning rates.

Nagarajan and Kolter (2017, Theorem 3.1) proved that, under assumptions on the equilibrium, the loss function and the curvature of the objective there (their Assumptions I–IV, which require both networks to be locally expressive enough), simultaneous gradient descent on the standard GAN objective is locally exponentially stable near an equilibrium. One assumption (their Assumption IV) is that all generators near the equilibrium produce distributions with the same support as the data. The same analysis shows that the Wasserstein GAN, whose objective is linear in the discriminator’s output, can have non-convergent limit cycles near equilibrium even then.

The Dirac-GAN

Mescheder, Geiger and Nowozin (2018) showed that the support assumption cannot be dropped, using an example in which each player has a single parameter. In the Dirac-GAN (their Definition 2.1) the data are a point mass at 0, the generator is a point mass at $\theta$, and the discriminator is linear, $D_\psi(x) = \psi x$. With the objective written as $L(\theta, \psi) = \mathbb{E}_{p(z)}[f(D_\psi(G_\theta(z)))] + \mathbb{E}_{p_{\text{data}}}[f(-D_\psi(x))]$, where $f(t) = -\log(1 + e^{-t})$ gives the original GAN (since $f(-t) = \log(1 - \sigma(t))$, this is the GAN article’s value function with the discriminator’s logit $-D_\psi$: only the sign of the output is flipped), the game becomes

$$L(\theta, \psi) = f(\psi\theta) + f(0).$$

The unique equilibrium is $\theta = \psi = 0$, and the Jacobian of the gradient vector field there has eigenvalues $\pm f'(0)\, i$, on the imaginary axis (Lemma 2.2). In this example linear discriminators lose nothing: taking the supremum over $\psi$ recovers the Jensen–Shannon divergence up to constants, and with $f(t) = t$ and a Lipschitz constraint, the Wasserstein distance. The authors examine the training dynamics in several forms:

• The continuous dynamics, training with infinitely small steps, keep $\theta(t)^2 + \psi(t)^2$ constant: they circle the equilibrium forever (Lemma 2.3).
• Simultaneous gradient descent has update eigenvalues $1 \pm h f'(0)\, i$, of modulus $\sqrt{1 + h^2 f'(0)^2} > 1$, so it is unstable for every learning rate, and the norm of the iterates grows monotonically (Lemma 2.4).
• Alternating gradient descent with $n_g$ generator and $n_d$ discriminator updates has eigenvalues on the unit circle when $\alpha = \sqrt{n_g n_d}\, h f'(0) \le 2$ and outside it beyond (Lemma 2.5); empirically it oscillates in stable cycles.
• The non-saturating loss gives the same eigenvalues $\pm f'(0)\, i$, so simultaneous gradient descent is again not locally convergent; its continuous dynamics do converge, but only at a logarithmic rate, $O(1/\sqrt{t})$ (supplement, Lemmas C.1 and C.2), which the main text calls extremely slow (§3).
• A WGAN with a fixed number of discriminator updates per generator update and a fixed learning rate does not generally converge (Lemma 3.1): the optimal discriminator must jump from $\psi = -1$ to $\psi = 1$ as $\theta$ changes sign, a discontinuity the finite updates cannot follow. WGAN-GP does not converge on the example either (§3.1), although the authors note that it has been applied successfully in practice.

Their explanation (§2.3) is geometric. When the generator is far from the data, the discriminator pushes it toward the data and grows steeper as it becomes more confident; when the generator reaches the data, the slope is largest and pushes it past. Near the equilibrium nothing pushes the discriminator toward zero slope on the data. In higher dimensions, when the data lie on a lower-dimensional manifold and the discriminator class is large, nothing drives the discriminator’s gradient orthogonal to the manifold to zero, even when the generator matches the data exactly, so the discriminator gives the generator no useful gradients orthogonal to the data and the generator does not converge.

The same analysis shows what stabilizes the game:

• Instance noise, Gaussian noise of standard deviation $\sigma$ added to the data, gives eigenvalues $f''(0)\sigma^2 \pm \sqrt{f''(0)^2\sigma^4 - f'(0)^2}$, all with negative real part when $f''(0) < 0$ and $\sigma > 0$ (Lemma 3.2). Above a critical level $\sigma^2_{\text{critical}} = |f'(0)| / |f''(0)|$ the eigenvalues become real and the rotation disappears.
• A zero-centred gradient penalty $\tfrac{\gamma}{2}\psi^2$, which is what a penalty on the squared norm of the discriminator’s gradient reduces to in the Dirac-GAN, gives eigenvalues $-\tfrac{\gamma}{2} \pm \sqrt{\tfrac{\gamma^2}{4} - f'(0)^2}$, with negative real part for every $\gamma > 0$, and no rotation at $\gamma_{\text{critical}} = 2|f'(0)|$ (Lemma 3.3).

They then proposed two simplified penalties (§4.1), on real data only and on generated data only:

$$R_1(\psi) = \frac{\gamma}{2}\, \mathbb{E}_{p_{\text{data}}(x)}\big[\lVert \nabla_x D_\psi(x) \rVert^2\big], \qquad R_2(\theta, \psi) = \frac{\gamma}{2}\, \mathbb{E}_{p_\theta(x)}\big[\lVert \nabla_x D_\psi(x) \rVert^2\big],$$

and proved that with either one, simultaneous and alternating gradient descent converge locally, at least linearly, for small enough learning rates, even when the data and generator distributions lie on lower-dimensional manifolds (Theorem 4.1; assumptions I–III, including a realizable generator and a discriminator able to detect any deviation from the equilibrium). Their Table 1 summarizes the results: unregularized training, DRAGAN, instance noise, consensus optimization and the zero-centred penalties converge locally when both distributions are absolutely continuous; in the general case, only instance noise, consensus optimization and the zero-centred penalties ($R_1$, $R_2$ and Roth et al.’s weighted penalty) do, while DRAGAN and unregularized training do not, and WGAN and WGAN-GP converge in neither case.

On images, they report that unregularized training quickly led to mode collapse for a single class-conditional GAN on all 1,000 ImageNet classes at $128 \times 128$ and for CelebA-HQ at $1024 \times 1024$, while $R_1$ gave stable training (§5 and supplement §F, which uses $\gamma = 10$ for the CelebA and LSUN models). $R_1$ is the regularizer that StyleGAN uses.

Remedies

Minibatch discrimination and other heuristics

Salimans et al. (2016) proposed several techniques “heuristically motivated to encourage convergence” (§3):

• Minibatch discrimination (§3.2). An intermediate feature vector $f(x_i) \in \mathbb{R}^A$ of each sample in a minibatch is multiplied by a learned tensor $T \in \mathbb{R}^{A \times B \times C}$, giving a matrix $M_i \in \mathbb{R}^{B \times C}$. For each of the $B$ rows, the closeness of two samples is $c_b(x_i, x_j) = \exp(-\lVert M_{i,b} - M_{j,b} \rVert_{L_1})$, and the layer outputs $o(x_i)_b = \sum_{j=1}^{n} c_b(x_i, x_j)$, which is concatenated with $f(x_i)$ and fed to the next layer. The text describes the sum as running over all other samples; as printed it includes $j = i$, which adds the constant $\exp(0) = 1$. The features are computed separately for real and for generated minibatches. The discriminator still classifies each image, but with the rest of its minibatch as side information, so a generated batch whose images are unusually close together is easy to detect. In their CIFAR-10 ablation, removing the minibatch features gave the largest drop in Inception Score of any removal, from 8.09 to 3.87 (Table 3). They also found that minibatch discrimination produced visually appealing samples quickly but did not work at all for their semi-supervised classifier, where feature matching worked much better (§3.2, §5).
• Feature matching (§3.1) trains the generator to match the mean of an intermediate discriminator layer on real and generated data, $\lVert \mathbb{E}_{x \sim p_{\text{data}}} f(x) - \mathbb{E}_{z} f(G(z)) \rVert_2^2$, instead of maximizing the discriminator’s output, so that it does not overtrain on the current discriminator.
• Historical averaging (§3.3) adds $\lVert \theta - \tfrac{1}{t}\sum_{i=1}^{t} \theta[i] \rVert^2$ to each player’s cost; on low-dimensional toy games where gradient descent goes into extended orbits, it found equilibria.

Salimans et al. also read the batch normalization in DCGAN discriminators as an implicit form of minibatch discrimination (§3.2); in our reading, because it makes each output depend on the other samples in the batch. The GAN article describes a later and simpler minibatch feature, the standard deviation across the minibatch used by Progressive GANs.

Unrolled GANs

Unrolled GANs (Metz et al., ICLR 2017; the paper was not read for this article, only its official code and later papers that describe it) give the generator foresight about how the discriminator will respond. The authors’ demo notebook (poolio/unrolled_gan, which states that its architecture and hyperparameters are not identical to the paper’s) writes the generator’s ideal objective as $f(\theta_G, \theta_D^\ast(\theta_G))$, with $\theta_D^\ast(\theta_G)$ the discriminator’s best response, and approximates the best response by a few steps of gradient ascent from the current discriminator, $\hat\theta_D(\theta_G)$. The generator is then updated with the gradient of $f(\theta_G, \hat\theta_D(\theta_G))$, differentiating through the unrolled discriminator steps (five in the notebook), while the discriminator itself takes ordinary steps. In our reading, a generator that collapses onto the discriminator’s current favourite point is penalized, because the unrolled discriminator has already learned to reject that point.

Later papers describe its cost and effect. VEEGAN’s authors report that it was a few orders of magnitude slower than the other methods they compared, because of the unrolling (Srivastava et al., 2017, §5.1), and Nagarajan and Kolter (2017, §1 and §3.4) relate their own regularizer closely to 1-unrolled GANs and report that in practice it can be as powerful as the more complex and slower 10-unrolled GANs.

PacGAN

PacGAN (Lin, Khetan, Fanti & Oh, 2018) packs samples: the discriminator receives $m$ samples concatenated together, all real or all generated, and classifies the pack. In their implementation everything else is kept from the “mother architecture”: the generator, the loss, the hyperparameters and the discriminator’s hidden layers; only the input layer grows by a factor of $m$ (for images, the samples are stacked as channels, so PacDCGAN4 on MNIST takes a $28 \times 28 \times 4$ input).

Their analysis explains why this penalizes collapse. A discriminator that sees packs effectively compares the product distributions $p_{\text{data}}^m$ and $p_g^m$. For the 0–1 loss, whose optimum gives the total variation distance, two generators can have the same distance $d_{\text{TV}}(p_{\text{data}}, p_g)$ yet very different collapse. Lin et al. derive the range of $d_{\text{TV}}(p_{\text{data}}^m, p_g^m)$ for pairs with a given unpacked distance $\tau$, whose upper end is $1 - (1 - \tau)^m$, and show that pairs with strong $(\varepsilon, \delta)$-mode collapse occupy the upper part of that range, where the packed distance grows fastest with $m$ (Theorem 2), while pairs without collapse occupy the bottom (Theorem 3). A generator that minimizes the packed distance therefore avoids strongly collapsing distributions. The authors contrast packing with minibatch discrimination: both show the discriminator several samples, but in one of their experiments, with the DCGAN discriminator of the unrolled-GAN implementation, the default discriminator had 585 weights, packing four samples added 162 and minibatch discrimination 1,225,732 (§1).

VEEGAN

VEEGAN (Srivastava, Valkov, Russell, Gutmann & Sutton, 2017) adds a reconstructor network $F_\theta$ that maps images back to noise and is trained both to map the real data to a standard Gaussian and to invert the generator (§3). The argument (their Figure 1): if the generator has dropped a mode, a reconstructor that inverts it was never trained on that mode, so its outputs on real images from the mode are unlikely to be Gaussian; conversely, if the reconstructor maps all real data to a Gaussian, a collapsed generator cannot map every $z$ back to itself, and the mismatch is a learning signal.

The objective (Eq. 5) combines a KL divergence between the joint distributions $q_\gamma(x \mid z)\, p_0(z)$ and $p_\theta(z \mid x)\, p(x)$, estimated with a discriminator on $(z, x)$ pairs as in BiGAN and ALI, with an autoencoder loss on the noise, $\mathbb{E}\,\lVert z - F_\theta(G_\gamma(z)) \rVert_2^2$. The noise is reconstructed, not the image: an $\ell_2$ loss is natural for Gaussian vectors but blurs images (§4). Because the reconstruction term does not pass through the discriminator, it gives the generator a signal even where the discriminator is constant, which is when, in the authors’ account, collapse occurs in methods whose objective depends on the parameters only through the discriminator, such as GANs, ALI and BiGAN (§2, §4). Their Proposition 1 shows that a global minimizer of the objective recovers the data distribution.

Gradient penalties and Wasserstein losses

The Wasserstein GAN replaces the Jensen–Shannon divergence by the Earth Mover’s distance, estimated by a critic that can be trained to optimality without its gradients vanishing. Its authors argue that training the critic to optimality makes collapse impossible, for the reason given under Intuition, and saw no evidence of collapse in any WGAN experiment (Arjovsky, Chintala & Bottou, 2017, §3–§4). WGAN-GP replaced the weight clipping with a penalty that pulls the critic’s gradient norm toward 1 on interpolates between real and generated images (Gulrajani et al., 2017). Both are described in the Wasserstein GAN article, and the GAN article’s example shows WGAN-GP covering all six shapes of its toy data where the non-saturating loss covers one or two.

Gradient penalties have three readings in this literature, which should not be confused:

• WGAN-GP’s one-centred penalty, $(\lVert \nabla D \rVert - 1)^2$, enforces a Lipschitz constraint for the Wasserstein critic.
• The zero-centred penalties $R_1$ and $R_2$ penalize any discriminator gradient on real or generated data, which pulls the discriminator toward the equilibrium discriminator and gives local convergence (Mescheder, Geiger & Nowozin, 2018). By their analysis WGAN-GP’s penalty does not guarantee it.
• Nagarajan and Kolter’s penalty acts on the generator’s update: it adds $\eta\, \lVert \nabla_{\theta_D} V(D_{\theta_D}, G_{\theta_G}) \rVert^2$ to the generator’s objective (their Eq. 4), the squared norm of the discriminator’s gradient with respect to the discriminator’s parameters. It makes equilibria locally exponentially stable for both the standard GAN and the WGAN (Theorem 3.2), and they report that it helps avoid mode collapse.

Other ways of bounding the discriminator, such as spectral normalization, and the two time-scale update rule of the FID paper are summarized in the GAN article.

How Collapse Is Measured

GANs have no likelihood, so collapse is measured through samples, and most measures need to know what the modes are. VEEGAN’s authors note that collapse can be measured accurately only on synthetic data with known modes (§5).

Mode counts on synthetic mixtures. The standard toy data are mixtures of Gaussians in two dimensions: 8 Gaussians on a ring and 25 on a grid, as specified in PacGAN (§4) and VEEGAN (§5.1). VEEGAN’s protocol counts a sample as high quality if it lies within three standard deviations of the nearest mode (ten in their 1,200-dimensional mixture), counts a mode as captured if it is the nearest mode of at least one high-quality sample, and reports both over 2,500 samples, averaged over five runs. Both numbers are needed: on the ring a standard GAN had 99.3% high-quality samples because all of them sat on one mode, while the unrolled GAN on the grid captured almost all modes by spreading its samples so widely that only 16% were of high quality. Their Table 1 reports, on the 8-mode ring, 1 mode for the GAN, 2.8 for ALI, 7.6 for the unrolled GAN and 8 for VEEGAN, and on the 25-mode grid 3.3, 15.8, 23.6 and 24.6. Mescheder, Geiger and Nowozin (2018, §5) used 2D problems differently, to measure convergence: they estimated the Wasserstein-1 distance from the generated samples to the true distribution on four 2D distributions, for six architectures and five training methods, reporting the best hyperparameter setting of each (Figure 5).

Stacked MNIST. To get many discrete modes from real images, three random MNIST digits are stacked as the three colour channels of one $28 \times 28 \times 3$ image, giving $10^3 = 1{,}000$ modes; a pretrained MNIST classifier labels each channel, and the three digits identify the mode (VEEGAN §5.2; PacGAN §4.2). Two numbers are reported: the number of modes generated, and a KL divergence, which PacGAN describes as between the generated distribution over modes and the expected uniform one (a reverse KL), and VEEGAN as between the generator and data distributions, used as a measure of sample quality. The table below combines VEEGAN’s Table 2 (26,000 samples, averaged over five runs), whose four rows PacGAN’s Table 1 copies, with PacGAN’s own rows (mean and standard error over ten trials). The two sets of rows do not share a set-up: VEEGAN’s methods share an off-the-shelf DCGAN generator but use different discriminators, PacGAN’s rows come from its own DCGAN-like implementation, and the two papers’ DCGAN rows differ (99 against 78.9 modes). Compare rows only within the same paper.

Method	Modes (of 1,000)	KL	Source
DCGAN	99	3.4	VEEGAN, Table 2
ALI	16	5.4	VEEGAN, Table 2
Unrolled GAN	48.7	4.32	VEEGAN, Table 2
VEEGAN	150	2.95	VEEGAN, Table 2
Minibatch discrimination	24.5 ± 7.67	5.49 ± 0.418	PacGAN, Table 1
DCGAN (PacGAN’s implementation)	78.9 ± 6.46	4.50 ± 0.127	PacGAN, Table 1
PacDCGAN2	1,000.0 ± 0.00	0.06 ± 0.003	PacGAN, Table 1
PacDCGAN3	1,000.0 ± 0.00	0.06 ± 0.003	PacGAN, Table 1
PacDCGAN4	1,000.0 ± 0.00	0.07 ± 0.005	PacGAN, Table 1

PacGAN’s authors found minibatch discrimination very unstable in this experiment, capturing fewer modes than their DCGAN, and suggest its parameter count as one cause: 47,976,773 discriminator parameters against 4,310,401 for the GAN and 4,324,801 for PacDCGAN4 (§4.2). They note the shortcomings of all these counts: the number of modes ignores class imbalance, and every such measure assumes the modes are known in advance (§4).

Collisions in a batch. Without labelled modes, diversity can be probed through near-duplicates. PacGAN (§4.3) estimates, on CelebA, the probability that a batch of 1,024 generated images contains at least one pair of near-duplicates, a test it takes from Arora and Zhang (2017). For a DCGAN with a 273K-parameter discriminator the probability was 1, and 0.33 with packing of two; with larger discriminators the DCGAN’s probability was 0.42 to 0.86 and the packed one’s 0 to 0.17 (Table 2).

Distances to real images. VEEGAN’s authors also report an “inference via optimization” measure, which they credit to the unrolled-GAN paper: for each real test image, optimize $z$ to minimize the $\ell_2$ distance between $G(z)$ and the image; a collapsed generator leaves some real images far from anything it can produce (§5.3). They warn that very dispersed generators can fool it and that $\ell_2$ may favour blurry generators.

Feature-space scores on real datasets. On natural images the modes are unknown, and the standard scores compare features of a pretrained classifier:

• The Fréchet Inception Distance compares Gaussians fitted to features of real and generated images. Lucic et al. (2018, §5) found it heavily influenced by missing modes: in their test, a “model” made of real images from only the first $k$ classes scored worse the fewer classes it kept. FID mixes fidelity and diversity, so a measure that separates them, such as precision and recall for generative models (described in the FID article), shows which one is lacking.
• The Inception Score rewards a spread of predicted classes, so it registers collapse across classes, but not collapse within a class: a generator that emits one image per class can score highly. Its article shows this with its own example. Salimans et al. (2016, §6.2) noted that models that even partially collapse had relatively low scores in their experiments.
• For class-conditional models, the conditional GAN article describes the within-class diversity measure of AC-GAN, MS-SSIM between samples of the same class.

Report distributions, not best runs. Lucic et al. (2018) trained many GANs under one protocol and concluded that, because of the randomness of optimization and model instability, comparisons need a summary of the distribution of results over random seeds and hyperparameters rather than the best run. They found that most models could reach similar scores with enough hyperparameter optimization and random restarts, and no evidence that any tested algorithm consistently beat the non-saturating GAN. Since collapse often varies from seed to seed, a single run says little about a method; the GAN article’s example illustrates this, with the surviving shapes changing from seed to seed.

Examples

A worked case on images is in the GAN article’s example, which trains small generators on six synthetic shapes with known templates and counts which shapes each loss covers. It uses a version of the mode-count protocol above: each generated image is assigned to its nearest shape, counted only if it is as close as nearly all real images are, and a shape is covered when it receives a minimum share of those images.

Examples from the papers:

• Mixtures of Gaussians. On the 8-mode ring, VEEGAN’s standard GAN collapsed all of its samples onto one mode (Srivastava et al., 2017, §5.1 and Figure 2), and the original WGAN paper describes a standard GAN rotating between the modes of such a mixture.
• ImageNet. Salimans et al. (2016, §6.4) note GANs’ tendency to underestimate the entropy of the distribution as a particular difficulty with 1,000 classes; Mescheder, Geiger and Nowozin (2018, supplement §F) found that unregularized training of their ImageNet GAN quickly led to collapse.
• CelebA. PacGAN’s collision test above estimated a probability of 1 that a batch of 1,024 faces from its smallest DCGAN contains a near-duplicate pair.
• Conditional GANs. Collapse is not limited to unconditional generators: the image-to-image translation article describes pix2pix’s generator learning to ignore its noise input, and the conditional GAN article describes AC-GAN’s collapse on many ImageNet classes.

Common Misconceptions

• “Mode collapse means the generator outputs one image.” Complete collapse is the extreme; the common failure is partial, some modes missing or under-represented, or variety missing within a mode (Lin et al., 2018, §1).
• “High sample quality rules out collapse.” It does not: VEEGAN’s collapsed GAN on the ring had the highest fraction of high-quality samples. Quality and coverage have to be measured together.
• “Collapse and non-convergence are the same problem.” They are related but distinct. A generator can converge to a distribution that misses modes, and the Dirac-GAN fails to converge with a single mode. Remedies address different parts: packing and minibatch discrimination change what the discriminator can see, zero-centred penalties and instance noise change the dynamics near equilibrium.
• “Gradient penalties are all the same.” WGAN-GP’s penalty is centred at 1 and enforces a Lipschitz constraint; $R_1$ and $R_2$ are centred at 0 and give local convergence under the authors’ assumptions. Mescheder, Geiger and Nowozin (2018) prove local convergence for the second kind and show that WGAN-GP does not converge on the Dirac-GAN.
• “A low FID or a high Inception Score means no collapse.” Inception Score ignores collapse within classes, and FID mixes coverage with fidelity in one number; mode-level measures need known modes.
• “Codebook collapse is mode collapse.” The VQ-VAE article describes codebook collapse, in which a vector quantizer uses only a few of its codes. It is a failure of the quantizer’s training, not of an adversarial game, although both leave a model using less variety than it has.

Where It Is Used

Mode collapse is the reference failure for every method that trains a generator against a discriminator: unconditional and conditional GANs, StyleGAN, whose $R_1$ penalty comes from the convergence analysis above, and image-to-image translation models. It is also part of the account of why diffusion models displaced GANs for most image synthesis: the generative model hub describes the trade-off between quality, coverage and speed in which GANs lack coverage. Coverage measures that do not need known modes, such as the recall of precision and recall for generative models (described in the FID article), apply to generative models of any family, and the hub uses recall to compare them.