GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

“GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium” by Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler and Sepp Hochreiter, of the LIT AI Lab and Institute of Bioinformatics at Johannes Kepler University Linz, was posted to arXiv on 26 June 2017 and published in Advances in Neural Information Processing Systems 30 (NIPS 2017). It made two contributions that took very different paths. The two time-scale update rule (TTUR) gives the generative adversarial network’s discriminator and generator separate learning rates, and the paper proves, with the theory of two time-scale stochastic approximation, that training then converges to a stationary local Nash equilibrium, also when the optimizer is Adam. To measure the improvement on images, the paper introduced the Fréchet Inception Distance (FID), which became the default score for image generators. The metric as understood today, with its formula, estimation and failure cases, is in the Fréchet Inception Distance article; this one covers what the 2017 paper proposed, proved and reported, and how its versions differ.

The arXiv record has six versions. The first four, from June and July 2017, list a sixth author, Günter Klambauer, claim convergence to a “Nash equilibrium” without “local”, and report a different set of experiments; the November 2017 version introduced the published title, author list and experiments. Unless a version is named, this article follows the NIPS proceedings paper and its supplement.

Problem

By 2017, GANs produced realistic images, but, as the paper’s abstract puts it, the convergence of their training had not been proved. Training is a game between two networks updated by stochastic gradient descent, and gradient descent on a game can cycle instead of settling. The convergence results available were restricted: some held for special GAN variants under extra assumptions, others held in expectation over training samples or as the number of examples goes to infinity, which the authors note excludes the mini-batch noise of real training. In the final version the paper adds that the local stability proof of Nagarajan and Kolter (2017) needs either strong assumptions or a linear discriminator.

Practitioners had meanwhile found ways to let the discriminator learn faster than the generator: in the paper’s account, a new generator objective from Salimans et al. (2016) slowed the generator so that it would not overtrain on the current discriminator, and the Wasserstein GAN takes several discriminator steps per generator step. The paper’s starting point was an analysis of actor-critic reinforcement learning, in which a critic that learns faster than the actor provably drives training to a stationary local Nash equilibrium.

The second problem was measurement. The paper calls the Inception Score of Salimans et al. a well-performing measure that correlates with human judgment, but objects that it never compares the statistics of generated images with those of real ones. To show that TTUR trains better GANs, the authors needed a score that does.

Contribution

The paper lists five contributions:

• the two time-scale update rule for GANs;
• a proof that GANs trained with TTUR converge to a stationary local Nash equilibrium;
• a description of Adam as a heavy ball with friction, governed by a second-order differential equation;
• the convergence of GANs trained with TTUR and Adam to a stationary local Nash equilibrium;
• the Fréchet Inception Distance, which the authors describe as more consistent than the Inception Score.

The paper defines a local Nash equilibrium as a point in parameter space with a neighbourhood in which neither network can lower its own loss by changing only its own parameters. It notes that gradient descent, being a local method, finds only local equilibria.

Method

The update rule

The discriminator $D(\cdot\,; w)$ has parameters $w$ and the generator $G(\cdot\,; \theta)$ has parameters $\theta$. Each is trained on its own loss, $\mathcal{L}_D$ or $\mathcal{L}_G$, which may be the original GAN loss, its non-saturating variant, a Wasserstein loss, or any pair of losses not tied together in a min-max game. With $g(\theta, w) = \nabla_w \mathcal{L}_D$ and $h(\theta, w) = \nabla_\theta \mathcal{L}_G$, a mini-batch gradient equals the true gradient plus noise, $M^{(w)}_n$ or $M^{(\theta)}_n$. TTUR updates the two networks with their own step sizes, $b(n)$ for the discriminator and $a(n)$ for the generator (Eq. 1):

$$w_{n+1} = w_n + b(n)\big(g(\theta_n, w_n) + M^{(w)}_n\big), \qquad \theta_{n+1} = \theta_n + a(n)\big(h(\theta_n, w_n) + M^{(\theta)}_n\big).$$

As printed, the updates add the gradients of the losses; in our reading the sign that makes them descent steps is left implicit.

Assumptions and the convergence theorem

The proof rests on five assumptions, stated in the main text and discussed at length in Section 2 of the supplement:

• (A1) The gradients $h$ and $g$ are Lipschitz. The paper notes that networks with smooth activations such as ELUs satisfy this and ReLU networks do not.
• (A2) Both step sizes satisfy the Robbins–Monro conditions, $\sum_n a(n) = \infty$, $\sum_n a^2(n) < \infty$ and likewise for $b(n)$, and $a(n) = o(b(n))$: the generator’s steps eventually become negligible next to the discriminator’s.
• (A3) The gradient noise terms are martingale difference sequences with bounded conditional second moments, which holds for randomly sampled mini-batches with bounded gradients.
• (A4) For each fixed generator $\theta$, the discriminator’s ordinary differential equation $\dot w(t) = g(\theta, w(t))$ has a locally asymptotically stable attractor $\lambda(\theta)$ that depends on $\theta$ in a Lipschitz way, and the generator’s equation $\dot\theta(t) = h(\theta(t), \lambda(\theta(t)))$ has a locally asymptotically stable attractor $\theta^\ast$. The paper suggests that weight decay can help ensure this by raising the eigenvalues of the Hessian.
• (A5) The iterates stay bounded, which the authors say the objective or weight decay usually ensures.

Theorem 1, which the paper attributes to Borkar (1997), states that under these assumptions the updates converge almost surely to $(\theta^\ast, \lambda(\theta^\ast))$, a stationary local Nash equilibrium. Borkar’s original result required unique, globally asymptotically stable equilibria; the paper takes the relaxation to local attractors from later work by Karmakar and Bhatnagar, reviewed in the supplement. The intuition is that once the generator moves slowly enough, the discriminator sees an almost fixed opponent, converges to its local optimum $\lambda(\theta)$ and tracks it, and the generator then descends against a nearly optimal discriminator. With equal time scales, the paper says, one can only prove that the updates return infinitely often to a neighbourhood of the solution, which can be very large. Figure 1 illustrates this with a panel taken from Zhang et al. (2007), in which a single time-scale algorithm on a network-flow problem keeps returning near the optimum when the gradient errors are small and diverges when they are large.

The paper also observes that the learning rates do not translate directly into perturbations. In its experiments the generator sometimes had the larger learning rate, and Section 5 of the supplement argues that this does not contradict the theory: how much the generator disturbs the discriminator also depends on its loss, its gradients, its architecture and the optimizer, so the learning rates only modulate it.

Adam as a heavy ball with friction

The authors used Adam to reduce mode collapse, reasoning that its running average of gradients acts as a velocity that carries the generator past small local minima, the ones associated with collapse, towards flat minima. Theorem 2 formalizes the heavy-ball reading, though not the claim about mode collapse. With a decaying learning rate $a(n) = a\, n^{-\tau}$ for $\tau \in (0, 1]$, decay rates tied to it, $\beta_1 = 1 - a(n+1)\, r(n)$ and $\beta_2 = 1 - \alpha\, a(n+1)\, r(n)$ for a memory function $r(n)$, the full gradient $\nabla f$ of a lower-bounded, continuously differentiable objective, and stationary second moments of the gradient, Adam follows the heavy ball with friction (HBF) equation

$$\ddot\theta_t + a(t)\, \dot\theta_t + \nabla f(\theta_t) = 0 .$$

The proof identifies Adam’s first-moment update with the stochastic heavy ball recursion of Gadat, Panloup and Saadane (2018), whose continuous limit is this equation. Under the stationarity assumption, the fluctuation of the second-moment estimate $v_n$ around its mean is treated as extra martingale noise, and the constant factor $1/\sqrt{v}$ as a rescaling of the parameters. The energy $\tfrac12 \lvert \dot\theta \rvert^2 + f(\theta)$ is a Lyapunov function that friction decreases, so the perturbed-ODE argument behind Theorem 1 carries over to Adam. The authors conclude that GANs trained with TTUR and Adam converge to a stationary local Nash equilibrium for stationary second moments.

The Fréchet Inception Distance as introduced

The FID is introduced in the experiments section as the performance measure. Two distributions are equal, the paper argues, when the expectations of every function in a basis agree; moments are such expectations of polynomials. The authors apply this to the last pooling layer of a pretrained Inception-v3 instead of raw pixels, keep only the first two moments for practical reasons, and compare the resulting Gaussians with the Fréchet distance. The formula, the Gaussian assumption and the paper’s naming of $d$ against $d^2$ are covered in the Fréchet Inception Distance article.

For the GAN experiments, the real statistics were computed from all images of the training set and the generated statistics from 50,000 samples. FID was computed every 1,000 DCGAN mini-batch updates and every 5,000 WGAN-GP outer iterations.

Results

Disturbed images and the Inception Score

Figure 3 adds six kinds of disturbance, each at four levels, to CelebA face images and plots FID against the level. The supplement (Section 1) specifies them: a mix $(1 - \alpha) X + \alpha N$ of the image $X$ with Gaussian noise $N$ scaled to $[0, 255]$, for $\alpha \in \{0, 0.25, 0.5, 0.75\}$; a Gaussian blur with standard deviation $\alpha \in \{0, 1, 2, 4\}$; five black rectangles of relative size $\alpha \in \{0, 0.25, 0.5, 0.75\}$ at random locations; a swirl of strength $\alpha \in \{0, 1, 2, 4\}$ around the image centre; salt-and-pepper noise flipping a fraction $\alpha \in \{0, 0.1, 0.2, 0.3\}$ of pixels to black or white; and replacing a fraction $\alpha \in \{0, 0.25, 0.5, 0.75\}$ of the CelebA images with images from a pool of 5,000 ImageNet images, five per class. The real statistics come from all CelebA images and the disturbed statistics from 50,000 randomly selected images.

The caption says FID captures the disturbance level very well by increasing monotonically. Read from the plots, FID rises with the level for five of the disturbances, reaching about 220 to 250 for the strongest noise, blur and rectangles, about 140 for the strongest swirl and about 150 at 75% contamination. Salt-and-pepper noise is the exception: FID jumps to about 370 at the first level and stays near that value at the next two.

The supplement compares FID with the Inception Score on the same images. To plot both as distances, the authors bound the score by the number of samples $m$ and define an “Inception Distance” (IND) as $m$ minus the score. In Figure 1 of the supplement (Figure A8 in arXiv v6), IND fluctuates or stays flat for noise, blur, rectangles and swirl, dips and then rises for salt-and-pepper noise, and falls steadily, from about 48 to about 3, as ImageNet images replace faces; read as a distance, this means the Inception Score improves as the data move away from CelebA. The caption summarizes that the Inception Score fluctuates, stays flat or, in the worst case, decreases. The plots’ axes are labelled IND while the caption calls them the Inception Score. The paper also says FID is consistent with human judgment, but it reports no human study; the evidence it gives is these controlled disturbances. The supplement adds sample grids from DCGANs on CelebA at FIDs of 500, 300, 133, 100, 45 and 13, and from a WGAN-GP at an FID of 3.

A toy saddle point

To show the difference between one and two time scales, the paper minimizes over $x$ and maximizes over $y$ the function $f(x, y) = (1 + x^2)(100 - y^2)$, which has a saddle point at $(0, 0)$ with value 100, using gradient steps with additive Gaussian noise. In Figure 4, equal learning rates of 0.01 diverge with large fluctuations, equal rates of 0.001 converge slowly, and a slow $x$ (0.0001) with a fast $y$ (0.01) converges fastest and moves directly to the saddle point; the reverse pairing also converges, more slowly.

GAN experiments

All image experiments compare TTUR with the “original” one time-scale training, in which both networks share one learning rate. Learning rates were tuned to be as large as possible while FID (or the Jensen–Shannon divergence for text) still decreased, and the stopping point was fixed at the update where the best models’ scores stopped improving. Each setting was trained eight times on images and ten times on text, and the training curves show the mean with the range from minimum to maximum.

• DCGAN (Radford et al., 2016) on CelebA, CIFAR-10, SVHN and LSUN Bedrooms (Figure 5). The original training improved faster at first and TTUR ended lower; the text adds that on CelebA and LSUN Bedrooms all one time-scale runs diverged. In every DCGAN setting of Table 1 the generator’s learning rate is larger than the discriminator’s, by a factor of 5 to 50.
• WGAN-GP (Gulrajani et al., 2017) on CIFAR-10 and LSUN Bedrooms (Figure 6). The original code trains the critic five times per generator update; TTUR updates it once, with a learning rate three times the generator’s, so the curves are aligned by wall-clock time. The supplement (Section 4.1) adds the iteration counts: TTUR ran 168k iterations against 53k on CIFAR-10 and 80k against 23k on LSUN in roughly the same time.
• WGAN-GP on the One Billion Word benchmark, a character-level language model whose generator and critic are 1D convolutional networks over sequences of 32 one-hot characters. FID does not apply to text, so the authors used the normalized Jensen–Shannon divergence of 4-gram and 6-gram statistics, as Gulrajani et al. had. The original code trains the critic ten times per generator update; TTUR, once.

Table 1 reports the best score of each method at its optimized number of updates and learning rates ($b$ for the discriminator, $a$ for the generator; lower is better):

Model and data	TTUR $b$, $a$	TTUR updates or time	TTUR score	Original $b = a$	Original updates or time	Original score
DCGAN, CelebA (FID)	1e-5, 5e-4	225k	12.5	5e-4	70k	21.4
DCGAN, CIFAR-10 (FID)	1e-4, 5e-4	75k	36.9	1e-4	100k	37.7
DCGAN, SVHN (FID)	1e-5, 1e-4	165k	12.5	5e-5	185k	21.4
DCGAN, LSUN Bedrooms (FID)	1e-5, 1e-4	340k	57.5	5e-5	70k	70.4
WGAN-GP, CIFAR-10 (FID)	3e-4, 1e-4	700 min	24.8	1e-4	800 min	29.3
WGAN-GP, LSUN Bedrooms (FID)	3e-4, 1e-4	1900 min	9.5	1e-4	2010 min	20.5
WGAN-GP, One Billion Word (4-gram JSD)	3e-4, 1e-4	1150 min	0.35	1e-4	1040 min	0.38
WGAN-GP, One Billion Word (6-gram JSD)	3e-4, 1e-4	1120 min	0.74	1e-4	1070 min	0.77

TTUR scores lower in every row, which the caption describes as a consistently better FID and JSD. The margins differ widely: on CIFAR-10 with DCGAN the gap is 0.8 FID points, on LSUN with WGAN-GP more than half the original score. The table gives no spread across runs. The SVHN row prints exactly the same two FIDs as the CelebA row, 12.5 and 21.4, in every version that contains the table; we cannot tell from the paper whether this is a coincidence or a copying error. Figure 6’s caption names CelebA, CIFAR-10, SVHN and LSUN Bedrooms, although the figure shows only CIFAR-10 and LSUN.

Section 4.3 of the supplement explains why BEGAN (Berthelot et al., 2017) is missing from the table. BEGAN keeps the generator’s and discriminator’s losses in a fixed ratio, which also fixes the relation between their gradients, so adjusting learning rates cannot impose two time scales; with stable learning rates, the curves of original and TTUR training on CelebA and LSUN Bedrooms were alike.

Versions

The dates are from the arXiv submission history; the comparisons are ours, made from the texts of the six versions, the NIPS paper and its supplement.

Version	Date	Changes
v1	26 Jun 2017	Six authors, title “…Converge to a Nash Equilibrium”; experiments with BEGAN, DCGAN and the improved WGAN (61 pages with appendices)
v2	27 Jun 2017	Code link added to the abstract; small corrections
v3	28 Jun 2017	Wording corrections; the appendix table of FIDs under disturbances (Table A7) revised; a v1–v2 remark that experiments ran for at most 200,000 updates removed
v4	13 Jul 2017	Three conditions in the appendix’s rate-of-convergence assumptions corrected (learning rates “nonincreasing” instead of “nondecreasing”, $\epsilon \ge 0$ instead of $\epsilon \le 0$, $\bar a \ge 0$ instead of $\bar a \le 0$)
v5	8 Nov 2017	Published title and five authors; arbitrary GAN losses; new experiment set and FID protocol (38 pages)
v6	12 Jan 2018	Related work rewritten; a sentence extending the setting to GANs that are not min-max games

Versions 1 to 4 formulate TTUR with the gradients of the original GAN loss, mentioning the non-saturating and Wasserstein variants, and describe the limit as a “stationary Nash equilibrium”, although their assumptions already use local attractors. Their experiments differ from the published ones. BEGAN on CelebA reached a best FID of 26.19 with TTUR against 28.55 with original training (their Table 1), and DCGAN on CelebA 17.88 against 20.09 (Table 2). On LSUN Bedrooms, BEGAN reached 112.0 against 112.8 and DCGAN 68.3 against 69.7 (Tables 5 and 6). The improved WGAN on One Billion Word reached JSDs of 0.351 against 0.374 for 4-grams and 0.736 against 0.756 for 6-grams (Table 3). The FID protocol was also smaller: real statistics from 100,000 randomly chosen CelebA or LSUN images, generated statistics from 5,000 samples, evaluated every 5,000 mini-batches; the disturbance experiment used the same 100,000 and 5,000. Figure 1’s oscillation example was on MNIST rather than CelebA. The disturbance FIDs were also printed, in Table A7: the strongest swirl raised FID only from 2.6 to 9.7 (2.0 to 8.5 after v3’s revision), whereas the swirl panel in v5 and later shows a visibly stronger swirl and FID up to about 140, although the appendix’s description of the swirl and its levels is unchanged.

Version 5 introduced the experiment set of the published paper: DCGAN on four image datasets, WGAN-GP on two and on text, with BEGAN moved to the appendix and its earlier TTUR advantage replaced by the finding that the two trainings behave alike. Version 6 changed the related work: where earlier versions cited Nagarajan and Kolter (2017) as showing local stability (v1 to v4: local asymptotic stability of the traditional GAN formulation), v6 says their proof needs strong, unrealistic assumptions or a linear discriminator.

The NIPS paper has the body of v6 in nine pages, with the contributions as a numbered sentence instead of a list. Its supplement has the content of v6’s Appendices A1 to A6 renumbered as Sections 1 to 6, so figure and equation numbers differ: the FID-versus-Inception-Score figure is Figure A8 in v6 and Figure 1 in the supplement, and the FID formula is equation 6 in the v5 and v6 bodies (equation 7 in their Appendix A1), unnumbered in the NIPS paper and equation 1 in its supplement.

Code Release

The paper’s code is in the bioinf-jku/TTUR repository on GitHub (Apache 2.0 licence), which the abstract has linked since v2. As of October 2026 its latest commit is ec65610, from December 2025; it contains TensorFlow code for the DCGAN, BEGAN and WGAN-GP experiments with FID evaluation, the FID-versus-Inception-Score experiment, and a stand-alone fid.py. The README links precalculated statistics for 64 × 64 cropped CelebA (all images), the LSUN bedroom, CIFAR-10 and SVHN training sets, and the ImageNet training and validation sets.

At that commit, fid.py loads the graph from TensorFlow’s inception-2015-12-05 archive, the same pretrained network the Inception Score code uses, and reads the 2048 activations of its pool_3:0 tensor. It feeds images of any size, with values from 0 to 255, into the graph’s ExpandDims:0 input, which in our reading leaves resizing to the graph itself, the step Parmar et al. (2022) later examined. Means and covariances are computed with NumPy’s mean and cov, so the covariance is the sample covariance. The matrix square root uses scipy.linalg.sqrtm; if the result is not finite, $10^{-6}$ is added to the diagonal of both covariances, and small imaginary parts are discarded after a check. This numerically stabler version replaced a direct computation in February 2018 (commit 88621e3), with credit to Dougal J. Sutherland. The README warns that fewer samples than the 2048 feature dimensions leave the covariance rank-deficient, and recommends at least 10,000 samples, since with fewer the true FID would be “underestimated”. Read literally, this conflicts with the upward bias of small-sample FID (see the FID article); in our reading the warning means the generator’s quality is underestimated.

Impact

The FID outgrew the paper that introduced it and became the standard score for image generators; the Fréchet Inception Distance and image generation articles describe its use. The large-scale GAN comparison of Lucic et al. (2018) used it as its main metric and examined it first. Comparing training and test images, they found low variance and a bias, at 10,000 samples, that their text calls high and their caption slight (FID 2.27 for CelebA, 5.19 for CIFAR-10, Figure 1a), strong sensitivity to dropped classes, and a Spearman rank correlation of 0.9 with the same distance computed on the 4096-dimensional FC7 layer of VGG on CelebA. With FID they showed that, given enough hyperparameter search, nearly all the GAN variants they tested reached similar scores.

The update rule was taken up in later GAN training recipes. Self-Attention GAN (Zhang et al., 2019) confirmed that TTUR is effective and advocated it to compensate for the slow learning of regularized discriminators, making it possible to use fewer discriminator steps per generator step; its stabilization experiments used one discriminator step per generator step, and its default learning rates with Adam were 0.0004 for the discriminator and 0.0001 for the generator.

Reviews of the submission, published with the proceedings, valued the convergence proof as a step towards a theory of GAN training and the FID as a useful tool, while one reviewer noted that the assumptions and theorem largely follow Borkar (1997) and the analysis of actor-critic algorithms.

Limitations

The experiments do not satisfy the theorem’s assumptions. Reviewing the submitted version, one reviewer pointed out that the assumptions treat the two networks symmetrically, that the experiments’ learning rates differ only by a constant factor rather than satisfying $a(n) = o(b(n))$, and that a much faster discriminator is known to hurt ordinary GAN training. Two more gaps follow from the paper’s own statements, in our reading: the DCGAN networks use ReLU-type activations, which the paper says violate (A1), and Theorem 2 needs Adam’s decay rates to approach 1 as the learning rate decays and the gradient’s second moments to be stationary, whereas the experiments report fixed learning rates. The supplement’s argument that learning rates only modulate perturbations explains why a faster generator need not contradict the theory, but it is an argument, not a test of the assumptions.

The assumptions can fail in prototypical cases. Mescheder, Geiger and Nowozin (2018) built a “Dirac-GAN”, a one-dimensional example in which data and generator are point masses and the discriminator is linear, on which the continuous training dynamics circle the equilibrium forever. They note that this does not contradict the paper, because the example violates its assumption that the optimal discriminator parameters depend continuously on the generator’s (Lipschitz dependence of $\lambda(\theta)$ in (A4) of the NIPS version, in our reading). They also found that two time-scale updates do not make it converge, and argued that such distributions, concentrated on low-dimensional manifolds, are typical of unregularized GAN training.

The results are thin evidence for the rule. Each comparison reports one best score, selected with the same FID against training-set statistics that is used to report it, without a held-out set or a spread across runs; Lucic et al. (2018) later argued that the randomness of GAN optimization makes it necessary to report the distribution of results rather than the best one, and training, validation and test sets and statistical significance in evaluation cover why both matter. The claim that Adam reduced mode collapse in MNIST experiments is stated without results. In v1 to v4, the BEGAN advantage that the later versions withdrew was reported as a TTUR result.

The FID’s validation was limited. The paper offers controlled disturbances of one dataset, CelebA, rather than a human study, and in the first four versions it estimated the generated statistics from 5,000 samples, half the minimum of 10,000 that its code’s README recommends. The monotonic rise that the caption claims does not hold for salt-and-pepper noise, and Bińkowski et al. (2018) found the Inception Score more monotonic under these disturbances than the paper had reported (see the FID article).

What Came After

The FID’s later critics took up points the paper had not tested: its small-sample bias, which Bińkowski et al. (2018) demonstrated and Chong and Forsyth (2020) corrected for by extrapolating in the sample size; its sensitivity to resizing and compression, which Parmar et al. (2022) traced to the implementations and addressed with Clean-FID; and its dependence on ImageNet classes, which Kynkäänniemi et al. (2023) exposed by matching class histograms. The Fréchet Inception Distance article covers these findings, the alternative metrics, and feature spaces other than Inception-v3.

For training, the paper’s two learning rates became one setting among several remedies for GAN instability, such as gradient penalties and spectral normalization, described in the GAN article. The convergence question it addressed moved on to which training methods converge at all: Mescheder et al. (2018) showed that unregularized training is not always convergent when the distributions are not absolutely continuous, that instance noise or zero-centred gradient penalties make it converge, and that WGAN and WGAN-GP with a fixed number of discriminator updates per generator update do not always converge; for WGANs they noted that two time-scale training might ensure convergence.