DCGAN

DCGAN, the deep convolutional generative adversarial network, is the generative adversarial network design of Radford, Metz and Chintala, posted in November 2015 and presented at ICLR 2016. Its generator turns a 100-dimensional noise vector into an image with a stack of transposed convolutions (the README’s “fractional-strided” convolutions), and its discriminator is a convolutional network with strided convolutions instead of pooling; both use batch normalization, and neither has fully connected hidden layers. The README shows generated bedrooms and faces (the face script produces $64 \times 64$ images). The work’s title claim was unsupervised representation learning: a discriminator trained only to tell real images from generated ones could serve as a feature extractor for classification. Later papers used the name both for the architecture and for its training set-up, as a baseline against which new GAN methods were compared.

The paper itself was not read for this article: its only open copy is on arXiv, which the site does not use. What follows is taken from the authors’ official code, the Theano repository Newmu/dcgan_code at its last commit (ee12b2d, May 2016), whose README summarizes the paper, and from later papers that describe or rerun DCGAN. The paper is listed among the references only to identify the work; every detail below is attributed to the code or to the later source that states it.

Problem Addressed

The GAN paper of 2014 defined the adversarial game; the generative adversarial network article covers its instability. DCGAN’s README states the aim plainly: to “stabilize Generative Adversarial networks with some architectural constraints”, and then to use what the networks had learned. Later papers took up both halves. Salimans et al. (2016, §2) built their improved GANs on “some of the DCGAN architectural innovations”, and Arjovsky, Chintala and Bottou (2017, §4.2) chose DCGAN, trained with the usual non-saturating (“$-\log D$”) loss, as the baseline for the Wasserstein GAN.

Architecture

The README lists the constraints, which it calls the summary of DCGAN:

• replace pooling layers with strided convolutions in the discriminator and fractionally strided convolutions in the generator;
• use batch normalization in both networks;
• remove fully connected hidden layers in deeper architectures, using average pooling at the end;
• use ReLU in every generator layer except the output, which uses $\tanh$;
• use LeakyReLU in every discriminator layer.

The code follows these with two exceptions that the README’s short list leaves out. In the face model (faces/train_uncond_dcgan.py), the generator’s output layer and the discriminator’s first layer have no batch normalization, and the discriminator ends not with average pooling but with a single linear map from its flattened $4 \times 4 \times 1024$ feature map to one sigmoid output, an output layer rather than a hidden one. The Generative Adversarial Network article gives the same guidelines as the paper states them.

Generator

The face generator maps $z \in [-1, 1]^{100}$, drawn uniformly, through a linear projection to $4 \times 4 \times 1024$ values, which are batch-normalized, passed through ReLU and reshaped into a $4 \times 4$ map with 1,024 channels. Four transposed convolutions with $5 \times 5$ kernels and stride 2 then double the resolution at each step while halving the channels, $4 \to 8 \to 16 \to 32 \to 64$ pixels and $1024 \to 512 \to 256 \to 128 \to 3$ channels, each followed by batch normalization and ReLU except the last, which ends in $\tanh$ so that pixel values lie in $[-1, 1]$. The code implements each transposed convolution as the gradient of a strided convolution with respect to its input (deconv in lib/ops.py), the identity that the Transposed Convolution article derives. With a kernel of 5 and a stride of 2, the kernel size is not a multiple of the stride, the uneven overlap that the same article connects to checkerboard artefacts.

By our count from the weight shapes in the script, the generator has about 18.9 million parameters, of which the initial projection holds 1.6 million (9%); the rest are in the transposed convolutions, mostly the first one ($1024 \times 512 \times 5 \times 5$ weights).

Discriminator

The face discriminator mirrors the generator: four convolutions with $5 \times 5$ kernels, stride 2 and padding 2 reduce a $64 \times 64 \times 3$ image to $32 \times 32 \times 128$, $16 \times 16 \times 256$, $8 \times 8 \times 512$ and $4 \times 4 \times 1024$, each followed by LeakyReLU with slope 0.2 (the default in lib/activations.py), and batch normalization on all but the first. Downsampling is learned by the strided convolutions; there is no pooling anywhere in either network. A linear layer and a sigmoid give the probability that the input is real. By the same count it has about 17.2 million parameters.

Other models in the repository

The repository holds three more configurations, which show how loosely the guidelines were applied:

• ImageNet, $32 \times 32$ (imagenet/load_pretrained.py, with released parameters): a 256-dimensional $z$, $3 \times 3$ kernels and 128 base filters, and seven layers in each network (“7l” in the model’s directory name), in which convolutions of stride 1 and stride 2 alternate, so that each resolution gets two convolutions.
• SVHN: the trained generator and discriminator parameters and the discriminator analysis script are released, but no training script; the model’s directory name records an all-convolutional DCGAN with a 100-dimensional Gaussian $z$, learning rate 0.0005 and minibatches of 64.
• Conditional MNIST (mnist/train_cond_dcgan.py, the model of the paper’s appendix according to its README): the one-hot digit label is concatenated to the input and to every layer of both networks, both networks have fully connected hidden layers of 1,024 units, batch normalization has no learned scale and shift, and the generator ends in a sigmoid for pixels in $[0, 1]$. The conditional GAN article covers this and the other ways of conditioning a GAN.

Training

The losses are the standard GAN cross-entropies with the non-saturating generator loss, as written in the face script: with $D(x) \in (0, 1)$ the discriminator’s output,

$$\mathcal{L}_D = -\,\mathbb{E}_{x}\big[\log D(x)\big] - \mathbb{E}_{z}\big[\log\big(1 - D(G(z))\big)\big], \qquad \mathcal{L}_G = -\,\mathbb{E}_{z}\big[\log D(G(z))\big],$$

each estimated on a minibatch. The Generative Adversarial Network article explains why the generator maximizes $\log D(G(z))$ rather than minimizing $\log(1 - D(G(z)))$.

The face script’s settings, all visible in its header:

Setting	Value in faces/train_uncond_dcgan.py
Data	350,000 of the faces in faces_364293_128px.hdf5 (the name suggests 364,293 images of 128 pixels), centre-cropped to $64 \times 64$, scaled to $[-1, 1]$
Minibatch	128
Optimizer	Adam, learning rate 0.0002, $\beta_1 = 0.5$, for both networks
Weight decay	L2, $10^{-5}$, added to the gradients of both networks
Updates	alternate one discriminator update and one generator update, each on its own minibatch
Schedule	25 epochs at constant learning rate (no decay)
Initialization	weights $\mathcal{N}(0, 0.02^2)$; batch-norm gains $\mathcal{N}(1, 0.02^2)$, shifts 0
Latent	$z \sim U[-1, 1]^{100}$

The faces README says that the data file was not released “due to size/data restrictions”, and that training takes about 12 hours; it does not say where the faces came from. The bedroom model’s training script is not in the repository; the README says that its samples after one epoch were produced with a 0.0002 learning rate.

Batch normalization during training. Real and generated minibatches pass through the discriminator in separate calls, so each is normalized with its own statistics, and the generator always normalizes with the statistics of the batch it is generating; lib/ops.py uses stored statistics only when they are passed in, as the SVHN analysis does. Salimans et al. (2016) note two consequences. Batch normalization makes a network’s output for one example depend on the other examples in its minibatch, which led them to virtual batch normalization against a fixed reference batch (§3.5); and the success of batch normalization in the DCGAN discriminator is “well explained” by seeing it as a way of letting the discriminator look at several examples together, the idea behind their minibatch discrimination (§3.2).

Monitoring. The face script prints no image-quality score. After every epoch it draws 100,000 samples and records the mean Euclidean distance from each of 10,000 real images to its nearest neighbour among the first 1,000, 10,000 and 100,000 samples (nnd_score in lib/metrics.py). Those 10,000 real images are the first 10,000 of the training file, not held-out data.

Inference

Generation is one forward pass: draw $z \sim U[-1, 1]^{100}$ and run the generator. Because the generator’s batch normalization uses the statistics of the current batch, a sample depends on the other samples generated with it. The ImageNet demo generates 400 samples, ranks them by the discriminator’s probability of being real, and writes them to a grid in that order.

The discriminator as a feature extractor. The README says the authors used the discriminator “as a pre-trained net for CIFAR-10 classification”. The released analysis script for SVHN (svhn/svhn_semisup_analysis.py) shows the procedure:

1. Replace each batch-normalization layer’s minibatch statistics with fixed ones: the mean and variance of each channel, averaged over all minibatches of the training set, whose labels are not used for this.
2. Take the output of every convolutional layer after its LeakyReLU, max-pool each to a $4 \times 4$ grid ($4 \times 4$ pooling at $16 \times 16$, $2 \times 2$ at $8 \times 8$, none at $4 \times 4$), flatten, and concatenate.
3. Train a linear support vector machine (scikit-learn’s LinearSVC) on 1,000 labelled images, 100 per class, choosing its regularization constant $C$ from 0.0001 to 0.01 by validation error averaged over 10 random labelled sets, then report the test error as mean and $1.96$ standard deviations over 100 random labelled sets.

The features are never fine-tuned; the linear probe only reads what adversarial training put there. The example below runs the same procedure on synthetic shapes.

Performance

The paper’s own tables could not be read. The results below are those of later papers that report or rerun DCGAN, each with its dataset and measure; the numbers are not comparable across rows, since the training budgets, implementations and measures differ.

Features for classification. Salimans et al. (2016, Figure 5) list DCGAN at 22.48% test error on SVHN with 1,000 labels, citing the DCGAN paper, against 23.56% for the stacked what-where autoencoder and 24.63% for virtual adversarial training; their own semi-supervised GAN reached 8.11% ± 1.3. The repository’s last commit (May 2016) is titled “updated svhn semisup result”; the README does not state the result.

Sample quality on CIFAR-10. Gulrajani et al. (2017, Table 3) list an unsupervised DCGAN at an Inception Score of 6.16 ± 0.07, a figure they take from Huang et al. (the SGAN paper), and a DCGAN with labels at 6.58, taken from the SteinGAN paper; their WGAN-GP ResNet scored 7.86 ± 0.07 unsupervised. In their Figure 3, WGAN-GP with Adam reached a score comparable to their DCGAN, which converged faster in wall-clock time. Their own standard DCGAN reached 7.24 on $32 \times 32$ ImageNet (Table 2 caption).

LSUN bedrooms and CelebA faces by FID. Heusel et al. (2017, Table 1) trained DCGAN with the original one time-scale update rule (one learning rate shared by both networks) and with their two time-scale update rule (TTUR), and report the best Fréchet Inception Distance over the number of updates and learning rates they tried: on LSUN Bedrooms 70.4 with the original training and 57.5 with TTUR, on CelebA 21.4 and 12.5, on CIFAR-10 37.7 and 36.9. They add that on CelebA and LSUN Bedrooms all the single-time-scale runs diverged. The FID paper article gives the full table and its learning rates.

Bedroom samples. Arjovsky et al. (2017, Figure 6) show that a DCGAN generator trained with the standard GAN loss on LSUN bedrooms at $64 \times 64$ produced high-quality samples, as did the same generator trained as a Wasserstein GAN. The README itself shows bedrooms after one and after five epochs, and notes repeated textures across samples after five epochs, which it reads as visual under-fitting.

Large, varied data. Salimans et al. (2016, §6.4) found that DCGANs without modification, trained on $128 \times 128$ ImageNet, learn some basic image statistics and produce contiguous shapes with somewhat natural colour and texture, but no recognizable objects.

Latent space. The README describes smooth interpolations between random points in $z$, in which every intermediate image plausibly looks like a bedroom; samples regenerated with the generator filters that draw windows dropped, in which some windows vanish and others turn into doors or mirrors; arithmetic on face codes (smiling woman minus neutral woman plus neutral man gives a smiling man); and a direction in $z$ along which faces turn. These are shown as figures in the repository, not measured.

Strengths

• A recipe that later work reused as is. Gulrajani et al. (2017, §5.2) trained DCGAN with “the default set of optimizer hyperparameters” its authors recommended, and the Wasserstein GAN paper kept the DCGAN architecture for its critic and discriminator while varying the generator (Arjovsky et al., 2017, §4.4).
• All-convolutional and simple. Resolution is set by the number of stride-2 layers, and every layer is a standard convolution or transposed convolution, so the design is easy to scale down for small images or to adapt to other inputs.
• Useful features without labels. In the SVHN comparison of Salimans et al. (2016), the DCGAN discriminator read with a linear classifier had a lower error than two other methods of the time, though well above theirs.

Limitations

• Instability remains. Heusel et al. (2017) report that every single-time-scale DCGAN run on CelebA and LSUN Bedrooms diverged. The standard GAN with a DCGAN discriminator failed to learn when the generator lost its batch normalization and had a constant number of filters, and showed marked mode collapse with an MLP generator (Arjovsky et al., 2017, Figures 7 and 8); Gulrajani et al. (2017, Figure 2) found that only WGAN-GP trained all their architecture variants on LSUN bedrooms with one set of hyperparameters.
• Low resolution and narrow data. The repository’s models are $64 \times 64$ or smaller, and the face and bedroom models each cover one kind of image; on all of ImageNet at $128 \times 128$ an unmodified DCGAN produced no objects (Salimans et al., 2016).
• Batch dependence. Batch normalization ties each output to its minibatch (Salimans et al., 2016, §3.5).
• Checkerboard artefacts. Its $5 \times 5$, stride-2 transposed convolutions overlap unevenly; Transposed Convolution explains the pattern and the remedies.
• No measure of its own. The code monitors only nearest-neighbour distances to training images, and the README’s evidence against memorization is visual (samples after one epoch, smooth interpolations).

Historical Importance

DCGAN (posted in 2015, presented at ICLR 2016) turned the convolutional GAN into a recipe that others could reuse: an all-convolutional generator with batch normalization, trained with Adam at 0.0002 and $\beta_1 = 0.5$. In the papers read for this article it appears both as the architecture to modify and as the baseline to beat. Salimans et al. (2016) built their improved techniques on it, Arjovsky et al. (2017) and Gulrajani et al. (2017) measured their Wasserstein GANs against it, Heusel et al. (2017) tested TTUR and FID on it, and Isola et al. (2017, §3.2) adapted the generator and discriminator of pix2pix from it, using its convolution–BatchNorm–ReLU modules. Its README’s latent-space demonstrations, interpolation and vector arithmetic on faces, are followed into later work by the latent space article.

Its own architecture was soon outgrown. Gulrajani et al. (2017) trained 101-layer ResNet generators and discriminators, which they believed to be the first very deep residual networks trained successfully as a GAN, and the later generators of Progressive GAN, BigGAN and StyleGAN left the plain DCGAN stack behind; the Generative Adversarial Network article traces that line.

Variants

• Conditional DCGANs. The repository’s MNIST model conditions both networks on the class by concatenation. The text-conditioned GAN of Reed et al. (2016), a DCGAN-style network, is described in Text-to-Image Generation.
• Video. VGAN (Vondrick, Pirsiavash & Torralba, 2016), which carried the recipe over to spatio-temporal convolutions, is described in Video Generation.
• Image-conditioned generators. pix2pix adapted DCGAN’s modules to an encoder–decoder generator with skip connections and a patch discriminator (Isola et al., 2017).
• Other losses on the same networks. Wasserstein GAN and WGAN-GP kept DCGAN networks while changing the objective, and WGAN-GP replaced batch normalization in the critic with layer normalization (Gulrajani et al., 2017, §5.2).

Implementations

The official code is the Theano repository Newmu/dcgan_code, under the MIT licence. As of November 2026 its last commit is ee12b2d of May 2016. It needs Theano with cuDNN and contains the face and conditional MNIST training scripts, the SVHN feature analysis, a demo that loads the released $32 \times 32$ ImageNet generator and discriminator, and the released SVHN generator and discriminator; the face data and the bedroom training script are not included. Its README links three other implementations: Torch (soumith/dcgan.torch), Chainer (mattya/chainer-DCGAN) and TensorFlow (carpedm20/DCGAN-tensorflow); Salimans et al. (2016) modified the TensorFlow one for their ImageNet experiments.

Example: The Discriminator as a Feature Extractor

The README’s claim that a DCGAN discriminator is a useful pre-trained network can be tested at toy scale. The script below trains a small DCGAN on 10,000 unlabelled synthetic $32 \times 32$ images, each showing one of five shapes (disk, square, triangle, ring or plus) at a random position, size, rotation and contrast, with pixel noise; pixel values lie in $[-1, 1]$, with the background set darker than the shape. It then reads the discriminator with the procedure of the official SVHN script, described under Inference, and compares it with two controls: an untrained discriminator with the same architecture and initialization, which shows what the architecture gives without adversarial training, and the raw pixels. The networks follow the face model, narrowed to 32 base filters and shortened to three stride-2 layers for $32 \times 32$ grayscale input, with the face script’s initialization, Adam settings, weight decay, uniform $z$ and alternating updates, for 10 epochs. Linear SVMs are trained on 10 or 100 labelled images per class from a separate pool, with $C$ chosen from the official grid on 1,000 labelled validation images (the official script also chooses $C$ on a validation set much larger than its training labels), and tested on 2,000 held-out images. Test accuracy is averaged over the same 10 labelled draws that choose $C$, where the official script draws 100 fresh labelled sets for testing; the smaller number keeps the runtime down. The data are the same for every seed; the seed, given as the script’s argument, sets the networks’ initialization, the GAN’s minibatch order and its noise. With PyTorch 2.14.1, NumPy 2.5.4 and scikit-learn 1.9.1 on one CPU thread, a run took four and a half to five and a half minutes on our machine, well above a minute; repeated runs with seed 0 gave identical output.

import sys
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.svm import LinearSVC

torch.set_num_threads(1)
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0

# Synthetic 32x32 images in [-1, 1]: one disk, square, triangle, ring or plus each,
# at a random position, size, rotation and contrast, with pixel noise.
def shapes(n, rng, size=32):
    yy, xx = np.mgrid[0:size, 0:size] + 0.5
    X = np.empty((n, size, size), np.float32)
    y = rng.integers(0, 5, n)
    for i in range(n):
        cx, cy = rng.uniform(10, 22, 2)
        r = rng.uniform(5, 9)
        a = rng.uniform(0, 2 * np.pi)
        u, v = xx - cx, yy - cy
        ur = np.cos(a) * u + np.sin(a) * v
        vr = -np.sin(a) * u + np.cos(a) * v
        k = y[i]
        if k == 0:                                          # disk
            m = u**2 + v**2 <= r**2
        elif k == 1:                                        # square
            m = (np.abs(ur) <= 0.8 * r) & (np.abs(vr) <= 0.8 * r)
        elif k == 2:                                        # triangle
            m = np.ones_like(u, bool)
            for t in range(3):
                b = a + 2 * np.pi * t / 3
                m &= np.cos(b) * u + np.sin(b) * v <= r / 2
        elif k == 3:                                        # ring
            d = np.sqrt(u**2 + v**2)
            m = (d <= r) & (d >= 0.55 * r)
        else:                                               # plus
            w = 0.3 * r
            m = ((np.abs(ur) <= w) & (np.abs(vr) <= r)) | ((np.abs(vr) <= w) & (np.abs(ur) <= r))
        fg, bg = rng.uniform(0.2, 1.0), rng.uniform(-1.0, -0.2)
        X[i] = np.clip(np.where(m, fg, bg) + rng.normal(0, 0.1, (size, size)), -1, 1)
    return X, y

rng = np.random.default_rng(1234)                           # the data are the same for every seed
X_unlab, _ = shapes(10000, rng)                             # GAN training images, labels discarded
X_pool, y_pool = shapes(2000, rng)                          # labelled pool for the probes
X_val, y_val = shapes(1000, rng)                            # choose the SVM's C
X_test, y_test = shapes(2000, rng)                          # held-out shapes

NZ, NGF, NDF, B = 100, 32, 32, 128

class Generator(nn.Module):                                 # official face model, 32x32 and narrower
    def __init__(s):
        super().__init__()
        s.fc = nn.Linear(NZ, 4 * NGF * 16, bias=False); s.bn0 = nn.BatchNorm1d(4 * NGF * 16)
        s.d1 = nn.ConvTranspose2d(4 * NGF, 2 * NGF, 5, 2, 2, output_padding=1, bias=False)
        s.bn1 = nn.BatchNorm2d(2 * NGF)
        s.d2 = nn.ConvTranspose2d(2 * NGF, NGF, 5, 2, 2, output_padding=1, bias=False)
        s.bn2 = nn.BatchNorm2d(NGF)
        s.d3 = nn.ConvTranspose2d(NGF, 1, 5, 2, 2, output_padding=1, bias=False)
    def forward(s, z):
        h = F.relu(s.bn0(s.fc(z))).view(-1, 4 * NGF, 4, 4)  # project and reshape
        h = F.relu(s.bn1(s.d1(h)))                          # 8x8
        h = F.relu(s.bn2(s.d2(h)))                          # 16x16
        return torch.tanh(s.d3(h))                          # 32x32, no batch norm

class Discriminator(nn.Module):
    def __init__(s):
        super().__init__()
        s.c1 = nn.Conv2d(1, NDF, 5, 2, 2, bias=False)       # no batch norm on the first layer
        s.c2 = nn.Conv2d(NDF, 2 * NDF, 5, 2, 2, bias=False); s.bn2 = nn.BatchNorm2d(2 * NDF)
        s.c3 = nn.Conv2d(2 * NDF, 4 * NDF, 5, 2, 2, bias=False); s.bn3 = nn.BatchNorm2d(4 * NDF)
        s.fc = nn.Linear(4 * NDF * 16, 1, bias=False)
    def layers(s, x):
        h1 = F.leaky_relu(s.c1(x), 0.2)                     # 16x16
        h2 = F.leaky_relu(s.bn2(s.c2(h1)), 0.2)             # 8x8
        h3 = F.leaky_relu(s.bn3(s.c3(h2)), 0.2)             # 4x4
        return h1, h2, h3
    def forward(s, x):
        return s.fc(s.layers(x)[2].flatten(1)).squeeze(1)   # logit

def init(net):                                              # N(0, 0.02); batch-norm gains N(1, 0.02)
    for m in net.modules():
        if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d, nn.Linear)):
            nn.init.normal_(m.weight, 0.0, 0.02)
        elif isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d)):
            nn.init.normal_(m.weight, 1.0, 0.02); nn.init.zeros_(m.bias)

torch.manual_seed(SEED)
G, D = Generator(), Discriminator(); init(G); init(D)
D_random = Discriminator(); init(D_random)                  # same architecture and init, never trained
opt_g = torch.optim.Adam(G.parameters(), 2e-4, betas=(0.5, 0.999), weight_decay=1e-5)
opt_d = torch.optim.Adam(D.parameters(), 2e-4, betas=(0.5, 0.999), weight_decay=1e-5)
Xt = torch.from_numpy(X_unlab)[:, None]
gen = torch.Generator().manual_seed(SEED)
ones, zeros = torch.ones(B), torch.zeros(B)
for epoch in range(10):
    perm = torch.randperm(len(Xt), generator=gen)
    for i in range(0, len(Xt) // B - 1, 2):                 # alternate: one D update, one G update
        x = Xt[perm[i * B:(i + 1) * B]]
        z = torch.rand(B, NZ, generator=gen) * 2 - 1        # z ~ U[-1, 1]^100
        loss_d = F.binary_cross_entropy_with_logits(D(x), ones) + \
                 F.binary_cross_entropy_with_logits(D(G(z).detach()), zeros)
        opt_d.zero_grad(); loss_d.backward(); opt_d.step()
        z = torch.rand(B, NZ, generator=gen) * 2 - 1
        loss_g = F.binary_cross_entropy_with_logits(D(G(z)), ones)   # non-saturating
        opt_g.zero_grad(); loss_g.backward(); opt_g.step()

def features(net, X):
    """As svhn_semisup_analysis.py: batch norm with averaged training-batch statistics,
    each layer max-pooled to a 4x4 grid, all layers concatenated."""
    Xu = torch.from_numpy(X_unlab)[:, None]
    m2 = v2 = m3 = v3 = 0; n = 0
    with torch.no_grad():
        for i in range(0, len(Xu), B):
            a2 = net.c2(F.leaky_relu(net.c1(Xu[i:i + B]), 0.2))
            m2 = m2 + a2.mean((0, 2, 3)); v2 = v2 + a2.var((0, 2, 3), unbiased=False)
            a3 = net.c3(F.leaky_relu(F.batch_norm(a2, a2.mean((0, 2, 3)), a2.var((0, 2, 3), unbiased=False),
                                                  net.bn2.weight, net.bn2.bias, eps=net.bn2.eps), 0.2))
            m3 = m3 + a3.mean((0, 2, 3)); v3 = v3 + a3.var((0, 2, 3), unbiased=False)
            n += 1
        net.bn2.running_mean, net.bn2.running_var = m2 / n, v2 / n
        net.bn3.running_mean, net.bn3.running_var = m3 / n, v3 / n
        net.eval()
        h1, h2, h3 = net.layers(torch.from_numpy(X)[:, None])
        f = torch.cat([F.max_pool2d(h1, 4).flatten(1), F.max_pool2d(h2, 2).flatten(1), h3.flatten(1)], 1)
    return f.numpy()

CS = [0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01]   # the official grid
def probe(Fp, Fv, Ft, per_class, draws=10):
    r = np.random.default_rng(0)                            # the same labelled draws for every condition
    sets = [np.concatenate([r.choice(np.where(y_pool == k)[0], per_class, replace=False)
                            for k in range(5)]) for _ in range(draws)]
    svm = lambda c, ix: LinearSVC(C=c, max_iter=20000, random_state=0).fit(Fp[ix], y_pool[ix])
    c = max(CS, key=lambda c: np.mean([svm(c, ix).score(Fv, y_val) for ix in sets]))
    acc = 100 * np.array([svm(c, ix).score(Ft, y_test) for ix in sets])
    return acc.mean(), 1.96 * acc.std()

print(f"seed {SEED}; GAN trained on {len(X_unlab)} unlabelled images; test set {np.bincount(y_test)} per class")
print("linear SVM test accuracy (%), mean +- 1.96 sd over 10 labelled draws; C chosen on validation")
print(f"{'features':22s}{'10 per class':>16s}{'100 per class':>16s}")
for name, fn in [("trained discriminator", lambda X: features(D, X)),
                 ("random discriminator", lambda X: features(D_random, X)),
                 ("raw pixels", lambda X: X.reshape(len(X), -1))]:
    Fp, Fv, Ft = fn(X_pool), fn(X_val), fn(X_test)
    row = [probe(Fp, Fv, Ft, k) for k in (10, 100)]
    print(f"{name:22s}" + "".join(f"{m:10.1f} +-{s:4.1f}" for m, s in row))

seed 0; GAN trained on 10000 unlabelled images; test set [436 405 409 372 378] per class
linear SVM test accuracy (%), mean +- 1.96 sd over 10 labelled draws; C chosen on validation
features                  10 per class   100 per class
trained discriminator       42.2 +- 4.9      63.8 +- 1.8
random discriminator        37.8 +- 4.4      57.5 +- 1.8
raw pixels                  31.8 +- 2.7      38.7 +- 1.9

Raw pixels are the weakest representation, below 40% even with 100 labels per class. In our reading, the shapes move, rotate and change contrast, so a linear classifier on pixels would need to learn each placement of each shape separately. The untrained discriminator is already much better, an advantage that comes from the architecture and the probe’s pooling alone; the untrained control plays the same role as in the self-supervised learning article’s example, which tests rotation pretraining (there, on other shapes and with a smaller network, the untrained features scored below raw pixels in the run shown). Adversarial training adds to it: the trained discriminator is about 4 points better than its untrained copy with 10 labels per class and about 6 points better with 100, on the same labelled draws. Its only training signal was to tell real shapes from the generator’s attempts, so what it gained over the untrained copy came without any labels.

Seeds 1 and 2 (with the script saved as dcgan_probe.py, python dcgan_probe.py 1 and python dcgan_probe.py 2) give the same order. Over seeds 0 to 2, the trained discriminator scored 42.2% to 45.5% with 10 labels per class against 36.7% to 38.6% for the untrained copy, and 63.5% to 65.2% with 100 labels against 57.3% to 58.4%; the pixel results do not depend on the seed. The margin over the untrained copy is the part of the result that can be credited to the GAN, and it is modest next to the margin the architecture gives over pixels. The absolute accuracies say nothing about the published SVHN or CIFAR-10 numbers: the networks are tiny, the GAN trained for 10 epochs on 10,000 images, and the probe sees at most 500 labelled images.