pix2pix

pix2pix is a conditional GAN for paired image-to-image translation: trained on aligned pairs such as label maps and street photographs, it learns to turn a new input image into a corresponding output image. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A. Efros presented it at CVPR 2017 under the title “Image-to-Image Translation with Conditional Adversarial Networks”; the name comes from the software released with the paper. The model has three parts: an objective that adds an $L^1$ distance to the target to a conditional adversarial loss, a U-Net generator, and a convolutional “PatchGAN” discriminator that judges local patches rather than whole images. The same architecture and objective, trained on different data, handled labels to photos, maps to aerial photos, edges to photos, grayscale to colour and day to night, problems that had each had special-purpose methods. Its high-resolution successor, pix2pixHD (Wang et al., 2018), keeps the conditional GAN and replaces the generator, the discriminator and the loss to reach $2048 \times 1024$ pixels.

The image-to-image translation article surveys the task, its datasets and the methods that followed; this article covers the model and pix2pixHD in detail. Unless a version is named, details of pix2pix come from the CVPR version of the paper. Its architecture and training details are in supplemental materials that the CVPR text points to online and that were not read for this article; they are taken here from the official code, which the article names each time.

Problem Addressed

A convolutional network trained to regress pixels needs a loss, and Isola et al. (§1) argue that choosing it is where the hand engineering had moved. Euclidean distance is minimized by averaging all plausible outputs, so it blurs; the same holds, they show, for $L^1$, which is minimized by the per-pixel median (§4.2). Earlier work had treated each output pixel as conditionally independent given the input, or had used structured losses that penalize the joint configuration of the output: conditional random fields, SSIM, feature matching, nonparametric losses, the convolutional pseudo-prior or matched covariance statistics (§2).

A GAN learns its loss instead: a discriminator trained to tell real from generated outputs penalizes any structure that differs between them, and “blurry images will not be tolerated since they look obviously fake” (§1). Image-conditional GANs already existed for single problems, such as images from surface normal maps, future frames, product photos and images from sparse annotations, and several papers, on inpainting and other problems, had added an unconditional GAN to a term such as $L^2$ regression. pix2pix’s stated contributions are to show that one conditional GAN gives reasonable results on a wide range of such problems, and to present a simple framework and analyse its main architectural choices (§1–§2).

Architecture

Both networks are adapted from those of DCGAN and use modules of the form convolution–BatchNorm–ReLU (§3.2). The discriminator sees the input image together with the output, so it judges whether a pair matches, not only whether the output looks real (Fig. 2).

Generator: U-Net with skips

Input and output in these problems differ in appearance but render the same structure, so edges and layout are aligned between them. An encoder–decoder would force all of that through its bottleneck; pix2pix adds skip connections that concatenate all channels at layer $i$ with those at layer $n - i$, where $n$ is the number of layers, following the shape of a U-Net (§3.2.1). Severing the skips turned the generator into an encoder–decoder that could not learn to generate realistic Cityscapes photos from label maps, and the U-Net was also better when both were trained with $L^1$ alone (§4.3, Fig. 4).

In the official code, the default unet_256 generator for $256 \times 256$ images has eight encoder layers of $4 \times 4$ convolutions with stride 2, taking the image down to $1 \times 1$ with 64, 128, 256 and then 512 channels, LeakyReLU with slope 0.2 and BatchNorm on every layer except the first and the innermost. Eight transposed convolutions bring it back up, with ReLU, BatchNorm and concatenated skips, and a tanh output (defineG_unet in the Torch models.lua; UnetGenerator in the PyTorch networks.py).

Noise as dropout

A conditional GAN given only $x$ would be deterministic and could not match any distribution other than a delta function, so earlier conditional GANs fed Gaussian noise $z$ alongside $x$. In Isola et al.’s first experiments the generator learned to ignore that noise. The final models provide noise only as dropout, applied in several generator layers at both training and test time, and even so the outputs showed only minor stochasticity; the authors leave generators that capture the full entropy of the conditional distribution as an open question (§3.1). In the Torch code, dropout with rate 0.5 follows the three innermost decoder layers; the PyTorch code leaves the innermost one without it and applies it after the next three.

Discriminator: PatchGAN

The discriminator is restricted to high-frequency structure. $L^1$ and $L^2$ blur, but they often capture the low frequencies correctly, so the $L^1$ term is trusted to enforce low-frequency correctness and the discriminator only needs to look at local patches (§3.2.2). The PatchGAN classifies each $N \times N$ patch as real or fake; it runs convolutionally across the image and averages all responses into the output of $D$. Isola et al. call it a Markovian discriminator: it treats the image as a Markov random field in which pixels farther apart than a patch diameter are independent, the assumption common to models of texture and style, so the PatchGAN “can therefore be understood as a form of texture/style loss” (§3.2.2). They credit Li and Wand’s Markovian GAN paper on real-time texture synthesis (ECCV 2016), not their CVPR 2016 paper on Markov random fields and CNNs, with a similar architecture for capturing local style statistics (§2). A smaller PatchGAN has fewer parameters, runs faster and applies to images of any size (§3.2.2).

The patch size is set by the discriminator’s depth (§4.4, footnote 1). In the official code the default “basic” discriminator is a stack of $4 \times 4$ convolutions on the six-channel concatenation of input and output: three with stride 2 (64, 128 and 256 channels, BatchNorm on all but the first, LeakyReLU 0.2), one with stride 1 (512 channels) and a final one-channel convolution with stride 1. The receptive field of each output unit is $70 \times 70$ pixels, and a $256 \times 256$ pair gives a $30 \times 30$ map of patch scores. Removing or adding stride-2 layers gives receptive fields of 16, 34, 70, 142 and 286 pixels for one to five of them, and a separate “PixelGAN” of three $1 \times 1$ convolutions gives a receptive field of one pixel (defineD_n_layers and defineD_pixelGAN in models.lua). The 286-pixel version covers the whole $256 \times 256$ image and is called the ImageGAN.

Training

Given pairs $(x, y)$ of input and target, the conditional adversarial objective is

$$\mathcal{L}_{cGAN}(G, D) = \mathbb{E}_{x,y}[\log D(x, y)] + \mathbb{E}_{x,z}[\log(1 - D(x, G(x, z)))],$$

which $D$ maximizes and $G$ minimizes (Eq. 1). For comparison the paper also trains an unconditional discriminator that does not see $x$, with $D(y)$ and $D(G(x, z))$ in place of $D(x, y)$ and $D(x, G(x, z))$ (Eq. 2). The generator also pays an $L^1$ distance to the target, which blurs less than $L^2$ (Eq. 3), and the final objective is

$$G^* = \arg\min_G \max_D \; \mathcal{L}_{cGAN}(G, D) + \lambda\, \mathbb{E}_{x,y,z}\big[\lVert y - G(x, z) \rVert_1\big] \qquad \text{(Eq. 4)},$$

with $\lambda = 100$ in the experiments (§4.2). Adding the $L^1$ term leaves the discriminator’s job unchanged; it only asks the generator to stay near the ground truth as well (§3.1).

Optimization follows the original GAN recipe of one gradient step on $D$, then one on $G$, with minibatch SGD and Adam (§3.3); batch sizes were between 1 and 10 depending on the experiment. The official code fills in the rest:

• Generator loss. $G$ is trained to make $D$ label its outputs real, the non-saturating form described in the generative adversarial network article, using binary cross-entropy (backward_G in pix2pix_model.py; the same in train.lua).
• Discriminator loss. The PyTorch code halves the sum of the real and fake terms before the update; the Torch code accumulates the gradients of both terms and reports their mean.
• Optimizer and schedule. Adam with learning rate 0.0002 and $\beta_1 = 0.5$, weights initialized from $\mathcal{N}(0, 0.02)$. The Torch default trains for 200 epochs at a constant rate; the PyTorch default trains 100 epochs at it and decays it linearly to zero over 100 more.
• Data. Batch size 1 by default; images loaded at $286 \times 286$, randomly cropped to $256 \times 256$ and flipped.

Batch normalization runs with the statistics of the current batch at test time as well as during training. With a batch of one, this is what Ulyanov et al. had called instance normalization (§3.3).

Inference

The generator runs exactly as during training: dropout stays on and batch normalization uses the test batch’s statistics rather than averages collected during training (§3.3). The input is a 1- to 3-channel image and so is the output (§4). Because both networks are fully convolutional, a generator trained on $256 \times 256$ map and aerial photos ran on $512 \times 512$ images (§4.4, Fig. 7). The PyTorch code’s U-Net needs sizes divisible by 256 (docs/tips.md).

Performance

FCN-score and the loss ablation

To score realism automatically, Isola et al. trained an FCN-8s semantic segmentation network on Cityscapes and segmented the photos generated from Cityscapes label maps; if the photos are realistic, a classifier trained on real images should recover the labels they came from (§4.1). The score is per-pixel accuracy, per-class accuracy and class IoU against those labels. The official repository’s evaluation notes add that the script uses the Cityscapes validation set, and that the FCN was trained on $256 \times 256$ images upsampled to $1024 \times 2048$, so results are synthesized at $256 \times 256$ and upsampled before scoring (docs/tips.md).

Loss (Cityscapes labels→photo)	Per-pixel acc.	Per-class acc.	Class IoU
$L^1$	0.42	0.15	0.11
GAN (unconditional)	0.22	0.05	0.01
cGAN	0.57	0.22	0.16
$L^1$ + GAN	0.64	0.20	0.15
$L^1$ + cGAN	0.66	0.23	0.17
Ground truth	0.80	0.26	0.21

These are Table 1 of the paper. $L^1$ alone gave reasonable but blurry photos, the cGAN alone sharper ones with artifacts on some applications, and the sum reduced the artifacts (§4.2, Fig. 3). The unconditional discriminator collapsed: the generator produced nearly the same output for every input, since the loss no longer penalized a mismatch between input and output. Adding $L^1$ also ties output to input, which is why $L^1$ + GAN scores almost as well as $L^1$ + cGAN (§4.2).

Patch size

With $L^1$ + cGAN throughout, the discriminator’s receptive field was varied on Cityscapes labels→photo (Table 2; input images $256 \times 256$, larger receptive fields padded with zeros):

Discriminator receptive field	Per-pixel acc.	Per-class acc.	Class IoU
$1 \times 1$ (PixelGAN)	0.39	0.15	0.10
$16 \times 16$	0.65	0.21	0.17
$70 \times 70$	0.66	0.23	0.17
$286 \times 286$ (ImageGAN)	0.42	0.16	0.11

The PixelGAN had no effect on spatial sharpness but made outputs more colourful: a bus painted grey under $L^1$ turned red. The $16 \times 16$ PatchGAN gave locally sharp results with tiling artifacts beyond the scale it can observe; the $70 \times 70$ one removed them, and was the default for every other experiment. The ImageGAN looked about as good as $70 \times 70$ but scored considerably lower, which the authors suggest may be because its extra parameters and depth make it harder to train (§4.4, Fig. 5).

The colour claim is measured as histogram intersection between the marginal distributions of output and ground-truth colours in Lab space on Cityscapes (Fig. 6d). $L^1$ gave 0.81, 0.69 and 0.70 for the L, a and b channels; cGAN 0.87, 0.74 and 0.84; $L^1$ + cGAN 0.86, 0.84 and 0.82; and the PixelGAN 0.83, 0.68 and 0.78 (the table’s row “PixelGAN” appears in the plots’ legend as “L1+pixelcGAN”). The plotted marginals show $L^1$ narrower than the ground truth, the grey average that $L^1$ favours when a colour is uncertain (§4.2). The PixelGAN’s intersections are higher than $L^1$’s in L and b but slightly lower in a; in our reading, the table alone does not show its larger colour spread, and the caption notes that the scores are dominated by the high-probability region, where the differences are imperceptible in the plots.

Perceptual studies

The human studies on Amazon Mechanical Turk followed the protocol of Zhang, Isola and Efros’s colorization paper: a real and a generated image appeared for one second each, and workers then had unlimited time to say which was fake. Each session had 10 practice trials with feedback and 40 without, tested one algorithm, and was limited to one per worker; about 50 workers rated each algorithm, at $256 \times 256$, without the vigilance trials of the original protocol. For colorization the real and fake images came from the same grayscale input; for maps and aerial photos they did not, to make the task harder and avoid results at the floor (§4.1).

Task	Method	Rated real
Map→aerial photo	$L^1$	0.8% ± 0.3%
Map→aerial photo	$L^1$ + cGAN	18.9% ± 2.5%
Aerial photo→map	$L^1$	2.8% ± 1.0%
Aerial photo→map	$L^1$ + cGAN	6.1% ± 1.3%
Colorization (ImageNet)	$L^2$ regression of Zhang et al.	16.3% ± 2.4%
Colorization (ImageNet)	Zhang et al. (2016), full method	27.8% ± 2.7%
Colorization (ImageNet)	pix2pix, $L^1$ + cGAN	22.5% ± 1.6%

These are Tables 3 and 4. Generated aerial photos fooled workers significantly more often than the $L^1$ baseline, but generated maps did not differ significantly from it by a bootstrap test, perhaps because errors are more visible in rigid map geometry than in chaotic aerial photos (§4.5). In colorization, trained on ImageNet and tested on the split of Zhang et al. and Larsson et al., pix2pix did not differ significantly from Zhang et al.’s $L^2$ variant and fell short of their full method, which was engineered for colorization (§4.5).

Semantic segmentation as translation

Run in reverse, on Cityscapes photo→labels, a cGAN without $L^1$ produced label maps with reasonable accuracy, which the authors describe as the first demonstration of a GAN generating nearly discrete labels. Yet $L^1$ regression alone scored better: per-pixel accuracy 0.86 against 0.74 for cGAN and 0.83 for $L^1$ + cGAN, and class IoU 0.35, 0.22 and 0.29 (Table 5). The cGAN’s outputs contained many small hallucinated objects (Fig. 11). The authors argue that vision problems with a less ambiguous answer need little more than a reconstruction loss (§4.6).

Strengths

• One recipe for many problems. The same architecture and objective served graphics tasks such as label→photo, edges→photo and colorization, with only the training data changed (Fig. 1).
• A learned structured loss. The discriminator penalizes whatever distinguishes outputs from real images in context, so no task-specific loss is needed.
• Small, local discriminator. The $70 \times 70$ PatchGAN scored higher on the FCN-score than the whole-image discriminator with fewer parameters, and like the generator it applies to images of any size.

Limitations

• Paired data. Every training example needs an aligned input and output; CycleGAN, by an overlapping group of authors, drops that requirement.
• Little diversity. Dropout gives only minor stochasticity (§3.1), so the model returns nearly one output per input although the problems are one-to-many.
• Hallucination and failures. The cGAN invents structure absent from the input, sharp “even if incorrect” (Fig. 5), adds small objects to label maps (Fig. 11), and has a common failure mode of grey or desaturated colorizations (Fig. 8).
• Resolution. Models were trained on $256 \times 256$ images, and Wang et al. (2018, §3.1) describe pix2pix’s results as up to that size; they found training unstable and quality unsatisfactory when they applied pix2pix directly to high-resolution images.
• Evaluation. The FCN-score depends on one segmenter trained at low resolution, and later papers ran other perceptual protocols, such as pix2pixHD’s pairwise preferences, so the human-study numbers are not comparable across papers.

Historical Importance

Image-conditional GANs had each been tailored to one application; pix2pix presented one that is not, and its authors framed this generality as the paper’s main contribution (§1–§2). The CVPR version also devotes a section to applications by the online community after the code was released, from #edges2cats to depth→street view, background removal, palette generation and pose transfer (§4.7, Fig. 10), and concludes that conditional adversarial networks are promising for translation tasks with highly structured graphical outputs (§5).

Its components were reused. CycleGAN used $70 \times 70$ PatchGANs for its discriminators (Zhu et al., 2017, §4), and pix2pixHD runs patch discriminators at several scales. Patch discriminators also became one term among others in the losses of autoencoders: VQGAN (Esser, Rombach and Ommer, 2021, §3.1) trains its autoencoder with a perceptual loss and a patch-based discriminator citing pix2pix, and the autoencoder of latent diffusion, based on VQGAN’s, does the same (Rombach et al., 2022, §3.1).

Variants

pix2pixHD (2018)

Wang, Liu, Zhu, Tao, Kautz and Catanzaro synthesized $2048 \times 1024$ images from semantic label maps, after pix2pix had become unstable at that size and Chen and Koltun’s cascaded refinement network (CRN), trained with a perceptual regression loss, gave high-resolution results that lacked fine detail (§1–§3.1). They made three changes (§3.2), and added two inputs for editing (§3.3–§3.4). Details below are from the CVPR version, which refers to its arXiv version for more training and architecture details.

Coarse-to-fine generator. A global generator $G_1$ works at $1024 \times 512$: a convolutional front-end, a set of residual blocks and a transposed-convolution back-end, built on the style-transfer network of Johnson et al. A local enhancer $G_2$ with the same three parts works at twice the resolution in each dimension; the input to its residual blocks is the element-wise sum of its own front-end features and the last feature map of $G_1$’s back-end. $G_1$ is trained first, then $G_2$, then both are fine-tuned together; a further enhancer would give $4096 \times 2048$ (§3.2, Fig. 3).

Multi-scale discriminators. A discriminator for high-resolution images needs a large receptive field, which takes a deeper network or larger kernels, with more capacity to overfit and more memory. pix2pixHD instead uses three discriminators $D_1, D_2, D_3$ of identical architecture on the real and synthesized images downsampled by factors 1, 2 and 4. The coarsest has the largest receptive field relative to the image and guides global consistency; the finest guides details. Without them, many repeated patterns appeared. A model can be extended to a higher resolution by adding one discriminator at the finest scale (§3.2):

$$\min_G \max_{D_1, D_2, D_3} \sum_{k=1}^{3} \mathcal{L}_{GAN}(G, D_k).$$

Feature-matching loss. With $D_k^{(i)}$ the first $i$ layers of discriminator $D_k$, $T$ layers and $N_i$ elements in layer $i$, and $s$ the label map,

$$\mathcal{L}_{FM}(G, D_k) = \mathbb{E}_{(s,x)} \sum_{i=1}^{T} \frac{1}{N_i} \Big[\big\lVert D_k^{(i)}(s, x) - D_k^{(i)}(s, G(s)) \big\rVert_1\Big]$$

compares the discriminator’s intermediate features of real and synthesized images (Eq. 3). The full objective adds $\lambda \sum_k \mathcal{L}_{FM}(G, D_k)$ to the adversarial terms with $\lambda = 10$; the discriminators only extract features for this term and do not maximize it (Eq. 4, §4). The objective has no $L^1$ term to the target: the authors report results without hand-crafted losses or pretrained networks (§1), and relate the feature-matching loss to the perceptual loss, whose features come from a fixed pretrained network rather than from the discriminator being trained. pix2pixHD uses the least-squares GAN loss. An optional VGG feature loss, $\lambda \sum_i \frac{1}{M_i} \lVert F^{(i)}(x) - F^{(i)}(G(s)) \rVert_1$ with $\lambda = 10$, slightly improved results; the variant without it is reported as “ours (w/o VGG loss)” (§4).

Instance boundaries and feature embedding. Semantic labels do not separate adjacent objects of one class, such as parked cars. pix2pixHD computes an instance boundary map, 1 where a pixel’s object ID differs from any of its four neighbours, and concatenates it with the one-hot label map at the input of the generator and the discriminator (§3.3, Fig. 4). For diverse outputs, an encoder $E$ maps the real image to a 3-dimensional feature per instance, averaged within each instance, and the generator receives $G(s, E(x))$; after training, K-means with $K = 10$ on each category’s features gives styles, such as asphalt or cobblestone for road, that a user picks per object (§3.4, §4).

Results. Segmenting synthesized Cityscapes images with PSPNet, as in pix2pix’s FCN-score, gave pixel accuracy 78.34 and mean IoU 0.3948 for pix2pix, 70.55 and 0.3483 for CRN, and 83.78 and 0.6389 for pix2pixHD, against 84.29 and 0.6857 for the original images (Table 1). The pix2pix baseline was trained on the high-resolution images with its default settings. In pairwise MTurk tests with unlimited time on the 500 Cityscapes test images, each compared 10 times, workers preferred pix2pixHD to pix2pix in 93.8% of judgments and to CRN in 86.2%, or 94.6% and 85.2% without the VGG loss (Table 2). The final model was preferred to a model trained with only the GAN loss in 68.55% of judgments and to one with GAN and feature-matching losses in 58.90%; car regions with instance maps were preferred in 64.34%; and on NYU Indoor RGBD the preference rates were 86.7% against pix2pix and 63.7% against CRN (§4.2). The CVPR version also reports edges→photo results on CelebA-HQ and cat images, label→photo on ADE20K, and interactive editing of facial parts on the Helen Face dataset (Figs. 2, 10, 11).

The official code (NVIDIA/pix2pixHD, as of November 2026) defaults to two discriminators for the $1024 \times 512$ global generator, and its script for $2048 \times 1024$ adds a third when it trains the local enhancer, the extension the paper describes. It weights each discriminator’s feature-matching term by $1/(\text{number of discriminators})$ and each layer’s by $4/(n + 1)$ for $n$ = n_layers_D (default 3), on top of $\lambda = 10$.

Other successors

The image-to-image translation article covers what followed: unpaired translation with CycleGAN, UNIT, MUNIT and CUT; SPADE’s spatially adaptive normalization for semantic image synthesis, which compared itself with pix2pixHD; and diffusion-based translation. The official PyTorch repository’s README (as of November 2026) points to the authors’ later one-step methods, pix2pix-turbo and CycleGAN-Turbo, built on a pretrained text-to-image diffusion model.

Implementations

• Torch (Lua): phillipi/pix2pix, the original code, BSD licence; last commit 89ff2a8 of August 2020 as of November 2026. Its options switch the discriminator’s conditioning, the GAN term and the $L^1$ term on and off (condition_GAN, use_GAN, use_L1), which is how the paper’s ablations can be run.
• PyTorch: junyanz/pytorch-CycleGAN-and-pix2pix, BSD licence, with CycleGAN in the same code base; last commit 2a7afba of August 2025 as of November 2026, whose README states support for PyTorch 2.4 and later. The pix2pix model defaults to the unet_256 generator, the 70 × 70 “basic” PatchGAN, batch normalization, the vanilla GAN loss and $\lambda = 100$; the README says it produces results comparable to or better than the Torch code, which it recommends for reproducing the paper.
• pix2pixHD: NVIDIA/pix2pixHD, BSD licence; last commit 14b3b3c of September 2024 as of November 2026.

Example: What Each Discriminator Adds

The image-to-image translation article’s example shows that a translator trained on a per-pixel loss lands on the per-pixel mean or median and draws no texture. This example asks what the PatchGAN’s receptive field changes, measured in the Fourier domain, where texture of a known period shows as power at a known frequency.

Each $32 \times 32$ input is a one-hot label map with three classes: background, a disk, and a rectangle drawn over it. The target paints the background at 0.2, the rectangle at 0.85, and the disk with vertical stripes $0.5 + 0.3 \sin(2\pi x/4 + \varphi)$ of period 4 pixels, whose phase $\varphi$ is drawn for each image and is not in the input. No translator can know the phase, so a per-pixel loss is minimized by a flat disk at the median, 0.5. Four translators share one small U-Net (four stride-2 levels down to $2 \times 2$, concatenated skips, dropout in the two innermost decoder blocks at training and test time): $L^1$ alone, and $L^1$ plus a conditional GAN term with $\lambda = 100$ and three discriminators built as the official code builds them, a $1 \times 1$ PixelGAN, a $16 \times 16$ PatchGAN (one stride-2 layer, half the image width) and a whole-image discriminator (two stride-2 layers, receptive field 34 pixels). Training follows the official code’s objective and optimizer with a batch of 32 for 200 steps; the batch size, the image size, the channel counts and the short schedule are ours, to fit the toy scale. On 500 held-out inputs, the script compares the radially averaged power spectrum $|\mathrm{FFT}|^2$ of outputs and targets in four bands of radius $r$ (cycles per 32 pixels): low ($r$ = 1–3, the shapes), mid (4–6), stripe (7–9, around the stripes’ $r = 8$) and high (10–16). It prints the ratio of output to target power in each band and the mean $L^1$ error to the target. The band power is blind to phase, so stripes of any phase count; the stripe band also contains part of the power of the shapes’ edges, so even a flat disk scores above zero.

import sys
import numpy as np
import torch
import torch.nn as nn

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
STEPS = int(sys.argv[2]) if len(sys.argv) > 2 else 200
torch.manual_seed(seed)
torch.set_num_threads(4)
rng = np.random.default_rng(seed)
H = 32
yy, xx = np.mgrid[:H, :H]


def labels(n):
    """Label maps: background (0), a disk (1), and a rectangle (2) drawn on top."""
    lab = np.zeros((n, H, H), np.int64)
    for k in range(n):
        cy, cx = rng.uniform(9, 23, 2)
        r = rng.uniform(6, 10)
        lab[k][(yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2] = 1
        h, w = rng.integers(6, 13, 2)
        y0, x0 = rng.integers(0, H - h + 1), rng.integers(0, H - w + 1)
        lab[k, y0:y0 + h, x0:x0 + w] = 2
    return lab


def paint(lab):
    """Targets in [0, 1]: class 0 flat 0.2, class 2 flat 0.85, class 1 vertical
    stripes of period 4 px whose phase is random and absent from the input."""
    phase = rng.uniform(0, 2 * np.pi, (len(lab), 1, 1))
    stripes = 0.5 + 0.3 * np.sin(2 * np.pi * xx / 4 + phase)
    return np.where(lab == 1, stripes, np.where(lab == 2, 0.85, 0.2))


def to_torch(lab, img):
    x = torch.tensor(np.eye(3, dtype=np.float32)[lab]).permute(0, 3, 1, 2)  # one-hot
    return x, torch.tensor(img[:, None] * 2 - 1, dtype=torch.float32)       # [-1, 1]


class Down(nn.Module):
    def __init__(s, i, o, norm=True):
        super().__init__()
        s.f = nn.Sequential(nn.LeakyReLU(0.2), nn.Conv2d(i, o, 4, 2, 1), nn.BatchNorm2d(o) if norm else nn.Identity())


class UNet(nn.Module):
    """U-Net as in pix2pix's code, scaled to 32x32: 4x4 stride-2 convolutions down to
    2x2, transposed convolutions up, skips by concatenation, dropout in the inner decoder."""
    def __init__(s, c=32):
        super().__init__()
        s.e1 = nn.Conv2d(3, c, 4, 2, 1)                                            # 16
        s.e2 = nn.Sequential(nn.LeakyReLU(0.2), nn.Conv2d(c, 2 * c, 4, 2, 1), nn.BatchNorm2d(2 * c))      # 8
        s.e3 = nn.Sequential(nn.LeakyReLU(0.2), nn.Conv2d(2 * c, 4 * c, 4, 2, 1), nn.BatchNorm2d(4 * c))  # 4
        s.e4 = nn.Sequential(nn.LeakyReLU(0.2), nn.Conv2d(4 * c, 4 * c, 4, 2, 1))                         # 2
        up = lambda i, o, drop: nn.Sequential(nn.ReLU(), nn.ConvTranspose2d(i, o, 4, 2, 1), nn.BatchNorm2d(o),
                                              nn.Dropout(0.5) if drop else nn.Identity())
        s.d4 = up(4 * c, 4 * c, True)
        s.d3 = up(8 * c, 2 * c, True)
        s.d2 = up(4 * c, c, False)
        s.d1 = nn.Sequential(nn.ReLU(), nn.ConvTranspose2d(2 * c, 1, 4, 2, 1), nn.Tanh())

    def forward(s, x):
        e1 = s.e1(x); e2 = s.e2(e1); e3 = s.e3(e2); e4 = s.e4(e3)
        d = s.d4(e4)
        d = s.d3(torch.cat([d, e3], 1))
        d = s.d2(torch.cat([d, e2], 1))
        return s.d1(torch.cat([d, e1], 1))


def patch_d(n_layers, c=32):
    """Conditional discriminator on concat(x, y), built as pix2pix's code builds it.
    n_layers = 0: 1x1 PixelGAN; 1: 16x16 receptive field; 2: 34x34 (whole 32x32 image)."""
    if n_layers == 0:
        return nn.Sequential(nn.Conv2d(4, c, 1), nn.LeakyReLU(0.2), nn.Conv2d(c, 2 * c, 1),
                             nn.BatchNorm2d(2 * c), nn.LeakyReLU(0.2), nn.Conv2d(2 * c, 1, 1))
    layers, m = [nn.Conv2d(4, c, 4, 2, 1), nn.LeakyReLU(0.2)], 1
    for n in range(1, n_layers):
        layers += [nn.Conv2d(c * m, c * 2 * m, 4, 2, 1), nn.BatchNorm2d(c * 2 * m), nn.LeakyReLU(0.2)]
        m *= 2
    layers += [nn.Conv2d(c * m, c * 2 * m, 4, 1, 1), nn.BatchNorm2d(c * 2 * m), nn.LeakyReLU(0.2),
               nn.Conv2d(c * 2 * m, 1, 4, 1, 1)]
    return nn.Sequential(*layers)


def init(m):
    if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
        nn.init.normal_(m.weight, 0, 0.02); nn.init.zeros_(m.bias)
    elif isinstance(m, nn.BatchNorm2d):
        nn.init.normal_(m.weight, 1, 0.02); nn.init.zeros_(m.bias)


def train(n_layers, x, y, lam=100.0):
    G = UNet(); G.apply(init)
    optG = torch.optim.Adam(G.parameters(), 2e-4, betas=(0.5, 0.999))
    if n_layers is not None:
        D = patch_d(n_layers); D.apply(init)
        optD = torch.optim.Adam(D.parameters(), 2e-4, betas=(0.5, 0.999))
    bce = nn.BCEWithLogitsLoss()
    for _ in range(STEPS):
        i = torch.randint(0, len(x), (32,))
        xb, yb = x[i], y[i]
        fake = G(xb)
        if n_layers is not None:                       # one D step: real pairs vs fake pairs
            pr, pf = D(torch.cat([xb, yb], 1)), D(torch.cat([xb, fake.detach()], 1))
            lossD = 0.5 * (bce(pr, torch.ones_like(pr)) + bce(pf, torch.zeros_like(pf)))
            optD.zero_grad(); lossD.backward(); optD.step()
        lossG = lam * (fake - yb).abs().mean()         # L1 term
        if n_layers is not None:                       # non-saturating cGAN term
            pf = D(torch.cat([xb, fake], 1))
            lossG = lossG + bce(pf, torch.ones_like(pf))
        optG.zero_grad(); lossG.backward(); optG.step()
    return G


R = np.rint(np.hypot(*np.meshgrid(np.fft.fftfreq(H) * H, np.fft.fftfreq(H) * H, indexing="ij"))).astype(int)
BANDS = {"low": (1, 3), "mid": (4, 6), "stripe": (7, 9), "high": (10, 16)}


def band_power(imgs):
    """Radially averaged power spectrum, averaged over images, in bands of radius r."""
    p = (np.abs(np.fft.fft2(imgs)) ** 2).mean(0)
    return {b: p[(R >= lo) & (R <= hi)].mean() for b, (lo, hi) in BANDS.items()}


lab_tr = labels(4000); img_tr = paint(lab_tr)
lab_te = labels(500); img_te = paint(lab_te)
x, y = to_torch(lab_tr, img_tr)
xt, _ = to_torch(lab_te, img_te)
pt = band_power(img_te)

print(f"{'translator':22s}" + "".join(f"{b:>8s}" for b in BANDS) + f"{'L1 err':>8s}")
for name, n_layers in [("L1 alone", None), ("L1 + 1x1 PixelGAN", 0),
                       ("L1 + 16x16 PatchGAN", 1), ("L1 + whole-image D", 2)]:
    G = train(n_layers, x, y)
    G.train()                                          # dropout and batch statistics at test time, as in pix2pix
    with torch.no_grad():
        out = (G(xt)[:, 0].numpy() + 1) / 2
    po = band_power(out)
    print(f"{name:22s}" + "".join(f"{po[b] / pt[b]:8.3f}" for b in BANDS) + f"{np.abs(out - img_te).mean():8.3f}")

Output (seed 0; PyTorch 2.14.1, NumPy 2.5.4, 4 CPU threads; a run took between about 70 seconds and 3.5 minutes on our machine, depending on the load from other jobs; a second run with seed 0 printed the identical table):

translator                 low     mid  stripe    high  L1 err
L1 alone                 0.979   0.740   0.122   0.168   0.047
L1 + 1x1 PixelGAN        1.011   0.752   0.107   0.199   0.046
L1 + 16x16 PatchGAN      1.011   0.852   0.475   0.255   0.052
L1 + whole-image D       1.057   0.882   0.738   0.211   0.059

Every translator matches the low band: the $L^1$ term pins down the shapes and their levels, the part of the target the input determines, as Isola et al. argue in motivating the PatchGAN. The differences are in the stripe band. $L^1$ alone leaves about a tenth of the target’s power there, consistent with flat disks whose band power comes from the shapes’ edges. The $1 \times 1$ PixelGAN leaves the stripe band at the level of $L^1$ alone: each of its decisions sees one pixel’s label and value, and a disk pixel of value 0.5 is a plausible pixel of a striped disk, so the stripes’ spatial pattern is invisible to it. This matches the paper’s report that the PixelGAN has no effect on spatial statistics; in our reading, the colour effect it found has no counterpart here because the target’s values within the disk are spread symmetrically about the median the $L^1$ term already gives. A discriminator whose receptive field spans several stripe periods sees that a flat disk is not a striped one, and the generator answers by drawing stripes: the stripe-band power rises to about half of the target’s with the $16 \times 16$ PatchGAN and to three quarters with the whole-image discriminator. The stripes it draws cannot have the target’s phase, so its $L^1$ error rises; the band ratios, blind to phase, register the texture the $L^1$ error penalizes.

The whole-image discriminator did better than the $16 \times 16$ PatchGAN here, while in the paper the $286 \times 286$ ImageGAN scored lower than $70 \times 70$ on the FCN-score. The settings differ: this image has a single texture at one scale, the whole-image discriminator is only one layer deeper than the patch one, and the measure is texture power, not recognizability. In our reading, 200 steps also leave every adversarial translator short of the target’s stripe power, so the gap between the two is a statement about this budget. The script takes the seed as its argument; over seeds 0, 1 and 2 the stripe-band ratio ranged from 0.098 to 0.122 for $L^1$ alone, 0.098 to 0.107 with the PixelGAN, 0.475 to 0.621 with the $16 \times 16$ PatchGAN and 0.738 to 0.847 with the whole-image discriminator, with the whole-image discriminator ahead on every seed; the low-band ratios stayed between 0.975 and 1.057, and the $L^1$ errors ranged from 0.045 to 0.047, 0.045 to 0.046, 0.050 to 0.052 and 0.057 to 0.060.