Image-to-Image Translation

Image-to-image translation is the task of converting an image of a scene from one representation into another: a label map into a photograph, a map into an aerial view, an edge sketch into a product photo, day into night. Isola et al. (2017) defined it by analogy with language translation and showed that one conditional GAN recipe, pix2pix, handles problems that each had special-purpose methods. Unlike image generation, the output must correspond to a given input; unlike segmentation, it is itself an image that must look real.

Problem Definition

Given images from a source domain $X$ and a target domain $Y$, learn a mapping $G: X \to Y$ such that $G(x)$ looks like a member of $Y$ and depicts the same scene as $x$. In the problems Isola et al. consider, input and output render the same structure, so they are roughly aligned pixel by pixel. Most problems are one-to-many: one label map fits many photographs. The target is therefore a conditional distribution $p(y \mid x)$ to sample from, not a single best guess.

Input and Output

The input is an image (photo, label map, edge map, grayscale or thermal image), optionally with noise or a style reference; the output is an image, usually of the same size. Because the networks are convolutional, a pix2pix generator trained on $256 \times 256$ maps ran on $512 \times 512$ inputs (Isola et al., §4.4, arXiv v3).

Variants

• Paired translation trains on aligned pairs $\{(x_i, y_i)\}$, such as Cityscapes photos and label maps.
• Unpaired translation trains on two unrelated collections; for horse↔zebra a paired output is not even well defined (Zhu et al., 2017).
• Multimodal translation returns several plausible outputs per input, controlled by a random or chosen code.
• Semantic image synthesis (label map to photo) has its own benchmarks.

Challenges

Multimodality and blur. Trained only on per-pixel error, a network predicts a summary of all plausible outputs: the per-pixel mean for $L^2$ and, as Isola et al. note, the median for $L^1$ (§4.2). Where edges or colours are uncertain, both are blurry or greyish.

Mode collapse and ignored noise. pix2pix’s generator learned to ignore a noise input, so the authors kept only dropout as noise and saw only minor stochasticity (§3.1); with an unconditional discriminator it produced nearly the same output for every input (§4.2). In CycleGAN, keeping the cycle loss in one direction only often caused instability and mode collapse (§5.1.3, ICCV).

Geometry changes. CycleGAN’s dog→cat translation degenerated into minimal changes to the input (§6, ICCV).

Unpaired correspondence. Many mappings give the same output distribution, so matching distributions does not fix which output belongs to which input (Zhu et al., §3.2); CycleGAN sometimes swapped tree and building labels (§6, ICCV).

Approaches

Paired translation with conditional GANs

pix2pix (Isola et al., 2017) trains a generator $G(x, z)$ and a discriminator $D(x, y)$ that sees the input with the real or generated output, and adds an $L^1$ term:

$$G^* = \arg\min_G \max_D \; \mathbb{E}_{x,y}[\log D(x, y)] + \mathbb{E}_{x,z}[\log(1 - D(x, G(x, z)))] + \lambda\, \mathbb{E}_{x,y,z}\big[\lVert y - G(x, z) \rVert_1\big],$$

with $\lambda = 100$ (§4.2) and the non-saturating generator loss (§3.3). $L^1$ fixes the low frequencies, and a PatchGAN discriminator models only high-frequency structure by classifying each $N \times N$ patch as real or fake and averaging the responses, a form of texture or style loss (§3.2.2). The generator is a U-Net, whose skips carry edges and layout across the encoder–decoder. On Cityscapes labels→photo, a $70 \times 70$ PatchGAN scored best; a $1 \times 1$ “PixelGAN” changed only colour, and a $286 \times 286$ “ImageGAN” scored lower (Table 2, CVPR).

pix2pixHD (Wang et al., 2018) reached $2048 \times 1024$ with a coarse-to-fine generator, three discriminators at three image scales, a feature-matching loss and instance boundary maps (§3, arXiv v2).

Unpaired translation

CycleGAN (Zhu et al., 2017) trains two generators, $G: X \to Y$ and $F: Y \to X$, each with an adversarial loss in its target domain, and ties them with a cycle-consistency loss:

$$\mathcal{L}_{\text{cyc}}(G, F) = \mathbb{E}_{x}\big[\lVert F(G(x)) - x \rVert_1\big] + \mathbb{E}_{y}\big[\lVert G(F(y)) - y \rVert_1\big],$$

weighted by $\lambda = 10$, with $70 \times 70$ PatchGANs and a least-squares adversarial loss (§4, arXiv v7; the ICCV version omits the value of $\lambda$).

Cycle consistency asks for a bijection, but an aerial photo holds more information than its map. Chu, Zhmoginov and Sandler (2017) found that a CycleGAN trained on maps hid details of the aerial photo, such as dots on a roof, in a low-amplitude, high-frequency signal in the generated map. Noise of standard deviation $3/256$ on the map changed the reconstruction almost as much as any noise level tested, JPEG compression destroyed it, and a small crafted perturbation of almost any map made the reverse generator produce a chosen photo (§3–§4).

Other methods change how the domains are linked. UNIT (Liu, Breuel & Kautz, 2017) assumes corresponding images share one latent code, with a VAE–GAN per domain and shared weights. MUNIT (Huang et al., 2018) splits an image into a domain-invariant content code and a domain-specific style code, and samples the style code to give multimodal outputs. CUT (Park et al., 2020) drops the reverse generator and uses a patchwise contrastive loss: an output patch should match the input patch at the same location rather than other patches of that input. It trained 40% faster with 31% less memory than CycleGAN, and improved horse→zebra FID from 77.2 to 45.5 (Table 1, arXiv v3).

Semantic image synthesis

SPADE (Park et al., 2019) argues that normalization layers “wash away” a label map fed only at the input. Every normalization layer instead normalizes per channel, as in batch normalization, and modulates with a scale and bias predicted from the mask $m$ at each location:

$$\gamma_{c,y,x}(m)\, \frac{h_{n,c,y,x} - \mu_c}{\sigma_c} + \beta_{c,y,x}(m).$$

On COCO-Stuff its Table 1 gives 37.4 mIoU and FID 22.6, against 14.6 and 111.5 for pix2pixHD; the text quotes 35.2 mIoU, the value in its ablation table (CVPR version and arXiv v2 alike).

Diffusion-based translation

Diffusion models sample the conditional distribution without adversarial training. SDEdit (Meng et al., 2022) needs no translation training: it noises a guide $x^{(g)}$, such as a stroke painting, to an intermediate time $t_0$ and runs the reverse process of an unconditional model of the target domain, starting, in DDPM notation (our restatement of their variance-preserving form), from

$$x_{t_0} = \sqrt{\bar\alpha_{t_0}}\, x^{(g)} + \sqrt{1 - \bar\alpha_{t_0}}\, \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I).$$

Larger $t_0$ gives more realistic but less faithful images; for LSUN stroke paintings, with time in $[0, 1]$, the authors chose $t_0 \in [0.3, 0.6]$ (Fig. 3, arXiv v2). Palette (Saharia et al., 2022) trains a conditional diffusion model that receives the source image by channel concatenation, as in latent diffusion, with one recipe for colorization, inpainting, uncropping and JPEG restoration; an $L^1$ denoising loss gave similar FID but lower diversity than $L^2$ (Table 6, arXiv v2). ControlNet (Zhang, Rao & Agrawala, 2023) adds conditions such as edges, depth, segmentation or pose to a frozen text-to-image model through a trainable copy of its encoder joined by zero-initialized convolutions.

Example: Why an L1 Translator Averages

The inputs are $16 \times 16$ disk masks; each output fills the disk with diagonal stripes of random phase on a dark (0.3, probability 0.7) or bright (0.9) base, so every mask has many correct outputs. Small CNNs trained on 4,000 pairs with $L^2$, $L^1$, and $L^1$ plus a noise channel are compared with the exact per-pixel minimizers and with a sampler drawing from the true conditional distribution. Inside the disk, on 500 held-out masks, the script reports the $L^1$ error to the ground truth, the mean level, the fraction of bright outputs (level $> 0.6$), the within-image standard deviation (texture), and the mean difference between two outputs for one input (spread).

import sys
import numpy as np
import torch
import torch.nn as nn

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.manual_seed(seed)
torch.set_num_threads(4)
rng = np.random.default_rng(seed)
H = 16
yy, xx = np.mgrid[:H, :H]

def masks(n):
    """Input images: binary disks of random position and radius."""
    cy, cx = rng.uniform(4.5, 10.5, (2, n, 1, 1))
    r = rng.uniform(3.0, 5.0, (n, 1, 1))
    return ((yy - cy)**2 + (xx - cx)**2 <= r**2).astype(np.float32)

def paint(mask):
    """Output images: the disk filled with diagonal stripes of random phase on a
    dark (p = 0.7) or bright (p = 0.3) base, so one mask has many correct outputs."""
    n = len(mask)
    base = np.where(rng.random((n, 1, 1)) < 0.7, 0.3, 0.9)
    phase = rng.uniform(0, 2 * np.pi, (n, 1, 1))
    return (mask * (base + 0.15 * np.sin(2 * np.pi * (yy + xx) / 4 + phase))).astype(np.float32)

def train(loss, use_z, x, y, steps=800):
    f = nn.Sequential(nn.Conv2d(2 if use_z else 1, 16, 3, padding=1), nn.ReLU(),
                      nn.Conv2d(16, 16, 3, padding=1), nn.ReLU(),
                      nn.Conv2d(16, 1, 3, padding=1))
    opt = torch.optim.Adam(f.parameters(), lr=5e-3)
    sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda t: 1 - t / steps)
    for _ in range(steps):
        i = torch.randint(0, len(x), (64,))
        inp = torch.cat([x[i], torch.randn_like(x[i])], 1) if use_z else x[i]  # noise channel
        err = f(inp) - y[i]
        l = err.abs().mean() if loss == "L1" else (err**2).mean()
        opt.zero_grad(); l.backward(); opt.step(); sched.step()
    def translate(m):
        with torch.no_grad():
            inp = torch.tensor(m)[:, None]
            if use_z:
                inp = torch.cat([inp, torch.randn_like(inp)], 1)
            return f(inp)[:, 0].numpy()
    return translate

mtr = masks(4000); otr = paint(mtr)
mte = masks(500);  ote = paint(mte)            # held-out inputs and their ground truth
x, y = torch.tensor(mtr)[:, None], torch.tensor(otr)[:, None]

def report(name, a, b):
    """a, b: two outputs for each test mask. All statistics are taken inside the disk."""
    inside = [m.astype(bool) for m in mte]
    l1 = np.mean([np.abs(a[k][s] - ote[k][s]).mean() for k, s in enumerate(inside)])
    level = np.array([a[k][s].mean() for k, s in enumerate(inside)])
    tex = np.mean([a[k][s].std() for k, s in enumerate(inside)])
    spread = np.mean([np.abs(a[k][s] - b[k][s]).mean() for k, s in enumerate(inside)])
    print(f"{name:18s}{l1:8.3f}{level.mean():8.2f}{(level > 0.6).mean():8.0%}{tex:9.3f}{spread:8.3f}")

print(f"{'translator':18s}{'L1 err':>8s}{'level':>8s}{'bright':>8s}{'texture':>9s}{'spread':>8s}")
for name, loss, use_z in [("L2 regression", "L2", False), ("L1 regression", "L1", False),
                          ("L1 + noise input", "L1", True)]:
    g = train(loss, use_z, x, y)
    report(name, g(mte), g(mte))
# Every pixel inside a disk has the same marginal distribution, so the exact
# per-pixel minimizers of L2 and L1 are constants: its mean and its median.
v = paint(np.ones((20000, 1, 1), np.float32))[:, 0, 0]
report("exact mean (L2)", mte * v.mean(), mte * v.mean())
report("exact median (L1)", mte * np.median(v), mte * np.median(v))
report("sampler", paint(mte), paint(mte))     # draws from the true conditional distribution

Output (seed 0; PyTorch 2.14.1, NumPy 2.5.4, about 30 seconds on 4 CPU threads; two runs gave identical output):

translator          L1 err   level  bright  texture  spread
L2 regression        0.245    0.48      0%    0.009   0.000
L1 regression        0.218    0.39      0%    0.010   0.000
L1 + noise input     0.218    0.39      0%    0.009   0.004
exact mean (L2)      0.244    0.48      0%    0.000   0.000
exact median (L1)    0.218    0.40      0%    0.000   0.000
sampler              0.298    0.48     30%    0.106   0.318

The trained networks land on the exact minimizers: flat disks at the mean, 0.48, for $L^2$ and at the per-pixel median, about 0.39, for $L^1$, levels no real disk has (real ones average 0.3 or 0.9). Neither draws stripes (texture near 0 against 0.106), and the noise channel is almost ignored, since under a per-pixel loss it says nothing about the target (pix2pix saw it ignored with an adversarial term too). Yet both have a lower $L^1$ error than the sampler, the only translator whose outputs look real: per-pixel error cannot rank translators by realism. The script takes the seed as its argument; over seeds 0, 1 and 2, the $L^1$ regressor’s error ranged from 0.218 to 0.246 against 0.298 to 0.339 for the sampler, and the networks’ texture stayed at or below 0.014 and their spread at or below 0.004.

Datasets and Benchmarks

Dataset	Pairing	Size as described	Source of the description
Cityscapes labels↔photo	Paired	2,975 training images; validation set for testing	Isola et al., §6.2, arXiv v3
CMP Facades labels→photo	Paired	400 training images	Isola et al., §6.2, arXiv v3
Maps↔aerial photo	Paired	1,096 training images scraped from Google Maps around New York City	Isola et al., §6.2, arXiv v3
Edges→shoes (edges2shoes)	Paired	50,000 UT Zappos50K images; edges from the HED detector with postprocessing	Isola et al., §4 and §6.2, arXiv v3
Horse↔zebra (horse2zebra)	Unpaired	939 horse and 1,177 zebra training images from ImageNet	Zhu et al., §7.1, arXiv v7

CUT gives horse→zebra 2,403 training images (§4.1, arXiv v3), more than the 2,116 in CycleGAN’s appendix. Semantic image synthesis also uses COCO-Stuff and ADE20K (Park et al., 2019), and Palette proposed an ImageNet-based protocol.

Evaluation Metrics

• FCN-score. pix2pix trained an FCN-8s on Cityscapes and scored generated photos by how well its segmentation matched the label maps they came from: per-pixel accuracy 0.42 for $L^1$ alone, 0.66 for $L^1$ + cGAN and 0.80 for real photos (Table 1, identical in the CVPR version and arXiv v3). Later work used other segmenters: PSPNet in pix2pixHD; DeepLabV2, UperNet101 and DRN-D-105 in SPADE.
• Fréchet Inception Distance compares translated and real target images as distributions and ignores whether each output matches its input, so SPADE and CUT pair it with segmentation scores.
• LPIPS (Zhang et al., 2018) compares unit-normalized deep features with channel weights fitted to human similarity judgments. Between outputs for one input it measures diversity: on edges→shoes, MUNIT’s Table 1 (arXiv v2) gives 0.011 for UNIT, 0.010 for CycleGAN, 0.109 for MUNIT and 0.293 for real image pairs.
• Human studies. In pix2pix’s Amazon Mechanical Turk protocol, a real and a generated image appeared for one second each and workers chose the fake, over 40 trials after 10 practice ones. Aerial photos from $L^1$ + cGAN passed as real in 18.9% of trials, against 0.8% for $L^1$ alone (Table 3, CVPR). Zhu et al. warn that their AMT numbers are not comparable with pix2pix’s (§5.1.1, arXiv v7). Palette reports a colorization fool rate of 47.8%, where 50% means indistinguishable (Table 1, arXiv v2).
• Per-pixel metrics such as PSNR and SSIM tend to prefer blurry regression outputs, an observation Palette cites from super-resolution work; the example shows the same for $L^1$ error.

Applications

Besides the benchmarks above, pix2pix handled day→night, thermal→colour and colorization. CycleGAN was applied to collection style transfer (the style of a whole set of Monet or Van Gogh paintings, not of one image), season transfer, object transfiguration and shallow depth of field for smartphone photos (§5.2, ICCV), and UNIT to domain adaptation. Super-resolution, colorization and inpainting can be posed as translation from a degraded image, as in Palette and latent diffusion’s concatenation-conditioned models.

Open Problems

• Diverse, faithful sampling. MUNIT’s outputs remained far less diverse than real images.
• Meaningful correspondence. Cycle consistency can be met by hidden signals; Chu et al. (2017) suggest modifying the loss for serious applications.
• Evaluation. FCN-scores depend on the segmenter, FID ignores the input, and human studies are costly and hard to compare across papers.