GAN Inversion

GAN inversion finds, for a given image, a latent code from which a pretrained generative adversarial network generator reproduces it. A GAN generator maps codes to images but, unlike a variational autoencoder, has no encoder that maps images back, so the reverse map has to be computed: by optimizing a code for each image, by training an encoder, or by combining the two. Once a photograph has a code, the operations that work on the generator’s own samples, moving along semantic directions, interpolating, mixing styles between layers, apply to it. The same step is called projection (Karras et al., 2020), embedding (Abdal, Qin & Wonka, 2019) or inversion (Zhu et al., 2020), and in the editing literature the routine is described as “invert first, edit later” (Richardson et al., 2021). The latent space article covers the spaces themselves and the directions found in them; the StyleGAN article covers the generator on which most of the methods below were developed.

Purpose

For a generator $G$ and an image $x$, inversion solves

$$z^\ast = \arg\min_{z} \; \ell\big(G(z), x\big) + \lambda\, R(z),$$

where $\ell$ measures how far the generated image is from the target and $R$ optionally keeps the code where the generator was trained to work. Two different things are asked of $z^\ast$. It should reconstruct $x$ (low distortion), and it should behave like the codes the generator was trained on, so that an edit $z^\ast + \alpha n$ along a direction $n$ changes the intended attribute and nothing else (editability). In our reading, the problem is hard for three reasons. The loss is non-convex in $z$, so gradient descent from a poor start lands in local minima. A real photograph usually lies outside the generator’s range, so no code reproduces it exactly and the method must decide what to give up. And the codes that come closest to an out-of-range image tend to be atypical ones, which is where editing directions learned from typical samples fail; the image editing article states this tension in general terms, and the rest of this article is about how inversion methods handle it.

Zhu, Shen, Zhao and Zhou (2020, §1.1) define the task the same way: given a fixed GAN model, find the most accurate latent code to recover the input image. They argue that reconstruction alone is the wrong criterion, since a code that recovers the pixels but does not lie in the semantic domain of the latent space cannot reuse what the GAN learned for editing.

When to Use It

Inversion is the entry point whenever a pretrained GAN’s latent space is to be used on images that it did not generate:

• Editing real photographs with directions found on generated samples, such as InterFaceGAN’s attribute normals or GANSpace’s principal components, in StyleGAN models of faces, cars, horses, churches or towers (Zhu et al., 2020, §3.4; the e4e and PTI READMEs).
• Image restoration and composition in which the generator supplies the missing content: inpainting, crossover of two images, local edits from scribbles and local style transfer (Abdal, Qin & Wonka, 2020, §7).
• Image-to-image translation through a fixed generator, when an encoder is trained to map sketches, segmentation maps or low-resolution images to codes (Richardson et al., 2021); the image-to-image translation article covers the task.
• Attribution and anomaly scoring. StyleGAN2’s authors use projection to test whether a generator could have produced an image (Karras et al., 2020, §5.1), and AnoGAN, described in the visual anomaly detection article, scores an image by how well the best code reproduces it.

It works best on images that resemble the training data in content and alignment. Abdal, Qin and Wonka (2019, §3.2) found StyleGAN embeddings of faces sensitive to translation, scaling and rotation, and the PTI repository asks that faces be aligned and cropped before inversion (README, “Preparing your Data”). For diverse images without a domain-specific GAN, diffusion models are inverted instead, with DDIM or null-text inversion; the image editing article covers those methods and has its own example.

How It Works

Reconstruction losses

Pixel losses alone smooth the reconstruction. Abdal, Qin and Wonka (2019, Eq. 1–2) minimize the sum of a pixel mean squared error, normalized by the number of pixel values, and a perceptual loss on the features of VGG-16 layers conv1_1, conv1_2, conv3_2 and conv4_2, with all weights set to 1. In their ablation (§5.2, Figure 9), the pixel loss alone embedded the general colours but smoothed details, and its embeddings of non-face images tended toward the average face; a single VGG layer was worse than several. StyleGAN2’s projector uses the LPIPS distance alone, after downsampling both images to $256 \times 256$ (Karras et al., 2020, Appendix D). Encoders add task-specific terms: pSp combines a pixel L2 loss, LPIPS and an identity loss on the cosine similarity of ArcFace face-recognition features (Richardson et al., 2021, §3.1). For faces the identity term matters; for cats and dogs the same authors trained without it (§4.5).

Inversion spaces

Where the code lives decides how much the generator can be bent toward an image. For StyleGAN, in order of increasing freedom:

• $\mathcal{Z}$, the 512-dimensional input of the mapping network, and $\mathcal{W}$, its 512-dimensional output, are the spaces in which the generator was trained to sample. Abdal, Qin and Wonka (2019, §3.3) found it was not easily possible to embed photographs into either, which is why they introduced the extended space below.
• $\mathcal{W}+$ gives each of the 18 style inputs its own $w$, an $18 \times 512$ code (Abdal, Qin & Wonka, 2019, §3.3). It reconstructs much better, and it can also embed images that are not faces at all (cats, dogs, paintings, cars into a face model, §3.1). StyleGAN2’s authors chose to stay in $\mathcal{W}$ for that reason: extending the space finds closer matches but “also enables projecting arbitrary images that should have no latent representation”, whereas codes in $\mathcal{W}$ correspond to images the generator could have produced (Karras et al., 2020, §5).
• The noise maps $n_i$ of StyleGAN’s per-layer noise inputs can also be optimized. Image2StyleGAN++ adds this noise space to $\mathcal{W}+$: 18 noise maps, two per resolution from $4 \times 4$ to $1024 \times 1024$ (Abdal, Qin & Wonka, 2020, §6).
• The generator’s weights themselves, which generator tuning changes (below).

The latent space article explains why $\mathcal{W}$ is better behaved than $\mathcal{Z}$ and how $\mathcal{W}+$ is used for morphing and style crossover; in our reading, each step down the list lowers distortion and admits codes the generator never produced in training.

Optimization-based inversion

The direct approach minimizes the loss over the code by gradient descent through the frozen generator. Image2StyleGAN (Abdal, Qin & Wonka, 2019, §5) optimizes a $\mathcal{W}+$ code with Adam at learning rate 0.01 for 5,000 steps, under 7 minutes per image on a V100 GPU. The loss on faces converged after about 1,000 steps, cats, dogs and cars after about 3,000, and paintings after about 5,000 (Figure 10). Initialization mattered: starting every layer from the mean code $\bar w$ gave faces a lower final loss (0.309 against 0.351, in units of $10^{-5}$) and a code closer to $\bar w$ (distance 30.67 against 35.60) than a random start from $U[-1, 1]$, while for cats, dogs, paintings and cars the random start reached a slightly lower loss (Table 2). Re-embedding an embedding seven times in a row slowly lost detail, which the authors suggest may be due to slow convergence near local optima (§5.3, Figure 11).

StyleGAN2’s projector (Karras et al., 2020, §5 and Appendix D) inverts into $\mathcal{W}$ together with the noise maps. It first runs 10,000 random codes through the mapping network to get the mean $\mu_w$ and the scale $\sigma_w^2 = \mathbb{E}_z \lVert f(z) - \mu_w \rVert_2^2$, starts from $w = \mu_w$ and unit Gaussian noise maps, and runs 1,000 Adam iterations with a maximum learning rate of 0.1, ramped up over the first 50 iterations and down with a cosine over the last 250. During the first three quarters the loss is evaluated at $\tilde w = w + \mathcal{N}(0, 0.05\, \sigma_w t^2)$, with $t$ falling from one to zero, which the authors say adds stochasticity and stabilizes finding the global optimum. Because the noise maps are free variables, the optimizer can hide image content in them; a regularizer penalizes their autocorrelation at several scales, on a pyramid of $2 \times 2$-averaged copies down to $8 \times 8$, and without it the noise maps visibly carried signal (Appendix D, Figure 8).

Image2StyleGAN++ (Abdal, Qin & Wonka, 2020, §4.2) optimizes $\mathcal{W}+$ and the noise space in sequence. Joint optimization failed in each form they tried: with the perceptual and pixel losses, details were lost and replaced by high-frequency artefacts; with the pixel loss only, reconstruction was almost perfect but so much of the image sat in the noise maps that resampling them destroyed it (Figure 2). Their recipe optimizes $w$ first (5,000 iterations, Adam at 0.01), then the noise maps with a pixel loss (3,000 iterations, Adam at learning rate 5), once each: a further round of $w$ optimization smoothed the image and dropped PSNR from 39.5 dB to 20 dB, and alternating again did not recover it (§4.2, §7.1). The reconstructions rose from 19–22 dB PSNR in $\mathcal{W}+$ to 39–45 dB with noise optimization (Figure 4), 44–45 dB with the iteration counts above (§7.1). The authors judged a representation that stores too much of the image in its noise maps unsuitable for editing (§4.2).

Encoder-based inversion

An encoder $E$ amortizes the optimization into one forward pass. The conventional encoder is trained on generated pairs, minimizing $\lVert z - E(G(z)) \rVert^2$ over sampled codes (Zhu et al., 2020, Eq. 1, describing Zhu et al., 2016, and others). Zhu et al. (2020, §2.1) argue that this supervision ignores the generator’s knowledge and never sees real images. Their domain-guided encoder instead feeds $E(x)$ to the fixed generator and is trained on real images with a pixel loss, a VGG conv4_3 feature loss and an adversarial loss from a discriminator trained alongside it with a gradient penalty on real images (Eq. 2–3; $\lambda_{\text{vgg}} = 5 \times 10^{-5}$, $\lambda_{\text{adv}} = 0.1$, $\gamma = 10$, §3.1). They invert into $\mathcal{W}$, but with a separate code per layer, unlike StyleGAN (§3.1, footnote 2).

pSp (“pixel2style2pixel”, Richardson et al., 2021) encodes directly into $\mathcal{W}+$. A feature pyramid over a ResNet-IR backbone pretrained on face recognition gives three feature maps; a small fully convolutional network (“map2style”) per style vector reduces its map to one 512-vector, styles 0–2 from the coarsest map, 3–6 from the middle and 7–18 from the finest, as the caption numbers them (Figure 2; supplementary §1). The output is an offset from the average code, $\mathrm{pSp}(x) = G(E(x) + \bar w)$, which they found a better initialization (§3). A single 512-vector expanded to all layers by one extra layer (“naive $\mathcal{W}+$”) already improved on a $\mathcal{W}$ encoder but lost fine details (Figure 5). The paper’s loss includes a term $\lVert E(x) - \bar w \rVert_2$ (Eq. 3) that pulls codes toward the average, which the authors liken to the truncation trick; the supplement sets its weight to 0.005 in every application except inversion, which does not use it (supplementary §1), although §4.5 lists this term among the losses of the cat and dog models, inversion included.

e4e (“encoder for editing”, Tov et al., 2021) was designed for the opposite priority. Its paper was not read here; its repository’s README, which reproduces the abstract, describes a study of StyleGAN’s latent space that identifies a distortion–editability trade-off and a distortion–perception trade-off, and two principles for designing encoders that control how close inversions stay to the regions StyleGAN was trained on, giving better editing on real images at a small cost in reconstruction accuracy. The README’s training command adds a discriminator on the codes (--w_discriminator_lambda 0.1, the “Dw loss multiplier” in options/train_options.py) and a progressive schedule (--progressive_start 20000), and it proposes a metric, latent editing consistency (LEC), with an example using an aging direction on FFHQ.

Hybrid inversion

Combining the two uses the encoder for a fast, in-domain start and the optimizer for accuracy. Zhu et al. (2020, §2.2) start from the domain-guided encoder’s code and keep the encoder in the objective as a regularizer:

$$z^{\text{inv}} = \arg\min_z \; \lVert x - G(z) \rVert_2 + \lambda_{\text{vgg}} \lVert F(x) - F(G(z)) \rVert_2 + \lambda_{\text{dom}} \lVert z - E(G(z)) \rVert_2,$$

with $F$ the VGG features and $\lambda_{\text{dom}} = 2$ (Eq. 4, §3.1). The last term asks that the code be a fixed point of encoder and generator, a code the encoder itself would produce from the reconstruction. In their ablation, $\lambda_{\text{dom}} = 0$ gave the best reconstruction and relatively poor manipulation, and $\lambda_{\text{dom}} = 40$ worse reconstruction and more satisfying manipulation (§3.5, Figure 9): the trade-off made into a dial. Richardson et al. (2021, §2) group inversion methods the same way, into optimization, encoders and hybrids that use an encoder to initialize optimization, and note that optimization typically reconstructs better but takes substantially longer.

Generator tuning

A further step changes the generator rather than the code. In pivotal tuning inversion (PTI; Roich et al., 2022, paper not read here), the repository’s README describes inverting the image to a code in $\mathcal{W}$, the “pivot”, and then briefly fine-tuning the generator so that the pivot reproduces the image, while a regularization term keeps nearby codes’ images unchanged; the edits that work in $\mathcal{W}$ then apply to a faithful reconstruction. In the official code at commit da94d59, the pivot comes from a $\mathcal{W}$ projector adapted from NVIDIA’s StyleGAN2-ADA code, run for 450 steps (first_inv_steps). The configuration also names a learning rate of 0.005 for it, but the projector’s schedule overwrites the optimizer’s rate at every step with its own peak of 0.01 (our reading of w_projector.py); the generator is then tuned with Adam at $3 \times 10^{-4}$ on an L2 plus LPIPS (AlexNet) loss for at most 350 steps, stopping early once LPIPS falls to 0.06 (configs/hyperparameters.py, single_id_coach.py). The locality regularizer samples codes from the mapping network at truncation 0.5, moves the pivot a fixed distance of 30 toward each and penalizes changes of the tuned generator’s images there against the original generator’s (localitly_regulizer.py). The README says the configuration holds the paper’s defaults, yet in that file the regularizer is switched off (use_locality_regularization = False), so the default run tunes on the reconstruction loss alone. Zhu et al. (2020, §1.1) also cite concurrent work by Pan et al. that optimizes the generator’s parameters together with the code, for image restoration and manipulation.

Editing after inversion

Once a code is found, editing is generation from a modified code. The linear edit of InterFaceGAN,

$$x^{\text{edit}} = G(z^{\text{inv}} + \alpha n),$$

moves along the normal $n$ of an attribute’s separating hyperplane (Zhu et al., 2020, Eq. 5, following Shen et al., 2020); interpolation between two inverted codes morphs between images. Zhu et al. compared Image2StyleGAN’s $\mathcal{W}+$ codes with their in-domain codes on both. Interpolating Image2StyleGAN codes distorted eyeglasses and gave an unnatural change of gender on faces, and artefacts and blur on towers; editing pose blurred hair, editing expression or eyeglasses changed identity, and adding clouds blurred the tower with the sky, whereas the in-domain codes changed mostly the intended content (§3.4, Figures 5–7).

Image2StyleGAN++ makes the edits local by masking. Its building blocks optimize $w$ under spatial masks on the losses and binary masks $w_m$, $n_m$ on which variables may change (§6). Inpainting optimizes only layers 1–9, 17 and 18 of $\mathcal{W}+$ from the mean face, so that early layers fill the missing content and the last layers keep colours consistent, and different random offsets of the start give different completions (§7.3, Figure 7); scribbles become local edits by re-embedding the scribbled image into the first 4 to 6 layers with a penalty that keeps $w$ near the original code (§7.4); and crossover, local style transfer and copying activation tensors at the fourth layer blend parts of two images (§7.2, §7.5, §7.6). The image inpainting article covers inpainting methods in general.

pSp turns the order around. Its encoder can be trained to map an input from another domain, a sketch, a segmentation map, a low-resolution image, directly to the codes of the corresponding face, so “invert first, edit later” becomes a single translation step that also handles inputs StyleGAN cannot represent (§1, §4). Multi-modal outputs come from style mixing: layers 1–7 from the encoder, layers 8–18 from a randomly drawn $w$ (§4.3).

How to Apply It

A practical inversion fixes four things.

1. The space. In our reading of the results below and of the example, choose $\mathcal{W}$, or a method that keeps codes near the regions the generator was trained on (e4e, a hybrid with a domain regularizer), when the code will be edited; choose $\mathcal{W}+$ and noise maps when only reconstruction or restoration matters; tune the generator when both are needed and per-image cost is acceptable.
2. The start. Start from the mean code for in-domain images (Abdal, Qin & Wonka, 2019, Table 2; Karras et al., 2020, Appendix D) or from an encoder’s output; a random start can help for images far from the domain.
3. The loss. Combine a pixel term with a feature-space term (VGG features or LPIPS), add an identity term for faces, and regularize any noise maps that are optimized.
4. The evaluation. Measure reconstruction and editing separately: a lower reconstruction error is not evidence that edits will work (Zhu et al., 2020, §1.1).

Example: inverting a small generator whose codes are known

Published inversions are judged on photographs whose true codes do not exist, so latent error cannot be measured there. This example uses a generator whose codes are known. The generator is not a GAN: it is a small coordinate network (an MLP over Fourier features of each pixel’s position) trained by regression to a renderer of soft disks on $16 \times 16$ images, with four factors in $[-1, 1]$: centre $x$, centre $y$, radius and brightness. It stands in for a GAN’s generator because inversion needs only a fixed, differentiable generator; nothing below depends on how it was trained. Like StyleGAN’s synthesis network, it has four layers whose features are scaled and shifted by learned affine maps of a code, so it has an input space in which one 4-dimensional code drives every layer (written $\mathcal{W}$ below, the space it was trained in) and an extended space with one code per layer ($\mathcal{W}+$, 16 numbers).

The test set has 100 in-domain images $G(z_{\text{true}})$ and 100 out-of-domain images, the same kind with a $3 \times 3$ square of value 0.8 pasted at a random position (redrawn until it changes at least one pixel), a detail the generator was never trained to draw and the stand-in for what a photograph contains beyond the generator’s range. Five methods invert them: an encoder into $\mathcal{W}$ and one into $\mathcal{W}+$ (MLPs trained through the frozen generator with a pixel loss on generated images from codes drawn over the full factor range, half of them with a square), Adam optimization of a $\mathcal{W}$ or $\mathcal{W}+$ code from the mean code 0 (300 steps), and a hybrid that starts the $\mathcal{W}$ optimization from the encoder’s output. Three measures are reported as medians: reconstruction RMS against the test image (square included); latent RMS against $z_{\text{true}}$, for $\mathcal{W}+$ over all four layers’ codes; and edit error, our measure of editability. For two edits along known factors, radius $+0.5$ and centre $x$ $+0.5$, added to the single code or to every layer’s code, it compares the change the edit makes from the inverted code with the change it makes from the true code:

$$\text{edit error} = \frac{\lVert [G(c + \delta) - G(c)] - [G(z_{\text{true}} + \delta) - G(z_{\text{true}})] \rVert}{\lVert G(z_{\text{true}} + \delta) - G(z_{\text{true}}) \rVert}.$$

It is 0 when the edit does to the inverted image exactly what it does to the true one, and about 1 or more when it is as wrong as the change itself; both differences are between generated images, so the square, which neither edit should touch, enters the measure only through the inverted code. The script (PyTorch 2.14.1, CPU, four threads) ran in 3.3 to 5.5 minutes on our machine, most of it training the generator and the two encoders.

import sys
import torch
import torch.nn as nn

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.manual_seed(seed)
torch.set_num_threads(4)

H, L, D = 16, 4, 4               # image size, modulated layers, code size per layer
ys, xs = torch.meshgrid(torch.arange(H, dtype=torch.float32),
                        torch.arange(H, dtype=torch.float32), indexing="ij")

def render(z):
    """Soft disk from 4 factors in [-1, 1]: centre x, centre y, radius, brightness."""
    cx, cy = 7.5 + 4.5 * z[:, 0], 7.5 + 4.5 * z[:, 1]
    rad, bri = 3 + 1.5 * z[:, 2], 0.7 + 0.3 * z[:, 3]
    d = torch.sqrt((xs - cx[:, None, None]) ** 2 + (ys - cy[:, None, None]) ** 2)
    return bri[:, None, None] * torch.sigmoid((rad[:, None, None] - d) / 0.5)

freqs = torch.tensor([1.0, 2.0, 4.0, 8.0]) * torch.pi / H
coords = torch.stack([xs, ys], -1).reshape(-1, 2)
fourier = torch.cat([torch.sin(coords[:, :, None] * freqs),
                     torch.cos(coords[:, :, None] * freqs)], -1).reshape(H * H, -1)

class Generator(nn.Module):
    """Coordinate network; each layer is modulated (scale and shift) by its own code."""
    def __init__(self, width=64):
        super().__init__()
        self.inp = nn.Linear(fourier.shape[1], width)
        self.lin = nn.ModuleList([nn.Linear(width, width) for _ in range(L)])
        self.A = nn.ModuleList([nn.Linear(D, width) for _ in range(L)])
        self.B = nn.ModuleList([nn.Linear(D, width) for _ in range(L)])
        self.out = nn.Linear(width, 1)

    def forward(self, w):
        """w: (N, D), one code for all layers (W), or (N, L, D), one per layer (W+)."""
        if w.dim() == 2:
            w = w[:, None, :].expand(-1, L, -1)
        h = self.inp(fourier)[None]
        for l in range(L):
            s, b = self.A[l](w[:, l])[:, None], self.B[l](w[:, l])[:, None]
            h = nn.functional.silu(self.lin[l](h) * (1 + s) + b)
        return torch.sigmoid(self.out(h)).reshape(-1, H, H)

prior = lambda n: torch.rand(n, D) * 2 - 1

# 1. Train the generator by regression to the renderer (a stand-in for a GAN generator).
G = Generator()
opt = torch.optim.Adam(G.parameters(), lr=2e-3)
for step in range(3000):
    z = prior(64)
    loss = ((G(z) - render(z)) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()
G.requires_grad_(False)
zc = prior(500)
print(f"PyTorch {torch.__version__}, seed {seed}")
print(f"generator vs renderer, RMS over 500 new codes: {((G(zc) - render(zc)) ** 2).mean().sqrt():.4f}")

def add_square(x, mask):
    """Paste a 3x3 square of value 0.8 where mask is True; redraw until a pixel changes."""
    x = x.clone()
    pending = torch.nonzero(mask).flatten()
    while len(pending):
        r, c = torch.randint(0, H - 2, (2, len(pending), 1, 1))
        sq = (ys >= r) & (ys < r + 3) & (xs >= c) & (xs < c + 3)
        new = torch.where(sq, torch.clamp(x[pending], min=0.8), x[pending])
        changed = (new != x[pending]).flatten(1).any(1)
        x[pending[changed]] = new[changed]
        pending = pending[~changed]
    return x

# 2. Test images: 100 in-domain G(z_true), 100 with a square the generator never draws.
N = 100
z_true = prior(2 * N)
z_true[:, [0, 2]] = z_true[:, [0, 2]] * 0.75 - 0.25    # cx, radius in [-1, 0.5]: edits stay in range
x_gen = G(z_true)
x_test = add_square(x_gen, torch.arange(2 * N) >= N)
changed = int(((x_test[N:] - x_gen[N:]).abs().amax((1, 2)) > 0).sum())
print(f"in-domain images equal G(z_true): {bool((x_test[:N] == x_gen[:N]).all())}; "
      f"images changed by the square: {changed}/{N}")

# 3. Encoders, trained through the frozen generator with a pixel loss on images like the test set.
def train_encoder(out):
    E = nn.Sequential(nn.Flatten(), nn.Linear(H * H, 128), nn.SiLU(),
                      nn.Linear(128, 128), nn.SiLU(), nn.Linear(128, out))
    o = torch.optim.Adam(E.parameters(), lr=1e-3)
    for step in range(2000):
        x = add_square(G(prior(128)), torch.rand(128) < 0.5)
        c = E(x).reshape(-1, L, D) if out == L * D else E(x)   # offset from the mean code 0
        loss = ((G(c) - x) ** 2).mean()
        o.zero_grad(); loss.backward(); o.step()
    return E

E_w, E_wp = train_encoder(D), train_encoder(L * D)

# 4. Optimization: Adam on the code, starting from the mean code 0 or from the encoder.
def optimize(x, init, steps=300, lr=0.05):
    c = init.clone().requires_grad_(True)
    o = torch.optim.Adam([c], lr=lr)
    for _ in range(steps):
        loss = ((G(c) - x) ** 2).sum()        # summed: each image's code gets its own gradient
        o.zero_grad(); loss.backward(); o.step()
    return c.detach()

with torch.no_grad():
    codes = {"enc-W": E_w(x_test), "enc-W+": E_wp(x_test).reshape(-1, L, D)}
codes["opt-W"] = optimize(x_test, torch.zeros(2 * N, D))
codes["opt-W+"] = optimize(x_test, torch.zeros(2 * N, L, D))
codes["hybrid-W"] = optimize(x_test, codes["enc-W"])

def measures(c):
    """Reconstruction RMS, latent RMS against z_true (every layer for W+), and the
    relative error of two edits (radius +0.5, centre x +0.5) against the true edit."""
    with torch.no_grad():
        rec = ((G(c) - x_test) ** 2).mean((1, 2)).sqrt()
        cw = c if c.dim() == 3 else c[:, None, :].expand(-1, L, -1)
        lat = ((cw - z_true[:, None, :]) ** 2).mean((1, 2)).sqrt()
        edits = []
        for k in (2, 0):
            e = torch.zeros(D); e[k] = 0.5
            d_inv, d_true = G(c + e) - G(c), G(z_true + e) - G(z_true)
            edits.append((d_inv - d_true).norm(dim=(1, 2)) / d_true.norm(dim=(1, 2)))
    return rec, lat, edits

print("\nmedians over 100 images     reconstruction  latent  latent<0.05  edit: radius  centre x")
for part, sl in (("in-domain", slice(0, N)), ("with square", slice(N, 2 * N))):
    for name, c in codes.items():
        rec, lat, (er, ex) = measures(c)
        print(f"{part:12s} {name:9s} {rec[sl].median():14.4f} {lat[sl].median():7.3f}"
              f" {(lat[sl] < 0.05).float().mean():11.0%} {er[sl].median():13.3f} {ex[sl].median():9.3f}")
_, _, (er, ex) = measures(z_true)
print(f"\nedit error of the true codes: {max(er.max(), ex.max()):.1e}")
_, _, (er, ex) = measures(prior(2 * N))
print(f"edit error of unrelated random codes: radius {er.median():.3f}, centre x {ex.median():.3f}")

Output (two runs with seed 0 gave identical output):

PyTorch 2.14.1, seed 0
generator vs renderer, RMS over 500 new codes: 0.0039
in-domain images equal G(z_true): True; images changed by the square: 100/100

medians over 100 images     reconstruction  latent  latent<0.05  edit: radius  centre x
in-domain    enc-W             0.0078   0.043         55%         0.070     0.061
in-domain    enc-W+            0.0122   0.691          0%         0.168     0.138
in-domain    opt-W             0.0000   0.000        100%         0.000     0.000
in-domain    opt-W+            0.0016   0.274          2%         0.053     0.036
in-domain    hybrid-W          0.0000   0.000        100%         0.000     0.000
with square  enc-W             0.1430   0.257          8%         0.570     0.487
with square  enc-W+            0.1307   0.944          0%         0.596     0.576
with square  opt-W             0.1341   0.127         41%         0.346     0.298
with square  opt-W+            0.0766   1.675          0%         0.590     0.612
with square  hybrid-W          0.1373   0.141         41%         0.383     0.311

edit error of the true codes: 0.0e+00
edit error of unrelated random codes: radius 1.140, centre x 1.345

In-domain images. Optimization in $\mathcal{W}$ recovers every true code and reconstructs exactly; the hybrid does too. Starting at the mean code, the optimizer came within 0.05 of the true code for all 100 images, so local minima did not trap it here. The $\mathcal{W}+$ optimizer also reconstructs almost exactly, but its codes sit a median 0.27 from the true ones. In our reading, the per-layer codes have directions that barely change an image the generator can already make, so many $\mathcal{W}+$ codes fit it and the optimizer lands on one that is not the generator’s own. Its edits are still close (edit errors of 0.05 and 0.04); in our reading, on images the generator can make, nothing pushes the codes far. The encoders reconstruct less exactly than optimization in the same space, in line with pSp’s and Zhu et al.’s tables, where optimization reconstructs better than the encoders, and the $\mathcal{W}+$ encoder’s codes are far from the true ones (0.69) although its images are close.

Out-of-domain images. The square separates the spaces. $\mathcal{W}+$ optimization paints part of it, lowering reconstruction error from 0.134 to 0.077, but only by moving its codes a median 1.68 from the true ones, most of the factor range’s width of 2; its edits are then wrong by 0.59 and 0.61 of their size, against 0.35 and 0.30 for $\mathcal{W}$. That is the distortion–editability trade-off in miniature: the space that fits the unfamiliar detail best is the one in which the edit defined on the generator’s own codes no longer does what it should. With this seed, even in $\mathcal{W}$ the square pulls the code away from the truth (median 0.13, only 41% within 0.05), which in our reading is the optimizer trading accuracy on the disk against the square’s pixels; the edit error grows with it. The encoders reconstruct worse than optimization in the same space. The $\mathcal{W}$ encoder is also worse than $\mathcal{W}$ optimization on latent and edit error, while with this seed the $\mathcal{W}+$ encoder’s codes stay nearer the truth than $\mathcal{W}+$ optimization’s (0.94 against 1.68), with similar edit errors. Unrelated random codes give edit errors of 1.14 and 1.35, so the measure separates the methods: with this seed no inversion of an image with a square is close to the true edit, but $\mathcal{W}$ optimization and the hybrid come closest.

With seeds 1 and 2 (the script’s argument), which retrain the generator and encoders and draw new test images, the in-domain rows came out the same way, and on images with a square $\mathcal{W}+$ optimization again had the lowest reconstruction error (0.127 and 0.101 against 0.150 for $\mathcal{W}$ optimization) and the larger latent and edit errors of the two optimizers (latent 0.56 and 1.52 against 0.013 and 0.021; edit errors 0.15 to 0.57 against 0.015 to 0.05). The size of the effect varies a lot between seeds: with seed 0, $\mathcal{W}$ optimization was pulled much further by the square than with seeds 1 and 2, and with seed 2 the hybrid ended further from the truth than $\mathcal{W}$ optimization from the mean (latent 0.127 against 0.021), so the start still mattered for some images. The seed-0 advantage of the $\mathcal{W}+$ encoder’s codes over $\mathcal{W}+$ optimization’s did not hold with seed 1 (0.60 against 0.56).

The toy omits what makes real inversion hard at scale: a perceptual loss, local minima that trap the optimizer even on images the generator can make, and generators whose $\mathcal{W}$ codes reconstruct photographs only roughly. What it does show is why reconstruction error alone ranks methods wrongly: in the out-of-domain rows of every seed, the method with the lowest reconstruction error has a larger latent error and a larger edit error than optimization in the generator’s own space.

Variants

• Optimization into $\mathcal{W}+$: Image2StyleGAN, mean-code initialization, VGG perceptual plus pixel loss.
• Optimization into $\mathcal{W}$ with noise: StyleGAN2’s projector, with noise added to the code and regularized noise maps.
• $\mathcal{W}+$ plus noise space, with masks: Image2StyleGAN++ for high-PSNR reconstruction, inpainting and local edits.
• Encoders: conventional encoders trained on generated pairs; the domain-guided encoder; pSp into $\mathcal{W}+$; e4e, which keeps inversions near the regions StyleGAN was trained on.
• Hybrids: encoder initialization followed by optimization, with the encoder as a regularizer in in-domain inversion.
• Generator tuning: PTI, and earlier joint optimization of code and generator parameters.
• Inversion of other generators: diffusion models are inverted by running a deterministic sampler backward or by optimizing guidance embeddings, which the image editing article covers; AnoGAN inverts a GAN to score anomalies.

Best Practices

• Initialize at the mean code for images from the training domain; for faces it gave Image2StyleGAN both a lower loss and a code closer to the mean (Table 2).
• Use feature-space losses with a pixel term: pixel loss alone smooths details and drifts toward the average face (Abdal, Qin & Wonka, 2019, §5.2).
• Optimize noise maps after the code, not with it, and regularize them (Abdal, Qin & Wonka, 2020, §4.2; Karras et al., 2020, Appendix D).
• Align inputs the way the training images were aligned (Abdal, Qin & Wonka, 2019, §3.2).
• Keep codes in the domain when they will be edited, with an encoder regularizer, a discriminator on the codes or a $\mathcal{W}$-space pivot.
• Report reconstruction and editing together, with the same images and settings across methods.

Pitfalls

• Overfitting with expressive spaces. $\mathcal{W}+$ and noise maps can reproduce images the generator cannot meaningfully represent, a car in a face model, and the code then edits badly (Abdal, Qin & Wonka, 2019, §3.1, §4.1; Zhu et al., 2020, §4, Figure 10).
• Content hidden in the noise. Jointly optimized noise maps carry the image, so resampling them destroys it and the code is unsuitable for editing (Abdal, Qin & Wonka, 2020, §4.2, Figure 2).
• Local minima. Non-convex losses make the start matter; StyleGAN2’s projector adds decaying noise to the code to explore (Appendix D), and repeated embedding drifts (Abdal, Qin & Wonka, 2019, Figure 11).
• Encoders that do not generalize. Encoders are fast but usually have problems generalizing beyond the training data, according to Abdal, Qin and Wonka (2019, §2); pSp’s global encoding loses details such as earrings and backgrounds, and faces far from frontal or with expressions absent from StyleGAN’s training set are hard to reconstruct (Richardson et al., 2021, §5, Figure 11).
• Judging by pixels alone. A low pixel error says nothing about whether $z^{\text{inv}} + \alpha n$ changes the intended attribute (Zhu et al., 2020, §1.1).
• Misuse. Faithful inversion of photographs makes realistic edits of real people easier. StyleGAN2’s authors note that generated images project almost exactly into their generator’s $\mathcal{W}$, which helps attribute them, while real images do not (§5.1, Figures 9–10).

Evidence and Impact

Comparisons between inversion methods use different generators, datasets and measures, so numbers are comparable only within one table.

Reconstruction, encoders against optimization. Richardson et al. (2021, Table 1) inverted faces into a StyleGAN2 generator trained on FFHQ (the table does not name its test images; the inversion figures of §4.1 and the supplement show CelebA-HQ test faces), measuring identity similarity with a CurricularFace network (different from the ArcFace network in their loss), LPIPS, MSE and runtime (units not printed in the table):

Method	Similarity ↑	LPIPS ↓	MSE ↓	Runtime
Optimization (Karras et al., 2020)	0.77	0.11	0.02	182.1
ALAE encoder	0.06	0.32	0.15	0.207
IDInvert encoder, no optimization	0.18	0.22	0.06	0.032
$\mathcal{W}$ encoder	0.35	0.23	0.06	0.064
Naive $\mathcal{W}+$ encoder	0.49	0.19	0.04	0.064
pSp without identity loss	0.19	0.17	0.03	0.105
pSp	0.56	0.17	0.03	0.105

Optimization reconstructs best on all three measures and is three orders of magnitude slower; among encoders, $\mathcal{W}+$ outputs beat a single $\mathcal{W}$ code, and the identity loss raises identity similarity without changing LPIPS or MSE.

Reconstruction and editing, in-domain inversion. Zhu et al. (2020) inverted 500 images per model into StyleGAN generators for FFHQ faces and LSUN towers, scoring reconstructions by FID, sliced Wasserstein discrepancy (SWD) and MSE (Table 1):

Method	Speed	Face FID	Face SWD	Face MSE	Tower FID	Tower SWD	Tower MSE
Conventional encoder	0.008 s	88.48	100.5	0.507	73.02	69.19	0.455
MSE-based optimization (Image2StyleGAN)	290 s	58.04	29.19	0.026	69.16	55.35	0.068
Domain-guided encoder	0.017 s	52.85	13.02	0.062	46.81	27.13	0.071
In-domain inversion	8 s	42.64	13.44	0.030	44.77	26.44	0.052

The text says in-domain inversion surpasses the other methods on all metrics (§3.3), but in the table the optimization baseline has the lower face MSE and the domain-guided encoder the lower face SWD. In-domain inversion was about 35 times faster than the optimization baseline. On interpolation and on attribute manipulation, its edited images had lower FID and SWD than Image2StyleGAN’s on both faces and towers (Table 2), and on 7,000 real faces, classifying age, gender, eyeglasses and pose from the inverted codes with InterFaceGAN boundaries gave better precision–recall curves for in-domain codes than for Image2StyleGAN codes (§3.2, Figure 3).

Reconstruction with noise optimization. Image2StyleGAN++’s $\mathcal{W}+$ embeddings reached 19–22 dB PSNR, and 39–45 dB after noise optimization, on the images of Figure 4 (§7.1).

Attribution. StyleGAN2’s projector found $\mathcal{W}$ codes for StyleGAN2’s own images with LPIPS distances so small that they could be “almost unambiguously attributed” to the network; with the original StyleGAN, projection did not find matching codes reliably even for its own images, which the authors write appears to be because the mapping from $\mathcal{W}$ to images is too complex to invert in practice (Karras et al., 2020, §5.1, Figure 10, LSUN Car and FFHQ).

In our reading, the line from Image2StyleGAN to PTI moved the question from whether a photograph can be embedded at all to which of its possible embeddings can be edited.