Latent Space

The latent space of a generative model is the space of codes from which it produces images. A GAN generator maps a vector $z$ drawn from a simple distribution, usually a standard Gaussian, to an image; a variational autoencoder decodes a code $z$ and also has an encoder that maps an image back to codes; a latent diffusion model generates a compressed spatial code that an autoencoder’s decoder turns into pixels. Because a code has far fewer numbers than the image, and the decoder is smooth, nearby codes give similar images and paths between codes give gradual changes. That is what makes latent spaces useful for interpolation, for attribute editing by moving along directions, and for asking whether a model has separated the factors that produced its training images. The codes are also called latent codes, latent vectors or latents; the image-side counterpart, a vector computed from an image for comparison, is an embedding.

Definition

For a generator $G : \mathcal{Z} \to \mathcal{X}$, the latent space is the domain $\mathcal{Z}$, together with the distribution $p(z)$ from which codes are drawn at sampling time. An image is a sample $x = G(z)$, $z \sim p(z)$; for a VAE the decoder outputs a distribution over images and the encoder $q(z \mid x)$ maps images into $\mathcal{Z}$. Most models also have intermediate latent spaces, representations inside the generator that can be edited in the same way: StyleGAN’s $\mathcal{W}$, the per-layer extension $\mathcal{W}+$, or a feature tensor at some layer. “The latent space” of a model therefore names whichever of these spaces is being sampled, interpolated or edited, and articles usually say which.

Intuition

Think of a model of small shape images whose training data vary in shape, size and position. A good latent space behaves like a set of knobs: points close together decode to similar images, a straight path from a small disk to a large one passes through medium disks, and somewhere there is a direction along which only the size changes. Nothing in the training objective of a GAN or a VAE asks for these properties directly. The generator is only asked to make images that look like the data when codes are drawn from $p(z)$; smoothness and meaningful directions are by-products, and how well they emerge depends on the architecture and the data.

Two complications recur. First, the prior is not a uniform blob. In high dimensions almost all of a Gaussian’s probability sits in a thin shell, so a path that leaves the shell, such as the straight line between two samples, passes through codes the generator was never trained on. Second, the data’s factors are usually correlated: if most large shapes in the training set are squares, the model has little reason to separate “large” from “square”, and a direction found for “square” may also enlarge the shape, as InterFaceGAN’s age direction added eyeglasses (below). Karras, Laine and Aila (2019, Figure 6) illustrate the second point for a training set in which one combination of two features is missing: the mapping from a fixed Gaussian $\mathcal{Z}$ to the image features must become curved so that the missing combination is never sampled, which entangles the factors in $\mathcal{Z}$.

Formal Definition

Let $z \in \mathbb{R}^d$ with prior $p(z) = \mathcal{N}(0, I_d)$. Its squared norm $\lVert z \rVert^2$ is a sum of $d$ independent squared standard normals, so it follows a chi-squared distribution with $d$ degrees of freedom, with mean $d$ and variance $2d$. The norm itself has mean close to $\sqrt{d}$ and a standard deviation that tends to $1/\sqrt{2}$, so the ratio $\lVert z \rVert / \sqrt{d}$ concentrates at 1 as $d$ grows: samples lie near a sphere of radius $\sqrt{d}$. Two independent samples are nearly orthogonal, since $z_1^\top z_2 / (\lVert z_1 \rVert \lVert z_2 \rVert)$ has mean 0 and a spread that shrinks like $1/\sqrt{d}$. (These are standard properties of the Gaussian; the embeddings article covers the related concentration of pairwise distances.)

Linear interpolation (lerp) between two codes is

$$\mathrm{lerp}(z_1, z_2; t) = (1 - t)\, z_1 + t\, z_2, \qquad t \in [0, 1].$$

For independent $z_1, z_2 \sim \mathcal{N}(0, I_d)$, $\mathrm{lerp}(z_1, z_2; t) \sim \mathcal{N}\big(0, ((1-t)^2 + t^2) I_d\big)$, so its norm at the midpoint is about $\sqrt{d/2} \approx 0.71 \sqrt{d}$: well inside the shell where the prior’s samples lie, closer to the origin than they are (our derivation).

Spherical interpolation (slerp) rotates $z_1$ toward $z_2$ in the plane they span. With $\theta$ the angle between them,

$$\mathrm{slerp}(z_1, z_2; t) = \frac{\sin\big((1 - t)\theta\big)}{\sin \theta}\, z_1 + \frac{\sin(t\theta)}{\sin \theta}\, z_2, \qquad \cos\theta = \frac{z_1^\top z_2}{\lVert z_1 \rVert\, \lVert z_2 \rVert}.$$

When $\lVert z_1 \rVert = \lVert z_2 \rVert$, every point of the path has that same norm. When the norms differ, the midpoint is $(z_1 + z_2) / (2 \cos(\theta / 2))$; for orthogonal codes its norm is the root mean square of the two endpoint norms (our derivation). The formula divides by $\sin\theta$ and is undefined for parallel or opposite codes. Karras, Laine and Aila (2019, §4.1) call slerp the most appropriate way to interpolate in StyleGAN’s normalized input latent space, citing Shoemake’s 1985 paper on animating rotations with quaternion curves and White’s “Sampling Generative Networks” (2016, not read here).

Directions. A unit vector $n \in \mathbb{R}^d$ defines an edit $z \mapsto z + \alpha n$. When a binary attribute is separated by the hyperplane $n^\top z = 0$, Shen et al. (2020, Eq. 1–2) model the attribute’s score as proportional to the signed distance $n^\top z$, so moving along $n$ changes the score linearly.

Properties

Interpolation

Interpolating between two codes and decoding the path is a common check that a model has learned a smooth space rather than memorized images; the variational autoencoder article does it for two shape codes and compares the result with a pixel cross-fade. Lerp and slerp differ in which codes the path visits. For a Gaussian prior, lerp dips toward the origin, where the density is higher but where the generator almost never received inputs during training (the example below measures this). Slerp stays on the shell. For spaces without a fixed prior the argument does not apply: StyleGAN’s $\mathcal{W}$ vectors “are not normalized in any fashion”, so its authors interpolate there with lerp (Karras et al., 2019, §4.1), and Abdal, Qin and Wonka (2019, §4.1) morph faces by lerp in $\mathcal{W}+$.

Interpolation also gives a measure. Karras, Laine and Aila (2019, §4.1) noted, after Laine, that features absent from both endpoints can appear in the middle of a path, a sign of entanglement. Their perceptual path length (PPL) averages, over many random paths and positions $t$ on them, the LPIPS distance between images at two points a small step $\epsilon$ apart, divided by $\epsilon^2$, with slerp in $\mathcal{Z}$ and lerp in $\mathcal{W}$; the StyleGAN article gives the formula and the headline scores. Two details matter for interpreting it. The authors measured both full paths ($t \sim U(0, 1)$) and path endpoints ($t \in \{0, 1\}$), arguing that $\mathcal{W}$ may contain regions that the generator renders poorly between points mapped from $\mathcal{Z}$, which biases the full-path measure in favour of $\mathcal{Z}$; as expected, the endpoint measure was lower for $\mathcal{W}$ (160.6 against 200.5 full-path for the style-based generator with noise) while $\mathcal{Z}$’s barely changed (415.3 against 412.0) (Table 3, FFHQ). And adding a mapping network to a traditional generator made $\mathcal{Z}$ much worse (full-path PPL 896.2, separability 170.29) while its intermediate space was better (324.5 and 6.52) at the same FID of 4.87 (Table 4), which the authors read as evidence that the input latent space can be arbitrarily entangled.

Latent arithmetic

Differences between codes can transfer attributes: if $z_a$ and $z_b$ differ only in one attribute, $z_c + (z_b - z_a)$ should add that attribute to $z_c$. Shen et al. (2020, §1) credit Radford, Metz and Chintala’s DCGAN paper (2016, not read here) with first observing this vector-arithmetic property in a GAN’s latent space. Abdal, Qin and Wonka (2019, §4) built three tests of whether an embedding into StyleGAN is semantically meaningful from the same operations: morphing by linear interpolation, style transfer by crossover (the first 9 of 18 layers’ codes from one image, $4^2$ to $64^2$, and the last 9 from another), and expression transfer by $w = w_1 + \lambda (w_3 - w_2)$, where $w_3 - w_2$ is the difference between an expressive and a neutral face of another person, with the channels of the difference whose norm fell below 1 set to zero to suppress background changes (§4.3). Face morphing worked; morphs of non-face images failed, and morphs between classes showed face contours in the middle (§4.1).

Semantic directions with labels: InterFaceGAN

Shen, Gu, Tang and Zhou (2020) asked how a GAN trained on faces encodes binary attributes. Their framework assumes that each attribute has a separating hyperplane in $\mathcal{Z}$, with the score linear in the signed distance $n^\top z$. Generated images are scored for each attribute, and a linear SVM per attribute gives the normal $n$. On a Progressive GAN trained on CelebA-HQ, SVMs for pose, smile, age, gender and eyeglasses scored 95.6% to 100.0% on a validation set of 6,000 samples with confident labels and 75.3% to 90.3% on 480,000 random samples (Table 1).

With $m$ attributes, $s = \Lambda N^\top z$ for $N = [n_1, \dots, n_m]$, so for $z \sim \mathcal{N}(0, I)$ the scores are Gaussian with covariance $\Lambda N^\top N \Lambda$ (Eq. 3–5). The attributes are uncorrelated exactly when the normals are orthogonal, and $n_i^\top n_j$ measures their entanglement. Pose and smile were almost orthogonal to the rest, while age, gender and eyeglasses were correlated (cosines 0.38 to 0.52 between their normals, Table 2), mirroring the attribute correlations of the generated images (Table 3) and, the authors suggest, the training data, in which older men more often wear glasses. Conditional manipulation removes the projection onto a direction to be preserved, $n_1 - (n_1^\top n_2)\, n_2$, so that editing attribute 1 leaves attribute 2 nearly unchanged; with several conditions, the projection onto the subspace they span is removed (§2.2).

Two further observations concern the geometry above. Their Property 2 bounds the probability that a Gaussian sample lies far from a hyperplane through the origin: for $d = 512$, almost every sample is within 5 units of the boundary (§2.1). Moving a code much further, beyond the threshold of 5 they chose, changed the face so much that it no longer looked like the same person (§3.2, Figure 5), which the authors explain by such codes being very unlikely under the prior. They also trained an SVM on 4,000 hand-labelled bad samples and found a “quality” direction along which artifacts could be corrected (§3.2, Figure 6). On StyleGAN they found $\mathcal{W}$ better than $\mathcal{Z}$ for editing, especially over long distances, but $\mathcal{W}$ had learned an age direction that already included eyeglasses and was nearly orthogonal to the eyeglasses direction, so conditional manipulation could not separate them there (§3.4).

Semantic directions without labels: GANSpace

Härkönen, Hertzmann, Lehtinen and Paris (2020) find directions by principal component analysis. For StyleGAN they sample many $z$, map them to $w$, and compute PCA of the $w$ vectors; an edit is $w' = w + V x$ with $V$ the principal basis and $x$ the slider values (Eq. 3). Since $z$ is isotropic, its own coordinate axes are equivalent to random directions (§3.3); in our reading, PCA of $z$ itself would find no preferred directions either. For BigGAN, which has no learned $w$, they compute PCA of the features at the first linear layer, the first layer with a non-isotropic distribution, and obtain a latent direction for each component by least-squares regression of the sampled $z$ on that component’s coordinates (Eq. 4).

The main tool is the layer-wise edit: applying a direction only to the codes of a range of layers, written $E(v_i, j\text{–}k)$. Component $v_1$ of StyleGAN2 on FFHQ mixes head rotation and gender, but applied only to layers 0–2 it gives a fairly pure rotation (§2.3, Figure 3). Across the models they examined, large changes of layout and viewpoint came from the first 20 components, later ones changed appearance and details; for StyleGAN2’s $\mathcal{W}$, the first 100 components sufficed to describe overall appearance, the other 412 controlling subtle changes (§3.1). Fixing the first eight principal coordinates of a StyleGAN2 cat model and randomizing the rest kept pose and camera while changing appearance, whereas fixing eight random directions changed almost everything and randomizing them changed almost nothing (§3.3, Figure 4). Some directions carried the training data’s associations, such as a “sporty” car edit that also changed the background to an open road, and some combinations were disallowed: a wrinkles edit had no significant effect on a child’s face (§3.2). Compared with supervised directions, the PCA edits were qualitatively close but sometimes more entangled (§3.3, Figure 5).

StyleGAN’s W and W+ spaces

In StyleGAN, an 8-layer mapping network $f$ turns $z \in \mathcal{Z}$ into $w \in \mathcal{W}$, and $w$ sets the style of every synthesis layer; the StyleGAN article covers the architecture, style mixing and the truncation trick in $\mathcal{W}$. The argument for $\mathcal{W}$ is about density. $\mathcal{Z}$ must be sampled with the training data’s density, so missing or rare combinations of factors force a warped mapping; $\mathcal{W}$’s density is induced by the learned mapping, so it can “unwarp” the factors (Karras et al., 2019, §4). Their second measure, linear separability, labels 200,000 generated images with 40 CelebA-attribute classifiers, keeps the more confident half, fits a linear SVM per attribute in the latent space and sums the conditional entropies of the classifier labels given the side of the hyperplane, reported as $\exp\big(\sum_i H(Y_i \mid X_i)\big)$ (§4.2). $\mathcal{W}$ was consistently more separable than $\mathcal{Z}$, and deeper mapping networks improved both separability and FID (Tables 3 and 4).

$\mathcal{W}+$ gives each of the 18 layers that receive a style its own 512-dimensional $w$, an $18 \times 512$ code. Abdal, Qin and Wonka (2019, §3.3) introduced it for embedding photographs, after finding that it was not easily possible to embed into $\mathcal{W}$ or $\mathcal{Z}$ directly (embedding into $\mathcal{W}$ did not give reasonable results, Figure 5). In our reading, $\mathcal{W}+$ is more expressive than the generator’s own sampling space: the authors embedded cats, dogs, paintings and cars into a face model with reasonable quality (§3.1), and found embeddings sensitive to translation, scaling and rotation of the face (§3.2, Table 1). Finding the code of a given photograph is GAN inversion, with its own methods and its own trade-off between how exactly a code reproduces a photo and how well it can then be edited; the image editing article summarizes that trade-off.

Disentanglement and its measures

A representation is informally called disentangled when a change in one underlying factor of variation changes one coordinate, or one group of coordinates, of the code; Locatello et al. (2019, §1) note that no formal definition is widely accepted. Karras, Laine and Aila (2019, §4) phrase the goal for generators as a latent space made of linear subspaces, each controlling one factor. The representation learning article places disentanglement among the properties sought of learned features.

Most measures need the ground-truth factors $v_1, \dots, v_K$ of a synthetic dataset and an encoder:

• BetaVAE metric (Higgins et al., 2017, as described by Kim & Mnih, 2018, §4, and Chen et al., 2018, §2.2). Fix one factor, vary the others, average the absolute differences of pairs of codes, and train a linear classifier to predict which factor was fixed. Chen et al. describe the score as the classifier’s accuracy; Kim and Mnih describe it as the error rate, yet write of 100% accuracy in the same passage. Kim and Mnih pointed out that it can score 100% when only $K - 1$ of $K$ factors are disentangled, and that it could be sensitive to the classifier’s optimization settings.
• FactorVAE metric (Kim & Mnih, 2018, §4). Normalize each code dimension by its standard deviation over the data, fix one factor, and take the dimension with the lowest variance; a majority-vote classifier from that dimension’s index to the factor’s index has no optimization hyperparameters. Kim and Mnih define the score as the classifier’s error rate, but they treat a high score as good, as for an accuracy: on 2D Shapes their best mean score was around 0.82 for FactorVAE against 0.73 for β-VAE (§6, Figure 5).
• Mutual information gap (MIG; Chen et al., 2018, §4.1). With $I(z_j; v_k)$ the mutual information between code dimension $j$ and factor $k$, and $H(v_k)$ the factor’s entropy,

$$\mathrm{MIG} = \frac{1}{K} \sum_{k=1}^{K} \frac{1}{H(v_k)} \Big( I\big(z_{j(k)}; v_k\big) - \max_{j \ne j(k)} I(z_j; v_k) \Big), \qquad j(k) = \arg\max_j I(z_j; v_k),$$

which lies between 0 and 1 and rewards one dimension per factor: a factor spread over two dimensions, as after a rotation of the code, scores low.

• Modularity, DCI Disentanglement and SAP, as summarized by Locatello et al. (2019, §4): whether each dimension depends on at most one factor (via mutual information), the entropy of each dimension’s normalized importance for predicting the factors, and the gap in prediction error between the two most predictive dimensions per factor.
• Encoder-free measures for generators. Perceptual path length and linear separability, above, need neither an encoder nor known factors (Karras et al., 2019, §4); InterFaceGAN’s cosine between attribute normals measures the entanglement of particular attributes.

The β-VAE of Higgins et al. (2017), as Locatello et al. (2019, §4) describe it, puts a hyperparameter $\beta$ in front of the VAE’s KL term to constrain the capacity of the bottleneck; the variational autoencoder article covers the objective. Chen et al. (2018, §3) decomposed the averaged KL term into an index–code mutual information, a total correlation between latent dimensions and per-dimension KL terms, and weighted only the total correlation by $\beta$ (β-TCVAE, §3); FactorVAE penalizes the same quantity with an auxiliary discriminator.

Locatello et al. (2019) tested these methods and measures at scale: six methods, six measures, seven datasets, more than 12,000 models with 50 seeds per setting. Their Theorem 1 shows that for any factorized prior there are infinitely many bijections that leave the distribution of codes unchanged but entangle every coordinate with every factor, so a method that sees only images cannot tell a disentangled model from an entangled one with the same data distribution (§3). Experimentally, the penalties reduced the total correlation of the sampled codes, but the mean codes usually used as the representation generally became more correlated as the penalty grew, DIP-VAE-I excepted (§5.1); all measures except Modularity were correlated, BetaVAE with FactorVAE and MIG with DCI most strongly (§5.2); the objective alone explained 37% of the variance of the scores on average, objective and regularization strength 59%, the rest being due to the seed (§5.3); and they found no reliable way to select good models without labels (§5.4).

Latent spaces of autoencoders for latent diffusion

A latent diffusion model separates the two jobs. Rombach et al. (2022, §1, §3.1) train an autoencoder for perceptual compression, which removes imperceptible high-frequency detail, and leave the semantic composition of the image to the diffusion model trained on its latents. The latent is spatial, $z \in \mathbb{R}^{h \times w \times c}$ with a downsampling factor $f = H/h$, and it is regularized only lightly, by a slight KL penalty toward a standard normal or by vector quantization, “to avoid arbitrarily high-variance latent spaces” (§3.1). Its two-dimensional structure is kept for the convolutional denoiser, whereas earlier two-stage models imposed an arbitrary one-dimensional ordering on the code to model it autoregressively (§3.1). The autoencoder and LDM articles cover the choice of $f$ and the rescaling of the latent to unit variance.

In our reading, such a latent is a different kind of latent space from a GAN’s $z$. It is close to an image (a grid of local codes), each image has essentially one code, and the distribution over codes is supplied by the diffusion model rather than by a fixed prior, so semantic edits are usually made through the diffusion process rather than by moving the autoencoder’s code. The diffusion model has a latent space of its own, the initial noise $x_T$: with a deterministic sampler, interpolating between two noise samples gives meaningful interpolations, as the diffusion sampling article describes for DDIM. That noise is a high-dimensional Gaussian ($64 \times 64 \times 4 = 16{,}384$ numbers for Stable Diffusion v1 at $512 \times 512$), so the norm argument above applies to it with more force (our reasoning).

Examples

Example: lerp and slerp on Gaussian latents

The NumPy example (NumPy 2.3.5, under a second) draws 20,000 pairs of independent Gaussian codes for dimensions 2, 16, 128 and 512 and measures what interpolation does to their norms. It prints the mean of $\lVert z \rVert / \sqrt{d}$ next to the exact mean of the chi distribution, the range that holds 98% of prior samples (1st to 99th percentile), the angle between the two codes, the norm of the lerp and slerp midpoints, the share of midpoints whose norm falls outside the prior’s range, and how much higher the Gaussian log-density is at the lerp midpoint than at the endpoints, on average.

import math
import sys
import numpy as np

N = int(sys.argv[1]) if len(sys.argv) > 1 else 20000   # pairs per dimension
rng = np.random.default_rng(0)

def slerp(z1, z2, t):
    """Spherical interpolation: rotate z1 toward z2 by a fraction t of the angle between them."""
    cos = np.sum(z1 * z2, -1, keepdims=True) / (
        np.linalg.norm(z1, axis=-1, keepdims=True) * np.linalg.norm(z2, axis=-1, keepdims=True))
    theta = np.arccos(np.clip(cos, -1, 1))
    return (np.sin((1 - t) * theta) * z1 + np.sin(t * theta) * z2) / np.sin(theta)

print(f"NumPy {np.__version__}; {N} pairs z1, z2 ~ N(0, I_d) per dimension d")
print(f"{'d':>4s} {'E|z|/sqrt d':>11s} {'chi mean':>9s} {'1-99% of |z|/sqrt d':>20s}"
      f" {'angle (deg)':>12s} {'lerp mid':>9s} {'slerp mid':>9s}"
      f" {'outside: lerp':>14s} {'slerp':>6s} {'logp gain':>10s}")
for d in [2, 16, 128, 512]:
    z1, z2 = rng.standard_normal((2, N, d))
    r = np.linalg.norm(z1, axis=1) / math.sqrt(d)
    chi = math.sqrt(2) * math.exp(math.lgamma((d + 1) / 2) - math.lgamma(d / 2)) / math.sqrt(d)
    lo, hi = np.quantile(r, [0.01, 0.99])                 # the prior's typical range of norms
    cos = np.sum(z1 * z2, 1) / (np.linalg.norm(z1, axis=1) * np.linalg.norm(z2, axis=1))
    ang = np.degrees(np.arccos(cos))
    m_lin = 0.5 * (z1 + z2)                               # lerp at t = 0.5
    m_sph = slerp(z1, z2, 0.5)
    r_lin = np.linalg.norm(m_lin, axis=1) / math.sqrt(d)
    r_sph = np.linalg.norm(m_sph, axis=1) / math.sqrt(d)
    out = lambda q: np.mean((q < lo) | (q > hi))
    # log N(m; 0, I) minus the mean log density of the two endpoints (constants cancel)
    gain = 0.25 * (np.sum(z1**2, 1) + np.sum(z2**2, 1)) - 0.5 * np.sum(m_lin**2, 1)
    print(f"{d:4d} {r.mean():11.3f} {chi:9.3f} {lo:9.3f} to {hi:6.3f}"
          f" {ang.mean():6.1f}±{ang.std():4.1f} {r_lin.mean():9.3f} {r_sph.mean():9.3f}"
          f" {out(r_lin):14.1%} {out(r_sph):6.1%} {gain.mean():10.1f}")

Output (two runs gave identical output):

NumPy 2.3.5; 20000 pairs z1, z2 ~ N(0, I_d) per dimension d
   d E|z|/sqrt d  chi mean  1-99% of |z|/sqrt d  angle (deg)  lerp mid slerp mid  outside: lerp  slerp  logp gain
   2       0.888     0.886     0.103 to  2.141   89.5±51.8     0.629     1.869           2.1%   8.7%        0.5
  16       0.986     0.985     0.608 to  1.416   89.9±14.8     0.697     0.994          24.2%   0.1%        4.0
 128       0.998     0.998     0.858 to  1.146   90.1± 5.1     0.705     0.999         100.0%   0.1%       32.0
 512       0.999     1.000     0.926 to  1.072   90.0± 2.5     0.707     1.000         100.0%   0.1%      128.0

The sample means match the chi distribution, and the range holding 98% of the prior’s norms narrows from a factor of 20 at $d = 2$ to 0.93–1.07 at $d = 512$: high-dimensional Gaussian codes live on a thin shell. Angles between independent codes average 90° at every $d$, but their spread falls from 52° to 2.5°, so in 512 dimensions two random codes are almost exactly orthogonal. Lerp midpoint norms approach $0.71\sqrt{d}$ as $d$ grows, as derived above. At $d = 2$ that is unremarkable (2.1% fall outside the prior’s range, about what the definition of the range allows), at $d = 16$ a quarter do, and from $d = 128$ essentially all do. In the last column the midpoint’s log-density exceeds the endpoints’ by $d/4$ nats on average, which follows from $\lVert z_1 \rVert^2 + \lVert z_2 \rVert^2 \approx 2d$ and $\lVert m \rVert^2 \approx d/2$. The gain is never negative: it equals $\lVert z_1 - z_2 \rVert^2 / 8$ (our derivation). In high dimensions the lerp midpoint is more probable per unit volume than almost every sample the prior produces, yet it is a kind of point that sampling essentially never produces.

Slerp midpoints keep the norm at $\sqrt{d}$ from $d = 16$ on. Only 0.1% of them fall outside the prior’s range, fewer than the 2% of prior samples the range excludes, because for nearly orthogonal codes the slerp midpoint’s norm is the root mean square of two endpoint norms, which varies less than one norm does. At $d = 2$ slerp misbehaves: the mean midpoint norm is 1.87 and 8.7% fall outside the range. In two dimensions the angle between independent codes is uniform between 0° and 180° (hence the spread of about 52°), so nearly opposite pairs of unequal norm are common, and for them the factor $1/(2\cos(\theta/2))$ in the midpoint formula is large. Near $\theta = \pi$ the midpoint norm grows like $1/(\pi - \theta)$, so at $d = 2$ its expected value is unbounded and the sample mean does not settle as more pairs are drawn (our derivation); the share outside the range is the more stable summary there. Whether lerp’s inner points matter for a given model is an empirical question about its generator; the example shows only that, under a Gaussian prior of the dimensions used in practice, lerp’s midpoints lie in a region of $\mathcal{Z}$ that the prior does not populate.

Examples in vision

• Face attribute editing. InterFaceGAN changes pose, smile, age, gender and eyeglasses of generated faces by moving along SVM normals and, combined with inversion, of photographs (Shen et al., 2020, §3.5); GANSpace finds rotation, aging, lighting and hair-color controls without labels.
• Scene and object edits. GANSpace’s layer-wise BigGAN and StyleGAN edits change viewpoint and zoom of objects, illumination, time of day and cloudiness of landscapes, and car color and type (Härkönen et al., 2020, §2.3, Figure 1).
• Morphing and style mixing. Linear paths and per-layer crossovers in $\mathcal{W}+$ morph between faces and transfer coarse or fine styles (Abdal et al., 2019, §4).
• Disentanglement benchmarks. dSprites, Cars3D, SmallNORB and Shapes3D, whose images are rendered from known factors, are among the datasets on which BetaVAE, FactorVAE, MIG and the other measures are computed (Locatello et al., 2019, §4).

Common Misconceptions

• “The densest region of the prior is where typical samples are.” The Gaussian density peaks at the origin, but in high dimensions almost no samples lie near it; in the example the lerp midpoints at $d = 128$ and $d = 512$ have a higher log-density than the mean log-density of their endpoints, yet essentially all lie outside the range of norms the prior produces.
• “The coordinate axes of $z$ are meaningful.” For an isotropic prior every orthogonal basis is equally likely, so axis-aligned edits in $\mathcal{Z}$ are random directions (Härkönen et al., 2020, §3.3). Meaningful directions have to be found, with labels or from the structure of a non-isotropic space such as $\mathcal{W}$.
• “A linear direction changes only its attribute.” A direction learned from correlated data carries the correlation: InterFaceGAN’s age direction added eyeglasses, and GANSpace’s car edits changed backgrounds. Conditional manipulation removes the projection onto directions to be preserved, but it cannot help when, as in StyleGAN’s $\mathcal{W}$, the correlated attribute was folded into the direction itself (Shen et al., 2020, §3.4).
• “Larger edits are stronger versions of the same edit.” Codes moved far from the hyperplane leave the region where the prior has mass and the face stops looking like the same person (Shen et al., 2020, §3.2).
• “A model that scores well on one disentanglement measure is disentangled.” The measures capture different notions, agree only partly, and depend more on the seed and the regularization strength than on the method; none can be computed without ground-truth factors except the encoder-free ones, which measure different things (Locatello et al., 2019).
• “An embedding and a latent code are the same thing.” Both are vectors that represent images, but an embedding is computed from an image to compare images, while a latent code is an input from which the model generates an image. An encoder’s code in a VAE is both.

Where It Is Used

• Image editing with GANs works by inverting a photo into a latent space and moving its code; diffusion editing works through the noise latent and the diffusion process instead.
• StyleGAN introduced $\mathcal{W}$ and the encoder-free measures, and its spaces are the usual setting for direction-finding and inversion.
• Latent diffusion and Stable Diffusion generate in an autoencoder’s spatial latent space and sample from a Gaussian noise latent.
• Variational autoencoders define a latent space with an encoder and a prior, and β-VAE and its relatives were designed for disentangled codes.
• Visual anomaly detection with AnoGAN searches a GAN’s latent space for the code that best reproduces a test image and scores the residual.
• Representation learning studies disentanglement as a property of learned features, and Locatello et al.’s results bound what can be learned without labels.