Perceptual Loss

A perceptual loss compares two images by passing both through a fixed, pretrained convolutional neural network and measuring the difference between their activations, instead of the difference between their pixels. The network, usually VGG trained on ImageNet classification, is not trained further; it only defines the distance. Two forms are standard. The feature reconstruction loss, also called the content loss or, in super-resolution papers, the VGG loss, compares the feature maps of one layer position by position. The style reconstruction loss, or style loss, compares Gram matrices, the correlations between the channels of a layer averaged over all positions, which keep the texture of an image and discard its layout. Johnson, Alahi and Fei-Fei (2016) used the name “perceptual loss functions” for both and trained feed-forward networks for style transfer and super-resolution with them; the two losses come from the texture synthesis and style transfer work of Gatys, Ecker and Bethge (2015, 2016) and from Mahendran and Vedaldi’s (2015) inversion of network features. Perceptual losses train super-resolution networks, style-transfer networks, the autoencoders of latent diffusion and inpainting models. LPIPS is the related metric: the same kind of feature distance, calibrated on human similarity judgments.

Definition

Let $\phi$ be a fixed network and $\phi_j(x)$ the activations of its layer $j$ for an image $x$, a feature map with $C_j$ channels on an $H_j \times W_j$ grid. A perceptual loss between an output image $\hat{y}$ and a target $y$ is a distance between $\phi_j(\hat{y})$ and $\phi_j(y)$, or between statistics of them, summed over one or more layers. The feature reconstruction loss compares the activations at corresponding positions, so it constrains what is where in the image. The style reconstruction loss compares the Gram matrix of each layer, a $C_j \times C_j$ matrix of channel co-activations averaged over positions, so it constrains which local patterns occur and how often, but not where.

“Perceptual” names the intent, not a property that was measured: the losses were proposed because distances in the feature space of a recognition network looked closer to human judgments than pixel distances. Whether they are was tested later, by the LPIPS study described below.

Intuition

Why not pixels. A per-pixel loss treats each pixel as independent, so it punishes a small shift of an edge as heavily as a wrong edge; the one-pixel-shift example of Johnson et al. is in the loss function article. A second failure matters more for image generation. When the target is ambiguous, as in 4× super-resolution, filling a large hole or reconstructing an image from abstract features, many sharp outputs are consistent with the input, and the output that minimizes the expected squared error is their average, which is blurred. Dosovitskiy and Brox (2016) put it this way: the features from which an image is reconstructed do not fix the exact locations of its details, a loss in image space averages over all likely locations, and what matters for perceived similarity is the distribution of details rather than their exact positions. Features of a trained network, they argued, are invariant to small, smooth deformations but sensitive to salient edges and textures, so a distance between features penalizes a missing texture more than a displaced one.

What a layer keeps. The easiest way to see what a feature loss constrains is to run it without a network to train: start from noise and change the pixels by gradient descent until their features match those of a target image. Mahendran and Vedaldi (2015) did this for each layer of an AlexNet-style network and found that the first layers are essentially an invertible code of the image, that all the convolutional layers keep a photographically faithful picture with increasing fuzziness, and that the fully connected layers return compositions of parts similar, but not identical, to those of the original. Gatys et al. (2016, Fig. 1) found reconstructions from conv1_2, conv2_2 and conv3_2 of VGG-19 almost perfect, and from conv4_2 and conv5_2 faithful in content but not in exact pixel values; Johnson et al. (Fig. 3) saw, for VGG-16, that reconstructions from higher layers keep content and overall spatial structure but not colour, texture and exact shape. A loss on a high layer therefore accepts many images as equally good, which is what a loss for an ambiguous problem should do, and also what lets artefacts through.

What a Gram matrix keeps. Multiplying a layer’s feature map with itself and averaging over positions throws away where each feature fired and keeps which features fire together. Matching only these statistics regenerates the texture of an image with its layout scrambled. With the Gram matrix of the first VGG layer alone, Gatys et al. (2015) obtained images with little structure, close to noise with a matched spectrum; adding the layers up to pool4 gave natural textures they found almost indistinguishable from the source, and the size of the regions in which local arrangement survived grew with the number of layers, which they explained by the growing receptive fields. Applied to a photograph that is not a texture, the same procedure gives a texturised version of it.

Formal Definition

Feature reconstruction loss

In Johnson et al.’s notation the feature reconstruction loss at layer $j$ is the squared, normalized Euclidean distance

$$\ell^{\phi,j}_{\mathrm{feat}}(\hat{y}, y) = \frac{1}{C_j H_j W_j} \big\lVert \phi_j(\hat{y}) - \phi_j(y) \big\rVert_2^2 ,$$

where the norm runs over all $C_j H_j W_j$ entries (their Eq. 2). The papers that use it differ in the normalization, which changes only how the term is weighted against the others:

• Gatys et al. (2016, Eq. 1) write the content loss as half the sum of squared differences, $\tfrac{1}{2}\sum_{i,j} (F^l_{ij} - P^l_{ij})^2$, with $F^l$ and $P^l$ the $N_l \times M_l$ matrices of the generated and content image’s responses ($N_l$ filters, $M_l$ positions); the caption of their Fig. 2 calls it a mean squared difference, which the equation is not.
• SRGAN (Ledig et al., 2017, Eq. 5) divides the sum over positions by $W_{i,j} H_{i,j}$, the size of the VGG-19 feature map; its terms and scaling are detailed in the SRGAN paper article.
• Mahendran and Vedaldi (2015, §2) divide by the squared norm of the target’s features, $\lVert \Phi(x) - \Phi_0 \rVert^2 / \lVert \Phi_0 \rVert^2$, so that the loss lies roughly in $[0, 1)$ near the optimum and can be balanced against an image prior.
• LaMa’s loss (Suvorov et al., 2022, Eq. 1) takes the mean of squared differences within each layer and then the mean over layers.

Style reconstruction loss

The Gram matrix of layer $j$ collects the inner products between channels $c$ and $c'$ over all positions,

$$G^{\phi}_j(x)_{c,c'} = \frac{1}{C_j H_j W_j} \sum_{h=1}^{H_j} \sum_{w=1}^{W_j} \phi_j(x)_{h,w,c}\, \phi_j(x)_{h,w,c'} ,$$

or, reshaping $\phi_j(x)$ into a $C_j \times H_j W_j$ matrix $\psi$, $G^\phi_j = \psi \psi^\top / (C_j H_j W_j)$ (Johnson et al., Eq. 3). Treating each grid position as a sample of a $C_j$-dimensional feature vector, $G_j$ is proportional to the uncentred covariance of the features. The style loss is the squared Frobenius norm of the difference, summed over a set of layers $J$:

$$\ell^{\phi,J}_{\mathrm{style}}(\hat{y}, y) = \sum_{j \in J} \big\lVert G^\phi_j(\hat{y}) - G^\phi_j(y) \big\rVert_F^2 .$$

Because $G_j$ is $C_j \times C_j$ whatever the image size, the loss is defined between images of different sizes. Gatys et al. (2015, Eqs. 1–3; 2016, Eqs. 3–5) use the unnormalized Gram matrix $G^l_{ij} = \sum_k F^l_{ik} F^l_{jk}$ and put the normalization in the per-layer loss, $E_l = \frac{1}{4 N_l^2 M_l^2} \sum_{i,j} (G^l_{ij} - A^l_{ij})^2$, with total $\sum_l w_l E_l$; in their style-transfer results $w_l = 1/5$ for each of the five style layers.

A Gram matrix is a large description. Each layer with $N_l$ feature maps contributes $N_l(N_l + 1)/2$ numbers, about 852,000 for all VGG-19 layers up to pool4 (Gatys et al., 2015). The authors found this heavily over-parameterized: one layer per scale (conv1_1 and pool1 to pool4) gave about 177,000 numbers with hardly any loss of quality, Gram matrices of the first 64 principal components about 10,000, and the 1,024 feature-map averages of those layers alone already produced interesting textures.

Combined objectives

A perceptual loss is almost always one term among several. Gatys et al. (2016, Eq. 7) synthesize a stylized image by minimizing $\alpha \mathcal{L}_{\mathrm{content}} + \beta \mathcal{L}_{\mathrm{style}}$ over the pixels, with $\alpha/\beta$ between $5 \times 10^{-4}$ and $5 \times 10^{-3}$ in their main figure and from $10^{-4}$ (a texturised version of the painting) to $10^{-1}$ (little stylization) in their trade-off figure. Johnson et al. add a total variation term, $\lambda_c \ell_{\mathrm{feat}} + \lambda_s \ell_{\mathrm{style}} + \lambda_{TV} \ell_{TV}$ (Eq. 5), with the total variation weight between $10^{-6}$ and $10^{-4}$ chosen per style by cross-validation. Dosovitskiy and Brox (2016, Eq. 1) combine a feature loss, an adversarial loss and a small pixel loss, $\lambda_{\mathrm{feat}} \mathcal{L}_{\mathrm{feat}} + \lambda_{\mathrm{adv}} \mathcal{L}_{\mathrm{adv}} + \lambda_{\mathrm{img}} \mathcal{L}_{\mathrm{img}}$, a family they call deep perceptual similarity metrics (DeePSiM), with $\lambda_{\mathrm{adv}} = 100$, $\lambda_{\mathrm{img}} = 2 \cdot 10^{-6}$ and $\lambda_{\mathrm{feat}} = 0.01$ for an AlexNet CONV5 comparator; they note that the large adversarial weight only compensates for the other two terms summing over many positions. LaMa weights its adversarial, perceptual, discriminator-feature and gradient-penalty terms by 10, 30, 100 and 0.001, found by beam search on a validation subset (Suvorov et al., 2022, §3). Because each paper normalizes differently, these weights cannot be carried from one implementation to another.

Properties

Optimizing an image or training a network

Gatys et al. minimize their loss over the pixels of one image with L-BFGS, starting from white noise, so every result is an optimization: at about $512 \times 512$ pixels it took up to an hour on an Nvidia K40 GPU (2016, §4). Johnson et al. moved the loss into training. A feed-forward image transformation network, an encoder–decoder with residual blocks, is trained on MS-COCO to minimize the same objective for one style, and then stylizes an image in one forward pass. On 50 COCO validation images with Picasso’s The Muse as style, the network reached values of the Gatys objective comparable to 50 to 100 L-BFGS iterations of the original method (Fig. 5); on a GTX Titan X it took 0.015 s at $256 \times 256$ against 15.86 s for 500 iterations, a speed-up of about 1,000 times (Table 1). Their qualitative comparison found the results similar, sometimes with more repetitive patterns. The same split runs through later uses: a perceptual loss can be the objective of an optimization over one image, as in feature inversion, or the training loss of a network that is applied many times. Style transfer by both routes is the subject of neural style transfer; here only the losses matter.

Which layers, and what they constrain

Work	Loss network	Feature (content) layer	Style layers
Gatys et al. (2015), texture synthesis	VGG-19, average pooling	none	all layers up to pool4; reduced model conv1_1, pool1–pool4
Gatys et al. (2016), style transfer	VGG-19, normalized, average pooling	conv4_2	conv1_1, conv2_1, conv3_1, conv4_1, conv5_1
Johnson et al. (2016), style transfer	VGG-16	relu3_3	relu1_2, relu2_2, relu3_3, relu4_3
Johnson et al. (2016), super-resolution	VGG-16	relu2_2	none
Dosovitskiy & Brox (2016)	AlexNet (CaffeNet)	CONV5 (after pooling)	none
SRGAN (Ledig et al., 2017)	VGG-19	$\phi_{2,2}$ or $\phi_{5,4}$, after activation	none
ESRGAN (Wang et al., 2018)	VGG-19	features before activation (layer not named)	none
LaMa (Suvorov et al., 2022)	segmentation ResNet-50 with dilated convolutions	several layers, averaged	none
VQGAN, LDM autoencoder	LPIPS (Zhang et al., 2018)	as in LPIPS	none

The layer names follow each paper. The VGG layers and their naming are described in the VGG article, which notes that the arXiv v1 preprint of Johnson et al. gives relu2_2 for the style-transfer feature loss as well; the authors’ copy of the ECCV paper read here gives relu3_3 (§4.1).

The layer sets the trade-off between fidelity and freedom. For content, Gatys et al. (2016, Fig. 5) compared conv2_2 with conv4_2 on the same painting and weights: with conv2_2 the painting’s texture looked merely blended over the photograph, while conv4_2 let the fine structure, edges and colour map change to agree with the style while the content stayed recognizable. For super-resolution Johnson et al. chose the low layer relu2_2, because their inversions showed more pronounced distortions from higher layers (§4.2). SRGAN tried both ends of VGG-19: a trend towards better texture detail with $\phi_{5,4}$, no significantly best loss by mean opinion score on Set5, and SRGAN-VGG54 significantly better than every other variant on Set14 (§3.4, Table 1). For style, adding higher layers matches local structure on a larger scale, giving what Gatys et al. describe as a smoother and more continuous result, which is why they used five layers up to conv5_1 (§3.2). Style and content interact as well: Gatys et al. note that there is usually no image that matches both constraints perfectly, and that one cannot render an image in the style of The Starry Night without structures that look like its stars (§3.1, §4).

Which network

The loss network’s training matters. Gatys et al. (2015, Fig. 3) found that a network with VGG’s architecture and random weights failed at texture synthesis, and that the Caffe reference network, which is close to AlexNet, matched the statistics less fully and overlaid the textures with an artifactual grid, which they attributed to its larger receptive fields and strides. Dosovitskiy and Brox (2016, Fig. 6) obtained their best reconstructions with the CONV5 features of supervised AlexNet, but a randomly initialized AlexNet and a network trained without labels on video frames also preserved key image features, and VAE samples trained with the unsupervised comparator looked qualitatively similar to those with AlexNet. Zhang et al. (2018) later found that agreement with human similarity judgments held across architectures and across supervised, self-supervised and unsupervised training, while a randomly initialized network did much worse.

The pretext task matters too. LaMa argued that classification networks focus on texture, which can bias inpainting, and used a ResNet-50 trained for segmentation, with dilated convolutions so that the receptive field grows quickly. In its loss ablation on Places (LaMa-Regular, segmentation-based masks, Table 3), the segmentation ResNet-50 with dilation gave FID 5.69 and LPIPS 0.059; a classification ResNet-50 with dilation, 5.87 and 0.059; without dilation, 6.00 and 0.061; and a classification VGG-19, 6.29 and 0.063, each with its weight searched separately. For super-resolution, ESRGAN also tried a VGG network fine-tuned for material recognition, which focuses on textures rather than objects, and reported only a marginal gain in the perceptual index of the PIRM challenge (Wang et al., 2018, §3.3).

Artefacts

A feature loss constrains an image only through what the network responds to, and the images that a layer maps to the same activations are not all natural. The papers describe several resulting artefacts:

• High-frequency noise and grids. Optimizing for similarity in a high-level feature space alone typically gives high-frequency artefacts, because feature representations are contractive and many non-natural images share the features of a natural one (Dosovitskiy & Brox, 2016, §1, §3; their Fig. 1(d) shows reconstructions with image and feature losses but no adversarial term looking very noisy). Johnson et al. found grid-like artefacts at the pixel level in many of their super-resolution outputs and saw the same artefacts in their optimization-based inversions, so they attributed them to the feature reconstruction loss rather than to the network architecture (§4.2). Gatys et al. (2016, §4) report low-level noise in stylized images that resembles the filters of the network’s units, most visible when both images are photographs.
• Lower PSNR. A network trained with $\ell_{\mathrm{feat}}$ is not trained to match pixels, so it scores lower on PSNR and SSIM. With identical architecture, data and training, Johnson et al.’s ×4 network trained with $\ell_{\mathrm{feat}}$ reached 27.09 dB / 0.7680 on Set5 against 28.40 dB / 0.8205 for the same network trained with a pixel loss (Fig. 8, means on the Y channel), while in an Amazon Mechanical Turk study a majority of five workers preferred the $\ell_{\mathrm{feat}}$ result over SRCNN on 96% of BSD100 images. They concluded that the pixel loss gives fewer artefacts and higher PSNR, the feature loss better fine detail.
• Repetition and scale. Johnson et al.’s feed-forward style networks sometimes produced more repetitive patterns than optimization, such as repeated yellow splotches for The Starry Night, and the effect grew at higher resolution (§4.1). Gatys et al. resized the style image to the content image’s size so that information was extracted on comparable scales (2016, §2.3); in our reading, style statistics are matched at the scale of the loss network’s receptive fields, so the size of the matched patterns is tied to the image resolution.
• Boundaries. In Gatys et al.’s textures, a yellow box from the corner of a source texture reappeared near the same corner; since Gram matrices store no positions, they concluded that some VGG features encode image boundaries, possibly because of zero padding, and found feature maps from conv3_1 onwards that respond only along the edges (2015, footnote 2).
• Sparse features and brightness. ESRGAN argued that VGG features after the activation are very sparse in deep layers, only 11.17% of neurons active after layer VGG19-54 (the fourth convolution before the fifth max-pooling) for its baboon image, which gives weak supervision, and that losses on them shift the brightness of outputs; it moved the loss before the activation (Wang et al., 2018, §3.3).

These artefacts are why perceptual losses come with priors. Mahendran and Vedaldi regularized inversions with an $\alpha$-norm and a total variation term $V^\beta$, and found that $\beta = 1$ produced “spikes” when the network subsamples, which $\beta = 2$ removed at the price of a washed-out image (§2); Johnson et al. kept a total variation term; Dosovitskiy and Brox replaced hand-made priors with an adversarial loss from a discriminator that judges local image statistics, which SRGAN, LaMa and the LDM autoencoder also use (see generative adversarial network).

The relation to LPIPS

Zhang et al. (2018) opened their paper with the question of how perceptual these “perceptual losses” are, and built the BAPPS dataset of human two-alternative judgments to test it. Deep features beat classic metrics such as SSIM by large margins, whatever the architecture and level of supervision, and they turned the finding into a metric, LPIPS, which differs from Johnson et al.’s feature loss in two steps: each position’s feature vector is unit-normalized across channels, and the channels of several layers are weighted by coefficients fitted to the human judgments. The formula, the variants, the evaluation and the conventions that change its value are in the LPIPS article; this article treats it only as a loss.

The calibrated metric came back as a loss. Esser, Rombach and Ommer (2021) trained VQGAN, a variant of the VQ-VAE, with the L2 reconstruction term replaced by a perceptual loss, citing Zhang et al., plus a patch-based adversarial loss (§3.1). The autoencoder of latent diffusion models follows them, with a perceptual loss again citing Zhang et al. and a patch-based adversarial objective, which, the authors write, keeps reconstructions on the image manifold and avoids the blur of pixel-space losses such as L2 or L1 alone (Rombach et al., 2022, §3.1; Appendix F of the supplement gives the full objective). Two consequences follow, in our reading. A model trained with an LPIPS loss and then evaluated with LPIPS is graded by its own objective, a caution the LPIPS article also gives. And because LPIPS is a network, the adversarial weaknesses reported for it as a metric apply to it as a loss.

Examples

Inverting a small network, layer by layer. VGG’s ImageNet weights are not downloaded here, so the example trains a stand-in: a VGG-like network of three blocks of two $3 \times 3$ convolutions with ReLU, average pooling between blocks, He initialization, global average pooling and a linear classifier, trained from scratch to recognize the one filled disk, square, triangle or cross in $32 \times 32$ RGB images with random size, position and colours. It is far smaller than VGG and its features are learned from four synthetic classes; it plays VGG’s role, not its part. As in Gatys et al., each channel is divided by its mean activation, here over 500 training images. For eight test images, two per class, the code starts from the same uniform noise and runs Adam on the pixels (2,000 steps, learning rate 0.05 with cosine decay, pixels clipped to $[0, 1]$ after each step, no image prior) to match either the features of one layer, relu1_2, relu2_2 or relu3_2, with Mahendran and Vedaldi’s normalized loss, or the Gram matrices of every layer up to that one, each term divided by the squared norm of the target Gram matrix (a normalization of our choosing).

The table reports, averaged over the eight images, the final loss; the pixel RMSE to the original; the RMSE after a Gaussian blur of 2 pixels, which keeps only coarse layout; a colour distance, the one-dimensional Wasserstein distance between the pixel-value distributions of each channel, which ignores where pixels are; the high-frequency ratio, the RMS of the Laplacian of the reconstruction over that of the original (above 1 means more fine-scale variation than the original); and the fraction of reconstructions that a second network of the same architecture, trained with another seed, assigns to the right shape, so the inverted network does not grade itself. An untrained copy of the network, with the same initialization and scaling, is a control: Gatys et al. (2015) found texture synthesis to fail with random VGG weights, while Dosovitskiy and Brox and Zhang et al. found random features weaker but not useless. Before the reconstructions, the code counts, from the Jacobian of each layer at one test image, the input directions to which the layer is almost blind. The example was planned with a base check that the reconstruction from relu1_2 recovers the image; pilot runs before the first run did not reach exact recovery, so the check applied was weaker: the relu1_2 reconstruction of the trained network must end far below the pixel error of the noise start, and that layer’s Jacobian must have no near-zero singular value. The budget, the learning-rate decay and the Jacobian count were set in the same pilots, before the first run. The run takes four to five minutes on four CPU threads, most of it training the two networks (PyTorch 2.14.1, NumPy 2.5.4, SciPy 1.18.1).

import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
from scipy.ndimage import gaussian_filter, laplace

torch.set_num_threads(4)

def make_shapes(n, rng, size=32):
    """One filled shape per 32x32 RGB image: 0 disk, 1 square, 2 triangle, 3 cross."""
    yy, xx = np.mgrid[0:size, 0:size] + 0.5
    X = np.empty((n, 3, size, size), np.float32); y = np.arange(n) % 4
    for i in range(n):
        r = rng.uniform(6, 11); cx, cy = rng.uniform(r + 1, size - r - 1, 2)
        dx, dy = xx - cx, yy - cy
        if y[i] == 0: m = dx**2 + dy**2 <= r**2
        elif y[i] == 1: m = (np.abs(dx) <= 0.8 * r) & (np.abs(dy) <= 0.8 * r)
        elif y[i] == 2: m = (dy <= 0.8 * r) & (dy >= -r + 2 * np.abs(dx))
        else: m = ((np.abs(dx) <= 0.3 * r) | (np.abs(dy) <= 0.3 * r)) & (np.abs(dx) <= r) & (np.abs(dy) <= r)
        fg, bg = rng.uniform(0, 1, 3), rng.uniform(0, 1, 3)          # random shape and background colours
        while np.abs(fg - bg).max() < 0.3: fg, bg = rng.uniform(0, 1, 3), rng.uniform(0, 1, 3)
        img = np.where(m[None], fg[:, None, None], bg[:, None, None]) + rng.normal(0, 0.03, (3, size, size))
        X[i] = np.clip(img, 0, 1)
    return torch.tensor(X), torch.tensor(y)

class Net(nn.Module):
    """VGG-like stand-in: three blocks of two 3x3 convolutions + ReLU, average pooling between blocks."""
    def __init__(s):
        super().__init__()
        c = lambda a, b: nn.Conv2d(a, b, 3, padding=1)
        s.blocks = nn.ModuleList([nn.ModuleList([c(3, 16), c(16, 16)]), nn.ModuleList([c(16, 32), c(32, 32)]),
                                  nn.ModuleList([c(32, 64), c(64, 64)])])
        s.fc = nn.Linear(64, 4)
        for m in s.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu"); nn.init.zeros_(m.bias)
    def feats(s, x):                                                   # relu1_2, relu2_2, relu3_2
        out = []
        for k, (c1, c2) in enumerate(s.blocks):
            if k: x = F.avg_pool2d(x, 2)
            x = F.relu(c2(F.relu(c1(x)))); out.append(x)
        return out
    def forward(s, x): return s.fc(s.feats(x)[-1].mean((2, 3)))

def train(seed, X, y, epochs=16):
    torch.manual_seed(seed); net = Net(); opt = torch.optim.Adam(net.parameters(), 1e-3)
    g = torch.Generator().manual_seed(seed)
    for _ in range(epochs):
        for idx in torch.randperm(len(X), generator=g).split(64):
            loss = F.cross_entropy(net(X[idx]), y[idx]); opt.zero_grad(); loss.backward(); opt.step()
    return net.eval().requires_grad_(False)

def scaled_features(net, Xref):          # as Gatys et al.: mean activation of every channel set to 1
    means = [f.mean((0, 2, 3)).clamp_min(1e-8)[None, :, None, None] for f in net.feats(Xref)]
    return lambda x: [f / m for f, m in zip(net.feats(x), means)]

def gram(f):                             # Johnson et al., Eq. 3
    b, c, h, w = f.shape; f = f.reshape(b, c, h * w); return f @ f.transpose(1, 2) / (c * h * w)

def reconstruct(phi, x0, target, l, steps=2000, lr=0.05):
    """Gradient descent on the pixels, from noise, to match layer l (feature) or Gram matrices of layers 1..l."""
    f0 = phi(x0); G0 = [gram(f) for f in f0]
    def loss_fn(x):                      # normalized by the target's own size, per image
        f = phi(x)
        if target == "feature":
            return ((f[l] - f0[l]) ** 2).sum((1, 2, 3)) / (f0[l] ** 2).sum((1, 2, 3))
        return sum(((gram(f[k]) - G0[k]) ** 2).sum((1, 2)) / (G0[k] ** 2).sum((1, 2)) for k in range(l + 1))
    x = torch.rand(x0.shape, generator=torch.Generator().manual_seed(3)).requires_grad_(True)
    opt = torch.optim.Adam([x], lr); sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, steps)
    for _ in range(steps):
        loss = loss_fn(x).sum(); opt.zero_grad(); loss.backward(); opt.step(); sched.step()
        with torch.no_grad(): x.clamp_(0, 1)                         # keep pixels in [0, 1]
    with torch.no_grad(): return x.detach(), loss_fn(x).mean().item()

def measures(x, x0, judge, y):
    a, b = x.numpy(), x0.numpy()
    rmse = np.sqrt(((a - b) ** 2).mean((1, 2, 3))).mean()
    blur = lambda v: gaussian_filter(v, (0, 0, 2, 2))                # coarse layout only
    rmse_blur = np.sqrt(((blur(a) - blur(b)) ** 2).mean((1, 2, 3))).mean()
    srt = lambda v: np.sort(v.reshape(*v.shape[:2], -1), -1)         # colour histogram, positions ignored
    w1 = np.abs(srt(a) - srt(b)).mean()
    lap = lambda v: np.sqrt(np.mean([laplace(v[i, c]) ** 2 for i in range(len(v)) for c in range(3)]))
    return rmse, rmse_blur, w1, lap(a) / lap(b), (judge(x).argmax(1) == y).float().mean().item()

rng = np.random.default_rng(0)
Xtr, ytr = make_shapes(4000, rng); Xte, yte = make_shapes(1000, rng)
net, judge = train(0, Xtr, ytr), train(1, Xtr, ytr)                 # inverted network and an independent judge
torch.manual_seed(2); rnd = Net().eval().requires_grad_(False)       # untrained control, same initialization
print(f"test accuracy: inverted net {(net(Xte).argmax(1) == yte).float().mean():.3f}, "
      f"judge {(judge(Xte).argmax(1) == yte).float().mean():.3f}")
sel = torch.cat([torch.where(yte == k)[0][:2] for k in range(4)]); x0, y0 = Xte[sel], yte[sel]

print("\nJacobian of each layer at test image 0: singular values below 1e-3 / 1e-6 of the largest (of 3072)")
for name, n in [("trained", net), ("random", rnd)]:
    phi = scaled_features(n, Xtr[:500])
    for l in range(3):
        J = torch.autograd.functional.jacobian(lambda v: phi(v[None])[l].flatten(), x0[0]).reshape(-1, 3072)
        s = torch.linalg.svdvals(J)
        print(f"  {name:8s} relu{l+1}_2: {(s < 1e-3 * s[0]).sum().item():5d} / {(s < 1e-6 * s[0]).sum().item():5d}")

print(f"\n{'network':8s} {'target':8s} {'layers':8s} {'loss':>7s} {'RMSE':>6s} {'blurRMSE':>8s} "
      f"{'colourW1':>8s} {'HFratio':>7s} {'judge':>5s}")
noise = torch.rand(x0.shape, generator=torch.Generator().manual_seed(3))
for label, x in [("noise", noise), ("original", x0)]:
    r = measures(x, x0, judge, y0)
    print(f"{'-':8s} {label:8s} {'-':8s} {'-':>7s} {r[0]:6.3f} {r[1]:8.3f} {r[2]:8.3f} {r[3]:7.2f} {r[4]:5.2f}")
for name, n in [("trained", net), ("random", rnd)]:
    phi = scaled_features(n, Xtr[:500])
    for target in ["feature", "gram"]:
        for l in range(3):
            x, loss = reconstruct(phi, x0, target, l); r = measures(x, x0, judge, y0)
            lay = f"relu{l+1}_2" if target == "feature" else f"1..{l+1}"
            print(f"{name:8s} {target:8s} {lay:8s} {loss:7.4f} {r[0]:6.3f} {r[1]:8.3f} {r[2]:8.3f} {r[3]:7.2f} {r[4]:5.2f}")

Output:

test accuracy: inverted net 0.990, judge 0.950

Jacobian of each layer at test image 0: singular values below 1e-3 / 1e-6 of the largest (of 3072)
  trained  relu1_2:     0 /     0
  trained  relu2_2:  1893 /  1341
  trained  relu3_2:  2868 /  2183
  random   relu1_2:     0 /     0
  random   relu2_2:  1763 /  1069
  random   relu3_2:  2330 /  2025

network  target   layers      loss   RMSE blurRMSE colourW1 HFratio judge
-        noise    -              -  0.400    0.258    0.267    6.61  0.25
-        original -              -  0.000    0.000    0.000    1.00  1.00
trained  feature  relu1_2   0.0024  0.118    0.098    0.075    1.52  0.75
trained  feature  relu2_2   0.0001  0.219    0.154    0.138    3.28  0.50
trained  feature  relu3_2   0.0005  0.329    0.218    0.217    5.01  0.38
trained  gram     1..1      0.0004  0.285    0.201    0.170    3.70  0.38
trained  gram     1..2      0.0038  0.276    0.193    0.159    3.47  0.38
trained  gram     1..3      0.0054  0.262    0.177    0.154    3.47  0.50
random   feature  relu1_2   0.2543  0.209    0.101    0.101    3.89  0.62
random   feature  relu2_2   0.0943  0.225    0.074    0.126    5.23  0.88
random   feature  relu3_2   0.0396  0.281    0.093    0.171    5.92  0.62
random   gram     1..1      0.0070  0.329    0.175    0.174    5.53  0.25
random   gram     1..2      0.2193  0.349    0.190    0.193    5.82  0.25
random   gram     1..3      0.1469  0.342    0.172    0.196    5.94  0.25

The Jacobian counts show what can be recovered at all, at the one image where they are computed. At relu1_2 no input direction is invisible to either network, so the image is determined by its features, at least locally. After one pooling step, more than a thousand of the 3,072 directions have singular values below $10^{-6}$ of the largest, and at relu3_2 more than two thousand: whole families of images share the same features, even though relu2_2 still has more outputs than the image has pixels; in our reading, inactive ReLUs and pooling discard the information.

At this budget the reconstruction from relu1_2 is not exact, although the printed count shows that layer to be invertible: its pixel error is far below the noise start’s but not zero, and its high-frequency ratio is above 1. That residual comes, in our reading, from the optimizer stopping short rather than from lost information. The differences between layers below therefore mix two things, the optimizer’s shortfall, which need not be the same at every layer, and what each layer discards; the example cannot separate them. With that caveat, the feature reconstructions from the trained network go the way of the count. The pixel error and the colour distance grow with depth, and the error after blurring grows less than the raw error, so coarse layout survives better than detail. The high-frequency ratio climbs towards that of the noise start: in our reading, from deep layers the optimizer has no reason to remove the starting noise in the many directions the count shows the layer does not see, which is the high-frequency artefact Mahendran and Vedaldi suppressed with an image prior and Dosovitskiy and Brox with an adversarial loss. The judge recognizes fewer deep reconstructions, but with eight images each step of 0.125 is one image, so these fractions are coarse.

The Gram reconstructions match their targets closely (losses of 0.006 or less) yet stay far from the original in pixels, with blurred errors between those of the relu2_2 and relu3_2 feature reconstructions: matching channel co-activations does not fix where the shape is, and the judge finds the right shape in only three or four of the eight, while the colour distance is lower than for relu3_2 features. The untrained control behaves differently. Its feature losses stay much higher at the same budget, so its reconstructions carry more of the optimizer’s shortfall; yet from relu2_2 and relu3_2 they keep coarse layout better than the trained network’s and the judge recognizes more of them, while at relu1_2 the two networks are close and the judge recognizes fewer of the control’s. The printed count gives the random network’s deeper layers fewer blind directions than the trained network’s; in our reading, the trained classifier has learned to discard what its task does not need, such as colour and position, which random filters keep. Its Gram reconstructions have among the largest pixel errors in the table, high-frequency ratios near the noise start and the judge at chance, in the direction of Gatys et al.’s finding that random weights fail for texture synthesis, though on a toy scale; for the Gram matrices of two and three layers the control’s loss also stays far above the trained network’s, so the optimizer’s shortfall is part of that failure too. One seed per network and eight images make all of these differences indicative only.

Common Misconceptions

• “A perceptual loss measures what people see.” The losses of Gatys et al. and Johnson et al. use features of a classification network with no calibration to human judgments; the name states the motivation. LPIPS was the step that measured and fitted the agreement, and even LPIPS was calibrated on small patches.
• “Matching the features means matching the image.” Only at low layers. A layer accepts every image with the same activations, and from deep layers that set includes images with different colours, textures, exact shapes and, without a prior, high-frequency noise; the example above shows it on a small scale.
• “The Gram matrix is the covariance of the features.” It is the uncentred second moment: the channel means are part of it. Gatys et al. (2015) found that the 1,024 channel means of five layers already produced interesting textures, so the means carry part of the style.
• “Higher layers are always better.” The best layer depends on the task: relu2_2 for Johnson et al.’s super-resolution, because higher layers distorted more; conv4_2 for Gatys et al.’s content, because it lets the style change the fine structure; SRGAN could not determine a significantly best loss by mean opinion score on Set5.
• “A perceptual loss replaces the pixel loss and the adversarial loss.” In most systems it does not. In Dosovitskiy and Brox’s ablation (Fig. 1), image and feature losses without the adversarial term gave noisy reconstructions, and image and adversarial losses without the feature term gave sharp images of objects that were not in the original (their introduction describes these two variants as the feature and adversarial losses “alone”, while the figure labels both with the image loss); SRGAN, ESRGAN (with an $L_1$ term), LaMa and the LDM autoencoder all combine it with other terms.
• “It must be VGG on ImageNet.” VGG is the common default, but AlexNet, unsupervised networks, a segmentation ResNet-50 and LPIPS have all served as loss networks, and LaMa’s ablation favoured the segmentation network.

Where It Is Used

• Super-resolution. Johnson et al.’s ×4 and ×8 networks with a relu2_2 loss, SRGAN with a VGG-19 loss and an adversarial term, ESRGAN with features before activation; these methods sit on the perception side of the trade-off between perceived quality and distortion that the super-resolution article describes.
• Style transfer. Gatys et al.’s optimization combines a feature loss for content with Gram losses for style, and Johnson et al.’s feed-forward networks learn to minimize the same objective; neural style transfer as a task, with its later normalization-based methods, builds on these losses.
• Autoencoders and generative first stages. Dosovitskiy and Brox trained networks to invert AlexNet features and a modified variational autoencoder with DeePSiM, obtaining sharper samples than with a squared-error loss; VQGAN and the autoencoder of latent diffusion, and therefore Stable Diffusion, use an LPIPS loss with a patch adversarial loss.
• Image inpainting. LaMa’s high-receptive-field perceptual loss supervises large-hole completion, where pixel losses average plausible fills; the authors call the loss crucial to their system, as their ablation shows (Table 3).
• Understanding networks. Mahendran and Vedaldi used feature reconstruction to ask what each layer of a network retains, an early tool of neural network interpretability, and Gatys et al. (2015) used Gram-matrix synthesis to show what each layer’s statistics capture.
• Evaluation. The same feature distances, calibrated, are the LPIPS metric, reported for super-resolution, inpainting and view synthesis.