The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

“The Unreasonable Effectiveness of Deep Features as a Perceptual Metric” is a paper by Richard Zhang, Phillip Isola and Alexei A. Efros of UC Berkeley (Isola also of OpenAI) and Eli Shechtman and Oliver Wang of Adobe Research. It was posted to arXiv on 11 January 2018, revised on 10 April 2018 as the camera-ready version, and published at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2018. The paper asked how well distances between the internal activations of convolutional networks, already used as “perceptual losses” for image synthesis, agree with human judgments of similarity. To answer it the authors collected the Berkeley-Adobe Perceptual Patch Similarity (BAPPS) dataset of human judgments on small image patches, found that features of networks trained for classification, for self-supervised tasks and even with an unsupervised initialization agreed with people far more often than PSNR or SSIM, and calibrated the networks on their data to obtain the Learned Perceptual Image Patch Similarity, LPIPS. This article covers the publication; the metric’s formula, conventions, library usage and failure cases are in the LPIPS article.

Problem

The paper opens with the familiar failure of per-pixel measures. The $\ell_2$ distance and PSNR treat pixels as independent, so a blur, which people notice at once, is a small $\ell_2$ change. The perceptually motivated metrics of the time, SSIM, MS-SSIM, FSIM and HDR-VDP, were hand-designed, and the authors argued that building such a metric is hard because human similarity depends on high-order structure, depends on context, and may not even satisfy the axioms of a distance. Fitting a function directly to human judgments might therefore not generalize, and the paper later reports a case where it did not.

Meanwhile a different notion of perceptual distance had spread through image synthesis. Neural style transfer, the perceptual losses of Johnson, Alahi and Fei-Fei (2016) for super-resolution and style transfer, and conditional image synthesis measured differences in the feature space of VGG (see loss function). The paper’s questions were whether these losses agree with people, whether they beat the traditional metrics, whether the architecture matters, whether the network must be trained on ImageNet classification, and whether it must be trained at all.

The existing test beds did not suit these questions. The full-reference image quality databases, LIVE, CSIQ, TID2008 and TID2013, held 25 to 30 reference images with 5 to 24 distortion types, at a few discrete levels except in LIVE, summarized as mean opinion scores (Table 1; see image quality assessment). The authors positioned their dataset as complementary: many more distortion types, including geometric distortions and the outputs of deep networks, on many more patches. Earlier work had trained a CNN to predict visual similarity (Kim and Lee) or used internal activations of VGG and AlexNet for quality assessment (Gao et al.’s DeepSim; Amirshahi et al.), and Berardino et al. had tested networks on a separate perceptual task; the paper set out to compare many architectures and training signals on a larger and more varied dataset.

Contribution

The camera-ready version lists four contributions:

• BAPPS, a perceptual similarity dataset with 484k human judgments, covering parameterized distortions and outputs of real algorithms, plus a second test of just noticeable differences (JND).
• The finding that deep features trained with supervised, self-supervised and unsupervised objectives model low-level perceptual similarity well and outperform the widely used metrics.
• A control showing that architecture alone does not explain this: untrained networks score much lower.
• Calibration: learning weights on a pretrained network’s features from the judgments improves agreement. The paper names the resulting family of trained metrics LPIPS.

The BAPPS Dataset

Distortions. Every 2AFC triplet consists of a $64 \times 64$ reference patch $x$ and two distorted versions $x_0, x_1$. The traditional distortions are photometric changes (lightness, colour, contrast, saturation), six kinds of noise and a checkerboard artifact, Gaussian and bilateral blur, spatial operations (shifts, affine, homography, linear and cubic warps, ghosting, chromatic aberration) and JPEG compression, with continuously sampled severity; 20 basic distortions and 308 sequential pairs of them (Table 2, §2.1). The CNN-based distortions come from 96 small “denoising autoencoders”, each trained for one epoch on the 1.3 million ImageNet images on a task such as autoencoding, denoising, colorization or super-resolution, with randomly varied input corruption, generator architecture, discriminator depth, loss weights ($\ell_1$, VGG and adversarial) and learning rate. The authors stress that these networks were not meant to solve their tasks, only to produce the kinds of artifacts deep networks make. Table 1 counts 425 distortion types; the text’s numbers add up to 424 (our arithmetic).

Real algorithms. Four test sets hold outputs of published methods, compared with the ground truth:

• Super-resolution: the three NTIRE 2017 tracks with “unknown” downsampling at $\times 2$, $\times 3$ and $\times 4$, about 20 submissions each, plus bicubic upsampling and four deep methods (VDSR, a sparse-prior network, SRGAN and EnhanceNet), with patches sampled from DIV2K images.
• Frame interpolation: three variants of flow-based interpolation, adaptive separable convolution and phase-based interpolation on the Middlebury data, with images randomly rescaled before sampling since the artifacts occur at different scales.
• Video deblurring: Photoshop Shake Reduction, Weighted Fourier Aggregation and three variants of a deep video deblurring network, on that network’s dataset.
• Colorization: pix2pix, Larsson et al. and variants of Zhang et al.’s colorization network, on ImageNet images at random scales.

Collection. Judgments were gathered on Amazon Mechanical Turk, with an average of about 3 seconds per judgment. Training triplets, from the MIT-Adobe 5k images, received 2 judgments each; validation triplets, from RAISE1k, received 5. Sentinels with obvious answers (much more Gaussian noise against a little) checked attention: about 90% of workers passed at least 14 of 15. The patch size was a deliberate choice: the space of full images is too large to cover, small patches limit the influence of semantics on which “respect of similarity” a judge attends to, and synthesis networks are trained with patch-based losses (§2.2).

JND test. Because a 2AFC judge can choose which aspect of similarity to attend to, the authors added a test with a correct answer: a reference and a distorted patch were flashed for 1 second each, 250 ms apart, and the observer said whether they were the same. Each worker saw 160 pairs and 40 sentinels (32 identical, 8 heavily noised), after a 10-pair training period with 4 identical pairs, chosen to prime workers to expect about 40% identical pairs; 36.4% of all responses were “same” (70.4% of sentinels, 27.9% of test pairs). Each pair received 3 judgments.

The splits, from Table 3 (the table prints the frame interpolation, deblurring and colorization counts without “k”, and labels the two RAISE1k 2AFC rows “Train” although their total row and the text make them validation sets):

Set	Source images	Split	Triplets or pairs	Judgments each
2AFC traditional, CNN-based, mixed	MIT-Adobe 5k	Train	56.6k, 38.1k, 56.6k (151.4k)	2
2AFC traditional, CNN-based	RAISE1k	Val	4.7k, 4.7k	5
2AFC super-resolution	DIV2K	Val	10.9k	5
2AFC frame interpolation	Middlebury	Val	1.9k	5
2AFC video deblurring	deblurring dataset	Val	9.4k	5
2AFC colorization	ImageNet	Val	4.7k	5
JND traditional, CNN-based	RAISE1k	Val	4.8k, 4.8k	3

The six 2AFC validation sets serve as the test sets throughout. By our arithmetic the 484k judgments of the contributions list are the 2AFC judgments alone (349.8k on distortions and 134.5k on real algorithms, Table 1); with the 28.8k JND judgments the dataset holds about 513k.

Method

Distances are computed by Eq. 1, a channel-weighted distance between unit-normalized activations of several layers that reduces to a cosine distance when all weights are 1 (§3). The networks are SqueezeNet (first convolution and fire modules), the single-tower AlexNet (conv1–conv5) and five convolutional layers of VGG, plus publicly available self-supervised networks (puzzle solving, cross-channel prediction or Split-Brain, learning from video, BiGAN and others), the stacked k-means initialization of Krähenbühl et al., and a network with Gaussian random weights. Three trained variants form LPIPS: lin, which learns only $w$ on a frozen network (1,472 parameters for VGG); tune, which fine-tunes the whole network from its classification weights; and scratch, which trains it from a random initialization. The LPIPS article gives the formula, the small network and loss used in training, and the weight constraint in full.

Results

A metric’s 2AFC score is the fraction of human judgments it agrees with, and the human ceiling is the expected score of a judge from the same population (§4; both are defined in the LPIPS article). The proceedings version shows the results as bar charts (Figure 4) and refers to an appendix for the numbers; the appendix is in the arXiv versions, where Table 5 (identical in v1 and v2) gives them, in percent:

Method	Trad.	CNN	Dist. avg	Super-res	Deblur	Color	Interp.	Real avg	All
Human	80.8	84.4	82.6	73.4	67.1	68.8	68.6	69.5	73.9
$\ell_2$	59.9	77.8	68.9	64.7	58.2	63.5	55.0	60.3	63.2
SSIM	60.3	79.1	69.7	65.1	58.6	58.1	57.7	59.8	63.1
FSIMc	61.4	78.6	70.0	68.1	59.5	57.3	57.7	60.6	63.8
HDR-VDP	57.4	76.8	67.1	64.7	59.0	53.7	56.6	58.5	61.4
Random (Gaussian)	60.5	80.7	70.6	64.9	59.5	62.8	57.2	61.1	64.3
K-means	66.6	83.0	74.8	67.3	59.8	63.1	59.8	62.5	66.6
Watching	66.5	80.7	73.6	69.6	60.6	64.4	61.6	64.1	67.2
Split-Brain	69.5	81.4	75.5	69.6	59.3	64.3	61.1	63.6	67.5
Puzzle	71.5	82.0	76.8	70.2	60.2	62.8	61.8	63.8	68.1
BiGAN	69.8	83.0	76.4	70.7	60.5	63.7	62.5	64.4	68.4
SqueezeNet	73.3	82.6	78.0	70.1	60.1	63.6	62.0	64.0	68.6
AlexNet	70.6	83.1	76.8	71.7	60.7	65.0	62.7	65.0	68.9
VGG	70.1	81.3	75.7	69.0	59.0	60.2	62.1	62.6	67.0
Squeeze-lin	76.1	83.5	79.8	71.1	60.8	65.3	63.2	65.1	70.0
Alex-lin	73.9	83.4	78.7	71.5	61.2	65.3	63.2	65.3	69.8
VGG-lin	76.0	82.8	79.4	70.5	60.5	62.5	63.0	64.1	69.2
Squeeze-scratch	74.9	83.1	79.0	71.1	60.8	63.0	62.4	64.3	69.2
Alex-scratch	77.6	82.8	80.2	71.1	61.0	65.6	63.3	65.2	70.2
VGG-scratch	77.9	83.7	80.8	71.1	60.6	64.0	62.9	64.6	70.0
Squeeze-tune	76.7	83.2	79.9	70.4	61.1	63.2	63.2	64.5	69.6
Alex-tune	77.7	83.5	80.6	69.1	60.5	64.8	62.9	64.3	69.7
VGG-tune	79.3	83.5	81.4	69.8	60.5	63.4	62.3	64.0	69.8

The table greys out the LPIPS values on the two distortion sets, since those variants were trained on the same distortion types (on different images).

Classifiers against low-level metrics. Averaged over all six test sets, SqueezeNet, AlexNet and VGG features scored 68.6%, 68.9% and 67.0%, despite models of 2.8, 9.1 and 58.9 MB (convolutional layers only), against 63.2%, 63.1% and 63.8% for $\ell_2$, SSIM and FSIMc. The margin is largest on the traditional distortions, where FSIMc reached 61.4% and the classifiers 70.1% to 73.3%; Appendix C of the arXiv v2 attributes this, as likely, to the geometric distortions, which SSIM-type metrics were not designed for. On the real algorithms the margins shrink: on super-resolution FSIMc (68.1%) is 0.9 to 3.6 points below the networks, and VGG’s raw features (62.6%) beat $\ell_2$ by 2.3 points.

Training signal. The four self-supervised networks in the table scored 67.2% to 68.4% overall, on par with the classifiers, and the k-means initialization (66.6%) beat every low-level metric. A Gaussian-initialized network gained little (64.3%), from which the authors concluded that the structure of the network helps only together with filters oriented by data.

Correlation with semantic tasks. For AlexNet-like networks, the authors correlated scores across methods on the 2AFC and JND tests with their PASCAL classification and detection results as summarized by Split-Brain (Table 4): .640 and .363 for 2AFC, .612 and .232 for JND, and .429 between classification and detection. The text cites this as Table 5, a numbering slip present since v1, where it reads Table 6.

JND. JND was scored as the mean average precision (mAP) of ranking pairs by distance against the “same” responses. Across methods the two perceptual tests correlated at $\rho = .928$, which the authors took as evidence that the 2AFC judgments carry an objective signal. The JND results appear only as a scatter plot (Figure 5): in our reading of it, the low-level metrics lie at about 44% to 50% JND mAP, the calibrated lin metrics at about 59% to 61%, and the raw networks between.

Calibration and training. On the distortion sets, tuning the whole network scored highest (79.9%, 80.6% and 81.4%), above scratch and lin, and close to the human 82.6%; the arXiv appendix reads this closeness as a sign that the 150k training pairs (300k judgments) nearly suffice for these distortions, while noting a small gap between tune and scratch (0.4 and 0.6 points for AlexNet and VGG). On the real algorithms, linear calibration improved 11 of the 12 per-task scores of the three networks, by 1.1, 0.3 and 1.5 points on average for SqueezeNet, AlexNet and VGG; the exception, from the table, is AlexNet on super-resolution, which fell from 71.7% to 71.5%. Training from scratch was slightly below lin for AlexNet and slightly above for VGG. Fine-tuning a pretrained network lowered transfer to the real algorithms, which the authors present as a negative result: optimizing directly for the low-level judgments did worse than transferring a representation trained for a high-level task. By the table, Alex-tune’s 64.3% is below even the uncalibrated AlexNet’s 65.0%. Alex-lin’s 65.3% was the best average on the real algorithms, and it became the default metric.

Qualitative differences. Comparing BiGAN features with SSIM on the traditional distortions, pairs BiGAN judged far and SSIM close generally contained blur, while BiGAN judged correlated noise patterns a smaller change than SSIM did (Figure 6; the CNN-based counterpart is Figure 11 of the arXiv v2 appendix).

TID2013. Appendix C of the arXiv v2 (Figure 7 of the main text in v1) computed Spearman correlations on TID2013 at five resolutions. Uncalibrated AlexNet features, averaged over all scales and layers, came close to the best metric, FSIMc. From the bar chart, FSIMc reaches about 0.85, AlexNet about 0.81, MS-SSIM and FSIM about 0.78 to 0.80, SqueezeNet about 0.76, VGG and ResNet-50 about 0.71 to 0.72, and PSNR and SSIM about 0.64 (our reading). On this mean-opinion-score benchmark, without geometric distortions, the deep features did not beat the best hand-designed metric.

Claims and Evidence

The abstract states that deep features beat all previous metrics by large margins on the paper’s dataset and that the result holds across architectures and levels of supervision. The tables support the first part for the 2AFC averages; the TID2013 appendix shows that it depends on the distortions included. The second part rests on the best self-supervised networks, as the introduction says. Figure 5 also plots six further self-supervised networks (egomotion, inpainting, context prediction, sound, video and colorization), and in our reading of it the egomotion and inpainting networks score at or below $\ell_2$ and SSIM on both tests.

The broader claims are worded as hypotheses. Perceptual similarity is described as an “emergent property” consistent with representations tuned to predict structure in the world; the conclusion’s “a good feature is a good feature” links classification and detection strength to perceptual agreement, citing Table 4 as suggestive evidence, which reports correlations of .640 and .363 over AlexNet-like networks without stating how many networks entered them. The paper reports no confidence intervals: the table marks values within 0.5 points of the best, while the gains from calibration on the real algorithms are 0.3 to 1.5 points (see statistical significance in evaluation for the tests such comparisons need). The gap to the human ceiling on the real algorithms, 69.5% against 65.3% for the best metric, remains larger than any of these gains (our arithmetic).

Versions

The paper has two arXiv versions and the proceedings version. The arXiv v2 ends with a changelog (Appendix D); the other differences below come from our comparison of the texts.

Version	Date	Content
arXiv v1	11 Jan 2018	Initial preprint; nine pages of main text with the TID2013 results and the SSIM–BiGAN comparisons on both distortion types; Appendices A–B
arXiv v2	10 Apr 2018	CVPR camera-ready text; TID2013 results, the CNN-based SSIM–BiGAN figure and some training details moved to Appendices A–C to fit eight pages; Appendix D changelog
CVPR 2018	June 2018	Proceedings pages 586–595; main text essentially that of v2 (typesetting, reference numbering and a few words differ; the proceedings cite Figure 6 where v2’s text cites Figure 11); no appendix

The changelog lists the clarifications in v2: SSIM was not designed for geometric distortions, BAPPS is a perceptual similarity dataset rather than an image quality assessment dataset, and the learned weights of Squeeze-lin and VGG-lin were added to the weight plots, which in v1 showed only the AlexNet model.

The reframing goes further than the changelog says (our comparison). The v1 abstract introduced “a new Full Reference Image Quality Assessment (FR-IQA) dataset” that was “orders of magnitude larger” than its predecessors, and found that deep features beat earlier metrics “by huge margins”; v2 speaks of a perceptual similarity dataset and of “large margins on our dataset”. v1 justified collecting on Mechanical Turk by its realistic range of displays and viewing conditions; v2 cites instead a study showing that the platform can replicate psychophysics experiments. v1 commented that the strong results of trained variants on the distortion test sets were unsurprising because training and test distributions were the same; v2 drops that sentence, while keeping the greyed values and the note on them in Table 5. The numbers in Tables 1 to 5 are the same in both versions.

The arXiv v2 is the most complete version: the proceedings refer to an appendix for the per-test-set scores and training details, and the CVPR open-access page, as of October 2026, offers the paper and a link to arXiv but no supplementary material.

Code and Weights

The repository richzhang/PerceptualSimilarity was created on 11 January 2018, the day v1 appeared, with the evaluation code, the lin weights for the three networks and the BAPPS download scripts; training code followed on 7 June 2018. A commit of 26 July 2018 titled “bug fix” introduced version 0.1: new weights for all three networks, the old ones kept as version 0.0, and a code comment explaining that the original release did not rescale the input before the network. The current code’s documentation says that version 0.0 corresponds to arXiv v1. The README asks users to state the version they report and says that retraining with the released script should roughly reproduce the Alex-lin row of Table 5; it does not give recomputed scores for version 0.1, so the paper’s numbers and the default weights may differ slightly (our observation).

The metric became a pip package, lpips, from version 0.1 on 6 September 2020; as of October 2026 the latest release is 0.1.4 of 25 August 2021, and the latest commit is of 19 December 2023. In the evaluation script at the pinned commit, the $\ell_2$ and SSIM baselines are computed in the Lab colour space by default; the paper does not state which colour space its baselines used (our reading). The LPIPS article covers the input range, spatial maps and other conventions of the package.

Impact

LPIPS became a standard learned full-reference metric for generated and restored images. It is reported beside PSNR and SSIM in novel view synthesis, for example in the NeRF paper and 3D Gaussian splatting, in super-resolution and image inpainting; it measures the diversity of outputs in multimodal image-to-image translation, and StyleGAN’s perceptual path length averages it along latent interpolations. In 2023 the DreamSim paper, which Zhang and Isola co-authored, still called LPIPS the predominant perceptual metric.

BAPPS became a benchmark in its own right. DISTS, E-LPIPS and ST-LPIPS report their 2AFC scores on it, Kumar et al. (2022) used it to study ImageNet classifiers, and DreamSim reports agreement on it. The paired-comparison idea had a concurrent counterpart at the same conference: PieAPP (Prashnani et al., 2018) collected probabilities that people prefer one distorted image over another and trained a network to predict a perceptual error from them.

In our reading, the paper also changed how VGG perceptual losses are understood: its finding that features from very different networks and training signals give similar agreement suggested that their usefulness is not specific to VGG or to classification (see self-supervised learning and transfer learning).

Limitations

The paper’s design limits what its numbers show:

• Patches and low-level similarity. Judgments concern $64 \times 64$ patches, chosen to reduce semantic influence; the dataset says nothing about whole-image composition or semantic correctness.
• Uncontrolled viewing. Judgments were collected online with sentinels as the only quality control, and training triplets have 2 judgments each, so the labels are noisy.
• Narrow real-algorithm tests. Four tasks, chosen from methods of 2017 and earlier, carry the paper’s main transfer test, and on them the best metric remains about 4 points below the human ceiling.
• Trained and tested on the same distortion types. The trained variants’ distortion scores are not comparable with the others’, as the paper’s own greying signals.
• No uncertainty estimates. Differences of a fraction of a point are ranked without confidence intervals.

Later work found further limits (details in the LPIPS article):

• Adversarial fragility. Kettunen, Härkönen and Lehtinen (2019) constructed image pairs that LPIPS rates close although they differ visibly, and the reverse.
• Shift sensitivity. Ghildyal and Liu (2022) showed that one-pixel shifts that people barely notice flip LPIPS’s 2AFC choices on a few percent of BAPPS triplets.
• Texture. Ding et al. (2022) designed DISTS to tolerate resampled texture that LPIPS, like other full-reference metrics, penalizes.
• Better classifiers are not better metrics. Kumar et al. (2022) trained ResNets, EfficientNets and Vision Transformers on ImageNet at $64 \times 64$ and found that past a point, higher ImageNet accuracy went with lower 2AFC scores on BAPPS, an inverse-U relationship that persisted with alternative distance functions. Shallow, early-stopped ResNets reached 70.2%, matching the best network trained on BAPPS itself. This qualifies the paper’s suggestion that stronger features are better models of similarity, which rested on AlexNet-like networks.
• Mid-level similarity. DreamSim’s NIGHTS dataset of synthetic image triplets that differ in layout, pose and content gave LPIPS (AlexNet-lin) 70.7% agreement with unanimous human votes on its test set, against 86.0% for DISTS and 96.2% for the tuned DreamSim ensemble (Fu et al., 2023, arXiv v3 Table 5). On the TID2013 and KADID-10k quality databases, LPIPS’s Spearman correlations of 0.838 and 0.883 were among the highest in the same paper (Table 8).

What Came After

Most successors, in our reading, kept the paper’s recipe, deep features calibrated or evaluated on human comparisons, and changed one ingredient. E-LPIPS, ST-LPIPS and DISTS each addressed one of the limits above, robustness, shift tolerance and texture; the LPIPS article compares them. DreamSim (2023), co-authored by Zhang and Isola, kept the 2AFC and JND protocols but moved them to whole synthetic images and to large pretrained vision transformers such as CLIP tuned on the judgments, targeting the mid-level similarity that BAPPS deliberately excluded. The paper’s broader claim, that perceptual similarity emerges in good representations, remains the reference point these studies test, and the image quality assessment article places LPIPS among the full-reference methods it influenced.