Image Quality Assessment: From Error Visibility to Structural Similarity

“Image Quality Assessment: From Error Visibility to Structural Similarity” is a 13-page paper by Zhou Wang and Eero P. Simoncelli of the Howard Hughes Medical Institute and New York University, and Alan C. Bovik and Hamid R. Sheikh of the Laboratory for Image and Video Engineering (LIVE) at the University of Texas at Austin. It appeared in IEEE Transactions on Image Processing in April 2004 (volume 13, number 4, pages 600–612), after submission in January 2003 and revision in August 2003. The paper argued that the standard way of building perceptual quality metrics, by predicting how visible the errors between a distorted image and its reference are, rests on assumptions that do not hold for natural images. In its place, it proposed measuring the loss of structural information. It defined the structural similarity (SSIM) index as one instance of this idea and showed that its mean over an image tracked human opinion scores of 344 JPEG and JPEG2000 images more closely than PSNR, the Sarnoff JND model and the authors’ earlier universal quality index.

The paper received the IEEE Signal Processing Society’s Best Paper Award for 2009 and its Sustained Impact Paper Award for 2016. In 2015 the four authors received an Engineering Emmy Award from the Television Academy for the SSIM video quality measurement model. This article covers the paper as published: the problem as the authors framed it, the index and the tests as printed, the released code, and what came after. The SSIM article covers the index’s formula, implementations and behaviour as they are understood today.

Problem

By 2003 the mean squared error and PSNR were the most widely used full-reference quality measures, despite being poorly matched to perceived quality. Most perceptual alternatives modified the MSE so that each error counted according to its visibility. The paper summarizes this error-sensitivity framework as a pipeline of five stages (Fig. 1, §II-A):

• Pre-processing: align and scale the images, and optionally convert colour spaces, map pixel values to display luminance, simulate the eye’s optics and light adaptation.
• Contrast sensitivity function (CSF) filtering: weight spatial frequencies by the visual system’s sensitivity to them.
• Channel decomposition: split the image into subbands tuned to frequency and orientation, from cortex-like transforms to the DCT or wavelets.
• Error normalization: divide the error in each channel by a visibility threshold that depends on the neighbouring signal energy (masking), converting it to units of just noticeable difference.
• Error pooling: combine the normalized errors $e_{l,k}$ over channels $l$ and coefficients $k$, usually with a Minkowski norm $\big(\sum_l \sum_k |e_{l,k}|^\beta\big)^{1/\beta}$ with $\beta$ between 1 and 4 (eq. 1).

The authors then listed five problems with this bottom-up approach (§II-B):

• Quality definition. Visible errors are not necessarily objectionable: multiplying all intensities by a constant is clearly visible but changes little of what the image shows.
• Suprathreshold. CSFs and masking models come from experiments at the threshold of visibility, while most image processing produces errors well above it.
• Natural image complexity. The psychophysics uses spots, bars and gratings, a few patterns at a time; natural images superimpose many.
• Decorrelation. Minkowski pooling implicitly treats errors at different locations as independent, but the coefficients of linear transforms such as wavelets remain strongly dependent for natural images.
• Cognitive interaction. Instructions, prior knowledge of the content, attention and fixations change quality judgments, and the models ignore them.

Two earlier publications had stated the alternative: Wang’s 2001 PhD thesis and a 2002 ICASSP paper by Wang, Bovik and Lu, both cited in §III-A. Wang and Bovik’s universal image quality index (UQI, 2002) had implemented it; as the 2004 paper describes it, UQI is the product of luminance, contrast and correlation terms without stabilizing constants, computed in $8 \times 8$ windows.

Contribution

• A top-down philosophy. The paper’s premise is that the visual system is highly adapted to extract structural information from a scene, so a measure of structural change should approximate perceived distortion (§III-A). Instead of accumulating the visibility of errors, a metric should judge how much of the scene’s structure survives. The authors argued that this sidesteps the suprathreshold problem, since it does not rely on threshold psychophysics, reduces the cognitive-interaction problem to a degree, and avoids the natural-image and decorrelation problems to some extent, since it compares complex signals directly rather than summing errors on simple patterns.
• A definition of structure. Because the luminance of a surface is the product of illumination and reflectance, while the objects’ structure does not depend on the illumination, the paper defines structural information as what remains in a local patch once its mean luminance and contrast are removed (§III-B).
• The SSIM index. A separable comparison of luminance, contrast and structure, combined into one index with stabilizing constants, which generalizes UQI (eqs. 5–13).
• A geometric reading. In the space of image components, the equal-distortion contours of a Minkowski error are circles of fixed size; component weighting makes them ellipses, and contrast masking scales them with the signal. SSIM compares vector lengths and angles separately, so its contours align with polar coordinates around each reference and both their size and shape adapt to the signal (Fig. 4).
• Best-case and worst-case validation. A method for probing any differentiable quality measure: search for the images that it scores best and worst among all images at a fixed MSE from the reference, and look at them (§IV-A).
• A subjective test on a database of JPEG and JPEG2000 images, evaluated with the protocol of the Video Quality Experts Group (VQEG).

Method

Definition. For two aligned, non-negative patches $x$ and $y$, the paper defines a luminance comparison $l$ from their means (eq. 6), a contrast comparison $c$ from their standard deviations (eq. 9) and a structure comparison $s$, the correlation coefficient of the patches with a small constant added (eq. 10). It requires the combined similarity to be symmetric, bounded by 1, and equal to 1 only for identical patches. The general index is $[l]^\alpha [c]^\beta [s]^\gamma$ (eq. 12); setting $\alpha = \beta = \gamma = 1$ and $C_3 = C_2/2$ gives the two-factor form in common use (eq. 13). The SSIM article gives the formulas and works through their behaviour.

Perceptual readings. The authors related the terms to known properties of vision. With $C_1$ negligible, $l$ depends only on the relative change of luminance, which they called qualitatively consistent with Weber’s law (eq. 8). For a given change of contrast, $c$ is less sensitive when the base contrast is high, consistent with contrast masking.

Constants, window and pooling. The paper fixed $K_1 = 0.01$ and $K_2 = 0.03$ in the stabilizing constants for its whole evaluation (eq. 7, §III-C). It applied the index locally, for four stated reasons: image statistics vary across space, so do distortions, only a local region is seen at high resolution at a time, and a map of local quality is useful in itself. UQI’s $8 \times 8$ square window, moved pixel by pixel, gave maps with blocking artifacts, so the paper used an $11 \times 11$ Gaussian weighting with a standard deviation of 1.5 samples, which made the maps locally isotropic (eqs. 14–16). The image score is the uniformly weighted mean of the local indices, the mean SSIM or MSSIM (eq. 17). The SSIM article gives the constants, window and pooling as formulas, with the authors’ remarks on them.

Test conditions. In the database experiment, the MSSIM was computed on the luminance component only, after averaging $2 \times 2$ blocks and downsampling by 2 (§IV-B). The authors found that the other colour components did not change the results significantly on this dataset, and cautioned that this should not be assumed for colour images in general. No parameter was trained on the database, since the index was meant for general-purpose quality assessment rather than for compression alone.

Results

Equal-MSE images (Fig. 2). The motivating example alters the 8-bit “Boat” image in five ways, each adjusted to an MSE of 210. The MSSIM values in the caption are 0.9900 for a mean shift, 0.9168 for a contrast stretch, 0.7748 for salt-and-pepper noise, 0.7052 for blur and 0.6949 for JPEG compression. The authors’ explanation was that the stretched and shifted images keep nearly all the structural information, which a pointwise linear luminance transform could recover except where intensities saturate, while compression and blur lose some of it permanently. They acknowledged that the contrast-stretched image is easily told apart from the reference. The mean squared error article describes the later, larger version of this demonstration in Wang and Bovik (2009).

Best and worst cases (Fig. 5). Starting from a distorted image $Y$ of the reference $X$, the method alternates a step along the gradient of the MSSIM, projected so that it is tangent to the sphere of constant MSE around $X$, with a rescaling of the error $Y - X$ back onto that sphere. Ascending gives a local maximum of the MSSIM and descending a local minimum, all at the same MSE. On a $100 \times 100$ crop of “Boat” with Gaussian white noise at an MSE of 2500, the caption gives MSSIM values of 0.3021 for the noisy start, 0.9337 for the best case and −0.5411 for the worst. In the best case the local structures of the original are well preserved; in the worst they are changed dramatically, in places with reversed contrast.

The database (§IV-B). Twenty-nine high-resolution 24-bit RGB images, typically $768 \times 512$, were compressed into 175 JPEG images (0.150 to 3.336 bits per pixel) and 169 JPEG2000 images (0.028 to 3.150 bits per pixel), with rates chosen so that subjective quality was roughly uniform over its range. Observers, mostly male college students, sat at a comfortable and only moderately controlled distance and rated each image on a continuous scale divided into five regions labelled Bad to Excellent; each JPEG image was rated by 13 to 20 subjects and each JPEG2000 image by 25. Raw scores were converted to z-scores per subject, rescaled to 1–100, and averaged into mean opinion scores (MOS) after outlier removal. The mean per-image standard deviations of the scores were 6.00 for JPEG, 7.33 for JPEG2000 and 6.65 overall. The images and scores were published on the LIVE website, and this set became known as the LIVE JPEG/JPEG2000 database (the name the MS-SSIM paper uses); the image quality assessment article describes the later LIVE releases and how such databases are built.

Maps (Figs. 6 and 7). Sample JPEG and JPEG2000 images with their SSIM maps and absolute-error maps show two effects. At low bit rates, fine detail such as roof tiles and trees is reproduced as smooth regions; their absolute errors are not larger than elsewhere, but the SSIM map marks them as poor, matching their appearance. In the other direction, for an image dominated by strong edges (“Bikes”), a JPEG2000 version at 1.1454 bits per pixel has a lower PSNR (33.47 dB) than a version of another image at 0.1982 bits per pixel (34.56 dB), yet looks better; its MSSIM is higher, 0.9747 against 0.9409.

Scatter plots and Table I. The comparison covers four models: PSNR, the Sarnoff model (JNDmetrix software, version 8.0 in the IEEE version’s Fig. 8 caption), UQI and MSSIM. Following VQEG’s Phase I full-reference television test, each model’s scores were mapped to MOS with a fitted logistic function before computing correlation and error measures; a second set of measures uses VQEG’s variance-weighted regression; the Spearman rank-order correlation measures monotonicity, and the outlier ratio is the fraction of predictions more than two standard deviations from the MOS. Table I of the paper:

Model	CC (nonlinear)	MAE	RMS	OR	CC (variance-weighted)	WMAE	WRMS	OR	SROCC
PSNR	0.905	6.53	8.45	0.157	0.903	6.18	8.26	0.140	0.901
Sarnoff	0.956	4.66	5.81	0.064	0.956	4.42	5.62	0.061	0.947
UQI	0.866	7.76	9.90	0.189	0.861	7.64	9.79	0.195	0.863
MSSIM	0.967	3.95	5.06	0.041	0.967	3.79	4.87	0.041	0.963

MSSIM is best on every criterion, as the text states. The margin over the Sarnoff model is small on the correlations (0.967 against 0.956 for the nonlinear CC, 0.963 against 0.947 for SROCC) and larger on the outlier ratio (0.041 against 0.064). Taking PSNR as the benchmark, the authors noted that the Sarnoff model performed quite well in this test, unlike earlier tests, including VQEG’s, in which most models were statistically equivalent to PSNR or RMSE. UQI scored below PSNR, which the authors attributed to its instability in nearly flat regions, the problem the constants $C_1$ and $C_2$ remove. The paper reports no significance tests for these differences; see statistical significance in evaluation for why that matters with 344 images.

Worked Example: Best and Worst Cases on a Synthetic Image

The example reimplements the search of §IV-A in NumPy and SciPy. The reference is a synthetic $100 \times 100$ grayscale image in $[0, 255]$ with a ramp, a bright disk, a dark bar and square-wave stripes; MSSIM uses the paper’s $11 \times 11$ Gaussian window with $\sigma = 1.5$, $K_1 = 0.01$, $K_2 = 0.03$, $L = 255$ and valid windows. The gradient of the MSSIM with respect to the distorted image is computed analytically through the local statistics and checked against finite differences. The search follows the paper’s two steps; the adaptive step size is our addition, since the paper does not give its step rule. Each run starts from Gaussian noise scaled to an MSE of exactly 2500, as in Fig. 5, for five noise seeds.

import numpy as np
from scipy.signal import fftconvolve

r = np.arange(11) - 5                                   # 11x11 Gaussian window, sigma 1.5
w = np.exp(-(r[:, None] ** 2 + r[None, :] ** 2) / (2 * 1.5 ** 2)); w /= w.sum()
C1, C2 = (0.01 * 255) ** 2, (0.03 * 255) ** 2           # K1 = 0.01, K2 = 0.03, L = 255
f = lambda a: fftconvolve(a, w, mode="valid")           # weighted local means (eqs. 14-16)

def mssim(x, y, grad=False):
    """Mean SSIM over valid windows (eqs. 13 and 17) and its gradient with respect to y."""
    mx, my, Exx, Eyy, Exy = f(x), f(y), f(x * x), f(y * y), f(x * y)
    A1, A2 = 2 * mx * my + C1, 2 * (Exy - mx * my) + C2
    B1, B2 = mx ** 2 + my ** 2 + C1, (Exx - mx ** 2) + (Eyy - my ** 2) + C2
    S = A1 * A2 / (B1 * B2)
    if not grad:
        return S.mean()
    dmy = S * (2 * mx / A1 - 2 * my / B1 - 2 * mx / A2 + 2 * my / B2)
    dExy, dEyy = 2 * S / A2, -S / B2
    back = lambda g: fftconvolve(g, w, mode="full")     # adjoint of the valid filtering
    return S.mean(), (back(dmy) + back(dExy) * x + back(dEyy) * 2 * y) / S.size

# A 100 x 100 synthetic reference in [0, 255]: a ramp, a bright disk, a dark bar, stripes.
yy, xx = np.mgrid[0:100, 0:100].astype(float)
X = 60 + 1.2 * xx
X[(yy - 35) ** 2 + (xx - 30) ** 2 < 18 ** 2] = 220
X[60:90, 10:45] = 30
X[55:95, 55:95] = 128 + 60 * np.sign(np.sin(2 * np.pi * xx[55:95, 55:95] / 8))

# Gradient check against central finite differences at a few pixels.
rng = np.random.default_rng(0)
Y0 = X + 50 * rng.standard_normal(X.shape)
_, g = mssim(X, Y0, grad=True)
for (i, j) in [(10, 10), (35, 48), (70, 70)]:
    E = np.zeros_like(X); E[i, j] = 1e-3
    fd = (mssim(X, Y0 + E) - mssim(X, Y0 - E)) / 2e-3
    print(f"pixel ({i},{j}): analytic {g[i, j]: .3e}  finite difference {fd: .3e}")

def search(X, Y, sign, mse=2500.0, iters=1500):
    """Section IV-A: step along the gradient projected onto the equal-MSE sphere, then
    rescale the error back to the sphere (sign +1 ascent, -1 descent). The step size
    lam grows after a successful step and halves after a failed one (our addition)."""
    s, lam = np.sqrt(mse * X.size), 1e6                 # s: norm of the error at this MSE
    v, g = mssim(X, Y, grad=True)
    for _ in range(iters):
        e = (Y - X) / np.linalg.norm(Y - X)
        Yn = Y + sign * lam * (g - e * np.sum(e * g))   # (1) Y + lam * P(X, Y) grad
        Yn = X + s * (Yn - X) / np.linalg.norm(Yn - X)  # (2) back to the MSE sphere
        vn, gn = mssim(X, Yn, grad=True)
        if sign * (vn - v) > 0:
            Y, v, g, lam = Yn, vn, gn, lam * 1.2
        else:
            lam /= 2
    return Y

def structure(x, y):  # mean of the structure term s (eq. 10 with C3 = C2/2) over windows
    mx, my = f(x), f(y)
    vx, vy, cxy = f(x * x) - mx ** 2, f(y * y) - my ** 2, f(x * y) - mx * my
    return np.mean((cxy + C2 / 2) / (np.sqrt(np.maximum(vx * vy, 0)) + C2 / 2))

edge = np.ones(X.shape, bool); edge[10:90, 10:90] = False  # pixels in fewer than 121 windows
print(f"\n{'seed':>4} {'MSE':>7} {'initial':>8} {'maximum':>8} {'minimum':>8} "
      f"{'s at min':>9} {'out of [0,255]':>15}")
for seed in range(5):
    rng = np.random.default_rng(seed)
    Y0 = X + 50 * rng.standard_normal(X.shape)                          # Gaussian noise
    Y0 = X + np.sqrt(2500 * X.size) * (Y0 - X) / np.linalg.norm(Y0 - X)  # MSE exactly 2500
    Yhi, Ylo = search(X, Y0, +1), search(X, Y0, -1)
    out = np.mean((Ylo < 0) | (Ylo > 255) | (Yhi < 0) | (Yhi > 255))
    print(f"{seed:4d} {np.mean((Yhi - X) ** 2):7.1f} {mssim(X, Y0):8.4f} {mssim(X, Yhi):8.4f} "
          f"{mssim(X, Ylo):8.4f} {structure(X, Ylo):9.3f} {out:15.2f}")

# Where each image puts its squared error (last seed), as shares of the total.
flat = (yy - 35) ** 2 + (xx - 30) ** 2 < 12 ** 2      # interior of the bright disk
stripes = np.zeros_like(flat); stripes[57:93, 57:93] = True
print(f"\n{'share of error':14s} {'border':>7s} {'flat disk':>10s} {'stripes':>8s}")
print(f"{'(share of area)':14s} {edge.mean():7.3f} {flat.mean():10.3f} {stripes.mean():8.3f}")
for name, Y in [("initial", Y0), ("maximum", Yhi), ("minimum", Ylo)]:
    e2 = (Y - X) ** 2 / np.sum((Y - X) ** 2)
    print(f"{name:14s} {e2[edge].sum():7.3f} {e2[flat].sum():10.3f} {e2[stripes].sum():8.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3; two runs gave identical output):

pixel (10,10): analytic  3.709e-08  finite difference  3.709e-08
pixel (35,48): analytic -1.531e-06  finite difference -1.531e-06
pixel (70,70): analytic  2.258e-06  finite difference  2.258e-06

seed     MSE  initial  maximum  minimum  s at min  out of [0,255]
   0  2500.0   0.2405   0.8993  -0.2241    -0.046            0.06
   1  2500.0   0.2433   0.8901  -0.2240    -0.046            0.07
   2  2500.0   0.2420   0.9033  -0.2241    -0.046            0.06
   3  2500.0   0.2407   0.8941  -0.2242    -0.047            0.07
   4  2500.0   0.2448   0.9067  -0.2240    -0.046            0.06

share of error  border  flat disk  stripes
(share of area)   0.360      0.044    0.130
initial          0.368      0.046    0.134
maximum          0.897      0.000    0.000
minimum          0.184      0.014    0.407

The range at one MSE. All three images of each run have the same MSE, and the MSSIM spans from about −0.22 to about 0.90, across all five seeds, which differ by less than 0.02 in the maximum. The ordering matches Fig. 5, but the numbers do not, since the reference image and the convergence of the search differ; the paper does not give its step size or number of iterations.

What the measure ignores. The ascent moves about 90% of the error energy into the band within 10 pixels of the border, which holds 36% of the pixels, and removes it almost entirely from the flat disk and the stripes; the descent does the opposite and keeps error away from the border. Our explanation: with valid windows, a pixel in the interior lies in $121$ windows of the map but a corner pixel in only one, so the MSSIM of eq. 17 weights pixels unevenly and error placed near the border costs little. In a $100 \times 100$ image the band is large; the paper’s crop has the same size, though the paper does not say whether its search used valid windows. This is the kind of insight the method was designed to give: the best case shows which errors the measure deems unimportant, here including an artifact of pooling rather than of perception.

What the measure punishes. The descent concentrates error in the stripes, where it can invert local structure: the mean structure term at the minimum is slightly negative, and the MSSIM itself is negative, as in the paper’s worst case. The search also does not keep intensities in $[0, 255]$: at 6 to 7% of the pixel positions, the best or the worst image leaves the valid range. The paper does not mention a range constraint; displaying such images requires clipping, which changes their MSE (our observation).

Versions

Two copies of the paper are in circulation. The authors’ preprint, a TeX document produced on 15 March 2004 and hosted on both the New York University and University of Waterloo sites, carries the journal’s header but its own page numbers 1–14. The IEEE version, pages 600–612, adds the submission and revision dates and the editor’s name. A word-level comparison of the two text layers found the same section structure, equation numbers, numbers in the figure captions and Table I values; the differences are copy-editing and a few wording changes. The IEEE abstract calls the framework “alternative complementary” rather than “alternative”, the IEEE text speaks of “perceived changes in structural information variation”, and the IEEE Fig. 8 caption names the Sarnoff software version (JNDmetrix 8.0). Both copies contain the same slip in §IV-B, comparing Fig. 7(g) with Fig. 6(j) where Fig. 7(j) is evidently meant (our reading).

Before publication the paper circulated under a different title. The header of the authors’ MATLAB file ssim_index.m (Version 1.0, copyright 2003) and the reference list of the MS-SSIM paper cite it as “Image quality assessment: From error measurement to structural similarity”, in volume 13, January 2004.

Code

The paper points to a MATLAB implementation on the authors’ New York University page; as of October 2026 that page still exists, and the project’s current page is at the University of Waterloo, last updated in November 2014. Both serve the same ssim_index.m (identical files). It implements eq. 13 with the paper’s defaults ($K = [0.01, 0.03]$, an $11 \times 11$ Gaussian window with $\sigma = 1.5$, $L = 255$), computes the local statistics with MATLAB’s filter2 in “valid” mode, so the map is smaller than the image by the window size minus one, and returns the mean of the map. It does not downsample. The paper’s experiments did, and the project page recommends downsampling by a factor that grows with the image size before calling ssim_index.m; a second file, ssim.m (Version 1.0, copyright 2009), performs that step automatically, and its header notes that the paper downsampled. The SSIM article shows how much these choices, and those of later libraries, change the score.

As of October 2026 the licensing is inconsistent: the project page says the code can be downloaded for free, changed and used anywhere provided the paper and page are cited, while the header of both MATLAB files permits educational and research use only and requires the authors’ permission before the code is used as the basis of a commercial product.

Impact

SSIM is usually reported next to PSNR: image restoration, super-resolution, compression and view-synthesis papers report both, and SSIM-based losses are used to train networks (see SSIM, image denoising and neural rendering). The LIVE Image Quality Assessment Database page asks users of its Release 2, which adds blur, white-noise and JPEG2000 bit-error images to JPEG and JPEG2000 ones and reports DMOS instead of MOS, to cite this paper (as of October 2026), and the authors’ project page reports SSIM on LIVE and five other rated databases (see image quality assessment).

The IEEE Signal Processing Society named it a recipient of its Best Paper Award for 2009, announced in February 2010, and of the Sustained Impact Paper Award for 2016, which honors journal articles of broad interest with sustained impact over many years. The Television Academy’s 67th Engineering Emmy Awards, announced in September 2015 and presented that October, honored the four authors for the SSIM video quality measurement model; the press release describes SSIM as a standard tool in broadcast and post-production.

Limitations

The paper itself named several:

• Arbitrary constants. $K_1$ and $K_2$ were not fitted; the authors reported only that performance was fairly insensitive to them.
• Colour. Only luminance was used, and the authors warned that the finding that colour did not matter should not be generalized.
• Optimization. They suggested using SSIM for rate–distortion optimization in compression, but noted that eq. 12 is mathematically more cumbersome than the MSE.
• One instance of a philosophy. They presented the index as one implementation, from an image-formation point of view, and expected others to follow.

Others follow from the design and the tests (our reading unless attributed):

• A narrow validation. The quantitative test covers two compression distortions, one database of 29 contents built in the lab of two of the authors, three competing models and no significance tests. The equal-MSE and best/worst-case demonstrations use a single image, “Boat”.
• A single scale. The window size is fixed in pixels, and the experiments depended on a fixed $2 \times 2$ downsampling. Wang, Simoncelli and Bovik (2003) called the single-scale design a drawback, since the right scale depends on display resolution and viewing distance, and reported on the same database that single-scale SSIM varied with the scale at which it was computed.
• What counts as structure. The argument that a contrast stretch or mean shift preserves structure because it is invertible makes the index tolerant of changes that viewers see, as the paper acknowledges for Fig. 2. Later analyses found the converse failures too, such as very low scores for nearly invisible changes in dark or flat regions (Nilsson & Akenine-Möller, 2020; see SSIM for the examples).
• Not a distance. The index is not a metric in the mathematical sense; Brunet, Vrscay and Wang (2012) derived related quantities that are.
• Uneven pooling. With valid windows, border pixels enter fewer windows than interior ones, which the example above shows the best-case search exploiting.

What Came After

• MS-SSIM. Wang, Simoncelli and Bovik’s multi-scale SSIM, presented at the Asilomar conference in November 2003 before the journal paper appeared, compares contrast and structure at five scales with weights calibrated in a perceptual experiment. On the same 344 images it reported a CC of 0.969 and an SROCC of 0.966. Its PSNR row and the rest of its Sarnoff row match Table I, but its Sarnoff outlier ratio is 3.20% against 0.064 in Table I. Its single-scale SSIM at scale 2, which applies the same $2 \times 2$ averaging and downsampling as the journal’s test, has a CC of 0.963 and an SROCC of 0.959, below Table I’s MSSIM (0.967 and 0.963). Neither paper explains these differences. The SSIM article gives its definition and weights.
• Analysis. Wang and Bovik (2009) restated the case against the MSE in a tutorial paper (see mean squared error); Brunet, Vrscay and Wang (2012) characterized SSIM as a normalized form of the MSE and studied its metric properties; Nilsson and Akenine-Möller (2020) collected its failure cases and implementation pitfalls.
• Optimization. The paper’s first open problem was taken up in video coding, for example by rate–distortion optimization built around SSIM (Wang, Rehman, Wang, Ma & Gao, 2012), and SSIM and MS-SSIM became training losses for restoration networks.
• Deep-feature metrics. Zhang et al. (2018) found that, averaged over their six two-alternative test sets, SSIM agreed with human choices no more often than the $\ell_2$ distance, and proposed LPIPS, which compares deep-network features instead (see the LPIPS paper). Ding, Ma, Wang and Simoncelli (2022), with two of the SSIM authors, returned to the structure idea in DISTS, which combines structure and texture similarity of deep features.