Image Quality Assessment

Image quality assessment (IQA) is the task of predicting how good an image looks to people. An IQA method takes a distorted image, and sometimes the pristine image it came from, and outputs a number that should agree with the scores human observers would give. The task exists because human judgments are the real target but are slow and expensive to collect: a compression codec, a camera pipeline or a denoising network needs a quality measure it can call thousands of times. The field therefore has two halves that this article connects. One is measurement: subjective experiments that turn viewers’ opinions into mean opinion scores, collected into public databases. The other is prediction: objective metrics, from PSNR and SSIM to learned networks, judged by how well they correlate with those scores. The main metrics have their own articles; this one compares them and explains how any of them is validated.

Problem Definition

The quantity to predict is perceived quality: the judgment a population of viewers would make of an image under stated viewing conditions. It is not the same as fidelity to a reference, although many methods measure fidelity and hope it tracks quality. Sheikh, Sabir and Bovik (2006) state the goal as agreement with human judgments regardless of the type of distortion, the image content or the strength of the distortion. All three clauses matter, because a metric can track human scores within one kind of distortion and fail when distortion types are mixed, as the TID2013 results and the example below show.

Two related questions are kept apart here. Aesthetic assessment asks whether a photograph is beautiful or well composed; Talebi and Milanfar (2018) separate this from technical quality, which concerns low-level degradations such as noise, blur and compression artefacts. Distribution-level evaluation of generative models asks whether a set of generated images resembles a set of real ones, which no single-image score answers; it is covered under Variants.

Input and Output

The input is a test image and, depending on the setting, more or less information about its undistorted original:

• Full reference (FR): the pristine reference image, pixel-aligned with the test image. Mean squared error, PSNR, SSIM and LPIPS are full-reference.
• Reduced reference (RR): a few features extracted from the reference, sent alongside the image. In Soundararajan and Bovik’s (2012) RRED indices, entropy differences of wavelet coefficients are computed from varying amounts of reference data, down to a single number per image.
• No reference (NR), or blind: the test image alone, as for NIQE. Hosu et al. (2020) call this the most challenging and the most practical of the three, since references are unavailable in many applications.

The output is usually a scalar, on whatever scale the method defines: decibels for PSNR, $[-1, 1]$ for SSIM, a distance for LPIPS or NIQE, or a predicted opinion score for methods trained on human ratings. Some methods also return a spatial quality map. The direction varies (higher is better for PSNR and SSIM, lower for LPIPS and NIQE), and so does the scale, which is why validation compares rankings or fits a mapping before comparing values.

Variants

• Synthetic versus authentic distortions. The classic databases apply controlled distortions (noise, blur, compression) at graded levels to a few dozen pristine images. Authentic databases such as KonIQ-10k and SPAQ collect photographs whose defects arose in capture and processing, with no reference at all. Hosu et al. (2020) report that every blind model they tested on both lost accuracy when trained on KonIQ-10k and tested on the authentic LIVE in the Wild database, and that conventional models lost much more than deep ones (BLIINDS-II fell from SROCC 0.585 on the KonIQ-10k test set to 0.090 and SSEQ from 0.604 to 0.245, while the deep models lost about 0.1; their Table 3 and §6.2.2).
• Opinion-aware and opinion-unaware NR methods. BRISQUE learns a mapping from natural-scene-statistics features to human scores; NIQE never sees a human score or a distorted image (see NIQE).
• Perceptual similarity. Instead of rating one image, observers choose which of two distorted versions is closer to a reference. Zhang et al.’s BAPPS dataset collected such two-alternative forced-choice (2AFC) judgments on 64 × 64 patches, and LPIPS was calibrated on them (see LPIPS for the 2AFC score and its human ceiling).
• Restoration outputs. Super-resolution and denoising networks, especially those trained with adversarial losses, produce plausible detail that differs from the reference. PIPAL (Gu et al., 2020) was built to test metrics on such outputs.
• Generative models. The Inception Score and the Fréchet Inception Distance score a model from a set of its samples, with no reference image for any of them. They answer a different question, whether the samples are recognizable, varied and distributed like real images, and are not validated against per-image opinion scores in the way IQA metrics are; image generation summarizes how weakly they tracked human judgment in one large study.
• Video. Video quality adds temporal artefacts. The Video Quality Experts Group (VQEG) ran the large validation studies of video metrics, whose analysis methods image IQA adopted (below).

Subjective Testing

Ground truth in IQA comes from experiments with human observers, and its form shapes everything downstream.

ITU-R BT.500. The reference methodology is ITU-R Recommendation BT.500, written for television and adopted for still images; the version in force as of October 2026 is BT.500-15 (May 2023). It asks for at least 15 observers unless a method states otherwise (fewer makes a study “informal”), sessions of no more than half an hour, observer screening and a 95% confidence interval for every mean score (Part 1, §§2.5–2.7, and Annex 1). Its main families of presentation are:

• Double-stimulus impairment scale (DSIS), the EBU method: the unimpaired reference is shown, then the impaired image, and the observer rates the impairment on a five-grade scale from 5, imperceptible, to 1, very annoying (Part 2, Annex 1).
• Double-stimulus continuous quality scale (DSCQS): the observer sees both versions without being told which is the reference and marks each on a continuous scale divided into five intervals labelled excellent to bad; the analysis uses the difference between the two marks (Part 2, Annex 2). VQEG’s first full-reference television study used DSCQS under the then-current BT.500-8 (VQEG, 2000, §4.3).
• Single-stimulus (SS) methods: one image at a time, rated on an adjectival five-grade scale, an 11-grade numerical scale or a continuous line (Part 2, Annex 3).
• Stimulus-comparison methods, in which observers compare two images directly (Part 2, Annex 4).

The Recommendation itself says that no method can yet be definitively recommended over the others and leaves the choice to the researcher.

MOS. The mean opinion score of an image is the average of the $N$ observers’ ratings $u_{ij}$ of image $j$,

$$\mathrm{MOS}_j = \frac{1}{N} \sum_{i=1}^{N} u_{ij} ,$$

on the scale of the experiment; higher means better.

DMOS. When each observer also rates the reference, a difference score removes the observer’s opinion of the content itself. The LIVE study used a single-stimulus design with the references hidden among the distorted images, because a double-stimulus study of 779 images was too long (Sheikh et al., 2006, §II-B). Each raw score $r_{ij}$ on a 1–100 slider became a difference $d_{ij} = r_{i,\mathrm{ref}(j)} - r_{ij}$ from the same observer’s score for the reference; differences were converted to z-scores per observer, averaged over observers, and mapped to a difference mean opinion score (DMOS) by a linear function fitted, separately for each of the seven sessions, to a 50-image double-stimulus realignment experiment (§II-C). Observers whose scores were too often outliers were rejected. In LIVE, a higher DMOS means a worse image. The convention is not universal: the KADID-10k files call their score DMOS, but its range is 1 to 5 and a higher value means higher quality (KADID-10k database page). The sign of a correlation means nothing until the direction of the score is checked.

Realignment. Every session sets its own scale, since observers spread their ratings over whatever range of quality they are shown. Sheikh et al. (2006, §III-C) argue that without realignment experts do not agree that scores from separate sessions may be combined, and they note that the residual misalignment after their correction is hard to quantify.

Paired comparisons. TID2013’s authors report that many participants found choosing between two images less annoying than grading one. TID2013 showed a reference with two distorted versions and asked which was better; each of the 120 distorted versions of a reference took part in nine comparisons per experiment, with pairs drawn as in a Swiss-system chess tournament so that images of similar quality met in later rounds, and wins were summed into a MOS from 0 to 9 (Ponomarenko et al., 2015, §4.1). PIPAL turned 1.13 million paired judgments into scores with the Elo rating system used for chess players, which also lets new distortion types be added without re-rating the old ones (Gu et al., 2020, §3).

Crowdsourcing. Large databases are rated online. KonIQ-10k used a five-point absolute category rating (ACR) scale (bad, poor, fair, good, excellent) and kept 1,459 of 2,302 crowd workers after reliability checks, for more than 1.2 million ratings and at least 120 per image (Hosu et al., 2020, §4). Viewing conditions are then uncontrolled; TID2013’s authors argue that slightly varying conditions resemble how images are actually viewed (§4.1).

Challenges

• Content and distortion interact. The same noise is less visible on a busy texture than on a flat region, the contrast masking that SSIM’s local normalization builds in (see SSIM), and the same MSE can look very different across distortions, as the equal-MSE demonstration in mean squared error shows. A metric must be consistent across contents and distortion types at once.
• Human scores are noisy. In the LIVE study, no metric was statistically as good as the “null model” that predicts DMOS exactly, judged by residuals against individual observers’ scores (Sheikh et al., 2006, Table IX); the gap between metrics and the mean of a panel includes the panel’s own noise.
• Saturation. Many images near the top of the scale look equally good, and their score differences are mostly measurement noise. Sheikh et al. (2006, §III-C) found that on one JPEG session Spearman correlation ranked the Sarnoff JND metric well below SSIM and VIF, while the fitted curves of the three were nearly identical; they recommend RMSE over rank correlation where quality saturates.
• Distortion coverage. Databases with five or six distortion types reward metrics tuned to those types. TID2013 added seven new types to TID2008’s 17, among them changes of colour saturation, chromatic aberrations and lossy compression of noisy images, and its “Exotic” subset, which includes mean shift, contrast change and transmission errors, caused problems for many metrics (Ponomarenko et al., 2015, §3 and §5).
• New kinds of distortion. Outputs of generative restoration methods contain plausible but wrong texture, a defect absent from the classic databases (Gu et al., 2020).
• Viewing conditions. Resolution, display and viewing distance change what is visible; LIVE used 21-inch CRT monitors at about 2 to 2.5 screen heights (Sheikh et al., 2006, §II-B), while crowdsourced studies cannot fix them.

Approaches

The method families are listed roughly in the order in which they appeared.

Signal fidelity. The mean squared error and PSNR, its logarithmic form, measure the average squared pixel difference. They are simple, differentiable and tied to an explicit noise model, and they ignore where and how errors occur. In VQEG’s 2000 study of video metrics, seven or eight of the nine models, depending on the performance measure, were statistically equivalent to each other and to PSNR (VQEG, 2000, executive summary). On LIVE, PSNR was among the best metrics for white noise but much worse than the others on the remaining distortions (Sheikh et al., 2006, §III-C).

Models of the visual system. A second line of work models properties of human vision and measures visible rather than raw errors. Sheikh et al. (2006, §II-D) group the Sarnoff JND Metrix among methods based on a model of the human visual system, DCTune and NQM among such models combined with application-specific modelling, and PQS among feature-based methods; on LIVE, the JND Metrix was in their best-performing group, while DCTune did statistically worse than PSNR (§III-C).

Structural similarity. SSIM compares local means, contrasts and correlations instead of raw differences, and MS-SSIM does so over several scales; both are full-reference and are usually reported next to PSNR.

Information fidelity. Sheikh and Bovik’s (2006) visual information fidelity (VIF) models wavelet coefficients of natural images as a Gaussian scale mixture, the distortion as attenuation plus additive noise, and the visual system as additive “visual noise”. VIF is the ratio of the mutual information between the reference signal and what the brain could extract from the test image to the corresponding information for the reference itself (their eq. 14). It equals 1 for an undistorted copy, and a contrast enhancement without added noise can push it above 1, which the authors read as an image better than its reference (§III-E). In the LIVE evaluation, VIF had the highest linear correlation after fitting on all data, 0.9533, against 0.9393 for MS-SSIM and 0.8709 for PSNR (Sheikh et al., 2006, Table III), and it never lost a significance test to any other metric on any LIVE session (§III-C).

Learned deep-feature full-reference metrics. LPIPS compares normalized features of an ImageNet classifier with channel weights calibrated on 2AFC judgments. DISTS (Ding et al., 2022) combines structure and texture terms from VGG features and was trained on KADID-10k to tolerate resampled texture; PieAPP and DISTS were the strongest metrics on PIPAL’s GAN-based outputs (below). These metrics fit human data better than pixel metrics on the outputs they were tested on, and can be attacked or misled in ways the LPIPS article describes.

Natural-scene-statistics NR methods. Locally normalized luminance of undistorted photographs follows regular statistics, which distortions break. BRISQUE regresses such features onto human scores, and NIQE measures the distance of an image’s features from a model of pristine photographs, without human scores (see NIQE).

Deep NR methods. With databases of thousands of rated images, convolutional networks pretrained on ImageNet and fine-tuned to opinion scores became the dominant blind approach, a form of transfer learning. NIMA (Talebi & Milanfar, 2018) predicts the whole histogram of ratings rather than the mean, trained with a squared earth mover’s distance loss, on the aesthetic database AVA and on TID2013 and LIVE in the Wild. KonCept512 (Hosu et al., 2020) fine-tuned InceptionResNetV2 on KonIQ-10k at $512 \times 384$ and reached SROCC 0.921 on its test set and 0.825 on LIVE in the Wild. HyperIQA (Su et al., 2020) separates content understanding from a “perception rule” generated per image by a hyper network, aimed at authentic distortions, and MUSIQ (Ke et al., 2021) is a multi-scale vision transformer that scores images at their native resolution and aspect ratio instead of resizing them to a fixed shape.

Datasets and Benchmarks

The databases differ in content, distortions, rating method and score convention, as their papers and official pages state them:

Database	References	Distorted images	Distortions	Rating protocol	Score
LIVE Release 2 (Sheikh et al., 2006)	29	779	5 types: JPEG2000, JPEG, white noise, Gaussian blur, fast-fading channel errors	Single stimulus with hidden reference, 1–100 slider, about 23 observers per image, about 25,000 judgments; 50-image realignment study	DMOS, higher = worse
CSIQ (Larson & Chandler, 2010)	30	866	6 types at 4–5 levels: JPEG, JPEG2000, global contrast decrement, pink noise, white noise, Gaussian blur	All versions of an original placed side by side across four calibrated monitors by perceived quality; 5,000 ratings	DMOS
TID2013 (Ponomarenko et al., 2015)	25	3,000	24 types × 5 levels, $512 \times 384$	Paired comparisons in Swiss-system rounds; 971 experiments, 524,340 elementary evaluations, labs and internet in five countries	MOS 0–9, higher = better
KADID-10k (Lin, Hosu & Saupe, 2019)	81	10,125	25 types × 5 levels	Crowdsourced degradation category ratings, 30 per image	“DMOS” 1–5, higher = better
KonIQ-10k (Hosu et al., 2020)	none (authentic)	10,073	In-the-wild photographs sampled from YFCC100M, $1024 \times 768$	Crowdsourced 5-point ACR, at least 120 ratings per image	MOS
SPAQ (Fang et al., 2020)	none (authentic)	11,125	Photos from 66 smartphones	Laboratory, more than 600 subjects, continuous 0–100 scale; also brightness, colourfulness, contrast, noisiness and sharpness scores, scene labels and EXIF data	MOS
PIPAL (Gu et al., 2020)	250	29k	40 types and 116 levels, including outputs of denoising, super-resolution and GAN-based methods; $288 \times 288$ patches	1.13 million paired judgments scored by Elo	Elo-based MOS

Some printed facts disagree between sources. TID2013’s abstract counts 985 experiments, while its Table 2 gives 971; the table is used above. The CSIQ paper (§4) reports 25 observers, while Larson’s CSIQ database page says 35. The LIVE figures, and the section and table numbers cited from Sheikh et al. (2006), come from the authors’ March 2006 draft of the journal paper, hosted by the LIVE laboratory. KADID-10k’s figures come from its official database page and the paper’s abstract, since the three-page paper was not read in full.

LIVE (Release 2), CSIQ and TID2013 are the traditional full-reference benchmarks: few references, synthetic distortions, laboratory ratings. KADID-10k scaled synthetic distortion up to make deep training possible. KonIQ-10k and SPAQ moved to authentic distortions, where only no-reference methods apply. PIPAL targets restoration outputs. The perceptual-similarity dataset BAPPS, with 2AFC judgments instead of opinion scores, is described with LPIPS.

Evaluation Metrics

A quality metric is validated by comparing its scores $x_j$ with the human scores $y_j$ (MOS or DMOS) over a database of $n$ images.

Rank correlations. The Spearman rank-order correlation coefficient (SROCC) is the Pearson correlation of the ranks; without ties,

$$\mathrm{SROCC} = 1 - \frac{6 \sum_{j=1}^{n} d_j^2}{n (n^2 - 1)} ,$$

where $d_j$ is the difference between the ranks of image $j$ by metric and by human score. Kendall’s rank-order correlation coefficient (KROCC) counts pairs of images: with $n_c$ pairs ordered the same way by both and $n_d$ ordered oppositely, $\tau = (n_c - n_d) / \binom{n}{2}$ when there are no ties (common implementations, such as SciPy’s, compute the tie-corrected $\tau_b$). Both measure prediction monotonicity, are unchanged by any monotonic transformation of the metric and need no fitting. VQEG used SROCC as its monotonicity metric (VQEG, 2000, §6.2.5), and TID2013 reports SROCC and KROCC only, because fitting procedures need not be unique (Ponomarenko et al., 2015, §5).

Linear correlation after a nonlinear mapping. The Pearson linear correlation coefficient (PLCC) measures prediction accuracy, but a metric’s relation to opinion scores is rarely linear: PSNR is logarithmic, SSIM crowds most images near 1. The convention, from VQEG’s video studies, is to fit a monotonic function from metric to human score first and compute PLCC between the fitted predictions $\hat{y}_j$ and $y_j$:

$$\mathrm{PLCC} = \frac{\sum_j (\hat{y}_j - \bar{\hat{y}})(y_j - \bar{y})}{\sqrt{\sum_j (\hat{y}_j - \bar{\hat{y}})^2}\,\sqrt{\sum_j (y_j - \bar{y})^2}} .$$

VQEG’s Phase I analysis tried a third-order polynomial and a four-parameter logistic, both constrained to be monotonic over the data, and used the logistic because it gave the higher correlation in virtually all cases (VQEG, 2000, §6.2.4); Larson and Chandler (2010, eq. 14) write it as follows, with the parameters chosen to minimize the mean squared error between predictions and opinion scores:

$$f(x) = \frac{\tau_1 - \tau_2}{1 + \exp\big((x - \tau_3)/\tau_4\big)} + \tau_2 .$$

Sheikh et al. (2006, eqs. 3–4) used a five-parameter logistic, a logistic with an added linear term and constrained to be monotonic,

$$Q(x) = \beta_1 \left( \frac{1}{2} - \frac{1}{1 + \exp\big(\beta_2 (x - \beta_3)\big)} \right) + \beta_4 x + \beta_5 ,$$

applied to the metric or its logarithm, whichever fitted better. The fit is done on the same images on which PLCC is then reported, so PLCC after fitting is an in-sample number, and papers that use different mapping functions can report different PLCCs for the same metric and data (our reasoning).

Errors after mapping. The root mean squared error between $\hat{y}_j$ and $y_j$ is in the units of the opinion scale. Sheikh et al. (2006, §III-C) found it carries the same information as fitted PLCC but is easier to interpret when all metrics do well. The outlier ratio, VQEG’s fourth metric, is the fraction of predictions whose error exceeds a threshold, but the threshold differs between sources: twice the standard error of the image’s DMOS in VQEG’s Phase I (VQEG, 2000, §6.2.6), twice the standard deviation of its individual ratings in Larson and Chandler (2010, eq. 15).

Statistical significance. Competing metrics often differ only in the second decimal place of a correlation (MS-SSIM, IFC and VIF span 0.939 to 0.953 on all of LIVE in Sheikh et al.’s Table III), and whether such differences are real depends on the number of images. Sheikh et al. (2006, §I) give an example: telling a linear correlation of 0.95 from one of 0.93 at 95% confidence, with a test on Fisher’s z-transformation, needs at least 260 images. They compared metrics with F-tests on the variances of the residuals after fitting (§III-B), the method VQEG used in its Phase II report. Bootstrap intervals and paired tests are covered in statistical significance in evaluation.

Splits for trained metrics. A metric fitted to human scores must be tested on images it was not trained on, and in IQA “not trained on” means different content: every distorted version of a reference goes to the same side of the split, or the model sees the test content in other distortions. NIQE’s paper, for example, reports medians over 1000 random 80/20 content-separated splits of LIVE (see NIQE); the KonIQ-10k paper used a split of 7,058 training, 1,000 validation and 2,015 test images (Hosu et al., 2020, §6.1). The general rules are in training, validation and test sets and cross-validation. The stronger test is cross-database: train on one database, test on another.

Reporting per distortion type. An overall correlation can hide a metric’s failure on one distortion and success on others. TID2013’s table makes the point: PSNR reaches SROCC 0.9134 on the “Simple” subset of additive Gaussian noise, Gaussian blur and JPEG compression, but 0.6395 on all 24 distortion types, against 0.8510 for FSIMc and 0.8513 for SFF, the two best on the full set (Ponomarenko et al., 2015, Table 4). The authors note that the best metrics reach only about 0.85 on the full set, below the over 0.9 they reach on TID2008.

Example

The example builds a small synthetic “subjective study” in NumPy and SciPy. Four $128 \times 128$ grayscale reference images in $[0, 255]$ (a ramp with a disk, fine stripes, overlapping random disks and smooth random texture) are distorted by five types at five levels: Gaussian noise, Gaussian blur, JPEG-like block DCT quantization with a scaled ITU-T T.81 luminance table, contrast reduction and a brightness shift, giving 100 images. The opinion scores are simulated, not human data. A toy model assigns each distorted image a visible impairment $v$ chosen by assumption: noise is less visible on busy images (a crude contrast-masking term), blur grows faster than linearly, and brightness shifts are mostly tolerated; the quality is $1 + 4e^{-v}$. Twenty-four simulated raters, each with a personal bias and rating noise, score each image and its hidden reference on a five-point ACR scale, giving a MOS and a DMOS. PSNR (MAX = 255) and SSIM (the paper’s $11 \times 11$ Gaussian window with $\sigma = 1.5$, valid windows) are computed as in the PSNR and SSIM articles and checked against scikit-image 0.25.2 with its paper-matching options. The script reports SROCC, KROCC and PLCC before and after the five-parameter logistic, per distortion type and over five simulated rater panels.

import numpy as np
from scipy.fft import dctn, idctn
from scipy.ndimage import gaussian_filter
from scipy.optimize import curve_fit
from scipy.stats import spearmanr, kendalltau, pearsonr
from skimage.metrics import structural_similarity  # scikit-image 0.25.2, cross-check only

# --- Four synthetic 128x128 grayscale references, 8-bit range [0, 255] ---------------
rng = np.random.default_rng(0)
yy, xx = np.mgrid[0:128, 0:128].astype(float)
smooth = 60 + 120 * xx / 127
smooth[(yy - 64) ** 2 + (xx - 64) ** 2 < 30 ** 2] = 200          # ramp and a disk
stripes = 128 + 70 * np.sin(2 * np.pi * (xx + 0.5 * yy) / 9)      # fine oriented stripes
leaves = np.full((128, 128), 128.0)                                # occluding random disks
for _ in range(400):
    cy, cx, r = rng.uniform(0, 128, 2).tolist() + [rng.uniform(3, 20)]
    leaves[(yy - cy) ** 2 + (xx - cx) ** 2 < r ** 2] = rng.uniform(30, 225)
texture = 128 + 45 * gaussian_filter(rng.standard_normal((128, 128)), 2) / 0.14  # smooth noise
refs = [np.clip(a, 0, 255) for a in (smooth, stripes, leaves, texture)]

# --- Five distortion types at five levels ---------------------------------------------
Q = np.array([[16, 11, 10, 16, 24, 40, 51, 61], [12, 12, 14, 19, 26, 58, 60, 55],
              [14, 13, 16, 24, 40, 57, 69, 56], [14, 17, 22, 29, 51, 87, 80, 62],
              [18, 22, 37, 56, 68, 109, 103, 77], [24, 35, 55, 64, 81, 104, 113, 92],
              [49, 64, 78, 87, 103, 121, 120, 101], [72, 92, 95, 98, 112, 100, 103, 99]])

def jpeg_like(a, s):  # 8x8 block DCT quantized with s * (ITU-T T.81 Table K.1)
    b = dctn((a - 128).reshape(16, 8, 16, 8), axes=(1, 3), norm="ortho")
    q = s * Q[None, :, None, :]
    return idctn(q * np.round(b / q), axes=(1, 3), norm="ortho").reshape(128, 128) + 128

LEVELS = {"noise":    [3, 6, 10, 16, 25],            # Gaussian noise sigma
          "blur":     [0.6, 1.0, 1.5, 2.2, 3.2],     # Gaussian blur sigma (pixels)
          "jpeg":     [1, 2, 4, 8, 16],              # quantization table scale
          "contrast": [0.85, 0.7, 0.55, 0.4, 0.25],  # contrast factor about mid-grey
          "shift":    [8, 16, 24, 32, 48]}           # added brightness (grey levels)

def distort(ref, kind, p, rng):
    if kind == "noise":    d = ref + p * rng.standard_normal(ref.shape)
    if kind == "blur":     d = gaussian_filter(ref, p)
    if kind == "jpeg":     d = jpeg_like(ref, p)
    if kind == "contrast": d = 128 + p * (ref - 128)
    if kind == "shift":    d = ref + p
    return np.clip(np.round(d), 0, 255)

def activity(ref):  # mean local standard deviation (Gaussian window, sigma 2)
    m = gaussian_filter(ref, 2)
    return np.mean(np.sqrt(np.maximum(gaussian_filter(ref ** 2, 2) - m ** 2, 0)))

# --- TOY opinion model (simulated, not human data) -------------------------------------
# Visible impairment v per distortion: an assumption chosen for illustration. Noise is
# masked by image activity; brightness shifts are tolerated; then quality = 1 + 4 exp(-v).
def impairment(kind, p, act):
    return {"noise": (p / 8) / (1 + act / 25), "blur": (p / 1.6) ** 1.5,
            "jpeg": 0.5 * np.log2(1 + p), "contrast": 2.0 * (1 - p),
            "shift": p / 80}[kind]

def panel(q_true, q_ref, rng, n=24):
    """n simulated raters on a 5-point ACR scale; each also rates the hidden reference."""
    bias = rng.normal(0, 0.35, n)[:, None]
    rate = lambda q: np.clip(np.round(q + bias + rng.normal(0, 0.6, (n, q.size))), 1, 5)
    s, s_ref = rate(q_true), rate(q_ref)
    mos = s.mean(0)
    dmos = (s_ref - s).mean(0)                       # difference scores, higher = worse
    ci95 = 1.96 * s.std(0, ddof=1) / np.sqrt(n)      # BT.500-style 95% interval half-width
    return mos, dmos, ci95

# --- Metrics: PSNR (MAX = 255) and SSIM (11x11 Gaussian, sigma 1.5, valid windows) -----
def psnr(x, y):
    return 10 * np.log10(255 ** 2 / np.mean((x - y) ** 2))

def ssim(x, y, L=255.0):
    f = lambda a: gaussian_filter(a, 1.5, truncate=5 / 1.5)[5:-5, 5:-5]  # 11x11, valid
    mx, my = f(x), f(y)
    vx, vy, cxy = f(x * x) - mx ** 2, f(y * y) - my ** 2, f(x * y) - mx * my
    C1, C2 = (0.01 * L) ** 2, (0.03 * L) ** 2
    return np.mean((2 * mx * my + C1) * (2 * cxy + C2) / ((mx ** 2 + my ** 2 + C1) * (vx + vy + C2)))

# --- 5-parameter logistic (Sheikh, Sabir & Bovik 2006, eqs. 3-4) ----------------------
def logistic5(x, b1, b2, b3, b4, b5):
    return b1 * (0.5 - 1 / (1 + np.exp(b2 * (x - b3)))) + b4 * x + b5

def fit_logistic(x, y):
    """Least-squares fit kept monotonic: b1, b2, b4 >= 0 on the score oriented to rise with y."""
    d = np.sign(spearmanr(x, y)[0]); x = d * x
    p0 = [np.ptp(y), 1 / np.std(x), np.mean(x), 0.0, np.mean(y)]
    lo, hi = [0, 0, -np.inf, 0, -np.inf], [np.inf] * 5
    p, _ = curve_fit(logistic5, x, y, p0=p0, bounds=(lo, hi), maxfev=20000)
    return logistic5(x, *p)

# --- Build the study -------------------------------------------------------------------
kinds, P, S_, Q_true = [], [], [], []
drng = np.random.default_rng(1)
for ref in refs:
    act = activity(ref)
    for kind, levels in LEVELS.items():
        for p in levels:
            d = distort(ref, kind, p, drng)
            kinds.append(kind); P.append(psnr(ref, d)); S_.append(ssim(ref, d))
            Q_true.append(1 + 4 * np.exp(-impairment(kind, p, act)))
kinds, P, S_, Q_true = map(np.array, (kinds, P, S_, Q_true))
sk = structural_similarity(ref, d, data_range=255, gaussian_weights=True, sigma=1.5,
                           use_sample_covariance=False)
print(f"SSIM check on the last image: NumPy {S_[-1]:.6f}, scikit-image {sk:.6f}")
q_ref = np.full(Q_true.size, 5.0)
mos, dmos, ci = panel(Q_true, q_ref, np.random.default_rng(2))
print(f"{len(mos)} distorted images, 24 simulated raters; mean 95% CI half-width of MOS "
      f"{ci.mean():.2f} on the 1-5 scale")
print(f"sign convention: SROCC(PSNR, MOS) = {spearmanr(P, mos)[0]:+.3f}, "
      f"SROCC(PSNR, DMOS) = {spearmanr(P, dmos)[0]:+.3f}")

def report(name, x, y):
    yhat = fit_logistic(x, y)
    rmse = np.sqrt(np.mean((yhat - y) ** 2))
    print(f"{name:5s} SROCC {spearmanr(x, y)[0]:.3f}  KROCC {kendalltau(x, y)[0]:.3f}  "
          f"PLCC raw {pearsonr(x, y)[0]:.3f}  PLCC fitted {pearsonr(yhat, y)[0]:.3f}  "
          f"RMSE fitted {rmse:.3f}")

print("\nAll 100 images against MOS:")
report("PSNR", P, mos); report("SSIM", S_, mos)

print("\nSROCC against MOS within each distortion type (20 images each):")
print("type       PSNR    SSIM   mean PSNR  mean SSIM  mean MOS")
for k in LEVELS:
    m = kinds == k
    print(f"{k:9s} {spearmanr(P[m], mos[m])[0]:6.3f}  {spearmanr(S_[m], mos[m])[0]:6.3f}"
          f"  {P[m].mean():9.2f}  {S_[m].mean():9.3f}  {mos[m].mean():8.2f}")

m = kinds != "shift"
print(f"\nwithout the brightness shifts (80 images): SROCC PSNR {spearmanr(P[m], mos[m])[0]:.3f}, "
      f"SSIM {spearmanr(S_[m], mos[m])[0]:.3f}")

print("\nAll-data SROCC and fitted PLCC over five simulated rater panels (seeds 2-6):")
for seed in range(2, 7):
    mo = panel(Q_true, q_ref, np.random.default_rng(seed))[0]
    out = []
    for x in (P, S_):
        out += [spearmanr(x, mo)[0], pearsonr(fit_logistic(x, mo), mo)[0]]
    print(f"seed {seed}: PSNR {out[0]:.3f} / {out[1]:.3f}   SSIM {out[2]:.3f} / {out[3]:.3f}")
print(f"noise-free toy quality: SROCC PSNR {spearmanr(P, Q_true)[0]:.3f}, "
      f"SSIM {spearmanr(S_, Q_true)[0]:.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2):

SSIM check on the last image: NumPy 0.935634, scikit-image 0.935634
100 distorted images, 24 simulated raters; mean 95% CI half-width of MOS 0.27 on the 1-5 scale
sign convention: SROCC(PSNR, MOS) = +0.429, SROCC(PSNR, DMOS) = -0.418

All 100 images against MOS:
PSNR  SROCC 0.429  KROCC 0.302  PLCC raw 0.444  PLCC fitted 0.451  RMSE fitted 0.859
SSIM  SROCC 0.799  KROCC 0.620  PLCC raw 0.667  PLCC fitted 0.800  RMSE fitted 0.578

SROCC against MOS within each distortion type (20 images each):
type       PSNR    SSIM   mean PSNR  mean SSIM  mean MOS
noise      0.877   0.907      28.72      0.750      3.05
blur       0.725   0.840      26.92      0.803      2.70
jpeg       0.769   0.703      29.19      0.890      2.35
contrast   0.941   0.818      22.37      0.850      2.75
shift      0.956   0.953      21.45      0.974      3.89

without the brightness shifts (80 images): SROCC PSNR 0.684, SSIM 0.725

All-data SROCC and fitted PLCC over five simulated rater panels (seeds 2-6):
seed 2: PSNR 0.429 / 0.451   SSIM 0.799 / 0.800
seed 3: PSNR 0.410 / 0.421   SSIM 0.785 / 0.784
seed 4: PSNR 0.428 / 0.445   SSIM 0.796 / 0.790
seed 5: PSNR 0.435 / 0.455   SSIM 0.831 / 0.822
seed 6: PSNR 0.427 / 0.442   SSIM 0.811 / 0.809
noise-free toy quality: SROCC PSNR 0.418, SSIM 0.809

Sign and scale. MOS and DMOS give correlations of opposite sign with the same metric, because one rises with quality and the other with impairment; the magnitudes differ slightly because the DMOS also carries the noise of each rater’s score for the reference.

Within types and across them. Inside each distortion type, PSNR ranks images reasonably well (SROCC 0.73 to 0.96), since within a type its value falls steadily with the distortion level. Pooled over all 100 images it drops to 0.43. The brightness shifts cause much of the drop: they have the lowest mean PSNR of any type but the highest mean MOS, because the toy viewers tolerate them, so they sit in the wrong place on PSNR’s scale relative to every other type. Removing them raises PSNR to 0.68; the rest of the gap comes from content: blur, quantization and contrast reduction give different PSNRs on different references, and the toy model masks noise on busy images, which PSNR ignores. This is the pattern of TID2013’s Simple and Full columns in miniature. SSIM, whose luminance term barely reacts to a uniform shift, ranks the pooled set much better (0.80), even though it is worse than PSNR within the JPEG and contrast types. That outcome reflects the toy model’s assumptions, chosen here to tolerate brightness shifts and to mask noise as SSIM’s normalization does; a toy model with different assumptions could favour PSNR. The example shows how pooling over distortion types changes the verdict, not which metric humans prefer.

What the logistic fit does. SROCC and KROCC are untouched by any monotonic mapping, so the fit can only change PLCC and RMSE. For SSIM it raises PLCC from 0.67 to 0.80, because SSIM’s scores bunch near 1 and relate to MOS along a curve, which the fit straightens. For PSNR it changes almost nothing (0.444 to 0.451): PSNR’s problem here is not a curved relation but a scatter between types that no single monotonic function can remove. KROCC runs lower than SROCC on its own scale but ranks the two metrics the same way.

Rater noise. Over five simulated panels, pooled SROCC moves by about 0.025 for PSNR (0.410 to 0.435) and 0.046 for SSIM (0.785 to 0.831), and the noise-free toy quality gives values inside those ranges. With 24 raters per image the MOS is precise enough (mean 95% interval half-width 0.27 on a 1–5 scale) that the metrics’ disagreement with it, not the raters’ noise, limits the correlations.

Applications

• Compression and transmission: comparing codecs and their settings, the setting of the early LIVE and VQEG studies, whose distortions were compression and transmission errors.
• Restoration benchmarks. Image denoising, super-resolution, deblurring and inpainting papers report PSNR, SSIM and often LPIPS against held-out references.
• Rendering and view synthesis. Novel view synthesis and neural rendering score rendered images against held-out photographs with the same full-reference metrics.
• Cameras and photo collections. SPAQ was collected to evaluate smartphone photography, where quality has become a basis for ranking devices (Fang et al., 2020); blind models such as NIMA were proposed to guide photo enhancement and pipeline tuning (Talebi & Milanfar, 2018).
• Training objectives. Differentiable metrics serve as loss functions: SSIM and MS-SSIM mixed with $\ell_1$, LPIPS as a perceptual loss, as their articles describe.

Using a metric versus evaluating one. Once a metric is optimized, it stops being an independent measure of the output (our reasoning, as in LPIPS): a network trained on an LPIPS or SSIM loss and then scored by the same metric is graded by its own objective, and the perception–distortion tradeoff (summarized in PSNR) shows that pushing a distortion measure down can cost realism beyond a point. PIPAL’s authors asked whether super-resolution methods were improving by beating IQA benchmarks; in their analysis of 23 super-resolution algorithms, the rank correlations printed between the algorithms’ mean human scores and their PSNR and SSIM were negative (−0.40 and −0.26), while PieAPP’s was 0.94 (Gu et al., 2020, Figure 3). PieAPP is a distance, lower for better images, so its positive value shows that the figure, like the paper’s tables, orients coefficients so that positive means agreement with the human scores (our reading). Validating a metric against human scores, and reporting a second, independent metric or a user study when the first was used for training, keeps the two roles apart.

Open Problems

• Generalization across distortions and databases. Metrics tuned on one database’s distortions lose accuracy on another’s. On PIPAL, PSNR’s SROCC is 0.5462 on outputs of PSNR-oriented super-resolution methods but 0.2839 on outputs of GAN-based ones, and the best metrics on the GAN subset, PieAPP, DISTS and WaDIQaM, reach only about 0.53 to 0.55 (Gu et al., 2020, Table 3). The blind cross-database drops reported for KonIQ-10k (above) are the no-reference counterpart.
• Generative distortions. Plausible but invented texture, typical of GAN-based restoration, is absent from the classic databases. Gu et al. (2020, §4) improved learned metrics on GAN outputs with anti-aliased pooling layers.
• Beyond the mean. A MOS discards the spread of opinions. NIMA predicts the rating distribution (Talebi & Milanfar, 2018). In LIVE’s analysis every metric remained statistically worse than a null model that predicts the panel’s DMOS exactly (Sheikh et al., 2006), so even there the metrics had not reached the limit set by the observers’ own disagreement.
• Viewing conditions and scale. Results depend on resolution and viewing distance, which laboratory studies control and crowdsourced studies cannot; MUSIQ’s native-resolution design is one response to the resizing that most networks impose (Ke et al., 2021).
• Agreement among protocols. Absolute ratings, difference scores, paired comparisons and 2AFC similarity judgments answer subtly different questions, and a metric’s ranking can change with the protocol, as the different NIQE results on PIRM and PIPAL illustrate (see NIQE).