Super-Resolution

Super-resolution (SR) estimates a high-resolution image from one or more low-resolution observations of the same scene. Single-image super-resolution (SISR) works from one image and must supply missing detail from prior knowledge of what images look like; multi-frame super-resolution combines several images of a scene taken with small relative shifts; video super-resolution does the same along a video, frame by frame. The task is also called upscaling, upsampling, image enlargement or zooming, and example-based single-image SR is also called “image hallucination” (Glasner et al., 2009), a name that states its central difficulty: a method can add detail that looks right without being in the scene.

Super-resolution is an inverse problem closely related to image denoising and deblurring, and many restoration networks handle all three.

Problem Definition

A low-resolution (LR) image $y$ is modelled as a degraded version of the unknown high-resolution (HR) image $x$. The form Zhang et al. (2021) call the traditional degradation model is

$$y = (x \otimes k)\downarrow_s + \, n ,$$

a convolution with a blur kernel $k$ (the point spread function of the optics and sensor), downsampling by the scale factor $s$, and additive white Gaussian noise $n$. The goal is an estimate $\hat{x}$ of size $s$ times that of $y$ in each dimension.

Most benchmarks use a simpler bicubic degradation: the LR image is produced by bicubic downscaling, in practice MATLAB’s imresize, as the NTIRE 2017 challenge stated for its “classic” track (Agustsson and Timofte, 2017, §3). MATLAB’s documentation states that imresize applies antialiasing by default when it shrinks an image with bicubic interpolation, so this degradation includes a prefilter. Zhang et al. note that bicubic degradation can be viewed as a special case of the traditional model, with a suitable kernel and no noise. A method trained on one degradation and tested on another is solving a different problem: Zhang et al. attribute the poor results of bicubic-trained methods on real images to this mismatch, and in particular to the blur kernel, for which the bicubic kernel is too simple.

Super-resolution is ill-posed: many HR images produce the same LR image. Dong et al. (2016, §1) describe it as an underdetermined inverse problem whose solution is not unique for any LR pixel. The information that is lost can be stated in frequency terms, following sampling and aliasing:

• Frequencies above the LR Nyquist limit fold onto lower frequencies when the image is downsampled. After sampling the alias and the original are indistinguishable, so a single image cannot reveal which one was there.
• Frequencies the blur removes cannot be restored at all where the blur’s transfer function is zero. Irani and Peleg (1991) make this point for an ideal low-pass blur: several HR images then give identical LR images, and an algorithm can only pick one.
• Frequencies the blur attenuates but does not remove can be amplified back, at the price of amplifying noise, as in deconvolution (the example below shows both effects).

In this sense, everything a method adds beyond these limits comes from a prior: a smoothness assumption, the recurrence of patches, a dictionary, or a network trained on example images.

Input and Output

• Input: one LR image (single-image SR), several LR images of a static scene that can be aligned by image registration (multi-frame SR), or a video. Non-blind methods also take the blur kernel and noise level; blind methods estimate or ignore them.
• Output: an image $s$ times larger in each dimension. Integer factors of 2, 3 and 4 are standard in benchmarks; ×8 appears in work on perceptual and face SR (Johnson et al., 2016; Saharia et al., 2022). Some networks handle several factors in one model.
• Colour: many methods, and most older benchmarks, super-resolve only the luma channel $Y$ of YCbCr and upscale the chroma channels by bicubic interpolation; Kim et al. (2016, §5.3) give the reason as human vision being more sensitive to detail in intensity than in colour.

Variants

• Single-image, multi-frame and video SR, which differ in where extra information comes from: priors alone, or other observations of the same scene. Glasner et al. (2009) combine classical multi-image and example-based SR within a single image: patches recurring within the image at subpixel offsets act like extra frames, and patches recurring across scales act like training examples.
• Bicubic (classical) and real-world SR. Real images carry unknown blur, sensor noise and compression artifacts, so real-world methods train on synthetic degradations far more varied than bicubic downscaling (see Approaches).
• Non-blind and blind SR, depending on whether the degradation is known.
• Fidelity-oriented and perception-oriented SR. Methods trained to minimize pixel error score highest on PSNR; perceptual, adversarial or diffusion objectives give sharper, more realistic images that score lower, a tradeoff that has been proved (Evaluation Metrics).
• Domain-specific SR, such as face super-resolution (“face hallucination”), which Yang et al. (2010) treat as a special case with its own trained dictionaries.

Challenges

• Regression to the mean. A method trained with a pixel loss on an ambiguous problem predicts an average of the plausible HR images, which is blurry. Ledig et al. (2017) illustrate this with their Figure 3: the MSE-optimal reconstruction lies between the sharp candidates and looks smooth.
• Degradation mismatch. Zhang et al. (2021) note that SR methods do not perform well when the assumed degradation deviates from that of real images, and that this holds for networks trained with adversarial losses too.
• Aliasing. Fine periodic structure above the LR Nyquist limit arrives as a coarser false pattern; no method can tell the two apart from a single frame, and the example below shows every method failing on such a pattern.
• Hallucinated detail. Realistic texture is not true texture. Ledig et al. (2017, §4) caution that methods hallucinating finer detail may be less suited to medical or surveillance uses.
• Cost. Networks applied to the HR grid scale with the output size; video and phone pipelines need real-time throughput.

Approaches

The history of super-resolution is a sequence of richer priors: interpolation assumes smoothness, multi-frame methods use the physics of image formation and extra observations, example-based methods learn from patches, and neural networks learn the whole mapping, first for fidelity and then for realism.

Interpolation

Nearest-neighbour, bilinear and bicubic interpolation, described in image warping, resample the LR image on a finer grid. They add no information: they cannot undo the blur or the aliasing, and they produce blocky (nearest neighbour) or smooth (bilinear, bicubic) results. Freeman et al. (2001, §1) show cubic spline interpolation blurring edges and detail when a teapot photograph is enlarged by 8, and sharpening the result helping little. Interpolation remains the baseline in benchmark tables, the first step of several learned methods, and the standard way to upscale chroma.

Multi-frame reconstruction

When several LR images of a scene are shifted by fractions of a pixel, each samples the scene at different points, and together they can determine detail finer than any single grid. Irani and Peleg (1991) cite earlier work by Tsai and Huang (1984), who used frequency-domain methods on translated images and disregarded the blur.

Irani and Peleg modelled the observed frames $g_k$ as the scene blurred by the point spread function, displaced, sampled and quantized, and reconstructed the HR image by iterative back-projection. They registered the frames to subpixel accuracy, estimating translation and rotation with a gradient-based method built on Lucas and Kanade and refined in a Gaussian pyramid. Starting from an initial guess $f^{(0)}$, they simulated the imaging process to obtain predicted LR images $g_k^{(n)}$, and spread each difference $g_k - g_k^{(n)}$ back onto the HR pixels in its receptive field. Schematically,

$$f^{(n+1)} = f^{(n)} + \sum_k B_k \big( g_k - g_k^{(n)} \big),$$

where $B_k$ is a back-projection operator built from a kernel the authors were free to choose; they likened the method to back-projection in computed tomography. They reported convergence usually in fewer than five iterations, and used the average of the registered LR images as the initial guess.

Multi-frame reconstruction is limited by registration accuracy and by the conditioning of the equations. Glasner et al. (2009, §1) summarize earlier analyses as showing that, in practice, it is numerically limited to resolution increases by factors smaller than 2.

Example-based and sparse-coding methods

Single-image methods replace extra frames with knowledge learned from other images. Freeman, Jones and Pasztor (2002; the details here are from their 2001 technical report) built a training set of HR images and blurred, subsampled LR versions; stored pairs of band-pass filtered, contrast-normalized LR patches and the corresponding high-frequency HR patches; and, for a new image, looked up candidate HR patches for each LR patch. They show that taking each patch’s nearest neighbour alone gives a result that looks like “oatmeal” (§3.1): similar LR patches have very different HR patches, so neighbouring choices must agree. They enforced agreement on overlapping pixels with a Markov network solved by belief propagation, and also gave a faster one-pass algorithm that processes patches in raster order. They aimed for plausible rather than correct detail.

Glasner et al. (2009) showed that a single image is its own training set: small patches recur within a natural image and across its downscaled versions, and a match found at a coarser scale reveals what the finer version of a patch looks like. Huang, Singh and Ahuja (2015) widened the search to patches transformed by perspective, guided by detected planes, and by local affine changes, and reported much better results on urban scenes.

Yang, Wright, Huang and Ma (2010) represented patches as sparse combinations of atoms from two jointly trained dictionaries, one for LR features and one for HR patches, learned so that a patch pair shares the same sparse code. For a new LR patch they solve an $\ell_1$-regularized least-squares problem (the Lasso) for its code over the LR dictionary, apply the code to the HR dictionary, and finally project the whole image onto the set of images consistent with the observation by gradient descent, a global back-projection step. Their dictionaries of 1,024 atoms were trained from 100,000 patch pairs. Neighbour embedding, random forests and anchored regression (ANR, A+) followed the same patch-to-patch pattern (Dong et al., 2016, §2.1), and SRCNN was compared with neighbour embedding, ANR and A+.

Convolutional networks

SRCNN. Dong, Loy, He and Tang (2014; journal version 2016, section numbers here from its arXiv v3 copy) replaced the patch pipeline by a three-layer convolutional neural network applied to the bicubic-upscaled image, whose layers they interpreted as patch extraction, non-linear mapping and reconstruction. They showed that a sparse-coding method can itself be viewed as such a network (2016, §3.2), with the difference that SRCNN optimizes all its steps jointly. The basic network has 8,032 parameters (2016, §4.1), and was trained with a mean squared error loss on LR inputs synthesized by Gaussian blur, subsampling and bicubic upscaling (§3.3).

Deeper and faster networks. Kim, Lee and Lee (2016) built VDSR, a network with 20 weight layers that predicts only the residual between the bicubic input and the HR image, a form of residual learning; this, with adjustable gradient clipping, allowed learning rates $10^4$ times higher than SRCNN’s, and one network trained on several scale factors handled them all. Shi et al. (2016) moved the computation to the LR grid: their network extracts features at low resolution and ends with a sub-pixel convolution layer that rearranges channels into an $s$ times larger image, avoiding the cost of processing the bicubic-upscaled input; the layer is described in transposed convolution. SRResNet, EDSR, SwinIR and BasicVSR all upsample at the end in this way.

EDSR. Lim et al. (2017) removed the batch normalization layers from the residual blocks of SRResNet, the generator of Ledig et al. (2017), arguing that normalizing features removes range flexibility, and reported about 40% less memory use in training as a side effect. Their large model has 32 residual blocks with 256 filters, about 43 million parameters, residual scaling by 0.1 for stable training, and an $L_1$ loss, which they found converged better than $L_2$. EDSR won the NTIRE 2017 challenge. EDSR+ uses geometric self-ensemble, averaging the outputs for flipped and rotated copies of the input, which the authors note is valid only for symmetric degradations such as bicubic downsampling.

Transformers. SwinIR (Liang et al., 2021) replaced the convolutional body with transformer blocks, residual Swin Transformer blocks that apply self-attention within shifted local windows, between a convolutional shallow feature extractor and an upsampling reconstruction module, and trained classical SR with an $L_1$ loss.

The table collects reported results on Set5 at ×4 with bicubic degradation, measured on Y with as many border pixels removed as the scale factor (Lim et al., 2017, §4.5; for SwinIR, its supplementary material and official test script). Numbers from different papers are not strictly comparable: each paper used its own evaluation code, and the same SRResNet scores 0.9019 SSIM in Ledig et al.’s table and 0.8910 in Lim et al.’s.

Method	Year	PSNR (dB)	SSIM	Source
Bicubic	–	28.42	0.8104	Lim et al. (2017), Table 3
A+ (anchored regression)	2014	30.28	0.8603	Lim et al. (2017), Table 3
SRCNN	2014	30.48	0.8628	Lim et al. (2017), Table 3
VDSR	2016	31.35	0.8838	Lim et al. (2017), Table 3
SRResNet	2017	32.05	0.8910	Lim et al. (2017), Table 3
EDSR	2017	32.46	0.8968	Lim et al. (2017), Table 3
SwinIR (trained on DIV2K and Flickr2K)	2021	32.92	0.9044	Liang et al. (2021), Table 2

Of SwinIR’s 4.5 dB gain over bicubic in this table, all but 0.46 dB had been reached by EDSR in 2017, a margin on which evaluation details matter.

Perceptual and adversarial losses

Johnson, Alahi and Fei-Fei (2016) trained ×4 and ×8 SR networks with a feature reconstruction loss, the squared distance between activations of a pretrained VGG-16 network at layer relu2_2, instead of a per-pixel loss; compared with the same network trained with a pixel loss, they report sharper edges and finer detail but lower PSNR (§4.2 of arXiv v1). The loss function article covers perceptual losses in general.

Ledig et al. (2017) added a generative adversarial network. SRGAN’s generator is a residual network trained with a perceptual loss, a VGG-19 feature loss plus $10^{-3}$ times an adversarial loss from a discriminator trained to tell SR outputs from real HR images (eq. 3). Its outputs scored lower in PSNR than the same generator trained with MSE (SRResNet), but in a mean opinion score test with 26 raters they received higher mean scores than every other method on Set5, Set14 and BSD100 at ×4 (Table 2).

ESRGAN (Wang et al., 2018) revised each part of SRGAN: residual-in-residual dense blocks without batch normalization, a relativistic discriminator that predicts whether a real image is more realistic than a fake one rather than whether an image is real, and a VGG loss computed on features before the activation rather than after. It won region 3 of the PIRM 2018 perceptual SR challenge, the region allowing the largest RMSE, with the best perceptual index (§1, Fig. 2). ESRGAN also interpolates the weights of a PSNR-oriented network and a GAN-trained network to move continuously between smooth and textured outputs, which the authors found reduced artifacts better than interpolating the output images.

Real-world degradations

Zhang et al. (2021, BSRGAN) synthesize training pairs with a randomly shuffled sequence of degradations: blur by isotropic and anisotropic Gaussian kernels, downsampling by nearest, bilinear or bicubic interpolation, Gaussian noise, JPEG compression with random quality, and camera sensor noise simulated through a reversed and re-applied camera processing pipeline. Real-ESRGAN (Wang et al., 2021) applies a classical degradation twice in sequence, a second-order process, and adds sinc filters to simulate common ringing and overshoot artifacts. Both train ESRGAN-style networks on these synthetic pairs and target real photographs, where no HR reference exists.

Diffusion models

Diffusion models sample an HR image conditioned on the LR input instead of predicting one. SR3 (Saharia et al., 2022; details from arXiv v2) adapts DDPM to SR: a U-Net receives the noisy HR image concatenated with the bicubic-upsampled LR image. In a two-alternative test on 16 × 16 → 128 × 128 face SR, viewers shown the input chose SR3’s output over the real photograph in 54.1% of trials, against 8.9% to 29.3% for the other methods, where 50% means indistinguishable (Figure 6). SR3’s PSNR and SSIM were below those of a regression baseline with the same architecture (Table 1), and the authors argue that these metrics penalize any high-frequency detail not aligned with the reference.

Video super-resolution

Video SR has many frames of a moving scene, so it is multi-frame SR with motion that must be estimated. Chan et al. (2021) decompose video SR networks into four components: propagation (how information moves between frames), alignment (spatial correction for motion), aggregation (combining aligned features) and upsampling. Sliding-window methods restore each frame from a few neighbours; recurrent methods carry features along the whole sequence. Their BasicVSR combines bidirectional recurrent propagation, alignment by optical flow and sub-pixel upsampling, and they report gains of up to 0.61 dB and up to 24 times faster inference over earlier state-of-the-art models (§1).

Datasets and Benchmarks

Test sets are small collections of HR images, degraded by bicubic downscaling at ×2, ×3 and ×4:

• Set5 (Bevilacqua et al., 2012): five images, baby, bird, butterfly, head and woman (Table 2).
• Set14 (Zeyde, Elad and Protter, 2012): 14 images, which SRCNN used as a larger test set beside Set5.
• BSD100 (B100): 100 test images of natural scenes from the Berkeley Segmentation Dataset, the BSD300 test set (Ledig et al., 2017, §3.1).
• Urban100 (Huang, Singh and Ahuja, 2015): 100 HR images of urban scenes, buildings and structures from Flickr, collected because the earlier benchmark sets contained few such images. It gives the lowest scores of the four in the tables of Kim et al. (2016) and Lim et al. (2017).
• DIV2K (Agustsson and Timofte, 2017): 1,000 high-quality images with about 2,000 pixels along at least one axis, split into 800 training, 100 validation and 100 test images. It was collected for the NTIRE 2017 challenge, whose tracks used bicubic downscaling and an “unknown” blur-and-decimation degradation. Since the test ground truth was not released, Lim et al. (2017, §4.1) report DIV2K validation results.

SRCNN was trained on 91 images or on ImageNet; EDSR and SwinIR train on DIV2K, SwinIR also with Flickr2K, and SwinIR’s tables add a fifth test set, Manga109. Real-world SR has no reference images, so it is judged visually or with no-reference metrics.

Evaluation Metrics

PSNR and SSIM against the HR reference remain the standard numbers, computed by convention on the luma channel with a border removed, with details that differ between papers: VDSR’s evaluation framework crops pixels near the boundary (Kim et al., 2016, §5.3) and SRGAN removed a 4-pixel strip (Ledig et al., 2017, §3.1). The PSNR article covers these conventions, including EDSR’s; the SSIM article covers SSIM’s window and constants.

The perception–distortion tradeoff. Blau and Michaeli (2018) proved that, for any distortion measure, the best achievable perceptual quality cannot improve as mean distortion decreases, so near that bound one can be improved only at the expense of the other; the PSNR article states the theorem. They concluded that restoration methods should be evaluated by a pair of measures, one full-reference measure for distortion and one no-reference measure for perceptual quality, and that GANs provide a principled way to approach the bound.

Perceptual measures. The PIRM 2018 challenge ranked methods within regions of root-mean-square error by a perceptual index combining two no-reference scores, Ma et al.’s learned score and NIQE (Wang et al., 2018, §1). LPIPS compares deep features of output and reference. Human studies are the reference: SRGAN used mean opinion scores, SR3 a forced choice between output and original. Image quality assessment describes how these metrics are validated, and how weakly they agree with people on the outputs of GAN-based methods.

Consistency. Since a valid SR output must reproduce the input when degraded again, SR3 also reports the MSE between the downsampled output and the LR input.

Example: Interpolation against a Learned Linear Upsampler

The example renders 96 × 96 synthetic images with exactly known content: random disks and rectangles, thin pen strokes, a mixture of both, and two oblique sinusoidal gratings with periods of 8 and 4 HR pixels. Each is degraded by a known model, a Gaussian blur with $\sigma = 1.2$ HR pixels followed by keeping every third pixel, so the LR images are 32 × 32. Three upsamplers return a 96 × 96 image: nearest neighbour, bicubic interpolation (cubic convolution with $a = -0.5$), and a ridge regression that maps each 5 × 5 LR neighbourhood linearly to the 3 × 3 block of HR pixels at its centre, trained on 40 other synthetic images. The ridge model is a learned linear filter for each of the nine sub-pixel positions, a minimal example-based method. Scores are PSNR and SSIM (Gaussian window, $\sigma = 1.5$) on $[0, 1]$ values with a 3-pixel border removed. The script also varies the training seed, trains with the wrong blur, and adds noise.

import numpy as np
from scipy.ndimage import gaussian_filter
from skimage.metrics import structural_similarity  # scikit-image 0.25.2

S, H = 3, 96                      # scale factor, high-resolution size
L = H // S                        # low-resolution size (32)

def render(rng, kind, period=None):
    """A 96x96 synthetic image in [0, 1], drawn at 4x and box-averaged (anti-aliased)."""
    n = 4 * H
    y, x = np.mgrid[0:n, 0:n] / 4.0
    img = np.full((n, n), 0.5 if rng is None else rng.uniform(0.2, 0.8))
    if kind in ("shapes", "mixed"):
        for _ in range(6):
            cx, cy, r = rng.uniform(10, 86), rng.uniform(10, 86), rng.uniform(4, 16)
            img[(x - cx) ** 2 + (y - cy) ** 2 < r ** 2] = rng.uniform()
            x0, y0 = rng.uniform(0, 80, 2)
            w, h = rng.uniform(5, 25, 2)
            img[(x > x0) & (x < x0 + w) & (y > y0) & (y < y0 + h)] = rng.uniform()
    if kind in ("strokes", "mixed"):
        for _ in range(10):                       # thin pen strokes, 1 to 3 pixels wide
            p, q = rng.uniform(5, 91, 2), rng.uniform(5, 91, 2)
            d, t = q - p, rng.uniform(1, 3)
            s = np.clip(((x - p[0]) * d[0] + (y - p[1]) * d[1]) / (d @ d), 0, 1)
            img[np.hypot(x - p[0] - s * d[0], y - p[1] - s * d[1]) < t / 2] = rng.choice([0.05, 0.95])
    if kind == "stripes":                         # oblique sinusoid; the LR grid resolves periods > 6
        img = 0.5 + 0.4 * np.sin(2 * np.pi * (x * np.cos(0.4) + y * np.sin(0.4)) / period)
    return img.reshape(H, 4, H, 4).mean(axis=(1, 3))

def degrade(hr, sigma, noise=0.0, rng=None):
    """Known degradation: Gaussian blur, keep every 3rd pixel (centre phase), optional noise."""
    lr = gaussian_filter(hr, sigma, mode="reflect")[1::S, 1::S]
    return lr + noise * rng.standard_normal(lr.shape) if noise else lr

def interp_matrix(kind):
    """HR-by-LR matrix for 1-D upsampling; HR pixel x sits at LR coordinate (x - 1) / 3."""
    M = np.zeros((H, L))
    for x in range(H):
        u = (x - 1) / S
        if kind == "nearest":
            M[x, min(max(int(np.floor(u + 0.5)), 0), L - 1)] = 1
            continue
        i0 = int(np.floor(u))
        for i in range(i0 - 1, i0 + 3):           # cubic convolution kernel, a = -0.5
            t, a = abs(u - i), -0.5
            w = ((a + 2) * t**3 - (a + 3) * t**2 + 1 if t <= 1
                 else a * t**3 - 5 * a * t**2 + 8 * a * t - 4 * a if t < 2 else 0.0)
            M[x, min(max(i, 0), L - 1)] += w      # clamp at the border
    return M

M_NN, M_BIC = interp_matrix("nearest"), interp_matrix("bicubic")
def upsample(lr, M): return M @ lr @ M.T

def patches(lr):
    """5x5 LR neighbourhood of every LR pixel (edge-padded), plus a bias: (L*L, 26)."""
    p = np.pad(lr, 2, mode="edge")
    X = np.stack([p[i:i + L, j:j + L] for i in range(5) for j in range(5)], -1).reshape(-1, 25)
    return np.hstack([X, np.ones((len(X), 1))])

def blocks(hr):
    """The 3x3 HR block centred on each LR pixel: (L*L, 9) regression targets."""
    return hr.reshape(L, S, L, S).transpose(0, 2, 1, 3).reshape(-1, S * S)

def train_ridge(seed, sigma, noise=0.0, n_img=40, lam=1e-5):
    """Fit one linear map from a 5x5 LR patch to its 3x3 HR block (ridge regression)."""
    rng = np.random.default_rng(seed)
    X, Y = [], []
    for _ in range(n_img):
        hr = render(rng, "mixed")
        X.append(patches(degrade(hr, sigma, noise, rng)))
        Y.append(blocks(hr))
    X, Y = np.vstack(X), np.vstack(Y)
    return np.linalg.solve(X.T @ X + lam * len(X) * np.eye(X.shape[1]), X.T @ Y)

def apply_ridge(W, lr):
    return (patches(lr) @ W).reshape(L, L, S, S).transpose(0, 2, 1, 3).reshape(H, H)

def score(hr, out):
    """PSNR and SSIM (Wang et al. settings) with 3 border pixels removed, values in [0, 1]."""
    a, b = hr[S:-S, S:-S], np.clip(out, 0, 1)[S:-S, S:-S]
    psnr = 10 * np.log10(1 / np.mean((a - b) ** 2))
    ssim = structural_similarity(a, b, data_range=1, gaussian_weights=True, sigma=1.5,
                                 use_sample_covariance=False)
    return psnr, ssim

SIGMA = 1.2                                       # the true blur, in HR pixels
tests = {"shapes": render(np.random.default_rng(100), "shapes"),
         "strokes": render(np.random.default_rng(101), "strokes"),
         "mixed": render(np.random.default_rng(102), "mixed"),
         "stripes 8": render(None, "stripes", period=8),
         "stripes 4": render(None, "stripes", period=4)}
W = train_ridge(seed=0, sigma=SIGMA)

print("1. x3, blur sigma 1.2, no noise: PSNR dB / SSIM")
print("image         nearest           bicubic           ridge (training seed 0)")
for k, hr in tests.items():
    lr = degrade(hr, SIGMA)
    outs = [upsample(lr, M_NN), upsample(lr, M_BIC), apply_ridge(W, lr)]
    print(f"{k:10s}" + "".join(f"    {p:6.2f} / {s:.3f}" for p, s in map(lambda o: score(hr, o), outs)))
p, s = score(tests["stripes 4"], np.full((H, H), 0.5))
print(f"flat grey 0.5 on stripes 4: {p:.2f} / {s:.3f}")

print("\n2. Ridge PSNR over training seeds 0-4 (40 different training images each)")
runs = [train_ridge(seed, SIGMA) for seed in range(5)]
for k, hr in tests.items():
    ps = [score(hr, apply_ridge(Wi, degrade(hr, SIGMA)))[0] for Wi in runs]
    print(f"{k:10s}  {min(ps):.2f}-{max(ps):.2f}")

print("\n3. Wrong degradation: ridge trained with blur sigma_t, tested with true sigma 1.2")
print("sigma_t   " + "".join(f"{k:>10s}" for k in tests))
for st in [0.3, 0.6, 0.9, 1.2, 1.5, 1.8, 2.4]:
    Wst = train_ridge(0, st)
    print(f"{st:7.1f}   " + "".join(f"{score(hr, apply_ridge(Wst, degrade(hr, SIGMA)))[0]:10.2f}"
                               for hr in tests.values()))

print("\n4. Noise sd 0.03 added to the LR image: PSNR dB")
W_noisy = train_ridge(0, SIGMA, noise=0.03)
print("method                " + "".join(f"{k:>10s}" for k in tests))
rows = {"bicubic": lambda z: upsample(z, M_BIC), "ridge trained clean": lambda z: apply_ridge(W, z),
        "ridge trained noisy": lambda z: apply_ridge(W_noisy, z)}
lrs = [degrade(hr, SIGMA, 0.03, np.random.default_rng(200 + i)) for i, hr in enumerate(tests.values())]
for name, f in rows.items():
    print(f"{name:20s}  " + "".join(f"{score(hr, f(lr))[0]:10.2f}" for hr, lr in zip(tests.values(), lrs)))

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; about 6 seconds):

1. x3, blur sigma 1.2, no noise: PSNR dB / SSIM
image         nearest           bicubic           ridge (training seed 0)
shapes         24.55 / 0.822     26.26 / 0.869     27.56 / 0.869
strokes        22.04 / 0.768     22.40 / 0.781     23.28 / 0.808
mixed          20.01 / 0.708     20.87 / 0.750     22.43 / 0.797
stripes 8      15.43 / 0.698     17.34 / 0.802     27.07 / 0.983
stripes 4      12.25 / 0.132     11.90 / 0.050     11.63 / 0.055
flat grey 0.5 on stripes 4: 11.82 / 0.014

2. Ridge PSNR over training seeds 0-4 (40 different training images each)
shapes      27.56-27.60
strokes     23.28-23.51
mixed       22.24-22.46
stripes 8   26.10-29.16
stripes 4   11.52-11.63

3. Wrong degradation: ridge trained with blur sigma_t, tested with true sigma 1.2
sigma_t       shapes   strokes     mixed stripes 8 stripes 4
    0.3        25.70     22.03     20.43     15.97     12.00
    0.6        26.55     22.52     21.22     18.26     11.96
    0.9        27.19     22.98     21.94     21.77     11.85
    1.2        27.56     23.28     22.43     27.07     11.63
    1.5        27.16     22.73     21.71     22.27     11.22
    1.8        25.34     21.07     19.43     17.14     10.42
    2.4        20.37     17.91     15.47     13.15      7.71

4. Noise sd 0.03 added to the LR image: PSNR dB
method                    shapes   strokes     mixed stripes 8 stripes 4
bicubic                    25.21     21.93     20.49     17.31     11.86
ridge trained clean        24.01     21.53     20.99     24.01     11.50
ridge trained noisy        25.02     22.06     21.13     24.44     11.63

Interpolation against learning. Bicubic interpolation beats nearest neighbour on every image except the period-4 grating, and the ridge model beats bicubic on the shapes, strokes and mixture by 0.9 to 1.6 dB. The gain comes from the blur: bicubic interpolation only fills in between samples, while the regression has learned an approximate inverse of the known Gaussian, a deblurring and interpolation filter in one. On the disks and rectangles the ridge model’s SSIM equals bicubic’s despite the higher PSNR, a reminder that the two metrics weigh errors differently.

What can and cannot be recovered. The period-8 grating lies below the LR Nyquist limit (a period of 6 HR pixels on a grid with a 3-pixel spacing), so its information is present in the LR image, only attenuated by the blur; by our arithmetic the Gaussian’s frequency response at that period is about 0.64. The ridge model restores it almost fully, a gain of nearly 10 dB. The period-4 grating lies above the limit and arrives aliased: every method scores between 11.6 and 12.3 dB with an SSIM of 0.13 or less, about what a flat grey image scores (11.82 dB), because no single-frame method can tell the true stripes from their alias.

Training data. Over five training sets the ridge model varies by less than 0.25 dB on the three images resembling the training content, but by 3 dB on the period-8 grating, a pattern absent from training: how a learned method treats unfamiliar content depends on what it happened to see.

Wrong degradation. Training with the true blur gives the best score on every image except the aliased grating. Assuming too little blur ($\sigma_t = 0.3$, close to no blur) gives results 0.4 to 1.4 dB below bicubic on all but the period-4 grating, while assuming too much ($\sigma_t = 2.4$) over-sharpens, costs 7.2 dB on the shapes and falls below nearest neighbour on every image. This asymmetry between under- and over-estimating the blur is a property of this example; the general lesson that a learned model inherits its training degradation is the one Zhang et al. (2021) draw for real images.

Noise. With noise of standard deviation 0.03 on the LR image, the model trained on clean data amplifies the noise and falls below bicubic on the shapes, the strokes and the aliased grating. Training with the same noise level recovers most of the loss and keeps the large gain on the period-8 grating, the trade-off between sharpening and noise amplification that learned restoration settles from its training data.

Applications

• Consumer imaging and displays: enlarging photographs and converting standard-definition video to HDTV (Freeman et al., 2001, §1), and making better use of high-resolution displays and low-cost sensors in phones (Yang et al., 2010, §I).
• Surveillance, medical and satellite imaging, named as applications by Kim et al. (2016, §1) and Yang et al. (2010, §I), with the caution about hallucinated detail above.
• Real-time video: sub-pixel networks were designed to super-resolve 1080p video in real time (Shi et al., 2016).
• Generation: cascades of diffusion SR models enlarge samples from low-resolution generators, as in SR3’s class-conditional ImageNet cascade, part of how high-resolution image generation is done.
• Other restoration: the same networks, trained on other degradations, perform denoising and JPEG artifact removal (SwinIR).

Open Problems

• Real degradations. Blind SR of photographs with unknown blur, noise and compression, without paired ground truth, remains the practical problem; synthetic degradation pipelines are an approximation.
• Faithfulness. Generative methods produce detail that looks real; telling users which detail is supported by the input, and keeping text, faces and medical structures faithful, is unsolved.
• Evaluation. Fidelity metrics penalize realism and no-reference metrics can be fooled; human studies are costly and rarely comparable across papers.
• Efficiency and scale. Large networks and many-step diffusion samplers are slow for high resolutions and video, and arbitrary, non-integer scale factors need models that do not train one network per factor.