Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

“Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network” is a paper by Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang and Wenzhe Shi, all of Twitter, published at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2017. It introduced two networks for 4× super-resolution: SRResNet, a deep residual network trained with a pixel-wise mean squared error, which the authors reported as a new state of the art by PSNR and SSIM; and SRGAN, the same generator trained with a “perceptual loss” that combines a distance between VGG feature maps with an adversarial loss from a discriminator, as in a generative adversarial network. SRGAN scored lower than SRResNet on PSNR and SSIM, yet in a mean opinion score (MOS) study with 26 raters it received the highest mean score of the methods compared on three benchmarks. Its argument that fidelity metrics and perceived quality part ways at large scale factors was taken up by later work on evaluating super-resolution, including the perception–distortion trade-off.

The paper went through five arXiv versions between September 2016 and May 2017, and the method, the author list and the tables changed along the way. This article was read in the CVF open-access copy of the CVPR paper and in all five arXiv versions of 1609.04802; section, table and figure numbers are those of the CVPR copy and arXiv v3–v5, which share them, and material found in some versions only is labelled.

Problem

By 2016, convolutional networks had raised super-resolution accuracy and speed: SRCNN learned the mapping from a bicubic-upscaled image, the efficient sub-pixel network (ESPCN) of Shi et al. (2016), several of whose authors are on this paper, computed features at low resolution and upsampled only at the end, and deeper recursive networks such as DRCN added accuracy. The paper describes the mean squared error (MSE) as the most widely used training objective for super-resolution, convenient because minimizing MSE maximizes PSNR, the measure the field used to rank methods.

The authors’ starting question, in the abstract, was how to recover finer texture at large upscaling factors. Their diagnosis (§1.1.3, Figure 3) was that a pixel-wise loss handles the ambiguity of the problem badly: many sharp high-resolution (HR) images are consistent with one low-resolution (LR) input, and the MSE-optimal prediction is their pixel-wise average, which is smooth. Such outputs have high PSNR but, the abstract argues, often lack high-frequency detail and fail to match the fidelity expected at the higher resolution. Figure 2 makes the point on one image at 4×: SRResNet scores 23.53 dB and SSIM 0.7832, SRGAN 21.15 dB and 0.6868, and bicubic interpolation 21.59 dB and 0.6423, and the authors present it as a case where the highest PSNR is not the perceptually better result.

Losses computed in the feature space of a pretrained network had already been proposed. The paper names Johnson, Alahi and Fei-Fei (2016) and Bruna, Sprechmann and LeCun (2016) as the closest work. Johnson et al. trained ×4 and ×8 super-resolution networks to match features of a pretrained VGG-16 network (see loss function for perceptual losses in general); Bruna et al., as the paper summarizes them, minimized squared error in the feature spaces of VGG19 and of scattering networks. The paper also describes Dosovitskiy and Brox (2016) as combining feature-space losses with adversarial training for image generation, and from v3 it cites Yu and Porikli (2016) for adding a discriminator loss to a pixel-wise MSE to super-resolve face images by 8×; Springer lists that chapter as first online on 16 September 2016, a day after arXiv v1. In our reading, the gap the paper aimed at was an adversarial loss for single-image super-resolution of general natural photographs rather than faces.

Contribution

The CVPR version lists three contributions (§1.2):

• SRResNet. A 16-block residual network optimized for MSE that the authors reported as a new state of the art at 4× by PSNR and SSIM.
• SRGAN. A GAN-based network trained with a perceptual loss in which the MSE content loss is replaced by a loss on VGG feature maps, which the paper describes as more invariant to changes in pixel space.
• A MOS study on images from three public benchmarks, which the authors presented as confirming that SRGAN was the new state of the art, by a large margin, for photo-realistic super-resolution at 4×.

The abstract adds the claim that, to the authors’ knowledge, SRGAN was the first framework able to infer photo-realistic natural images at 4×. In v1 the contributions were two: SRResNet (then with 15 blocks) and the perceptual loss; the MOS study entered in v3, after v1 had listed collecting mean opinion scores as future work.

Method

The LR image $I^{LR}$ is a downsampled version of the HR image $I^{HR}$, and a generator $G_{\theta_G}$ is trained to minimize the average of a super-resolution loss $l^{SR}$ over training pairs (eq. 1). The paper describes the degradation in two ways that do not match: §2 says $I^{LR}$ is obtained by applying a Gaussian filter and then downsampling, while §3.2 says the LR training images were made by downsampling with a bicubic kernel. Both sentences are in every version, the CVPR copy included.

Generator and discriminator

Figure 4 shows both networks; the CVPR text and figure, and v4–v5, describe the following.

• Generator (SRResNet). A 9 × 9 convolution with 64 feature maps and a parametric ReLU, then $B = 16$ residual blocks, each of two 3 × 3 convolutions with 64 feature maps, batch normalization and a ParametricReLU (the activation of the PReLU paper), with an identity shortcut. A further convolution and batch normalization follow, and a long skip connection adds the output of the first layer, a residual connection around the whole body. Two upsampling stages, each a 3 × 3 convolution to 256 maps, a ×2 pixel shuffle and a PReLU, give 4×, and a final 9 × 9 convolution outputs three channels. The block layout follows Gross and Wilber’s ResNet study, as Johnson et al. did, and the upsampling layers are the sub-pixel convolutions of Shi et al. (2016), described in transposed convolution.
• Discriminator. Following the DCGAN guidelines of Radford et al., it uses LeakyReLU ($\alpha = 0.2$) and no max-pooling: eight 3 × 3 convolutional layers whose feature maps double from 64 to 512, as in VGG, with four stride-2 convolutions, one at each width, reducing the resolution (Figure 4), then dense layers of 1024 and 1 units and a sigmoid. It is trained against the generator in the minimax game of eq. 2, the formulation of the GAN paper.

Batch normalization statistics are frozen at test time so that the output depends only on the input (§3.2, from v2).

Perceptual loss

The loss SRGAN minimizes is a content loss plus a weighted adversarial loss (eq. 3):

$$l^{SR} = l^{SR}_{X} + 10^{-3}\, l^{SR}_{\mathrm{Gen}} .$$

The content loss $l^{SR}_X$ is either the pixel-wise MSE (eq. 4) or a VGG loss (eq. 5), the squared Euclidean distance between feature maps of a pretrained 19-layer VGG network, averaged over the spatial positions of the map:

$$l^{SR}_{VGG/i,j} = \frac{1}{W_{i,j}H_{i,j}} \sum_{x=1}^{W_{i,j}} \sum_{y=1}^{H_{i,j}} \Big( \phi_{i,j}(I^{HR})_{x,y} - \phi_{i,j}\big(G_{\theta_G}(I^{LR})\big)_{x,y} \Big)^2 .$$

Here $\phi_{i,j}$ is the feature map of the $j$-th convolution, after its ReLU, before the $i$-th max-pooling layer; the words “after activation” were added in v3. The paper uses two layers: $\phi_{2,2}$, low-level features (in the common VGG layer naming, relu2_2), and $\phi_{5,4}$, high-level features from the last convolutional block (relu5_4). Only arXiv v5 states how the terms were scaled: LR inputs in $[0, 1]$, HR images in $[-1, 1]$ so the MSE is computed on that range, and VGG feature maps divided by 12.75 to bring the VGG loss to a scale comparable with the MSE, equivalent to multiplying eq. 5 by about 0.006 (our check: $1/12.75^2 \approx 0.00615$). These sentences are absent from the CVPR copy and from v1–v4, so the relative scale of the two terms, and with it the meaning of the $10^{-3}$ weight, cannot be recovered from the proceedings alone (our reading).

The adversarial term is the non-saturating generator loss (eq. 6),

$$l^{SR}_{\mathrm{Gen}} = \sum_{n=1}^{N} -\log D_{\theta_D}\big(G_{\theta_G}(I^{LR})\big),$$

which the paper minimizes instead of $\log(1 - D)$ for better gradient behaviour, following Goodfellow et al.; the generative adversarial network article explains the difference.

Training

All networks were trained on an NVIDIA Tesla M40 GPU with a random sample of 350,000 ImageNet images, using mini-batches of 16 random 96 × 96 HR crops and Adam with $\beta_1 = 0.9$ (§3.2). SRResNet was trained for $10^6$ iterations at a learning rate of $10^{-4}$. SRGAN was initialized from the trained MSE SRResNet to avoid poor local optima and trained for $10^5$ iterations at $10^{-4}$ and $10^5$ more at $10^{-5}$, alternating one generator and one discriminator update ($k = 1$). The implementation used Theano and Lasagne. From v4 the paper adds that the images were in BGR channel order. The supplementary material in arXiv v3–v5 (Appendix A.2) shows the generator on Set14’s baboon image: after 20,000 iterations it had moved far from the SRResNet initialization, producing much high-frequency content including noise, and the output changed little over the last 50,000 to 100,000 iterations.

Results

All experiments are at 4×, a 16-fold reduction in pixels, on Set5, Set14 and BSD100, the test set of BSD300 from the Berkeley Segmentation Dataset (§3.1). PSNR and SSIM were computed on the luma channel after removing a 4-pixel border, with the daala package at commit 8d03668. Outputs of nearest neighbour, bicubic, SRCNN and SelfExSR came from the supplementary material of Huang et al. (2015) and DRCN’s from its authors; the paper does not say where the ESPCN outputs came from. Statistical tests were paired two-sided Wilcoxon signed-rank tests at $p < 0.05$. The authors warned that values computed with this shared framework might deviate slightly from those in the original papers.

The MOS study

Twenty-six raters gave each image an integer score from 1 (bad) to 5 (excellent) (§3.3). On Set5 and Set14 they rated 12 versions of each image: nearest neighbour, bicubic, SRCNN, SelfExSR, DRCN, ESPCN, SRResNet with MSE and with VGG22 loss, SRGAN with MSE, VGG22 and VGG54 content losses, and the original. On BSD100 three of the variants were left out, giving 9 versions. Each rater thus scored 1,128 images, presented in random order without labels; Appendix A.3 counts 29,328 ratings in all. Raters were calibrated on 20 images from the BSD300 training set, told that the nearest-neighbour version should score 1 and the original 5, and a pilot study with 26 raters on 10 BSD100 images, shown twice within a larger set, found good test–retest reliability. The image quality assessment article describes how such opinion scores are usually collected and validated.

Content losses (Table 1)

Table 1 compares the generator trained without and with the adversarial term, for different content losses, on Set5 and Set14. An asterisk marks a MOS significantly higher than the other losses in its category.

Set	Metric	SRResNet-MSE	SRResNet-VGG22	SRGAN-MSE	SRGAN-VGG22	SRGAN-VGG54
Set5	PSNR (dB)	32.05	30.51	30.64	29.84	29.40
Set5	SSIM	0.9019	0.8803	0.8701	0.8468	0.8472
Set5	MOS	3.37	3.46	3.77	3.78	3.58
Set14	PSNR (dB)	28.49	27.19	26.92	26.44	26.02
Set14	SSIM	0.8184	0.7807	0.7611	0.7518	0.7397
Set14	MOS	2.98	3.15*	3.43	3.57	3.72*

Every alternative to SRResNet-MSE has lower PSNR and SSIM on both sets, and SRGAN-VGG54 the lowest PSNR. On Set5 the authors found no significantly best loss, and the two SRGAN variants with the highest mean MOS there are not VGG54; on Set14 SRGAN-VGG54 was significantly better than every other variant. The paper attributes the minor artifacts it saw in some SRGAN-MSE outputs to competition between the MSE and adversarial terms, reports a trend of better texture with $\phi_{5,4}$ than with $\phi_{2,2}$, and from here calls SRGAN-VGG54 simply “SRGAN”. Since v4 the text adds that SRResNet-VGG22 was trained with an extra total variation loss weighted $2 \times 10^{-8}$; the CVPR copy and v3 do not mention it.

Comparison with other methods (Table 2)

Set	Metric	Nearest	Bicubic	SRCNN	SelfExSR	DRCN	ESPCN	SRResNet	SRGAN	HR
Set5	PSNR (dB)	26.26	28.43	30.07	30.33	31.52	30.76	32.05	29.40	∞
Set5	SSIM	0.7552	0.8211	0.8627	0.872	0.8938	0.8784	0.9019	0.8472	1
Set5	MOS	1.28	1.97	2.57	2.65	3.26	2.89	3.37	3.58	4.32
Set14	PSNR (dB)	24.64	25.99	27.18	27.45	28.02	27.66	28.49	26.02	∞
Set14	SSIM	0.7100	0.7486	0.7861	0.7972	0.8074	0.8004	0.8184	0.7397	1
Set14	MOS	1.20	1.80	2.26	2.34	2.84	2.52	2.98	3.72	4.32
BSD100	PSNR (dB)	25.02	25.94	26.68	26.83	27.21	27.02	27.58	25.16	∞
BSD100	SSIM	0.6606	0.6935	0.7291	0.7387	0.7493	0.7442	0.7620	0.6688	1
BSD100	MOS	1.11	1.47	1.87	1.89	2.12	2.01	2.29	3.56	4.46

SRResNet has the highest PSNR and SSIM on all three sets, and SRGAN the highest MOS short of the originals. On BSD100 all MOS differences were highly significant except SRCNN against SelfExSR, and Figure 5 shows the distribution of the 2,600 ratings per method (100 images × 26 raters). The table also shows what the text leaves implicit (our reading): on BSD100 SRGAN’s PSNR and SSIM are below bicubic interpolation’s, and on Set14 its PSNR roughly equals bicubic’s while its SSIM is lower, yet raters gave it more than twice bicubic’s mean score on both. Even the originals averaged 4.32 to 4.46, not 5.

The abstract (v3 onward) claims that SRGAN’s MOS is closer to the originals’ than to that of any state-of-the-art method. That holds on Set14 and BSD100 but not on Set5, where SRGAN’s 3.58 is 0.21 from SRResNet’s 3.37 and 0.74 from the originals’ 4.32 (our arithmetic); the supplement calls the Set5 results somewhat inconclusive because of the very small sample size and images with comparably little detail. The ranking against the other methods rests on MOS alone: no table shows SRGAN ahead on a computed metric.

Versions

The arXiv record has five versions; the CVF open-access page links to v5 and offers no separate supplementary file. The differences below come from our comparison of the texts.

Version	Date	Main changes
arXiv v1	15 Sep 2016	Nine authors. Generator with $B = 15$ blocks, ReLU, no long skip connection, and two “deconvolution layers (stride = 0.5)” after Shi et al. Loss: content loss + $10^{-3}$ adversarial loss + $2 \times 10^{-8}$ total variation regularizer. SRResNet trained on 50,000 ImageNet images; the GAN pretrained with the $\phi_{2,2}$ loss, which worked better than MSE pretraining, then trained on 350,000 images for about a day. Table 1: SRResNet against other methods, e.g. Set5 31.92 dB / 0.8998. Table 2: PSNR and SSIM of SRGAN-MSE, -VGG22 and -VGG54 on all three sets (Set5 VGG54 28.74 dB / 0.8435), captioned “NEEDS TO BE UPDATED”. No MOS; Figure 2 gives SRResNet 23.44 dB / 0.7777 and SRGAN 20.34 dB / 0.6562
arXiv v2	19 Sep 2016	Same results. Adds Bruna et al. and Dosovitskiy and Brox to the related work, the 4-pixel border crop and the daala commit, a link to result images, a pointer to an independent GAN super-resolution project on GitHub limited to faces, BN frozen at test time, and the caution that hallucinated detail may suit medical or surveillance uses less
arXiv v3	21 Nov 2016	Eleven authors (Cunningham and Acosta added). New method and results: $B = 16$, long skip connection, sub-pixel convolution layers, no total variation term, all networks on 350,000 images, SRGAN initialized from the MSE SRResNet, the $10^5 + 10^5$ iteration schedule. MOS study, Wilcoxon tests, nearest neighbour added, the present Tables 1 and 2 and Figure 2 values, and the supplementary Appendices A.1–A.6 (v1 and v2 had only an appendix of visual comparisons). Text and Figure 4 still show ReLU
arXiv v4	13 Apr 2017	ParametricReLU in the text and figure, LeakyReLU $\alpha = 0.2$, kernel sizes in Figure 4, BGR channel order, the TV note for SRResNet-VGG22, and the $10^{-3}$ weight restored in the §3.4 equation (v3 printed it there without the weight). Adds that text and structured scenes remain hard
arXiv v5	25 May 2017	Adds the input ranges and the 1/12.75 VGG rescaling; otherwise v4. The arXiv comment calls it the main paper plus some supplementary material and says the paper was accepted for oral presentation at CVPR
CVPR 2017	2017	Pages 105–114 (Crossref; the CVF copy is numbered 4681–4690). Main text as in v4–v5 with the same tables and figures, but without the TV note, the input and VGG scaling, the supplementary material and the sentences pointing to it

The table values did not change from v3 to the proceedings, although the text changed the generator’s activation from ReLU to PReLU in v4. The paper does not say whether the networks were retrained; unchanged numbers suggest that v4 corrected the description rather than the model (our inference). Readers who want the training scale and the supplement need arXiv v5; for v1-era results, the numbers are from a different generator and training schedule and should not be mixed with the CVPR tables.

Example: Why the Average Wins on PSNR

The paper’s Figure 3 argument can be checked exactly on a synthetic image whose ambiguity is known. A 128 × 128 image is the sum of smooth content (disks and bars, band-limited below the LR Nyquist frequency) and a fine random-phase grain between 0.15 and 0.45 cycles per pixel. Downsampling by 4 after an ideal low-pass filter keeps the smooth content exactly and removes the grain entirely, so every image “smooth content + any grain of the same statistics” is consistent with the LR input and equally plausible under this toy prior. The posterior mean, the estimate an MSE-trained network approaches, is then the smooth content alone. The script compares it with bicubic interpolation, with one plausible image (a stand-in for a GAN’s output: right statistics, invented detail), and with the true grain shifted by one pixel; “fine detail” is the energy above the LR Nyquist frequency relative to the truth.

import numpy as np
from scipy.ndimage import zoom
from skimage.metrics import structural_similarity  # scikit-image 0.25.2

N, S = 128, 4                                   # HR size, scale factor
fy, fx = np.meshgrid(np.fft.fftfreq(N), np.fft.fftfreq(N), indexing="ij")
f = np.hypot(fx, fy)                            # radial frequency, cycles per HR pixel
LR_NYQ = 0.5 / S                                # 0.125: the finest detail the LR grid can hold

def band(img, lo, hi):
    """Keep only spatial frequencies with lo <= |f| < hi (periodic image)."""
    return np.real(np.fft.ifft2(np.fft.fft2(img) * ((f >= lo) & (f < hi))))

def scene(rng):
    """Smooth content: disks and bars, band-limited below the LR Nyquist frequency."""
    y, x = np.mgrid[0:N, 0:N]
    img = np.full((N, N), 0.5)
    for _ in range(8):
        cx, cy, r = rng.uniform(0, N, 2).tolist() + [rng.uniform(8, 24)]
        img[(x - cx) ** 2 + (y - cy) ** 2 < r ** 2] = rng.uniform(0.25, 0.75)
    return band(img, 0, 0.12)

def texture(rng, sd=0.08):
    """Fine grain: random-phase noise between 0.15 and 0.45 cycles/pixel, above the LR limit."""
    t = band(rng.standard_normal((N, N)), 0.15, 0.45)
    return sd * t / t.std()

def degrade(hr):
    """Ideal low-pass at the LR Nyquist frequency, then keep every 4th pixel."""
    return band(hr, 0, LR_NYQ)[::S, ::S]

def fourier_upsample(lr):
    """Exact for content below the LR Nyquist frequency: zero-pad the spectrum."""
    F = np.fft.fftshift(np.fft.fft2(lr))
    p = (N - lr.shape[0]) // 2
    return np.real(np.fft.ifft2(np.fft.ifftshift(np.pad(F, p)))) * S * S

def scores(ref, out):
    psnr = 10 * np.log10(1 / np.mean((ref - out) ** 2))
    ssim = structural_similarity(ref, out, data_range=1, gaussian_weights=True, sigma=1.5,
                                 use_sample_covariance=False)
    hf = lambda im: np.sum(band(im, LR_NYQ, 1) ** 2)
    return psnr, ssim, hf(out) / hf(ref)          # last: fine-detail energy relative to truth

base = scene(np.random.default_rng(0))
truth_tex = texture(np.random.default_rng(1))
hr = base + truth_tex                            # the unknown high-resolution image
lr = degrade(hr)                                 # 32 x 32 observation
base_hat = fourier_upsample(lr)                  # recovers the smooth content exactly
print(f"LR keeps the texture? max |degrade(texture)| = {np.abs(degrade(truth_tex)).max():.1e}")
print(f"base recovered exactly? max error = {np.abs(base_hat - base).max():.1e}\n")

rng = np.random.default_rng(2)
samples = [base_hat + texture(rng) for _ in range(64)]   # plausible HR images: all fit the LR exactly
rows = {"bicubic (scipy zoom, order 3)": zoom(lr, S, order=3, mode="grid-wrap", grid_mode=True),
        "average of 64 plausible images": np.mean(samples, axis=0),
        "posterior mean (smooth content only)": base_hat,
        "one plausible image (new texture)": samples[0],
        "true texture shifted by 1 pixel": base_hat + np.roll(truth_tex, 1, axis=1)}
print(f"{'estimate':38s} {'PSNR dB':>8s} {'SSIM':>6s} {'fine detail':>12s}")
for name, out in rows.items():
    p, s, e = scores(hr, out)
    print(f"{name:38s} {p:8.2f} {s:6.3f} {e:12.2f}")

print("\nAveraging k plausible images (seeds 0-9 for the samples): PSNR range, mean fine detail")
for k in [1, 2, 4, 16, 64]:
    ps, es = [], []
    for seed in range(10):
        r = np.random.default_rng(100 + seed)
        out = np.mean([base_hat + texture(r) for _ in range(k)], axis=0)
        p, _, e = scores(hr, out)
        ps.append(p); es.append(e)
    print(f"k = {k:2d}: {min(ps):.2f}-{max(ps):.2f} dB, fine detail {np.mean(es):.2f}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; under a second, identical over two runs):

LR keeps the texture? max |degrade(texture)| = 1.7e-17
base recovered exactly? max error = 6.7e-16

estimate                                PSNR dB   SSIM  fine detail
bicubic (scipy zoom, order 3)             21.44  0.187         0.00
average of 64 plausible images            21.89  0.208         0.02
posterior mean (smooth content only)      21.94  0.209         0.00
one plausible image (new texture)         18.93  0.123         1.00
true texture shifted by 1 pixel           19.88  0.290         1.00

Averaging k plausible images (seeds 0-9 for the samples): PSNR range, mean fine detail
k =  1: 18.91-18.99 dB, fine detail 1.00
k =  2: 20.14-20.25 dB, fine detail 0.50
k =  4: 20.93-21.02 dB, fine detail 0.25
k = 16: 21.66-21.72 dB, fine detail 0.06
k = 64: 21.86-21.90 dB, fine detail 0.02

The smooth answer wins on PSNR. The posterior mean has no grain at all and the best PSNR; a plausible image with the right statistics scores 3 dB lower. That gap is not a property of this seed: when the grain is independent of the input, a sample from the posterior has on average twice the squared error of the posterior mean, $10 \log_{10} 2 \approx 3.01$ dB less PSNR (our arithmetic), and the ten sample seeds stay within 18.91 to 18.99 dB. Averaging $k$ plausible images walks from one end to the other: each doubling of $k$ halves the fine-detail energy and raises PSNR, exactly the pixel-wise averaging of Figure 3. Bicubic interpolation lands near the mean because it, too, adds no detail.

A shift is punished like a wrong texture. Putting the true grain one pixel off costs 2 dB of PSNR relative to the blank answer, although every structure in it is real. SSIM disagrees here, ranking the shifted grain above the mean, because its local variance and contrast terms reward texture of the right strength; SSIM, too, punishes the plausible image with new grain. In our reading, neither metric rewards what the paper’s raters responded to, detail of the right kind, which this toy measures only through its fine-detail column. SRGAN’s lower PSNR in Table 2 is what this example predicts for a network that produces plausible detail instead of the average, though the example cannot say whether that detail looks right; that was the job of the MOS study.

Impact

SRGAN became a reference point for super-resolution aimed at appearance rather than fidelity. ESRGAN (Wang et al., 2018) calls it a seminal work and a milestone on the way to visually pleasing results, keeps its high-level architecture, and revises each component (see What Came After). SRResNet, its MSE-trained generator, became a baseline in its own right: EDSR (Lim et al., 2017) starts from it, compares their residual blocks in its Figure 2, and lists a reproduced SRResNet in its tables.

The paper’s evaluation argument had a longer reach. Blau and Michaeli (2018) evaluated 16 super-resolution methods on BSD100, including SRResNet and SRGAN variants, and placed SRGAN among the first methods, with EnhanceNet, Johnson et al. and others, to reach the high perceptual-quality region of their perception–distortion plane. They also found every full-reference measure they tested inconsistent with the human opinion scores that had rated SRGAN highly, while no-reference measures agreed with them, and recommended reporting one measure of each kind. The perception–distortion trade-off they proved, stated in the PSNR article, gave a theoretical form to the gap between Table 2’s PSNR and MOS columns. Learned metrics such as LPIPS, published at CVPR 2018 like Blau and Michaeli’s paper, were developed to measure perceptual similarity better than PSNR and SSIM.

Limitations

The authors named several limits (§4):

• Fidelity. The ideal loss depends on the application; methods that hallucinate detail may be less suited to medical or surveillance use.
• Speed. The model was not optimized for real-time video, unlike ESPCN, though preliminary experiments suggested shallower networks as efficient alternatives.
• Depth and stability. Deeper SRResNets ($B > 16$) improved accuracy at higher cost (Appendix A.1 of the arXiv versions), but deeper SRGAN variants were increasingly hard to train because high-frequency artifacts appeared.
• Content. Perceptually convincing reconstruction of text and structured scenes was left to future work (from v4).

Further limits are visible in the paper itself (our reading):

• A small human study. The ranking rests on 26 raters, whose background and recruitment the paper does not describe, and on Set5 the content-loss comparison was not significant. It supports the ordering on three small benchmarks at one scale rather than a general claim.
• One scale factor, one degradation. Every result is 4× on synthetically downsampled images, described inconsistently as Gaussian-filtered or bicubic.
• Underspecified scaling. The weighting that makes $10^{-3}$ meaningful, the VGG rescaling, appears only in arXiv v5.
• Related-work imprecision. The paper states that Johnson et al. and Bruna et al. formulated their loss on VGG19 features; Johnson et al. (2016) used VGG-16, at relu2_2 for super-resolution.

Later work located specific weaknesses. Wang et al. (2018) observed that the batch normalization layers in SRGAN-style generators are more likely to introduce artifacts in deeper GAN-trained networks, and argued that the VGG features SRGAN compares after activation are very sparse in deep layers, giving weak supervision and output brightness inconsistent with the ground truth.

What Came After

ESRGAN (Wang et al., 2018) revised all three parts of SRGAN: residual-in-residual dense blocks without batch normalization, a relativistic discriminator that judges whether one image is more realistic than another, and a VGG loss on features before activation. It won region 3 of the PIRM 2018 perceptual super-resolution challenge with the best perceptual index, and it interpolates between the weights of a PSNR-oriented and a GAN-trained network to trade smoothness for texture. EDSR took the opposite branch from SRResNet, dropping batch normalization for higher PSNR. Real-world degradation pipelines such as BSRGAN and Real-ESRGAN, and later diffusion-based methods that sample a high-resolution image rather than predict one, carried the perceptual line further; the super-resolution article covers these methods and the evaluation practice that grew from the paper’s argument.