Visualizing and Understanding Convolutional Networks

“Visualizing and Understanding Convolutional Networks” is a paper by Matthew D. Zeiler and Rob Fergus of New York University, published at the European Conference on Computer Vision (ECCV 2014) after three arXiv versions in November 2013. It asked what the layers of an ImageNet-trained convolutional network respond to and whether knowing that helps build better networks. Its tool was a deconvolutional network, or deconvnet, that maps a single activation back to pixel space; its check was an occlusion experiment that slides a grey square over the image and records the classifier’s output. Inspecting the first two layers of AlexNet this way, the authors changed its first layer to $7 \times 7$ filters at stride 2, and the resulting network, often called the ZF net, beat AlexNet’s single-model validation error. The paper also ablated the network’s layers and showed that its features, with only a new softmax classifier, beat the state-of-the-art results the authors compared against on Caltech-101 and Caltech-256. The European Computer Vision Association lists it, under the arXiv v1 title, as an ECCV 2014 Best Paper Honorable Mention. Explanation methods for vision networks in general, including those that followed this paper, are the subject of interpretability of neural networks; this article covers the publication and its versions.

Problem

By the authors’ account, convolutional networks had done well on digit classification and face detection since LeCun et al. introduced them in the early 1990s, and in the 18 months before the paper they had led on harder benchmarks: Cireşan et al. on NORB and CIFAR-10, Krizhevsky et al. on the ImageNet 2012 classification benchmark with 16.4% error against 26.1% for the second-placed entry, and Girshick et al. on PASCAL VOC detection. The authors credited larger labeled training sets, GPU implementations and regularization such as dropout. Nobody, they wrote, understood how these models worked internally, which left the design of better models to trial and error.

Earlier visualizations were mostly limited to the first layer, whose filters can be shown as pixels directly; higher layers need other methods. Erhan et al. (2009) found each unit’s optimal stimulus by gradient descent in image space, which in the authors’ view needs careful initialization and says nothing about the unit’s invariances; Le et al. (2010), extending Berkes and Wiskott, approximated invariances with the Hessian around the optimal response, which the authors judged too simple for the complex invariances of higher layers. Zeiler and Fergus wanted a non-parametric view: which patterns from real images drive a given feature map. The paper also framed its transfer experiments as a supervised alternative to the unsupervised pre-training of Hinton et al. (2006) and others.

Contribution

• Deconvnet visualization. A procedure that reconstructs, from one activation in any convolutional layer, the input pattern that caused it, reusing the trained network’s own pooling switches and filters.
• Diagnosis and redesign. Visualizations of AlexNet’s first two layers revealed filters with little mid-frequency coverage and aliasing, and the redesigned first layer improved ImageNet accuracy (in arXiv v3 and ECCV; see Versions).
• Occlusion sensitivity. Classifier output and feature activity as a function of where a grey square hides the image, used both to show that the classifier depends on the object and to check the visualizations.
• Ablations of layer count and size, and transfer of the ImageNet features to Caltech-101, Caltech-256 and PASCAL VOC 2012 by retraining only the classifier, with a layer-by-layer analysis of how discriminative the features are.
• In the arXiv versions only, analyses of invariance to translation, scale and rotation, and of implicit correspondence between object parts.

Versions

The paper exists in three arXiv versions and the ECCV version, and they differ more than revisions usually do. The dates are from the arXiv submission history; the comparison is ours.

Version	Date	Title and main content
arXiv v1	12 Nov 2013	“Visualizing and Understanding Convolutional Neural Networks”; citations print as “(?)” and the reference list is missing
arXiv v2	13 Nov 2013	Same text as v1 with the bibliography restored
arXiv v3	28 Nov 2013	Shorter title; rewritten around a new architecture with a $7 \times 7$, stride-2 first layer chosen from the visualizations; new architecture-selection section and filter renormalization
ECCV	LNCS 8689, pp. 818–833, 2014	v3 without the invariance and correspondence analyses; adds the comparison of the deconvnet with gradients, Simonyan et al. and Girshick et al. in related work, and new rows (DeCAF, Howard, Clarifai) and an Oquab et al. column in the results tables

In v1 and v2 the network keeps AlexNet’s $11 \times 11$, stride-4 first layer and widens convolutional layers 3, 4 and 5 to 512, 1,024 and 512 maps. The text there credits the gain to the ablation study, which found that wider middle layers help, and the abstract says the ablation enabled the better architectures. In v3 and ECCV the abstract says instead that the visualizations, used diagnostically, found the better architectures, and the wider layers survive as one variant in an ensemble. Other numbers change between the versions: the validation set is 65,000 images in v1 and v2 and 50,000 in v3 and ECCV, and training took about 24 days in v1 and v2 against 12 days in v3 and ECCV, both for 70 epochs on one GTX 580. Figure and table numbers below are the ECCV version’s unless stated; arXiv v3 has two more figures and one more table, so its numbers run one or two higher.

Method

The deconvnet

The deconvnet comes from Zeiler, Taylor and Fergus (2011), where, the paper says, it was proposed for unsupervised learning. Here it is not trained at all; it is attached to an already trained convnet as a probe, with one deconvnet layer for each convnet layer (Fig. 1). To examine one activation, the image is passed forward, every other activation in that layer is set to zero, and the result is passed down through three operations per layer until it reaches the pixels:

1. Unpooling. Max pooling cannot be inverted, but the forward pass can record the location of each pooling region’s maximum in “switch” variables. Unpooling puts each value from the layer above back at its switch location, leaving the rest of the region at zero.
2. Rectification. The signal is passed through a ReLU, so the reconstruction stays positive, as feature maps after a ReLU are.
3. Filtering. The signal is convolved with transposed versions of the layer’s own trained filters, which in practice means flipping each filter vertically and horizontally (the same operation as a transposed convolution). Contrast normalization is skipped on the way down (stated in the ECCV version).

Because the switches come from the image being examined, the reconstruction looks like a piece of that image, weighted by how much each structure contributed to the activation. The authors stress that it is not a sample from the model, since nothing generative is involved. For Fig. 2 they projected, for a random subset of feature maps in layers 2 to 5, the nine strongest activations over the validation set, each projected separately and shown next to the image patch that produced it.

Relation to the gradient

The ECCV version adds that the procedure resembles backpropagating a single strong activation, that is, computing $\partial h / \partial X_n$ for the feature-map element $h$ and the input image $X_n$, except that the ReLU is imposed independently and contrast normalization is left out. A footnote reports that rectifying with the binary mask of the forward ReLU, which is what the gradient does, gave visualizations that were significantly less clear.

Simonyan, Vedaldi and Zisserman (2014), citing the arXiv version, worked out the correspondence layer by layer. The gradient through a convolution is a convolution with the flipped kernel, as in the deconvnet’s filtering step, and the gradient through max pooling routes the signal to the maximum’s location, which is the switch. Only the ReLU differs. With $R_{n+1}$ the signal arriving from above and $X_n$ the ReLU’s forward input,

$$\text{gradient: } R_n = R_{n+1} \cdot \mathbb{1}(X_n > 0), \qquad \text{deconvnet: } R_n = R_{n+1} \cdot \mathbb{1}(R_{n+1} > 0).$$

They concluded that, apart from the ReLU, the deconvnet reconstruction equals the gradient, and that gradient-based visualization generalizes it to any layer, including the fully connected class scores they visualized. Zeiler and Fergus project only from convolutional layers; their related-work section contrasts this with Simonyan et al.’s projections from the fully connected layers.

Architecture and training

The network of Fig. 3 (arXiv v3 and ECCV) follows Krizhevsky et al. (2012) with three groups of changes. Krizhevsky’s sparse connections, a consequence of splitting the model across two GPUs, become dense; the text places them in layers 3, 4 and 5, although in AlexNet the restricted layers are 2, 4 and 5 (see AlexNet). The first layer uses $7 \times 7$ filters at stride 2 instead of $11 \times 11$ at stride 4 (§4.1), and the figure, like §5.1, gives layer 2 a stride of 2 as well. And because visualizing the first-layer filters during training showed a few of them dominating, every convolutional filter whose RMS value exceeds $10^{-1}$ is rescaled to that value, which the authors call crucial in the first layer, whose inputs lie roughly in $[-128, 128]$.

Layer	Filters	Size, stride	Output after pooling
1	96	$7 \times 7$, 2	$55 \times 55$ (ReLU, $3 \times 3$ max pool at stride 2, contrast normalization)
2	256	$5 \times 5$, 2	$13 \times 13$ (same operations)
3	384	$3 \times 3$, 1	$13 \times 13$
4	384	$3 \times 3$, 1	$13 \times 13$
5	256	$3 \times 3$, 1	$6 \times 6$ ($3 \times 3$ max pool at stride 2)
6, 7	4,096 units each	fully connected	input $6 \cdot 6 \cdot 256 = 9{,}216$
Output	$C$-way softmax

Training used the ImageNet 2012 training set (1.3 million images, 1,000 classes). Each image was resized so its shorter side was 256, center-cropped to $256 \times 256$ and mean-subtracted per pixel, and ten $224 \times 224$ sub-crops (corners and center, with and without horizontal flips) were used. Stochastic gradient descent ran with mini-batches of 128, learning rate $10^{-2}$ annealed by hand when validation error plateaued, and momentum 0.9; dropout at rate 0.5 was applied in layers 6 and 7; all weights were initialized to $10^{-2}$ and biases to 0. Training stopped after 70 epochs, about 12 days on one GTX 580, with code based on Krizhevsky et al.’s.

Results

What the layers respond to

In Fig. 2, layer 2 responds to corners and other conjunctions of edges and color; layer 3 to textures such as mesh patterns and text; layer 4 to more class-specific parts such as dog faces and birds’ legs; and layer 5 to whole objects in varied poses, such as keyboards and dogs. The image patches vary more than the projections, which show only the discriminative structure: in one layer-5 map the patches seemed to have little in common until the projections showed that the map responded to grass in the background. The caption notes strong grouping within each map, more invariance at higher layers and exaggeration of discriminative parts such as dogs’ eyes and noses. The authors read the features as compositional, increasingly invariant and increasingly class-discriminative with depth, far from random, uninterpretable patterns.

Fig. 4 follows the strongest activation of several maps over all training examples at epochs 1, 2, 5, 10, 20, 30, 40 and 64. The lower layers settle within a few epochs, while the upper layers develop only after 40 to 50 epochs, which the authors take as a reason to train until fully converged. Sudden jumps in appearance mark a change of the image that gives the strongest activation.

Architecture selection

Visualized, AlexNet’s first-layer filters were a mix of extremely high and low frequencies with little in the middle, and its second layer showed aliasing artifacts that the authors attribute to the stride-4 first layer (§4.1; on aliasing, see sampling and aliasing). With $7 \times 7$ filters at stride 2, the first layer had more distinctive and fewer “dead” filters, and the second layer was cleaner, with no visible aliasing (Fig. 5). In the ECCV version, the Fig. 5 caption describes five panels, (a) to (e), copied from arXiv v3, but the figure prints four, and the text and caption assign the panels differently.

Invariance (arXiv only)

The arXiv versions (v3 §4, Fig. 5; v1 and v2 §4.2, Fig. 4) translated five images vertically by up to 60 pixels, scaled them by up to 1.8 times and rotated them from 0° to 350°, and plotted the Euclidean distance between the original and transformed feature vectors at layers 1 and 7, and the probability of the true class. Small transformations changed layer 1 dramatically and layer 7 less, roughly linearly for translation and scaling. The output was stable to translation and scaling but not to rotation; v3 adds an exception for objects with rotational symmetry, such as an entertainment center. The ECCV version drops this analysis.

Occlusion sensitivity

Fig. 6 slides a grey square over three test images and records, for each position, the summed activity of the layer-5 feature map that responds most strongly to the unoccluded image, the probability of the correct class and the most probable class; it also shows projections of that map. For a Pomeranian, the strongest map responds to the dog’s face, and covering the face lowers both that map’s activity and the Pomeranian probability; with the face hidden but a ball visible, the network answers “tennis ball”. For a car wheel, the strongest layer-5 map responds to text on the car, but the classifier is most sensitive to the wheel. For an Afghan hound in an image with several objects, the strongest map picks out faces, but the classifier is sensitive to the dog, which the authors attribute to its use of many feature maps. The authors drew two conclusions: the classifier localizes the object rather than relying only on scene context, and the drop in a map’s activity when the square covers the region its projection shows validates the projections in Figs. 2 and 4. The occlusion method in later use is described in interpretability of neural networks.

Correspondence (arXiv only)

The arXiv versions (v3 §4.3; v1 and v2 §4.4) asked whether the network implicitly matches object parts across images. In five frontal dog images, the same part (right eye, left eye or nose) or a random region was masked. With $\boldsymbol{\epsilon}_i^l = \mathbf{x}_i^l - \tilde{\mathbf{x}}_i^l$ the change of image $i$’s layer-$l$ feature vector,

$$\Delta^l = \sum_{i \neq j} H\!\left(\operatorname{sign}(\boldsymbol{\epsilon}_i^l), \operatorname{sign}(\boldsymbol{\epsilon}_j^l)\right),$$

with $H$ the Hamming distance, so a lower value means masking the part changes the features more consistently. At layer 5 the eyes and nose scored 0.067 to 0.079 against 0.107 for random regions; at layer 7 all four were 0.068 to 0.073, which the authors suggest may come from upper layers discriminating between dog breeds. The table’s values are fractions headed “Mean Feature Sign Change”, so they must be normalized in a way the formula does not state.

ImageNet 2012

Table 1 gives top-1 and top-5 error on the ILSVRC 2012 validation set and top-5 error on its test set, in percent (see accuracy for top-$k$ error); selected rows are shown below. The ECCV version adds the DeCAF, Howard and Clarifai rows and titles the table “ImageNet 2012/2013”.

Model	Val top-1	Val top-5	Test top-5
Krizhevsky et al., 1 convnet	40.7	18.2	–
Krizhevsky et al., 5 convnets	38.1	16.4	16.4
Replication of Krizhevsky et al., 1 convnet	40.5	18.1	–
1 convnet as per Fig. 3	38.4	16.5	–
5 convnets as per Fig. 3, (a)	36.7	15.3	15.3
1 convnet, layers 3–5 with 512, 1,024, 512 maps, (b)	37.5	16.0	16.1
6 convnets, (a) and (b)	36.0	14.7	14.8
Howard (ECCV only)	–	–	13.5
Clarifai (ECCV only)	–	–	11.7

The replication came within 0.1 points of Krizhevsky et al.’s single-network top-5 validation error. The text says the new network beats that single model by 1.7% in test top-5 error, but the table gives no test error for either single model: 1.7 is the validation difference (18.2 against 16.5). The only single model with a test error is variant (b), at 16.1%. In v1 and v2 the claim was 1.8% on validation, and their table did list a single-model test error. The six-network ensemble reached 14.8% test error; v3 called this the best published performance on the dataset, with a footnote that the ImageNet 2013 competition had surpassed it, while ECCV calls it close to Howard’s data-augmentation approach and some way short of the 2013 winner. The introduction gives the second-placed 2012 entry as 26.1%, while Table 1 lists it (Gunji et al.) at 26.2%; Russakovsky et al. (2015) give 26.17%.

ILSVRC 2013

The archived official results page for ILSVRC 2013 lists classification entries by error, without ranks, and shades those trained with outside data. Among entries without outside data, the lowest error is Clarifai’s 0.11743, an average of several models, followed by NUS at 0.12953 and by team ZF at 0.13511 for five models of four architectures; ZF’s other four entries range from 0.13748 to 0.14079. Two Clarifai entries with outside data reached 0.11197 and 0.11537. The page’s team information lists ZF as Matthew D. Zeiler and Rob Fergus of New York University, with architectures chosen with the help of deconvnet visualizations, and Clarifai as Matthew Zeiler of Clarifai, whose architecture was chosen with a visualization technique based on the same deconvolutional networks. Russakovsky et al. (2015) name Clarifai the winner of the 2013 classification task, noting that its architectures were chosen with Zeiler and Fergus’s visualization technique, and list ZF at 13.5%. The paper does not report the ZF team’s entries or explain its relation to Clarifai; its Table 1 cites Clarifai’s 11.7% with a reference to “Zeiler, M.: Clarifai (2013)” and the results page.

Layer ablations

Table 2 retrains each variant from scratch; the errors below are top-5 on validation. In Krizhevsky et al.’s architecture, removing the fully connected layer 7 changed it from 18.1% to 18.4%, removing layers 6 and 7 raised it to 22.4%, and removing convolutional layers 3 and 4 raised it to 22.1%. Removing all four left a network of four layers with 50.1%. The authors found the small cost of dropping the fully connected layers surprising, since they hold most of the parameters, and concluded that overall depth, rather than any single section, matters. In their own network, 2,048 or 8,192 units in layers 6 and 7 gave 17.6% and 17.0% against 16.5%, wider convolutional layers 3 to 5 gave 16.0%, and widening both gave 16.9% with a training top-1 error of 10.0%, which they call overfitting. AlexNet discusses what these numbers say about that architecture.

Transfer to other datasets

For transfer, layers 1 to 7 of the ImageNet network were kept fixed and a new softmax was trained on the target training set; a second strategy trained the same architecture from random weights on the target data. Using normalized correlation, the authors found 44 of Caltech-101’s 9,144 images and 243 of Caltech-256’s 30,607 images in their ImageNet training set, removed them and retrained the ImageNet models.

• Caltech-101 (Table 3; mean per-class accuracy over five splits, up to 50 test images per class). The pretrained features reached 83.8 ± 0.5% with 15 and 86.5 ± 0.5% with 30 training images per class, against 46.5% for the network trained from scratch with 30. The text says this beats Bo et al.’s result by 2.2%, but the table lists Bo et al. at 81.4%, 5.1 points lower; 2.2 is the margin over Yang et al.’s 84.3%. The ECCV version adds that the result agrees with Donahue et al.’s 86.1%.
• Caltech-256 (Table 4). With 60 training images per class, 74.2 ± 0.3% against 55.2% for Bo et al.; with 6 per class, Fig. 7 shows the pretrained network already above Bo et al.’s best, which used ten times as many.
• PASCAL VOC 2012 (Table 5). A 20-way softmax trained on the train and validation images, which the authors call not ideal because their model makes one exclusive prediction per image while PASCAL images contain several objects. The mean was 79.0, 3.2 points below the best competition entry (Yan et al., 82.2), but higher on five classes; the ECCV version adds Oquab et al.’s 82.8 (their paper). The table labels the per-class values “Acc %”, although PASCAL VOC classification is scored by average precision.
• Features by layer (Table 6). A linear SVM or softmax on the features of one layer improves steadily from layer 1 (44.8% on Caltech-101, 24.6% on Caltech-256) to layer 5. The text says the best results use all layers, but on Caltech-101 the SVM on layer 5 (86.2%) beats both classifiers on layer 7 (85.5% and 85.4%); on Caltech-256 layer 7 is best (72.6%).

The authors attributed the weaker PASCAL result to dataset bias between ImageNet’s object images and PASCAL’s full scenes, and suggested that a loss allowing several objects per image would help and would extend the network to detection.

Worked Example: Deconvnet, Gradient and Guided Backpropagation

The example trains a small network on rendered shapes and projects the strongest activation of its third convolutional layer back to the pixels with three ReLU rules: the true gradient, the deconvnet’s rule, and guided backpropagation, which applies both masks. It first checks that a deconvnet written out as in §2.1 (unpool with the recorded switches, rectify, filter with the transposed trained filters) equals a backward pass in which only the ReLU’s rule is replaced. It then measures, over 60 probe images, the Pearson correlation of each projection with the image inside the unit’s receptive field, for the trained network and for the same architecture with fresh random weights, and the correlation between the two networks’ projections. Three training seeds show the spread. It needs PyTorch (the output is from version 2.14.1) and runs in about 11 seconds on one CPU thread.

import numpy as np
import torch
from torch import nn
F = nn.functional
torch.set_num_threads(1)                    # one thread: identical output on every run

def make_images(n, seed):
    """32x32 noisy images of a disc, a square or a triangle (classes 0, 1, 2) at random
    positions and sizes."""
    rng = np.random.default_rng(seed)
    yy, xx = np.mgrid[:32, :32]
    X, Y = [], []
    for i in range(n):
        label = i % 3
        r = rng.uniform(5, 8)
        cy, cx = rng.uniform(r + 1, 31 - r, size=2)
        if label == 0:
            obj = (yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2
        elif label == 1:
            obj = np.maximum(abs(yy - cy), abs(xx - cx)) <= 0.886 * r
        else:                                              # upward-pointing triangle
            obj = (yy <= cy + r) & (abs(xx - cx) <= (yy - (cy - r)) * 0.6)
        fg, bg = rng.uniform(0.6, 1.0), rng.uniform(0.0, 0.3)
        X.append(bg + (fg - bg) * obj + rng.normal(0, 0.15, (32, 32)))
        Y.append(label)
    return torch.tensor(np.array(X), dtype=torch.float32).unsqueeze(1), torch.tensor(Y)

class Net(nn.Module):
    """conv-relu-maxpool, conv-relu-maxpool, conv-relu (layer 3), global average pool, linear."""
    def __init__(self):
        super().__init__()
        self.c1 = nn.Conv2d(1, 16, 5, padding=2)
        self.c2 = nn.Conv2d(16, 32, 3, padding=1)
        self.c3 = nn.Conv2d(32, 32, 3, padding=1)
        self.fc = nn.Linear(32, 3)
    def layer3(self, x, relu=F.relu):
        x = F.max_pool2d(relu(self.c1(x)), 2)
        x = F.max_pool2d(relu(self.c2(x)), 2)
        return relu(self.c3(x))
    def forward(self, x):
        return self.fc(self.layer3(x).mean((2, 3)))

def train(seed, X, y, steps=400):
    torch.manual_seed(seed)
    net = Net()
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    g = torch.Generator().manual_seed(seed)
    for _ in range(steps):
        idx = torch.randint(len(y), (32,), generator=g)
        loss = F.cross_entropy(net(X[idx]), y[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    return net

# ---- three ways back through a ReLU; pooling and convolution are handled by autograd ----------
class BackReLU(torch.autograd.Function):
    """Forward: max(x, 0). Backward by rule: 'gradient' masks by the forward input (true gradient),
    'deconvnet' rectifies the signal coming down, 'guided' does both (Springenberg et al.)."""
    @staticmethod
    def forward(ctx, x, rule):
        ctx.save_for_backward(x); ctx.rule = rule
        return x.clamp(min=0)
    @staticmethod
    def backward(ctx, g):
        x, = ctx.saved_tensors
        if ctx.rule == "gradient":
            return g * (x > 0), None
        if ctx.rule == "deconvnet":
            return g.clamp(min=0), None
        return g.clamp(min=0) * (x > 0), None

def strongest(net, X):
    """Index of the strongest layer-3 activation (map, row, col) for each image."""
    with torch.no_grad():
        a = net.layer3(X)
    return a.flatten(1).argmax(1)

def project(net, X, rule, idx):
    """Keep one layer-3 activation per image, zero the rest, and map it back to pixels."""
    X = X.clone().requires_grad_(True)
    a = net.layer3(X, relu=lambda t: BackReLU.apply(t, rule)).flatten(1)
    seed = torch.zeros_like(a)
    seed[torch.arange(len(X)), idx] = a[torch.arange(len(X)), idx].detach()
    a.backward(seed)
    return X.grad[:, 0]

def deconvnet_by_hand(net, X, idx):
    """Zeiler & Fergus Sec. 2.1 written out: unpool with switches, rectify, filter with the
    transposed (flipped) filters, layer by layer down to pixels. Biases are not used."""
    with torch.no_grad():
        r1 = F.relu(net.c1(X)); p1, s1 = F.max_pool2d(r1, 2, return_indices=True)
        r2 = F.relu(net.c2(p1)); p2, s2 = F.max_pool2d(r2, 2, return_indices=True)
        r3 = F.relu(net.c3(p2)).flatten(1)
        R = torch.zeros_like(r3)
        R[torch.arange(len(X)), idx] = r3[torch.arange(len(X)), idx]
        R = R.view(-1, 32, 8, 8)
        R = F.conv_transpose2d(F.relu(R), net.c3.weight, padding=1)               # layer 3
        R = F.conv_transpose2d(F.relu(F.max_unpool2d(R, s2, 2)), net.c2.weight, padding=1)
        R = F.conv_transpose2d(F.relu(F.max_unpool2d(R, s1, 2)), net.c1.weight, padding=2)
    return R[:, 0]

def corr(A, B):
    """Mean Pearson correlation of paired maps over the pixels where the first map is nonzero
    (the unit's receptive field); B may be the images themselves."""
    out = []
    for a, b in zip(A, B):
        s = a != 0
        out.append(np.corrcoef(a[s].numpy(), b[s].numpy())[0, 1])
    return float(np.mean(out))

Xtr, ytr = make_images(1500, seed=0)
Xte, yte = make_images(600, seed=1)
Xp, _ = make_images(60, seed=2)                     # probe images for the projections
print(f"torch {torch.__version__}")
for seed in range(3):
    net = train(seed, Xtr, ytr)
    torch.manual_seed(100 + seed); rand = Net()          # same architecture, fresh weights
    with torch.no_grad():
        acc = (net(Xte).argmax(1) == yte).float().mean().item()
        a_rand = rand.layer3(Xp).flatten(2)
    idx = strongest(net, Xp)                             # trained net: strongest activation
    pos = idx % 64                                       # random net: strongest map, same place
    idx_r = a_rand[torch.arange(len(Xp)), :, pos].argmax(1) * 64 + pos
    diff = (deconvnet_by_hand(net, Xp, idx) - project(net, Xp, "deconvnet", idx)).abs().max()
    print(f"\nseed {seed}: test accuracy {acc:.3f}; "
          f"deconvnet by hand vs modified backward pass, max |difference| {diff:.1e}")
    print("  rule        corr. with image: trained  random   corr. trained vs random")
    for rule in ["gradient", "deconvnet", "guided"]:
        m, mr = project(net, Xp, rule, idx), project(rand, Xp, rule, idx_r)
        print(f"  {rule:10s}  {corr(m, Xp[:, 0]):+24.2f}  {corr(mr, Xp[:, 0]):+6.2f}"
              f"   {corr(m, mr):+.2f}")

Output:

torch 2.14.1

seed 0: test accuracy 0.993; deconvnet by hand vs modified backward pass, max |difference| 0.0e+00
  rule        corr. with image: trained  random   corr. trained vs random
  gradient                       +0.29   +0.07   +0.02
  deconvnet                      +0.39   +0.21   +0.29
  guided                         +0.48   +0.55   +0.42

seed 1: test accuracy 0.997; deconvnet by hand vs modified backward pass, max |difference| 0.0e+00
  rule        corr. with image: trained  random   corr. trained vs random
  gradient                       +0.24   +0.14   +0.05
  deconvnet                      +0.49   +0.38   +0.29
  guided                         +0.52   +0.53   +0.40

seed 2: test accuracy 0.992; deconvnet by hand vs modified backward pass, max |difference| 0.0e+00
  rule        corr. with image: trained  random   corr. trained vs random
  gradient                       +0.29   +0.17   +0.11
  deconvnet                      +0.46   +0.35   +0.37
  guided                         +0.50   +0.57   +0.54

The zero difference confirms Simonyan et al.’s analysis in code: unpooling with switches is the gradient of max pooling, and filtering with the flipped filters is the gradient of the convolution, so the deconvnet is the backward pass with only the ReLU changed.

In the trained networks, the deconvnet and guided projections resemble the image more than the gradient does (correlations of 0.39–0.52 against 0.24–0.29). In our reading, this is a measurable version of the authors’ footnote that the forward-ReLU mask gave less clear pictures. The random networks show where that resemblance comes from. The gradient of a random network barely resembles the image (0.07–0.17) or the trained network’s gradient (0.02–0.11), so it depends on the learned weights. The deconvnet projection of a random network still correlates 0.21–0.38 with the image and 0.29–0.37 with the trained projection, and guided backpropagation keeps all of its resemblance (0.53–0.57, at least as high as when trained). Part of what makes these projections look meaningful is supplied by the image itself, through the switches and the ReLU masks, rather than by what the network learned. Nie et al. (2018) found the same effect in VGG-16 and explained it theoretically; this tiny network with three seeds only illustrates it.

Impact

Fig. 2’s progression from edges to textures, parts and whole objects is the description of the feature hierarchy that the articles on feature maps and convolutional networks cite. Occluding the input and watching the output became one of the perturbation methods for explaining classifiers.

The architectural changes carried into competition and later networks. Russakovsky et al. (2015) record that the ILSVRC 2013 classification winner, Clarifai, chose its architectures with the paper’s visualization technique. Simonyan and Zisserman (2015) describe the top-performing ILSVRC 2013 entries, citing Zeiler and Fergus and OverFeat, as using $7 \times 7$ first-layer filters at stride 2, and went further with $3 \times 3$ filters throughout (see the VGG paper). Faster R-CNN (Ren et al., 2015) used the Zeiler and Fergus model, with five shareable convolutional layers, as one of its two backbones.

The transfer results appeared alongside DeCAF and the work of Girshick et al. and Oquab et al., which the ECCV version cites as concurrent; how reusing ImageNet features became standard practice is told in transfer learning. The deconvnet itself became the starting point of the backpropagation-based methods: Simonyan et al. (2014) related it to the gradient, and Springenberg et al. (2015) built guided backpropagation from it.

Limitations

• One activation at a time. The authors note that the method shows a single activation, not the joint activity of a layer.
• Conditioned on the image. Springenberg et al. (2015) point out that because the switches come from a forward pass, the reconstruction is conditioned on an image and does not directly visualize learned features; in their all-convolutional network, which has no pooling and so no switches, the deconvnet failed to give sharp reconstructions from higher layers, while guided backpropagation worked without switches.
• Image recovery rather than explanation. Nie et al. (2018) argue theoretically and show on VGG-16 that DeconvNet and guided backpropagation perform partial image recovery: their maps are almost the same whichever class logit is chosen, and almost unchanged by an adversarial perturbation that changes the prediction. In a random three-layer network, adding max pooling turned DeconvNet’s map from noise into a recognizable image, and in a random VGG-16 both methods reconstructed fine detail of the input. Their experiments propagate back from class logits rather than from single feature-map activations as Zeiler and Fergus did, so they bear on the paper’s projections by extension. Adebayo et al. (2018) found guided backpropagation, which builds on the deconvnet, largely insensitive to randomizing the higher layers’ weights; they did not test the deconvnet itself and cite Nie et al. for it.
• Diagnosis versus ablation. The $7 \times 7$, stride-2 change is supported by visualizations and by the comparison of the replication (18.1%) with the new network (16.5%). In our reading, that comparison also replaces the grouped connections with dense ones, the text does not say whether the replication used the filter renormalization, and no row isolates the first-layer change; v1 and v2 had credited a gain of similar size to widening the middle layers instead.
• Occlusion with a grey square inserts a pattern the network never saw in training, which confounds removing evidence with an unfamiliar input (see interpretability of neural networks).
• Text and tables disagree in several places: the 1.7% “test” improvement, the 2.2% Caltech-101 margin, the claim that all layers give the best transfer results, the 26.1% against 26.2% runner-up, the five-panel caption of a four-panel Fig. 5 (ECCV), and in v1 and v2 a Fig. 5 caption describing four occlusion examples when three are printed.

What Came After

Simonyan, Vedaldi and Zisserman (2014) generalized the projection to gradients of class scores, giving image-specific saliency maps. Springenberg et al. (2015) combined the deconvnet’s ReLU rule with the gradient’s into guided backpropagation, which produced sharp projections from high layers without pooling switches. Later methods, such as class activation maps and Grad-CAM, moved attention from single units to class evidence, and sanity checks (Adebayo et al., 2018) and the analysis of Nie et al. (2018) showed that a sharp, image-like map is a poor guide to whether it reflects the model; interpretability of neural networks covers that development. On the architecture side, Simonyan and Zisserman (2015) replaced large early filters with stacks of $3 \times 3$ filters and contrasted their design with the $7 \times 7$, stride-2 first layer of Zeiler and Fergus (see the VGG paper).