Feature Visualization

Feature visualization shows what a part of a trained network responds to: a single unit, a whole channel (a feature map), a class output or an entire layer. Where a saliency map answers a local question about one prediction (“which pixels made this image score high for dog?”), feature visualization asks a global one (“what does channel 97 of this layer detect?”). Olah, Mordvintsev and Schubert (2017) describe it as answering what a network, or part of it, is looking for by generating examples, and they set it beside attribution as one of the two main threads of neural network interpretability. The examples can be synthesized, by optimizing an input until the unit fires strongly (activation maximization), or found, by ranking real images by the unit’s response (dataset examples). A third family asks what information a layer keeps, by inverting its activations into an image, and a fourth labels units automatically by how well their activations segment human-annotated concepts (Network Dissection). DeepDream, also called Inceptionism, is a variant of activation maximization that starts from a photograph and amplifies whatever a layer already sees in it.

The techniques share one weakness, which runs through this article: a picture of what maximizes a unit is not a description of what the unit computes. Optimized images depend on the regularizer that made them look natural, dataset examples confound what the unit detects with what co-occurs with it, and a recent line of work shows that visualizations can be manipulated without changing a network’s behaviour on real images.

Purpose

A vision network’s intermediate layers are its learned features, and one line of evidence that they form a hierarchy from edges to textures to object parts comes from visualizing them (see the Zeiler and Fergus paper and convolutional neural network). Feature visualization serves several purposes:

• Understanding what was learned. Which units detect textures, which object parts, and how this changes with depth, architecture and training data. Bau et al. (2017) used Network Dissection to compare networks trained on objects and on scenes, with and without dropout and batch normalization, and with supervised and self-supervised objectives.
• Debugging. Mordvintsev, Olah and Tyka (2015) visualized a class “dumbbell” and found that the generated dumbbells came with muscular arms attached; they suggested that the network might never have been shown a dumbbell without an arm holding it.
• Architecture changes. Zeiler and Fergus changed AlexNet’s first two layers after visualizing them, as the paper article describes.
• Concept labels at scale. Dissection labels every unit of a layer without a person looking at each.

When to Use It

Feature visualization fits when the question is about a component of the network rather than a decision: what a channel responds to, whether a layer contains detectors for some concept, how two networks’ representations differ. It needs white-box access: activation maximization needs gradients with respect to the input, dataset examples need forward passes over a large image collection, and dissection needs a densely labelled probe set such as Bau et al.’s Broden.

It fits less well when:

• the question is why one image received one prediction, which is the job of saliency maps, Grad-CAM and perturbation methods;
• the claim to be supported is causal (“the model uses texture X to decide Y”), which needs an intervention, as the interpretability hub describes for shortcut studies;
• the concept of interest is spread across many units, where single-unit pictures can mislead (Fong & Vedaldi, 2018; Olah et al., 2017);
• the network may have been built, or fine-tuned, by a party with an interest in what an audit sees (Geirhos et al., 2024).

How It Works

Activation maximization

Let $a(\mathbf{x})$ be the quantity to visualize, as a function of an input image $\mathbf{x}$. Olah et al. (2017) list the usual choices: a single neuron $\text{layer}_n[x, y, z]$ at one position, a whole channel $\text{layer}_n[:, :, z]$ over all positions, a layer through the DeepDream objective $\lVert \text{layer}_n[:, :, :] \rVert^2$, and a class through its logit before the softmax or its probability after it. Activation maximization solves

$$\mathbf{x}^\star = \arg\max_{\mathbf{x}} \; a(\mathbf{x}) - \lambda R(\mathbf{x})$$

by gradient ascent from noise or from a constant image, where $R$ is a regularizer or prior and the gradient $\nabla_{\mathbf{x}} a$ comes from one backward pass, as for a saliency map. Olah et al. credit Erhan et al. (2009) with the core idea, with minimal regularization; Geirhos et al. (2024) also date the method to that report, which was not read for this article. Simonyan, Vedaldi and Zisserman (2014), as the interpretability hub describes, applied it to ImageNet classifiers to visualize class models; as Nguyen, Yosinski and Clune (2015) describe their work, an L2 penalty gave “class appearance models” with recognizable parts such as dog faces, fox ears and cup handles.

For a linear unit the problem has a closed form. If $a(\mathbf{x}) = \mathbf{w}^\top \mathbf{x} + b$ and the image is constrained to a ball $\lVert \mathbf{x} \rVert_2 \le r$, the Cauchy–Schwarz inequality gives

$$\mathbf{x}^\star = r \, \frac{\mathbf{w}}{\lVert \mathbf{w} \rVert}, \qquad a(\mathbf{x}^\star) = r \lVert \mathbf{w} \rVert + b ,$$

so the visualization of a linear unit is its weight template. It follows that a first-layer filter, before its nonlinearity, is its own visualization, and its weights can simply be displayed. Deeper units are not linear in the image, and their maximizers depend on the starting point, the step rule and the regularizer.

Probabilities or logits. Olah et al. (2017) note that the easiest way to raise a class probability is often to lower the other classes rather than raise the class of interest, and in their experience optimizing pre-softmax logits gives better images.

The high-frequency problem

Unregularized maximization of a deep network’s units does not give natural-looking images. Olah et al. (2017) describe the results as a kind of optical illusion, full of noise and high-frequency patterns to which the network responds strongly. They relate the patterns to adversarial examples and say that strided convolutions and pooling seem to be an important part of the cause, since each creates checkerboard patterns in the gradient. Nguyen, Yosinski and Clune (2015) made the link concrete: by gradient ascent on the softmax output of ImageNet networks, starting from the mean image, they produced mostly unrecognizable images that the networks classified with 99.99% confidence. The interpretation problem follows: if the most activating input is a pattern no natural image contains, the visualization shows how to drive the unit, not what it detects in practice.

Regularizers and priors

Olah et al. (2017) arrange the remedies on a spectrum from weak to strong regularization, between the unregularized extreme (adversarial patterns) and the other extreme of searching only over real images (dataset examples). In between are three families:

• Frequency penalization. Olah et al. credit Mahendran and Vedaldi (2015) with introducing the total variation regularizer, which those authors write as $\sum_{i,j} \big((x_{i,j+1} - x_{ij})^2 + (x_{i+1,j} - x_{ij})^2\big)^{\beta/2}$, with $\beta = 2$ rather than the usual $\beta = 1$, together with an $\alpha$-norm that keeps pixel values in range ($\alpha = 6$). Olah et al. credit Nguyen et al. (2015) with introducing image blurring, and note that blurring is equivalent, in the Fourier domain, to an L2 penalty on each frequency component weighted by its frequency, and that these methods suppress legitimate edges along with noise; a bilateral filter, which preserves edges, does slightly better.
• Transformation robustness. The image is stochastically jittered, rotated or scaled before each step, so the result must activate the unit under small transformations. Olah et al. credit jitter and multi-scale optimization to Mordvintsev et al. (2015) and find even a small amount effective, especially combined with a high-frequency regularizer. Formally the objective becomes $\mathbb{E}_{t}\, a(t(\mathbf{x}))$ over random transformations $t$.
• Learned priors. A model of natural images is learned and enforced, either by optimizing in the latent space of a generator, $\mathbf{x} = G(\mathbf{h})$, or by adding the gradient of a learned log-density. Nguyen et al. (2017) cast activation maximization as sampling from $p(\mathbf{x} \mid y) \propto p(\mathbf{x})\, p(y \mid \mathbf{x})$, with one gradient step towards a generic natural image (the prior, modelled by a denoising autoencoder) and one towards a higher score for the chosen class or neuron, plus noise. Their earlier DGN-AM, which they describe as optimizing the code of a generator trained to reconstruct images from a classifier’s features, gave realistic images but little diversity; Plug & Play Generative Networks added a prior on the code. Olah et al. state the trade-off: learned priors give the most photorealistic visualizations, but it may be unclear what came from the model being visualized and what came from the prior.

Preconditioning and parameterization. Olah et al. (2017) separate regularizers from changes to how the image is parameterized. Blurring the gradient instead of the image, they argue, is gradient ascent in a stretched parameterization: it changes which directions are steepest and so which local optimum is reached, but not where the optima are. Their own images were optimized in a colour-decorrelated Fourier basis with frequencies scaled to equal energy, using Adam at learning rate 0.05 for 2,560 steps, with padding by 16 pixels, jitter by up to 16 pixels, random scaling by up to 5%, rotation by up to 5 degrees and a second jitter of up to 8 pixels at every step. Whether the preconditioner only accelerates the optimization or also regularizes it, they write, is hard to tell.

Diversity

A single optimized image shows one extreme input. Olah et al. (2017) describe how a unit that seemed, from one visualization, to detect the tops of dog heads turned out, with a diversity term, to respond mostly to fur texture, which dataset examples broadly confirmed (including a spoon with dog-fur colouring). Their diversity term optimizes several images jointly and penalizes the cosine similarity between their Gram matrices $G_{ij} = \sum_{x,y} \text{layer}_n[x, y, i]\, \text{layer}_n[x, y, j]$, the statistic used in neural style transfer:

$$C_{\text{diversity}} = - \sum_a \sum_{b \neq a} \frac{\operatorname{vec}(G_a) \cdot \operatorname{vec}(G_b)}{\lVert \operatorname{vec}(G_a) \rVert \, \lVert \operatorname{vec}(G_b) \rVert}.$$

The same article reports neurons that respond to unrelated things, such as one that responds to two kinds of animal faces and to car bodies, and suggests that neurons are not necessarily the right semantic units. As Olah et al. describe earlier work, Wei et al. clustered a class’s activations over the training set and optimized for the cluster centres, and Nguyen, Yosinski and collaborators started optimizations from diverse dataset examples; Nguyen et al. (2017) report that activation maximization often converges to the same mode from different random initializations.

Dataset examples

The simplest visualization ranks real images, or image patches, by the unit’s activation and shows the top ones. As the paper article describes, Zeiler and Fergus showed the nine strongest activations of sampled channels next to their deconvnet projections, and Zhou et al.’s work on scene networks, as Bau et al. (2017) describe it, used human evaluation to find that individual units behave as object detectors in a network trained to classify scenes from Places. Dataset examples are real, so they cannot be adversarial patterns, and a whole range of activation levels can be shown, not only the top. Their weakness, as Olah et al. put it, is that optimization separates the things causing behaviour from things that merely correlate with them, and dataset examples do not: if every image of stripes also contains a disc, the top images of a stripe detector show discs.

Fong and Vedaldi (2018) add a statistical caution: showing only maximally activating inputs ignores how the unit behaves on the rest of the distribution. Using Network Dissection’s masks, they found that the best single filter for a concept scored an IoU of 0 on many individual images of that concept, because no activation survived the threshold, and they proposed showing examples at each decile of the per-image IoU distribution instead.

Projections from activations

As the Zeiler and Fergus paper article describes, their deconvnet projects one strong activation back to the pixels with the switches recorded by max pooling, a ReLU on the backward signal and the transposed filters. Its relation to the gradient (it differs only at ReLUs), its experiments and the finding of Nie et al. that such projections largely recover the input image rather than reflect the decision belong to that article and to saliency maps. Because the projection is conditioned on one image, it is, in our reading, a hybrid of a dataset example and an attribution map rather than a picture of the unit’s preferred input.

Feature inversion

Inversion asks what a whole layer retains, rather than what one unit wants. Mahendran and Vedaldi (2015) found an image whose representation matches a target, $\arg\min_{\mathbf{x}} \ell(\Phi(\mathbf{x}), \Phi_0) + \lambda R(\mathbf{x})$, with the same regularizers as above, and noted that the loss could be replaced by one that optimizes selected neural responses, which gives activation maximization. Dosovitskiy and Brox (2016), in “Inverting Visual Representations with Convolutional Networks”, instead trained an up-convolutional network to predict the expected pre-image of a feature vector, the average of the natural images that could have produced it. Inverting AlexNet, they found that colours and rough contours of an image could be reconstructed even from the top layer, the predicted class probabilities, and that the reconstruction depended more on the small probabilities of non-predicted classes than on the top predictions. The layer-by-layer results and a worked inversion of a small network are in perceptual loss.

DeepDream

Mordvintsev, Olah and Tyka (2015) described two uses of the same machinery. Starting from noise, they maximized a class such as “banana”, which by their account did not work very well until they imposed a prior that neighbouring pixels be correlated. Starting from a photograph, they picked a layer and asked the network to enhance whatever it detected, which Olah et al. (2017) write as maximizing the squared activations of the whole layer. Lower layers then add strokes and ornament-like patterns; higher layers turn clouds into birds and horizons into towers, in a feedback loop in which a faint resemblance is amplified step by step. Applied iteratively to its own output with zooming, even from random noise, the procedure gave what they called neural net “dreams”; they named the technique Inceptionism, after the network architecture used. The authors present it as a qualitative sense of the level of abstraction a layer has reached; the output is shaped by the input image, as their examples show, so in our reading it is an amplification of the network’s response to that image, not a portrait of a unit.

Network Dissection

Network Dissection (Bau et al., 2017) replaces looking with scoring. Its probe set, Broden, merges densely labelled images from ADE, OpenSurfaces, Pascal-Context, Pascal-Part and the Describable Textures Dataset, with colour labels for every pixel, into concepts of six kinds: objects, parts, scenes, materials, textures and colours. For each unit $k$ the method:

1. collects the activation map $A_k(\mathbf{x})$ over all Broden images and sets a threshold $T_k$ with $P(a_k > T_k) = 0.005$ over all positions;
2. upsamples each map bilinearly to the annotation’s resolution, anchoring each value at the centre of the unit’s receptive field, and thresholds it, $M_k(\mathbf{x}) = S_k(\mathbf{x}) \ge T_k$;
3. scores the unit against each concept $c$ by a dataset-wide intersection over union,

$$\mathrm{IoU}_{k,c} = \frac{\sum_{\mathbf{x}} |M_k(\mathbf{x}) \cap L_c(\mathbf{x})|}{\sum_{\mathbf{x}} |M_k(\mathbf{x}) \cup L_c(\mathbf{x})|},$$

where $L_c$ is the concept’s mask and the sums run over images that have a label of the concept’s kind.

A unit is called a detector for its best concept when $\mathrm{IoU}_{k,c} > 0.04$; the authors report that their qualitative results are insensitive to this threshold, which changes the number of detectors but not the orderings of networks. Interpretability of a layer is summarized as the number of unique concepts among its detectors. The method needs only forward passes and works only as well as the probe set: a unit that matches a concept absent from Broden scores low.

One property of the measure follows from the definitions (our reasoning, checked in the example below). Because $M_k$ covers about 0.5% of all positions, a unit can reach at most about $\mathrm{IoU} = 0.005 / f_c$ on a concept that covers a fraction $f_c > 0.005$ of the pixels, even if it fires exactly inside the concept. Detector thresholds and IoU values are therefore comparable across units only for concepts of similar frequency.

Successors. Fong and Vedaldi’s Net2Vec (2018) learns, for each concept, a weight vector over all of a layer’s filters, used to segment or classify the concept, and compares it with the best single filter. Combinations matched or beat the best single filter for most concepts (for segmentation, 91.6% of concepts at AlexNet’s conv1 and 75.7% at conv5; for classification, 85.0–90.2% across layers), and around 15% of conv1 filters were the best filter for at least 20 or 30 concepts, depending on the task. They read this as evidence that concepts are mostly encoded by several filters and that filters are shared across concepts, and they treat the learned weights as concept embeddings whose arithmetic gives meaningful results. TCAV, which learns a concept direction in activation space from example images and tests a class’s sensitivity to it, is described in the interpretability hub.

How to Apply It

The PyTorch example below runs activation maximization, dataset examples and Network Dissection on a network whose units are known, so that each technique can be compared with the truth. It is built like the example in the Grad-CAM article: fixed, hand-made oriented detectors, spatial averaging and a logistic-regression head, the only fitted part. Nothing is trained by gradient descent.

Data. The images are $64 \times 64$ grey images with background 0.5 and Gaussian noise of standard deviation 0.05. Each contains two non-overlapping discs of radius 9, each filled with one of five textures: gratings of period 6 pixels and amplitude 0.4 with random phase, whose stripes run horizontally, vertically, along “" or along “/” (with the $y$ axis pointing down, “" means $x - y$ is constant along a stripe), or a flat bright disc of value 0.9. With probability 0.4 an image contains vertical stripes; its other disc is then the bright disc with probability 0.8. The concept masks are the pixels of each disc. There are 600 training images and 600 probe images.

Network. Layer 1 has five $9 \times 9$ filters applied with stride 2 to the image minus 0.5 (zero padding is then padding with the background): four even Gabor filters, one per stripe orientation (Gaussian envelope of standard deviation 2.5, wavelength 6, zero mean, unit norm), and a Gaussian blob of the same width for brightness. Each is followed by a threshold of 0.15, three times the standard deviation of a unit-norm filter’s response to the background noise, and a ReLU, giving five $32 \times 32$ maps. The unit to visualize is the logit of a logistic regression (scikit-learn, $C = 1$) on the five spatial means of these maps, standardized with the training set’s means and standard deviations, fitted to predict “contains vertical stripes”. The logit is linear in the pooled features and nonlinear in the image.

The script has four parts:

• A. A linear unit, the response of the vertical filter at one position before its ReLU, maximized inside the ball $\lVert \mathbf{x} \rVert_2 \le r$ of centred images, where $r$ is the median norm of the centred probe images. Projected gradient ascent takes 400 steps of length $0.05\,r$ in the direction of the normalized gradient, from five noise images (seeds 0–4), with three regularizers: none; jitter, where the gradient is taken at the image shifted by a random integer offset of up to 4 pixels in each direction; and blur, a Gaussian blur of standard deviation 0.5 pixels after each step. The result is compared with the closed form $\mathbf{x}^\star = r\,\mathbf{w}/\lVert\mathbf{w}\rVert$ above.
• B. The same procedure on the logistic unit. The script reports its logit, the share of the result’s spectral power above 0.25 cycles per pixel (the “HF fraction”; the gratings have 1/6 cycle per pixel), and the channel profile of seed 0, the five pooled layer-1 responses normalized to sum to 1 (in the order horizontal, vertical, “", “/”, disc).
• C. The nine probe images with the highest logit, with the share of them containing each concept, their mean channel profile, and its cosine with each maximized image’s profile.
• D. Network Dissection of the five layer-1 units as Bau et al. define it ($\tau = 0.005$, bilinear upsampling anchored at each unit’s receptive-field centre, detector if IoU > 0.04), with the bound $\tau / f_c$ for each concept and a check of the IoU code on an oracle unit that fires most strongly deep inside the vertical-stripe masks. It ends with Bau et al.’s rotation test: a random orthogonal $5 \times 5$ matrix mixes the five post-ReLU maps, and the dissection is repeated.

The script uses PyTorch 2.14.1, NumPy 2.5.4, SciPy 1.18.1 and scikit-learn 1.9.1, in double precision on one CPU thread, and runs in about 4 to 15 seconds depending on machine load; two runs gave identical output.

import numpy as np
import torch
import torch.nn.functional as F
from scipy.ndimage import distance_transform_edt
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

torch.set_num_threads(1)
torch.set_default_dtype(torch.float64)
IMG, R, PERIOD, AMP, NOISE = 64, 9, 6.0, 0.4, 0.05
CONCEPTS = ["horizontal", "vertical", "diag \\", "diag /", "disc"]
THETA, TAU = 3 * NOISE, 0.005                 # unit threshold; Network Dissection quantile

def stripes(kind, yy, xx):
    """Coordinate across the stripes (y points down)."""
    return {"horizontal": yy, "vertical": xx,
            "diag \\": (xx - yy) / np.sqrt(2), "diag /": (xx + yy) / np.sqrt(2)}[kind]

def make_set(n, seed):
    """Two non-overlapping textured discs per image; vertical stripes come with the bright
    disc 80% of the time. Returns images, concept masks (n, 5, 64, 64) and labels."""
    rng = np.random.default_rng(seed)
    yy, xx = np.mgrid[:IMG, :IMG].astype(float)
    X = np.zeros((n, IMG, IMG)); M = np.zeros((n, 5, IMG, IMG), bool); y = np.zeros(n, int)
    for i in range(n):
        if rng.random() < 0.4:
            other = "disc" if rng.random() < 0.8 else ["horizontal", "diag \\", "diag /"][rng.integers(3)]
            kinds, y[i] = ["vertical", other], 1
        else:
            kinds = list(rng.choice(["horizontal", "diag \\", "diag /", "disc"], 2, replace=False))
        while True:
            c = rng.uniform(R + 2, IMG - 1 - R - 2, size=(2, 2))
            if np.hypot(*(c[0] - c[1])) >= 2 * R + 2:
                break
        img = np.full((IMG, IMG), 0.5)
        for kind, (cy, cx) in zip(kinds, c):
            m = (yy - cy) ** 2 + (xx - cx) ** 2 <= R ** 2
            pat = 0.9 if kind == "disc" else \
                0.5 + AMP * np.cos(2 * np.pi * stripes(kind, yy, xx) / PERIOD + rng.uniform(0, 2 * np.pi))[m]
            img[m] = pat
            M[i, CONCEPTS.index(kind)] |= m
        X[i] = img + rng.normal(0, NOISE, (IMG, IMG))
    return X, M, y

def filters():
    """Four even Gabor filters (one per stripe orientation) and a Gaussian brightness blob, unit norm."""
    yy, xx = np.mgrid[-4:5, -4:5].astype(float)
    env = np.exp(-(xx ** 2 + yy ** 2) / (2 * 2.5 ** 2))
    W = []
    for kind in CONCEPTS[:4]:
        g = env * np.cos(2 * np.pi * stripes(kind, yy, xx) / PERIOD)
        g -= g.mean(); W.append(g / np.linalg.norm(g))
    W.append(env / np.linalg.norm(env))
    return torch.tensor(np.array(W))[:, None]

W = filters()
def layer1(xc):                       # xc: centred images (n, 64, 64) -> ReLU maps (n, 5, 32, 32)
    return F.relu(F.conv2d(xc[:, None], W, stride=2, padding=4) - THETA)

# ---- data, the logistic unit, and data checks ---------------------------------------------------
Xtr, Mtr, ytr = make_set(600, seed=1)
Xp, Mp, yp = make_set(600, seed=2)
with torch.no_grad():
    Atr, Ap = layer1(torch.tensor(Xtr - 0.5)), layer1(torch.tensor(Xp - 0.5))
sc = StandardScaler().fit(Atr.mean((2, 3)).numpy())
lr = LogisticRegression(C=1.0, max_iter=5000).fit(sc.transform(Atr.mean((2, 3)).numpy()), ytr)
mu, sd = torch.tensor(sc.mean_), torch.tensor(sc.scale_)
w, b = torch.tensor(lr.coef_[0]), float(lr.intercept_[0])
def logit(pooled):                    # the "vertical stripes" unit on pooled layer-1 features
    return ((pooled - mu) / sd) @ w + b
def unit(xc):                         # ... as a function of one centred image
    return logit(layer1(xc[None]).mean((2, 3))[0])

has = Mp.any((2, 3))                  # (600, 5): which concepts each probe image contains
print(f"torch {torch.__version__}; data checks: positives {yp.mean():.3f}, "
      f"disc | vertical {has[yp == 1, 4].mean():.3f}, disc | no vertical {has[yp == 0, 4].mean():.3f}, "
      f"overlapping concept pixels {(Mp.sum(1) > 1).sum()}, objects per image {sorted(set(has.sum(1).tolist()))}")
print("nonzero activations per unit:", " ".join(f"{v:.3f}" for v in (Ap > 0).double().mean((0, 2, 3)).tolist()))
zp = logit(Ap.mean((2, 3))).numpy()
print("unit weights (" + ", ".join(CONCEPTS) + "):", " ".join(f"{v:+.1f}" for v in w.tolist()),
      f" bias {b:+.1f} (standardized features);  probe accuracy {((zp > 0) == yp).mean():.3f}")

# ---- activation maximization ----------------------------------------------------------------------
r = float(np.median(np.linalg.norm((Xp - 0.5).reshape(600, -1), axis=1)))
k = torch.exp(-torch.arange(-2, 3.) ** 2 / (2 * 0.5 ** 2)); k /= k.sum()
def blur(x):                          # Gaussian blur, sd 0.5 px, zero padding on the centred image
    x = F.conv2d(x[None, None], k.view(1, 1, 1, 5), padding=(0, 2))
    return F.conv2d(x, k.view(1, 1, 5, 1), padding=(2, 0))[0, 0]

def maximize(objective, reg, seed, steps=400, eta=0.05, jit=4):
    """Projected, normalized gradient ascent on a centred image inside the ball ||x|| <= r."""
    g = torch.Generator().manual_seed(seed)
    x = torch.randn(IMG, IMG, generator=g); x *= r / x.norm()
    for _ in range(steps):
        x.requires_grad_(True)
        if reg == "jitter":
            s = torch.randint(-jit, jit + 1, (2,), generator=g).tolist()
            val = objective(torch.roll(x, s, dims=(0, 1)))
        else:
            val = objective(x)
        grad, = torch.autograd.grad(val, x)
        if grad.norm() == 0:
            raise RuntimeError("zero gradient")
        x = (x + eta * r * grad / grad.norm()).detach()
        if reg == "blur":
            x = blur(x)
        x = x * min(1.0, r / float(x.norm()))
    return x

def hf_fraction(x, cut=0.25):         # share of spectral power above 0.25 cycles/pixel
    P = np.abs(np.fft.fft2(x - x.mean())) ** 2
    f = np.fft.fftfreq(IMG); fr = np.hypot(*np.meshgrid(f, f, indexing="ij"))
    return P[fr > cut].sum() / P.sum()

def profile(xc):                      # share of each layer-1 channel in the pooled response
    p = layer1(xc[None]).mean((2, 3))[0]
    return (p / p.sum()).detach().numpy()

def rng_str(v):
    return f"{min(v):.6f} to {max(v):.6f}" if max(v) - min(v) > 5e-7 else f"{v[0]:.6f}"

REGS, SEEDS = ["none", "jitter", "blur"], range(5)
f_vert = W[1, 0]
lin = lambda x: (f_vert * x[28:37, 28:37]).sum() - THETA
x_star = torch.zeros(IMG, IMG); x_star[28:37, 28:37] = r * f_vert
z_star = r - THETA
print(f"\nA. linear unit (vertical filter at one position), radius r = {r:.3f}, z* = {z_star:.3f}")
print("   regularizer   cos(x, x*)                   z / z*")
for reg in REGS:
    xs = [maximize(lin, reg, s) for s in SEEDS]
    cs = [float(F.cosine_similarity(x.flatten(), x_star.flatten(), 0)) for x in xs]
    zs = [float(lin(x)) / z_star for x in xs]
    print(f"   {reg:10s}    {rng_str(cs):27s}  {rng_str(zs)}")

print("\nB. logistic unit (seeds 0-4)                  channel profile of seed 0")
print("   regularizer   logit z            HF fraction    " + " ".join(f"{c[:5]:>6s}" for c in CONCEPTS))
am = {}
for reg in REGS:
    xs = [maximize(unit, reg, s) for s in SEEDS]
    am[reg] = xs[0]
    zs = [float(unit(x)) for x in xs]; hs = [hf_fraction(x.numpy()) for x in xs]
    print(f"   {reg:10s}    {min(zs):.3f} to {max(zs):.3f}    {min(hs):.3f} to {max(hs):.3f}    "
          + " ".join(f"{v:6.2f}" for v in profile(xs[0])))
print(f"   start noise HF fraction {hf_fraction(torch.randn(IMG, IMG, generator=torch.Generator().manual_seed(0)).numpy()):.3f};"
      f" probe images: median HF fraction {np.median([hf_fraction(x) for x in Xp]):.3f}, logit max {zp.max():.3f}")

# ---- dataset examples ----------------------------------------------------------------------------
top = np.argsort(-zp)[:9]
print("\nC. top-9 probe images vs. the probe set: share containing each concept")
print("   " + " ".join(f"{c:>11s}" for c in CONCEPTS))
print("   top 9     " + " ".join(f"{v:11.2f}" for v in has[top].mean(0)))
print("   all 600   " + " ".join(f"{v:11.2f}" for v in has.mean(0)))
ptop = Ap[top].mean((0, 2, 3)).numpy(); ptop /= ptop.sum()
print("   channel profile of the top 9:", " ".join(f"{v:.2f}" for v in ptop))
print("   cosine with the top-9 profile:", ", ".join(
    f"{reg} {float(np.dot(profile(am[reg]), ptop) / np.linalg.norm(profile(am[reg])) / np.linalg.norm(ptop)):.3f}"
    for reg in REGS))

# ---- network dissection --------------------------------------------------------------------------
def upsample(A):                      # (n, k, 32, 32) -> (n, k, 64, 64); unit (i, j) sits at pixel (2i, 2j)
    t = np.arange(IMG) / 2; i0 = np.floor(t).astype(int); i1 = np.minimum(i0 + 1, 31); a = t - i0
    A = A[:, :, i0] * (1 - a)[:, None] + A[:, :, i1] * a[:, None]
    return A[:, :, :, i0] * (1 - a) + A[:, :, :, i1] * a

def dissect(A):
    T = np.quantile(A.transpose(1, 0, 2, 3).reshape(A.shape[1], -1), 1 - TAU, axis=1)
    S = upsample(A) >= T[None, :, None, None]
    iou = np.zeros((A.shape[1], 5))
    for u in range(A.shape[1]):
        for c in range(5):
            iou[u, c] = (S[:, u] & Mp[:, c]).sum() / (S[:, u] | Mp[:, c]).sum()
    return iou

fc = Mp.mean((0, 2, 3))
L = Mp[:, 1]
O = np.stack([distance_transform_edt(m) for m in L]) + 1e-6 * np.random.default_rng(5).random(L.shape)
Mo = O >= np.quantile(O, 1 - TAU)
print(f"\nD. dissection; oracle check (vertical): IoU {(Mo & L).sum() / (Mo | L).sum():.5f}, tau/f_c {TAU / fc[1]:.5f}")
print("   IoU          " + " ".join(f"{c:>10s}" for c in CONCEPTS) + "   best")
iou = dissect(Ap.numpy())
for u, name in enumerate(CONCEPTS):
    print(f"   unit {name:7s} " + " ".join(f"{v:10.3f}" for v in iou[u]) + f"   {CONCEPTS[iou[u].argmax()]}")
print("   bound tau/f_c" + " ".join(f"{v:10.3f}" for v in TAU / fc))
Q, Rq = np.linalg.qr(np.random.default_rng(0).normal(size=(5, 5))); Q *= np.sign(np.diag(Rq))
iou_rot = dissect(np.einsum("kj,njhw->nkhw", Q, Ap.numpy()))
for name, m in [("original", iou), ("rotated", iou_rot)]:
    det = m.max(1) > 0.04
    print(f"   {name:9s}: best IoU per unit " + " ".join(f"{v:.3f}" for v in m.max(1))
          + f"; detectors {det.sum()}, unique concepts {len(set(m.argmax(1)[det]))}")

Output:

torch 2.14.1; data checks: positives 0.410, disc | vertical 0.772, disc | no vertical 0.520, overlapping concept pixels 0, objects per image [2]
nonzero activations per unit: 0.060 0.064 0.061 0.062 0.102
unit weights (horizontal, vertical, diag \, diag /, disc): -0.7 +4.0 -1.4 -1.4 -0.7  bias -0.9 (standardized features);  probe accuracy 1.000

A. linear unit (vertical filter at one position), radius r = 8.400, z* = 8.250
   regularizer   cos(x, x*)                   z / z*
   none          1.000000                     1.000000
   jitter        -0.423779 to -0.121051       -0.445089 to -0.141333
   blur          0.734568 to 0.736694         0.516504 to 0.516646

B. logistic unit (seeds 0-4)                  channel profile of seed 0
   regularizer   logit z            HF fraction     horiz  verti  diag   diag    disc
   none          55.421 to 61.024    0.029 to 0.045      0.00   0.84   0.08   0.07   0.01
   jitter        54.284 to 59.133    0.001 to 0.004      0.00   0.84   0.09   0.06   0.01
   blur          23.828 to 28.303    0.001 to 0.002      0.00   0.93   0.00   0.00   0.06
   start noise HF fraction 0.804; probe images: median HF fraction 0.171, logit max 6.502

C. top-9 probe images vs. the probe set: share containing each concept
    horizontal    vertical      diag \      diag /        disc
   top 9            0.00        1.00        0.00        0.00        1.00
   all 600          0.33        0.41        0.31        0.33        0.62
   channel profile of the top 9: 0.02 0.18 0.05 0.05 0.71
   cosine with the top-9 profile: none 0.261, jitter 0.269, blur 0.308

D. dissection; oracle check (vertical): IoU 0.19622, tau/f_c 0.19622
   IoU          horizontal   vertical     diag \     diag /       disc   best
   unit horizontal      0.132      0.000      0.000      0.000      0.000   horizontal
   unit vertical      0.000      0.097      0.000      0.000      0.000   vertical
   unit diag \       0.000      0.000      0.181      0.000      0.000   diag \
   unit diag /       0.000      0.000      0.000      0.161      0.000   diag /
   unit disc         0.000      0.000      0.000      0.000      0.107   disc
   bound tau/f_c     0.248      0.196      0.257      0.245      0.129
   original : best IoU per unit 0.132 0.097 0.181 0.161 0.107; detectors 5, unique concepts 5
   rotated  : best IoU per unit 0.057 0.058 0.095 0.179 0.065; detectors 5, unique concepts 3

The linear unit (A). Without a regularizer, every start converges to the closed-form optimum: the gradient of a linear unit is its template, and each projection back onto the ball shrinks the part of the start noise that is orthogonal to it. The two regularizers move the result away from the unit’s own maximizer. Blur leaves a pattern whose cosine with the template is about 0.73 and which reaches only half the possible response; in our reading, the period-6 Gabor is itself fine structure, and repeated blurring removes part of it. Jitter does worse for this objective: the cosine is negative for all five seeds. A single-position neuron is not shift-invariant, and in our reading the random shifts of up to 4 pixels, more than half the 6-pixel period, make successive gradients disagree in sign, so the last iterate reflects the last few shifts rather than the template. In our reading, transformation robustness assumes an objective that tolerates the transformation, such as a channel averaged over positions; Olah et al. used the channel objective for most of their images.

The logistic unit (B). From noise whose HF fraction is 0.80, unregularized ascent ends at 0.029–0.045, and jitter and blur at 0.001–0.004, so both regularizers remove most of the remaining fine structure. Jitter costs little activation (logits 54–59 against 55–61), blur roughly half or more (24–28). All of them exceed the highest-scoring probe image (6.5), by factors of about 3.7 to 9.4: the optimized image is a superstimulus, stronger than anything in the data. The seed-0 profiles are dominated by the vertical channel (0.84–0.93), with small responses of the diagonal channels to the same pattern; the unit’s largest weight is on that channel, and it has negative weights on the others. In our reading, because the unit averages over positions, the best use of the norm budget is a vertical grating spread over the whole image rather than a disc-shaped patch like those in the data.

Dataset examples (C). All nine top probe images contain vertical stripes, which only 41% of probe images do, but all nine also contain the bright disc, which 62% do. Their channel profile is dominated by the disc channel (0.71), and its cosine with the maximized images’ profiles is only 0.26–0.31. The unit’s weight on the disc channel is negative (−0.7), so the disc does not raise the logit through that channel. The printed output does not show why disc-bearing images rank highest. In our reading, one possible reason is that the vertical Gabor may also respond to the left and right edges of a bright disc; the example does not test this. Either way, someone reading the top nine images would conclude that the unit likes discs with stripes, while optimization shows stripes alone. This is Olah et al.’s point about causes and correlates in miniature, and the disc co-occurs with vertical stripes because the data were built that way.

Dissection (D). The oracle check reproduces the bound $\tau / f_c$ to five decimals. Every unit’s best concept is the one it was built for, with IoUs from 0.097 to 0.181, all above Bau et al.’s detector threshold of 0.04 and below the bound for their concept (0.129–0.257). Even these units, built as detectors, reach only 49–83% of the bound, so either part of their thresholded, upsampled responses falls outside the concept masks or the upsampled masks cover less than 0.5% of the pixels; the printed output does not separate the two. After a random rotation, the five mixed units still pass the 0.04 threshold but name only three distinct concepts. Bau et al. found 80% fewer unique detectors after rotating AlexNet’s conv5; here the drop is from 5 to 3, and in our reading the detector count does not fall because the 0.04 threshold sits far below the bounds of this data set.

Process. One planned check failed as first planned: the logistic unit was to reach a probe accuracy of at least 0.99, and in the pilot, with the logistic regression fitted with $C = 1$ on the unscaled pooled features, it failed. The penalty dominated a fit on unscaled features, and the reachable value in the plan had been computed for a threshold on the vertical feature rather than for the planned fit, which was a planning error. On the editor’s decision, the pooled features were standardized before the first run, with $C = 1$ and the criterion unchanged, and the pilot was repeated once. At the same time, three formatting changes were approved: logits printed to three decimals, plain integers in the data check, and ranges written with “to”. The output above is the first run.

The example shows mechanics on a network whose units are known; it does not show that any visualization is faithful for a trained network, which is the subject of the evidence below.

Variants

• Neuron, channel, layer and class objectives (Olah et al., 2017): one position, one channel over all positions, the DeepDream objective over a layer, or a class logit or probability.
• Combinations and directions. Olah et al. (2017) optimize sums of channels (adding a “black and white” neuron to a “mosaic” neuron gave a black-and-white mosaic), interpolate between channel objectives, and visualize random directions in activation space. Net2Vec’s concept embeddings are directions learned from labels (Fong & Vedaldi, 2018).
• Image parameterizations: pixels, a decorrelated Fourier basis (Olah et al., 2017), or the latent code of a generator (Nguyen et al., 2017).
• Diversity: joint optimization with a Gram-matrix diversity term (Olah et al., 2017), clustering-based facets, or sampling with a learned prior (Nguyen et al., 2017).
• Dataset examples: top-$k$ images or patches, examples sampled across activation deciles (Fong & Vedaldi, 2018), and segmentations of the top images (Bau et al., 2017).
• Inversion: optimization-based (Mahendran & Vedaldi, 2015) or learned (Dosovitskiy & Brox, 2016); see perceptual loss.
• Dissection: single-unit IoU against Broden (Bau et al., 2017) and multi-filter concept embeddings (Fong & Vedaldi, 2018).

Best Practices

• Show optimized images and dataset examples together. Olah et al. (2017) checked a hypothesis formed from optimized images against dataset examples; the example above shows how the two can disagree when a correlate travels with the cause.
• Report the whole recipe. Objective, image parameterization, regularizers and their strengths, transformations, optimizer, steps and starting point all change the picture; Olah et al. list theirs in full.
• Prefer logits to probabilities for class visualizations (Olah et al., 2017).
• Show several optimizations, with different seeds or a diversity term, before describing a unit by one image.
• Look beyond the top examples. Sample across the activation range or IoU deciles (Fong & Vedaldi, 2018).
• Check the scale of dissection scores. Compare IoU with the bound $\tau / f_c$ of each concept, and remember that a unit can match only concepts in the probe set (Bau et al., 2017).
• Check that visualizations activate the network like natural images of the same class do, with the sanity check of Geirhos et al. (2024) described below.

Pitfalls

• Reading the regularizer as the unit. Strong priors make realistic images, but it can be unclear what came from the model and what from the prior (Olah et al., 2017). In the example, blur halved the linear unit’s response and changed its picture.
• Reading adversarial patterns as features. Unregularized optimization finds high-frequency inputs that drive a unit far beyond anything natural (Olah et al., 2017; Nguyen et al., 2015).
• One facet for the whole unit. A single optimization shows one extreme; a unit can respond to unrelated things (Olah et al., 2017).
• Correlates in dataset examples. Top images show what co-occurs with the feature as well as the feature.
• Assuming units are the right basis. Bau et al. (2017) found that interpretability drops when the basis is rotated, which supports reading single units in the networks they studied; Olah et al., describing Szegedy et al.’s earlier finding that random directions seemed as meaningful as units, report that in their experience random directions are often interpretable, though at a lower rate; and Net2Vec found most concepts better encoded by several filters.
• Treating dissection counts as accuracy. Bau et al. (2017) report that interpretability is neither a prerequisite of nor an obstacle to discriminative power: a rotated layer has the same discriminative power and far fewer unique detectors.
• Trusting visualizations of a model one did not build. Geirhos et al. (2024) show that visualizations can be manipulated to show arbitrary patterns.

Evidence and Impact

What dissection measured. Bau et al. (2017) checked their labels with Amazon Mechanical Turk raters on AlexNet trained on Places205. Among units whose human-written labels raters agreed with, the method’s labels were rated descriptive for 37% of the units in conv1 and 71% in conv5, against 82% and 91% for a second check of the human labels. A random rotation of conv5 left 80% fewer unique detectors, which they take as evidence that interpretability is an axis-aligned property of the representation and not an inevitable result of discriminative power. Across networks, they found interpretability ranked ResNet > VGG > GoogLeNet > AlexNet, and Places-trained networks above ImageNet-trained ones; self-supervised AlexNets had many texture detectors but few object detectors, and the colorization model, trained on colourless images, almost no colour detectors. Different random initializations gave similar levels of interpretability; removing dropout gave more texture and fewer object detectors; batch normalization seemed to decrease interpretability significantly; and widening AlexNet’s conv5 from 256 to 768 units, with global average pooling, gave many more detectors at similar accuracy. Colour and texture concepts dominated the lower layers, and more object and part detectors emerged in conv5.

Distributed concepts. Net2Vec (Fong & Vedaldi, 2018) found that combinations of filters segment and classify most Broden concepts as well as or better than the best single filter, from AlexNet’s first layer on, and that single filters are often the best filter for many concepts at once. In our reading, single-unit labels therefore describe the extreme responses of a unit, not the whole of what it encodes.

Feature visualization under adversarial construction. Geirhos et al. (2024) built a “fooling circuit”: a small classifier, trained to tell natural images from feature visualizations (99.49% accuracy on held-out data), gates which of two units’ outputs passes on. With one such circuit per unit, the last layer of Inception-V1 showed arbitrary visualizations, a permutation of other units’ or a hard-coded picture such as the Mona Lisa, while its ImageNet top-1 accuracy changed from 69.146% to 68.744% and top-5 from 88.858% to 88.330%. A second construction added “silent” units that never fire on natural images but dominate optimization; it made all 512 units of a ResNet-50 layer show near-identical visualizations with no change in validation accuracy. The authors frame the threat as an audit of a model whose owner controls it.

Feature visualization in ordinary networks. The same paper proposes a sanity check: feed the visualizations of Inception-V1’s last-layer units back into the network and compare each layer’s activations with those of natural images of the same class, using Spearman rank correlation and scaling so that 1 is the similarity between natural images of one class and 0 the similarity between images of different classes. Through the first two thirds of the layers, the visualizations were about as similar to their own class’s natural images as to images of unrelated classes; similarity rose only in the last third. The authors conclude that the visualizations are processed along different paths from natural images and so do not explain how the network processes them. They also prove that, without strong assumptions on the function, knowing its maximizer and minimizer cannot guarantee useful predictions of its value elsewhere: their Table 1 answers “no” for black-box functions, neural networks, piecewise-affine, monotonic and convex functions, and affine functions of more than one input, and “yes” only for affine functions of one input and constants; for a Lipschitz function with known constant, whether a value is closer to the maximum or the minimum can be predicted only for small constants. Human studies they cite found that people often struggle to make sense of feature visualizations (Borowski et al., 2021, and Zimmermann et al., 2021, as Geirhos et al. report them).

Use. Feature visualization is behind several familiar descriptions of what convolutional networks learn: the edge-to-object hierarchy from Zeiler and Fergus’s projections, detector units in scene networks, and the visualizations of every GoogLeNet channel in Olah et al.’s appendix. Olah et al. (2017) conclude that feature visualization by itself will never give a completely satisfactory understanding and present it as one building block, to be combined with other tools. The evidence since then points the same way: visualizations suggest hypotheses about units, and dissection-style scores, interventions and checks such as Geirhos et al.’s are needed to test them.