Interpretability of Neural Networks

Interpretability of neural networks is the study of what a trained network has learned and why it produces a given output. Doshi-Velez and Kim (2017) define interpretability, for machine learning systems, as the “ability to explain or to present in understandable terms to a human”. The closely related word explainability, and the labels explainable AI and XAI, are used for the same field, often with the emphasis on methods that explain a model after training; DARPA’s XAI programme used the term for both explainable models and the interfaces that present their explanations. Lipton (2016; journal version 2018) separated two notions that the literature had mixed: transparency, where the model itself can be understood, and post-hoc explanation, where a separate procedure extracts information from a model that may remain opaque, with saliency maps as his example of the second. Rudin (2019) draws the same line more sharply. She calls a model interpretable when its form is constrained, for example to be sparse, monotonic or case-based, so that its explanation is the computation itself, and calls post-hoc explanations of black boxes explainable ML. She argues the second can mislead in high-stakes decisions, because an explanation that only approximates the model is not faithful to it. This article uses “interpretability” for the field and “explanation” for the output of any method.

For a convolutional neural network classifying images, the questions are concrete: which pixels made the network say “dog”, what a given feature map responds to, whether the network uses a concept such as “striped”, and whether it relies on the object or on something that merely co-occurs with it. Explanations of image models are often called heat maps, saliency maps, relevance maps or attribution maps.

Definition

An explanation method takes a trained model, and usually an input, and returns something a person can inspect: a map over the pixels, a set of example images, a synthesized image, or a score for a human-defined concept. The main distinctions are:

• Local versus global. A local explanation accounts for one prediction; a global one describes what the model does in general. Doshi-Velez and Kim (2017) tie the first to justifying a specific decision and the second to scientific understanding or detecting bias. A saliency map is local; a visualization of what a channel detects across a dataset is global.
• Post-hoc versus intrinsic. A post-hoc method explains a model trained without regard to explanation. An intrinsically interpretable model, such as a sparse linear model or a prototype network, is built so that its computation can be read directly.
• Model-specific versus model-agnostic. Gradient and activation methods need the network’s internals; perturbation methods such as LIME query only inputs and outputs.
• Audience. Lipton (2016) lists different desiderata that interpretability is asked to serve: trust, causality, transferability, informativeness, and fair and ethical decision-making. A developer debugging a model, a radiologist judging one case and a regulator auditing a system need different explanations.

Intuition

A classifier is a function from millions of pixel values to a few class scores. An explanation compresses that function into something small enough to look at, and every compression discards information. A saliency map says where the evidence was, but not what the network did with it. Rudin (2019) makes this point with two maps, for “Siberian husky” and for “transverse flute”, that look almost the same on one image. A synthesized image that maximizes a unit shows one pattern the unit responds to, but not every pattern it responds to. The useful question about any method is therefore what it claims to show, and whether that claim can be checked.

Two errors pull in opposite directions. An explanation can be faithful to the model but unhelpful to a person, or helpful-looking but unrelated to the model. Jacovi and Goldberg (2020) name the two properties: an interpretation is faithful when it accurately represents the reasoning behind the prediction, and plausible when it is convincing to people. Adebayo et al. (2018) observe that many saliency methods were proposed on the strength of their visual appeal, which is plausibility. Faithfulness needs tests.

Formal Definition

Let $f_c(\mathbf{x})$ be the score (logit) of class $c$ for an image $\mathbf{x}$. A local attribution method returns a map $R_c(\mathbf{x})$ with one value per pixel, or per region, measuring that part’s contribution to $f_c(\mathbf{x})$. The families below differ in how they define “contribution”.

Gradient. $R_c(\mathbf{x}) = \left| \partial f_c / \partial \mathbf{x} \right|$, the local sensitivity of the score to each pixel. Simonyan, Vedaldi and Zisserman (2014) take the maximum magnitude over colour channels.

Integrated gradients (Sundararajan et al., 2017) accumulate gradients along the straight path from a baseline image $\mathbf{x}'$ (a black image, for example) to $\mathbf{x}$:

$$\mathrm{IG}_i(\mathbf{x}) = (x_i - x'_i) \int_0^1 \frac{\partial f_c\big(\mathbf{x}' + \alpha(\mathbf{x} - \mathbf{x}')\big)}{\partial x_i}\, d\alpha , \qquad \sum_i \mathrm{IG}_i(\mathbf{x}) = f_c(\mathbf{x}) - f_c(\mathbf{x}').$$

The second identity, which the authors call completeness, follows from the fundamental theorem of calculus for path integrals.

Grad-CAM (Selvaraju et al., 2017) works on the feature maps $A^k$ of a convolutional layer. It weights each map by the spatial average of the gradient of the score with respect to it, sums, and keeps the positive part:

$$\alpha_k^c = \frac{1}{Z} \sum_{i,j} \frac{\partial f_c}{\partial A^k_{ij}}, \qquad L^c_{\text{Grad-CAM}} = \operatorname{ReLU}\Big(\sum_k \alpha_k^c A^k\Big),$$

where $Z$ is the number of positions. The map has the layer’s resolution and is upsampled to the image.

Occlusion. $R_c$ at a region $\Omega$ is $f_c(\mathbf{x}) - f_c(\mathbf{x}_{\Omega \leftarrow v})$, the drop in the score when the region is replaced by a constant $v$.

A concept score answers a global question instead. Kim et al. (2018) learn a concept activation vector $\mathbf{v}_C^l$ in the activation space of layer $l$, the normal of a linear classifier that separates examples of concept $C$ from random images. They then report the fraction of class-$k$ inputs for which moving the activations along $\mathbf{v}_C^l$ increases the class score:

$$\mathrm{TCAV}_{C,k,l} = \frac{\big|\{\mathbf{x} \in X_k : \nabla h_{l,k}(f_l(\mathbf{x})) \cdot \mathbf{v}_C^l > 0\}\big|}{|X_k|}.$$

Families of Methods

Gradient attribution

Simonyan, Vedaldi and Zisserman (2014) computed an image-specific class saliency map with a single backpropagation pass and used it, with a GraphCut colour model, for weakly supervised object segmentation. They also showed that Zeiler and Fergus’s deconvnet reconstruction (below) equals the gradient computation except at ReLU layers, where the deconvnet zeroes negative values of the signal coming down instead of using the forward pass’s mask. Raw gradients are noisy and measure local sensitivity, not contribution. A saturated unit can matter to the output while having zero gradient, which is one of the failures Sundararajan et al. (2017) built integrated gradients to fix: their method satisfies two axioms they propose, sensitivity and implementation invariance, and they report that 20 to 300 steps suffice to approximate the integral. SmoothGrad (Smilkov et al., 2017) averages gradient maps over copies of the image with added Gaussian noise; the authors found noise of 10–20% of the input range useful on Inception v3 and little visible change beyond 50 samples. Saliency maps in this family are cheap and have pixel resolution. Their weakness is that the result depends on the baseline, the noise level and the score being differentiated, and that sharp, plausible maps can come from operations that ignore the model (see below).

Class activation maps

Zhou et al. (2016) observed that a network ending in global average pooling followed by one linear layer computes each class score as a weighted sum of spatially averaged feature maps. The same weights, applied to the maps before averaging, give a class activation map (CAM) that shows where the evidence for that class lies. From image labels alone, their GoogLeNet-GAP reached 37.1% top-5 localization error on the ILSVRC 2014 test set (with a heuristic that takes boxes from the maps of the top three classes; 42.9% without), against 34.2% for a fully supervised AlexNet. CAM requires that architecture. Grad-CAM (Selvaraju et al., 2017) replaces the linear-layer weights by averaged gradients, applies to a wide variety of CNN-based models (the authors use the last convolutional layer), and reduces to CAM for the GAP architecture. Its maps are class-discriminative but coarse, at the spatial resolution of the chosen layer, so the authors multiply them pointwise with the pixel-level map of Guided Backpropagation (Guided Grad-CAM) when fine detail is wanted. The authors also apply Grad-CAM to captioning and visual question answering models.

Perturbation methods

Perturbation methods change the input and watch the output. Zeiler and Fergus (2014, §4.2, Fig. 6) slid a grey square over test images and plotted the probability of the correct class as a function of its position. Covering a Pomeranian’s face lowered the Pomeranian probability, and with only the face hidden the network answered “tennis ball”. They concluded that the network localizes the object rather than relying only on context. LIME (Ribeiro et al., 2016) fits a sparse linear model to the classifier’s outputs on perturbed versions of one input, in which groups of pixels (superpixels) are switched on or off; the surrogate is meant to be faithful only near that input. SHAP (Lundberg & Lee, 2017) assigns each feature its Shapley value from cooperative game theory, the unique additive attribution with properties the authors call local accuracy, missingness and consistency, and shows that LIME and several other methods are special cases of one additive framework. RISE (Petsiuk et al., 2018) multiplies the image by thousands of random smooth masks (4,000 for VGG16 and 8,000 for ResNet-50 in their experiments) and averages the masks weighted by the resulting class score. The common cost is that a perturbed image is out of the training distribution: a grey square or a blurred patch is itself a pattern the network may react to, so a drop in score mixes the removal of evidence with the network’s response to an unfamiliar input.

Feature visualization

Feature visualization asks what a unit, channel or class responds to, globally. Erhan et al. (2009) found the input that maximizes a unit’s activation by gradient ascent in image space and applied it to unsupervised deep networks; Olah et al. (2017) credit them with the core idea, and Simonyan et al. (2014) applied it to ImageNet classifiers to visualize class models. Zeiler and Fergus (2014, §2.1) projected single strong activations back to pixel space with a deconvnet, which undoes pooling with the max locations recorded in the forward pass and filters with the transposed filters. Their visualizations of the nine strongest activations of sampled channels gave the familiar picture of features that grow from edges to object parts with depth. Inspecting AlexNet’s first two layers the same way showed few mid-frequency filters and aliasing from the stride-4 first layer, and they changed those layers in response (§4.1). Unconstrained optimization tends to produce high-frequency patterns that excite a unit without resembling anything natural. Olah et al. (2017) relate these patterns to adversarial examples and to strided convolution and pooling, and survey the regularizers used against them: frequency penalties, robustness to small transformations, and learned image priors, with stronger regularization giving more realistic images at the risk of misleading correlations. They note that optimization gives one extreme example, while dataset examples show the range of inputs that activate a unit.

Concept-based explanations

Pixel maps answer “where”; concept methods answer “what” in human terms. Network Dissection (Bau et al., 2017) labels individual channels. It thresholds each channel’s upsampled activation at its top 0.5% quantile over the Broden dataset, which merges densely labelled images of objects, parts, scenes, materials, textures and colours. It scores the channel against each concept’s masks by intersection over union, and calls a channel a detector for a concept when that score exceeds a threshold. TCAV (Kim et al., 2018) starts from the concept instead: given example images of “striped” and random counterexamples, it estimates how sensitive the class “zebra” is to that direction in a layer’s activations (the formula above), and tests the score against concept vectors learned from random image sets. Concept bottleneck models (Koh et al., 2020) build concepts into the architecture: the network first predicts annotated concepts, such as bird attributes or radiographic findings, and then predicts the label only from them. A user can correct a predicted concept at test time and see the label change. The authors report accuracy competitive with end-to-end models on knee X-ray grading and bird identification, at the price of concept annotations for training.

Attention maps

Models built on attention, such as the vision transformer, produce attention weights that are easy to display over the image, and they are often shown as explanations. The evidence for treating them that way comes mostly from NLP and recurrent models. Jain and Wallace (2019), working mainly with BiLSTM encoders on text classification, question answering and inference tasks, found attention weights often uncorrelated with gradient-based importance, and found very different attention distributions that gave the same predictions. Wiegreffe and Pinter (2019) answered that whether attention is an explanation depends on the definition of explanation. They proposed tests including a uniform-weights baseline and adversarially trained attention, and concluded that the earlier experiments do not show attention to be useless for explanation. For deep transformers, Abnar and Zuidema (2020) argue that information from different tokens becomes mixed across layers, which makes raw attention unreliable as an explanation probe, and propose attention rollout and attention flow, which combine the weights of all layers. None of the three papers tested vision models, so their results transfer to image transformers only as cautions. An attention map shows how one head in one layer mixed its inputs, which is not the same as what the prediction depended on (see self-attention).

Models designed to be interpretable

The alternative to explaining a black box is a model whose reasoning is visible by construction. Rudin (2019) calls the assumed necessary trade-off between accuracy and interpretability a myth, particularly for structured data with meaningful features. For computer vision, where deep networks gave large gains, she argues that some forms of interpretability can still be built in without losing accuracy. ProtoPNet (Chen et al., 2019) is her group’s example. A prototype layer on top of a convolutional backbone compares patches of the image’s feature map with learned prototypes, each of which is projected onto, and so equal to, a feature-map patch of a training image of its class. The class scores are a weighted sum of these similarities, so a prediction reads as “this part looks like that part of a training image”. On cropped CUB-200-2011 bird images the authors report accuracy comparable to the same backbones without the prototype layer, at most 3.5 points lower. Such models move the question rather than settle it: the prototypes are faithful to the computation, but whether two patches are similar for the reason a person would assume still needs checking.

Evaluating Explanations

There is no ground-truth explanation for a real network, so evaluation relies on tests that a faithful method must pass.

Sanity checks. Adebayo et al. (2018) proposed two randomization tests. In the model-parameter randomization test, the weights are re-initialized, either all at once, cascading from the top layer down, or one layer at a time, and the explanation is recomputed. A method that explains the model should change when the model is destroyed. In the data randomization test, a model is trained on permuted labels, which it can only memorize, and its explanations are compared with those of the properly trained model. They compared maps by Spearman rank correlation (with and without absolute values), SSIM and HOG similarity. On Inception v3 and other models, plain gradients and Grad-CAM passed. Guided Backpropagation and Guided Grad-CAM stayed nearly unchanged however many layers were randomized, behaving like an edge detector that needs neither model nor data. They also warn that visual inspection misleads: maps from gradient times input and integrated gradients still showed the bird’s outline after several blocks were randomized, while their rank correlation without absolute values fell to about zero.

Removal metrics. If the highlighted pixels matter, removing them should hurt. Petsiuk et al. (2018) define a deletion metric, the area under the class-probability curve as the most important pixels are removed (lower is better), and an insertion metric, the area under the curve as they are added to a blurred image (higher is better). Hooker et al. (2019) argue that without retraining such scores confound lost information with the shift to images the network never saw. Their ROAR benchmark removes the top-ranked pixels from training and test images and retrains. On their image classification datasets, many popular methods were no better than a random ranking, and only the ensembling variants SmoothGrad-Squared and VarGrad were better.

Human studies. Doshi-Velez and Kim (2017) distinguish application-grounded evaluation (real users on the real task), human-grounded evaluation (lay users on simplified tasks, such as predicting the model’s output from an explanation) and functionally grounded evaluation (formal proxies with no humans). In LIME’s “husky versus wolf” study, Ribeiro et al. (2016) trained a deliberately bad classifier on 20 images in which every wolf photo, and no husky photo, had snow in the background. Among 27 graduate students, 10 trusted the model before seeing explanations and 3 after, and 12 named snow as a possible feature before against 25 after. Such studies measure whether an explanation helps people, not whether it is faithful.

Examples

Gradient saliency, occlusion and Grad-CAM on a shortcut. The PyTorch example below (PyTorch 2.14.1, about 17 seconds on one CPU thread) trains a small CNN to tell discs from squares in noisy $32 \times 32$ images. In training, a $4 \times 4$ bright marker in the top-left corner accompanies 95% of the squares and 5% of the discs, a spurious cue. Accuracy is measured on three test sets: the training distribution, no marker, and the marker on every disc and no square. On 100 squares that all carry the marker, the script computes the three attribution maps for the class “square”. It reports the share of attribution falling on the square and on the marker, and in brackets that share divided by the region’s share of the pixels. It then applies Adebayo et al.’s model-parameter randomization test, re-initializing all weights at once, and reports the mean Spearman rank correlation between each trained map and the map from the random network. Three training seeds show the spread, and the rank correlation of seeds 1 and 2’s maps with seed 0’s gives a reference for how similar maps of two trained models are.

import numpy as np
import torch
from torch import nn
from scipy.stats import spearmanr
F = nn.functional
torch.set_num_threads(1)                    # one thread: identical output on every run

def make_images(n, p_marker, seed):
    """32x32 noisy images: class 0 = disc, class 1 = square, at random positions.
    A 4x4 bright marker in the top-left corner appears with probability
    p_marker[label]. Returns images, labels, object masks and marker masks."""
    rng = np.random.default_rng(seed)
    yy, xx = np.mgrid[:32, :32]
    X, Y, OBJ, MRK = [], [], [], []
    for i in range(n):
        label = i % 2
        r = rng.uniform(5, 8)                              # half-size of the shape
        cy, cx = rng.uniform(r + 8, 31 - r, size=2)        # keeps clear of the corner
        if label == 0:
            obj = (yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2
        else:
            obj = np.maximum(abs(yy - cy), abs(xx - cx)) <= r * 0.886   # same area as the disc
        mrk = np.zeros((32, 32), bool)
        if rng.random() < p_marker[label]:
            mrk[1:5, 1:5] = True
        fg, bg = rng.uniform(0.6, 1.0), rng.uniform(0.0, 0.3)
        img = bg + (fg - bg) * (obj | mrk) + rng.normal(0, 0.15, (32, 32))
        X.append(img); Y.append(label); OBJ.append(obj); MRK.append(mrk)
    X = torch.tensor(np.array(X), dtype=torch.float32).unsqueeze(1)
    return X, torch.tensor(Y), np.array(OBJ), np.array(MRK)

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 32, 3, padding=1), nn.ReLU())     # 32 maps of 8x8: Grad-CAM layer
        self.head = nn.Linear(32, 2)
    def forward(self, x, keep=False):
        a = self.body(x)
        if keep:
            self.maps = a                                   # kept for Grad-CAM
        return self.head(a.mean((2, 3)))                    # global average pooling

def train(seed, X, y, steps=600, batch=32):
    torch.manual_seed(seed)
    net = SmallCNN()
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    g = torch.Generator().manual_seed(seed)
    for _ in range(steps):
        idx = torch.randint(len(y), (batch,), generator=g)
        loss = F.cross_entropy(net(X[idx]), y[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    return net

def accuracy(net, X, y):
    with torch.no_grad():
        return (net(X).argmax(1) == y).float().mean().item()

# ---- three attribution methods, each returning a nonnegative 32x32 map per image --------------
def gradient_saliency(net, X, c):
    """|d logit_c / d pixel| (Simonyan et al. 2014)."""
    X = X.clone().requires_grad_(True)
    net(X)[:, c].sum().backward()
    return X.grad.abs()[:, 0].numpy()

def grad_cam(net, X, c):
    """ReLU(sum_k alpha_k A^k), alpha_k = spatial mean of d logit_c / d A^k (Selvaraju et al. 2017)."""
    out = net(X, keep=True)
    A = net.maps
    dA, = torch.autograd.grad(out[:, c].sum(), A)
    cam = F.relu((dA.mean((2, 3), keepdim=True) * A).sum(1, keepdim=True))
    return F.interpolate(cam, size=32, mode="bilinear", align_corners=False)[:, 0].detach().numpy()

def occlusion(net, X, c, size=4, stride=4):
    """Drop in logit_c when a size x size patch is set to the image median (Zeiler & Fergus 2014)."""
    with torch.no_grad():
        base = net(X)[:, c]
        heat = torch.zeros(len(X), 32, 32); count = torch.zeros(32, 32)
        fill = X.flatten(1).median(1).values.view(-1, 1, 1)
        for i in range(0, 32 - size + 1, stride):
            for j in range(0, 32 - size + 1, stride):
                Xo = X.clone()
                Xo[:, 0, i:i + size, j:j + size] = fill
                drop = (base - net(Xo)[:, c]).view(-1, 1, 1)
                heat[:, i:i + size, j:j + size] += drop
                count[i:i + size, j:j + size] += 1
        return (heat / count).clamp(min=0).numpy()

METHODS = {"gradient": gradient_saliency, "occlusion": occlusion, "Grad-CAM": grad_cam}

def mass(maps, region):
    """Fraction of total attribution falling inside a region, averaged over images."""
    tot = maps.sum((1, 2)) + 1e-12
    return float(np.mean((maps * region).sum((1, 2)) / tot))

def rank_corr(A, B):
    """Mean Spearman correlation between paired maps; maps that are all zero are skipped."""
    ok = (A.reshape(len(A), -1).std(1) > 0) & (B.reshape(len(B), -1).std(1) > 0)
    if not ok.any():
        return "   all zero"
    r = np.mean([spearmanr(a.ravel(), b.ravel())[0] for a, b in zip(A[ok], B[ok])])
    return f"{r:+.2f}" + (f" ({ok.sum()} maps)" if not ok.all() else "")

# ---- data: the marker accompanies 95% of squares and 5% of discs in training ------------------
Xtr, ytr, _, _ = make_images(2000, p_marker=(0.05, 0.95), seed=0)
tests = {"as in training": make_images(1000, (0.05, 0.95), seed=1),
         "no marker":      make_images(1000, (0.0, 0.0), seed=2),
         "marker on discs only": make_images(1000, (1.0, 0.0), seed=3)}
# probe set: squares that all carry the marker
Xp, yp, obj, mrk = make_images(200, (0.0, 1.0), seed=4)
sq = yp.numpy() == 1
Xp, obj, mrk = Xp[sq], obj[sq], mrk[sq]
print(f"torch {torch.__version__}; probe set: {len(Xp)} squares with the marker; "
      f"object {obj.mean():.1%} of pixels, marker {mrk.mean():.1%}")

first = {}
for seed in range(3):
    net = train(seed, Xtr, ytr)
    torch.manual_seed(100 + seed); rand = SmallCNN()        # same architecture, fresh weights
    print(f"\nseed {seed}, test accuracy: " +
          ", ".join(f"{k} {accuracy(net, *v[:2]):.3f}" for k, v in tests.items()))
    print("  method     share on object  share on marker  rank corr. with seed 0  with random weights")
    for name, f in METHODS.items():
        m, mr = f(net, Xp, 1), f(rand, Xp, 1)               # attributions for "square"
        first.setdefault(name, m)
        print(f"  {name:9s}  {mass(m, obj):.2f} ({mass(m, obj) / obj.mean():.1f}x)"
              f"      {mass(m, mrk):.2f} ({mass(m, mrk) / mrk.mean():.1f}x)"
              f"     {rank_corr(m, first[name]) if seed else '  (reference)':>13s}"
              f"          {rank_corr(m, mr)}")

Output:

torch 2.14.1; probe set: 100 squares with the marker; object 12.9% of pixels, marker 1.6%

seed 0, test accuracy: as in training 0.987, no marker 0.892, marker on discs only 0.790
  method     share on object  share on marker  rank corr. with seed 0  with random weights
  gradient   0.32 (2.5x)      0.06 (3.8x)       (reference)          +0.10
  occlusion  0.31 (2.4x)      0.17 (10.8x)       (reference)          -0.18
  Grad-CAM   0.24 (1.8x)      0.10 (6.1x)       (reference)             all zero

seed 1, test accuracy: as in training 0.980, no marker 0.967, marker on discs only 0.720
  method     share on object  share on marker  rank corr. with seed 0  with random weights
  gradient   0.28 (2.2x)      0.05 (3.3x)             +0.49          +0.02
  occlusion  0.34 (2.6x)      0.13 (8.1x)             +0.56          +0.00
  Grad-CAM   0.24 (1.8x)      0.08 (5.3x)             +0.72          -0.00

seed 2, test accuracy: as in training 0.973, no marker 0.983, marker on discs only 0.821
  method     share on object  share on marker  rank corr. with seed 0  with random weights
  gradient   0.36 (2.8x)      0.04 (2.4x)             +0.46          +0.04
  occlusion  0.33 (2.5x)      0.10 (6.6x)             +0.47          +0.23
  Grad-CAM   0.28 (2.2x)      0.06 (4.0x)             +0.54          +0.08 (99 maps)

The networks learn mostly the shape: with no marker at all, accuracy stays between 89% and 98%. But putting the marker on every disc lowers it to 72–82%, so each network also uses the marker. All three methods register this. Relative to its area, the marker receives 2.4 to 10.8 times its share of the pixels, against 1.8 to 2.8 times for the square, and more than the square in every run except seed 2’s gradient map. Occlusion, which measures the effect of removing a region directly, gives it the largest share. In absolute terms, though, the marker’s share is small, 4–6% of the gradient map and at most 17% of any map. In our reading, someone inspecting individual gradient maps could take a faint corner spot for noise. The test set that separates the cue from the label measured the reliance; the maps showed where to look.

In the randomization test, the maps of a freshly initialized network have rank correlations between −0.18 and +0.23 with the trained maps, against +0.46 to +0.72 between maps of two trained seeds. All three methods therefore depend on the learned weights, as Adebayo et al. found for gradients and Grad-CAM. Grad-CAM of a random network can be entirely zero (all 100 maps for seed 0, one map for seed 2): when the summed evidence for the class is negative everywhere, the ReLU removes it. The seed-to-seed correlations are themselves well below one. Three networks of similar accuracy give partly different explanations of the same images, so a single map describes one trained model, not the task.

Uses in Vision

Debugging shortcut learning. Geirhos et al. (2020) call decision rules that score well on standard benchmarks but fail under harder conditions shortcuts. Explanations are one way to find them. Lapuschkin et al. (2019) found that a Fisher-vector classifier trained on PASCAL VOC 2007 recognized horses largely by a source tag in the image corner, present in about a fifth of the horse images. Removing the tag removed the “horse” decision, while the heat map of a deep network fine-tuned on the same data pointed at the horse and rider. LIME’s husky-versus-wolf study (above) staged the same failure. Both are cases of dataset bias that a test set drawn from the same collection does not reveal.

From hypothesis to test. DeGrave, Janizek and Lee (2021) trained chest-radiograph classifiers for COVID-19 the way several published models had been trained, on positive and negative images from different sources. Their ROC AUC fell from 0.992–0.995 on held-out images from the training sources to 0.70–0.76 on a second dataset (their Fig. 1c, both training directions). Saliency maps highlighted text and laterality markers, and images edited by a generative adversarial network to flip the prediction changed the markers and the radiopacity at the image borders. The authors then tested those hypotheses by editing radiographs: swapping laterality markers between a positive and a negative image shifted the output significantly more than swapping random patches of the same size. The explanation suggested where to look, and the intervention showed that the cue was used.

Medical imaging caution. Saliency maps are often shown to clinicians as decision aids. Saporta et al. (2022) compared seven saliency methods on chest X-ray classifiers with a human benchmark for localizing pathologies. Grad-CAM generally localized best of the seven, but all seven were significantly worse than the human benchmark. Grad-CAM’s gap to the human benchmark was largest for small pathologies with complex shapes, and its localization was positively correlated with model confidence. In our reading, a heat map over the right region does not show that the model used that region for the right reason, and one over the wrong region may reflect the method’s coarse resolution rather than the model.

Model development and science. Zeiler and Fergus’s architecture changes, and network dissection’s comparison of how many concept detectors different networks and training regimes produce, are uses of interpretability to understand and improve models rather than to justify single predictions. Probing what learned representations encode with linear classifiers is a neighbouring practice.

History

Feature visualization came first. Erhan et al. (2009) visualized units of unsupervised deep networks by optimizing the input. When convolutional networks won ImageNet in 2012, the question became what such networks had learned. Zeiler and Fergus posted their deconvnet and occlusion study in November 2013 (arXiv; ECCV 2014), and Simonyan, Vedaldi and Zisserman posted gradient saliency and class-model visualization in December 2013 (ICLR 2014 workshop). The next step moved from individual units to class evidence that could be localized: CAM in 2016, and in 2016–2017 the model-agnostic LIME and SHAP, Grad-CAM, and axiomatic methods such as integrated gradients. Network Dissection (2017) and TCAV (2018) shifted the question from pixels to concepts.

DARPA’s Explainable Artificial Intelligence (XAI) programme held its proposers day on 11 August 2016 (Gunning, 2016). Its planned schedule ran from 2017 to 2021 in two phases, technology demonstrations and comparative evaluations, and aimed at more explainable models that keep high prediction accuracy together with interfaces that turn them into explanations for users. As of October 2026, DARPA’s programme page describes it as complete. Doshi-Velez and Kim (2017) and Lipton (2016) asked at the same time what interpretability should mean and how to evaluate it.

From 2018 the critical work grew: sanity checks (2018), Rudin’s case for interpretable models and ProtoPNet (2019), the attention debate and ROAR (2019), and definitions of faithfulness (2020). The emphasis moved from producing maps to testing them.

Common Misconceptions

• “A saliency map shows why the network decided.” It shows where the score is sensitive or where evidence was found, not what the network did with it, and maps for different classes, or for a trained and a randomized network, can look alike (Rudin, 2019; Adebayo et al., 2018).
• “A map that highlights the object proves the model is right.” Plausibility is not faithfulness (Jacovi & Goldberg, 2020). In the example above, every method gave the square 1.8 to 2.8 times its share of the pixels, yet accuracy fell from 97–99% to 72–82% when the corner marker moved to the discs.
• “Interpretable models must be less accurate.” Rudin (2019) calls a necessary trade-off a myth, especially for structured data, and ProtoPNet reached accuracy comparable to its non-interpretable backbones on CUB-200-2011 (Chen et al., 2019). The trade-off can exist; it has to be shown case by case.
• “Feature visualizations show the feature.” An optimized image shows one input that strongly excites a unit, shaped by the regularizer; dataset examples and diverse optimizations are needed to see more of what the unit responds to (Olah et al., 2017).

Where It Is Used

• Image classification is the main setting for saliency maps, Grad-CAM, occlusion and concept tests.
• Weakly supervised localization uses CAM-style maps as object locations learned from image labels (Zhou et al., 2016; Simonyan et al., 2014).
• Dataset auditing and medical imaging use explanations to look for shortcuts before deployment, with the limits described above.
• Architecture design and analysis: visualizing feature maps, the receptive field of units, and attention in vision transformers.