Saliency Maps

A saliency map, in the sense used for explaining classifiers, assigns each input pixel a score for how much a network’s output for one class depends on it. The family covered here computes those scores from gradients: the derivative of the class score with respect to the pixels, and refinements that integrate, average, multiply or modify that derivative. It belongs to the broader field of neural network interpretability, which also covers class activation maps such as Grad-CAM, perturbation methods, concept tests and how all of them are evaluated. Because a gradient is computed by one backpropagation pass, a saliency map costs about as much as a training step on one image and has the input’s full resolution.

The name collides with an older one. Salient object detection and fixation prediction also output “saliency maps”, but there the map predicts what a person would notice or look at, is learned from human annotations or eye tracking, and is scored against them. A classifier’s saliency map makes no claim about human attention: it describes one model’s sensitivity, and it is right or wrong only relative to that model.

Purpose

An image classifier maps tens of thousands of pixel values to a handful of class scores. For a single prediction, one wants to know which pixels the score responded to: whether a “husky” decision rests on the dog or on the snow, whether a radiograph classifier looks at the lung or at a laterality marker. The interpretability hub describes such cases, including the chest-radiograph shortcuts that DeGrave et al. found with saliency maps and then confirmed by editing the images (see also radiology AI).

Gradient methods answer this with the network’s own derivatives, without retraining it or querying it thousands of times. Sundararajan, Taly and Yan (2017) frame the task as attribution: given an input $x$ and a baseline input $x'$, return a vector $(a_1, \dots, a_n)$ whose entry $a_i$ is the contribution of feature $x_i$ to the prediction. Their examples include telling a doctor which part of a retinal image drove a diabetic retinopathy grade, and helping developers debug a model. A saliency map is that vector arranged as an image.

When to Use It

Gradient-based maps fit when:

• the model is differentiable and its weights are available (a white-box setting);
• one wants a per-pixel, per-image, per-class explanation, at the cost of a few backward passes;
• the question is local: why this prediction, on this image.

They fit less well when:

• the model is only available as an API, where perturbation methods such as occlusion or LIME (see the interpretability hub) are the option;
• one wants to know what a unit or class detects in general, which is the question of feature visualization;
• a coarse, class-discriminative region is more useful than pixel detail, which is what Grad-CAM and its relatives provide;
• the map will be used as evidence for a decision about a person without a test of whether it reflects the model (see Evidence and Impact).

How It Works

Let $f_c(x)$ be the score of class $c$ for an image $x$ with pixels $x_i$, usually the logit before the softmax. All the methods below derive the map from gradients $\partial f_c / \partial x_i$, computed by reverse-mode automatic differentiation.

Vanilla gradients

The plain gradient map is

$$M_i = \left| \frac{\partial f_c}{\partial x_i}(x) \right|,$$

the local sensitivity of the class score to each pixel. As the interpretability hub describes, Simonyan, Vedaldi and Zisserman (2014) computed it as an image-specific class saliency map, took the maximum magnitude over colour channels, and used the maps, with a GraphCut colour model, for weakly supervised object segmentation. The gradient is a first-order Taylor coefficient: near $x$, $f_c(x + \delta) \approx f_c(x) + \nabla f_c(x)^\top \delta$. It answers “which pixels, changed slightly, would change the score most”, which is not the same question as “which pixels produced the score”. Shrikumar, Greenside and Kundaje (2017) summarize the authors’ point that the computation is similar to the Zeiler and Fergus deconvnet except at ReLUs.

Two well-known problems follow from this definition. Gradient maps of real networks look speckled and noisy, and SmoothGrad, as Sturmfels, Lundberg and Lee (2020) describe it, was designed to sharpen them. And a gradient can be zero where the input matters, which motivated integrated gradients.

Gradient times input

Multiplying the gradient elementwise by the input,

$$M_i = x_i \, \frac{\partial f_c}{\partial x_i}(x),$$

keeps the sign and weights each pixel by its value. For a linear model $f(x) = w^\top x$ the products $w_i x_i$ sum exactly to the score, which is why Sundararajan et al. call gradient times input “a reasonable starting point”: practitioners inspect coefficient-times-value products to debug linear models, and the gradient plays the role of the coefficients in a deep network. Shrikumar et al. (2017) report that, without the modifications added for numerical stability, the original rules of layer-wise relevance propagation (LRP; Bach et al., 2015) are equivalent within a scaling factor to gradient times input, a result they credit to their own 2016 preprint and to Kindermans et al. (2016). They also note that gradient times input implicitly measures against an all-zero reference, since it is the first-order Taylor approximation of the change from an input of zeros.

Deconvnet and guided backpropagation

Backpropagation-based visualizations change one rule of the backward pass. With $R_{n+1}$ the signal arriving at a ReLU from above and $X_n$ the ReLU’s forward input, the true gradient passes $R_{n+1}$ where $X_n > 0$; the deconvnet of Zeiler and Fergus passes it where $R_{n+1} > 0$; and guided backpropagation (Springenberg et al., 2015) passes it only where both hold. The Zeiler and Fergus paper article derives the three rules, checks in code that a deconvnet is the backward pass with only the ReLU rule replaced, and compares the three projections on trained and random networks.

Guided backpropagation gives sharp, image-like maps, and that sharpness turned out to be the problem. Shrikumar et al. point out that by zeroing negative signals, the deconvnet and guided backpropagation can fail to highlight inputs that contribute negatively. Nie, Zhang and Patel (2018) argue theoretically, and support with experiments, that guided backpropagation and the deconvnet essentially perform partial image recovery that is unrelated to the network’s decision; they identify the backward ReLU and the local connections of CNNs as the two main causes of the compelling pictures. Sanity checks (Adebayo et al., 2018), summarized below as the interpretability hub reports them, point to the same insensitivity from the other side.

Integrated gradients

Integrated gradients (Sundararajan et al., 2017) replace the single gradient with an average of gradients along the straight path from a baseline $x'$ to the input:

$$\mathrm{IG}_i(x) = (x_i - x'_i) \int_0^1 \frac{\partial f\big(x' + \alpha (x - x')\big)}{\partial x_i}\, d\alpha .$$

In practice the integral is a Riemann sum over $m$ steps (their Eq. 3),

$$\mathrm{IG}_i(x) \approx (x_i - x'_i) \, \frac{1}{m} \sum_{k=1}^{m} \frac{\partial f\big(x' + \tfrac{k}{m}(x - x')\big)}{\partial x_i},$$

which is $m$ gradient evaluations, batched in any framework.

The method is defined by axioms rather than by a picture:

• Sensitivity(a). If the input and baseline differ in one feature and the predictions differ, that feature gets a nonzero attribution. Plain gradients break it. The authors’ example is the one-ReLU network $f(x) = 1 - \mathrm{ReLU}(1 - x)$ with baseline $x' = 0$ and input $x = 2$: the output goes from 0 to 1, but $f$ is flat at $x = 2$, so the gradient there is 0. They state that the deconvnet and guided backpropagation break it too, because they pass signal through a ReLU only if the ReLU is on at the input.
• Implementation invariance. Two networks that compute the same function get the same attributions. Gradients satisfy it, since the chain rule does not depend on how the function is implemented. Methods that backpropagate “discrete gradients”, such as DeepLIFT and LRP, break it in the authors’ analysis, because the chain rule does not hold for ratios of differences.
• Completeness. The attributions add up to the difference between the outputs at the input and at the baseline: $$\sum_i \mathrm{IG}_i(x) = f(x) - f(x').$$ This is the fundamental theorem of calculus for path integrals (their Proposition 1), and it implies Sensitivity(a).
• Sensitivity(b), or dummy. A variable the function does not depend on gets zero attribution.
• Linearity. Attributions for $a f_1 + b f_2$ are the corresponding combination of the attributions for $f_1$ and $f_2$.

Citing Friedman (2004), the authors state that path methods, which integrate gradients along any monotone path from $x'$ to $x$, are the only attribution methods that always satisfy implementation invariance, Sensitivity(b), linearity and completeness. Among path methods, the straight line is the only one that is symmetry-preserving: two variables that play interchangeable roles, with equal values in the input and equal values in the baseline, get equal attributions (their Theorem 1). In cost-sharing terms, integrated gradients is the Aumann–Shapley method. Averaging over many paths gives other methods with the same axioms, such as the Shapley–Shubik method; for $f = \min(x_1, x_2)$ with input $(1, 3)$ and baseline $(0, 0)$, integrated gradients assigns the whole change to $x_1$ while Shapley–Shubik splits it in half, and the authors call the preference between them “somewhat subjective”.

The authors’ practical advice is to check completeness: between 20 and 300 steps usually approximated the integral within 5% in their experiments, and if the attributions do not add up to $f(x) - f(x')$, $m$ should be increased. Their neural machine translation model, which is highly nonlinear, needed 100 to 1,000 steps.

The baseline

Every attribution with completeness explains a difference, $f(x) - f(x')$, so the baseline decides what question is answered. Sundararajan et al. recommend a baseline whose score is near zero, so the attributions can be read as a distribution of the output itself, and one that conveys a complete absence of signal. For images they use the black image, note that “an image consisting of noise” has the same property, and warn against baselines such as an adversarially constructed image with zero score, whose artifacts would appear in the attributions.

Sturmfels, Lundberg and Lee (2020) show what the choice costs. Since $\mathrm{IG}_i$ contains the factor $x_i - x'_i$, any pixel equal to the baseline gets zero attribution, so a black baseline cannot highlight black pixels even when they form the object; every constant-colour baseline is blind to its own colour. They compare four alternatives on Inception V4:

• the maximum-distance baseline, the farthest valid image from the input in $L_1$ distance, which avoids blindness but contains much of the image’s information;
• a blurred copy of the input, following Fong and Vedaldi’s use of blur to represent missing information, which tends to favour high-frequency detail;
• a uniform-noise image in the valid range;
• a Gaussian baseline centred on the input.

DeepLIFT’s authors made a similar observation on CIFAR-10: a blurred reference highlighted the outlines of key objects, while an all-zero reference highlighted hard-to-interpret background pixels. Averaging the attributions over a distribution of baselines, with the interpolation point also sampled, gives expected gradients, a formulation Sturmfels et al. credit to Erion et al. They find visual inspection misleading. In their ablation tests on 1,000 images, the training-distribution and uniform baselines did well when the top-ranked pixels were removed, and all baselines did about equally, and only slightly better than random noise, when a box around the map’s centre of mass was removed. They then question whether ablation tests are well founded at all, since ablated images lie outside the training distribution.

SmoothGrad

SmoothGrad (Smilkov et al., 2017), as Sturmfels et al. describe it, averages saliency maps over $n$ copies of the image with added Gaussian noise:

$$M_{\mathrm{SG}}(x) = \frac{1}{n} \sum_{j=1}^{n} M\big(x + \varepsilon_j\big), \qquad \varepsilon_j \sim \mathcal{N}(0, \sigma^2 I).$$

As the interpretability hub reports, the authors found noise of 10–20% of the input range useful on Inception v3 and little visible change beyond 50 samples. Sturmfels et al. show that the gradient-times-input variant of SmoothGrad is close to expected gradients with a Gaussian baseline. The two differ in that SmoothGrad multiplies the gradient by $x + \varepsilon$ rather than by $\varepsilon$, and it always samples the end of the path instead of points along it. They suggest that SmoothGrad’s smoothness comes from the implicit assumption that pixels are independent Gaussian, which breaks the correlations between neighbouring pixels. A way to read SmoothGrad that follows from its definition is that the averaged gradient estimates the gradient of the smoothed score $\mathbb{E}_\varepsilon f(x + \varepsilon)$, so $\sigma$ sets the scale at which the network is examined. Whether one averages signed gradients and then takes magnitudes, or averages magnitudes, is a further choice, and the example below shows that it matters.

Attention rollout and attention flow

For transformers, and the vision transformer in particular, the attention weights of the class token over image patches are easy to display, but they describe how one layer mixes its inputs. Abnar and Zuidema (2020) observe that information from different tokens becomes increasingly mixed across layers, so raw attention in higher layers is unreliable as an explanation. They model the network as a graph whose nodes are tokens at each layer and whose edge weights are attention weights, averaged over heads. To account for residual connections, they replace each layer’s attention matrix $W_{\mathrm{att}}$ by

$$A = 0.5\, W_{\mathrm{att}} + 0.5\, I .$$

• Attention rollout treats weights as proportions of information and multiplies them along every path, summing over paths. For layers $1$ to $L$ this is the matrix product $\tilde A = A^{(L)} A^{(L-1)} \cdots A^{(1)}$, and the class token’s row gives a weight per input token.
• Attention flow treats the weights as edge capacities and computes the maximum flow from a node in a higher layer to each input token, so a path is limited by its weakest edge rather than by the product of its weights.

Rollout costs $O(d\,n^2)$ and flow $O(d^2 n^4)$ for depth $d$ and $n$ tokens, by the authors’ count. Rollout is the method behind the class-token attention maps of the ViT paper. Neither method uses gradients; they are included here because they answer the same question for transformers and are compared with gradient maps.

Chefer, Gur and Wolf (2021) point out that rollout’s result is fixed for a given image whatever the target class, and that it cannot separate positive from negative contributions. Their method propagates LRP-style relevance through all layers, multiplies each block’s attention relevance by the gradient of the attention map with respect to the target class, keeps the positive part, averages over heads, adds the identity, and multiplies the blocks together as rollout does. The result is class-specific. Their rollout baseline, written as $I + \mathbb{E}_h A^{(b)}$ with the mean taken over heads, adds the identity without Abnar and Zuidema’s halving. Since $I + W = 2(0.5\,W + 0.5\,I)$, the two products differ by the constant factor $2^B$ for $B$ blocks, and the relative token weights are the same.

How to Apply It

The PyTorch example below makes the main properties of integrated gradients and SmoothGrad visible on a network whose behaviour is known. Nothing is trained. The network is a fixed, hand-built detector of striped texture:

• four Sobel edge filters, each followed by a ReLU that passes responses above 0.5;
• a $5 \times 5$ average of the summed edge maps, followed by a ReLU above 0.3, which fires where edges are dense;
• the mean over positions times 8 gives the logit $z$, and the score is capped, $f = \min(z, 1)$, like Sundararajan et al.’s $1 - \mathrm{ReLU}(1 - x)$.

The $48 \times 48$ image has a flat background of 0.3 and a disc of radius 10 filled with vertical stripes of period 4 pixels, plus Gaussian noise of standard deviation 0.03, clipped to $[0, 1]$. The disc is the ground truth: with the stripes replaced by background, $z$ is exactly 0, so every unit that fires sees the disc within its $7 \times 7$ receptive field. A second image with stronger stripes pushes $z$ above the cap. Maps are compared by the share of their absolute mass on the disc and on the “far background”, pixels more than 3 pixels from the disc, and by their total variation per unit mass,

$$\mathrm{TV}(M) = \frac{\sum |M_{r,c+1} - M_{r,c}| + \sum |M_{r+1,c} - M_{r,c}|}{\sum |M_{r,c}|},$$

a measure of how speckled a map is.

The script runs four parts:

• A. It measures the completeness error of integrated gradients from a black baseline against the number of steps $m$.
• B. It computes the gradient, gradient times input and integrated gradients on the saturated image.
• C. It compares baselines: black, blurred (Gaussian, $\sigma = 5$ pixels), a single uniform-noise image (five draws, printed as a range) and the average over 16 uniform-noise images. The first rows are the gradient and gradient times input on the same image, as references. Their share-on-object measure follows the example in Interpretability of Neural Networks, which computes it for gradients of a trained CNN.
• D. It runs SmoothGrad on the logit $z$ for several noise levels and sample counts, with both map definitions (magnitude of the mean gradient, and mean of the gradient magnitudes), over noise seeds 0 to 4.

It used PyTorch 2.14.1, NumPy 2.5.4 and SciPy 1.18.1, in double precision on one CPU thread, and took about 4–6 seconds.

import numpy as np
import torch
import torch.nn.functional as F
from scipy import ndimage

torch.set_num_threads(1)                      # one thread: identical output on every run
torch.set_default_dtype(torch.float64)
N = 48
yy, xx = np.mgrid[:N, :N]
OBJ = (yy - 20) ** 2 + (xx - 27) ** 2 <= 10 ** 2          # object mask: disc of radius 10
FAR = ~ndimage.binary_dilation(OBJ, iterations=3)          # more than 3 px from the object


def make_image(amp=0.3, with_object=True, seed=0):
    """Disc of vertical stripes (period 4 px) on a flat background of 0.3, plus noise."""
    rng = np.random.default_rng(seed)
    stripes = 0.5 + amp * np.sin(np.pi * xx / 2)
    img = np.where(OBJ & with_object, stripes, 0.3) + rng.normal(0, 0.03, (N, N))
    return torch.tensor(np.clip(img, 0, 1))


# ---- a fixed, hand-built "striped texture" detector (no training) ----------------------------
SX = torch.tensor([[-1., 0, 1], [-2, 0, 2], [-1, 0, 1]])
K = torch.stack([SX, -SX, SX.T, -SX.T]).unsqueeze(1)       # four Sobel edge detectors


def logit(x):                                              # x: (B, 48, 48) -> (B,)
    h1 = F.relu(F.conv2d(x.unsqueeze(1), K) - 0.5)          # edges stronger than 0.5
    e = F.avg_pool2d(h1.sum(1, keepdim=True), 5, stride=1)  # local edge energy, 5x5
    return 8.0 * F.relu(e - 0.3).mean((1, 2, 3))            # textured area, scaled


def score(x):
    return torch.clamp(logit(x), max=1.0)                   # capped: f = min(z, 1)


def gradient(x, fn=score):                                 # d fn / dx for a batch
    x = x.clone().requires_grad_(True)
    fn(x).sum().backward()
    return x.grad


def integrated_gradients(x, base, m):
    """Right Riemann sum of Sundararajan et al.'s Eq. 3 with m steps."""
    alphas = torch.arange(1, m + 1).double().view(-1, 1, 1) / m
    g = torch.cat([gradient(base + a * (x - base)) for a in alphas.split(1024)])
    return (x - base) * g.mean(0)


def smoothgrad(x, sigma, n, seed):
    """Gradients of the logit at n noisy copies; returns |mean| and mean of |.|."""
    gen = torch.Generator().manual_seed(seed)
    g = gradient(x + sigma * torch.randn(n, N, N, generator=gen), logit)
    return g.mean(0).abs(), g.abs().mean(0)


def share(M, region):                                      # fraction of |M| in a region
    M = M.abs().numpy()
    return (M * region).sum() / M.sum()


def tv(M):                                                 # total variation per unit mass
    M = M.abs()
    return float(((M[:, 1:] - M[:, :-1]).abs().sum() + (M[1:] - M[:-1]).abs().sum()) / M.sum())


f = lambda im: score(im[None]).item()
x, xs, x0 = make_image(), make_image(amp=0.45), make_image(with_object=False)
black = torch.zeros(N, N)
blur = torch.tensor(ndimage.gaussian_filter(x.numpy(), 5))
print(f"torch {torch.__version__}; object {OBJ.mean():.1%} of pixels, far background {FAR.mean():.1%}")
print(f"z: image {logit(x[None]).item():.4f}, high-contrast image {logit(xs[None]).item():.4f}, "
      f"object removed {logit(x0[None]).item():.4f}, black {f(black):.4f}, blurred {f(blur):.4f}")

print("\nA. completeness, black baseline: |sum IG - (f(x) - f(x'))| / |f(x) - f(x')|")
for m in (4, 16, 64, 256, 1024, 4096):
    errs = [abs(integrated_gradients(im, black, m).sum().item() - (f(im) - f(black))) / abs(f(im) - f(black))
            for im in (x, xs)]
    print(f"  m = {m:4d}:  image {errs[0]:.2e}   high-contrast (saturated) image {errs[1]:.2e}")

print("\nB. saturated input (z > 1, so f = 1)")
gs = gradient(xs[None])[0]
igs = integrated_gradients(xs, black, 1024)
print(f"  max |gradient| = {gs.abs().max():.1e}, sum gradient x input = {(gs * xs).sum():.4f}")
print(f"  sum IG = {igs.sum():.4f} (f(x) - f(x') = {f(xs) - f(black):.4f}), "
      f"share on object {share(igs, OBJ):.2f}, far background {share(igs, FAR):.2f}")

print("\nC. baselines (m = 1024)       f(x')   sum    f(x)-f(x')  object  far bg   TV")
g = gradient(x[None])[0]
for name, M in [("gradient |df/dx|", g.abs()), ("gradient x input", g * x)]:
    total = f"{M.sum():6.3f}" if name.startswith("gradient x") else "     -"
    print(f"  {name:26s}    -   {total}       -      {share(M, OBJ):.2f}    {share(M, FAR):.2f}   {tv(M):.2f}")
for name, base in [("IG, black", black), ("IG, blurred (sigma 5 px)", blur)]:
    M = integrated_gradients(x, base, 1024)
    print(f"  {name:26s} {f(base):.3f}  {M.sum():6.3f}    {f(x) - f(base):6.3f}    "
          f"{share(M, OBJ):.2f}    {share(M, FAR):.2f}   {tv(M):.2f}")
noise = [torch.tensor(np.random.default_rng(1000 + s).uniform(0, 1, (N, N))) for s in range(16)]
rows = []
for b in noise[:5]:
    M = integrated_gradients(x, b, 1024)
    rows.append((f(b), M.sum().item(), f(x) - f(b), share(M, OBJ), share(M, FAR), tv(M)))
lo, hi = np.min(rows, 0), np.max(rows, 0)
print("  IG, uniform noise, 1 draw (range over 5 draws):")
print("   " + "  ".join(f"{a:.2f} to {b:.2f}" for a, b in zip(lo, hi)))
M = sum(integrated_gradients(x, b, 1024) for b in noise) / 16
fb = np.mean([f(b) for b in noise])
print(f"  {'IG, uniform noise, 16 avg':26s} {fb:.3f}  {M.sum():6.3f}    {f(x) - fb:6.3f}    "
      f"{share(M, OBJ):.2f}    {share(M, FAR):.2f}   {tv(M):.2f}")

print("\nD. SmoothGrad of the logit z, range over noise seeds 0-4      |mean grad|            mean |grad|")
print("                                            TV          object       TV          object")
sg0 = smoothgrad(x, 0.0, 50, 0)
print(f"  check: sigma 0 equals the gradient, max difference {(sg0[0] - gradient(x[None], logit)[0].abs()).abs().max():.1e}")
for sigma, n in [(0.0, 50), (0.05, 50), (0.15, 50), (0.3, 50), (0.6, 50),
                 (0.15, 1), (0.15, 5), (0.15, 20), (0.15, 200)]:
    r = np.array([[v for M in smoothgrad(x, sigma, n, s) for v in (tv(M), share(M, OBJ))] for s in range(5)])
    print(f"  sigma {sigma:4.2f}, n {n:3d}:  " +
          "  ".join(f"{a:.2f}-{b:.2f}" for a, b in zip(r.min(0), r.max(0))))

Output:

torch 2.14.1; object 13.8% of pixels, far background 77.9%
z: image 0.7585, high-contrast image 1.5930, object removed 0.0000, black 0.0000, blurred 0.0000

A. completeness, black baseline: |sum IG - (f(x) - f(x'))| / |f(x) - f(x')|
  m =    4:  image 2.66e-01   high-contrast (saturated) image 1.66e-01
  m =   16:  image 7.87e-02   high-contrast (saturated) image 1.95e-02
  m =   64:  image 1.78e-02   high-contrast (saturated) image 5.83e-03
  m =  256:  image 4.39e-03   high-contrast (saturated) image 1.14e-03
  m = 1024:  image 1.06e-03   high-contrast (saturated) image 9.60e-05
  m = 4096:  image 2.84e-04   high-contrast (saturated) image 2.06e-04

B. saturated input (z > 1, so f = 1)
  max |gradient| = 0.0e+00, sum gradient x input = 0.0000
  sum IG = 1.0001 (f(x) - f(x') = 1.0000), share on object 0.95, far background 0.00

C. baselines (m = 1024)       f(x')   sum    f(x)-f(x')  object  far bg   TV
  gradient |df/dx|              -        -       -      0.85    0.00   1.78
  gradient x input              -    1.740       -      0.91    0.00   1.92
  IG, black                  0.000   0.759     0.758    0.95    0.00   1.99
  IG, blurred (sigma 5 px)   0.000   0.759     0.758    0.98    0.00   2.15
  IG, uniform noise, 1 draw (range over 5 draws):
   1.00 to 1.00  -0.24 to -0.24  -0.24 to -0.24  0.53 to 0.56  0.32 to 0.36  2.08 to 2.18
  IG, uniform noise, 16 avg  1.000  -0.241    -0.242    0.55    0.37   1.29

D. SmoothGrad of the logit z, range over noise seeds 0-4      |mean grad|            mean |grad|
                                            TV          object       TV          object
  check: sigma 0 equals the gradient, max difference 1.4e-17
  sigma 0.00, n  50:  1.78-1.78  0.85-0.85  1.78-1.78  0.85-0.85
  sigma 0.05, n  50:  1.76-1.77  0.85-0.85  1.69-1.71  0.85-0.85
  sigma 0.15, n  50:  1.61-1.65  0.70-0.71  0.68-0.71  0.55-0.57
  sigma 0.30, n  50:  1.60-1.63  0.48-0.48  0.31-0.32  0.20-0.20
  sigma 0.60, n  50:  1.60-1.63  0.42-0.43  0.27-0.28  0.19-0.19
  sigma 0.15, n   1:  1.48-1.55  0.55-0.60  1.48-1.55  0.55-0.60
  sigma 0.15, n   5:  1.51-1.56  0.59-0.64  0.97-1.03  0.53-0.59
  sigma 0.15, n  20:  1.58-1.63  0.66-0.69  0.76-0.79  0.55-0.59
  sigma 0.15, n 200:  1.63-1.65  0.72-0.73  0.63-0.64  0.55-0.56

Completeness (A). On the unsaturated image, the error falls about fourfold for each fourfold increase in steps, as expected of a Riemann sum whose integrand is piecewise constant in $\alpha$ (the network is piecewise linear and the path is a straight line). It is under 5% from $m = 64$ on both images and 7.9% at $m = 16$ on the unsaturated one, in line with Sundararajan et al.’s 20 to 300 steps. On the saturated image the error is not monotone (9.6e-05 at 1,024 steps, 2.1e-04 at 4,096): there the gradient drops to zero where the path crosses the cap, and it follows from the right Riemann sum that its error depends on where that crossing falls between steps. Printing $\sum_i \mathrm{IG}_i$ against $f(x) - f(x')$, as the authors recommend, is the cheapest check that $m$ is large enough.

Saturation (B). On the high-contrast image the score is capped, so the gradient is exactly zero at every pixel and gradient times input sums to zero: by either map, nothing in the image matters. Integrated gradients still distributes the full output, $f(x) - f(0) = 1$, with 95% of its mass on the disc and none on the far background. This is Sundararajan et al.’s one-ReLU counterexample in image form. In our reading, a trained network’s softmax saturates more gently than this cap, but confident predictions sit where the curve is flat, which is where the gradient understates the evidence.

Baselines (C). The black and blurred baselines both score 0, so their attributions add up to the image’s score; both put almost all of the mass on the disc and none far from it. The blurred baseline puts slightly more on the disc (0.98 against 0.95). In our reading, the reason is the factor $x - x'$: blurring removes the stripes but keeps the flat background, so background pixels near the disc differ little from the baseline. The uniform-noise baseline behaves differently. To an edge-energy detector, noise is texture, and every noise draw scores the full 1. The attributions therefore explain $f(x) - f(x') = -0.24$, the image being less striped than noise, and about a third of their mass falls on the far background, where the noise baseline differed from the flat background. A noise image is not an absence of signal for this network, although Sundararajan et al. suggest that a noise image can be. Averaging over 16 draws barely changes the shares but lowers TV from about 2.1 to 1.29, because attributions that differ from draw to draw partly cancel. Gradient times input sums to 1.740, more than twice the score: without a baseline it has no completeness, and the thresholds of this network mean that the first-order Taylor expansion around the zero image, which Shrikumar et al. identify as its implicit reference, is not exact. The integrated-gradient maps have higher TV than the plain gradient; in our reading, because the factor $x_i - x'_i$ follows the stripes.

SmoothGrad (D). A noise level of 0.05 changes little. The two map definitions part at higher noise:

• Mean of magnitudes. It becomes much smoother (TV from 1.78 to 0.27) but spreads off the disc (share from 0.85 to 0.19). It follows from the construction that noise of standard deviation $\sigma$ gives Sobel responses with standard deviation $\sqrt{12}\,\sigma$, about 0.5 at $\sigma = 0.15$ and 1.0 at $\sigma = 0.3$. That is at or above the first threshold, so in the noisy copies the background itself fires. A map of magnitudes accumulates that background response.
• Magnitude of the mean. Signed gradients from background units that fire on noise partly cancel, so the share on the disc stays higher (0.42 at $\sigma = 0.6$). TV hardly falls (1.78 to about 1.60); in our reading, because the mean converges to the gradient of the smoothed score, which still alternates with the stripes.

At $\sigma = 0.15$, more samples make the magnitude-of-mean map slightly less smooth (TV from 1.48–1.55 at $n = 1$ to 1.63–1.65 at $n = 200$) and more concentrated on the disc (0.55–0.60 to 0.72–0.73). A single noisy gradient is a different, noisier map than the smoothed gradient that the average approaches. The spread over seeds is small from $n = 50$. The example shows that $\sigma$ is not a free denoising knob: it decides at what scale, and in what regime of the network, the gradient is measured, and the two definitions give different answers.

Two parts of the process are disclosed. In the pilot, SmoothGrad differentiated the capped score $f$, and several rows of part D printed undefined measures: a noisy copy whose logit exceeds the cap has zero gradient, and a map that is entirely zero gives 0/0 in the share and in TV. On the editor’s decision, part D was changed to differentiate the uncapped logit $z$, the class score that saliency methods differentiate. The change was piloted once before the first run, and parts A to C were not changed. Second, the planned check that SmoothGrad at $\sigma = 0$ equals the plain gradient exactly failed: the printed difference is 1.4e-17, not 0, because averaging 50 identical double-precision gradients rounds. Expecting exact equality was a planning error about rounding. The check is kept as printed.

The example covers only properties that hold by construction, on a network whose relevant region is known. It does not show that any of these maps is faithful on a trained network, which is the question of the next section.

Variants

• DeepLIFT (Shrikumar et al., 2017) compares each neuron’s activation with its activation on a reference input and propagates contribution scores that satisfy a “summation-to-delta” property, in one backward pass. Because it uses differences rather than derivatives, it propagates a signal where the gradient is zero. Separate treatment of positive and negative contributions (its RevealCancel rule) is meant to expose dependencies that other methods miss. Sundararajan et al. argue that it breaks implementation invariance.
• Layer-wise relevance propagation (Bach et al., 2015), as Chefer et al. describe it, propagates relevance from the predicted class back to the input, with the total relevance kept constant across layers. Chefer et al. adapt such rules to transformers, whose skip connections and non-ReLU activations make naive propagation unstable.
• Expected gradients and baseline distributions (Sturmfels et al., 2020) average integrated gradients over a distribution of baselines (uniform, Gaussian, blurred or drawn from the data) instead of committing to one image.
• Other path methods integrate along non-straight paths; they keep completeness and implementation invariance but lose symmetry preservation (Sundararajan et al., 2017).
• SmoothGrad on other maps. The averaging can wrap any saliency method. Sturmfels et al. analyse the gradient-times-input version. The interpretability hub reports that, in Hooker et al.’s ROAR benchmark, only the ensembling variants SmoothGrad-Squared and VarGrad beat a random ranking.
• Guided Grad-CAM multiplies a Grad-CAM map with a guided-backpropagation map, inheriting guided backpropagation’s problems (Shrikumar et al., 2017).
• Attention rollout, attention flow and gradient-weighted relevance for transformers, as above.

Best Practices

• State the score and the class. Differentiate the logit of a named class rather than a probability, whose softmax can saturate, and say which one was used.
• Check completeness. For integrated gradients, print $\sum_i \mathrm{IG}_i$ against $f(x) - f(x')$ and raise $m$ until they agree (Sundararajan et al. found 20–300 steps enough in practice).
• Check the baseline’s score. A baseline should score near zero for the explained class; report $f(x')$. Compare more than one baseline (black, blurred, noise or a distribution), since constant baselines are blind to their own colour (Sturmfels et al., 2020).
• Treat SmoothGrad’s noise level as a parameter of the explanation. Report $\sigma$, $n$ and whether magnitudes are taken before or after averaging.
• Keep signs when they carry information. Absolute values hide evidence against a class. Integrated gradients and gradient times input are signed; deconvnet and guided backpropagation can fail to show negative contributions.
• Run a randomization test before trusting a method on a new model. Recompute the maps after re-initializing the weights, the test Adebayo et al. propose, as the interpretability hub describes. The interpretability hub shows such a test in code.
• Test the hypothesis a map suggests. Remove or move the highlighted feature and measure the prediction, as DeGrave et al. did with laterality markers.

Pitfalls

• Reading sensitivity as contribution. A zero gradient can sit on decisive evidence (part B above), and a large gradient can sit on a pixel that contributed nothing at this input.
• Trusting sharp maps. Guided backpropagation and the deconvnet produce clean, image-like maps that change little when the model is randomized (Adebayo et al., 2018, through the hub) and that, by Nie et al.’s analysis, largely recover the image.
• Choosing a baseline without saying so. The same image gives different maps from black, blurred and noise baselines. A baseline that scores high, like the noise image in the example, turns the map into an explanation of a difference the reader did not ask about.
• Too few integration steps. A coarse Riemann sum can miss completeness by tens of percent (27% at four steps in the example).
• Class-agnostic maps. Chefer et al. report that some attribution-propagation methods, the LRP variants among them, give nearly the same map whatever class is explained, and that rollout is fixed per image by construction. Compute maps for two classes before reading a map as class evidence.
• Visual judgement. A map that outlines the object looks right, but Sturmfels et al.’s decoy-MNIST example shows an edge detector that looks more plausible than a faithful map of a network that reads a class code from a corner patch. Plausibility is not faithfulness (see the hub).
• Confusing the two saliencies. A classifier’s saliency map is not a prediction of human attention, and salient-object or fixation benchmarks do not evaluate it.

Evidence and Impact

Sanity checks. As the interpretability hub summarizes, Adebayo et al. (2018) re-initialized a trained network’s weights, layer by layer or all at once, and trained models on randomly permuted labels, then recomputed the maps. Plain gradients and Grad-CAM changed as they should, while guided backpropagation and Guided Grad-CAM stayed nearly unchanged however many layers were randomized. Maps from gradient times input and integrated gradients still showed the bird’s outline visually after several blocks were randomized, although their rank correlation with the original maps, without absolute values, fell to about zero. The hub and the Zeiler and Fergus paper article each reproduce part of this in code. A method that passes is not shown to be faithful; one that fails is shown not to depend on what the model learned.

Theory of backpropagation-based maps. Nie et al. (2018) explain guided backpropagation’s and the deconvnet’s insensitivity: in their analysis the backward ReLU, together with the local connections of convolutional networks, makes the maps a partial recovery of the input image, unrelated to the decision. The Zeiler and Fergus article reports their experiments in more detail.

Removal benchmarks. The hub summarizes Hooker et al.’s ROAR, which retrains the model after removing the top-ranked pixels. On their image classification datasets, many methods did no better than a random ranking, and only SmoothGrad-Squared and VarGrad did better. Sundararajan et al. had already argued that removal tests cannot separate artifacts of the perturbed data, of the model and of the attribution method, which is why they turned to axioms.

Axioms. Integrated gradients’ axioms (completeness, sensitivity, implementation invariance, symmetry) are guarantees about the accounting, not about faithfulness to a human notion of evidence. They fix how a difference $f(x) - f(x')$ is shared out, given a baseline. Sturmfels et al. show that the choice of baseline then changes the map substantially and that the available ablation tests do not agree on which baseline is best.

Transformers. On a six-layer transformer trained for subject–verb number agreement (accuracy 0.96), Abnar and Zuidema measured the Spearman correlation of each method’s token weights, in the last layer, with blank-out ablation scores over 2,000 test sentences: 0.29 ± 0.39 for raw attention, 0.71 ± 0.24 for rollout and 0.70 ± 0.22 for flow (their Table 1). Against input gradients the correlations were 0.46, 0.54 and 0.61 (Table 2). On a DistilBERT model fine-tuned for SST-2 sentiment, all three correlated weakly with input gradients (0.09 to 0.14 in the last layer, Table 3). Chefer et al. evaluated on ViT-Base. On the ImageNet-segmentation dataset, treating relevance maps as segmentations, their method reached 61.95% mIoU against 55.42% for rollout, 46.37% for raw attention, 40.82% for Grad-CAM and 32.89% for LRP. In perturbation tests on the ImageNet validation set, pixels were masked in order of relevance and the area under the top-1 accuracy curve was recorded. For the predicted class, their method scored 17.03% when the most relevant pixels were masked first (lower is better), against 20.05% for rollout, and 54.16% when the least relevant were masked first (higher is better), against 53.1%. They did not evaluate attention flow, for computational reasons. All of these test agreement with other signals, not ground truth, and the NLP results transfer to vision only as cautions.

Use. Gradient saliency is part of the standard toolkit for auditing image classifiers: Sundararajan et al. used integrated gradients on a diabetic retinopathy model, and the hub describes chest-radiograph shortcut studies, as well as Saporta et al.’s finding that seven saliency methods localized pathologies significantly worse than a human benchmark (see chest X-ray analysis). From 2018, as the interpretability hub’s history puts it, the emphasis moved from producing maps to testing them. A saliency map can point to a feature worth testing, and an intervention on that feature confirms whether the model uses it.