Attention Mechanism

An attention mechanism lets a neural network decide, for each output it computes, which parts of its input to draw on. It compares a query with a set of keys, turns the comparison scores into weights that sum to one, and returns the weighted average of the values that go with the keys. The weights are computed from the data, so they change from input to input: a captioning model can look at the dog while it writes “dog” and at the frisbee while it writes “frisbee”. The mechanism is also called soft attention, content-based attention or, in its first machine-translation form, an alignment model.

Attention entered deep learning in sequence-to-sequence models for translation and image captioning around 2014–2015, spread to vision as spatial and channel attention and non-local blocks, and became the core of the transformer. This article covers the general mechanism: queries, keys and values, the scoring functions, soft and hard attention, its history and its cost. Self-attention, where queries, keys and values come from the same set of tokens, has its own article with multi-head attention and masks, as does the transformer architecture; attention from one set of tokens to another, as in a transformer decoder or a text-conditioned image generator, is called cross-attention.

Definition

An attention function maps a query $\mathbf{q}$ and a set of $m$ key–value pairs $(\mathbf{k}_j, \mathbf{v}_j)$ to an output, the weighted sum of the values, where each weight comes from a compatibility function of the query with the corresponding key (Vaswani et al., 2017, §3.2). In three steps:

1. Score each key: $e_j = \operatorname{score}(\mathbf{q}, \mathbf{k}_j)$.
2. Normalize the scores with a softmax: $\alpha_j = \exp(e_j) / \sum_{l} \exp(e_l)$.
3. Read out the weighted average $\mathbf{c} = \sum_j \alpha_j \mathbf{v}_j$, often called the context vector.

The weights $\alpha_j$ form the attention map or alignment for that query. Soft attention uses the weighted average as it stands; hard attention selects one value, by sampling an index from the weights or by taking the largest. In most models keys and values are two learned projections of the same features, and queries come from the part of the network that needs information: the decoder state in translation, a target description in search, every token in self-attention.

Intuition

Think of a scene cut into a grid of patches and a question, “where is the disk?”. Each patch advertises what it contains through its key; the query describes what is wanted; the score measures the match. The softmax turns the scores into a budget of attention that sums to one, so patches compete for it: a clearly matching patch takes almost all of it, two equally good matches share it. The output is then a blend of what the patches carry in their values, dominated by the winners.

Three things set attention apart from a fixed layer. The mixing weights are computed from the input, whereas a convolution applies the same weights to every image. The number of keys can vary, so one set of parameters handles a 10-word sentence and a 50-word one, or 196 feature-map positions. And nothing in the weighted sum depends on the order of the keys, which is why models that need position must add it to the keys and values (see positional encoding).

Formal Definition

Let $\mathbf{q} \in \mathbb{R}^{d_q}$ be a query and $K = [\mathbf{k}_1, \dots, \mathbf{k}_m]$, $V = [\mathbf{v}_1, \dots, \mathbf{v}_m]$ the keys and values. Then

$$\alpha_j = \frac{\exp\big(\operatorname{score}(\mathbf{q}, \mathbf{k}_j)\big)}{\sum_{l=1}^{m} \exp\big(\operatorname{score}(\mathbf{q}, \mathbf{k}_l)\big)}, \qquad \operatorname{Attention}(\mathbf{q}, K, V) = \sum_{j=1}^{m} \alpha_j \mathbf{v}_j .$$

With $n$ queries, the $m$ keys and the $m$ values stacked as the rows of $Q$, $K$ and $V$, and a score that is a dot product, the whole computation is the matrix form $\operatorname{softmax}(QK^\top)V$ with the softmax applied row by row.

The first neural translation form. In Bahdanau, Cho and Bengio (2015), an encoder reads the source sentence into annotations $\mathbf{h}_1, \dots, \mathbf{h}_{T_x}$, and before emitting target word $i$ the decoder, with hidden state $\mathbf{s}_{i-1}$, computes

$$e_{ij} = a(\mathbf{s}_{i-1}, \mathbf{h}_j), \qquad \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}, \qquad \mathbf{c}_i = \sum_{j=1}^{T_x} \alpha_{ij} \mathbf{h}_j$$

(their Eqs. 5–6). In the terms above, the decoder state is the query and the annotations are both keys and values. The authors call $a$ an alignment model, read $\alpha_{ij}$ as the probability that target word $i$ is aligned to source word $j$, so that $\mathbf{c}_i$ is an expected annotation, and describe the result as implementing a mechanism of attention in the decoder (§3.1). Because the alignment is soft rather than a latent variable, the gradient flows through it and the alignment model is trained jointly with the translator.

Scoring functions. Four forms recur. The table writes them with this article’s query $\mathbf{q}$ and key $\mathbf{k}$, and $\mathbf{w}$ for the additive score’s output vector (the letter $\mathbf{v}$ is taken by the values); the last column gives each as its source prints it:

Name	$\operatorname{score}(\mathbf{q}, \mathbf{k})$	As printed	Source
Additive	$\mathbf{w}^\top \tanh(W_q \mathbf{q} + W_k \mathbf{k})$	$\mathbf{v}_a^\top \tanh(W_a \mathbf{s}_{i-1} + U_a \mathbf{h}_j)$	Bahdanau et al. (2015), Appendix A.1.2
Dot	$\mathbf{q}^\top \mathbf{k}$	$\mathbf{h}_t^\top \bar{\mathbf{h}}_s$	Luong et al. (2015), Eq. 8
General (bilinear, multiplicative)	$\mathbf{q}^\top W \mathbf{k}$	$\mathbf{h}_t^\top W_a \bar{\mathbf{h}}_s$	Luong et al. (2015), Eq. 8
Scaled dot product	$\mathbf{q}^\top \mathbf{k} / \sqrt{d_k}$	$\operatorname{softmax}(QK^T / \sqrt{d_k})V$	Vaswani et al. (2017), Eq. 1

The additive score is a multilayer perceptron with one hidden layer of width $d_a$ applied to the pair. Bahdanau et al. point out that $W_k \mathbf{k}_j$ does not depend on the query and can be computed once per key. The multiplicative forms need only matrix products. Luong et al. also tried a location-based score that ignores the keys and predicts the weights from the query alone, $\boldsymbol{\alpha} = \operatorname{softmax}(W \mathbf{q})$, and found that it did not learn good alignments (their Eq. 9 and §5.3). Luong et al.’s third content-based score, “concat”, is printed differently in the two versions of their paper: the arXiv version (v5) gives $\mathbf{v}_a^\top \tanh(W_a[\mathbf{h}_t; \bar{\mathbf{h}}_s])$, which is the additive form, while the EMNLP proceedings print $W_a[\mathbf{h}_t; \bar{\mathbf{h}}_s]$ (Eq. 8), without the $\tanh$ and the vector $\mathbf{v}_a$.

Which score works better has depended on the setting. In Luong et al.’s English–German experiments on newstest2014, dot worked best for global attention and general for their local attention (their Table 4); concat is not in the table, and the text reports that their implementation of it did not perform well (§5.3). Britz et al. (2017), sweeping architectures on WMT’15 English→German, compared additive attention, $\langle \mathbf{v}, \tanh(W_1 \mathbf{h}_j + W_2 \mathbf{s}_i) \rangle$, with an unscaled multiplicative score, $\langle W_1 \mathbf{h}_j, W_2 \mathbf{s}_i \rangle$ (their Eqs. 6–7). Their text reports additive attention slightly but consistently better, with the attention dimensionality having little effect. Their Table 5 (mean BLEU on newstest2013) is less even: at dimension 256 both averaged 22.33 ($p = 0.5$ in the text), and at 1024 the multiplicative model fell to 18.22 against 22.10 for additive.

Why scale by $\sqrt{d_k}$. Vaswani et al. (2017) cite that result: additive attention outperforms unscaled dot-product attention for larger $d_k$. They suspected that large dot products push the softmax into regions with extremely small gradients, and divided by $\sqrt{d_k}$; their footnote notes that the dot product of two vectors with independent zero-mean, unit-variance components has variance $d_k$. The self-attention article works through the variance argument and the softmax Jacobian, and shows on image patches how unscaled weights collapse onto one key as $d_k$ grows. Additive attention avoids the problem without a constant, in our reading, because its $\tanh$ bounds every hidden unit and the learned $\mathbf{w}$ sets the scale of the scores. Vaswani et al. note that, at similar theoretical complexity, dot-product attention is much faster and more space-efficient in practice, because all scores are one matrix product.

Soft and hard attention. Xu et al. (2015) define both for image captioning. Their decoder attends over $L = 196$ feature vectors $\mathbf{a}_i$ of a $14 \times 14$ convolutional feature map. Hard attention treats the attended location as a latent variable $\mathbf{s}_t$, one-hot and sampled from the attention weights, and feeds the selected feature $\hat{\mathbf{z}}_t = \sum_i s_{t,i} \mathbf{a}_i$ to the decoder. Since sampling has no gradient, they maximize a variational lower bound on $\log p(\mathbf{y} \mid \mathbf{a})$ with a Monte Carlo gradient estimate that they note is equivalent to the REINFORCE rule, with a moving-average baseline and an entropy term to reduce variance. Soft attention feeds the expectation instead,

$$\mathbb{E}_{p(\mathbf{s}_t \mid \mathbf{a})}[\hat{\mathbf{z}}_t] = \sum_{i=1}^{L} \alpha_{t,i} \mathbf{a}_i ,$$

which is the Bahdanau weighted sum and makes the whole model differentiable. Hard attention reads one location per step; soft attention computes a score for every location. Luong et al.’s local attention sits between them: in its predictive form (local-p) the model predicts a source position, scores only a window around it, and multiplies the weights by a Gaussian centred there, which keeps it differentiable while touching fewer positions (their §3.2). The image captioning article covers the captioning model itself.

Properties

• A convex combination. The weights are positive and sum to one, so the output lies in the convex hull of the values. Soft attention can blend but not extrapolate; when two keys match equally, it returns their average, which need not resemble either (the example below shows this for locations).
• Order-free over keys. Permuting the key–value pairs together leaves the output unchanged. Attention over image patches sees a set, not a grid, unless position is put into the keys or values.
• Dot products depend on norms. The dot score grows with the length of the key as well as with its direction, so a bright, high-contrast patch can outscore a better-matching but faint one. Learned projections can compensate; an unprojected template comparison, as in the example, cannot.
• Cost. $n$ queries against $m$ keys need $nm$ scores, each costing $O(d)$ for a dot product, and an $n \times m$ weight matrix. Bahdanau et al. noted that their model evaluates the alignment for every source word at every target word, acceptable for sentences of 15–40 words but possibly limiting elsewhere (§6.1). In self-attention $n = m$, so cost and memory grow quadratically with the number of tokens; for images that number is the number of patches or feature-map positions, which is why pixel-level global attention is avoided (see self-attention and Vision Transformer).
• Efficient variants. Windowed attention restricts each query to a neighbourhood, as in Swin Transformer (see self-attention). Sparse Transformers factorize the attention pattern to $O(n\sqrt{n})$ (Child et al., 2019). Linear attention replaces the softmax with a kernel feature map so that matrix products can be reassociated, reducing the cost to $O(n)$ in sequence length (Katharopoulos et al., 2020). FlashAttention computes exact attention with tiling that reduces reads and writes between GPU memory levels (Dao et al., 2022). The first three change what is computed; the last changes only how.

History

The fixed-length bottleneck. Neural translation models of 2014 encoded the source sentence into one vector from which a decoder generated the translation. Bahdanau, Cho and Bengio conjectured that this vector was a bottleneck, pointing to evidence that such models degrade quickly as sentences get longer, and let the decoder search the source annotations afresh for each target word. Their model, RNNsearch, kept its BLEU score on long sentences where the fixed-vector model dropped sharply (their Fig. 2). On the English–French WMT’14 test set (news-test-2014), RNNsearch-50 scored 26.75 BLEU against 17.82 for the encoder–decoder baseline, and 34.16 against 26.71 on sentences without unknown words, where a longer-trained RNNsearch-50⋆ reached 36.15 against 35.63 for the phrase-based Moses system (their Table 1). The paper was posted to arXiv in September 2014 and presented as an oral at ICLR 2015; in 2025 ICLR named it the runner-up for its Test of Time Award.

Luong, Pham and Manning (2015) simplified the computation path, compared the scoring functions above, and introduced global and local attention. Their ensemble reached 25.9 BLEU on WMT’15 English to German, which they reported as a new state of the art for that task.

Who introduced attention? It depends on what counts. Bahdanau et al. themselves name Graves (2013) as a similar approach: for handwriting synthesis, Graves’s network read the text through a soft window, a mixture of Gaussians over character positions whose location could only move forward, which suits handwriting but not the reordering translation needs. Mnih et al. (2014, on arXiv in June, before Bahdanau et al.’s September preprint) built a recurrent model that looks at an image through a sequence of glimpses, a hard attention trained with reinforcement learning. Xu et al. (2015) place their work in a long line of attention in neural networks, naming Larochelle and Hinton (2010) as the earliest of the works in the same spirit. Luong et al. (2015) describe attention as a concept already gaining popularity across modalities, which Bahdanau et al. applied to translation; ICLR’s award announcement credits the paper with introducing a form of attention. Schlag, Irie and Schmidhuber (2021) show that linearized self-attention is formally equivalent to the fast weight programmers of the early 1990s, in which a network writes key–value outer products into the weights of another. What Bahdanau et al. introduced, on this reading, is soft, content-based attention over every input position, trained end to end, the form later models inherited.

Attention in vision: where to look. Mnih et al.’s recurrent attention model extracts a retina-like glimpse, sharp at the centre and coarse around it, at a location chosen by a learned policy, so the computation is set by the number of glimpses rather than the image size. On their 60×60 cluttered translated MNIST task, four glimpses gave 4.96% test error against 8.09% for a convolutional network with a similar number of parameters (their Table 2a). Xu et al. (2015) then carried Bahdanau-style soft attention, and a hard variant, to caption generation over convolutional features, and showed the per-word attention maps as a view of where the model looked. On COCO, their hard-attention model scored 25.0 BLEU-4 and soft attention 24.3, while on METEOR soft attention led, 23.90 to 23.04 (their Table 1).

Spatial transformers: a different mechanism. Jaderberg et al. (2015) let a network predict the parameters of a spatial transformation, such as a crop-and-scale $\begin{bmatrix} s & 0 & t_x \\ 0 & s & t_y \end{bmatrix}$, and resample the feature map with a differentiable bilinear kernel. They present it as usable for tasks that need attention, trainable by backpropagation alone, unlike reinforcement-learned glimpses, and as a generalization of differentiable attention to any spatial transformation. Four transformers constrained to that attention form, feeding part classifiers, reached 84.1% accuracy on the CUB-200-2011 bird dataset (their Table 3). The relation is close but not identity, in our reading: each output pixel of a spatial transformer is a weighted sum of input pixels (their Eq. 3), but the weights come from a sampling kernel evaluated at predicted coordinates, nonzero only for the nearest input pixels, not from comparing a query with keys. It decides where to look by geometry; query–key attention decides by content.

Channel and spatial gating. Squeeze-and-excitation blocks (Hu, Shen and Sun, 2018) rescale each channel of a feature map by a sigmoid gate computed from globally pooled features, and are often called channel attention. Successors such as CBAM, described in that article, add a spatial map of gates. These gates are independent sigmoids rather than weights that sum to one, and nothing is averaged, so they share the idea of input-dependent emphasis rather than the query–key form; the squeeze-and-excitation article compares them.

Non-local blocks and the transformer. Wang et al. (2018) defined a non-local operation that computes the response at each position as a normalized weighted sum over all positions, $\mathbf{y}_i = \frac{1}{\mathcal{C}(\mathbf{x})} \sum_j f(\mathbf{x}_i, \mathbf{x}_j)\, g(\mathbf{x}_j)$, inspired by non-local means, and noted that the then-new transformer’s self-attention is a special case, their embedded Gaussian version. Vaswani et al. (2017) had built the transformer from scaled dot-product self-attention alone, without recurrence or convolution. The Vision Transformer (Dosovitskiy et al., 2021) applied that architecture to sequences of image patches, and attention is now a standard component of vision backbones, detectors and generators.

Examples

Attention over the patches of a rendered scene. The code draws a $48 \times 48$ image with a square, a disk, a triangle and a cross, each of random size, offset and brightness in a random cell of a $4 \times 4$ grid, plus noise. The 16 cells are the keys and values (144 pixels each), and the query is a clean template of one shape. It first scores the patches by a raw dot product with the template, then trains the parameters of an additive and a scaled dot-product score (attention width $d_a = 32$) so that the weights fall on the cell holding the queried shape; nothing else in the model is learned. Training runs 600 Adam steps on fresh random scenes, for three seeds each, and testing uses 1,000 held-out scenes (NumPy 2.3.5, about 20 seconds for all examples).

import numpy as np

G, C = 4, 12                     # a 4x4 grid of 12x12-pixel cells: 48x48 image, 16 patches
SHAPES = ["square", "disk", "triangle", "cross"]

def draw(shape, size, cy, cx):
    """Binary mask of one shape of the given size, centred at (cy, cx) in a CxC cell."""
    y, x = np.mgrid[0:C, 0:C] + 0.5
    dy, dx, r = y - cy, x - cx, size / 2
    if shape == "square":
        m = (abs(dy) <= r) & (abs(dx) <= r)
    elif shape == "disk":
        m = dy**2 + dx**2 <= r**2
    elif shape == "triangle":
        m = (abs(dy) <= r) & (abs(dx) <= (dy + r) / 2)
    else:                        # cross
        m = ((abs(dy) <= r) & (abs(dx) <= r / 3)) | ((abs(dx) <= r) & (abs(dy) <= r / 3))
    return m.astype(float)

def scene(rng, shapes=SHAPES):
    """Each shape in its own random cell, with random size, offset and brightness, plus noise."""
    img = np.zeros((G * C, G * C))
    cells = rng.choice(G * G, size=len(shapes), replace=False)
    for s, cell in zip(shapes, cells):
        r, c = divmod(int(cell), G)
        cy, cx = C / 2 + rng.uniform(-1, 1, 2)
        img[r*C:(r+1)*C, c*C:(c+1)*C] = rng.uniform(0.6, 1.0) * draw(s, rng.uniform(6, 9), cy, cx)
    return img + 0.1 * rng.standard_normal(img.shape), cells

def patches(img):                # 16 tokens of 144 pixels, in row-major cell order
    return img.reshape(G, C, G, C).transpose(0, 2, 1, 3).reshape(G * G, C * C)

TEMPLATES = np.stack([draw(s, 8, C / 2, C / 2).ravel() for s in SHAPES])

def batch(rng, n):
    """n scenes; the query is a clean template of one shape, the label is its cell."""
    K, Q, T = [], [], []
    for _ in range(n):
        img, cells = scene(rng)
        t = rng.integers(len(SHAPES))
        K.append(patches(img)); Q.append(TEMPLATES[t]); T.append(cells[t])
    return np.array(Q), np.array(K), np.array(T)

def softmax(e):
    e = np.exp(e - e.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

D, DA = C * C, 32                # patch dimension, attention dimension

def scores(kind, P, Q, K):
    """Additive: w^T tanh(Wq q + Wk k).  Scaled dot product: (Wq q).(Wk k) / sqrt(DA)."""
    a, b = Q @ P["Wq"].T, K @ P["Wk"].T            # (n, DA) and (n, 16, DA)
    if kind == "additive":
        h = np.tanh(a[:, None, :] + b)
        return h @ P["w"], (a, b, h)
    return np.einsum("nd,nkd->nk", a, b) / np.sqrt(DA), (a, b, None)

def gradients(kind, P, Q, K, T):
    """Cross-entropy of the attention weights against the target cell, and its gradient."""
    e, (a, b, h) = scores(kind, P, Q, K)
    n = len(T)
    de = softmax(e)
    de[np.arange(n), T] -= 1
    de /= n
    if kind == "additive":
        du = de[:, :, None] * P["w"] * (1 - h**2)
        return {"w": np.einsum("nk,nkd->d", de, h),
                "Wq": du.sum(1).T @ Q, "Wk": np.einsum("nkd,nkf->df", du, K)}
    da = np.einsum("nk,nkd->nd", de, b) / np.sqrt(DA)
    db = de[:, :, None] * a[:, None, :] / np.sqrt(DA)
    return {"Wq": da.T @ Q, "Wk": np.einsum("nkd,nkf->df", db, K)}

def train(kind, seed, steps=600, bs=64, lr=3e-3):
    """Adam on fresh random scenes; only the scoring function is learned."""
    rng = np.random.default_rng(seed)
    P = {"Wq": rng.standard_normal((DA, D)) / np.sqrt(D),
         "Wk": rng.standard_normal((DA, D)) / np.sqrt(D)}
    if kind == "additive":
        P["w"] = rng.standard_normal(DA) / np.sqrt(DA)
    m = {k: np.zeros_like(w) for k, w in P.items()}
    s = {k: np.zeros_like(w) for k, w in P.items()}
    for t in range(1, steps + 1):
        g = gradients(kind, P, *batch(rng, bs))
        for k in P:
            m[k] = 0.9 * m[k] + 0.1 * g[k]
            s[k] = 0.999 * s[k] + 0.001 * g[k] ** 2
            P[k] -= lr * (m[k] / (1 - 0.9**t)) / (np.sqrt(s[k] / (1 - 0.999**t)) + 1e-8)
    return P

Q_test, K_test, T_test = batch(np.random.default_rng(100), 1000)
idx = np.arange(1000)

def report(name, w):
    print(f"{name:28s} argmax on target {(w.argmax(1) == T_test).mean():6.1%}"
          f"   mean weight on target {w[idx, T_test].mean():.3f}")

report("raw template dot product", softmax(np.einsum("nd,nkd->nk", Q_test, K_test)))
models = {}
for kind in ["additive", "dot"]:
    for seed in range(3):
        P = train(kind, seed)
        models.setdefault(kind, P)                   # keep seed 0 for later
        report(f"{kind}, seed {seed}", softmax(scores(kind, P, Q_test, K_test)[0]))
print("parameters: additive", sum(w.size for w in models["additive"].values()),
      " dot", sum(w.size for w in models["dot"].values()))

raw template dot product     argmax on target  36.0%   mean weight on target 0.367
additive, seed 0             argmax on target  99.5%   mean weight on target 0.976
additive, seed 1             argmax on target  99.3%   mean weight on target 0.975
additive, seed 2             argmax on target  99.3%   mean weight on target 0.974
dot, seed 0                  argmax on target  93.7%   mean weight on target 0.914
dot, seed 1                  argmax on target  93.3%   mean weight on target 0.909
dot, seed 2                  argmax on target  94.0%   mean weight on target 0.910
parameters: additive 9248  dot 9216

A raw template dot product finds the queried shape in only about a third of the scenes: as the Properties section notes, the score rewards overlap with bright pixels, and every shape overlaps the template a good deal. Both learned scores fix this; over the three seeds additive attention puts 97.4–97.6% of the weight on the right cell on average and dot-product attention 90.9–91.4%, with the same number of parameters to within the 32 entries of $\mathbf{w}$. That additive attention wins here is in line with Britz et al.’s finding, although their multiplicative score was unscaled; this is one small task with a fixed training budget, not a general ranking.

Continuing, one test scene shows where each score puts the weight of the query “disk”:

# One test scene: where does the query "disk" put its weight?
img, cells = scene(np.random.default_rng(7))
Kx, qx = patches(img)[None], TEMPLATES[[1]]
print("shape cells (square, disk, triangle, cross):", [divmod(int(c), G) for c in cells])
for kind in ["additive", "dot"]:
    w = softmax(scores(kind, models[kind], qx, Kx)[0])[0]
    print(f"{kind} weights for the query 'disk':")
    print(np.array2string(w.reshape(G, G), precision=3, suppress_small=True))
w_raw = softmax(Kx[0] @ qx[0])
print("raw template dot product, largest weights:",
      {divmod(int(j), G): round(float(w_raw[j]), 3) for j in np.argsort(w_raw)[::-1][:3]})

shape cells (square, disk, triangle, cross): [(3, 0), (2, 0), (3, 2), (2, 2)]
additive weights for the query 'disk':
[[0.    0.    0.    0.   ]
 [0.    0.    0.    0.   ]
 [0.989 0.    0.005 0.   ]
 [0.002 0.    0.003 0.   ]]
dot weights for the query 'disk':
[[0.002 0.001 0.001 0.001]
 [0.001 0.    0.004 0.003]
 [0.501 0.007 0.466 0.001]
 [0.001 0.005 0.004 0.001]]
raw template dot product, largest weights: {(2, 2): 0.899, (3, 0): 0.071, (2, 0): 0.03}

The additive score concentrates on the disk’s cell, (2, 0). The dot-product score splits its weight almost evenly between the disk and the cross at (2, 2), and the raw template comparison puts 90% on the cross: a cross of the same size overlaps a disk template in most of its pixels.

Soft and hard attention. Continuing with the trained additive model, the code builds scenes with two disks and lets the values be the pixel coordinates of each cell’s centre, so the attention output is a location.

# Scenes with two disks: the query "disk" now has two good matches.
def two_disk_weights(seed):
    img, cells = scene(np.random.default_rng(seed), shapes=["disk", "square", "disk", "cross"])
    w = softmax(scores("additive", models["additive"], TEMPLATES[[1]], patches(img)[None])[0])[0]
    return w, cells

shares = [two_disk_weights(sd)[0][two_disk_weights(sd)[1][[0, 2]]].sum() for sd in range(30)]
print(f"30 two-disk scenes: weight on the two disks {min(shares):.3f} to {max(shares):.3f}")

w, cells2 = two_disk_weights(10)
print("scene 10: disk cells", divmod(int(cells2[0]), G), divmod(int(cells2[2]), G),
      " weights", round(w[cells2[0]], 3), round(w[cells2[2]], 3))

# Values: the centre of each cell in pixels (row, col), so attention returns a location.
centres = np.stack(np.divmod(np.arange(G * G), G), axis=1) * C + C / 2
soft = w @ centres                                   # expectation under the weights
hard = centres[w.argmax()]                           # argmax selection
samples = np.random.default_rng(0).choice(G * G, size=10_000, p=w)
print("soft location:", soft.round(1), "  argmax location:", hard)
print("sampled hard attention, fraction per disk:",
      [round(float(np.mean(samples == c)), 3) for c in cells2[[0, 2]]],
      " mean sampled location:", centres[samples].mean(0).round(1))

# Derivative of the output's column with respect to the score of the second disk.
e = np.log(w)
de = np.zeros(G * G)
de[cells2[2]] = 1e-4
for name, f in [("soft", lambda e: softmax(e) @ centres[:, 1]),
                ("argmax", lambda e: centres[e.argmax(), 1])]:
    print(f"{name:6s} d(column)/d(score) = {(f(e + de) - f(e - de)) / 2e-4:.2f} pixels")

30 two-disk scenes: weight on the two disks 0.722 to 1.000
scene 10: disk cells (2, 2) (0, 3)  weights 0.524 0.475
soft location: [18.6 35.7]   argmax location: [30. 30.]
sampled hard attention, fraction per disk: [0.523, 0.476]  mean sampled location: [18.6 35.7]
soft   d(column)/d(score) = 3.00 pixels
argmax d(column)/d(score) = 0.00 pixels

Across the 30 scenes the query puts 72–100% of its weight on the two disks, split between them differently from scene to scene. In scene 10 the split is close to even, and the soft output, the weighted average of the two centres, lands at (18.6, 35.7), on empty background between the disks. The argmax picks one disk. Sampling locations from the weights, as Xu et al.’s hard attention does, picks each disk in proportion to its weight, and the average of the samples matches the soft output, the expectation in the equation above. The last two lines show why hard attention needs a different training method: nudging a score moves the soft output by about 3 pixels per unit, but leaves the argmax unchanged, so its derivative is zero; Xu et al. and Mnih et al. use sampling with REINFORCE-style gradients instead.

Common Misconceptions

• “Attention means transformers.” Attention was used with recurrent decoders for translation and captioning, and with glimpse-based vision models, years before the transformer, which removed the recurrence and kept the attention.
• “The weights measure similarity.” A dot-product score also grows with the norm of the key, and learned projections decide which features count. The weights say which values were mixed for this output under the learned scoring, not how alike two inputs are.
• “The $\sqrt{d_k}$ factor is part of attention.” It belongs to scaled dot-product attention. Additive attention needs no such constant, and Luong et al.’s dot score had none.
• “Hard attention is soft attention followed by an argmax.” Xu et al.’s hard attention samples a location and is trained with a variational bound equivalent to REINFORCE; an argmax would pass no gradient at all.
• “An attention map explains the prediction.” Bahdanau et al. and Xu et al. presented attention maps as qualitative alignments, and they are a cheap first look. Whether they explain predictions is disputed: on NLP tasks, Jain and Wallace (2019) found attention weights often uncorrelated with gradient-based importance, and Wiegreffe and Pinter (2019) argued that the answer depends on the definition of explanation. Interpretability of Neural Networks covers the debate and compares attention maps with gradient and perturbation methods.

Where It Is Used

• Captioning and vision-language models. A text decoder attends over image features at every word, from Show, Attend and Tell to modern vision-language models (see image captioning).
• Transformers for vision. Self-attention mixes patch tokens in the Vision Transformer and its hierarchical successors; DETR decodes object queries by attending to image features.
• Convolutional networks. Non-local blocks add self-attention to video and image CNNs, and squeeze-and-excitation and related gating blocks reweight channels and positions.
• Matching and generation. Feature matching methods attend within and between two images, and text-to-image models such as Stable Diffusion condition each denoising step on a prompt through cross-attention.
• Efficient perception. Glimpse models and spatial transformers process only selected regions at full resolution, trading global coverage for computation set by the number of glimpses.