SuperGlue

SuperGlue is a neural network that matches two sets of local features, finding the correspondences between them and rejecting the points that have none. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich (ETH Zurich and Magic Leap) presented it at CVPR 2020. It takes keypoints and descriptors from any detector, such as SuperPoint or SIFT, and replaces the nearest-neighbour search and hand-tuned filters of classical feature matching. The authors call it a learnable “middle-end”: it sits between a front-end that detects and describes points and a back-end, such as pose estimation or bundle adjustment, that uses the matches.

The model has two parts. An attentional graph neural network refines every descriptor with alternating self-attention, within an image, and cross-attention, across the two images. An optimal matching layer scores every pair of refined descriptors, adds a “dustbin” row and column for points without a partner, and solves a differentiable optimal transport problem with the Sinkhorn algorithm, which yields a soft partial assignment. Unless stated otherwise, details below come from the CVF open-access version of the paper and its supplementary material.

Problem Addressed

The classical pipeline that feature matching describes finds each point’s nearest neighbour in descriptor space and then filters the candidates with Lowe’s ratio test, the mutual check and further heuristics before RANSAC. The authors argue that these steps throw away information: learned inlier classifiers such as PointCN and OANet work on sets of matches that are still found by nearest-neighbour search, so they ignore the structure of the assignment and the appearance of the points (§2). Graph matching methods that use such structure are usually posed as quadratic assignment problems, which are NP-hard, and the authors left them out of their comparisons as orders of magnitude too slow for more than 500 keypoints (§2, §5.2).

The paper states two physical constraints (§3): a keypoint has at most one correspondence in the other image, and some keypoints have none, because of occlusion or a failed detection. Correspondences therefore form a partial assignment between the $M$ keypoints of image $A$ and the $N$ keypoints of image $B$. SuperGlue predicts a soft version of it, a matrix $\mathbf P \in [0, 1]^{M \times N}$ with

$$\mathbf P \mathbf 1_N \le \mathbf 1_M, \qquad \mathbf P^\top \mathbf 1_M \le \mathbf 1_N$$

(Eq. 1), so that each row and column carries at most one unit of match probability and the rest is left to “no match”.

Architecture

Each local feature $i$ has a position $\mathbf p_i = (x, y, c)_i$, with image coordinates and detection confidence $c$, and a visual descriptor $\mathbf d_i \in \mathbb R^D$. Both sets go through the graph network together; the optimal matching layer then compares them.

Keypoint Encoder

A multilayer perceptron embeds the position into the descriptor space and adds it to the descriptor (Eq. 2):

$${}^{(0)}\mathbf x_i = \mathbf d_i + \mathrm{MLP}_{\mathrm{enc}}(\mathbf p_i).$$

The authors present it as an instance of the positional encoder used in language models, which lets later layers reason about appearance and position jointly. In the supplement, the MLP has five layers with widths 32, 64, 128, 256 and $D$, about 100k parameters, and keypoint coordinates are normalized by the larger image dimension before it.

Attentional Graph Neural Network

The keypoints of both images form one complete graph with two kinds of undirected edges, which the authors call a multiplex graph: self edges join a keypoint to every other keypoint of its image, and cross edges join it to every keypoint of the other image. Each layer $\ell$ updates every node with a residual message-passing step (Eq. 3):

$${}^{(\ell+1)}\mathbf x_i^A = {}^{(\ell)}\mathbf x_i^A + \mathrm{MLP}\left(\left[{}^{(\ell)}\mathbf x_i^A \,\|\, \mathbf m_{\mathcal E \to i}\right]\right),$$

where $[\cdot \,\|\, \cdot]$ is concatenation and the same update runs for image $B$. Layers alternate between the two edge types: self edges on odd layers, cross edges on even ones. The message is an attention-weighted average of values (Eqs. 4–5), $\mathbf m_{\mathcal E \to i} = \sum_{j : (i, j) \in \mathcal E} \alpha_{ij} \mathbf v_j$ with $\alpha_{ij} = \mathrm{Softmax}_j(\mathbf q_i^\top \mathbf k_j)$, where the query comes from the receiving keypoint and the keys and values from the keypoints of the source image. Each layer has its own projections, shared by both images, and uses multi-head attention. A final linear projection gives the matching descriptors $\mathbf f_i$ (Eq. 6).

Compared with the transformer, the authors describe two changes (§3.4): the self-attention is embedded in a graph network, and a symmetric cross-attention is added. Because the same layers serve both images, the network is equivariant to permuting the keypoints within an image and also to swapping the two images, a symmetry the authors say earlier handcrafted and learned matchers lack. They describe the cross-attention as inspired by the way people look back and forth between two images when matching. The appendix measures the spatial extent of attention on ScanNet pairs and finds that it shrinks by more than a factor of ten from the first to the last layer: early layers attend widely, and late cross-attention concentrates near the true match.

Optimal Matching Layer

Scores. Every pair is scored by the inner product of matching descriptors (Eq. 7),

$$\mathbf S_{i,j} = \langle \mathbf f_i^A, \mathbf f_j^B \rangle, \qquad (i, j) \in \mathcal A \times \mathcal B.$$

Unlike learned visual descriptors, matching descriptors are not normalized, so their magnitude can change per feature and during training to express confidence. The paper notes that maximizing $\sum_{i,j} \mathbf S_{i,j} \mathbf P_{i,j}$ under Eq. 1 is a linear assignment problem, the problem the Hungarian algorithm solves exactly.

Dustbins. To let points go unmatched, each set gets a dustbin, a device the authors note is common in graph matching and was also used in SuperPoint’s detector head for cells without a point. The score matrix is augmented to $\bar{\mathbf S} \in \mathbb R^{(M+1) \times (N+1)}$ by a new row and column filled with one learnable parameter $z$ (Eq. 8):

$$\bar{\mathbf S}_{i, N+1} = \bar{\mathbf S}_{M+1, j} = \bar{\mathbf S}_{M+1, N+1} = z.$$

A keypoint goes to one keypoint of the other image or to the dustbin, but each dustbin can absorb as many points as the other image has. With $\mathbf a = [\mathbf 1_M^\top \; N]^\top$ and $\mathbf b = [\mathbf 1_N^\top \; M]^\top$, the augmented assignment must satisfy (Eq. 9)

$$\bar{\mathbf P} \mathbf 1_{N+1} = \mathbf a, \qquad \bar{\mathbf P}^\top \mathbf 1_{M+1} = \mathbf b.$$

Sinkhorn. Maximizing $\langle \bar{\mathbf S}, \bar{\mathbf P} \rangle$ under Eq. 9 is an optimal transport problem between the discrete distributions $\mathbf a$ and $\mathbf b$. The paper uses its entropy-regularized form, which gives a soft assignment and is solved on the GPU with the Sinkhorn algorithm: $\exp(\bar{\mathbf S})$ is normalized alternately along rows and columns, similar to row and column softmax, for $T$ iterations, after which the dustbins are dropped and $\mathbf P = \bar{\mathbf P}_{1:M, 1:N}$ (§3.2). The paper calls this a differentiable version of the Hungarian algorithm; Feature Matching’s summary calls it a differentiable optimal transport layer.

In our notation, with a temperature $\varepsilon > 0$, the regularized problem and its solution are

$$\bar{\mathbf P}_\varepsilon = \arg\max_{\bar{\mathbf P} \ge 0,\; \bar{\mathbf P}\mathbf 1 = \mathbf a,\; \bar{\mathbf P}^\top \mathbf 1 = \mathbf b} \; \langle \bar{\mathbf S}, \bar{\mathbf P} \rangle - \varepsilon \sum_{i,j} \bar P_{ij} (\log \bar P_{ij} - 1), \qquad \bar{\mathbf P}_\varepsilon = \operatorname{diag}(\mathbf u) \, e^{\bar{\mathbf S} / \varepsilon} \operatorname{diag}(\mathbf v),$$

and Sinkhorn alternates $\mathbf u \leftarrow \mathbf a \oslash (e^{\bar{\mathbf S}/\varepsilon} \mathbf v)$ and $\mathbf v \leftarrow \mathbf b \oslash ((e^{\bar{\mathbf S}/\varepsilon})^\top \mathbf u)$. The paper runs Sinkhorn on $\exp(\bar{\mathbf S})$, which is $\varepsilon = 1$: because the scores are unnormalized and learned, their scale plays the role of an inverse temperature. In the example below the scores are cosines in $[-1, 1]$, so $\varepsilon$ stands in for that learned score scale.

Three consequences of these definitions are worth stating; each is our derivation, not a statement of the paper.

• The exact problem is a linear assignment. Copy the dustbin of $A$ into $N$ rows and the dustbin of $B$ into $M$ columns, all with score $z$. The $(M+N) \times (N+M)$ square problem has the same optimum as the transport problem with marginals $\mathbf a$ and $\mathbf b$, since transport problems with integer marginals have integer optimal solutions, and scipy.optimize.linear_sum_assignment solves it exactly. The construction resembles the non-assignment matrix in Hungarian algorithm, except that every dustbin copy can take any point, rather than one item each, and dustbin-to-dustbin pairs also score $z$ rather than zero.
• The dustbin acts as a threshold at $z$. With $k$ real matches, the total score is $\sum_{\text{matches}} \mathbf S_{ij} + z(M - k) + z(N - k) + z k$: unmatched rows and columns each pair with a dustbin copy, and the $k$ dustbin rows and columns left over pair with each other. That equals $z(M+N) + \sum_{\text{matches}} (\mathbf S_{ij} - z)$, so the exact layer is a maximum-weight one-to-one matching with weights $\mathbf S_{ij} - z$, in which a pair is only worth taking when its score exceeds $z$.
• The regularized optimum approaches the exact one linearly in $\varepsilon$. Write $H(\bar{\mathbf P}) = -\sum \bar P_{ij}(\log \bar P_{ij} - 1)$ and let $\bar{\mathbf X}$ be an exact optimum. Since $\bar{\mathbf P}_\varepsilon$ maximizes the regularized objective, $\langle \bar{\mathbf S}, \bar{\mathbf P}_\varepsilon \rangle + \varepsilon H(\bar{\mathbf P}_\varepsilon) \ge \langle \bar{\mathbf S}, \bar{\mathbf X} \rangle + \varepsilon H(\bar{\mathbf X})$, so the gap in total score is at most $\varepsilon [H(\bar{\mathbf P}_\varepsilon) - H(\bar{\mathbf X})]$, which goes to zero with $\varepsilon$. This bounds the objective, not the plan: the plan itself approaches an exact optimum only slowly when other assignments score almost as much.

The second point relates SuperGlue to Wasserstein GAN, whose example uses the fact that optimal transport between equal weights on equally many points is an assignment. Here the weights are unequal, because the dustbins carry mass $N$ and $M$, and the copying of dustbins restores the assignment form.

Training

SuperGlue is trained with supervision from ground-truth matches $\mathcal M = \{(i, j)\} \subset \mathcal A \times \mathcal B$, estimated from known relative transformations: homographies, or camera poses with depth maps. The same transformations label sets $\mathcal I \subseteq \mathcal A$ and $\mathcal J \subseteq \mathcal B$ of unmatched keypoints. The loss is the negative log-likelihood of the augmented assignment (Eq. 10):

$$\mathrm{Loss} = -\sum_{(i,j) \in \mathcal M} \log \bar{\mathbf P}_{i,j} - \sum_{i \in \mathcal I} \log \bar{\mathbf P}_{i, N+1} - \sum_{j \in \mathcal J} \log \bar{\mathbf P}_{M+1, j},$$

which, in the authors’ words, aims at maximizing precision and recall at once. Both the graph network and the Sinkhorn iterations are differentiable, so gradients reach the scores and, if wanted, the descriptor network.

Appendix E gives the labelling rule. The $M \times N$ matrix of reprojection errors is computed with the ground-truth transformation; a pair is a correspondence when its error is the minimum along both its row and its column and below 3, 5 and 3 pixels for homographies, indoor and outdoor data. For homographies, every other point is unmatched. Indoors and outdoors, where poses and depth have errors, a point is labelled unmatched only if its smallest reprojection error exceeds 15 or 5 pixels; points in between get no label but are still constrained by the normalization of the Sinkhorn layer.

Three models were trained, each with Adam at a constant learning rate of $10^{-4}$ for the first 200k, 100k or 50k iterations, followed by exponential decay until iteration 900k:

• Homographies: synthetic pairs made by random homographies and photometric distortions of the 1M distractor images of the Oxford and Paris retrieval dataset, generated on the fly so that no pair is seen twice; batches of 32 pairs with 512 SuperPoint keypoints each.
• Indoor: ScanNet, with pairs chosen by an overlap score between 0.4 and 0.8 computed from ground-truth poses and depth, which the authors argue gives wider baselines than selecting pairs by time difference or SIFT-based covisibility; batches of 64 pairs with 400 keypoints.
• Outdoor: MegaDepth, with scenes of the PhotoTourism test set removed, initialized from the homography model because of the limited number of training scenes; batches of 16 pairs with 1024 keypoints.

SuperPoint’s detection and description run on the fly during training to allow data augmentation, and random keypoints are added for batching and robustness (§4). Unless stated otherwise, the descriptor network is frozen, so that comparisons with other matchers use the same features.

Example: Sinkhorn with Dustbins

The example isolates the optimal matching layer. Nothing is trained: fixed patch descriptors stand in for the matching descriptors $\mathbf f$, so it is closest to the paper’s “No Graph Neural Net” ablation, which keeps the keypoint encoder and the matching layer but replaces the graph network by one linear projection. It checks that Sinkhorn with dustbins meets the marginals of Eq. 9 and that, as $\varepsilon \to 0$, the soft assignment approaches the exact one, computed by linear_sum_assignment on the dustbin-augmented cost. It then varies the dustbin score $z$ and the temperature $\varepsilon$, with mutual nearest neighbours as one reference row. Feature matching compares nearest-neighbour rules on synthetic descriptors, and the Hungarian algorithm article solves gated assignments; this example compares neither.

• Images. Image $A$ is the bundled chelsea photograph in grayscale, $300 \times 451$ pixels with intensities in $[0, 1]$. Image $B$ is $A$ warped bilinearly by a known homography $H$: a rotation of 8° and scale 0.9 about the centre, a mild perspective term and a shift of 60 pixels right and 10 down. Points are $(x, y)$ = (column, row), and $H$ maps a point of $A$ to its position in $B$. Part of $A$ leaves the frame, and a $90 \times 135$ block of $B$ is replaced by a crop of the bundled coins image as an occluder.
• Keypoints and scores. The 150 strongest Harris corners per image, at least 8 pixels from the border (and in $B$, from the warp’s edge), with a minimum spacing of 4 pixels. Each descriptor is a $15 \times 15$ patch of the Gaussian-smoothed image ($\sigma = 1$) at the keypoint, minus its mean, with unit norm; $\mathbf S$ holds the cosines, inner products as in Eq. 7.
• Ground truth. SuperGlue’s homography rule from Appendix E: a pair matches when its reprojection error is the minimum along its row and column and below 3 pixels. Points of $A$ that land outside $B$ or under the occluder, and points of $B$ on the occluder, cannot match. The script checks the coordinate convention by warping a Gaussian dot and comparing its centroid with $H(\mathbf x)$.
• Solvers. Sinkhorn in the log domain on $e^{\bar{\mathbf S}/\varepsilon}$, with the dustbin score $z \in \{0.3, 0.5, 0.7, 0.9\}$. For each $z$, the temperature goes down a ladder from $\varepsilon = 0.1$ to $0.001$, each step starting from the previous potentials; at each step Sinkhorn runs in blocks of 100 iterations until the largest row-marginal error is below $10^{-9}$, or for at most 20,000 iterations. SuperGlue’s own budget, $T = 100$ iterations from a cold start, is run at four temperatures. The exact limit is linear_sum_assignment on the $(M+N) \times (N+M)$ expansion described above.
• Measures. Matches from a soft plan are the pairs with $\mathbf P_{ij} > 0.2$, the paper’s confidence threshold (§4), scored by precision and recall against the ground truth. On the ladder the script also prints the largest row and column marginal errors, the gap between the exact optimum and the plan’s total score $\langle \bar{\mathbf S}, \bar{\mathbf P}_\varepsilon \rangle$, the largest entrywise difference from the exact assignment $\mathbf X$, and the number of entries where $\mathbf P > 0.5$ disagrees with $\mathbf X$.

The planned checks were: the data checks (dot error at most 0.05 pixels, a one-to-one ground truth, and the ground truth scoring precision and recall 1); column-marginal errors at most $10^{-9}$ in every Sinkhorn row; row-marginal errors at most $10^{-9}$ at $\varepsilon = 0.1$ and $0.05$; a gap that decreases strictly along the ladder for every $z$; and a gap at $\varepsilon = 0.001$ between 0 and 0.02 for every $z$. The hypotheses were: H1, at every $\varepsilon$, raising $z$ lowers the number of matches and raises their precision; H2, for every $z$, fewer entries of $\mathbf P > 0.5$ disagree with the exact assignment at $\varepsilon = 0.001$ than at $0.1$; H3, for every $z$, the number of matches at $\mathbf P > 0.2$ is closer to the exact count at $\varepsilon = 0.001$ than at $0.1$; and H4, no row beats mutual nearest neighbours on precision and recall at once. Before the plan was written, two diagnostic scripts ran: the first, on chelsea, found too few keypoints and ground-truth matches and exposed a sign error in the expanded cost, which was fixed; the second compared keypoint and ground-truth counts on four bundled images under two homographies; together they set the detector settings, the homography and the choice of chelsea. A planning script that fixed the tolerances printed the exact-assignment rows, the $T = 100$ rows and, on the ladder, the marginal errors, gaps and largest entrywise differences before the plan was written, so H1–H4 were tested on quantities that it did not print: the ladder’s matches above 0.2 and its disagreements at 0.5. The example uses NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2, has no random elements, and takes about two minutes on one CPU thread.

import os
for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[k] = "1"                      # one BLAS thread, set before NumPy is imported
import numpy as np
from scipy.ndimage import binary_erosion
from scipy.optimize import linear_sum_assignment
from scipy.special import logsumexp
from skimage import color, data, feature, filters, transform

MARGIN, R_PATCH, K = 8, 7, 150

# --- Image pair with a known homography, a region out of view and an occluder ---
# Points are (x, y) = (column, row); H maps a point of image A to its position in image B.
img_a = color.rgb2gray(data.chelsea())                       # 300 x 451, bundled
h, w = img_a.shape
cx, cy, th, s = w / 2, h / 2, np.deg2rad(8.0), 0.9
H = (np.array([[1, 0, cx + 60], [0, 1, cy + 10], [0, 0, 1]])          # translate
     @ np.array([[1, 0, 0], [0, 1, 0], [1e-4, -5e-5, 1]])              # perspective
     @ np.array([[s * np.cos(th), -s * np.sin(th), 0], [s * np.sin(th), s * np.cos(th), 0], [0, 0, 1]])
     @ np.array([[1, 0, -cx], [0, 1, -cy], [0, 0, 1]]))                # about the centre
tf = transform.ProjectiveTransform(matrix=H)
img_b = transform.warp(img_a, tf.inverse, output_shape=(h, w), order=1, cval=0.0)
inside_b = transform.warp(np.ones((h, w)), tf.inverse, output_shape=(h, w), order=0) > 0.5
occ = np.zeros((h, w), bool)
occ[int(.15 * h):int(.15 * h) + int(.3 * h), int(.55 * w):int(.55 * w) + int(.3 * w)] = True
img_b[occ] = (data.coins() / 255.0)[50:50 + int(.3 * h), 50:50 + int(.3 * w)].ravel()  # occluder
safe_b = binary_erosion(inside_b, iterations=MARGIN)

def detect(img, valid):                      # Harris corners, top K
    r = feature.corner_harris(img, method="k", k=0.05, sigma=1.5)
    r[~valid] = 0
    pk = feature.corner_peaks(r, min_distance=4, threshold_rel=1e-4, exclude_border=MARGIN, num_peaks=K)
    return pk[:, ::-1].astype(float)

def describe(img, xy):                       # fixed descriptor: normalized 15 x 15 patch
    g = filters.gaussian(img, sigma=1.0)
    p = np.array([g[y - R_PATCH:y + R_PATCH + 1, x - R_PATCH:x + R_PATCH + 1].ravel()
                  for x, y in xy.astype(int)])
    p -= p.mean(1, keepdims=True)
    return p / np.linalg.norm(p, axis=1, keepdims=True)

kp_a, kp_b = detect(img_a, np.ones((h, w), bool)), detect(img_b, safe_b)
S = describe(img_a, kp_a) @ describe(img_b, kp_b).T          # scores: cosine similarity
M, N = S.shape

# Ground truth as in SuperGlue's Appendix E: reprojection error minimal along its row and
# column and below 3 px; A points that land outside B or under the occluder, and B points
# on the occluder, cannot match.
proj = tf(kp_a)
pi = np.clip(np.round(proj).astype(int), 0, [w - 1, h - 1])
seen = (proj >= MARGIN).all(1) & (proj[:, 0] < w - MARGIN) & (proj[:, 1] < h - MARGIN)
seen &= safe_b[pi[:, 1], pi[:, 0]] & ~occ[pi[:, 1], pi[:, 0]]
on_occ = occ[kp_b[:, 1].astype(int), kp_b[:, 0].astype(int)]
D = np.linalg.norm(proj[:, None] - kp_b[None], axis=2)
D[~seen] = np.inf
D[:, on_occ] = np.inf
r_min, c_min = D.argmin(1), D.argmin(0)
gt = np.zeros((M, N), bool)
i = np.flatnonzero((D[np.arange(M), r_min] < 3) & (c_min[r_min] == np.arange(M)))
gt[i, r_min[i]] = True

yy, xx = np.mgrid[0:h, 0:w]                  # convention check: a warped dot lands at H(x)
dot_err = []
for x, y in [(150.3, 120.7), (220.6, 180.2), (100.4, 60.8)]:
    wd = transform.warp(np.exp(-((xx - x) ** 2 + (yy - y) ** 2) / 8.0), tf.inverse, output_shape=(h, w))
    dot_err.append(np.hypot(*(tf([[x, y]])[0] - [(wd * xx).sum() / wd.sum(), (wd * yy).sum() / wd.sum()])))
print(f"M={M} N={N}  A points visible in B={seen.sum()}  B points on occluder={on_occ.sum()}  "
      f"ground-truth matches={gt.sum()}")
print(f"checks: dot error {max(dot_err):.4f} px, ground truth one-to-one "
      f"{gt.sum(1).max() <= 1 and gt.sum(0).max() <= 1}")

def score(pred):
    tp = (pred & gt).sum()
    return f"n={pred.sum():3d} P={tp / max(pred.sum(), 1):.4f} R={tp / gt.sum():.4f}"

nn_ab, nn_ba = S.argmax(1), S.argmax(0)
mnn = np.zeros((M, N), bool)
i = np.flatnonzero(nn_ba[nn_ab] == np.arange(M))
mnn[i, nn_ab[i]] = True
print("ground truth itself      ", score(gt))
print("mutual nearest neighbour ", score(mnn))

# --- Optimal matching layer: dustbins, marginals a = [1_M, N], b = [1_N, M] (Eqs. 8-9) ---
a, b = np.r_[np.ones(M), N], np.r_[np.ones(N), M]

def sinkhorn(Sbar, eps, iters, f=None, g=None):
    """Log-domain Sinkhorn on exp(Sbar / eps); f, g are the dual potentials (warm start)."""
    f = np.zeros(M + 1) if f is None else f
    g = np.zeros(N + 1) if g is None else g
    for _ in range(iters):
        f = eps * (np.log(a) - logsumexp((Sbar + g[None, :]) / eps, axis=1))
        g = eps * (np.log(b) - logsumexp((Sbar + f[:, None]) / eps, axis=0))
    return np.exp((Sbar + f[:, None] + g[None, :]) / eps), f, g

def lsa_augmented(Sbar):
    """The limit eps -> 0: linear_sum_assignment with each dustbin copied N or M times."""
    big = np.full((M + N, N + M), Sbar[-1, -1])
    big[:M, :N] = S
    r, c = linear_sum_assignment(big, maximize=True)
    X = np.zeros((M, N))
    keep = (r < M) & (c < N)
    X[r[keep], c[keep]] = 1
    return X, big[r, c].sum()

for z in [0.3, 0.5, 0.7, 0.9]:               # dustbin score z of Eq. (8)
    Sbar = np.full((M + 1, N + 1), z)
    Sbar[:M, :N] = S
    X, best = lsa_augmented(Sbar)
    print(f"\nz={z}  linear_sum_assignment on the augmented cost: {score(X.astype(bool))}")
    print("    eps  iters  row err    col err    gap        max|P-X|  P>0.5 vs LSA  matches P>0.2")
    f = g = None
    for eps in [0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001]:   # eps-scaling: warm starts
        for block in range(1, 201):                           # at most 20,000 iterations
            P, f, g = sinkhorn(Sbar, eps, 100, f, g)
            if np.abs(P.sum(1) - a).max() < 1e-9:
                break
        row, col = np.abs(P.sum(1) - a).max(), np.abs(P.sum(0) - b).max()
        gap = best - (Sbar * P).sum()        # optimal total score minus the plan's total score
        diff = ((P[:M, :N] > 0.5) != X.astype(bool)).sum()
        print(f"  {eps:5.3f} {100 * block:6d}  {row:.3e}  {col:.3e}  {gap:.4e}  "
              f"{np.abs(P[:M, :N] - X).max():.4f}  {diff:12d}  {score(P[:M, :N] > 0.2)}")
    for eps in [0.1, 0.05, 0.02, 0.01]:      # SuperGlue's fixed budget, T = 100, cold start
        P, _, _ = sinkhorn(Sbar, eps, 100)
        print(f"  T=100, eps={eps:<5} row err {np.abs(P.sum(1) - a).max():.3e}  "
              f"col err {np.abs(P.sum(0) - b).max():.3e}  {score(P[:M, :N] > 0.2)}")

Output:

M=150 N=150  A points visible in B=91  B points on occluder=37  ground-truth matches=72
checks: dot error 0.0121 px, ground truth one-to-one True
ground truth itself       n= 72 P=1.0000 R=1.0000
mutual nearest neighbour  n= 77 P=0.8571 R=0.9167

z=0.3  linear_sum_assignment on the augmented cost: n=147 P=0.4558 R=0.9306
    eps  iters  row err    col err    gap        max|P-X|  P>0.5 vs LSA  matches P>0.2
  0.100    100  1.137e-13  1.137e-13  3.2541e+01  0.9922           143  n= 55 P=0.9636 R=0.7361
  0.050    300  2.210e-10  8.527e-14  9.7239e+00  0.9891            90  n=146 P=0.4795 R=0.9722
  0.020  20000  6.375e-05  3.979e-13  1.2956e+00  0.9653            41  n=195 P=0.3590 R=0.9722
  0.010  20000  1.228e-04  7.105e-13  3.4673e-01  0.9131            33  n=190 P=0.3684 R=0.9722
  0.005  20000  2.741e-04  9.663e-13  1.0292e-01  0.7509            18  n=181 P=0.3812 R=0.9583
  0.002  20000  2.993e-04  2.274e-12  2.4570e-02  0.6515             1  n=163 P=0.4172 R=0.9444
  0.001  20000  1.909e-04  3.695e-12  7.0044e-03  0.4913             0  n=151 P=0.4437 R=0.9306
  T=100, eps=0.1   row err 1.137e-13  col err 1.137e-13  n= 55 P=0.9636 R=0.7361
  T=100, eps=0.05  row err 1.485e-04  col err 1.421e-13  n=146 P=0.4795 R=0.9722
  T=100, eps=0.02  row err 5.039e-01  col err 2.109e-15  n=195 P=0.3590 R=0.9722
  T=100, eps=0.01  row err 2.922e+00  col err 8.527e-14  n=194 P=0.3608 R=0.9722

z=0.5  linear_sum_assignment on the augmented cost: n=136 P=0.5000 R=0.9444
    eps  iters  row err    col err    gap        max|P-X|  P>0.5 vs LSA  matches P>0.2
  0.100    100  1.554e-15  5.684e-14  2.7916e+01  0.9892           136  n= 37 P=0.9730 R=0.5000
  0.050    100  8.527e-14  2.842e-14  1.1677e+01  0.9896            85  n=102 P=0.6667 R=0.9444
  0.020  20000  2.129e-08  1.990e-13  1.7345e+00  0.9918            41  n=161 P=0.4348 R=0.9722
  0.010  20000  1.515e-04  3.411e-13  3.8358e-01  0.9919            30  n=165 P=0.4242 R=0.9722
  0.005  20000  2.171e-04  6.253e-13  1.0499e-01  0.9916            24  n=160 P=0.4375 R=0.9722
  0.002  20000  2.423e-04  3.155e-12  2.4526e-02  0.9901            17  n=154 P=0.4416 R=0.9444
  0.001  20000  1.432e-04  4.320e-12  1.1062e-02  0.9868            13  n=149 P=0.4564 R=0.9444
  T=100, eps=0.1   row err 1.554e-15  col err 5.684e-14  n= 37 P=0.9730 R=0.5000
  T=100, eps=0.05  row err 1.181e-13  col err 8.527e-14  n=102 P=0.6667 R=0.9444
  T=100, eps=0.02  row err 1.446e-01  col err 5.684e-14  n=161 P=0.4348 R=0.9722
  T=100, eps=0.01  row err 7.426e-01  col err 1.705e-13  n=167 P=0.4192 R=0.9722

z=0.7  linear_sum_assignment on the augmented cost: n=104 P=0.6154 R=0.8889
    eps  iters  row err    col err    gap        max|P-X|  P>0.5 vs LSA  matches P>0.2
  0.100    100  1.137e-13  2.842e-14  1.6230e+01  0.9779           104  n=  0 P=0.0000 R=0.0000
  0.050    100  8.527e-14  2.842e-14  9.8273e+00  0.9751            94  n= 61 P=0.9180 R=0.7778
  0.020    700  3.942e-10  1.110e-15  2.3802e+00  0.9798            42  n=101 P=0.6436 R=0.9028
  0.010  20000  2.887e-05  1.421e-13  5.5705e-01  0.9556            24  n=116 P=0.5690 R=0.9167
  0.005  20000  1.996e-04  8.527e-14  1.4181e-01  0.8365            14  n=115 P=0.5739 R=0.9167
  0.002  20000  1.352e-04  4.832e-13  2.2400e-02  0.5809             7  n=111 P=0.5946 R=0.9167
  0.001  20000  2.054e-04  4.974e-12  6.0665e-03  0.3694             0  n=107 P=0.6075 R=0.9028
  T=100, eps=0.1   row err 1.137e-13  col err 2.842e-14  n=  0 P=0.0000 R=0.0000
  T=100, eps=0.05  row err 2.842e-14  col err 5.684e-14  n= 61 P=0.9180 R=0.7778
  T=100, eps=0.02  row err 6.652e-03  col err 2.842e-14  n=101 P=0.6436 R=0.9028
  T=100, eps=0.01  row err 2.102e-01  col err 5.684e-14  n=116 P=0.5690 R=0.9167

z=0.9  linear_sum_assignment on the augmented cost: n= 45 P=0.9333 R=0.5833
    eps  iters  row err    col err    gap        max|P-X|  P>0.5 vs LSA  matches P>0.2
  0.100    100  1.421e-13  5.684e-14  4.5292e+00  0.9664            45  n=  0 P=0.0000 R=0.0000
  0.050    100  7.958e-13  1.137e-13  2.1285e+00  0.9442            45  n=  7 P=1.0000 R=0.0972
  0.020    100  1.421e-13  5.684e-14  1.1695e+00  0.9438            35  n= 29 P=0.9310 R=0.3750
  0.010    200  1.836e-11  2.842e-14  5.3933e-01  0.9619            19  n= 35 P=0.9143 R=0.4444
  0.005   8800  9.209e-10  1.421e-13  1.6158e-01  0.9700            11  n= 38 P=0.9474 R=0.5000
  0.002  20000  9.685e-05  8.527e-14  3.4736e-02  0.9720             6  n= 42 P=0.9524 R=0.5556
  0.001  20000  2.501e-05  3.411e-13  9.9145e-03  0.9699             2  n= 44 P=0.9318 R=0.5694
  T=100, eps=0.1   row err 1.421e-13  col err 5.684e-14  n=  0 P=0.0000 R=0.0000
  T=100, eps=0.05  row err 1.990e-12  col err 2.842e-14  n=  7 P=1.0000 R=0.0972
  T=100, eps=0.02  row err 6.537e-13  col err 2.842e-14  n= 29 P=0.9310 R=0.3750
  T=100, eps=0.01  row err 7.365e-06  col err 5.684e-14  n= 35 P=0.9143 R=0.4444

The data checks hold: the warped dot lands within 0.0121 pixels of $H(\mathbf x)$, the 72 ground-truth matches are one-to-one and score precision and recall 1.0000 against themselves, and 59 of image $A$’s 150 points are out of view or occluded in $B$, so the dustbins have points to absorb. The marginal checks pass: every column error is below $10^{-11}$, because each Sinkhorn iteration ends with the column update, and the ladder’s row errors at $\varepsilon = 0.1$ and $0.05$ are at most $2.2 \times 10^{-10}$. Below $\varepsilon = 0.05$, Sinkhorn often uses its full 20,000 iterations, and row errors of up to $3 \times 10^{-4}$ remain, which is the slow convergence of Sinkhorn at low temperature.

Convergence to the exact assignment. The objective converged: for every $z$ the gap falls strictly at each step of the ladder, from between 4.5 and 32.5 at $\varepsilon = 0.1$ to between 0.006 and 0.011 at $\varepsilon = 0.001$, which passes the planned checks. Entrywise convergence to the exact assignment failed within the budget while the objective converged: at $\varepsilon = 0.001$ the largest $|\mathbf P - \mathbf X|$ is still 0.49, 0.99, 0.37 and 0.97 for $z = 0.3$, 0.5, 0.7 and 0.9. The batch plan asked for “convergence as ε → 0 to linear_sum_assignment on the augmented cost”, which the entrywise numbers do not show; our tolerances were set on the gap because a planning run had already shown that the plan does not reach the vertex within 20,000 iterations per temperature. Rounded at 0.5, the plan agrees exactly with the exact assignment for $z = 0.3$ and $0.7$, differs in 13 entries for $z = 0.5$ and in 2 for $z = 0.9$. In our reading, these are pairs whose scores leave other assignments almost as good: the bound derived above caps the loss in total score at $\varepsilon$ times an entropy difference, but says nothing about how far the plan is from a vertex when two vertices nearly tie.

Dustbin score. In the exact rows, raising $z$ from 0.3 to 0.9 cuts the matches from 147 to 45, raises precision from 0.4558 to 0.9333 and lowers recall from 0.9306 to 0.5833, as the threshold reading of the dustbin predicts: only pairs whose cosine exceeds $z$ are worth matching. Hypothesis H1 failed at $\varepsilon = 0.1$: there the soft plan has 55, 37, 0 and 0 entries above 0.2 for the four values of $z$, so the count does not fall strictly from 0.7 to 0.9 and precision drops to zero instead of rising, because at that temperature no entry reaches 0.2 once the dustbin scores 0.7 or more. H1 held at every $\varepsilon \le 0.05$, on the ladder and in the $T = 100$ rows, where counts fall and precision rises monotonically with $z$ (at $\varepsilon = 0.001$, 151, 149, 107 and 44 matches with precision 0.4437, 0.4564, 0.6075 and 0.9318).

Temperature. H2 held for every $z$: the entries where $\mathbf P > 0.5$ disagrees with the exact assignment fall from 143, 136, 104 and 45 at $\varepsilon = 0.1$ to 0, 13, 0 and 2 at $\varepsilon = 0.001$. H3 held for every $z$: the number of matches above 0.2 ends 4, 13, 3 and 1 away from the exact count, against 92, 99, 104 and 45 at $\varepsilon = 0.1$. A high temperature spreads each point’s unit of mass over many partners and the dustbin, so few entries pass the threshold; a low one concentrates it.

More matches than keypoints. At $z = 0.3$ and $\varepsilon = 0.02$, 195 entries exceed 0.2 although each image has only 150 points, and the row-marginal error there is $6.4 \times 10^{-5}$. Our planning note had attributed such counts to unmet row marginals under the fixed $T = 100$ budget; that note was wrong. The correct reason follows from the marginals (our derivation, not the paper’s): when a real row sums to 1 with its dustbin entry, at most four of its entries can exceed 0.2, since five would sum to more than 1. So a threshold of 0.2 alone does not give a one-to-one set, whereas a threshold above 0.5 does, since at most one entry per row and per column can exceed 0.5.

SuperGlue’s budget. With $T = 100$ iterations from a cold start, the plan is converged at $\varepsilon = 0.1$, where it equals the ladder’s first row, and nearly so at $0.05$. At lower temperatures the row marginals are not met: the row error reaches 2.922 at $z = 0.3$ and $\varepsilon = 0.01$, so the plan does not meet Eq. 9: some row of $\bar{\mathbf P}$ misses its marginal by up to 2.9. In SuperGlue the learned scale of the scores sets the effective temperature, and in our reading a network trained with $T = 100$ learns scores whose scale suits that budget.

Mutual nearest neighbours. H4 held: no row beats mutual nearest neighbours (precision 0.8571, recall 0.9167) on both counts. Rows with higher precision have recall of at most 0.7778, and rows with recall of at least 0.9167 have precision of at most 0.6667. In our reading, the dustbin of an exact or nearly exact layer is a single global threshold on fixed scores, while the mutual check already gives a one-to-one set and removes many of the unmatched points; the transport layer alone adds little to fixed descriptors. This agrees with the paper’s ablation, in which the matching layer without the graph network gained 2.16 points of AUC@20° over NN + mutual and the graph network brought most of the gain.

Inference

At test time SuperGlue takes the keypoints and descriptors of two images and returns the soft assignment $\mathbf P$. Matches can be kept with a confidence threshold, for which the authors chose 0.2, or all of them can be passed on with their confidences to a later step, such as weighted pose estimation (§4). The paper states only the threshold; the example above shows that it does not by itself make matches one-to-one.

The model uses $D = 256$, the size of SuperPoint’s descriptors, for all intermediate representations, $L = 9$ layers of alternating self- and cross-attention with 4 heads each, and $T = 100$ Sinkhorn iterations. It has 12M parameters; the supplement counts 0.66M per layer and 18 layers in total. On an NVIDIA GTX 1080, a forward pass for an indoor image pair took 69 ms on average (15 FPS), and the supplement reports 14.5 and 11.5 FPS for 512 and 1024 keypoints per image, with the graph network and the matching layer costing about the same. Because attention and the score matrix cover all pairs of keypoints, the cost grows quadratically with the number of keypoints, as LightGlue’s authors note.

Performance

All numbers are from the paper’s tables, read from the CVF copy. The baselines are matchers applied to the same features: nearest neighbour (NN) with handcrafted filters (the mutual check, the ratio test, a descriptor-distance threshold, GMS) or with the learned inlier classifiers PointCN, OANet and NG-RANSAC.

Homography estimation (Table 1). On 1,024 test pairs of $640 \times 480$ images with synthetic homographies and SuperPoint features (512 keypoints), the corner reprojection error is summarized as the area under its cumulative curve up to 10 pixels. SuperGlue reached an AUC of 53.67 with RANSAC and 65.85 with the direct linear transform (DLT), a least-squares solver with no robustness, at a match precision of 90.7% and recall of 98.3%. The best baseline, NN + OANet, reached 44.55 and 52.29, with precision 82.8% and recall 64.7%. NN + mutual reached 0.24 with DLT. The authors note that SuperGlue’s matches are clean enough for DLT to beat RANSAC.

Indoor pose (Table 2). On 1,500 ScanNet test pairs, the pose error is the larger of the rotation and translation angular errors, with poses from essential-matrix estimation with RANSAC; the table reports its AUC at 5°, 10° and 20°, the match precision (P) and the matching score (MS), correct matches over detected keypoints. With SuperPoint, SuperGlue reached 16.16, 33.81 and 51.84, P 84.4% and MS 31.5%, against 11.76, 26.90 and 43.85 for NN + OANet and 9.43, 21.53 and 36.40 for NN + mutual. With root-normalized SIFT, SuperGlue reached 6.71, 15.70 and 28.67, against 5.83, 13.06 and 22.47 for the ratio test. The authors also report up to 10 times more correct matches than the ratio test with SIFT, which they attribute to operating on all possible matches rather than on nearest neighbours.

Outdoor pose (Table 3). On the PhotoTourism set of the CVPR 2019 Image Matching Challenge, with models trained on MegaDepth, the same measures gave 34.18, 50.32 and 64.16 with precision 84.9% for SuperPoint + SuperGlue, against 21.03, 34.08 and 46.88 for SuperPoint + NN + OANet. With SIFT, SuperGlue reached 23.68, 36.44 and 49.44, against 15.19, 24.72 and 35.30 for the ratio test.

Ablation (Table 4, indoor, SuperPoint). Pose AUC@20° and precision were 36.40 and 50.4% for NN + mutual. SuperGlue without the graph network reached 38.56 and 66.0%; without cross-attention, 42.57 and 74.0%; without the positional encoding, 47.12 and 75.8%; with 3 layers, 46.93 and 79.9%; and the full 9-layer model 51.84 and 84.4%. The authors read this as the optimal matching layer alone improving on nearest neighbours while the graph network brings most of the gain. Training SuperPoint’s descriptor network through SuperGlue raised AUC@20° from 51.84 to 53.38 (§5.4).

The supplement also evaluates the homography model on HPatches and SuperPoint with SuperGlue on the Aachen Day-Night visual localization benchmark, where the authors report results similar to or better than existing approaches with fewer keypoints.

Strengths

• Any front-end. SuperGlue needs only keypoints and descriptors, learned or handcrafted, and improved pose accuracy over the other matchers in the paper with both SuperPoint and SIFT.
• Joint matching and rejection. One model finds correspondences and decides which points have none, instead of a nearest-neighbour search followed by separate filters.
• Structure built in. The transport layer enforces the one-to-one structure of Eq. 1 in a soft way, which the authors liken to a soft mutual check embedded in training (§3.4); the attention layers make the result symmetric in the two images.
• Confidences. The soft assignment gives each match a value that later stages can use as a weight.

Limitations

• Quadratic cost. Every layer attends over all keypoint pairs and the matching layer fills an $M \times N$ matrix, so time and memory grow with the product of the keypoint counts; LightGlue’s authors name this, with the expense of the Sinkhorn iterations, as a reason for their revision.
• Training. LightGlue’s authors describe SuperGlue as hard to train, needing computing resources that many practitioners lack, and report that follow-up works did not reach the original model’s performance.
• The dustbin couples all points. In LightGlue’s analysis, a single dustbin entangles the similarity scores of all points and gives suboptimal training dynamics; they replace it with a separate matchability score per point.
• Only the last layer is supervised. Because Sinkhorn is expensive, SuperGlue predicts and is supervised only after its final layer, so it cannot stop early on easy pairs (LightGlue, §3.5).
• The threshold is not a matching rule. As the example shows, a threshold of 0.2 on a soft assignment can keep several partners for one point.
• Licence. The official repository, which holds the code and trained weights, carries a noncommercial research licence (see Implementations).

Historical Importance

SuperGlue moved learning from the descriptor into the matching step. Before it, learning in matching pipelines meant better detectors and descriptors, such as SuperPoint and the other learned local features, or classifiers that pruned nearest-neighbour matches; the matcher itself stayed a hand-designed nearest-neighbour search. SuperGlue showed that a network reading both keypoint sets at once, with an assignment layer at the end, could raise precision and recall together, and it presented the result as a step towards end-to-end deep SLAM.

Two of its ideas spread. Interleaved self- and cross-attention between two images was taken up by later matchers, LightGlue among them: LoFTR applied it to dense feature maps without a detector, and offers an optimal transport layer in SuperGlue’s style as one of its two matching layers, alongside a dual-softmax. The differentiable partial assignment with dustbins gave a way to train matching end to end with an explicit “no match” option. SuperPoint with SuperGlue is among the combinations that the hloc toolbox, described in visual localization, implements.

Variants

• SuperGlue with SIFT. The paper trains separate models for root-normalized SIFT features, with 1024 keypoints and 24 pairs per batch.
• End-to-end SuperPoint and SuperGlue. Two copies of SuperPoint are initialized from the original weights; the detector is frozen and the descriptor network is trained through SuperGlue’s loss alone (Appendix E).
• LightGlue (Lindenberger, Sarlin and Pollefeys, ICCV 2023) revisits SuperGlue’s design (§3.5). It replaces the absolute positional encoding with a relative, rotary encoding added in every self-attention layer, and the Sinkhorn layer and dustbin with a lighter head: a similarity matrix normalized by softmax over rows and over columns, multiplied by a sigmoid matchability score for each point (Eq. 8). The lighter head allows an assignment and supervision after every layer, so the network can stop early on easy pairs and prune points it judges unmatchable. In their ablation on synthetic homographies (Table 5), with SuperPoint features and both models trained in the same set-up, SuperGlue reached a precision of 74.6% and recall of 90.5%, and LightGlue 86.8% and 96.3%; the text summarizes the difference as +4% recall and +12% precision, and the table’s recall difference is 5.8 points (the +4% recall also appears in the caption of their Figure 5, after 5M training pairs). The authors add that SuperGlue can reach similar accuracy with long enough training.

Implementations

The official repository, magicleap/SuperGluePretrainedNetwork, holds the code and trained weights, as the paper and supplement state. As of November 2026, its licence is a noncommercial research-use agreement for academic or nonprofit use, which states that using or downloading the software means accepting its terms; only the licence was read for this article. The hloc toolbox described in visual localization implements localization pipelines with SuperPoint features matched by SuperGlue.