LoFTR

LoFTR (Local Feature TRansformer) is a model for feature matching between two images, introduced by Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao and Xiaowei Zhou at CVPR 2021. It has no keypoint detector. A convolutional network computes feature maps of both images. A transformer of interleaved self-attention and cross-attention layers rewrites the coarse map at 1/8 of the image resolution so that each feature depends on both images. Every coarse position of one image is then scored against every coarse position of the other. The confident mutual matches are refined at 1/2 resolution by an expectation over a small correlation window. The result is a semi-dense set of matches, in the authors’ words, that can include points in low-texture regions where detectors find few repeatable keypoints. LoFTR belongs to the family of learned local features and learned matchers, and it adapts the attention design of SuperGlue to dense feature maps.

Problem Addressed

The classical matching pipeline runs in three stages: detect interest points in each image, describe a patch around each one, then match the descriptors by nearest-neighbour search or a learned matcher. SuperPoint learned the first two stages, and SuperGlue learned the third. LoFTR’s authors argue that the detector itself limits the pipeline. A detector may not fire on the same scene points in both images, because of poor texture, repetitive patterns, viewpoint or illumination change and motion blur. The authors add that, without repeatable interest points, correct correspondences cannot be found even with perfect descriptors (§1). They name indoor scenes, where low-texture regions or repetitive patterns sometimes occupy most of the field of view, as the case where this is most prominent.

Detector-free methods existed before LoFTR. The authors’ related work traces dense feature matching to SIFT Flow, and it names NCNet as the method that learned dense correspondences end to end. NCNet builds a 4D cost volume over all pairs of positions and regularizes it with 4D convolutions to enforce neighbourhood consensus; Sparse NCNet and the concurrent DRC-Net followed it. LoFTR’s authors argue that these convolutional methods see only a limited neighbourhood of each match. Their alternative is consensus over the whole image, from the global receptive field of attention. Their example is a white wall, whose positions can be told apart by where they lie relative to the edges of the wall (§1).

Architecture

LoFTR has four components, numbered in Fig. 2 of the paper:

a local feature CNN that produces coarse and fine feature maps;
the LoFTR module, a transformer on the coarse maps;
a differentiable matching layer that turns the transformed features into a confidence matrix and selects coarse matches;
a coarse-to-fine module that refines each selected match.

The notation follows the paper: images $I^A$ and $I^B$ , coarse maps $\tilde F^A, \tilde F^B$ at 1/8 resolution, and fine maps $\hat F^A, \hat F^B$ at 1/2 resolution.

Local Feature CNN

The backbone is a modified ResNet-18 with a feature pyramid network, trained from random initialization. The authors give two reasons for the CNN: its inductive bias of translation equivariance and locality suits local feature extraction, and its downsampling shortens the sequence the transformer must process (§3.1). In the official configuration (cvpr_ds_config.py), the coarse features have 256 channels and the fine features 128.

LoFTR Module

The coarse maps are flattened to sequences of $(H/8)(W/8)$ vectors. A 2D sinusoidal positional encoding is added once, following the 2D extension of DETR; unlike DETR, it is not added again at every layer (§3.2). The sequences then pass through $N_c = 4$ pairs of layers, each a self-attention layer within each image followed by a cross-attention layer between the two. This interleaving follows SuperGlue, which the authors describe as a graph neural network over two sets of detected keypoints. In LoFTR every coarse position takes part, so attention is not limited to detected points.

A sequence of $N$ positions makes full attention cost $O(N^2)$ ; at 1/8 of a $640 \times 480$ image, $N = 80 \times 60 = 4{,}800$ per image. The authors call vanilla attention impractical here even at 1/8 resolution. They use the linear transformer of Katharopoulos et al. (2020), as they describe it. The softmax kernel $\exp(\mathbf q^\top \mathbf k)$ is replaced by $\phi(\mathbf q)^\top \phi(\mathbf k)$ with $\phi(\cdot) = \operatorname{elu}(\cdot) + 1$ . Associativity then lets $\phi(K)^\top V$ be computed first, which costs $O(N)$ because the feature dimension is much smaller than $N$ . The official linear_attention.py implements this kernel. Attention mechanism places linear attention among the other efficient variants.

The authors attribute two properties to the transformed features $\tilde F^A_{tr}, \tilde F^B_{tr}$ . They are context-dependent, through the global receptive field, and position-dependent, through the positional encoding. Fig. 4(c) visualizes the features with PCA: on a uniform white wall, the transformed features vary smoothly with position, so each position is distinct even though the input colour is not.

Matching Layer

The score matrix between all coarse positions is

\mathcal S(i, j) = \frac{1}{\tau}\,\big\langle \tilde F^A_{tr}(i), \tilde F^B_{tr}(j) \big\rangle .

The paper offers two differentiable layers. The first is an optimal-transport layer as in SuperGlue, with $-\mathcal S$ as the cost of a partial assignment, solved with three Sinkhorn iterations in LoFTR’s implementation. The second is the dual-softmax, credited to NCNet and DISK, which applies a softmax along both dimensions:

\mathcal P_c(i, j) = \operatorname{softmax}\big(\mathcal S(i, \cdot)\big)_j \cdot \operatorname{softmax}\big(\mathcal S(\cdot, j)\big)_i .

The paper calls this the probability of soft mutual nearest-neighbour matching. Coarse matches are the pairs whose confidence reaches a threshold $\theta_c$ and that are mutual nearest neighbours in $\mathcal P_c$ :

\mathcal M_c = \big\{ (i, j) \;\big|\; (i, j) \in \operatorname{MNN}(\mathcal P_c),\; \mathcal P_c(i, j) \ge \theta_c \big\},

with $\theta_c = 0.2$ (§3.6). In the official coarse_matching.py, each feature is first divided by $\sqrt{C}$ , so the dot product is scaled by $1/C$ as well as by $1/\tau$ . The default temperature is 0.1. The threshold test is strict ( $>$ ). Two border cells are masked out in both images, and the mutual check compares each entry with the maxima of its row and column over the full matrix. The main paper does not state the $1/\sqrt{C}$ scaling or the border mask. The optimal-transport variant imports log_optimal_transport from SuperGlue’s superglue.py and adds a learnable dustbin score; the README says the authors cannot redistribute that file because of its licence.

Coarse-to-Fine Module

Each coarse match $(\tilde i, \tilde j)$ is located on the fine maps as $(\hat i, \hat j)$ , and a $w \times w$ window with $w = 5$ is cropped around each position. A smaller LoFTR module with $N_f = 1$ self- and cross-attention pair transforms the two windows. The centre vector of A’s window is correlated with all $w^2$ vectors of B’s window. The softmax of these correlations is a heatmap of match probability, and its expectation gives the refined position $\hat j'$ in $I^B$ to sub-pixel accuracy (§3.4). The authors say this correlation-based approach is inspired by Wang et al. (2020). A’s point stays fixed: only the position in B is refined.

In the official fine_matching.py, the heatmap is $\operatorname{softmax}(\mathbf f_A^\top \mathbf f_B / \sqrt{C})$ , and the expectation is taken in normalized window coordinates. It is scaled by $\lfloor w/2 \rfloor$ times the fine stride of 2 pixels and added to the coarse position in B. fine_preprocess.py crops the windows with stride 4 on the 1/2-resolution map, centred on the coarse cell, and adds coarse context before the fine transformer. §3.6 says the coarse transformer features are upsampled and concatenated with the fine features. In fine_preprocess.py the coarse feature of each matched cell is projected and concatenated with every vector of its window. Coarse position $k$ maps to pixel $8k$ in the code, and fine position $m$ to pixel $2m$ . With $w = 5$ at stride 2, the window spans $\pm 4$ pixels, exactly half a coarse cell, so a correct coarse match always leaves the true position inside the window.

This expectation is a soft-argmax over a window. Heatmap regression describes its failure mode: the softmax gives every cell some weight, which pulls the expectation towards the centre of the map. In LoFTR the centre of the window is the coarse match, so the pull is back towards the coarse position. Coarse-to-fine estimation and PWC-Net refine dense flow across a pyramid by warping. LoFTR instead refines each sparse match once, in a fixed window, without warping.

Training

The indoor model was trained on ScanNet and the outdoor model on MegaDepth, following SuperGlue (§3.6). Ground-truth matches come from camera poses and depth maps. The coarse ground truth $\mathcal M_c^{gt}$ is the set of mutual nearest neighbours between the two 1/8-resolution grids, with the distance between grid cells measured by reprojecting their centres. In the official supervision.py, the grid points are warped, rounded to the nearest cell and kept when the round trip returns to the starting cell.

The loss is $\mathcal L = \mathcal L_c + \mathcal L_f$ . For the dual-softmax, the coarse loss is the negative log-likelihood of the ground-truth cells:

\mathcal L_c = -\frac{1}{|\mathcal M_c^{gt}|} \sum_{(\tilde i, \tilde j) \in \mathcal M_c^{gt}} \log \mathcal P_c(\tilde i, \tilde j).

With optimal transport, the authors use SuperGlue’s loss. The fine loss is an $\ell_2$ distance weighted by the inverse of the heatmap’s total variance $\sigma^2(\hat i)$ , so that confident refinements count more:

\mathcal L_f = \frac{1}{|\mathcal M_f|} \sum_{(\hat i, \hat j') \in \mathcal M_f} \frac{1}{\sigma^2(\hat i)} \big\| \hat j' - \hat j'_{gt} \big\|_2 ,

where $\hat j'_{gt}$ is $\hat i$ warped with the ground-truth pose and depth. Matches whose warped point falls outside the window are ignored, and no gradient flows through $\sigma^2$ (§3.5). On ScanNet, training used Adam with an initial learning rate of $10^{-3}$ and a batch of 64, and converged after 24 hours on 64 GTX 1080Ti GPUs. The authors sampled 230 million training pairs with overlap scores between 0.4 and 0.8, following SuperGlue’s procedure (§4.2). During training, the official coarse_matching.py pads the coarse matches sent to the fine level with ground-truth matches, so the fine stage also trains on ground-truth coarse matches, not only on predicted ones.

The example trains no network. It keeps LoFTR’s matching layer and coarse-to-fine module as the official code implements them, and replaces the CNN and both transformers with fixed, hand-set descriptors. This shows what the two stages do with the features they are given, and where they fail when the features are not distinctive.

Data and conventions:

Images: two bundled scikit-image images in grey, coffee ( $400 \times 600$ ) and rocket ( $427 \times 640$ , with a large sky).
Pairs: image B is image A warped by a known homography $H$ : a rotation of 4° and a scale of 1.04 about the image centre, a shift of $(11.3, -6.6)$ pixels and a mild perspective term.
Points are $(x, y)$ = (column, row), $H$ maps A’s pixels to B’s, and $B(H\mathbf x) = A(\mathbf x)$ , with bilinear interpolation and zeros outside.
Noise: each image is run clean and with Gaussian noise of standard deviation 0.03 added to B, with noise seeds 0, 1 and 2, for eight pairs.

Features and matching:

Coarse descriptors. For the cell at pixel $8k$ : a $25 \times 25$ patch of the image blurred with $\sigma = 2$ , sampled every 3 pixels and normalized to zero mean and unit length (81 values).
Coarse matching. The dual-softmax is applied to cosines divided by $\tau_c$ . This equals the code’s $\langle \mathbf f/\sqrt C, \mathbf g/\sqrt C\rangle/\tau$ for feature vectors of norm $\sqrt C$ ; the equivalence is our reasoning. Selection follows coarse_matching.py: threshold 0.2 (tested with $\ge$ rather than the code’s $>$ ), two border cells removed, and a mutual maximum over the full matrix.
Fine descriptors. $5 \times 5$ patches of the half-resolution image (blur $\sigma = 1$ , even pixels), normalized the same way.
Refinement. The heatmap is the softmax of the 25 cosines divided by $\tau_f$ , and the refined position is $8j + 2\,\mathbb E[\mathbf d]$ for offsets $\mathbf d \in \{-2, \dots, 2\}^2$ . The value $\tau_f = 1/\sqrt{128} \approx 0.0884$ is the code’s $1/\sqrt C$ for the fine width of 128, again for vectors of norm $\sqrt C$ . The argmax limit ( $\tau_f \to 0$ ) and the uniform limit ( $\tau_f = \infty$ ) are computed exactly.

Measures:

Ground truth. Coarse ground truth follows supervision.py. A selected coarse match is correct if it is a ground-truth pair.
End-point error (EPE). The distance in pixels between a match’s position in B and $H$ applied to A’s point $8i$ .
Teacher-forced refinement. The fine table refines the ground-truth coarse matches, as in the ground-truth padding of LoFTR’s fine training, so refinement is measured apart from coarse errors.
Default pipeline. A last block runs the code’s defaults, $\tau_c = 0.1$ , $\theta_c = 0.2$ and $\tau_f = 1/\sqrt{128}$ .

Checks printed before the tables:

E1: the dual-softmax against its log-domain form $\exp(2S_{ij} - \operatorname{lse}_k S_{ik} - \operatorname{lse}_k S_{kj})$ , within $10^{-12}$ .
E2: the uniform limit returns the coarse position, within $10^{-9}$ pixels.
E3 and D1: the coarse-only EPE and the ground-truth counts equal values computed in advance from the homography alone: 3.0461 and 3.0453 pixels, and 2,976 and 3,395 pairs (3,105, 3,552, 3,220 and 3,675 for the base pairs below).
E4: $\tau_f = 10^{-5}$ reproduces the argmax, within $10^{-3}$ pixels.
B1: under a whole-cell shift of $(16, 8)$ pixels, the mutual nearest neighbours of the cosine matrix, written MNN( $S$ ), and the full coarse rule at $\tau_c = 0.005$ each recover at least 0.99 of the ground-truth pairs, and the correct matches have a coarse EPE of exactly 0.
B2: under a shift of $(6, 2)$ pixels, which falls on the fine grid, the argmax limit finds the true position exactly for at least 0.99 of matches, with a mean EPE of at most 0.01 pixels, against a mean coarse-only EPE of 2.8284 pixels.

Before the run, our plan stated these hypotheses for every pair, with the noisy conditions counting only if they held for each seed:

H1a: the number of selected coarse matches does not increase with $\tau_c$ .
H1b: precision at the largest $\tau_c$ with at least 20 selected matches is at least that at $\tau_c = 0.005$ .
H1c: at $\tau_c = 0.005$ the selected matches are exactly MNN( $S$ ). The plan added, in its words, “selected ⊆ MNN by construction”.
H2: at $\tau_f = 1/\sqrt{128}$ the refined EPE is below the coarse-only EPE.
H3: the smallest EPE over the $\tau_f$ sweep is at a finite positive $\tau_f$ and is below the argmax limit.
H4: above that $\tau_f$ , the EPE rises steadily to the coarse-only value.
H5: in the default pipeline, refinement lowers the mean EPE of the correct selected matches.

The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2, and runs in about 15 seconds.

import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"; os.environ["OMP_NUM_THREADS"] = "1"; os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np
import skimage
from skimage import io, color, filters, transform
from scipy.special import logsumexp

DATA = os.path.join(os.path.dirname(skimage.__file__), "data")
C, F, W, BORDER, THETA = 8, 2, 5, 2, 0.2   # coarse stride, fine stride, window, border cells, threshold
TAU_C = [0.005, 0.01, 0.02, 0.05, 0.1, 0.2]
TAU_F = [0.0, 0.005, 0.01, 0.02, 0.05, 1 / np.sqrt(128), 0.2, 0.5, 1.0, np.inf]

def load(name):
    return color.rgb2gray(io.imread(os.path.join(DATA, name)))

def homography(h, w):
    """Rotation 4 deg and scale 1.04 about the centre, shift (11.3, -6.6), mild perspective."""
    cx, cy, a, s = w / 2, h / 2, np.deg2rad(4.0), 1.04
    T1 = np.array([[1, 0, -cx], [0, 1, -cy], [0, 0, 1.0]])
    RS = np.array([[s * np.cos(a), -s * np.sin(a), 0], [s * np.sin(a), s * np.cos(a), 0], [0, 0, 1.0]])
    T2 = np.array([[1, 0, cx + 11.3], [0, 1, cy - 6.6], [0, 0, 1.0]])
    P = np.array([[1, 0, 0], [0, 1, 0], [2e-5, -1.5e-5, 1.0]])
    return T2 @ RS @ T1 @ P

def shift(tx, ty):
    return np.array([[1, 0, tx], [0, 1, ty], [0, 0, 1.0]])

def apply(H, xy):
    p = np.c_[xy, np.ones(len(xy))] @ H.T
    return p[:, :2] / p[:, 2:]

def unit(v):
    v = v - v.mean(axis=-1, keepdims=True)
    return v / (np.linalg.norm(v, axis=-1, keepdims=True) + 1e-12)

def coarse_desc(img):
    """Cell k (pixel 8k): 25x25 patch of the sigma-2 image sampled every 3 px, normalized."""
    g = np.pad(filters.gaussian(img, 2.0), 12, mode="edge")
    hc, wc = img.shape[0] // C, img.shape[1] // C
    ys, xs = np.mgrid[0:hc, 0:wc] * C
    oy, ox = np.meshgrid(np.arange(-12, 13, 3), np.arange(-12, 13, 3), indexing="ij")
    return unit(g[ys.ravel()[:, None] + oy.ravel() + 12, xs.ravel()[:, None] + ox.ravel() + 12])

def fine_desc(fmap, u, v):
    """5x5 patch of the half-resolution map at fine index (u, v), normalized."""
    p = np.pad(fmap, 2 + W, mode="edge")
    oy, ox = np.meshgrid(np.arange(-2, 3), np.arange(-2, 3), indexing="ij")
    return unit(p[v[:, None] + oy.ravel() + 2 + W, u[:, None] + ox.ravel() + 2 + W])

def interior(idx, hc, wc):
    y, x = idx // wc, idx % wc
    return (y >= BORDER) & (y < hc - BORDER) & (x >= BORDER) & (x < wc - BORDER)

def ground_truth(H, hc, wc):
    """LoFTR's coarse labels: warp grid points, round to a cell, keep mutual pairs."""
    pts = np.c_[np.arange(hc * wc) % wc, np.arange(hc * wc) // wc] * C
    r01 = np.round(apply(H, pts) / C).astype(int)
    r10 = np.round(apply(np.linalg.inv(H), pts) / C).astype(int)
    inb = lambda r: (r[:, 0] >= 0) & (r[:, 0] < wc) & (r[:, 1] >= 0) & (r[:, 1] < hc)
    n1 = np.where(inb(r01), r01[:, 0] + r01[:, 1] * wc, -1)
    n0 = np.where(inb(r10), r10[:, 0] + r10[:, 1] * wc, -1)
    i = np.arange(hc * wc)
    ok = n1 >= 0
    ok[ok] &= n0[n1[ok]] == i[ok]
    ok &= interior(i, hc, wc)
    ok[ok] &= interior(n1[ok], hc, wc)
    return dict(zip(i[ok], n1[ok]))

def dual_softmax(cos, tau):
    S = cos / tau
    R = np.exp(S - S.max(1, keepdims=True)); R /= R.sum(1, keepdims=True)
    K = np.exp(S - S.max(0, keepdims=True)); K /= K.sum(0, keepdims=True)
    return R * K

def select(P, hc, wc, theta):
    """Threshold, remove border cells, then mutual maximum of P (coarse_matching.py)."""
    keep = interior(np.arange(P.shape[0]), hc, wc)
    i = np.arange(P.shape[0]); j = P.argmax(1)
    ok = (P[i, j] >= theta) & (P.argmax(0)[j] == i) & keep & keep[j]
    return dict(zip(i[ok], j[ok]))

def refine(fa, fb, pairs, wc, tau):
    """Expectation over the 5x5 window of B around each coarse match, in pixels."""
    i = np.array([p[0] for p in pairs]); j = np.array([p[1] for p in pairs])
    ia = np.c_[i % wc, i // wc] * (C // F); jb = np.c_[j % wc, j // wc] * (C // F)
    q = fine_desc(fa, ia[:, 0], ia[:, 1])
    dy, dx = np.meshgrid(np.arange(-2, 3), np.arange(-2, 3), indexing="ij")
    d = np.c_[dx.ravel(), dy.ravel()]
    cand = np.stack([fine_desc(fb, jb[:, 0] + a, jb[:, 1] + b) for a, b in d], 1)
    sim = np.einsum("mc,mkc->mk", q, cand)
    if tau == 0:                         # argmax limit
        wgt = np.eye(W * W)[sim.argmax(1)]
    elif np.isinf(tau):                  # uniform limit
        wgt = np.full(sim.shape, 1.0 / (W * W))
    else:
        z = sim / tau; wgt = np.exp(z - z.max(1, keepdims=True)); wgt /= wgt.sum(1, keepdims=True)
    return jb * F + F * (wgt @ d)

def epe(H, pairs, wc, pos):
    i = np.array([p[0] for p in pairs])
    truth = apply(H, np.c_[i % wc, i // wc] * C)
    return np.linalg.norm(pos - truth, axis=1)

def make_pair(name, H, noise_seed):
    A = load(name)
    B = transform.warp(A, transform.ProjectiveTransform(np.linalg.inv(H)), order=1, cval=0.0)
    if noise_seed is not None:
        B = B + np.random.default_rng(noise_seed).normal(0, 0.03, B.shape)
    return A, B

def coarse_pos(pairs, wc):
    j = np.array([p[1] for p in pairs])
    return np.c_[j % wc, j // wc].astype(float) * C

conds = [(n, s) for n in ["coffee.png", "rocket.jpg"] for s in [None, 0, 1, 2]]
labels = [f"{n[:6]}{'' if s is None else '/s' + str(s)}" for n, s in conds]
coarse_rows, fine_rows, pipe_rows, checks = {}, {}, [], {"E1": 0.0, "E2": 0.0, "E4": 0.0}
for (name, seed), lab in zip(conds, labels):
    A, B = make_pair(name, homography(*load(name).shape), seed)
    H = homography(*A.shape)
    hc, wc = A.shape[0] // C, A.shape[1] // C
    gt = ground_truth(H, hc, wc)
    cos = coarse_desc(A) @ coarse_desc(B).T
    for tau in TAU_C:
        P = dual_softmax(cos, tau)
        if tau == 0.1:
            S = cos / tau
            Plog = np.exp(2 * S - logsumexp(S, 1)[:, None] - logsumexp(S, 0)[None, :])
            checks["E1"] = max(checks["E1"], np.abs(P - Plog).max())
        sel = select(P, hc, wc, THETA)
        correct = [(i, j) for i, j in sel.items() if gt.get(i) == j]
        coarse_rows.setdefault(f"{tau:g}", []).append(f"{len(sel)}/{len(correct)}")
        if tau == 0.1:
            default_correct = correct
    mnn = select(cos, hc, wc, -np.inf)  # mutual maximum of S itself
    coarse_rows.setdefault("MNN(S)", []).append(f"{len(mnn)}/{sum(gt.get(i) == j for i, j in mnn.items())}")
    fa = filters.gaussian(A, 1.0)[::F, ::F]; fb = filters.gaussian(B, 1.0)[::F, ::F]
    pairs = sorted(gt.items())
    cpos = coarse_pos(pairs, wc)
    fine_rows.setdefault("coarse only", []).append(epe(H, pairs, wc, cpos).mean())
    for tau in TAU_F:
        pos = refine(fa, fb, pairs, wc, tau)
        key = "argmax" if tau == 0 else ("uniform" if np.isinf(tau) else f"{tau:.4g}")
        fine_rows.setdefault(key, []).append(epe(H, pairs, wc, pos).mean())
        if np.isinf(tau):
            checks["E2"] = max(checks["E2"], np.abs(pos - cpos).max())
        if tau == 0:
            pos0 = pos
    checks["E4"] = max(checks["E4"], np.abs(refine(fa, fb, pairs, wc, 1e-5) - pos0).max())
    if seed is None:
        checks[f"E3 {name[:6]}"] = epe(H, pairs, wc, cpos).mean()
        checks[f"D1 {name[:6]}"] = len(gt)
    if default_correct:
        pc = coarse_pos(default_correct, wc)
        pf = refine(fa, fb, default_correct, wc, 1 / np.sqrt(128))
        pipe_rows.append(f"{lab:10s} {len(default_correct):5d} {epe(H, default_correct, wc, pc).mean():9.4f} {epe(H, default_correct, wc, pf).mean():9.4f}")
    else:
        pipe_rows.append(f"{lab:10s} {0:5d}       n/a       n/a")

# Base checks: B1 coarse stage under a whole-cell shift, B2 fine stage under a fine-grid shift.
base = []
for name in ["coffee.png", "rocket.jpg"]:
    A, B = make_pair(name, shift(16, 8), None)
    hc, wc = A.shape[0] // C, A.shape[1] // C
    gt = ground_truth(shift(16, 8), hc, wc)
    cos = coarse_desc(A) @ coarse_desc(B).T
    mnn = select(cos, hc, wc, -np.inf)
    sel = select(dual_softmax(cos, 0.005), hc, wc, THETA)
    good = [(i, j) for i, j in sel.items() if gt.get(i) == j]
    e = epe(shift(16, 8), good, wc, coarse_pos(good, wc)).max() if good else np.nan
    base.append(f"B1 {name[:6]}: GT {len(gt)}  in MNN(S) {sum(gt.get(i) == j for i, j in mnn.items()) / len(gt):.4f}"
                f"  selected at tau_c=0.005 {len(good) / len(gt):.4f}  max coarse EPE of correct {e:.1e}")
    A, B = make_pair(name, shift(6, 2), None)
    gt = ground_truth(shift(6, 2), hc, wc)
    pairs = sorted(gt.items())
    fa = filters.gaussian(A, 1.0)[::F, ::F]; fb = filters.gaussian(B, 1.0)[::F, ::F]
    e0 = epe(shift(6, 2), pairs, wc, refine(fa, fb, pairs, wc, 0.0))
    ec = epe(shift(6, 2), pairs, wc, coarse_pos(pairs, wc))
    base.append(f"B2 {name[:6]}: GT {len(gt)}  argmax EPE exactly 0 in {np.mean(e0 == 0):.4f}"
                f"  mean argmax EPE {e0.mean():.4f}  mean coarse EPE {ec.mean():.4f}")

print("Checks")
for k, v in checks.items():
    print(f"  {k:10s} {v:.1e}" if k[0] in "E" and not k.startswith("E3") else f"  {k:10s} {v:.4f}" if k.startswith("E3") else f"  {k:10s} {v}")
for line in base:
    print(" ", line)
print("\nCoarse matches, selected/correct (theta_c = 0.2, mutual check, 2 border cells removed)")
print(f"{'tau_c':>8s} " + " ".join(f"{l:>11s}" for l in labels))
for k, v in coarse_rows.items():
    print(f"{k:>8s} " + " ".join(f"{x:>11s}" for x in v))
print("\nMean end-point error (px) of the ground-truth coarse matches, by fine temperature tau_f")
print(f"{'tau_f':>11s} " + " ".join(f"{l:>9s}" for l in labels))
for k, v in fine_rows.items():
    print(f"{k:>11s} " + " ".join(f"{x:9.4f}" for x in v))
print("\nDefault pipeline (tau_c = 0.1, theta_c = 0.2, tau_f = 1/sqrt(128)): correct matches, mean EPE coarse, fine")
for r in pipe_rows:
    print(r)

Checks
  E1         7.8e-16
  E2         0.0e+00
  E4         2.4e+00
  E3 coffee  3.0461
  D1 coffee  2976
  E3 rocket  3.0453
  D1 rocket  3395
  B1 coffee: GT 3105  in MNN(S) 1.0000  selected at tau_c=0.005 0.9961  max coarse EPE of correct 0.0e+00
  B2 coffee: GT 3220  argmax EPE exactly 0 in 1.0000  mean argmax EPE 0.0000  mean coarse EPE 2.8284
  B1 rocket: GT 3552  in MNN(S) 0.9983  selected at tau_c=0.005 0.5512  max coarse EPE of correct 0.0e+00
  B2 rocket: GT 3675  argmax EPE exactly 0 in 1.0000  mean argmax EPE 0.0000  mean coarse EPE 2.8284

Coarse matches, selected/correct (theta_c = 0.2, mutual check, 2 border cells removed)
   tau_c      coffee   coffee/s0   coffee/s1   coffee/s2      rocket   rocket/s0   rocket/s1   rocket/s2
   0.005   1869/1315   1717/1188   1721/1193   1734/1199     931/604     788/480     788/487     799/479
    0.01   1684/1251   1550/1129   1551/1129   1562/1145     781/514     707/452     726/464     711/447
    0.02    1206/977    1131/901    1120/894    1136/911     581/405     522/365     529/367     539/372
    0.05     371/337     351/317     347/312     355/318     175/139     157/120     166/131     158/115
     0.1         0/0         0/0         0/0         0/0         1/1         1/1         1/1         1/1
     0.2         0/0         0/0         0/0         0/0         0/0         0/0         0/0         0/0
  MNN(S)   1798/1270   1660/1142   1660/1163   1665/1155   1643/1186     798/469     785/478     823/468

Mean end-point error (px) of the ground-truth coarse matches, by fine temperature tau_f
      tau_f    coffee coffee/s0 coffee/s1 coffee/s2    rocket rocket/s0 rocket/s1 rocket/s2
coarse only    3.0461    3.0461    3.0461    3.0461    3.0453    3.0453    3.0453    3.0453
     argmax    1.2104    2.2250    2.1811    2.1736    0.9997    3.5936    3.5854    3.5642
      0.005    1.0568    2.0435    2.0180    2.0353    0.8846    3.5298    3.5023    3.4820
       0.01    1.0429    1.9524    1.9351    1.9512    0.8205    3.4669    3.4366    3.4125
       0.02    1.0656    1.8624    1.8585    1.8654    0.8405    3.3509    3.3165    3.2893
       0.05    1.1729    1.7865    1.7867    1.7941    1.2223    3.0836    3.0605    3.0327
    0.08839    1.3354    1.8159    1.8168    1.8174    1.6212    2.8843    2.8727    2.8472
        0.2    1.7917    2.0662    2.0713    2.0684    2.1604    2.7273    2.7253    2.7111
        0.5    2.3719    2.4853    2.4878    2.4902    2.6046    2.7988    2.8008    2.7949
          1    2.6732    2.7295    2.7312    2.7322    2.8073    2.8948    2.8961    2.8928
    uniform    3.0461    3.0461    3.0461    3.0461    3.0453    3.0453    3.0453    3.0453

Default pipeline (tau_c = 0.1, theta_c = 0.2, tau_f = 1/sqrt(128)): correct matches, mean EPE coarse, fine
coffee         0       n/a       n/a
coffee/s0      0       n/a       n/a
coffee/s1      0       n/a       n/a
coffee/s2      0       n/a       n/a
rocket         1    0.4500    1.6541
rocket/s0      1    0.4500    2.0950
rocket/s1      1    0.4500    3.0320
rocket/s2      1    0.4500    2.8244

Checks. E1, E2, E3 and D1 pass: the dual-softmax matches its log-domain form to $7.8 \times 10^{-16}$ , the uniform limit returns the coarse position exactly, and the coarse-only EPE and the ground-truth counts equal the values computed in advance. B2 passes for both images. Under a fine-grid shift, the argmax limit finds the true position for every ground-truth match, where the coarse cells alone are 2.8284 pixels off on average.

E4 failed. At $\tau_f = 10^{-5}$ the refined positions differ from the argmax by up to 2.4 pixels on one axis, against a tolerance of $10^{-3}$ . The following derivation is ours. If the best candidate in a window has cosine $s_1$ and another candidate has $s_1 - g$ , the softmax gives the second a weight $e^{-g/\tau_f}$ times that of the first, so the expectation moves towards it by a share of their separation. Offsets in a window differ by at most 8 pixels on an axis, so a 2.4-pixel shift needs at least 0.3 of the weight off the argmax. With 24 other candidates, at least one must then hold a weight of $0.3/24$ or more, against at most 0.7 for the argmax. Its gap is therefore at most $\tau_f \ln(0.7 \cdot 24 / 0.3) \approx 4\tau_f = 4 \times 10^{-5}$ . At least one window thus had a runner-up within that distance of its best cosine, and a softmax at any finite temperature splits such near-ties. Our tolerance assumed no near-ties at that scale.

B1 coffee passed; B1 rocket failed for the full rule. For coffee, MNN( $S$ ) recovers 1.0000 of the ground-truth pairs under the whole-cell shift, and the full rule 0.9961. For rocket, MNN( $S$ ) recovers 0.9983, which passes, but the full coarse rule at $\tau_c = 0.005$ recovers only 0.5512 against the criterion of at least 0.99. So at least 0.447 of the ground-truth pairs are mutual nearest neighbours of the cosines that the full rule rejects, at its threshold or at its mutual check on $\mathcal P_c$ . In both images the correct matches have a coarse EPE of exactly 0, as planned. A derivation (ours) shows when the threshold rejects a mutual nearest neighbour. Write $r_i = \max_k \cos_{ik}$ and $m_j = \max_k \cos_{kj}$ , so that

\tau \log \mathcal P_c(i, j) = 2\cos_{ij} - \tau\operatorname{lse}_k\!\frac{\cos_{ik}}{\tau} - \tau\operatorname{lse}_k\!\frac{\cos_{kj}}{\tau}, \qquad \tau\operatorname{lse}_k\!\frac{\cos_{ik}}{\tau} = r_i + \tau \log \sum_k e^{(\cos_{ik} - r_i)/\tau}.

For a mutual best pair, $\cos_{ij} = r_i = m_j$ , and $\mathcal P_c(i, j) = 1/(n_i n_j)$ , where $n_i \ge 1$ and $n_j \ge 1$ are the sums $\sum_k e^{(\cos_{ik} - r_i)/\tau}$ over its row and its column. The pair fails the threshold of 0.2 when $n_i n_j > 5$ : when its row and column together hold the equivalent of more than five cells within a few $\tau$ of the best cosine. In our reading, this is rocket’s sky, where many patches of smooth gradient look alike to a normalized-patch descriptor. LoFTR’s transformer is meant to remove exactly this ambiguity by making features position-dependent.

Coarse selection (H1).

H1a held in all eight pairs. Counts fall with $\tau_c$ , to 0 matches at $\tau_c = 0.2$ for every pair, and 0 at 0.1 for coffee.
H1b held in all eight pairs. At $\tau_c = 0.05$ , the largest temperature with at least 20 matches everywhere, precision is 0.9084, 0.9031, 0.8991 and 0.8958 for coffee (clean, seeds 0–2), against 0.7036, 0.6919, 0.6932 and 0.6915 at 0.005. For rocket it is 0.7943, 0.7643, 0.7892 and 0.7278, against 0.6488, 0.6091, 0.6180 and 0.5995. A higher temperature keeps fewer matches, and a larger share of them is correct.
H1c failed in all eight pairs. At $\tau_c = 0.005$ , coffee selects 1,869, 1,717, 1,721 and 1,734 matches, against 1,798, 1,660, 1,660 and 1,665 in MNN( $S$ ). Rocket selects 931, 788, 788 and 799, against 1,643, 798, 785 and 823.

The plan’s “selected ⊆ MNN by construction” was wrong, as the coffee counts show. The reason, by our derivation, is the identity above. The mutual check in LoFTR’s rule is a mutual maximum of $2\cos_{ij} - \tau\operatorname{lse}_{\text{row}\,i} - \tau\operatorname{lse}_{\text{col}\,j}$ , not of $\cos_{ij}$ . A row’s best column under $\mathcal P_c$ can be a column that is less contested, rather than the column with the highest cosine. At the same time, the threshold implies $2\cos_{ij} - r_i - m_j \ge \tau\log\theta_c$ , which is $-0.008$ at $\tau = 0.005$ . So every selected pair is within $0.008$ in cosine of being a mutual nearest neighbour, without having to be one. Only in the limit $\tau \to 0$ do both sets coincide. The dual-softmax is a soft mutual-nearest-neighbour rule, as the paper calls it, and at finite temperature it differs from the hard rule in both directions: in clean rocket it selects 712 fewer pairs than MNN( $S$ ) contains, and in clean coffee it selects 71 more, so at least 71 of its pairs are not mutual nearest neighbours of the cosines.

Refinement (H2 to H4).

H2 held in all eight pairs. At $\tau_f = 1/\sqrt{128}$ the refined EPE is 1.3354, 1.8159, 1.8168 and 1.8174 pixels for coffee, and 1.6212, 2.8843, 2.8727 and 2.8472 for rocket, against 3.0461 and 3.0453 for the coarse positions.
H3 held in all eight pairs. The minimum is at $\tau_f = 0.01$ for clean coffee (1.0429, against 1.2104 for the argmax) and clean rocket (0.8205 against 0.9997). It is at 0.05 for noisy coffee (1.7865, 1.7867 and 1.7941 against 2.2250, 2.1811 and 2.1736), and at 0.2 for noisy rocket (2.7273, 2.7253 and 2.7111 against 3.5936, 3.5854 and 3.5642).
H4 held in all eight pairs. Above each minimum, the EPE rises at every step of the sweep to the coarse-only value at the uniform limit.

The sweep shows the trade-off a soft-argmax over a window has to make. A sharp heatmap picks one candidate, so its accuracy is limited by the 2-pixel spacing of the fine grid. A flat heatmap returns the window centre, the coarse match, which is the centre bias Heatmap Regression describes. In between, the expectation interpolates between neighbouring candidates and beats the argmax. The best temperature moves up with noise, from 0.01 clean to 0.05 and 0.2: a flatter heatmap averages out noisy correlations.

The plan did not predict one result in the table. With noise, rocket’s argmax-limit EPE (3.5936, 3.5854 and 3.5642) is worse than leaving the coarse match alone (3.0453). In our reading, the noise dominates the small fine patches where the image is smooth, so the argmax lands on an arbitrary offset in the window, which on average is farther from the truth than the window centre.

Default pipeline (H5). H5 failed for rocket and is undecidable for coffee. At the code’s defaults, coffee gives 0 correct matches in all four pairs. Rocket gives one correct match per pair, whose coarse EPE is 0.4500 pixels and whose refined EPE is 1.6541, 2.0950, 3.0320 and 2.8244. One match per pair also cannot show an average effect.

The coarse collapse follows from the threshold (our reasoning). $\mathcal P_c \ge 0.2$ needs both factors to be at least 0.2, so all other cells of a row must together weigh at most four times the best one: $\sum_{k \ne j} e^{-(\cos_{ij} - \cos_{ik})/\tau} \le 4$ . With coffee’s 3,750 cells and $\tau = 0.1$ , Jensen’s inequality then requires the other cosines to lie on average at least $0.1 \ln(3{,}749/4) \approx 0.68$ below the best one. In our reading, normalized intensity patches from one image rarely do that, which is consistent with the printed counts at $\tau_c = 0.1$ . LoFTR’s trained features are optimized so that the dual-softmax concentrates on the right cell, and its threshold and temperature are set for such features. With fixed features, only temperatures well below the default keep matches, at a lower precision (H1b).

Inference

At test time, LoFTR takes two images and returns the fine matches $\mathcal M_f$ with their coarse confidences. A’s points lie on the 1/8 grid, and B’s are refined to sub-pixel positions. Because every coarse cell can match, the number of matches depends on the image pair and the threshold rather than on a keypoint budget. For HPatches, the authors capped the output at 1,000 matches (§4.1). The full dual-softmax model ran in 116 ms for a $640 \times 480$ pair on an RTX 2080Ti, and the optimal-transport model in 130 ms (§3.6). Downstream geometry uses RANSAC: OpenCV’s homography estimation for HPatches, and the essential matrix for relative pose (§4).

Performance

The authors compared LoFTR with three groups of baselines: detector-based features matched by mutual nearest neighbours (D2-Net, R2D2, DISK); SuperPoint with SuperGlue; and the detector-free Sparse-NCNet and DRC-Net. All numbers below are from the paper, as printed.

Homography estimation on HPatches (Table 1). The measure is the area under the curve (AUC) of the corner error at 3, 5 and 10 pixels, with images resized to a shorter side of 480. LoFTR-DS, trained on MegaDepth, reached 65.9, 75.6 and 84.6. SuperPoint + SuperGlue reached 53.9, 68.3 and 81.7, and DRC-Net 50.6, 56.2 and 68.3. The table’s caption cites HPatches as reference [7], which in the paper’s list is ScanNet; the text cites it correctly as [1].
Indoor relative pose on ScanNet (Table 2). On the 1,500 test pairs of SuperGlue’s evaluation, the measure is the AUC of the pose error at 5°, 10° and 20°. LoFTR-DS reached 22.06, 40.8 and 57.62, and LoFTR-OT 21.51, 40.39 and 57.96. SuperPoint + SuperGlue reached 16.16, 33.81 and 51.84. Trained on MegaDepth, LoFTR-OT reached 16.88, 33.62 and 50.62, against 7.69, 17.93 and 30.49 for DRC-Net trained on MegaDepth.
Outdoor relative pose on MegaDepth (Table 3). On 1,500 pairs from two scenes, LoFTR-DS reached 52.8, 69.19 and 81.18, and LoFTR-OT 50.31, 67.14 and 79.93. SuperPoint + SuperGlue (with HLoc’s set-up) reached 42.18, 61.16 and 75.96, and DRC-Net 27.01, 42.96 and 58.31. The text’s 61% and 13% gains at AUC@10° are relative: 69.19 against 42.96 and against 61.16.
Visual localization (Tables 4 and 5). On the Aachen Day-Night v1.1 local-feature track (night queries, within 0.25 m and 2°, 0.5 m and 5°, 1 m and 10°), LoFTR-DS scored 72.8, 88.5 and 99.0, against 73.3, 88.0 and 98.4 for SuperPoint + SuperGlue. The text says LoFTR-DS outperforms all baselines on this track, but at the tightest threshold the table puts it 0.5 below SuperPoint + SuperGlue and LISRD + SuperPoint + AdaLAM (73.3). On InLoc with HLoc, LoFTR-OT scored 47.5, 72.2 and 84.8 on DUC1 and 54.2, 74.8 and 85.5 on DUC2. HLoc + SuperPoint + SuperGlue scored 49.0, 68.7 and 80.8, and 53.4, 77.1 and 82.4. The text says LoFTR-OT outperforms all published methods on InLoc, but the table shows SuperPoint + SuperGlue higher at DUC1’s tightest threshold and at DUC2’s middle one.
Ablation on ScanNet (Table 6). These runs use optimal-transport matching, according to the text. Replacing the LoFTR module with convolutions of a comparable parameter count gave 14.98, 32.04 and 49.92. Coarse and fine resolutions of 1/16 and 1/4 gave 16.75, 34.82 and 54.0, at 104 ms. Positional encoding at every layer, as in DETR, gave 18.02, 35.64 and 52.77. Doubling the layers ( $N_c = 8$ , $N_f = 2$ ) gave 20.87, 40.23 and 57.56.

The paper’s tables disagree on the full model. Table 6’s full model ( $N_c = 4$ , $N_f = 1$ ) scores 20.06, 40.8 and 57.62. Its last two values equal LoFTR-DS in Table 2, and none of the three equals LoFTR-OT in Table 2 (21.51, 40.39, 57.96), although the text says the ablations used optimal transport.

Strengths

No repeatability requirement. Matches are not limited to points a detector fires on twice. The authors’ Fig. 1 shows LoFTR finding correspondences on a texture-less wall and a floor with repetitive patterns, where, in the caption’s words, detector-based methods struggle to find repeatable interest points; they also attribute the HPatches gain partly to the larger number of match candidates.
Context and position in the descriptor. Attention over both whole images gives each coarse feature global context and, through the positional encoding, a position. The ablation shows the largest drop when the transformer is replaced by convolutions (Table 6, row 1).
Sub-pixel output from a coarse search. The search runs at 1/8 resolution, and the expectation in the fine window recovers sub-pixel positions. The example shows that the expectation beats both the coarse cell and the argmax over the fine grid, at a suitable temperature.
Two interchangeable matching layers. Dual-softmax and optimal transport performed comparably indoors, and dual-softmax was better outdoors (§4.2).

Limitations

Coarse decisions are final. The fine stage searches only $\pm 4$ pixels around a coarse match, so a wrong coarse cell cannot be repaired. The example’s teacher-forced table measures refinement only on correct coarse matches.
Cost grows with resolution. The coarse sequence grows with the image area, and the 4,800-position sequences of a $640 \times 480$ pair already needed linear attention. Coarser grids (1/16) were faster but less accurate (Table 6).
Fixed grid in A. A’s matched points lie on the 1/8 grid, and only B’s side is refined. Matches are semi-dense rather than placed at salient points.
Thresholds tied to the learned features. In the example, the code’s defaults ( $\tau = 0.1$ , $\theta_c = 0.2$ ) kept almost no matches with fixed features. The confidence threshold presupposes features as distinctive as the trained ones.
Training data. Training needs posed images with depth (ScanNet, MegaDepth) and large compute (24 hours on 64 GPUs for ScanNet).

Historical Importance

LoFTR brought SuperGlue’s interleaved self- and cross-attention from detected keypoints to dense feature maps. In its own tables, a detector-free matcher outperformed SuperPoint + SuperGlue on HPatches homographies and on ScanNet and MegaDepth pose. Before it, NCNet-style methods regularized 4D cost volumes with local convolutions; LoFTR replaced that local consensus with attention over the whole image.

As of November 2026, the official README points to QuadTreeAttention as a more efficient attention mechanism for LoFTR, and states that LoFTR is part of the kornia library from version 0.5.11. The DUSt3R article reports that MASt3R’s authors, on the Map-free benchmark, called LoFTR with KBR depth the second-best published method, although their table also lists other methods above it.

Variants

LoFTR-DS and LoFTR-OT. The same network with the dual-softmax or the optimal-transport matching layer. Pretrained indoor and outdoor models of each were released (README, as of November 2026).
Resolution (8, 2) and (16, 4). The official configuration accepts coarse and fine strides of 8 and 2, or 16 and 4. The second is the faster variant of Table 6.
Detector-based counterpart. SuperPoint + SuperGlue keeps the detector and matches keypoint sets with attention and optimal transport. Feature matching summarizes it and LightGlue, its faster revision.

Implementations

Official repository. zju3dv/LoFTR is released under the Apache License 2.0 (as of November 2026). It contains inference and training code, with the configuration values cited above at commit df7ca80. The README links pretrained weights for the indoor and outdoor models with both matching layers, and the ScanNet and MegaDepth test sets.
kornia. According to the README, LoFTR is available as kornia.feature.LoFTR from kornia 0.5.11.