SuperPoint

SuperPoint is a fully convolutional network that finds interest points and computes a descriptor for each of them in a single forward pass over a full image. Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich of Magic Leap presented it at the CVPR 2018 Workshops. It addresses feature detection and description for multiple-view geometry: the keypoints and descriptors that feature matching, structure from motion and SLAM start from. The network has a VGG-style encoder shared by two heads, one for points and one for 256-dimensional descriptors.

SuperPoint’s main idea is its training signal. Interest points have no agreed definition that people could label, so the authors used no human labels. They first trained a detector called MagicPoint on rendered shapes whose corners are known. They then used a procedure they named Homographic Adaptation to turn that detector’s output on real, unlabelled images into pseudo-ground-truth: the detector is run on many randomly warped copies of each image, and the responses are averaged after warping them back. The descriptor head is trained on pairs of images related by a known homography. Unless stated otherwise, the details below come from the open-access CVF version of the paper, which includes the appendix (sections, equations and tables are numbered as in that copy).

Problem Addressed

In 2018, learned keypoint detectors for semantic tasks, such as human pose, could be trained from large sets of labelled 2D locations. Geometric interest points have no comparable labels, because whether a pixel is a good interest point is, in the authors’ words, “semantically ill-defined”. Earlier systems of learned local features therefore covered only part of the pipeline, as the paper’s Table 1 summarizes:

UCN and DeepDesc learn descriptors but, as the paper notes, do not detect interest points. Of the two, only UCN takes full images as input.
LIFT keeps the classical detect-then-describe recipe on patches and, as the paper describes it, needs supervision from a classical structure-from-motion system.
TILDE learns a detector only.

The table’s caption claims that SuperPoint is the only method to compute both interest points and descriptors in a single network in real time. A second gap was robustness. On the authors’ Synthetic Shapes test set (Table 2), the classical FAST, Harris and Shi–Tomasi detectors had a mean average precision of 0.405, 0.678 and 0.686 for corner detection without noise, falling to 0.061, 0.213 and 0.157 with added imaging noise. MagicPoint reached 0.979 and 0.971.

Architecture

The network takes a grayscale image of size $H \times W$ and returns two outputs: a point-probability map of size $H \times W$ and a descriptor map of size $H \times W \times D$ . Both heads read the same encoder output and upsample without learned layers, which the authors chose to keep the model fast and easy to train (Figure 3).

Shared Encoder

The encoder follows VGG: eight $3 \times 3$ convolution layers with 64, 64, 64, 64, 128, 128, 128 and 128 channels, with a $2 \times 2$ max pool after every two layers (§6). Three pooling layers reduce the image to a grid of $H_c \times W_c = H/8 \times W/8$ cells, each covering $8 \times 8$ pixels. Every convolution in the network is followed by a ReLU and batch normalization.

Interest Point Decoder

The detector head applies a $3 \times 3$ convolution with 256 channels and a $1 \times 1$ convolution with 65 channels, giving $\mathcal{X} \in \mathbb{R}^{H_c \times W_c \times 65}$ . Sixty-four channels stand for the 64 pixel positions inside a cell. The 65th is a “no interest point” dustbin. A channel-wise softmax is taken, the dustbin is dropped, and the remaining $H_c \times W_c \times 64$ tensor is reshaped into the $H \times W$ map. The paper notes that this parameter-free reshape is known as sub-pixel convolution, “depth to space” or “pixel shuffle”. It avoids upconvolution, which the authors say adds computation and can cause checkerboard artifacts. It follows from the 65-way softmax that each cell can put most of its probability on at most one pixel. This dustbin is a per-cell class; it is unrelated to the dustbin row and column that SuperGlue adds to its score matrix, as feature matching describes.

Descriptor Decoder

The descriptor head applies a $3 \times 3$ convolution with 256 channels and a $1 \times 1$ convolution with $D = 256$ channels. This gives one descriptor per cell, a semi-dense grid with one vector every 8 pixels, a model the authors describe as similar to UCN’s. The decoder then interpolates the grid bicubically to full resolution and L2-normalizes each vector. According to the paper, learning descriptors semi-densely rather than densely reduces training memory and keeps the run time tractable.

Training

Training has three stages (Figure 2): pre-training a base detector on synthetic data, self-labelling real images with Homographic Adaptation, and joint training of points and descriptors on the self-labelled images.

Synthetic Shapes and MagicPoint

Synthetic Shapes is a rendered dataset of triangles, quadrilaterals, lines, cubes, checkerboards, stars and ellipses. Interest points are placed where they are unambiguous: Y-, L- and T-junctions, the centers of small ellipses and the end points of line segments (§4.1). The images are also warped by homographies, and the data are generated on the fly, so the network never sees an example twice. MagicPoint is the SuperPoint architecture without the descriptor head, trained for 200,000 iterations on this data (§6).

Appendix B reports two sizes, MagicPointL and MagicPointS, and lists MagicPointL’s layers as 64-64-64-64-128-128-128-128-128, nine numbers against the eight convolution layers of §6. Both sizes averaged a mean average precision near 0.98 without noise (Table 5). The authors found that MagicPoint transferred to real images “reasonably well”, especially on corner-like scenes such as tables and windows. However, it was less repeatable under viewpoint changes than the classical detectors, and it missed many potential points on diverse textures. That gap motivated the next step.

Homographic Adaptation

Let $f_\theta$ be the base detector and $\mathcal{H}$ a homography. In the paper’s notation, $\mathcal{H}(I)$ is the warped image and $\mathcal{H}\mathbf{x}$ the warped points. A detector is covariant with homographies if its points move with the image:

\mathcal{H}\mathbf{x} = f_\theta(\mathcal{H}(I)), \qquad \text{equivalently} \qquad \mathbf{x} = \mathcal{H}^{-1} f_\theta(\mathcal{H}(I)).

A real detector is not exactly covariant, so different homographies give different points. Homographic Adaptation averages over a sample of $N_h$ random homographies (Eq. 10):

\hat{F}(I; f_\theta) = \frac{1}{N_h} \sum_{i=1}^{N_h} \mathcal{H}_i^{-1} f_\theta(\mathcal{H}_i(I)).

In practice the detector’s heatmap is computed on each warped copy and warped back to the original frame before averaging (Figure 5). The authors typically fix the first homography to the identity, so that $N_h = 1$ means no adaptation. Homographies are built by composing simpler transformations: translation, scale, in-plane rotation and symmetric perspective distortion, each drawn from a truncated normal within preset ranges, combined with a root center crop that helps avoid border artifacts (§5.2, Figure 6). The paper’s text does not give the ranges.

The authors justify the homography as the transformation to sample. It is exact or nearly exact for a camera rotating about its center, for distant scenes and for planar scenes, and it needs no 3D information (§5.1). On held-out MS-COCO images, they report a repeatability of .67 without adaptation, a boost of 21% with $N_h = 100$ and 22% with $N_h = 1000$ , and they conclude that more than 100 homographies add little (§5.2). In our reading of Figure 14, whose bars rise from about 0.67 to just above 0.8, these percentages are relative changes, not percentage points.

Appendix C separates within-scale aggregation, averaging over warps at one resolution, from across-scale aggregation over an image pyramid. Within a scale, rotations and translations should not create or remove points, so the authors liken within-scale aggregation to an intersection of sets and across-scale aggregation to a union. They found that within-scale aggregation had the larger effect on repeatability. The procedure can be iterated: the pseudo-labels for SuperPoint came from Homographic Adaptation with $N_h = 100$ applied to MagicPoint, and then a second time with the model trained on the first round’s labels (§6).

The paper does not present consistency under transformations as a new idea. It says that TILDE used a similar principle, and it relates its method to Honari et al.’s “equivariant landmark transform”. In our reading, what it adds is the use of homography averaging to label real images for a large fully convolutional network.

Joint Training and Losses

Pseudo-labels were generated for the 80,000 images of the MS-COCO 2014 training split, resized to $240 \times 320$ and converted to grayscale (COCO). For each training example, a homography $\mathcal{H}$ is sampled, from a more restrictive set than during adaptation (the authors avoid extreme in-plane rotations, which they say are rare in HPatches). The image and its pseudo-labels are warped to form a pair. The loss sums a detector term for each image and a weighted descriptor term (Eq. 1):

\mathcal{L}(\mathcal{X}, \mathcal{X}', \mathcal{D}, \mathcal{D}'; Y, Y', S) = \mathcal{L}_p(\mathcal{X}, Y) + \mathcal{L}_p(\mathcal{X}', Y') + \lambda \mathcal{L}_d(\mathcal{D}, \mathcal{D}', S).

The detector loss $\mathcal{L}_p$ is the mean over cells of a 65-way cross-entropy between the cell’s logits $\mathbf{x}_{hw}$ and its label $y_{hw}$ (Eqs. 2–3). The label is the position of the pseudo-ground-truth point in the cell, or, in our reading of the 65-class set-up, the dustbin if there is none; if two points fall in one cell, one is chosen at random.

The descriptor loss compares every cell of the first image with every cell of the second. Cells $(h, w)$ and $(h', w')$ correspond when the warped center $\widehat{\mathcal{H}\mathbf{p}_{hw}}$ lies within 8 pixels of $\mathbf{p}_{h'w'}$ (Eq. 4):

s_{hwh'w'} = \begin{cases} 1, & \text{if } \lVert \widehat{\mathcal{H}\mathbf{p}_{hw}} - \mathbf{p}_{h'w'} \rVert \le 8 \\ 0, & \text{otherwise.} \end{cases}

Each pair contributes a hinge loss on the dot product of the two descriptors (Eqs. 5–6), averaged over all $(H_c W_c)^2$ pairs:

l_d(\mathbf{d}, \mathbf{d}'; s) = \lambda_d \, s \max(0, m_p - \mathbf{d}^\top \mathbf{d}') + (1 - s) \max(0, \mathbf{d}^\top \mathbf{d}' - m_n).

Negative pairs vastly outnumber positive ones, so the weight $\lambda_d = 250$ balances them. The margins are $m_p = 1$ and $m_n = 0.2$ , and the two losses are balanced with $\lambda = 0.0001$ . Training used PyTorch with mini-batches of 32 and Adam (learning rate 0.001, $\beta = (0.9, 0.999)$ ), with random Gaussian noise, motion blur and brightness changes as augmentation (§6). The paper calls its approach self-supervised, and more specifically self-training: the detector supervises itself through its own averaged outputs. In our reading, positives and negatives defined by a known warp make the descriptor loss a pair-based metric-learning set-up.

Homographic Adaptation with a Harris Base Detector

The example trains no network. The averaging in Eq. 10 works with any base detector, so it can be tested with a classical one. The script below uses the Harris corner detector as $f_\theta$ (scikit-image’s corner_harris, $k = 0.05$ , $\sigma = 1$ ). It runs on three images bundled with scikit-image (camera, astronaut and coffee, in grayscale), each halved in size to $256 \times 256$ , $256 \times 256$ and $200 \times 300$ , close to the $240 \times 320$ the paper used for repeatability.

Homographies come from our own sampler, not the authors’. Scale is $e^z$ with $z \sim \mathcal{N}(0, 0.15^2)$ , rotation has a 15° standard deviation, translation 4% of the image size, and perspective $0.15$ divided by the image size. Each is drawn from a normal truncated at two standard deviations, and the transformations are composed about the image center. The same family generates the adaptation warps and ten held-out test homographies per image.

Conventions:

Points are $(x, y)$ = (column, row).
$H$ maps a source point to the target image, and the warped image is $H(I)(\mathbf{x}') = I(H^{-1}\mathbf{x}')$ , with bilinear interpolation and zeros outside.
Pixels within 8 pixels of the frame or of zero-filled areas are excluded, so that Harris does not fire on the borders a warp creates.

The adapted map averages the unwarped Harris maps over the first $N_h$ warps, the first being the identity, for $N_h \in \{1, 3, 10, 30, 100\}$ . Each pixel is averaged over the warps in which it was valid. Harris responses are unbounded, unlike MagicPoint’s probabilities, so the code prints two aggregation choices: “raw” averages the responses, and “norm” divides each warp’s map by its maximum first.

The 300 strongest local maxima in a $9 \times 9$ window are kept, matching the paper’s 300 points and its NMS setting of 4, which we read as a radius in pixels. Each test pair is an image and its warp by a held-out homography. Each image of the pair gets its own independent adaptation warps, as two photographs would.

Repeatability is the paper’s measure (Appendix A, Eqs. 13–14). A point is correct if a point of the other image lies within $\varepsilon$ of it after mapping, and

\text{Rep} = \frac{1}{N_1 + N_2}\left(\sum_i \text{Corr}(\mathbf{x}_i) + \sum_j \text{Corr}(\mathbf{x}_j)\right),

counted over the points that both images see, with distances measured in the second image. It is printed at the paper’s $\varepsilon = 3$ pixels and at 1.5 pixels, the distance Mikolajczyk and Schmid used. The feature detection article defines their repeatability score, which divides the number of correspondences by the smaller of the two point counts instead of averaging over both images. This example uses Eq. 14 at both distances. A row of 300 random points per image gives the chance level.

The script also prints three checks per image:

the largest error between a warped Gaussian blob’s centroid and $H\mathbf{x}$ , which confirms the point and image conventions agree;
the smallest fraction of the image that the test warps keep in view;
the repeatability of an image against itself.

Before the run, our plan stated four hypotheses:

H1: for every image and both choices, mean repeatability at 3 pixels is higher with $N_h = 100$ than with $N_h = 1$ . We expected the gain to be small because Harris is nearly rotation-invariant.
H1b: the same at 1.5 pixels.
H2: at 3 pixels and averaged over the images, the gain from $N_h = 1$ to 10 exceeds the gain from 30 to 100 (diminishing returns).
H3: the gain from 1 to 100 is smaller at 1.5 pixels than at 3, because averaging unwarped maps might blur localization.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"
import sys
import numpy as np
from scipy import ndimage, stats
from skimage import color, data, feature, transform

# Homographic adaptation (DeTone et al., 2018, Eq. 10) with Harris as the base detector.
# Coordinates: points are (x, y) = (column, row); a homography H maps (x, y, 1) of the
# source image to the target image, and the warped image is H(I)(x') = I(H^-1 x').
T_TEST = int(sys.argv[1]) if len(sys.argv) > 1 else 10   # held-out homographies per image
NH_LIST = [1, 3, 10, 30, 100]   # number of adaptation warps; the first warp is the identity
K, NMS, MARGIN = 300, 4, 8      # points kept, NMS radius (px), distance kept from invalid pixels
EPS = (3.0, 1.5)                # SuperPoint's threshold and Mikolajczyk and Schmid's


def load_images():
    ims = {"camera": data.camera() / 255.0,
           "astronaut": color.rgb2gray(data.astronaut()),
           "coffee": color.rgb2gray(data.coffee())}
    return {k: transform.rescale(v, 0.5, anti_aliasing=True) for k, v in ims.items()}


def sample_homography(rng, shape):
    """Scale, in-plane rotation, translation and perspective about the image center,
    each drawn from a normal truncated at two standard deviations."""
    h, w = shape
    tn = lambda s: stats.truncnorm.rvs(-2, 2, scale=s, random_state=rng)
    s, th = np.exp(tn(0.15)), np.deg2rad(tn(15.0))
    tx, ty, px, py = tn(0.04 * w), tn(0.04 * h), tn(0.15 / w), tn(0.15 / h)
    C = np.array([[1, 0, -(w - 1) / 2], [0, 1, -(h - 1) / 2], [0, 0, 1.0]])
    S = np.diag([s, s, 1.0])
    R = np.array([[np.cos(th), -np.sin(th), 0], [np.sin(th), np.cos(th), 0], [0, 0, 1.0]])
    Tr = np.array([[1, 0, tx], [0, 1, ty], [0, 0, 1.0]])
    P = np.array([[1, 0, 0], [0, 1, 0], [px, py, 1.0]])
    H = np.linalg.inv(C) @ Tr @ P @ R @ S @ C
    return H / H[2, 2]


def warp(img, H, order=1):
    """H(I): resample img so that a source pixel at x lands at H x."""
    return transform.warp(img, transform.ProjectiveTransform(matrix=np.linalg.inv(H)),
                          order=order, mode="constant", cval=0.0)


def map_points(H, pts):
    q = np.c_[pts, np.ones(len(pts))] @ H.T
    return q[:, :2] / q[:, 2:]


def safe_region(valid):
    """Pixels at least MARGIN px from any invalid pixel and from the frame edge."""
    v = valid.copy()
    v[0, :] = v[-1, :] = v[:, 0] = v[:, -1] = False
    return ndimage.distance_transform_edt(v) > MARGIN


def harris(img):
    return feature.corner_harris(img, method="k", k=0.05, sigma=1.0)


def adapted_maps(img, valid, Hs):
    """Average of unwarped Harris maps over the first N_h warps, for every N_h in NH_LIST.
    'raw' averages the responses; 'norm' divides each warp's map by its maximum first."""
    acc = {"raw": np.zeros(img.shape), "norm": np.zeros(img.shape)}
    cnt = np.zeros(img.shape)
    out = {}
    for i, H in enumerate(Hs, start=1):
        w_img = warp(img, H)
        ok = safe_region(warp(valid.astype(float), H, order=0) > 0.5)
        R = np.where(ok, harris(w_img), 0.0)
        back = transform.warp(R, transform.ProjectiveTransform(matrix=H), order=1,
                              mode="constant", cval=0.0, clip=False)
        back_ok = transform.warp(ok.astype(float), transform.ProjectiveTransform(matrix=H),
                                 order=1, mode="constant", cval=0.0) > 0.999
        acc["raw"] += np.where(back_ok, back, 0.0)
        acc["norm"] += np.where(back_ok, back / R[ok].max(), 0.0)
        cnt += back_ok
        if i in NH_LIST:
            out[i] = {c: np.where(cnt > 0, a / np.maximum(cnt, 1), -np.inf) for c, a in acc.items()}
    return out


def detect(F, region):
    """Top-K local maxima (NMS radius 4) with a positive response inside region, as (x, y)."""
    peak = (F == ndimage.maximum_filter(F, size=2 * NMS + 1)) & region & (F > 0)
    rc = np.argwhere(peak)
    order = np.argsort(-F[peak], kind="stable")[:K]
    return rc[order][:, ::-1].astype(float)


def inside(pts, region):
    r, c = np.round(pts[:, 1]).astype(int), np.round(pts[:, 0]).astype(int)
    ok = (r >= 0) & (r < region.shape[0]) & (c >= 0) & (c < region.shape[1])
    out = np.zeros(len(pts), bool)
    out[ok] = region[r[ok], c[ok]]
    return out


def repeatability(p1, p2, H, reg1, reg2, eps):
    """DeTone et al. (2018), Eq. 13-14, on the points that both images see.
    Distances are measured in the second image's frame."""
    q1 = map_points(H, p1)
    k1, k2 = inside(q1, reg2), inside(map_points(np.linalg.inv(H), p2), reg1)
    a, b = q1[k1], p2[k2]
    if len(a) == 0 or len(b) == 0:
        return 0.0, 0
    d = np.linalg.norm(a[:, None, :] - b[None, :, :], axis=2)
    return ((d.min(1) <= eps).sum() + (d.min(0) <= eps).sum()) / (len(a) + len(b)), (len(a) + len(b)) / 2


def blob_check(shape, H):
    """Warp a Gaussian blob at a known sub-pixel point; compare its centroid with H x."""
    h, w = shape
    x0 = np.array([(w - 1) / 2 + 3.3, (h - 1) / 2 - 2.7])
    yy, xx = np.mgrid[0:h, 0:w]
    blob = np.exp(-((xx - x0[0]) ** 2 + (yy - x0[1]) ** 2) / (2 * 2.0 ** 2))
    wb = warp(blob, H)
    target = map_points(H, x0[None])[0]
    win = np.hypot(xx - target[0], yy - target[1]) < 12
    cen = np.array([(xx * wb * win).sum(), (yy * wb * win).sum()]) / (wb * win).sum()
    return np.linalg.norm(cen - target)


imgs = load_images()
print(f"T_TEST={T_TEST}, K={K}, NMS={NMS}, MARGIN={MARGIN}, N_h={NH_LIST}")
results = {}
for gi, (name, img) in enumerate(imgs.items()):
    full = np.ones(img.shape, bool)
    reg1 = safe_region(full)
    trng = np.random.default_rng([1, gi])
    tests = [sample_homography(trng, img.shape) for _ in range(T_TEST)]

    # Data checks: blob centroid error, shared area, identity pair at N_h = 1.
    blob_err = max(blob_check(img.shape, H) for H in tests)
    shared = [inside(map_points(H, np.argwhere(reg1)[:, ::-1].astype(float)), safe_region(
        warp(full.astype(float), H, order=0) > 0.5)).mean() for H in tests]
    p_id = detect(adapted_maps(img, full, [np.eye(3)])[1]["raw"], reg1)
    rep_id = [repeatability(p_id, p_id, np.eye(3), reg1, reg1, e)[0] for e in EPS]
    print(f"{name}: {img.shape}, max blob centroid error {blob_err:.4f} px, "
          f"shared fraction min {min(shared):.3f}, identity rep {rep_id[0]:.4f} / {rep_id[1]:.4f}")

    # Adapted detections on the original image and on each held-out warp.
    def adapt_hs(*key):
        rng = np.random.default_rng([0, gi, *key])
        return [np.eye(3)] + [sample_homography(rng, img.shape) for _ in range(max(NH_LIST) - 1)]

    maps1 = adapted_maps(img, full, adapt_hs(0))
    for t, Ht in enumerate(tests):
        img2 = warp(img, Ht)
        valid2 = warp(full.astype(float), Ht, order=0) > 0.5
        reg2 = safe_region(valid2)
        maps2 = adapted_maps(img2, valid2, adapt_hs(1, t))
        for nh in NH_LIST:
            for ch in ("raw", "norm"):
                p1, p2 = detect(maps1[nh][ch], reg1), detect(maps2[nh][ch], reg2)
                for e in EPS:
                    r, n = repeatability(p1, p2, Ht, reg1, reg2, e)
                    results.setdefault((name, ch, nh, e), []).append((r, n))
        rrng = np.random.default_rng([2, gi, t])
        rp1 = np.argwhere(reg1)[rrng.choice(reg1.sum(), K, replace=False)][:, ::-1].astype(float)
        rp2 = np.argwhere(reg2)[rrng.choice(reg2.sum(), K, replace=False)][:, ::-1].astype(float)
        for e in EPS:
            results.setdefault((name, "random", 0, e), []).append(
                repeatability(rp1, rp2, Ht, reg1, reg2, e))

print()
print(f"repeatability over {T_TEST} held-out homographies per image: mean [min-max]")
print(f"{'image':<10}{'maps':<7}{'N_h':>4}   {'eps = 3 px':<22}{'eps = 1.5 px':<22}{'points':>6}")
for name in imgs:
    for ch in ("raw", "norm", "random"):
        for nh in ([0] if ch == "random" else NH_LIST):
            row = []
            for e in EPS:
                r = np.array([v[0] for v in results[(name, ch, nh, e)]])
                row.append(f"{r.mean():.4f} [{r.min():.3f}-{r.max():.3f}]  ")
            npts = np.mean([v[1] for v in results[(name, ch, nh, EPS[0])]])
            print(f"{name:<10}{ch:<7}{('-' if nh == 0 else nh):>4}   {row[0]}{row[1]}{npts:6.1f}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; two runs printed identical output; the script ran in about 25 seconds on a shared machine):

T_TEST=10, K=300, NMS=4, MARGIN=8, N_h=[1, 3, 10, 30, 100]
camera: (256, 256), max blob centroid error 0.0297 px, shared fraction min 0.648, identity rep 1.0000 / 1.0000
astronaut: (256, 256), max blob centroid error 0.0395 px, shared fraction min 0.591, identity rep 1.0000 / 1.0000
coffee: (200, 300), max blob centroid error 0.0222 px, shared fraction min 0.709, identity rep 1.0000 / 1.0000

repeatability over 10 held-out homographies per image: mean [min-max]
image     maps    N_h   eps = 3 px            eps = 1.5 px          points
camera    raw       1   0.7282 [0.602-0.842]  0.6669 [0.526-0.782]   284.2
camera    raw       3   0.6875 [0.573-0.817]  0.6083 [0.453-0.750]   279.2
camera    raw      10   0.7213 [0.609-0.842]  0.6534 [0.538-0.799]   278.1
camera    raw      30   0.7333 [0.646-0.833]  0.6771 [0.561-0.794]   274.6
camera    raw     100   0.7526 [0.672-0.849]  0.7061 [0.601-0.821]   275.6
camera    norm      1   0.7282 [0.602-0.842]  0.6669 [0.526-0.782]   284.2
camera    norm      3   0.6930 [0.583-0.809]  0.6105 [0.462-0.739]   279.6
camera    norm     10   0.7230 [0.608-0.845]  0.6566 [0.546-0.816]   278.9
camera    norm     30   0.7360 [0.647-0.830]  0.6778 [0.563-0.791]   276.4
camera    norm    100   0.7469 [0.654-0.845]  0.7040 [0.590-0.813]   277.0
camera    random    -   0.1519 [0.086-0.216]  0.0380 [0.020-0.072]   281.6
astronaut raw       1   0.7561 [0.669-0.841]  0.7067 [0.582-0.809]   266.8
astronaut raw       3   0.7258 [0.654-0.812]  0.6528 [0.555-0.769]   261.6
astronaut raw      10   0.7737 [0.698-0.816]  0.7252 [0.648-0.775]   258.3
astronaut raw      30   0.7913 [0.695-0.848]  0.7395 [0.661-0.796]   257.2
astronaut raw     100   0.7987 [0.724-0.867]  0.7493 [0.671-0.823]   256.9
astronaut norm      1   0.7561 [0.669-0.841]  0.7067 [0.582-0.809]   266.8
astronaut norm      3   0.7217 [0.647-0.803]  0.6325 [0.521-0.737]   261.7
astronaut norm     10   0.7798 [0.689-0.830]  0.7276 [0.633-0.780]   260.2
astronaut norm     30   0.7880 [0.699-0.833]  0.7377 [0.660-0.782]   258.7
astronaut norm    100   0.7950 [0.712-0.841]  0.7456 [0.661-0.805]   259.7
astronaut random    -   0.1515 [0.091-0.212]  0.0400 [0.020-0.062]   277.9
coffee    raw       1   0.7543 [0.705-0.817]  0.7060 [0.676-0.778]   264.4
coffee    raw       3   0.7745 [0.718-0.846]  0.7223 [0.627-0.794]   256.1
coffee    raw      10   0.8079 [0.772-0.863]  0.7623 [0.704-0.825]   251.7
coffee    raw      30   0.8073 [0.733-0.884]  0.7635 [0.674-0.855]   248.9
coffee    raw     100   0.8097 [0.745-0.874]  0.7626 [0.694-0.832]   243.8
coffee    norm      1   0.7543 [0.705-0.817]  0.7060 [0.676-0.778]   264.4
coffee    norm      3   0.7754 [0.711-0.845]  0.7154 [0.611-0.793]   256.5
coffee    norm     10   0.8113 [0.775-0.871]  0.7662 [0.713-0.834]   252.9
coffee    norm     30   0.8082 [0.742-0.891]  0.7645 [0.682-0.862]   247.9
coffee    norm    100   0.8101 [0.744-0.886]  0.7651 [0.684-0.839]   243.4
coffee    random    -   0.1692 [0.126-0.224]  0.0440 [0.024-0.061]   281.3

The checks behave as planned:

The self-pair repeatability is exactly 1.
Blob centroids land within 0.04 pixels of $H\mathbf{x}$ , so the image warps and the point mapping agree.
The test warps keep at least 59% of each image’s usable area in view.
Unadapted Harris ( $N_h = 1$ ) repeats about three quarters of its points at 3 pixels, against 15–17% for random points, so the measure is far from both chance and saturation.
The raw and norm rows are identical at $N_h = 1$ , as they must be: the two maps differ by a positive factor, and detection uses only ranks and the sign.

The outcomes of the four hypotheses:

H1 held for all three images and both choices. The gains are 0.024 (raw) and 0.019 (norm) for camera, 0.043 and 0.039 for astronaut, and 0.055 and 0.056 for coffee. They are small next to the spread over test homographies: for camera the per-pair range moves from 0.602–0.842 to 0.672–0.849.
H1b held for all six cases, with gains between 0.037 and 0.059 at 1.5 pixels.
H2 held for both choices. Averaged over the images, the gain from 1 to 10 warps was 0.021 (raw) and 0.025 (norm), against 0.010 and 0.007 from 30 to 100. It does not hold image by image: for camera the mean at 10 warps is still below that at 1 warp, and most of its gain comes after 10.
H3 failed. For camera and coffee, under both choices, the gain at 1.5 pixels was larger than at 3 pixels; for camera it was 1.6 (raw) and 2.0 (norm) times as large. For astronaut, the gains computed from the printed means are the same at both distances (0.0426 raw, 0.0389 norm), so the output cannot show the smaller gain that H3 predicts, and H3 is not supported there either. Averaging did not cost localization at this scale. The adapted points repeat better at the tighter distance too.

Two patterns that were not hypothesized appear in the table:

With $N_h = 3$ , repeatability falls below the unadapted detector for camera and astronaut, under both choices and at both distances.
The number of points in the shared region is lower at $N_h = 100$ than at $N_h = 1$ for every image and choice; for coffee it falls at every step, from 264.4 to 243.8 (raw).

In our reading, the first comes from the independent warps given to the two images. With three warps, each image’s map is a different average of three noisy maps, and the two averages need not agree better than the two single maps do. With many warps, both approach the same expected map, which Eq. 10 estimates, and the gain appears. The fall in shared points means that either more of the adapted detector’s 300 points fall outside the region the other image sees, or fewer than 300 positive maxima exist. The printed output does not tell which.

The paper’s 21% relative gain was measured with MagicPoint on MS-COCO, with the authors’ sampler; the relative gains at 3 pixels here are 2.6–7.4%, with Harris and our sampler. The set-ups differ, so the numbers are not comparable. What carries over is the mechanism: averaging over warps rewards points that the base detector finds consistently across viewpoints, and, averaged over the images, the benefit shrinks as more warps are added.

Inference

At test time SuperPoint takes a grayscale image and runs one forward pass. The detector head gives a full-resolution probability map, from which points are taken after non-maximum suppression. The descriptor head gives the semi-dense descriptor grid, and descriptors are sampled at the detected points by bicubic interpolation and L2 normalization, so the dense map need not be built. The authors timed the forward pass at about 11.15 ms for a $480 \times 640$ input on a Titan X GPU with Caffe’s timing tool. Sampling descriptors at 1000 points took about 1.5 ms on a CPU, for an estimated total of about 13 ms, or 70 frames per second (§7.1). Homographic Adaptation is a training-time procedure and is not run at inference.

In the paper’s matching evaluation, descriptors were matched by nearest neighbour from the first image to the second, and the homography was estimated with OpenCV’s findHomography() with RANSAC on all matches (§7.3).

Performance

The paper evaluates on HPatches, which it describes as 116 scenes with 696 images: 57 scenes with large illumination changes and 59 with large viewpoint changes. It follows Mikolajczyk et al.’s protocol, except that detections are compared by the distance between point centers, since the network has no patches. For multi-scale methods, distances are measured at the highest-resolution scale (Appendix A).

Detector repeatability (Table 3). This was measured at $240 \times 320$ , with 300 points per image, $\varepsilon = 3$ pixels, and non-maximum suppression of 4 or 8. The detectors other than MagicPoint and SuperPoint were run through OpenCV.

Detector	Illumination, NMS 4	Illumination, NMS 8	Viewpoint, NMS 4	Viewpoint, NMS 8
SuperPoint	.652	.631	.503	.484
MagicPoint	.575	.507	.322	.260
FAST	.575	.472	.503	.404
Harris	.620	.533	.556	.461
Shi	.606	.511	.552	.453
Random	.101	.103	.100	.104

SuperPoint was the most repeatable under illumination changes and with NMS 8 on viewpoint changes. On viewpoint changes with NMS 4, however, Harris (.556) and Shi (.552) were more repeatable than SuperPoint (.503), which tied with FAST. The table’s caption calls SuperPoint “competitive” on viewpoint changes, and the text says that Homographic Adaptation gave a large boost “especially under large viewpoint changes”. That boost is relative to MagicPoint (.322 to .503); the table does not show SuperPoint beating the classical detectors there.

Homography estimation and matching (Table 4). This was measured at $480 \times 640$ , with at most 1000 points per system and $\varepsilon = 3$ pixels for the detector and descriptor metrics. LIFT used its authors’ pre-trained model (named Picadilly in the paper), and SIFT and ORB OpenCV’s defaults. A homography counts as correct when the four image corners, mapped by the estimated and true homographies, differ on average by at most $\varepsilon$ (Eq. 15).

Method	Correct, $\varepsilon = 1$	$\varepsilon = 3$	$\varepsilon = 5$	Rep.	MLE	NN mAP	M. Score
SuperPoint	.310	.684	.829	.581	1.158	.821	.470
LIFT	.284	.598	.717	.449	1.102	.664	.315
SIFT	.424	.676	.759	.495	0.833	.694	.313
ORB	.150	.395	.538	.641	1.157	.735	.266

SuperPoint had the highest share of correct homographies at 3 and 5 pixels and the best descriptor metrics: nearest-neighbour mean average precision (NN mAP) and matching score (M. Score). SIFT was best at 1 pixel and had the lowest mean localization error (MLE), which the authors attribute to its sub-pixel refinement. ORB had the highest repeatability but the worst homography accuracy. The authors explain this by its detections clustering in parts of the image, and conclude that optimizing repeatability alone does not give better matching, a point close to the one feature detection makes about repeatability versus matchability. The paper summarizes SuperPoint as performing “comparably to SIFT” for homography estimation.

Strengths

One network, one pass. Points and descriptors share the encoder’s computation, and both heads upsample without learned layers. The authors estimated 70 frames per second at $480 \times 640$ on a 2018 GPU.
No manual labels. Synthetic shapes provide the initial supervision, and Homographic Adaptation extends it to any unlabelled image collection, which the paper frames as synthetic-to-real domain adaptation.
Noise robustness. On Synthetic Shapes with noise, MagicPoint’s mean average precision (0.971) was far above that of FAST, Harris and Shi–Tomasi (Table 2).
Descriptor quality. On HPatches, the learned descriptors led the nearest-neighbour mAP and matching score of the methods compared, and produced many correct matches spread over the image (Table 4, Figure 8).

Limitations

In-plane rotation. Figure 8 shows a failure of SuperPoint and LIFT under extreme in-plane rotation, which the authors attribute to such rotations being absent from training. Joint training deliberately excludes them.
Localization. Points are classified within $8 \times 8$ cells without sub-pixel refinement. SIFT’s lower localization error and better accuracy at $\varepsilon = 1$ point to this cost (Table 4).
Viewpoint repeatability. Under large viewpoint changes with NMS 4, SuperPoint was less repeatable than Harris and Shi (Table 3).
Homographies as the only model. Homographic Adaptation and the descriptor pairs use homographies, which are exact only for planar scenes, distant scenes and rotating cameras (§5.1). Occlusions and depth discontinuities are not modelled; feature detection notes the same limit of homography-based benchmarks.
Inherited bias. The pseudo-labels come from MagicPoint, so they include only the kinds of points the base detector can be made to fire on. In our reading, a point type absent from Synthetic Shapes can enter the labels only if the base detector already responds to it on some warps; Homographic Adaptation selects among responses rather than creating new ones.
Licence. The official repository is released under a licence for noncommercial research use (see Implementations), which excludes commercial use.

Historical Importance

In the comparison of the paper’s Table 1, SuperPoint was the only method that computed both interest points and descriptors in a single network in real time. LIFT, the other learned detector and descriptor in that table, worked on patches and relied on structure-from-motion supervision. The authors draw three conclusions: knowledge can be transferred from a synthetic dataset to real images, sparse detection and description can be cast as one efficient convolutional network, and the result works for geometric matching tasks such as homography estimation. They also expected such a network to serve as a learned front end for visual SLAM in robotics and augmented reality.

Later systems described on this site use SuperPoint as their local feature:

SuperGlue. The learned matcher of Sarlin, DeTone, Malisiewicz and Rabinovich (2020), which feature matching summarizes, is paired with SuperPoint in the localization results that visual localization reports.
Hierarchical localization. The pipeline in visual localization matches SuperPoint keypoints and descriptors after NetVLAD retrieval. HF-Net distils SuperPoint into the keypoint and local-descriptor heads of a single multi-task network.
SLAM benchmarks. In the TartanAir SLAM competition results that DROID-SLAM compares against, the best submission combined SuperPoint, SuperGlue and COLMAP.

Variants

MagicPoint is the detector-only base network trained on Synthetic Shapes, in large and small sizes (Appendix B). A footnote in the paper places an earlier version of these experiments in the authors’ “Toward Geometric DeepSLAM” preprint.
Iterated Homographic Adaptation. The SuperPoint models in the paper used two rounds of adaptation. The authors note that more rounds are possible, and suggest applying the technique to semantic segmentation and object detection as future work.
Distilled SuperPoint. HF-Net’s local head is trained by knowledge distillation to reproduce SuperPoint’s outputs, as visual localization describes.

Implementations

Official repository. Magic Leap’s repository magicleap/SuperPointPretrainedNetwork is released, as of November 2026, under a software licence agreement for academic and non-profit organizations that grants a licence to use the software for “noncommercial research purposes”. The agreement states that using or downloading the software counts as accepting its terms, and its definition of the software includes the documentation. We therefore read only the licence, and say nothing here about the repository’s code or contents.
hloc. The hierarchical localization toolbox described in visual localization implements that pipeline with interchangeable retrieval, features and matchers, and its README reports localization results for SuperPoint features matched by SuperGlue.