Visual Place Recognition

Visual place recognition (VPR) decides where a photograph was taken by recognizing the place it shows: given a query image, find images of the same place in a database of images whose positions are known, and report their positions. The query might be a phone photo of a street corner, a frame from a car’s dashcam or a robot’s camera returning to a corridor it mapped an hour earlier. The task is also called visual geo-localization or image geo-localization, the name Berton et al. (2022) use for the same retrieval problem, and in robotics appearance-based localization, loop-closure detection or simply “have I been here before?”. Lowry et al. (2016) put the question as whether a human, animal or robot can decide whether an image shows a place it has already seen.

Almost every modern system casts the task as image retrieval over a geotagged database, built on the local feature descriptors and the aggregation and re-ranking machinery that the image retrieval article covers for instance search. What makes place recognition its own task is what counts as a match (a position within a radius, not a labelled object), the changes a place goes through between visits (night, snow, roadworks), the repetitive look of streets, and its use inside robots, where a wrong match can corrupt a map. It is distinct from scene recognition, which classifies the kind of place (a kitchen, a street) rather than identifying this place.

Problem Definition

A database (the map or reference set) holds images $I_1, \dots, I_N$ with known positions $x_1, \dots, x_N$, usually GPS coordinates, sometimes full camera poses. A query image $q$, taken at an unknown position $x_q$, arrives; the system ranks the database and estimates $x_q$ from the positions of the best-ranked images. Arandjelović et al. (2016) state the usual formulation: an image representation $f$ maps every image to a fixed-size vector, the database vectors are computed offline, and the query is matched by nearest-neighbour search under Euclidean distance, exactly or approximately.

Relevance is geometric. In the protocol of the Pittsburgh and Tokyo benchmarks, a database image counts as correct if it lies within a distance $d$ of the query’s true position, with $d = 25$ m (Torii et al., 2013; Torii et al., 2015); Mapillary SLS adds a viewing-direction condition (Warburg et al., 2020). Two consequences follow. A photograph of a famous tower taken from far away is not a correct match for a query standing at the tower’s foot, although it shows the same object; NetVLAD’s authors treat such pairs explicitly as negatives (Arandjelović et al., 2016). And the ground truth comes from the camera’s position, not from what it sees, so two images within the radius may share no visible content if they face different ways.

What a “place” is depends on the application. Lowry et al. (2016) quote Kuipers’ two views, a place as a point or as the abstraction of a region, and note that a place, unlike a pose, has no orientation, which makes recognition regardless of heading a standing difficulty. Retrieval benchmarks sidestep the question by defining places through the radius.

Input and Output

The input is one query image (or a short sequence) and a database prepared offline: one compact vector per image for global-descriptor methods, plus local features per image when results are re-ranked. The output is a ranked list of database images, from which the system takes either the position of the first result, a shortlist of candidate places, or, in robotics, a decision “same place as keyframe $k$” or “new place” with a confidence.

Database images are usually perspective crops of street-level panoramas, for example 24 crops of 640 × 480 pixels per Google Street View panorama in Pittsburgh (Torii et al., 2013), or frames of vehicle-mounted cameras. Queries come from another time and often another camera: phone photographs in Tokyo 24/7, crowd-sourced dashcam sequences in Mapillary SLS.

Variants

• Single image and sequences. Most benchmarks match one query image against single images. Mapillary SLS defines four tasks, image-to-image, sequence-to-sequence, sequence-to-image and image-to-sequence (Warburg et al., 2020). In robotics, sequence matching such as SeqSLAM exploits the fact that a vehicle traverses the same route; Lowry et al. (2016) describe it as among the most successful approaches in changing environments.
• Place recognition and metric localization. Place recognition returns a nearby database image; visual localization estimates the full six-degree-of-freedom camera pose. Berton et al. (2022) distinguish the two and note that geo-localization methods can serve as one stage of a localization pipeline; such a pipeline typically goes on to match features against the retrieved images or a 3D model and solve perspective-n-point.
• Place recognition and landmark retrieval. Landmark retrieval finds every image of a landmark however far away it was taken, over a discrete set of landmarks; place recognition datasets cover a continuous area and count only nearby images (Berton et al., 2022).
• Loop closure. Inside SLAM, place recognition compares the current frame against the robot’s own earlier keyframes, with no external database (below).
• Other modalities. The same retrieval formulation is applied to LiDAR scans and across views, such as ground-level to aerial images; this article treats camera images.

Challenges

• Appearance change. A place changes with time of day, weather and season, and over years through construction and demolition. Torii et al. (2015) built Tokyo 24/7 around day, sunset and night queries; Nordland covers the same train route in four seasons; Mapillary SLS spans images up to seven years apart (Warburg et al., 2020).
• Viewpoint change. A query taken from the other side of the street, at another distance or another heading overlaps only partly with the database images. Torii et al. (2015) showed that matching across a large appearance change becomes much easier when query and database image are taken from about the same viewpoint, which is what their view synthesis exploits.
• Perceptual aliasing. Different places can look alike: rows of identical windows, suburban houses, forest along a railway. Lowry et al. (2016) name this problem; a system must reject matches between such places while accepting matches between very different images of one place. Torii et al. (2013) show that repeated facade elements also distort bag-of-words scores within one image.
• Distractors. Cars, people and vegetation change between visits. NetVLAD’s authors found that their trained network learned to focus on building facades and skylines and to ignore cars and people, and also, without Time Machine training, that recognizing parked cars helped because the same cars appeared in all images of a place in that data (Arandjelović et al., 2016).
• Scale. City-scale databases hold hundreds of thousands to millions of images. Berton et al. (2022) compute that ResNet-50 NetVLAD descriptors for the 1.05 million images of San Francisco would need about 256 GB of memory, which makes compression and approximate search part of the problem.
• Domain and geography. Models trained on one city or one kind of imagery transfer imperfectly. Warburg et al. (2020) found clear differences in recall across the cities of Mapillary SLS and called the trained networks geographically biased, and Berton et al. (2022) report that the same network trained on Pittsburgh or on Mapillary SLS can differ by about 30 points of Recall@1 when tested on St Lucia.

Approaches

Retrieval with local features and bags of words

Before deep learning, place recognition followed the instance-retrieval pipeline: local invariant features such as SIFT, quantized into a bag of visual words, searched through an inverted file, often re-ranked by RANSAC on tentative matches. Torii et al. (2013) studied what repetitive facades do to this model. A repeated window produces the same visual word many times, which inflates its weight in the tf-idf score; earlier “burstiness” weighting down-weighted words that occur many times in one image. They instead detected groups of repeated features in each image and soft-assigned each feature to fewer visual words the larger its repeated group, and capped the weights of repeated words. On their new Pittsburgh benchmark, the two components improved recall at $N = 10$ and $d = 25$ m by 8.92 and 10.30 points over burstiness weighting, and by 11.97 points together.

FAB-MAP: probabilistic appearance-based localization

Robotics developed its own lineage around loop closure. FAB-MAP (Cummins and Newman, 2008) is described here through Lowry et al.’s survey, because the paper itself (on the SAGE site) could not be read for this article. It represents each image as a bag of words of SIFT or SURF features and computes, in a training phase, how distinctive each word is. Rather than assuming the words independent (naive Bayes), it models their co-occurrence with a Chow–Liu tree, a tree-structured approximation of the joint distribution in which each word depends on one other. Its answer to perceptual aliasing is probabilistic: two images that share many words get a low match probability if those words are common in the world, because the normalizing term sums over all places seen so far and over a model of places not yet visited. Lowry et al. note that the successor FAB-MAP 2.0 (Cummins and Newman’s later work, not read for this article) used a 100,000-word vocabulary and an inverted index, and that the set of unvisited places was originally modelled by sampling from the Chow–Liu tree. The design reflects the robotics priority Lowry et al. describe: until recently, recall at 100% precision was the key measure, because one false loop closure could wreck a map.

Aggregated descriptors: VLAD and DenseVLAD

A bag of words needs very large vocabularies and inverted files. Jégou, Douze, Schmid and Pérez (2010) proposed the vector of locally aggregated descriptors (VLAD) as a compact alternative, which they describe as a simplification of the Fisher kernel. With $k$ visual words $c_1, \dots, c_k$ from k-means and each $d$-dimensional local descriptor $x$ assigned to its nearest word, VLAD accumulates the residuals,

$$v_{i,j} = \sum_{x:\, \mathrm{NN}(x) = c_i} \left( x_j - c_{i,j} \right), \qquad i = 1, \dots, k, \; j = 1, \dots, d,$$

a $k \times d$ vector that is then $\ell_2$-normalized. Because each word stores a $d$-dimensional sum rather than a count, small vocabularies such as $k = 64$ suffice, and the vector can be reduced by PCA and quantized for search in very large databases (the bag of visual words article gives their retrieval results). Arandjelović and Zisserman (2013) added intra-normalization: each word’s $d$-dimensional block is $\ell_2$-normalized before the whole vector, so that a few bursty words, such as those of repeated structures, cannot dominate the similarity.

Torii et al. (2015) built the strongest pre-deep place descriptor on this basis. Their DenseVLAD extracts RootSIFT descriptors on a dense grid (stride 2 pixels, four scales of 16 to 40 pixels) rather than at detected keypoints, aggregates them with a 128-word vocabulary learned from the database images into an intra-normalized VLAD, and reduces it with PCA and whitening to 4,096 dimensions. Densely sampled descriptors matched better across day–night changes than descriptors at detected keypoints, which detectors place differently under different lighting. Their second idea attacked viewpoint: using the approximate piecewise-planar depth maps that come with Street View panoramas, they synthesized virtual views on a 5 m grid, so that the database covered viewpoints close to any query (597,744 synthesized views at 49,812 virtual positions, added to 75,984 views of 6,332 panoramas). Dense descriptors plus synthesized views gave the largest gains for sunset and night queries.

NetVLAD

Arandjelović, Gronat, Torii, Pajdla and Sivic (2016) made VLAD a trainable layer of a convolutional network and trained the whole network for place recognition. The last convolutional layer (conv5 of AlexNet or VGG-16, before the ReLU) is read as an $H \times W$ grid of $D$-dimensional local descriptors. Hard assignment to the nearest centre, the only non-differentiable part of VLAD, is replaced by a softmax over centres,

$$V(j, k) = \sum_{i=1}^{N} \frac{e^{w_k^\top x_i + b_k}}{\sum_{k'} e^{w_{k'}^\top x_i + b_{k'}}} \left( x_i(j) - c_k(j) \right),$$

which follows from a soft assignment proportional to $e^{-\alpha \lVert x_i - c_k \rVert^2}$ with $w_k = 2 \alpha c_k$ and $b_k = -\alpha \lVert c_k \rVert^2$, and recovers VLAD as $\alpha \to \infty$. Training decouples $\{w_k, b_k\}$ from the anchors $\{c_k\}$, giving three sets of parameters per cluster. The output is intra-normalized, flattened and $\ell_2$-normalized; with $K = 64$ it has about 16,000 dimensions on AlexNet and 32,000 on VGG-16, optionally reduced to 4,096 by PCA with whitening.

The training data came from the Google Street View Time Machine: panoramas of the same places taken at different times. Their GPS tags give only weak supervision: images within a short distance are potential positives, which may face elsewhere, and images far away are definite negatives. The weakly supervised ranking loss therefore uses the best-matching potential positive,

$$L_\theta = \sum_j \max\!\left( \min_i d_\theta^2(q, p_i^q) + m - d_\theta^2(q, n_j^q),\; 0 \right),$$

a triplet loss in the style of multiple-instance learning, summed over hard negatives $n_j^q$ with margin $m$. On Pitts250k-test, trained AlexNet NetVLAD reached a Recall@1 of 81.0% against 55.0% for off-the-shelf AlexNet with standard VLAD pooling; VLAD pooling beat max pooling for both off-the-shelf and trained networks; and VGG-16 NetVLAD with whitening beat VLAD of dense RootSIFT on all benchmarks and, without synthesized views of its own, Torii et al.’s view-synthesis method on Tokyo 24/7, the only benchmark on which that method could run because it needs Street View depth data. Training mattered most at the NetVLAD layer, and Time Machine data mattered: on Pitts30k-val, max-pooled AlexNet went from 33.5% Recall@1 off the shelf to 38.7% when trained on same-time images and to 68.5% when trained with Time Machine images (their Table 1 and §5.2).

GeM and other global descriptors

The global descriptors of instance retrieval carried over. Generalized-mean (GeM) pooling (Radenović, Tolias & Chum, 2019) pools each channel of the feature map by a power mean with exponent $p$, between average pooling ($p = 1$) and max pooling ($p \to \infty$); the image retrieval article gives the formula and the authors’ self-supervised training from structure-from-motion reconstructions. In the authors’ code the GeM layer’s exponent starts at $p = 3$ and is a single learned parameter shared by all channels; a separate GeMmp layer learns one per channel (cnnimageretrieval-pytorch at commit 1e66a41). A GeM descriptor is as long as the network has channels (512 to 2,048), much shorter than NetVLAD’s.

Which aggregation is best depends on the training data, according to Berton et al.’s (2022) controlled benchmark, which retrained every combination of backbone and aggregation with the same protocol. With a ResNet-50 trained on the small Pitts30k, the full 65,536-dimensional NetVLAD beat 1,024-dimensional GeM on every test set (for example 69.8% against 41.5% Recall@1 on Tokyo 24/7). Trained on the large Mapillary SLS, GeM with a fully connected layer outscored NetVLAD and the related CRN on Tokyo 24/7 and San Francisco even when they kept 32 times more dimensions (64.0% against 62.8% and 63.6% Recall@1 on Tokyo 24/7, at 2,048 against 65,536 dimensions), which the authors attribute to GeM having been designed for large training sets; reducing NetVLAD to GeM’s size by PCA cost it noticeably. Their benchmark also found partial hard-negative mining nearly as good as mining the whole database at a fraction of the cost, and lower image resolution often harmless or even helpful.

Re-ranking with geometric verification

A global descriptor discards where features are, so a second stage can re-rank the top candidates by checking local correspondences geometrically, as spatial verification does in instance retrieval. Patch-NetVLAD (Hausler, Garg, Xu, Milford & Fischer, 2021) derives the local features from NetVLAD itself. After retrieving the top 100 candidates with ordinary NetVLAD, it computes a NetVLAD descriptor (with the same projection and PCA) for every $d_x \times d_y$ patch of the conv5 feature map, with stride $s_p$, finds mutual nearest neighbours between the patch descriptors of the query and each candidate, and scores the pair by the number of inliers of a homography fitted with RANSAC to the patch centres, with the stride as inlier tolerance; a faster “rapid spatial scoring” rewards matched patches whose displacements agree with the mean displacement instead. Because a patch descriptor is a sum over its $1 \times 1$ cells before projection, an integral feature map gives the descriptors of every patch size cheaply (“IntegralVLAD”), and scores from patch sizes 2, 5 and 8 (a single patch of size 5 covers about 228 × 228 pixels of the 640 × 480 image) are fused with weights 0.45, 0.15 and 0.4, chosen by grid search on RobotCar Seasons v2 and used for every dataset.

In their Table 1, Patch-NetVLAD raised Recall@1 over NetVLAD from 13.0% to 44.9% on Nordland, from 58.6% to 79.5% on the Mapillary SLS validation set, from 64.4% to 86.0% on Tokyo 24/7 and from 35.1% to 48.1% on the withheld Mapillary challenge test set, a result that won the Mapillary place recognition challenge at ECCV 2020 (the text gives the validation figure as “79.%”, the table 79.5). It beat a SuperPoint–SuperGlue re-ranking on average, while DELG’s local-feature re-ranking scored higher on several datasets at roughly ten times the computation. The multi-scale fusion added about 2 points of Recall@1 on average over a single patch size.

Learned aggregation and foundation-model features

Later single-stage methods question whether re-ranking is needed. MixVPR (Ali-bey, Chaib-draa & Giguère, 2023) treats each of the $c$ activation maps of an intermediate ResNet layer as a flattened global feature, passes them through a cascade of $L = 4$ MLP blocks with skip connections that mix information across the whole map, and projects the result channel-wise and row-wise to a 4,096- or 2,048-dimensional descriptor. Trained on GSV-Cities with a multi-similarity loss and with every baseline retrained on the same data and backbone, it reported Recall@1 of 94.6% on Pitts250k-test, 88.0% on MSLS-val and 58.4% on Nordland, against 90.5%, 82.6% and 32.6% for retrained NetVLAD, and 64.0% on the MSLS challenge set without re-ranking, above Patch-NetVLAD’s 48.1%.

Self-supervised foundation models then supplied the backbone. AnyLoc (Keetha et al.) uses dense per-patch features of a DINOv2 vision transformer, taken from a chosen layer and “facet” and used without fine-tuning (“as is”, in SALAD’s description), and aggregates them with an unsupervised VLAD whose vocabulary is fitted by clustering a set of descriptors, such as the database’s (AnyLoc README at commit a9fda68; the paper, in IEEE Robotics and Automation Letters, was not read for this article). SALAD (Izquierdo & Civera, 2024) fine-tunes the last four blocks of a DINOv2 ViT-B on GSV-Cities and replaces NetVLAD’s softmax assignment with an optimal-transport assignment computed by the Sinkhorn algorithm, with an extra “dustbin” cluster that absorbs uninformative features; its 8,448-dimensional descriptor (64 clusters of 128 dimensions plus a 256-dimensional global token) trained in about 30 minutes on one GPU. In SALAD’s tables it reached 75.0% Recall@1 on the MSLS challenge set and 76.0% on Nordland, against 42.2% and 16.1% for AnyLoc’s frozen features, and its ablations show that a fine-tuned DINOv2 with plain NetVLAD aggregation was already close (75.8% and 71.8%, with a descriptor of 24,576 dimensions), which in our reading locates much of the gain in the fine-tuned backbone; the authors stress that aggregation also matters and point to the weaker Nordland result of DINOv2 NetVLAD.

These results compare methods within each paper’s own protocol; the Nordland numbers in particular come from different versions of the dataset (below) and should not be compared across papers.

Place Recognition and Loop Closure in SLAM

In SLAM, drift accumulates while the robot explores, and recognizing a return to a mapped place, a loop closure, adds a constraint that lets the back end correct the whole trajectory; the SLAM article shows the effect of one such constraint on a pose graph. The place-recognition front end must answer quickly, against a database that grows as the robot moves, and with very few false positives. Bag-of-words retrieval with temporal and geometric consistency checks, as in DBoW2, is the common solution described in the SLAM article, and a robust back end can still reject a false loop by the residuals it induces. Lowry et al. (2016) describe a shift from demanding 100% precision of the place recognizer towards accepting more candidate matches and correcting mismatches afterwards with topological or geometric checks, which matters most in changing environments where strict matching fails. Patch-NetVLAD’s authors present their method as a component for SLAM as well as for stand-alone localization against a prior map.

Loop closure differs from benchmark retrieval in that the query’s own recent frames must be excluded (they trivially match), the decision is binary rather than a ranking, and the cost of an error is asymmetric. The methods above are evaluated mostly on retrieval benchmarks, so their behaviour inside a full SLAM system depends on this verification layer.

Example: Pooling, VLAD and Geometric Re-ranking on a Synthetic Street

The script builds a street whose ground truth is known exactly and runs the three stages discussed above: global average pooling of local descriptors, VLAD of the same descriptors, and VLAD followed by geometric re-ranking. The street is a facade strip of buildings 5 to 11 m wide (10 pixels per metre). Each building takes one of four shared styles of window grid and wall tone, so different places look alike, as rows of similar houses do; what tells places apart is small: doors, signs carrying random glyph patterns, drainpipes, and the order of the buildings. Images of 96 × 128 pixels are rendered with a pinhole camera by intersecting every image ray with the facade plane, so the facade is warped by a true homography.

The database has one image every 2 m along 400 m of street, taken frontally from 12 m away in daylight. Queries are taken at random positions on the same street from 9.6 to 15 m away, turned by up to 20° of yaw and 3° of roll. Day queries change brightness and contrast and add a shading gradient across the image; night queries darken the walls, light each window independently with probability 0.4, and change brightness. Following the Pittsburgh protocol, a query is correct at $N$ if one of its top $N$ database images was taken within $d = 5$ m of the query’s camera position; with about five database images inside that radius, a random ranking would give a Recall@1 near 2.5%.

Local descriptors are dense DAISY descriptors from scikit-image 0.25.2 (a 6-pixel grid, 104 dimensions each), in the spirit of DenseVLAD’s dense RootSIFT. Average pooling takes their mean. VLAD uses $k = 32$ words, intra-normalization and global $\ell_2$ normalization. Re-ranking follows Patch-NetVLAD in outline, with DAISY descriptors in place of patch-NetVLAD descriptors: mutual nearest neighbours between query and candidate descriptors, a RANSAC homography on the descriptor positions with the 6-pixel grid stride as inlier tolerance, and the top 20 candidates re-ordered by inlier count. Where the vocabulary should come from is left open by the sources (Torii et al., 2015, learned DenseVLAD’s vocabulary from the database images; NetVLAD learns its centres on separate training data), so the script prints both: words learned on the test street’s database images and on a separate 100 m training street.

The script ran in 47 to 82 seconds in our runs, with Python 3.13, NumPy 2.3.5 and SciPy 1.16.3. It was planned, with its hypotheses, before the first run. Its first two runs stopped with errors before producing any output (the last building overran the facade array, and the training street had fewer descriptors than the k-means sample asked for); after fixing both, using 10,000 descriptors for each vocabulary, the third run completed in about 18 minutes but printed only one re-ranking row, because the two re-ranking rows shared a label: it stood in the database-vocabulary position and held the training-vocabulary results. The labels were then reworded and made distinct and NumPy’s threading was limited to one thread, which removed the overhead on small matrix products; every row printed by the third run was reproduced exactly.

import os
os.environ["OMP_NUM_THREADS"] = "1"        # small matrices: threads only add overhead
import numpy as np
from scipy import ndimage
from skimage.feature import daisy  # scikit-image 0.25.2

PX = 10                      # facade pixels per metre
FH, HC = 160, 80             # facade height; camera height (facade row)
IH, IW, F = 96, 128, 110.0   # image size and focal length (pixels)
D0 = 120.0                   # database camera distance to the facade (pixels)
STEP, RAD = 6, 12            # DAISY grid stride and radius


def make_styles(rng, n=4):
    """Shared building styles: window size, spacing, rows and wall tone."""
    return [dict(ww=rng.integers(8, 14), wh=rng.integers(10, 18), sp=rng.integers(18, 26),
                 rows=rng.integers(3, 5), tone=rng.uniform(0.45, 0.7)) for _ in range(n)]


def make_street(rng, length_m, styles):
    """A facade strip in [0, 1] and the rectangles of its windows."""
    L = length_m * PX + 400                          # margin so every camera sees facade
    fac = np.zeros((FH, L + 120))                    # room for the last building
    windows, x = [], 0
    while x < L:
        w = int(rng.integers(50, 110))
        st = styles[rng.integers(len(styles))]
        wall = st["tone"] + rng.uniform(-0.04, 0.04)
        fac[:, x:x + w] = wall + 0.03 * ndimage.gaussian_filter(rng.standard_normal((FH, w)), 1)
        fac[::6, x:x + w] -= 0.03                    # brick courses
        y0 = 20
        for r in range(st["rows"]):
            yy = y0 + r * (st["wh"] + 12)
            for xx in range(x + 6, x + w - st["ww"] - 4, st["sp"]):
                fac[yy - 1:yy + st["wh"] + 1, xx - 1:xx + st["ww"] + 1] = wall + 0.15  # frame
                fac[yy:yy + st["wh"], xx:xx + st["ww"]] = 0.12
                windows.append((yy, xx, st["wh"], st["ww"]))
        dx = x + int(rng.integers(4, max(5, w - 18)))  # door
        fac[FH - 50:FH - 22, dx:dx + 14] = 0.08
        if rng.random() < 0.7:                       # a sign with a random glyph pattern
            sw = int(rng.integers(20, 36))
            sx = x + int(rng.integers(2, max(3, w - sw - 2)))
            g = np.kron(rng.random((3, sw // 4)) < 0.5, np.ones((4, 4)))
            fac[FH - 66:FH - 54, sx:sx + g.shape[1]] = 0.9 - 0.75 * g[:12]
        if rng.random() < 0.3:                       # drainpipe
            px = x + int(rng.integers(2, w - 2))
            fac[10:FH - 22, px:px + 2] = 0.25
        fac[:, x] = 0.1                              # building boundary
        x += w
    fac[FH - 22:] = 0.3                              # pavement
    return np.clip(fac[:, :L], 0, 1), windows


def render(fac, xcam, yaw=0.0, dist=D0, roll=0.0):
    """Pinhole camera at street position xcam (pixels), looking at the facade plane:
    each image ray is intersected with the plane, so the warp is a homography."""
    r, c = np.mgrid[0:IH, 0:IW].astype(float)
    xc, yc = c - IW / 2, r - IH / 2
    cr, sr = np.cos(roll), np.sin(roll)
    xc, yc = cr * xc - sr * yc, sr * xc + cr * yc
    zc = np.full_like(xc, F)
    xw = xc * np.cos(yaw) + zc * np.sin(yaw)
    zw = -xc * np.sin(yaw) + zc * np.cos(yaw)
    t = dist / zw
    X, Y = xcam + t * xw, HC + t * yc
    return ndimage.map_coordinates(fac, [Y, X], order=1, mode="nearest")


def night_facade(rng, fac, windows):
    """Dark walls; each window lit independently with probability 0.4."""
    n = 0.35 * fac
    for (y, x, h, w) in windows:
        if rng.random() < 0.4:
            n[y:y + h, x:x + w] = 0.85
    return n


def features(img):
    d = daisy(img, step=STEP, radius=RAD, rings=2, histograms=6, orientations=8)
    rr, cc = np.mgrid[0:d.shape[0], 0:d.shape[1]]
    pos = np.c_[RAD + STEP * rr.ravel(), RAD + STEP * cc.ravel()].astype(float)
    return d.reshape(-1, d.shape[2]), pos


def unit(v):
    return v / (np.linalg.norm(v, axis=-1, keepdims=True) + 1e-12)


def kmeans(X, k, rng, iters=15):
    C = X[rng.choice(len(X), k, replace=False)].copy()
    for _ in range(iters):
        lab = np.argmin((C ** 2).sum(1) - 2 * X @ C.T, axis=1)
        for j in range(k):
            if np.any(lab == j):
                C[j] = X[lab == j].mean(0)
    return C


def vlad(D, C):
    """VLAD: sum of residuals per nearest word, intra-normalized, then l2-normalized."""
    lab = np.argmin((C ** 2).sum(1) - 2 * D @ C.T, axis=1)
    V = np.zeros_like(C)
    np.add.at(V, lab, D - C[lab])
    return unit(unit(V).ravel())


def homography_inliers(p, q, rng, n_hyp=300, tol=STEP):
    """RANSAC: largest number of matches consistent with one homography p -> q."""
    m = len(p)
    if m < 4:
        return 0
    idx = np.argsort(rng.random((n_hyp, m)), axis=1)[:, :4]          # 4 distinct matches each
    P, Q = p[idx], q[idx]                                   # (n_hyp, 4, 2)
    z, o = np.zeros((n_hyp, 4)), np.ones((n_hyp, 4))
    x, y, u, v = P[..., 0], P[..., 1], Q[..., 0], Q[..., 1]
    A = np.concatenate([np.stack([x, y, o, z, z, z, -u * x, -u * y, -u], -1),
                        np.stack([z, z, z, x, y, o, -v * x, -v * y, -v], -1)], 1)
    H = np.linalg.svd(A)[2][:, -1].reshape(n_hyp, 3, 3)     # DLT, smallest singular vector
    ph = np.c_[p, np.ones(m)]
    proj = np.einsum("hij,mj->hmi", H, ph)
    w = proj[..., 2]
    ok = np.abs(w) > 1e-9
    err = np.linalg.norm(proj[..., :2] / np.where(ok, w, 1)[..., None] - q[None], axis=2)
    return int(((err < tol) & ok).sum(1).max())


def mutual_nn(Dq, Dd):
    S = Dq @ Dd.T
    a, b = S.argmax(1), S.argmax(0)
    i = np.flatnonzero(b[a] == np.arange(len(Dq)))
    return i, a[i]


def recall_at(rank_pos, qpos, ns=(1, 5, 10, 20), d_m=5.0):
    """Fraction of queries with a database image within d_m metres among the top N."""
    hit = np.abs(rank_pos - qpos[:, None]) <= d_m * PX
    first = np.where(hit.any(1), hit.argmax(1), 10 ** 9)
    return {n: float(np.mean(first < n)) for n in ns}


def run(seed, street_m=400, train_m=100, n_q=100, k=32, top=20):
    rng = np.random.default_rng(seed)
    styles = make_styles(rng)
    fac, windows = make_street(rng, street_m, styles)
    train_fac, _ = make_street(rng, train_m, styles)
    noise = lambda im: np.clip(im + rng.normal(0, 0.02, im.shape), 0, 1)

    db_x = 200 + np.arange(0, street_m * PX, 2 * PX, dtype=float)   # every 2 m
    db = [noise(render(fac, x)) for x in db_x]
    train = [noise(render(train_fac, x)) for x in 200 + np.arange(0, train_m * PX, 2 * PX)]

    queries, q_x, cond = [], [], []
    for c in ("day", "night"):
        for _ in range(n_q):
            x = rng.uniform(200, 200 + street_m * PX - 2 * PX)
            yaw, roll = np.deg2rad(rng.uniform(-20, 20)), np.deg2rad(rng.uniform(-3, 3))
            dist = D0 * rng.uniform(0.8, 1.25)
            src = fac if c == "day" else night_facade(rng, fac, windows)
            im = render(src, x, yaw, dist, roll)
            if c == "day":                                   # sun and shadow
                im = im * (1 + rng.uniform(-0.2, 0.2) * np.linspace(-1, 1, IW))[None, :]
            im = rng.uniform(0.75, 1.25) * im + rng.uniform(-0.05, 0.05)
            queries.append(noise(np.clip(im, 0, 1)))
            q_x.append(x)
            cond.append(c)
    q_x, cond = np.array(q_x), np.array(cond)

    fdb = [features(im) for im in db]
    fq = [features(im) for im in queries]
    ftr = [features(im) for im in train]

    out = {}
    avg_db = unit(np.array([f[0].mean(0) for f in fdb]))
    avg_q = unit(np.array([f[0].mean(0) for f in fq]))
    out["average pooling"] = np.argsort(-(avg_q @ avg_db.T), axis=1, kind="stable")

    sample = lambda fs: np.concatenate([f[0] for f in fs])
    Xdb, Xtr = sample(fdb), sample(ftr)
    vocabs = {"database street": kmeans(Xdb[rng.choice(len(Xdb), 10000, replace=False)], k, rng),
              "training street": kmeans(Xtr[rng.choice(len(Xtr), 10000, replace=False)], k, rng)}
    cache = {}
    for vname, C in vocabs.items():
        Vdb = np.array([vlad(f[0], C) for f in fdb])
        Vq = np.array([vlad(f[0], C) for f in fq])
        order = np.argsort(-(Vq @ Vdb.T), axis=1, kind="stable")
        out[f"VLAD, {vname} vocabulary"] = order
        rer = []
        for n, o in enumerate(order):
            score = []
            for d in o[:top]:
                if (n, d) not in cache:
                    i, j = mutual_nn(fq[n][0], fdb[d][0])
                    prng = np.random.default_rng([seed, n, int(d)])
                    cache[(n, d)] = homography_inliers(fq[n][1][i][:, ::-1], fdb[d][1][j][:, ::-1], prng)
                score.append(cache[(n, d)])
            head = o[:top][np.argsort(-np.array(score), kind="stable")]
            rer.append(np.r_[head, o[top:]])
        out[f"  + re-ranking top {top} ({vname.split()[0]})"] = np.array(rer)
    return out, db_x, q_x, cond


if __name__ == "__main__":
    import sys, time
    t0 = time.time()
    seeds = [0, 1, 2]
    res = [run(s) for s in seeds]
    names = list(res[0][0])
    print(f"seeds {seeds}; 200 database images (every 2 m), 100 day + 100 night queries per seed")
    print(f"Recall@N at d = 5 m: R@1 mean [min, max] over seeds, then means of R@5, R@10, R@20")
    for c in ("day", "night"):
        print(f"--- {c} queries")
        for nm in names:
            rs = []
            for out, db_x, q_x, cond in res:
                sel = cond == c
                rs.append(recall_at(db_x[out[nm][sel]], q_x[sel]))
            r1 = [r[1] for r in rs]
            print(f"{nm:38s} R@1 {np.mean(r1):.2f} [{min(r1):.2f}, {max(r1):.2f}]  "
                  f"R@5 {np.mean([r[5] for r in rs]):.2f}  R@10 {np.mean([r[10] for r in rs]):.2f}  "
                  f"R@20 {np.mean([r[20] for r in rs]):.2f}")
    print("--- R@1 against the localization radius d (all queries, mean over seeds)")
    for nm in names:
        line = []
        for d_m in (2, 5, 10):
            line.append(np.mean([recall_at(r[1][r[0][nm]], r[2], d_m=d_m)[1] for r in res]))
        print(f"{nm:38s} d=2 m {line[0]:.2f}   d=5 m {line[1]:.2f}   d=10 m {line[2]:.2f}")
    print(f"runtime {time.time() - t0:.0f} s", file=sys.stderr)

Output:

seeds [0, 1, 2]; 200 database images (every 2 m), 100 day + 100 night queries per seed
Recall@N at d = 5 m: R@1 mean [min, max] over seeds, then means of R@5, R@10, R@20
--- day queries
average pooling                        R@1 0.23 [0.18, 0.29]  R@5 0.54  R@10 0.72  R@20 0.87
VLAD, database street vocabulary       R@1 0.69 [0.63, 0.76]  R@5 0.89  R@10 0.95  R@20 0.98
  + re-ranking top 20 (database)       R@1 0.88 [0.88, 0.88]  R@5 0.97  R@10 0.98  R@20 0.98
VLAD, training street vocabulary       R@1 0.70 [0.67, 0.73]  R@5 0.91  R@10 0.95  R@20 0.97
  + re-ranking top 20 (training)       R@1 0.88 [0.87, 0.89]  R@5 0.96  R@10 0.97  R@20 0.97
--- night queries
average pooling                        R@1 0.04 [0.01, 0.07]  R@5 0.15  R@10 0.24  R@20 0.37
VLAD, database street vocabulary       R@1 0.07 [0.02, 0.10]  R@5 0.16  R@10 0.29  R@20 0.43
  + re-ranking top 20 (database)       R@1 0.17 [0.11, 0.25]  R@5 0.30  R@10 0.37  R@20 0.43
VLAD, training street vocabulary       R@1 0.07 [0.05, 0.10]  R@5 0.20  R@10 0.33  R@20 0.47
  + re-ranking top 20 (training)       R@1 0.17 [0.08, 0.26]  R@5 0.33  R@10 0.41  R@20 0.47
--- R@1 against the localization radius d (all queries, mean over seeds)
average pooling                        d=2 m 0.05   d=5 m 0.13   d=10 m 0.17
VLAD, database street vocabulary       d=2 m 0.19   d=5 m 0.38   d=10 m 0.45
  + re-ranking top 20 (database)       d=2 m 0.28   d=5 m 0.53   d=10 m 0.58
VLAD, training street vocabulary       d=2 m 0.18   d=5 m 0.39   d=10 m 0.47
  + re-ranking top 20 (training)       d=2 m 0.28   d=5 m 0.53   d=10 m 0.59

By day the three stages separate as the retrieval pipeline predicts. The mean descriptor is dominated by what every place shares, wall texture and window grids, so average pooling finds the right place first in only about a quarter of the queries, although its Recall@20 shows that the right place is usually somewhere near the top. VLAD keeps which kinds of local pattern occur and how they deviate from their word, which distinguishes buildings of the same style through their signs and doors, and triples Recall@1. Re-ranking then resolves most remaining confusions inside the shortlist: among places that look alike, only the right one has descriptors whose positions agree with a single homography of the facade. Its Recall@1 approaches, but stays below, the VLAD shortlist’s Recall@20, which bounds what re-ranking a top-20 list can reach, which is why geometric verification cannot rescue a first stage that misses the place altogether. The two vocabulary sources differ by a few points at most, within the spread across seeds.

The night queries show the limit of descriptors built on image gradients. Lighting random windows does not just dim the image; it replaces dark rectangles with bright ones in a pattern that changes from night to night, and the gradient-based descriptors change with it. Average pooling and VLAD fall to a Recall@1 between about 1.5 and 3 times the chance level, and VLAD’s advantage almost disappears, against the plan’s hypothesis that “VLAD beats average pooling clearly (guess +15 to +30 points R@1)”; by day the gain exceeded that guess. Re-ranking still helps, more than doubling Recall@1, because the facade’s geometry survives where the descriptors partly agree (doors, signs, building edges), but it is capped by a shortlist that contains a correct image for fewer than half of the queries. This mirrors, at small scale, why Torii et al. (2015) and NetVLAD’s authors treated night queries separately and why learned, condition-invariant descriptors were needed; the example has no learned component and makes no claim about how large such gaps are on real data.

The last block varies the localization radius. With $d = 2$ m, Recall@1 roughly halves for every method. A query camera turned by 20° at 12 m views facade about 4.4 m to the side of its own position, so its best visual match can be a database image taken outside a 2 m radius: the protocol defines a match by where the camera stood, not by what it saw. Widening $d$ to 10 m adds fewer queries than narrowing it removes. Radii are a protocol choice and change every number; comparisons are valid only at the same radius.

Datasets and Benchmarks

Benchmark	Database	Queries	Change between query and database	Correct if
Pitts250k (Torii et al., 2013)	254,064 Street View crops of 10,586 panoramas	24,000 crops of 1,000 panoramas from another session	viewpoint, illumination, some season	within 25 m
Pitts30k (Arandjelović et al., 2016)	10,000 per split	about 6,800 in the test split (Berton et al., 2022)	as Pitts250k	within 25 m
Tokyo 24/7 (Torii et al., 2015)	75,984 Street View crops of 6,332 panoramas	315 phone photos (of 1,125 collected)	day, sunset, night; structure	within 25 m, after spatial non-maximum suppression
Nordland (Sünderhauf et al., 2013, as described by Warburg et al., 2020)	one season of a 182 km train journey	another season	season; almost no viewpoint change	within a few frames (version-dependent)
Mapillary SLS (Warburg et al., 2020)	1.68 million images from 30 cities; test cities disjoint	one query frame per sequence	all of the above, over up to 7 years, many cameras	within 25 m and 40° of heading

Pittsburgh (Pitts250k, Pitts30k). Torii et al. (2013) generated 24 perspective images of 640 × 480 pixels (60° horizontal field of view) from each of 10,586 Google Street View panoramas of Pittsburgh, at 12 horizontal directions and two elevation angles (the paper labels these “pitch” and “yaw” the other way round), giving 254,064 database images; 24,000 query images came from 1,000 panoramas of the Google Pittsburgh Research Data Set, captured in a different session. They noted that Street View GPS positions are often snapped to the middle of the street, so the ground truth itself is accurate only to roughly 7 to 15 m. Arandjelović et al. (2016) split the data geographically into three roughly equal parts of about 83,000 database images and 8,000 queries each for training, validation and testing (Pitts250k-train, -val and -test), and defined a smaller Pitts30k with 10,000 database images per split. Later papers describe Pitts30k-test differently (about 6,800 queries and 10,000 database images in Berton et al.’s table, “8k queries and 8k references” in MixVPR’s text), so its query count should be taken from the split files used. Patch-NetVLAD’s authors called Pittsburgh and Tokyo 24/7 “near-solved” in 2021.

Tokyo 24/7. Torii et al. (2015) photographed 125 locations in Tokyo with two smartphones, in three viewing directions and at three times of day (daytime, sunset, night): 1,125 queries, with ground-truth positions placed by hand on a map and estimated accurate to 5 m. The evaluation uses the 315 queries inside the roughly 1.6 × 1.6 km area covered by the database. Torii et al. diversified the shortlists by spatial non-maximum suppression on the map, assigning the score of each synthesized view to the closest Street View panorama, and NetVLAD followed this protocol on Tokyo 24/7. For training, Arandjelović et al. (2016) collected a geographically disjoint Tokyo Time Machine set.

Nordland. Nordland is footage from a camera at the front of a train on a Norwegian railway, traversed once in each season. Warburg et al. (2020) list it as four sequences of a 182 km journey (728 km in total, about 115,000 frames) with seasonal change but little variation of viewpoint, camera or time of day. Because consecutive frames look alike and the camera follows the rails, Nordland isolates appearance change and aliasing. Its protocols vary: Patch-NetVLAD removes tunnels and stops, as earlier work did, and counts a match within 10 frames, while MixVPR and SALAD use the version distributed with the VPR-Bench framework, which SALAD describes as tolerating two frames (MixVPR’s text applies its 25 m rule to all benchmarks). Results on the two versions differ greatly (NetVLAD scored 13.0% Recall@1 in Patch-NetVLAD’s table and 32.6% in MixVPR’s retrained version) and are not comparable. The dataset paper by Sünderhauf, Neubert and Protzel (2013) was not read for this article.

Mapillary Street-Level Sequences (MSLS). Warburg et al. (2020) curated about 1.68 million images from the Mapillary platform: 30 cities on six continents, hundreds of cameras, seasons, day and night, and revisits up to seven years apart, each image with GPS, compass heading, capture time and a day/night label. Images are grouped into sequences, sub-sequences within 30 m and 40° of each other are clustered as the same place, and one query frame per sequence is evaluated, so that every place weighs equally. A database image is correct if it lies within 25 m of the query and its viewing direction differs by less than 40°. The test cities (Miami, Athens, Buenos Aires, Stockholm, Bengaluru, Kampala) are disjoint from the training cities, and test metadata are withheld, so test results come from the challenge server; most papers report the public validation set (“MSLS-val”) and some the withheld set (“MSLS challenge”), which Patch-NetVLAD’s authors found harder (48.1% against 79.5% Recall@1). The paper reports Recall@5 for summer/winter, day/night and old/new subsets on the test set. The text says all models were especially challenged by night-to-day; in the single-image rows of their Table 3 night-to-day is the lowest score of every model (tied with winter-to-summer for the two CaffeNet models), though not in every multi-view row (GeM’s image-to-sequence row is lowest for winter-to-summer). NetVLAD retrained on MSLS reached 48% Recall@1 overall against 28% for NetVLAD trained on Pittsburgh (their Table 3).

Other commonly used sets include San Francisco (the revisited R-SF in Berton et al.’s benchmark) and the larger SF-XL, SPED (static surveillance cameras over a year), St Lucia, Eynsham, Oxford RobotCar and the RobotCar Seasons and Extended CMU Seasons localization benchmarks, which score poses by translation and rotation error rather than retrieval radius (Hausler et al., 2021; Berton et al., 2022; Izquierdo & Civera, 2024).

Evaluation Metrics

Recall@N with a localization radius. The standard measure counts a query as correctly localized if at least one of the top $N$ retrieved database images lies within $d$ of its true position, and reports the percentage of such queries as a function of $N$ (Torii et al., 2013; Arandjelović et al., 2016). It is a hit rate, not the recall of information retrieval: one correct image in the top $N$ is enough, however many lie within the radius, as the Recall@K article explains. Three settings change the number:

• The radius $d$. 25 m is the convention for street-level benchmarks; Berton et al. (2022) also report how results change with the threshold. Because ground truth is the camera’s position, a small radius can count a visually perfect match as wrong when the camera turned toward a neighbouring facade, and GPS error (7 to 15 m in Pittsburgh) sets a floor below which the radius stops being meaningful.
• Extra conditions. Mapillary SLS also requires a heading within 40°; Nordland counts frames instead of metres.
• Database redundancy. When several database crops come from one panorama, a method can fill its top $N$ with near-duplicates (our reading of why diversification matters); Tokyo 24/7’s protocol applies spatial non-maximum suppression before counting.

Recall@N says nothing about whether a system knows when it is wrong. Precision–recall curves, and recall at 100% precision, measure that: the system accepts or rejects each best match by its confidence, and loop closure needs high precision because a false match corrupts the map (Lowry et al., 2016). Practical comparisons also report descriptor dimension, memory, extraction and matching time; Berton et al. (2022) argue that recall alone hides these costs, and that approximate search such as IVF-PQ can cut matching time and memory by 98.5% at a cost of a few points of Recall@1.

Applications

• Loop closure and relocalization in SLAM, described above; also map merging between sessions or robots.
• Localization against a prior map for robots, drones and vehicles where GNSS is unavailable or imprecise, often as the coarse stage before pose estimation against a 3D map (robot perception, autonomous driving perception).
• Augmented reality, where retrieving the right part of a map is the first step of placing virtual content in a city.
• Large-scale reconstruction: Warburg et al. (2020) list efficient 3D reconstruction as a use, since retrieval chooses which images to match in structure from motion.
• Geo-localizing photographs, including archival imagery, which NetVLAD’s authors cite as a motivation.

Open Problems

• Extreme appearance change. Night-to-day was the hardest MSLS case for every single-image model in Warburg et al. (2020), and Nordland and the MSLS challenge set are the benchmarks on which recent methods still leave much room (Izquierdo & Civera, 2024).
• Generalization across places and domains. Training data from a few cities biases models geographically (Warburg et al., 2020); comparisons across papers are confounded by different training sets, which Berton et al. (2022) show can change results by tens of points.
• Knowing when not to match. Retrieval benchmarks always contain a correct answer; a robot must also detect that it is somewhere new, and calibrated confidences for that decision are much less studied than Recall@N.
• Cost. Re-ranking with local features improves accuracy at many times the cost of a single global search (SALAD’s table lists about 8.4 s of re-ranking per query for Patch-NetVLAD, a latency taken from the R2Former authors’ measurements, against 2.4 ms for SALAD’s single global search, measured by SALAD’s authors on another GPU), while compact global descriptors need large databases in memory; single-stage methods such as MixVPR and SALAD claim the accuracy of two-stage ones, and whether that holds across all conditions is open.
• Benchmark protocols. Different versions of Nordland and Pitts30k, the choice of radius and of validation or withheld test sets make published numbers hard to compare, and saturated benchmarks such as Pittsburgh no longer separate methods.