Oxford Buildings

The Oxford Buildings dataset, usually called Oxford 5K or Oxford5k, is a set of 5,062 Flickr photographs of Oxford with a manually labelled ground truth for 55 query regions on 11 landmarks. James Philbin, Ondřej Chum, Michael Isard, Josef Sivic and Andrew Zisserman of the University of Oxford’s Visual Geometry Group (VGG) and Microsoft Research introduced it at CVPR in 2007 to evaluate large-vocabulary bag-of-visual-words retrieval. Its companion, Paris 6K, was collected the same way from Paris landmarks in 2008. For a decade the pair, with the Oxford 100K distractor set, was the standard test for instance-level image retrieval: finding every photograph of the same building as a query. In 2018 Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis and Ondřej Chum relabelled both datasets as Revisited Oxford and Paris (ROxford and RParis). They added harder queries, three protocols called Easy, Medium and Hard, and a set of one million distractors, R1M.

Average precision itself is defined in average precision; the retrieval methods evaluated on these datasets are in image retrieval.

Purpose

Philbin et al. (2007) wanted to measure how a retrieval system’s quality depends on its visual vocabulary and on spatial re-ranking, and how it degrades as the collection grows. They chose building facades as the test domain because repeated architectural elements, such as windows and doors, make them hard to tell apart, and Flickr because it holds many tourist photographs of the same buildings. The task is particular-object retrieval: the user selects a region of a query image, and the system ranks the whole collection by how likely each image is to contain that object. Because judges may disagree when an object is occluded or distorted, the authors chose to rank all images rather than to make a binary decision. Their labels include a “Junk” class so that such borderline images do not count either way.

Paris served a different purpose at first. Philbin et al. (2008) collected it to train visual vocabularies on images of similar scenes, buildings in a related architectural style, that do not contain the Oxford landmarks. Training the vocabulary on Paris and testing on Oxford shows how a system behaves when the images it will search are not known in advance. Paris later became a second test set with the same protocol (see Dataset Size).

Radenović et al. (2018) revisited the two datasets because of annotation errors, their small size, and near-perfect scores that no longer separated methods; they argued that the lack of difficulty came from the annotation, not from the images (see Limitations and Bias).

Dataset Size

Set	Images	Queries	Source
Oxford 5K	5,062 (5,063 in some later papers)	55	2007 paper, Table 1; official page
Paris 6K	6,300 in the 2008 paper; 6,412 on the official page, 20 of them corrupted	55	2008 paper, Table 1; official page; Radenović et al. (2018)
Oxford 100K	99,782	none (distractors)	2007 paper
Philbin et al.’s 1M	1,040,801 at 500 × 333 pixels	none (distractors)	2007 paper
Flickr1M	1 million	none (distractors)	Jégou et al. (2008)
ROxford	4,993 database images	70	Radenović et al. (2018)
RParis	6,322 database images	70	Radenović et al. (2018)
R1M	1,001,001 at 1024 × 768 pixels	none (distractors)	Radenović et al. (2018)

Oxford 5K. The 2007 paper gives 5,062 images of 1024 × 768 pixels and 16,334,970 Hessian-affine SIFT features. The official page also gives 5,062, and the README of its precomputed features lists 5,062 image names. Tolias, Sicre and Jégou (2016) and the dataset section of Radenović et al. (2018) give 5,063, while Radenović et al.’s introduction gives 5,062. Removing the 70 query source images from 5,063 gives ROxford’s 4,993. We could not resolve the one-image difference without downloading the images.

Paris 6K. The 2008 paper describes 6,300 images of 1024 × 768 pixels. The official page lists 6,412 and notes that 20 supplied images are corrupted but kept for backward compatibility. Radenović et al. (2018) and Arandjelović and Zisserman (2012) count 6,392, which is 6,412 less the 20 corrupted images, and Tolias et al. (2016) give 6,412. The 2008 paper defines no Paris queries. The official page offers ground-truth files in the Oxford format but states neither the number of queries nor when they were added. Arandjelović and Zisserman (2012) describe 55 queries, and Radenović et al. (2018) give them as 5 for each of 11 landmarks.

Distractors. Oxford 100K and the 1M set were crawled from Flickr’s 145 and 450 most popular tags, with exact duplicates removed. The authors assumed that they contain none of the landmarks, so that they act as pure distractors. Jégou, Douze and Schmid (2008) built a separate distractor set of one million Flickr images, called Flickr1M, which they described as slightly biased compared with their Holidays dataset because it contains low-resolution images and more photographs of people. Later papers combine the sets: Oxford 105K is Oxford 5K plus Oxford 100K, and Tolias et al. (2016) added a million Flickr distractors from Jégou and colleagues to go further.

Classes

There are no classes in the classification sense: each landmark defines one query group.

• Oxford. The 11 landmarks are All Souls, Ashmolean, Balliol, Bodleian, Christ Church, Cornmarket, Hertford, Keble, Magdalen, Pitt Rivers and Radcliffe Camera (2007 paper, Fig. 2). A “landmark” here means a particular part of a building. The official page lists 17 Flickr searches used to collect the images: the 11 landmarks, five other colleges (Jesus, New, Oriel, Trinity, Worcester) and “Oxford” alone. The 5,062 image names in the README of the precomputed features carry the same 17 search prefixes. The extra searches add images of the same city that mostly show none of the 11 landmarks and so act as distractors (our reading; the 2007 paper names only the “Oxford” search as a source of distractors), and the paper notes that the collection contains difficult distractors that are easily confused with the queries.
• Paris. The official page lists 12 searches: La Défense, the Eiffel Tower, Hôtel des Invalides, the Louvre, the Moulin Rouge, the Musée d’Orsay, Notre Dame, the Panthéon, the Centre Pompidou, Sacré-Cœur, the Arc de Triomphe, and “Paris” alone.

Annotation Types

Query regions. Each query is an image and a bounding box marking the object of interest. The five queries of a landmark come from different images and vary widely in scale, position and lighting (2007 paper, Fig. 2). Radenović et al. note that the boxes simulate both a user cropping away clutter and large occlusions.

Original labels. Every database image received one of four labels for each landmark (2007 paper, §2):

• Good: a nice, clear picture of the object.
• OK: more than 25% of the object is clearly visible.
• Junk: less than 25% of the object is visible, or there is a very high level of occlusion or distortion.
• Absent: the object is not present. The official page calls this label “Bad”.

The paper reports between 7 and 220 Good and OK images per landmark. All five queries of a landmark share one list of labels.

Revisited labels. Five annotators relabelled both datasets in three steps (Radenović et al., 2018, §2.2):

1. Potential positives. Each annotator inspected the whole dataset for any image showing any side or version of each landmark, also using an interactive retrieval tool. These images, merged with every image originally labelled positive or Junk, formed the list of potential positives.
2. Labels. Each annotator labelled each potential positive for each query group. Easy means the image shows the landmark from the same side as the query, without a large change of viewpoint, significant occlusion, extreme illumination change or severe clutter. Hard means the landmark is shown under conditions that are difficult to match but is recognizable without context. Unclear means the image possibly shows the landmark but the overlap with the query region cannot be decided, or it shows another side of a partially symmetric building. Negative means none of these applies; an image with any physical overlap with the query is never Negative.
3. Majority vote. A first vote decided between {Easy, Hard}, Unclear and Negative, with ties going to Unclear. A second vote decided between Easy and Hard, with ties going to Hard. The authors then inspected images in order of decreasing label entropy.

The revision also changed the query groups. In the original data, the second Balliol query and the fourth Christ Church query show a different side from the other four, and the Arc de Triomphe queries mix three day and two night photographs. Splitting these gave 13 groups for Oxford and 12 for Paris. Fifteen new, harder queries per dataset, three on each of five original landmarks, raised the totals to 70 queries in 26 groups (ROxford) and 25 groups (RParis). The images from which the queries are cropped were removed from the databases, so that a method that preprocesses the database cannot use any part of a query.

Radenović et al.’s Table 1 counts how labels moved. On ROxford, 222 originally Junk images became Hard and 50 became Easy, while 72 originally Absent images became Hard and 1 Easy. On RParis, 813 Junk images became Hard and 147 Absent images became Hard. The movement from Absent to Easy or Hard counts errors in the original labels; the movement from Junk to Hard reflects the new protocol’s greater challenge. One originally positive image on ROxford became Negative, and six on RParis.

R1M. The distractors are high-resolution (1024 × 768) images from YFCC100M, a collection of Flickr photographs and videos published under Creative Commons licenses (Thomee et al., 2016). Radenović et al. randomly chose 5 million geotagged images, excluded the UK, France and Las Vegas (for its replicas of the Eiffel Tower and the Arc de Triomphe), and kept about 4.1 million available at high resolution. They ranked these by retrieval against each landmark, manually inspected the top 2,000 per landmark, and removed 110 images that showed a query landmark, for example through faulty GPS tags, toy models, or paintings and photographs of the landmark. Finally, they ran all 70 queries against the 4.1 million images with three methods, scored each distractor by the fraction of Easy and Hard images ranked below it, summed over queries and methods, and kept the one million most distracting. Mean average precision on ROxford with this set was close to that with all 4 million distractors and clearly lower than with a random million, and adding either of two other methods to the selection changed it little (their Table 2).

Splits

There is no training split. Every image is a test image, and the benchmarks expect anything a method learns, such as a vocabulary, a PCA whitening or a fine-tuned network, to come from other data. This rule was established gradually rather than stated with the dataset:

• Philbin et al. (2007) computed their vocabularies on the Oxford 5K images themselves. Their 2008 paper added vocabularies trained on Paris, and found that every method lost accuracy when the vocabulary came from the other city.
• Jégou et al. (2008) learned all parameters on Flickr60K, a set of Flickr images unrelated to the test sets.
• Arandjelović and Zisserman (2012) reported their results separately for vocabularies learned on the test dataset and on an independent one. Tolias et al. (2016) learned PCA on Paris when testing on Oxford and vice versa, and compared their results directly only with methods that do not learn on the test set.

The revisited protocol makes two further rules explicit: the query images are no longer in the database, and only the cropped region may be used as the query, never the full image, because the labels describe only what is inside the box.

Tasks

• Particular-object and landmark retrieval from a query region, scored by mean average precision.
• Large-scale retrieval, with the distractor sets added to the database, which tests memory and speed as well as accuracy.
• Training data for vocabularies, whitening or descriptors used on the other city, mainly Paris for Oxford.
• Object localization, occasionally: Tolias et al. (2016) used the query boxes of the other four queries of a landmark to score how well their method localized the object.

Evaluation

The original protocol

The official program is compute_ap.cpp on the Oxford page, unchanged between the copy archived in May 2009 and the live copy of October 2026. For each query it reads three lists, <query>_good.txt, <query>_ok.txt and <query>_junk.txt, together with the system’s ranked list of image names.

1. Positives are the union of Good and OK. Absent images, which appear in no list, are negatives.
2. Junk images are deleted from the ranked list before anything is counted. They do not occupy a rank, so returning one at the top neither helps nor hurts. The 2007 paper calls them null examples, treated as though they are not in the database.
3. AP is accumulated over the remaining list as trapezoids under the precision–recall curve, starting from precision 1 at recall 0. The recall denominator is the total number of positives in the ground truth, so positives missing from a truncated list lower the score. The trapezoidal rule, and how it differs from the mean of the precisions at the positives, is explained in average precision.
4. mAP is the mean of AP over the 55 queries. The 2007 paper averaged the five queries of each landmark first and then the landmarks, which gives the same number because every landmark has five queries.

The queries are the regions, not the images. The Paris ground truth uses the same file format, so the same program scores it.

The revisited protocol

The revisitop code (python/evaluate.py and python/example_evaluate.py, commit be39832, June 2022; the MATLAB code mirrors it) loads, for each query, three lists of image indices: easy, hard and junk. Read against the paper’s definitions, the list the files call junk holds the Unclear images. Each setup passes compute_map one list of positives and one list of images to ignore:

Setup	Positive	Ignored	Negative
Easy (E)	Easy	Hard, Unclear	all others
Medium (M)	Easy, Hard	Unclear	all others
Hard (H)	Hard	Easy, Unclear	all others

compute_map deletes ignored images as compute_ap.cpp deletes Junk: it shifts each positive’s rank up by the number of ignored images ranked above it, then adds the same trapezoids. A query with no positives in a setup is left out of that setup’s mean, which can happen in the Hard setup. The code also reports mean precision at rank $K$ (mP@K, with $K$ = 1, 5 and 10). Its denominator is not $K$ but the smaller of $K$ and the rank of the last positive. A query with three positives at ranks 1 to 3 therefore scores 1 at $K = 10$, not 0.3.

Ignoring Easy images in the Hard setup means that ranking them first costs nothing: the setup measures how well a method finds the difficult views, whatever it does with the easy ones. Finding the original protocol closest to Easy and Easy close to saturation, Radenović et al. reported only Medium and Hard, with and without R1M, and added mP@10 to mAP because it reflects what a user would inspect and correlates with the success of later steps such as query expansion.

Example: one ranking, five scores

The script below implements both protocols in NumPy and applies them to synthetic ranked lists, with no images and no dataset files. The first part scores a nine-item list three ways. The second builds a synthetic benchmark of 11 landmarks with 5 queries each and 5,000 database images. Every image has a “true” relation to each landmark: Easy, Hard, Unclear or Negative, in random numbers per landmark. An old-style annotation is drawn from it with probabilities we chose: Easy views are almost always Good, Hard views are mostly Junk, Unclear ones are split, and a few negatives are mislabelled Good. Two simulated systems rank the images by a score equal to a per-label mean plus Gaussian noise. System A separates Easy views well and Hard views poorly. System B gives up a little on Easy views to rank Hard views much higher. Each query of a landmark draws its own noise. The script runs in about 2 seconds.

import numpy as np


def trapezoid_ap(labels, positive, ignore):
    """AP as compute_ap.cpp and revisitop's compute_ap compute it.

    labels: label of each database image in ranked order (best first).
    Ignored images are deleted from the list; each positive adds a trapezoid
    of width 1/n_pos between the precision before it and the precision at it.
    """
    keep = ~np.isin(labels, ignore)
    hits = np.isin(labels[keep], positive)
    n_pos = hits.sum()
    if n_pos == 0:
        return np.nan                      # query left out of the mean
    r = np.flatnonzero(hits)               # 0-based ranks after deletion
    k = np.arange(n_pos)                   # positives ranked above each one
    p_before = np.where(r == 0, 1.0, k / np.maximum(r, 1))
    p_at = (k + 1) / (r + 1)
    return np.sum((p_before + p_at) / 2) / n_pos


def precision_at(labels, positive, ignore, K=10):
    """mP@K as revisitop computes it: denominator min(K, rank of last positive)."""
    keep = ~np.isin(labels, ignore)
    r = np.flatnonzero(np.isin(labels[keep], positive)) + 1
    kq = min(r.max(), K)
    return np.sum(r <= kq) / kq


# Which labels count as positive and which are ignored in each protocol.
ORIGINAL = dict(positive=["good", "ok"], ignore=["junk"])          # compute_ap.cpp
EASY = dict(positive=["easy"], ignore=["unclear", "hard"])         # revisitop
MEDIUM = dict(positive=["easy", "hard"], ignore=["unclear"])
HARD = dict(positive=["hard"], ignore=["unclear", "easy"])

# 1. One short ranked list: what deleting junk does.
ranked = np.array(["good", "junk", "junk", "ok", "absent", "good", "junk", "absent", "ok"])
print("official (junk ignored):  AP = %.3f" % trapezoid_ap(ranked, **ORIGINAL))
print("junk counted as negative: AP = %.3f" % trapezoid_ap(ranked, ["good", "ok"], []))
print("junk counted as positive: AP = %.3f" % trapezoid_ap(ranked, ["good", "ok", "junk"], []))

# 2. A synthetic benchmark: 11 landmarks x 5 queries, 5,000 database images.
# Each image has a "true" relation to a landmark (easy, hard, unclear, negative);
# the old annotation saw it through a coarser, noisier lens ("good" stands for Good or OK).
OLD_GIVEN_TRUE = {          # P(good, junk, absent | true relation); our choice
    "easy":     [0.97, 0.02, 0.01],
    "hard":     [0.20, 0.70, 0.10],
    "unclear":  [0.20, 0.50, 0.30],
}
MEANS = {"A": {"easy": 3.0, "hard": 1.0, "unclear": 0.8},     # strong on easy views
         "B": {"easy": 2.6, "hard": 2.0, "unclear": 0.8}}     # better on hard views


def benchmark(seed, n_db=5000, n_landmarks=11, n_queries=5):
    rng = np.random.default_rng(seed)
    rows = {s: [] for s in "AB"}
    for _ in range(n_landmarks):
        true = np.full(n_db, "negative", dtype=object)
        n = {"easy": rng.integers(10, 80), "hard": rng.integers(5, 60),
             "unclear": rng.integers(3, 20)}
        idx = rng.permutation(n_db)
        start = 0
        for lab, m in n.items():
            true[idx[start:start + m]] = lab
            start += m
        old = np.full(n_db, "absent", dtype=object)
        for lab, p in OLD_GIVEN_TRUE.items():
            sel = true == lab
            old[sel] = rng.choice(["good", "junk", "absent"], size=sel.sum(), p=p)
        old[(true == "negative") & (rng.random(n_db) < 2e-4)] = "good"  # false positives
        for _ in range(n_queries):
            noise = rng.normal(size=n_db)
            for s in "AB":
                mu = np.array([MEANS[s].get(t, 0.0) for t in true])
                order = np.argsort(-(mu + noise))
                rows[s].append([trapezoid_ap(old[order], **ORIGINAL),
                                trapezoid_ap(old[order], ["good", "ok"], []),
                                *[trapezoid_ap(true[order], **P) for P in (EASY, MEDIUM, HARD)],
                                precision_at(true[order], **HARD)])
    return {s: np.nanmean(np.array(r), axis=0) for s, r in rows.items()}


cols = ["Oxford", "Ox junk=neg", "Easy", "Medium", "Hard", "Hard mP@10"]
res = {s: np.array([benchmark(seed)[s] for seed in range(10)]) for s in "AB"}
print("\nmean over 10 seeds (min-max), x100")
print("%-6s" % "" + "".join("%20s" % c for c in cols))
for name, v in [("A", res["A"]), ("B", res["B"]), ("B - A", res["B"] - res["A"])]:
    m, lo, hi = 100 * v.mean(0), 100 * v.min(0), 100 * v.max(0)
    print("%-6s" % name + "".join("%7.1f [%5.1f,%5.1f]" % t for t in zip(m, lo, hi)))

Output (NumPy 2.3.5):

official (junk ignored):  AP = 0.835
junk counted as negative: AP = 0.569
junk counted as positive: AP = 0.918

mean over 10 seeds (min-max), x100
                    Oxford         Ox junk=neg                Easy              Medium                Hard          Hard mP@10
A        56.4 [ 52.4, 60.0]   55.1 [ 51.1, 58.9]   66.6 [ 63.8, 68.9]   47.2 [ 42.8, 53.0]    3.3 [  2.7,  4.2]    8.3 [  4.5, 10.9]
B        46.1 [ 42.7, 48.9]   41.2 [ 37.5, 45.4]   50.9 [ 47.7, 53.4]   48.8 [ 45.8, 50.2]   20.9 [ 17.8, 24.5]   41.1 [ 32.5, 51.5]
B - A   -10.3 [-11.9, -9.2]  -13.8 [-15.0,-12.5]  -15.7 [-16.1,-15.3]    1.6 [ -2.8,  5.3]   17.5 [ 15.1, 20.4]   32.8 [ 26.4, 41.5]

In the short list, the three Junk images sit among the positives. Deleting them, as the official code does, moves every later positive up the list. Counting them as negatives lowers AP by about 0.27, and counting them as positives raises it by 0.08, so the rule for borderline images alone changes a single query’s score substantially.

On the synthetic benchmark the two systems swap places depending on the protocol. The original labels and the Easy setup prefer A by 9 to 16 points in every seed, because they count A’s strength, the easy views, and ignore or barely count B’s, the hard views, which the old annotation mostly put in Junk. Medium mixes the two kinds of view and leaves the systems within a few points of each other, with the sign of the difference changing between seeds. Hard, which ignores the easy views, prefers B by 15 to 20 points, and the gap in mP@10 is larger still. Counting Junk as negative widens A’s lead under the old labels: B retrieves more of the hard views, and those now count as false positives. Which system is better is a question about what the benchmark should count as relevant, and the original annotation answered it in a way that hides progress on difficult views. This is the argument Radenović et al. made with real methods. In their Table 3 a fine-tuned ResNet-101 with GeM pooling scored 87.8 mAP on the original Oxford and 84.8, 64.7 and 38.5 in the Easy, Medium and Hard setups on ROxford. The synthetic magnitudes, including the Oxford column falling below Easy because of the injected label noise, are not those of the real datasets.

We checked both functions against the official code. On 40 random queries over 300 images, compute_map from revisitop (commit be39832, run with NumPy 2.3.5) gave the same per-query AP and mP@1, mP@5 and mP@10 in all three setups, up to floating-point rounding, including skipping a query with no Hard images. compute_ap.cpp, compiled and run on Good, OK and Junk files for 40 random queries, agreed to within $5 \times 10^{-7}$; it accumulates in single precision.

License

As of October 2026, the Oxford and Paris pages state that the datasets consist of images provided by Flickr and that use of these images must respect Flickr’s Terms and Conditions of Use. In addition, use of the datasets must follow VGG’s Dataset Terms of Access. These terms cover the “content”, defined as the images and videos VGG curates, and grant researchers a non-exclusive, non-transferable licence to use it free of charge strictly for non-commercial research, for 12 months, after which the content must be deleted. They forbid making the content available to third parties except as strictly necessary, and attempting to identify any living or deceased individual from it. Neither page names a license for the ground-truth files or the evaluation code, so their terms are unclear. Both pages now state that the image download links are no longer available from the site; the ground-truth files are still listed, and the Oxford page also lists compute_ap.cpp and precomputed features.

The revisitop repository has no license file, and the project page names no license for the revisited annotations. The R1M images come from YFCC100M, whose photographs were published on Flickr under Creative Commons licenses, commercial or non-commercial depending on the photo (Thomee et al., 2016). Each R1M image keeps its own license, and the project page states none for the set as a whole.

INRIA Holidays is different: its official page (archived September 2022; the original address returns 404 as of October 2026) states that INRIA is the copyright holder of all its images, provides the data “as is” without warranty, and asks users to cite Jégou et al. (2008). The evaluation package is marked GPL.

Limitations and Bias

• Annotation errors. Radenović et al. found false positives and false negatives in the original labels and showed extreme examples (their Fig. 2), such as Absent images that clearly show the landmark.
• Junk hides difficulty. Images with little of the landmark visible, or with severe occlusion or distortion, were removed from scoring. By the standards of 2018 many of them were views a good method should retrieve (see the Junk-to-Hard counts above), and a method that found them earned nothing.
• Shared ground truth across different views. One list of labels per landmark ignored that some queries showed another side of a building or night instead of day.
• Saturation. By 2012 bag-of-words systems reported above 0.9 mAP on Oxford 5K (see Historical Milestones).
• Uncleaned distractors. Oxford 100K was not manually cleaned and contains images of the Oxford and Paris landmarks (Radenović et al., Fig. 4), so a correct result can count as a false positive. Radenović et al. also note that some distractor sets in use are biased because their resolution differs from that of the test images; the 1M set of 2007, for example, is at 500 × 333 pixels.
• Small query sets. With 55 or 70 queries, and 11 to 26 query groups, differences of a point or two can come from a few queries; Radenović et al. show per-query AP (their Figs. 6 and 7). See statistical significance in evaluation.
• Learning on the test set. Results with vocabularies or whitening learned on the test images are not comparable with results learned elsewhere, and papers have not always said which they report; Arandjelović and Zisserman’s 2012 results show how large the gap can be (see Historical Milestones). The general risk is discussed in benchmark overfitting.
• Geographic and subject scope. Two European cities, mostly building exteriors photographed by tourists and uploaded to Flickr. The benchmarks test viewpoint, scale, lighting and occlusion on architecture, not other object types, and say nothing about retrieval of people or products.
• People in the images. Tourist photographs include passers-by. VGG’s privacy notice (as of October 2026) states that, because of how its datasets were built, the people in them could not be notified, citing the exemption in Article 14(5)(b) of the UK GDPR for cases where notice is impossible or disproportionate, in particular in research, and gives an email address for copyright holders and people whose privacy is affected to request removal. Both pages list face annotations (face-annotations-v1.json). The revisitop README (June 2022) says its download scripts were switched to face-blurred versions of the Oxford and Paris images at the suggestion of the original dataset owners, quoting them as finding a negligible loss of accuracy for benchmarking; we found no paper reporting that experiment.

Common Uses

• The standard evaluation of instance retrieval from 2007 to about 2018: bag-of-words vocabularies, spatial verification and query expansion, aggregated descriptors such as VLAD, and the first CNN descriptors (embeddings), usually as mAP on Oxford 5K, Oxford 105K and Paris 6K.
• ROxford and RParis, Medium and Hard, with and without R1M, as the main successors, reported with mAP and often mP@10.

Historical Milestones

Results are mAP on Oxford 5K under the original protocol unless stated, as reported by their authors; the original papers gave mAP between 0 and 1, later ones in percent.

• 2007, the benchmark. Philbin et al. reached 0.618 with a one-million-word vocabulary built by approximate k-means on the Oxford images and 0.647 with spatial re-ranking (their Table 4; Table 5 gives 0.645 for the same setting). Adding the 100K and 1M distractors lowered the scores to 0.490 and 0.393 (0.541 and 0.465 with re-ranking).
• 2008, soft assignment and an independent vocabulary. Philbin et al. assigned each descriptor to several nearby words. With spatial verification and query expansion, soft assignment raised mAP over hard assignment from 0.801 to 0.825 with an Oxford vocabulary and from 0.654 to 0.718 with a Paris vocabulary (their Table 5). The gap between the two vocabularies is the cost of not training on the test city.
• 2008, Hamming embedding. Jégou et al., with all parameters learned on Flickr60K, reported up to 0.547 on Oxford and 0.751 on Holidays with Hamming embedding and weak geometric consistency (their Table 2, 20,000 words, best Hamming threshold for each).
• 2012, above 0.9. Arandjelović and Zisserman combined RootSIFT, database-side augmentation and discriminative query expansion. With a vocabulary learned on the test dataset they reported 0.929 on Oxford 5K, 0.891 on Oxford 105K and 0.910 on Paris 6K; with an independent vocabulary, 0.809, 0.722 and 0.765 (their Table 3).
• 2014–2016, CNN descriptors. Babenko et al. (2014) reported 0.435 for the 4,096-dimensional layer-6 neural code of an ImageNet network and 0.545 after retraining it on landmark photographs (their Table 1). Tolias et al. (2016) reported 66.9 on Oxford 5K and 83.0 on Paris 6K for 512-dimensional R-MAC vectors from VGG16 with PCA learned on the other city (their Table 2).
• 2018, the revisit. On the new protocols the best methods in Radenović et al.’s evaluation, which start diffusion from images verified with local features, reached 79.1 and 80.2 mAP on ROxford Medium but at most 48.7 on ROxford Hard with R1M (their Table 5), against 90.0 on the original Oxford for GeM descriptors with diffusion (their Table 3). The authors concluded that image retrieval was far from solved, especially at large scale and under difficult viewing conditions.

Related Datasets

INRIA Holidays. Jégou, Douze and Schmid (2008) collected 1,491 high-resolution photographs, mostly their own holiday photos plus images taken on purpose to test robustness to rotations, viewpoint and illumination changes and blur. The images form 500 groups, each showing one scene or object; the first image of each group is the query and the other 991 images are the correct results. The protocol, from the official page and the evaluation package (holidays_map.py, 2008):

• mAP is averaged over all 500 queries, and the script refuses result files that omit any query.
• The query image, if returned, is ignored, “junk” as in the Oxford software, neither a true nor a false positive.
• AP uses the same trapezoids as compute_ap.cpp; the official page describes the package as inspired by the Oxford software.
• The authors learned all parameters, such as the vocabulary, on a distinct dataset and supplied Flickr60K descriptors for that purpose.

Holidays groups are small, which Radenović et al. contrasted with Oxford and Paris, where queries have up to hundreds of positives. Some images are rotated by 90 degrees. Jégou et al. handled this with an orientation prior that allowed quarter turns. Babenko et al. (2014), following several earlier works, manually rotated all images to their natural orientation and found results about 0.03 mAP lower in most settings on the unrotated version, so the two versions are not comparable. In our reading, rotating the images also removes part of what some of the photographs were taken to test.

UKBench. The University of Kentucky benchmark of Nistér and Stewénius (2006), in the version Philbin et al. (2007) and Babenko et al. (2014) used, has 10,200 images of 2,550 objects, four per object, each image used as a query. It is scored by the average number of the object’s four images among the top four results, from 0 to 4. The image retrieval article describes its versions and scoring.

Other landmark benchmarks. Google Landmarks Dataset v2, with millions of images, a long-tailed set of landmarks and mostly non-landmark queries, is described in image retrieval. Its scale addresses the size problem that R1M addresses for Oxford and Paris.

Resources

• Oxford Buildings page: ground-truth files (gt_files_170407.tgz), compute_ap.cpp, precomputed SIFT features and visual words for Oxford 5K, face annotations and the terms.
• Paris page: ground-truth files in the Oxford format, the list of corrupted images and face annotations.
• Revisiting Oxford and Paris: the revisited annotations with the 70 query boxes (MATLAB and Python), the R1M image lists and images.
• revisitop on GitHub: MATLAB and Python code to load, crop and evaluate, with example features and their expected scores.
• INRIA Holidays page (archived) and its evaluation package (archived).