Benchmarks and Leaderboards

A benchmark is a fixed way of measuring how well a method solves a vision task, so that methods from different groups and different years can be compared on the same footing. ImageNet’s classification challenge, PASCAL VOC and COCO for detection, KITTI for driving scenes, Cityscapes for street-scene segmentation and MOTChallenge for tracking each fix a task, a set of images with annotations, a protocol saying how predictions are matched and scored, and a metric. Most of them also keep the test answers away from the people being tested: they release the test images without labels, score submissions on an evaluation server and ration how often a team may ask. Around these sit annual challenges with workshops, and leaderboards that rank every submitted method. This article describes how those parts work, using the published rules of these benchmarks, and the ways benchmark scores mislead. Two of those failure modes are summarized briefly: benchmark overfitting, the inflation of scores when a community reuses one test set, and the noise in a score measured on a finite test set, which statistical significance in evaluation treats in full. How a single study divides its data is the subject of training, validation and test sets.

Definition

A benchmark has four parts:

• A task: what a method must output for each input, such as one label per image, a set of scored boxes, a class per pixel or a set of tracks per video.
• Data: images or videos with reference annotations, divided into splits. Training and validation annotations are normally public; the test split is the one that produces the reported score.
• A protocol: the rules that turn predictions into a score. They state which training data are allowed, how predictions are matched to the reference (an intersection-over-union threshold, a set of ignored regions, a maximum number of detections), which subsets are scored and how results are averaged.
• A metric: the number the protocol produces, such as top-5 error, average precision or accuracy.

The PASCAL VOC organizers described their challenge as two components: a public dataset of images and annotation with standardized evaluation software, and an annual competition with a workshop. They gave each a goal: the dataset provides a ready-made harness for training and testing so that algorithms can be compared, and the competition measures the state of the art each year (Everingham et al., 2010, §1). ILSVRC adopted the same two-part design (Russakovsky et al., 2015, §1). The retrospective on VOC adds that what made results comparable across papers was providing the splits, the protocol and the evaluation code together, and recommends that every challenge supply code for its standard assessment (Everingham et al., 2015, §7.1.1).

Four terms describe how the test data are run:

• A held-out test set is kept apart from training and model selection. Its images, and sometimes its annotations, may be public.
• A test set with withheld annotations releases the images but not the labels; only the organizers can score it.
• A sequestered test set is not released at all: participants submit software, and the organizers run it on data the participants never see.
• An evaluation server is the service that scores submitted predictions against withheld annotations, usually with a limit on how often each team may submit. A leaderboard is the public table of server scores, and a challenge is a time-limited competition on a benchmark, usually ending in a workshop.

Intuition

A benchmark is an examination that many candidates sit, set by examiners who want the marks to mean the same thing next year as this year. Publishing the questions in advance lets candidates practise them; publishing the answers lets them memorize the answers. So the examiners publish the syllabus and practice papers (the training and validation sets), hand out the exam questions but keep the answer key (withheld annotations), mark each script themselves and limit how many times a candidate may resit. The scoreboard then ranks the candidates. Two things still go wrong: if everyone resits often enough, the marks start to reflect the answer key rather than the subject, and if two candidates differ by less than the marking noise, their order on the scoreboard is a matter of luck.

Formal Definition

A benchmark can be written as a tuple $(\mathcal{T}, D_{\text{train}}, D_{\text{val}}, D_{\text{test}}, P, M)$: a task $\mathcal{T}$, data splits drawn from some distribution of images, a protocol $P$ and a metric $M$. A method $f$, trained under the rules of $P$ on $D_{\text{train}}$ (and possibly $D_{\text{val}}$), receives the score

$$s(f) = M\big(P(f, D_{\text{test}})\big),$$

an estimate of the score $\bar{s}(f)$ that $f$ would obtain on the whole distribution the test set was drawn from. A leaderboard over methods $f_1, \dots, f_m$ sorts them by $s(f_i)$; the rank it shows is a statement about $\bar{s}$ only if the differences between the $s(f_i)$ are larger than their sampling error.

The PASCAL VOC organizers list three ways to obtain the test score (Everingham et al., 2015, §7.1.2):

1. Release the test images and annotations, and let participants score themselves.
2. Release the test images but withhold the annotations; participants submit predictions and the organizers score them, for example on an evaluation server.
3. Release nothing; participants submit software, which the organizers run.

Each option trades the opportunity to adapt to $D_{\text{test}}$ against the cost of running the benchmark. Under option 1 a method can be tuned on $D_{\text{test}}$ directly, so $s(f)$ is optimistically biased. Under option 2 each submission still reveals one number about $D_{\text{test}}$, and a team that submits $k$ times can select among $k$ variants; a submission limit bounds $k$, and how fast the bias can grow with $k$ is the subject of benchmark overfitting. Option 3 also hides the images, which prevents inspecting or hand-labelling them, at the cost of running every participant’s software.

Properties

Held-out, withheld and sequestered test sets

The early VOC challenges released test annotations, and the organizers learned from it. Everingham et al. (2010, §4.1) report that the VOC2005 test annotations were released and “led to some ‘optimistic’ reported results”, where several parameter settings had been run on the test set and only the best reported. The retrospective names “some evidence” of the same with the VOC 2007 release as one reason for withholding test annotations from VOC 2008 onward (Everingham et al., 2015, §7.1.2). ILSVRC chose the same option, releasing test annotations only for the 2010 classification task; option 3 was judged infeasible for test sets of 40,000 to 100,000 images, and option 1 to leave too much room for overfitting to the test data (Russakovsky et al., 2015, Appendix F of arXiv v3, Appendix 6 of the journal version). Cityscapes released 2,975 training and 500 validation images with annotations and withheld the annotations of its 1,525 test images, with the split made by city so that no city appears in two splits (Cordts et al., 2016, §2.3). KITTI withholds the test ground truth of every benchmark in its suite, and MOTChallenge never released the annotations of its test sequences (Dendorfer et al., 2021).

Withholding the annotations of images does not make the test set fully secret. Because the test data are photographs, participants could in principle label them by hand or judge results by eye; the VOC organizers relied on participants’ honesty and on the short time between releasing the test images and the submission deadline (Everingham et al., 2010, §4.1). Truly sequestered data avoids this. NIST’s face recognition evaluations receive algorithms as compiled black-box libraries and run them on datasets that are sequestered at NIST, meaning that developers have no access to them for training or testing (one set of web images in that evaluation was not truly sequestered, but its composition was unknown to the developers; Grother, Ngan & Hanaoka, 2019, executive summary). The face recognition article describes these evaluations. The price is the one VOC named: the organizers must run every participant’s code and pay for its computation.

COCO’s 2015 test set showed a middle path. Its roughly 80,000 test images were divided into four splits of about 20,000 whose identities, apart from test-dev, were not revealed: test-dev for unlimited debugging, test-standard for comparisons in papers, test-challenge for the annual competition and test-reserve, whose scores were never published and served as a check: a substantial difference between a method’s test-standard and test-reserve scores would prompt an investigation. In 2017 COCO dropped test-standard and test-reserve, keeping test-dev and test-challenge of about 20,000 images each, because after several years it had seen no evidence of overfitting to particular splits; it credits the complexity of the output and the size of the test set with protecting against simple attacks such as boosting (COCO test guidelines, as of October 2026).

Evaluation servers and submission limits

A server enforces the protocol identically for everyone and keeps the annotations private. Its submission rules are the benchmark’s main defence against tuning on the test set, and each benchmark words them differently:

Benchmark	Rule	Source
PASCAL VOC	One result per method in the challenge; the server limited how often a participant could submit results for earlier years, whose test images are subsets of the current year’s	Everingham et al. (2010, §4.1; 2015, §2.4.2)
ILSVRC	At most two submissions per week to the test server	Russakovsky et al. (2015, App. F); ILSVRC page (as of October 2026)
COCO (2017 splits)	Validation: unlimited, for debugging uploads. Test-dev: 5 uploads per day, one public leaderboard entry per participant. Test-challenge: 5 uploads in total per challenge. No extra accounts to get around the limits	COCO test guidelines (as of October 2026)
KITTI	72 hours between submissions (from 28 June 2012); at most 3 submissions per month, counted per benchmark (from 26 July 2016); only novel work aimed at a peer-reviewed paper, other work evaluated on a split of the training set	KITTI home page changelog; policy on the benchmark pages, e.g. object detection (as of October 2026)
MOTChallenge	One submission per method and benchmark expected; resubmission after 72 hours, at most 4 per tracker and challenge, a new tracker every 30 days, no second account	Dendorfer et al. (2021, App. A.1 of arXiv v2)
NIST FRTE 1:1	Algorithm submissions no more often than every four calendar months	FRTE 1:1 page (as of October 2026)

The limits share a logic. A submission returns a score computed on the test set, so frequent submissions let a team select among variants by test score. MOTChallenge states the aim directly: the waiting period is meant to discourage using the server for training and parameter tuning on the test data (Dendorfer et al., 2021, App. A.1 of arXiv v2). A per-account limit is only as strong as the link between accounts and teams, hence the rules against multiple accounts at COCO and MOTChallenge; MOTChallenge also required an institutional rather than a generic e-mail address, for accountability and to prevent abuse through several accounts (Dendorfer et al., 2021, App. A). ILSVRC’s appendix stated that in practice researchers had never abused the server (Russakovsky et al., 2015, App. F); the incident described under Failure Modes was announced a few months after that version, and the submissions it concerned had begun two months before it.

Servers also shape what can be measured. ILSVRC observed that a detection method returning ten boxes for each of the 200 classes in each of its 40,000 test images would produce about 80 million detections and a 2.24 GB file, so methods are implicitly required to submit only their most confident detections (Russakovsky et al., 2015, §4.3 of arXiv v3). And because servers are rationed, most papers report the public validation split instead; on ImageNet the 50,000 validation images serve as the de facto test set, a practice the split article discusses.

Challenges and workshops

A challenge runs a benchmark on a calendar. The VOC and ILSVRC cycle was: release the training and validation data with a development kit containing the evaluation code, release the unlabelled test images, accept submissions until a deadline, about three months after the test images were released for VOC (Everingham et al., 2010, §4.1) and about four months after the data release for ILSVRC (Russakovsky et al., 2015, App. F), then present the results and the most interesting methods at a workshop. ILSVRC’s workshop was held at ICCV or ECCV in alternate years (Russakovsky et al., 2015, §1), and COCO reveals test-challenge results at its workshop (COCO test guidelines).

Challenge rules add structure that a permanent leaderboard lacks:

• Tracks by training data. VOC defined two competitions for classification, detection and person layout: one using only the VOC training and validation data and one allowing any data except the VOC test images (Everingham et al., 2010, §2.1). ILSVRC 2015 had a “provided data” track and an “external data” track, and allowed each team one result per algorithm for up to five algorithms, stating that changed parameters do not make a different algorithm (ILSVRC 2015 page).
• Disclosure. ILSVRC 2015 introduced “open” entries, expected to reveal most details of the method, and “closed” entries that need only an abstract, to let industrial teams take part (ILSVRC 2015 page).
• New tasks. VOC’s “taster” competitions, segmentation and person layout from 2007, asked for methods well ahead of the state of the art while the main competitions stayed fixed to track progress (Everingham et al., 2010, §7.1, §7.2).
• Growing data with fixed old splits. VOC added images each year while keeping earlier years’ assignments to training, validation and test, so all methods from 2009 to 2012 could be compared on the 2009 test images (Everingham et al., 2015, §7.1.3).

The workshop is also where organizers can reward something other than rank. The VOC organizers gave workshop time to particularly interesting methods as well as particularly successful ones (Everingham et al., 2010, §7.1) and later tried novelty prizes, which they found hard to judge objectively (Everingham et al., 2015, §7.2).

Leaderboards

A leaderboard turns server scores into a public ranking that lasts beyond any one challenge. The rules about who may appear on it are part of the benchmark:

• One entry per participant. COCO allows a single published test-dev result per participant on its public leaderboard, although a paper may report several (COCO test guidelines).
• Anonymous entries for review. MOTChallenge allowed anonymous results for blind review if the venue and paper ID were given, and hid them after six months of inactivity (Dendorfer et al., 2021, App. A). KITTI deletes entries that are still anonymous or have no paper after six months (KITTI policy, as of October 2026).
• Disclosure of conditions. MOTChallenge required each entry to state whether it used extra training data and whether it used the provided detections, and displayed both in the table (Dendorfer et al., 2021, App. A.1). COCO requires every submission to list any external training data (COCO test guidelines).
• Significance. ILSVRC reported bootstrap confidence intervals for the top entries of 2012–2014 (Russakovsky et al., 2015, §6.2), and the VOC organizers added a leaderboard to their server with the stated intention of adding bootstrap intervals so that differences between entries could be judged (Everingham et al., 2015, §7.3).

A leaderboard depends on its server. MOTChallenge’s website and evaluation server went offline in 2026; as of October 2026 the site returns a notice that the evaluation server, submissions and user accounts are offline, while the dataset archives and a static copy of the leaderboards, with all results published up to 16 April 2026, remain available (MOTChallenge offline notice). Because the test annotations were never released, new results on those test sets can no longer be obtained (MOTChallenge).

Failure Modes

Adaptive overfitting. Every score a server returns tells its recipient something about the test set, and a community that keeps the variants that score best can, in principle, fit the test set itself. The theory says how fast this can happen and how leaderboards can release less information per submission; replications of CIFAR-10, ImageNet and MNIST test sets and Kaggle’s public and private leaderboards found the effect on rankings small, with most of the drops on new test sets due to harder images rather than reuse. All of this is in benchmark overfitting.

Gaps smaller than the noise. A test set is a sample, so every score has a sampling error, and leaderboard neighbours often differ by less. ILSVRC’s 99.9% bootstrap intervals show both situations in one table: the 2014 classification winner, GoogLeNet, at 6.66% top-5 error (6.40–6.92) was significantly ahead of VGG at 7.32% (7.05–7.60), but MSRA at 8.06% (7.78–8.34) and AHoward at 8.11% (7.83–8.39) were 0.05 points apart with nearly identical intervals (Russakovsky et al., 2015, Table 8 of arXiv v3). Those intervals are for each entry on its own; a paired comparison of two methods on the same images is more sensitive, as statistical significance in evaluation explains with intervals, paired tests and the variance from training seeds. The example below shows how often a leaderboard’s order is right when its gaps are of this size.

Circumventing the limits. In May 2015 the ILSVRC organizers announced that a group had evaded the limit of two test-server submissions per week by creating several ImageNet accounts. Their follow-up of 2 June 2015 stated that between 28 November 2014 and 13 May 2015 a team from Baidu had used at least 30 accounts to submit at least 200 times, including more than 40 submissions in five days in March. The organizers explained why such results were not directly comparable to others: testing many slightly different solutions lets a team pick the best by test performance, gaining a small but potentially significant advantage, and lets test data rather than training and validation data guide research choices. They asked the team not to submit to the server or the challenge for 12 months, and published the team’s apology (ILSVRC announcements, 19 May and 2 June 2015).

Numbers that are not comparable. A score means something only under its protocol, and papers often change the protocol. The VOC organizers observed that where a benchmark supplies no evaluation code, papers compute the same measure differently, for example averaging over images in one paper and over classes in another (Everingham et al., 2015, §7.1.1). Papers that report a validation split as their test result, or carve their own validation splits from KITTI’s training data, produce numbers that cannot be placed on the official leaderboard. Musgrave, Belongie and Lim (2020) re-examined deep metric learning, whose papers had reported large gains over the contrastive and triplet losses: many compared methods with different backbones, embedding sizes and augmentations, and most used no validation set, checking test accuracy during training and reporting the best. Retrained under one protocol with cross-validated hyperparameters, the newer losses performed only marginally better than, and sometimes on par with, the classic ones. A paper’s “state of the art” claim is a claim about a protocol; when the protocol changed, the claim has to be read with it.

Metric fixation and lost diversity. A leaderboard rewards whatever its metric measures. The VOC retrospective calls reduced diversity of methods the biggest risk of running a competition: the safest entry is an incremental improvement on last year’s winner, and new ideas may be dropped before they mature because they do not yet beat refined old ones. It notes that the metric of each VOC competition had typically been changed or refined at least once, that methods may overfit to the chosen metric, and suggests publishing several metrics, or a measure of how different each entry is from the others (Everingham et al., 2015, §7.2). Their 2010 paper had already argued that datasets “have a shelf life”: once performance saturates, a dataset encourages fine-tuning of implementation details rather than fundamental progress, and real progress may be lost in the noise (Everingham et al., 2010, §7.1).

A test set that no longer measures the task. Near the top of an old leaderboard, label errors and ambiguous images in the test set can be as large as the gaps between entries, so a better model may not score higher; re-annotations of the ImageNet validation set and estimates of label errors in common test sets are summarized in benchmark overfitting. A test set can also be unrepresentative in ways its users do not see. Cordts et al. (2016, §2.3) checked their own split: a network trained on the validation images scored 3.8 points higher on the test images recorded at the end of the year than on the whole test set. They hypothesized softer light on cloudy autumn days as the cause and supported it with a further test: 4.5 points higher on images taken at low temperatures (cloudy) and 0.9 points lower at high ones (sunny). A model ranked first on one benchmark has been shown to be best on that benchmark’s images; whether that carries over to other cameras, places and label conventions is the question studied under dataset bias.

Examples

The script builds a leaderboard on synthetic images and asks how often its order is right. The images are 16×16 grayscale seven-segment glyphs of the digits 0–9, drawn with a random contrast between 0.6 and 1.0 on a zero background and covered by Gaussian noise of standard deviation 0.7. Twelve “entries” are template classifiers whose templates carry fixed random errors of increasing size, so their true accuracies range from about 78% to 88% and the top entries are a few tenths of a point apart, as on a mature leaderboard. A pool of 1,000,000 scored images stands in for the population: each entry’s accuracy on it is its true accuracy. Part 1 scores all entries on one test set of 2,000 images and computes 95% bootstrap ranges for their ranks, resampling test images and ranking all entries on each of 2,000 resamples, as Everingham et al. (2015, §3.5) did for VOC. Part 2 draws 500 test sets of each of four sizes from the pool and compares each leaderboard’s order with the true order. It runs in about four seconds with NumPy 2.3.5, and two runs give identical output.

import numpy as np

S = 16                                           # image size (pixels)
SEG = {"a": (2, 3, 4, 12), "b": (3, 8, 10, 12), "c": (8, 13, 10, 12), "d": (12, 13, 4, 12),
       "e": (8, 13, 4, 6), "f": (3, 8, 4, 6), "g": (7, 9, 4, 12)}   # rows r0:r1, cols c0:c1
DIGITS = ["abcdef", "bc", "abdeg", "abcdg", "bcfg", "acdfg", "acdefg", "abc", "abcdefg", "abcdfg"]
T = np.zeros((10, S, S))                         # seven-segment glyphs of the digits 0-9
for k, segs in enumerate(DIGITS):
    for s in segs:
        r0, r1, c0, c1 = SEG[s]
        T[k, r0:r1, c0:c1] = 1

def images(n, rng):
    """n grayscale digit images: glyph contrast 0.6-1.0 on a zero background, Gaussian noise sd 0.7."""
    y = rng.integers(0, 10, n)
    X = T[y] * rng.uniform(0.6, 1.0, (n, 1, 1)) + rng.normal(scale=0.7, size=(n, S, S))
    return X.reshape(n, -1), y

# Twelve leaderboard entries: template classifiers whose templates carry fixed random errors of
# growing size, so their true accuracies differ by amounts like those between real entries.
names = list("KDHAJFBLCGEI")
W = []
for i, strength in enumerate(np.linspace(0.0, 0.10, 12)):
    w = T.reshape(10, -1) + strength * np.random.default_rng(100 + i).normal(size=(10, S * S))
    w -= w.mean(1, keepdims=True)
    W.append(w / np.linalg.norm(w, axis=1, keepdims=True))

# A pool of 1,000,000 images stands in for the population; we keep only who got what right.
rng = np.random.default_rng(0)
correct = []
for _ in range(10):
    X, y = images(100_000, rng)
    correct.append(np.stack([(X @ w.T).argmax(1) == y for w in W], axis=1))
correct = np.concatenate(correct)                # shape (1,000,000, 12), boolean
true_acc = correct.mean(0)
true_rank = np.argsort(np.argsort(-true_acc)) + 1

def ranks(acc, rng):
    """Rank 1 = highest accuracy, per row; exact ties are broken at random."""
    key = -acc + 1e-9 * rng.random(acc.shape)
    return np.argsort(np.argsort(key, axis=-1), axis=-1) + 1

# ---- Part 1: one public leaderboard on a test set of 2,000 images, with bootstrap rank ranges
n = 2000
test = correct[rng.choice(len(correct), n, replace=False)]
acc = test.mean(0)
boot = rng.integers(0, n, (2000, n))             # resample test images, all entries together
boot_rank = ranks(np.stack([test[b].mean(0) for b in boot]), rng)
lo, hi = np.percentile(boot_rank, [2.5, 97.5], axis=0)
print(f"Leaderboard on {n} test images (true accuracy from 1,000,000 images)")
print("rank  entry  test acc  95% rank range  true acc  true rank")
for r, j in enumerate(np.argsort(-acc, kind="stable"), 1):
    print(f"{r:4d}  {names[j]:>5s}  {100 * acc[j]:7.2f}   {int(lo[j]):3d} - {int(hi[j]):2d}     "
          f"{100 * true_acc[j]:7.2f}  {true_rank[j]:6d}")
top = np.sort(true_acc)[::-1]
print("true gaps between neighbours, ranks 1-6 (points):", np.round(100 * -np.diff(top[:6]), 2))
b1, b2 = np.argsort(-true_acc)[:2]
d = correct[:, b1].astype(float) - correct[:, b2]
print(f"standard error at n={n}: one score {100 * np.sqrt(top[0] * (1 - top[0]) / n):.2f} points, "
      f"difference of the true top two (paired) {100 * d.std() / np.sqrt(n):.2f} points")

# ---- Part 2: 500 independent test sets of each size drawn from the pool
print("\n   n   SE of one score  P(true best ranked 1st)  mean |rank error|  entries off by >= 3 ranks")
for n in (500, 2000, 10_000, 50_000):
    first, err, far = [], [], []
    for t in range(500):
        idx = rng.choice(len(correct), n, replace=False)
        r = ranks(correct[idx].mean(0), rng)
        first.append(r[true_rank == 1][0] == 1)
        err.append(np.abs(r - true_rank).mean())
        far.append(np.sum(np.abs(r - true_rank) >= 3))
    se = np.sqrt(top[0] * (1 - top[0]) / n)
    print(f"{n:6d}   {100 * se:6.2f} points     {np.mean(first):10.2f}            {np.mean(err):6.2f}"
          f"             {np.mean(far):5.2f} (max {np.max(far)})")

Output:

Leaderboard on 2000 test images (true accuracy from 1,000,000 images)
rank  entry  test acc  95% rank range  true acc  true rank
   1      K    87.50     1 -  3       87.52       1
   2      D    87.50     1 -  3       87.40       2
   3      H    87.10     1 -  4       87.15       3
   4      A    86.35     3 -  5       86.73       4
   5      J    86.10     3 -  6       86.03       5
   6      F    85.35     5 -  6       85.44       6
   7      B    83.85     7 -  8       84.09       7
   8      L    82.70     7 -  9       83.58       8
   9      C    81.05     8 - 10       82.28       9
  10      G    79.75     9 - 11       81.01      10
  11      E    79.30    10 - 12       79.51      11
  12      I    77.60    11 - 12       78.38      12
true gaps between neighbours, ranks 1-6 (points): [0.12 0.25 0.41 0.7  0.6 ]
standard error at n=2000: one score 0.74 points, difference of the true top two (paired) 0.25 points

   n   SE of one score  P(true best ranked 1st)  mean |rank error|  entries off by >= 3 ranks
   500     1.48 points           0.36              0.77              0.48 (max 5)
  2000     0.74 points           0.60              0.33              0.03 (max 2)
 10000     0.33 points           0.87              0.05              0.00 (max 0)
 50000     0.15 points           0.99              0.00              0.00 (max 0)

This particular test set happened to put every entry in its true place, with the top two tied, but the bootstrap ranges show that it could not have known: the three leading entries share the range 1–3 or 1–4, and only the groups separated by about a point or more (ranks 1–6 against 7–12) are told apart reliably. Part 2 shows how often the order would be wrong on another test set of the same size. With 2,000 images the true best entry comes first in 60% of test sets; with 500 it is first in only about a third of them, and an average of about one entry in every two test sets lands three or more places from its true rank. The reason is visible in the gaps: the top three are separated by 0.12 and 0.25 points, while one score’s standard error at $n = 2{,}000$ is 0.74 points. The paired standard error of the difference between the top two is three times smaller, 0.25 points, because the entries are similar and fail on mostly the same images; this is why the ranking is more stable than the separate errors suggest, and why comparisons should be paired. Even so, the 0.12-point gap is half that paired error, so only around 50,000 images rank the leader correctly almost every time. Real leaderboards add what this simulation leaves out: noise from training seeds, and the selection among many submissions that benchmark overfitting describes.

Common Misconceptions

• “A server with hidden labels makes the test set unseen.” The images are seen, and every returned score carries information about the labels. Withholding annotations stops direct tuning on the labels; submission limits, single-entry rules and sequestered data address the rest, each partially.
• “First place on the leaderboard is the best method.” It is the best score on one test sample under one protocol. When the gap to second place is within the sampling error, as for many adjacent ILSVRC entries and for the leaders in the example above, another test set of the same size could reverse it.
• “Validation numbers and test numbers are interchangeable.” Many papers report a public validation split because servers are rationed; such numbers are useful, but they are not the benchmark’s official score, and papers that carve their own splits are not comparable with each other.
• “A higher number under a modified protocol is still progress on the benchmark.” Changing the backbone, the training data, the averaging or the matching threshold changes the measurement. Leaderboards that record external data and detection sources, and challenges with separate tracks, exist to keep such differences visible.
• “A benchmark is a permanent measure of a task.” Benchmarks age: their metrics are revised, their best scores approach the level of label noise, and their servers can close.

Where It Is Used

Every task in computer vision is tracked through benchmarks. Image classification progress was measured on ILSVRC and its ImageNet validation set, where the AlexNet paper and the ResNet paper reported their results; object detection on PASCAL VOC and then COCO; driving perception, stereo, flow and odometry on KITTI; urban segmentation on Cityscapes; and multi-object tracking on MOTChallenge. The dataset articles describe each benchmark’s own splits, protocol and server. Benchmarks that later replicated or stressed these test sets, such as ImageNetV2, are discussed in benchmark overfitting; reporting a result with its uncertainty is covered in statistical significance in evaluation; and the choice of a metric for each task in its metric article, such as average precision for detection.