Training, Validation, and Test Sets

A model is trained to do well on images it has never seen, so its quality has to be measured on images it has never seen. The standard way to arrange this is to divide the labelled data into three disjoint parts: a training set that fits the model’s parameters, a validation set that guides the choices made around training, and a test set that is used once, at the end, to report how well the finished model generalizes. Keeping the division honest is the hard part: near-duplicate images, frames of one video, statistics computed over all the data and repeated peeking at the test score all let information about the test images into the model, and the reported number then overstates what it will do on new data. The training workflow article shows where the split sits in a training run; this article covers the split itself, how large each part should be, and the ways it leaks.

Definition

The three sets have three jobs, fit, select and report:

• Training set: the images whose loss the optimizer minimizes; the model’s weights are fitted to them.
• Validation set (also called the development or held-out set): images the optimizer never sees, used to estimate performance while choices are still being made: hyperparameters, architecture, augmentation, the stopping epoch. Using it is model selection.
• Test set: images used for nothing that shapes the model, scored once to estimate the finished model’s error on new data from the same source. Using it is model assessment.

Hastie, Tibshirani and Friedman (2009, §7.2) name the two goals model selection and model assessment, and recommend for a data-rich situation a random division into the three sets, with the test set “kept in a vault” until the analysis ends. Goodfellow, Bengio and Courville (2016, §5.3) give the reason the validation set exists: hyperparameters that control capacity, such as a polynomial’s degree or the weight-decay strength, cannot be learned on the training set, because the training error always prefers more capacity. Since the test set may not inform any choice, hyperparameters included, they always build the validation set from the training data.

Intuition

A class that sees last year’s exam while studying can score well without learning the subject. The training set is the study material, the validation set is the practice exams used to decide how to study, and the test set is the final exam, sealed until the end. Practice-exam scores flatter the student slightly, because the study plan was chosen to do well on them; the final exam is fair only if nothing from it leaked into the study material, and only the first time it is sat.

Formal Definition

Let images and labels $(\mathbf{x}, y)$ come from a distribution $\mathcal{D}$, and let $\hat{f}$ be a trained classifier. What we want is its population error, the probability that it mislabels a new image from $\mathcal{D}$. A test set $S = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ estimates it by the test error

$$\hat{e}_S = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}\big[\hat{f}(\mathbf{x}_i) \neq y_i\big].$$

This is an unbiased estimate of the population error when two conditions hold: the test images are drawn from $\mathcal{D}$, the distribution the claim is about, and $\hat{f}$ does not depend on them. Recht et al. (2019) use the same setup and call the bias that appears when the second condition fails, because models were adapted to $S$, the adaptivity gap. Leakage breaks the first or second condition through the data; tuning on the test set breaks the second through the experimenter.

When, in addition, the test images are independent of one another, the number of errors is binomial and the standard error of an accuracy $a$ on $n$ test images is $\sqrt{a(1-a)/n}$. Inverting it gives the test-set size needed for a 95% interval of half-width $h$ (our arithmetic):

$$n \approx \frac{1.96^2\, a(1-a)}{h^2}.$$

At $a = 0.9$, a half-width of 1 percentage point needs about 3,460 test images and half a point about 13,800; at $a = 0.99$ the same half-widths need about 380 and 1,520, because errors are rarer.

When test images come in correlated groups (frames of one video, crops of one photograph, scans of one patient), the formula with $n$ images is wrong. The errors within a group are correlated, so the effective sample size lies between the number of groups and the number of images, and the binomial standard error understates the uncertainty (our reasoning, demonstrated in the example below).

Properties

How large each part should be

Hastie et al. (2009, §7.2) say no general rule exists, since the right sizes depend on the signal-to-noise ratio and the training-set size, and give 50% training, 25% validation and 25% test as a typical split. Goodfellow et al. (2016, §5.3) describe using about 80% of the training data for training and 20% for validation. In our reading, both are proportions for small and medium datasets, and the formula above suggests reasoning in absolute numbers instead: the test set must be large enough that its standard error is small next to the differences it is meant to detect, and everything beyond that is better spent on training. A million-image dataset needs no 25% test set; a few thousand images may need a larger share, or cross-validation, which rotates the held-out part through the data so every image is used for both fitting and evaluation.

The validation set has a second, subtler error. It is used to pick the best of many configurations, and the winner’s validation score is the maximum of several noisy estimates, so it is optimistic. Goodfellow et al. note that the validation error therefore underestimates the generalization error, though typically by less than the training error does; this is why the selected model is scored once more on the test set.

Stratification

A random split of a small dataset can, by chance, leave a rare class under-represented in the validation or test set, which makes its per-class scores noisy and shifts the overall accuracy. A stratified split samples each class separately in the desired proportions, so every set has the same class frequencies as the whole. It is the usual default for classification (the accuracy article’s example uses scikit-learn’s stratify option). Stratification only balances labels; it does nothing about the leakage below, and with group splits it has to be done at the level of groups.

The test set is used once

Every decision informed by the test score (keeping one model over another, stopping a sweep, changing an augmentation because the test number dropped) makes the final model depend on the test images, and the reported score is then biased upward. Hastie et al. warn that choosing the model with the smallest test error makes its test error underestimate the true error, sometimes by a large margin. Goodfellow et al. extend the point to a research community: when one test set is used for years and many groups try to beat the state of the art on it, evaluations become optimistic and benchmarks go stale. Recht et al. (2019) tested this for ImageNet by collecting a new test set with the original procedure: accuracies dropped by 11–14 percentage points, yet the models’ order was almost exactly preserved, and they attributed the drop to a distribution gap, slightly harder images, rather than to adaptive overfitting. Test-set reuse across a community, and the replication studies that measure it, are the subject of benchmark overfitting.

Data Leakage in Vision

Kapoor and Narayanan (2023) define leakage as a spurious relationship between the inputs and the target created by how the data were collected, sampled or pre-processed, one that will not exist in the data the model is meant for; it usually inflates measured performance. Their taxonomy, built from 22 reviews in 17 fields that together found leakage in 294 papers, starts with failures to keep the training and test data separate (no test set, pre-processing or feature selection on all data, duplicates) and continues with illegitimate features, temporal leakage, non-independence between training and test samples, and sampling bias. Images are prone to several of these.

Duplicates and near-duplicates

Datasets built by web search can contain the same photograph several times, resized, recompressed, cropped or colour-shifted, and a random split puts copies on both sides. Barz and Denzler (2020) searched the CIFAR test sets for near-duplicates by nearest-neighbour retrieval in a CNN’s feature space followed by manual inspection. In CIFAR-10 they found 286 test images with a duplicate in the training set and 39 more duplicated within the test set, 3.25% of the test set in all; in CIFAR-100, 891 and 104, or 10%. (Their abstract describes all of these as duplicates in the training set; the counts in §3.3 separate the two kinds.) On the test images with training duplicates, ResNet-110 made 0% and 2.9% errors, against 5.3% and 26.1% on the full test sets. After replacing the duplicates with new images (the ciFAIR test sets), errors rose by 0.42 points on CIFAR-10 and 2.74 points on CIFAR-100 on average over the six architectures of their Table 2 (our mean of the table’s gaps; the text gives 0.41 and 2.73), and the ranking of the architectures barely changed. Almost all were near-duplicates rather than exact copies, so pixel matching would have missed them. The COCO authors removed near-duplicates across splits when building the dataset (Lin et al., 2015, extended version). The same problem appears between datasets when a network is pretrained on one and tested on another, as the transfer learning article describes for ImageNet and PASCAL VOC.

The same subject, scene, video or camera

Images of the same object instance, person, place or recording share appearance a model can memorize: the background, the lighting, the sensor’s noise, the person’s clothes. If such images land on both sides of the split, the test measures recognition of the instance rather than generalization to new ones. The fix is a group split (also called a grouped or subject-wise split): assign each group as a whole to one set. Benchmarks do this whenever the group structure is known:

• Video frames. Consecutive frames are near-duplicates. UCF101’s official splits keep all clips cut from one long video on the same side, and its page warns that splitting them inflates accuracy (UCF101).
• People. Market-1501, a person re-identification benchmark of 1,501 identities, divides them into 750 training and 751 test identities (Zheng et al., 2015), so a model is tested on people it has never seen.
• Scenes and cities. Cityscapes is split at the city level, a city lying entirely in one split (Cordts et al., 2016). For monocular depth, Eigen et al. (2014) split the KITTI raw recordings by scene, 28 scenes for training and 28 for testing, and trained on NYU Depth v2 raw frames from the training scenes only (monocular depth estimation, NYU Depth v2).
• Photographers. COCO grouped images by photographer and date taken before splitting (Lin et al., 2015, extended version).
• Patients. Kapoor and Narayanan cite a histopathology study whose training and test sets contained different observations of the same patient, so a claim about new patients was tested on old ones.

The right grouping depends on the claim being made: a face verifier for new people needs identity-disjoint splits, while a system that will only ever see the same enrolled people may legitimately be tested on new photographs of them (our reasoning, following Kapoor and Narayanan’s point that non-independence is leakage unless the claim concerns data with the same dependence).

Time

A model deployed on future images should be tested on images captured after its training images. Kapoor and Narayanan call a test set containing data from before the training data temporal leakage, because the model is built with data from after the period it is tested on. In vision, a random split of images collected over months spreads each period’s lighting, season, camera firmware and scene changes over all three sets, while a split by capture date shows how much performance decays as conditions drift (our reasoning).

Pre-processing fitted on all the data

Any statistic computed before splitting carries information from the held-out images into training. Hastie et al. (2009, §7.10.2) show the extreme case: with 50 samples, 5,000 predictors independent of the labels and a true error of 50%, choosing the 100 predictors most correlated with the labels on all the data and then cross-validating a nearest-neighbour classifier gave an average estimated error of 3%. The fix is to fit every step that uses labels on the training part only. They add a qualification: a screening step that ignores the labels, such as keeping the highest-variance predictors, may be done on all samples. For images, the common label-free steps are per-channel normalization constants and a PCA or whitening basis; in our view they too are best fitted on the training set, since the test set stands in for data that do not exist yet, although their bias is usually small. Label-using steps such as feature selection or rebalancing by oversampling must never see the held-out data: Kapoor and Narayanan note that oversampling before splitting places copies of training samples in the test set.

Augmentation and evaluation data

Data augmentation belongs to training. Augmenting before the split is the duplicate problem in another form, since flipped or cropped copies of one image land on both sides; augmenting validation or test images at random makes their scores vary from run to run. Test-time augmentation is not a leak: it is part of the inference procedure, and is reported as such.

Benchmarks with Hidden Test Labels

Many vision benchmarks enforce the test set’s vault themselves: they release the test images without labels and score submissions on an evaluation server that limits how often a team may submit. ImageNet has released only the 2010 test labels; COCO withholds test-dev and test-challenge annotations and asks papers to report test-dev; Cityscapes withholds its 1,525 test images’ annotations (Cordts et al., 2016); KITTI withholds every benchmark’s test ground truth and, under its policy as of October 2026, accepts only work aimed at publication.

Because servers are rationed, papers routinely report the public validation split, and on ImageNet the 50,000 validation images serve as the de facto test set: Recht et al. (2019) replicated the ImageNet validation set because most papers developed and tested models on it, and use “validation set” and “test set” interchangeably for ImageNet. Two consequences follow. First, a paper that reports validation numbers needs its own held-out data for model selection, usually carved from the training set; otherwise it selects and reports on the same images. Second, papers carve those splits differently, so “validation” numbers from different papers are not always comparable, as the KITTI article notes. Following this site’s convention, articles say when a paper reports a benchmark’s validation split as its test result.

Examples

The script builds synthetic “videos” of 16×16 grayscale images with floating-point intensities centred near 0 (shape contrast 0.4 to 1.0, noise standard deviation 0.3): each video shows one of four shapes (the class) over its own smooth random background, and its ten frames differ only by a one-pixel camera shake and sensor noise. It evaluates a 1-nearest-neighbour classifier, which memorizes every training frame, and a nearest-centroid classifier, which keeps one mean template per class, under two 80/20 splits of 200 videos: a leaky split that picks test frames at random, and a group split that holds out whole videos. Both are compared with accuracy on 2,000 new videos, which stands in for the deployment data. For one seed it also computes the test accuracy’s standard error two ways: the binomial formula with frames as independent images, and the spread of per-video accuracies. It runs in about ten seconds with NumPy 2.3.5 and SciPy 1.16.3.

import numpy as np
from scipy.ndimage import gaussian_filter, shift

K, S, FRAMES = 4, 16, 10                     # classes, image size (pixels), frames per video

# One 16x16 shape per class: horizontal bar, vertical bar, square ring, diagonal stripe.
T = np.zeros((K, S, S))
T[0, 6:10, 2:14] = 1
T[1, 2:14, 6:10] = 1
T[2, 3:13, 3:13] = 1; T[2, 5:11, 5:11] = 0
for i in range(2, 14):
    T[3, i, i - 2:i + 2] = 1

def make_videos(n_per_class, rng):
    """Synthetic 'videos': one object over a video-specific textured background.
    Frames of a video differ only by a one-pixel camera jitter and sensor noise."""
    X, y, vid = [], [], []
    for c in range(K):
        for _ in range(n_per_class):
            bg = 1.5 * gaussian_filter(rng.normal(size=(S + 1, S + 1)), 1.5)
            obj = rng.uniform(0.4, 1.0) * shift(T[c], rng.integers(-2, 3, size=2), order=0)
            for _ in range(FRAMES):
                oy, ox = rng.integers(0, 2, size=2)
                X.append(bg[oy:oy + S, ox:ox + S] + obj + rng.normal(scale=0.3, size=(S, S)))
                y.append(c)
                vid.append(len(vid) // FRAMES)
    return np.array(X).reshape(-1, S * S), np.array(y), np.array(vid)

def sqdist(A, B):
    return (A ** 2).sum(1)[:, None] - 2 * A @ B.T + (B ** 2).sum(1)[None]

def nearest_neighbour(Xtr, ytr, Xte):                 # memorizes every training frame
    return ytr[sqdist(Xte, Xtr).argmin(1)]

def nearest_centroid(Xtr, ytr, Xte):                  # one mean template per class
    M = np.stack([Xtr[ytr == c].mean(0) for c in range(K)])
    return sqdist(Xte, M).argmin(1)

res = {m: [] for m in ("1-NN", "centroid")}
for seed in range(20):
    rng = np.random.default_rng(seed)
    X, y, vid = make_videos(50, rng)                  # 200 videos, 2,000 frames
    Xf, yf, _ = make_videos(500, rng)                 # 2,000 new videos: the "deployment" data
    n, V = len(y), vid.max() + 1
    leaky_te = np.isin(np.arange(n), rng.permutation(n)[: n // 5])   # 20% of frames at random
    group_te = np.isin(vid, rng.permutation(V)[: V // 5])           # 20% of videos, all frames
    for name, clf in (("1-NN", nearest_neighbour), ("centroid", nearest_centroid)):
        acc_leaky = np.mean(clf(X[~leaky_te], y[~leaky_te], X[leaky_te]) == y[leaky_te])
        hit = clf(X[~group_te], y[~group_te], X[group_te]) == y[group_te]
        acc_new = np.mean(clf(X[~group_te], y[~group_te], Xf) == yf)
        res[name].append((acc_leaky, hit.mean(), acc_new))
        if seed == 0 and name == "1-NN":
            a, m = hit.mean(), len(hit)
            per_video = hit.reshape(-1, FRAMES).mean(1)               # accuracy of each test video
            print(f"seed 0, 1-NN, group split: accuracy {a:.3f} on {m} frames from {len(per_video)} videos")
            print(f"  binomial SE treating frames as independent: {np.sqrt(a * (1 - a) / m):.3f}")
            print(f"  SE from per-video accuracies:               {per_video.std(ddof=1) / np.sqrt(len(per_video)):.3f}")

print("\n20 seeds   leaky split  group split  new videos  | leaky - new  group - new (sd)")
for name, r in res.items():
    r = np.array(r)
    lo, hi = r.min(0), r.max(0)
    d = r[:, 1] - r[:, 2]
    print(f"{name:9s}  " + "  ".join(f"{lo[j]:.3f}-{hi[j]:.3f}" for j in range(3))
          + f" |   {np.mean(r[:, 0] - r[:, 2]):+.3f}       {d.mean():+.3f} ({d.std(ddof=1):.3f})")

Output:

seed 0, 1-NN, group split: accuracy 0.657 on 400 frames from 40 videos
  binomial SE treating frames as independent: 0.024
  SE from per-video accuracies:               0.057

20 seeds   leaky split  group split  new videos  | leaky - new  group - new (sd)
1-NN       1.000-1.000  0.557-0.828  0.679-0.763 |   +0.277       -0.006 (0.066)
centroid   0.708-0.802  0.472-0.838  0.667-0.712 |   +0.066       -0.007 (0.082)

Under the leaky split the nearest-neighbour classifier is perfect in all 20 runs: every test frame has a sibling from the same video in the training set, closer than any frame of another video, so the classifier matches backgrounds rather than recognizing shapes. On new videos the same classifier is right about seven times in ten. The group split recovers that number: averaged over seeds, its estimate is within one point of the new-video accuracy. The nearest-centroid classifier is inflated far less, by about 7 points on average against about 28, because averaging a class’s frames blurs out any single video’s background; leakage rewards memorization most, which is why Barz and Denzler worried about high-capacity models.

The group split is unbiased but noisy. Its estimates range over about 27 points for 1-NN and 37 for the centroid classifier, and the seed-to-seed spread of its error against the new-video accuracy is close to the per-video standard error of the seed shown, not to the binomial one, which is less than half as large. The ten frames of a video are mostly right or wrong together: by our arithmetic, $a(1-a)/\text{SE}^2$ with the per-video standard error puts the 400 test frames of seed 0 at the information of about 70 independent images. Reporting the binomial interval for frame-level accuracy would claim more than twice the precision the test set has.

Common Misconceptions

• “A random split guarantees a fair test.” Only if the images are independent; the example’s random frame split reports 100% for a classifier that is right about 70% of the time on new videos.
• “The validation score is the model’s accuracy.” It belongs to the configuration that won a selection on those very images, so it is optimistic.
• “A good test score means the model works in deployment.” The test set measures performance on the distribution it was drawn from. A model tested on one hospital’s or one city’s images may fail on another’s; that is a different question, of distribution or domain shift, which a held-out split from the same source cannot answer. Kapoor and Narayanan cite pneumonia models trained on one hospital’s images that did not generalize to another hospital’s.

Where It Is Used

Every learned vision model is developed with some version of this split, from image classification on ImageNet and CIFAR to detection on COCO, action recognition on UCF101 and depth estimation on KITTI. Early stopping, hyperparameter search and the choice of regularization strength run on the validation set, and every metric, from accuracy to average precision, is only as trustworthy as the test set it is computed on. When a single split is too small, cross-validation reuses the data; when differences between models are small, the test-set uncertainty calls for statistical significance testing.