Vision Systems & Engineering

Dataset Curation

How a training set for a vision model is assembled and maintained, from defining what it should represent through filtering, deduplication, balancing, splitting, documentation and versioning, and how data pruning ranks examples so that a smaller set trains as well as a larger one.

intermediate

Dataset curation is the work of deciding which images, with which labels, go into a dataset, and of keeping that decision correct as the data and the task change. It covers defining what the data should represent, collecting and licensing it, filtering out unusable or unwanted items, removing duplicates within the set and against evaluation sets, deciding whether to rebalance, splitting it into training, validation and test sets, documenting it and versioning it. Each of these steps changes what a model learns and what its reported score means, which is why dataset bias, leakage between splits and class imbalance all trace back to curation decisions.

One part of curation has a theory of its own: data pruning, which ranks the examples of a training set by a difficulty score and keeps only a fraction of them. Sorscher, Geirhos, Shekhar, Ganguli and Morcos (2022) showed in a perceptron model that whether to keep the easy or the hard examples depends on how much data there is to begin with, and reported the same shift for a ResNet18 on CIFAR-10. The example in this article tests that prediction on synthetic shape images with a convex classifier.

Purpose

A model can only learn what its training set shows, and a benchmark can only measure what its test set contains. Curation addresses three problems that a raw collection of images has:

  • It is not the target distribution. Images gathered by web search, by one camera or by one annotation vendor differ from the images the system will see; the dataset bias article describes how selection, capture and labelling make a dataset an unrepresentative sample, and how cross-dataset tests measure the damage.
  • It contains items that distort training or evaluation. Exact and near-duplicate copies inflate test scores when they fall on both sides of a split, as the train, validation and test split article shows for CIFAR; mislabelled, corrupt or off-task images add noise; some content should not be in the set at all.
  • It is larger than it needs to be, or larger in the wrong places. Redundant images cost compute without adding information, while rare classes and conditions stay rare. Sorscher et al. (2022, §1) start from the observation that test error often falls only as a slow power law of dataset size, and take this to suggest that many training examples are highly redundant.

When to Use It

Every dataset is curated, whether or not the steps are written down; the question is how deliberately. In our reading, explicit curation pays most when:

  • the data come from the web or another uncontrolled source, where duplicates and off-task images are common;
  • the same data serve training and evaluation, so that duplicates or related images can leak across the split;
  • a model will be retrained as new data arrive, so that the dataset needs versions;
  • compute is the limit, so that a smaller set that trains as well is worth finding (data pruning);
  • other people will use the data, so that its composition and limits must be documented.

Curation does not replace evaluation on data drawn from the deployment conditions. In our reading, a curated set is a hypothesis about what matters, and only a test on the target distribution checks it.

How It Works

The steps below are our summary of the workflow, in a working order; in practice the loop repeats as errors found in training and evaluation send the curator back to earlier steps.

  1. Define the target. State the task, the conditions the model must handle (viewpoints, lighting, sensors, places, populations) and the claim the test set must support. Every later step is judged against this definition.
  2. Source and license. Record where each image came from and under what terms. The LAION article shows the cost of a collection assembled from web links: its datasets were withdrawn in December 2023 over links to suspected child sexual abuse material and re-released as Re-LAION-5B in August 2024.
  3. Filter. Remove images that are corrupt, off-task or unwanted, with rules, classifiers or a similarity score. LAION filtered image–text pairs by the similarity of their CLIP embeddings; its article works through how the threshold trades the precision of the kept set against the recall of good pairs. As the scaling laws article reports, Goyal et al. (2024) found that the value of such a filter depends on the compute budget: training a CLIP ViT-B/32 on the DataComp medium pool, a LAION-style filter keeping about 10% of the pairs beat no filtering at small budgets and lost to it at large ones.
  4. Deduplicate. Remove copies within the set and, above all, copies of evaluation images in the training set. Methods differ in what they treat as “the same” (our summary): exact hashes of file bytes or pixels catch only identical copies; perceptual hashes and distances in pixel or feature space also catch re-encoded, resized or slightly edited ones. The CIFAR article describes both ends: the original report removed duplicates by comparing images with the L2 norm, and Barz and Denzler (2020) later found near-duplicates of test images in the training sets by nearest-neighbour search in a CNN’s feature space followed by manual review; its example compares pixel distance with a shift- and colour-invariant space. LAION’s deduplication, as its article describes it, used Bloom filters on URLs and captions and so did not compare image content. LAION’s later MD5 hash matching and the PhotoDNA perceptual-hash matching in the Stanford Internet Observatory’s report served to find known abuse material, not to deduplicate in this sense.
  5. Decide on balance. Keeping the natural frequencies, resampling or reweighting are choices with consequences for every class; the class imbalance article covers them. Pruning can unbalance a dataset by itself: Sorscher et al. (2022, §5) found that all the pruning metrics they benchmarked on ImageNet amplified class imbalance, and used a class-balancing ratio of 50% in all their ImageNet experiments.
  6. Split. Assign images to training, validation and test sets at the level of the groups that make images dependent (videos, people, places, photographers, patients, times), as the split article explains. COCO’s authors, as its article reports, removed near-duplicates across splits and grouped images by photographer and date.
  7. Document. Record motivation, composition, collection, preprocessing, labelling, recommended uses and maintenance. Gebru et al. (2021) proposed a datasheet with questions on each of these, as the dataset bias article describes.
  8. Version and maintain. Every change to the files, the labels or the splits makes a new version, and results are reported against a version. Withdrawn and corrected datasets, such as 80 Million Tiny Images and LAION-5B in the dataset bias article, show that copies and derived models outlive the original.
  9. Iterate. Errors found while training and evaluating (mislabelled images, missing conditions, groups with poor results) feed back into collection, labelling and filtering. Label errors and their detection, and the choice of which images to label next, are the subjects of annotation quality and of active learning; the image annotation article covers the labelling workflow itself.

Large pretraining sets combine these steps at scale. As the DINO article describes it, DINOv2’s LVD-142M started from 1.2 billion web images left after unsafe-content filtering, hash deduplication and face blurring, removed near-duplicates of benchmark test and validation images with a copy-detection pipeline, and then retrieved images close, in a self-supervised embedding, to images of curated datasets. As the foundation models article reports, DINOv2’s curated 142 million images beat as many uncurated ones on most benchmarks (Oquab et al., 2024, §6.2).

Data pruning

Data pruning keeps a fraction ff of a training set chosen by a score, and trains from scratch on the kept examples. Sorscher et al. (2022, §2.1) describe earlier scores as ordering examples from “easiest” to “hardest”, and note that earlier pruning work kept the hardest. They distinguish pruning, a one-shot choice of a subset, from active learning, which alternates between training a model and choosing new inputs to be labelled.

The theory. Their analysis (§3) uses a perceptron in the teacher–student setting: inputs are i.i.d. Gaussian vectors in RN\mathbb{R}^N, labels are given by a teacher perceptron TT, so the labels are realizable by construction, and a probe student JprobeJ_\text{probe} gives each example a margin mμ=Jprobe⋅(yμxμ)m_\mu = J_\text{probe} \cdot (y_\mu x_\mu), large for easy examples and small for hard ones. Keeping the fPfP hardest of PP examples and training a new perceptron to completion, they compute the test error as a function of αtot=P/N\alpha_\text{tot} = P/N, the number of examples per parameter, of ff, and of the angle θ\theta between probe and teacher. Two predictions follow:

  • The best strategy depends on the amount of data. For small αtot\alpha_\text{tot}, keeping the hardest examples is worse than random pruning, and keeping the easiest is better than keeping the hardest; for large αtot\alpha_\text{tot}, keeping the hardest is better. The authors’ explanation is that easy examples give coarse information about the target function and hard ones fine information, which helps only once the basics are learned. The transition becomes sharper at small ff (Fig. 1C). They report the same shift in a ResNet18 trained on pruned subsets of CIFAR-10 under the EL2N score (Fig. 1D, §3), so the prediction is tested both in the perceptron theory and in a network.
  • Pruning can beat power-law scaling. At a fixed ff, error eventually falls as the classical power law ε∝αprune−1\varepsilon \propto \alpha_\text{prune}^{-1}, but pruning more aggressively as the initial dataset grows traces a Pareto frontier that falls at least exponentially in the pruned size. With an imperfect score (θ>0\theta > 0) this crosses over to a power law, and there is a minimum useful fraction: for θ=10°\theta = 10° (20°20°) one can do no better asymptotically than keeping 24% (46%) of the data (§3).

The experiments. The authors report better than power-law scaling with pruned dataset size for ResNets trained on SVHN, CIFAR-10 and ImageNet (§4, Fig. 3). Fine-tuning an ImageNet-21K-pretrained ViT on 10% of CIFAR-10 chosen by EL2N could match or exceed fine-tuning on all of it, and pretraining ResNet50s on as little as 50% of ImageNet could match pretraining on all of it for CIFAR-10 fine-tuning (§4, Fig. 4). Benchmarking ten scores on ImageNet (§5), they found substantial diversity in the rankings, though EL2N, DDD and memorization had pairwise Spearman correlations above 0.7, and only a few scores matched full-data top-5 validation accuracy when keeping about 80% of the data; memorization in particular performed strongly. They remark that ImageNet is already carefully curated, which may make it harder to prune.

A self-supervised score. The earlier scores they review all need labels (§2.1). Sorscher et al. (§6) run k-means in the embedding of a self-supervised model (SwAV, pretrained on ImageNet) and define difficulty as the Euclidean distance to the nearest centroid, or prototype: the most prototypical images are the easiest. This label-free score matched or exceeded memorization until only 70–80% of the data was kept, and they report it robust to the number of clusters within an order of magnitude of the number of classes. Computing one prototype per class instead (supervised prototypes) also performed close to memorization. In their summary, the self-supervised score lets them discard 20% of ImageNet without losing performance (§7).

They name the limits themselves (§7): exponential scaling needs a high-quality score; training on a pruned set for the same number of epochs saves compute, but the same number of iterations often did better; class balancing was essential; and pruning may affect groups differently, so fairness tests specific to the deployment are still needed.

How to Apply It

The example tests the first prediction, keep easy when data are scarce and hard when abundant, outside the perceptron model. Images are 24×2424 \times 24 renderings of four classes (disk, square, triangle and plus sign) with an exactly known atypicality a∈[0,1]a \in [0, 1]: 80% of images are central, with a=∣N(0,0.12)∣a = |\mathcal{N}(0, 0.12)| clipped to [0,1][0, 1], and 20% form a tail with aa uniform on [0.3,1][0.3, 1]. As aa grows, the shape loses contrast (from 0.6 to 0.09 above a background of 0.3), becomes blurrier (Gaussian σ=0.5+1.5a\sigma = 0.5 + 1.5a pixels) and is rotated by up to 45a45a degrees and stretched by up to a factor 1±0.6a1 \pm 0.6a; every image has a position jitter of up to 2 pixels and Gaussian noise with standard deviation 0.08. Coordinates are (row, column) with the origin at the image centre, and shapes are rendered with 4×44 \times 4 supersampling.

Features are a fixed, label-free map: HOG from scikit-image (9 orientations, 8×88 \times 8 pixel cells, one cell per block, L2-Hys), giving 81 numbers per image, standardized with the mean and standard deviation of a fixed unlabelled reference set of 2,000 images. The learner is multinomial logistic regression from scikit-learn (L-BFGS, C=1C = 1, at most 2,000 iterations). Since CC multiplies the summed loss, a smaller kept set is regularized relatively more at the same CC.

Three scores rank each pool:

  • prototype, label-free, the analogue of the paper’s self-supervised score: k-means with k=4k = 4 (the number of classes, our choice) on the pool’s features, distance to the nearest centroid, small = easy;
  • supervised, the analogue of the theory’s probe margin: the full-pool model’s logit for the true class minus the largest other logit, computed on the pool itself, large = easy;
  • random.

Two pools are drawn per seed, P=200P = 200 and P=20,000P = 20{,}000, that is αtot=P/81\alpha_\text{tot} = P/81 of 2.5 and 247 examples per feature. In the paper’s Fig. 1C, which plots αtot\alpha_\text{tot} from 1 to about 30, 2.5 lies in the keep-easy region for every ff used here, and 247 lies beyond the right edge of the plotted range, past the transition. From each, fractions f∈{0.8,0.6,0.4,0.2}f \in \{0.8, 0.6, 0.4, 0.2\} are kept, either the easiest or the hardest, ranked over the whole pool (“global”) or within each class so that every class keeps the same share (“per-class”). Sorscher et al. used a class-balancing ratio on ImageNet but none in the theory, so both rankings are run. Each kept set trains a new model, scored on 5,000 fresh test images per seed. The outcome concerns this analogue, HOG features and logistic regression on images with an atypical tail, not the paper’s perceptron or networks.

The planned checks were: C1, the full-pool model is learnable, with test accuracy above 0.40 (chance is 0.25) and below 0.99 at both pools; C2, it does not separate its training data, with training accuracy below 1.0; C3, the data have the planned properties (a tail share within 0.09 of 0.20 at P=200P = 200 and within 0.03 at P=20,000P = 20{,}000, exactly P/4P/4 images per class, and lower test accuracy on tail images, a≥0.3a \ge 0.3, than on central ones); and C4, every fit converges. The hypotheses, on means over seeds 0–4 with a tolerance of 0.005, were: H1, at P=200P = 200 keeping easy is at least as accurate as keeping hard, for each score, ranking and ff; H2, at P=20,000P = 20{,}000 keeping hard is at least as accurate as keeping easy, likewise; and H3, the Spearman correlation between the prototype distance and the supervised hardness (minus the margin) is at least 0.3 on every seed at both pools. Two printed values decide nothing: each score’s Spearman correlation with the true atypicality aa, and the smallest class share in each kept set (minimum over seeds).

The script loops over seeds 0–4, given on the command line (python prune.py 0 1 2 3 4). It ran with NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2 on one BLAS thread, in about 30 seconds; two runs printed identical output. It makes 370 fits.

# Data pruning after Sorscher et al. (2022): keep easy or keep hard, on synthetic shapes.
# Usage: python prune.py SEED [SEED ...]   (pilot: 0; first run: 0 1 2 3 4)
import os, sys
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"; os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np
from scipy.ndimage import gaussian_filter
from scipy.stats import spearmanr
from skimage.feature import hog
from sklearn.cluster import KMeans
from sklearn.linear_model import LogisticRegression

S, SS = 24, 4                                   # image size, supersampling factor
g = (np.arange(S * SS) + 0.5) / SS - S / 2      # supersampled pixel centres, origin at the image centre
YY, XX = np.meshgrid(g, g, indexing="ij")       # rows, columns

def render(cls, a, rng):
    """One 24x24 image of class cls (disk, square, triangle, plus) with atypicality a in [0, 1]."""
    th = np.deg2rad(45 * a * rng.uniform(-1, 1))    # rotation grows with a
    st = 1 + 0.6 * a * rng.uniform(-1, 1)           # stretch grows with a
    cx, cy = rng.uniform(-2, 2, 2)                  # position jitter, independent of a
    x, y = XX - cx, YY - cy
    u = (np.cos(th) * x + np.sin(th) * y) / st
    v = (-np.sin(th) * x + np.cos(th) * y) * st
    r = 7.0
    if cls == 0:   m = u**2 + v**2 <= r**2
    elif cls == 1: m = (np.abs(u) <= 0.85 * r) & (np.abs(v) <= 0.85 * r)
    elif cls == 2: m = (v <= 0.6 * r) & (v >= -r + 2 * np.abs(u)) & (np.abs(u) <= r)
    else:          m = ((np.abs(u) <= r) & (np.abs(v) <= 0.3 * r)) | ((np.abs(v) <= r) & (np.abs(u) <= 0.3 * r))
    cov = m.reshape(S, SS, S, SS).mean(axis=(1, 3))                      # area coverage per pixel
    img = 0.3 + 0.6 * (1 - 0.85 * a) * gaussian_filter(cov, 0.5 + 1.5 * a, mode="nearest")  # contrast falls, blur grows
    return img + rng.normal(0, 0.08, img.shape)

def make(n, rng):
    y = np.repeat(np.arange(4), n // 4); rng.shuffle(y)                 # balanced classes
    tail = rng.random(n) < 0.2                                          # 20% atypical tail
    a = np.where(tail, rng.uniform(0.3, 1.0, n), np.clip(np.abs(rng.normal(0, 0.12, n)), 0, 1))
    X = np.array([hog(render(c, ai, rng), orientations=9, pixels_per_cell=(8, 8), cells_per_block=(1, 1),
                      block_norm="L2-Hys") for c, ai in zip(y, a)])    # 81 label-free features
    return X, y, a, tail

ref = make(2000, np.random.default_rng(999))[0]                         # fixed unlabeled reference set
mu, sd = ref.mean(0), ref.std(0) + 1e-12
not_conv = [0, 0]                                                       # [fits not converged, fits]

def fit(X, y):
    m = LogisticRegression(C=1.0, max_iter=2000).fit(X, y)
    not_conv[0] += int(m.n_iter_[0] >= 2000); not_conv[1] += 1
    return m

def keep(order, y, f, per_class):
    """order: indices from easiest to hardest. Returns (easy set, hard set) of fraction f."""
    if not per_class:
        k = int(round(f * len(order)))
        return order[:k], order[-k:]
    E, H = [], []
    for c in range(4):
        oc = order[y[order] == c]; k = int(round(f * len(oc)))
        E.append(oc[:k]); H.append(oc[-k:])
    return np.concatenate(E), np.concatenate(H)

FRACS = [0.8, 0.6, 0.4, 0.2]
CONDS = [(s, d, r) for s in ("prototype", "supervised") for r in ("global", "per-class") for d in ("easy", "hard")]

def run(seed):
    rng = np.random.default_rng(seed)
    Xt, yt, at, tt = make(5000, rng); Xt = (Xt - mu) / sd
    res = {}
    for P in (200, 20000):
        X, y, a, tail = make(P, rng); X = (X - mu) / sd
        m = fit(X, y)
        acc = lambda mod: (mod.predict(Xt) == yt).mean()
        z = m.decision_function(X)
        zt = z[np.arange(P), y].copy(); z[np.arange(P), y] = -np.inf
        margin = zt - z.max(1)                                          # supervised score: large = easy
        proto = KMeans(4, n_init=10, random_state=seed).fit(X).transform(X).min(1)  # label-free: small = easy
        r = dict(full=acc(m), train=(m.predict(X) == y).mean(), central=(m.predict(Xt[~tt]) == yt[~tt]).mean(),
                 tailacc=(m.predict(Xt[tt]) == yt[tt]).mean(), tailshare=tail.mean(), counts=np.bincount(y, minlength=4),
                 rho=spearmanr(proto, -margin).statistic, rho_pa=spearmanr(proto, a).statistic,
                 rho_ma=spearmanr(-margin, a).statistic)
        orders = {"prototype": np.argsort(proto, kind="stable"), "supervised": np.argsort(-margin, kind="stable")}
        for f in FRACS:
            for s in ("prototype", "supervised"):
                for rk in ("global", "per-class"):
                    E, H = keep(orders[s], y, f, rk == "per-class")
                    for d, idx in (("easy", E), ("hard", H)):
                        r[(s, d, rk, f)] = (acc(fit(X[idx], y[idx])), np.bincount(y[idx], minlength=4).min() / len(idx))
            idx = rng.permutation(P)[: int(round(f * P))]
            r[("random", f)] = (acc(fit(X[idx], y[idx])), np.bincount(y[idx], minlength=4).min() / len(idx))
        res[P] = r
    return res

seeds = [int(s) for s in sys.argv[1:]]
R = {s: run(s) for s in seeds}
for P in (200, 20000):
    print(f"\n=== pool P = {P} (alpha_tot = P/81 = {P/81:.1f}) ===")
    print("seed  full-acc  train-acc  central  tail    tail-share  class-counts       rho(proto,sup)  rho(proto,a)  rho(sup,a)")
    for s in seeds:
        r = R[s][P]
        print(f"{s:4d}  {r['full']:.4f}    {r['train']:.4f}     {r['central']:.4f}  {r['tailacc']:.4f}  {r['tailshare']:.4f}"
              f"      {str(r['counts'].tolist()):18s} {r['rho']:+.4f}         {r['rho_pa']:+.4f}       {r['rho_ma']:+.4f}")
    print("test accuracy, mean over seeds [min-max]; last column: smallest class share in the kept set (min over seeds)")
    print(f"{'condition':30s}" + "".join(f"{'f=' + str(f):>22s}" for f in FRACS))
    rows = [(f"{s} {d} ({rk})", lambda f, s=s, d=d, rk=rk: [R[q][P][(s, d, rk, f)] for q in seeds]) for s, d, rk in CONDS]
    rows.append(("random", lambda f: [R[q][P][("random", f)] for q in seeds]))
    for name, get in rows:
        line = f"{name:30s}"
        for f in FRACS:
            v = np.array([x[0] for x in get(f)]); cs = min(x[1] for x in get(f))
            line += f"  {v.mean():.4f} [{v.min():.3f}-{v.max():.3f}] {cs:.2f}"
        print(line)

print("\n=== checks and hypotheses (tolerance 0.005 on seed means) ===")
for P in (200, 20000):
    fa = np.array([R[s][P]["full"] for s in seeds]); tr = np.array([R[s][P]["train"] for s in seeds])
    ts = np.array([R[s][P]["tailshare"] for s in seeds]); tol = 0.03 if P == 20000 else 0.09
    ce = np.array([R[s][P]["central"] for s in seeds]); ta = np.array([R[s][P]["tailacc"] for s in seeds])
    cnt = all((R[s][P]["counts"] == P // 4).all() for s in seeds)
    print(f"P={P}: C1 full test acc min {fa.min():.4f} max {fa.max():.4f} (bar >0.40, <0.99): {'held' if fa.min() > 0.40 and fa.max() < 0.99 else 'failed'}")
    print(f"P={P}: C2 train acc max {tr.max():.4f} (bar <1.0): {'held' if tr.max() < 1.0 else 'failed'}")
    print(f"P={P}: C3 tail share max |dev| {np.abs(ts - 0.2).max():.4f} (bar {tol}); counts equal {cnt}; "
          f"min(central - tail acc) {(ce - ta).min():+.4f} (bar >0): "
          f"{'held' if np.abs(ts - 0.2).max() <= tol and cnt and (ce - ta).min() > 0 else 'failed'}")
for P, H, want in ((200, "H1", "easy"), (20000, "H2", "hard")):
    other = "hard" if want == "easy" else "easy"
    for s in ("prototype", "supervised"):
        for rk in ("global", "per-class"):
            for f in FRACS:
                d = np.array([R[q][P][(s, want, rk, f)][0] - R[q][P][(s, other, rk, f)][0] for q in seeds])
                print(f"{H} P={P} {s:10s} {rk:9s} f={f}: mean(keep-{want} - keep-{other}) {d.mean():+.4f} "
                      f"[{d.min():+.4f}..{d.max():+.4f}] -> {'held' if d.mean() >= -0.005 else 'failed'}")
for P in (200, 20000):
    rh = np.array([R[s][P]["rho"] for s in seeds])
    print(f"H3 P={P}: rho(prototype distance, supervised hardness) per seed {' '.join(f'{v:+.4f}' for v in rh)} "
          f"(bar >= 0.3): {'held' if rh.min() >= 0.3 else 'failed'}")
print(f"C4 fits not converged: {not_conv[0]} of {not_conv[1]}: {'held' if not_conv[0] == 0 else 'failed'}")
=== pool P = 200 (alpha_tot = P/81 = 2.5) ===
seed  full-acc  train-acc  central  tail    tail-share  class-counts       rho(proto,sup)  rho(proto,a)  rho(sup,a)
   0  0.5746    1.0000     0.6424  0.3140  0.2350      [50, 50, 50, 50]   -0.2762         -0.4271       +0.3815
   1  0.6098    1.0000     0.6716  0.3523  0.2050      [50, 50, 50, 50]   -0.3642         -0.3538       +0.3430
   2  0.6202    1.0000     0.6889  0.3466  0.2350      [50, 50, 50, 50]   -0.3525         -0.4044       +0.4578
   3  0.6150    1.0000     0.6893  0.3201  0.2200      [50, 50, 50, 50]   -0.4224         -0.4285       +0.3880
   4  0.6100    1.0000     0.6920  0.2873  0.1700      [50, 50, 50, 50]   -0.3415         -0.3741       +0.3672
test accuracy, mean over seeds [min-max]; last column: smallest class share in the kept set (min over seeds)
condition                                      f=0.8                 f=0.6                 f=0.4                 f=0.2
prototype easy (global)         0.5832 [0.561-0.597] 0.22  0.5595 [0.544-0.585] 0.19  0.5318 [0.494-0.582] 0.12  0.4666 [0.420-0.553] 0.10
prototype hard (global)         0.6115 [0.585-0.632] 0.22  0.6236 [0.591-0.636] 0.18  0.6274 [0.611-0.656] 0.15  0.5919 [0.546-0.637] 0.10
prototype easy (per-class)      0.5855 [0.571-0.597] 0.25  0.5579 [0.545-0.583] 0.25  0.5308 [0.500-0.567] 0.25  0.4755 [0.416-0.554] 0.25
prototype hard (per-class)      0.6176 [0.592-0.634] 0.25  0.6284 [0.595-0.644] 0.25  0.6380 [0.628-0.647] 0.25  0.6100 [0.578-0.646] 0.25
supervised easy (global)        0.6447 [0.629-0.674] 0.23  0.6643 [0.642-0.681] 0.21  0.6614 [0.656-0.675] 0.19  0.6259 [0.573-0.650] 0.07
supervised hard (global)        0.6050 [0.575-0.620] 0.21  0.5962 [0.571-0.614] 0.18  0.5137 [0.496-0.538] 0.20  0.3454 [0.300-0.378] 0.10
supervised easy (per-class)     0.6446 [0.627-0.672] 0.25  0.6663 [0.644-0.679] 0.25  0.6606 [0.648-0.666] 0.25  0.6415 [0.608-0.657] 0.25
supervised hard (per-class)     0.6041 [0.573-0.618] 0.25  0.5914 [0.563-0.614] 0.25  0.5150 [0.495-0.549] 0.25  0.3494 [0.296-0.380] 0.25
random                          0.6093 [0.586-0.633] 0.21  0.5848 [0.567-0.607] 0.19  0.5941 [0.555-0.622] 0.16  0.5309 [0.405-0.587] 0.10

=== pool P = 20000 (alpha_tot = P/81 = 246.9) ===
seed  full-acc  train-acc  central  tail    tail-share  class-counts       rho(proto,sup)  rho(proto,a)  rho(sup,a)
   0  0.7670    0.7721     0.8639  0.3944  0.1978      [5000, 5000, 5000, 5000] -0.2698         -0.3349       +0.4919
   1  0.7814    0.7749     0.8738  0.3967  0.1981      [5000, 5000, 5000, 5000] -0.2632         -0.3340       +0.5013
   2  0.7672    0.7714     0.8671  0.3695  0.2026      [5000, 5000, 5000, 5000] -0.2661         -0.3464       +0.4987
   3  0.7732    0.7748     0.8698  0.3897  0.1952      [5000, 5000, 5000, 5000] -0.2741         -0.3364       +0.4880
   4  0.7622    0.7648     0.8661  0.3534  0.2049      [5000, 5000, 5000, 5000] -0.2643         -0.3442       +0.5013
test accuracy, mean over seeds [min-max]; last column: smallest class share in the kept set (min over seeds)
condition                                      f=0.8                 f=0.6                 f=0.4                 f=0.2
prototype easy (global)         0.7670 [0.759-0.780] 0.21  0.7626 [0.756-0.773] 0.18  0.7516 [0.746-0.760] 0.16  0.7128 [0.702-0.721] 0.13
prototype hard (global)         0.7712 [0.764-0.781] 0.23  0.7701 [0.763-0.779] 0.21  0.7667 [0.762-0.776] 0.18  0.7614 [0.750-0.769] 0.15
prototype easy (per-class)      0.7646 [0.759-0.779] 0.25  0.7527 [0.740-0.763] 0.25  0.7186 [0.700-0.734] 0.25  0.6130 [0.571-0.657] 0.25
prototype hard (per-class)      0.7729 [0.768-0.782] 0.25  0.7694 [0.765-0.780] 0.25  0.7648 [0.758-0.772] 0.25  0.7558 [0.743-0.765] 0.25
supervised easy (global)        0.7715 [0.765-0.781] 0.23  0.7709 [0.764-0.779] 0.24  0.7676 [0.759-0.777] 0.20  0.7623 [0.750-0.770] 0.13
supervised hard (global)        0.7712 [0.763-0.783] 0.22  0.7717 [0.767-0.781] 0.23  0.5121 [0.444-0.537] 0.23  0.1536 [0.148-0.158] 0.18
supervised easy (per-class)     0.7697 [0.763-0.780] 0.25  0.7711 [0.766-0.780] 0.25  0.7658 [0.756-0.775] 0.25  0.7510 [0.740-0.763] 0.25
supervised hard (per-class)     0.7705 [0.763-0.783] 0.25  0.7711 [0.764-0.781] 0.25  0.4974 [0.468-0.508] 0.25  0.1622 [0.155-0.170] 0.25
random                          0.7697 [0.762-0.778] 0.25  0.7670 [0.761-0.777] 0.24  0.7638 [0.758-0.773] 0.24  0.7593 [0.750-0.772] 0.24

=== checks and hypotheses (tolerance 0.005 on seed means) ===
P=200: C1 full test acc min 0.5746 max 0.6202 (bar >0.40, <0.99): held
P=200: C2 train acc max 1.0000 (bar <1.0): failed
P=200: C3 tail share max |dev| 0.0350 (bar 0.09); counts equal True; min(central - tail acc) +0.3194 (bar >0): held
P=20000: C1 full test acc min 0.7622 max 0.7814 (bar >0.40, <0.99): held
P=20000: C2 train acc max 0.7749 (bar <1.0): held
P=20000: C3 tail share max |dev| 0.0048 (bar 0.03); counts equal True; min(central - tail acc) +0.4695 (bar >0): held
H1 P=200 prototype  global    f=0.8: mean(keep-easy - keep-hard) -0.0284 [-0.0492..-0.0148] -> failed
H1 P=200 prototype  global    f=0.6: mean(keep-easy - keep-hard) -0.0640 [-0.0816..-0.0448] -> failed
H1 P=200 prototype  global    f=0.4: mean(keep-easy - keep-hard) -0.0957 [-0.1236..-0.0558] -> failed
H1 P=200 prototype  global    f=0.2: mean(keep-easy - keep-hard) -0.1253 [-0.1878..-0.0140] -> failed
H1 P=200 prototype  per-class f=0.8: mean(keep-easy - keep-hard) -0.0322 [-0.0512..-0.0156] -> failed
H1 P=200 prototype  per-class f=0.6: mean(keep-easy - keep-hard) -0.0705 [-0.0878..-0.0496] -> failed
H1 P=200 prototype  per-class f=0.4: mean(keep-easy - keep-hard) -0.1073 [-0.1400..-0.0758] -> failed
H1 P=200 prototype  per-class f=0.2: mean(keep-easy - keep-hard) -0.1345 [-0.1822..-0.0458] -> failed
H1 P=200 supervised global    f=0.8: mean(keep-easy - keep-hard) +0.0397 [+0.0238..+0.0646] -> held
H1 P=200 supervised global    f=0.6: mean(keep-easy - keep-hard) +0.0680 [+0.0486..+0.0796] -> held
H1 P=200 supervised global    f=0.4: mean(keep-easy - keep-hard) +0.1477 [+0.1250..+0.1598] -> held
H1 P=200 supervised global    f=0.2: mean(keep-easy - keep-hard) +0.2806 [+0.2568..+0.3308] -> held
H1 P=200 supervised per-class f=0.8: mean(keep-easy - keep-hard) +0.0404 [+0.0226..+0.0640] -> held
H1 P=200 supervised per-class f=0.6: mean(keep-easy - keep-hard) +0.0749 [+0.0634..+0.0852] -> held
H1 P=200 supervised per-class f=0.4: mean(keep-easy - keep-hard) +0.1456 [+0.1160..+0.1640] -> held
H1 P=200 supervised per-class f=0.2: mean(keep-easy - keep-hard) +0.2920 [+0.2582..+0.3336] -> held
H2 P=20000 prototype  global    f=0.8: mean(keep-hard - keep-easy) +0.0042 [+0.0008..+0.0072] -> held
H2 P=20000 prototype  global    f=0.6: mean(keep-hard - keep-easy) +0.0075 [+0.0060..+0.0108] -> held
H2 P=20000 prototype  global    f=0.4: mean(keep-hard - keep-easy) +0.0151 [+0.0132..+0.0178] -> held
H2 P=20000 prototype  global    f=0.2: mean(keep-hard - keep-easy) +0.0486 [+0.0426..+0.0576] -> held
H2 P=20000 prototype  per-class f=0.8: mean(keep-hard - keep-easy) +0.0083 [+0.0030..+0.0108] -> held
H2 P=20000 prototype  per-class f=0.6: mean(keep-hard - keep-easy) +0.0167 [+0.0086..+0.0250] -> held
H2 P=20000 prototype  per-class f=0.4: mean(keep-hard - keep-easy) +0.0463 [+0.0338..+0.0608] -> held
H2 P=20000 prototype  per-class f=0.2: mean(keep-hard - keep-easy) +0.1428 [+0.1088..+0.1810] -> held
H2 P=20000 supervised global    f=0.8: mean(keep-hard - keep-easy) -0.0003 [-0.0018..+0.0014] -> held
H2 P=20000 supervised global    f=0.6: mean(keep-hard - keep-easy) +0.0008 [-0.0024..+0.0042] -> held
H2 P=20000 supervised global    f=0.4: mean(keep-hard - keep-easy) -0.2555 [-0.3198..-0.2214] -> failed
H2 P=20000 supervised global    f=0.2: mean(keep-hard - keep-easy) -0.6088 [-0.6216..-0.5942] -> failed
H2 P=20000 supervised per-class f=0.8: mean(keep-hard - keep-easy) +0.0008 [-0.0018..+0.0030] -> held
H2 P=20000 supervised per-class f=0.6: mean(keep-hard - keep-easy) -0.0000 [-0.0030..+0.0028] -> held
H2 P=20000 supervised per-class f=0.4: mean(keep-hard - keep-easy) -0.2684 [-0.2926..-0.2556] -> failed
H2 P=20000 supervised per-class f=0.2: mean(keep-hard - keep-easy) -0.5888 [-0.6086..-0.5772] -> failed
H3 P=200: rho(prototype distance, supervised hardness) per seed -0.2762 -0.3642 -0.3525 -0.4224 -0.3415 (bar >= 0.3): failed
H3 P=20000: rho(prototype distance, supervised hardness) per seed -0.2698 -0.2632 -0.2661 -0.2741 -0.2643 (bar >= 0.3): failed
C4 fits not converged: 0 of 370: held

Checks. C1 held at both pools: full-pool test accuracy lies between 0.5746 and 0.6202 at P=200P = 200 and between 0.7622 and 0.7814 at P=20,000P = 20{,}000. C2 failed at P=200P = 200, where the full-pool model reached training accuracy 1.0000 on every seed, and held at P=20,000P = 20{,}000 (at most 0.7749). C3 held at both pools: the largest deviation of the tail share from 0.20 is 0.0350 and 0.0048, every pool has exactly P/4P/4 images per class, and tail images are classified less accurately than central ones on every seed (the smallest gap is 0.3194 at P=200P = 200). C4 held: none of the 370 fits stopped at the iteration limit.

H1 (small pool). For the supervised score, H1 held in all eight cells: keeping the easy examples beat keeping the hard ones by 0.04 at f=0.8f = 0.8, growing to 0.28 (global) and 0.29 (per-class) at f=0.2f = 0.2, with every seed’s difference positive. Keeping the easiest 20% by margin (0.6259 global, 0.6415 per-class) was also more accurate than keeping a random 20% (0.5309). For the prototype score, H1 failed in all eight cells: keeping the hard examples was more accurate at every fraction, on every seed, with mean differences from −0.0284 (global, f=0.8f = 0.8) to −0.1345 (per-class, f=0.2f = 0.2).

H2 (large pool). For the prototype score, H2 held in all eight cells, with mean differences from +0.0042 (global, f=0.8f = 0.8) to +0.1428 (per-class, f=0.2f = 0.2). For the supervised score, H2 held at f=0.8f = 0.8 and 0.60.6 under both rankings, with mean differences within the tolerance (between −0.0003 and +0.0008), and failed at f=0.4f = 0.4 and 0.20.2 under both rankings: keeping the hardest 40% gave 0.5121 (global) and 0.4974 (per-class) against about 0.77 for the easiest, and keeping the hardest 20% gave 0.1536 and 0.1622, below the chance level of 0.25. At this pool no score’s kept set was more accurate than a random subset of the same size by more than 0.005 at any fraction (random kept 0.7697 at f=0.8f = 0.8 and 0.7593 at f=0.2f = 0.2).

So the supervised score shows the paper’s keep-easy side at the small pool but not its keep-hard side at the large one, and the prototype score favours keeping hard at both pools; neither shows the crossover.

H3. H3 failed at both pools. The Spearman correlation between the prototype distance and the supervised hardness was negative on every seed at both pools (the five values per pool are on the H3 lines), against a bar of +0.3. The prototype distance’s correlation with the true atypicality aa was also negative on every seed at both pools. The supervised hardness’s correlation with aa was positive on every seed at both pools.

Rankings. Ranking within each class kept every class at exactly 25% by construction, while global ranking let the smallest class fall to 0.07 of the kept set (supervised easy, f=0.2f = 0.2, P=200P = 200) and to 0.10 for a random 20% of the small pool. The two rankings gave the same held or failed outcome in every cell of H1 and H2; the largest difference in accuracy between them is for keeping the easiest 20% by the prototype score at the large pool (0.7128 global, 0.6130 per-class).

Process. The example was planned before it ran, and the plan’s hypotheses and bars were written before any script. A planning script on seed 100, outside the seeds shown, then printed the checks’ reachable values, H3’s correlation and the supervised-global f=0.2f = 0.2 cell of H1 and of H2. H3’s correlation and the H2 cell already failed there; those two outcomes are therefore illustrations rather than tests. The bar of 0.3 for H3 had been fixed before that script ran and was not changed. Compared with the batch plan for this example, five choices were made before the pilot and approved with it: per-class ranking was added beside global ranking (D1); the correlations with aa and the smallest class shares were added as printed values that decide nothing (D2); the plan’s open choices were fixed as four classes, 81 HOG features, k=4k = 4, C=1C = 1, an in-sample margin, a fresh test set per seed, a tolerance of 0.005 on seed means and an H3 bar of 0.3 set before the planning script, where the batch plan had “the bar from the reachable-value script” (D3); the data check C3 and the convergence check C4 were added (D4); and f=1f = 1 is the full-pool fit while random has no easy or hard direction (D5). A pilot on seed 0 gave the same held or failed outcome for every check and hypothesis as the five-seed run shown. Nothing was changed after the pilot.

Variants

  • Deduplication by exact hash, perceptual hash, pixel distance or learned features, within a set or against evaluation sets; the CIFAR example compares two feature spaces for near-duplicate retrieval.
  • Score filtering, keeping items above a threshold on a quality or relevance score, as in LAION’s CLIP filter.
  • Retrieval-based curation, collecting images that resemble a curated reference set in an embedding, as in DINOv2’s LVD-142M (DINO).
  • Data pruning by a difficulty score, keeping easy or hard examples (above).
  • Mining during training. Hard negative mining adds the negatives a detector currently gets wrong in rounds of retraining; it chooses examples within a training run rather than fixing a dataset once.
  • Active learning chooses which unlabelled images to send for labelling next, repeatedly, by scores computed from the current model; Sorscher et al. separate it from pruning, which selects once.

Best Practices

These recommendations are ours unless a source is named.

  • Deduplicate against every evaluation set before training, including public benchmarks you will report on, and record what was removed.
  • Split by group, after deduplication, so that copies and related images cannot straddle the split (train, validation and test split).
  • Keep the raw collection and the curation code, so that each filtered or pruned version can be rebuilt and the filters re-run when they change.
  • Report a pruning or filtering result against random selection of the same size, as Sorscher et al. do, and at more than one dataset size, since the best direction can change with it.
  • Check what a filter or pruning rule removes, per class and per condition; Sorscher et al. found that pruning amplified class imbalance on ImageNet.
  • Write the datasheet while curating, not afterwards, so that collection and filtering decisions are recorded when they are made.

Pitfalls

  • Assuming hardest is best. Earlier pruning work kept the hardest examples; Sorscher et al.’s theory and CIFAR-10 experiment show that with scarce data keeping the easiest can be better, and in the example above keeping the hardest 20% of a large pool by margin gave accuracy below chance.
  • Trusting a label-free score without checking its ranking. In the example, the k-means prototype distance on HOG features failed H3: its Spearman correlation with the supervised hardness was negative on every seed.
  • Deduplicating by URL or caption only. The same picture at two URLs stays in twice, as the LAION article notes.
  • Filtering without regard to compute. A filter that helps at one training budget can hurt at another (Goyal et al., 2024, as the scaling laws article reports).
  • Pruning away whole classes or conditions. A global ranking can drop rare classes; the example’s easiest 20% by margin, ranked globally at P=200P = 200, held as little as 7% of one class.
  • Treating curation as finished. Data and requirements change; an undocumented, unversioned dataset cannot be corrected or withdrawn cleanly (dataset bias).

Evidence and Impact

The evidence for deliberate curation comes mostly from what happens without it: near-duplicates of CIFAR test images in its training sets (CIFAR), datasets withdrawn over their content (dataset bias, LAION), and filtering whose value depends on compute (scaling laws). On the positive side, DINOv2’s curated data beat uncurated data of the same size on most benchmarks (foundation models).

For data pruning, Sorscher et al. (2022) give both a theory, in which the best strategy changes from keeping easy to keeping hard examples as data grow and in which a good enough score can turn power-law scaling into exponential scaling, and experiments: the same change of strategy for a ResNet18 on CIFAR-10, better than power-law scaling on SVHN, CIFAR-10 and ImageNet, and a label-free prototype score that matched the best supervised score on ImageNet down to 70–80% of the data. The main limitation they name is that exponential scaling needs a high-quality score (§7). In the example, neither score produced the change of strategy with logistic regression on HOG features of shapes with an atypical tail, which tests our analogue rather than the paper’s setting.

Related

  • CIFAR

    CIFAR-10 and CIFAR-100, two labelled sets of 60,000 colour images of 32×32 pixels taken from the 80 Million Tiny Images collection in 2009, with 10 classes or 100 classes in 20 superclasses, the standard small-scale benchmarks for image classification.

  • LAION

    LAION-400M and LAION-5B, open collections of hundreds of millions to billions of image URLs paired with their web alt-text and filtered by CLIP similarity, which trained OpenCLIP and Stable Diffusion, were withdrawn in December 2023 over links to suspected child sexual abuse material, and were re-released as Re-LAION-5B in August 2024.

  • Class Imbalance

    When some classes have far more training images than others, as in long-tailed recognition, classifiers favour the frequent classes; how imbalance is measured, why it hurts, and the remedies of resampling, reweighting, rebalanced losses, decoupled training and post-hoc logit adjustment for a prior shift.

  • Scaling Laws

    Empirical laws that describe how a network's error falls as training data, model size and compute grow, covering power laws and their saturating forms, compute-optimal allocation, the evidence from vision (JFT-300M, Instagram hashtags, Big Transfer, ViT scaling, ViT-22B, OpenCLIP), downstream and transfer scaling, where the laws break, and how to fit and extrapolate them.

  • Hard Negative Mining

    Training a detector's classifier on the background examples it gets wrong rather than on a random sample of background, from bootstrapping in sliding-window and part-model training to online hard example mining, SSD's fixed-ratio selection, and the focal loss that reweights instead.

  • Image Annotation

    How people attach labels to images for training and evaluating vision models, from image tags and bounding boxes to masks, keypoints, tracks and 3D cuboids, and how guidelines, agreement checks, quality control, model assistance and fair pay decide whether those labels can be trusted.

References

  1. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S. & Morcos, A. S. (2022). Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems 35 (NeurIPS).
  2. Barz, B. & Denzler, J. (2020). Do We Train on Test Data? Purging CIFAR of Near-Duplicates. Journal of Imaging, 6(6), 41 (not read).
  3. Gebru, T., Morgenstern, J., Vecchione, B., Wortman Vaughan, J., Wallach, H., Daumé III, H. & Crawford, K. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86–92 (not read).
  4. Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2024). DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research (TMLR). arXiv:2304.07193 (not read).
  5. Goyal, S., Maini, P., Lipton, Z. C., Raghunathan, A. & Kolter, J. Z. (2024). Scaling Laws for Data Filtering—Data Curation cannot be Compute Agnostic. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (not read).