Dataset Bias

Dataset bias is the difference between the images and labels a dataset contains and the visual world it is meant to represent. Every dataset is gathered by particular people, from particular sources, with particular cameras, keywords, filters and label definitions, and a model trained on it learns those choices along with the task. The symptoms are familiar: a classifier that scores well on its own test set and much worse on images collected elsewhere, a detector that finds side views of cars but not front views, a face analysis system whose errors fall unevenly on groups that were rare in its data. Related terms describe parts of the same problem: dataset shift or distribution shift (training and test data drawn from different distributions), domain shift (a change of camera, style or setting), sampling or selection bias, and representation bias (groups or regions under-represented). This article covers the kinds of bias that vision datasets show, how Torralba and Efros (2011) measured them with cross-dataset experiments, how replication studies separate shift from test-set reuse, what bias means when the images show people (demographic and geographic representation, offensive labels, consent and withdrawn datasets), and what is done about it. How a benchmark is built and run is the subject of benchmarks and leaderboards.

Definition

A dataset is biased, for a given task, when the joint distribution of its images and labels differs from the distribution of images and labels the task will meet in use. Because “the visual world” is not available as a dataset, bias is in practice measured relative to other datasets or to a target population: it shows up as a gap between a model’s accuracy on the test split of its own dataset and its accuracy on data gathered another way.

Torralba and Efros (2011, §3.1) named four sources of such gaps in object recognition datasets:

• Selection bias: datasets prefer particular kinds of images, such as street scenes, nature scenes, or whatever an Internet keyword search returns.
• Capture bias: photographers tend to take pictures of objects in similar ways, for example with the object in the centre of the frame or seen from a canonical side.
• Category or label bias: semantic categories are poorly defined, and different labellers assign different labels to the same kind of object (“grass” or “lawn”, “painting” or “picture”).
• Negative-set bias: the negative examples define what a dataset considers “the rest of the world”; if they are unrepresentative or unbalanced, classifiers become overconfident and less discriminative.

Datasets of people add questions that are not about accuracy alone: whose images were collected and with what consent, how people are categorized and labelled, and what happens to a dataset once its problems are known.

Intuition

Torralba and Efros (2011, §1) opened with a game, Name That Dataset: shown three images each from twelve popular recognition datasets, guess which image came from which dataset. In their labs most people got more than 75% right. Each dataset has a recognizable “signature”: the kind of scene, the camera, the framing, the post-processing. Anything a person can use to recognize the source, a classifier can also use, and what it learns about “car” is then partly what a car looks like in that dataset.

A dataset is like an opinion poll taken in one town. The answers are correct for that town, and nothing inside the poll tells you how far they are from the national result; only a second poll taken elsewhere does. Cross-dataset evaluation is that second poll. It also shows that the bias is not only in the data: a model that relies on cues tied to one dataset’s habits fails where those habits differ, whereas a model that learned what the object itself looks like does not care. Torralba and Efros (2011, §5) make the same point from the other side: a dataset that defines “car” as the rear view of a race car leaves no reasonable algorithm able to call a side view of a family sedan a car.

Formal Definition

Let each dataset $k$ be a sample from a distribution $P_k(x, y)$ over images $x$ and labels $y$, and let $P^{*}(x, y)$ be the distribution of the intended application. Dataset bias is $P_k \neq P^{*}$; dataset shift between datasets $k$ and $l$ is $P_k \neq P_l$. Writing $P(x, y) = P(y \mid x)\,P(x) = P(x \mid y)\,P(y)$ separates the common cases, which Storkey (2008) catalogues for machine learning in general:

• Covariate shift (Storkey’s “simple covariate shift”): only $P(x)$ changes; the labelling rule $P(y \mid x)$ is the same.
• Prior probability shift: only the class frequencies $P(y)$ change.
• Sample selection bias: the data pass through a selection process, unknown to the learner, that depends on the example. With a selection indicator $s$,

$$P_k(x, y) = \frac{P(s = 1 \mid x, y)\,P^{*}(x, y)}{P(s = 1)},$$

so the dataset over-represents whatever the selection favours (a search engine’s ranking, an annotator’s acceptance, a filter’s threshold).

• Domain shift: the measurement changes, as when the same scenes are photographed with a different camera.

In our reading, the vision terms map onto these: selection bias is sample selection; capture bias changes $P_k(x \mid y)$ through viewpoint, framing and imaging; label bias changes $P_k(y \mid x)$ itself, since the same image would be labelled differently; and negative-set bias is selection acting on the negatives, $P_k(x \mid y = 0)$ covering a small and unrepresentative part of the world’s non-objects.

Cross-dataset generalization. Train a model on each of $n$ datasets with the same classes and test it on all of them, giving a matrix $A_{ij}$ (trained on $i$, tested on $j$) of accuracy or average precision. Torralba and Efros (2011, Table 1) summarize row $i$ by its diagonal entry (“self”), the mean of its off-diagonal entries and the relative drop:

$$\bar{A}_i = \frac{1}{n - 1} \sum_{j \neq i} A_{ij}, \qquad \text{drop}_i = 1 - \frac{\bar{A}_i}{A_{ii}}.$$

A large drop means the model learned dataset $i$ rather than the task; column averages measure how easy dataset $j$ is for everyone else’s models. Name That Dataset is the complementary test: train a classifier to predict $k$ from $x$. Accuracy far above chance $1/n$ shows that the $P_k$ differ in ways visible to that classifier.

Properties

Datasets can be told apart, even for one object

Torralba and Efros (2011, §1) trained a 12-way linear SVM on 1,000 images from the training portion of each dataset, tested it on 300 per dataset and repeated this 20 times. With the best of four descriptors (thumbnails, gist and bags of HOG words) it was right 39% of the time against 8% chance, with no sign of saturation as training data grew. The confusions grouped scene-centric datasets apart from object-centric ones, with Caltech-101 and Caltech-256 confused with each other. Restricting the images to crops of cars from five datasets, which remove most of the scene, the classifier still reached 61% against 20% chance: Caltech cars were mostly side views, ImageNet favoured racing cars and PASCAL cars came from non-canonical viewpoints.

Training on one dataset and testing on another costs much of the accuracy

The same paper trained “car” and “person” classifiers (bag of words with a Gaussian-kernel SVM) and detectors (HOG with a linear SVM) on each of six datasets, SUN09, LabelMe, PASCAL VOC 2007, ImageNet, Caltech-101 and MSRC, and tested them on all six. The authors warn that the protocol differs from each dataset’s own, so the differences in AP matter rather than the values. For car classification, the mean AP when training and testing on the same dataset was 53.4, and 27.5 when testing on the others, a drop of 48%. The drops differed widely between training sets, from 7% for PASCAL to 73% for Caltech-101; PASCAL and ImageNet generalized best most of the time, and Caltech-101 and MSRC were the easiest datasets to score well on (§3.1, Table 1). Some person-classification drops were negative, because a model trained on PASCAL or ImageNet scored higher on the easy Caltech-101 and MSRC test sets than on its own. Testing each dataset’s detectors on its own positives against negatives pooled from all six datasets, the measured form of negative-set bias, lowered car and person detection AP by 15–22% for SUN09, LabelMe and PASCAL, left Caltech-101 unchanged, raised ImageNet’s (whose negatives were already varied) and lowered MSRC’s by 6–7% (Table 2; the text says that ImageNet, Caltech-101 and MSRC show no drop). A further experiment priced a training example from one dataset in examples of another: one LabelMe car was worth 0.26 PASCAL cars on the PASCAL benchmark, and every sample was worth less than one when used on a different dataset (§4, Table 3). Torralba and Efros concluded that Caltech-101 should have been retired long ago, as Ponce et al. (2006) had argued (§5); the Caltech-101 article records its results.

New datasets react to old biases and acquire their own

Torralba and Efros (2011, §2.1) read the history of recognition datasets as a series of reactions: COIL-100’s objects on black backgrounds, Corel’s professional photographs, Caltech-101’s web images of single centred objects, MSRC and LabelMe’s cluttered scenes, PASCAL VOC’s stricter training and testing standards, and finally Tiny Images, ImageNet and SUN09 as a reaction to datasets too small for the world’s complexity. Each removed a bias of its predecessors and introduced others, a cycle the authors attribute to the lack of a clear understanding of the types and sources of bias.

Shift appears even when a collection procedure is repeated

Bias is not only a matter of different groups collecting different things. Recht et al. (2019) rebuilt CIFAR-10’s and ImageNet’s test sets by following the original procedures from the same image sources, and still measured top-1 accuracy drops of about 3 to 15 points on CIFAR-10 and 10 to 14 points for convolutional networks on ImageNet, with the ranking of models almost unchanged; their leading hypothesis for the drop was a small distribution gap, mostly harder images, rather than reuse of the old test sets (ImageNetV2; benchmark overfitting separates the two effects). In the terms above, this is selection bias on a small scale: a new time window and new annotators changed which images passed the selection step (our reading).

Geography and income

Web-collected datasets inherit where their photographs were taken. Shankar et al. (2017) geolocated, at country level, about 2 million of Open Images’ 9 million images: more than 32% of that sample came from the United States and 60% from the six most represented countries, all in North America and Europe, while China and India contributed 1% and 2%. For ImageNet’s fall 2011 release, with lower coverage, about 45% of the located images were from the United States. Models trained on either dataset were much less likely to recognize “bridegroom” in web images that crowd workers in Hyderabad chose to represent their own communities than in the standard test images. DeVries et al. (2019) tested six publicly available object-recognition systems, in their February 2019 versions, on the Dollar Street photographs of household items from 264 homes in 54 countries: for every system, top-5 accuracy was roughly 10% lower for homes with an income below US$50 per month than for those above US$3,500 (§2; the introduction words this, inverted, as an error rate 10% lower), and one system’s accuracy was about 15 points higher for items photographed in the United States than in Somalia or Burkina Faso. Their examples suggest that the errors come from objects that look different (dish soap) or appear in different contexts (toothbrushes outside bathrooms), and they name two causes in the training data: the geography of ImageNet, COCO and Open Images, whose Flickr images under-sample Africa, India, China and South-East Asia, and collection by English search terms, which miss what is tagged in other languages.

Demographic representation in face datasets

For images of people, under-representation becomes a question of whom a system works for. Buolamwini and Gebru (2018) labelled skin type on the Fitzpatrick scale, grouping types I–III as lighter and IV–VI as darker, in two face analysis benchmarks: lighter-skinned subjects made up 79.6% of IJB-A (500 public figures, as the dataset stood in April and May 2017) and 86.2% of the 2,194 Adience subjects that could be labelled (Table 3). Their Figure 3 shows darker-skinned women as 4.4% of IJB-A’s subjects and 7.4% of Adience’s (the caption assigns the 4.4% to Adience, but the bars, the text of §3.5 and the darker-skinned totals of Table 3, 20.4% and 13.8%, agree with the values above). They built a more balanced benchmark of 1,270 parliamentarians from three African and three European countries (21.3% darker-skinned women) and found that three commercial gender classifiers, tested in April and May 2017, erred most on darker-skinned women; the study evaluated gender classification, not face recognition. The most extensive measurement for recognition, NIST’s 2019 report on demographic effects, used sequestered government photographs rather than a research dataset; the face recognition article summarizes both studies’ results and scope. The disclaimer on the Labeled Faces in the Wild home page, added in 2019 according to Peng, Mathur and Narayanan (2021), states that the benchmark has very few children, no babies, very few people over 80, a relatively small proportion of women and little or no representation of many ethnicities, and should not be used to conclude that an algorithm is suitable for commercial use (archived copy of January 2025). IBM’s Diversity in Faces approached the problem from the annotation side, releasing ten facial coding schemes computed for one million face images from the YFCC-100M Flickr collection so that coverage could be measured (Merler et al., 2019).

Label bias when the labels describe people

Label bias in an object dataset makes a “car” ambiguous; when the labels describe people, it can be harmful. ImageNet took its vocabulary from WordNet. Yang et al. (2020) examined the 2,832 categories of the person subtree of the full ImageNet (about 8.3% of its images; the 1,000 ILSVRC classes contain only three person categories). Twelve in-house annotators flagged 1,593 as unsafe, either offensive or sensitive, associated with 600,040 images, although the 2009 construction had already removed synsets whose WordNet gloss marked them as offensive or a slur. Of the remaining 1,239, crowd workers rated only 158 as imageable, with a score of at least 4, the threshold the authors propose; for the rest, such as “irreligionist”, nothing in a photograph shows the category, and its images, in the authors’ words, are “simply demonstrating annotators’ bias”. Crowd annotation of perceived gender, skin colour and age on 100 images each from the 139 safe, imageable synsets with at least 100 images found women under-represented overall and an average share of dark skin of only 6.2% across categories, with the largest shares in categories that align with stereotypes, such as “rapper” (66.4%). The ImageNet article records what was removed from the full release.

Birhane and Prabhu (2021) extended the critique to datasets inherited from WordNet without audit. In 80 Million Tiny Images they found slurs and other offensive nouns as class labels (Figure 1); in ILSVRC-2012, a model-assisted census estimated 83,436 images showing 101,070 to 132,201 people, with 62 misogynistic images confirmed by hand (Figure 5). Their Table 1 lists ImageNet, Tiny Images, Open Images, Places, JFT-300M and Tencent ML-Images among large datasets containing images of people, with zero images collected with the subjects’ consent, and they argue that a Creative Commons license addresses copyright, not consent to be in a training set.

Withdrawal and the life of a dataset

Some datasets have been withdrawn because of what their images or labels contained. On 29 June 2020 the creators of 80 Million Tiny Images withdrew it, citing a preprint of Birhane and Prabhu’s audit, and asked the community to stop using it and delete existing copies; the CIFAR article, whose datasets were drawn from Tiny Images, summarizes their statement. Peng, Mathur and Narayanan (2021) followed three face and person recognition datasets, Labeled Faces in the Wild (LFW), MS-Celeb-1M and the multi-camera tracking benchmark DukeMTMC, through 946 citing papers, samples of about 20% of the papers citing the last two and 5% of those citing LFW. In their account, MS-Celeb-1M and DukeMTMC were retracted in April 2019 after an investigation by Harvey and Laplace raised concerns about how the data were collected and used; Microsoft never gave ethical reasons for removing MS-Celeb-1M, and Peng et al. count the removal as a retraction because it responded to those concerns. Their findings:

• Retraction had limited effect. Copies, derived datasets and pre-trained models of MS-Celeb-1M remained available, and in the sampled papers published in 2020 MS-Celeb-1M or its derivatives were used 54 times and DukeMTMC or its derivatives 73 times. Neither retraction clearly told users to stop. Peng et al. report, citing press coverage, that Microsoft explained the removal by the end of the research challenge, and, citing a letter in a university newspaper, that a DukeMTMC creator apologized for a violation of IRB guidelines, an explanation that did not appear in the dataset’s official channels (Table 2). The face recognition article gives the details for MS-Celeb-1M.
• Derivatives raise new questions. Of 41 derivatives of the three datasets, 21 enabled new applications (DukeMTMC-ReID turned the tracking benchmark into a re-identification benchmark, used far more than the original) and 7 added annotations such as gender, race or “attractiveness”.
• Licenses did not restrict production use. MS-Celeb-1M’s license permitted only non-commercial research, but only 3 of 21 repositories of models pre-trained on it carried that restriction. The licenses of MS-Celeb-1M and DukeMTMC left unclear whether the restriction covers trained models, ImageNet’s did not prevent redistributing such models under commercial licenses, and Peng et al. found 18 such repositories for MS-Celeb-1M and 9 for ImageNet (§5, Table 3).
• Concerns change with time. LFW was introduced in 2007 as a research benchmark; by their count from archived copies of its results page, only 3 of 38 results added before 2014 came from commercial systems, against 41 of 49 after 2016 (Figure 2).

They conclude that a dataset’s harms cannot all be anticipated when it is created and call for stewardship over its whole life: clear and persistent licenses and documentation, explicit retractions, terms that govern derivatives, and checks by conference program committees that papers do not rely on retracted data.

Examples

The script builds three synthetic datasets of the same task, classifying a filled disc, square or triangle in a 32×32 grayscale image, that differ only in capture: studio images show a large light object centred on a plain grey backdrop, catalog images a mid-size dark object roughly centred on white, and street images an object anywhere in the frame on a smoothly cluttered background, lighter or darker than its surroundings. Each image is described by HOG cells pooled over the whole image and over its four quadrants, and every classifier is a logistic regression on those features. The script first plays Name That Dataset, then trains a shape classifier on each dataset and tests it on all three, then trains on two datasets together and tests on the third. Seed 0 is shown in full, and seeds 1–5 give the spread. It runs in about 15 seconds with NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2, and two runs give identical output.

import numpy as np
from scipy.ndimage import gaussian_filter
from skimage.feature import hog
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

S = 32                                            # image size (pixels)
yy, xx = np.mgrid[0:S, 0:S] + 0.5
NAMES = ["studio", "catalog", "street"]
SHAPES = ["disc", "square", "triangle"]

def shape_mask(k, cx, cy, r, th):
    """Filled disc (k=0), square (k=1) or triangle (k=2) of size r at (cx, cy), rotated by th."""
    u, v = xx - cx, yy - cy
    a, b = np.cos(th) * u + np.sin(th) * v, -np.sin(th) * u + np.cos(th) * v
    if k == 0:
        return u**2 + v**2 <= r**2
    if k == 1:
        return (np.abs(a) <= 0.8 * r) & (np.abs(b) <= 0.8 * r)
    return (b <= 0.6 * r) & (b >= -r + 1.7 * np.abs(a))

def render(style, n, rng):
    """n images of the three shapes. The styles differ only in how the pictures were 'taken'."""
    y = rng.integers(0, 3, n)
    X = np.empty((n, S, S))
    for i in range(n):
        if style == "studio":        # centred, large, light object on a plain grey backdrop
            cx, cy = S / 2 + rng.normal(0, 1, 2); r = rng.uniform(9, 11)
            bg = np.full((S, S), 0.45); fg = rng.uniform(0.8, 0.95)
        elif style == "catalog":     # roughly centred, mid-size, dark object on white
            cx, cy = S / 2 + rng.normal(0, 2.5, 2); r = rng.uniform(6, 9)
            bg = np.full((S, S), 0.92); fg = rng.uniform(0.05, 0.3)
        else:                        # "street": anywhere in the frame, on a cluttered background
            r = rng.uniform(6, 9); cx, cy = rng.uniform(r, S - r, 2)
            bg = 0.5 + 1.5 * gaussian_filter(rng.normal(0, 0.15, (S, S)), 1.5)
            fg = 0.5 + rng.choice([-1, 1]) * rng.uniform(0.3, 0.45)
        m = shape_mask(y[i], cx, cy, r, rng.uniform(-0.3, 0.3))
        X[i] = np.clip(np.where(m, fg, bg) + rng.normal(0, 0.04, (S, S)), 0, 1)
    return X, y

def features(X):
    """HOG cells (4x4 pixels, 9 orientations), pooled over the image and over four quadrants."""
    F = []
    for x in X:
        h = hog(x, orientations=9, pixels_per_cell=(4, 4), cells_per_block=(2, 2),
                feature_vector=False).reshape(7, 7, -1)
        quads = [h[a, b].max((0, 1)) for a in (slice(0, 4), slice(3, 7)) for b in (slice(0, 4), slice(3, 7))]
        F.append(np.concatenate([h.mean((0, 1)), h.max((0, 1))] + quads))
    return np.array(F)

def classifier():
    return make_pipeline(StandardScaler(), LogisticRegression(C=0.1, max_iter=3000))

def experiment(seed, n_train=1500, n_test=600):
    rng = np.random.default_rng(seed)
    data = {}
    for name in NAMES:
        Xtr, ytr = render(name, n_train, rng)
        Xte, yte = render(name, n_test, rng)
        data[name] = (features(Xtr), ytr, features(Xte), yte)
    # 1. Name That Dataset: predict which dataset an image came from (chance 1/3).
    d_tr, d_te = np.repeat(np.arange(3), n_train), np.repeat(np.arange(3), n_test)
    m = classifier().fit(np.vstack([data[n][0] for n in NAMES]), d_tr)
    name_acc = np.mean(m.predict(np.vstack([data[n][2] for n in NAMES])) == d_te)
    # 2. Cross-dataset generalization of the shape classifier: train on row i, test on column j.
    A = np.zeros((3, 3))
    for i, a in enumerate(NAMES):
        m = classifier().fit(data[a][0], data[a][1])
        for j, b in enumerate(NAMES):
            A[i, j] = np.mean(m.predict(data[b][2]) == data[b][3])
    # 3. Leave one dataset out: train on the other two together, test on the held-out one.
    held = []
    for h in NAMES:
        tr = [n for n in NAMES if n != h]
        m = classifier().fit(np.vstack([data[n][0] for n in tr]), np.concatenate([data[n][1] for n in tr]))
        held.append(np.mean(m.predict(data[h][2]) == data[h][3]))
    return name_acc, A, np.array(held)

name_acc, A, held = experiment(seed=0)
print(f"Name That Dataset (3 datasets, chance 33.3%): {100 * name_acc:.1f}% correct\n")
print("Shape accuracy (%), train on row, test on column")
print("train \\ test " + "".join(f"{n:>9s}" for n in NAMES) + "     self  mean others  drop")
for i, a in enumerate(NAMES):
    others = np.mean(np.delete(A[i], i))
    print(f"{a:>12s} " + "".join(f"{100 * v:9.1f}" for v in A[i])
          + f"   {100 * A[i, i]:6.1f}   {100 * others:9.1f}  {100 * (1 - others / A[i, i]):4.0f}%")
print("\nTrained on the other two, tested on the held-out dataset (%):",
      ", ".join(f"{n} {100 * v:.1f}" for n, v in zip(NAMES, held)))

runs = [experiment(seed) for seed in range(1, 6)]
na = np.array([r[0] for r in runs])
drop = np.array([[1 - np.mean(np.delete(r[1][i], i)) / r[1][i, i] for i in range(3)] for r in runs])
ho = np.array([r[2] for r in runs])
print("\nSeeds 1-5, min-max")
print(f"  Name That Dataset: {100 * na.min():.1f}-{100 * na.max():.1f}%")
for i, n in enumerate(NAMES):
    print(f"  {n:>8s}: drop {100 * drop[:, i].min():3.0f}% to {100 * drop[:, i].max():3.0f}%,"
          f" held-out accuracy {100 * ho[:, i].min():.1f}-{100 * ho[:, i].max():.1f}%")

Output:

Name That Dataset (3 datasets, chance 33.3%): 94.8% correct

Shape accuracy (%), train on row, test on column
train \ test    studio  catalog   street     self  mean others  drop
      studio     100.0     96.3     70.3    100.0        83.3    17%
     catalog      89.3     98.2     59.2     98.2        74.2    24%
      street      96.2     93.0     89.7     89.7        94.6    -5%

Trained on the other two, tested on the held-out dataset (%): studio 98.0, catalog 95.0, street 70.7

Seeds 1-5, min-max
  Name That Dataset: 93.9-94.9%
    studio: drop  14% to  17%, held-out accuracy 97.7-98.2%
   catalog: drop  19% to  25%, held-out accuracy 94.2-95.2%
    street: drop  -8% to  -2%, held-out accuracy 71.5-75.5%

The same features that classify shapes identify the source of an image about 94% of the time against 33% chance: object size, where the strongest gradients fall and whether the background is plain or textured are all visible in pooled HOG, just as framing and scene type were visible to Torralba and Efros’s descriptors. The cross-dataset table reproduces the pattern of their Table 1 on a small scale. The studio and catalog classifiers are nearly perfect at home and lose 14–25% of their accuracy elsewhere, most of it on street images, where objects are off-centre, smaller than in studio images and surrounded by edges that the centred datasets never showed them. The street classifier is the hardest to train and the worst at home, yet the best away: it scores higher on the two easy datasets than on its own test set, giving a negative drop, as PASCAL and ImageNet did on Caltech-101 and MSRC. Pooling two datasets is not a cure. Studio and catalog together share the same blind spot, centred objects on plain backgrounds, and reach only 70.7–75.5% on street images over the six seeds; training on varied data (street plus either centred set) transfers to the remaining centred set at 94–98%. Every image here shows the same three shapes; the only thing that changed between datasets was how the pictures were taken.

Reducing and Documenting Bias

No dataset is unbiased, so the remedies aim to make bias smaller, measurable and known to users:

• During collection. Torralba and Efros (2011, §6) suggest gathering images from multiple sources (several search engines in several countries, as ImageNet did), or starting from a large collection of unannotated images and labelling them by crowdsourcing rather than searching for each category. Against capture bias they suggest flipping and jittering images and automatic crops, the transformations that data augmentation now applies routinely; against negative-set bias, adding negatives from other datasets, or, following a suggestion of Mark Everingham, mining hard negatives from a large unlabelled set with standard algorithms and removing true positives by hand. DeVries et al. (2019) add searching in languages other than English.
• During training. Khosla et al. (2012) treated several datasets as biased samples of one visual world and learned, in a max-margin framework, a weight vector shared by all datasets plus a bias vector for each. On “car”, “person”, “bird”, “chair” and “dog” from PASCAL VOC 2007, LabelMe, Caltech-101 and SUN09, holding one dataset out, the shared weights beat a single SVM trained on the pooled data by an overall 2.8% mean AP for classification (better in 24 of the 25 comparisons of their Fig. 2) and 0.7% for detection, and the per-dataset bias vectors alone could play Name That Dataset (in their Table 2 a dataset’s own bias vector scores best in seven of eight cases; the text says all). Domain adaptation and transfer learning address the related case in which some data from the target setting are available.
• During evaluation. Report accuracy on more than one dataset, including one collected independently of the training set, and use leave-one-dataset-out tests; Torralba and Efros proposed that every new dataset be run through their battery of tests. For people, report errors per group rather than an average, as Buolamwini and Gebru did and as NIST’s reports do, and check that each group has enough examples for its error rate to mean something, the limitation the LFW disclaimer itself states.
• Re-balancing. Yang et al. (2020) chose not to release their image-level demographic annotations, because mistaken labels of gender, skin colour or age could harm the people depicted, and proposed instead a web interface that re-balances a category toward a target distribution of one attribute by removing images of over-represented groups. To protect the removed images’ annotations from being inferred, it returns only 90% of the minority group’s images and requires at least two groups of at least ten images each. They estimated it could balance 131 categories for gender, 117 for skin colour and 81 for age, and noted that balancing on two attributes at once leaves too few images.
• Documentation. Gebru et al. (2021) proposed that every dataset come with a datasheet, by analogy with the datasheets of electronic components, answering questions about its motivation, composition, collection process, preprocessing and labelling, recommended uses, distribution and maintenance. The questions in the arXiv version (v8, December 2021) include whether the dataset is a sample and whether it is representative of the larger set, whether it identifies subpopulations, and, for data about people, whether they were notified, whether they consented and whether they can revoke consent. Birhane and Prabhu (2021) propose “dataset audit cards” in the same spirit, recording known shortcomings alongside a dataset’s release.
• Stewardship. Peng, Mathur and Narayanan (2021) add the steps after release: persistent licenses and documentation in standard repositories, licenses that say whether their restrictions extend to trained models and derivatives, retractions stated explicitly wherever the data remain, and users and program committees checking that a dataset has not been retracted.

Common Misconceptions

• “A bigger dataset is an unbiased dataset.” Size reduces sampling error, not selection. Tiny Images and ImageNet were both built by querying image search engines with WordNet nouns, at very different scales, and both inherited WordNet’s offensive person categories; Torralba and Efros found that a training example from one dataset is worth less than one example from another when used on the other’s test set.
• “High accuracy on a held-out split shows that a model generalizes.” A held-out split is drawn from the same $P_k$ as the training data, so it measures generalization within one dataset’s habits. Only data collected differently, as in the cross-dataset matrix or a replication such as ImageNetV2, shows how much of the accuracy survives a change of source.
• “Dataset bias is about people and fairness.” The term began with cars, persons and mugs: Torralba and Efros’s capture-bias example is that almost all mugs from an image search have their handle on the right. Demographic under-representation is one consequence of the same selection processes, with higher stakes.
• “Balancing the classes removes the bias.” Balancing changes $P(y)$ or the frequency of a group; it does not change how each class or group was photographed, framed or labelled, and it can remove so much data that error rates per group can no longer be estimated.
• “Withdrawing a dataset removes it.” Copies, derived datasets and models trained on the data outlive the original, as MS-Celeb-1M’s history shows; a withdrawal that does not say why, and that is not visible where the data remain, leaves users unsure whether to stop.
• “Augmentation fixes capture bias.” Flips and crops add variation along the axes they cover, such as left–right orientation and small shifts, and nothing along others, such as viewpoint, background or the photographer’s choice of subject.

Where It Is Used

Dataset bias is the reason benchmarks are read with caution: a leaderboard ranks methods on one dataset’s habits, and benchmark overfitting and ImageNetV2 separate test-set reuse from distribution shift. The dataset articles each record their own biases: centred, aligned objects in Caltech-101; seasonal collection effects in PASCAL VOC; the frequency of the person class in COCO; label errors, offensive person categories and incidental faces in ImageNet; the withdrawn Tiny Images source of CIFAR; and the writers behind MNIST. In tasks, it governs how far image classification and object detection results transfer to new cameras and places, motivates transfer learning and domain adaptation, and is central to face recognition and face detection, where demographic differentials and the provenance of web-collected training sets are measured and debated.