Vision Systems & Engineering
Annotation Quality and Label Noise
How to measure and repair the quality of human labels for vision datasets; raw and chance-corrected agreement (Cohen's kappa, Scott's pi, Fleiss' kappa, Krippendorff's alpha) and why agreement is not accuracy, latent-truth aggregation with Dawid–Skene EM against the majority vote, finding wrong labels with confident learning, and what label noise does to training and evaluation.
intermediate
Every supervised vision model is trained and tested against labels that people produced, and people make mistakes and disagree. Annotation quality is how close those labels are to the truth the guidelines intend; label noise is the part that is wrong, a given label that differs from the true one. Image annotation describes how labels are collected, how guidelines, qualification tests, gold questions and redundancy keep them in check, and how agreement is measured for boxes, masks and keypoints. This article goes one step further for categorical labels, such as an image’s class or an object’s category: how to measure agreement so that it is not fooled by chance, why agreement is not the same as accuracy, how to turn several noisy labels into one estimate of the truth without any ground truth, and how to find the wrong labels in a dataset from a model’s predictions.
Definition
Let each image have a true class that nobody observes directly. An annotator, or a pipeline such as a web search or a report parser, produces a given label . The labels are noisy where . Northcutt, Jiang and Chuang (2021) call the noisy (given) label and the latent, uncorrupted label. Label noise can be told apart by what the probability of a flip depends on (our grouping; the terms are Northcutt et al.’s and Patrini et al.’s):
- Uniform (symmetric) noise: a label is replaced, with some probability, by a class drawn uniformly at random, whatever the image and its class. The label smoothing and loss function articles discuss training under this kind of noise.
- Class-conditional noise: the flip probability depends on the true class only, , which can be written as a noise matrix. Northcutt et al. adopt this assumption, which they attribute to Angluin and Laird’s (1988) classification noise process, because some confusions are far more likely than others: in their ImageNet example a leopard is more likely to be labelled jaguar than bathtub. Patrini et al. (2017) call the same object the noise transition matrix , with , rows indexed by the true class.
- Noise that depends on the image: the flip probability also depends on the image itself, for example on how small, blurred or ambiguous the object is. Northcutt et al. note that there are exceptions to the class-conditional assumption; the annotators simulated below are of this kind, because their error rate grows with each image’s difficulty.
Two further quantities describe the people rather than the labels. Agreement is how often annotators give the same label to the same item; accuracy is how often an annotator’s label is the true one. They are different: two annotators can agree on the same wrong answer, and with a skewed class distribution they can agree often by chance.
Intuition
Suppose two annotators sort images of four shapes. If one shape makes up 55% of the images, two annotators who knew nothing and simply answered in proportion to that frequency would still agree on about two images in five (the sum of the squared class frequencies, 0.39). Raw agreement therefore has to be read against the agreement chance alone would give, and that baseline grows as the classes become more imbalanced. Chance-corrected coefficients such as Cohen’s kappa subtract it.
When several annotators label the same image, the obvious way to combine them is a majority vote. Raykar et al. (2010) point out that a vote assumes all annotators are equally good: if one expert and several novices label an image and the novices give the same wrong label, the vote follows the novices. If each annotator’s typical mistakes were known, their labels could be weighted accordingly. Those mistakes can be estimated without any ground truth, because an annotator whose labels rarely match everyone else’s consensus reveals that in the data; the estimate and the consensus are then refined together. This is the idea of Dawid and Skene’s (1979) model, as Raykar et al. describe it. It breaks down in the case it cannot see: errors that several annotators share look exactly like agreement.
To find wrong labels in a dataset with one label per image, a model trained on the same noisy labels can serve as a second opinion, provided its predictions for each image come from a model that did not see that image’s label (cross-validation gives such out-of-sample predictions). An image whose label the model finds unlikely, while it confidently predicts another class, is a candidate error. Confident learning makes this precise by counting, class by class, how many images look like they belong to another class than their label.
Formal Definition
Chance-corrected agreement
For two coders labelling items, the observed agreement is the fraction of items they label alike. Artstein and Poesio (2008) write the three classic coefficients in one form,
where is the agreement expected by chance, , and the coefficients differ only in how , the chance that coder assigns an arbitrary item to category , is estimated (their §2.4):
- Bennett, Alpert and Goldstein’s (1954) assumes a uniform distribution over the categories, . Artstein and Poesio note that it can be raised simply by adding categories nobody uses.
- Scott’s (Scott, 1955) assumes the same distribution for every coder and estimates it from both coders’ labels pooled, , so .
- Cohen’s (Cohen, 1960) gives each coder its own distribution, estimated from that coder’s labels, so .
With more than two coders, agreement on each item is measured as the fraction of agreeing pairs of judgments, the solution Artstein and Poesio attribute to Fleiss (1971). With coders and of them assigning item to category , Artstein and Poesio (§2.5.1) write
The expected agreement uses one distribution pooled over all coders, so this coefficient, known as Fleiss’ kappa, is a generalization of Scott’s , not of Cohen’s . Artstein and Poesio call it multi-, note that Siegel and Castellan’s is the same coefficient, and point to Davies and Fleiss (1982) for a true multi-coder ; they trace much confusion in the literature to the name. With two coders, multi- reduces to Scott’s (our derivation; check C2 in the example confirms it numerically). Krippendorff’s is defined in terms of disagreement, so that some disagreements can count more than others; it handles multiple coders and missing values, and when all disagreements are weighted equally it is nearly identical to multi-, with a small-sample correction (their §2.6).
The difference between the two families has a closed form. With the annotator bias , which Artstein and Poesio state (referring to an earlier report of theirs for the proof) equals the summed variance of the coders’ category proportions divided by ,
It follows that , and the two coincide when the coders’ marginal distributions agree; with more coders the difference shrinks (their §3.1).
The gap between raw and chance-corrected agreement also has a closed form, which follows from the definition (our derivation):
For a fixed raw agreement below 1, the gap grows with , and grows as the label distribution becomes more skewed. Artstein and Poesio (§3.2) call this the prevalence problem: with two categories, one rare, identical marginals, a proportion of agreement on the rare category and for half the disagreement,
which is about the fraction of one coder’s rare labels that the other coder also gave. They argue that this is justified rather than paradoxical: when one category dominates, indiscriminate coding also gives high agreement and high accuracy, so reliability shows in agreement on the rare categories.
Aggregation with a latent truth
The model behind Dawid and Skene (1979), in the form Raykar et al. (2010) describe (their classifier-free binary algorithm, item 4 of §2.7, combined here with the per-annotator multinomial confusion parameters of their multi-class extension, §3), gives each annotator a confusion matrix and the classes a prior . If annotators err independently given the true class, the probability of the labels that item received is
The true classes are hidden, so the parameters are fitted by expectation–maximization. The E step computes each item’s posterior ; the M step re-estimates as the mean of and as the posterior-weighted fraction of class- items that annotator labelled . Missing labels are handled by summing only over the items each annotator labelled (Raykar et al., §2.7, item 5). Raykar et al. note that EM is only guaranteed to reach a local maximum, that restarts from several initializations can help, and that they initialize from the majority vote (§2.5). Their own method goes further and learns a classifier jointly with the annotators’ error rates; the classifier-free version is the one they describe as similar to Dawid and Skene’s. The model also has a symmetry that the initialization resolves: renaming the classes consistently in every and in leaves the likelihood unchanged, so an EM run started without reference to the votes can converge to a relabelled solution (our derivation).
Confident learning
Northcutt, Jiang and Chuang (2021) start from out-of-sample predicted probabilities from any model trained on the noisy labels, and the given labels. For each class the threshold is the average self-confidence, the mean predicted probability of class among the images labelled (their Eq. 2):
The confident joint counts, for each given label , the images whose predicted probability for some class reaches ; if several classes reach their thresholds, the one with the largest probability is taken (their Eq. 1). Images with no class above its threshold are not counted. The counts are calibrated so that each row sums to the number of images with that given label and the matrix sums to one, giving an estimate of the joint distribution of given and true labels (Eq. 3), from which the noise matrix and the prior of the true labels follow. Northcutt et al. index with the given label first.
They compare five ways to turn these counts into a set of suspected errors (§3.2): the baseline , which flags every image whose predicted class differs from its label (a single iteration of Chen et al.’s INCV method, as they describe it); , the images in the off-diagonal cells of the confident joint; prune by class (PBC), the images of each class with the lowest self-confidence; prune by noise rate (PBNR), for each off-diagonal cell the images labelled with the largest margin ; and C+NR, the images both PBC and PBNR flag. The paper uses by default, because it matches the conditions of their Theorem 2. The open-source cleanlab package, in version 2.9.0, defaults find_label_issues to filter_by="prune_by_noise_rate", that is PBNR (cleanlab/filter.py, line 63; Apache-2.0). The per-class thresholds are what Northcutt et al. credit for robustness to class imbalance and to classes whose probabilities run high or low; they show that fails when the distributions of predicted probabilities differ between classes, for example under class imbalance (their Theorem 2), while the confident joint exactly recovers the label errors under their stated conditions, including when the predicted probabilities equal the true noise rates and each diagonal entry of the noise matrix is the largest in its row and column (Condition 1 and Theorem 1).
Properties
- Chance correction depends on the class distribution, not only on the annotators. The same two annotators, with the same error process, get a lower kappa on a more skewed dataset (the example shows it). Values are therefore comparable only across studies with similar class distributions (our reading of the prevalence formula).
- Interpretation scales are conventions. Artstein and Poesio reproduce Landis and Koch’s (1977) bands, from “poor” below 0 through “slight”, “fair”, “moderate” and “substantial” to the top band from 0.8, and contrast them with the stricter content-analysis convention attributed to Krippendorff: above 0.8 good, 0.67 to 0.8 for tentative conclusions, and in his later work 0.8 as a minimum. They describe the choice of an adequate level as little more than a black art (§4.1.3).
- Which coefficient to report depends on the question. For the reliability of an annotation scheme, Artstein and Poesio recommend a single-distribution coefficient such as or , chosen by the task rather than by the observed marginals; individual-distribution coefficients such as say more about the trustworthiness of particular annotators’ data (§3.1). They also recommend testing reliability with more than two coders, to make it less likely that coders share a deviant reading of the guidelines.
- Latent-truth aggregation weights annotators by estimated quality, so spammers and biased annotators lose influence, and it estimates each annotator’s confusion matrix as a by-product, which can drive feedback or payment (our reading). It assumes errors are independent given the true class.
- Shared errors are invisible to agreement. Neither a vote nor a model that assumes independent errors can tell a shared misreading from the truth, since both see consistent labels. Image Annotation’s example shows the geometric version of this for masks.
- Confident learning needs good out-of-sample probabilities. Its guarantees are stated for ideal probabilities and for probabilities perturbed in the ways Northcutt et al.’s §4 specifies; a model that fits its own training labels exactly would give no signal (our reasoning), so the probabilities must come from held-out folds. Northcutt et al. used folds in their cross-validated experiments.
Consequences for training. Deep networks can fit wrong labels. Arpit et al. (2017) compared networks trained on real data with networks trained on data whose labels or inputs had been randomized, and concluded that networks trained with SGD variants first use patterns rather than brute-force memorization to fit real data; they also found that suitably tuned explicit regularization such as dropout can degrade training on noise without hurting generalization on real data. When the noise matrix is known, Patrini et al. (2017) correct the loss in two ways: the backward correction multiplies the vector of per-class losses by , giving an unbiased estimate of the clean loss, and the forward correction multiplies the model’s predicted probabilities by before the loss. They estimate from the model’s predictions on “perfect examples” of each class (in most of their experiments the 97th percentile rather than the maximum), and in their experiments the forward correction did better than the backward one, with the noise estimator as the bottleneck. Confident learning instead removes the suspected errors and reweights the remaining classes before retraining (Northcutt et al., §3.2). Robust losses, smoothing and sample selection belong to learning with noisy labels; the loss function article gives the condition under which a loss is robust to uniform noise, and label smoothing its relation to the backward correction.
Consequences for evaluation. Errors in test labels change the measurement itself. Image Annotation reports how many errors confident learning and human review found in the test sets of ten benchmarks, in Northcutt, Athalye and Mueller’s (2021) study, a different paper from Northcutt, Jiang and Chuang’s description of confident learning; and benchmark overfitting and benchmarks and leaderboards discuss what happens to rankings when score differences approach the level of label noise. The examples in CelebA and chest X-ray analysis score models against noisy and true labels side by side.
Examples
Agreement, aggregation and error finding on rendered shapes
The script below renders 2,000 grayscale images of pixels, each showing one of four shapes (disk, square, triangle, cross), under two class priors: balanced, and imbalanced at 0.55, 0.25, 0.15 and 0.05. Every image has a known difficulty that lowers the shape’s contrast from 1 to 0.3 and raises a Gaussian blur from 0.5 to 2 pixels, with pixel noise of standard deviation 0.15. Seven simulated annotators label the images; each answers wrongly with a probability that rises linearly with , and a wrong answer is drawn from the annotator’s own error pattern. A1 and A2 are careful (), A3 is good (), A4 () answers “disk” when wrong (“square” when the image is a disk), A5 () answers the next class, A6 answers at random (error 0.75, which makes its labels uniform) and A7 is near-random (0.60). A5, A6 and A7 label only a random 60% of the images.
- Part A, agreement. Raw agreement, Cohen’s and Scott’s for the careful pair, beside each annotator’s true accuracy, and Fleiss’ for all seven on the images that all of them labelled. The hand-written is checked against scikit-learn’s
cohen_kappa_score(unweighted,weights=None), Fleiss’ against statsmodels’fleiss_kappa, which takes an items × categories count table and requires the same number of raters on every item, and multi- on the pair against . - Part B, aggregation. Majority vote against Dawid–Skene EM in four conditions: B1, the seven annotators above; B2, seven equal annotators (, labelling everything); B3a and B3b, five good annotators () of whom two or three also share one misreading, answering “square” for every triangle with . Accuracy is scored against the truth, beside an oracle that applies the posterior with the exact confusion matrices (averaged over ) and the true prior. The script also prints the error of the estimated confusion matrices (mean absolute difference from each annotator’s empirical confusion matrix against the truth) for EM and for counting against the vote, the number of tied votes, EM’s iteration count and, in B3, the accuracy on the affected items.
- Part C, finding planted errors. The true labels are flipped with a known class-conditional matrix, rows the true class and columns the given label, with 0.80 on the diagonal (an expected 20% of flips). Logistic regression on HOG features gives out-of-fold probabilities from 4-fold stratified cross-validation, and the three methods , and PBNR are written from Northcutt et al.’s equations (not ported from cleanlab). Precision and recall are measured against the known flips, overall and for the images whose true class is triangle or cross, the rare classes under the imbalanced prior.
Conventions: confusion matrices have the true class in rows and the given label in columns; tied votes are broken uniformly at random with the seed’s generator; EM starts from the vote fractions, as Raykar et al. initialize it, a modelling choice; EM adds a pseudo-count of 0.01 to every count, so it maximizes a posterior rather than the likelihood, and the monotonicity check is on that objective; PBNR’s counts are rounded to the nearest integer, and its selections for different cells of a row are united.
The checks: C0, the data have the planned properties (class fractions within 0.035 of the prior, each annotator’s accuracy within 4 standard errors of its exact expected value, the subset annotators’ missing fraction within 0.04 of 0.40, the flip rate within 4 standard errors of 0.20); C1 and C2, the hand-written coefficients equal the libraries’ to ; C3, EM’s objective never decreases (by more than ); C4, EM’s accuracy is within 0.015 of the oracle’s in B1 and B2; C5, with ideal probabilities (each image’s probabilities set to the row of the flip matrix for its true class, Northcutt et al.’s Condition 1), and flag exactly the flipped images; C6, a label-free check that the task is learnable but not trivial, the out-of-fold accuracy on the true labels between 0.60 and 0.98 and the training-fold accuracy on the noisy labels at most 0.99 (the model does not fit its training labels). The hypotheses, from the plan written before the code ran: H1, the careful pair’s is below its raw agreement and the gap is wider under the imbalanced prior; H2a, in B1 Dawid–Skene is right on at least as many images as the vote; H2b, in B2 the two differ by no more than , with and the images that exactly one of them gets right; H3, in B3b, where three of five annotators share the misreading, Dawid–Skene does no better than the vote on the affected images (within the same bound); H4, the precision of and of PBNR exceeds the flip rate. The comparison of the methods on the rare classes has no hypothesis; Northcutt et al. claim that confident learning is more robust to class imbalance than . The script takes the seeds as arguments (with none it runs seed 0 only); it was run with the arguments 0 1 2 3 4, prints the first seed in full and every seed’s deciding values in a table. It uses NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2, scikit-learn 1.7.2 and statsmodels 0.14.5 on one thread, runs in about 16 seconds, and two runs gave identical output.
import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
os.environ[v] = "1" # one BLAS thread, set before NumPy loads
import sys
import numpy as np
from scipy.ndimage import gaussian_filter
from skimage.draw import disk, polygon
from skimage.feature import hog
from skimage.transform import downscale_local_mean
from sklearn.metrics import cohen_kappa_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from statsmodels.stats.inter_rater import fleiss_kappa
K, N, S = 4, 2000, 24
NAMES = ["disk", "square", "triangle", "cross"]
PRIORS = {"balanced": [0.25, 0.25, 0.25, 0.25], "imbalanced": [0.55, 0.25, 0.15, 0.05]}
# ---------------------------------------------------------------- images
def render(cls, rng):
"""One 24x24 image of a shape of class cls; returns image and difficulty d."""
up = 4 # draw at 4x, then average
m = np.zeros((S * up, S * up))
s = rng.uniform(6, 9) * up
r0, c0 = (S / 2 + rng.uniform(-2, 2, 2)) * up
if cls == 0:
m[disk((r0, c0), s, shape=m.shape)] = 1
elif cls == 1:
h = 0.85 * s
m[int(r0 - h):int(r0 + h), int(c0 - h):int(c0 + h)] = 1
elif cls == 2:
a = np.deg2rad([-90, 30, 150])
m[polygon(r0 + s * np.sin(a), c0 + s * np.cos(a), m.shape)] = 1
else:
w = 0.3 * s
m[int(r0 - s):int(r0 + s), int(c0 - w):int(c0 + w)] = 1
m[int(r0 - w):int(r0 + w), int(c0 - s):int(c0 + s)] = 1
m = downscale_local_mean(m, (up, up))
d = rng.uniform() # known per-image difficulty
img = rng.uniform(0.1, 0.3) + (1 - 0.7 * d) * m # contrast falls with d
img = gaussian_filter(img, 0.5 + 1.5 * d) # blur grows with d
return img + rng.normal(0, 0.15, img.shape), d
def make_images(prior, rng):
y = rng.choice(K, size=N, p=prior)
out = [render(c, rng) for c in y]
imgs = np.array([o[0] for o in out]); d = np.array([o[1] for o in out])
X = np.array([hog(im, orientations=9, pixels_per_cell=(6, 6), cells_per_block=(2, 2)) for im in imgs])
return y, d, X
# ---------------------------------------------------------------- annotators
UNI = (np.ones((K, K)) - np.eye(K)) / (K - 1) # wrong answers uniform over the others
TO0 = np.zeros((K, K)); TO0[1:, 0] = 1; TO0[0, 1] = 1 # wrong answers go to class 0 (0 -> 1)
NEXT = np.roll(np.eye(K), 1, axis=1) # wrong answer y -> y+1 mod 4
def ann(a, b, R, frac=1.0, bias=False):
return dict(a=a, b=b, R=R, frac=frac, bias=bias) # error probability e(d) = a + b d
B1 = [ann(.02, .16, UNI), ann(.02, .16, UNI), ann(.05, .20, UNI), ann(.05, .30, TO0),
ann(.05, .30, NEXT, .6), ann(.75, 0, UNI, .6), ann(.60, 0, UNI, .6)]
B2 = [ann(.15, .40, UNI) for _ in range(7)]
B3a = [ann(.05, .20, UNI, bias=j < 2) for j in range(5)]
B3b = [ann(.05, .20, UNI, bias=j < 3) for j in range(5)]
CONDS = [("B1 quality differs", B1), ("B2 all equal", B2),
("B3a 2 of 5 share bias", B3a), ("B3b 3 of 5 share bias", B3b)]
def annotate(y, d, anns, rng):
L = np.full((N, len(anns)), -1)
for j, A in enumerate(anns):
lab = rng.uniform(size=N) < A["frac"]
wrong = rng.uniform(size=N) < A["a"] + A["b"] * d
alt = np.array([rng.choice(K, p=A["R"][c]) for c in y])
lj = np.where(wrong, alt, y)
if A["bias"]:
lj = np.where((y == 2) & (d > 0.5), 1, lj) # shared misreading: hard triangles -> square
L[lab, j] = lj[lab]
return L
def pop_confusion(A):
"""Exact confusion matrix P(label | true) averaged over d ~ U(0,1)."""
e = A["a"] + A["b"] / 2
C = (1 - e) * np.eye(K) + e * A["R"]
if A["bias"]: # d > 0.5: always square; d < 0.5: as usual
e0 = A["a"] + A["b"] / 4
C[2] = 0.5 * ((1 - e0) * np.eye(K)[2] + e0 * A["R"][2]); C[2, 1] += 0.5
return C
# ---------------------------------------------------------------- agreement
def cohen(l1, l2):
C = np.zeros((K, K)); np.add.at(C, (l1, l2), 1); C /= C.sum()
Ao = np.trace(C); Ae = C.sum(1) @ C.sum(0) # individual marginals
return Ao, (Ao - Ae) / (1 - Ae)
def scott(l1, l2):
Ao = np.mean(l1 == l2)
P = np.bincount(np.r_[l1, l2], minlength=K) / (2 * len(l1)) # pooled marginal
Ae = P @ P
return (Ao - Ae) / (1 - Ae)
def multi_pi(L):
"""Fleiss's multi-pi from an items x coders label array (every item fully labelled)."""
i, c = L.shape
n = np.stack([(L == k).sum(1) for k in range(K)], 1) # n_ik
Ao = (n * (n - 1)).sum() / (i * c * (c - 1))
P = n.sum(0) / (i * c)
return Ao, (Ao - P @ P) / (1 - P @ P), n
# ---------------------------------------------------------------- aggregation
def majority(L, rng):
cnt = np.stack([(L == k).sum(1) for k in range(K)], 1)
top = cnt == cnt.max(1, keepdims=True)
ties = int((top.sum(1) > 1).sum())
pick = np.array([rng.choice(np.flatnonzero(t)) for t in top]) # random tie-break
return pick, cnt, ties
def posterior(L, prior, conf):
lp = np.tile(np.log(prior), (N, 1))
for j in range(L.shape[1]):
m = L[:, j] >= 0
with np.errstate(divide="ignore"): # log 0 = -inf for impossible labels (oracle)
lp[m] += np.log(conf[j][:, L[m, j]]).T
return lp
def dawid_skene(L, cnt, alpha=0.01, tol=1e-10, max_it=1000):
T = cnt / cnt.sum(1, keepdims=True) # initialization: vote fractions
J = L.shape[1]; objs = []
for it in range(1, max_it + 1):
prior = (T.sum(0) + alpha) / (N + K * alpha) # M-step (Dirichlet pseudo-count alpha)
conf = []
for j in range(J):
m = L[:, j] >= 0
Cj = np.full((K, K), alpha)
for k in range(K):
Cj[:, k] += T[m & (L[:, j] == k)].sum(0)
conf.append(Cj / Cj.sum(1, keepdims=True))
lp = posterior(L, prior, conf) # E-step
mx = lp.max(1, keepdims=True)
ll = (mx[:, 0] + np.log(np.exp(lp - mx).sum(1))).sum()
objs.append(ll + alpha * (np.log(prior).sum() + sum(np.log(c).sum() for c in conf)))
Tn = np.exp(lp - mx); Tn /= Tn.sum(1, keepdims=True)
done = np.abs(Tn - T).max() < tol
T = Tn
if done:
break
return T.argmax(1), conf, np.diff(objs), it
def emp_conf(ref, L, j):
m = L[:, j] >= 0
C = np.zeros((K, K)); np.add.at(C, (ref[m], L[m, j]), 1)
with np.errstate(invalid="ignore"):
return C / C.sum(1, keepdims=True)
def cm_error(est, y, L):
return np.nanmean([np.abs(est[j] - emp_conf(y, L, j)) for j in range(L.shape[1])])
# ---------------------------------------------------------------- confident learning
T_NOISE = np.array([[.80, .12, .04, .04], [.15, .80, .03, .02], [.03, .15, .80, .02], [.02, .03, .15, .80]])
def confident_learning(P, g):
"""Northcutt, Jiang & Chuang (2021): Eqs. 1-3, C_confusion, C_{y~,y*} and PBNR."""
t = np.array([P[g == j, j].mean() for j in range(K)]) # Eq. 2: per-class thresholds
above = P >= t
yhat = np.where(above.any(1), np.where(above, P, -np.inf).argmax(1), -1) # Eq. 1, collisions by argmax
C = np.zeros((K, K), int); m = yhat >= 0
np.add.at(C, (g[m], yhat[m]), 1) # confident joint
nX = np.bincount(g, minlength=K)
rows = C.sum(1, keepdims=True)
Cc = np.divide(C, rows, out=np.zeros((K, K)), where=rows > 0) * nX[:, None]
Q = Cc / Cc.sum() # Eq. 3
cj = m & (yhat != g) # CL method 2: off-diagonals of C
conf = P.argmax(1) != g # baseline C_confusion
pbnr = np.zeros(len(g), bool) # CL method 4: prune by noise rate
for i in range(K):
idx = np.flatnonzero(g == i)
for j in range(K):
if i == j:
continue
mnum = int(np.round(len(g) * Q[i, j]))
if mnum > 0:
marg = P[idx, j] - P[idx, i]
pbnr[idx[np.argsort(-marg, kind="stable")[:mnum]]] = True
est = len(g) * (Q.sum() - np.trace(Q))
return {"C_confusion": conf, "C_conf.joint": cj, "PBNR": pbnr}, est
def pr(flag, truth):
tp = (flag & truth).sum()
return tp / max(flag.sum(), 1), tp / max(truth.sum(), 1)
# ---------------------------------------------------------------- one seed
def run(seed, verbose):
rng = np.random.default_rng(seed)
res = {"checks": {}}
ck = res["checks"]
ck.update(c0a=0.0, c0b=0.0, c0c=0.0, c0d=0.0, c1=0.0, c2=0.0, c2b=0.0, c3=np.inf, c4=0.0,
c5=0, c6lo=1.0, c6hi=0.0, c6tr=0.0)
for pname, prior in PRIORS.items():
y, d, X = make_images(prior, rng)
ck["c0a"] = max(ck["c0a"], np.abs(np.bincount(y, minlength=K) / N - prior).max())
# ------------------------------ Part A: agreement (annotators of B1)
L = annotate(y, d, B1, rng)
accs = []
for j, A in enumerate(B1):
m = L[:, j] >= 0
acc = np.mean(L[m, j] == y[m]); ea = 1 - (A["a"] + A["b"] / 2)
ck["c0b"] = max(ck["c0b"], abs(acc - ea) / np.sqrt(ea * (1 - ea) / m.sum()))
if A["frac"] < 1:
ck["c0c"] = max(ck["c0c"], abs((~m).mean() - 0.4))
accs.append(acc)
Ao, kap = cohen(L[:, 0], L[:, 1])
ck["c1"] = max(ck["c1"], abs(kap - cohen_kappa_score(L[:, 0], L[:, 1])))
pi2 = scott(L[:, 0], L[:, 1])
ck["c2b"] = max(ck["c2b"], abs(multi_pi(L[:, :2])[1] - pi2))
full = (L >= 0).all(1)
Ao7, f7, n7 = multi_pi(L[full])
ck["c2"] = max(ck["c2"], abs(f7 - fleiss_kappa(n7)))
res[("A", pname)] = dict(Ao=Ao, kappa=kap, pi=pi2, gap=Ao - kap, nfull=int(full.sum()),
Ao7=Ao7, fleiss=f7)
if verbose:
print(f"\n=== seed {seed}, {pname} prior {prior}")
print("Part A true accuracy of A1..A7: " + " ".join(f"{a:.3f}" for a in accs))
print(f" careful pair A1-A2: agreement {Ao:.4f} Cohen kappa {kap:.4f} Scott pi {pi2:.4f}"
f" gap (agreement - kappa) {Ao - kap:.4f}")
print(f" all seven on the {full.sum()} fully labelled images: pairwise agreement {Ao7:.4f}"
f" Fleiss kappa {f7:.4f}")
print("Part B condition vote D-S oracle ties iters CM err D-S / vote"
" affected: n vote D-S")
# ------------------------------ Part B: aggregation
for cname, anns in CONDS:
Lc = L if cname.startswith("B1") else annotate(y, d, anns, rng)
if cname.startswith("B2"):
for j, A in enumerate(anns):
acc = np.mean(Lc[:, j] == y); ea = 1 - (A["a"] + A["b"] / 2)
ck["c0b"] = max(ck["c0b"], abs(acc - ea) / np.sqrt(ea * (1 - ea) / N))
mv, cnt, ties = majority(Lc, rng)
ds, conf, steps, its = dawid_skene(Lc, cnt)
ck["c3"] = min(ck["c3"], steps.min() if len(steps) else np.inf)
orc = posterior(Lc, np.array(prior), [pop_confusion(A) for A in anns]).argmax(1)
cmv = [emp_conf(mv, Lc, j) for j in range(Lc.shape[1])]
r = dict(mv=int((mv == y).sum()), ds=int((ds == y).sum()), orc=int((orc == y).sum()),
b=int(((mv == y) & (ds != y)).sum()), c=int(((mv != y) & (ds == y)).sum()))
aff = (y == 2) & (d > 0.5)
r.update(naff=int(aff.sum()), mv_aff=int((mv[aff] == 2).sum()), ds_aff=int((ds[aff] == 2).sum()),
b_aff=int(((mv[aff] == 2) & (ds[aff] != 2)).sum()),
c_aff=int(((mv[aff] != 2) & (ds[aff] == 2)).sum()))
res[("B", pname, cname[:3])] = r
if cname[:2] in ("B1", "B2"):
ck["c4"] = max(ck["c4"], abs(r["ds"] - r["orc"]) / N)
if verbose:
aff_s = (f" {r['naff']:4d} {r['mv_aff'] / r['naff']:.3f} {r['ds_aff'] / r['naff']:.3f}"
if cname.startswith("B3") else "")
print(f" {cname:<24} {r['mv'] / N:.4f} {r['ds'] / N:.4f} {r['orc'] / N:.4f} {ties:5d} {its:5d}"
f" {cm_error(conf, y, Lc):.4f} / {cm_error(cmv, y, Lc):.4f}{aff_s}")
# ------------------------------ Part C: confident learning on planted flips
g = np.array([rng.choice(K, p=T_NOISE[c]) for c in y])
flipped = g != y
ck["c0d"] = max(ck["c0d"], abs(flipped.mean() - 0.2) / np.sqrt(0.2 * 0.8 / N))
pipe = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=5000))
cv = StratifiedKFold(4, shuffle=True, random_state=seed)
acc_true = np.mean(cross_val_predict(pipe, X, y, cv=cv) == y) # label-free check
tr = [np.mean(pipe.fit(X[a], g[a]).predict(X[a]) == g[a]) for a, _ in cv.split(X, g)]
P = cross_val_predict(pipe, X, g, cv=cv, method="predict_proba")
ck["c6lo"] = min(ck["c6lo"], acc_true); ck["c6hi"] = max(ck["c6hi"], acc_true)
ck["c6tr"] = max(ck["c6tr"], np.mean(tr))
flags, est = confident_learning(P, g)
ideal, _ = confident_learning(T_NOISE[y], g) # Condition 1 probabilities
ck["c5"] = max(ck["c5"], int((ideal["C_conf.joint"] != flipped).sum()),
int((ideal["C_confusion"] != flipped).sum()))
rare = y >= 2
res[("C", pname)] = dict(rate=flipped.mean(), **{k: pr(v, flipped)[0] for k, v in flags.items()})
if verbose:
print(f"Part C out-of-fold accuracy on true labels {acc_true:.4f}; mean training-fold accuracy"
f" on given labels {np.mean(tr):.4f}")
print(f" flipped labels {flipped.sum()} ({flipped.mean():.4f}); errors estimated from Q {est:.1f}")
print(" method flagged precision recall | true class 2-3: flagged precision recall")
for k, v in flags.items():
p_, r_ = pr(v, flipped); pr_, rr_ = pr(v & rare, flipped & rare)
print(f" {k:<14} {v.sum():8d} {p_:.4f} {r_:.4f} | {(v & rare).sum():16d} {pr_:.4f} {rr_:.4f}")
return res
if __name__ == "__main__":
seeds = [int(s) for s in sys.argv[1:]] or [0]
allres = {s: run(s, s == seeds[0]) for s in seeds}
print("\n=== per-seed hypothesis values (counts out of N = 2000; bounds = 2 sqrt(b + c) / n)")
print("seed prior H1 gap H2a vote/D-S H2b |diff| bound H3 aff vote D-S bound H4 rate CJ prec PBNR prec")
verdict = {h: True for h in ["H1", "H2a", "H2b", "H3", "H4"]}
for s, R in allres.items():
for p in PRIORS:
A, C = R[("A", p)], R[("C", p)]
b1, b2, b3 = R[("B", p, "B1 ")], R[("B", p, "B2 ")], R[("B", p, "B3b")]
dif, bd = abs(b2["ds"] - b2["mv"]) / N, 2 * np.sqrt(b2["b"] + b2["c"]) / N
n3 = b3["naff"]; a_mv, a_ds = b3["mv_aff"] / n3, b3["ds_aff"] / n3
bd3 = 2 * np.sqrt(b3["b_aff"] + b3["c_aff"]) / n3
verdict["H1"] &= A["kappa"] < A["Ao"]
verdict["H2a"] &= b1["ds"] >= b1["mv"]
verdict["H2b"] &= dif <= bd
verdict["H3"] &= a_ds <= a_mv + bd3
verdict["H4"] &= C["C_conf.joint"] > C["rate"] and C["PBNR"] > C["rate"]
print(f"{s:4d} {p:<11} {A['gap']:.4f} {b1['mv']:5d}/{b1['ds']:5d} {dif:.4f} {bd:.4f}"
f" {a_mv:.4f} {a_ds:.4f} {bd3:.4f} {C['rate']:.4f} {C['C_conf.joint']:.4f} {C['PBNR']:.4f}")
verdict["H1"] &= R[("A", "imbalanced")]["gap"] > R[("A", "balanced")]["gap"]
print("hypotheses held on every seed and prior: " + ", ".join(f"{h} {'held' if v else 'failed'}"
for h, v in verdict.items()))
ck = {k: (max if k not in ("c3", "c6lo", "c5") else (min if k in ("c3", "c6lo") else max))(
R["checks"][k] for R in allres.values()) for k in allres[seeds[0]]["checks"]}
print("checks over all seeds:")
print(f" C0 data: max |class fraction - prior| {ck['c0a']:.4f} (<= 0.035); max |z| annotator accuracy"
f" {ck['c0b']:.2f} (<= 4); max |missing - 0.40| {ck['c0c']:.4f} (<= 0.04); flip-rate |z| {ck['c0d']:.2f} (<= 4)")
print(f" C1 max |kappa - sklearn| {ck['c1']:.1e}; C2 max |Fleiss - statsmodels| {ck['c2']:.1e},"
f" |multi-pi(pair) - pi| {ck['c2b']:.1e} (<= 1e-12)")
print(f" C3 smallest EM objective step {ck['c3']:.1e} (>= -1e-9); C4 max |D-S - oracle| accuracy, B1-B2"
f" {ck['c4']:.4f} (<= 0.015)")
print(f" C5 items differing from the planted flips with ideal probabilities: {ck['c5']} (= 0)")
print(f" C6 out-of-fold accuracy on true labels {ck['c6lo']:.4f} to {ck['c6hi']:.4f} (in [0.60, 0.98]);"
f" max training-fold accuracy {ck['c6tr']:.4f} (<= 0.99)")
=== seed 0, balanced prior [0.25, 0.25, 0.25, 0.25]
Part A true accuracy of A1..A7: 0.904 0.897 0.863 0.794 0.804 0.232 0.396
careful pair A1-A2: agreement 0.8165 Cohen kappa 0.7553 Scott pi 0.7553 gap (agreement - kappa) 0.0612
all seven on the 395 fully labelled images: pairwise agreement 0.5185 Fleiss kappa 0.3575
Part B condition vote D-S oracle ties iters CM err D-S / vote affected: n vote D-S
B1 quality differs 0.9730 0.9920 0.9935 61 155 0.0014 / 0.0070
B2 all equal 0.8840 0.8915 0.8895 153 51 0.0093 / 0.0112
B3a 2 of 5 share bias 0.9380 0.9680 0.9720 104 46 0.0061 / 0.0201 248 0.649 0.915
B3b 3 of 5 share bias 0.8600 0.8630 0.9620 36 25 0.0587 / 0.0585 248 0.000 0.000
Part C out-of-fold accuracy on true labels 0.6740; mean training-fold accuracy on given labels 0.7407
flipped labels 382 (0.1910); errors estimated from Q 921.3
method flagged precision recall | true class 2-3: flagged precision recall
C_confusion 989 0.3165 0.8194 | 468 0.3141 0.7989
C_conf.joint 802 0.3404 0.7147 | 361 0.3352 0.6576
PBNR 792 0.3396 0.7042 | 365 0.3425 0.6793
=== seed 0, imbalanced prior [0.55, 0.25, 0.15, 0.05]
Part A true accuracy of A1..A7: 0.900 0.902 0.841 0.815 0.802 0.245 0.411
careful pair A1-A2: agreement 0.8155 Cohen kappa 0.7109 Scott pi 0.7109 gap (agreement - kappa) 0.1046
all seven on the 425 fully labelled images: pairwise agreement 0.5174 Fleiss kappa 0.2979
Part B condition vote D-S oracle ties iters CM err D-S / vote affected: n vote D-S
B1 quality differs 0.9670 0.9910 0.9915 66 93 0.0027 / 0.0143
B2 all equal 0.8930 0.9235 0.9265 149 102 0.0128 / 0.0207
B3a 2 of 5 share bias 0.9600 0.9815 0.9855 81 104 0.0065 / 0.0200 153 0.634 0.882
B3b 3 of 5 share bias 0.9120 0.9145 0.9770 34 35 0.0555 / 0.0541 153 0.000 0.000
Part C out-of-fold accuracy on true labels 0.7080; mean training-fold accuracy on given labels 0.7365
flipped labels 411 (0.2055); errors estimated from Q 1005.4
method flagged precision recall | true class 2-3: flagged precision recall
C_confusion 955 0.3592 0.8345 | 253 0.2688 0.8095
C_conf.joint 938 0.3443 0.7859 | 220 0.3000 0.7857
PBNR 834 0.3741 0.7591 | 202 0.3119 0.7500
=== per-seed hypothesis values (counts out of N = 2000; bounds = 2 sqrt(b + c) / n)
seed prior H1 gap H2a vote/D-S H2b |diff| bound H3 aff vote D-S bound H4 rate CJ prec PBNR prec
0 balanced 0.0612 1946/ 1984 0.0075 0.0083 0.0000 0.0000 0.0000 0.1910 0.3404 0.3396
0 imbalanced 0.1046 1934/ 1982 0.0305 0.0101 0.0000 0.0000 0.0000 0.2055 0.3443 0.3741
1 balanced 0.0623 1931/ 1973 0.0045 0.0079 0.0000 0.0000 0.0000 0.1955 0.3622 0.3753
1 imbalanced 0.0963 1942/ 1980 0.0175 0.0089 0.0000 0.0000 0.0000 0.1915 0.3431 0.3656
2 balanced 0.0649 1947/ 1979 0.0015 0.0081 0.0000 0.0000 0.0000 0.2115 0.3692 0.3826
2 imbalanced 0.0990 1936/ 1980 0.0125 0.0084 0.0000 0.0000 0.0000 0.2000 0.3415 0.3768
3 balanced 0.0626 1943/ 1981 0.0000 0.0085 0.0000 0.0000 0.0000 0.1970 0.3615 0.3762
3 imbalanced 0.1103 1940/ 1982 0.0145 0.0079 0.0000 0.0000 0.0000 0.2105 0.3452 0.3764
4 balanced 0.0580 1941/ 1987 0.0035 0.0085 0.0000 0.0000 0.0000 0.1995 0.3691 0.3873
4 imbalanced 0.1043 1946/ 1983 0.0220 0.0108 0.0000 0.0000 0.0000 0.2130 0.3843 0.4246
hypotheses held on every seed and prior: H1 held, H2a held, H2b failed, H3 held, H4 held
checks over all seeds:
C0 data: max |class fraction - prior| 0.0205 (<= 0.035); max |z| annotator accuracy 3.09 (<= 4); max |missing - 0.40| 0.0260 (<= 0.04); flip-rate |z| 1.45 (<= 4)
C1 max |kappa - sklearn| 2.2e-16; C2 max |Fleiss - statsmodels| 1.7e-16, |multi-pi(pair) - pi| 0.0e+00 (<= 1e-12)
C3 smallest EM objective step -1.8e-11 (>= -1e-9); C4 max |D-S - oracle| accuracy, B1-B2 0.0060 (<= 0.015)
C5 items differing from the planted flips with ideal probabilities: 0 (= 0)
C6 out-of-fold accuracy on true labels 0.6705 to 0.7265 (in [0.60, 0.98]); max training-fold accuracy 0.7528 (<= 0.99)
Agreement. Under both priors the careful pair agree on about 82% of the images (0.8165 balanced, 0.8155 imbalanced on seed 0), while their kappa falls from 0.7553 to 0.7109: the same people with the same error process lose chance-corrected agreement on the skewed dataset. The chance agreement implied by the printed values, , is 0.250 under the balanced prior and 0.362 under the imbalanced one, and the gap formula above turns the larger into the larger gap. H1 held on every seed: kappa is below raw agreement in all ten cases, and the gap under the imbalanced prior is larger than under the balanced one on each seed (0.1046 against 0.0612, 0.0963 against 0.0623, 0.0990 against 0.0649, 0.1103 against 0.0626, 0.1043 against 0.0580). Cohen’s kappa and Scott’s pi agree to four decimals here; by construction the two careful annotators share one error process, the case in which the bias term above is close to zero. Fleiss’ kappa over all seven annotators is far lower (0.3575 and 0.2979), and in our reading the accuracy row shows why it cannot be read as an average quality: it mixes two careful annotators near 0.90 with a random one near 0.24 and two biased ones near 0.80, and says nothing about which is which.
Aggregation. With annotators of different quality (B1), H2a held in all ten seed–prior cases: on seed 0 Dawid–Skene is right on 0.9920 and 0.9910 of the images against the vote’s 0.9730 and 0.9670, close to the oracle’s 0.9935 and 0.9915 (C4), and its confusion-matrix estimates are several times closer to the annotators’ true error patterns than counts against the vote (0.0014 against 0.0070, and 0.0027 against 0.0143). With seven equal annotators (B2), H2b held under the balanced prior on all five seeds ( 0.0075, 0.0045, 0.0015, 0.0000 and 0.0035 against bounds 0.0083, 0.0079, 0.0081, 0.0085 and 0.0085) and failed under the imbalanced prior on all five seeds (0.0305, 0.0175, 0.0125, 0.0145 and 0.0220 against 0.0101, 0.0089, 0.0084, 0.0079 and 0.0108).
When two of five annotators share the misreading (B3a), the vote keeps the true class on 0.649 and 0.634 of the affected images on seed 0, and Dawid–Skene on 0.915 and 0.882. When three of five share it (B3b), H3 held on every seed and prior: Dawid–Skene and the vote both score 0.0000 on the affected images. Over all images in B3b on seed 0, the vote, Dawid–Skene and the oracle score 0.8600, 0.8630 and 0.9620 (balanced) and 0.9120, 0.9145 and 0.9770 (imbalanced).
Finding the planted errors. The label-free check C6 held: the out-of-fold accuracy on the true labels runs from 0.6705 to 0.7265 over the seeds, above what a constant answer reaches (0.25 under the balanced prior, 0.55 for “disk” under the imbalanced one) and far from perfect, and the training-fold accuracy (at most 0.7528) shows that the classifier does not fit its noisy training labels. H4 held in all ten cases; on seed 0, and PBNR reach a precision of 0.3404 and 0.3396 against a flip rate of 0.1910 under the balanced prior, and 0.3443 and 0.3741 against 0.2055 under the imbalanced one. In absolute terms the precision is low: on seed 0 about two of every three flagged images carry a correct label, and the error count estimated from (921.3 and 1005.4 on seed 0) is more than twice the true count (382 and 411). With ideal probabilities, C5 held: and flag exactly the flipped images, as Northcutt et al.’s Theorem 1 states. On seed 0, flags the most images and has the highest recall of the three under both priors; its precision is the lowest of the three under the balanced prior, but above that of under the imbalanced one (0.3592 against 0.3443). On the rare classes under the imbalanced prior on seed 0, the two confident-learning methods have higher precision than (0.3000 and 0.3119 against 0.2688) and lower recall (0.7857 and 0.7500 against 0.8095); this comparison had no hypothesis, and one seed does not test the paper’s claim. In practice, the flagged set is a review queue for people, ordered by how confident the model is, not a list of errors to delete (our reading).
The other checks held: C0 (largest class-fraction deviation 0.0205, largest annotator-accuracy 3.09, missing-fraction deviation 0.0260, flip-rate 1.45), C1 and C2 (differences of at most ), C3 (smallest objective step , floating-point rounding in our reading) and C4 (largest difference from the oracle 0.0060).
How the example was made. The hypotheses are the batch plan’s. Their operational forms, including the bound used for H2b and H3, the set-up and every check’s bar were written in a hypotheses file before any script ran; no bar or tolerance was written or changed after a script. A reachable-value script then ran the example’s code on another seed, together with an exact calculation of the careful pair’s expected agreement and kappa; on its seed it printed H2b failing under the imbalanced prior, before the plan was frozen, and the hypothesis was not changed. One edit to the code before the plan was frozen silenced a log-of-zero warning in the oracle’s posterior and changed no printed value. A pilot on seeds 100 to 104 gave the same outcome for every hypothesis and check as the first run shown here. The example departs from the plan’s description in ten ways, all fixed before the pilot and approved with it: the shared bias is run with two of five annotators, as the plan’s set-up says, and with three of five, because H3 is worded for a bias the majority shares, and H3 is decided on the latter; the oracle row, the vote-based confusion-matrix error, tie counts, iteration counts and the affected-item accuracy were added as reference prints; the plan’s “all equal” condition was specified as B2; Part B runs under both priors; Part A adds Scott’s pi and the seven annotators’ pairwise agreement; the planted errors are class-conditional flips of the true labels rather than annotator labels, was added beside the planned PBNR and , the error count estimated from was added, and 4 folds follow the paper; the rare classes are the true classes triangle and cross, printed under both priors; checks C4 and C5 were added; five seeds are run instead of one; and H2b’s “sampling error” and H3’s “no better” were made operational with the bound above.
Common Misconceptions
- “High agreement means accurate labels.” Agreement measures consistency. Two annotators who share a misreading agree perfectly and are both wrong (B3b above), and agreement between two independent annotators understates each one’s accuracy, as Image Annotation’s example shows for masks.
- “Kappa is a property of the annotators.” It also depends on the class distribution: the same annotators get a lower kappa on a skewed dataset (H1 above), so a kappa reported on one dataset does not transfer to another.
- “Fleiss’ kappa generalizes Cohen’s kappa.” It generalizes Scott’s , with one pooled distribution of categories; a multi-coder version of Cohen’s is a different coefficient (Artstein and Poesio, §2.5).
- “0.6 is substantial agreement, so the labels are good enough.” Landis and Koch’s bands are a convention, and the content-analysis convention that Artstein and Poesio describe sets the bar at 0.8.
- “More annotators always fix errors.” A vote or a latent-truth model removes independent errors; it adopts an error that a majority shares, whatever the number of annotators.
- “Confident learning finds the errors.” It finds candidates. Its exactness results assume ideal probabilities; in Part C above, with probabilities from a classifier, about two of every three flagged labels were correct, so flagged items go to review.
- “cleanlab’s default is the paper’s method.” The paper’s default is ; cleanlab 2.9.0’s
find_label_issuesdefaults to PBNR. Results from the two are not interchangeable.
Where It Is Used
- Dataset construction: agreement studies on a doubly annotated sample decide whether guidelines are clear enough; Image Annotation describes qualification, gold questions and redundant labelling, where latent-truth aggregation replaces the vote.
- Crowdsourcing: annotators’ estimated confusion matrices can single out near-random and systematically biased workers (our reading).
- Medical imaging: labels mined from radiology reports carry noise of their own (radiology AI, CheXpert, chest X-ray analysis), and in whole-slide image classification several of PANDA’s top teams treated label noise from subjective grading as a key problem.
- Cleaning training data: confident learning and similar filters remove or relabel suspected errors before training, and noise-matrix corrections adjust the loss instead; supervised learning introduces label noise in its general setting.
- Evaluation: test-label errors bound how finely models can be compared (benchmarks and leaderboards, benchmark overfitting), and scaling laws name label noise among the causes of an error floor.
- Imbalanced data: with rare classes, raw agreement and accuracy look good by default; chance-corrected agreement and per-class checks, as in class imbalance, are needed to see the rare classes.
Related
- Confusion Matrix
The table that counts how often each true class is predicted as each class, with true and false positives and negatives, row and column normalization, the metrics read from it, and how to see from it which classes a vision model confuses.
- Expectation–Maximization
An iterative method for maximum likelihood estimation when part of the data is hidden or missing, alternating an E step that infers the hidden variables with an M step that refits the parameters; its lower-bound derivation, the monotone increase of the likelihood and what is proved about convergence, the full updates for Gaussian mixtures, its variants, and its uses in vision from colour models and background subtraction to mixtures of motions.
- Label Smoothing
Training a classifier on targets that mix the one-hot label with a uniform distribution, so the cross-entropy loss has a finite minimum with a fixed gap between the true-class logit and the others; its definition in Inception-v3, the equivalent penalty against a uniform prior, its effects on accuracy, calibration, label noise, transfer and knowledge distillation, and its place in ImageNet and transformer training recipes.
- Loss Function
The function that scores how far a network's output is from the desired output on one example; training minimizes its average over the training images, and the choice of loss encodes what counts as a good prediction for each vision task.
- Benchmark Overfitting
How reusing one test set for years of model selection can inflate benchmark scores, what the theory of adaptive data analysis says about it, and what replication studies of CIFAR-10, ImageNet, MNIST and Kaggle found when they measured it.
- Cross-Validation
How k-fold cross-validation and its variants (leave-one-out, repeated, stratified, grouped, time-ordered and nested) reuse a small labelled image set for both training and evaluation, what the estimate measures, how noisy it is, and why tuning and reporting on the same folds is optimistic.
References
- Artstein, R. & Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4), 555–596.
- Northcutt, C. G., Jiang, L. & Chuang, I. L. (2021). Confident Learning: Estimating Uncertainty in Dataset Labels. Journal of Artificial Intelligence Research, 70, 1373–1411.
- Raykar, V. C., Yu, S., Zhao, L. H., Hermosillo Valadez, G., Florin, C., Bogoni, L. & Moy, L. (2010). Learning From Crowds. Journal of Machine Learning Research, 11, 1297–1322.
- Patrini, G., Rozza, A., Menon, A. K., Nock, R. & Qu, L. (2017). Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y. & Lacoste-Julien, S. (2017). A Closer Look at Memorization in Deep Networks. Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70, 233–242.
- cleanlab developers. cleanlab v2.9.0, cleanlab/filter.py (Apache-2.0), read November 2026.
- Dawid, A. P. & Skene, A. M. (1979). Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, 28(1), 20–28 (not read).
- Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46 (not read).
- Scott, W. A. (1955). Reliability of Content Analysis: The Case of Nominal Scale Coding. Public Opinion Quarterly, 19(3), 321–325 (not read).
- Fleiss, J. L. (1971). Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin, 76(5), 378–382 (not read).
- Landis, J. R. & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174 (not read).