Vision Systems & Engineering
Bias and Fairness in Computer Vision
What it means for a vision system's decisions to be fair to different groups of people, the formal criteria (demographic parity, equalized odds and equal opportunity, calibration within groups), why they cannot all hold at once, how post-processing and training-time methods trade one against another, and what an audit of a deployed system reports.
intermediate
A face verifier, a pedestrian detector or a skin-lesion classifier makes decisions about people, and the people it decides about belong to groups: by sex, age, skin type, or the camera and clinic their images came from. Fairness asks whether the system’s errors and decisions are distributed acceptably across such groups. The question has a formal side, a set of criteria that can be computed from a system’s outputs, and an older social side about what counts as acceptable, which no criterion settles. This article covers the formal side: the main group-fairness criteria and their names, the theorem that rules out satisfying them together, post-processing that enforces one of them and what it costs in the others, training-time methods evaluated on vision benchmarks, and what an audit of a vision system reports. Where unfairness comes from in the data, such as who is in a dataset and how people are labelled, belongs to dataset bias, and the measured demographic differentials of face recognition belong to face recognition and biometrics.
Definition
A model produces a score or a decision for an input whose true outcome is and whose owner belongs to a group . Hardt, Price and Srebro (2016) call the protected attribute and Kleinberg, Mullainathan and Raghavan (2017) speak of two groups; in vision may be a demographic category or any attribute of the image that should not matter, such as whether it is in colour, which Wang et al. (2020) call the domain. A group-fairness criterion is a condition on the joint distribution of that equalizes some rate across the values of . Hardt et al. call such criteria oblivious: they depend only on that joint distribution, not on the features, the model’s form or how it was trained (Def. 2.3), so they can be checked on a deployed system from its outputs, the labels and the group of each case.
NIST SP 1270 (March 2022), which describes itself as a first step toward detailed guidance and states that it establishes no legal standard, sorts bias in AI into three categories: systemic (institutional or historical) bias from the practices of institutions, statistical and computational bias from unrepresentative samples and modelling choices, and human bias in how people perceive and act on information (Schwartz et al., 2022, §2.1.2). The criteria below measure outcomes of all three; they do not say which caused a gap.
Intuition
Imagine a screening model that flags images for a specialist’s review and is used on two groups of people. It can be fair in three different senses. It can flag the two groups at the same rate. It can be equally good at catching true cases and equally prone to false alarms in both groups. Or its scores can mean the same thing in both groups: of the cases scored 0.3, about 30% are positive whichever group they come from.
Each sense sounds reasonable, and each has been argued for. The trouble starts when the groups differ in how common the condition is. If 30% of one group and 10% of the other are positive, flagging at equal rates means flagging many negatives in one group or missing many positives in the other. In the example below, scores that mean the same thing in both groups, cut at one threshold, give the high-prevalence group more flags, more true cases caught and more false alarms. Making catch rates and false-alarm rates equal then makes a flag mean different things in the two groups. The example below computes all three on scores whose meaning is exact by construction.
Formal Definition
For a binary decision and groups :
- Demographic parity (Hardt et al.’s term; Kleinberg et al. call it statistical parity) requires the decision to be independent of the group:
- Equalized odds (Hardt et al., Def. 2.1) requires and to be independent given ; for binary , equal true positive rates (recall) and equal false positive rates:
- Equal opportunity (Def. 2.2) keeps only the equation: equal true positive rates.
For a score , Kleinberg et al. define three conditions on a “risk assignment” that places people in bins with scores (§1.1):
- (A) Calibration within groups: in each group, a fraction of the people in bin are positive, so for every group. This is the per-group version of the property that expected calibration error measures.
- (B) Balance for the negative class: the mean score of the negatives is the same in both groups, .
- (C) Balance for the positive class: the same for the positives.
The authors describe the balance conditions as generalizations of equal false positive and false negative rates. A binary classifier is the special case of two bins, each scored by its fraction of positives (§1.2). For a thresholded decision, in our derivation, condition (A) then requires equal precision (positive predictive value, PPV) and equal false omission rate (FOR, the fraction of positives among those not selected) across groups, and with shared bin scores, (B) and (C) reduce to equal false positive and true positive rates.
Properties
The criteria conflict
Kleinberg et al.’s Theorem 1 states that if a risk assignment satisfies (A), (B) and (C), the instance must either allow perfect prediction (every feature vector’s probability of being positive is 0 or 1) or have equal base rates in the two groups. Theorem 2 extends this to approximate versions: if a risk assignment satisfies the three conditions approximately, the instance must lie close to one of the two special cases. The result holds however the scores are produced, by an algorithm or by people (§1.2). Hardt et al. summarize it as showing that a meaningful score calibrated within each group can meet “a criterion equivalent to equalized odds” only by being perfectly accurate. Kleinberg et al.’s own term is balance; in our reading, balance equalizes only the mean score of each class, which is weaker than Hardt et al.’s equalized odds for a score ( independent of given ), and the two coincide for a two-bin decision with shared bin scores.
Two consequences matter for the example below, in our reading of the theorem. Under equal opportunity, a two-bin decision with shared bin scores is guaranteed only (C), so the theorem requires (A) or (B) to fail; it does not say which. Under equalized odds, (B) and (C) both hold, so (A) must fail: the groups’ PPV or FOR must differ. Writing and for the shared true and false positive rates and for a group’s base rate, in our derivation
which differs between groups whenever , and . This is the relation between precision and prevalence that the radiology AI article shows for screening populations, here appearing between groups.
The second escape clause, equal base rates, is weaker than it sounds. Kleinberg et al.’s construction for that case gives everyone the same score. A calibrated score that is informative can still violate balance when base rates are equal: in our derivation, for a score drawn from with , the mean score of the positives is , which changes with the spread at a fixed base rate . In our derivation, only identical score distributions guarantee that a shared threshold, whatever its value, satisfies every criterion; the example’s second control shows the difference.
Demographic parity and unawareness
Hardt et al. (§1) give two objections to demographic parity: it permits accepting the qualified members of one group and random members of another as long as the rates match, and when the outcome is correlated with the group it rules out the perfect predictor . Equalized odds, by contrast, always allows . They also note that fairness through unawareness, simply leaving the protected attribute out, does not work because other features encode it (“redundant encodings”). Wang et al. (§4.2) observe the same in images: a linear SVM trained on the features of a CIFAR-10 classifier that never saw a colour–class correlation predicted whether a new image was colour or grayscale with 82% accuracy.
Post-processing a score
Hardt et al. (§3) show how to derive a fair decision from any score and the group, without retraining. Each group’s achievable (false positive rate, true positive rate) pairs form the region under its ROC curve. Equal opportunity picks a threshold per group so that the true positive rates match; with continuous ROC curves, points on the curves themselves suffice and no randomization is needed. Equalized odds needs one point shared by both groups, which lies under every group’s ROC curve; the best such point is on the pointwise minimum of the curves, and for a group whose curve is higher there it is generally reached by randomizing between two thresholds: below the lower threshold never select, above the upper one always select, in between select with a fixed probability (§3.2 of the NIPS version). The equalized-odds solution therefore depends on the worst group’s ROC curve, which Hardt et al. argue gives the builder an incentive to make predictions accurate in every group. Their Theorem 4.5 bounds how far such a derived predictor is from the optimal fair one in terms of how far the score is from the Bayes-optimal one.
In Hardt et al.’s case study on credit scores (§5), with a loss in which a false positive costs 82/18 times a false negative, a single threshold earned 99.3% of the maximum profit, equal opportunity 92.8%, equalized odds 80.2% and demographic parity 69.8%.
Examples
Three criteria on calibrated scores
The script below computes the three kinds of criteria on synthetic risk scores, with no images and no people. Each group’s score is drawn from a Beta distribution and the true label from , so the score is calibrated within each group by construction and every population rate is exact: given the score follows and given it follows , whose tails SciPy evaluates. Beside each exact value the script prints the estimate from 20,000 samples per group and a two-sample for the difference between groups.
- Main scenario: group A , base rate 0.30; group B , base rate 0.10.
- Control 1: both groups .
- Control 2: A and B , both with base rate 0.30 but different spread.
A missed positive costs 4 and a false alarm 1, so for a calibrated score the cost-minimizing rule is to select when , that is , in every group. The main scenario compares that shared threshold with Hardt et al.’s equal opportunity (one threshold per group, common true positive rate, minimum pooled cost) and equalized odds (a common false and true positive rate on the lower ROC curve, one group randomized). The fair thresholds are computed from the exact distributions, as Hardt et al. assume the joint distribution is known, and applied to the samples. Pooled cost and accuracy average the two groups with equal weight. Per-group calibration is checked with scikit-learn’s calibration_curve with 10 equal-width bins (n_bins=10, strategy="uniform"; the defaults are 5 and uniform), whose binning the script reproduces to get bin counts.
Before the run, the plan fixed the criteria: a rate “differs” between groups when the exact difference is at least 0.01 and ; rates are “equal within sampling error” when ; constructed equalities must hold exactly to ; calibration passes when no bin with at least 30 samples deviates by more than 3.5 standard errors. The checks labelled C1–C6 in the output test the data and the machinery: samples against the construction (C1), closed forms against numerical integration (C2), calibration (C3), the optimizers against grids of 100,001 points (C4), the shared threshold against 9,999 others (C5) and the randomized rule against its target rates (C6); C7 requires two runs to give identical output. It runs in about two seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2.
import os, sys
os.environ["OPENBLAS_NUM_THREADS"] = "1" # one BLAS thread, set before NumPy is imported
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np
from scipy import stats, integrate, optimize
from sklearn.calibration import calibration_curve
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
N, C_FN, C_FP = 20_000, 4.0, 1.0 # samples per group; cost of a miss and of a false alarm
T0 = C_FP / (C_FP + C_FN) # Bayes threshold of a calibrated score: 0.2
SCEN = {"main": ((3, 7), (1, 9)), # base rates 0.30 and 0.10
"control 1": ((3, 7), (3, 7)), # equal base rates, identical scores
"control 2": ((3, 7), (0.75, 1.75))} # equal base rates 0.30, different spread
rng = np.random.default_rng(SEED)
# Exact population quantities: s ~ Beta(a, b), y ~ Bernoulli(s), so s|y=1 ~ Beta(a+1, b), s|y=0 ~ Beta(a, b+1)
pi_ = lambda g: g[0] / (g[0] + g[1])
tpr = lambda t, g: stats.beta.sf(t, g[0] + 1, g[1])
fpr = lambda t, g: stats.beta.sf(t, g[0], g[1] + 1)
roc = lambda phi, g: tpr(stats.beta.isf(phi, g[0], g[1] + 1), g)
mpos = lambda g: (g[0] + 1) / (g[0] + g[1] + 1)
mneg = lambda g: g[0] / (g[0] + g[1] + 1)
def exact_rates(T, F, g):
p = pi_(g); sr = p * T + (1 - p) * F
return dict(SR=sr, TPR=T, FPR=F, PPV=p * T / sr, FOR=p * (1 - T) / (1 - sr),
acc=p * T + (1 - p) * (1 - F), cost=C_FN * p * (1 - T) + C_FP * (1 - p) * F)
def sample_rates(sel, y):
T, F = sel[y].mean(), sel[~y].mean()
return dict(SR=sel.mean(), TPR=T, FPR=F, PPV=y[sel].mean(), FOR=y[~sel].mean(),
acc=(sel == y).mean(), cost=(C_FN * (y & ~sel) + C_FP * (sel & ~y)).mean()), \
dict(SR=sel.size, TPR=y.sum(), FPR=(~y).sum(), PPV=sel.sum(), FOR=(~sel).sum())
def z_prop(pa, na, pb, nb): return (pa - pb) / np.sqrt(pa * (1 - pa) / na + pb * (1 - pb) / nb)
def z_mean(xa, xb): return (xa.mean() - xb.mean()) / np.sqrt(xa.var(ddof=1) / xa.size + xb.var(ddof=1) / xb.size)
def draw(g):
s = rng.beta(g[0], g[1], N); y = rng.random(N) < s
return s, y
def report(title, ex, sm, dn):
print(f" {title}")
print(" measure exact A exact B sample A sample B z")
for k in ["SR", "TPR", "FPR", "PPV", "FOR"]:
z = z_prop(sm[0][k], dn[0][k], sm[1][k], dn[1][k])
print(f" {k:7s} {ex[0][k]:8.4f} {ex[1][k]:8.4f} {sm[0][k]:8.4f} {sm[1][k]:8.4f} {z:+7.2f}")
for k in ["acc", "cost"]:
print(f" {k:7s} {ex[0][k]:8.4f} {ex[1][k]:8.4f} {sm[0][k]:8.4f} {sm[1][k]:8.4f}"
f" pooled exact {0.5 * (ex[0][k] + ex[1][k]):.5f} sample {0.5 * (sm[0][k] + sm[1][k]):.5f}")
return 0.5 * (ex[0]["cost"] + ex[1]["cost"])
def check(name, value, ok):
print(f" check {name}: {value} -> {'pass' if ok else 'FAIL'}")
print(f"NumPy {np.__version__}, SciPy {__import__('scipy').__version__}, "
f"scikit-learn {__import__('sklearn').__version__}, seed {SEED}")
quad_err, costs = [], {}
for name, groups in SCEN.items():
print(f"== {name}: A Beta{groups[0]} base rate {pi_(groups[0]):.2f}, B Beta{groups[1]} base rate {pi_(groups[1]):.2f}")
data = [draw(g) for g in groups]
for g, (s, y) in zip(groups, data): # C1: the samples match the construction
za = (y.mean() - pi_(g)) / np.sqrt(pi_(g) * (1 - pi_(g)) / N)
zs = (s.mean() - pi_(g)) / np.sqrt(stats.beta(*g).var() / N)
check("C1 data base rate z, mean score z", f"{za:+.2f}, {zs:+.2f}", max(abs(za), abs(zs)) < 3)
for lab, f, cls in (("mean score | y=1", mpos, True), ("mean score | y=0", mneg, False)):
z = z_mean(data[0][0][data[0][1] == cls], data[1][0][data[1][1] == cls])
print(f" {lab}: exact A {f(groups[0]):.4f} B {f(groups[1]):.4f}, sample A "
f"{data[0][0][data[0][1] == cls].mean():.4f} B {data[1][0][data[1][1] == cls].mean():.4f}, z {z:+.2f}")
worst = [] # C3: calibration within each group, 10 equal-width bins
for s, y in data:
prob_true, prob_pred = calibration_curve(y, s, n_bins=10, strategy="uniform")
ids = np.searchsorted(np.linspace(0, 1, 11)[1:-1], s)
n_b, k_b, s_b = (np.bincount(ids, w, minlength=11) for w in (None, y, s))
nz = n_b > 0
same = np.allclose(prob_true, k_b[nz] / n_b[nz], rtol=0, atol=1e-12) and \
np.allclose(prob_pred, s_b[nz] / n_b[nz], rtol=0, atol=1e-12)
check("C3 own binning equals calibration_curve", same, same)
use = n_b >= 30
m = s_b[use] / n_b[use]
worst.append(np.max(np.abs(k_b[use] / n_b[use] - m) / np.sqrt(m * (1 - m) / n_b[use])))
check("C3 calibration max |z| over bins (A, B)", f"{worst[0]:.2f}, {worst[1]:.2f}", max(worst) <= 3.5)
for g in groups: # C2: closed forms against direct integration
f = stats.beta(*g).pdf; p = pi_(g)
quad_err += [abs(integrate.quad(lambda s: s * s * f(s), 0, 1)[0] / p - mpos(g)),
abs(integrate.quad(lambda s: s * (1 - s) * f(s), 0, 1)[0] / (1 - p) - mneg(g))]
# shared threshold
sm, dn = zip(*[sample_rates(s > T0, y) for s, y in data])
ex = [exact_rates(tpr(T0, g), fpr(T0, g), g) for g in groups]
costs[name, "shared"] = report(f"shared threshold {T0}", ex, sm, dn)
ts = np.linspace(0.001, 0.999, 9999)
grid = 0.5 * sum(exact_rates(tpr(ts, g), fpr(ts, g), g)["cost"] for g in groups)
check("C5 cost at 0.2 minus grid minimum", f"{costs[name, 'shared'] - grid.min():.1e}",
costs[name, "shared"] <= grid.min() + 1e-9)
for g in groups:
quad_err += [abs(integrate.quad(lambda s: s * stats.beta(*g).pdf(s), T0, 1)[0] / pi_(g) - tpr(T0, g)),
abs(integrate.quad(lambda s: (1 - s) * stats.beta(*g).pdf(s), T0, 1)[0] / (1 - pi_(g)) - fpr(T0, g))]
if name != "main":
continue
A, B = groups
# equal opportunity: common TPR tau, group thresholds, minimum pooled cost (Hardt et al. 2016, Sec. 3.2)
eo_cost = lambda tau: 0.5 * sum(exact_rates(tau, fpr(stats.beta.isf(tau, g[0] + 1, g[1]), g), g)["cost"] for g in groups)
tau = optimize.minimize_scalar(eo_cost, bounds=(1e-6, 1 - 1e-6), method="bounded", options={"xatol": 1e-12}).x
th = [stats.beta.isf(tau, g[0] + 1, g[1]) for g in groups]
taus = np.linspace(1e-5, 1 - 1e-5, 100_001)
check("C4 EO optimum minus grid minimum", f"{eo_cost(tau) - eo_cost(taus).min():.1e}", eo_cost(tau) <= eo_cost(taus).min() + 1e-9)
sm, dn = zip(*[sample_rates(s > t, y) for (s, y), t in zip(data, th)])
ex = [exact_rates(tpr(t, g), fpr(t, g), g) for g, t in zip(groups, th)]
costs[name, "EO"] = report(f"equal opportunity: thresholds A {th[0]:.4f}, B {th[1]:.4f}", ex, sm, dn)
print(f" exact TPR difference {ex[0]['TPR'] - ex[1]['TPR']:.1e}")
# equalized odds: common (FPR, TPR) on the lower of the two ROC curves, minimum pooled cost
od_cost = lambda phi: 0.5 * sum(exact_rates(np.minimum(roc(phi, A), roc(phi, B)), phi, g)["cost"] for g in groups)
phi = optimize.minimize_scalar(od_cost, bounds=(1e-6, 1 - 1e-6), method="bounded", options={"xatol": 1e-12}).x
check("C4 EOdds optimum minus grid minimum", f"{od_cost(phi) - od_cost(taus).min():.1e}", od_cost(phi) <= od_cost(taus).min() + 1e-9)
lo, hi = (0, 1) if roc(phi, A) <= roc(phi, B) else (1, 0)
T = roc(phi, groups[lo])
t_lo = stats.beta.isf(phi, groups[lo][0], groups[lo][1] + 1)
k = T / phi # the other group: mix "select above t2" with "select nobody"
phi2 = optimize.brentq(lambda x: roc(x, groups[hi]) - k * x, phi, 1 - 1e-12, xtol=1e-15)
t2 = stats.beta.isf(phi2, groups[hi][0], groups[hi][1] + 1); p = phi / phi2
err = max(abs(p * fpr(t2, groups[hi]) - phi), abs(p * tpr(t2, groups[hi]) - T))
check("C6 randomized group's exact (FPR, TPR) error", f"{err:.1e}", err < 1e-9)
print(f" group {'AB'[lo]}: select if s > {t_lo:.4f}; group {'AB'[hi]}: select if s > {t2:.4f} "
f"with probability p = {p:.4f}")
sel = [None, None]
sel[lo] = data[lo][0] > t_lo
sel[hi] = (data[hi][0] > t2) & (rng.random(N) < p)
sm, dn = zip(*[sample_rates(sl, y) for sl, (s, y) in zip(sel, data)])
ex = [None, None]
ex[lo] = exact_rates(tpr(t_lo, groups[lo]), fpr(t_lo, groups[lo]), groups[lo])
ex[hi] = exact_rates(p * tpr(t2, groups[hi]), p * fpr(t2, groups[hi]), groups[hi])
costs[name, "EOdds"] = report(f"equalized odds: FPR {phi:.4f}, TPR {T:.4f} in both groups", ex, sm, dn)
print(f" exact TPR difference {ex[0]['TPR'] - ex[1]['TPR']:.1e}, FPR difference {ex[0]['FPR'] - ex[1]['FPR']:.1e}")
for g, t in ((A, th[0]), (B, th[1]), (groups[lo], t_lo), (groups[hi], t2)):
quad_err += [abs(integrate.quad(lambda s: s * stats.beta(*g).pdf(s), t, 1)[0] / pi_(g) - tpr(t, g)),
abs(integrate.quad(lambda s: (1 - s) * stats.beta(*g).pdf(s), t, 1)[0] / (1 - pi_(g)) - fpr(t, g))]
print(f" H4 exact pooled cost steps: EO - shared {costs[name, 'EO'] - costs[name, 'shared']:.2e}, "
f"EOdds - EO {costs[name, 'EOdds'] - costs[name, 'EO']:.2e}")
check("C2 max |quad - closed form|", f"{max(quad_err):.1e}", max(quad_err) < 1e-8)
NumPy 2.3.5, SciPy 1.16.3, scikit-learn 1.7.2, seed 0
== main: A Beta(3, 7) base rate 0.30, B Beta(1, 9) base rate 0.10
check C1 data base rate z, mean score z: +0.88, +0.97 -> pass
check C1 data base rate z, mean score z: -0.31, -0.35 -> pass
mean score | y=1: exact A 0.3636 B 0.1818, sample A 0.3641 B 0.1832, z +58.20
mean score | y=0: exact A 0.2727 B 0.0909, sample A 0.2735 B 0.0906, z +146.32
check C3 own binning equals calibration_curve: True -> pass
check C3 own binning equals calibration_curve: True -> pass
check C3 calibration max |z| over bins (A, B): 1.51, 1.40 -> pass
shared threshold 0.2
measure exact A exact B sample A sample B z
SR 0.7382 0.1342 0.7433 0.1363 +154.59
TPR 0.8791 0.3758 0.8773 0.3709 +43.55
FPR 0.6778 0.1074 0.6851 0.1104 +125.66
PPV 0.3573 0.2800 0.3574 0.2705 +9.28
FOR 0.1385 0.0721 0.1447 0.0724 +13.68
acc 0.4893 0.8409 0.4852 0.8381 pooled exact 0.66511 sample 0.66165
cost 0.6195 0.3463 0.6262 0.3494 pooled exact 0.48291 sample 0.48782
check C5 cost at 0.2 minus grid minimum: -8.7e-09 -> pass
check C4 EO optimum minus grid minimum: -3.0e-13 -> pass
equal opportunity: thresholds A 0.2790, B 0.1081
measure exact A exact B sample A sample B z
SR 0.5197 0.3571 0.5212 0.3508 +34.87
TPR 0.7045 0.7045 0.6972 0.7021 -0.41
FPR 0.4406 0.3185 0.4447 0.3121 +24.36
PPV 0.4067 0.1973 0.4052 0.1988 +30.48
FOR 0.1846 0.0460 0.1915 0.0456 +33.03
acc 0.6030 0.6838 0.5983 0.6894 pooled exact 0.64340 sample 0.64383
cost 0.6630 0.4048 0.6768 0.3995 pooled exact 0.53388 sample 0.53812
exact TPR difference 0.0e+00
check C4 EOdds optimum minus grid minimum: -1.4e-13 -> pass
check C6 randomized group's exact (FPR, TPR) error: 2.2e-16 -> pass
group A: select if s > 0.3000; group B: select if s > 0.0652 with probability p = 0.7509
equalized odds: FPR 0.3828, TPR 0.6496 in both groups
measure exact A exact B sample A sample B z
SR 0.4628 0.4095 0.4639 0.4076 +11.37
TPR 0.6496 0.6496 0.6462 0.6553 -0.74
FPR 0.3828 0.3828 0.3847 0.3803 +0.81
PPV 0.4211 0.1586 0.4219 0.1597 +40.09
FOR 0.1957 0.0593 0.1999 0.0578 +32.16
acc 0.6269 0.6205 0.6247 0.6232 pooled exact 0.62370 sample 0.62395
cost 0.6884 0.4847 0.6968 0.4795 pooled exact 0.58654 sample 0.58815
exact TPR difference -2.2e-16, FPR difference -1.7e-16
H4 exact pooled cost steps: EO - shared 5.10e-02, EOdds - EO 5.27e-02
== control 1: A Beta(3, 7) base rate 0.30, B Beta(3, 7) base rate 0.30
check C1 data base rate z, mean score z: +0.45, +0.44 -> pass
check C1 data base rate z, mean score z: -0.12, +2.50 -> pass
mean score | y=1: exact A 0.3636 B 0.3636, sample A 0.3626 B 0.3663, z -1.46
mean score | y=0: exact A 0.2727 B 0.2727, sample A 0.2736 B 0.2751, z -0.98
check C3 own binning equals calibration_curve: True -> pass
check C3 own binning equals calibration_curve: True -> pass
check C3 calibration max |z| over bins (A, B): 1.34, 1.32 -> pass
shared threshold 0.2
measure exact A exact B sample A sample B z
SR 0.7382 0.7382 0.7372 0.7422 -1.14
TPR 0.8791 0.8791 0.8733 0.8802 -1.15
FPR 0.6778 0.6778 0.6785 0.6833 -0.84
PPV 0.3573 0.3573 0.3571 0.3553 +0.32
FOR 0.1385 0.1385 0.1454 0.1393 +0.89
acc 0.4893 0.4893 0.4878 0.4855 pooled exact 0.48928 sample 0.48667
cost 0.6195 0.6195 0.6268 0.6221 pooled exact 0.61951 sample 0.62448
check C5 cost at 0.2 minus grid minimum: -1.1e-08 -> pass
== control 2: A Beta(3, 7) base rate 0.30, B Beta(0.75, 1.75) base rate 0.30
check C1 data base rate z, mean score z: +0.25, +1.83 -> pass
check C1 data base rate z, mean score z: +0.19, -1.50 -> pass
mean score | y=1: exact A 0.3636 B 0.5000, sample A 0.3656 B 0.4951, z -36.65
mean score | y=0: exact A 0.2727 B 0.2143, sample A 0.2743 B 0.2124, z +31.75
check C3 own binning equals calibration_curve: True -> pass
check C3 own binning equals calibration_curve: True -> pass
check C3 calibration max |z| over bins (A, B): 1.62, 1.27 -> pass
shared threshold 0.2
measure exact A exact B sample A sample B z
SR 0.7382 0.5601 0.7403 0.5587 +38.77
TPR 0.8791 0.8786 0.8833 0.8779 +0.91
FPR 0.6778 0.4236 0.6788 0.4216 +44.77
PPV 0.3573 0.4706 0.3589 0.4723 -18.44
FOR 0.1385 0.0828 0.1352 0.0832 +9.32
acc 0.4893 0.6671 0.4903 0.6684 pooled exact 0.57817 sample 0.57935
cost 0.6195 0.4422 0.6150 0.4416 pooled exact 0.53085 sample 0.52835
check C5 cost at 0.2 minus grid minimum: -8.7e-09 -> pass
check C2 max |quad - closed form|: 2.1e-11 -> pass
Every check held, C7 (two runs give identical output, compared outside the script) included. The samples match the construction (largest C1 2.50), the closed forms agree with direct integration to , the score is calibrated in all six groups (largest bin deviation 1.62 standard errors), both optimizers are at least as good as the best grid point, 0.2 is the cheapest shared threshold, and the randomized rule reaches its target rates to .
One threshold on a calibrated score (H1, H1b: held). In the main scenario the shared threshold selects 74% of group A and 14% of group B, catches 88% of A’s positives but 37% of B’s, and raises false alarms on 69% of A’s negatives against 11% of B’s: demographic parity and equalized odds both fail, at of 43 or more. The score itself is calibrated in both groups, yet the mean score of A’s positives (0.364 exactly) is twice that of B’s (0.182), and A’s negatives average three times B’s (0.273 against 0.091): Kleinberg et al.‘s conditions (B) and (C) fail while (A) holds, as Theorem 1 allows with unequal base rates. Nothing about the score is wrong in either group; the gaps come from the groups’ different score distributions meeting one threshold, the same mechanism that the face recognition example shows for verification scores.
Equal opportunity (H2: held). Lowering B’s threshold to 0.108 and raising A’s to 0.279 makes the true positive rates equal (exactly, and in the sample). The false positive rates still differ (0.445 against 0.312 in the sample), and so does precision: exactly 0.407 in A against 0.197 in B, a gap of 0.21 where the shared threshold’s was 0.08. In the terms of the theorem, (C) now holds for the two-bin decision, and both (A) and (B) fail.
Equalized odds (H2b: held). The optimum lies on group A’s ROC curve, which is the lower one here, so A keeps a plain threshold (0.300) while B is selected above 0.0652 with probability 0.7509. In our derivation A’s threshold is where the slope of its ROC curve, the likelihood ratio of a calibrated score, equals the ratio of the pooled cost weights, which is 1 for these base rates and costs. Both error rates are now equal (exactly, and and in the sample), and precision differs even more: 0.421 against 0.159 exactly. This is the failure of (A) that Theorem 1 forces once (B) and (C) hold, and it follows the formula for PPV above: the same true and false positive rates applied to base rates of 0.30 and 0.10.
Cost of each constraint (H4: held, a check of the optimizers). The pooled cost rises from 0.483 for the shared threshold to 0.534 for equal opportunity and 0.587 for equalized odds, and pooled accuracy falls from 0.665 to 0.643 and 0.624. The ordering is not a finding about fairness: the shared threshold is optimal without constraints, and equalized odds adds a constraint to equal opportunity, so any correct optimizer must give it. The sizes depend on the costs, base rates and score distributions chosen here; Hardt et al.’s credit-score case gives other numbers for other data.
The controls (H3, H3b: held). With identical score distributions (control 1), every criterion holds within sampling error at the shared threshold; the largest among the five rates and the two balance means is 1.46. With equal base rates but a wider spread in group B (control 2), the true positive rates happen to agree (0.8791 and 0.8786 exactly), but the false positive rates (0.678 against 0.424), the selection rates and the mean score of the positives (0.364 against 0.500) all differ. Equal base rates are not enough: the score distributions themselves must match.
Process note. The plan’s example entry named the two groups’ base rates, the Beta construction, the three criteria with calibration_curve, Hardt et al.’s two post-processing rules and an equal-base-rate control. This example departs from it in ten planned ways, each logged before the pilot: (1) a cost-weighted loss (a miss costs 4, a false alarm 1) with 0.2 as the shared Bayes threshold, expected cost printed beside accuracy, because under 0–1 loss the threshold 0.5 would select only about of group B; (2) the false omission rate added, since Kleinberg et al.’s condition (A) for a two-bin decision concerns both bins; (3) a hypothesis on the balance conditions (H1b); (4) a hypothesis for equalized odds (H2b), and false positive rates added to H2, after checking the plan against Theorem 1, which under equal opportunity requires only that (A) or (B) fail and forces (A) to fail only under equalized odds; (5) the control split into identical distributions and equal base rates with a different spread, because the plan’s control left the distributions open and the outcome depends on them; (6) the cost ordering H4, as a check of the optimizers; (7) 10 equal-width calibration bins with a per-bin criterion; (8) thresholds from the exact distributions; (9) the higher-ROC group randomized with its upper threshold at 1, a choice that changes no printed rate (Hardt et al.’s Figure 4 caption notes the mixture is not unique); (10) the checks C1–C7. All bars were written before any script ran; a planning script then printed the exact values of the measures, before the plan was frozen. The pilot ran the same script with seed 1; every check and every hypothesis held, as in the first run with seed 0 shown above, so no outcome differs between them. After the pilot, the scenario labels were changed from “C1” and “C2” to “control 1” and “control 2”, with the editor’s approval, to avoid a clash with the check labels; only those label strings changed.
Mitigation on a vision benchmark
Wang et al. (2020) built CIFAR-10S to compare mitigation methods where the bias is controlled exactly. Each CIFAR-10 class’s 5,000 training images are split 95% to 5% between colour and grayscale, five classes mostly colour and five mostly grayscale; testing uses colour and grayscale copies of the whole test set, and the primary metric is mean per-class, per-domain accuracy. A ResNet-18 trained from scratch on the skewed set scored 89.0% on colour test images, while one trained only on grayscale images, never having seen colour, scored 93.0% on them: the skewed model had learned to associate colour with five of the classes (§3). Their Table 1 (means over five runs, ±2 standard deviations) gives 88.5 ± 0.3% for the baseline, 89.1 ± 0.4% with oversampling, 83.8 ± 1.1% and 84.1 ± 1.0% for two forms of adversarial training (the text gives 83.4% for the first), 90.3 ± 0.5% for a domain-discriminative classifier with prior-shift inference, and 92.0 ± 0.1% for their domain-independent model, which trains a separate classifier head per domain on a shared representation and sums the heads’ activations at test time. Adversarial training enforced confusion between the domains, they report, but also introduced confusion between classes (their Figure 2), and they connect this to redundant encoding. On CelebA attribute recognition with gender as the domain (test set, 34 attributes, an ImageNet-pretrained ResNet-50), the domain-independent model raised a mean average precision weighted to give men and women equal weight from 74.7% to 76.3%, against 71.9% for adversarial training (Table 3).
The authors point out that the analogy between colour and gender is limited: they treat colour and grayscale as binary and disjoint, and the transformation between them is linear, whereas gender is more fluid and its manifestation more complex. Citing Zhao et al. (2017) for showing that a gender correlation in training data can become amplified in a model, they also report Zhao et al.’s bias amplification measure beside accuracy. NIST SP 1270 (§3.2.2) groups mitigation methods into pre-processing (changing the training data), in-processing (changing the training objective or constraints, as Wang et al.’s methods do) and post-processing (changing a trained model’s outputs, as Hardt et al.’s thresholds do).
Auditing a Vision System
An audit reports the criteria above for the decision the system actually makes. The published practice in vision comes from faces: NIST’s 2019 evaluation of demographic effects argues that because systems run at one fixed threshold, both error rates should be reported per group at that threshold, as the face recognition article describes, and Buolamwini and Gebru (2018) reported gender-classification error for subgroups formed by skin type and gender together, an intersectional breakdown, as dataset bias describes. In outline, an audit of a vision system:
- defines the groups, including capture conditions such as camera, site or lighting where those may matter, and records how group labels were obtained;
- reports each group’s rates at the operating threshold, with intervals, since a group’s sample may be small; statistical significance in evaluation gives the methods, and dataset bias the warning that each group needs enough examples for its error rate to mean something;
- reports more than one criterion, since by Kleinberg et al.’s theorem they will not all agree when base rates differ, and states which one the deployment is meant to satisfy;
- checks calibration within each group if scores are shown to people;
- for medical devices, follows the subgroup reporting items that clinical validation summarizes.
Group-specific thresholds are not neutral engineering choices. NIST SP 1270 (March 2022) notes that methods relying on demographic information, as most pre-, in- and post-processing methods do, may carry higher risk in regulated settings where disparate treatment must be avoided (§3.2.2), and the biometrics article reports that NIST’s digital identity guidelines (SP 800-63B-4, July 2025) require biometric systems used for U.S. federal authentication to meet their false match rate for all demographic groups with a fixed threshold. This article draws no legal conclusion about them.
Common Misconceptions
- “A calibrated model is fair.” Calibration within groups is one criterion. The example’s scores are exactly calibrated in both groups and still give different selection, true positive and false positive rates at one threshold, and different mean scores among positives and negatives.
- “Removing the protected attribute removes the bias.” Other inputs encode it (Hardt et al.’s redundant encodings); in images, Wang et al.’s linear SVM predicted the colour domain with 82% accuracy from features of a network that had never been trained on the colour–class skew.
- “Equal error rates make a decision mean the same thing in every group.” Under equalized odds the example’s precision is 0.42 in one group and 0.16 in the other; with unequal base rates and imperfect prediction, Kleinberg et al.’s theorem makes such a gap unavoidable.
- “Equal base rates remove the conflict.” They make the three conditions achievable together, but a calibrated score with a different spread in each group still fails balance and equal false positive rates, as the second control shows.
- “Fairness is a property of the model.” The same scores satisfy or fail a criterion depending on the threshold, the population and the criterion chosen. NIST AI 100-1 (§3.7) adds that systems in which harmful biases are mitigated are not necessarily fair: predictions somewhat balanced across demographic groups may still be inaccessible to people with disabilities, for example.
- “Gender Shades measured face recognition.” It evaluated commercial gender classification; NIST’s demographic study evaluated recognition, as the face recognition article explains.
Where It Is Used
The criteria apply wherever a vision model’s errors fall on people: face recognition and biometrics, where demographic differentials at a fixed threshold are measured; medical imaging, where clinical validation asks for subgroup results and radiology AI shows how precision depends on prevalence; and attribute recognition on datasets such as CelebA, on which Wang et al. tested mitigation. They rest on the rates of precision and recall, the per-group ROC curves that post-processing works on, and per-group calibration. Upstream of all of them, dataset bias covers how the composition and labelling of training and test data create the gaps that these criteria measure.
Related
- Face Recognition
Deciding whether two face images show the same person (verification) or which enrolled person a face belongs to (identification, closed- or open-set), with the detect, align, embed and compare pipeline, its benchmarks and evaluation protocols, and NIST's measurements of demographic differentials.
- Biometric Recognition Systems
Recognizing people automatically from their face, iris, fingerprints or other body traits; how systems enrol people and then verify or identify them, how their errors are measured (FMR and FNMR, FAR and FRR, FPIR and FNIR, DET curves and the equal error rate), and how presentation attacks, template protection, demographic differentials and regulation shape their use.
- ROC Curve and AUC
The area under the receiver operating characteristic curve, a threshold-free measure of how well a classifier's scores rank positives above negatives, with its probabilistic meaning, computation with tied scores, partial and multi-class variants, and the operating-point summaries used in vision.
- Statistical Significance in Evaluation
How much of a reported difference between two vision models is noise from the finite test set or from training randomness, and the confidence intervals, paired tests, multiple-comparison corrections and reporting practices that tell a real improvement from a lucky seed.
- Clinical Validation and Regulation of Medical Imaging AI
How the claims made for a medical image analysis model are tested, from internal, temporal and external validation through reader studies, prospective studies and randomized trials, with the reporting guidelines that describe such studies, the statistics of rare findings, the regulatory pathways in outline, and monitoring after deployment.
- Computer Vision in Radiology
How computer vision is used on radiological images such as chest X-rays, mammograms, CT, MRI and ultrasound, for computer-aided detection and diagnosis, triage of reading worklists, quantification and report generation, and how such systems are evaluated, why they fail through shortcut learning and dataset shift, and how regulators classify them.
References
- Hardt, M., Price, E. & Srebro, N. (2016). Equality of Opportunity in Supervised Learning. Advances in Neural Information Processing Systems 29 (NIPS).
- Kleinberg, J., Mullainathan, S. & Raghavan, M. (2017). Inherent Trade-Offs in the Fair Determination of Risk Scores. 8th Innovations in Theoretical Computer Science Conference (ITCS 2017), Leibniz International Proceedings in Informatics (LIPIcs), 67, 43:1–43:23.
- Wang, Z., Qinami, K., Karakozis, I. C., Genova, K., Nair, P., Hata, K. & Russakovsky, O. (2020). Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
- Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A. & Hall, P. (2022). Towards a Standard for Identifying and Managing Bias in Artificial Intelligence. NIST Special Publication 1270, National Institute of Standards and Technology.
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January 2023.
- Buolamwini, J. & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Conference on Fairness, Accountability and Transparency (FAT*), Proceedings of Machine Learning Research, 81, 77–91.
- Zhao, J., Wang, T., Yatskar, M., Ordonez, V. & Chang, K.-W. (2017). Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. Conference on Empirical Methods in Natural Language Processing (EMNLP) (not read).