Vision Systems & Engineering
Testing and Validating Vision Systems
How a vision system is shown to meet its requirements before and after it goes into service, from the test plan and held-out tests to slice analysis, behavioural and metamorphic tests, stress tests at the requirement's limits, regression tests and the field acceptance test, with an inspection example in which the aggregate passes, one slice fails and a label-free test points to it.
intermediate
A vision system is tested to find out where it fails and validated to show that it meets the requirements written for its use. The two words are easily run together, and the methods overlap, but the questions differ: a test asks whether the model, the code around it or the whole station behaves as specified on cases chosen to expose faults, while validation asks whether the system, as installed, does what its users need under the conditions it will meet. The NIST AI Risk Management Framework, quoting ISO 9000:2015, defines validation as confirmation by objective evidence that the requirements for a specific intended use have been fulfilled (NIST AI 100-1, §3.1), and groups the whole activity as test, evaluation, verification and validation (TEVV). This article covers the parts of that work that are specific to learned vision systems: the test plan, held-out tests against field validation, slice analysis, behavioural (metamorphic) tests, stress tests at the limits of the requirements, regression tests across model versions and the acceptance test in the field. Requirements and the outline of validation are set in designing a computer vision system; the intervals and corrections that every count below needs are derived in statistical significance in evaluation.
Purpose
A single score on a held-out test set answers a narrow question: how often the model is right on images drawn the way the test set was drawn. A deployed system needs answers to other questions. Does it meet the requirement under each condition the requirement names, including rare ones? Does it fail in a way that a change of lighting, camera or product would trigger? Did the new model version break something the old one did right? Will it keep working on the real line or site? Testing and validation turn each question into a test case with a pass or fail criterion fixed in advance, so that the decision to ship rests on evidence rather than on an average.
The NIST framework puts the same point in general terms: accuracy measurements should be paired with clearly defined test sets that are representative of the conditions of expected use, together with details of the test method, and they may be broken down by data segment (NIST AI 100-1, §3.1). Its examples of computational accuracy measures are false-positive and false-negative rates, quantities in which a vision system’s requirements can be written.
When to Use It
Every system that is deployed gets tested in some form; the question is how much of the plan below is worth its cost.
- Required by the application. Inspection systems that reject parts, medical devices and vehicles have written requirements and someone who signs off against them. Industrial visual inspection works in escape rates and false-reject rates; clinical validation of medical AI covers the study designs that regulators and clinicians expect, which are more demanding than anything here.
- Heterogeneous conditions. When the system will meet several cameras, sites, lighting states or product variants, the conditions become slices that must each be tested, because an average over them can hide one that fails.
- Model updates. Each retrained or re-exported model is a new system; regression tests decide whether it may replace the old one.
- Less useful for one-off analyses whose output a person reviews anyway, where an honest held-out score and a look at the errors may be enough (our judgement).
How It Works
The test plan
A test plan links each written requirement to the tests that check it and to the criterion that decides each test. A requirement such as “recall of at least 0.95 for spot defects, under illumination from 50% to 100% of nominal, on both camera models” generates several test cases: a held-out test set drawn from the expected field mix, a slice for each camera and illumination band, a stress set at 50% illumination, and the acceptance test on the line. Each test case names its data, its metric, its criterion and the sample size needed for the criterion to be decidable. Writing the criterion before the data are seen is what separates a test from an exploration of results (our reading; benchmark overfitting covers the theory and the replication studies of what repeated use of one test set does to its scores).
The plan also says who runs the tests and who judges them. The NIST framework states that, ideally, the people who carry out verification and validation are distinct from those who perform test and evaluation, and that tests can be planned as early as the design phase, in accordance with the design requirement (NIST AI 100-1, Appendix A). It lists TEVV tasks for each phase of the lifecycle: validating design assumptions and data collection against the intended context; model validation during development; system validation and integration testing at deployment; and, in operation, ongoing monitoring, periodic re-testing and the tracking of incidents and errors.
Held-out tests and field validation answer different questions
A held-out test set measures performance on images like the training data. Training, validation and test sets covers how to split so that the score is honest: frames of one video, or images of one part, stay on one side. A held-out condition test keeps whole sites, cameras or dates out of training, which estimates the drop under shifts that are expected; the dataset bias article shows how much a change of dataset costs, and Vision System Design summarizes the camera-trap and WILDS results for a change of site.
Koh et al. (2021) distinguish two kinds of shift in WILDS. In domain generalization the test domains are unseen in training, such as new hospitals or new camera traps; in subpopulation shift the test domains were all seen in training but appear in different proportions, and the aim is to do well on each of them. For subpopulation shift they typically seek to minimize the maximum error over the test domains, and they evaluate by the worst case: the lowest true positive or true negative rate over eight demographic groups for a text dataset, and, for satellite images in a hybrid setting that also shifts in time, the accuracy on the worst geographical region (their Sections 3 and 4). In our reading, field validation in a vision system meets both kinds: new installations are new domains, and the mix of conditions at an installation changes over the seasons.
Slices
A slice is a subset of the test data defined by metadata: camera, site, lighting, product variant, object size, time of day. Slice analysis reports the metric per slice, with its interval and its sample size, and judges the worst slice against the requirement as well as the average. Subsets that no metadata field marks, such as subtle findings among obvious ones, are what Oakden-Rayner et al. (2020) call hidden stratification in the title of their paper; chest X-ray analysis discusses studies that document such gaps between subsets. Three properties of slices decide how they can be used, as the example below shows in numbers:
- The aggregate is a weighted mean of the slices, with weights equal to their shares of the test data, so a slice that is 2% of the data can fail badly while the aggregate passes.
- A slice’s interval depends on the slice’s own count, not on the size of the whole test set. Rare slices need targeted collection to be bounded at all.
- Testing many slices at once is a multiple-comparisons problem. A rule that every slice must pass at 95% confidence controls each slice’s error separately; a rule meant to hold for all slices together needs a correction such as Bonferroni’s or Holm’s, as statistical significance in evaluation explains. The test plan states which one applies.
Behavioural and metamorphic tests
Ribeiro et al. (2020) adapted behavioural (black-box) testing from software engineering to NLP models in CheckList. It organizes tests in a matrix of capabilities against three test types (their Section 2.2):
- a minimum functionality test (MFT), a set of simple labelled cases that targets one behaviour, like a unit test;
- an invariance test (INV), which applies label-preserving perturbations and expects the prediction not to change;
- a directional expectation test (DIR), which applies a perturbation whose effect on the label is known and checks that the prediction moves that way.
The authors describe the last two as inspired by metamorphic tests in software, and note that they test relationships between predictions rather than predictions against labels, so they run on unlabelled data. In a user study they report that practitioners using CheckList created more than twice as many tests and found almost three times as many bugs as practitioners without it. CheckList is an NLP paper; transferring its test types to images is our step, not the authors’.
For a vision system, the transfer is direct (our examples). An MFT is a small rendered or curated set for one behaviour: a defect of each type on a clean part, a sign at the minimum size. An INV test applies a change the requirement says must not matter, such as a shift of a pixel or two, a gain change within the specified illumination range, or a horizontal flip for a class that is symmetric, and counts the decisions that change. A DIR test applies a change with a known direction: deepening a defect’s contrast should not lower its score. The label for an invariance test comes from the requirement, not from an annotator, so these tests can run on every field image without ground truth. Care is needed with the perturbation: a flip changes the label of a turn-left sign, as traffic sign recognition points out with its mirror pairs, and a gain change beyond the specified range is a stress test, not an invariance.
Stress tests
A stress test renders or collects data at the edges of the requirements: the minimum illumination, the maximum blur at the highest conveyor speed, the most extreme viewing angle. Vision System Design describes ImageNet-C’s corruptions and suggests rendering a requirement’s conditions in the same way, and Traffic Sign Recognition runs a network under five corruptions. Stress data answer a question the field-mix test cannot: the field mix may contain no image at the requirement’s limit, so it cannot show that the limit is met. The criterion is the same requirement, applied to the stress set alone.
Regression tests across versions
A new model, a new export or a new preprocessing library is a new system. Regression tests rerun the whole plan, slices included, on the new version, and add tests of agreement with the old one: the fraction of decisions that change on a fixed golden set, and which slices those changes fall in (our outline). Deploying vision models covers the numerical side, parity tests of the serving pipeline against a golden set, and the release side, canaries and rollback, which it reports from Breck et al. (2017).
Acceptance in the field
The last test runs the system on the real line or site against the written requirements. Vision System Design describes running it first in shadow, with decisions recorded but not acted on; Deploying Vision Models notes that the sources it cites do not describe shadow deployment. What is counted in shadow is the same as in the plan: decisions per slice, against labels from a reviewer or from a downstream check, with the sample size each criterion needs fixed beforehand. For rare failures that size is large: Vision System Design derives the zero-failure size for a miss probability , and industrial visual inspection gives the rule of three for zero observed escapes. After acceptance, the same counts continue in service as model monitoring and data drift detection.
How to Apply It
In outline:
- Write each requirement as a testable statement with its metric, threshold, conditions and confidence.
- For each requirement, list the test cases: held-out field mix, each slice the conditions name, stress sets at the limits, behavioural tests for the invariances the requirement implies.
- Fix each criterion and its sample size before collecting results, and state whether slice criteria hold per slice or family-wise.
- Collect or render the data each test needs, including targeted collection for rare slices.
- Run the tests, report every slice with its count and interval, and record the verdicts.
- Rerun the plan on every new version, and continue the counts in the field.
Example: an aggregate that passes, a slice that fails
The script renders floating-point images of a flat part surface lit at illumination : a defective part carries a dark spot that reduces reflectance by a contrast , so the image is , where coverage is the fraction of each pixel inside the disc, computed by supersampling (pixel covers , row first). Camera A is sharp; camera B blurs with a Gaussian of pixel. Sensor noise is Gaussian with standard deviation 0.05, added after the blur and independent of . Each image carries metadata drawn from a field mix: illumination nominal (, share 0.8) or dim (, 0.2); camera A (0.7) or B (0.3); spot large (radius 3 pixels, 0.6) or small (radius 1.5, 0.4), with the spot centre uniform in in both coordinates. The rare cell, dim light on camera B with a small spot, is 2.4% of defective parts.
The detector is fixed and untrained: it subtracts the image median, so that a dark spot gives positive values, correlates the result with a unit-norm disc template of radius 1.5 (a matched filter, as in ROC AUC; template matching describes such correlation scores), and takes the maximum over all valid positions. A part is called defective when the score exceeds , the 99th percentile of the scores of 2,000 good parts at nominal light on camera A, so that the target false-positive rate (FPR) is 1%. The written requirement is recall of at least 0.95 with illumination from 0.5 to 1.0.
The script measures, on a field-mix test set of 6,000 defective and 6,000 good parts:
- aggregate recall and FPR, with exact (Clopper–Pearson) 95% intervals from SciPy’s
binomtest(...).proportion_ci(), whose default is the exact two-sided interval; every “lower bound” below is the lower end of this two-sided interval, equivalently a one-sided 97.5% bound; - recall per cell (8 cells) beside a population recall from 20,000 further images per cell, and FPR per illumination × camera cell (4 cells, since a good part has no spot size);
- the flip rate per cell under two transformations that the requirement says must not change the decision: a one-pixel circular shift along the columns and a digital gain of 0.9 (which keeps nominal and dim light inside the 0.5–1.0 range), each separately and either;
- three acceptance verdicts: the aggregate lower bound against 0.95; every cell’s lower bound at 95% each (per cell); and every cell’s lower bound at confidence (family-wise, Bonferroni over 8 cells); with, for each cell, the smallest test count whose exact lower bound would clear 0.95 at the cell’s population recall, and the field-mix test size that count implies;
- recall on a stress set of 2,000 defective and 2,000 good parts at , the requirement’s minimum, which the field mix never contains;
- a test of the size Vision System Design’s zero-failure rule gives for the aggregate requirement, defective parts from the field mix, and the rare cell’s share of it.
Checks confirm the data (every defective noise-free image has a spot, good ones are flat, the noise has the stated deviation), the matched filter against scipy.signal.correlate2d, the per-cell counts against scikit-learn’s recall_score, the vectorized bound against binomtest, the zero-failure size against a direct search, and whether the aggregate FPR interval contains the 1% target. The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2, runs in about 12 seconds, and two runs printed identical output.
import os
os.environ["OMP_NUM_THREADS"] = "1" # one BLAS thread, set before NumPy loads
os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from numpy.lib.stride_tricks import sliding_window_view
from scipy.ndimage import gaussian_filter
from scipy.signal import correlate2d
from scipy.stats import beta, binomtest
from sklearn.metrics import recall_score
SIZE, SIGMA, SS, CONTRAST, REQ = 32, 0.05, 8, 0.25, 0.95
ILLUM = {"nominal": 1.0, "dim": 0.6}; L_MIN = 0.5 # requirement: L from 0.5 to 1.0
BLUR = {"A": 0.0, "B": 1.0} # camera B: Gaussian blur, sigma 1 px
RADIUS = {"large": 3.0, "small": 1.5}
SHARE = {"nominal": 0.8, "dim": 0.2, "A": 0.7, "B": 0.3, "large": 0.6, "small": 0.4}
CELLS = [(i, b, r) for i in ILLUM for b in BLUR for r in RADIUS]
RARE = ("dim", "B", "small")
def disc_coverage(cy, cx, r):
"""Fraction of each pixel inside a disc of radius r centred at (cy, cx)
(pixel (i, j) covers [i, i+1) x [j, j+1)); 8 x 8 supersampling of a
10 x 10 pixel patch around the centre."""
cov = np.zeros((len(cy), SIZE, SIZE))
oy, ox = np.floor(cy).astype(int) - 4, np.floor(cx).astype(int) - 4
sub = (np.arange(10 * SS) + 0.5) / SS
for s in range(0, len(cy), 2000):
dy = oy[s:s + 2000, None] + sub - cy[s:s + 2000, None]
dx = ox[s:s + 2000, None] + sub - cx[s:s + 2000, None]
inside = dy[:, :, None] ** 2 + dx[:, None, :] ** 2 <= r * r
patch = inside.reshape(-1, 10, SS, 10, SS).mean(axis=(2, 4))
for j, k in enumerate(range(s, s + len(patch))):
cov[k, oy[k]:oy[k] + 10, ox[k]:ox[k] + 10] = patch[j]
return cov
g = (np.arange(5 * SS) + 0.5) / SS - 2.5 # matched filter: the small disc
TEMPLATE = ((g[:, None] ** 2 + g[None, :] ** 2) <= 1.5 ** 2).reshape(5, SS, 5, SS).mean(axis=(1, 3))
TEMPLATE /= np.linalg.norm(TEMPLATE)
def scores(img):
"""Fixed, untrained detector: median background minus image (a dark spot
gives positive values), correlated with the template; the maximum over
all 28 x 28 'valid' positions is the score."""
r = np.median(img, axis=(1, 2), keepdims=True) - img
win = sliding_window_view(r, (5, 5), axis=(1, 2))
return np.einsum("nijkl,kl->nij", win, TEMPLATE).max(axis=(1, 2))
def make(n, rng, defective, illum=None, cam=None, size=None):
"""Render n part images. Metadata not fixed by the caller is drawn from the
field mix. Returns images, noise-free images and the metadata."""
il = np.full(n, illum) if illum else np.where(rng.random(n) < SHARE["nominal"], "nominal", "dim")
cm = np.full(n, cam) if cam else np.where(rng.random(n) < SHARE["A"], "A", "B")
sz = np.full(n, size) if size or not defective else np.where(rng.random(n) < SHARE["large"], "large", "small")
L = np.array([ILLUM[v] for v in il]) if illum != "min" else np.full(n, L_MIN)
cov = np.zeros((n, SIZE, SIZE))
if defective:
cy, cx = rng.uniform(8, 24, n), rng.uniform(8, 24, n)
for r in RADIUS:
m = sz == r
cov[m] = disc_coverage(cy[m], cx[m], RADIUS[r])
clean = L[:, None, None] * (1.0 - CONTRAST * cov)
b = cm == "B"
clean[b] = gaussian_filter(clean[b], sigma=(0, BLUR["B"], BLUR["B"]), mode="nearest")
return clean + rng.normal(0, SIGMA, clean.shape), clean, il, cm, sz
def ci(k, n, conf=0.95): # exact (Clopper-Pearson), two-sided
r = binomtest(int(k), int(n)).proportion_ci(confidence_level=conf)
return r.low, r.high
def cp_low(k, n, conf): # the same lower bound, vectorized
return np.where(k > 0, beta.ppf((1 - conf) / 2, np.maximum(k, 1), n - k + 1), 0.0)
def needed(p, conf):
"""Smallest n whose exact lower bound at k = round(p n) clears REQ."""
if p <= REQ:
return None
n = np.arange(1, 200001)
ok = np.nonzero(cp_low(np.round(p * n), n, conf) >= REQ)[0]
return int(n[ok[0]]) if ok.size else None
s_val, s_test, s_stress, s_pop, s_vsd = [np.random.default_rng(c) for c in np.random.SeedSequence(0).spawn(5)]
# Threshold from good parts only: 99th percentile of 2,000 nominal camera-A scores
val = scores(make(2000, s_val, False, "nominal", "A")[0])
T = np.quantile(val, 0.99, method="higher")
print(f"threshold {T:.5f}; validation FPR {np.mean(val > T):.4f} (target 0.01)")
# Field-mix test set, its metamorphic copies and decisions
D, Dc, Di, Db, Dr = make(6000, s_test, True)
G, Gc, Gi, Gb, _ = make(6000, s_test, False)
dec, flip = {}, {}
for name, X in (("def", D), ("good", G)):
d0 = scores(X) > T
dsh = scores(np.roll(X, 1, axis=2)) > T # 1-pixel circular shift
dga = scores(0.9 * X) > T # digital gain 0.9
dec[name], flip[name] = d0, (dsh != d0, dga != d0, (dsh != d0) | (dga != d0))
LD = np.array([ILLUM[v] for v in Di])
print("C1 data: defective images with no spot below L:",
int(np.sum(Dc.min(axis=(1, 2)) >= LD - 1e-12)),
f"| good noise-free max |I - L| {np.max(np.abs(Gc - np.array([ILLUM[v] for v in Gi])[:, None, None])):.1e}",
f"| noise std {np.std(G - Gc):.5f} (sigma {SIGMA})")
k, n = int(dec["def"].sum()), len(D)
lo, hi = ci(k, n)
fp = int(dec["good"].sum()); flo, fhi = ci(fp, len(G))
print(f"\nM1 aggregate recall {k}/{n} = {k / n:.4f} [{lo:.4f}, {hi:.4f}] "
f"FPR {fp}/{len(G)} = {fp / len(G):.4f} [{flo:.4f}, {fhi:.4f}]")
print("\nM2/M3 defective cells: share, test n (expected), recall [exact 95%], population recall,"
" flip rate under shift / gain / either")
rows, pop, c3_bad = {}, {}, 0
for cell in CELLS:
m = (Di == cell[0]) & (Db == cell[1]) & (Dr == cell[2])
w = SHARE[cell[0]] * SHARE[cell[1]] * SHARE[cell[2]]
kc, nc = int(dec["def"][m].sum()), int(m.sum())
c3_bad += round(recall_score(np.ones(nc, int), dec["def"][m].astype(int)) * nc) != kc
pop[cell] = np.mean(scores(make(20000, s_pop, True, *cell)[0]) > T)
fl = [int(f[m].sum()) for f in flip["def"]]
rows[cell] = (kc, nc, fl, w)
l, h = ci(kc, nc)
print(f" {' '.join(cell):17s} {w:.3f} {nc:5d} ({w * n:6.1f}) {kc / nc:.4f} [{l:.4f}, {h:.4f}]"
f" pop {pop[cell]:.4f} flips {fl[0] / nc:.4f} {fl[1] / nc:.4f} {fl[2] / nc:.4f} ({fl[2]})")
print("M2/M3 good cells: test n (expected), FPR [exact 95%], flip rate under shift / gain / either")
gflip = {}
for i in ILLUM:
for b in BLUR:
m = (Gi == i) & (Gb == b)
w = SHARE[i] * SHARE[b]
fc, nc = int(dec["good"][m].sum()), int(m.sum())
fl = [int(f[m].sum()) for f in flip["good"]]
gflip[(i, b)] = (fl[2], nc)
l, h = ci(fc, nc)
print(f" {i} {b:11s} {nc:5d} ({w * len(G):6.1f}) {fc / nc:.4f} [{l:.4f}, {h:.4f}]"
f" flips {fl[0] / nc:.4f} {fl[1] / nc:.4f} {fl[2] / nc:.4f} ({fl[2]})")
print("\nM4 acceptance verdicts")
print(f" aggregate rule: lower bound {lo:.4f} -> {'PASS' if lo >= REQ else 'FAIL'}")
c4 = []
for conf, label in ((0.95, "per-cell rule (95% each)"), (1 - 0.05 / 8, "family-wise rule (Bonferroni, 8 cells)")):
lows = {c: ci(rows[c][0], rows[c][1], conf)[0] for c in CELLS}
c4 += [abs(lows[c] - cp_low(np.array(rows[c][0]), np.array(rows[c][1]), conf)) for c in CELLS]
worst = min(lows, key=lows.get)
print(f" {label}: lowest bound {lows[worst]:.4f} ({' '.join(worst)}) -> "
f"{'PASS' if lows[worst] >= REQ else 'FAIL'}")
print(" test count each cell would need (95% / family-wise), and the field-mix total it implies")
for cell in CELLS:
out = []
for conf in (0.95, 1 - 0.05 / 8):
nn = needed(pop[cell], conf)
if nn is not None:
c4.append(abs(ci(round(pop[cell] * nn), nn, conf)[0] - cp_low(np.round(pop[cell] * nn), nn, conf)))
w = rows[cell][3]
out.append("never" if nn is None else f"{nn} (total {int(np.ceil(round(nn / w, 9)))})") # 72 / 0.036 = 2000.0000000000002
print(f" {' '.join(cell):17s} pop {pop[cell]:.4f}: {out[0]} / {out[1]}")
S, _, _, _, _ = make(2000, s_stress, True, "min")
SG, _, _, _, _ = make(2000, s_stress, False, "min")
ks, fs = int((scores(S) > T).sum()), int((scores(SG) > T).sum())
slo, shi = ci(ks, 2000)
print(f"\nM5 stress set at L = {L_MIN}: recall {ks}/2000 = {ks / 2000:.4f} [{slo:.4f}, {shi:.4f}] "
f"FPR {fs}/2000 = {fs / 2000:.4f} [{ci(fs, 2000)[0]:.4f}, {ci(fs, 2000)[1]:.4f}]")
n59 = int(np.ceil(np.log(0.05) / np.log(REQ))) # miss rate 1 - REQ: ln(1 - p) = ln REQ
n_search = next(m for m in range(1, 10000) if REQ ** m <= 0.05) # independent check
V, _, Vi, Vb, Vr = make(n59, s_vsd, True)
dv = scores(V) > T
mr = (Vi == RARE[0]) & (Vb == RARE[1]) & (Vr == RARE[2])
print(f"\nM6 zero-failure test size for recall >= {REQ}: {n59} defective images; "
f"misses {int((~dv).sum())} -> {'PASS' if dv.all() else 'FAIL'}")
if mr.sum():
vl, vh = ci(int(dv[mr].sum()), int(mr.sum()))
print(f" rare cell: {int(dv[mr].sum())}/{int(mr.sum())} detected, exact [{vl:.4f}, {vh:.4f}]")
else:
vl = None
print(" rare cell: no images, so no bound")
print("\nchecks")
print(f"C1 every cell has images: {min(r[1] for r in rows.values())} defective, "
f"{min(v[1] for v in gflip.values())} good (smallest cell)")
X = D[:100].tolist() + G[:100].tolist()
c2 = max(abs(correlate2d(np.median(x) - np.array(x), TEMPLATE, mode="valid").max() - s)
for x, s in zip(X, scores(np.array(X))))
print(f"C2 matched filter vs scipy correlate2d: max |diff| {c2:.1e}")
print(f"C3 cells where recall_score disagrees with the counts: {c3_bad}")
print(f"C4 beta.ppf vs binomtest bound: max |diff| {max(c4):.1e}")
print(f"C6 zero-failure size: formula {n59}, search {n_search}, equal: {n59 == n_search}")
print(f"C5 aggregate FPR interval [{flo:.4f}, {fhi:.4f}] contains 0.01: {flo <= 0.01 <= fhi}")
print("\nhypotheses")
kr, nr = rows[RARE][0], rows[RARE][1]
agg_pop = sum(rows[c][3] * pop[c] for c in CELLS)
print(f"H1 aggregate {k / n:.4f}, lower {lo:.4f}; rare cell upper {ci(kr, nr)[1]:.4f}; "
f"population aggregate {agg_pop:.4f}, rare {pop[RARE]:.4f}")
fr, nrr = rows[RARE][2], rows[RARE][1]
for t, name in enumerate(("shift", "gain", "either")):
others = [c for c in CELLS if c != RARE]
best = max(others, key=lambda c: rows[c][2][t] / rows[c][1])
strict = all(fr[t] * rows[c][1] > rows[c][2][t] * nrr for c in others)
print(f"H2 {name:6s}: rare {fr[t] / nrr:.4f}, highest other {rows[best][2][t] / rows[best][1]:.4f} "
f"({' '.join(best)}); rare strictly highest: {strict}")
gbest = max(gflip, key=lambda c: gflip[c][0] / gflip[c][1])
print(f"H2 good cells: highest 'either' flip rate {gflip[gbest][0] / gflip[gbest][1]:.4f} ({' '.join(gbest)}),"
f" rare cell above it: {fr[2] * gflip[gbest][1] > gflip[gbest][0] * nrr}")
print(f"H3 stress {ks / 2000:.4f} < field mix {k / n:.4f}: {ks / 2000 < k / n}; "
f"H3b stress upper {shi:.4f} < field-mix lower {lo:.4f}: {shi < lo}")
print(f"H4 rare cell in the {n59}-image test: "
+ ("no images" if vl is None else f"lower bound {vl:.4f}, below {REQ}: {vl < REQ}"))
Output (NumPy 2.3.5, SciPy 1.16.3, scikit-learn 1.7.2; identical over two runs):
threshold 0.21153; validation FPR 0.0095 (target 0.01)
C1 data: defective images with no spot below L: 0 | good noise-free max |I - L| 0.0e+00 | noise std 0.05002 (sigma 0.05)
M1 aggregate recall 5945/6000 = 0.9908 [0.9881, 0.9931] FPR 44/6000 = 0.0073 [0.0053, 0.0098]
M2/M3 defective cells: share, test n (expected), recall [exact 95%], population recall, flip rate under shift / gain / either
nominal A large 0.336 2037 (2016.0) 1.0000 [0.9982, 1.0000] pop 1.0000 flips 0.0000 0.0000 0.0000 (0)
nominal A small 0.224 1321 (1344.0) 1.0000 [0.9972, 1.0000] pop 1.0000 flips 0.0000 0.0000 0.0000 (0)
nominal B large 0.144 864 ( 864.0) 1.0000 [0.9957, 1.0000] pop 1.0000 flips 0.0000 0.0000 0.0000 (0)
nominal B small 0.096 598 ( 576.0) 0.9950 [0.9854, 0.9990] pop 0.9986 flips 0.0000 0.0050 0.0050 (3)
dim A large 0.084 553 ( 504.0) 1.0000 [0.9934, 1.0000] pop 1.0000 flips 0.0000 0.0000 0.0000 (0)
dim A small 0.056 317 ( 336.0) 0.9968 [0.9826, 0.9999] pop 0.9953 flips 0.0000 0.0063 0.0063 (2)
dim B large 0.036 199 ( 216.0) 1.0000 [0.9816, 1.0000] pop 1.0000 flips 0.0000 0.0000 0.0000 (0)
dim B small 0.024 111 ( 144.0) 0.5405 [0.4433, 0.6355] pop 0.5694 flips 0.0000 0.2162 0.2162 (24)
M2/M3 good cells: test n (expected), FPR [exact 95%], flip rate under shift / gain / either
nominal A 3311 (3360.0) 0.0082 [0.0054, 0.0118] flips 0.0000 0.0069 0.0069 (23)
nominal B 1488 (1440.0) 0.0060 [0.0028, 0.0115] flips 0.0020 0.0054 0.0074 (11)
dim A 861 ( 840.0) 0.0093 [0.0040, 0.0182] flips 0.0000 0.0081 0.0081 (7)
dim B 340 ( 360.0) 0.0000 [0.0000, 0.0108] flips 0.0000 0.0000 0.0000 (0)
M4 acceptance verdicts
aggregate rule: lower bound 0.9881 -> PASS
per-cell rule (95% each): lowest bound 0.4433 (dim B small) -> FAIL
family-wise rule (Bonferroni, 8 cells): lowest bound 0.4073 (dim B small) -> FAIL
test count each cell would need (95% / family-wise), and the field-mix total it implies
nominal A large pop 1.0000: 72 (total 215) / 113 (total 337)
nominal A small pop 1.0000: 72 (total 322) / 113 (total 505)
nominal B large pop 1.0000: 72 (total 500) / 113 (total 785)
nominal B small pop 0.9986: 72 (total 750) / 113 (total 1178)
dim A large pop 1.0000: 72 (total 858) / 113 (total 1346)
dim A small pop 0.9953: 72 (total 1286) / 156 (total 2786)
dim B large pop 1.0000: 72 (total 2000) / 113 (total 3139)
dim B small pop 0.5694: never / never
M5 stress set at L = 0.5: recall 1756/2000 = 0.8780 [0.8629, 0.8920] FPR 17/2000 = 0.0085 [0.0050, 0.0136]
M6 zero-failure test size for recall >= 0.95: 59 defective images; misses 0 -> PASS
rare cell: 1/1 detected, exact [0.0250, 1.0000]
checks
C1 every cell has images: 111 defective, 340 good (smallest cell)
C2 matched filter vs scipy correlate2d: max |diff| 2.2e-16
C3 cells where recall_score disagrees with the counts: 0
C4 beta.ppf vs binomtest bound: max |diff| 4.5e-13
C6 zero-failure size: formula 59, search 59, equal: True
C5 aggregate FPR interval [0.0053, 0.0098] contains 0.01: False
hypotheses
H1 aggregate 0.9908, lower 0.9881; rare cell upper 0.6355; population aggregate 0.9893, rare 0.5694
H2 shift : rare 0.0000, highest other 0.0000 (nominal A large); rare strictly highest: False
H2 gain : rare 0.2162, highest other 0.0063 (dim A small); rare strictly highest: True
H2 either: rare 0.2162, highest other 0.0063 (dim A small); rare strictly highest: True
H2 good cells: highest 'either' flip rate 0.0081 (dim A), rare cell above it: True
H3 stress 0.8780 < field mix 0.9908: True; H3b stress upper 0.8920 < field-mix lower 0.9881: True
H4 rare cell in the 59-image test: lower bound 0.0250, below 0.95: True
The aggregate passes with room to spare: 5,945 of 6,000 defects found, and an exact lower bound of 0.9881 against the requirement of 0.95. The cell table shows what the aggregate hides. Seven cells are at or near perfect recall, but dim light on camera B with a small spot finds 60 of 111 defects, an interval of 0.4433 to 0.6355, and its population recall of 0.5694, inside that interval, shows that the test set did not draw an unlucky sample. The cell’s share is 0.024, so even a recall of zero there would cost the aggregate less than two and a half points, which is why the aggregate cannot fail on its account. The construction says why this cell is the weak one: dim light scales the spot’s depth by 0.6, the blur of camera B spreads a small spot over more pixels than the radius-1.5 template covers, and the noise does not shrink with the light, so the three together leave the spot’s response close to the threshold, while each alone does not (our reading of the construction, consistent with the high recall of the dim camera-A and nominal camera-B small-spot cells).
The metamorphic test needs no labels and points at the same cell. Because the gain multiplies the image, its median and so every window sum by 0.9, a decision flips under the gain only when a score lies just above the threshold, above and at most . In the rare cell 24 of 111 defects (0.2162) lie in that band; in every other defective cell the rate is at most 0.0063, and in the good cells at most 0.0081. A team that ran the gain test on unlabelled field images, grouped by the metadata it has, would see one group whose decisions are fragile and would know where to collect labelled defects; grouping defective images by spot size, as here, uses the rendered size, which in the field would be known only after review. The circular shift changed no decision in any defective cell; among good parts it changed decisions only in the nominal, camera-B cell (rate 0.0020).
The acceptance verdicts differ by rule. The aggregate rule passes; the per-cell rule fails at the rare cell’s lower bound of 0.4433, and so does the family-wise rule at 0.4073. The needed-count column shows why the per-cell rule is demanding even where a cell is perfect: an exact lower bound of 0.95 needs 72 defects with none missed at 95% confidence, and 113 at the Bonferroni level, so every cell needs at least that many, and the dim, camera-B, large-spot cell alone needs a field-mix test of 2,000 defective parts to collect its 72. Dim light on camera A with small spots needs 156 at the family-wise level, because at its population recall of 0.9953 the expected count at 113 parts, rounded, already includes one miss. For the rare cell no count suffices, since its recall is below the requirement.
The stress set at finds 1,756 of 2,000 defects, a recall of 0.8780 with an upper bound of 0.8920, below both the requirement and the field-mix result. The field-mix test contains no part at that illumination, so only the stress set can show that the requirement fails at its own limit. The zero-failure test of 59 defective parts, the size that bounds an aggregate miss rate of 5% at one-sided 95% confidence when no defect is missed (fewer than the 72 above, whose two-sided intervals put 2.5% in each tail), passes with no miss; it contains one part from the rare cell, which was detected, and an interval of 0.0250 to 1.0000 from one part says nothing about that cell.
All checks but one held. The data checks held (no defective image without a spot, good images flat to 0.0e+00, noise standard deviation 0.05002), the matched filter agreed with correlate2d to 2.2e-16, the counts agreed with recall_score in every cell, the vectorized bound agreed with binomtest to 4.5e-13, and the zero-failure size from the formula equalled the direct search (59). The check that the aggregate FPR interval contains the 1% target failed: the interval is 0.0053 to 0.0098, with a validation FPR of 0.0095.
The four planned hypotheses came out as follows. H1, that the aggregate meets the requirement (point and lower bound) while the rare cell’s upper bound falls below it, held, and so did its population form (aggregate population recall 0.9893, rare cell 0.5694). H2, that the flip rate is strictly highest in the rare cell, held under the gain and under either transformation, and held against the good cells; it failed under the circular shift alone, where every defective cell printed 0.0000. H3, that stress recall is below field-mix recall, held, and so did its stronger form that the stress set’s upper bound (0.8920) lies below the field-mix lower bound (0.9881). H4, that a test of the zero-failure size cannot bound the rare cell’s recall above 0.95, held: the cell’s lower bound is 0.0250.
How the example was planned and changed. The hypotheses, measures and checks were written before any script ran. The contrast was then chosen at planning from the grid 0.15, 0.20, 0.25 and 0.30 by a rule fixed beforehand: the value whose rare-cell population recall came closest to 0.6, provided every other cell reached 0.97 and the aggregate 0.96. H1 and H3 therefore illustrate a constructed failure rather than test for one. The planning scripts were a first version that stopped with an error while building the template and printed nothing, and a corrected version that computed the population recalls and was run under a timer before the plan was written; a third computed the test counts. Several choices were made at planning where the outline the example was written to left them open or could not be followed:
- FPR is reported for 4 illumination × camera cells, not 8, because a good part has no spot size.
- The planned exact detection probabilities have no closed form for a maximum over overlapping windows, so population recalls from 20,000 rendered images per cell stand in for them.
- The contrast was chosen by the rule above.
- The shift and the gain are applied separately and as either, with flip rates for good cells too, and the grouping by spot size uses the rendered size.
- The family-wise verdict and its needed counts were added beside the per-cell verdict, because the choice between them is open.
- The stronger form of H3 was added.
- The zero-failure size follows from Vision System Design’s formula with a 5% miss rate.
skimage.feature.match_templatewas not used as a cross-check, because it normalizes the correlation.- The requirement’s minimum illumination, 0.5, lies below the field mix’s dim level, 0.6.
- The detector subtracts the image median before correlating.
- The checks on the FPR target and on the noise level were added.
The tolerance of the check that good images are flat was set to rather than exact equality before any run, because camera B’s blur sums floating-point weights. The pilot run found one broken line: the zero-failure size divided by instead of and printed a test of 2 parts. The line was corrected, and a check comparing the formula with a direct search (the smallest with ) was added; nothing else changed, and the repeated pilot, fixed in advance as final, is the first run. The outcomes that changed between the two runs are the zero-failure test (2 parts with no rare-cell part in the pilot, 59 parts with one in the first run) and H4, which the pilot printed as “no images” for its 2-part test; the C6 line is new, and every other printed line was identical. One change was made after the first run, at the fact-check: two field-mix totals in the needed-count list were one too high because the division was rounded up in floating point (72 / 0.036 evaluates to slightly more than 2,000), so the quotient is now rounded to nine decimals before it is rounded up. This changed those two printed totals, from 501 to 500 and from 2,001 to 2,000, and no other line.
Variants
- Offline test suites on stored images, run on every model version: held-out field mix, slices, MFT sets and stress sets.
- Label-free tests on field data: invariance and directional tests run on unlabelled images in service, which flag fragile decisions without ground truth.
- Rendered stress and scenario tests, where the scene can be simulated: the requirement’s limits are rendered deliberately rather than waited for.
- Shadow and canary field tests, in which a new version runs on live inputs before its decisions are used (deploying vision models).
- Prospective and randomized studies in medicine, where the test is a clinical study (clinical validation of medical AI).
Best Practices
- Write criteria before results. A threshold, a slice list or a sample size chosen after seeing the numbers is a selection, not a test.
- Report the worst slice next to the average, with each slice’s count and interval, as WILDS reports the worst domain for subpopulation shift.
- Size the test for the smallest slice that has a criterion, not for the aggregate; collect rare conditions on purpose.
- State per-slice or family-wise for every rule that covers several slices, and correct for multiplicity when the claim is about all of them.
- Run invariance tests continuously: they need no labels and can point to the slice where labelled data should be collected.
- Test at the requirement’s limits with stress data, since a field-mix test may contain none.
- Keep validation independent: the NIST framework recommends that, ideally, verification and validation be done by people other than those who perform test and evaluation.
Pitfalls
- An aggregate that passes for the wrong reason. A slice that is a few percent of the data can fail completely without moving the average below the requirement.
- Intervals from the whole test set applied to a slice. A slice’s uncertainty is set by its own count; 111 images give an interval about 0.2 wide at a recall near 0.5.
- Many slices, one significance level. Eight slices each at 95% do not give 95% confidence that all eight pass.
- Perturbations that change the label. A mirror flip of an asymmetric sign or a gain outside the specified range is not an invariance.
- Untested thresholds. A threshold set from one sample of good parts is itself an estimate, and its false-positive rate on new data is a test case of its own.
- Reusing the test set during development, which turns it into a validation set (benchmark overfitting).
Evidence and Impact
The evidence for behavioural testing comes from NLP: Ribeiro et al. (2020) found failures in commercial and research models that held-out accuracy had not shown, and, in their user studies, more tests and more bugs found with the CheckList matrix than without it. For distribution shift, WILDS documents lower out-of-distribution than in-distribution performance on each of its ten datasets, and evaluates subpopulation shift by the worst domain (Koh et al., 2021); the camera-trap numbers that Vision System Design summarizes show the size of the gap for a change of site. The NIST AI RMF (2023) sets out the practices in general form: test sets representative of expected use, results broken down by segment, TEVV throughout the lifecycle, and, ideally, verification and validation by people other than those who perform test and evaluation. In our reading, the most established practice for vision is in regulated domains, where clinical validation studies define what counts as evidence; elsewhere the methods above are engineering practice adapted from software testing and statistics rather than the subject of controlled comparisons.
Related
- Deploying Vision Models
Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.
- Industrial Visual Inspection
Using cameras and vision algorithms to check manufactured parts on a production line, from lighting and optics to golden-template comparison, supervised defect models and anomaly detection trained on good parts only, judged by how many defects escape and how many good parts are rejected.
- Clinical Validation and Regulation of Medical Imaging AI
How the claims made for a medical image analysis model are tested, from internal, temporal and external validation through reader studies, prospective studies and randomized trials, with the reporting guidelines that describe such studies, the statistics of rare findings, the regulatory pathways in outline, and monitoring after deployment.
- Training, Validation, and Test Sets
Why labelled images are divided into a training set that fits the model, a validation set that selects it and a test set that reports it once, how large each must be, and how leakage between them, from near-duplicate images to frames of one video, inflates the reported score.
- Benchmark Overfitting
How reusing one test set for years of model selection can inflate benchmark scores, what the theory of adaptive data analysis says about it, and what replication studies of CIFAR-10, ImageNet, MNIST and Kaggle found when they measured it.
- Dataset Bias
How the way a dataset's images are chosen, photographed and labelled makes it an unrepresentative sample of the visual world, how cross-dataset tests measure the damage, and what bias means when the images show people, from demographic representation to withdrawn datasets.
References
- Ribeiro, M. T., Wu, T., Guestrin, C. & Singh, S. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 4902–4912.
- Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2021). WILDS: A Benchmark of in-the-Wild Distribution Shifts. Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139, 5637–5664.
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1.
- Oakden-Rayner, L., Dunnmon, J., Carneiro, G. & Ré, C. (2020). Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. Proceedings of the ACM Conference on Health, Inference, and Learning (CHIL), 151–159 (not read).