Vision Systems & Engineering

Model Monitoring and Data Drift

How a deployed vision system is watched for change once it is in service, from what can be observed without labels to sequential detectors (Shewhart, CUSUM and EWMA charts), windowed two-sample tests, label-shift estimation with a black-box classifier, accuracy estimates without labels, feedback loops and failure triage, with an inspection-camera example in which three drifts meet three detectors.

intermediate

Model monitoring is the work of watching a deployed system for the changes that make its validation out of date. A vision model is accepted on test data and in a field test (testing and validating vision systems); afterwards the camera ages, the lights dim, the lens drifts out of focus, the product mix changes and new software is released, and none of this announces itself. Monitoring turns such changes into alarms: it computes statistics of the images, of the model’s outputs and of whatever labels arrive, and decides, frame by frame or window by window, whether they still look like the reference period. Data drift is the everyday name for a change in the input distribution, and is used loosely for any change of the joint distribution of images and labels; the dataset bias article gives the precise vocabulary (covariate shift, prior probability shift, sample selection bias, domain shift), which this article uses where it matters. What to watch is listed in deploying vision models: inputs, predictions, system health and labelled samples. This article is about how to decide that something has changed, how quickly that can be decided at a given false-alarm rate, what can be estimated without labels, and what to do next.

Purpose

A model’s accuracy is a property of the model and the data it sees. When the data change, the accuracy changes, but the labels that would measure it arrive late, rarely, or only for the cases someone chose to review. Monitoring fills the gap with proxies that are available at once: brightness, sharpness and colour statistics of the frames, the distribution of scores and predicted classes, the rate of alarms or rejections. Its job is to detect a change early enough that the system can be stopped, recalibrated or retrained before the requirement it was accepted against is violated for long, while raising few enough false alarms that operators keep paying attention.

Sculley et al. (2015, §7) put the case for it: the external world is rarely stable, and unit and end-to-end tests are not enough evidence that a running system works as intended without comprehensive live monitoring. Their starting points, prediction bias (the distribution of predicted labels should usually equal that of observed labels), action limits and up-stream producers, which in a vision system include the camera, lens, lights and firmware, are summarized in designing a computer vision system.

When to Use It

Every deployed model needs some monitoring; what differs is how much can be observed.

  • Fixed cameras on a controlled scene (inspection lines, gauging stations, fixed traffic or ward cameras) have a stable reference period, so changes in image statistics are meaningful and control charts apply.
  • Uncontrolled scenes (phones, vehicles, web images) vary so much that a single reference distribution is broad, and small drifts hide inside it; monitoring then relies more on prediction statistics, slices and labelled samples.
  • Rare positive classes (defects, findings, intrusions) make label-based monitoring slow, because errors on the class that matters accumulate slowly; the post-market monitoring of medical devices meets the same arithmetic.
  • Systems whose decisions select their own future data (only rejected parts are inspected, only flagged images are reviewed) need monitoring of the selection itself, because their labelled samples are biased.

Monitoring does not replace validation, a parity test or a planned re-validation: it can show that the data have moved, but it cannot show by itself that the system is still correct.

How It Works

What can be observed, and when

Three layers of evidence arrive at different speeds. Input statistics (mean intensity, contrast, colour means, a focus measure such as the variance of the Laplacian, image size, decode failures) are available for every frame at once. Output statistics (score distributions, predicted-class frequencies, box counts, the reject rate) are also immediate, and they respond to whatever the model responds to. Outcome statistics (accuracy, recall and calibration against labels) need ground truth, which arrives late and in small numbers. A change can show in one layer and not in another: an illumination drop moves the inputs, a change in what arrives (more defective parts) moves the outputs but not necessarily the inputs, and a change in what a label means (the definition of a defect is tightened) moves neither until labels arrive. The example below shows the first two cases on one stream.

Kinds of drift, as a monitor sees them

Writing the joint distribution as P(x,y)=P(y∣x) P(x)=P(x∣y) P(y)P(x, y) = P(y \mid x)\,P(x) = P(x \mid y)\,P(y) separates the cases that matter for monitoring (the factorization and the names are those of dataset bias):

  • Appearance or covariate drift: P(x)P(x) changes, for example with lighting, focus, a new camera or a dirty lens. Input monitors see it. Whether the labelling rule P(y∣x)P(y \mid x) stays the same depends on whether the change destroys the evidence: a blurred scratch is still a scratch, but the model may no longer see it.
  • Label shift (prior probability shift): P(y)P(y) changes while P(x∣y)P(x \mid y) does not, for example when a supplier sends more defective parts. Lipton, Wang and Smola (2018) call it label shift and motivate it with diagnosis and recognition, where the label causes the observation. Input monitors may not see it at all, because each class still looks as it did; output monitors do, and it can be estimated (below). Class imbalance derives what a prior shift does to a classifier’s posterior, and domain adaptation shows how it harms alignment methods.
  • Concept drift: P(y∣x)P(y \mid x) changes, because what counts as a defect, a lesion or a person changes, or because the labelling team changes its practice (annotation quality). Only labels reveal it.
  • Pipeline and sensor drift: firmware that changes colour processing, a resize library upgraded, a lens refocused. These look like appearance drift, but the remedy is to fix the pipeline, not to retrain; deploying vision models shows a bug (a channel swap) that aggregate statistics cannot see.

Sequential detectors: control charts

A monitored quantity xtx_t, one value per frame or per batch of frames, is compared with its in-control mean μ0\mu_0 and standard deviation σ\sigma estimated on a reference period. Statistical process control, as the NIST/SEMATECH e-Handbook (§6.3) presents it, offers three classic charts.

A Shewhart chart alarms when a single point falls outside μ0±3σ\mu_0 \pm 3\sigma. Under normality the probability of that is 0.0027 per point, so the average run length between false alarms is 1/0.0027≈3701/0.0027 \approx 370 points; the Handbook tabulates it as 371 and gives an average run length of 44.0 points for a mean shift of 1σ1\sigma (§6.3.2.3.1).

The CUSUM chart accumulates evidence. In the tabular form, which the Handbook says most users prefer to the older V-mask (§6.3.2.3),

Shi(i)=max⁡(0,  Shi(i−1)+xi−μ^0−k),Slo(i)=max⁡(0,  Slo(i−1)+μ^0−k−xi),S_{hi}(i) = \max\bigl(0,\; S_{hi}(i-1) + x_i - \hat{\mu}_0 - k\bigr), \qquad S_{lo}(i) = \max\bigl(0,\; S_{lo}(i-1) + \hat{\mu}_0 - k - x_i\bigr),

with Shi(0)=Slo(0)=0S_{hi}(0) = S_{lo}(0) = 0; the process is declared out of control when either sum exceeds the decision limit hh. The reference value kk is the slack that keeps in-control noise from accumulating; the rule of thumb the Handbook quotes from Montgomery is to set kk to half the shift to be detected and hh to about 4 or 5, in units of σ\sigma. Its average-run-length table for k=0.5k = 0.5 gives, for one one-sided chart, 336 points with no shift at h=4h = 4 and 930 at h=5h = 5, against 8.38 and 10.4 for a shift of 1σ1\sigma. From those runs the Handbook concludes that the Shewhart chart is better for large shifts and CUSUM faster for small ones, and states in §6.3.2.3 that CUSUM charts are better than Shewhart charts for mean shifts of 2σ2\sigma or less. (The Handbook’s text beside the table says to add 0.5 to the first column to obtain the shift and places the run length 8.38 at the entry 0.5, while the table prints 8.38 in the row 1.00; this article reads the first column as the shift itself, which matches the Shewhart column’s 371 at 0.00.)

The EWMA chart smooths instead: EWMAt=λYt+(1−λ) EWMAt−1\mathrm{EWMA}_t = \lambda Y_t + (1 - \lambda)\,\mathrm{EWMA}_{t-1}, started at the historical mean, with λ\lambda usually between 0.2 and 0.3 and control limits at ±k sewma\pm k\,s_{\mathrm{ewma}} with sewma2≈λ2−λs2s_{\mathrm{ewma}}^2 \approx \frac{\lambda}{2 - \lambda} s^2 and k=3k = 3 or taken from published tables (§6.3.2.4). A small λ\lambda gives more weight to older data, and the choice of λ\lambda can make the chart sensitive to small or gradual drifts.

The average run length (ARL) is the design currency of all three. The in-control ARL is the false-alarm budget: on a camera running at 10 frames per second, an ARL of 930 frames is one false alarm in about a minute and a half per chart (our arithmetic), far too many for an operator, so in practice the chart runs on per-minute or per-batch means, or the limit is raised. The out-of-control ARL for the shift that matters is the detection delay. Both depend on the data being independent and close to normal; image statistics often are neither, so the false-alarm rate is measured on a stream with no change, as the example does.

Windowed two-sample tests

The other family compares a window of recent data with a reference sample. For one feature, the two-sample Kolmogorov–Smirnov (KS) test uses the largest gap between the two empirical distribution functions, D=sup⁡x∣Fref(x)−Fwin(x)∣D = \sup_x |F_{\mathrm{ref}}(x) - F_{\mathrm{win}}(x)|, and detects any change of distribution, not only of the mean. In SciPy 1.16.3, scipy.stats.ks_2samp is two-sided by default and computes the exact p-value when neither sample exceeds 10,000 values (method 'auto'), the asymptotic one otherwise; its documentation assumes samples from continuous distributions, so features are kept as floats rather than quantized to 8 bits, which would create ties. For many features at once, the maximum mean discrepancy (MMD) of Gretton et al. (2012) is the largest difference in expectations over functions in the unit ball of a reproducing kernel Hilbert space; it gives a multivariate test that can be computed in quadratic time, with linear-time approximations.

For images, the high dimension is first reduced. Rabanser, Günnemann and Lipton (2019) compared reductions and tests on simulated shifts; they tested each reduced dimension with KS and combined the results with a Bonferroni correction, rejecting when the smallest p-value is below the significance level divided by the number of tests, and found, despite the heavy correction, performance comparable to multivariate kernel tests. Their sample-size results for black-box shift detection on a classifier’s softmax outputs are summarized in deploying vision models. They name shift detection on online data, with its correlation between adjacent time steps, as an open question. That is the situation of a deployed camera: a window slid one frame at a time is tested thousands of times against one reference sample, the tests are strongly dependent, and the stated significance level says little about how often such a monitor alarms on unchanged data (our reasoning). The rate that matters is alarms per 1,000 frames on a stream with no change, which has to be measured; the multiple-comparison corrections in statistical significance in evaluation apply across features, not across time.

Estimating label shift without labels

Under label shift, a fixed classifier’s confusion behaviour is unchanged, which lets the new class frequencies be recovered from its predictions alone. Lipton, Wang and Smola (2018) note that the prediction y^=f(x)\hat{y} = f(x) depends on yy only through xx, so q(y^∣y)=p(y^∣y)q(\hat{y} \mid y) = p(\hat{y} \mid y) when p(x∣y)=q(x∣y)p(x \mid y) = q(x \mid y), and

q(y^)=∑yp(y^∣y) q(y)=∑yp(y^,y) q(y)p(y).q(\hat{y}) = \sum_{y} p(\hat{y} \mid y)\, q(y) = \sum_{y} p(\hat{y}, y)\, \frac{q(y)}{p(y)}.

With the joint confusion matrix C^y^,y\hat{C}_{\hat{y},y} estimated on labelled reference data, the predicted class frequencies μ^y^\hat{\mu}_{\hat{y}} in a window of unlabelled production data and the reference class frequencies ν^y\hat{\nu}_y, their black box shift estimation (BBSE) is

w^=C^y^,y−1 μ^y^,μ^y=diag⁡(ν^y) w^.\hat{w} = \hat{C}_{\hat{y},y}^{-1}\, \hat{\mu}_{\hat{y}}, \qquad \hat{\mu}_y = \operatorname{diag}(\hat{\nu}_y)\, \hat{w}.

Of the classifier it needs only that the expected confusion matrix be invertible; the classifier may be biased, inaccurate or uncalibrated, and better classifiers give tighter estimates (their Theorem 3). For two classes it reduces to p^=(r−FPR)/(TPR−FPR)\hat{p} = (r - \mathrm{FPR}) / (\mathrm{TPR} - \mathrm{FPR}), with rr the predicted-positive rate (our algebra). They also build a test for label shift on the same quantities, and use the estimated weights to correct the classifier. The assumption is the catch: under appearance drift p(x∣y)p(x \mid y) changes, the reference confusion matrix no longer applies, and the estimate is wrong, as the example shows.

The same formula explains why the raw predicted-positive rate is a biased monitor of prevalence. Its expectation is p TPR+(1−p) FPRp\,\mathrm{TPR} + (1 - p)\,\mathrm{FPR}, so a rise of the true prevalence by Δp\Delta p raises it by only (TPR−FPR) Δp(\mathrm{TPR} - \mathrm{FPR})\,\Delta p (our derivation): in expectation, an imperfect classifier under-reports a rise in prevalence.

Estimating accuracy without labels

Several methods try to predict accuracy on new data from the model’s own outputs. Guillory et al. (2021) studied this for image classifiers: the average confidence on the new distribution overestimates accuracy for poorly calibrated models, whereas the difference of confidences (DoC), the average confidence on the base distribution minus that on the new one, gave useful estimates of the accuracy change across synthetic, natural and adversarial shifts, while common distributional distances such as the Fréchet distance and MMD did not give reliable estimates. The authors fit regression models on a set of shifts and apply them to others, and their limits matter for monitoring: on 2 of their 8 synthetic shifts, defocus blur and Gaussian blur, DoC made the predictions worse; ImageNet-A, collected adversarially, gave the largest errors of the shifts studied, for every approach; and they write that the problem is far from solved. An accuracy estimate of this kind is a trigger for labelling, not a measurement. Expected calibration error and uncertainty estimation cover the confidence scores themselves.

Feedback loops and selective labels

A deployed model can shape the data it later learns from or is judged on. Sculley et al. (2015, §4) distinguish direct feedback loops, in which a model influences the selection of its own future training data, from hidden feedback loops, in which two systems influence each other through the world; they suggest some randomization, or isolating parts of the data from the model’s influence, as mitigations for direct loops. Ensign et al. (2018) analyse the mechanism in a setting outside vision: when a system only receives feedback where its own decisions sent attention, the sampling bias can compound into a runaway loop, which they model with urn processes and avoid by changing the system’s inputs. The vision analogue is the inspection station that only sends rejected parts to a human reviewer: the reviewed sample contains the false positives and no misses, so precision can be measured from it and recall cannot (our reasoning). A small random audit of accepted parts, or of unflagged images, is the randomization that breaks the loop.

Triage

An alarm says that something changed, not what. In our reading the first questions are about the physical and software layers, because they are cheapest to check and to fix: did the camera, lens, light or firmware change (input statistics move, and a reference target imaged at intervals, which designing a computer vision system suggests, moves with them); did the code or a library change (a parity test against stored golden outputs answers it); did the input population change (slices of the statistics by line, camera, shift or product localize it, as Sculley et al. recommend slicing prediction bias); or did the labels change. Then the decision follows the cause: repair the pipeline or the hardware, recalibrate a threshold, estimate and correct a prior shift, or collect and label new data and retrain, followed by re-validation against the original requirement.

How to Apply It

In outline:

  1. Fix a reference period under accepted conditions, with labels, and compute every monitored statistic on it.
  2. Choose monitors for each failure the requirements care about: an input statistic for each physical cause (brightness for lighting, a focus measure for defocus), output statistics for population changes, and a labelled audit sample, drawn at random, for what only labels reveal.
  3. Choose a detector per monitor and set its limits for a false-alarm budget, then measure the actual alarm rate on unchanged data.
  4. Simulate or replay each anticipated drift and measure the detection delay against the time at which the requirement is violated.
  5. Write the triage and response for each alarm, and re-validate after every repair or retraining.

Example: three drifts on an inspection camera

The script renders a fixed inspection camera looking at metal plates. Each 32×3232 \times 32 frame shows a bright square plate (reflectance 0.75, 24 pixels across, placed with a random offset of up to one pixel) on a dark background (0.15); a defective plate carries a faint horizontal scratch six pixels long with a Gaussian cross-profile (standard deviation 0.6 pixels) that removes 6% of the plate’s reflectance at its centre. The illumination varies by 10% from frame to frame (a standard deviation of 0.1 around its mean), defocus is a Gaussian blur of σb\sigma_b pixels, and sensor noise of standard deviation 0.02 is added after the optics. Pixel jj covers [j,j+1)[j, j+1); coverage of the plate, and of the scratch along its length, is computed exactly, and the cross-profile is sampled at pixel centres. The classifier is fixed and untrained: it subtracts the median of the plate’s interior and correlates the result with a unit-norm line template (template matching), and calls a frame defective when the best match exceeds a threshold set at the 99.5th percentile of good frames in a labelled reference set of 4,000 frames with 5% defects. The written requirement is a recall of at least 0.80 on defective frames.

The monitored stream has three segments, each starting after maintenance with every detector restarted, and each with its event at local frame 500:

  • A, lamp ageing: the illumination drops abruptly by 15% (1,000 frames).
  • B, defocus: σb\sigma_b rises linearly from 0 at frame 500 to about 2 pixels at the end (1,500 frames).
  • C, label shift: the defect rate rises from 5% to 20% while each class’s images are generated exactly as before (1,500 frames).

Four quantities are monitored on every frame: the mean intensity; the sharpness, the variance of scipy.ndimage.laplace over the frame; a normalized sharpness, the sharpness divided by the squared mean intensity; and the classifier’s score. Each is watched by two detectors. The CUSUM is the Handbook’s tabular form on features standardized by the reference mean and standard deviation, with k=0.5k = 0.5 and h=5h = 5, both one-sided sums, and both reset after an alarm. The KS monitor compares the last 100 frames with a fixed reference window of 100 reference frames at every frame, alarms when p<αp < \alpha, and waits for the window to refill after an alarm. The level α\alpha is chosen on a separate calibration stream of 10,000 unchanged frames, as the value on a fixed grid (10−1.510^{-1.5} down to 10−610^{-6} in quarter decades) whose alarm rate on mean intensity comes closest to the CUSUM’s. A control stream of 10,000 unchanged frames measures every monitor’s false alarms. BBSE runs on non-overlapping 500-frame windows with the reference set’s confusion matrix. Because a 500-frame window holds only a few dozen defective frames, the moment at which the requirement is violated during the defocus is taken from a population curve instead: recall on 2,000 defective frames at each blur from 0 to 2 pixels, interpolated to the blur at which it crosses 0.80.

The hypotheses were written before any script ran, apart from the changes listed at the end of the example: H1, on the illumination drop the CUSUM on mean intensity alarms sooner than the KS monitor on it; H2, on the defocus both sharpness monitors, with both detectors, alarm before recall falls below 0.80; H3a, during the label shift no input monitor alarms more often than a Poisson 99% bound set by its own control-stream rate; H3b, BBSE’s estimate in each post-shift window lies within τ=0.04\tau = 0.04 of the window’s true defect rate; H3c, the predicted-positive rate rises by less than the true defect rate. A precondition, C-match, asks that the KS and CUSUM control-stream alarm rates on mean intensity agree within a factor of two, so that H1 compares the two at matched false-alarm rates. Checks C1–C7 test the renderer, the matched filter against scipy.signal.correlate2d, the CUSUM against the Handbook’s worked example and its run-length table, BBSE against its two-class closed form, the absence of ties in the KS reference window, and the stacked Laplacian against a per-frame one. The script uses NumPy 2.3.5 and SciPy 1.16.3 with one BLAS thread, takes about 8 seconds, and printed identical output in two runs.

import os
os.environ["OMP_NUM_THREADS"] = "1"          # one BLAS/OpenMP thread, set before NumPy loads
os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from scipy import ndimage, signal, stats
from numpy.lib.stride_tricks import sliding_window_view

# ---- renderer: a square plate under a fixed camera, with an optional dark scratch ----
N, IDX = 32, np.arange(32)
BG, PLATE, NOISE, JIT, C = 0.15, 0.75, 0.02, 0.10, 0.06     # C: scratch contrast
g = np.exp(-np.array([-1.0, 0.0, 1.0]) ** 2 / (2 * 0.6 ** 2))
TEMPLATE = np.outer(g, np.ones(6))
TEMPLATE /= np.linalg.norm(TEMPLATE)                         # unit-norm matched filter


def cover(a, b):
    """Length of [a, b) inside each pixel [j, j+1) for a, b of shape (n,)."""
    return np.clip(np.minimum(b[:, None], IDX + 1) - np.maximum(a[:, None], IDX), 0, None)


def render(rng, defect, L0=1.0, blur=None, noise=True, jitter=True):
    n = defect.size
    dr, dc = rng.uniform(-1, 1, n), rng.uniform(-1, 1, n)
    plate = cover(4 + dr, 28 + dr)[:, :, None] * cover(4 + dc, 28 + dc)[:, None, :]
    r0, c0 = rng.uniform(11, 21, n), rng.uniform(10, 16, n)
    prof = np.exp(-((IDX + 0.5)[None, :] - r0[:, None]) ** 2 / (2 * 0.6 ** 2))
    scratch = prof[:, :, None] * cover(c0, c0 + 6)[:, None, :]
    refl = BG + (PLATE - BG) * plate - C * PLATE * scratch * defect[:, None, None]
    z = rng.standard_normal(n)
    L = L0 * (1 + JIT * z) if jitter else np.full(n, L0)
    img = L[:, None, None] * refl                            # radiance
    if blur is not None:                                     # defocus, sigma in pixels
        blur = np.broadcast_to(blur, (n,))
        for i in range(n):
            if blur[i] > 0:
                img[i] = ndimage.gaussian_filter(img[i], blur[i], mode="nearest")
    if noise:
        img = img + NOISE * rng.standard_normal(img.shape)   # sensor noise after the optics
    return img


def scores(img):
    inner = img[:, 9:23, 9:23]
    R = np.median(inner, axis=(1, 2))[:, None, None] - inner   # dark residual
    win = sliding_window_view(R, (3, 6), axis=(1, 2))
    return np.einsum("nijab,ab->nij", win, TEMPLATE).max(axis=(1, 2))


NAMES = ["intensity", "sharpness", "norm. sharpness", "score"]
INPUTS = [0, 1, 2]


def features(img):
    m = img.mean(axis=(1, 2))
    v = ndimage.laplace(img, axes=(1, 2)).var(axis=(1, 2))   # variance of the Laplacian
    return np.stack([m, v, v / m ** 2, scores(img)], axis=1)


# ---- detectors ----
def cusum(z, k=0.5, h=5.0, reset=True):
    """Tabular CUSUM (NIST/SEMATECH e-Handbook 6.3.2.3); returns alarm indices and S_hi."""
    s_hi = s_lo = 0.0
    alarms, trace = [], []
    for i, x in enumerate(z):
        s_hi = max(0.0, s_hi + x - k)
        s_lo = max(0.0, s_lo - x - k)
        trace.append(s_hi)
        if s_hi > h or s_lo > h:
            alarms.append(i)
            if reset:
                s_hi = s_lo = 0.0
    return alarms, trace


W = 100


def ks_pvalues(ref, x):
    p = np.full(x.size, np.nan)
    for t in range(W - 1, x.size):
        p[t] = stats.ks_2samp(ref, x[t - W + 1:t + 1]).pvalue
    return p


def ks_alarms(p, alpha):
    alarms, next_test = [], W - 1
    for t in range(W - 1, p.size):
        if t >= next_test and p[t] < alpha:
            alarms.append(t)
            next_test = t + W                                # the window refills
    return alarms


ss = np.random.SeedSequence(0).spawn(7)
rng_ref, rng_str, rng_ctl, rng_cal, rng_pop, rng_arl, rng_chk = [np.random.default_rng(s) for s in ss]

# ---- reference set: threshold, standardization, confusion matrix, KS reference window ----
y_ref = rng_ref.random(4000) < 0.05
F_ref = features(render(rng_ref, y_ref))
T = np.quantile(F_ref[~y_ref, 3], 0.995, method="higher")
pred_ref = F_ref[:, 3] > T
mu, sd = F_ref.mean(0), F_ref.std(0, ddof=1)
ref_win = F_ref[:W]
Cmat = np.array([[np.mean((pred_ref == i) & (y_ref == j)) for j in (0, 1)] for i in (0, 1)])
nu = np.array([np.mean(~y_ref), np.mean(y_ref)])
TPR_ref, FPR_ref = pred_ref[y_ref].mean(), pred_ref[~y_ref].mean()
print(f"reference set: 4000 frames, {y_ref.sum()} defective; threshold T = {T:.5f}; "
      f"recall {TPR_ref:.4f}, FPR {FPR_ref:.4f}")

# ---- the monitored stream: three segments, each restarted after maintenance ----
SEG = 500                     # onset at local frame 500 in every segment
yA = rng_str.random(1000) < 0.05
FA = np.concatenate([features(render(rng_str, yA[:SEG], L0=1.0)),
                     features(render(rng_str, yA[SEG:], L0=0.85))])
yB = rng_str.random(1500) < 0.05
blurB = np.where(np.arange(1500) < SEG, 0.0, 2.0 * (np.arange(1500) - SEG) / 1000)
FB = features(render(rng_str, yB, blur=blurB))
yC = np.concatenate([rng_str.random(SEG) < 0.05, rng_str.random(1000) < 0.20])
FC = features(render(rng_str, yC))
segments = [("A illumination -15%", FA, yA), ("B defocus ramp", FB, yB), ("C label shift 5%->20%", FC, yC)]

# ---- control and calibration streams (no change) ----
y_ctl = rng_ctl.random(10000) < 0.05
F_ctl = features(render(rng_ctl, y_ctl))
y_cal = rng_cal.random(10000) < 0.05
F_cal = features(render(rng_cal, y_cal))

# ---- choose the KS level: match CUSUM's calibration-stream rate on intensity ----
cal_cusum = len(cusum((F_cal[:, 0] - mu[0]) / sd[0])[0]) / 10.0
p_cal = ks_pvalues(ref_win[:, 0], F_cal[:, 0])
grid = [10 ** (-1.5 - 0.25 * j) for j in range(19)]
cal_ks = {a: len(ks_alarms(p_cal, a)) / 10.0 for a in grid}
alpha = min(grid, key=lambda a: (abs(cal_ks[a] - cal_cusum), -a))
print(f"calibration stream (10000 frames), intensity: CUSUM {cal_cusum:.2f} alarms/1000; "
      f"KS alpha chosen {alpha:.2e} -> {cal_ks[alpha]:.2f} alarms/1000")


def run_detectors(F, start=0):
    """Alarm frames per (feature, detector) on one segment or stream."""
    out = {}
    for f in range(4):
        out[(f, "CUSUM")] = cusum((F[:, f] - mu[f]) / sd[f])[0]
        out[(f, "KS")] = ks_alarms(ks_pvalues(ref_win[:, f], F[:, f]), alpha)
    return out


# ---- control stream: false alarms ----
ctl = run_detectors(F_ctl)
print("\ncontrol stream, alarms per 1000 frames (count over 10000):")
rate = {}
for d in ("CUSUM", "KS"):
    row = []
    for f in range(4):
        rate[(f, d)] = len(ctl[(f, d)]) / 10.0
        row.append(f"{NAMES[f]} {rate[(f, d)]:.2f} ({len(ctl[(f, d)])})")
    anyin = len(set().union(*[ctl[(f, d)] for f in INPUTS])) / 10.0
    print(f"  {d:5s}: " + ", ".join(row) + f"; any input monitor {anyin:.2f}")
ratio = rate[(0, "KS")] / rate[(0, "CUSUM")]
print(f"C-match  KS/CUSUM control rate on intensity = {ratio:.2f} (in [0.5, 2]: {0.5 <= ratio <= 2})")

# ---- population recall (the requirement is recall >= 0.80) ----
sig = np.round(np.arange(0, 2.01, 0.1), 10)
rb = np.array([np.mean(scores(render(rng_pop, np.ones(2000, bool), blur=s)) > T) for s in sig])
r85 = np.mean(scores(render(rng_pop, np.ones(2000, bool), L0=0.85)) > T)
fpr_pop = np.mean(scores(render(rng_pop, np.zeros(10000, bool))) > T)
print("\npopulation recall against blur sigma (2000 defective frames each):")
print("  " + " ".join(f"{s:.1f}:{r:.3f}" for s, r in zip(sig, rb)))
print(f"population recall at illumination 0.85: {r85:.4f}; population FPR {fpr_pop:.4f}")
j = np.nonzero(rb < 0.80)[0][0]
s_star = sig[j - 1] + (rb[j - 1] - 0.80) / (rb[j - 1] - rb[j]) * (sig[j] - sig[j - 1])
t_req = SEG + 1000 * s_star / 2.0
print(f"recall falls below 0.80 at blur {s_star:.3f} px: local frame t_req = {t_req:.1f} "
      f"({t_req - SEG:.1f} frames after onset)")

# ---- detection delays per event ----
print("\ndetection delay (frames after onset at local frame 500; 'none' = no alarm in the segment)")
print("and alarms before onset, per feature and detector:")
first = {}
for name, F, y in segments:
    al = run_detectors(F)
    print(f"  {name}")
    for d in ("CUSUM", "KS"):
        cells = []
        for f in range(4):
            post = [a for a in al[(f, d)] if a >= SEG]
            pre = len([a for a in al[(f, d)] if a < SEG])
            first[(name[0], f, d)] = post[0] - SEG if post else None
            first[(name[0], f, d, "n_post")] = len(post)
            cells.append(f"{NAMES[f]} {post[0] - SEG if post else 'none'} (pre {pre})")
        print(f"    {d:5s}: " + ", ".join(cells))

# ---- per-window truth, predicted-positive rate and BBSE ----
print("\n500-frame windows: true prevalence, predicted-positive rate, BBSE estimate, accuracy, recall")
bbse_rows, closed_diff = [], []
for name, F, y in segments:
    pred = F[:, 3] > T
    for w in range(len(y) // 500):
        s = slice(500 * w, 500 * (w + 1))
        mu_hat = np.array([np.mean(~pred[s]), np.mean(pred[s])])
        q = nu * np.linalg.solve(Cmat, mu_hat)               # BBSE: w = C^-1 mu, q = nu * w
        closed = (mu_hat[1] - FPR_ref) / (TPR_ref - FPR_ref)
        closed_diff.append(abs(q[1] - closed))
        k, n = int(np.sum(pred[s] & y[s])), int(np.sum(y[s]))
        acc = np.mean(pred[s] == y[s])
        bbse_rows.append((name[0], w, y[s].mean(), mu_hat[1], q[1]))
        print(f"  {name[0]}{w}: prevalence {y[s].mean():.4f}, predicted {mu_hat[1]:.4f}, BBSE {q[1]:.4f}, "
              f"accuracy {acc:.4f}, recall {k}/{n} = {k / n:.4f}")

# ---- hypotheses ----
print("\nhypotheses")
dc, dk = first[("A", 0, "CUSUM")], first[("A", 0, "KS")]
print(f"H1  illumination drop, intensity: CUSUM delay {dc}, KS delay {dk} -> "
      f"{'held' if dc is not None and (dk is None or dc < dk) else 'failed'}")
for f in (1, 2):
    for d in ("CUSUM", "KS"):
        dl = first[("B", f, d)]
        ok = dl is not None and SEG + dl < t_req
        print(f"H2  defocus, {NAMES[f]} {d}: alarm at local frame {SEG + dl if dl is not None else 'none'}, "
              f"t_req {t_req:.1f} -> {'held' if ok else 'failed'}")
for f in INPUTS:
    for d in ("CUSUM", "KS"):
        nmax = stats.poisson.ppf(0.99, rate[(f, d)])
        npost = first[("C", f, d, "n_post")]
        print(f"H3a label shift, {NAMES[f]} {d}: {npost} alarms in 1000 post-onset frames, "
              f"bound {nmax:.0f} -> {'held' if npost <= nmax else 'failed'}")
TAU = 0.04
for (seg, w, prev, r, q) in bbse_rows:
    if seg == "C" and w >= 1:
        print(f"H3b window C{w}: BBSE {q:.4f}, true {prev:.4f}, |error| {abs(q - prev):.4f}, tau {TAU:.2f} -> "
              f"{'held' if abs(q - prev) <= TAU else 'failed'}")
predC = FC[:, 3] > T
rpre, ppre = predC[:SEG].mean(), yC[:SEG].mean()
rpost, ppost = predC[SEG:].mean(), yC[SEG:].mean()
print(f"H3c predicted-positive rate {rpre:.4f} -> {rpost:.4f} (rise {rpost - rpre:+.4f}); "
      f"true prevalence {ppre:.4f} -> {ppost:.4f} (rise {ppost - ppre:+.4f}) -> "
      f"{'held' if rpost - rpre < ppost - ppre else 'failed'}")

# ---- checks ----
print("\nchecks")
# C1 data
seed = rng_chk.integers(2 ** 32)
d1 = rng_chk.random(200) < 0.5
clean = render(np.random.default_rng(seed), d1, noise=False, jitter=False)
noisy = render(np.random.default_rng(seed), d1, noise=True, jitter=False)
dev = np.abs(clean[~d1, 9:23, 9:23] - PLATE).max()
nodark = int(np.sum(clean[d1, 9:23, 9:23].min(axis=(1, 2)) >= PLATE))
nstd = (noisy - clean).std()
print(f"C1  noise-free good interior max |I - 0.75| = {dev:.1e} (< 1e-12: {dev < 1e-12}); "
      f"defective frames without a darker interior pixel: {nodark} (must be 0); "
      f"noise std {nstd:.5f} (within 2% of 0.02: {abs(nstd - 0.02) <= 0.0004})")
bchk = [blurB[t] - 2.0 * (t - SEG) / 1000 for t in (500, 1000, 1499)]
print(f"C1  defocus sigma at local frames 500, 1000, 1499: {blurB[500]:.3f}, {blurB[1000]:.3f}, "
      f"{blurB[1499]:.3f}; max deviation from formula {max(map(abs, bchk)):.1e}")
for name, F, y in segments:
    print(f"C1  {name[0]} labels: before onset {y[:SEG].sum()}/{SEG} = {y[:SEG].mean():.4f}, "
          f"after {y[SEG:].sum()}/{y.size - SEG} = {y[SEG:].mean():.4f}")
# C2 matched filter against SciPy
imgs = render(rng_chk, rng_chk.random(100) < 0.5)
mine = scores(imgs)
ref2 = []
for im in imgs:
    inner = im[9:23, 9:23]
    ref2.append(signal.correlate2d(np.median(inner) - inner, TEMPLATE, mode="valid").max())
d2 = np.abs(mine - np.array(ref2)).max()
print(f"C2  matched filter vs correlate2d: max |diff| = {d2:.1e} (< 1e-9: {d2 < 1e-9})")
# C3 the e-Handbook's worked example (no reset)
hb = [324.925, 324.675, 324.725, 324.350, 325.350, 325.225, 324.125, 324.525, 325.225, 324.600,
      324.625, 325.150, 328.325, 327.250, 327.825, 328.500, 326.675, 327.775, 326.875, 328.350]
al3, tr3 = cusum(np.array(hb) - 325, k=0.3175, h=4.1959, reset=False)
print(f"C3  Handbook example: first signal at point {al3[0] + 1} (Handbook 14), S_hi(20) = {tr3[-1]:.4f} "
      f"(Handbook 19.04; within 0.01: {abs(tr3[-1] - 19.04) <= 0.01})")
# C4 in-control ARL of one one-sided chart on N(0, 1)
for h, book in ((4.0, 336), (5.0, 930)):
    s, rl, alive, t = np.zeros(2000), np.zeros(2000), np.ones(2000, bool), 0
    while alive.any():
        t += 1
        s = np.where(alive, np.maximum(0, s + rng_arl.standard_normal(2000) - 0.5), s)
        hit = alive & (s > h)
        rl[hit] = t
        alive &= ~hit
    print(f"C4  one-sided ARL0, k = 0.5, h = {h:.0f}: {rl.mean():.1f} (Handbook {book}; "
          f"within 10%: {abs(rl.mean() - book) <= 0.1 * book})")
# C5 BBSE matrix form against the binary closed form
print(f"C5  BBSE matrix vs closed form: max |diff| = {max(closed_diff):.1e} (< 1e-12: "
      f"{max(closed_diff) < 1e-12}); det(C) = {np.linalg.det(Cmat):.5f}")
# C6 ties in the KS reference window
ties = [ref_win[:, f].size - np.unique(ref_win[:, f]).size for f in range(4)]
print(f"C6  tied values in the KS reference window per feature: {ties} (must be all 0)")
# C7 stacked Laplacian against per-frame
lap_s = ndimage.laplace(imgs, axes=(1, 2))
d7 = max(np.abs(lap_s[i] - ndimage.laplace(imgs[i])).max() for i in range(100))
print(f"C7  stacked vs per-frame Laplacian: max |diff| = {d7:.1e} (< 1e-12: {d7 < 1e-12})")

Output:

reference set: 4000 frames, 212 defective; threshold T = 0.07673; recall 0.8774, FPR 0.0048
calibration stream (10000 frames), intensity: CUSUM 1.70 alarms/1000; KS alpha chosen 3.16e-02 -> 0.10 alarms/1000

control stream, alarms per 1000 frames (count over 10000):
  CUSUM: intensity 1.70 (17), sharpness 2.80 (28), norm. sharpness 2.20 (22), score 6.70 (67); any input monitor 6.30
  KS   : intensity 0.20 (2), sharpness 0.20 (2), norm. sharpness 0.00 (0), score 0.20 (2); any input monitor 0.40
C-match  KS/CUSUM control rate on intensity = 0.12 (in [0.5, 2]: False)

population recall against blur sigma (2000 defective frames each):
  0.0:0.877 0.1:0.865 0.2:0.889 0.3:0.864 0.4:0.812 0.5:0.716 0.6:0.609 0.7:0.512 0.8:0.415 0.9:0.367 1.0:0.271 1.1:0.223 1.2:0.173 1.3:0.141 1.4:0.106 1.5:0.087 1.6:0.058 1.7:0.053 1.8:0.043 1.9:0.040 2.0:0.019
population recall at illumination 0.85: 0.6910; population FPR 0.0058
recall falls below 0.80 at blur 0.413 px: local frame t_req = 706.4 (206.4 frames after onset)

detection delay (frames after onset at local frame 500; 'none' = no alarm in the segment)
and alarms before onset, per feature and detector:
  A illumination -15%
    CUSUM: intensity 4 (pre 3), sharpness 7 (pre 4), norm. sharpness 24 (pre 0), score none (pre 1)
    KS   : intensity 27 (pre 0), sharpness 48 (pre 0), norm. sharpness 381 (pre 0), score none (pre 0)
  B defocus ramp
    CUSUM: intensity 55 (pre 1), sharpness 186 (pre 2), norm. sharpness 165 (pre 1), score 251 (pre 7)
    KS   : intensity none (pre 0), sharpness 198 (pre 3), norm. sharpness 195 (pre 1), score none (pre 0)
  C label shift 5%->20%
    CUSUM: intensity 219 (pre 1), sharpness 676 (pre 3), norm. sharpness 170 (pre 3), score 41 (pre 2)
    KS   : intensity none (pre 0), sharpness none (pre 0), norm. sharpness 610 (pre 1), score 117 (pre 0)

500-frame windows: true prevalence, predicted-positive rate, BBSE estimate, accuracy, recall
  A0: prevalence 0.0480, predicted 0.0440, BBSE 0.0450, accuracy 0.9840, recall 19/24 = 0.7917
  A1: prevalence 0.0400, predicted 0.0260, BBSE 0.0244, accuracy 0.9740, recall 10/20 = 0.5000
  B0: prevalence 0.0640, predicted 0.0640, BBSE 0.0679, accuracy 0.9880, recall 29/32 = 0.9062
  B1: prevalence 0.0400, predicted 0.0340, BBSE 0.0335, accuracy 0.9900, recall 16/20 = 0.8000
  B2: prevalence 0.0540, predicted 0.0120, BBSE 0.0083, accuracy 0.9500, recall 4/27 = 0.1481
  C0: prevalence 0.0500, predicted 0.0540, BBSE 0.0564, accuracy 0.9880, recall 23/25 = 0.9200
  C1: prevalence 0.1920, predicted 0.1780, BBSE 0.1985, accuracy 0.9780, recall 87/96 = 0.9062
  C2: prevalence 0.1900, predicted 0.1740, BBSE 0.1940, accuracy 0.9640, recall 82/95 = 0.8632

hypotheses
H1  illumination drop, intensity: CUSUM delay 4, KS delay 27 -> held
H2  defocus, sharpness CUSUM: alarm at local frame 686, t_req 706.4 -> held
H2  defocus, sharpness KS: alarm at local frame 698, t_req 706.4 -> held
H2  defocus, norm. sharpness CUSUM: alarm at local frame 665, t_req 706.4 -> held
H2  defocus, norm. sharpness KS: alarm at local frame 695, t_req 706.4 -> held
H3a label shift, intensity CUSUM: 2 alarms in 1000 post-onset frames, bound 5 -> held
H3a label shift, intensity KS: 0 alarms in 1000 post-onset frames, bound 2 -> held
H3a label shift, sharpness CUSUM: 1 alarms in 1000 post-onset frames, bound 7 -> held
H3a label shift, sharpness KS: 0 alarms in 1000 post-onset frames, bound 2 -> held
H3a label shift, norm. sharpness CUSUM: 5 alarms in 1000 post-onset frames, bound 6 -> held
H3a label shift, norm. sharpness KS: 1 alarms in 1000 post-onset frames, bound 0 -> failed
H3b window C1: BBSE 0.1985, true 0.1920, |error| 0.0065, tau 0.04 -> held
H3b window C2: BBSE 0.1940, true 0.1900, |error| 0.0040, tau 0.04 -> held
H3c predicted-positive rate 0.0540 -> 0.1760 (rise +0.1220); true prevalence 0.0500 -> 0.1910 (rise +0.1410) -> held

checks
C1  noise-free good interior max |I - 0.75| = 0.0e+00 (< 1e-12: True); defective frames without a darker interior pixel: 0 (must be 0); noise std 0.01997 (within 2% of 0.02: True)
C1  defocus sigma at local frames 500, 1000, 1499: 0.000, 1.000, 1.998; max deviation from formula 0.0e+00
C1  A labels: before onset 24/500 = 0.0480, after 20/500 = 0.0400
C1  B labels: before onset 32/500 = 0.0640, after 47/1000 = 0.0470
C1  C labels: before onset 25/500 = 0.0500, after 191/1000 = 0.1910
C2  matched filter vs correlate2d: max |diff| = 2.8e-17 (< 1e-9: True)
C3  Handbook example: first signal at point 14 (Handbook 14), S_hi(20) = 19.0350 (Handbook 19.04; within 0.01: True)
C4  one-sided ARL0, k = 0.5, h = 4: 325.9 (Handbook 336; within 10%: True)
C4  one-sided ARL0, k = 0.5, h = 5: 880.7 (Handbook 930; within 10%: True)
C5  BBSE matrix vs closed form: max |diff| = 2.8e-17 (< 1e-12: True); det(C) = 0.04380
C6  tied values in the KS reference window per feature: [0, 0, 0, 0] (must be all 0)
C7  stacked vs per-frame Laplacian: max |diff| = 0.0e+00 (< 1e-12: True)

False alarms first. The KS level chosen on the calibration stream was α=3.16×10−2\alpha = 3.16 \times 10^{-2}, which gave 0.10 alarms per 1,000 calibration frames against the CUSUM’s 1.70. On the control stream the CUSUM monitors raised between 1.70 and 6.70 alarms per 1,000 frames, the classifier score most often. For comparison, C4’s simulation of a single one-sided chart on normal data printed run lengths of 325.9 and 880.7 frames against the Handbook’s 336 and 930; the stream’s features are not normal, and the score in particular comes from a mixture of good and defective frames (our reading of the construction), so their actual rates had to be measured rather than read from the table. On the control stream the KS monitors raised between 0.00 and 0.20 alarms per 1,000 frames.

C-match failed: the ratio of the KS to the CUSUM control rate on mean intensity printed 0.12, outside the planned range of 0.5 to 2.

The illumination drop. H1 held: the CUSUM on mean intensity alarmed 4 frames after the drop and the KS monitor 27 frames after it. Because C-match failed, this comparison is not at matched false-alarm rates.

The raw sharpness also alarmed on the lighting change (after 7 and 48 frames). That follows from the construction: the Laplacian is linear in the image, so the variance of its noise-free part scales with the square of the illumination, and a dimmer lamp looks like a softer focus to this measure (our derivation). The normalized sharpness removes that factor from the noise-free part and still alarmed, later (after 24 and 381 frames). The classifier’s score raised no alarm at all on this segment, although the window after the drop printed a recall of 10/20 = 0.5000 and the population recall at 85% illumination printed 0.6910, below the requirement. In our reading this follows from the construction: after the median is subtracted, a good frame’s score depends only on the sensor noise, which does not change with the lamp, and good frames make up 95% of the stream, so the score distribution hardly moves while the defects become harder to see. An output monitor can stay quiet while the requirement is violated.

The defocus. The population curve put the violation at a blur of 0.413 pixels, local frame 706.4. H2 held in all four cases: the raw sharpness alarmed at local frames 686 (CUSUM) and 698 (KS), and the normalized sharpness at 665 (CUSUM) and 695 (KS), each shortly before that frame. The score’s CUSUM alarmed only 251 frames after onset and its KS monitor not at all, so, on this stream, the output monitor would have reported the defocus after the requirement was already violated. The windows show how the accuracy figure hides it: in the last window of the defocus segment, recall printed 4/27 = 0.1481 while accuracy printed 0.9500, because about 95% of frames are good and are still called good. The CUSUM on mean intensity also alarmed during this segment (first 55 frames after onset); such an alarm cannot by itself be told apart from the false alarms the same monitor raised on the control stream.

The label shift. The output monitors saw it: the score’s CUSUM alarmed 41 frames after the rise and its KS monitor 117 frames after it. The input monitors mostly did not. H3a held for five of the six input monitors: 2 alarms against a bound of 5 (intensity, CUSUM), 0 against 2 (intensity, KS), 1 against 7 (sharpness, CUSUM), 0 against 2 (sharpness, KS) and 5 against 6 (normalized sharpness, CUSUM).

H3a failed for the KS monitor on normalized sharpness, with 1 alarm against a bound of 0.

BBSE recovered the new defect rate. H3b held in both post-shift windows: 0.1985 against a true 0.1920 and 0.1940 against 0.1900, errors of 0.0065 and 0.0040 against the tolerance of 0.04. In the same windows the predicted-positive rate was 0.1780 and 0.1740, and H3c held: from the window before the shift to the two after it, the predicted-positive rate rose by 0.1220 while the true defect rate rose by 0.1410, the under-reporting that the derivation above predicts for any classifier with TPR−FPR<1\mathrm{TPR} - \mathrm{FPR} < 1. Read Sculley et al.’s prediction-bias diagnostic with that in mind: when labels are available, the gap between predicted and observed rates grows with the prevalence even if the classifier is unchanged (our derivation from the expectation above).

BBSE is wrong where its assumption fails. After the illumination drop it estimated 0.0244 for a true 0.0400, and in the last window of the defocus 0.0083 for a true 0.0540: in both segments the images of each class changed, so the reference confusion matrix no longer applies, and by the two-class formula above the estimate falls with the predicted-positive rate, which the lower recall pulls down, instead of following the defect rate. A BBSE estimate is only meaningful when the input monitors are quiet.

Checks. C1 held: the noise-free good interior equals 0.75 (maximum deviation 0.0e+00), every noise-free defective frame has a darker interior pixel (0 failures), the noise standard deviation printed 0.01997, and the blur schedule matches its formula. C2 held (maximum difference from correlate2d 2.8e-17). C3 held: on the Handbook’s worked example the implementation signals first at point 14, as the Handbook does, and ends with Shi(20)=19.0350S_{hi}(20) = 19.0350 against the Handbook’s printed 19.04. C4 held for both decision limits (within 10% of the table). C5 held (BBSE’s matrix form and the two-class formula agree to 2.8e-17; the confusion matrix’s determinant printed 0.04380). C6 held (no ties in the KS reference window) and C7 held (the stacked Laplacian equals the per-frame one). Two runs printed identical output.

How the example was made. The hypotheses, measures and checks were written before any script ran, and three planning scripts on another seed then computed the values that set the remaining choices. Several of those choices were made after a planning script had printed values, and the example is therefore an illustration of the mechanisms, not a test of them:

  • The scratch contrast was first to be chosen from 0.15 to 0.40 as the value whose nominal recall came closest to 0.93, accepted only between 0.88 and 0.97, and the rule said to stop and report, not widen the grid, if no value fitted. Every value gave a recall of 1.0000. Against that instruction, the rule was replaced at planning by a grid from 0.04 to 0.10 with a target recall of 0.90, accepted between 0.86 and 0.95; the whole replacement was written after that planning script, and its target after the script had also printed H3c’s expected margin under the first contrasts. The contrast of 0.06 came from the replacement rule.
  • The plate was first 20 pixels across; it was enlarged to 24 after the first planning script showed the blurred plate edge entering the classifier’s search region under strong defocus.
  • The requirement was first written as a recall of 0.85 and set to 0.80 after a planning script printed the nominal recall and the recall against blur.
  • The normalized sharpness was added as a fourth feature after a planning script showed the raw sharpness moving under the illumination drop, because whether a focus measure is normalized for brightness is a modelling choice the sources leave open, so both forms are printed.
  • The BBSE tolerance τ=0.04\tau = 0.04 comes from a formula fixed in advance (three times the estimate’s standard deviation at a 500-frame window, rounded up), whose value a planning script computed.

Other choices of the design: the three events sit in separate segments with restarts, so that the lighting alarm does not mask the later events; both one-sided CUSUM sums run on every feature, since a monitor does not know the sign of the next drift; α\alpha is chosen by a fixed matching rule on a calibration stream, because a window tested at every frame has no closed-form alarm rate; the requirement’s violation during the defocus comes from a population curve; H3a’s “no alarm” is a Poisson bound on each monitor’s control-stream rate; BBSE runs on 500-frame windows and is printed for all eight; the frame-to-frame illumination variation is 10%; and checks C3 to C7 were added to the planned measures. H1 is our own comparison: the Handbook’s statement about shifts of 2σ2\sigma or less compares CUSUM with Shewhart charts, not with a KS test. The outcomes test these analogues on rendered frames, not any paper’s setting. The script was run as a pilot with these settings (twice, with identical output); it was approved unchanged, and the first run’s output, shown above, is identical to the pilot’s. After the pilot, one diagnostic script examined the KS p-values on the calibration stream; nothing in this article rests on it.

Variants

  • Per-frame against per-batch statistics. Charting the mean of each batch of frames lowers the variance, so the same limits detect smaller shifts and raise fewer false alarms, at the cost of one batch of delay (our reasoning).
  • Shewhart, CUSUM and EWMA charts for single statistics, and their multivariate counterparts (the e-Handbook lists Hotelling, principal-component and multivariate EWMA charts in §6.3).
  • Windowed two-sample tests: KS per feature with a correction across features, MMD on embeddings, or tests on a classifier’s outputs (black-box shift detection in Rabanser et al.).
  • Label-shift estimation with BBSE, followed by correction of the classifier’s prior (class imbalance gives the adjustment for a known prior).
  • Accuracy estimation without labels from confidences (DoC and related methods), used to decide where to spend labelling effort.
  • Labelled audits: a random sample of production images labelled at a fixed rate, the only monitor of concept drift and of recall when decisions select the reviewed cases.
  • Physical references: a calibration target imaged at fixed intervals, whose statistics are free of product variation.

Best Practices

  • Measure the false-alarm rate, do not assume it. Run every monitor on a stretch of unchanged data and report alarms per 1,000 frames; in the example the CUSUM rates ranged from 1.70 to 6.70 alarms per 1,000 frames across features with the same limits.
  • Match each monitor to a cause. Lighting, focus, population and labelling each need their own signal; in the example the score monitor missed the lighting drop entirely and caught the defocus only after the requirement was violated.
  • Normalize physical monitors against confounders. A focus measure that scales with brightness alarms on lighting changes; dividing by the squared mean removes that factor from the noise-free image, though in the example the normalized form still alarmed on the lighting drop, later.
  • Express delays against the requirement, not in frames alone: an alarm is useful if it comes before the system fails its specification.
  • Estimate label shift only when the inputs are stable. BBSE assumes unchanged class-conditional images; check the input monitors first.
  • Audit at random. Labels gathered only where the system raised an alarm or rejected a part are a biased sample of its errors.
  • Restart and re-baseline after maintenance, and record each restart, so that alarms before and after a repair are not mixed.

Pitfalls

  • Treating a sliding test’s significance level as its false-alarm rate. Tests on overlapping windows against one reference sample are dependent; only the measured rate on unchanged data counts.
  • Accuracy as the headline monitor for a rare class. With 5% defective frames, accuracy stayed at 0.9500 in a window whose recall was 0.1481.
  • Reading the predicted-positive rate as prevalence. In expectation it scales every change in prevalence by TPR−FPR\mathrm{TPR} - \mathrm{FPR}.
  • Feedback through selective review, which hides misses and lets measured precision stand in for a recall nobody measures.
  • Thresholds left unchanged after retraining, which Sculley et al. note may become invalid when a model is updated on new data.

Evidence and Impact

The detection theory is classical: the e-Handbook’s run-length tables, quoted above, quantify the trade-off between false alarms and delay. For high-dimensional data, Rabanser et al. (2019) found two-sample tests on a classifier’s softmax outputs the best overall of the reductions they compared, and aggregated univariate tests comparable to multivariate kernel tests, on simulated shifts of MNIST and CIFAR-10 images; deploying vision models gives their sample-size results. Lipton et al. (2018) proved BBSE consistent, bounded its error and reported accurate estimates on natural images. Guillory et al. (2021) showed that confidence-based estimates of accuracy can track accuracy changes across many shifts, and also where they fail. On the systems side, the evidence is from engineering experience rather than controlled comparison: Sculley et al. (2015) describe the maintenance costs of changing external worlds and feedback loops, and Ensign et al. (2018) analyse how selective feedback compounds. In our reading, what transfers to vision from all of these is the discipline of the example: a reference period, a measured false-alarm rate, a delay stated against the requirement, and labelled audits for what no proxy can show.

Related

  • Designing a Computer Vision System

    How a working vision system is designed from its requirements, through the choice of sensor, optics, lighting, compute and model, to the data it is trained on, its validation in the field and its maintenance once deployed, with the requirement arithmetic for a measurement station worked through in code.

  • MLOps for Computer Vision

    The engineering practices that keep a vision model reproducible and maintainable after its first release, from versioning images, annotations and models by their content through pipelines that re-run only the stages a change affects, experiment tracking and the model registry to continuous training and lineage, with an example that caches a shape classifier's pipeline by content hash.

  • Domain Adaptation

    Adapting a model trained on labelled images from one domain to a shifted target domain, usually with only unlabelled target images; the settings, the theory of what alignment can and cannot guarantee, discrepancy, adversarial, pixel-level, self-training and test-time methods, domain generalization, and the benchmarks.

  • Dataset Bias

    How the way a dataset's images are chosen, photographed and labelled makes it an unrepresentative sample of the visual world, how cross-dataset tests measure the damage, and what bias means when the images show people, from demographic representation to withdrawn datasets.

  • Clinical Validation and Regulation of Medical Imaging AI

    How the claims made for a medical image analysis model are tested, from internal, temporal and external validation through reader studies, prospective studies and randomized trials, with the reporting guidelines that describe such studies, the statistics of rare findings, the regulatory pathways in outline, and monitoring after deployment.

  • Statistical Significance in Evaluation

    How much of a reported difference between two vision models is noise from the finite test set or from training randomness, and the confidence intervals, paired tests, multiple-comparison corrections and reporting practices that tell a real improvement from a lucky seed.

References

  1. NIST/SEMATECH. e-Handbook of Statistical Methods, Sections 6.3.2.3 (Cusum Control Charts), 6.3.2.3.1 (Cusum Average Run Length) and 6.3.2.4 (EWMA Control Charts) (accessed November 2026).
  2. Lipton, Z. C., Wang, Y.-X. & Smola, A. J. (2018). Detecting and Correcting for Label Shift with Black Box Predictors. Proceedings of the 35th International Conference on Machine Learning (ICML), PMLR 80, 3122–3130.
  3. Rabanser, S., Günnemann, S. & Lipton, Z. C. (2019). Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. Advances in Neural Information Processing Systems 32 (NeurIPS).
  4. Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B. & Smola, A. (2012). A Kernel Two-Sample Test. Journal of Machine Learning Research, 13, 723–773.
  5. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F. & Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NIPS).
  6. Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T. & Schmidt, L. (2021). Predicting with Confidence on Unseen Distributions. IEEE/CVF International Conference on Computer Vision (ICCV).
  7. Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C. & Venkatasubramanian, S. (2018). Runaway Feedback Loops in Predictive Policing. Conference on Fairness, Accountability and Transparency (FAT*), Proceedings of Machine Learning Research, 81, 160–171.