Clinical Validation and Regulation of Medical Imaging AI

Clinical validation is the work of testing whether a medical image analysis model does what is claimed for it, in the people, places and workflow where it is meant to be used. A model that scores well on a held-out split of its own dataset has passed one test of many: whether the score survives a later period, another hospital or another scanner, whether readers do better with the model than without it, whether patients are better off when it is used, and whether all of this stays true after the software is installed and the scanners, protocols and populations around it change. Regulation enters because in the United States such software is regulated as a medical device, and the EU Artificial Intelligence Act names the Medical Devices Regulation among the product legislation under which AI systems can be high-risk; the evidence a regulator asks for shapes the studies that get done. Related terms are clinical evaluation, external validation and real-world evaluation, and the reporting side is covered by guidelines such as TRIPOD+AI, STARD-AI and CLAIM.

Computer Vision in Radiology gives the reference standards, the reader study of McKinney et al. (2020), claims of radiologist-level performance, the MASAI trial and the gap between enriched test sets and screening populations, and Medical Computer Vision summarizes evaluation across the field. This article goes one level deeper into the designs themselves: the kinds of validation, reader studies in outline, prospective and randomized evaluation, reporting guidelines, the statistics of rare findings, regulatory pathways in outline and monitoring after deployment. It describes methods, evidence and limits for practitioners and researchers. It gives no clinical guidance and draws no clinical or legal conclusions of its own, and it attributes every reported result to its authors with the design behind it.

Definition

The word validation means different things in different communities, and most confusion about clinical validation starts there. In machine learning, the validation set is the data used to tune hyperparameters and select models, and the test set is held out for the final estimate (Training, Validation, and Test Sets). The EU Artificial Intelligence Act uses the same convention: validation data serve to evaluate the trained system and tune its non-learnable parameters, while testing data provide an independent evaluation that confirms the expected performance before the system is placed on the market (Regulation (EU) 2024/1689, Article 3, points 30 and 32). In clinical prediction research, by contrast, validation usually means evaluating a model’s performance, and external validation means doing so on new data. TRIPOD+AI notes this clash in its glossary and, to avoid ambiguity, calls data used to estimate performance evaluation data, which should be distinct from the data used for training, hyperparameter tuning and model selection, with no overlap in participants, and should represent the population in which the model is to be used (Collins et al., 2024, Box 1).

With that vocabulary, the TRIPOD statement of 2015 distinguishes the main kinds of evaluation (Collins et al., 2015):

• Apparent performance is measured on the data used to develop the model and tends to be optimistic because of overfitting.
• Internal validation uses only the original study sample, by data splitting, cross-validation or bootstrapping, to estimate and correct that optimism; TRIPOD calls it a necessary part of model development.
• External validation applies the finished model, unchanged, to participants other than those used to develop it. TRIPOD lists its forms: data from the same investigators with the same definitions but sampled from a later period (temporal or narrow validation); data from other investigators in another hospital or country, sometimes with different definitions and measurements (geographic or broad validation); similar participants from an intentionally different setting, such as primary instead of secondary care; and other types of participants altogether.

These are evaluations of the model’s outputs against a reference standard. A model can pass all of them and still fail to help, because the claim that matters clinically is usually about a decision made with the model, by a reader or a care pathway. Evaluations of that claim, reader studies, prospective studies and randomized trials, form the second half of clinical validation.

Intuition

Each design answers one question and leaves others open. A useful way to read any study is to ask which of these it answers.

Design	Question it answers	What it cannot show
Internal test split	Does the model generalize to new images from the same source?	Transfer to other sites, periods or scanners
Temporal validation	Does it still work on later data from the same source?	Transfer to other sites
Geographic or setting validation	Does it work at another hospital, in another population or setting?	Whether its use changes decisions
Standalone study on an enriched set	How well does it discriminate on difficult, chosen cases?	Predictive values and workload at real prevalence
Reader study (MRMC)	Do readers perform better with it than without it, on a fixed case set?	Behaviour under real working conditions
Prospective observational study	What happens when it is used on cases as they arrive?	Causal effect, if use is chosen by the clinicians
Randomized trial	What is the effect of using it on a prespecified outcome?	Long-term outcomes it was not powered for; other sites
Post-market monitoring	Is it still performing as validated?	Anything not measured in routine data

The table is our summary of the designs described below. Two things run through it. Each step moves the evaluation closer to the decision the model is meant to support and further from the convenient data it was built on. And each step is more expensive, which is why most published studies stop early: Kim et al. (2019) examined 516 studies, published between 1 January and 17 August 2018, that evaluated algorithms for diagnostic analysis of medical images, and found that only 31 (6%) performed external validation, none of which combined the three design features they considered fundamental for validating real-world performance: a diagnostic cohort design rather than a diagnostic case–control design, data from multiple institutions, and prospective data collection.

Validation Designs

Internal validation and its limits

Internal validation estimates how a model will do on new images drawn from the same process as the training images. Its standard failure in imaging is leakage: images of one patient on both sides of a split, which inflates performance in ways Medical Image Classification demonstrates. Even done correctly, by patient, it cannot reveal what is shared by every image of the source: the scanners, protocols, referral patterns, prevalence and labelling habits. Kim et al. (2019) point out that test sets in computer science are usually random subsamples of the original dataset, which cannot show clinical performance elsewhere, and that convenience case–control data, the data a developer can most easily collect, may be prone to selection bias and artificial disease prevalence.

Temporal, geographic and setting validation

Temporal validation holds out a later period from the same source: the model is developed on data up to a date and tested on data acquired afterwards. It tests robustness to drift in equipment, protocol and population at one site; by our reading it is also the closest offline imitation of deployment at the developing hospital, since a deployed model always meets data from after its training. Geographic validation tests transfer to other hospitals, where acquisition, populations and labelling all differ. Setting validation moves the model to a different place in the care pathway: TRIPOD’s example is from secondary to primary care, and in imaging a move from diagnostic work-up, where most patients are referred because something is suspected, to screening, where most are healthy, is of the same kind.

When the data come from several hospitals, the hospitals themselves are a source of variation. TRIPOD+AI asks authors to describe how heterogeneity in model performance across clusters such as hospitals or countries was handled and quantified, and to report it if examined, pointing to TRIPOD-Cluster for details (Collins et al., 2024, items 12d and 23b). A related design, which on our reading follows from treating hospitals as the unit, is to hold out each hospital in turn and report the spread of performance over hospitals rather than one pooled number. STARD-AI asks, for an external evaluation on an independent dataset, for an assessment of how it differs from the training, validation and test sets, and for whether the datasets represent the distribution of the target condition expected in the intended use population (Sounderajah et al., 2025, items 28 and 29).

External validation is necessary but not sufficient: shortcuts learned from one source may survive at another (Computer Vision in Radiology), and a model can keep its ranking of images while losing its operating point and calibration, as the example below shows.

Reader studies and multi-reader multi-case designs

Most imaging AI is meant to assist a reader, so the claim to test is often about the reader’s performance with the model. In a reader study, a set of readers interprets a set of cases under two or more conditions, such as unaided and aided, and their performance, usually an ROC AUC or sensitivity and specificity, is compared. The difficulty is that both readers and cases are samples: another set of readers, or another set of cases, would give different results, and because the same readers read the same cases under both conditions, the measurements are correlated. Multi-reader multi-case (MRMC) analysis treats both as random.

The Obuchowski–Rockette model, as the documentation of the MRMCaov package writes it, describes the performance $\hat{\theta}_{ij}$ of reader $j$ with test (or condition) $i$, estimated over the cases, as

$$\hat{\theta}_{ij} = \mu + \tau_i + R_j + (\tau R)_{ij} + \epsilon_{ij},$$

with a fixed test effect $\tau_i$, a random reader effect $R_j$, a random test-by-reader interaction $(\tau R)_{ij}$ and an error term $\epsilon_{ij}$ that carries the case sampling. The errors are correlated in three ways, with covariances for the same reader under two tests, two readers under the same test, and two readers under two tests, and these covariances are estimated from the case-level data by the jackknife or, for the empirical AUC, by the method of DeLong and colleagues or an unbiased estimator (Smith, Hillis & Pesce, MRMCaov user guide). The package’s documentation credits the analysis-of-variance approach to Dorfman, Berbaum and Metz (1992) and to Obuchowski and Rockette (1995), later unified by Hillis and colleagues. Its worked example is a factorial design, in which each of five radiologists read all 114 cases (45 with the condition and 69 without) under both imaging modalities; the package also supports designs in which cases are nested within readers or within tests, and partially paired designs (MRMCaov user guide). The iMRMC software from the FDA’s Center for Devices and Radiological Health analyses MRMC studies that compare AUCs between two modalities and can also be used to size such a study (iMRMC README).

Regulators ask for this kind of evidence. The special controls for medical image analyzers (CADe) require a description of pre-specified performance testing that assesses whether the device improves reader performance as intended and characterizes standalone performance, where the testing includes one or more standalone tests, side-by-side comparisons or a reader study, as applicable, and the results must show improved reader performance in the intended use population (Food and Drug Administration, 2020, reclassification order). The 2025 order for detection and diagnosis software (CADe/x) asks for assisted-read performance in the indicated user population and, separately, standalone performance to characterize the device for labeling (Food and Drug Administration, 2025).

A reader study still tests readers in an experiment: a fixed, often enriched case set, read outside the clinical routine, usually without the consequences of a real report. It measures what readers can do with the model under those conditions, not what they will do in practice. That is the step to prospective evaluation.

Prospective studies

A prospective study applies the model to cases as they arrive, with its output available to the people making decisions. STARD-AI defines the distinction by timing: a study is prospective when data collection was planned before the index test and the reference standard were performed, and retrospective when after (Sounderajah et al., 2025, item 5). The first prospective use is usually small. DECIDE-AI is a reporting guideline for this early-stage live clinical evaluation, in which the supported decisions affect patient care, and it is defined by stage rather than by study design (Vasey et al., 2022). Its authors compare the stage to phases 1 and 2 of drug trials and to the development and exploratory stages of surgical innovation, and they stress human factors: how users interact with the system, how their variability biases results, and how the system changes through version updates. The guideline has 17 AI-specific items, with 28 subitems, and ten generic ones.

Prospective is not the same as controlled. PRAIM, a prospective, observational, multicentre implementation study at 12 German screening sites, compared double reading with and without an AI system, with radiologists choosing examination by examination whether to read in the AI-supported viewer (Eisemann et al., 2025). Some preferred the AI viewer for examinations the AI tagged as normal, which made cancers more frequent in the group read without AI; the authors adjusted for this confounding by overlap weighting on propensity scores, list the absence of random assignment first among the study’s limitations, and report that a simulation and sensitivity analyses supported their adjustment. Mammography and Breast Cancer Screening gives the study’s design and results. The study shows what a prospective design gains, decisions actually made with the model in routine work, and what it costs when clinicians choose when to use the model: the comparison is only as good as the adjustment for why they chose it.

Randomized trials

Randomization removes that choice. He et al. (2023) ran a blinded, randomized non-inferiority trial of the initial assessment of left ventricular ejection fraction (LVEF) from echocardiograms, comparing an AI model with sonographers. Of 3,769 transthoracic echocardiogram studies performed at one academic medical centre in 2019, 274 were excluded for image quality and 3,495 were randomized 1:1; cardiologists then made the final assessment, starting from either the AI’s or the sonographer’s tracing without being told which. The primary end point, a change in LVEF of more than 5% between the initial and final assessment (the paper’s definition), occurred in 16.8% of AI-initialized and 27.2% of sonographer-initialized studies, a difference of −10.5% (95% CI −13.2% to −7.7%) in the paper’s Table 2 (the abstract and text give −10.4%), which met the prespecified non-inferiority margin and was also superior. The cardiologists guessed the source correctly for 32.3% of studies, wrongly for 24.2% and were unsure for 43.4%, and the authors report a blinding index within the range usually described as good. The model had been trained on 144,184 echocardiogram videos from another institution and was not changed during the trial, so the authors also describe the trial as a prospective external validation. They list as limitations that it was a single-centre trial, that it was not powered for long-term outcomes, and that it used previously acquired studies interpreted prospectively.

The trial illustrates design choices that standalone evaluation never faces: an end point defined by what clinicians do with the output, a margin fixed in advance, blinding of the people whose behaviour is measured, and a frozen model version. The MASAI screening trial, described in Computer Vision in Radiology, randomized screening examinations, allocating women 1:1 to AI-supported or standard reading, and measured screening outcomes. CONSORT-AI and SPIRIT-AI extend the general guidelines for trial reports and protocols to AI interventions; Medical Computer Vision summarizes what CONSORT-AI asks for.

Statistics at Low Prevalence

Many imaging tasks look for findings that are rare in the population where the model will be used. Prevalence then governs the precision of every estimate that depends on the positive cases, and it governs what predictive values mean (Computer Vision in Radiology works through the predictive values).

Intervals depend on the number of positives. Sensitivity is a proportion among the positive cases only. Its confidence interval therefore depends on the number of positives, not on the size of the test set: a test set of 1,000 images at 1% prevalence holds about ten positives, and an interval for sensitivity from ten cases is wide whatever method computes it. The Wald interval behaves badly for small counts and proportions near 1, and the Wilson interval is preferred there, as Statistical Significance in Evaluation explains; the Clopper–Pearson (“exact”) interval is the other common choice. As a planning rule that follows from the normal approximation, estimating a sensitivity near $\mathrm{Se}$ to within $\pm d$ needs about

$$n_{+} \approx \frac{1.96^2\, \mathrm{Se}\,(1 - \mathrm{Se})}{d^2}$$

positives, and a test set of about $n_{+}/\pi$ cases at prevalence $\pi$ (our arithmetic): for $\mathrm{Se} = 0.9$ and $d = 0.05$, about 138 positives, or about 13,800 cases at a prevalence of 1%. Specificity, estimated from the many negatives, is precise at the same size.

Zero events. When no errors of a kind are observed among $n$ cases, the point estimate is zero but the uncertainty is not. Solving $(1 - p)^n = 0.05$ gives the upper end of a one-sided 95% interval, $p \approx 3/n$ for large $n$: with no missed findings among 30 positives, a miss rate as high as about 10% is still compatible with the data (our derivation).

Calibration needs events too. Van Calster et al. (2019) distinguish four increasingly strict levels of calibration: in the mean (calibration-in-the-large, the average predicted risk against the observed rate), weak (a calibration intercept near 0 and a calibration slope near 1), moderate (a flexible calibration curve close to the diagonal) and strong, which they call utopic. A slope below 1 means risks too extreme, above 1 too moderate; a negative intercept means overestimation. They note that a minimum of 200 patients with and 200 without the event has been suggested for a precise calibration curve, that in small datasets it is defensible to assess only intercept and slope, that calibration-in-the-large is irrelevant at internal validation but highly relevant at external validation, and that poorly calibrated predictions at a new site call for updating, which may start with the intercept. At 1% prevalence, 200 events take 20,000 cases (our arithmetic). Expected Calibration Error covers binned measures and reliability diagrams; TRIPOD+AI describes calibration as best assessed graphically, with a smoothed flexible calibration curve in the individual data (Collins et al., 2024, Box 1).

Example: Sensitivity Intervals and Calibration at an External Site

The script trains a lesion detector on synthetic patches from one simulated site and evaluates it at a second site. Each patch is $24 \times 24$ pixels of smooth background texture with Gaussian noise; positives carry a faint round blob (Gaussian, $\sigma = 2$ pixels) at a random position. Site A, where the model is developed, has noise of standard deviation 0.3 and a lesion amplitude of 1.0, with a prevalence of 20%, like a development set enriched with positive cases. Site B, the external site, is noisier (0.6) with weaker lesion contrast (0.8), like a different detector, and has a prevalence of 1%. The model is a logistic regression on two features: the maximum of a difference-of-Gaussians blob filter (built from Gaussian filters) in the central $16 \times 16$ pixels, and the patch’s standard deviation. It is fitted twice on the same data, with scikit-learn’s default L2 penalty ($C = 1$) and with almost none ($C = 10^6$), because the calibration results depend on that choice. A threshold for 90% sensitivity is set on site A’s validation data.

The first part reports sensitivity and specificity with Wilson and Clopper–Pearson 95% intervals (SciPy’s binomtest) on site A’s internal test set and on the first 1,000, 5,000 and 20,000 images of site B’s evaluation sample. The second part measures calibration on both sites: the observed rate, the mean predicted risk, and the calibration intercept and slope, which the script estimates by logistic regression of the outcome on the model’s log-odds $\ell$, the slope as the coefficient $b$ in $\operatorname{logit} P(y = 1) = a + b\,\ell$ and the intercept as $a$ with $b$ fixed at 1. It prints the intercept that prevalence alone would produce, $\log \frac{\pi_B (1 - \pi_A)}{\pi_A (1 - \pi_B)}$ (the derivation is in Medical Image Classification), and then recalibrates on a separate local sample of 5,000 site-B images, once by updating the intercept only and once by refitting intercept and slope, and measures calibration again on the evaluation sample. The script loops over seeds 0–4 and both penalty settings and runs in about 8 seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2.

Disclosure, as required for examples changed after their first run: the site parameters, data sizes, features and outputs were planned before the first run. In the first run the detector did not work at its own site, because the background texture had the same scale as the lesion (blur $\sigma = 2$, unit standard deviation) and the blob filter’s maximum was taken over the whole patch, including its borders. A second run made the texture smoother and fainter (blur $\sigma = 6$, standard deviation 0.5) but also lowered the lesion amplitudes (0.6 and 0.45) and site B’s noise (0.5); the detector remained weak at site A. The third run keeps the smoother texture, restores the planned lesion amplitudes and noise levels, and takes the filter maximum over the central region where lesions are placed. It fitted only the default penalty. A check with almost no penalty then gave very different calibration slopes, so the penalty loop was added afterwards; the data, features and measurements are unchanged, and the results for the default penalty are identical to the third run’s.

import numpy as np
from scipy import ndimage, optimize, stats
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

H = 24  # patch side in pixels
SITES = {"A": dict(noise=0.3, amp=1.0),   # development site
         "B": dict(noise=0.6, amp=0.8)}   # external site: noisier, lower lesion contrast

def patches(n, prevalence, site, rng):
    """Synthetic patches: smooth texture plus noise; label 1 adds a faint Gaussian blob."""
    p = SITES[site]
    y = (rng.random(n) < prevalence).astype(int)
    tex = ndimage.gaussian_filter(rng.normal(size=(n, H, H)), (0, 6, 6))
    img = 0.5 * tex / tex.std() + rng.normal(scale=p["noise"], size=(n, H, H))
    yy, xx = np.mgrid[:H, :H]
    for i in np.flatnonzero(y):
        cy, cx = rng.uniform(6, H - 6, 2)
        img[i] += p["amp"] * np.exp(-((yy - cy) ** 2 + (xx - cx) ** 2) / (2 * 2.0 ** 2))
    return img, y

def features(img):
    dog = ndimage.gaussian_filter(img, (0, 1.5, 1.5)) - ndimage.gaussian_filter(img, (0, 4, 4))
    return np.column_stack([dog[:, 4:-4, 4:-4].max(axis=(1, 2)), img.std(axis=(1, 2))])

def ci(k, n):
    r = stats.binomtest(k, n)
    w, e = r.proportion_ci(method="wilson"), r.proportion_ci(method="exact")
    return f"{k / n:.2f} [{w.low:.2f}-{w.high:.2f}] [{e.low:.2f}-{e.high:.2f}]"

def calibration(lp, y):
    """Calibration intercept (slope fixed at 1) and slope, from logistic regression on the logit."""
    nll = lambda a: np.sum(np.logaddexp(0, a + lp) - y * (a + lp))
    a = optimize.minimize_scalar(nll, bounds=(-10, 10), method="bounded").x
    b = LogisticRegression(penalty=None).fit(lp[:, None], y).coef_[0, 0]
    return a, b

PREV = {"A": 0.2, "B": 0.01}
cal_rows = []
for seed in [0, 1, 2, 3, 4]:
    rng = np.random.default_rng(seed)
    Xtr, ytr = patches(3000, PREV["A"], "A", rng)
    Xva, yva = patches(2000, PREV["A"], "A", rng)
    Xte, yte = patches(3000, PREV["A"], "A", rng)
    Xlo, ylo = patches(5000, PREV["B"], "B", rng)    # local sample at site B, for recalibration
    Xex, yex = patches(20000, PREV["B"], "B", rng)   # external evaluation sample at site B
    for C in [1.0, 1e6]:   # scikit-learn's default L2 penalty, and almost none
        clf = LogisticRegression(C=C).fit(features(Xtr), ytr)
        lp = {k: clf.decision_function(features(X)) for k, X in
              [("va", Xva), ("te", Xte), ("lo", Xlo), ("ex", Xex)]}
        t = np.quantile(lp["va"][yva == 1], 0.10)        # threshold for 90% sensitivity at site A
        print(f"seed {seed}, C={C:g}: sensitivity / specificity with Wilson and exact 95% intervals")
        pos = lp["te"] >= t
        print(f"  A test   n=3000  pos={yte.sum():3d}  sens {ci(int(pos[yte == 1].sum()), int(yte.sum()))}"
              f"  spec {ci(int((~pos[yte == 0]).sum()), int((yte == 0).sum()))}")
        for n in [1000, 5000, 20000]:
            s, y = lp["ex"][:n], yex[:n]
            pos = s >= t
            print(f"  B first n={n:5d} pos={y.sum():3d}  sens {ci(int(pos[y == 1].sum()), int(y.sum()))}"
                  f"  spec {ci(int((~pos[y == 0]).sum()), int((y == 0).sum()))}")
        # calibration
        shift = np.log(PREV["B"] * (1 - PREV["A"]) / (PREV["A"] * (1 - PREV["B"])))
        a_lo, b_lo = calibration(lp["lo"], ylo)
        lp_int = lp["ex"] + a_lo                                   # intercept-only update
        lr = LogisticRegression(penalty=None).fit(lp["lo"][:, None], ylo)
        lp_full = lr.decision_function(lp["ex"][:, None])          # intercept and slope update
        row = [seed, C, roc_auc_score(yte, lp["te"]), roc_auc_score(yex, lp["ex"]), shift]
        for name, l, y in [("A test", lp["te"], yte), ("B", lp["ex"], yex),
                           ("B int", lp_int, yex), ("B int+slope", lp_full, yex)]:
            a, b = calibration(l, y)
            row += [y.mean(), (1 / (1 + np.exp(-l))).mean(), a, b]
        cal_rows.append(row)

print("\nseed      C  AUC A  AUC B  prior shift |  A test: obs  mean_p  int  slope |   B: obs  mean_p   int  slope |"
      " B, intercept updated: mean_p  int slope | B, int+slope updated: mean_p  int slope")
for r in cal_rows:
    print(f"{r[0]:4d}  {r[1]:>5g}  {r[2]:.3f}  {r[3]:.3f}    {r[4]:+.2f}    |  {r[5]:.3f}  {r[6]:.3f} {r[7]:+.2f} {r[8]:.2f} |"
          f"  {r[9]:.4f} {r[10]:.4f} {r[11]:+.2f} {r[12]:.2f} |  {r[14]:.4f} {r[15]:+.2f} {r[16]:.2f} |"
          f"  {r[18]:.4f} {r[19]:+.2f} {r[20]:.2f}")

seed 0, C=1: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=603  sens 0.89 [0.86-0.91] [0.86-0.91]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos= 12  sens 0.75 [0.47-0.91] [0.43-0.95]  spec 0.95 [0.94-0.97] [0.94-0.97]
  B first n= 5000 pos= 45  sens 0.69 [0.54-0.80] [0.53-0.82]  spec 0.95 [0.94-0.96] [0.94-0.96]
  B first n=20000 pos=196  sens 0.54 [0.47-0.61] [0.47-0.61]  spec 0.95 [0.95-0.95] [0.95-0.95]
seed 0, C=1e+06: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=603  sens 0.89 [0.86-0.91] [0.86-0.91]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos= 12  sens 0.67 [0.39-0.86] [0.35-0.90]  spec 0.97 [0.96-0.98] [0.96-0.98]
  B first n= 5000 pos= 45  sens 0.60 [0.45-0.73] [0.44-0.74]  spec 0.97 [0.97-0.98] [0.97-0.98]
  B first n=20000 pos=196  sens 0.48 [0.41-0.55] [0.41-0.55]  spec 0.97 [0.97-0.97] [0.97-0.97]
seed 1, C=1: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=625  sens 0.87 [0.84-0.89] [0.84-0.89]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos=  9  sens 0.33 [0.12-0.65] [0.07-0.70]  spec 0.96 [0.94-0.97] [0.94-0.97]
  B first n= 5000 pos= 58  sens 0.55 [0.42-0.67] [0.42-0.68]  spec 0.96 [0.95-0.96] [0.95-0.96]
  B first n=20000 pos=214  sens 0.62 [0.55-0.68] [0.55-0.69]  spec 0.96 [0.95-0.96] [0.95-0.96]
seed 1, C=1e+06: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=625  sens 0.88 [0.85-0.90] [0.85-0.90]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos=  9  sens 0.22 [0.06-0.55] [0.03-0.60]  spec 0.96 [0.95-0.97] [0.95-0.98]
  B first n= 5000 pos= 58  sens 0.50 [0.38-0.62] [0.37-0.63]  spec 0.96 [0.96-0.97] [0.96-0.97]
  B first n=20000 pos=214  sens 0.58 [0.52-0.65] [0.51-0.65]  spec 0.96 [0.96-0.97] [0.96-0.97]
seed 2, C=1: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=577  sens 0.93 [0.91-0.95] [0.90-0.95]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos=  9  sens 0.56 [0.27-0.81] [0.21-0.86]  spec 0.91 [0.90-0.93] [0.90-0.93]
  B first n= 5000 pos= 37  sens 0.54 [0.38-0.69] [0.37-0.71]  spec 0.91 [0.91-0.92] [0.91-0.92]
  B first n=20000 pos=182  sens 0.69 [0.62-0.75] [0.62-0.76]  spec 0.92 [0.92-0.92] [0.92-0.92]
seed 2, C=1e+06: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=577  sens 0.93 [0.91-0.95] [0.91-0.95]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos=  9  sens 0.44 [0.19-0.73] [0.14-0.79]  spec 0.94 [0.93-0.95] [0.92-0.96]
  B first n= 5000 pos= 37  sens 0.51 [0.36-0.67] [0.34-0.68]  spec 0.94 [0.93-0.95] [0.93-0.95]
  B first n=20000 pos=182  sens 0.66 [0.59-0.72] [0.59-0.73]  spec 0.94 [0.94-0.95] [0.94-0.95]
seed 3, C=1: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=613  sens 0.90 [0.87-0.92] [0.87-0.92]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos= 11  sens 0.82 [0.52-0.95] [0.48-0.98]  spec 0.95 [0.93-0.96] [0.93-0.96]
  B first n= 5000 pos= 53  sens 0.53 [0.40-0.66] [0.39-0.67]  spec 0.95 [0.94-0.95] [0.94-0.96]
  B first n=20000 pos=202  sens 0.60 [0.53-0.66] [0.53-0.67]  spec 0.95 [0.94-0.95] [0.94-0.95]
seed 3, C=1e+06: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=613  sens 0.90 [0.87-0.92] [0.87-0.92]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos= 11  sens 0.73 [0.43-0.90] [0.39-0.94]  spec 0.95 [0.94-0.96] [0.94-0.96]
  B first n= 5000 pos= 53  sens 0.49 [0.36-0.62] [0.35-0.63]  spec 0.95 [0.95-0.96] [0.95-0.96]
  B first n=20000 pos=202  sens 0.59 [0.52-0.65] [0.52-0.66]  spec 0.95 [0.95-0.95] [0.95-0.95]
seed 4, C=1: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=563  sens 0.91 [0.88-0.93] [0.88-0.93]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos= 12  sens 0.42 [0.19-0.68] [0.15-0.72]  spec 0.94 [0.92-0.95] [0.92-0.95]
  B first n= 5000 pos= 54  sens 0.57 [0.44-0.70] [0.43-0.71]  spec 0.94 [0.93-0.94] [0.93-0.94]
  B first n=20000 pos=185  sens 0.65 [0.58-0.72] [0.58-0.72]  spec 0.94 [0.94-0.94] [0.94-0.94]
seed 4, C=1e+06: sensitivity / specificity with Wilson and exact 95% intervals
  A test   n=3000  pos=563  sens 0.91 [0.88-0.93] [0.88-0.93]  spec 1.00 [1.00-1.00] [1.00-1.00]
  B first n= 1000 pos= 12  sens 0.33 [0.14-0.61] [0.10-0.65]  spec 0.96 [0.94-0.97] [0.94-0.97]
  B first n= 5000 pos= 54  sens 0.50 [0.37-0.63] [0.36-0.64]  spec 0.96 [0.95-0.96] [0.95-0.96]
  B first n=20000 pos=185  sens 0.60 [0.53-0.67] [0.53-0.67]  spec 0.96 [0.96-0.96] [0.96-0.96]

seed      C  AUC A  AUC B  prior shift |  A test: obs  mean_p  int  slope |   B: obs  mean_p   int  slope | B, intercept updated: mean_p  int slope | B, int+slope updated: mean_p  int slope
   0      1  0.999  0.866    -3.21    |  0.201  0.204 -0.05 3.77 |  0.0098 0.1969 -3.71 1.19 |  0.0093 +0.06 1.19 |  0.0091 +0.10 0.93
   0  1e+06  0.999  0.867    -3.21    |  0.201  0.200 +0.21 1.02 |  0.0098 0.1460 -7.77 0.33 |  0.0081 +0.53 0.33 |  0.0090 +0.10 0.93
   1      1  0.999  0.905    -3.21    |  0.208  0.207 +0.01 3.73 |  0.0107 0.1959 -3.62 1.36 |  0.0101 +0.06 1.36 |  0.0104 +0.04 1.01
   1  1e+06  0.999  0.906    -3.21    |  0.208  0.210 -0.21 1.01 |  0.0107 0.1870 -8.62 0.37 |  0.0106 +0.02 0.37 |  0.0104 +0.04 1.01
   2      1  0.999  0.897    -3.21    |  0.192  0.193 -0.01 3.84 |  0.0091 0.1946 -3.76 1.31 |  0.0104 -0.15 1.31 |  0.0100 -0.12 0.97
   2  1e+06  0.999  0.897    -3.21    |  0.192  0.191 +0.22 1.12 |  0.0091 0.1586 -7.88 0.38 |  0.0090 +0.02 0.38 |  0.0100 -0.12 0.97
   3      1  0.999  0.890    -3.21    |  0.204  0.209 -0.08 3.69 |  0.0101 0.2014 -3.73 1.29 |  0.0104 -0.03 1.30 |  0.0107 -0.08 1.00
   3  1e+06  0.999  0.891    -3.21    |  0.204  0.208 -0.67 0.96 |  0.0101 0.2047 -9.47 0.34 |  0.0119 -0.49 0.34 |  0.0107 -0.07 0.99
   4      1  0.999  0.903    -3.21    |  0.188  0.192 -0.07 3.09 |  0.0092 0.2031 -3.80 1.33 |  0.0102 -0.11 1.33 |  0.0103 -0.12 1.03
   4  1e+06  0.999  0.903    -3.21    |  0.188  0.192 -0.82 0.80 |  0.0092 0.1991 -9.42 0.35 |  0.0107 -0.41 0.35 |  0.0103 -0.13 1.02

At its own site the task is easy (AUC 0.999 in every seed, with either penalty) and the threshold delivers about the intended sensitivity, 0.87–0.93. At site B, with 182–214 positives among 20,000 images, sensitivity falls to 0.48–0.69, and the intervals at that size exclude site A’s values, so the drop is real; specificity falls from 1.00 to 0.92–0.97. The same site evaluated on its first 1,000 images, with 9–12 positives, tells a vaguer story. The point estimates range from 0.22 to 0.82 over the seeds and penalties, and the Wilson intervals are 0.43–0.54 wide; in seed 3, and in seed 0 with the default penalty, they still include site A’s sensitivity, so a study of that size would not have detected the loss. The specificity intervals, from about 990 negatives or more, are narrow at every size. The Clopper–Pearson intervals are wider than Wilson’s, most visibly with few positives. This is the low-prevalence problem in miniature: the precision of a sensitivity estimate is set by the number of positives, and an external test set that looks large can contain only a handful.

The calibration table separates the model’s ranking from its probabilities, and shows how much the probabilities depend on a modelling choice. With the default penalty, the mean predicted risk at site A matches the observed rate, but the slope is 3.1–3.8, far above 1: the probabilities are too moderate. With almost no penalty the internal slope is 0.80–1.12, so the moderate probabilities come from the penalty, which keeps the weights small when the two classes are almost separable; internal validation is where such a slope is detected. At site B both models overestimate risk in the mean, about 0.15–0.20 predicted against an observed rate of about 0.01. Prevalence alone would shift the intercept by −3.21. With the default penalty the measured intercepts, −3.62 to −3.80, are more negative, which on our reading reflects the acquisition change on top of the prevalence change (noisier images raise the filter responses of negative patches), and the slope is 1.19–1.36. Without the penalty the slope is 0.33–0.38: the noisier images make the unpenalized model’s risks far too extreme, and the intercept, estimated with the slope fixed at 1, absorbs that too (−7.77 to −9.47). On our reading the default penalty’s too-moderate probabilities and the external site’s too-extreme ones nearly cancel, so the external slope near 1 under the default penalty says little about how well the model transfers. Updating the intercept on the local sample brings the mean predicted risk to within 0.002 of the observed rate under both settings but leaves the slope where it was; refitting both brings the slope to 0.93–1.03 under both. Both updates are monotone in the score, so the AUC at site B, 0.87–0.91, is unchanged by them, and neither recovers the sensitivity lost to the noisier images; only a new threshold, or a better model, can do that. The local sample of 5,000 images at 1% prevalence is expected to hold only about 50 positives, which is why the updated intercepts still scatter around zero, by up to about 0.15 with the default penalty; without the penalty, the intercept-only update leaves them up to about 0.5 from zero, since it fits an intercept while the slope stays far from 1. The synthetic patches differ from clinical images in every respect except the arithmetic of counting positives and fitting probabilities, which holds for any model.

Reporting Guidelines

Reporting guidelines list what a study must state so that a reader can judge its design and risk of bias; they do not prescribe how to run the study. STARD-AI sets out the family of AI extensions and their scopes (Sounderajah et al., 2025, Table 1):

Guideline	Scope
STARD-AI	Diagnostic accuracy of an AI-based test
TRIPOD+AI	Developing and validating a prediction model by regression or AI methods
TRIPOD-LLM	Prediction models that use large language models
CLAIM	Developing and validating a medical imaging AI model
DECIDE-AI	Early-stage clinical evaluation of an AI decision support system
SPIRIT-AI	Protocols of clinical trials of AI interventions
CONSORT-AI	Reports of clinical trials of AI interventions
CHEERS-AI	Health economic evaluations of AI interventions

STARD-AI (2025) extends STARD 2015, whose 30 items cover diagnostic accuracy studies in general, to 40 items, of which four are modified and 14 are new, developed through a modified Delphi process with more than 240 international stakeholders and a consensus meeting. Several of the new or modified items concern data rather than models: eligibility at both the participant and the data level, the source of the data, who annotated them and how, the devices (manufacturer and model) and software versions used, acquisition protocols and preprocessing, how the index test was developed through training, validation, testing and external evaluation, with sample sizes, and the baseline characteristics of the training, validation and test sets. Another concerns the analysis: error analysis and assessments of algorithmic bias and fairness. Items carried over from STARD 2015 remain, among them whether positivity cut-offs were prespecified or exploratory, whether participants formed a consecutive, random or convenience series, and the distribution of disease severity and of alternative diagnoses. For a study whose primary aim is a prediction model rather than diagnostic accuracy, the authors point to TRIPOD+AI, and they mention CLAIM for developing or validating an imaging model (Sounderajah et al., 2025).

TRIPOD+AI (2024) replaces the TRIPOD 2015 checklist with 27 main items and 52 subitems, each marked as applying to development, evaluation or both (Collins et al., 2024). The items that bear on evaluation ask how the evaluation data differ from the development data in setting, eligibility, outcome and predictors; a comparison of the distributions of important predictors with the development data; performance estimates with confidence intervals, including for key subgroups such as sociodemographic ones; the measures and plots used, naming discrimination, calibration and clinical utility; and any model updating such as recalibration, with the updated model’s performance. Medical Computer Vision summarizes the statement’s scope and how a 2020 review measured adherence to its predecessor.

CLAIM, the Checklist for Artificial Intelligence in Medical Imaging, was published in 2020 (Mongan et al., 2020) and updated in 2024 (Tejani et al., 2024); its text could not be read for this article. DECIDE-AI (2022), described above, covers the first live clinical use.

Regulation in Outline

The summary below is drawn only from official texts that could be read: final orders in the Federal Register of January 2020 and June 2025, the US Code as published on govinfo.gov, and the EU Artificial Intelligence Act as published on 12 July 2024 (later amendments are not covered). It is an outline as of November 2026, not legal analysis. Computer Vision in Radiology gives the US device types for radiological software, the De Novo and 510(k) mechanism and the 2026 decision on a petition for exemption, and the AI Act’s classification of AI systems in medical devices as high-risk.

What premarket evidence looks like in the US orders. The special controls for the radiological device types translate the validation designs above into requirements. Besides the performance testing described under reader studies, they require labeling that describes the patient population for which the device is indicated, the intended reading protocol, the intended user and user training, the device’s inputs and outputs, and the compatible imaging hardware and protocols (Food and Drug Administration, 2020, order on lesions suspicious for cancer). The order for triage and notification software requires the labeling to discuss situations in which the device may fail or may not operate at its expected performance level (Food and Drug Administration, 2020, triage and notification order). On our reading, the labeling thus states the setting in which the device was validated.

Postmarket surveillance. Section 522 of the Federal Food, Drug, and Cosmetic Act lets the FDA, by order at clearance or approval or at any time afterwards, require a manufacturer to conduct postmarket surveillance of a class II or class III device that meets one of the criteria the section lists, such as a device whose failure would be reasonably likely to have serious adverse health consequences; the manufacturer then submits a surveillance plan for approval (21 U.S.C. 360l). When the FDA reclassified medical image analyzers in 2020, a commenter asked for special controls requiring postmarket data, periodic retraining of clinicians and quality assurance. The FDA declined. It cited nearly 20 years of regulating these devices as class III without requiring postmarket studies, no medical device reports for them in the previous ten years, and its remaining authority under section 522; it wrote that manufacturers should consider the need for, and the most appropriate methods, to evaluate potential changes in CADe performance over time (Food and Drug Administration, 2020, reclassification order).

Planned changes. Section 515C, added to the Act in December 2022, allows a predetermined change control plan, describing planned changes to a device, to be approved or cleared with it; changes consistent with the plan then need no new supplemental application or 510(k). The FDA may approve or clear such a plan if the device remains safe and effective without the changes and, for cleared devices, would remain substantially equivalent to its predicate. The FDA may require the plan to include labeling for the changed device, notification requirements if the device does not function as intended under the plan, and performance requirements for the changes, and versions changed under a plan cannot serve as predicates for other devices (21 U.S.C. 360e-4).

The EU AI Act. For high-risk AI systems that are products, or safety components of products, covered by the legislation in Section A of Annex I, which includes the Medical Devices Regulation (EU) 2017/745, the provider follows the conformity assessment procedure of that legislation, and the AI Act’s requirements become part of that assessment (Article 43(3)). Among those requirements, training, validation and testing data must be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose, and must take into account, to the extent the intended purpose requires, the geographical, contextual, behavioural or functional setting in which the system is to be used (Article 10(3) and (4)). Providers must establish a post-market monitoring system that actively and systematically collects, documents and analyses data on the system’s performance throughout its lifetime, based on a plan that is part of the technical documentation; where the product legislation already requires post-market surveillance, the provider may integrate the elements into the existing system and plan, provided that this achieves an equivalent level of protection (Article 72). Article 113 sets the dates: the obligations attached to Article 6(1), the provision that makes AI in such products high-risk, apply from 2 August 2027.

Post-Market Monitoring

After deployment the inputs change: new scanners and software versions, changed protocols, a different referral mix and, when screening programmes change, a different prevalence. By our reasoning from the example, monitoring has to catch a fall in sensitivity at a fixed threshold and a shift in calibration, neither of which need show in the AUC, and what can be monitored depends on what is observable. Input statistics (acquisition parameters, intensity distributions, the proportion of images flagged) are available at once and can signal a change of scanner or population. Performance against outcomes needs a reference standard, which arrives late, in screening often only after the next round, and for rare findings accumulates slowly: by the arithmetic above, a hospital that sees 50 positive cases a year needs almost three years to estimate sensitivity to within ±0.05 at a sensitivity of 0.9. It follows from the event counts that calibration can be checked in the mean, by comparing the average predicted risk with the observed rate, long before there are enough events for a curve.

The guidelines and texts above give this monitoring its places: DECIDE-AI for the first live use, TRIPOD+AI’s items on model updating for recalibration studies, section 522 and predetermined change control plans in the United States, and Article 72 of the AI Act in the EU. What a monitoring plan should contain for a given device is set by those instruments and by regulators’ guidance documents.

Common Misconceptions

• “Validated” means one thing. A model validated on a held-out split, at a second hospital, in a reader study and in a randomized trial has passed four different tests; a claim should name which.
• External validation shows the model is safe to use. It shows that discrimination and calibration transfer to the data tested, at one time. It does not show that readers use the output well, and shortcuts may survive it.
• A large test set, or a good AUC at a new site, settles the question. Precision depends on the number of positives, and the ranking can transfer while the operating point and the probabilities do not (the example above).
• Prospective means unbiased. A prospective study in which clinicians choose when to use the model can be confounded by that choice, as PRAIM’s authors found and adjusted for.
• Following a reporting guideline makes a study good. DECIDE-AI’s authors state that adherence to a reporting guideline should not be taken as a sign of methodological quality, which is the domain of methodological guidance and risk-of-bias tools (Vasey et al., 2022).

Where It Is Used

Every claim of clinical performance in the neighbouring articles rests on one of these designs. Computer Vision in Radiology and Digital Pathology report reader studies, external validations and trials in their fields; Medical Image Classification and Medical Image Segmentation describe the splits and metrics on which internal validation depends; Chest X-ray Analysis and mammography screening, the setting of MASAI and PRAIM, apply these designs to their own tasks. The general tools are in the evaluation articles: ROC AUC for discrimination, Expected Calibration Error for calibration, Statistical Significance in Evaluation for intervals and comparisons, and Dataset Bias for why data from one source misrepresent others.