Computer Vision in Radiology

Radiology is the medical specialty that acquires and interprets images of the inside of the body: X-ray radiographs, mammograms, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound and others. A radiologist reads each examination and writes a report, and the report, not the image, is what usually reaches the referring physician. Computer vision enters at several points of this work. It can mark suspicious regions for the reader (computer-aided detection, CADe), estimate how likely a finding is to be malignant (computer-aided diagnosis, CADx), reorder the queue of examinations waiting to be read (triage or worklist prioritization), measure structures that readers would otherwise estimate by eye (quantification), and draft text for the report (report generation). Work in this area is also called radiology AI, medical image analysis for radiology, or, in older literature, CAD.

This article is an overview. It describes the images, the tasks and the methods in outline and links to the articles that treat them in depth; it gives most of its space to what decides whether a reported result means anything: the reference standard, the comparison with readers, the case mix, and whether the study was retrospective, external or prospective. It describes methods, evidence and limits for practitioners and researchers. It gives no diagnostic or clinical guidance, draws no clinical conclusions of its own, and attributes every reported result to its authors with the design behind it.

Problem

A radiological examination produces one or more images, from a pair of chest radiograph views to CT and MRI volumes with hundreds of slices, and a radiologist must search them for findings, characterize what is found, compare with prior examinations and report. Two features of this work make it a target for computer vision. The first is volume and repetition: screening programs read large numbers of mostly normal examinations. McKinney et al. (2020) state that more than 42 million screening mammography examinations are performed each year in the United States and the United Kingdom combined, and in the UK each is read by two readers. The second is variability between readers. The same authors describe high variability in experts’ cancer detection accuracy, and Ouyang et al. (2020) describe considerable inter-observer variability in the human assessment of cardiac function from ultrasound.

A computer can help in different ways, and the way decides both how it should be evaluated and how regulators classify it. Doi (2007), in a historical review, describes the original idea of computer-aided diagnosis: the radiologist uses the computer output as a “second opinion” and makes the final decision, so the computer need not match the radiologist, only complement them. He contrasts this with automated computer diagnosis, which rests on the algorithm alone. The regulatory definitions below are written around the first model; screening studies such as MASAI test a third arrangement, in which the computer decides how many humans read an examination.

History

Rule-based and feature-based CAD. Litjens et al. (2017), in a survey of deep learning in medical image analysis, describe the field’s earlier phases: from the 1970s to the 1990s, systems chained low-level pixel processing (edge and line detectors, region growing) with mathematical modelling into rule-based programs, which they compare to the brittle expert systems of the time; at the end of the 1990s, supervised methods became increasingly popular, including feature extraction followed by statistical classifiers for computer-aided detection and diagnosis, with the features still designed by hand. They note that convolutional networks were already applied to medical images in 1995, by Lo et al., to detect nodules in X-ray images; the survey’s text calls it a four-layer network, while its table lists a two-layer CNN with twelve 5×5 filters in each layer.

Mammography CAD in practice. Doi (2007) notes that a large number of CAD systems had been employed to assist in the early detection of breast cancer on mammograms. Lehman et al. (2015) write that after the US Food and Drug Administration (FDA) approved CAD for mammography in 1998, and the Centers for Medicare and Medicaid Services provided increased payment in 2002, CAD technology disseminated rapidly; Fenton et al. (2007) also date the FDA approval to 1998. Large observational studies then measured its effect in community practice. Fenton et al. (2007) compared screening performance before and after seven of 43 facilities in three US states adopted CAD (1998–2002, 429,345 mammograms of 222,135 women, cancers counted within a year of screening): specificity fell from 90.2% to 87.2%, the positive predictive value from 4.1% to 3.2% and the biopsy rate rose by 19.7%, while the rise in sensitivity from 80.4% to 84.0% was not significant. Lehman et al. (2015) compared 495,818 digital screening mammograms read with CAD and 129,807 without (2003–2009, 271 radiologists at 66 facilities of the Breast Cancer Surveillance Consortium, cancers identified by linkage to tumor registries within a year), with radiologist-level random effects and adjustment for patient factors. They found no improvement on any measure (sensitivity 85.3% with and 87.3% without CAD; specificity 91.6% and 91.4%), and among 107 radiologists who read both ways, sensitivity was lower with CAD. Both studies are observational, so they measure an association in practice rather than the effect of CAD under controlled conditions; McKinney et al. (2020) summarize this generation of software as having failed to improve reader performance in real-world settings.

Deep learning. Litjens et al. (2017) date the turn to AlexNet’s ImageNet result of December 2012, with papers applying deep learning to medical images growing rapidly in 2015 and 2016. Large public chest radiograph datasets followed: ChestX-ray8 and its extension ChestX-ray14 in 2017 (Wang et al., 2017), CheXpert and MIMIC-CXR in 2019 (Irvin et al., 2019; Johnson et al., 2019). Comparisons of models with radiologists on these data followed (Rajpurkar et al., 2017; Irvin et al., 2019), and the section on evaluation examines what they show.

Regulation and trials. Between July 2017 and May 2018, the FDA created three new device types for radiological software by De Novo classification, for diagnostic, triage and detection-and-diagnosis software, and in January 2020 it moved the older CADe device type out of the highest risk class; the section on regulation gives the details. Randomized trials followed: the MASAI trial in Sweden, whose authors knew of no earlier randomized trial of AI-supported mammography screening, enrolled women from April 2021 (Lång et al., 2023).

Visual Data

Radiology uses several imaging physics, each with its own image statistics, artifacts and conventions. They are summarized here only as far as a vision practitioner needs; the linked sensor articles give the physics.

• Projection radiography (X-ray). X-rays pass through the body to a detector on the other side, and the image records the “shadows” of the structures they pass through; digital detectors have largely replaced screen-film systems (NIBIB, Medical X-rays). Each pixel integrates attenuation along a ray, so structures at different depths overlap, and a single frontal view can hide findings; the CheXNet authors cite the claim that up to 15% of accurate diagnoses need the lateral view (Rajpurkar et al., 2017). Irvin et al. (2019) call chest radiography the most common imaging examination globally, and most large public datasets are of chest radiographs.
• Mammography. A digital X-ray detector images the breast, compressed between the detector support and a paddle (NIBIB, Medical X-rays). A complete screening examination in the data of McKinney et al. (2020) has four standard views, two of each breast, and readers compare with prior examinations; the reader study of McKinney et al. (2020) gave readers up to four sets of priors where available. Breast density affects how visible abnormalities are (NIBIB, Medical X-rays).
• CT. CT reconstructs cross-sectional slices, and usually whole volumes, of X-ray attenuation in Hounsfield units, calibrated to water and air, so that intensities have a physical meaning that camera values lack, although the same tissue can still give different values on different scanners and tube voltages; Computed Tomography covers projection, reconstruction, windowing and artifacts.
• MRI. MRI samples the Fourier transform of the body’s magnetization and reconstructs images whose contrast depends on tissue relaxation times and on the sequence; intensities are not calibrated, and one examination contains several contrasts. Magnetic Resonance Imaging covers k-space, sequences and the DICOM and NIfTI formats in which images reach vision code.
• Ultrasound. A transducer sends sound pulses into the body and records their echoes; from the speed of sound and the echo times the scanner computes distances to tissue boundaries and forms images, displayed in 2D, 3D or 3D over time (NIBIB, Ultrasound). Ultrasound does not image bone or air-filled tissue such as the lungs well (NIBIB, Ultrasound). For vision, it is the radiological modality that is natively video: EchoNet-Dynamic, for example, works on echocardiogram videos and uses information across several cardiac cycles (Ouyang et al., 2020), a task related to video understanding.

Two properties of the data cut across modalities. Images are usually stored and exchanged in the DICOM format (MIMIC-CXR, for example, distributes its images in DICOM) inside picture archiving and communication systems (PACS), with acquisition metadata that a model can exploit, intentionally or not (see shortcut learning below). And labels are expensive: they need expert readers, and pathology confirmation or follow-up takes months or years (McKinney et al., 2020, followed women for 27 to 39 months). Large datasets therefore often take their labels from the free-text reports by natural language processing (NLP), as ChestX-ray14, CheXpert and MIMIC-CXR do, which brings label noise of its own.

Vision Tasks Involved

Reading support: detection and diagnosis

Computer-aided detection marks locations that may be abnormal (a nodule on a chest radiograph, a mass or calcifications on a mammogram) and is a form of object detection, usually evaluated per lesion. Computer-aided diagnosis assigns a finding or an examination a likelihood of disease or a category, a form of image classification. The FDA’s device definitions keep the two apart: a medical image analyzer directs the clinician’s attention to portions of an image and is not intended to recommend a diagnosis, while CADx software for lesions suspicious for cancer characterizes lesions and provides information about them, with decisions made by the clinical user (the 2020 Federal Register orders on medical image analyzers and on lesions suspicious for cancer). A system can do both: the mammography system of McKinney et al. (2020) outputs an examination-level cancer score and a ranked list of boxes around suspicious areas.

Triage and worklist prioritization

Radiologists read from worklists of waiting examinations. A triage system scores incoming examinations and moves those likely to contain urgent findings forward, or flags likely normal examinations. The FDA’s definition of radiological computer aided triage and notification software captures one cautious form: it notifies a designated list of clinicians of time-sensitive images, does not mark locations in the image, does not remove cases from the reading queue, and operates in parallel with the standard of care (Food and Drug Administration, 2020, triage and notification order).

The benefit of triage is measured in time, not only in accuracy. Annarumma et al. (2019) trained an ensemble of two convolutional networks on 470,388 adult chest radiographs (2007–2017) labelled critical, urgent, nonurgent or normal by an NLP system applied to the reports, and tested prioritization in a simulation on 15,887 further radiographs. Normal radiographs were identified with a sensitivity of 71% and a specificity of 95%, and in the simulation the average reporting delay for critical findings fell from 11.2 to 2.7 days compared with historical data. Because the comparison is a simulation against historical reporting times, it estimates what reordering could achieve, not what happened when a department used the system.

Screening offers a second form of triage, in which the score decides how many readers see an examination. In McKinney et al.’s (2020) simulation of UK double reading, the second reader was skipped whenever the AI agreed with the first reader, and the authors report performance equivalent to double reading with 88% of the second reader’s work saved. MASAI tested a related protocol prospectively (see evaluation).

Quantification

Many radiological decisions depend on measurements: the volume of a tumor and its change between examinations, the size of a nodule, the fraction of blood the left ventricle ejects per beat. Vision supplies these measurements through segmentation, often with a U-Net configured by a framework such as nnU-Net, and through registration to compare examinations of the same patient over time. EchoNet-Dynamic illustrates the combination in ultrasound video. In their abstract (the source read for this article; it does not describe the reference measurements), Ouyang et al. (2020) report that the model, trained on echocardiogram videos, segments the left ventricle with a Dice similarity coefficient of 0.92, estimates the ejection fraction with a mean absolute error of 4.1% and classifies heart failure with reduced ejection fraction with an AUC of 0.97, and that on an external dataset from another health system the error was 6.0% and the AUC 0.96; a prospective evaluation with repeated human measurements found the model’s variance comparable to or less than that of human experts. The authors describe it as surpassing human experts on these tasks, in that set-up. A measurement carries its own error model: segmentation metrics such as the Dice coefficient say how well a mask overlaps a reference, not how well a derived volume or ejection fraction agrees with the clinical value, which is reported separately, as here.

Report generation

Report generation produces free text from images, a task related to image captioning. Jing et al. (2018) framed it as predicting tags and generating a multi-sentence findings paragraph, with a co-attention mechanism over image regions and a hierarchical recurrent decoder, evaluated on the Indiana University chest X-ray collection (IU X-Ray, 7,470 image–report pairs) and a pathology image set. Chen et al. (2020) used a Transformer with a relational memory and were, in their words, the first to report generation results on MIMIC-CXR. Their evaluation shows the measurement problem of the task: besides n-gram overlap metrics from captioning (BLEU, METEOR, ROUGE-L), they report “clinical efficacy” metrics, obtained by running the CheXpert labeller on generated and reference reports and comparing the 14 extracted labels by precision, recall and F1. Overlap metrics reward fluent text that resembles the reference and do not detect a plausible sentence about a finding that is not in the image; the label-based metrics detect it only for the 14 observations the labeller extracts. Such sentences are the failure that multimodal hallucination describes for vision-language models in general.

Image formation

Vision methods also act before reading, in reconstruction of CT and MRI images from fewer or noisier measurements. Their outputs are the images that every task above consumes, and their own failure mode, structures that look plausible but were not measured, is discussed in that article.

Common Methods

The methods are those of general computer vision, adapted to the data. Examination-level classifiers are convolutional networks or Transformers, which can be initialized by transfer learning from natural images; CheXNet, for example, is a 121-layer DenseNet trained on ChestX-ray14 (Rajpurkar et al., 2017). Detection uses box or point detectors on 2D images and slices, segmentation uses 2D and 3D encoder–decoders, and volumetric tasks bring the anisotropic voxels and large inputs described in Medical Image Segmentation.

Three adaptations are specific to radiology:

• Weak labels from reports. Report-derived labels are cheap but noisy and are usually examination-level, without locations. Wang et al. (2017) mined eight, then fourteen, findings from reports and trained classifiers with only image-level labels, then derived heatmaps for localization. CheXpert’s labeller marks each of 14 observations as positive, negative or uncertain, and Irvin et al. (2019) compared several ways of treating the uncertain label in training; different treatments worked best for different observations.
• Multiple images per decision. A mammography examination has four views and priors; a chest radiograph study may have frontal and lateral views; an MRI study has several contrasts. Models combine them at the input or feature level, and the evaluation must state the unit (image, view, study or patient).
• Operating points. Most outputs are continuous scores, and a clinical decision needs a threshold. McKinney et al. (2020) chose each threshold on a separate validation set, never on the test set, to match or exceed a particular human benchmark; the example below shows why the threshold has to be chosen with the target population’s prevalence in mind.

Datasets

Most clinical image collections are private. The public datasets below are named with their official pages; their facts come from their papers and pages, no data were downloaded for this article, and the terms of access were read where the page states them.

Dataset	Content	Labels	Access (as stated)
ChestX-ray14	112,120 frontal chest radiographs (NIH Clinical Center)	14 findings mined from reports by NLP; boxes for a subset	distributed by the NIH Clinical Center (terms not read for this article)
CheXpert	224,316 chest radiographs of 65,240 patients (Stanford Hospital, 2002–2017)	14 observations, positive, negative or uncertain, from a rule-based report labeller; radiologist-labelled validation and test sets	through the Stanford AIMI website (terms not read for this article)
MIMIC-CXR	377,110 images from 227,835 studies of 65,379 patients (Beth Israel Deaconess Medical Center, 2011–2016)	free-text reports	credentialed users who sign a data use agreement; PhysioNet Credentialed Health Data License 1.5.0 (v2.1.0 page)
LIDC-IDRI	1,018 thoracic CT scans	nodule marks, outlines and ratings by four thoracic radiologists	CC BY 3.0 (TCIA collection page)

ChestX-ray14. The CVPR paper introduced ChestX-ray8, 108,948 frontal radiographs of 32,717 patients with eight findings mined from reports; after the CVPR submission, the authors extended the label set to 14 findings and 112,120 images, which is the version described in the appendix of the final arXiv version (Wang et al., 2017). They evaluated their NLP labelling against reports annotated by people, reporting a precision of 0.90 and a recall of 0.91 overall on the OpenI reports for the eight-finding set, so about one label in ten that the method assigned was wrong on that evaluation set before any model was trained.

CheXpert. The validation set of 200 studies was labelled by three board-certified radiologists, and the test set of 500 studies by eight: the majority vote of five forms the reference standard and the other three serve as the human comparison (Irvin et al., 2019; CheXpert page). Keeping the reference readers separate from the comparison readers avoids scoring a reader against a reference that reader helped define.

MIMIC-CXR. The images are in DICOM format and come with their free-text reports, de-identified to the HIPAA Safe Harbor standard, which makes it usable for report generation (Johnson et al., 2019); Chen et al. (2020) report generation results on it. Access requires PhysioNet credentialing, a training course and a signed data use agreement that forbids sharing the data and attempts at re-identification (PhysioNet project page, v2.1.0, published 23 July 2024).

LIDC-IDRI. Four experienced thoracic radiologists read each scan in two phases: first blinded, then reviewing their own marks alongside the anonymized marks of the other three, without being forced to agree. Of 7,371 lesions marked as a nodule by at least one radiologist, 2,669 were marked as a nodule of at least 3 mm by at least one, and only 928 of these (34.7%) by all four (Armato et al., 2011). The collection page lists 1,010 patients, noting that eight patients have two scans from different time points (TCIA collection page). The numbers make reader disagreement concrete: the reference standard for “nodule” depends on how many of the four readers one requires.

Evaluation

A reported performance figure in radiology needs at least five pieces of context: the unit of evaluation, the reference standard, the comparison, the case mix and the study design. Nagendran et al. (2020), in a systematic review of records from 2010 to June 2019, examined studies that compared deep learning algorithms in medical imaging with expert clinicians; radiology was the most common specialty, with 36 of the 81 non-randomized studies. They found 10 records of randomized trials, of which two were published, and 81 non-randomized studies, of which nine were prospective and six were tested in a real-world clinical setting; the median number of experts in the comparator group was four; data and code were unavailable for 95% and 93% of studies; 58 of 81 were at high risk of bias; and 61 of 81 stated in their abstract that the performance of the AI was at least comparable to (or better than) that of clinicians. The design elements below are the ones those numbers concern.

Reference standards

The label a model is scored against is itself a measurement. The studies in this article range from pathology and long-term follow-up (McKinney et al., 2020) through the majority vote of several radiologists (CheXpert’s test set) to NLP applied to reports (ChestX-ray14’s labels). McKinney et al. (2020) used biopsy-confirmed cancer within a follow-up interval chosen to include the next screening round (39 months in the UK, 27 months in the US), and required negatives to have a later non-cancer screen. They argue that short follow-up favours readers, because readers are the gatekeepers to biopsy: a cancer the reader missed is not diagnosed within a short interval and so does not count against the reader, while a longer interval also counts cancers found at the next screen. They show measured reader sensitivity falling as the interval lengthens.

Standalone performance, reader studies and MRMC analysis

A standalone study scores the algorithm alone, usually by sensitivity, specificity and the ROC curve and its area. A reader study asks readers to interpret a set of cases, with or without the algorithm, or compares them with it. Because both readers and cases vary, and the same readers read the same cases, the statistics must treat both as random samples; multiple-reader multiple-case (MRMC) methods do this. McKinney et al. (2020) compared their system with six radiologists by the Obuchowski–Rockette–Hillis procedure, with the jackknife to estimate covariance terms, a method originally formulated for comparing imaging modalities and adapted to compare a standalone algorithm with a population of readers; Digital Pathology describes a similar comparison in CAMELYON16, and Statistical Significance in Evaluation covers the general methods.

The reader study of McKinney et al. (2020) shows how such a design is built and what it measures. An external clinical research organization ran it; six US board-certified radiologists, two of them breast imaging specialists, read 500 mammograms sampled from the US test set, with patient age, history and prior examinations where available. The sample was enriched by design: 125 cases with biopsy-proven cancer within 27 months, 125 with a negative biopsy and 250 not biopsied, chosen, the authors write, to increase the difficulty of the task and the statistical power; 465 cases met the inclusion criteria. Readers gave BI-RADS scores, which were turned into a six-point suspicion scale to fit ROC curves. The system’s AUC was 0.740 (95% CI 0.696–0.794) against a mean radiologist AUC of 0.625 (SD 0.032), a difference of 0.115 (95% CI 0.055–0.175). Since the sample was made difficult on purpose and scored against a long follow-up, these AUCs describe this case mix, not screening performance; this is our reading of the design. On the full test sets, the system was compared with the historical decisions of the original readers: on the UK test set (25,856 women screened at two English centres in 2012–2015, a random sample of 10%, 414 with cancer), it showed an absolute specificity improvement of 1.2% and was non-inferior in sensitivity compared with the first reader, and non-inferior to the second reader and to the consensus decision; on the US test set (3,097 women from one academic medical centre, 2001–2018, enriched with all biopsied women and weighted to correct for the sampling), it improved specificity by 5.7% and sensitivity by 9.4% compared with the original radiologists’ readings. All of this is retrospective; the authors write that prospective studies will be required to understand how such uses benefit patient care.

Claims of radiologist-level performance

“Radiologist-level” means as much as the comparison behind it. CheXNet’s title claims radiologist-level pneumonia detection. Its comparison used 420 frontal chest radiographs from ChestX-ray14, each labelled by four Stanford radiologists; each radiologist and the model were scored by F1 against each of the other four labellers in turn, and the model’s mean F1 of 0.435 (95% CI 0.387–0.481) exceeded the radiologists’ average of 0.387 (0.330–0.442) (Rajpurkar et al., 2017). The authors list as limitations that only frontal views were shown and that neither model nor radiologists had the patient history. The training labels came from report mining (Wang et al., 2017). CheXpert’s comparison is narrower and more careful in its reference: on Cardiomegaly, Edema and Pleural Effusion, the model achieved higher performance than all three comparison radiologists but not their majority vote, judged by where the radiologists’ operating points lay relative to the model’s ROC and precision–recall curves; on Consolidation it exceeded two of three, and on Atelectasis all three radiologists were better (Irvin et al., 2019). Both are retrospective comparisons, and both papers state their claims within those set-ups.

External validation and prospective studies

Internal test sets, drawn from the same hospitals and period as the training data, share their acquisition protocols, populations and labelling habits. External validation tests on data from elsewhere: McKinney et al. (2020) trained a version of their system on UK data only and tested it on the US data, where it still exceeded the US radiologists’ specificity and sensitivity at a threshold chosen on separate US validation data. External validation is necessary but not sufficient: DeGrave et al. (2021) argue from their results that external evaluation is insufficient to show that a model relies on medically relevant features, because shortcuts may not impair performance in new hospitals.

Prospective studies apply the system to cases as they arrive, and randomized trials compare workflows with and without it. MASAI is a randomized, controlled, population-based trial at four screening sites in Sweden (Lång et al., 2023). Women aged 40–80 who did not opt out were randomly assigned by the PACS, after image acquisition, to AI-supported screening or to standard double reading without AI. In the intervention arm, an AI system (Transpara version 1.7.0) gave each examination a malignancy risk score from 1 to 10; examinations scoring 1–9 were read by one radiologist and those scoring 10 by two, and readers saw the risk scores and, for scores 8–10, the system’s detection marks. The participants and radiographers, but not the radiologists, were masked to the allocation. The 2023 report is the prespecified clinical safety analysis after 80,033 women were randomized (April 2021 to July 2022). Cancer detection rates were 6.1 per 1,000 screened (95% CI 5.4–6.9) with AI support and 5.1 (4.4–5.8) without, a ratio of 1.2 (95% CI 1.0–1.5, p = 0.052); recall rates were 2.2% and 2.0%, the false positive rate was 1.5% in both arms, the positive predictive value of recall was 28.3% and 24.8%, and the screen-reading workload was 44.3% lower with AI. The trial’s primary endpoint, the interval cancer rate, was to be assessed after 100,000 participants and two years of follow-up, and is not part of this report. The outcomes MASAI reports, detection rate, recall rate and PPV of recall in a screened population, depend on prevalence, so it follows that an enriched retrospective study cannot estimate them directly; the example below shows why.

Prevalence, enrichment and predictive values

Sensitivity and specificity are measured within the positive and the negative cases separately, so they carry over from an enriched test set to a screening population if the cases are otherwise similar. Positive predictive value does not, because it mixes both groups (Precision and Recall explains why). By Bayes’ rule, a test with sensitivity $\mathrm{Se}$ and specificity $\mathrm{Sp}$ applied at prevalence $\pi$ has

$$\mathrm{PPV} = \frac{\mathrm{Se}\,\pi}{\mathrm{Se}\,\pi + (1 - \mathrm{Sp})(1 - \pi)}.$$

When $\pi$ is small, the false-positive term dominates the denominator unless $1 - \mathrm{Sp}$ is of the order of $\pi$ or smaller, so at screening prevalence a useful PPV requires a very high specificity. McKinney et al. (2020) give a real-world reference point: 22.8% of UK examinations recalled by consensus double reading and 4.87% of US examinations recalled by single reading were ultimately diagnosed with cancer in their data.

Example: from an Enriched Test Set to a Screening Population

The script below trains a lesion classifier on synthetic radiograph-like patches, tests it on an enriched set of the kind a reader study uses, and then applies it to a large screening-like sample at a prevalence of 0.5%. The patches are 32 × 32 pixels of smooth soft-tissue texture with rib-like bands, detector noise and, in positives, a faint round blob of random size and position. The 0.5% prevalence, 5 lesions per 1,000 images, is our choice, of the same order as the cancer detection rates per 1,000 screened in MASAI; the lesion contrast was set so that the classifier is good but imperfect, with an AUC around 0.9. The classifier is a logistic regression on a few features: a difference-of-Gaussians blob filter (built from Gaussian filters), the maximum response in each of 64 cells, and the cells sorted by response so that the features do not depend on where the lesion is. Two thresholds are chosen on an enriched validation set, never on test data: one for 90% sensitivity and one for 98% specificity. For each, the script prints sensitivity, specificity and PPV on the enriched test set, the PPV that Bayes’ rule predicts at 0.5% prevalence from those test values, and what the threshold actually does on the screening sample, whose number of lesions is printed per seed: sensitivity, specificity, PPV, images flagged per 1,000 and lesions found per 1,000. It loops over seeds 0–4 and runs in about 15 seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2.

import numpy as np
from scipy import ndimage
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

H = 32  # patch side in pixels

def radiographs(n, prevalence, rng):
    """Synthetic radiograph-like patches; label 1 means a faint round lesion is present."""
    y = (rng.random(n) < prevalence).astype(int)
    img = 3 * ndimage.gaussian_filter(rng.normal(size=(n, H, H)), (0, 3, 3))  # soft-tissue texture
    rows = np.arange(H)[None, :, None]
    img += 0.25 * np.sin(2 * np.pi * rows / rng.uniform(7, 11, (n, 1, 1))     # rib-like bands
                         + rng.uniform(0, 2 * np.pi, (n, 1, 1)))
    yy, xx = np.mgrid[:H, :H]
    for i in np.flatnonzero(y):                                               # lesion: Gaussian blob
        cy, cx = rng.uniform(8, H - 8, 2)
        s = rng.uniform(1.5, 3.0)
        img[i] += 1.5 * np.exp(-((yy - cy) ** 2 + (xx - cx) ** 2) / (2 * s ** 2))
    img += rng.normal(scale=0.15, size=img.shape)                             # detector noise
    return img, y

def features(img):
    """Blob filter (difference of Gaussians), max over 4x4 cells, cells sorted by response."""
    dog = ndimage.gaussian_filter(img, (0, 1.5, 1.5)) - ndimage.gaussian_filter(img, (0, 5, 5))
    cells = dog.reshape(len(img), 8, 4, 8, 4).max(axis=(2, 4)).reshape(len(img), -1)
    return -np.sort(-cells, axis=1)

def rates(score, y, t):
    pos = score >= t
    return pos[y == 1].mean(), 1 - pos[y == 0].mean(), (pos & (y == 1)).sum() / pos.sum(), pos.mean()

def ppv_at(sens, spec, prev):  # Bayes' rule
    return sens * prev / (sens * prev + (1 - spec) * (1 - prev))

PREV = 0.005  # screening prevalence (our choice): 5 lesions per 1,000 images
rows = []
for seed in [0, 1, 2, 3, 4]:
    rng = np.random.default_rng(seed)
    Xtr, ytr = radiographs(4000, 0.5, rng)    # enriched training set
    Xva, yva = radiographs(2000, 0.5, rng)    # enriched validation set: sets the thresholds
    Xte, yte = radiographs(2000, 0.5, rng)    # enriched test set, like a reader-study sample
    Xsc, ysc = radiographs(60000, PREV, rng)  # screening-like population
    clf = LogisticRegression(max_iter=2000).fit(features(Xtr), ytr)
    s_va, s_te, s_sc = (clf.decision_function(features(X)) for X in (Xva, Xte, Xsc))
    for name, t in [("sens 90%", np.quantile(s_va[yva == 1], 0.10)),   # operating points from
                    ("spec 98%", np.quantile(s_va[yva == 0], 0.98))]:  # validation data only
        se, sp, ppv_te, _ = rates(s_te, yte, t)
        se_s, sp_s, ppv_s, flag = rates(s_sc, ysc, t)
        rows.append((seed, name, roc_auc_score(yte, s_te), se, sp, ppv_te, ppv_at(se, sp, PREV),
                     se_s, sp_s, ppv_s, 1000 * flag, 1000 * se_s * ysc.mean(), int(ysc.sum())))

print("seed lesions threshold | enriched test: AUC  sens  spec  PPV | PPV at 0.5% |"
      " screening: sens  spec   PPV  flagged/1000 found/1000")
for r in rows:
    print(f"{r[0]:4d} {r[12]:7d} {r[1]} |              {r[2]:.3f} {r[3]:.2f} {r[4]:.3f} {r[5]:.2f} |"
          f"    {r[6]:.3f}    |       {r[7]:.2f} {r[8]:.3f} {r[9]:.3f}    {r[10]:5.1f}     {r[11]:4.2f}")

seed lesions threshold | enriched test: AUC  sens  spec  PPV | PPV at 0.5% | screening: sens  spec   PPV  flagged/1000 found/1000
   0     273 sens 90% |              0.914 0.90 0.705 0.75 |    0.015    |       0.88 0.710 0.014    292.9     4.02
   0     273 spec 98% |              0.914 0.56 0.977 0.96 |    0.110    |       0.56 0.978 0.105     24.4     2.55
   1     274 sens 90% |              0.921 0.90 0.722 0.75 |    0.016    |       0.89 0.731 0.015    272.2     4.08
   1     274 spec 98% |              0.921 0.61 0.979 0.96 |    0.126    |       0.57 0.977 0.102     25.5     2.62
   2     287 sens 90% |              0.929 0.90 0.802 0.81 |    0.022    |       0.89 0.771 0.018    232.3     4.23
   2     287 spec 98% |              0.929 0.61 0.979 0.97 |    0.130    |       0.57 0.978 0.113     24.2     2.73
   3     306 sens 90% |              0.921 0.88 0.773 0.79 |    0.019    |       0.89 0.757 0.018    246.2     4.55
   3     306 spec 98% |              0.921 0.60 0.976 0.96 |    0.113    |       0.59 0.976 0.110     27.4     3.02
   4     279 sens 90% |              0.917 0.92 0.716 0.77 |    0.016    |       0.92 0.724 0.015    278.8     4.30
   4     279 spec 98% |              0.917 0.57 0.979 0.96 |    0.119    |       0.58 0.979 0.114     23.9     2.72

On the enriched test set, both operating points look useful: 75–81% of positive calls are right at the high-sensitivity threshold, and 96–97% at the high-specificity one. On the screening samples the same thresholds give PPVs of 1.4–1.8% and 10–11%. Sensitivity and specificity carry over approximately, as they should, since each is computed within one class (the largest differences, about 0.04 in sensitivity and 0.03 in specificity, are of the size a test set of about 1,000 images per class allows); the PPV collapses because almost all images are negative, and even the roughly 2% of negatives flagged at the strict threshold outnumber the lesions it finds about eight to one. Bayes’ rule applied to the enriched test’s sensitivity and specificity predicts the screening PPV to within a few percentage points, but it is higher in every row. Two things visible in the output account for the gap. Four of the five screening samples contain fewer than the expected 300 lesions (273–287, a realized prevalence of 0.46–0.48%), which lowers their PPV below the value at exactly 0.5%. And the test-set estimates are themselves noisy, which matters most where the sample has the expected number of lesions or more: on seed 3, with 306 lesions, the prediction is still slightly high, and on seed 2 the high-sensitivity threshold’s specificity is 0.802 on the test set but 0.771 on the screening sample. On seed 1’s strict threshold both act: a test sensitivity of 0.61 against 0.57 on the screening sample, together with that sample’s 274 lesions, takes the PPV from a predicted 0.126 to a measured 0.102. At low prevalence PPV depends so strongly on specificity that a test set of about 1,000 negatives cannot fix it precisely.

The flagged and found columns are the screening program’s view. The 90%-sensitivity threshold finds 88–92% of the lesions (4.0–4.6 per 1,000 images) but flags 230–290 images per 1,000, far above the recall rates of 2.0–2.2% reported in MASAI; the 98%-specificity threshold flags 24–27 per 1,000, a rate of the same order, but finds only 56–59% of the lesions (2.5–3.0 per 1,000). Choosing between them, or anything in between, is a decision about what the program values: missed lesions against recalls and reading time. The AUC, 0.91–0.93 over the seeds and the same for both thresholds, says nothing about that choice. The synthetic lesions differ from real findings in every respect except the arithmetic, and the arithmetic holds for any classifier.

Challenges

• Labels. Expert labels are scarce and disagree (LIDC-IDRI above), and report-mined labels depend on how each institution’s radiologists phrase uncertainty: Zech et al. (2018) attribute part of Mount Sinai’s high pneumonia prevalence to NLP labels assigned more liberally there than at the other sites.
• Case mix. McKinney et al. (2020) note that most earlier mammography studies evaluated on small, enriched datasets with limited follow-up; predictive values measured on such sets do not carry over to a screening population (the example above).
• Context. Radiologists read with the clinical question, the history and prior examinations; an evaluation that withholds them, as CheXNet’s did, tests a narrower task (Rajpurkar et al., 2017).
• Volume and multi-image inputs. CT and MRI volumes, multi-view and multi-contrast studies and ultrasound video strain memory and require explicit decisions about the unit of prediction.

Failure Modes and Risks

Shortcut learning

A model trained to predict a label learns whatever in the image predicts it, including features that have nothing to do with the disease. In radiology these shortcuts come from how images are acquired and collected. Zech et al. (2018), in a retrospective cross-sectional study, trained pneumonia classifiers on 158,323 chest radiographs from three hospital systems: the NIH Clinical Center (112,120 radiographs), Mount Sinai Hospital (42,396) and the Indiana University Network for Patient Care (3,807). Pneumonia prevalence was 34.2% at Mount Sinai and about 1% at the other two, so sorting radiographs by hospital alone achieved an AUC of 0.861 on the joint Mount Sinai–NIH data. A network trained to identify the hospital system was correct for 99.95% of NIH and 99.98% of Mount Sinai test radiographs, using features from many image regions, with metal laterality tokens placed by technologists particularly influential. Models performed worse on external than on internal data in three of five comparisons; the model trained on pooled Mount Sinai and NIH data reached an internal AUC of 0.931 but 0.815 on Indiana. When the authors engineered training cohorts in which one site had ten times the pneumonia rate of the other, internal performance rose, but the gain did not carry over to the external Indiana data. They also suggest that a model could learn that portable radiographs, taken of inpatients too ill to travel, are more likely to show pneumonia.

DeGrave et al. (2021) examined COVID-19 detection models trained, as several published ones were, on positive and negative radiographs from different sources. Their dataset I combined COVID-19-positive images from a public collection of images from publications and other online sources with negative images from ChestX-ray14; dataset II drew both classes from two repositories from a single region, published by a shared research team (BIMCV-COVID-19+ and PadChest). Models reached ROC AUCs of 0.992 ± 0.006 and 0.995 ± 0.003 on held-out images from their own dataset, but 0.76 ± 0.04 and 0.70 ± 0.05 on the other dataset. Saliency maps and generative models pointed to laterality markers, text and the radiopacity of image borders, and swapping laterality markers between images changed the predictions. The authors describe such training data as having the potential for near worst-case confounding, and conclude that even datasets drawn from more similar sources, such as hospitals in one region, may be confounded. Interpretability of Neural Networks discusses what saliency maps can and cannot show, and Dataset Bias the general problem.

Dataset shift

A model meets different scanners, protocols, populations and disease prevalence when it moves between sites or over time. Prevalence shift alone changes predictive values (the example above), and calibration fails with it: Zech et al. (2018) found that a model trained on NIH data was poorly calibrated on Mount Sinai data and vice versa (see Expected Calibration Error for how calibration is measured). Shifts in acquisition change the images themselves. External validation detects some shift but, as DeGrave et al. (2021) point out, shortcuts may not impair performance at a new site, so good external results do not rule them out.

Regulation

Regulatory status decides whether a system may be marketed for clinical use. In the United States, the FDA regulates radiological image analysis software as medical devices. The summary below is taken only from final orders published in the Federal Register, as of October 2026; it draws no legal conclusions. The orders name four device types under 21 CFR part 892, which a 2026 order groups as CADx, CADe, CADt and CADe/x.

• Medical image analyzers (CADe), 21 CFR 892.2070. By a final order of 22 January 2020, effective 21 February 2020, the FDA reclassified medical image analyzers for mammography breast cancer, ultrasound breast lesions, radiograph lung nodules and radiograph dental caries detection from class III, the class requiring premarket approval, to class II with special controls, subject to premarket notification (510(k)). The device type is identified as one that marks, highlights or otherwise directs clinicians’ attention to portions of a radiology image that may reveal abnormalities, is not intended to replace review by a qualified radiologist, and is not intended for triage or to recommend a diagnosis. Its special controls require performance testing, by standalone tests, side-by-side comparisons or a reader study, showing that the device improves reader performance in the intended use population, with enough cases from important cohorts, such as subsets defined by confounders and by image acquisition characteristics, to characterize performance in each (Food and Drug Administration, 2020, reclassification order).
• CADx for lesions suspicious for cancer, 21 CFR 892.2060. Classified into class II on 19 July 2017 on a De Novo request for QuantX, and codified in January 2020: software that characterizes lesions identified on images such as MRI, mammography, radiography or CT as suspicious for cancer, with diagnostic and management decisions made by the clinical user (Food and Drug Administration, 2020, order on lesions suspicious for cancer).
• Computer aided triage and notification (CADt), 21 CFR 892.2080. Classified into class II on 13 February 2018 on a De Novo request for Viz.ai’s ContaCT, and codified in January 2020, with the definition quoted under triage above. Its special controls require performance testing showing that the device provides effective triage, for example improved time to review of prioritized images (Food and Drug Administration, 2020, triage and notification order).
• Computer-assisted detection and diagnosis (CADe/x), 21 CFR 892.2090. Classified into class II on 24 May 2018 on a De Novo request for OsteoDetect, and codified on 13 June 2025: software that aids the detection, localization and characterization of fractures, lesions or other disease-specific findings on images such as radiographs, MRI and CT, informing decisions made by the clinical user and not intended to replace a complete clinician’s review (Food and Drug Administration, 2025).

The De Novo orders explain the mechanism: a device not in commercial distribution before 28 May 1976 is automatically in class III, and De Novo classification can place it in class I or II, after which it can serve as a predicate so that later devices of the same type use the 510(k) process (Food and Drug Administration, 2020, triage and notification order). On 22 October 2025 the FDA received a petition asking for a partial exemption of devices of all four types from premarket notification under certain conditions, such as a manufacturer’s earlier clearance and a post-market plan. The FDA denied it on 1 April 2026 and published its final determination on 17 September 2026: manufacturers must continue to submit, and receive clearance of, a 510(k) before marketing these devices (Food and Drug Administration, 2026).

In the European Union, the Artificial Intelligence Act, Regulation (EU) 2024/1689, classifies an AI system as high-risk when it is a product, or the safety component of a product, covered by the legislation listed in its Annex I and that product must undergo a third-party conformity assessment (Article 6(1)); Annex I lists the Medical Devices Regulation, Regulation (EU) 2017/745. Its recitals note that more than one piece of Union harmonisation legislation, such as Regulation (EU) 2017/745, may apply to one product. This summary reflects the text of the Regulation as published on 12 July 2024.

Deployment Considerations

• Intended use. The regulatory definitions above tie each device type to a role: directing attention, characterizing lesions, notifying clinicians, or detecting and characterizing findings. A system evaluated and cleared for one role, such as notification in parallel with the standard of care, has not been shown to work in another, such as removing cases from a worklist.
• Local operating points. Thresholds chosen on one site’s validation data may not suit another site’s prevalence and equipment; the example above shows how strongly predictive values and workload depend on prevalence. The special controls for CADe and CADt ask for performance by subgroups defined by acquisition characteristics and confounders (the 2020 reclassification and triage and notification orders), the information a deploying site needs to judge transfer.
• Monitoring. Scanner replacements, protocol changes and changes in the referred population shift the inputs after deployment. It follows from the evidence on dataset shift that performance measured before deployment is not a guarantee after it, which argues for ongoing comparison with outcomes.
• Workflow effects. Standalone metrics do not measure the combined performance of readers and system or its effect on reading time and recalls. The CAD experience in mammography (Fenton et al., 2007; Lehman et al., 2015) and the workflow outcomes MASAI reports (Lång et al., 2023) are measurements of that combination.
• Data governance. Clinical images and reports are protected health data. Public datasets such as MIMIC-CXR are de-identified and released only under data use agreements, and much clinical data cannot be released at all: Zech et al. (2018) state that their Mount Sinai data cannot be released under the terms of their institutional review board approval.