Medical Computer Vision

Medical computer vision applies the methods of computer vision to images of the body: radiographs, computed tomography and magnetic resonance volumes, ultrasound video, scanned microscope slides and photographs of the eye, skin or the inside of the body. It is also called medical image analysis or medical image computing. The images are made to answer clinical questions, and the outputs of vision on them are measurements, delineations, alignments, flags and probabilities that a clinician or a researcher acts on. Most of its methods come from general vision, with convolutional networks and U-Net-style encoder–decoders at the centre, but the data differ from photographs in almost every respect that matters for learning: how the images are formed, how many labelled examples exist, who may see them, how much they change from one hospital to the next, and what counts as the right answer.

This article is the hub for the medical side of the category. It surveys the modalities, the tasks and the methods in a sentence or two each and links to the articles that treat them, and gives its own space to the problems they share: scarce and disputed labels, privacy, shift between scanners and sites, evaluation and validation, regulation and deployment in outline, and the failure modes that the literature documents. It describes methods, evidence and limits for practitioners and researchers. It gives no diagnostic or clinical guidance, draws no clinical conclusions of its own, and attributes every reported result to its authors with the design behind it.

Problem

Clinical imaging produces images faster than they can be studied by eye in detail. Radiologists read examinations and write reports, pathologists examine stained tissue sections, and both search for findings, grade them and compare them with earlier examinations. Two features of this work invite automation. The first is volume: Litjens et al. (2017) note that picture archiving and communication systems (PACS) had been routine in most Western hospitals for at least a decade and held millions of images, and that few other domains have this much image data acquired for specific purposes in well-structured archives. The second is variability: the same image is read differently by different experts, so a reproducible measurement has value in itself.

Vision can enter at every stage. It can form the image from the scanner’s raw measurements, align images with one another, outline organs and lesions so they can be measured, detect and classify findings, reorder the reading queue, predict a label for a whole slide or patient, and draft text for the report. Each use asks a different question of the evidence, and the way a system is meant to be used decides both how it should be evaluated and how regulators classify it (Computer Vision in Radiology works this through for radiology).

History

The history below is an outline; the linked articles give each strand with its sources.

• Images of the inside of the body. Wilhelm Röntgen found a penetrating, previously unknown radiation in 1895 while studying cathode rays, and received the first Nobel Prize in Physics in 1901 for it (NobelPrize.org, Röntgen facts). The first patient was scanned with Godfrey Hounsfield’s clinical CT prototype in 1972 (Hounsfield, 1979), and in 1973 Paul Lauterbur described how adding magnetic field gradients made it possible to image, by nuclear magnetic resonance, a cross-section of tubes of ordinary water surrounded by heavy water (Lauterbur, 1973; Nobel press release, 2003). Hounsfield shared the 1979 Nobel Prize in Physiology or Medicine with Allan Cormack, and Lauterbur the 2003 prize with Peter Mansfield (NobelPrize.org, 1979; Nobel press release, 2003).
• A common format. With the introduction of CT and other digital diagnostic imaging modalities in the 1970s, the American College of Radiology and the National Electrical Manufacturers Association formed a joint committee in 1983 to standardize the transfer of images between vendors’ devices; its ACR-NEMA standard (1985, revised in 1988) was replaced in 1993 by Digital Imaging and Communications in Medicine (DICOM), which added networking and a file format for offline media (NEMA, 2026, PS3.1, Section 1.3). DICOM was extended to whole-slide microscopy images in 2010 (NEMA, 2010).
• Rule-based, then feature-based analysis. Litjens et al. (2017) describe medical image analysis from the 1970s to the 1990s as rule-based systems chaining low-level pixel processing with mathematical modelling, and the late 1990s as the time supervised methods became increasingly popular, with features still designed by hand. They note that a convolutional network was applied to medical images, for lung nodule detection, as early as 1995, by Lo et al.
• Deep learning. Litjens et al. (2017) date the turn to AlexNet’s ImageNet result of December 2012, with medical papers growing rapidly in 2015 and 2016. U-Net, designed for biomedical segmentation with few annotated images, appeared at MICCAI 2015 (Ronneberger et al., 2015). Large public chest radiograph datasets with comparisons against radiologists, and challenges on whole-slide images, followed; the radiology and digital pathology overviews give their dates and designs.
• Configuration, scale and regulation. The Medical Segmentation Decathlon of 2018 asked for one method that would work unchanged on ten tasks, and was won by nnU-Net, which configures a plain U-Net from the data (Antonelli et al., 2022; Isensee et al., 2021). The US Food and Drug Administration created new device types for radiological and pathology software from 2017 (see Regulation and Deployment). From 2023, several groups pretrained large self-supervised encoders on 100,000 to more than a million pathology slides (Digital Pathology), and MedSAM adapted the Segment Anything Model to medical images (Ma et al., 2024).

Visual Data

Medical images come from different physics, and each brings its own intensity scale, resolution, artifacts and conventions. The sensor articles give the physics; this section says what a vision practitioner meets.

• Projection radiography and mammography. X-rays pass through the body to a detector, and the image records the “shadows” of the structures along each path (NIBIB, Medical X-rays), so structures at different depths overlap. Chest radiographs, the subject of most large public datasets, and mammograms, read in screening programs with several views and prior examinations, are described in Computer Vision in Radiology; X-ray radiography, chest X-ray and mammography are named here as subjects in their own right.
• Computed tomography (CT). Computed Tomography explains how a scanner measures X-ray attenuation along many lines through a slice and computes cross-sections and volumes from them. Its values are calibrated in Hounsfield units relative to water and air, so intensities have a physical meaning that camera pixels lack, although the same tissue can give different values on different scanners and X-ray spectra; the article covers Hounsfield units and windowing, scanner geometries, artifacts and dose.
• Magnetic resonance imaging (MRI). Magnetic Resonance Imaging explains how the scanner samples the Fourier transform of the body’s magnetization (k-space) and how sequence timing produces T1-weighted, T2-weighted and many other contrasts of the same anatomy. Its intensities are not calibrated, and one examination contains several contrasts; the article also covers the DICOM and NIfTI formats in which volumes reach vision code.
• Ultrasound. A transducer sends sound pulses and records their echoes, and the scanner computes distances to tissue boundaries from the echo times (NIBIB, Ultrasound). It is the radiological modality that is natively video. Ultrasound, nuclear imaging such as positron emission tomography (PET), and cardiac imaging are named here; the radiology overview describes echocardiography video as an example.
• Microscopy and pathology. Digital Pathology covers stained tissue slides scanned into whole-slide images of billions of pixels, stored as multi-resolution pyramids; color depends on the laboratory’s staining and the scanner. Whole-slide imaging as a sensor is named here. Microscopy of cells and tissues for biological research is the subject of Bioimage Analysis, which shares many methods with pathology.
• Photographs and video. Fundus photographs of the retina, dermatology photographs and endoscopy or surgical video are camera images, closer to ordinary photographs; ophthalmic, dermatological and surgical applications are named here. Litjens et al. (2017) cite a skin lesion classifier trained on dermoscopic and standard photographs from 18 public datasets, more than $10^5$ training images.

Three properties cut across modalities. Images are often 3D, with voxels that have a physical size, frequently larger between slices than within them, which Medical Image Segmentation shows changing both training and metrics. Images carry acquisition metadata (scanner, protocol, spacing, orientation) in their DICOM headers, which a pipeline needs and a model can exploit in unintended ways (see Failure Modes). And labels need clinical experts, or are mined from reports (see Data below).

Vision Tasks Involved

The tasks are those of general vision, adapted to the data and to the clinical question. Each has its own article.

• Reconstruction. Medical Image Reconstruction treats the computation of CT and MRI images from their raw measurements as an inverse problem: filtered back-projection, iterative and statistical methods, parallel imaging, compressed sensing and learned or unrolled networks, with benchmarks such as fastMRI. Fewer measurements mean a lower radiation dose in CT and a shorter scan in MRI, and learned methods raise a question the earlier ones did not: whether a plausible structure in the output was actually measured. Low-dose CT denoising is a related subject.
• Segmentation. Medical Image Segmentation assigns a label to every voxel, delineating organs, tumors and lesions so they can be measured. It covers 2D, 2.5D and 3D networks, losses for tiny targets, voxel spacing and anisotropy, inter-rater variation, overlap and surface-distance metrics with the Metrics Reloaded recommendations, and benchmarks such as the Medical Segmentation Decathlon and BraTS. nnU-Net is the framework that has become the common baseline: from a fingerprint of the training data it sets spacing, normalization, patch size and network depth for a plain U-Net and selects among 2D, 3D and cascade configurations by cross-validation. Whole-body CT models such as TotalSegmentator, which its authors trained with nnU-Net to segment 104 anatomical structures (Wasserthal et al., 2022), and organ, tumor and lesion segmentation tasks are named here.
• Registration. Medical Image Registration finds the spatial correspondence between images: a CT and an MRI of the same patient, a follow-up and its baseline, or a patient and an atlas. It covers rigid, affine and deformable transforms, mutual information for images of different modalities, diffeomorphic methods, atlas-based segmentation and learned registration, and the difficulty of knowing whether an alignment is right; image registration holds the general method.
• Detection, classification and triage. Computer-aided detection marks suspicious locations (object detection) and computer-aided diagnosis assigns a likelihood or category (image classification); triage reorders worklists. Computer Vision in Radiology covers these with their history, from the mammography systems of the 1990s and 2000s to deep learning on chest radiographs, and with quantification and report generation. Computer-aided detection and diagnosis is named here as a subject of its own.
• Cells, tissue and slides. In pathology, nuclei are segmented and classified, tissue regions are labelled tile by tile, and a whole slide’s label is predicted from thousands of tiles by multiple-instance learning (Digital Pathology); stain normalization and whole-slide image classification are named here. Cell Segmentation treats the instance segmentation of touching cells and nuclei in microscopy and histology, with StarDist, Cellpose and HoVer-Net described from their papers.

Scientific imaging shares several of these tasks: Bioimage Analysis applies segmentation, tracking and measurement to microscopy for biology, and Remote Sensing and Earth Observation meets a related version of the calibration and shift problems below, with satellites in place of scanners.

Common Methods

Most methods are general vision methods with medical adaptations.

• Encoder–decoders for dense prediction. The U-Net family is the usual choice for segmentation, in 2D and 3D, and nnU-Net’s authors argue that how a U-Net is configured matters more than architectural novelty (nnU-Net gives their evidence and its limits).
• Classifiers and detectors from natural images. Examination-level classifiers and detectors are convolutional networks or Transformers, often initialized by transfer learning from natural images, as the radiology overview describes.
• Weak labels. Labels mined from radiology or pathology reports are cheap but noisy and usually apply to the whole examination or slide; image-level classifiers and multiple-instance learning are built to use them.
• Self-supervised pretraining and foundation models. Unlabelled images are far more plentiful than labels, so self-supervised pretraining on large archives is used to obtain general encoders, the medical foundation models. Pathology foundation models are described in Digital Pathology, and MedSAM, a box-prompted 2D model trained on 1,570,263 image–mask pairs from 10 imaging modalities (Ma et al., 2024), in Medical Image Segmentation; medical foundation models in general are named here.
• Physics and priors in the loop. Unrolled reconstruction networks build the scanner’s forward model into their layers, and learned registration such as VoxelMorph is trained on an image similarity term plus a smoothness regularizer of the kind classical methods use; the two method articles describe how.

Data: Scarcity, Annotation and Privacy

Scarcity and labels

Litjens et al. (2017) argue that the familiar complaint about a lack of data is only partially correct: hospital archives hold millions of images, and the main challenge is to obtain relevant annotations for them. Turning free-text reports into labels needs text mining, and task-specific annotation by radiologists or pathologists takes much time, especially for 3D segmentation drawn slice by slice. Public datasets are accordingly small by the standards of natural images: the Medical Segmentation Decathlon’s left-atrium task had 20 training volumes (Antonelli et al., 2022). Litjens et al. also point to class imbalance (screening archives are mostly normal) and to heterogeneous “normal” classes that include rare benign findings and artifacts, on which systems trained as binary classifiers can fail.

Expert labels disagree. In LIDC-IDRI, four radiologists marked lung nodules on CT, reviewed each other’s marks and were not forced to agree; Litjens et al. (2017) note that the nodules they did not all agree on outnumbered those they did by a factor of three. Computer Vision in Radiology gives the counts from the dataset paper, which for lesions marked as nodules of at least 3 mm show a ratio nearer two to one. The reference standard is therefore a choice: one expert, a consensus panel, pathology confirmation, follow-up, or labels mined from reports, each with its own errors. Image annotation covers agreement measures and consensus labels in general.

Access is the other constraint. Many public medical datasets require registration, credentialing or a signed data use agreement, and much clinical data cannot be released at all: Zech et al. (2018) state that their Mount Sinai data cannot be released under the terms of their institutional review board approval. The linked articles name datasets with their official pages and stated terms.

Privacy and de-identification

Medical images are personal health data, and identifying information hides in more places than file names. DICOM’s Attribute Confidentiality Profiles, in Annex E of PS3.15, define how to create a de-identified copy of a dataset (NEMA, 2026, PS3.15). Their Basic Application Level Confidentiality Profile, intended for clinical trials, teaching files, publication and registries, removes information about the identity and demographics of the patient, responsible parties and family, the personnel and organizations involved, private attributes, and identifiers such as UIDs, dates and times that could match a copy to its original. Options then remove more or keep more:

• Clean Pixel Data removes identifying text burned into the pixels; the standard notes that CT images do not normally contain such text, whereas ultrasound images routinely do, and that the stored pixel values must be blacked out, not covered by an overlay.
• Clean Recognizable Visual Features distorts pixel data from which a person could be recognized; the standard notes the suggestion that a 3D rendering of high-resolution head and neck imaging may allow identification, and that this option can make the data unusable for its purpose.
• Retain options keep longitudinal dates (in full or modified), patient characteristics, device identity, institution identity, UIDs or safe private attributes that a specific use needs.

The standard is explicit that de-identifying the attributes does not de-identify the information object, and that the profiles are part of a de-identification process, not a replacement for one; such a process would at least involve determining the purpose and the recipients, interpreting the applicable regulations and assessing the risk of re-identification (NEMA, 2026, PS3.15, Annex E). Regulations differ by country. MIMIC-CXR, for example, de-identified its radiographs and reports to the HIPAA Safe Harbor standard of US law (Johnson et al., 2019).

Two consequences for vision follow from the profile, by our reading. Unless the Retain Device Identity and Retain Institution Identity options are used, de-identification removes attributes such as the station name, the device serial number and the institution name, which identify the individual scanner and site, so a dataset’s de-identification choices limit the analyses of shift it supports. The manufacturer and model name are not among the attributes that the profile’s table lists. And burned-in text and markers that survive de-identification are both a privacy risk and a source of shortcuts (see Failure Modes).

Domain Shift Between Scanners and Sites

A model trained on images from some hospitals is used on images from others, acquired with other scanners, protocols and populations. Castro et al. (2020) name the two obstacles to translating medical imaging models into practice as data scarcity and data mismatch, and classify the mismatch, dataset shift, by what changes between the development data and the deployment data, with sample selection as a related, separate mechanism:

• population shift: the characteristics of the patients, such as age, differ;
• acquisition shift: different scanners or imaging protocols, which they call one of the most notorious and well-studied sources of dataset shift in medical imaging;
• annotation shift: the same image would be labelled differently, for example by centres with different grading policies or readers of different experience;
• prevalence shift: the class balance differs;
• manifestation shift: the way a disease appears in the anatomy differs;
• sample selection (which they distinguish from dataset shift, and which can produce selection bias): inclusion criteria, quality control or training choices such as patch selection change which images are in the data.

They give a prostate MRI example: a model developed on a research dataset from a high-resolution 3 T scanner, with mostly younger patients, and deployed on conventional 1.5 T scanners meets both acquisition and population shift, and could learn that image quality predicts cancer risk. Dataset Bias covers the general vocabulary; domain shift in medical imaging is named here as a subject of its own.

Acquisition shift is often invisible to the eye and survives preprocessing. Glocker et al. (2019) built an age- and sex-matched set of 592 T1-weighted brain MRIs, 296 from each of two studies (Cam-CAN and UK Biobank) acquired on 3 T Siemens scanners with similar protocols, a case they describe as close to a best case for pooling. After skull stripping, bias-field correction, rigid registration and intensity whitening inside the brain, a random forest identified the study a scan came from with 96.96% accuracy in two-fold cross-validation (98.82% after affine registration), and site differences remained in derived gray-matter maps. In sex classification with rigid registration, accuracy was 81–84% within each study and 78% and 81% when training on one study and testing on the other; with affine registration it was 78–81% within each study and 62% and 74% across them. A dataset confounded by design (all females from one study, all males from the other) gave 94.59% with rigid registration, which the authors call overly optimistic. The authors conclude that multi-site harmonization remains an open challenge. This is a short workshop paper on one task and two studies; its value is in showing how easily a classifier finds the site.

Clinical studies show the same pattern. Zech et al. (2018) found that a network could identify the hospital system of a chest radiograph almost perfectly and that pneumonia models performed worse on external than on internal data in three of five comparisons; Computer Vision in Radiology gives the details. In PANDA, the specificity of a representative prostate grading algorithm fell from 92.9% internally to 75.2% and 84.3% on two external sets, which the authors attribute partly to distribution shift (Bulten et al., 2022; Digital Pathology).

The standard responses are spatial alignment and resampling to a common resolution, intensity normalization, and domain adaptation or harmonization methods (Castro et al., 2020). CT’s calibrated scale allows a fixed mapping, and nnU-Net clips and normalizes CT intensities with statistics from the whole training set but z-scores MRI and other images per image (nnU-Net); color deconvolution and stain normalization play this role in pathology (Digital Pathology). Castro et al. caution that domain adaptation may fail, or harm, when class prevalence also changes. The example below shows what per-image normalization does and does not repair.

Example: Training on One Scanner, Testing on Another

The script simulates two scanners imaging the same synthetic anatomy. Each anatomy image is a smooth elliptical “body” in arbitrary physical units, with faint texture; half of the images contain a small bright lesion. Scanner A maps the anatomy to intensities with a gain of 1000 and Gaussian noise; scanner B, like an MRI scanner from another vendor or protocol, uses a different gain (1600) and offset, twice the noise, a smooth multiplicative bias field across the image and a slight blur. Both apply a small random gain per image. A logistic regression is trained on scanner A images only, on four features computed inside a body mask found by Otsu thresholding: the strongest difference-of-Gaussians blob response, a high percentile of the smoothed intensity, and the mean and standard deviation of the intensity. It is then tested on the same test anatomy imaged by A and by B, once with raw intensities and once with per-image z-score normalization inside the mask, the kind of per-image scheme nnU-Net applies to MRI. The scanner settings were fixed before any run; the anatomy’s contrast, how far from the centre a lesion may lie and the mask erosion were corrected after the first two runs showed that the in-domain detector itself was not working (a lesion as faint as the texture, and a blob feature dominated by the body’s edge), and the third run is shown whatever it showed. The script loops over seeds 0–4, prints the area under the ROC curve, accuracy at the default probability threshold of 0.5, and on scanner B the sensitivity and specificity, then the range over seeds. It runs in about 16 seconds with NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2.

import numpy as np
from scipy import ndimage
from skimage.filters import threshold_otsu
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

H = 48  # image side in pixels
SCANNERS = {  # arbitrary intensity scale, as in MRI: gain, offset, noise, bias field, blur
    "A": dict(gain=1000, offset=0, noise=20, bias=0.0, blur=0.0),
    "B": dict(gain=1600, offset=200, noise=40, bias=0.15, blur=0.7),
}

def anatomy(n, rng):
    """Scanner-independent 'tissue' images in [0, 1]; label 1 = a small bright lesion."""
    y = (rng.random(n) < 0.5).astype(int)
    yy, xx = np.mgrid[:H, :H]
    z = np.zeros((n, H, H))
    for i in range(n):
        cy, cx = H / 2 + rng.uniform(-2, 2, 2)
        body = ((yy - cy) / 20) ** 2 + ((xx - cx) / 16) ** 2 <= 1
        tex = ndimage.gaussian_filter(rng.normal(size=(H, H)), 4)
        z[i] = body * (0.5 + 0.03 * tex / tex.std())
        if y[i]:
            r, a = rng.uniform(0, 6), rng.uniform(0, 2 * np.pi)
            ly, lx, s = cy + r * np.sin(a), cx + r * np.cos(a), rng.uniform(1.5, 3.0)
            z[i] += 0.08 * body * np.exp(-((yy - ly) ** 2 + (xx - lx) ** 2) / (2 * s ** 2))
    return z, y

def scan(z, p, rng):
    """Image the same anatomy with one scanner's settings."""
    out = np.empty_like(z)
    yy, xx = np.mgrid[:H, :H] / (H / 2) - 1
    for i, zi in enumerate(z):
        t = rng.uniform(0, 2 * np.pi)
        field = 1 + p["bias"] * (np.cos(t) * xx + np.sin(t) * yy)  # smooth multiplicative bias field
        img = p["gain"] * rng.lognormal(0, 0.05) * field * zi + p["offset"]
        if p["blur"]:
            img = ndimage.gaussian_filter(img, p["blur"])               # coarser resolution
        out[i] = img + rng.normal(scale=p["noise"], size=zi.shape)
    return out

def features(imgs, normalize):
    """Blob response and intensity statistics inside an Otsu body mask; optional z-scoring."""
    f = []
    for img in imgs:
        sm = ndimage.gaussian_filter(img, 1.5)
        mask = ndimage.binary_erosion(sm > threshold_otsu(sm), iterations=8)
        if normalize:  # per-image z-score inside the mask
            img = (img - img[mask].mean()) / img[mask].std()
            sm = ndimage.gaussian_filter(img, 1.5)
        dog = sm - ndimage.gaussian_filter(img, 5)
        f.append([dog[mask].max(), np.percentile(sm[mask], 99), img[mask].mean(), img[mask].std()])
    return np.array(f)

print("seed | normalization | AUC A->A  AUC A->B | acc A->A  acc A->B | sens A->B  spec A->B")
results = {False: [], True: []}
for seed in [0, 1, 2, 3, 4]:
    rng = np.random.default_rng(seed)
    z_tr, y_tr = anatomy(2000, rng)
    z_te, y_te = anatomy(2000, rng)
    tr = scan(z_tr, SCANNERS["A"], rng)
    te_a, te_b = scan(z_te, SCANNERS["A"], rng), scan(z_te, SCANNERS["B"], rng)  # same anatomy, two scanners
    for norm in (False, True):
        sc = StandardScaler().fit(features(tr, norm))
        clf = LogisticRegression(max_iter=1000).fit(sc.transform(features(tr, norm)), y_tr)
        row = []
        for te in (te_a, te_b):
            X = sc.transform(features(te, norm))
            pred = clf.predict(X)
            row.append((roc_auc_score(y_te, clf.decision_function(X)), (pred == y_te).mean(),
                        pred[y_te == 1].mean(), 1 - pred[y_te == 0].mean()))
        results[norm].append([row[0][0], row[1][0], row[0][1], row[1][1], row[1][2], row[1][3]])
        r = results[norm][-1]
        print(f"{seed:4d} | {'z-score' if norm else 'none':13s} |   {r[0]:.3f}     {r[1]:.3f}  |"
              f"   {r[2]:.3f}    {r[3]:.3f}  |   {r[4]:.3f}      {r[5]:.3f}")
for norm in (False, True):
    a = np.array(results[norm])
    print(f"range over seeds, {'z-score' if norm else 'none':7s}: " + ", ".join(
        f"{k} {lo:.3f}-{hi:.3f}" for k, lo, hi in zip(
            ["AUC A->A", "AUC A->B", "acc A->A", "acc A->B", "sens A->B", "spec A->B"],
            a.min(0), a.max(0))))

seed | normalization | AUC A->A  AUC A->B | acc A->A  acc A->B | sens A->B  spec A->B
   0 | none          |   0.761     0.648  |   0.689    0.494  |   1.000      0.000
   0 | z-score       |   0.753     0.634  |   0.688    0.567  |   0.784      0.355
   1 | none          |   0.734     0.634  |   0.671    0.500  |   1.000      0.000
   1 | z-score       |   0.723     0.610  |   0.653    0.565  |   0.791      0.338
   2 | none          |   0.716     0.623  |   0.658    0.505  |   1.000      0.000
   2 | z-score       |   0.713     0.609  |   0.648    0.576  |   0.757      0.391
   3 | none          |   0.737     0.626  |   0.686    0.514  |   1.000      0.000
   3 | z-score       |   0.732     0.606  |   0.657    0.577  |   0.783      0.357
   4 | none          |   0.751     0.640  |   0.676    0.502  |   1.000      0.000
   4 | z-score       |   0.743     0.618  |   0.682    0.582  |   0.763      0.399
range over seeds, none   : AUC A->A 0.716-0.761, AUC A->B 0.623-0.648, acc A->A 0.658-0.689, acc A->B 0.494-0.514, sens A->B 1.000-1.000, spec A->B 0.000-0.000
range over seeds, z-score: AUC A->A 0.713-0.753, AUC A->B 0.606-0.634, acc A->A 0.648-0.688, acc A->B 0.565-0.582, sens A->B 0.757-0.791, spec A->B 0.338-0.399

On its own scanner the detector is weak but clearly better than chance, with or without normalization. Moved to scanner B without normalization, it calls every image positive: its features are in scanner A’s intensity units, B’s larger gain and offset push every image past the learned decision boundary, and accuracy falls to chance. The AUC on B, 0.62–0.65, is well above that accuracy because AUC does not depend on the threshold: the ranking of images partly survives the change of scale, and the operating point does not. Per-image z-scoring removes the global gain and offset, and the classifier’s operating point on B moves back towards balance, with accuracy of 0.565–0.582 over the seeds. But it stays far from the in-domain accuracy, still errs heavily towards positive calls (specificity 0.34–0.40), and its AUC on B is no better than without normalization, slightly lower in every seed. That is what the design predicts, on our reading: z-scoring cannot undo the bias field, the blur or the noise, which relative to B’s gain is a quarter higher than A’s; these change the shape of the intensity distribution inside the body and the contrast of a small lesion against its surroundings. In-domain, z-scoring costs a little AUC in every seed, probably because it turns two of the four features, the mean and standard deviation inside the mask, into constants. The simulation is a caricature of a scanner, and the numbers belong to it; its lesson is consistent with Glocker et al.’s (2019) finding that scanner effects survived whitening inside the brain. By our reading, intensity normalization is needed when intensities have no fixed scale, but it is not a test of whether a model transfers; only evaluation on data from the new scanner is.

Evaluation and Validation

A reported result in medical imaging means little without its design. The radiology and pathology overviews work through reader studies, multiple-reader multiple-case statistics and the gap between enriched test sets and screening populations; this section summarizes the parts every medical task shares.

• The unit and the metric. Results may be per voxel, lesion, image, examination, slide or patient, and the same model can look very different at different units. The metric articles own their definitions: the Dice coefficient and surface distances for segmentation, ROC curves and AUC and precision and recall for detection and classification, calibration for probabilities. Metrics Reloaded is a consensus framework for choosing metrics from the properties of a problem (Maier-Hein et al., 2024), summarized in Medical Image Segmentation.
• The reference standard. Which labels the model is scored against (a single reader, a panel, pathology, follow-up, report mining) sets a ceiling on what the result can show, and the training labels may come from a weaker standard than the test labels.
• The comparison. A comparison with clinicians needs to say how many readers, of what experience, with what information and under what conditions read the cases. Nagendran et al. (2020), reviewing deep learning studies in medical imaging that compared algorithms with clinicians (2010 to June 2019), found a median of five clinicians in the comparison groups, of whom a median of four were experts, 58 of 81 non-randomized studies at high risk of bias, and 61 of 81 stating in their abstract that the AI performed at least as well as clinicians. “Expert-level” means as much as the comparison behind it.
• Internal, external and prospective validation. An internal test set shares the training data’s hospitals, scanners and labelling habits; external validation tests on data from elsewhere; prospective studies apply the system to cases as they arrive, and randomized trials compare workflows with and without it. In Nagendran et al.’s review, nine of 81 non-randomized studies were prospective. External validation is necessary but not sufficient: DeGrave et al. (2021) argue that shortcuts need not impair performance at a new hospital, so good external results do not prove that a model uses medically relevant features.
• Case mix and prevalence. Enriched test sets, which oversample positive and difficult cases, measure discrimination but not the predictive values a screening population would see; Computer Vision in Radiology shows the arithmetic in an example.
• Uncertainty and comparisons between methods. Small test sets make confidence intervals wide, and rankings change with the metric and the aggregation; Statistical Significance in Evaluation covers the general methods, and cross-validation is how frameworks such as nnU-Net estimate performance on small datasets. Clinical validation is named here as a subject of its own.

Reporting guidelines

Reporting guidelines list what a paper must state so that readers can judge its design. The official TRIPOD website describes TRIPOD+AI as a 27-item checklist that updates and replaces TRIPOD 2015, for studies that develop, validate or update a prediction model, whether by regression or machine learning and whether for diagnosis, prognosis, monitoring or screening; it was published in 2024 (TRIPOD website; Collins et al., 2024). Nagendran et al. (2020) assessed the studies in their review against a modified version of TRIPOD 2015 and found adherence below 50% for 12 of the 29 items they assessed. CONSORT-AI extends the CONSORT 2010 guideline for randomized trials with 14 items for AI interventions, developed with SPIRIT-AI for trial protocols through a Delphi survey and a consensus meeting (Liu et al., 2020). It asks trial reports to describe the algorithm version, how input data were acquired, selected and preprocessed, how poor-quality or unavailable input data were handled, the human–AI interaction and an analysis of errors. For medical imaging specifically, the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) was published in Radiology: Artificial Intelligence in 2020 (Mongan et al., 2020).

Regulation and Deployment

Regulatory status decides whether a system may be marketed for clinical use. This section is an outline taken from final orders published in the Federal Register, as of October 2026; the radiology and pathology articles give the details, and no legal conclusion is drawn here. In the United States, the FDA regulates image analysis software as medical devices. New kinds of device were placed in classes through De Novo requests, after which later devices of the same type can be cleared through premarket notification (510(k)): for radiology, device types for computer-assisted diagnosis, triage and notification, and detection and diagnosis software, classified from 2017 to 2018 (Computer Vision in Radiology); for pathology, whole-slide imaging systems in 2017 and software to assist users in digital pathology in 2021 (Digital Pathology). In January 2020, the FDA moved medical image analyzers for mammography, breast ultrasound, lung nodules on radiographs and dental caries from class III to class II with special controls (Food and Drug Administration, 2020). In September 2026 it published its denial of a petition for a partial exemption of the four radiological detection, diagnosis and triage device types from premarket notification, so a 510(k) must still be submitted and cleared before marketing them (Food and Drug Administration, 2026). The European Union’s rules, including the Artificial Intelligence Act’s treatment of AI systems in medical devices, are summarized in Computer Vision in Radiology.

Several of these orders write the problems of this article into their requirements. The special controls for medical image analyzers ask for test data with enough cases from important cohorts, including subsets defined by confounders and by image acquisition characteristics, to characterize performance in each (Food and Drug Administration, 2020). Those for digital pathology software ask for training data representing pre-analytical variables, multiple sites and demographics, an independent validation set, and performance stratified by confounders such as scanning equipment (Food and Drug Administration, 2023).

Deployment adds considerations that a benchmark does not measure:

• Intended use. A system evaluated for one role, such as notifying clinicians in parallel with routine reading, has not been shown to work in another, such as removing cases from a worklist.
• Integration. Images arrive through PACS and DICOM networks; DICOM was created so that devices from different vendors could exchange images and connect to PACS and hospital information systems (NEMA, 2026, PS3.1).
• Local checks and monitoring. Thresholds chosen at one site may not suit another’s prevalence and equipment, and scanner replacements, protocol changes and changes in the referred population shift the inputs after deployment. It follows from the evidence on domain shift that performance measured before deployment is not a guarantee after it.
• Workflow and people. Standalone accuracy does not measure what readers do with a system’s output; the observational studies of mammography CAD and the MASAI screening trial described in Computer Vision in Radiology measure that combination.
• Data governance. Training, validation and monitoring all need data that are de-identified, permitted for the purpose and documented, which the privacy section above describes.

Failure Modes and Risks

• Shortcut learning. A model learns whatever in the image predicts the label, including hospital-specific markers, text and acquisition details; Zech et al. (2018) suggest, for example, that portable radiographs, taken of patients too ill to travel, could act as a proxy for pneumonia. They found a network identifying the hospital system of chest radiographs almost perfectly, with metal markers placed by technologists among the influential features, and DeGrave et al. (2021) found COVID-19 detectors relying on laterality markers, text and image borders when the two classes came from different sources (Computer Vision in Radiology). Interpretability of Neural Networks discusses what saliency maps can show about such shortcuts.
• Acquisition and site shift. Scanner effects survive standard preprocessing (Glocker et al., 2019), and performance falls on external data in clinical studies (Zech et al., 2018; Bulten et al., 2022). The example above shows a classifier that fails silently on a new scanner even though its ranking of images partly survives.
• Confounded training sets. When a class comes mostly from one site, sex or scanner, the model can learn the confounder; Glocker et al.’s design with all females from one study and all males from the other inflated accuracy, and Castro et al. (2020) describe how acquisition and population shift together can teach a model that image quality predicts disease.
• Label noise and disputed references. Report-mined labels and single-reader references carry errors that a model learns and that an evaluation then scores against (Litjens et al., 2017).
• Rare and unseen findings. Systems trained on common classes can fail on rare benign findings and artifacts grouped into “normal” (Litjens et al., 2017), and a model trained to separate tumor from normal tissue may not recognize other pathologies (Digital Pathology).
• Hallucinated structure. Learned reconstruction and report generation can produce plausible content that was not measured or not present (Medical Image Reconstruction; the radiology overview for report generation).
• Metric and evaluation pitfalls. A segmentation that misses a lesion has no surface distance, so averages flatter it unless missed cases are scored, rankings change with the metric, and enriched test sets overstate predictive values (Medical Image Segmentation, Computer Vision in Radiology).
• Lost geometry. Code that keeps a voxel array and drops its header loses the spacing and orientation, so, by our reading, volumes, distances and the patient’s left and right can no longer be recovered from it; Magnetic Resonance Imaging describes the geometry that DICOM and NIfTI headers carry.
• Privacy leaks. Burned-in text, recognizable features in head imaging and private header attributes can identify a patient after a careless de-identification (NEMA, 2026, PS3.15, Annex E).