Medical Image Classification

Medical image classification assigns a category to a medical image or to a group of images: a finding present or absent on a chest radiograph, normal or abnormal for a set of musculoskeletal views, a disease grade for a retinal photograph, a diagnosis for a skin lesion or a brain MRI. It is image classification with the same networks and losses, but the data change what has to be decided before training starts: which unit the label belongs to, where the label came from, how rare the positive class is, how the images are split so that the test measures new patients, and which operating point and calibration the use requires.

Medical Computer Vision and Computer Vision in Radiology describe the problems that all medical tasks share: scarce and disputed labels, shortcut learning, dataset shift between scanners and sites, and the way predictive values depend on prevalence. This article takes them as given and treats the classification task itself. Chest X-ray analysis and whole-slide image classification, two settings with their own large datasets and methods, are named here as subjects of their own. Like its neighbours, this article describes methods, evidence and limits for practitioners and researchers; it gives no diagnostic or clinical guidance, and every reported result is attributed to its authors with its set-up.

Problem Definition

A classifier maps an input $x$ to scores $s_k(x)$ for classes $k = 1, \dots, K$, usually read as estimated probabilities. Three choices define a medical version of the task.

The unit of the label. A label can belong to one image, to a view, to a series or volume, to a study (one examination, possibly with several images), to a lesion photographed several times, or to a patient. These units nest, and a label is often recorded at a coarser unit than the input. In MURA, a dataset of upper-extremity radiographs, each study of one or more views was labelled normal or abnormal by a radiologist at the time of clinical interpretation, so the label belongs to the study, not to each image (Rajpurkar et al., 2018). In brain MRI classification of Alzheimer’s disease the diagnosis belongs to the subject, and a 2D network trained on slices inherits that label for every slice (Wen et al., 2020).

The label space. Binary tasks (abnormal against normal) are common, but many are multi-label, with several findings that can co-occur and each scored as its own binary problem; others are multi-class (seven diagnostic categories of pigmented skin lesions in HAM10000; Tschandl et al., 2018) or ordinal, as with the five-grade diabetic retinopathy scale that Raghu et al. (2019) collapse into a binary referable-or-not decision.

The claim. A model is almost always meant for patients it has never seen, often at hospitals it has never seen. The claim fixes the population the test set must represent, and therefore how the data are split (see Challenges and the example below).

Input and Output

Inputs range from single 2D images (radiographs, dermoscopic and fundus photographs), through multi-view studies and multi-contrast examinations, to 3D CT and MRI volumes and gigapixel whole-slide images. 2D images are usually resized to a network’s input size: Raghu et al. (2019) worked with $587 \times 587$ fundus photographs and CheXpert radiographs resized to $224 \times 224$, and MedMNIST v2 reduces every image to $28 \times 28$ or $28 \times 28 \times 28$ (Yang et al., 2023), a choice made for lightweight benchmarking rather than to preserve diagnostic detail. Volumes and slides do not fit a standard network at full resolution, so they are cut into slices, patches or tiles whose predictions are combined (see Approaches).

The output is a vector of scores. A decision needs a threshold on each score, chosen for an operating point: a sensitivity and specificity pair that suits the intended use. MURA’s ensemble, for instance, reached an AUROC of 0.929 with an operating point of 0.815 sensitivity and 0.887 specificity on its 207-study test set (Rajpurkar et al., 2018). Whether the scores can also be read as probabilities is the separate question of calibration (see Evaluation Metrics).

Variants

• Image-level classification. One image, one label: a dermoscopic image of one lesion, a fundus photograph, a histology tile.
• Study-level classification from several images. MURA’s baseline applies a 169-layer DenseNet to each view and averages the per-view probabilities to score the study (Rajpurkar et al., 2018). Mammography with its four standard views and priors, and chest radiographs with frontal and lateral views, pose the same problem; Computer Vision in Radiology describes how models combine views.
• Volume classification. Wen et al. (2020) group the CNN approaches to Alzheimer’s disease classification from T1-weighted MRI into four types: 2D slice-level, 3D patch-level, region-of-interest-based and 3D subject-level networks. In their own comparison, the three 3D approaches performed similarly, the 2D slice approach was lower, and none of the CNNs did better than a linear support vector machine on voxel-based gray-matter features.
• Slide-level classification. A whole-slide image is classified from thousands of tiles by multiple-instance learning, as Digital Pathology outlines; whole-slide image classification is the subject in its own right.
• Multi-label findings. Chest radiograph label sets list several findings, each scored as its own binary problem (five CheXpert findings in Raghu et al., 2019); chest X-ray analysis holds those label sets and their datasets.

Challenges

Labels

Labels come from different sources with different errors: a radiologist’s reading at the time of the examination (MURA), labels mined from free-text reports by natural language processing, a consensus of several experts, histopathology, or follow-up. In HAM10000, more than half of the lesions were confirmed by pathology, and the ground truth for the rest came from follow-up, expert consensus or in-vivo confocal microscopy (Tschandl et al., 2018). Report-mined labels are cheap and plentiful but noisy, and they describe the examination rather than a location in it; Computer Vision in Radiology gives the precision and recall that ChestX-ray14’s labeller achieved and the uncertainty labels of CheXpert. A model trained on such labels learns to reproduce them, errors included, and a test set labelled the same way scores it against the same errors; a reference standard of higher quality for the test set is the usual remedy (CheXpert’s radiologist-labelled test set is one).

Class imbalance

Positive cases are rare in most clinical populations, and public datasets are imbalanced in their own ways: MURA has 9,045 normal and 5,818 abnormal studies (Rajpurkar et al., 2018), while 6,705 of HAM10000’s 10,015 images, in seven classes, show melanocytic nevi (Tschandl et al., 2018, Table 1). Imbalance affects training, because a network can reach a low loss by favouring the majority class, and evaluation, because accuracy then rewards the same behaviour. Buda et al. (2018) studied imbalance systematically for CNNs on MNIST, CIFAR-10 and ImageNet, not on medical images: they found it detrimental to classification, found oversampling the minority class to be the dominant remedy in almost all scenarios, recommended oversampling to the level that removes the imbalance entirely, and found that oversampling did not cause the CNNs to overfit, in contrast to some classical models. They recommend thresholding to compensate for prior class probabilities when the number of correctly classified cases is what matters.

Several images per patient

Medical datasets hold several images of most patients: slices and patches of one volume, several views of one study, repeat examinations over time, several photographs of one lesion. These images share the patient’s anatomy, the scanner, the acquisition protocol and the label, so a network can learn to recognize the patient and recall the label. If images of one patient fall on both sides of a split, the test measures that recall, not performance on new patients. Training, Validation, and Test Sets treats this group leakage in general, with Kapoor and Narayanan’s taxonomy; two medical studies show its size.

Wen et al. (2020) reviewed 32 studies that classified Alzheimer’s disease from anatomical MRI with CNNs. They labelled data leakage as clear in 6 and as unclear, for lack of explanation, in 10, which they summarized as about half of the surveyed studies possibly reporting biased results; the first cause they list is a split that is not made at the subject level, which happens when slices or patches are extracted from a 3D image or when a subject was scanned at several time points. In their own experiments with a 2D slice-level CNN (AD against cognitively normal subjects), a subject-level split gave a mean validation balanced accuracy of 0.79 over five folds, and a slice-level split gave 1.00 in every fold (Table 5 of the arXiv version). On their independent test sets the leaky model’s balanced accuracy was 0.75 on ADNI, 0.80 on AIBL and 0.68 on OASIS, close to the 0.76, 0.76 and 0.65 of the model trained with the subject-level split (Table 6): the perfect validation score said nothing about new subjects.

Yagis et al. (2021) measured the same effect directly, with three 2D CNNs (two VGG16 variants and ResNet-18) and nested five-fold cross-validation on four brain MRI datasets, for Alzheimer’s and Parkinson’s disease. On OASIS-200 (100 patients and 100 controls), mean slice-level test accuracy was 66.0–68.9% with a subject-level split and 94.2–99.0% with a slice-level split, a gap of 28–31 percentage points; on the smaller OASIS-34 the gap was 41–45 points (their Table 4; the abstract rounds the OASIS gap to 30%). With labels assigned to subjects at random, the slice-level split still gave 93.7–95.9% test accuracy (about 96% in the abstract) and the subject-level split 49.3–53.4%, the chance level of a balanced random labelling. The authors conclude that the effect is severe, especially for small datasets.

Shift and shortcuts

A classifier finds whatever in the image predicts the label, including the hospital, the scanner and markers placed by technologists, and its performance changes when the population, the acquisition or the prevalence changes. Computer Vision in Radiology gives the documented cases (Zech et al. on pneumonia across three hospital systems; DeGrave et al. on COVID-19 shortcuts) and Medical Computer Vision the taxonomy of dataset shift; both are the reason that a split by site, or a separate external test set, is needed in addition to a split by patient.

Approaches

Hand-crafted features and classical classifiers

Before deep learning, medical classifiers computed designed features (intensity statistics, texture descriptors, shape measures of a segmented region) and fed them to a classifier such as a support vector machine; Medical Computer Vision outlines that history. Such pipelines remain useful baselines: none of Wen et al.’s CNNs did better than their linear SVM (see Variants), a check that a deep model’s accuracy is not an artifact of the evaluation.

Transfer from natural images

The common recipe is to fine-tune a network pretrained on ImageNet, described in general in Transfer Learning. Tajbakhsh et al. (2016) compared fine-tuning with training from scratch on four applications in radiology, cardiology and gastroenterology (polyp detection, pulmonary embolism detection, colonoscopy frame classification and intima–media boundary segmentation) from three modalities. They reported that a pretrained CNN with adequate fine-tuning outperformed, or at worst matched, a CNN trained from scratch, that fine-tuned CNNs were more robust to small training sets, and that neither tuning only the last layers nor tuning all layers was best everywhere, so they proposed choosing the depth of fine-tuning layer by layer for the data at hand.

Raghu et al. (2019) questioned how much of the benefit came from reusing ImageNet features. They trained ResNet-50, Inception-v3 and a family of small convolution–batch-norm–ReLU networks (CBR, from one-third to one-twentieth of the size of the standard ImageNet models), each three times from random initialization and from ImageNet weights, on two large tasks: referable diabetic retinopathy from retinal fundus photographs (about two hundred thousand images, scored by AUC) and five findings in CheXpert. Transfer changed the retinal AUC by a fraction of a percentage point; Transfer Learning gives the numbers and the small-data experiment, in which transfer helped ResNet-50 but barely changed the small models. On CheXpert (their Table 2, ResNet-50 and four CBR models), transfer raised the edema AUC for every model, by 0.6 to 4.9 points; for the other four findings it changed per-finding AUCs by between about −2 and +2.6 points. The text summarizes this as transfer being slightly worse for atelectasis, cardiomegaly and consolidation and helping with edema and pleural effusion. The table agrees for consolidation (lower with transfer for four of the five models) but not for cardiomegaly, where transfer raised the AUC for all four CBR models and lowered it only for ResNet-50; and for ResNet-50, transfer was slightly better on atelectasis and slightly worse on pleural effusion. The small models, much worse than ResNet-50 on ImageNet, were about as good on both medical tasks, so ImageNet accuracy did not predict medical accuracy in this set-up. The authors trace part of the apparent benefit of transfer to the overparametrization of the standard models, locate meaningful feature reuse in the lowest layers, and show that pretrained weight scalings alone speed up convergence.

Pretraining on medical images

Unlabelled medical images are far more plentiful than labels, so self-supervised pretraining on them is the next step. Azizi et al. (2021) pretrained ResNets with SimCLR, a contrastive method, first on ImageNet and then on unlabelled images of the target domain, and fine-tuned on two tasks: classification of 27 skin conditions (26 common conditions and “other”) from consumer camera photographs collected by a US tele-dermatology service, labelled by a panel of dermatologists and split into development and test sets with no patient overlap; and five findings in CheXpert. They also introduced Multi-Instance Contrastive Learning (MICLe), which forms a positive pair from two different images of the same patient case instead of two augmentations of one image, so the patient structure that causes leakage in evaluation becomes a training signal in pretraining. Their best dermatology model (a ResNet-152 (2×), pretrained with SimCLR on ImageNet and then with MICLe on dermatology images) reached a top-1 accuracy of 70.02% against 63.36% for the same architecture with supervised ImageNet pretraining; on CheXpert the best mean AUC, from SimCLR pretraining on ImageNet and then CheXpert without MICLe, was 0.7729 against 0.7625 (their Tables 3 and 4). For CheXpert they re-split the original training set at random into 67,429 training, 22,240 validation and 33,745 test images because the official validation set is small, and they note that their numbers are therefore not comparable with the CheXpert paper or its leaderboard; the paper describes the re-split in images and does not say whether patients were kept apart. Pretrained encoders of this kind, trained at larger scale, are the medical foundation models, which Digital Pathology describes for pathology; medical foundation models in general are named here.

Combining several images

When the label belongs to a study, a volume or a slide, image-level predictions are combined, from MURA’s fixed average of per-view probabilities to the learned pooling of multiple-instance learning (see Variants); 3D convolutional networks take a volume directly when memory allows.

Handling imbalance

Imbalance is handled in the loss, in the sampling, or at the threshold. MURA’s baseline weights the binary cross-entropy loss by study type $T$, with the positive term weighted by the fraction of normal images and the negative term by the fraction of abnormal images,

$$L(X, y) = -w_{T,1}\, y \log p(Y = 1 \mid X) - w_{T,0}\, (1 - y) \log p(Y = 0 \mid X), \qquad w_{T,1} = \frac{|N_T|}{|A_T| + |N_T|}, \quad w_{T,0} = \frac{|A_T|}{|A_T| + |N_T|},$$

where $|A_T|$ and $|N_T|$ are the numbers of abnormal and normal training images of type $T$ (Rajpurkar et al., 2018). Oversampling the minority class, which Buda et al. (2018) found most effective in their experiments on natural images, achieves a similar balance through the data. Either way the trained model’s scores no longer reflect the training prevalence, so a threshold, and any probability that is reported, must be set afterwards on validation data (see Evaluation Metrics).

Example: Splitting by Image and by Patient

The script builds synthetic patients, each with eight $32 \times 32$ image patches. Every patient has an appearance shared by all of their images, as scans of one person on one scanner do: a brightness offset, a contrast and a texture scale drawn once per patient, with a fresh texture realization and noise in each image. Half of the 60 patients are labelled diseased, and in each of their images a faint Gaussian blob (amplitude 0.25 against a texture amplitude of about 0.3) appears with probability 0.7. A second run sets the blob amplitude to zero, so that the labels carry no image signal at all, as in the random-label experiment of Yagis et al. (2021). Four features per image (mean, standard deviation, a Laplacian-to-intensity ratio for texture and a normalized difference-of-Gaussians maximum for blobs) go into a random forest with a fixed seed. The out-of-fold scores of five-fold cross-validation are scored by ROC AUC twice: with folds drawn over images, as a naive split would, and with folds drawn over patients (scikit-learn’s GroupKFold), both per image and per patient (the mean of a patient’s image scores). All parameters were fixed before the script was first run and not changed afterwards. The script loops over seeds 0–4 and prints the range. It runs in about 10 seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2.

import numpy as np
from scipy import ndimage
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import KFold, GroupKFold
from sklearn.metrics import roc_auc_score

N_PAT, N_IMG, H = 60, 8, 32  # patients, images per patient, patch side in pixels

def make_patients(lesion_amp, rng):
    """Synthetic patches: every image of a patient shares that patient's appearance;
    the label belongs to the patient, and a lesion shows in 70% of a diseased patient's images."""
    y_pat = rng.permutation(np.repeat([0, 1], N_PAT // 2))
    yy, xx = np.mgrid[:H, :H]
    X, y, g = [], [], []
    for p in range(N_PAT):
        bright, contrast = rng.normal(0, 0.5), rng.lognormal(0, 0.2)   # patient-specific
        tex_sigma = rng.uniform(1, 4)                                 # patient-specific texture scale
        for _ in range(N_IMG):
            tex = ndimage.gaussian_filter(rng.normal(size=(H, H)), tex_sigma)
            img = bright + contrast * 0.3 * tex / tex.std()
            if y_pat[p] and rng.random() < 0.7:
                cy, cx = rng.uniform(6, H - 6, 2)
                img += lesion_amp * np.exp(-((yy - cy) ** 2 + (xx - cx) ** 2) / (2 * 2.0 ** 2))
            X.append(img + rng.normal(scale=0.1, size=(H, H)))
            y.append(y_pat[p]); g.append(p)
    return np.array(X), np.array(y), np.array(g)

def features(imgs):
    """Intensity, texture and blob features of each patch."""
    f = []
    for im in imgs:
        s = im.std()
        dog = ndimage.gaussian_filter(im, 1.5) - ndimage.gaussian_filter(im, 3.0)
        f.append([im.mean(), s, ndimage.laplace(im).std() / s, dog.max() / s])
    return np.array(f)

def oof_scores(F, y, splits):
    """Out-of-fold probability of 'diseased' for every image."""
    score = np.empty(len(y))
    for tr, te in splits:
        rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(F[tr], y[tr])
        score[te] = rf.predict_proba(F[te])[:, 1]
    return score

print("lesion  seed  image split  patient split  patient split, per patient")
summary = {}
for amp in (0.25, 0.0):
    for seed in range(5):
        rng = np.random.default_rng(seed)
        X, y, g = make_patients(amp, rng)
        F = features(X)
        s_img = oof_scores(F, y, KFold(5, shuffle=True, random_state=seed).split(F))
        g_perm = rng.permutation(N_PAT)[g]           # shuffle which patients share a fold
        s_pat = oof_scores(F, y, GroupKFold(5).split(F, y, g_perm))
        per_pat = [s_pat[g == p].mean() for p in range(N_PAT)]
        y_pat = [y[g == p][0] for p in range(N_PAT)]
        r = (roc_auc_score(y, s_img), roc_auc_score(y, s_pat), roc_auc_score(y_pat, per_pat))
        summary.setdefault(amp, []).append(r)
        print(f"{amp:5.2f}  {seed:4d}  {r[0]:11.3f}  {r[1]:13.3f}  {r[2]:13.3f}")
for amp, rs in summary.items():
    lo, hi = np.min(rs, axis=0), np.max(rs, axis=0)
    print(f"lesion {amp:.2f}, range over seeds 0-4: image split {lo[0]:.3f}-{hi[0]:.3f}, "
          f"patient split {lo[1]:.3f}-{hi[1]:.3f}, per patient {lo[2]:.3f}-{hi[2]:.3f}")

lesion  seed  image split  patient split  patient split, per patient
 0.25     0        0.921          0.512          0.528
 0.25     1        0.962          0.578          0.610
 0.25     2        0.916          0.537          0.548
 0.25     3        0.925          0.499          0.503
 0.25     4        0.864          0.491          0.486
 0.00     0        0.926          0.508          0.508
 0.00     1        0.967          0.572          0.596
 0.00     2        0.912          0.533          0.547
 0.00     3        0.941          0.513          0.518
 0.00     4        0.875          0.481          0.483
lesion 0.25, range over seeds 0-4: image split 0.864-0.962, patient split 0.491-0.578, per patient 0.486-0.610
lesion 0.00, range over seeds 0-4: image split 0.875-0.967, patient split 0.481-0.572, per patient 0.483-0.596

The image-level split reports AUCs near 0.9 in every run, and it reports them just as high when the images contain no lesion at all. That is the signature of leakage: with seven of each patient’s eight images in the training folds, the forest learns to recognize patients by their brightness, contrast and texture scale and returns the patient’s label, which is exactly what Yagis et al. (2021) saw with random labels. Split by patient, the same features and forest score near 0.5 in both runs, and averaging a patient’s images does not change that. So even with a lesion present, nothing the forest learned carries over to new patients; the output does not say why, and our reading, which the script does not test, is that a blob this faint does not stand out from each patient’s texture in these four features. The settings were not tuned to make the lesion visible. The leaky split would have reported a strong classifier, and only the patient split shows that it had learned nothing that transfers to new patients. The general group-split example in Training, Validation, and Test Sets shows the same mechanism with video frames; what this one adds is a label that belongs to the patient, not to the image.

Datasets and Benchmarks

Most clinical image collections are private, and public benchmarks differ in what their protocols fix. The facts below come from the papers and official pages; no data were downloaded for this article.

Benchmark	Data	Task and unit	Split	Official metric
MedMNIST v2	12 2D and 6 3D datasets resized to $28^2$ or $28^3$	binary, multi-class, ordinal and multi-label, per image or volume	official train, validation and test split for each dataset	AUC and accuracy
MURA	40,561 radiographs in 14,863 studies of 12,173 patients	normal or abnormal, per study	patient-disjoint train, validation and test; hidden test set	Cohen’s kappa against a radiologist majority vote
ISIC 2018 Part 3	10,015 HAM10000 dermoscopic images for training; 1,512 test images	7 diagnostic classes, per image	test set split into internal and external partitions	balanced accuracy

MedMNIST v2 standardizes 18 datasets into one format with fixed splits and benchmark results, licensed CC BY 4.0 except DermaMNIST, which is CC BY-NC 4.0 (Yang et al., 2023; MedMNIST page). Its authors use a source dataset’s official split where one exists, to avoid data leakage, and otherwise split at the patient level 7:1:2. For DermaMNIST, built from HAM10000, the paper describes splitting the images 7:1:2. HAM10000’s own paper states that the number of images does not correspond to the number of unique lesions, because some lesions were photographed at several magnifications or angles or with different cameras, which its authors present as a natural form of augmentation; the dataset is distributed under CC BY-NC 4.0 (Tschandl et al., 2018). Whether a DermaMNIST split keeps such views together is not stated in the passage read, so a user who needs lesion-disjoint evaluation would have to check the split against HAM10000’s lesion identifiers (our reasoning).

MURA used a patient-disjoint split: 11,184 patients for training, 783 for validation and 206 (207 studies) for testing (Rajpurkar et al., 2018). Six board-certified radiologists labelled each test study; three chosen at random formed the reference standard by majority vote, and the other three were compared with the model by Cohen’s kappa. The model’s overall kappa was 0.705 (95% CI 0.700–0.710 in their Table 2; the text gives 0.700–0.709), against 0.731 to 0.778 for the three radiologists; it was comparable to the best radiologist on finger studies and equal to two of them on wrist studies, and lower than the best on the other five study types. The radiologists had no clinical information, and the comparison is retrospective. Its competition used a hidden test set, with teams submitting code that was run on data they could not read, and is now closed (MURA page).

ISIC 2018 Part 3 trained on HAM10000 and evaluated on 1,512 held-out images, of which 1,196 came from the same two institutions as the training data (in Austria and Australia) and 316 from institutions not represented in it (in Turkey, New Zealand, Sweden and Argentina) (Codella et al., 2019). The organizers chose balanced accuracy, the mean recall over classes, because the dataset’s prevalence may not reflect real-world prevalence, and they report that most of the 141 submissions did better on the internal than on the external partition, though some of the best did not, and that the choice of metric changed the ranking of submissions considerably.

CheXpert, ChestX-ray14 and MIMIC-CXR belong to chest X-ray analysis, where Computer Vision in Radiology describes them; CAMELYON and PANDA, in Digital Pathology, belong to whole-slide image classification.

Evaluation Metrics

State the unit. The same predictions give different numbers per image, per study and per patient, and the unit must match the claim. Yagis et al. (2021) report slice-level accuracies, while Wen et al. (2020) combine slice predictions into a subject-level decision by weighted soft voting; MURA scores studies; a screening claim concerns patients.

Threshold-free scores. ROC AUC is the standard summary of a score’s ability to rank positives above negatives, insensitive to prevalence, and multi-label tasks report it per finding and as a mean over findings (Raghu et al., 2019; Azizi et al., 2021). MedMNIST pairs AUC with accuracy, noting that AUC is less sensitive to class imbalance (Yang et al., 2023).

Operating points. A deployed classifier makes decisions, so the result that matters is sensitivity and specificity at a threshold (Precision and Recall defines them). The threshold must be chosen on validation data, never on the test set, for a stated target such as a minimum sensitivity; predictive values at the deployment prevalence follow from sensitivity and specificity, as Computer Vision in Radiology shows. When classes are imbalanced and several decisions are scored at once, balanced accuracy (ISIC 2018) or agreement statistics such as Cohen’s kappa (MURA) are used.

Uncertainty. Test sets are often small, and images of one patient are correlated, so confidence intervals should resample patients rather than images; Statistical Significance in Evaluation explains why grouped data shrink the effective sample size.

Calibration. When the score itself is used, to triage cases, to defer to a reader or to report a risk, it must be calibrated: among cases given a score of 0.3, about 30% should be positive. Expected Calibration Error covers how this is measured and repaired, for example by temperature scaling on validation data. Two medical causes of miscalibration follow from the sections above. Reweighting or oversampling for imbalance shifts the scores away from the training prevalence. And a change of prevalence between sites shifts them even for a model that was calibrated where it was developed: if the images of each class look the same at both sites, Bayes’ rule gives the posterior odds at a site with prevalence $\pi'$ from those at the development prevalence $\pi$ as

$$\frac{p'}{1 - p'} = \frac{p}{1 - p} \cdot \frac{\pi' (1 - \pi)}{\pi (1 - \pi')},$$

so the ranking, and the AUC, are unchanged while every probability moves (our derivation, under that assumption; in practice acquisition and population change too). Zech et al. found a pneumonia model trained on NIH data poorly calibrated on Mount Sinai data and vice versa, as Computer Vision in Radiology reports.

Applications

Many of the results that the neighbouring articles report are classification results. Chest radiographs (findings from report-mined label sets) and whole-slide images (cancer detection, grading and molecular status) are the two largest, each a subject in its own right; mammography screening, with its views, priors and screening metrics, is another. Other settings include diabetic retinopathy grading from fundus photographs (Raghu et al., 2019), skin lesion diagnosis from dermoscopic images (HAM10000 and the ISIC challenges) and from consumer photographs (Azizi et al., 2021), abnormality detection in musculoskeletal radiographs (MURA), and diagnosis of neurological disease from brain MRI (Wen et al., 2020; Yagis et al., 2021). Ophthalmic image analysis is named here as a subject of its own.

Open Problems

• Evaluation that matches the claim. Patient-level and site-level splits are easy to state and often missed, and many papers do not say which they used; Wen et al. (2020) could not assess leakage in 10 of 32 studies for lack of explanation.
• Labels. Report-mined and single-reader labels limit both training and testing, and better reference standards are expensive.
• Value of pretraining. Whether ImageNet features, in-domain self-supervised pretraining or large medical foundation models help depends on the task, the amount of labelled data and the model size (Raghu et al., 2019; Azizi et al., 2021), and comparisons are hard when papers re-split datasets differently.
• Calibration and shift. A model validated at one site must keep its operating point and calibration at others, under prevalence and acquisition shift.
• From scores to studies. Most results are retrospective and standalone; how a classifier changes readers’ decisions and patient outcomes needs reader studies and prospective trials, which Computer Vision in Radiology describes and Clinical Validation and Regulation of Medical Imaging AI treats in detail.