Chest X-ray Analysis

Chest X-ray analysis applies computer vision to chest radiographs: classifying the findings a radiograph shows, locating them, and, increasingly, describing them in text. Chest radiography is the most common imaging examination worldwide, in the words of the CheXpert, CheXNeXt and MIMIC-CXR authors (Irvin et al., 2019; Rajpurkar et al., 2018; Johnson et al., 2019), and several of the largest public datasets in medical vision, among them ChestX-ray14, CheXpert, MIMIC-CXR and PadChest, are chest radiographs with labels mined from their reports. That combination, abundant images with labels that nobody drew on the images, shapes the methods, the evaluation and the failures described here.

X-ray Radiography covers how a projection radiograph is formed and stored, Medical Image Classification the task in general, and Computer Vision in Radiology the history, triage, report generation, the CheXNet and CheXpert comparisons with radiologists, the shortcut studies of Zech et al. and DeGrave et al., and predictive values at clinical prevalence. This article goes deeper into what is specific to the chest radiograph. It describes methods, evidence and limits for practitioners and researchers, gives no diagnostic or clinical guidance, and attributes every reported result to its authors with its set-up.

Problem

A chest radiograph study holds one or more images, usually a frontal view and often a lateral view, and a free-text report written by a radiologist during routine care (Johnson et al., 2019). The vision problem is usually posed as multi-label classification: for each of a fixed list of findings, estimate the probability that it is present in the image or the study. CheXpert’s task, for example, is to predict the probability of 14 observations from multi-view radiographs (Irvin et al., 2019). Localization, marking where a finding is, and report generation extend the task; the second is described in Computer Vision in Radiology.

Three features make the chest version of the task its own subject. The labels are mostly not drawn on images but extracted from reports by natural language processing, so they inherit what the report says and what the extraction gets wrong. The images come from different projections and patient positions, which change how the same anatomy looks. And the findings overlap in appearance and in vocabulary: the CheXNeXt authors note that consolidation, infiltration and pneumonia are all manifestations of airspace opacities yet are separate labels in ChestX-ray14 (Rajpurkar et al., 2018).

Visual Data

Projections and positions

A radiograph is a central projection from the X-ray source onto the detector, so a structure’s magnification depends on its distance from the detector, as X-ray Radiography derives. Chest radiographs are taken in a few standard ways. The PadChest authors, who extracted projection and positioning from 160,868 radiographs of one Spanish hospital, identified mainly posteroanterior (PA), anteroposterior (AP) and lateral projections, with the patient erect (standing or sitting), supine or in lateral decubitus, and protocols for different clinical situations: standard standing PA and lateral views, mobile radiography in AP erect or AP supine position for patients unable to stand, a paediatric protocol, lordotic views and rib views (Bustos et al., 2020). The names give the direction in which the beam crosses the chest: posteroanterior from back to front, anteroposterior from front to back.

The projection changes the image, not only its geometry label. Bustos et al. (2020) give two examples. In AP views the heart silhouette is enlarged, an effect of the projection that should not be read as cardiomegaly and that their Figure 3 explains by the dependence of anatomical ratios on the distance to the X-ray source; and a pleural effusion in a standing patient shows a meniscus sign that is absent in a decubitus projection, where the fluid spreads along the most dependent part of the chest. They also name landmarks that radiologists use to recognize a PA projection, such as air in the gastric chamber and the scapulae projected outside the lung fields, and warn that a model trained on data in which AP views are rare may attribute projection-induced heart enlargement to cardiomegaly. The geometry in X-ray Radiography gives the same result: the magnification $M = \text{SID}/\text{SOD}$ is larger for a structure farther from the detector, and the heart, which lies in the front of the chest, is farther from a detector at the patient’s back (our reasoning).

The mix of projections differs between datasets and within them. In MIMIC-CXR, 65.94% of the DICOM images come from examinations coded CHEST (PA AND LAT) and 33.49% from CHEST (PORTABLE AP), and the authors note that not all patients are healthy enough to stand for a PA radiograph or sit upright for an AP one (Johnson et al., 2019). ChestX-ray14 holds frontal views only, both PA and AP (Rajpurkar et al., 2018). VinDr-CXR released only adult PA views, after a classifier removed lateral, paediatric and non-chest images (Nguyen et al., 2022). PadChest grouped its images into six classes (standard PA, standard lateral, AP erect, AP supine, paediatric and rib views) and, for 20,367 images whose DICOM headers carried no projection, assigned one with a pretrained classifier; the authors warn that these automatically assigned projections should not be used to train another projection classifier (Bustos et al., 2020).

What the stored image contains

The DICOM header records the view position, and MIMIC-CXR-JPG copies it into a metadata table with values such as AP, PA and LATERAL, alongside the coded view and patient orientation (erect or recumbent) (Johnson et al., 2024). Images may also carry burned-in text placed by the modality: MIMIC-CXR’s de-identification removed dates and identifiers from the pixels but kept radiologically relevant text such as orientation (Johnson et al., 2019). The images vary in technique and positioning, may omit anatomy or contain objects that obstruct it, and may be collimated at the modality, all of which the MIMIC-CXR authors leave in because they occur in daily practice.

Datasets also differ in the preprocessing applied before release, which is part of the data a model learns from. The ChestX-ray8 paper describes its images as extracted from DICOM, rescaled with the default window stored in the header and resized to $1024 \times 1024$ (Wang et al., 2017). MIMIC-CXR-JPG normalizes each image to the range 0–255 using its own minimum and maximum, inverts it where the photometric interpretation requires, histogram-equalizes it and saves it as JPEG with quality factor 95 (Johnson et al., 2024). PadChest rescaled the dynamic range with the DICOM window and did not resize (Bustos et al., 2020). Medical Image Preprocessing treats such choices in general.

Findings and Label Sets

Every dataset fixes a list of findings, and the lists differ in size, vocabulary and kind.

• ChestX-ray14 started as eight findings chosen from radiologists’ feedback (atelectasis, cardiomegaly, effusion, infiltration, mass, nodule, pneumonia and pneumothorax) and was extended to fourteen by adding consolidation, edema, emphysema, fibrosis, pleural thickening and hernia; an image without any of them is labelled “No findings” (Wang et al., 2017).
• CheXpert chose 14 observations from their prevalence in the reports and their clinical relevance, conforming to the Fleischner Society’s glossary where applicable, including “Support Devices” and “No Finding”. Pneumonia was included, although the authors call it a clinical diagnosis, to represent images that suggested primary infection (Irvin et al., 2019). MIMIC-CXR-JPG uses the same 14 labels (Johnson et al., 2024).
• PadChest labelled every entity its physicians found in the reports: 174 radiographic findings, 19 differential diagnoses and 104 anatomic locations in the preprint’s abstract (its dataset section counts 170 findings), mapped to UMLS concepts and organized in hierarchies (Bustos et al., 2020). The authors separate radiographic findings, which a radiologist can assert from the image alone, from differential diagnoses, which need clinical information and additional tests; they argue that expecting a model to match doctors on diagnoses such as pneumonia from the image alone may be unrealistic.
• VinDr-CXR has 22 local labels, findings marked with boxes, and 6 global labels for the diagnostic impression (lung tumour, pneumonia, tuberculosis, other diseases, COPD and no finding), corresponding in the authors’ description to the findings and impression sections of a report (Nguyen et al., 2022).

Two consequences follow. A model trained on one label set does not answer the questions of another: MIMIC-CXR’s authors note that CheXpert’s labels do not perfectly overlap ChestX-ray14’s (Johnson et al., 2019), and a pooled analysis across datasets has to restrict itself to shared labels, as Seyyed-Kalantari et al. (2021) did. And the label for a normal image means different things: in ChestX-ray8’s labelling rules a report was coded “Normal” if no disease at all was detected in it, not only none of the eight, or if it contained the text-mined concept “normal” or “normal size” (Wang et al., 2017); in CheXpert’s labeller, “No Finding” is positive when no pathology is classified as positive or uncertain (Irvin et al., 2019).

Labels Mined from Reports

How the labellers work

The NIH labeller (ChestX-ray14). Wang et al. (2017) detected disease concepts in the findings and impression sections with two tools, DNorm and MetaMap, and merged their outputs to maximize recall. They then removed negated and uncertain mentions with hand-crafted rules on a syntactic dependency graph rather than on the raw text, so that a phrase such as “clear of focal airspace disease, pneumothorax, or pleural effusion” negates all three findings, which a regular expression would not capture. Uncertain mentions (“cannot exclude”, “may represent”, “suggesting”) were removed rather than kept.

The CheXpert labeller. Irvin et al. (2019) extract mentions from the impression section with phrase lists curated by radiologists, classify each mention as negative, uncertain or positive by rules on the universal dependency parse, and aggregate the mentions into one label per observation: positive if any mention is positive, uncertain if none is positive and one is uncertain, negative if a mention is negative, and blank if the observation is not mentioned. The uncertainty classification runs before and after negation, so “cannot exclude pneumothorax” becomes uncertain rather than negative. The uncertain label, in their description, captures both the radiologist’s diagnostic uncertainty and ambiguity in the report, as in “heart size is stable”. MIMIC-CXR-JPG releases labels from this labeller and from NegBio, run with CheXpert’s mention patterns, which differ from those used for ChestX-ray14 (Johnson et al., 2024).

PadChest. Trained physicians annotated 27,593 reports (27% of the dataset in the preprint’s abstract; 27,593 of the 109,931 labelled reports, about 25%, in its Table 7), working sentence by sentence (22,120 unique sentences, helped by the redundancy of report language), and a recurrent network with attention trained on those annotations labelled the remaining reports (Bustos et al., 2020). A special label “Unchanged” marks sentences that report no change in a finding not named in the report; the authors say it should be replaced by the patient’s labels from a prior study, or removed, before training an image model.

How noisy the labels are

The labellers are evaluated against people reading the reports, and those evaluations measure only one of two gaps: between the report and its labels, not between the report and the image.

On the report side, Wang et al. (2017, appendix B) had two annotators label 900 reports for the 14 findings and resolve conflicts, and report an overall precision and recall of 94.4% each for their method, against 89.8% precision and 85.0% recall for the same 14 findings on the public OpenI reports, where pneumonia’s precision was 44.4% (the main text’s 0.90 and 0.91 on OpenI are for the original eight findings). Irvin et al. (2019) built a report evaluation set of 1,000 reports labelled by two radiologists and compared their labeller by F1 with the one used to label ChestX-ray14, which they call the NIH labeler, for detecting mentions, negations and uncertainty. The NIH labeller’s negation F1 for cardiomegaly was 0.000 against 0.909 for CheXpert’s, and its uncertainty F1 for pneumothorax 0.167 against 0.762; CheXpert’s labeller reached micro-averaged F1 scores of 0.969, 0.952 and 0.848 for mentions, negations and uncertainty. PadChest’s labelling network reached a micro-F1 of 0.939 (its Table 4; the text rounds to 0.93) on 500 manually labelled sentences drawn from reports of 2009–2013 (2009–2014 in its conclusions), a different period from the 2014–2017 reports it was trained on (Bustos et al., 2020).

The second gap, between what a report says and what the image shows, is measured when report labels are scored against radiologists who read the images. CheXpert’s test set allows this: the authors scored the labeller’s output on the test studies against the consensus of five radiologists who annotated the images, with uncertain labels assigned either all correctly (an upper bound) or all wrongly (a lower bound) (Irvin et al., 2019). The upper bound performed comparably to the three comparison radiologists on most of the five tasks, but not everywhere: for atelectasis the upper-bound operating point had a true positive rate of 0.51 at a false positive rate of 0.10, while the three radiologists’ true positive rates were 0.71 to 0.92 at false positive rates of 0.18 to 0.31 (their Figure 3). A report written for a clinician need not mention every finding visible in the image, which is one possible reading of the gap (ours); the paper itself reports the operating points without decomposing their causes.

Report labels also inherit the habits of their hospitals. Computer Vision in Radiology describes how Zech et al. attribute part of one hospital’s high pneumonia prevalence to more liberal labelling there. Seyyed-Kalantari et al. (2021) add that the labellers’ performance in different patient subpopulations has not been explored, and that labels extracted from clinical records may already contain biases of clinical care.

What the uncertain label does in training

CheXpert’s uncertain labels are frequent for some observations: in the training set, consolidation is uncertain in 12.78% of studies and positive in 6.78% (Irvin et al., 2019). The authors compared five policies on a validation set of 200 studies labelled by three radiologists: ignoring uncertain labels in the loss (U-Ignore), mapping them all to negative (U-Zeros) or to positive (U-Ones), self-training, in which a U-Ignore model relabels them with its predicted probabilities (U-SelfTrained), and treating “uncertain” as a third class (U-MultiClass). Only two differences were significant. On atelectasis U-Ones reached an AUC of 0.858 against 0.811 for U-Zeros, and on cardiomegaly U-MultiClass reached 0.854 against 0.828 for U-Ignore. The authors’ explanations are linguistic: they expect that phrases such as “possible atelectasis” were meant to describe the most likely finding, while many uncertain cardiomegaly mentions are borderline (“minimal cardiac enlargement”) or uninformative (“stable cardiac contours”). The final model used a different policy for each observation, chosen on the validation set.

CheXNeXt took a different approach to ChestX-ray14’s labels, which its authors describe as partially incorrect (Rajpurkar et al., 2018). An ensemble trained on the original labels relabelled the training and tuning sets, with a label made positive if either the original label or the ensemble’s thresholded prediction was positive, and new networks were trained on the relabelled data. The rule can only add positives, so it treats missed findings as the main error (our reading). Relabelling increased the number of positive labels for every pathology, and performance improved on 11 pathologies and worsened on 3.

Example: Training and Testing with Report-Like Label Noise

The example below generates 32 × 32 patches that stand in for regions of a radiograph: a smooth random background, three bright horizontal bands, additive noise and, in 30% of patches, a Gaussian “opacity” whose amplitude varies from subtle to obvious. The true labels are known. A second set of labels imitates a report-mining pipeline: a true opacity is missed with probability 0.5 if it is subtle and 0.1 otherwise, and a patch without one receives a false positive label with probability 0.03. A small convolutional network is trained once on true labels and once on the noisy labels, on the same images, and both are tested on 2,000 new patches against both kinds of label. The script also reports the AUC on subtle positives against all negatives, and the mean predicted probability. It uses PyTorch 2.14.1 and NumPy 2.5.4 and loops over three seeds; running it twice gives identical output.

import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(1)
S = 32  # patch size in pixels

def make_patches(n, rng, prevalence=0.30):
    """Synthetic radiograph patches: smooth background, rib-like bands, noise;
    positives carry a Gaussian 'opacity' of random amplitude."""
    f = np.fft.fftfreq(S)
    lowpass = np.exp(-(f[:, None] ** 2 + f[None, :] ** 2) / (2 * 0.08 ** 2))
    yy, xx = np.mgrid[0:S, 0:S]
    X = np.empty((n, S, S), np.float32)
    y = (rng.random(n) < prevalence).astype(int)
    amp = np.where(y == 1, rng.uniform(0.15, 0.6, n), 0.0)
    for i in range(n):
        bg = np.real(np.fft.ifft2(np.fft.fft2(rng.normal(size=(S, S))) * lowpass))
        bg *= 0.3 / bg.std()
        for r in rng.integers(0, S, 3):  # three bright horizontal bands
            bg += 0.8 * np.exp(-((yy - r) ** 2) / 4.0)
        if y[i]:
            cy, cx = rng.uniform(6, S - 6, 2)
            bg += amp[i] * 4 * np.exp(-((yy - cy) ** 2 + (xx - cx) ** 2) / (2 * 3.0 ** 2))
        X[i] = bg + rng.normal(0, 0.5, (S, S))
    subtle = (y == 1) & (amp < 0.375)
    return X, y, subtle

def report_labels(y, subtle, rng):
    """Labels as a report-mining pipeline might give them: missed findings
    (more often when subtle) and a few false mentions."""
    u = rng.random(len(y))
    miss = np.where(subtle, 0.5, 0.1)
    return np.where(y == 1, (u >= miss).astype(int), (u < 0.03).astype(int))

def auc(score, label):
    r = np.argsort(np.argsort(score)) + 1.0
    pos = label == 1
    n1, n0 = pos.sum(), (~pos).sum()
    return (r[pos].sum() - n1 * (n1 + 1) / 2) / (n1 * n0)

def train(X, labels, seed):
    torch.manual_seed(seed)
    net = nn.Sequential(nn.Conv2d(1, 8, 5, padding=2), nn.ReLU(), nn.MaxPool2d(2),
                        nn.Conv2d(8, 16, 5, padding=2), nn.ReLU(),
                        nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(16, 1))
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    Xt = torch.from_numpy(X[:, None]); yt = torch.from_numpy(labels.astype(np.float32))
    g = torch.Generator().manual_seed(seed)
    for epoch in range(8):
        for idx in torch.randperm(len(Xt), generator=g).split(64):
            loss = nn.functional.binary_cross_entropy_with_logits(net(Xt[idx])[:, 0], yt[idx])
            opt.zero_grad(); loss.backward(); opt.step()
    return net

results = {"clean": [], "noisy": []}
for seed in (0, 1, 2):
    rng = np.random.default_rng(seed)
    Xtr, ytr, sub_tr = make_patches(4000, rng)
    Xte, yte, sub_te = make_patches(2000, rng)
    ytr_noisy, yte_noisy = report_labels(ytr, sub_tr, rng), report_labels(yte, sub_te, rng)
    neg = yte == 0
    stratum = sub_te | neg  # subtle positives against all negatives
    for name, lab in (("clean", ytr), ("noisy", ytr_noisy)):
        net = train(Xtr, lab, seed)
        with torch.no_grad():
            p = torch.sigmoid(net(torch.from_numpy(Xte[:, None]))[:, 0]).numpy()
        row = (auc(p, yte), auc(p, yte_noisy), auc(p[stratum], yte[stratum]), p.mean())
        results[name].append(row)
        print(f"seed {seed} {name:5s} labels: AUC clean test {row[0]:.3f} | "
              f"AUC noisy test {row[1]:.3f} | AUC subtle {row[2]:.3f} | mean p {row[3]:.3f}")
    print(f"seed {seed} training-label prevalence: true {ytr.mean():.3f}, noisy {ytr_noisy.mean():.3f}")
for name, rows in results.items():
    lo, hi = np.min(rows, axis=0), np.max(rows, axis=0)
    print(f"{name:5s} range over seeds: AUC clean test {lo[0]:.3f}-{hi[0]:.3f}, "
          f"AUC noisy test {lo[1]:.3f}-{hi[1]:.3f}, AUC subtle {lo[2]:.3f}-{hi[2]:.3f}, "
          f"mean p {lo[3]:.3f}-{hi[3]:.3f}")

seed 0 clean labels: AUC clean test 0.834 | AUC noisy test 0.794 | AUC subtle 0.756 | mean p 0.285
seed 0 noisy labels: AUC clean test 0.831 | AUC noisy test 0.792 | AUC subtle 0.753 | mean p 0.247
seed 0 training-label prevalence: true 0.299, noisy 0.236
seed 1 clean labels: AUC clean test 0.826 | AUC noisy test 0.802 | AUC subtle 0.745 | mean p 0.308
seed 1 noisy labels: AUC clean test 0.822 | AUC noisy test 0.800 | AUC subtle 0.741 | mean p 0.283
seed 1 training-label prevalence: true 0.304, noisy 0.239
seed 2 clean labels: AUC clean test 0.840 | AUC noisy test 0.800 | AUC subtle 0.763 | mean p 0.239
seed 2 noisy labels: AUC clean test 0.738 | AUC noisy test 0.709 | AUC subtle 0.677 | mean p 0.213
seed 2 training-label prevalence: true 0.306, noisy 0.230
clean range over seeds: AUC clean test 0.826-0.840, AUC noisy test 0.794-0.802, AUC subtle 0.745-0.763, mean p 0.239-0.308
noisy range over seeds: AUC clean test 0.738-0.831, AUC noisy test 0.709-0.800, AUC subtle 0.677-0.753, mean p 0.213-0.283

The clearest effect is in the evaluation, not the training. Every one of the six models scores 0.02 to 0.04 lower in AUC when its predictions are compared with noisy test labels instead of true ones, although the predictions are the same: missed opacities are counted as negatives that the model ranks high, and false labels as positives it ranks low. A test set labelled the same way as the training data therefore understates a model that is better than its labels, and cannot show that it is better (our reasoning). CheXpert’s authors make the related point that benchmarking on ChestX-ray14 is problematic because its test labels are extracted by an automatic labeller, and they gave CheXpert radiologist-labelled validation and test sets (Irvin et al., 2019).

Training on the noisy labels cost little ranking quality in two of the three seeds (an AUC 0.003 to 0.004 lower against true labels) and much more in the third (0.840 against 0.738). Three runs show that the outcome varies between runs; they do not establish its average size. In every seed the subtle positives are ranked worse than the whole test set (AUC 0.677 to 0.763 against 0.738 to 0.840): the overall number hides weaker performance on exactly the cases this labeller misses most, the pattern discussed under subgroups below. The models trained on noisy labels predict lower probabilities in every seed, by 0.025 to 0.038 in the mean, moving towards the noisy labels’ prevalence (0.230 to 0.239) and away from the true one (about 0.30); but the clean-label models’ means also range from 0.239 to 0.308, so after eight epochs even the mean probability varies between runs by more than the noise shifts it, and probabilities would have to be recalibrated on data with trusted labels before being read as risks (our reasoning; CheXpert’s authors measured calibration before and after isotonic regression and Platt scaling). Expected Calibration Error covers how calibration is measured.

The example was planned before it was run, and the version shown is the third. In the first two runs even the models trained on true labels learned little, so the example could not show anything about label noise. The first run had a background of unit standard deviation and a network with 16 and 32 channels; for the second, the background was smoothed more (a low-pass width of 0.03 instead of 0.08 cycles per pixel) and the channels were halved to 8 and 16 to shorten the run. A fixed matched filter, applied without training or labels, then showed that the background texture swamped the opacity in both versions. The background was therefore scaled down, choosing the smallest of three candidate scales (0.5, 0.4, 0.3) at which the matched filter reached an AUC of at least 0.8, a criterion independent of label noise, and the smoothing was returned to its first value; the channel change was kept. The noise model, prevalence, amplitudes, optimizer, epochs, batch size, seeds and printed quantities were not changed, and the third run is reported as it came out.

Common Methods

The networks are those of image classification, usually initialized from ImageNet, and the label handling is described above. ChestX-ray14’s baselines used AlexNet, GoogLeNet, VGGNet-16 and ResNet-50, with ResNet-50 best (Wang et al., 2017), and CheXpert’s authors compared ResNet-152, DenseNet-121, Inception-v4 and SE-ResNeXt-101 and chose DenseNet-121 (Irvin et al., 2019). The choices specific to chest radiographs are these.

• Multi-label output and imbalance. One sigmoid output per finding, trained with binary cross-entropy. Wang et al. (2017) weighted positive and negative terms by their frequencies in each batch, because positive labels are sparse; the weighting raised the cardiomegaly AUC from 0.7262 to 0.8141 and pneumonia from 0.5164 to 0.6333 on ChestX-ray8.
• Resolution. Radiographs are typically about 3000 × 2000 pixels and many findings are small (Wang et al., 2017), but networks take far less: 512 pixels for CheXNeXt (Rajpurkar et al., 2018) and 320 for CheXpert’s model (Irvin et al., 2019). The CheXNeXt authors list the reduced resolution, 1,024 pixels for their radiologists and 512 for the algorithm against the more than 2,000 usual in practice, among their limitations.
• Several views. CheXpert’s model takes one image and outputs, for a study with several views, the maximum probability across them (Irvin et al., 2019).

Localization

Weakly supervised: maps from a classifier

Most chest datasets have no locations, so localization is often read from a classifier. Wang et al. (2017) replaced the final layers of pretrained networks with a transition layer, a global pooling layer and a prediction layer, so that multiplying the transition layer’s activations by the prediction weights gives a heatmap per finding; they compared average, max and log-sum-exp (LSE) pooling, which interpolates between the two with a parameter $r$,

$$x_p = \frac{1}{r} \log \left( \frac{1}{|\mathcal{S}|} \sum_{(i,j) \in \mathcal{S}} \exp(r\, x_{ij}) \right),$$

where $x_{ij}$ are the activations in the pooling region $\mathcal{S}$; it tends to the maximum as $r \to \infty$ and to the average as $r \to 0$. Boxes were cut from thresholded heatmaps and scored against 1,600 boxes that a board-certified radiologist drew on 983 images for the eight original findings, used only for evaluation. Because the heatmaps were $32 \times 32$ for $1024 \times 1024$ images, the predicted boxes were often larger than the drawn ones, so the authors counted a box as correct if either its intersection over union or its intersection over the predicted box’s own area (IoBB) exceeded a threshold. On ChestX-ray14 at an IoBB threshold of 0.5, the fraction of findings correctly localized was 0.9452 for cardiomegaly, 0.3058 for masses and 0 for nodules (appendix Table 19); at an IoU threshold of 0.5 on ChestX-ray8, the best was 0.178, for cardiomegaly (Table 7). Small findings such as nodules were localized worst.

CheXNet, CheXNeXt and CheXpert all show class activation maps or Grad-CAM heatmaps as illustrations (Rajpurkar et al., 2017; Rajpurkar et al., 2018; Irvin et al., 2019). Saporta et al. (2022) measured how well such maps localize. On CheXpert’s test set, two radiologists drew reference segmentations for ten pathologies and three other radiologists drew benchmark segmentations; seven saliency methods were compared on three architectures, with Grad-CAM on DenseNet-121 generally best. Its segmentations had a significantly lower mean IoU than the human benchmark for five of the ten pathologies and a higher one for two (atelectasis and consolidation); averaged over the pathologies, its mean IoU was 24.0% worse. Support devices had one of the highest classification AUROCs (0.969) but a mean IoU of 0.163, and lung lesions showed the largest gap (76.2% worse than the benchmark). The gap was largest for small pathologies with complex shapes, and higher model confidence went with better localization. A heatmap that appears to highlight a finding in an illustration is therefore weak evidence of where the finding is, least of all for small findings (our reading); Interpretability of Neural Networks discusses saliency methods in general.

Supervised: boxes drawn by radiologists

Detection trained on boxes needs annotated datasets. VinDr-CXR provides boxes for its 22 local labels on 18,000 PA radiographs from two Vietnamese hospitals, collected between 2018 and 2020: each of the 15,000 training images was labelled independently by three of 17 radiologists, with every annotator’s boxes released, and each of the 3,000 test images was labelled by three radiologists whose work two more experienced radiologists reviewed to reach a consensus (Nguyen et al., 2022). Training images therefore carry three possibly disagreeing sets of boxes, a form of label noise that a detector trained on them must handle, while the test set has one reference (our reading). Object Detection covers the detectors, and Lesion Detection covers detection of findings with its free-response evaluation.

Datasets

The facts below come from the papers and official pages; no data were downloaded for this article, and no data use agreement was read or accepted. Computer Vision in Radiology describes ChestX-ray14, CheXpert and MIMIC-CXR in more detail.

Dataset	Images and source	Views	Training labels	Reference for evaluation	Access (as of November 2026)
ChestX-ray14	112,120 images of 30,805 patients, NIH Clinical Center	frontal (PA and AP)	14 findings, NIH labeller	report-mined labels; boxes on 983 images	NIH Clinical Center (terms not read)
CheXpert	224,316 images of 65,240 patients, Stanford Hospital, 2002–2017	frontal and lateral	14 observations, positive, negative or uncertain	200 validation studies (3 radiologists), 500 test studies (5 of 8 radiologists)	Stanford (terms not read)
MIMIC-CXR(-JPG)	377,110 images, 227,835 studies, 65,379 patients, Beth Israel Deaconess, 2011–2016	frontal and lateral; PA and portable AP	reports; CheXpert and NegBio labels in the JPG version	test-set reports labelled by one radiologist (v2.1.0)	credentialed access, data use agreement, PhysioNet Credentialed Health Data License 1.5.0
PadChest	160,868 images of 67,625 patients, Hospital San Juan, Alicante, 2009–2017	PA, lateral, AP erect and supine, paediatric, ribs	174 findings, 19 diagnoses and 104 locations from Spanish reports; 27% of reports by physicians, the rest by a network	none designated in the preprint	formal request; users undertake not to share data or re-identify
VinDr-CXR	18,000 adult PA images, two hospitals in Vietnam, 2018–2020	PA only	22 boxed findings and 6 impressions, 3 radiologists per image	3,000 test images, consensus of 5 radiologists	credentialed access, data use agreement, PhysioNet Credentialed Health Data License 1.5.0

Sources: Wang et al. (2017) and Rajpurkar et al. (2018) for ChestX-ray14; Irvin et al. (2019); Johnson et al. (2019, 2024); Bustos et al. (2020) and the BIMCV page; Nguyen et al. (2022) and its PhysioNet page (Nguyen et al., 2021). The PadChest preprint disagrees with itself: its methods text gives 168,861 images in the collection where its Table 7 gives 168,171 in the initial set, and its dataset section pairs the 160,868 labelled images with 69,882 patients, the initial set’s count in Table 7, where the table gives 67,625 patients for the final labelled set (the abstract says about 67,000). The table above follows Table 7. Its official page repeats the abstract’s figures.

The splits matter as much as the data. ChestX-ray14’s appendix describes a split at the patient level, about 70/10/20, with the split files published (Wang et al., 2017); results on random image-level re-splits, or on the label set of one dataset tested on another, are not comparable with results on the official split; Medical Image Classification shows what splitting images rather than patients does to reported accuracy. MIMIC-CXR-JPG provides a recommended split file (Johnson et al., 2024). VinDr-CXR’s training and test sets differ in their label prevalences (atelectasis is 0.41% of training images and 2.87% of test images, by the majority vote of the annotators), which affects any threshold chosen on training data.

Shortcut Learning and Hidden Stratification

Projection and acquisition as shortcuts

The general mechanism, a model using whatever predicts the label including the hospital and markers placed by technologists, is described with Zech et al.’s and DeGrave et al.‘s studies in Computer Vision in Radiology. The chest radiograph supplies a specific candidate: the projection. It is easy to learn, it changes the appearance of the heart and pleura (Bustos et al., 2020), and it is tied to the patient’s condition, since mobile AP radiographs are taken of patients unable to stand (Bustos et al., 2020; Johnson et al., 2019). DeGrave et al. (2021) trained networks to predict AP against PA view on their two COVID-19 datasets: they reached AUCs of 0.99917 and 0.992 on internal test data, and 0.949 and 0.993 on external data (their Figure 5), and the authors attribute part of their COVID-19 models’ attention to image edges, the diaphragm and the cardiac silhouette to differences in projection and positioning between the sources. Cropping images to their central 75% to remove text markers did not improve generalization; the authors suggest that a model has many redundant shortcuts and moves to another when one is removed. They restricted their data to PA and upright AP images, excluding laterals and AP supine views.

A recorded view position allows two checks that the image alone does not: stratifying results by projection, and testing whether a model’s errors follow the projection mix of a new site (our reasoning). PadChest’s authors provide the metadata for this reason, to help control confounders such as projection and patient positioning (Bustos et al., 2020).

Failures in subgroups

An overall AUC averages over subsets that the labels do not mark: subtle against obvious findings, one patient group against another. Oakden-Rayner et al. (2020) named such unmarked subsets hidden stratification in the title of their paper, which could not be read for this article. Two of the studies read for it document performance that differs between subsets.

Seyyed-Kalantari et al. (2021) trained DenseNet-121 classifiers on MIMIC-CXR, CheXpert, ChestX-ray14 and the three combined, and measured underdiagnosis, which they define as the false positive rate of the “No Finding” output at a single threshold chosen to maximize F1, in subgroups by sex, age, race or ethnicity and insurance type (the last as a proxy for socioeconomic status; only MIMIC-CXR records race and insurance). On MIMIC-CXR, female patients, patients under 20, Black and Hispanic patients and patients with Medicaid insurance had higher underdiagnosis rates, and intersections of these groups often higher still; on ChestX-ray14 the pattern differed, with male patients and patients over 80 most affected, which the authors partly attribute to small subgroups (37 “No Finding” test images for patients over 80) and to that hospital’s selective admission. The results are averages over five training seeds on a fixed patient-disjoint split, with images resized to $256 \times 256$. The false negative rate moved in the opposite direction, which the authors read as selective underdiagnosis rather than undirected noise. They caution that their labels come from clinical records and may carry the same bias the model shows.

Saporta et al. (2022) found the localization gap largest for small, complex-shaped pathologies, and CheXNeXt’s authors attribute its poorest results, on emphysema and hiatal hernia, to a symmetric global appearance in the first and the lowest training prevalence in the second (Rajpurkar et al., 2018). The example above reproduces the mechanism in miniature: the subtle opacities, the ones the simulated reports miss, are ranked worse in every run while the overall AUC stays higher.

Evaluation and Results with Their Study Design

Chest X-ray results depend on the reference standard, the comparison, the case mix and the design, and the published comparisons differ in all four. All those below are retrospective; none measured an effect on patients.

CheXNet (read in arXiv v3) compared a DenseNet-121 trained on ChestX-ray14 with four radiologists on pneumonia, scoring each by F1 against the others; Computer Vision in Radiology gives the design and the numbers.

CheXNeXt (Rajpurkar et al., 2018) extended the comparison to all 14 ChestX-ray14 findings. The 420 frontal radiographs of its validation set were sampled to contain at least 50 cases of each pathology, an enriched set rather than a clinical one. The reference standard was the majority vote of three cardiothoracic specialist radiologists, and the comparison readers were six board-certified radiologists from three academic institutions and three senior radiology residents, who had no patient information and no knowledge of the prevalence. Radiologists’ AUCs were estimated from their yes/no answers by fitting a ROC curve, and differences were judged with Bonferroni-corrected bootstrap intervals. The radiologists had significantly higher AUCs on cardiomegaly (0.888 against 0.831), emphysema (0.911 against 0.704) and hiatal hernia (0.985 against 0.851); the algorithm was significantly higher on atelectasis (0.862 against 0.808); the other ten differences were not significant. The authors list as limitations frontal views only, although they cite the claim that up to 15% of accurate diagnoses need the lateral view; no history or prior examinations; reduced image resolution; a reference standard without CT, laboratory or pathology data; and data from one institution. They also note that their readers labelled every finding regardless of clinical significance, unlike routine reporting.

CheXpert (Irvin et al., 2019) used a test set of 500 studies read from the images by eight radiologists, five forming the reference and three compared, with “uncertain likely” counted as positive; Computer Vision in Radiology gives the comparison. On the five tasks compared with radiologists, the model’s AUCs ranged from 0.85 (atelectasis) to 0.97 (pleural effusion). No statistical test of model against radiologists was performed, and neither had history or prior examinations. The report-label bounds described above come from the same test set.

Benchmarks on report-mined labels. Results on ChestX-ray14’s official test split are scored against labels from the NIH labeller (Wang et al., 2017), so they measure agreement with report-mined labels; the example above shows how such a test understates a model’s agreement with the truth, and why it cannot rank models whose differences are smaller than the label noise (our reasoning). Reported AUCs per finding, such as the 0.6580 for pneumonia and 0.8717 for hernia of the ChestX-ray14 ResNet-50 baseline (appendix Table 17), are best read as agreement with those labels.

The metrics are the general ones of Medical Image Classification: ROC AUC per finding, sensitivity, specificity and F1 at a threshold (Precision and Recall), agreement with readers and calibration. With 14 findings or more, comparisons need correction for multiple testing, as CheXpert’s validation comparisons (Benjamini–Hochberg) and CheXNeXt’s (Bonferroni) applied; Statistical Significance in Evaluation treats both.

Failure Modes and Risks

The sections above give the evidence; in summary, a chest radiograph model can learn the labeller’s omissions and negation errors, which a test labelled the same way cannot reveal; use projection, positioning, markers and preprocessing as shortcuts that do not transfer; answer a label set that mixes findings with diagnoses that need clinical information; fail on subsets that overall metrics hide, including under-served patient groups; and come with heatmaps that localize worse than radiologists. The evaluations described here also withheld the lateral view or the history and prior studies, so they test a narrower task than reporting (Rajpurkar et al., 2018; Irvin et al., 2019).

Deployment Considerations

A model built on public chest data meets a site with its own projection mix, detector processing, label conventions and prevalence. The view position and patient orientation recorded in DICOM can be used to route images and to report results by projection; preprocessing must match that of the training data, which for MIMIC-CXR-JPG includes per-image normalization and histogram equalization (Johnson et al., 2024); and thresholds and probabilities set on enriched or differently labelled data must be re-established on local data with trusted labels and checked by patient subgroup. The public datasets’ terms restrict redistribution (see Datasets). Reader studies, prospective evaluation, regulation and post-market monitoring are the subject of clinical validation of medical imaging AI.