CheXpert

CheXpert is a collection of chest radiographs from Stanford, each study labelled for 14 observations as positive, negative or uncertain by a rule-based program that reads the radiology report. Irvin, Rajpurkar and colleagues at Stanford presented it at AAAI-19: 224,316 chest radiographs of 65,240 patients, examined between October 2002 and July 2017 in inpatient and outpatient centres, with frontal and lateral views (Irvin et al., 2019). Two things set it apart from the earlier ChestX-ray14: an explicit “uncertain” label, and validation and test sets labelled by board-certified radiologists who read the images, with three further radiologists’ reads kept for comparison. The dataset came with a public competition on five observations, now closed, whose leaderboard lists for each model an “AUC” and the number of radiologists below the model’s curves.

Chest X-ray Analysis covers the CheXpert labeller, the label sets of other chest datasets and the results of the five uncertainty policies. Computer Vision in Radiology covers the reader comparison. This article covers the dataset itself: its label format and counts, the splits and how their reference standards were built, the competition’s scoring and what its page leaves undefined, and access. The scoring is worked through on synthetic data, because no patient image is used here.

Purpose

The authors name three things that progress in automated radiograph reading needed: datasets that are large, that have strong reference standards, and that provide expert human performance for comparison (Irvin et al., 2019). They give ChestX-ray14 as the most commonly used benchmark before theirs, and call it problematic as a benchmark for comparing models because its test labels, like its training labels, come from an automatic labeller. CheXpert keeps report-mined labels for training, where size matters, and puts radiologists’ reads of the images in the evaluation sets, where the reference matters.

The second aim is the uncertainty label. Reports hedge (“may represent”, “cannot exclude”), and the labeller marks such mentions as uncertain instead of dropping them or forcing them to a decision. The paper then asks how a classifier should be trained on them, which is the question the example below takes up.

Dataset Size

The paper and the competition page give 224,316 chest radiographs of 65,240 patients (Irvin et al., 2019; CheXpert page). The AIMI dataset page gives the same two counts (AIMI page, as of November 2026).

Table 1 of the paper counts studies in the training set by label, read here from the rendered table. For the five observations that the competition scores:

Observation	Positive (%)	Uncertain (%)	Negative (%)
Atelectasis	29,333 (15.63)	29,377 (15.66)	128,931 (68.71)
Cardiomegaly	23,002 (12.26)	6,597 (3.52)	158,042 (84.23)
Consolidation	12,730 (6.78)	23,976 (12.78)	150,935 (80.44)
Edema	48,905 (26.06)	11,571 (6.17)	127,165 (67.77)
Pleural Effusion	75,696 (40.34)	9,419 (5.02)	102,526 (54.64)

Each row of Table 1 sums to 187,641 studies (for atelectasis, 29,333 + 29,377 + 128,931), so that is the number of training studies the table describes (our arithmetic). The table has no column for unmentioned observations, although the label format has a blank for them; in our reading the “Negative” column includes them. Uncertain labels are common for some observations: for consolidation the uncertain count is almost twice the positive count, which the authors note when they discuss ignoring uncertain labels. Support Devices (56.4%) and Lung Opacity (49.39%) have the highest positive shares, and Pleural Other (1.3%) the lowest.

The validation set has 200 studies of 200 patients, and the test set 500 studies of 500 patients (see Splits). The paper does not give the number of radiographs in either.

Classes

There are 14 observations: No Finding, Enlarged Cardiomediastinum, Cardiomegaly, Lung Lesion, Lung Opacity, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other, Fracture and Support Devices. Twelve are pathologies; “Support Devices” and “No Finding” are the other two (Irvin et al., 2019). Chest X-ray Analysis gives the authors’ reasons for the choice, including why pneumonia, a clinical diagnosis, was kept.

“No Finding” is derived, not mined: it is positive when no pathology is labelled positive or uncertain. Its uncertain count in Table 1 is therefore 0.

The competition scores five of the 14, which the authors call the competition tasks, chosen for clinical importance and prevalence in the validation set: atelectasis, cardiomegaly, consolidation, edema and pleural effusion.

Annotation Types

Training labels from reports. Each study has one label per observation, mined from the Impression section of its report in three stages: mention extraction, mention classification and mention aggregation. Chest X-ray Analysis describes the stages and the labeller’s accuracy against the earlier NIH labeller. The aggregation rules fix the label (Irvin et al., 2019):

• positive (1) if at least one mention is classified positive;
• uncertain if no mention is positive and at least one is uncertain;
• negative (0) if there is at least one negative mention and none of the above;
• blank if the observation is not mentioned.

The competition page gives the file encoding: blank for unmentioned, 0 for negative, −1 for uncertain and 1 for positive. The paper writes the uncertain label as $u$; the page uses −1 in this encoding and $u$ in its description of the policies. The labeller is open source in the stanfordmlgroup/chexpert-labeler repository, under the MIT License according to GitHub’s repository record (its code was not read for this article). The page notes that MIMIC-CXR’s labels come from the same labeller.

Reference reads of the images. In the validation and test sets, each radiologist read the radiographs, not the report, and classified each observation into one of four categories: present, uncertain likely, uncertain unlikely and absent. The four were binarized: present and uncertain likely count as positive, absent and uncertain unlikely as negative. The reference label is the majority vote of the binarized reads (Irvin et al., 2019).

The report evaluation set. A separate set of 1,000 reports from 1,000 patients was labelled by two radiologists, from the reports, as confidently present, confidently absent, uncertainly present or not mentioned, with disagreements resolved by consensus. It measures the labeller, not image models.

Splits

• Training. The bulk of the dataset, with report-mined labels (Table 1 counts 187,641 studies; see Dataset Size).
• Validation. 200 studies of 200 patients, sampled at random from the full dataset with no patient overlap with the report evaluation set. Three board-certified radiologists annotated each study; the majority vote of their binarized reads is the reference.
• Test. 500 studies of 500 patients, sampled at random from the 1,000 studies of the report test set. Eight board-certified radiologists annotated each study. The majority vote of five is the reference: three of the five also annotated the validation set, and the other two were sampled at random. The remaining three are the comparison radiologists (Irvin et al., 2019).

The test labels are now public: a README in the rajpurkarlab/cheXpert-test-set-labels repository, last changed on 21 October 2022, describes the release of each radiologist’s annotations (the five reference readers and the three benchmark readers), the majority-vote reference, the CheXpert and CheXbert labeller outputs for these studies, and predictions of some leaderboard models. It links the test images on Stanford AIMI’s portal. The repository has no licence file, so only its README and file listing were read here; the listing shows CSV files and no code. None of its files is used in this article.

Keeping the reference readers apart from the comparison readers matters for the reader comparison, as Computer Vision in Radiology explains.

Tasks

The task the paper defines is multi-label: from multi-view chest radiographs, predict the probability of each of the 14 observations. The baseline model takes one radiograph and outputs 14 probabilities; for a study with several views, it outputs the maximum probability across them (Irvin et al., 2019). The competition scores five of the observations at the study level. CheXpert is also used for medical image classification research beyond the competition (see Common Uses).

Evaluation

The paper’s protocol

The paper scores each competition task by the area under the ROC curve; ROC AUC covers the measure. On the validation set it reports 95% confidence intervals by DeLong’s non-parametric method, and, for each task, tests whether the best approach’s AUC is significantly greater than the worst’s with a one-sided DeLong test for correlated ROC curves, controlling for multiple testing with the Benjamini–Hochberg procedure (Irvin et al., 2019). ROC AUC describes DeLong’s method, and Statistical Significance in Evaluation the Benjamini–Hochberg procedure.

The values compared are not single networks. For each uncertainty policy the authors kept the best 10 checkpoints per run by average AUC over the competition tasks, ran each model three times, and averaged the output probabilities of the 30 checkpoints (Table 3 caption and text). The final model chose a policy per task on the validation set: U-Ones for atelectasis and edema, U-MultiClass for cardiomegaly and pleural effusion, U-SelfTrained for consolidation.

On the test set, each of the three comparison radiologists, and their majority vote, is scored by sensitivity, specificity and precision against the five-reader reference, and plotted as a point in ROC and precision–recall space beside the model’s curves. The authors “examine whether the radiologist operating points lie below the curves” (Irvin et al., 2019). No statistical test of model against radiologists was performed, which the paper lists as a limitation. The paper’s test results: AUCs from 0.85 (atelectasis) to 0.97 (pleural effusion); on cardiomegaly, edema and pleural effusion the model performs better than all three radiologists but not their majority vote; on consolidation above two of the three; on atelectasis all three radiologists perform better than the model.

The competition

The leaderboard on the competition page has two score columns, “AUC” and “Num Rads Below Curve”, and the page now says the competition is closed (CheXpert page, as of November 2026). Its entries are dated from January 2019 to November 2022. The page defines neither column beyond its description of the test set and the sentence about radiologist operating points. No evaluation code is published with it: the page links none, the test-labels repository holds data files only, and the chexpert-labeler repository is the labeller.

What the page leaves open, and the readings this article takes:

• “AUC”. Not defined. The Stanford Baseline (ensemble) row, dated 23 January 2019 and linked to the paper, lists 0.907; the five AUCs printed in the paper’s Figure 3 (0.85, 0.90, 0.90, 0.92, 0.97) average 0.908. In our reading the column is the mean of the five per-task AUCs; the sources do not explain the difference of 0.001 (rounding in the figure’s two-decimal legend is one possible reading, which the public material cannot check), and the page does not define the column.
• “Num Rads Below Curve”. Every value on the leaderboard is a multiple of 0.2 between 0 and 3.0. In our reading it is the number of comparison radiologists (0 to 3) below the model’s curve, averaged over the five tasks, so that 2.8 means 14 of 15 cases. The page does not say whether the ROC curve, the precision–recall curve or both decide “below”, or whether a point on the curve counts.
• Step or interpolated curve. An empirical ROC curve is a staircase of achievable thresholds; the straight lines between its vertices are reachable only by randomizing between two thresholds. A radiologist point between two vertices can be above the staircase and below the interpolated line. The curves in the paper’s Figure 3 are drawn as staircases in the rendered figure; which one the leaderboard used is not stated.

A disagreement between the page and the paper. The Stanford Baseline (ensemble) row lists 1.8 radiologists below the curve. The paper’s text gives 3 of 3 for cardiomegaly, edema and pleural effusion, 2 of 3 for consolidation and 0 for atelectasis, and the panel titles of Figure 3 read “(>0 rads)”, “(>3 rads)”, “(>2 rads)”, “(>3 rads)” and “(>3 rads)”. That is 11 of 15, a mean of 2.2, against the leaderboard’s 1.8. The sources do not explain the difference; a different curve convention, a different rule for points on the curve or a different submission are possible readings, and none can be checked from the public material.

Example: the five uncertainty policies at their population optimum

The paper compares five ways of training on uncertain labels (Irvin et al., 2019):

• U-Ignore masks uncertain labels out of the binary cross-entropy loss;
• U-Zeros maps every uncertain label to 0 (the paper’s text and the page spell it U-Zeroes, its Table 3 U-Zeros);
• U-Ones maps every uncertain label to 1;
• U-SelfTrained trains a U-Ignore model to convergence, then relabels each uncertain label with that model’s predicted probability (a soft label) and trains again, a form of semi-supervised learning;
• U-MultiClass treats “uncertain” as a third class with probabilities $p_0, p_1, p_u$, and outputs at test time the positive probability “after applying a softmax restricted to the positive and negative classes”.

Their validation AUCs and the two significant differences are in Chest X-ray Analysis. This example asks what the policies are worth before any finite-sample effect: what each one’s training loss is minimized by, and how those minimizers score under the leaderboard’s two columns.

Population targets. Let $x$ be the input and $p_0, p_1, p_u$ the probabilities that the report label is negative, positive or uncertain given $x$. A loss that is cross-entropy against a target $t$ is minimized, for each $x$, by the expected target. So the population-optimal outputs are:

Policy	Expected target given $x$
U-Zeros	$p_1$
U-Ones	$p_1 + p_u$
U-Ignore	$q = p_1 / (p_0 + p_1)$
U-SelfTrained	$p_1 + p_u\,q$
U-MultiClass	$p_1 / (p_0 + p_1)$ (restricted softmax of the three-class optimum)

Since $p_0 + p_1 = 1 - p_u$, the U-SelfTrained target is $p_1 + p_u p_1/(1 - p_u) = p_1/(1-p_u) = q$. U-Ignore, U-SelfTrained and U-MultiClass therefore have the same population optimum, and any difference between them, such as the paper’s significant U-MultiClass over U-Ignore difference on cardiomegaly, comes from finite data, optimization or ensembling, not from what the losses aim at (our reading, supported by the identity and by the check below). The paper’s other significant difference, U-Ones over U-Zeros on atelectasis, is not covered by this argument: those two targets differ, and the example shows that the difference in targets alone can separate them.

Why a second variable is needed. AUROC depends only on how a score orders the cases. If the report’s uncertainty depends only on how strong the finding is, every target above is a function of finding strength alone, and if all of them increase with it, they order the cases identically and all five tie. A separation needs the uncertainty to depend on something else as well, such as the view (anteroposterior, AP, or posteroanterior, PA). The tie also needs every score to order the strength bins alike, so the code prints how many pairs of cells each score orders against the best possible ranking.

Set-up. Everything is computed exactly over 20 cells; nothing is sampled or trained.

• Input. $x = (b, v)$: a finding-strength bin $b \in \{0, \dots, 9\}$ with $P(b) = 1/10$, and a view $v$, PA with probability 0.6 and AP with 0.4. Each cell is rendered as a noise-free $48 \times 48$ synthetic patch: a Gaussian blob of amplitude $b/9$ (the finding) beside a disc of radius 12 (PA) or 14 (AP), the disc being our stand-in for a view difference. A fixed matched filter reads $b$ and the disc’s width reads $v$, and the code checks that both are recovered for all 20 cells.
• Truth and report. $P(y = 1 \mid b) = \sigma(b - 6)$, the same in both views. With probability $p_u(b, v, y)$ the report label is uncertain; otherwise it is positive with probability 0.9 if $y = 1$ and 0.03 if $y = 0$.
• Five synthetic tasks, standing in for the five competition tasks so that a mean over tasks and a count of radiologists exist. They are uncertainty mechanisms, not models of the real observations:
  • T1, borderline: $p_u = 0.4\exp(-((b-6)/1.5)^2)$;
  • T2, uncertain means likely: $p_u = 0.30$ if $y = 1$, else 0.05;
  • T3, view-driven: $p_u = 0.35$ on AP, 0.05 on PA, whatever $b$ and $y$;
  • T4, likely on AP: $p_u = 0.45$ for AP positives, else 0.05;
  • T5, unlikely on AP: $p_u = 0.45$ for AP negatives, else 0.05.
• Readers. All readers see the image, that is $b$, and read independently of each other given it, with $P(\text{read} = 1 \mid b) = \sigma(s\,(b - c))$. The reference is the majority (at least 3 of 5) of five readers with $s = 2.0$, $c = 6.0$. The three compared readers are R1 ($s = 1.5$, $c = 6.0$), R2 (1.5, 7.0, a cautious reader) and R3 (1.0, 5.0, a liberal and less consistent one). Because the readers see only $b$, the reference does not depend on the view, and its two views tie.
• The ceiling. Beside the five policies the code scores $P(\text{ref} = 1 \mid x)$, the exact posterior of the reference. By the Neyman–Pearson lemma, ranking by it gives the highest interpolated ROC curve, and so the highest AUROC, that any score of $x$ can reach; it is the yardstick for each policy.
• Measures. Exact AUROC against the reference, as a sum over weighted cells with ties counted one half, and the mean over the five tasks. Each reader’s (FPR, TPR) against the reference. The number of readers strictly below each curve (margin above $10^{-12}$), under the step curve (the best deterministic threshold whose FPR does not exceed the reader’s) and under the interpolated curve, given per task, as a total and as a mean. “Below” uses the ROC curve only.
• Checks. Each closed-form target is compared with a numerical minimizer of its policy’s expected loss (SciPy); the AUROC sum with scikit-learn’s roc_auc_score and with the trapezoid under the interpolated curve; and the dependence of $p_u$ on $b$, $v$ and $y$ with the task list. Scores are rounded to 12 decimals before ranking, so that cells that are equal in exact arithmetic (the same $b$ in two views) are tied rather than ordered by floating-point noise.

The hypotheses, fixed before the pilot:

• H1: U-Ignore, U-SelfTrained and U-MultiClass give identical scores, AUROCs and reader counts in every task.
• H2: in T1 and T2, all five policies reach the ceiling’s AUROC (gap at most $10^{-12}$), given that none orders a pair of cells against it.
• H3: in T3, U-Ignore reaches the ceiling, and U-Zeros and U-Ones fall below it.
• H4: in T4, U-Ones has the highest AUROC of the five, strictly above U-Zeros and U-Ignore.
• H5: in T5, U-Zeros has the highest AUROC of the five, strictly above U-Ones and U-Ignore.
• H6: for every score and task, at least as many readers lie below the interpolated curve as below the step curve.
• H7a: under the interpolated convention, all three readers lie below the ceiling’s curve in every task.
• H7b: a score that orders no pair of cells against the ceiling has the ceiling’s reader count under both conventions.

"""CheXpert's five uncertainty policies at their population optimum, scored as the competition page scores:
mean AUROC over five tasks against a five-reader majority reference, and "Num Rads Below Curve" for three
compared readers, under a step and an interpolated ROC curve. Exact sums over 20 cells; nothing is trained."""
import os
for k in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[k] = "1"
import numpy as np
from scipy.optimize import minimize, minimize_scalar
from scipy.stats import binom
from sklearn.metrics import roc_auc_score

B, VIEWS = 10, ("PA", "AP")
P_B = np.full(B, 1 / B)
P_V = np.array([0.6, 0.4])
P_Y = 1 / (1 + np.exp(-(np.arange(B) - 6.0)))          # P(y = 1 | b), the same for both views
SENS_LAB, FPR_LAB = 0.9, 0.03                           # report labeller when it does not say "uncertain"
SIG = lambda z: 1 / (1 + np.exp(-z))
# readers see the image, i.e. the finding strength b, and read independently of each other given it:
# P(read = 1 | b) = SIG(slope * (b - centre)); five reference readers alike, three compared readers
REF = (2.0, 6.0)
READERS = {"R1": (1.5, 6.0), "R2": (1.5, 7.0), "R3": (1.0, 5.0)}
POLICIES = ("U-Zeros", "U-Ones", "U-Ignore", "U-SelfTrained", "U-MultiClass")

def p_uncertain(task, b, v, y):
    if task == "T1 borderline":
        return 0.4 * np.exp(-((b - 6.0) / 1.5) ** 2)
    if task == "T2 u-means-likely":
        return 0.30 if y == 1 else 0.05
    if task == "T3 view-driven":
        return 0.35 if v == 1 else 0.05
    if task == "T4 likely-on-AP":
        return 0.45 if (v == 1 and y == 1) else 0.05
    if task == "T5 unlikely-on-AP":
        return 0.45 if (v == 1 and y == 0) else 0.05
TASKS = ("T1 borderline", "T2 u-means-likely", "T3 view-driven", "T4 likely-on-AP", "T5 unlikely-on-AP")

# ---------- 1. images: one noise-free 48 x 48 patch per cell; the model input x = (b, v) is read from it ----------
yy, xx = np.mgrid[0:48, 0:48]
TEMPLATE = np.exp(-((yy - 24) ** 2 + (xx - 36) ** 2) / (2 * 3.0 ** 2))   # the finding: a Gaussian blob, sigma 3
WIN = (slice(15, 34), slice(29, 45))                    # matched-filter window, clear of the disc

def render(b, v):
    radius = 12 if v == 0 else 14                       # a wider disc marks the AP view (our stand-in)
    disc = ((yy - 24) ** 2 + (xx - 13) ** 2 <= radius ** 2).astype(float)
    return 0.5 * disc + (b / 9) * TEMPLATE

def read_features(img):
    t = TEMPLATE[WIN]
    amp = (img[WIN] * t).sum() / (t * t).sum()          # matched filter: least-squares amplitude of the blob
    width = int((img[24, :30] > 0.25).sum())            # disc width on the centre row
    return int(round(9 * amp)), int(width > 2 * 12 + 1)

cells = [(b, v) for b in range(B) for v in range(2)]
read = [read_features(render(b, v)) for b, v in cells]
print(f"data: {len(cells)} cells (10 bins x 2 views); features recovered from images: "
      f"{sum(r == c for r, c in zip(read, cells))}/{len(cells)}")

# ---------- 2. exact AUROC over weighted cells (ties count 1/2) and the two curve conventions ----------
def auroc(s, wp, wn):
    tot = 0.0
    for i in range(len(s)):
        tot += wp[i] * (wn[s < s[i]].sum() + 0.5 * wn[s == s[i]].sum())
    return tot / (wp.sum() * wn.sum())

def roc_vertices(s, wp, wn):
    fpr, tpr = [0.0], [0.0]
    for t in np.unique(s)[::-1]:                        # threshold "score >= t", from high to low
        tpr.append(wp[s >= t].sum() / wp.sum())
        fpr.append(wn[s >= t].sum() / wn.sum())
    return np.array(fpr), np.array(tpr)

def curve_at(f, fpr, tpr, how):
    if how == "step":                                   # best deterministic threshold with FPR <= f
        return tpr[fpr <= f + 1e-15].max()
    best = 0.0                                          # interpolated: highest segment through f
    for k in range(len(fpr) - 1):
        f0, f1, t0, t1 = fpr[k], fpr[k + 1], tpr[k], tpr[k + 1]
        if f0 <= f <= f1:
            best = max(best, t1 if f1 == f0 else t0 + (t1 - t0) * (f - f0) / (f1 - f0))
    return best

# ---------- 3. population quantities for one task ----------
def population(task):
    P = np.zeros((B, 2, 2, 3))                          # P(r | b, v, y) over r = (0, 1, u)
    for b in range(B):
        for v in range(2):
            for y in range(2):
                pu = p_uncertain(task, b, v, y)
                p1 = (1 - pu) * (SENS_LAB if y else FPR_LAB)
                P[b, v, y] = [1 - pu - p1, p1, pu]
    joint = P_B[:, None, None] * P_V[None, :, None] * np.stack([1 - P_Y, P_Y], -1)[:, None, :]   # P(b, v, y)
    pr = (joint[..., None] * P).sum(2)                  # P(b, v, r)
    px = pr.sum(-1)
    p0, p1, pu = (pr[..., i] / px for i in range(3))
    q = p1 / (p0 + p1)
    scores = {"U-Zeros": p1, "U-Ones": p1 + pu, "U-Ignore": q,
              "U-SelfTrained": p1 + pu * q, "U-MultiClass": p1 / (p0 + p1)}
    p_ref = binom.sf(2, 5, SIG(REF[0] * (np.arange(B) - REF[1])))[:, None]   # P(ref = 1 | b): >= 3 of 5
    pxv = joint.sum(-1)                                 # P(b, v)
    wpos = pxv * p_ref                                  # P(b, v, ref = 1)
    wneg = pxv * (1 - p_ref)
    scores["ceiling"] = wpos / (wpos + wneg)            # P(ref = 1 | x), the best possible ranking
    readers = {}
    for name, (slope, centre) in READERS.items():
        p_read = SIG(slope * (np.arange(B) - centre))[:, None]
        tp = (wpos * p_read).sum() / wpos.sum()
        fpr_ = (wneg * p_read).sum() / wneg.sum()
        readers[name] = (fpr_, tp)
    return P, (p0, p1, pu, q), scores, wpos, wneg, readers

# ---------- 4. checks of the derivation: each closed form minimizes its policy's expected loss ----------
def bce_argmin(a, c):                                   # minimize -(a log s + c log(1 - s))
    return minimize_scalar(lambda s: -(a * np.log(s) + c * np.log(1 - s)),
                           bounds=(1e-12, 1 - 1e-12), method="bounded", options={"xatol": 1e-13}).x

def multiclass_argmin(p0, p1, pu):                      # logits (0, z1, zu); restricted softmax at test time
    f = lambda z: -(p0 * 0 + p1 * z[0] + pu * z[1]) + np.log(1 + np.exp(z[0]) + np.exp(z[1]))
    z = minimize(f, np.zeros(2), method="BFGS", options={"gtol": 1e-12}).x
    return np.exp(z[0]) / (1 + np.exp(z[0]))

worst_min, worst_sk, worst_trap, worst_sum, worst_id = 0.0, 0.0, 0.0, 0.0, 0.0
results = {}
for task in TASKS:
    P, (p0, p1, pu, q), S, wpos, wneg, readers = population(task)
    worst_sum = max(worst_sum, np.abs(P.sum(-1) - 1).max())
    for b in range(B):
        for v in range(2):
            a0, a1, au, qq = p0[b, v], p1[b, v], pu[b, v], q[b, v]
            num = {"U-Zeros": bce_argmin(a1, a0 + au), "U-Ones": bce_argmin(a1 + au, a0),
                   "U-Ignore": bce_argmin(a1, a0), "U-SelfTrained": bce_argmin(a1 + au * qq, a0 + au * (1 - qq)),
                   "U-MultiClass": multiclass_argmin(a0, a1, au)}
            for k in POLICIES:
                worst_min = max(worst_min, abs(num[k] - S[k][b, v]))
    worst_id = max(worst_id, np.abs(S["U-SelfTrained"] - S["U-Ignore"]).max(),
                   np.abs(S["U-MultiClass"] - S["U-Ignore"]).max())
    s_flat = {k: np.round(v.ravel(), 12) for k, v in S.items()}   # equal in exact arithmetic -> tied
    wp, wn = wpos.ravel(), wneg.ravel()
    auc = {}
    for k, s in s_flat.items():
        auc[k] = auroc(s, wp, wn)
        sk = roc_auc_score(np.r_[np.ones(20), np.zeros(20)], np.r_[s, s], sample_weight=np.r_[wp, wn])
        worst_sk = max(worst_sk, abs(sk - auc[k]))
        f, t = roc_vertices(s, wp, wn)
        worst_trap = max(worst_trap, abs(np.sum((f[1:] - f[:-1]) * (t[1:] + t[:-1]) / 2) - auc[k]))
    grid = np.array([[[p_uncertain(task, b, v, y) for y in range(2)] for v in range(2)] for b in range(B)])
    depends = [np.ptp(grid, axis=ax).max() > 0 for ax in (0, 1, 2)]     # does pu vary with b, with v, with y?
    disc = {k: int(sum((s_flat[k][i] - s_flat[k][j]) * (s_flat["ceiling"][i] - s_flat["ceiling"][j]) < 0
                       for i in range(20) for j in range(20)) // 2) for k in s_flat}
    results[task] = (auc, readers, s_flat, wp, wn, depends, disc)

print(f"checks: |sum_r P(r|b,v,y) - 1| max {worst_sum:.1e}; closed forms vs numerical loss minimizers max "
      f"|diff| {worst_min:.1e}; U-SelfTrained and U-MultiClass vs U-Ignore scores max |diff| {worst_id:.1e}")
print(f"        AUROC (pairs) vs sklearn roc_auc_score max |diff| {worst_sk:.1e}; "
      f"vs trapezoid under interpolated curve {worst_trap:.1e}")
print("pu depends on (b, v, y):", "; ".join(f"{t.split()[0]} {tuple(int(d) for d in results[t][5])}" for t in TASKS))
print("cell pairs ordered against the ceiling (discordant), per task:")
for t in TASKS:
    d = results[t][6]
    print(f"  {t:<19}" + "  ".join(f"{k} {d[k]}" for k in POLICIES))

# ---------- 5. the competition's two columns ----------
print("\nAUROC against the 5-reader majority reference")
cols = POLICIES + ("ceiling",)
print(f"{'task':<19}" + "".join(f"{k:>15}" for k in cols) + f"{'max-min (5)':>13}")
for t in TASKS:
    a = results[t][0]
    spread = max(a[k] for k in POLICIES) - min(a[k] for k in POLICIES)
    print(f"{t:<19}" + "".join(f"{a[k]:>15.6f}" for k in cols) + f"{spread:>13.1e}")
print(f"{'mean of 5 tasks':<19}" + "".join(f"{np.mean([results[t][0][k] for t in TASKS]):>15.6f}" for k in cols))
print("ceiling minus AUROC (decides ties at 1e-12):")
for t in TASKS:
    a = results[t][0]
    print(f"  {t:<19}" + "  ".join(f"{k} {a['ceiling'] - a[k]:.1e}" for k in POLICIES))

print("\nreader operating points against the reference (FPR, TPR):")
for t in TASKS:
    print(f"  {t:<19}" + "  ".join(f"{n} ({fp:.4f}, {tp:.4f})" for n, (fp, tp) in results[t][1].items()))

for how in ("step", "interpolated"):
    print(f"\nreaders below the curve, {how} convention (margin = curve TPR - reader TPR at the reader's FPR)")
    counts = {k: [] for k in cols}
    for t in TASKS:
        auc, readers, s_flat, wp, wn, _, _ = results[t]
        line = []
        for k in cols:
            f, tt = roc_vertices(s_flat[k], wp, wn)
            m = [curve_at(fp, f, tt, how) - tp for fp, tp in readers.values()]
            counts[k].append(sum(x > 1e-12 for x in m))
            line.append(f"{k} {counts[k][-1]} [" + " ".join(f"{x:+.4f}" for x in m) + "]")
        print(f"  {t:<19}" + "  ".join(line))
    print(f"  {'total / mean':<19}" + "  ".join(f"{k} {sum(c)} / {np.mean(c):.1f}" for k, c in counts.items()))

data: 20 cells (10 bins x 2 views); features recovered from images: 20/20
checks: |sum_r P(r|b,v,y) - 1| max 0.0e+00; closed forms vs numerical loss minimizers max |diff| 5.8e-07; U-SelfTrained and U-MultiClass vs U-Ignore scores max |diff| 1.1e-16
        AUROC (pairs) vs sklearn roc_auc_score max |diff| 2.2e-16; vs trapezoid under interpolated curve 2.2e-16
pu depends on (b, v, y): T1 (1, 0, 0); T2 (0, 0, 1); T3 (0, 1, 0); T4 (0, 1, 1); T5 (0, 1, 1)
cell pairs ordered against the ceiling (discordant), per task:
  T1 borderline      U-Zeros 0  U-Ones 0  U-Ignore 0  U-SelfTrained 0  U-MultiClass 0
  T2 u-means-likely  U-Zeros 0  U-Ones 0  U-Ignore 0  U-SelfTrained 0  U-MultiClass 0
  T3 view-driven     U-Zeros 7  U-Ones 17  U-Ignore 0  U-SelfTrained 0  U-MultiClass 0
  T4 likely-on-AP    U-Zeros 5  U-Ones 0  U-Ignore 0  U-SelfTrained 0  U-MultiClass 0
  T5 unlikely-on-AP  U-Zeros 2  U-Ones 18  U-Ignore 0  U-SelfTrained 0  U-MultiClass 0

AUROC against the 5-reader majority reference
task                       U-Zeros         U-Ones       U-Ignore  U-SelfTrained   U-MultiClass        ceiling  max-min (5)
T1 borderline             0.993260       0.993260       0.993260       0.993260       0.993260       0.993260      0.0e+00
T2 u-means-likely         0.993260       0.993260       0.993260       0.993260       0.993260       0.993260      0.0e+00
T3 view-driven            0.987837       0.987390       0.993260       0.993260       0.993260       0.993260      5.9e-03
T4 likely-on-AP           0.982563       0.993260       0.993260       0.993260       0.993260       0.993260      1.1e-02
T5 unlikely-on-AP         0.993260       0.976990       0.993260       0.993260       0.993260       0.993260      1.6e-02
mean of 5 tasks           0.990036       0.988832       0.993260       0.993260       0.993260       0.993260
ceiling minus AUROC (decides ties at 1e-12):
  T1 borderline      U-Zeros 0.0e+00  U-Ones 0.0e+00  U-Ignore 0.0e+00  U-SelfTrained 0.0e+00  U-MultiClass 0.0e+00
  T2 u-means-likely  U-Zeros 0.0e+00  U-Ones 0.0e+00  U-Ignore 0.0e+00  U-SelfTrained 0.0e+00  U-MultiClass 0.0e+00
  T3 view-driven     U-Zeros 5.4e-03  U-Ones 5.9e-03  U-Ignore 0.0e+00  U-SelfTrained 0.0e+00  U-MultiClass 0.0e+00
  T4 likely-on-AP    U-Zeros 1.1e-02  U-Ones -1.1e-16  U-Ignore -2.2e-16  U-SelfTrained -2.2e-16  U-MultiClass -2.2e-16
  T5 unlikely-on-AP  U-Zeros 4.0e-12  U-Ones 1.6e-02  U-Ignore -1.1e-16  U-SelfTrained -1.1e-16  U-MultiClass -1.1e-16

reader operating points against the reference (FPR, TPR):
  T1 borderline      R1 (0.0774, 0.8572)  R2 (0.0245, 0.6728)  R3 (0.2048, 0.9073)
  T2 u-means-likely  R1 (0.0774, 0.8572)  R2 (0.0245, 0.6728)  R3 (0.2048, 0.9073)
  T3 view-driven     R1 (0.0774, 0.8572)  R2 (0.0245, 0.6728)  R3 (0.2048, 0.9073)
  T4 likely-on-AP    R1 (0.0774, 0.8572)  R2 (0.0245, 0.6728)  R3 (0.2048, 0.9073)
  T5 unlikely-on-AP  R1 (0.0774, 0.8572)  R2 (0.0245, 0.6728)  R3 (0.2048, 0.9073)

readers below the curve, step convention (margin = curve TPR - reader TPR at the reader's FPR)
  T1 borderline      U-Zeros 2 [-0.0041 +0.1803 +0.0887]  U-Ones 2 [-0.0041 +0.1803 +0.0887]  U-Ignore 2 [-0.0041 +0.1803 +0.0887]  U-SelfTrained 2 [-0.0041 +0.1803 +0.0887]  U-MultiClass 2 [-0.0041 +0.1803 +0.0887]  ceiling 2 [-0.0041 +0.1803 +0.0887]
  T2 u-means-likely  U-Zeros 2 [-0.0041 +0.1803 +0.0887]  U-Ones 2 [-0.0041 +0.1803 +0.0887]  U-Ignore 2 [-0.0041 +0.1803 +0.0887]  U-SelfTrained 2 [-0.0041 +0.1803 +0.0887]  U-MultiClass 2 [-0.0041 +0.1803 +0.0887]  ceiling 2 [-0.0041 +0.1803 +0.0887]
  T3 view-driven     U-Zeros 3 [+0.0816 +0.0676 +0.0911]  U-Ones 3 [+0.0531 +0.1803 +0.0903]  U-Ignore 2 [-0.0041 +0.1803 +0.0887]  U-SelfTrained 2 [-0.0041 +0.1803 +0.0887]  U-MultiClass 2 [-0.0041 +0.1803 +0.0887]  ceiling 2 [-0.0041 +0.1803 +0.0887]
  T4 likely-on-AP    U-Zeros 2 [+0.0816 -0.0467 +0.0911]  U-Ones 3 [+0.0531 +0.1803 +0.0903]  U-Ignore 3 [+0.0816 +0.1803 +0.0911]  U-SelfTrained 3 [+0.0816 +0.1803 +0.0911]  U-MultiClass 3 [+0.0816 +0.1803 +0.0911]  ceiling 2 [-0.0041 +0.1803 +0.0887]
  T5 unlikely-on-AP  U-Zeros 3 [+0.0816 +0.1803 +0.0911]  U-Ones 3 [+0.0531 +0.0112 +0.0903]  U-Ignore 3 [+0.0531 +0.1803 +0.0903]  U-SelfTrained 3 [+0.0531 +0.1803 +0.0903]  U-MultiClass 3 [+0.0531 +0.1803 +0.0903]  ceiling 2 [-0.0041 +0.1803 +0.0887]
  total / mean       U-Zeros 12 / 2.4  U-Ones 13 / 2.6  U-Ignore 12 / 2.4  U-SelfTrained 12 / 2.4  U-MultiClass 12 / 2.4  ceiling 10 / 2.0

readers below the curve, interpolated convention (margin = curve TPR - reader TPR at the reader's FPR)
  T1 borderline      U-Zeros 3 [+0.1356 +0.2217 +0.0920]  U-Ones 3 [+0.1356 +0.2217 +0.0920]  U-Ignore 3 [+0.1356 +0.2217 +0.0920]  U-SelfTrained 3 [+0.1356 +0.2217 +0.0920]  U-MultiClass 3 [+0.1356 +0.2217 +0.0920]  ceiling 3 [+0.1356 +0.2217 +0.0920]
  T2 u-means-likely  U-Zeros 3 [+0.1356 +0.2217 +0.0920]  U-Ones 3 [+0.1356 +0.2217 +0.0920]  U-Ignore 3 [+0.1356 +0.2217 +0.0920]  U-SelfTrained 3 [+0.1356 +0.2217 +0.0920]  U-MultiClass 3 [+0.1356 +0.2217 +0.0920]  ceiling 3 [+0.1356 +0.2217 +0.0920]
  T3 view-driven     U-Zeros 3 [+0.1356 +0.1106 +0.0920]  U-Ones 3 [+0.0542 +0.2217 +0.0903]  U-Ignore 3 [+0.1356 +0.2217 +0.0920]  U-SelfTrained 3 [+0.1356 +0.2217 +0.0920]  U-MultiClass 3 [+0.1356 +0.2217 +0.0920]  ceiling 3 [+0.1356 +0.2217 +0.0920]
  T4 likely-on-AP    U-Zeros 2 [+0.1356 -0.0036 +0.0920]  U-Ones 3 [+0.1356 +0.2217 +0.0920]  U-Ignore 3 [+0.1356 +0.2217 +0.0920]  U-SelfTrained 3 [+0.1356 +0.2217 +0.0920]  U-MultiClass 3 [+0.1356 +0.2217 +0.0920]  ceiling 3 [+0.1356 +0.2217 +0.0920]
  T5 unlikely-on-AP  U-Zeros 3 [+0.1356 +0.2217 +0.0920]  U-Ones 3 [+0.0542 +0.0551 +0.0903]  U-Ignore 3 [+0.1356 +0.2217 +0.0920]  U-SelfTrained 3 [+0.1356 +0.2217 +0.0920]  U-MultiClass 3 [+0.1356 +0.2217 +0.0920]  ceiling 3 [+0.1356 +0.2217 +0.0920]
  total / mean       U-Zeros 14 / 2.8  U-Ones 15 / 3.0  U-Ignore 15 / 3.0  U-SelfTrained 15 / 3.0  U-MultiClass 15 / 3.0  ceiling 15 / 3.0

What the output shows. The checks pass: all 20 cells are read back from their patches, the closed forms agree with the numerical minimizers to within the optimizer’s tolerance ($5.8 \times 10^{-7}$), and the AUROC sum agrees with scikit-learn and with the trapezoid to rounding error. U-SelfTrained and U-MultiClass reproduce U-Ignore’s scores to $1.1 \times 10^{-16}$, so the three columns are the same in every row of every table.

The AUROC table splits the tasks in two. In T1 and T2 the uncertainty does not depend on the view, so every target is a function of finding strength alone; every score orders the cells as the ceiling does (zero discordant pairs), and all five policies reach 0.993260. In T3 to T5 the uncertainty depends on the view, and the policies that change the target of uncertain cases pay for it where the change is wrong for the task: U-Zeros and U-Ones in T3, U-Zeros in T4 (1.1e-02 below the ceiling), U-Ones in T5 (1.6e-02). The U-Ignore family reaches the ceiling in all five tasks. In T3 this follows from the construction: the uncertainty there does not depend on $y$ given $x$, so dropping uncertain cases leaves $P(r = 1 \mid x, r \neq u) = 0.9\,P(y=1 \mid b) + 0.03\,(1 - P(y=1 \mid b))$, which orders the cells as the ceiling does. On the mean over tasks, the leaderboard’s AUC column, U-Zeros scores 0.990036 and U-Ones 0.988832 against 0.993260.

The reader counts depend on the convention as much as on the score. Under the interpolated convention every reader lies below the ceiling’s curve in every task, as the Neyman–Pearson lemma requires for readers who see only the image and are not threshold rules, and every policy except U-Zeros in T4 (2 of 3) keeps all three. Under the step convention the ceiling has only 2 readers below in every task: R1’s point (FPR 0.0774, TPR 0.8572) sits between two vertices of its staircase, 0.0041 above the step and 0.1356 below the interpolated line. The step convention also reverses the AUROC ranking in T3: U-Zeros and U-Ones, whose AUROCs are lower, have 3 readers below their staircases against the ceiling’s 2, and on the step totals the best-ranked score has the lowest count (10, a mean of 2.0, against 12 to 13 for the policies). A count of radiologists below the curve is therefore not a function of the ranking alone; with the convention undefined, small differences in the leaderboard’s second column are hard to interpret (our reading).

Outcomes against the hypotheses, from the printed output:

Hypothesis	Outcome
H1: U-Ignore, U-SelfTrained, U-MultiClass identical	held (scores to 1.1e-16; equal AUROCs, gaps and counts in every task)
H2: T1 and T2, all five at the ceiling	held (0 discordant pairs, gaps 0.0e+00 for all five in both tasks)
H3: T3, U-Ignore at the ceiling, U-Zeros and U-Ones below	held (gap 0.0e+00; 5.4e-03 and 5.9e-03)
H4: T4, U-Ones strictly highest	failed: U-Ones is at the ceiling (gap −1.1e-16) but tied with the U-Ignore family (−2.2e-16); it is above U-Zeros (1.1e-02)
H5: T5, U-Zeros strictly highest	failed: U-Zeros is 4.0e-12 below the ceiling, where the U-Ignore family sits (−1.1e-16), with 2 discordant pairs; it is above U-Ones (1.6e-02)
H6: interpolated count at least the step count	held, for every score and task
H7a: all three readers below the ceiling’s interpolated curve	held, 3 in every task
H7b: no discordant pairs, same count as the ceiling	held under the interpolated convention in every task and under the step convention in T1–T3; failed under the step convention in T4 (U-Ones and the U-Ignore family 3 against the ceiling’s 2) and T5 (the U-Ignore family 3 against 2)

H4 failed because U-Ignore also reached the ceiling in T4: it orders no pair of cells against the ceiling (printed). In our reading, dropping the uncertain AP positives lowers U-Ignore’s score in each AP cell by less than that score rises from one bin to the next, so the order across bins survives.

H5 failed by a margin of 4.0e-12, which is real but tiny. In our reading, from the construction: in T5 more AP negatives are uncertain, so U-Zeros, whose target is $p_1 = 0.9\,(1 - p_u^{+})P + 0.03\,(1 - p_u^{-})(1 - P)$, with $P = P(y = 1 \mid b)$ and $p_u^{+}$, $p_u^{-}$ the uncertainty rates of positives and negatives ($p_u^{+} = 0.05$ in both views), gives AP cells $0.855P + 0.0165(1 - P)$ and PA cells $0.855P + 0.0285(1 - P)$. Only at the lowest bins, where $P$ is 0.0025, 0.0067 and 0.0180 for $b = 0, 1, 2$, is that gap larger than the rise in $P$ from one bin to the next: $(1, \text{AP})$ scores about 0.0221 against 0.0305 for $(0, \text{PA})$, and $(2, \text{AP})$ about 0.0316 against 0.0340 for $(1, \text{PA})$ (our arithmetic). These two pairs match the printed count of 2.

H7b failed under the step convention in T4 and T5 and held elsewhere. In our reading, those scores order the cells as the ceiling does but break its ties between the two views of a bin, which adds vertices to the staircase and lifts it at some readers’ FPR; the interpolated curve is unchanged, because the added vertices lie on its segments.

Within this model, the population targets separate the policies only when the uncertainty depends on something other than the finding, and the U-Ignore family never fell below the ceiling in these five mechanisms. That is a property of these scenarios, not a ranking of the policies on real data, where the paper’s validation results differ by observation.

How the example was planned, and what changed. The scenarios, the five-way comparison and the hypotheses H1–H6 were written down before any of this article’s scripts ran, after a planning script from the batch’s research had printed AUROCs against the true label for three scenarios, one of them view-driven like T3. Three planning runs preceded the pilot:

• The first planning run modelled readers as reading the true label $y$ with fixed sensitivity and false positive rate. Every reader then lay above every curve in every task, because they used information that the image, the model’s only input, does not carry. The reader model was changed to readers who see only $b$, with the parameters above chosen once; H7 was rewritten as H7a and H7b before the next run. H4 and H5 had already failed in that run and were kept as written.
• The same run counted pairs of cells as ordered where exact arithmetic gives a tie (the same $b$ in two views, computed along different floating-point paths); rounding to 12 decimals before ranking was added as a precision repair.
• The line “ceiling minus AUROC” was added before the pilot, because six decimals cannot decide ties at $10^{-12}$.

Every outcome above was therefore seen in a planning run before the plan was fixed, and the example is an illustration of the mechanisms, not a test of them. The pilot and the first run printed the same output as the last planning run, and nothing was changed after the first run. The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2 on one BLAS thread, takes about a second, and two runs printed identical output.

License

As of November 2026, the AIMI dataset page for CheXpert has “Terms & Conditions” and “License” headings with no text under them on the public page. Its “Download here” link leads to Stanford AIMI’s portal, which was not opened, so the dataset’s licence (not read) is not described here beyond what the public pages below say. The dataset page asks users to cite the dataset by its DOI, 10.71718/y7pj-4v93.

The public AIMI pages describe the terms in general, as of November 2026:

• AIMI’s datasets are available to the research community at no charge for non-commercial research use, “subject to the applicable data use terms”, and are downloaded through Redivis (AIMI data page).
• The commercial-use page says the datasets are shared under a licence restricted to non-commercial purposes, and that companies can license a dataset by application to a Stanford committee, a signed licence agreement and an annual fee, given as $70,000 per dataset for agreements contracted in FY25 (through 31 August 2025).
• AIMI expects users to acknowledge Stanford AIMI in publications and presentations, in a wording given on the same data page.

No licence name is given here, because the licence text was not read. The test-labels repository has no licence file. The labeller’s repository is under the MIT License, according to GitHub’s record.

Limitations and Bias

• Training labels from reports. The training labels are the labeller’s reading of the report’s Impression section, with its errors and with the report’s own hedging. Chest X-ray Analysis gives the labeller’s measured accuracy and the upper and lower bounds of the report labels against the test reference.
• One institution. All studies come from one hospital system’s inpatient and outpatient centres. The sources name it differently: “Stanford Hospital” in the paper and on the competition page, “Stanford Health Care” on the AIMI page.
• Small evaluation sets. 200 validation and 500 test studies; the paper’s confidence intervals on the validation AUCs are several points wide (for atelectasis under U-Ignore, 0.759 to 0.877).
• No clinical context. Neither the radiologists nor the model had patient history or prior examinations, a limitation the authors acknowledge.
• The reader comparison. No statistical test of the model against the radiologists was performed (Irvin et al., 2019), and the leaderboard’s “Num Rads Below Curve” is undefined beyond the page’s description, as the example shows; its value for the paper’s own model differs between page and paper.
• Subgroups. Chest X-ray Analysis describes a study of underdiagnosis in patient subgroups that trained classifiers on CheXpert among other datasets.

Common Uses

• Benchmarking chest radiograph classifiers on the five competition tasks against a radiologist reference. Medical Foundation Models reports CheXzero’s zero-shot results on the CheXpert test set of 500 radiographs.
• Studying uncertain and noisy labels. The uncertain label and the paper’s five policies; Chest X-ray Analysis gives the paper’s results.
• Transfer and pretraining studies. Medical Image Classification describes Raghu et al.’s transfer study, which used CheXpert radiographs resized to 224 × 224.
• Explanation maps. The paper illustrates its model with Grad-CAM maps, and Chest X-ray Analysis describes a later study that measured saliency maps against radiologists’ segmentations on the CheXpert test set.

Historical Milestones

• January 2019, first leaderboard entry. The leaderboard’s earliest entry, the Stanford Baseline (ensemble), is dated 23 January 2019, with an AUC of 0.907 (CheXpert page).
• 2019, AAAI. The paper presents the dataset, the labeller, the five uncertainty policies on the validation set, and the comparison with three radiologists on the test set. Its model was a DenseNet-121, chosen over ResNet-152, Inception-v4 and SE-ResNeXt-101, on $320 \times 320$ inputs (Irvin et al., 2019).
• Co-release with MIMIC-CXR. The competition page announces MIMIC-CXR, 371,920 chest X-rays in 227,943 studies from Beth Israel Deaconess Medical Center (2011–2016), labelled by the same labeller, as a co-release; the paper describes MIMIC-CXR as announced but not yet public. Chest X-ray Analysis gives the counts of its later versions.
• 2019–2022, leaderboard. Entries are dated up to November 2022; the page now marks the competition as closed.
• October 2022, test labels released. The rajpurkarlab repository publishes the test set’s individual and majority-vote labels, and the README links the test images.

Related Datasets

• ChestX-ray14 (article): frontal radiographs from the NIH Clinical Center with report-mined labels in its test set as well, the dataset the CheXpert authors contrast theirs with.
• MIMIC-CXR: co-released, from another hospital, with labels from the CheXpert labeller.
• OpenI and PLCO: the paper lists OpenI, 7,470 frontal-view radiographs with reports labelled by human annotators, and the PLCO lung screening dataset, 185,421 images with a low prevalence of important pathologies, as earlier public chest radiograph collections without strong test reference standards.
• Chest X-ray Analysis compares CheXpert with ChestX-ray14, MIMIC-CXR, PadChest and VinDr-CXR in one table.

Resources

• The AAAI-19 paper in the publisher’s open-access proceedings (Irvin et al., 2019).
• The competition page with the closed leaderboard, https://stanfordmlgroup.github.io/competitions/chexpert/.
• The AIMI dataset page, https://aimi.stanford.edu/datasets/chexpert-chest-x-rays, with the dataset DOI 10.71718/y7pj-4v93; access is through AIMI’s portal and data use terms.
• The test-set labels, https://github.com/rajpurkarlab/cheXpert-test-set-labels (no licence file).
• The labeller, https://github.com/stanfordmlgroup/chexpert-labeler (MIT License).
• No official evaluation code was released; the competition’s scoring is described above from the page and the paper.