ChestX-ray14

ChestX-ray14 is a collection of frontal chest radiographs from the NIH Clinical Center in Bethesda, each labelled with the findings that a text-mining program read from its radiology report. Wang, Peng, Lu, Lu, Bagheri and Summers introduced it at CVPR 2017 as ChestX-ray8: 108,948 frontal images of 32,717 patients, collected between 1992 and 2015, with eight findings. They then extended it to fourteen findings, the version known as ChestX-ray14 (Wang et al., 2017). The authors describe it as at least an order of magnitude larger than OpenI, the largest public chest radiograph collection before it. It also came with a small set of boxes drawn by a radiologist, and a protocol for scoring boxes cut from a classifier’s heatmaps. The CheXpert authors called it the most commonly used benchmark for chest radiograph models at the time of their work (Irvin et al., 2019).

Chest X-ray Analysis covers the labeller, its measured accuracy, CheXNet and the label sets of other datasets. Computer Vision in Radiology places the dataset in the history of the field. This article covers the dataset itself: its two versions, the box set, the splits, the localization protocol as the paper defines it, and what that protocol rewards. The protocol is worked through on synthetic heatmaps, because no patient image is used here.

Purpose

The authors set out to test whether the image and report archive of a hospital’s picture archiving and communication system (PACS) could feed deep learning without manual labelling. Their argument has three parts (Wang et al., 2017):

• crowd workers cannot label pathology;
• findings are often small compared with a radiograph of about 3000 × 2000 pixels;
• the ImageNet-pretrained models that captioning and visual question answering rely on do not carry over directly to medical diagnosis, in their view.

Their answer was to mine labels from the reports and to train classifiers whose maps could also localize findings, a weakly supervised set-up. They describe fully automated reading of chest X-rays as still “an arduous journey”.

Dataset Size

ChestX-ray8 (CVPR 2017). The paper gives these counts (Wang et al., 2017):

• 108,948 frontal-view images of 32,717 patients;
• 24,636 images with one or more of the eight findings;
• 84,312 images labelled normal.

The paper’s Table 1 gives the images per finding:

Finding	Images	Finding	Images
Atelectasis	5,789	Mass	6,046
Cardiomegaly	1,010	Nodule	1,971
Effusion	6,331	Pneumonia	1,062
Infiltration	10,317	Pneumothorax	2,793

These counts sum to 35,319, more than the 24,636 abnormal images, because an image can carry several labels. The same table counts 2,435 reports in the OpenI subset that the authors used for checking their labeller.

The images were extracted from DICOM files, with their intensities rescaled by the default window stored in the DICOM header, and resized to 1024 × 1024 (Wang et al., 2017).

ChestX-ray14. The extended version has 112,120 images of 30,805 patients with fourteen findings. This version is described in the appendix of the final arXiv version, which could not be read for this article; the counts here are those that Chest X-ray Analysis and Computer Vision in Radiology give. The extension adds 3,172 images but has 1,912 fewer patients than ChestX-ray8. The sources read here do not explain the difference.

Classes

ChestX-ray8’s eight findings were short-listed on radiologists’ feedback (Wang et al., 2017): atelectasis, cardiomegaly, effusion, infiltration, mass, nodule, pneumonia and pneumothorax. ChestX-ray14 adds consolidation, edema, emphysema, fibrosis, pleural thickening and hernia. An image with none of the fourteen is labelled “No findings”, as Chest X-ray Analysis reports from the appendix.

The CVPR paper’s label for an image without findings is “Normal”. A report was coded Normal under either of two conditions:

• no disease at all was detected in it, not only none of the eight;
• it contained the mined concepts “normal” or “normal size”.

So a Normal image is not merely one without the listed findings (Wang et al., 2017). In the CVPR paper’s networks, a Normal image has an all-zero label vector, so the eight-way task is multi-label rather than a choice of one class.

Annotation Types

Image labels. Every label comes from the report, not from a reading of the image. Two tools, DNorm and MetaMap, found disease mentions in the findings and impression sections, and their results were merged to maximize recall. Rules on a syntactic dependency graph then removed negated and uncertain mentions (Wang et al., 2017). Chest X-ray Analysis describes the rules and the labeller’s measured precision and recall. The authors retrieved 3,851 OpenI reports with human annotations and used a subset of them, 2,435 reports in Table 1, as the gold standard; on it, the CVPR paper reports an overall precision of 0.90, recall of 0.91 and F1 of 0.90 for the eight findings.

Boxes. The authors selected 200 instances of each of the eight findings, 1,600 instances on 983 images (Wang et al., 2017). The annotation worked as follows:

• a board-certified radiologist was given an image and a finding name and drew a box around that finding only;
• each instance was stored in its own XML file, so an image with several findings has several files;
• the boxes were used only as ground truth for evaluating localization, never for training.

The authors suggested the boxes could also serve one-shot or low-shot learning. ChestX-ray14’s six added findings have no boxes in the CVPR paper. For other chest datasets with boxes, such as VinDr-CXR with three radiologists per training image, see Chest X-ray Analysis.

Splits

The two versions describe the split differently.

• CVPR paper. For ChestX-ray8, the authors “randomly shuffled the entire dataset” into training (70%), validation (10%) and testing (20%), and report classification results on the test set (Wang et al., 2017). The CVPR text does not say whether the shuffle kept each patient’s images together.
• Appendix. For ChestX-ray14, the appendix describes a split at the patient level, about 70/10/20, with the split files published, as Chest X-ray Analysis reports.

The two descriptions are reported here as they stand, not reconciled. Because patients have several images on average (108,948 images of 32,717 patients), an image-level shuffle would put images of the same patient on both sides. Medical Image Classification shows what that does to reported accuracy. Results from the two splits, or from random re-splits, are not comparable.

Tasks

• Multi-label classification: one binary decision per finding, each scored by its own ROC curve. The CVPR paper fine-tuned AlexNet, GoogLeNet, VGGNet-16 and ResNet-50 initialized from ImageNet. It added a weighting of positive and negative terms in the loss, because positive labels are sparse (Wang et al., 2017).
• Weakly supervised localization: the classifier’s maps are turned into boxes and compared with the radiologist’s boxes. No box is used in training.

Evaluation

Classification

Each finding is scored by the area under its ROC curve; ROC AUC gives the CVPR paper’s per-finding values for ResNet-50 and discusses the measure. The test labels come from the same labeller as the training labels, so the score measures agreement with mined labels. The CheXpert authors call using it to compare models problematic for that reason, and contrast it with CheXpert’s validation and test sets, which radiologists labelled (Irvin et al., 2019).

The CVPR paper disagrees with itself here: its text says the AUC values “are given in Table 4”, but they are in Table 3, and Table 4 holds the localization results.

Localization: the paper’s protocol

No official evaluation code was released. The protocol is defined in the CVPR paper’s text and Table 4 (Wang et al., 2017).

1. Heatmaps. For each finding, the network gives an $S \times S$ map, with $S \in \{8, 16, 32\}$ depending on the model. The map is the transition layer’s activations weighted by the prediction layer’s weights for that finding. The paper evaluates $32 \times 32$ maps for $1024 \times 1024$ images.
2. Boxes. The map’s intensities are normalized to $[0, 255]$ and thresholded at 60 and at 180 “individually”. Boxes are then drawn around the isolated regions of each binary map, the connected components.
3. Matching. Each computed box is compared with the radiologist’s box $G$ by two measures. One is intersection over union. The other is the intersection over the detected box’s own area, IoBB, which the paper likens to “Area of Precision or Purity”:

$$\mathrm{IoBB}(P, G) = \frac{|P \cap G|}{|P|}, \qquad \mathrm{IoU}(P, G) = \frac{|P \cap G|}{|P \cup G|}.$$

4. Correctness. Because the heatmaps are coarse, the authors observe that “the computed B-Boxes are often larger” than the drawn ones. They therefore count a localization as correct if $\mathrm{IoU} > T(\mathrm{IoU})$ or $\mathrm{IoBB} > T(\mathrm{IoBB})$.
5. Reporting. Table 4 gives, for each finding, the localization accuracy (Acc.) and the “average false positive number” (AFP) at $T(\mathrm{IoBB}) \in \{0.1, 0.25, 0.5, 0.75, 0.9\}$. It annotates 0.25 as “two times larger on both x and y axis than ground truth B-Boxes”. Results at varying $T(\mathrm{IoU})$ are in the supplementary material. No supplement is listed on the CVF page, so those results could not be read.

The paper leaves several steps open, and each choice below is our reading or is printed both ways in the example:

• Upsampling. The paper does not say how a $32 \times 32$ map becomes a $1024 \times 1024$ one. The example prints bilinear and nearest-neighbour upsampling, both on the centred grid that Image Interpolation describes.
• Normalization and threshold. We read the normalization as a min–max rescaling of the upsampled map, kept as floating point, and the thresholds as strict (> 60, > 180).
• Connectivity. The example uses 8-connectivity and prints how often 4-connectivity would change the boxes.
• Acc. The paper does not define it. We read it as the fraction of reference boxes hit by at least one correct box.
• AFP. The paper does not define it either. We read it as the number of computed boxes per image that fail the criterion. Table 4 fits this reading, since AFP never falls in any column as the threshold rises (nodule’s stays at 0.0772 from 0.5 on).
• $T(\mathrm{IoU})$. Table 4 is indexed by $T(\mathrm{IoBB})$ alone and does not say which $T(\mathrm{IoU})$ went with it. If the two thresholds are equal, the “or” rule is the IoBB rule, because IoBB is never smaller than IoU: $|P| \le |P \cup G|$. The example checks this box by box.

Two consequences of the definition, which the example also checks, decide what the criterion rewards. For a box that contains the reference, $P \cap G = G$ and $P \cup G = P$, so IoBB and IoU are both $|G|/|P|$. For a box inside the reference, $P \cap G = P$, so IoBB is 1 whatever its size. In our reading, therefore, IoBB gives boxes that contain the finding, the oversized case the authors cite as their reason for using it, no more credit than IoU does. Its leniency goes to boxes smaller than the finding.

Example: the protocol on synthetic heatmaps

The script below implements the protocol on synthetic data with exact reference boxes. It uses no radiograph and trains no network. Each “heatmap” is an error model, not a classifier’s output: the exact fraction of each $32 \times 32$-pixel cell that the reference box covers, blurred with a Gaussian of 0.8 cells and scaled to a peak of 1.

• Size classes. There are three, 60 images each: large (260–360 by 320–460 pixels), medium (90–150) and small (24–48). The sizes are our choice and are not statistics of the dataset’s boxes.
• Noisy maps. These add, with probability 0.4, a distractor blob away from the box, and Gaussian noise with a standard deviation of 0.03.
• Localizers. Four are scored:
  • the paper’s pipeline, with each upsampling and each threshold;
  • a degenerate localizer that returns a 6 × 6 box at the map’s peak;
  • one that returns the whole image;
  • for a check, an oracle that returns the reference box.
• Box convention. Boxes are half-open pixel ranges (row and column start included, end excluded), so a box’s area is its height times its width, with no “+1” (see Bounding Box for the conventions).

The script also does three other things:

• In Part A it scores six exact boxes as fractions.
• It checks the hand-written bilinear upsampling against scikit-image, find_objects boxes against regionprops, the box formula against pixel counts, and the IoU/IoBB identities on every scored box.
• In Part C it asks which denominators reproduce the Acc. values of the paper’s Table 4. Those values were typed from the table as printed.

import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
from fractions import Fraction
import numpy as np
from scipy import ndimage
from skimage.measure import regionprops
from skimage.transform import resize

IMG, CELL = 1024, 32                      # 1024 x 1024 image, 32 x 32 heatmap
TS = [Fraction(t) for t in ("0.1", "0.25", "0.5", "0.75", "0.9")]   # thresholds of the paper's Table 4

# Boxes are (r0, c0, r1, c1) in pixels, half-open: rows r0..r1-1, columns c0..c1-1.
def area(b): return max(0, b[2] - b[0]) * max(0, b[3] - b[1])
def inter(p, g): return area((max(p[0], g[0]), max(p[1], g[1]), min(p[2], g[2]), min(p[3], g[3])))
def iou(p, g): i = inter(p, g); return Fraction(i, area(p) + area(g) - i)
def iobb(p, g): return Fraction(inter(p, g), area(p))   # intersection over the detected box

# ---- Part A: the two measures on exact boxes ----
g = (400, 400, 500, 500)                                  # 100 x 100 reference box
cases = {"exact": g, "2x per axis, centred": (350, 350, 550, 550),
         "3x per axis, centred": (300, 300, 600, 600), "whole image": (0, 0, IMG, IMG),
         "10 x 10 inside": (445, 445, 455, 455), "shifted by half a width": (400, 450, 500, 550)}
print("Part A: exact boxes against a 100 x 100 reference")
for name, p in cases.items():
    u, b = iou(p, g), iobb(p, g)
    print(f"  {name:24s} IoU {str(u):>7s} = {float(u):.4f}   IoBB {str(b):>5s} = {float(b):.4f}"
          f"   IoBB>0.25 {b > Fraction(1, 4)!s:5s}  IoBB>=0.25 {b >= Fraction(1, 4)}")

# ---- Part B: synthetic heatmaps with exact reference boxes ----
SIZES = {"large": ((260, 360), (320, 460)), "medium": ((90, 150), (90, 150)), "small": ((24, 48), (24, 48))}
N = 60                                                     # images per size class
rng = np.random.default_rng(0)

def coverage(b):                          # fraction of each 32 x 32 cell covered by box b
    edges = np.arange(0, IMG + 1, CELL)
    fr = np.clip(np.minimum(edges[1:], b[2]) - np.maximum(edges[:-1], b[0]), 0, None) / CELL
    fc = np.clip(np.minimum(edges[1:], b[3]) - np.maximum(edges[:-1], b[1]), 0, None) / CELL
    return np.outer(fr, fc)

def make_case(size):
    (hlo, hhi), (wlo, whi) = SIZES[size]
    h, w = rng.integers(hlo, hhi + 1), rng.integers(wlo, whi + 1)
    r0, c0 = rng.integers(32, IMG - 32 - h + 1), rng.integers(32, IMG - 32 - w + 1)
    box = (int(r0), int(c0), int(r0 + h), int(c0 + w))
    clean = ndimage.gaussian_filter(coverage(box), 0.8, mode="constant")
    clean /= clean.max()                                   # finding response peaks at 1
    noisy = clean.copy()
    has_d = rng.random() < 0.4
    if has_d:                                              # a distractor blob somewhere else
        while True:
            cr, cc = rng.uniform(2, 30, size=2)
            if abs(cr * CELL - (r0 + h / 2)) > h / 2 + 96 or abs(cc * CELL - (c0 + w / 2)) > w / 2 + 96:
                break
        yy, xx = np.mgrid[0:32, 0:32] + 0.5
        noisy += rng.uniform(0.3, 0.9) * np.exp(-((yy - cr) ** 2 + (xx - cc) ** 2) / 2.0)
    noisy += rng.normal(0, 0.03, noisy.shape)
    return box, clean, noisy, has_d

cases_b = {s: [make_case(s) for _ in range(N)] for s in SIZES}

def hand_bilinear(hm):                    # centred grid, edge values held, separable
    x = (np.arange(IMG) + 0.5) / CELL - 0.5
    i0 = np.clip(np.floor(x).astype(int), 0, 31); i1 = np.clip(i0 + 1, 0, 31)
    t = np.clip(x - i0, 0, 1)
    rows = hm[i0] * (1 - t)[:, None] + hm[i1] * t[:, None]
    return rows[:, i0] * (1 - t) + rows[:, i1] * t

def upsample(hm, order): return resize(hm, (IMG, IMG), order=order, mode="edge", anti_aliasing=False)

def boxes_from(hm, order, thr, conn=8):   # the paper's box generation
    up = upsample(hm, order)
    norm = 255 * (up - up.min()) / (up.max() - up.min())
    st = np.ones((3, 3)) if conn == 8 else None
    lab, n = ndimage.label(norm > thr, structure=st)
    return [(s[0].start, s[1].start, s[0].stop, s[1].stop) for s in ndimage.find_objects(lab)], lab

def tiny_box(hm):                          # degenerate localizer: 6 x 6 box at the peak
    up = upsample(hm, 1)
    r, c = np.unravel_index(np.argmax(up), up.shape)
    return [(int(r) - 3, int(c) - 3, int(r) + 3, int(c) + 3)]

def score(preds_gts, rule, T):
    """Acc: fraction of reference boxes hit by a correct box; AFP: incorrect boxes per image."""
    hit, fp = 0, 0
    for preds, gt in preds_gts:
        ok = [(iou(p, gt) if rule == "IoU" else iobb(p, gt)) > T for p in preds]
        hit += any(ok); fp += len(ok) - sum(ok)
    return hit / len(preds_gts), fp / len(preds_gts)

# checks of the components
hm0 = cases_b["medium"][0][2]
print("\nChecks")
print(f"  hand bilinear vs skimage resize(order=1): max |diff| = {np.abs(hand_bilinear(hm0) - upsample(hm0, 1)).max():.1e}")
print(f"  np.repeat vs skimage resize(order=0): differing pixels = "
      f"{int((np.repeat(np.repeat(hm0, 32, 0), 32, 1) != upsample(hm0, 0)).sum())}")
mism = conn_diff = 0
for s in SIZES:
    for box, clean, noisy, _ in cases_b[s]:
        for thr in (60, 180):
            b8, lab = boxes_from(noisy, 1, thr)
            rp = sorted((r.bbox for r in regionprops(lab)), key=lambda t: t)
            mism += sorted(b8) != [tuple(int(v) for v in t) for t in rp]
            conn_diff += sorted(b8) != sorted(boxes_from(noisy, 1, thr, conn=4)[0])
print(f"  find_objects vs regionprops boxes: differing box sets = {mism} of {2 * 3 * N}")
print(f"  8- vs 4-connectivity: differing box sets = {conn_diff} of {2 * 3 * N}")

# run every localizer
runs, allpairs = {}, []
for s in SIZES:
    for variant in ("noise-free", "noisy"):
        k = 1 if variant == "noise-free" else 2
        for order, oname in ((1, "bilinear"), (0, "nearest"))[:1 if variant == "noise-free" else 2]:
            for thr in (60, 180):
                runs[(variant, "heatmap", oname, thr, s)] = [(boxes_from(c[k], order, thr)[0], c[0]) for c in cases_b[s]]
        runs[(variant, "tiny box", "bilinear", "-", s)] = [(tiny_box(c[k]), c[0]) for c in cases_b[s]]
        runs[(variant, "whole image", "-", "-", s)] = [([(0, 0, IMG, IMG)], c[0]) for c in cases_b[s]]
for v in runs.values():
    allpairs += [(p, gt) for preds, gt in v for p in preds]

# data and definition checks
oracle = [([c[0]], c[0]) for s in SIZES for c in cases_b[s]]   # the reference box itself
print("  oracle (reference box as prediction): Acc 1 and AFP 0 for every T and rule:",
      all(score(oracle, r, T) == (1.0, 0.0) for r in ("IoU", "IoBB") for T in TS))
for s in SIZES:
    hs = [c[0][2] - c[0][0] for c in cases_b[s]]; ws = [c[0][3] - c[0][1] for c in cases_b[s]]
    inside = all(0 <= c[0][0] and c[0][2] <= IMG and 0 <= c[0][1] and c[0][3] <= IMG for c in cases_b[s])
    print(f"  data {s:6s}: {len(hs)} images, heights {min(hs)}-{max(hs)}, widths {min(ws)}-{max(ws)}, "
          f"boxes in image {inside}, distractors {sum(c[3] for c in cases_b[s])}")
viol = sum(iobb(p, gt) < iou(p, gt) for p, gt in allpairs)
cont = [(p, gt) for p, gt in allpairs if p[0] <= gt[0] and p[1] <= gt[1] and p[2] >= gt[2] and p[3] >= gt[3]]
insd = [(p, gt) for p, gt in allpairs if p[0] >= gt[0] and p[1] >= gt[1] and p[2] <= gt[2] and p[3] <= gt[3]]
print(f"  {len(allpairs)} scored boxes: IoBB < IoU in {viol}; containing the reference {len(cont)}, "
      f"IoBB != IoU in {sum(iobb(p, g) != iou(p, g) for p, g in cont)}; inside it {len(insd)}, "
      f"IoBB != 1 in {sum(iobb(p, g) != 1 for p, g in insd)}")
either = sum((iou(p, gt) > T or iobb(p, gt) > T) != (iobb(p, gt) > T) for p, gt in allpairs for T in TS)
print(f"  'IoU > T or IoBB > T' differs from 'IoBB > T' in {either} of {len(allpairs) * len(TS)} box-threshold pairs")
pm = 0
for p, gt in allpairs[:400]:
    mp = np.zeros((IMG, IMG), bool); mg = np.zeros((IMG, IMG), bool)
    mp[max(p[0], 0):p[2], max(p[1], 0):p[3]] = True; mg[gt[0]:gt[2], gt[1]:gt[3]] = True
    pm += Fraction(int((mp & mg).sum()), int(mp.sum())) != iobb(p, gt)
print(f"  box formula vs pixel count (first 400 boxes): IoBB mismatches = {pm}")

# ---- results: Acc / AFP for each T ----
print("\nPart B: Acc / AFP at T = " + ", ".join(str(float(t)) for t in TS))
show = [("heatmap", "bilinear", 60), ("heatmap", "bilinear", 180), ("heatmap", "nearest", 60),
        ("heatmap", "nearest", 180), ("tiny box", "bilinear", "-"), ("whole image", "-", "-")]
for variant in ("noise-free", "noisy"):
    print(f"[{variant}]")
    for loc, up, thr in (show[:2] if variant == "noise-free" else show):
        for s in SIZES:
            for rule in ("IoU", "IoBB"):
                if loc == "whole image" and rule == "IoBB": continue
                cells = [score(runs[(variant, loc, up, thr, s)], rule, T) for T in TS]
                lab = f"{loc} {up} {thr}" if loc == "heatmap" else loc
                print(f"  {lab:20s} {s:6s} {rule:4s} " + " ".join(f"{a:.3f}/{f:.3f}" for a, f in cells))

# ---- Part C: which denominators fit Table 4's accuracies? ----
table4 = {"Atelectasis": [0.7277, 0.5500, 0.2833, 0.1666, 0.1333], "Cardiomegaly": [0.9931, 0.9794, 0.8767, 0.7260, 0.6849],
          "Effusion": [0.7124, 0.5424, 0.3333, 0.2418, 0.2091], "Infiltration": [0.7886, 0.5772, 0.4227, 0.3252, 0.2520],
          "Mass": [0.4352, 0.2823, 0.1411, 0.1176, 0.0588], "Nodule": [0.1645, 0.0506, 0.0126, 0.0126, 0.0126],
          "Pneumonia": [0.7500, 0.5583, 0.3833, 0.2583, 0.2416], "Pneumothorax": [0.4591, 0.3469, 0.1836, 0.1020, 0.0816]}
def fits(n, v, mode):                    # is there an integer k with k/n shown as v?
    t = round(v * 10000)
    return any((10000 * k // n if mode == "trunc" else (20000 * k + n) // (2 * n)) == t for k in range(n + 1))
print("\nPart C: denominators n <= 200 that reproduce every Acc. of a Table 4 column")
for name, vals in table4.items():
    out = []
    for mode in ("trunc", "round"):
        ns = [n for n in range(1, 201) if all(fits(n, v, mode) for v in vals)]
        out.append(f"{mode}: {ns[:4]}{' ...' if len(ns) > 4 else ''} ({len(ns)})")
    print(f"  {name:13s} " + "   ".join(out))

Output:

Part A: exact boxes against a 100 x 100 reference
  exact                    IoU       1 = 1.0000   IoBB     1 = 1.0000   IoBB>0.25 True   IoBB>=0.25 True
  2x per axis, centred     IoU     1/4 = 0.2500   IoBB   1/4 = 0.2500   IoBB>0.25 False  IoBB>=0.25 True
  3x per axis, centred     IoU     1/9 = 0.1111   IoBB   1/9 = 0.1111   IoBB>0.25 False  IoBB>=0.25 False
  whole image              IoU 625/65536 = 0.0095   IoBB 625/65536 = 0.0095   IoBB>0.25 False  IoBB>=0.25 False
  10 x 10 inside           IoU   1/100 = 0.0100   IoBB     1 = 1.0000   IoBB>0.25 True   IoBB>=0.25 True
  shifted by half a width  IoU     1/3 = 0.3333   IoBB   1/2 = 0.5000   IoBB>0.25 True   IoBB>=0.25 True

Checks
  hand bilinear vs skimage resize(order=1): max |diff| = 2.2e-16
  np.repeat vs skimage resize(order=0): differing pixels = 0
  find_objects vs regionprops boxes: differing box sets = 0 of 360
  8- vs 4-connectivity: differing box sets = 0 of 360
  oracle (reference box as prediction): Acc 1 and AFP 0 for every T and rule: True
  data large : 60 images, heights 264-358, widths 320-456, boxes in image True, distractors 18
  data medium: 60 images, heights 94-146, widths 90-148, boxes in image True, distractors 26
  data small : 60 images, heights 24-48, widths 24-48, boxes in image True, distractors 26
  1977 scored boxes: IoBB < IoU in 0; containing the reference 1034, IoBB != IoU in 0; inside it 638, IoBB != 1 in 0
  'IoU > T or IoBB > T' differs from 'IoBB > T' in 0 of 9885 box-threshold pairs
  box formula vs pixel count (first 400 boxes): IoBB mismatches = 0

Part B: Acc / AFP at T = 0.1, 0.25, 0.5, 0.75, 0.9
[noise-free]
  heatmap bilinear 60  large  IoU  1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000 0.000/1.000
  heatmap bilinear 60  large  IoBB 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000 0.000/1.000
  heatmap bilinear 60  medium IoU  1.000/0.000 1.000/0.000 0.767/0.233 0.000/1.000 0.000/1.000
  heatmap bilinear 60  medium IoBB 1.000/0.000 1.000/0.000 0.767/0.233 0.000/1.000 0.000/1.000
  heatmap bilinear 60  small  IoU  0.600/0.400 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000
  heatmap bilinear 60  small  IoBB 0.600/0.400 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000
  heatmap bilinear 180 large  IoU  1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000 0.000/1.000
  heatmap bilinear 180 large  IoBB 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000
  heatmap bilinear 180 medium IoU  1.000/0.000 1.000/0.000 1.000/0.000 0.000/1.000 0.000/1.000
  heatmap bilinear 180 medium IoBB 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000
  heatmap bilinear 180 small  IoU  1.000/0.000 1.000/0.000 0.650/0.350 0.167/0.833 0.033/0.967
  heatmap bilinear 180 small  IoBB 1.000/0.000 1.000/0.000 0.650/0.350 0.167/0.833 0.033/0.967
[noisy]
  heatmap bilinear 60  large  IoU  1.000/0.300 1.000/0.300 1.000/0.300 0.000/1.300 0.000/1.300
  heatmap bilinear 60  large  IoBB 1.000/0.300 1.000/0.300 1.000/0.300 0.000/1.300 0.000/1.300
  heatmap bilinear 60  medium IoU  1.000/0.433 1.000/0.433 0.000/1.433 0.000/1.433 0.000/1.433
  heatmap bilinear 60  medium IoBB 1.000/0.433 1.000/0.433 0.000/1.433 0.000/1.433 0.000/1.433
  heatmap bilinear 60  small  IoU  0.350/1.083 0.000/1.433 0.000/1.433 0.000/1.433 0.000/1.433
  heatmap bilinear 60  small  IoBB 0.350/1.083 0.000/1.433 0.000/1.433 0.000/1.433 0.000/1.433
  heatmap bilinear 180 large  IoU  1.000/0.050 1.000/0.050 1.000/0.050 1.000/0.050 0.000/1.050
  heatmap bilinear 180 large  IoBB 1.000/0.050 1.000/0.050 1.000/0.050 1.000/0.050 1.000/0.050
  heatmap bilinear 180 medium IoU  1.000/0.133 1.000/0.133 0.983/0.150 0.050/1.083 0.000/1.133
  heatmap bilinear 180 medium IoBB 1.000/0.133 1.000/0.133 1.000/0.133 1.000/0.133 1.000/0.133
  heatmap bilinear 180 small  IoU  1.000/0.133 0.983/0.150 0.450/0.683 0.033/1.100 0.000/1.133
  heatmap bilinear 180 small  IoBB 1.000/0.133 0.983/0.150 0.450/0.683 0.033/1.100 0.000/1.133
  heatmap nearest 60   large  IoU  1.000/0.283 1.000/0.283 1.000/0.283 0.100/1.183 0.000/1.283
  heatmap nearest 60   large  IoBB 1.000/0.283 1.000/0.283 1.000/0.283 0.100/1.183 0.000/1.283
  heatmap nearest 60   medium IoU  1.000/0.433 1.000/0.433 0.100/1.333 0.000/1.433 0.000/1.433
  heatmap nearest 60   medium IoBB 1.000/0.433 1.000/0.433 0.100/1.333 0.000/1.433 0.000/1.433
  heatmap nearest 60   small  IoU  0.467/0.967 0.000/1.433 0.000/1.433 0.000/1.433 0.000/1.433
  heatmap nearest 60   small  IoBB 0.467/0.967 0.000/1.433 0.000/1.433 0.000/1.433 0.000/1.433
  heatmap nearest 180  large  IoU  1.000/0.050 1.000/0.050 1.000/0.050 0.967/0.083 0.200/0.850
  heatmap nearest 180  large  IoBB 1.000/0.050 1.000/0.050 1.000/0.050 1.000/0.050 1.000/0.050
  heatmap nearest 180  medium IoU  1.000/0.133 1.000/0.133 0.933/0.200 0.367/0.767 0.000/1.133
  heatmap nearest 180  medium IoBB 1.000/0.133 1.000/0.133 1.000/0.133 1.000/0.133 0.900/0.233
  heatmap nearest 180  small  IoU  1.000/0.133 0.950/0.183 0.383/0.750 0.000/1.133 0.000/1.133
  heatmap nearest 180  small  IoBB 1.000/0.133 0.950/0.183 0.500/0.633 0.067/1.067 0.017/1.117
  tiny box             large  IoU  0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000
  tiny box             large  IoBB 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000
  tiny box             medium IoU  0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000
  tiny box             medium IoBB 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000 1.000/0.000
  tiny box             small  IoU  0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000
  tiny box             small  IoBB 1.000/0.000 1.000/0.000 0.967/0.033 0.900/0.100 0.867/0.133
  whole image          large  IoU  0.800/0.200 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000
  whole image          medium IoU  0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000
  whole image          small  IoU  0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000 0.000/1.000

Part C: denominators n <= 200 that reproduce every Acc. of a Table 4 column
  Atelectasis   trunc: [180] (1)   round: [] (0)
  Cardiomegaly  trunc: [146] (1)   round: [] (0)
  Effusion      trunc: [153] (1)   round: [] (0)
  Infiltration  trunc: [123] (1)   round: [] (0)
  Mass          trunc: [85, 170] (2)   round: [] (0)
  Nodule        trunc: [79, 158] (2)   round: [] (0)
  Pneumonia     trunc: [120] (1)   round: [] (0)
  Pneumothorax  trunc: [98, 196] (2)   round: [] (0)

Part A. The exact boxes show the two identities in their simplest form.

• The centred boxes two and three times the reference’s width and height contain it, so their IoBB equals their IoU: 1/4 and 1/9.
• The whole image scores 625/65536 by both measures.
• A 10 × 10 box inside the reference scores IoBB 1 against IoU 1/100.
• The box shifted by half a width scores 1/2 against 1/3. Only this partly overlapping box and the box inside the reference separate the two measures.

The box twice as large on each axis sits exactly on Table 4’s 0.25 threshold, the value the table annotates as “two times larger”. Under the strict inequality of the paper’s text it is not a correct localization; under “≥” it would be. The paper does not say which side of the boundary it meant.

Checks. Every check came out exact:

• the hand-written bilinear upsampling matches scikit-image to $2.2 \times 10^{-16}$;
• nearest-neighbour upsampling is pixel replication;
• find_objects and regionprops give the same boxes;
• 4- and 8-connectivity never changed a box set on these maps;
• the oracle scores Acc. 1 and AFP 0 everywhere;
• among the 1,977 boxes scored, none has IoBB below IoU, all 1,034 boxes containing their reference have IoBB equal to IoU, and all 638 boxes inside it have IoBB 1;
• the “or” rule never differs from the IoBB rule at equal thresholds;
• the box formula agrees with pixel counts.

Noise-free maps. These show what the protocol does to a perfect response.

• Large boxes, threshold 60. The IoU and IoBB rows are identical, as boxes containing the reference would give. Both fall to 0 at $T = 0.9$.
• Threshold 180, large and medium. IoBB counts every reference found up to $T = 0.9$, while IoU drops to 0 at 0.9 for large and at 0.75 for medium. In our reading, these boxes lie inside or almost inside the reference.
• Small boxes. The IoBB and IoU rows are identical at both thresholds. At threshold 180 they fall from 1.000 at $T = 0.25$ to 0.033 at $T = 0.9$. At threshold 60 only 0.600 of the small references are found even at $T = 0.1$.

Noise is not needed for this: the noise-free maps already miss small findings. This echoes the paper’s Table 4, where nodules reach Acc. 0.1645 at $T(\mathrm{IoBB}) = 0.1$ against 0.9931 for cardiomegaly. Our heatmaps are an error model, though, not the paper’s network.

Noisy maps.

• AFP. The distractors add false positives. At threshold 60, AFP is 0.300 for large references up to $T = 0.5$ and 0.433 for medium ones up to $T = 0.25$, equal to the 18 and 26 distractors among 60 images. At the higher thresholds of $T$, AFP also counts the boxes around the finding itself that fail.
• Tiny box. It never passes an IoU threshold, but under IoBB it is correct at every threshold for every large and medium reference. For small references it reaches 0.867 at $T = 0.9$, where the paper’s pipeline at threshold 60 reaches 0.
• Whole image. It passes only at $T = 0.1$, for 0.800 of the large references, those that cover more than a tenth of the image.
• Nearest-neighbour upsampling. It changes individual cells: on noisy maps, 0.100 against 0.000 for large references at threshold 60 and $T = 0.75$, and 0.900 against 1.000 for medium references at threshold 180 under IoBB at $T = 0.9$. It does not change the pattern.

Part C. Every Acc. column of Table 4 is reproduced by $k/n$ truncated to four decimals for one or two denominators $n \le 200$. Rounded to four decimals, no $n \le 200$ reproduces any column. For example, 13/79 = 0.16456 is printed as 0.1645. In our reading, the table’s values were truncated rather than rounded, and they fit the denominators printed above. The arithmetic does not show how many boxes the authors scored.

Hypotheses. We wrote down H1–H8 before any code ran; H9 and H10 were written after a planning run had printed the values they predict.

Hypothesis	Outcome
H1: IoBB ≥ IoU for every box; equal for boxes containing the reference; 1 for boxes inside it	held (0, 0 and 0 exceptions)
H2: with equal thresholds, the “or” rule equals the IoBB rule	held (0 of 9,885)
H3a: the tiny box never passes an IoU threshold	held
H3b: at $T = 0.9$, the tiny box’s IoBB accuracy is at least that of the bilinear pipeline at threshold 60, for every size	held (1.000, 1.000 and 0.867 against 0.000 each)
H4: the whole image fails every threshold from 0.25 and fails 0.1 for medium and small references	held
H5: the box twice as large on each axis has IoBB = IoU = 1/4 and passes “≥ 0.25” but not “> 0.25”	held
H6: on noisy maps at $T = 0.5$ under IoBB, small references are found less often than large ones at both thresholds	held (0.000 against 1.000 at 60; 0.450 against 1.000 at 180)
H7: on noise-free maps at $T = 0.1$, every reference is found with no false positive, for every size and both thresholds	failed for small boxes at threshold 60 (Acc. 0.600, AFP 0.400); held for the other five size-and-threshold conditions
H8: under truncation, every Table 4 column has a denominator $n \le 200$	held
H9: IoBB credits boxes inside the reference (threshold 180, large, noisy, $T = 0.9$: IoBB 1.000 against IoU 0.000; tiny box at 1.000 for large and medium)	held
H10: rounding fits no column	held

Because H9 and H10 were written after the values were seen, the example illustrates those two points rather than testing them.

In our reading, H7 fails because the threshold-60 region spreads beyond small boxes. A box containing the reference has IoBB $|G|/|P|$, which drops below 0.1 once the box is more than $\sqrt{10} \approx 3.16$ times the reference’s width and height. By construction, a Gaussian blur of 0.8 cells (25.6 pixels) stays above 60/255 of its peak out to $25.6\sqrt{2\ln(255/60)} \approx 43.5$ pixels from its centre, so even a response concentrated at one point gives a region about 87 pixels across, more than $\sqrt{10}$ times a 24-pixel box (76 pixels).

Two changes were made to the planned set-up after a planning run, in which H7 had already failed for small boxes at threshold 60, and before the pilot; both were logged before the pilot ran:

• the noise-free block prints only the bilinear pipeline;
• the oracle check was added as the base check of the scorer, in place of H7, which tests the localizer rather than the implementation.

The pilot and the first run printed the same output, and nothing was changed after the first run. The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2 on one BLAS thread. It took about 32–35 seconds, and two runs printed identical output.

License

As of November 2026, the licence or terms of use of ChestX-ray14 could not be read for this article. The CVPR paper gives an NIH Clinical Center page for access, and the dataset is distributed by the NIH Clinical Center, as Computer Vision in Radiology notes. Neither the page nor the distribution site was read here. So no licence name is given here. Readers should check the terms on the NIH Clinical Center’s distribution page before use.

Limitations and Bias

• Labels from reports. Training and test labels come from the same text-mining program, so test results measure agreement with mined labels (Irvin et al., 2019). The labeller’s errors, measured against human annotation, are in Chest X-ray Analysis. On OpenI, the CVPR paper’s own Table 2 gives its lowest F1 for mass, 0.52 with a recall of 0.40, and a precision of 0.66 for pneumonia. Its text says the method had higher precision than MetaMap for all diseases, which the eight finding rows bear out; the table’s “Normal” row is lower, 0.87 against MetaMap’s 0.93.
• Few boxes, one reader. 1,600 boxes on 983 images, drawn by a board-certified radiologist, for the original eight findings only. The authors call the number relatively small but say it may still be sufficient for a reasonable estimate (Wang et al., 2017). The paper reports no agreement between readers.
• A lenient localization criterion. As the protocol section and the example show, IoBB counts any box inside the reference as perfect. A localizer can therefore score well with boxes far smaller than the finding, and Table 4’s Acc. under IoBB is not comparable with IoU-based detection results.
• One institution. All images come from one hospital’s archive (Wang et al., 2017). Chest X-ray Analysis reports subgroup differences in underdiagnosis on ChestX-ray14 that differ from those on MIMIC-CXR. Computer Vision in Radiology describes a study whose COVID-19 negatives came from ChestX-ray14 and whose models learned source differences.
• Split descriptions. The CVPR text and the appendix describe the split differently (see Splits).

Common Uses

• Training and benchmarking chest radiograph classifiers. DenseNet and Computer Vision in Radiology describe CheXNet, a DenseNet-121 trained on ChestX-ray14, and its pneumonia comparison with radiologists. Chest X-ray Analysis describes CheXNeXt’s comparison on all fourteen findings.
• Pooled studies. Studies that pool several chest datasets include ChestX-ray14 alongside CheXpert and MIMIC-CXR, as in the subgroup study that Chest X-ray Analysis describes.
• Weakly supervised localization. Classifier maps such as class activation maps and Grad-CAM are the kind of output the box set’s protocol scores. Chest X-ray Analysis discusses how reliable such maps are as localization.

Historical Milestones

• 2017, CVPR. ChestX-ray8 introduced, with eight findings, the 1,600-box evaluation set, and baselines: ResNet-50 had the best AUC for seven of the eight findings in Table 3 (AlexNet for mass), which the text summarizes as ResNet-50 achieving the best results, and localization was reported as Acc. and AFP under IoBB (Wang et al., 2017).
• After the CVPR submission, extension. The label set grew to fourteen findings and 112,120 images, described in the appendix of the final arXiv version, as Computer Vision in Radiology reports.
• 2017–2018, CheXNet and CheXNeXt. Comparisons of models trained on ChestX-ray14 with radiologists, described in Computer Vision in Radiology and Chest X-ray Analysis.
• 2019, CheXpert. Irvin et al. (2019) compared their labeller with the “NIH labeler” and built radiologist-labelled evaluation sets, citing the automatically labelled test set of ChestX-ray14 as a reason.

Related Datasets

• OpenI (Indiana Network for Patient Care), 3,955 reports and 7,470 images, was the largest public chest radiograph collection before ChestX-ray8. Its human-annotated reports were used to check the labeller (Wang et al., 2017).
• CheXpert, MIMIC-CXR, PadChest and VinDr-CXR differ from ChestX-ray14 in views, label sets, reference standards and access. Chest X-ray Analysis compares them in one table.

Resources

• The CVPR 2017 paper on the CVF open-access site (Wang et al., 2017).
• The NIH Clinical Center page that the paper gives for access, https://www.cc.nih.gov/drd/summers.html (not read for this article).
• No official evaluation code was released. The localization protocol is reconstructed above from the paper’s text and Table 4.