Lesion Detection

Lesion detection finds every lesion of a given kind in a radiology image or volume and marks each with a location and a score: the nodules in a chest CT, the masses on a mammogram, the enlarged lymph nodes or liver lesions in an abdominal scan. It is the vision task behind computer-aided detection (CADe), and a form of object detection with its own constraints. The number of lesions per image is unknown and often zero, the lesions are small against a large volume full of structures that resemble them, experts disagree about which findings count, and results are reported per lesion and per scan, not per image class. Computer Vision in Radiology gives the history of CAD and the regulation of detection software; Medical Image Classification covers the image- and patient-level labels that detection refines. This article covers the task itself: formulations, detectors, free-response evaluation and the benchmarks’ hit rules. Like its neighbours, it describes methods and evidence for researchers and practitioners and gives no clinical guidance.

Problem Definition

Given an image or volume $I$, a detector outputs a set of marks $\{(\mathbf{p}_i, s_i)\}_{i=1}^{N}$, where $\mathbf{p}_i$ locates a suspected lesion (a point, a box or a mask) and $s_i$ is a score, higher for findings more likely to be real. $N$ varies from scan to scan and can be zero. A mark is a true positive if it satisfies a hit criterion with respect to a lesion of the reference standard, and each reference lesion can be found at most once. Unlike image classification, there is no fixed set of negatives: any location without a lesion is a potential false positive, so specificity is undefined and false positives are counted per image or per scan instead (Maier-Hein et al., 2024).

What counts as a lesion is part of the task definition. LUNA16, a public benchmark for nodule detection in CT, counts only nodules of at least 3 mm accepted by at least three of four readers; marks on smaller nodules, other abnormalities and nodules with less agreement count neither for nor against a system, because, its organizers argue, a mark on another abnormality is not truly a false positive (Setio et al., 2017). A detector is always evaluated against a defined class of findings, not against “all lesions”.

Input and Output

Inputs are CT and MRI volumes, radiographs and mammograms, usually stored as DICOM; LUNA16 converted its scans to MetaImage files (LUNA16 Data page). Volumes are often anisotropic, with slices several millimetres apart; LUNA16 kept only thin-slice scans, which guidelines it cites recommend for managing nodules (see Datasets and Benchmarks).

Outputs take three forms:

• Points. LUNA16 asks for one line per mark: the scan identifier, the $x$, $y$, $z$ position in world coordinates (millimetres) and a degree of suspicion (LUNA16 Evaluation page). The reference is a centre and a diameter per nodule.
• Boxes. Detectors adapted from Faster R-CNN or one-stage designs output 2D bounding boxes or 3D cuboids. DeepLesion derives its boxes from radiologists’ measurements: a box drawn tightly around the two RECIST diameters, padded by 5 pixels (Yan et al., 2018, CVPR). VinDr-CXR’s radiologists drew rectangles directly (Nguyen et al., 2022).
• Masks. A segmentation network gives a voxel map, and connected components become detections; Tumor Segmentation describes how LiTS matched such components lesion by lesion.

Detection is often coupled with characterization: Jaeger et al. (2019) detect lung lesions and label each benign or malignant, and the same model can output a score per patient.

Variants

• Single-type and universal detection. Most systems target one lesion type, such as lung nodules, and are tuned to its shape and contrast. DeepLesion, mined from routine measurements across the body, contains lung, abdominal, mediastinal, liver, pelvic, soft-tissue, kidney and bone lesions (Yan et al., 2018, CVPR), and its journal paper’s title names the goal of detecting all of them with one model, universal lesion detection (Yan, Wang, Lu & Summers, 2018).
• 2D, 2.5D and 3D processing. Slice-wise 2D detectors can take neighbouring slices as extra input channels; multi-view (“2.5D”) networks classify several planes through a candidate; 3D networks process sub-volumes. Litjens et al. (2017) describe the multi-view approach as a way to bring 3D context into 2D networks without full 3D convolutions.
• Detection or segmentation. A lesion can be detected as an object or segmented and then split into components; the formulations differ in what they are trained on and in how scores per lesion are obtained (see Approaches).
• Lesion-, image- and patient-level outputs. A screening decision concerns an examination, not a mark. Scores per patient are often the maximum over lesions, and Jaeger et al. (2019) warn that such scores cannot tell whether a positive patient was found for the right lesion.
• Other image types. Point detection of cells and mitoses in histology follows similar matching rules at a different scale; Cell Detection covers it. The detection of masses and calcifications in mammography and of nodules on chest radiographs belongs to the applications Mammography and Breast Cancer Screening and Chest X-ray Analysis.

Challenges

• Small targets in large volumes. Almost every location is background. Litjens et al. (2017) note that most of these negatives are easy, which keeps a network from concentrating on the hard ones, that few papers they reviewed addressed this imbalance directly, and point to selective sampling of misclassified examples (van Grinsven et al., 2016, in retinal images) as one answer.
• Mimics. In LUNA16, most false positives of the combined system were large vessels, mediastinal structures, scarring and spinal abnormalities, with others caused by motion artifacts and extremely noisy scans (Setio et al., 2017). Nodules attached to vessels or the pleura are hard to separate from the structures they touch, which is why several candidate detectors treat them separately.
• Varied appearance. Solid, subsolid and very large nodules need different detectors in classical systems (see Approaches), and the nodules a combined system still missed at 1 false positive per scan were small or irregular in shape, and most of the small ones were already lost at the candidate stage (Setio et al., 2017).
• Disputed and incomplete references. Radiologists disagree on what constitutes a nodule. After merging marks closer than the sum of their radii, LUNA16 counted 2,290, 1,602, 1,186 and 777 nodules marked by at least one, two, three and four readers (Setio et al., 2017); Computer Vision in Radiology gives the counts of the LIDC-IDRI paper itself. References also miss lesions: LUNA16’s observer study accepted as nodules some marks counted as false positives, mostly small nodules, nodules in scans with several nodules, or nodules inside larger abnormalities (Setio et al., 2017).
• Data from different sources. Scanners, dose, slice thickness and reconstruction differ between institutions, and published systems were trained on different data. In the LUNA16 complete-detection track, one system was trained on National Lung Screening Trial data and used LUNA16 only for validation, another on Chinese patient data; the organizers note that the spread of results reflects the training data as well as the methods.

Approaches

The CAD pipeline: candidates, then false-positive reduction

Classical CAD systems, and many learned ones, share a three-stage design: preprocessing (standardizing intensities, restricting the search to the organ, reducing noise), candidate detection at very high sensitivity with many false positives, and false-positive reduction, which classifies the candidates and produces the final marks (Setio et al., 2017). The split matches the imbalance: a cheap first stage discards almost all of the volume, and an expensive second stage looks closely at a few hundred locations per scan.

LUNA16 describes the classical candidate detectors it combined, each built for a type of nodule:

• Shape. The detector of Murphy et al. (2009), as LUNA16 describes it, computes at every lung voxel the shape index and curvedness from the principal curvatures $k_1$ and $k_2$ of the Gaussian-smoothed image ($\sigma = 1$ voxel), $$\mathrm{SI} = \frac{2}{\pi} \arctan\frac{k_1 + k_2}{k_1 - k_2}, \qquad \mathrm{CV} = \sqrt{k_1^2 + k_2^2},$$ as the paper prints them, thresholds both to obtain seed voxels on nodule surfaces, grows and merges clusters and uses their centres as candidates.
• Density. A subsolid-nodule detector keeps voxels between −750 and −300 Hounsfield units, removes vessel and airway borders by morphological opening and discards components smaller than 34 mm³; a large-nodule detector thresholds at −300 HU and keeps components with an equivalent diameter of 8–40 mm.
• Enhancement filters. Other detectors use multiscale dot- and vessel-enhancement filters, or the divergence of the normalized gradient, with separate rules for isolated, vessel-attached and pleura-attached nodules.

No single detector found everything. Their sensitivities ranged from 31.8% to 92.9%, and merging the candidates of all five (candidates closer than 5 mm merged, those outside a lung mask with a 10 mm margin discarded) found 98.3% of the 1,186 reference nodules with about 850 candidates per scan (Setio et al., 2017, Table 1). Candidate detection sets the ceiling: a nodule missed there cannot be recovered by the second stage.

The second stage was traditionally a feature classifier. The classical systems in LUNA16 computed geometric and intensity features per candidate (radius, sphericity, mean and standard deviation of intensity, entropy) and classified them with small neural networks (Setio et al., 2017).

Convolutional networks for false-positive reduction

Convolutional networks entered at the second stage, where the task reduces to classifying patches around candidates; Computer Vision in Radiology gives their first use for nodule detection, in 1995. Litjens et al. (2017) note that most deep detection systems at the time of their survey still classified pixels or patches and post-processed the result into candidates.

The LUNA16 false-positive reduction track, which supplied a common list of candidates, shows the designs that followed (Setio et al., 2017):

• Multi-view 2D networks. The system based on Setio et al. (2016), as the LUNA16 paper describes it, extracts nine 65 × 65 patches, covering 50 × 50 mm, on different planes through each candidate, processes each with a 2D network stream with shared weights and fuses the streams before the classifier; the submitted version averaged three such networks trained from different random seeds.
• 3D networks with several contexts. The system of Dou et al. (2016), as described there, combines three 3D networks with receptive fields of 20 × 20 × 6, 30 × 30 × 10 and 40 × 40 × 26 voxels, so that small and large nodules each see enough surrounding tissue, and fuses their probabilities by a weighted sum; intensities were clipped to −1,000 to 400 HU and nodules were augmented by small translations and rotations against the class imbalance.
• Ensembles. Averaging the probabilities of the five submitted networks raised the score of the best single system; the organizers attribute the gain to differences in architecture and input patches, even though all five were convolutional networks.

End-to-end detectors and segmentation formulations

Learned candidate generation followed. In LUNA16’s complete-detection track, the leading system applied a U-Net to each axial slice, thresholded the probability map and took the centres of connected components as candidates, then classified them with a second network (Setio et al., 2017). Detectors from natural images, two-stage (Faster R-CNN, Mask R-CNN) and one-stage designs built on a feature pyramid network, were adapted to 2D slices with neighbouring slices as channels and to 3D, as in the comparison of Jaeger et al. (2019) below.

The choice between detection and segmentation is a choice about supervision and about scoring. A segmentation network trained on voxel labels needs heuristics to turn its map into scored objects; a box detector scores objects directly but is trained only on boxes, discarding voxel annotations that medical datasets often have. Jaeger et al. (2019) combined both in Retina U-Net, a one-stage RetinaNet detector whose feature pyramid gets two further high-resolution levels with skip connections, so that it resembles a U-Net, trained with a segmentation loss while the detection heads stay on the coarser levels. They evaluated it in 5-fold cross-validation on lesions in LIDC-IDRI CT (1,035 scans in their count; raters’ outlines fused by what they call pixel-wise majority voting, malignancy ratings averaged into benign and malignant) and on an in-house breast diffusion MRI set of 331 patients, by mean average precision at an IoU threshold of 0.1 in 3D. In 2D with neighbouring slices, Retina U-Net reached 50.2 mAP on LIDC against 45.4 for Mask R-CNN, 48.2 for RetinaNet and 41.1 for a U-Net whose components were turned into boxes, each scored by its highest voxel probability; in 3D, a two-stage variant scored slightly higher on LIDC (50.5 against 49.8). The authors suggest that the U-Net suffered from high-confidence false positives produced by that maximum-score aggregation; aggregating by the mean instead did not help.

Segmentation-based detection remains natural when the masks are needed anyway, for volume and growth measurements; lesion-wise scoring of segmentations is how LiTS reported detection.

Datasets and Benchmarks

LUNA16

LUNA16 (LUng Nodule Analysis 2016) was organized by researchers at Radboud University Medical Center in Nijmegen and the Polytechnic University of Turin to compare nodule detectors on the largest public reference database, LIDC-IDRI (Setio et al., 2017; LUNA16 Home page).

• Data. 888 CT scans from LIDC-IDRI’s 1,018, after discarding scans with inconsistent slice spacing or missing slices and thick-slice scans. The post-print read for this article gives the slice-thickness cut-off as larger than 3 mm in its text but as ≤ 2.5 mm in its Table 4, and the challenge’s Data page states that scans thicker than 2.5 mm were excluded. LIDC-IDRI is licensed CC BY 3.0, and the challenge data CC BY 4.0 (LUNA16 Data page).
• Reference standard. 1,186 nodules of at least 3 mm accepted by at least three of four readers; 1,104 nodules marked by fewer readers, 11,509 marks of nodules under 3 mm and 19,004 non-nodule marks are irrelevant findings, ignored in the evaluation.
• Tracks. Complete nodule detection, where the input is only the scan, and false-positive reduction, where teams classify a supplied list of candidates, added, the organizers write, to let teams with image classification experience but no chest CT background take part. The paper’s Table 1 gives the merged set of 754,975 candidates a sensitivity of 0.983, about 1,166 of the 1,186 nodules, as the Data page states for this list (candidates_V2); the paper’s text and the Description page give 1,162 of 1,186, which is 98.0%.
• Training and testing. The data are public, annotations included, so LUNA16 is an open challenge: teams train and test on the same scans in 10-fold cross-validation over ten fixed subsets, merging the per-fold results.
• Hit criterion and score. A mark is a true positive if it lies within the radius of a reference nodule from its centre; if several marks hit one nodule, the highest-scoring is selected; marks on irrelevant findings are discarded; at most 100 marks per scan are evaluated. FROC curves are summarized by the competition performance metric (see Evaluation Metrics), with 95% confidence intervals from 1,000 bootstrap resamples of scans.

As of 31 October 2016, seven systems had been evaluated in the complete track and five in the false-positive reduction track. The best complete system scored a CPM of 0.811, and the commercial system evaluated (Visia CT Lung CAD 5.3, designed to alert the radiologist to regions after the initial reading) 0.715. In the false-positive reduction track, the 3D networks of Dou et al. scored 0.908, the best single result, and the best combination of networks 0.952, with a sensitivity over 95% at fewer than one false positive per scan (Setio et al., 2017, Table 2 and Fig. 1). The organizers name the limits of the design themselves: with public annotations, teams could tune to the data set; cross-validation is easy to get wrong, and one team withdrew excellent results because its network weights were not reinitialized between folds; and training and test data from the same sources may bias results upward. They suggested a hidden test set, or teams submitting their algorithms as containers, for a future challenge. The Home page reports that new submissions stopped being processed in January 2018 (as of November 2026).

LUNA16 also illustrates why results on “LIDC-IDRI” are not comparable across papers. Table 4 of the paper lists published CAD systems evaluated on LIDC subsets of 58 to 949 scans, with different nodule size limits, agreement levels from one to four readers, and sensitivities reported at different false-positive rates. Its predecessor, ANODE09, had compared detectors on 50 screening scans from a single centre and scanner protocol, with few large nodules (Setio et al., 2017).

The observer study of LUNA16 re-read the marks of the combined system that counted as false positives at 0.25 per scan, and the paper says the accepted nodules were shared as additional annotations; the Data page lists that file as “available soon” (as of November 2026). In the post-print read for this article, the counts of accepted marks are placeholders, and the caption of its Table 3 (222 false positives) differs from its text (127 marks after filtering); the journal version was not read.

DeepLesion

DeepLesion was mined from the picture archiving system of the authors’ institution, which the paper calls a major medical institute (the authors are at the NIH Clinical Center): radiologists’ RECIST measurements of significant findings, collected as “bookmarks” over close to two decades, were converted into boxes. After removing erroneous annotations, it contains 32,120 axial CT slices, mostly 512 × 512, from 10,594 studies of 4,427 patients, with one to three lesions per slice, 32,735 lesions in all (Yan et al., 2018, CVPR). The CVPR paper uses the data for lesion retrieval, clustering and matching: 30% of the lesions were labelled into eight coarse types and verified by a radiologist, and those were split 25/25/50% into seed, validation and test sets with no patient overlap. Because the boxes come from lesions a radiologist chose to measure, unmeasured lesions in the same slice are unlabelled; a detector’s marks on them count as false positives under any matching rule (our reading of the construction). The journal paper on DeepLesion and universal lesion detection (Yan, Wang, Lu & Summers, 2018) could not be read for this article (its publisher’s site served a bot challenge), so its split and detection protocol are not described here.

VinDr-CXR

VinDr-CXR supplies radiologists’ boxes for 22 local findings on chest radiographs, among them nodule/mass and “other lesion”, so it can serve as a detection benchmark for radiographs (Nguyen et al., 2022); Chest X-ray Analysis describes the dataset, its readers and its access terms. The paper does not define a matching rule or detection metric; a slightly modified version of the data was used for a Kaggle detection competition, whose scoring the paper does not describe.

Evaluation Metrics

The free-response ROC curve

Free-response methods come from observer-performance research in radiology. Conventional ROC analysis asks a reader for one rating per image and ignores where the reader looked, so a reader who rates an image highly because of a lesion mimic is credited as if the lesion had been found; location ROC (LROC) asks for one mark per image, and the free-response paradigm allows any number of marks, each with a rating (Thompson et al., 2013). For a detector, a threshold $t$ on the score selects the marks with $s_i \geq t$, giving

$$S(t) = \frac{\#\{\text{reference lesions hit by a mark with } s_i \geq t\}}{N_{\text{lesions}}}, \qquad F(t) = \frac{\#\{\text{false-positive marks with } s_i \geq t\}}{N_{\text{scans}}},$$

and the FROC curve plots the lesion sensitivity $S$ against the mean number of false positives per scan (or per image) $F$ as $t$ falls. LUNA16 evaluates every threshold that gives a distinct point and connects the lowest point to $(0, 0)$ (LUNA16 Evaluation page). Unlike an ROC curve, the FROC curve has no fixed right end: it follows from the definition that it stops at the lowest-scoring mark, below a sensitivity of 1 if some lesions were never marked.

Summaries: CPM, FPPI and AFROC

A single number is needed to rank systems. The competition performance metric (CPM) averages the sensitivity at seven false-positive rates,

$$\mathrm{CPM} = \frac{1}{7} \sum_{f \in \{1/8,\, 1/4,\, 1/2,\, 1,\, 2,\, 4,\, 8\}} S\big(F^{-1}(f)\big),$$

where $S(F^{-1}(f))$ is the sensitivity of the curve at $f$ false positives per scan. According to the LUNA16 paper, the metric was introduced in ANODE09 and is called the CPM in Niemeijer et al. (2011). Its range is 0 to 1. The organizers justify the low rates: CAD systems in clinical use, they write, operate at about 1 to 4 false positives per scan, and the rates down to 1/8 test whether a system finds many nodules with very few false alarms, as a more autonomous system would need (LUNA16 Evaluation page). Because the rates double from one to the next, every factor of two in false-positive rate weighs equally, and the rates below one per scan carry three of the seven terms (our observation); the LUNA16 organizers observe that differences in sensitivity at low rates substantially influenced the scores. Other summaries are a sensitivity at one fixed rate, or the false positives per image (FPPI) needed to reach a fixed sensitivity, the counting metric Metrics Reloaded pairs with an FROC score for its object detection examples, cell detection in microscopy video and multiple sclerosis lesion detection in brain MRI (Maier-Hein et al., 2024).

Reader studies use different summaries of free-response data. Alternative FROC (AFROC) analysis keeps only the highest-rated false mark on each image, plotting the lesion localization fraction against the resulting false-positive fraction, and the jackknife AFROC (JAFROC) figure of merit is the probability that lesion marks are rated above the highest false marks on normal images (JAFROC-1 when there are no normal images) (Thompson et al., 2013). These are built for comparing readers and modalities with multi-reader, multi-case statistics, and treat normal images differently from images with lesions, a distinction the FROC curve does not make.

Average precision is the summary from natural-image detection. Metrics Reloaded recommends either AP or an FROC score as the threshold-free metric for object detection: the FROC score for easy interpretation, AP as a standardized metric, complementing a counting metric at a chosen operating point when the detections carry continuous scores (Maier-Hein et al., 2024). Precision depends on how many lesions the test set contains; false positives per scan does not, so FROC results from a set with few lesions per scan are easier to relate to a reading workload (our reasoning).

Matching criteria

The hit criterion decides what the numbers mean. Metrics Reloaded splits it into a localization criterion, which says whether a prediction can match a reference object (centre distance, point inside a mask or box, box or mask IoU), and an assignment strategy for ambiguous matches; for those two examples it recommends box IoU, with greedy matching by score and double assignments counted as false positives (Maier-Hein et al., 2024). The benchmarks above differ on both:

• Distance or overlap. LUNA16 accepts a mark within one nodule radius of its centre, a criterion that scales with lesion size. Jaeger et al. (2019) match 3D boxes at an IoU of 0.1, a threshold they chose for the coarse localization clinicians need, far below the 0.5 common in object detection. LUNA16’s observer study shows a limit of radius criteria: marks on non-nodular annotated lesions were not well captured by a nodule radius (Setio et al., 2017).
• Acceptance radius. In reader studies, the acceptance radius around a lesion classifies marks as lesion or non-lesion localizations, and more lenient radii inflate the figure of merit (Thompson et al., 2013). The same holds for detectors, and Cell Detection shows how a match radius moved published point-detection scores.
• Duplicates. LUNA16 selects the highest-scoring mark on a nodule; Metrics Reloaded counts additional matches as false positives. The choice matters for systems without non-maximum suppression or with loose merging.
• Excluded findings. Ignoring marks on irrelevant findings, as LUNA16 does, keeps systems from being penalized for finding real abnormalities outside the reference class. Without it, a system that finds sub-threshold nodules looks worse.

Uncertainty and aggregation

LUNA16 computes confidence intervals by resampling scans, not marks (LUNA16 Evaluation page), which respects the correlation between marks in one scan (our reasoning); Statistical Significance in Evaluation covers tests for comparing systems. Patient-level scores taken as the maximum over marks turn detection into classification, scored by ROC AUC; Jaeger et al. (2019) report such a patient-level AP next to their lesion-level mAP and caution that it can reward a correct patient score obtained from the wrong lesion.

Example: An FROC Curve and Its CPM From a Two-Stage Detector

The script builds synthetic CT-like volumes with known lesions, runs a two-stage detector and scores it as LUNA16 does. Each volume has 32 × 64 × 64 voxels, treated as 1 mm isotropic, with a smooth low-contrast background, noise and six bright tubes as vessel-like mimics. Reference nodules, up to four per volume, are soft spheres 4–10 mm in diameter with random contrast; 2 mm blobs play the role of irrelevant findings, and marks within 1.5 mm of one are ignored. Stage 1 generates candidates as the local maxima of a scale-normalized Laplacian of Gaussian over three scales (the response of the Laplacian operator to bright blobs), keeping at most 100 per scan. Stage 2 reduces false positives with a logistic regression trained on 30 separate volumes, on four features per candidate: the LoG response, its best scale, the local intensity, and the ratio of the smallest to the largest magnitude of the Hessian eigenvalues, near 1 for a sphere and near 0 for a tube. The ratio is our construction, in the spirit of the dot- and vessel-enhancement filters of the classical detectors. Sensitivities between curve points are read by linear interpolation, our reading of the Evaluation page’s “connected” points; the LUNA16 evaluation script was not read. NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2 were used, and the run takes a few seconds.

import numpy as np
from scipy import ndimage as ndi
from sklearn.linear_model import LogisticRegression

SHAPE = (32, 64, 64)                       # voxels, treated as 1 mm isotropic
SIGMAS = (1.5, 2.5, 3.5)                   # LoG scales for candidate generation
RATES = np.array([1/8, 1/4, 1/2, 1, 2, 4, 8])   # FPs per scan used by the CPM

def blob(grid, c, r):                      # soft sphere of radius r at centre c
    d = np.sqrt(sum((g - ci) ** 2 for g, ci in zip(grid, c)))
    return 1 / (1 + np.exp((d - r) / 0.7))

def make_volume(rng):
    grid = np.indices(SHAPE).astype(float)
    vol = 2.0 * ndi.gaussian_filter(rng.normal(size=SHAPE), 4) + rng.normal(0, 0.08, SHAPE)
    for _ in range(6):                     # vessel-like tubes: the main source of false positives
        p, q = rng.uniform(0, SHAPE), rng.uniform(0, SHAPE)
        t = np.clip(sum((g - pi) * (qi - pi) for g, pi, qi in zip(grid, p, q)) / np.sum((q - p) ** 2), 0, 1)
        d = np.sqrt(sum((g - pi - t * (qi - pi)) ** 2 for g, pi, qi in zip(grid, p, q)))
        vol += rng.uniform(0.5, 0.9) / (1 + np.exp((d - rng.uniform(1, 2)) / 0.5))
    nodules, small = [], []
    for _ in range(min(rng.poisson(1.5), 4)):          # reference nodules, 4-10 mm diameter
        c, r = rng.uniform((6, 8, 8), np.array(SHAPE) - (6, 8, 8)), rng.uniform(2, 5)
        vol += rng.uniform(0.3, 0.9) * blob(grid, c, r)
        nodules.append((c, r))
    for _ in range(rng.poisson(1.0)):                  # "irrelevant findings": 2 mm blobs
        c = rng.uniform(4, np.array(SHAPE) - 4)
        vol += 0.5 * blob(grid, c, 1.0)
        small.append((c, 1.0))
    return vol, nodules, small

def candidates(vol):
    """Stage 1: scale-normalised LoG maxima. Stage 2 features from the Hessian at the best scale."""
    resp = np.stack([-s ** 2 * ndi.gaussian_laplace(vol, s) for s in SIGMAS])
    best, r = resp.argmax(0), resp.max(0)
    peaks = (r == ndi.maximum_filter(r, size=5)) & (r > 0.05)
    pts = np.argwhere(peaks)
    pts = pts[np.argsort(-r[peaks])][:100]             # at most 100 marks per scan
    pairs = [(i, j) for i in range(3) for j in range(i, 3)]
    hess = {}                                          # second derivatives at each scale
    for k, s in enumerate(SIGMAS):
        for i, j in pairs:
            order = [0, 0, 0]; order[i] += 1; order[j] += 1
            hess[k, i, j] = s ** 2 * ndi.gaussian_filter(vol, s, order=order)
    feats = []
    for p in pts:
        k = best[tuple(p)]
        H = np.empty((3, 3))
        for i, j in pairs:
            H[i, j] = H[j, i] = hess[k, i, j][tuple(p)]
        lam = np.sort(np.linalg.eigvalsh(H))           # l1 <= l2 <= l3; bright blob: all negative
        ratio = max(-lam[2], 0) / max(-lam[0], 1e-6)   # near 1 for a sphere, near 0 for a tube
        feats.append([r[tuple(p)], ratio, SIGMAS[k], vol[tuple(p)]])
    return pts.astype(float), np.array(feats).reshape(-1, 4)

def match(pts, scores, nodules, small, dup_fp=False, ignore_small=True):
    """Label each mark TP (1), FP (0) or ignored (-1), LUNA16-style: hit within the nodule radius,
    highest-scoring mark per nodule is the TP."""
    lab = np.zeros(len(pts), int)
    for c, rad in nodules:
        hit = np.flatnonzero(np.linalg.norm(pts - c, axis=1) <= rad)
        if len(hit):
            top = hit[np.argmax(scores[hit])]
            lab[hit] = 0 if dup_fp else -1
            lab[top] = 1
    for c, rad in small if ignore_small else []:
        near = np.linalg.norm(pts - c, axis=1) <= max(rad, 1.5)
        lab[near & (lab == 0)] = -1
    return lab

def froc(scans, n_nodules):
    """scans: list of (scores, labels). Returns CPM and sensitivity at the seven rates."""
    s = np.concatenate([sc for sc, lb in scans]); l = np.concatenate([lb for sc, lb in scans])
    keep = l >= 0; s, l = s[keep], l[keep]
    order = np.argsort(-s, kind="stable"); s, l = s[order], l[order]
    last = np.r_[s[1:] != s[:-1], True]                # one curve point per unique threshold
    sens = np.r_[0, np.cumsum(l == 1)[last] / n_nodules]
    fps = np.r_[0, np.cumsum(l == 0)[last] / len(scans)]
    at = np.interp(RATES, fps, sens)                   # points connected; flat beyond the last
    return at.mean(), at, fps[np.argmax(sens)]         # CPM, the 7 sensitivities, FPs/scan at max

def run(seed, n_train=30, n_test=40, verbose=False):
    rng = np.random.default_rng(seed)
    data = [make_volume(rng) for _ in range(n_train + n_test)]
    cand = [candidates(v) for v, _, _ in data]
    X, y = [], []
    for (pts, f), (_, nod, sm) in zip(cand[:n_train], data[:n_train]):
        lab = match(pts, f[:, 0], nod, sm)
        X.append(f[lab >= 0]); y.append(lab[lab >= 0])
    clf = LogisticRegression(max_iter=1000).fit(np.vstack(X), np.concatenate(y))
    test = list(zip(cand[n_train:], data[n_train:]))
    n_nod = sum(len(nod) for _, (_, nod, _) in test)
    out = {}
    for name, stage2, kw in [("stage 1 (LoG score)", False, {}),
                             ("stage 2 (FP reduction)", True, {}),
                             ("stage 2, duplicates = FP", True, {"dup_fp": True}),
                             ("stage 2, <3 mm blobs = FP", True, {"ignore_small": False})]:
        scans = []
        for (pts, f), (_, nod, sm) in test:
            sc = clf.predict_proba(f)[:, 1] if stage2 else f[:, 0]
            scans.append((sc, match(pts, sc, nod, sm, **kw)))
        out[name] = froc(scans, n_nod)
    if verbose:
        found = sum((match(p, f[:, 0], nod, sm) == 1).sum() for (p, f), (_, nod, sm) in test)
        print(f"test scans {n_test}, nodules {n_nod}, candidates/scan "
              f"{np.mean([len(p) for (p, f), _ in test]):.1f}, candidate sensitivity {found / n_nod:.3f}")
        print("protocol                     " + " ".join(f"{r:>5g}" for r in RATES) + "   CPM  FPs/scan at max")
        for name, (cpm, at, fmax) in out.items():
            print(f"{name:28s} " + " ".join(f"{a:5.3f}" for a in at) + f" {cpm:5.3f} {fmax:8.2f}")
        dup = small = 0                                # marks the variants re-label
        for (pts, f), (_, nod, sm) in test:
            sc = clf.predict_proba(f)[:, 1]
            a = match(pts, sc, nod, sm)
            dup += ((a == -1) & (match(pts, sc, nod, sm, dup_fp=True) == 0)).sum()
            small += ((a == -1) & (match(pts, sc, nod, sm, ignore_small=False) == 0)).sum()
        print(f"marks re-labelled as FP: duplicates {dup}, on <3 mm blobs {small}")
    return out, test, clf, n_nod

out, test, clf, n_nod = run(0, verbose=True)
# bootstrap over scans (LUNA16: 1,000 resamples of scans) for the stage-2 CPM
scans = [(clf.predict_proba(f)[:, 1], p, nod, sm) for (p, f), (_, nod, sm) in test]
rng = np.random.default_rng(0); cpms = []
for _ in range(1000):
    idx = rng.integers(0, len(scans), len(scans))
    sub = [(scans[i][0], match(scans[i][1], scans[i][0], scans[i][2], scans[i][3])) for i in idx]
    cpms.append(froc(sub, sum(len(scans[i][2]) for i in idx))[0])
lo, hi = np.percentile(cpms, [2.5, 97.5])
print(f"stage-2 CPM 95% bootstrap interval: {lo:.3f}-{hi:.3f}")
print("seed  CPM stage 1  CPM stage 2")
for seed in range(5):
    o = run(seed)[0]
    print(f"{seed:4d}  {o['stage 1 (LoG score)'][0]:11.3f}  {o['stage 2 (FP reduction)'][0]:11.3f}")

test scans 40, nodules 53, candidates/scan 78.7, candidate sensitivity 0.906
protocol                     0.125  0.25   0.5     1     2     4     8   CPM  FPs/scan at max
stage 1 (LoG score)          0.019 0.019 0.132 0.208 0.283 0.340 0.434 0.205    29.95
stage 2 (FP reduction)       0.396 0.547 0.717 0.792 0.830 0.906 0.906 0.728     2.95
stage 2, duplicates = FP     0.396 0.547 0.717 0.792 0.830 0.906 0.906 0.728     2.95
stage 2, <3 mm blobs = FP    0.396 0.547 0.717 0.792 0.830 0.906 0.906 0.728     3.33
marks re-labelled as FP: duplicates 0, on <3 mm blobs 40
stage-2 CPM 95% bootstrap interval: 0.619-0.833
seed  CPM stage 1  CPM stage 2
   0        0.205        0.728
   1        0.310        0.796
   2        0.241        0.799
   3        0.304        0.824
   4        0.277        0.842

The candidate stage sets the ceiling. About nine in ten nodules produce a candidate, and no later stage can do better: both curves level off at that sensitivity. Ranked by the LoG response alone, most nodules fall behind other candidates, mostly on the tubes, which also give strong Laplacian responses (our reading; the script does not print where false positives lie), and the curve needs about 30 false positives per scan to reach its ceiling, far beyond the CPM’s range. The second stage keeps the same candidates and only re-orders them. The shape ratio separates spheres from tubes, so the nodules move to the front, the ceiling is reached at about 3 false positives per scan, and the CPM rises from 0.205 to 0.728. In relative terms the gain is largest at the lowest rates, where the LoG ranking finds almost nothing, the part of the curve that LUNA16’s choice of rates weights most; in absolute terms it is largest between 1/2 and 1 false positive per scan. Over data seeds 0–4 the CPM ranges from 0.205 to 0.310 for stage 1 and from 0.728 to 0.842 for stage 2; seed 0, shown in detail, is the lowest stage-2 value of the five.

The two protocol variants leave every sensitivity unchanged in this run, for reasons the last lines show. No nodule received two marks (the $5^3$ maximum filter keeps at most one maximum per neighbourhood, which we take to be the reason), so counting duplicates as false positives changes nothing. Forty marks fell on the 2 mm blobs, and counting them as false positives moves the point where the curve reaches its ceiling from 2.95 to 3.33 false positives per scan; that shift lies between the rates of 2 and 4 at which the CPM samples the curve, and the lower points are unaffected, so all seven sensitivities stay the same. With a detector that scored such blobs higher, or that produced several maxima per nodule, the same rules would change the CPM (our reasoning). The bootstrap interval over the 40 test scans, 0.619–0.833, is wider than the spread between seeds: with 53 nodules, a difference in CPM of 0.1 between two systems would not be clearly resolved (our reading), which is why LUNA16 reports intervals with its curves.

The script was changed after its first run, as recorded in the drafting plan. The first run stopped with an error in the bootstrap block, where match received its arguments in the wrong order, after the seed-0 table had been printed; the second run stopped at the bootstrap’s print statement, a format-string error. Both were fixed without changing any computation. After the first complete run showed identical rows for the two variants, a print-only line was added (the numbers of duplicate and blob marks, the highest stage-2 score of a blob mark and the lowest of a true positive), then replaced, because it did not by itself explain the identical rows, by two print-only additions: the last column, the false-positive rate at which each curve reaches its maximum sensitivity, and the line counting the marks each variant re-labels. No data, parameter or measurement was changed, and every value that both the earlier runs and the final script print is the same in the output above. The script was run twice and gave identical output.

Applications

Lesion detection underlies two of the device roles that Computer Vision in Radiology describes with their regulatory definitions: marks that direct a reader’s attention, and detection combined with characterization of each finding. LUNA16’s organizers motivate nodule detection by the reading volume of CT lung cancer screening, and the commercial system they evaluated is designed as a second look after the radiologist’s own reading (Setio et al., 2017). Detection also starts quantitative work: a detected lesion is segmented and measured, and tracked across examinations, which Tumor Segmentation and Medical Image Registration cover. DeepLesion’s authors aimed at organizing the lesions in a hospital archive, retrieving similar lesions and matching a patient’s lesions between studies (Yan et al., 2018, CVPR). In mammography and chest radiography the same methods detect masses, calcifications and nodules, within the applications Mammography and Breast Cancer Screening and Chest X-ray Analysis.

Open Problems

• The reference standard. Which findings count, how many readers must agree and how lesions missed by all readers are handled change both training labels and scores, as LUNA16’s observer study and DeepLesion’s measured-lesion labels show.
• Evaluation that predicts use. Open benchmarks with public annotations and cross-validation on data from the same sources may favour the systems tested (Setio et al., 2017). Standalone FROC results do not measure what readers do with the marks; reader studies and prospective evaluation, discussed in Computer Vision in Radiology and Clinical Validation and Regulation of Medical Imaging AI, measure that.
• Comparable protocols. Hit criteria, duplicate rules, excluded findings, normalization per scan or per image, the number of marks evaluated and interpolation between curve points can all change scores, and results on the same source data use different subsets and references (Setio et al., 2017, Table 4); results are comparable only under the same protocol and code.
• Generalization and universal detection. Detectors tuned to one lesion type, scanner population or slice thickness may not transfer; universal detection across lesion types depends on archive-mined labels whose completeness is unknown.