Tumor Segmentation

Tumor segmentation delineates a tumor, and often its internal parts, voxel by voxel in a CT or MRI volume, so that its extent, volume and composition can be measured. It is a case of medical image segmentation, which covers the general methods, losses for small targets, voxel spacing and the overlap and surface metrics. Tumors add their own problems: they have no fixed shape, number or position; their parts are defined by how they appear on particular scans and are nested inside each other; some are a few millimetres across and some cases contain none of a given part; and experts drawing the same tumor disagree more than they do for organs. Much of what is known about these problems comes from public challenges, above all BraTS for brain tumors, LiTS for liver tumors and KiTS for kidney tumors, which this article takes as cases. It describes methods and evaluation for researchers and gives no clinical guidance.

Problem Definition

Given a volume, often with several co-registered channels, predict a label map in which each voxel is background, healthy tissue of an organ of interest, or one of several tumor classes. Two features distinguish the task from organ segmentation.

Labels and evaluated regions differ. Annotators assign each voxel one exclusive label, but the regions that matter clinically are unions of labels that contain one another. In BraTS 2021 the labels are necrotic tumor core (NCR, label 1), peritumoral edematous or infiltrated tissue (ED, label 2) and gadolinium-enhancing tumor (ET, label 4), and the evaluated regions are the whole tumor (WT, all three labels), the tumor core (TC, NCR and ET) and the enhancing tumor (ET alone) (Baid et al., 2021). The organizers describe the TC as the bulk of the tumor that is typically considered for surgical excision, and the WT as the complete extent of the disease. KiTS21 used the same idea for the kidney, with kidney and masses (all regions), masses (tumors and cysts) and tumor alone as its “hierarchical evaluation classes” (Heller et al., 2023).

Definitions are image-based. The BraTS sub-regions are defined by appearance on MRI and, in the organizers’ words, do not reflect strict biologic entities: ET is tissue that is hyperintense on post-contrast T1 compared with pre-contrast T1 and with healthy white matter, NCR is hypointense on post-contrast T1, and ED is the hyperintense envelope on T2-FLAIR. The organizers note that other criteria could be set and would give slightly different sub-regions (Baid et al., 2021). A change of definition changes the ground truth: BraTS 2012–2016 also labelled non-enhancing tumor (NET, label 3), which BraTS 2017 merged into NCR because some annotators overestimated it on little image evidence, so that rankings risked rewarding agreement with a particular test-set annotator rather than algorithmic performance (Bakas et al., 2018).

When each lesion must be counted, as with many liver metastases, the task overlaps with lesion detection and instance segmentation. KiTS19 chose semantic segmentation on purpose, because the border between two abutting tumors, or between the two halves of a horseshoe kidney, is not well defined (Heller et al., 2021).

Input and Output

Brain. BraTS supplies four MRI sequences per patient: native T1-weighted, post-contrast T1-weighted (T1Gd), T2-weighted and T2-FLAIR (see MRI for the contrasts). All scans are converted from DICOM to NIfTI, co-registered to the SRI24 anatomical template, resampled to 1 mm isotropic voxels and skull-stripped, so every case arrives on the same grid (Baid et al., 2021). The original acquisitions vary: the 2018 report lists slice thicknesses of 1–5 mm for T1 and 2–4 mm for T2, with axial, sagittal or coronal acquisitions depending on the sequence and site (Bakas et al., 2018).

Liver. LiTS supplies contrast-enhanced abdominal CT volumes converted to NIfTI with no other processing, with in-plane pixel sizes of 0.56–1.0 mm, slice thicknesses of 0.45–6.0 mm and 42 to 1026 slices (Bilic et al., 2023). Intensities are in Hounsfield units, and the contrast between tumor and liver in those units turned out to matter for every method (see Challenges).

Kidney. KiTS19 supplies CT in the late arterial phase and KiTS21 in the corticomedullary phase. KiTS19 first released the volumes in their original spacing and, at participants’ request, a second version resampled to the dataset’s median spacing of 3 × 0.781 × 0.781 mm (Heller et al., 2021).

The output is a label map on the input grid. A network can predict the exclusive labels (softmax over NCR, ED, ET and background) or the overlapping regions directly (one sigmoid output per region), which changes how errors are penalized (see Approaches).

Variants

• Brain tumors. Gliomas in pre-operative multiparametric MRI are the BraTS task; BraTS 2014–2016 also included longitudinal scans to assess disease progression (Bakas et al., 2018).
• Liver tumors. LiTS segments the liver and every tumor in it, primary (such as hepatocellular carcinoma and cholangiocarcinoma) or secondary (metastases of colorectal, breast and lung cancers), as a single tumor label, in pre- and post-therapy CT (Bilic et al., 2023).
• Kidney tumors and cysts. KiTS19 segments kidney and tumor; KiTS21 adds renal cysts as a third class (Heller et al., 2021, 2023).
• Other organs. The Medical Segmentation Decathlon, described in Medical Image Segmentation, includes lung, pancreatic and colon tumor tasks; its brain and liver tasks reuse BraTS and LiTS data.

Challenges

Nested and ambiguous boundaries

The sub-regions are nested, and the inner ones are the hardest. In the BRATS 2012–2013 data the human raters agreed best on the whole tumor and worst on the enhancing tumor: comparing pairs of raters, the median Dice was 87% for the whole tumor, 94% for the core of high-grade gliomas but 82% for that of low-grade gliomas, and 77% for the active (enhancing) tumor, which was scored on high-grade cases only (Menze et al., 2015, Fig. 5); the corresponding means, 74–85%, are the range the report’s abstract gives. In BraTS 2018 the results of the participating teams followed the same order: most teams reached a median Dice of about 0.9 on the whole tumor, while on the enhancing tumor (called active tumor in that report) the median Dice of the top 54 of 63 teams ranged from 0.74 to 0.85 and their mean Dice only from 0.61 to 0.77, a gap the organizers attribute to outlier cases; they add that Dice is more sensitive to errors on the enhancing tumor because of its smaller volume (Bakas et al., 2018). Low-grade gliomas are hard for another reason: they often show little contrast enhancement or edema, and the boundary between tumor and healthy tissue is uncertain even for experts (Bakas et al., 2018).

Small, many and absent targets

In LiTS a scan contains between 0 and 12 tumors, and methods did well on large tumors but poorly on small ones. Grouping lesions by diameter (under 10 mm, 10–20 mm, over 20 mm), the organizers found that top teams at ISBI 2017 reached an F1 score of only around 0.10 for detecting the small lesions; the worst results came from scans with a single small tumor (Bilic et al., 2023). In KiTS19, a multivariate regression over the 100 teams’ tumor Dice per case found tumor size to be the only significant predictor among those tested (slice thickness, multifocality, field of view, size and subtype) (Heller et al., 2021).

Absent targets are just as common. Some low-grade gliomas have no enhancing tumor at all (Bakas et al., 2018), and 7 of the 201 LiTS scans contain no lesion (Bilic et al., 2023). For such a case Dice is $0/0$ if nothing is predicted and 0 if anything is, so the challenge’s convention (see Evaluation Metrics) can outweigh everything else. The reverse problem also exists: every KiTS19 case contained at least one renal tumor, and its organizers warn that methods with high scores on this task should not be trusted to perform as well on a cohort that includes healthy controls (Heller et al., 2021).

Mimics and contrast

Structures that look like tumor cause false positives. The BraTS 2021 organizers list the errors automated methods typically make: vessels within the edema labelled as enhancing tumor or as edema, the choroid plexus and T1-bright blood products labelled as edema, and periventricular white matter hyperintensities segmented as tumor (Baid et al., 2021). In KiTS19 a single test case was missed (Dice below 0.05) by each of the top five teams and most others, because a cyst in the affected kidney was consistently mistaken for the tumor (Heller et al., 2021). In LiTS the methods did best when the tumor was 40–60 HU denser than the surrounding liver and worst when the difference was below 20 HU (Bilic et al., 2023).

Disagreement between annotators

Tumor boundaries are uncertain, and each challenge measured that uncertainty differently, so the numbers are not directly comparable:

• BRATS 2012–2013. The 30 training cases were each annotated by four raters and the 2012 test cases by three; the bar an algorithm should reach was set by comparing pairs of raters (Menze et al., 2015). The authors note that this inter-rater score somewhat overestimates variability, while comparing a rater with the consensus underestimates it, because the rater contributed to that consensus.
• LiTS. A new radiologist re-annotated 15 CT scans from scratch and reached a median Dice of 70.2% against the existing consensus annotation, while a board-certified radiologist who corrected the existing annotations agreed with them at 95.2%; the authors compare this with the best leaderboard result of 82.5% as of 4 April 2022 (Bilic et al., 2023).
• KiTS19. The whole annotation process was repeated on 30 training cases after the challenge, giving a Dice of 0.983 for the kidney-plus-tumor region (called “kidney” in the challenge paper’s abstract) and 0.923 for the tumor (Heller et al., 2019, 2021). This measures how well the process agrees with itself, not two independent experts.
• BraTS 2021. Each case was refined by a single annotator and approved by senior neuroradiologists, so inter-rater agreement could not be assessed (Baid et al., 2021).

A score above one of these figures is therefore not above “human performance” in general, only above that challenge’s measurement of it. The image annotation article covers agreement measures and consensus labels.

Approaches

Before deep learning

The BRATS report sorts earlier brain tumor methods into two families (Menze et al., 2015). Generative methods combine explicit models of anatomy and appearance, for example treating the tumor as an outlier with respect to healthy tissue or deriving its prior from tumor growth models; they generalize well but depend on registration, which large lesions disturb. Discriminative methods learn the mapping from local intensity features to labels with classifiers such as support vector machines or decision trees; they need training data from the same imaging protocol and careful intensity calibration. The LiTS report groups liver tumor methods similarly, into thresholding with spatial regularization, local features with learning algorithms, and deep learning (Bilic et al., 2023). In BraTS 2017 all winning entries were at least partly based on encoder–decoder networks (Isensee et al., 2018).

Cascades and region-based training

Two ways of exploiting the nesting appear in the challenge entries. A cascade segments the outer region first and the inner regions within it: Wang et al.’s BraTS 2017 entry segmented the whole tumor, then the tumor core within it, then the enhancing tumor, and the KiTS21 winner first segmented the kidney region coarsely, cropped around each kidney and then segmented kidney, tumor and masses at a finer scale with four networks trained in the nnU-Net framework (Isensee et al., 2018; Heller et al., 2023). Region-based training keeps one network but changes its output: instead of a softmax over the exclusive labels, a sigmoid per evaluated region (WT, TC, ET), each optimized directly. Isensee et al. (2018) argued that optimizing the evaluated regions, not the cascade itself, explained Wang et al.’s result, and in their BraTS 2018 cross-validation on the 285 training cases adding region-based training raised the mean Dice of the enhancing tumor from 73.43 to 73.81 and of the tumor core from 82.17 to 82.87 (Table 1 of arXiv v2). nnU-Net version 2 supports region-based training for such hierarchical labels; the nnU-Net article describes it.

Our reading of why regions help: under a softmax over labels, a voxel of necrosis predicted as enhancing tumor is an error for both labels, although it lies inside the tumor core either way; a region output scores that voxel as correct for the core and wrong only for the enhancing tumor. The example below shows this, and KiTS21’s organizers give the same reason for their hierarchical classes: they avoid penalizing a model twice for labelling a tumor as a cyst, or part of a mass as healthy kidney (Heller et al., 2023).

Configured U-Nets, ensembles and fusion

As in other segmentation tasks, carefully configured U-Nets did well against modified architectures. KiTS19 was won by a residual 3D U-Net trained with the nnU-Net pipeline on the official training data only, with CT intensities clipped to [−79, 304] HU and normalized, while the organizers note that the plain nnU-Net performed best despite several teams’ attempts to enhance it, and that the winners’ own experiments showed no meaningful benefit from popular architectural additions (Heller et al., 2021). nnU-Net with BraTS-specific changes (region-based training, more aggressive augmentation, a larger batch, batch normalization, in two of the three variants a batch Dice loss, and the postprocessing below), as an ensemble of three variants chosen on the validation set, took first place in BraTS 2020 with test-set Dice scores of 88.95, 85.06 and 82.03 for the whole tumor, tumor core and enhancing tumor; its authors add that their experiments do not show which of the changes contributed most (Isensee et al., 2020).

Fusion of several segmentations is an older and recurring idea. BRATS fused its raters’ annotations by a hierarchical majority vote that assumes increasing severity from edema to non-enhancing core to necrotic core to enhancing core and assigns each voxel the most severe label that at least half of the raters chose; the authors report that STAPLE, which ignores the order of labels, gave implausible results such as edema surrounded by core (Menze et al., 2015). Applied to the outputs of the best-ranked half of the BRATS 2012–2013 algorithms, the same vote ranked above every individual algorithm, in all three regions and for both Dice and Hausdorff distance (Menze et al., 2015; Bakas et al., 2018). BraTS 2021 used the same idea for its annotations: a STAPLE fusion of nnU-Net, DeepScan and DeepMedic outputs was refined manually by neuroradiologists (Baid et al., 2021).

Postprocessing for absent targets

Because of how BraTS scores cases without enhancing tumor, both of Isensee et al.’s entries removed small enhancing-tumor predictions: if the total predicted enhancing tumor was below a threshold chosen on the training cases, all of it was relabelled as necrosis, which keeps those voxels in the tumor core (Isensee et al., 2018, 2020). In the 2018 cross-validation this step raised the mean enhancing-tumor Dice from 75.94 to 78.68, without changing the other two regions (Isensee et al., 2018, Table 1). The 2020 paper describes the trade-off: the rule removes some true positives in small tumors, but can gain more perfect scores on cases with no enhancing tumor than it loses; one of the validation cases it shows scored an enhancing-tumor Dice of 0 because the postprocessing had removed a small lesion the network had found (Isensee et al., 2020). Size filters on connected components serve the same purpose elsewhere: the KiTS21 winner removed predicted kidneys, tumors and cysts smaller than 20,000, 200 and 50 voxels as too small to be realistic (Heller et al., 2023).

Datasets and Benchmarks

BraTS

The Multimodal Brain Tumor Image Segmentation Benchmark has been held every year since 2012, as the table of editions in Baid et al. (2021) lists; Medical Image Segmentation summarizes its 2015 report.

• Data. The dataset grew from 50 cases in 2012 to 660 in 2020 and 2,040 in 2021 (1,251 for training, 219 for validation and 570 for testing, Table 1 of Baid et al., 2021, and its abstract); the body text of the same paper gives 2,000 cases. Since 2017 a validation set without labels lets teams check their methods on unseen data, and since 2017 only scans without prior instrumentation are included (Bakas et al., 2018; Baid et al., 2021). The data are retrospective and multi-institutional.
• Annotation protocol. Annotators start from the outside: the whole tumor on T2-FLAIR, drawn on every third slice and filled by morphological operations, then the tumor core on every other slice, then the enhancing tumor by an intensity threshold within the core on T1Gd; since 2017, FLAIR hyperintensities in the opposite hemisphere or around the ventricles are excluded from edema unless contiguous with it (Bakas et al., 2018).
• Metrics. Dice and the 95th-percentile Hausdorff distance (HD95) for each of the three regions, with sensitivity and specificity reported as well (Bakas et al., 2018; Baid et al., 2021). BRATS adopted HD95 because a few small false-positive regions, which postprocessing could remove, can dominate the maximum distance (Menze et al., 2015).
• Ranking. From 2017 each team is ranked against the others on every test case, region and metric (in 2018, 191 cases × 3 regions × 2 metrics, 1,146 rankings); the ranks are averaged per case and then over cases. Permutation tests (100,000 permutations) of the case-level ranks decide whether neighbouring teams differ, and in 2017 and 2018 they led to a shared third place (Bakas et al., 2018).
• Rules (2021). Ranked methods may use only the BraTS data, with no models pretrained on other data, and the final test is run on submitted containers so that the test data stay hidden (Baid et al., 2021).

The winners were Kamnitsas et al.’s ensemble of three architectures (a 3D FCN, a 3D U-Net and DeepMedic) in 2017 and NVIDIA’s entry in 2018, with the DKFZ U-Net second (Bakas et al., 2018; Isensee et al., 2018). The papers read for this article do not state a data licence for BraTS; access goes through the challenge platforms.

LiTS

The Liver Tumor Segmentation Benchmark was held at ISBI 2017 and MICCAI 2017, and its data formed the liver task of the 2018 Medical Segmentation Decathlon (Bilic et al., 2023).

• Data. 201 contrast-enhanced abdominal CT volumes from seven institutions, 131 for training and 70 for testing, split 2:1 with a similar distribution of centres, so generalization to unseen centres was not tested; 194 scans contain lesions. The data are licensed CC BY-NC-SA.
• Annotation. A radiologist with more than three years of experience in oncologic imaging labelled tumor and healthy liver slice by slice; three further readers, blinded to the first segmentation, verified it, with the most senior breaking ties. The paper adds that scans with very small and uncertain lesion-like structures “were omitted in the annotation”, without saying whether the structures or the scans were left out.
• Ranking. At ISBI 2017 and MICCAI 2017 teams were ranked by Dice computed per case and averaged over cases (the 2018 Decathlon by Dice and ASD); a post-challenge ranking summed the ranks for Dice, average symmetric surface distance (ASD) and relative volume difference (RVD).
• Results. The best tumor Dice was 0.674 at ISBI 2017, 0.702 at MICCAI 2017 and 0.739 at MICCAI 2018 (nnU-Net), against 0.963 for the liver. At ISBI 2017 and MICCAI 2017 the top three tumor methods did not differ significantly on any of the three metrics. The best lesion-wise recall was 0.458, 0.515 and 0.554 in the three events, and the MICCAI 2017 winner by Dice had a lesion-wise precision of 0.148, which the organizers read as many false positives. Under the post-challenge ranking the first place changed in all three events (Table 7).
• After the challenge. The evaluation server stayed open; by April 2022 it had scored more than 3,414 valid submissions. Test images were released to participants, so the organizers cannot exclude overfitting through repeated submissions or manual correction.

Two numbers in the paper do not agree: the text gives tumor sizes between 38 and 1,231 mm³, while Table 3 gives a median tumor volume of $16.11 \times 10^3$ mm³ in the training set.

KiTS19 and KiTS21

KiTS19, held with MICCAI 2019, released 210 training and 90 test CT scans of patients who underwent partial or radical nephrectomy at the University of Minnesota between 2010 and mid-2018 (Heller et al., 2021). Patients with tumor thrombus, and two whose suspected tumor turned out to be a cyst, were excluded; cysts were labelled as kidney. More than half of the imaging came from over 50 referring institutions. The data are licensed CC BY-NC-SA 4.0.

• Score. For each test case, the mean of the kidney Dice and the tumor Dice, averaged over the 90 cases. Each team could enter one submission and see an approximate score, on 45 sampled cases, at most twice.
• Results. 106 teams submitted and 100 met all requirements. The winner scored 0.974 on the kidney and 0.851 on the tumor (composite 0.912), followed by composites of 0.906, 0.903, 0.902 and 0.899. Across teams the mean tumor Dice was 0.580 against 0.915 for the kidney, and tumor precision exceeded recall. Pairwise significance was assessed by bootstrap with Holm–Bonferroni correction.
• Design limits named by the organizers. A single health system, a random rather than temporal or external test split (internal validation in PROBAST’s terms), test images released to participants, and label errors found after the data freeze.

KiTS21 answered several of these (Heller et al., 2023). Its 300 training cases are the KiTS19 cases, re-annotated with cysts as their own class, and its 100 test cases come from another institution, the Cleveland Clinic. Each region was localized by a medical trainee, who also placed guidance pins along the intended boundary, and then delineated three times by laypeople, so each case has $3^N$ possible composite segmentations for $N$ regions. At test time a random sample of these composites was scored with Dice and the surface Dice (a tolerance-based boundary score; see Medical Image Segmentation) on each hierarchical class, and teams were ranked by a rank-then-aggregate procedure, with tumor Dice as the tie-breaker. Of 25 teams on the final leaderboard, the top five reached tumor Dice scores of 0.86, 0.83, 0.83, 0.82 and 0.81 and missed no tumor completely, whereas in KiTS19 every team had missed at least one. The winner was not statistically better than any other top-five team. The KiTS21 paper compares its results with an inter-rater agreement of 0.88 for the tumor, citing KiTS19, whose own papers report 0.923 for the repeated annotation; it also attributes the winner’s 0.86 to the kidney region in one place and to the tumor in another.

Evaluation Metrics

The Dice coefficient and boundary distances are defined in their own articles and in Medical Image Segmentation; what is specific to tumors is how challenges handle nested regions (BraTS’s WT, TC and ET, KiTS21’s hierarchical classes), absent targets and ranking.

Absent targets. BraTS scores a case whose reference has no enhancing tumor as Dice 1 and HD95 0 if the prediction is also empty, and as Dice 0 and HD95 373.13 if any voxel is predicted (Isensee et al., 2020; the 2018 paper states the Dice values). Isensee et al. (2020) observe that with case-wise ranking this makes the enhancing tumor of such cases binary: every team shares either the first or the last rank. The fixed value of 373.13 also dominates any mean HD95 on the validation set, where for their final submission even the 90th percentile of the enhancing tumor’s HD95 was 11.32. LiTS computed its lesion detection metrics over the whole test set rather than per case, to avoid problems with scans that contain no tumor (Bilic et al., 2023). The Dice article lists the conventions in use for empty masks.

Aggregation and ranking. LiTS averages Dice per case, which, its organizers note, penalizes errors more in cases with fewer lesions (Bilic et al., 2023), whereas BraTS ranks first and averages ranks. Isensee et al. (2020) re-implemented the BraTS ranking for model selection, because the order of their variants by mean Dice differed from their order under the BraTS ranking: the best of nine variants by mean Dice in cross-validation was third under the ranking (Table 3). The statistical significance article covers the tests used to decide whether ranks differ.

Lesion-wise detection. LiTS defined lesions as 3D connected components and matched them by their overlap, measured by IoU, after merging components that a prediction splits or merges; any metric computed on merged reference lesions is attributed to each constituent lesion. From the matches it reported precision, recall and F1 by lesion size (Bilic et al., 2023, Appendix C). These scores ranked teams differently from Dice: not all top segmentation methods detected tumors well. Precision and recall explains the underlying counts.

Example: Nested Regions, Empty Targets and Their Scores

Two NumPy and SciPy scripts (NumPy 2.3.5, SciPy 1.16.3; under a second each) use synthetic tumors with known labels. The first builds a glioma-like tumor in a $64^3$ volume with 1 mm voxels from three ellipsoids, labelled with the BraTS 2021 values (NCR inside an enhancing rim, inside edema), and scores two predictions by region and by label: one with every boundary 1 mm too far inside, and one with correct whole tumor and core but a necrosis that is too large, so that part of the enhancing rim is labelled necrosis.

import numpy as np
from scipy import ndimage as ndi

N = 64                                          # 64^3 voxels, 1 mm isotropic
z, y, x = np.mgrid[0:N, 0:N, 0:N] - 32.0        # coordinates relative to the tumour centre

def ellipsoid(rz, ry, rx):
    return (z / rz) ** 2 + (y / ry) ** 2 + (x / rx) ** 2 <= 1.0

def label_map(wt, tc, ncr):
    """BraTS 2021 labels: 1 necrotic core (NCR), 2 edema (ED), 4 enhancing tumour (ET)."""
    lab = np.zeros((N, N, N), np.uint8)
    lab[wt] = 2
    lab[tc] = 4
    lab[ncr & tc] = 1
    return lab

REGIONS = {"WT": (1, 2, 4), "TC": (1, 4), "ET": (4,)}   # evaluated regions (nested)
LABELS = {"ED": (2,), "NCR": (1,), "ET": (4,)}          # the labels themselves

def dice(p, g):
    return 2 * (p & g).sum() / (p.sum() + g.sum())

def scores(pred, ref, groups):
    return {k: dice(np.isin(pred, v), np.isin(ref, v)) for k, v in groups.items()}

ref = label_map(ellipsoid(20, 16, 14), ellipsoid(10, 9, 8), ellipsoid(6, 5, 4.5))
preds = {
    "A: every boundary 1 mm inside": label_map(ellipsoid(19, 15, 13), ellipsoid(9, 8, 7),
                                               ellipsoid(5, 4, 3.5)),
    "B: necrosis too large": label_map(ellipsoid(20, 16, 14), ellipsoid(10, 9, 8),
                                       ellipsoid(8, 7, 6.5)),
}

print("region  voxels  boundary voxels / voxels")
for k, v in REGIONS.items():
    m = np.isin(ref, v)
    print(f"{k:6s}  {m.sum():6d}  {(m & ~ndi.binary_erosion(m)).sum() / m.sum():.2f}")

print("\nprediction                       region Dice: WT     TC     ET  "
      "| label Dice: ED    NCR    ET")
for name, p in preds.items():
    r, l = scores(p, ref, REGIONS), scores(p, ref, LABELS)
    print(f"{name:31s}  {r['WT']:16.3f}  {r['TC']:.3f}  {r['ET']:.3f}  "
          f"|  {l['ED']:12.3f}  {l['NCR']:.3f}  {l['ET']:.3f}")

Output:

region  voxels  boundary voxels / voxels
WT       18697  0.15
TC        2979  0.27
ET        2420  0.45

prediction                       region Dice: WT     TC     ET  | label Dice: ED    NCR    ET
A: every boundary 1 mm inside               0.906  0.826  0.728  |         0.860  0.688  0.728
B: necrosis too large                       1.000  1.000  0.746  |         1.000  0.533  0.746

The same 1 mm error costs each region in proportion to its share of boundary voxels. The enhancing tumor is almost as large as the core here, yet it scores lowest, because as a rim it has an inner and an outer boundary and nearly half its voxels lie on one of them; volume alone does not decide how sensitive Dice is, shape does. This is the pattern of the BraTS results, where enhancing tumor scores trail tumor core and whole tumor. Prediction B gets the extent of the tumor and its core exactly right: scored by region, only the enhancing tumor is penalized, while scored by label the same misplaced voxels count against both necrosis and enhancing tumor, and necrosis, the smaller label, drops to 0.533. That double charge is what region-based evaluation, and region-based training, avoid.

The second script simulates 40 cases per seed: 30 spherical tumors with a 2-voxel enhancing rim around necrosis, of random size, and 10 tumors with no enhancing tumor. A simulated predictor, an error model rather than a trained network, shifts the true enhancing tumor by one voxel and, with probability 0.5, adds a 7-voxel false-positive blob in the edema, standing in for a vessel mistaken for enhancement. Each case’s enhancing-tumor Dice is computed under three conventions for an empty reference: the BraTS rule, skipping the case as undefined, and adding 1 to numerator and denominator. The postprocessing of Isensee et al. then removes the predicted enhancing tumor when it has fewer than $T$ voxels.

import sys
import numpy as np

N = 48
z, y, x = np.mgrid[0:N, 0:N, 0:N] - 24.0
r = np.sqrt(z ** 2 + y ** 2 + x ** 2)

def make_case(rng, has_et):
    """Spherical tumour: ET rim 2 voxels thick around necrosis, edema shell outside.
    A case without ET has a core that is all necrotic / non-enhancing."""
    rc = rng.uniform(3, 12)
    et = (r <= rc) & (r > rc - 2) if has_et else np.zeros_like(r, bool)
    ed = (r <= rc + 6) & (r > rc)
    return et, ed

def predict(rng, et, ed):
    """Simulated error model: a 1-voxel shift, and with probability 0.5 a 7-voxel
    false-positive blob somewhere in the edema (a vessel mistaken for enhancement)."""
    p = np.roll(et, 1, axis=2)
    if rng.random() < 0.5:
        c = np.argwhere(ed)[rng.integers(ed.sum())]
        p = p | blob(c)
    return p

def blob(c):
    m = np.zeros((N, N, N), bool)
    m[tuple(c)] = True
    for ax in range(3):
        for d in (-1, 1):
            cc = c.copy(); cc[ax] += d
            m[tuple(cc)] = True
    return m

def dice_brats(p, g):      # empty/empty = 1, any false positive on an empty reference = 0
    if not g.any():
        return 1.0 if not p.any() else 0.0
    return 2 * (p & g).sum() / (p.sum() + g.sum())

def dice_skip(p, g):       # undefined (NaN) when the reference is empty
    return np.nan if not g.any() else 2 * (p & g).sum() / (p.sum() + g.sum())

def dice_smooth(p, g):     # +1 in numerator and denominator
    return (2 * (p & g).sum() + 1) / (p.sum() + g.sum() + 1)

def cohort(seed, n=40, n_empty=10):
    rng = np.random.default_rng(seed)
    cases = []
    for i in range(n):
        et, ed = make_case(rng, has_et=i >= n_empty)
        cases.append((et, predict(rng, et, ed)))
    return cases

def evaluate(cases, T):
    """Postprocessing: drop the predicted ET if it has fewer than T voxels."""
    res = {"brats": [], "skip": [], "smooth": []}
    empty_ok = removed = 0
    for g, p in cases:
        if p.sum() < T:
            removed += g.any() and p.any()
            p = np.zeros_like(p)
        empty_ok += (not g.any()) and (not p.any())
        res["brats"].append(dice_brats(p, g))
        res["skip"].append(dice_skip(p, g))
        res["smooth"].append(dice_smooth(p, g))
    return (np.mean(res["brats"]), np.nanmean(res["skip"]), np.mean(res["smooth"]),
            empty_ok, removed)

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
cases = cohort(seed)
sizes = np.array([g.sum() for g, _ in cases[10:]])
d = np.array([dice_brats(p, g) for g, p in cases[10:]])
print(f"seed {seed}: 30 cases with ET ({sizes.min()}-{sizes.max()} voxels), 10 without")
print(f"Dice of the smallest / largest ET: {d[sizes.argmin()]:.3f} / {d[sizes.argmax()]:.3f}")
print("T (voxels)  mean ET Dice: BraTS  skip empty  smoothed | empty scored 1  true ET removed")
for T in (0, 10, 50, 100):
    b, s, m, e, k = evaluate(cases, T)
    print(f"{T:10d}  {b:18.3f}  {s:10.3f}  {m:8.3f} | {e:14d}  {k:15d}")

lo = {T: [] for T in (0, 10)}
for s in range(10):
    c = cohort(s)
    for T in lo:
        lo[T].append(evaluate(c, T)[0])
for T, v in lo.items():
    print(f"seeds 0-9, T={T}: BraTS-convention mean ET Dice {min(v):.3f}-{max(v):.3f}")

Output:

seed 0: 30 cases with ET (224-2984 voxels), 10 without
Dice of the smallest / largest ET: 0.759 / 0.751
T (voxels)  mean ET Dice: BraTS  skip empty  smoothed | empty scored 1  true ET removed
         0               0.733       0.744     0.742 |              7                0
        10               0.808       0.744     0.808 |             10                0
        50               0.808       0.744     0.808 |             10                0
       100               0.808       0.744     0.808 |             10                0
seeds 0-9, T=0: BraTS-convention mean ET Dice 0.652-0.733
seeds 0-9, T=10: BraTS-convention mean ET Dice 0.802-0.808

The smallest and the largest enhancing rims score almost the same Dice under the one-voxel shift, although one has thirteen times the volume of the other: a rim of constant thickness keeps the same share of boundary voxels at every size. On the empty cases, a 7-voxel blob that would barely move Dice on a case with enhancing tumor decides the score under the BraTS rule. Three of the ten empty cases receive 0, and the mean is 0.733; removing predictions under 10 voxels turns all ten into perfect scores and raises it to 0.808. Over seeds 0 to 9 the range narrows from 0.652–0.733 to 0.802–0.808, which in our reading shows that most of the variation between seeds came from how many empty cases happened to receive a blob. Skipping empty cases makes the score blind to these false positives (0.744 at every threshold), and the smoothed Dice scores a blob on an empty case as $1/8$ rather than 0, which keeps it close to the skipped mean at $T = 0$. No true enhancing rim in this cohort is smaller than 224 voxels, so thresholds up to 100 cost nothing here; the trade-off Isensee et al. describe, losing true enhancing tumor in small tumors, would appear only with targets as small as the threshold, which this cohort does not contain.

Applications

The challenge papers name the uses that motivate tumor segmentation, without claiming clinical validity for their methods. The BraTS 2021 organizers list surgical treatment planning, image-guided interventions, monitoring tumor growth and radiotherapy maps, and note that manual delineation is time-consuming and subjective (Baid et al., 2021); BraTS also used segmentations as the input to survival prediction (Bakas et al., 2018). The LiTS organizers point to tumor staging under RECIST, which measures the diameter of target lesions, and to localization as a prerequisite for ablation, radiotherapy, resection and embolization (Bilic et al., 2023). KiTS19 was motivated by nephrometry scores, manual scores of tumor complexity that segmentation could automate, and its organizers state the intended use as characterizing tumors in patients already diagnosed with one (Heller et al., 2021). Each of these uses needs clinical validation beyond a challenge score.

Open Problems

• Generalization. LiTS tested on data from the same centres as training and KiTS19 on a random split from one health system; BraTS 2021 planned an out-of-distribution test cohort, and KiTS21’s external test set comes from a single other institution (Bilic et al., 2023; Heller et al., 2021, 2023; Baid et al., 2021).
• Small and absent targets. Lesion-level detection of small tumors remains poor, and scores on cases without a target depend on conventions.
• Reference quality. Single-annotator references (BraTS 2021; LiTS, whose limitations section names one rater per centre although its protocol describes verification by three further readers) cannot show where a method falls within expert variation, and the challenges’ measurements of agreement are not comparable with each other.
• Image-based definitions. Sub-regions follow appearance on particular sequences and phases, and redefinitions such as the merging of BraTS’s non-enhancing tumor label change the task.
• Beyond the training distribution. The BraTS organizers name blood products, resection cavities in post-operative scans, non-glioblastoma entities and low-grade gliomas without enhancing tumor as effects that methods still need to handle (Bakas et al., 2018).
• Ranking stability. Top teams are often not significantly different (BraTS, LiTS, KiTS21), and rankings change with the metric and the aggregation.