Medical Image Segmentation

Medical image segmentation assigns a label to every voxel (or pixel) of a medical image, so that an organ, a tumor, a lesion, a vessel tree or a tissue class is delineated as a region with a measurable size and shape. It is also called delineation or contouring, and in radiology the result is often called a mask or a label map. It is the medical form of semantic segmentation, but three things set it apart: most images are 3D volumes whose voxels have physical sizes, often different along each axis; the target can occupy a tiny fraction of the volume; and the reference itself is drawn by experts who disagree with each other.

Problem Definition

Given an image $I$ defined on a voxel grid, predict a label map $L$ on the same grid, with $L(\mathbf{v}) \in \{0, 1, \dots, K\}$, where 0 is background and $1, \dots, K$ are the target structures. Two kinds of target dominate:

• Anatomy: organs and substructures such as the liver, the hippocampus, the left atrium of the heart or the zones of the prostate. Their shape and position are fairly consistent across patients, and they are usually large.
• Lesions: tumors, metastases, multiple sclerosis lesions and other abnormalities. They vary in number, size, shape and position, may be absent, and can be a few voxels across.

Labels can be exclusive (each voxel has one class) or nested, as in brain tumor segmentation, where BraTS scored the whole tumor, the tumor core and the enhancing (“active”) tumor as overlapping regions (Menze et al., 2015). When individual lesions must be counted, the problem becomes instance segmentation or detection; Metrics Reloaded treats the choice of problem category as the first step of validation, because a semantic score can hide missed lesions (Maier-Hein et al., 2024).

Input and Output

The input is usually a 3D volume from computed tomography or magnetic resonance imaging, sometimes with several co-registered channels: the Decathlon’s brain task gives four MRI sequences per patient (FLAIR, T1-weighted, post-contrast T1-weighted and T2-weighted) and its prostate task two (Antonelli et al., 2022). 2D inputs include X-ray radiographs, ultrasound frames, fundus photographs and microscopy images. Alongside the voxel array, the file header stores the voxel spacing in millimetres and the orientation of the axes. CT and MRI volumes are often anisotropic: voxels are finer within a slice than between slices, for example $0.8 \times 0.8$ mm pixels within slices that are 3 mm apart. The Decathlon converted every image to the NIfTI format and transposed all of them, without resampling, to a consistent axis orientation, to spare participants the format and orientation problems of medical data (Antonelli et al., 2022).

The output is a label map on the input grid, or per-class probabilities from which the label map is taken by argmax or a threshold. Volumes and distances computed from it are only meaningful in physical units, through the spacing: a voxel count is not a volume until it is multiplied by the voxel volume.

Variants

2D, 2.5D and 3D. A 2D network segments each slice independently and sees no context across slices. A 3D network convolves over all three axes and sees the whole neighbourhood of a voxel, at the cost of memory, which limits the patch it can process. A 2.5D network sits between them: it takes a few adjacent slices as input channels, or combines 2D networks applied along the three orthogonal planes. In the Decathlon, the second-placed team in the mystery phase (NVDLMED) ensembled three models, each trained on a different view (axial, coronal and sagittal), initialized from 2D pretrained models (Antonelli et al., 2022). nnU-Net trains a 2D U-Net, a 3D U-Net at full resolution and a 3D cascade, and picks among them by cross-validation; its authors expected the 2D configuration to work well on anisotropic data such as prostate MRI and found the 3D full-resolution U-Net the best overall (Isensee et al., arXiv v2, 2020).

Fully automatic, interactive and promptable. Interactive methods refine a result from clicks, scribbles or boxes. MedSAM adapts the Segment Anything Model to medical images: it is a promptable 2D model, steered by a bounding box, that handles 3D images as a series of 2D slices; its authors trained it on 1,570,263 image–mask pairs covering 10 imaging modalities and compared it, by median Dice per task over 86 internal and 60 external validation tasks, with SAM and with specialist U-Net and DeepLabV3+ models trained separately for each modality. The abstract reports better accuracy and robustness than the specialist models; the results section describes performance on par with or better than them (Ma et al., 2024).

Sparse and weak annotation. Fully labelled volumes are expensive. 3D U-Net was designed to learn dense volumetric segmentation from volumes in which only some slices are annotated (Çiçek et al., 2016).

Microscopy. Cell and nucleus segmentation in microscopy and histology is usually an instance problem, with touching objects to separate, and is treated as a task of its own: cell segmentation.

Challenges

• Class imbalance and small targets. A lesion can fill well under 1% of a volume. Salehi et al. (2017) note that in lesion detection the non-lesion voxels typically outnumber lesion voxels by more than 500 times, and that training on such data biases predictions towards high precision and low recall.
• Anisotropic spacing. Slices several times thicker than the in-plane pixel size make the volume look different along each axis, complicate resampling, and change what a distance in voxels means.
• Memory. Whole volumes rarely fit through a 3D network at full resolution, so training and inference work on patches; in nnU-Net the patch size of the 3D full-resolution U-Net is limited by GPU memory, and for large images it may be too small to see enough context (Isensee et al., arXiv v2, 2020).
• Low contrast and ambiguous boundaries. Some targets differ little in intensity from their surroundings. In the Decathlon’s pancreas task the median over participants of the mean Dice was 0.69 for the pancreas and 0.21 for the tumor mass, a gap the organizers consider likely explained by the very small intensity difference between the two (Antonelli et al., 2022).
• Annotation cost and inter-rater variation. References are drawn by experts, often slice by slice. In the BraTS benchmark, Dice scores between pairs of expert raters ranged from 74% to 85% across tumor sub-regions (Menze et al., 2015); the Decathlon’s data were annotated by a single rater per case, so no inter-rater estimates exist for its tasks (Antonelli et al., 2022). See image annotation for agreement measures and consensus labels.
• Few training cases and domain shift. The Decathlon’s left-atrium task had 20 training volumes (Antonelli et al., 2022). Scanners, protocols and sites change intensities and resolution, so a model trained at one site may degrade at another (dataset bias).

Approaches

From classical methods to U-Net

Before deep learning, medical segmentation combined intensity thresholding, region growing, deformable contours and shape models, and atlas-based segmentation, which registers labelled reference images to a new scan and transfers their labels (medical image registration); the general methods are surveyed in image segmentation. These methods encode prior knowledge by hand and are sensitive to the contrast and anatomy they were tuned for.

U-Net (Ronneberger, Fischer & Brox, 2015) made learned segmentation practical with few annotated images, through an encoder–decoder with skip connections and heavy elastic augmentation; it was designed for microscopy. 3D U-Net (Çiçek et al., 2016) replaced every 2D operation with its 3D counterpart, and V-Net (Milletari, Navab & Ahmadi, 2016) trained a volumetric network end to end on prostate MRI with a loss based on the Dice coefficient, introduced to cope with strong foreground–background imbalance. The U-Net article covers these and later variants (Attention U-Net, UNet++).

nnU-Net: configuration over architecture

Many architectural extensions of U-Net were proposed after 2016. Analysing the 2019 Kidney and Kidney Tumor Segmentation (KiTS) leaderboard, the nnU-Net authors found that its top 15 methods were all derived from the 3D U-Net and that none of the common modifications (residual or dense connections, attention, dilated convolutions) was necessary for good performance; details of the pipeline’s configuration mattered more (Isensee et al., arXiv v2, 2020). nnU-Net, described in that preprint and published as Isensee et al. (2021), keeps a plain U-Net and automates the choices around it: from a “dataset fingerprint” of the training data (image sizes, voxel spacings, class ratios and, for CT, foreground intensities) it sets the preprocessing, target spacing, patch and batch size and network topology by heuristic rules, and chooses between its configurations, ensembles and post-processing empirically by cross-validation (arXiv v2). It won the Medical Segmentation Decathlon, and in the two years after the challenge, sometimes with minor modifications, it won 33 of 53 further segmentation tasks it entered, as reported by the Decathlon organizers (Antonelli et al., 2022). The nnU-Net article describes the rules and the versions.

Losses for class imbalance

Plain cross-entropy averages a per-voxel loss, so when 99% of voxels are background, predicting background everywhere is already a low-loss solution. Remedies act on the loss or on the sampling:

• Weighted cross-entropy raises the weight of the rare class, as U-Net’s weight map does for cell borders.
• Overlap losses. The soft Dice loss of Milletari et al. (2016) and the generalized Dice loss of Sudre et al. (2017), which weights each class by the inverse square of its volume, normalize the loss by the size of the structure; their formulas and gradients are in the Dice coefficient article. The Tversky loss (Salehi, Erdogmus & Gholipour, 2017) replaces Dice by the Tversky index, $$S(P, G; \alpha, \beta) = \frac{|P \cap G|}{|P \cap G| + \alpha\,|P \setminus G| + \beta\,|G \setminus P|},$$ which weights false positives by $\alpha$ and false negatives by $\beta$; $\alpha = \beta = 0.5$ gives Dice. On multiple sclerosis lesion segmentation in MRI the authors (arXiv v1 of the paper) obtained their best $F_2$ score and Dice with $\beta = 0.7$ ($\alpha = 0.3$), a setting that favours recall.
• Boundary losses. Kervadec et al. (2021) argue that regional losses such as Dice and cross-entropy sum over regions whose sizes differ by orders of magnitude, and propose a boundary loss, a distance between contours written as a regional integral of the softmax output against a precomputed distance map of the reference. It is combined with a regional loss rather than used alone; they evaluated it on ischemic stroke lesions (ISLES) and white matter hyperintensities (WMH), whose foreground may be 500 times smaller than the background (arXiv v4).
• Compound losses. nnU-Net trains with the sum of cross-entropy and Dice loss (Isensee et al., arXiv v2, 2020). Among the Decathlon finalists who described their methods, the Dice loss was the most common choice (29%), followed by cross-entropy (21%) (Antonelli et al., 2022).
• Sampling. Patch-based training can control how often the target appears. nnU-Net draws one third of its training patches so that they are guaranteed to contain foreground, and the rest at random locations (Isensee et al., arXiv v2, 2020).

The example below compares cross-entropy and Dice loss on a small synthetic target.

Spacing, resampling and anisotropy

Networks assume that a voxel step means roughly the same physical distance everywhere in the training data, so images are usually resampled to a common target spacing. The choice is a trade-off: a coarse spacing loses detail, and a fine one makes the volume larger than a patch can cover. nnU-Net’s rules, as described in arXiv v2 of its paper, show the issues involved (Isensee et al., 2020); the nnU-Net article gives them in full and compares them with the current code:

• The default target spacing is the median of the training spacings, per axis. For anisotropic datasets this default can cause severe interpolation artifacts or a loss of information, so when the data are anisotropic by more than a factor of three (in the paper’s terms, both voxel and spacing anisotropy), the low-resolution axis takes the 10th percentile of its training spacings instead.
• For anisotropic images (largest spacing more than three times the smallest), image data are resampled in-plane with third-order splines and out-of-plane with nearest-neighbour interpolation, to avoid resampling artifacts; label maps are converted to one-hot channels, interpolated linearly (with nearest-neighbour interpolation along the low-resolution axis of anisotropic cases) and recombined by argmax.
• Downsampling is applied first along the high-resolution axes only, until their spacing is within a factor of two of the coarse axis; when the initial spacing ratio exceeds two, the kernel size along the out-of-plane axis is 1 until then.

The example network follows the same idea on a small scale: its first convolution in each block is $1 \times 3 \times 3$ and its pooling halves only the in-plane axes.

Datasets and Benchmarks

Medical segmentation is evaluated mostly through challenges, each with fixed training data, a hidden test set and its own scoring rules. Maier-Hein et al. (2018) counted 150 biomedical image analysis challenges up to the end of 2016, with 549 tasks, 70% of them segmentation.

Medical Segmentation Decathlon. Held at MICCAI 2018 to find a general-purpose algorithm, the Decathlon combined ten 3D tasks, 2,633 retrospectively acquired images in all, from several institutions: brain tumors (multiparametric MRI), left atrium and hippocampus (MRI), liver and liver tumors, lung tumors, pancreas and pancreatic tumors (CT), and the prostate (multiparametric MRI) in a development phase; colon cancer, hepatic vessels and tumors, and the spleen (CT) in a mystery phase (Antonelli et al., 2022). Its protocol:

• Splits. For 8 of the 10 tasks, two thirds of the cases were released with labels for training and one third without labels for testing; the brain and liver tasks kept the splits of the earlier challenges they came from.
• Phases. In the development phase teams could submit once a day to a leaderboard. In the mystery phase they had to solve three unseen tasks with one submission, without changing the architecture or hyperparameters. 180 teams registered, 31 submitted valid development results and 19 completed the mystery phase.
• Metrics. Dice and the normalized surface distance (NSD), both computed on 3D volumes, with NSD tolerances set per task by the clinicians who annotated it: 1 mm for the hippocampus, 2 mm for the lung, 3 mm for the hepatic vessels and spleen, 4 mm for the heart, prostate and colon, 5 mm for the brain and pancreas, and 7 mm for the liver. The organizers stress that these metrics are neither task-specific nor sufficient for clinical validation.
• Ranking. Undefined metric values were set to 0. For each target region and metric, the teams were compared pairwise with a Wilcoxon signed-rank test, and each team’s score was the number of teams it beat significantly; the ranks from these scores were averaged over the target regions of a task and then over the tasks of each phase.
• Licence. The data are released under CC-BY-SA 4.0 (Antonelli et al., 2022; Decathlon website, as of October 2026).

nnU-Net ranked first in both phases, with a median Dice of 0.79 over the development tasks and 0.71 over the mystery tasks. The median over participants of the mean Dice per task ranged from 0.16 for colon cancer to 0.94 for the liver and spleen, a spread that shows how much difficulty depends on the target (Antonelli et al., 2022). Two inconsistencies in the paper: it calls NSD both the normalized surface Dice and the normalized surface distance, and its Table 2 lists the liver task as 210 volumes while its training and test counts (131 and 70) and Fig. 1 give 201.

BraTS. The Multimodal Brain Tumor Image Segmentation Benchmark, first held at MICCAI 2012 and 2013, segments glioma sub-regions in multi-contrast MRI. Its 2015 report evaluated twenty algorithms on 65 scans annotated by up to four raters and 65 simulated ones, found that no single algorithm ranked among the top five for all three sub-regions, and used the agreement between human raters as the bar an algorithm should reach (Menze et al., 2015). BraTS continued in later years; its 2020 edition was won by an nnU-Net entry (Antonelli et al., 2022).

Other challenges cover single organs or diseases, such as the 2019 Kidney and Kidney Tumor Segmentation Challenge (KiTS), where eight nnU-Net derivatives ranked among the top 15 entries (Antonelli et al., 2022). Challenge participants usually register to get access to the data (Antonelli et al., 2022), and some datasets also require a data-use agreement.

How much a ranking says. Maier-Hein et al. (2018) showed that challenge rankings are fragile. Re-ranking the 2015 segmentation challenges by the 95th-percentile Hausdorff distance instead of the Hausdorff distance moved one algorithm from last (10th) place to first; changing how test cases are aggregated reshuffled almost all teams in some tasks; leaving out a single test case let a team other than the winner rank first in up to 16% of cases; and reference annotations by two different observers produced different rankings. In 62% of segmentation tasks the number of annotators of the reference was not reported. The statistical significance article covers paired tests and confidence intervals for such comparisons.

Evaluation Metrics

Overlap

The Dice coefficient is the standard overlap score; Maier-Hein et al. (2018) found it in 92% of 383 segmentation tasks. The Dice article covers its definition, its relation to IoU, aggregation, empty masks and its sensitivity to small structures. Dice counts voxels, not where the errors lie, so it is complemented by boundary-based metrics.

Surface distances

Let $\partial P$ and $\partial G$ be the boundary voxels (or surface elements) of the prediction and the reference, with coordinates in millimetres, so that the voxel spacing enters every distance, and let $d(\mathbf{x}, S) = \min_{\mathbf{y} \in S} \|\mathbf{x} - \mathbf{y}\|$. The Hausdorff distance is the larger of the two directed maxima (Taha and Hanbury, 2015, define it this way for finite point sets, with distances in voxels):

$$\mathrm{HD}(P, G) = \max\Big\{ \max_{\mathbf{p} \in \partial P} d(\mathbf{p}, \partial G),\; \max_{\mathbf{g} \in \partial G} d(\mathbf{g}, \partial P) \Big\}.$$

It reports the single worst boundary point, so one stray voxel far from the target sets its value; the Chamfer distance article discusses the same quantity for point sets. The 95th-percentile Hausdorff distance (HD95) replaces each maximum by a percentile, a form of the quantile method of Huttenlocher et al., which Taha and Hanbury describe as one way to handle the noise and outliers common in medical segmentations. In DeepMind’s surface-distance code, HD95 is the larger of the two directed 95th percentiles, with each surface element weighted by its area (DeepMind surface-distance, compute_robust_hausdorff).

The average symmetric surface distance (ASSD) averages instead of taking a maximum. Definitions differ: one pools the distances of both boundaries,

$$\mathrm{ASSD}(P, G) = \frac{\sum_{\mathbf{p} \in \partial P} d(\mathbf{p}, \partial G) + \sum_{\mathbf{g} \in \partial G} d(\mathbf{g}, \partial P)}{|\partial P| + |\partial G|},$$

Taha and Hanbury’s average distance takes the larger of the two directed means, and Metrics Reloaded lists ASSD and the mean absolute surface distance (MASD) as separate metrics. The normalized surface distance at tolerance $\tau$, also called the surface Dice, is the fraction of both boundaries that lies within $\tau$ of the other:

$$\mathrm{NSD}_\tau(P, G) = \frac{\big|\{\mathbf{p} \in \partial P : d(\mathbf{p}, \partial G) \le \tau\}\big| + \big|\{\mathbf{g} \in \partial G : d(\mathbf{g}, \partial P) \le \tau\}\big|}{|\partial P| + |\partial G|},$$

with sizes measured as surface areas in DeepMind’s implementation (compute_surface_dice_at_tolerance). It is 1 when every boundary point is within the tolerance, and it ignores errors smaller than $\tau$, which is how it absorbs annotation imprecision; the tolerance is chosen per structure, as in the Decathlon.

Metrics Reloaded

Metrics Reloaded is a consensus framework by a large international group for choosing metrics from the properties of a problem (Maier-Hein et al., 2024). For semantic segmentation it recommends:

• computing metrics per image and then aggregating over images, respecting the hierarchy of the data (for example several images per patient), rather than pooling all voxels of a dataset;
• an overlap metric, Dice or IoU by default, unless the targets are consistently small relative to the voxel grid and the reference may be noisy; the $F_\beta$ score when false positives and false negatives have different costs, and the centerline Dice (clDice) for tubular structures;
• complementing it with a boundary-based metric: NSD when annotation imprecision should be compensated for; otherwise, depending on how errors should be penalized, NSD or Boundary IoU (by existence), ASSD or MASD (by distance over the whole contour), or HD or a percentile of HD (by distance, focused on outliers);
• not using boundary-based metrics for semantic segmentation when target structures can touch or overlap (Maier-Hein et al., 2024, Extended Data Fig. 7), and confidence intervals for all metrics when the test set is small.

Pitfalls

The companion paper by Reinke et al. (2024) catalogues how metrics mislead. For segmentation:

• Small structures. Single-voxel differences change Dice greatly, and one erroneous voxel in the reference can inflate the Hausdorff distance of a small structure; HD95 was designed to tolerate such outliers.
• Implementations differ. Boundary extraction is not standardized: the Liver Tumor Segmentation challenge’s implementation and DeepMind’s can give different ASSD values, which matters most for metrics sensitive to small contour changes such as HD.
• Empty references and predictions make several metrics undefined (division by zero, or no boundary to measure from). A method that misses a lesion entirely has no surface distance, so an average over the remaining cases flatters it unless missed cases are scored explicitly; the Decathlon set undefined values to 0 for ranking (Antonelli et al., 2022). The example shows this.
• Spacing. Distances computed in voxel units, or with the wrong spacing, are not comparable across scans, and with anisotropic voxels they weigh errors across slices differently from errors within a slice (our reading); the voxel spacing is part of the metric.

Metrics also measure agreement with one reference, not correctness: when expert raters agree at a Dice of 0.74 to 0.85, as in BraTS (Menze et al., 2015), differences between methods within that range are hard to interpret.

Example: Cross-Entropy and Dice Loss on a Small 3D Target

The PyTorch script (PyTorch 2.14.1, NumPy 2.5.4, SciPy 1.18.1; about 40 s on a laptop CPU) generates synthetic CT-like volumes of $12 \times 32 \times 32$ voxels with a spacing of $3 \times 0.8 \times 0.8$ mm: a large “organ” with one faint spherical “lesion” of radius 2.5 to 5 mm, which is the target, in correlated noise. A two-level 3D U-Net is trained for 150 steps with plain cross-entropy (CE) and with a soft Dice loss computed per volume, for three seeds each (the seed sets the initialization and the order of batches; the data are the same). On 40 test volumes it reports mean Dice, HD95 in millimetres and NSD at a 2 mm tolerance (both averaged over the volumes where the network predicted something), HD95 recomputed as if the voxels were isotropic, the number of empty predictions, and the ratio of predicted to true lesion volume. Surfaces are boundary voxels, unweighted, a simplification of the area-weighted surfaces of DeepMind’s code.

import sys
import numpy as np
import torch
from torch import nn
import torch.nn.functional as F
from scipy import ndimage as ndi

torch.set_num_threads(4)
SEEDS = [int(a) for a in sys.argv[1:]] or [0, 1, 2]
SPACING = np.array([3.0, 0.8, 0.8])        # mm per voxel: slice thickness, rows, columns
SHAPE = (12, 32, 32)                       # 36 x 25.6 x 25.6 mm field of view

def make_volume(rng):
    """Noisy synthetic CT-like volume: a large 'organ' with one small, faint 'lesion'.
    Returns the image and the lesion mask (the segmentation target)."""
    z, y, x = np.meshgrid(*(np.arange(n) * s for n, s in zip(SHAPE, SPACING)), indexing="ij")
    organ = ((z - 18) / 15) ** 2 + ((y - 12.8) / 10) ** 2 + ((x - 12.8) / 10) ** 2 <= 1
    r = rng.uniform(2.5, 5.0)                                    # lesion radius in mm
    cz, cy, cx = 18 + rng.uniform(-6, 6), 12.8 + rng.uniform(-4, 4), 12.8 + rng.uniform(-4, 4)
    lesion = (z - cz) ** 2 + (y - cy) ** 2 + (x - cx) ** 2 <= r ** 2
    noise = ndi.gaussian_filter(rng.standard_normal(SHAPE), (0.5, 1.0, 1.0))
    img = 0.5 * organ + 0.6 * lesion + 0.4 * noise / noise.std()
    return img.astype(np.float32), lesion

def dataset(n, seed):
    rng = np.random.default_rng(seed)
    imgs, masks = zip(*(make_volume(rng) for _ in range(n)))
    return torch.tensor(np.stack(imgs))[:, None], torch.tensor(np.stack(masks)).long()

def block(cin, cout):
    # the first convolution is in-plane only (1x3x3), the second 3x3x3
    return nn.Sequential(nn.Conv3d(cin, cout, (1, 3, 3), padding=(0, 1, 1)), nn.ReLU(),
                         nn.Conv3d(cout, cout, 3, padding=1), nn.ReLU())

class TinyUNet3D(nn.Module):
    """Two-level 3D U-Net; pooling halves only the in-plane axes (anisotropic voxels)."""
    def __init__(self, c=8):
        super().__init__()
        self.enc0, self.enc1 = block(1, c), block(c, 2 * c)
        self.up = nn.ConvTranspose3d(2 * c, c, (1, 2, 2), stride=(1, 2, 2))
        self.dec0 = block(2 * c, c)
        self.head = nn.Conv3d(c, 2, 1)

    def forward(self, x):
        e0 = self.enc0(x)
        e1 = self.enc1(F.max_pool3d(e0, (1, 2, 2)))
        return self.head(self.dec0(torch.cat([self.up(e1), e0], 1)))

def soft_dice_loss(logits, target, eps=1.0):
    p = logits.softmax(1)[:, 1].flatten(1)                      # foreground probability
    g = target.flatten(1).float()
    dice = (2 * (p * g).sum(1) + eps) / (p.sum(1) + g.sum(1) + eps)
    return 1 - dice.mean()                                       # per volume, then averaged

def surface_distances(pred, gt, spacing):
    """Distances (mm) from each boundary voxel of one mask to the other mask's boundary."""
    border = lambda m: m & ~ndi.binary_erosion(m)
    bp, bg = border(pred), border(gt)
    d_to_g = ndi.distance_transform_edt(~bg, sampling=spacing)
    d_to_p = ndi.distance_transform_edt(~bp, sampling=spacing)
    return d_to_g[bp], d_to_p[bg]                                 # pred->gt, gt->pred

def evaluate(net, x, y, tol=2.0):
    with torch.no_grad():
        pred = net(x).argmax(1).numpy().astype(bool)
    gt = y.numpy().astype(bool)
    dice, hd95, hd95_iso, nsd, empty = [], [], [], [], 0
    for p, g in zip(pred, gt):
        dice.append(2 * (p & g).sum() / (p.sum() + g.sum()))
        if not p.any():                       # empty prediction: distances undefined
            empty += 1
            continue
        a, b = surface_distances(p, g, SPACING)
        hd95.append(max(np.percentile(a, 95), np.percentile(b, 95)))
        nsd.append(((a <= tol).sum() + (b <= tol).sum()) / (len(a) + len(b)))
        a, b = surface_distances(p, g, [0.8, 0.8, 0.8])   # spacing wrongly taken as isotropic
        hd95_iso.append(max(np.percentile(a, 95), np.percentile(b, 95)))
    mean = lambda v: f"{np.mean(v):.2f}" if v else "n/a"     # over non-empty predictions only
    return np.mean(dice), mean(hd95), mean(hd95_iso), mean(nsd), empty, pred.sum() / gt.sum()

x_tr, y_tr = dataset(160, seed=1)
x_te, y_te = dataset(40, seed=2)
print(f"foreground fraction (training): {y_tr.float().mean():.4f}; "
      f"lesion size {y_te.sum((1, 2, 3)).min()}-{y_te.sum((1, 2, 3)).max()} voxels")
print("loss  seed  Dice   HD95 mm  HD95 if isotropic  NSD@2mm  empty  pred/true volume")
results = {}
for loss_name in ("CE", "Dice"):
    for seed in SEEDS:
        torch.manual_seed(seed)
        net = TinyUNet3D()
        opt = torch.optim.Adam(net.parameters(), lr=3e-3)
        g = torch.Generator().manual_seed(seed)
        for step in range(150):
            idx = torch.randint(0, len(x_tr), (4,), generator=g)
            logits = net(x_tr[idx])
            if loss_name == "CE":
                loss = F.cross_entropy(logits, y_tr[idx])
            else:
                loss = soft_dice_loss(logits, y_tr[idx])
            opt.zero_grad()
            loss.backward()
            opt.step()
        r = evaluate(net, x_te, y_te)
        results.setdefault(loss_name, []).append(r)
        print(f"{loss_name:4s}  {seed:4d}  {r[0]:.3f}  {r[1]:>7s}  {r[2]:>17s}  {r[3]:>7s}"
              f"  {r[4]:5d}  {r[5]:16.2f}")
for loss_name, rs in results.items():
    d = [r[0] for r in rs]
    print(f"{loss_name}: Dice over seeds {min(d):.3f}-{max(d):.3f}")

Output:

foreground fraction (training): 0.0102; lesion size 33-274 voxels
loss  seed  Dice   HD95 mm  HD95 if isotropic  NSD@2mm  empty  pred/true volume
CE       0  0.462     3.86               2.57     0.76      3              0.71
CE       1  0.000      n/a                n/a      n/a     40              0.00
CE       2  0.000      n/a                n/a      n/a     40              0.00
Dice     0  0.520     6.77               4.09     0.68      0              1.69
Dice     1  0.564     4.91               2.93     0.79      0              0.86
Dice     2  0.564     5.01               3.03     0.79      0              0.87
CE: Dice over seeds 0.000-0.462
Dice: Dice over seeds 0.520-0.564

The lesions fill 1% of the training voxels. With cross-entropy, two of the three runs converge to predicting background everywhere. In our reading, that answer is already right for 99% of the voxels, so it is a low-loss solution that a short training run does not leave; an empty prediction scores a Dice of 0 on every volume. The one CE run that learned still under-segments, with 71% of the true lesion volume, and misses 3 of the 40 lesions entirely. With the Dice loss, whose value depends on the overlap with each lesion and not on the number of background voxels, all three runs learn; one of them over-segments (1.69 times the true volume), and its HD95 is worse accordingly. The script takes seeds as arguments: run with seeds 0 to 9 (python seg3d.py 0 1 2 3 4 5 6 7 8 9, about 2 minutes), cross-entropy collapsed to an empty prediction in 9 of 10 runs and the Dice loss in 2 of 10 (seeds 4 and 8), and the Dice runs that learned reached a mean Dice of 0.487 to 0.579. The comparison holds for this budget only: a small network, 150 steps and no foreground oversampling. The example does not show whether longer training, the foreground oversampling of nnU-Net or a compound loss would close the gap.

Two evaluation pitfalls are visible too. The CE seed-0 run has the lowest HD95 of all six runs, because surface distances are averaged only over the 37 volumes where it predicted something; the three lesions it missed count as a Dice of 0 but are absent from its HD95. Reporting a surface distance alone, without the number of missed targets or a penalty for them, would rank the run that misses lesions first. And computing HD95 as if the voxels were isotropic at 0.8 mm shrinks the values by 33 to 40%, because an error of one slice is then counted as 0.8 mm instead of 3 mm; the error is not a constant factor, since it depends on how much of each discrepancy lies across slices.

Applications

Segmentation turns an image into regions that can be measured or used by later steps. The Decathlon’s organizers name computer-aided diagnosis, biomarker extraction, surgical intervention planning and disease prognosis (Antonelli et al., 2022), and the MedSAM authors diagnosis, treatment planning and the monitoring of disease progression, noting that manual delineation is time-consuming and requires expertise (Ma et al., 2024). The Decathlon’s own tasks show the range of targets: brain tumor sub-regions, the left atrium, the hippocampus, the prostate zones, the liver, pancreas, spleen and hepatic vessels, and tumors of the liver, lung, pancreas and colon. In biology, segmentation of cells, nuclei and tissue in microscopy and histology plays the same role. A segmentation used in patient care needs clinical validation, which the benchmark metrics above do not provide on their own (Antonelli et al., 2022); this article describes methods and evaluation, not clinical use.

Open Problems

• Generalization across sites and scanners. Benchmarks test on data from the same sources as training; performance at a new hospital, on a new scanner or protocol, is not guaranteed.
• Small and absent targets. Overlap metrics and losses behave badly on tiny structures and on cases with no target, and lesion-level detection and segmentation are often evaluated separately.
• Reference quality. Single-rater references, as in the Decathlon, cannot show where a method falls within expert variation; multi-rater datasets and metrics that use them are rare.
• Metric choice and reporting. Rankings change with the metric, the aggregation and the annotator (Maier-Hein et al., 2018), and metric implementations differ (Reinke et al., 2024); Metrics Reloaded is a step towards problem-driven choices.
• Uncertainty. Most methods output one label map; estimating where the prediction is unreliable, and how that relates to inter-rater variation, remains open.
• General and promptable models. nnU-Net automated configuration for one dataset at a time; promptable foundation models such as MedSAM aim at one model for many tasks, and how they compare with dataset-specific training across modalities and target sizes is still being studied.