Organ Segmentation

Organ segmentation delineates anatomical organs, such as the liver, spleen, kidneys, pancreas and the large vessels, voxel by voxel in CT and MRI volumes, usually many organs at once. The masks give organ volumes and shapes, define the organs to spare in radiotherapy planning, and serve as the first step of many other analyses. It is the anatomical branch of medical image segmentation, which covers the general task, the losses, and the overlap and surface metrics; this article covers what is particular to organs: a strong anatomical prior that can be carried from labelled examples by registration, many labels of very different sizes in one volume, and benchmarks built around multi-organ label sets.

Two families of methods have dominated. Atlas-based methods register labelled reference scans to the new scan and carry their labels along, and multi-atlas methods combine the labels of several such references by label fusion. Learned methods train a network on labelled volumes, and whole-body models such as TotalSegmentator now segment more than a hundred structures in one pass.

Problem Definition

Given a volume $I$ on a voxel grid $\Omega$ with known voxel spacing, predict a label map $L: \Omega \to \{0, 1, \dots, K\}$, where 0 is background and $1, \dots, K$ are organs from a fixed list. Labels are usually exclusive, one organ per voxel. The list and its numbering are part of the problem: the 2015 Multi-Atlas Labeling Beyond the Cranial Vault challenge (BTCV) numbered 13 abdominal structures from (1) spleen to (13) left adrenal gland, and AMOS used 15 (BTCV pages; Ji et al., 2022).

What sets organs apart from lesions, the subject of tumor segmentation, is that every patient has them, in roughly the same arrangement. Shape, position and neighbourhood are informative, which is why a labelled example of another patient, deformed to fit, is already a reasonable segmentation. The prior is not absolute: the BTCV organizers note that some of their patients have no right kidney or no gallbladder, and TotalSegmentator’s test set contains a removed spleen and a transplanted kidney (BTCV pages; Wasserthal et al., 2023). A method must therefore handle organs that are absent, displaced or duplicated, and a metric must say what an absent organ scores.

Input and Output

The input is a CT or MRI volume. Abdominal CT is often acquired after contrast injection, at a phase chosen for the question: the BTCV scans were taken in the portal venous phase, 512 × 512 pixels with 85 to 198 slices, in-plane pixels of 0.54 to 0.98 mm and slices 2.5 to 5.0 mm thick (BTCV pages). CT intensities are in Hounsfield units, so a fixed intensity window means the same tissue range on every calibrated scanner. MRI has no such scale and varies by sequence: CHAOS gave T1-weighted in-phase and opposed-phase images (registered to each other, so one reference served both) and T2-weighted SPIR images (not registered to the T1 images), with pixels of 0.72 to 2.03 mm and slices 4.4 to 8.0 mm thick (Kavur et al., 2021). The spacing, read from the file header with the orientation (medical image preprocessing), enters every volume and distance computed later; computed tomography and MRI describe how the images are formed.

The output is a label map on the input grid, or one probability map per organ. TotalSegmentator, for example, writes one mask file per structure, or with an option a single file holding all labels (TotalSegmentator README, as of October 2026).

Variants

• Abdominal multi-organ segmentation in contrast-enhanced CT is the most benchmarked form: BTCV, AMOS and the abdominal tasks of other challenges.
• Whole-body segmentation covers organs, bones, muscles and vessels throughout the body: TotalSegmentator’s first version segmented 104 structures (27 organs, 59 bones, 10 muscles and 8 vessels) (Wasserthal et al., 2023).
• Organs at risk in radiotherapy planning are the organs near a tumour that the dose plan must spare, such as the parotid glands in head and neck CT (Iglesias & Sabuncu, 2015).
• MRI and cross-modality segmentation. CHAOS asked for the liver, kidneys and spleen in MRI, and in two of its five tasks for one system that segments both CT and MRI; AMOS has CT and MRI subsets with the same 15 labels (Kavur et al., 2021; Ji et al., 2022).
• Partial labels. Datasets label different subsets of organs. CHAOS’s fourth task made this explicit: its CT scans had only the liver labelled and its MRI scans four organs, and one model had to produce the right outputs for both, because, as the organizers explain, a scan acquired for one organ may not show the others completely (Kavur et al., 2021).

Challenges

• Sizes spanning orders of magnitude. In AMOS the liver has about 200 times as many labelled voxels as an adrenal gland, and the baseline networks segmented large organs (spleen, liver, kidneys) well and small ones (duodenum, adrenal glands) poorly (Ji et al., 2022). An average over organs mixes these regimes.
• Low contrast between neighbours. Some organs differ little in intensity from what surrounds them; the pancreas is the usual example, and in the Medical Segmentation Decathlon it was one of the harder targets (medical image segmentation gives the figures).
• Deformable, mobile anatomy. Abdominal organs shift and change shape between patients and with breathing, bowel content and position. Iglesias and Sabuncu (2015) note that multi-atlas methods were applied to the abdomen despite the relatively poor performance of registration there compared with brain MRI, because organs shift within the abdominal cavity.
• Pathology and anatomical variants. Tumours, resections and transplants change the expected anatomy. TotalSegmentator’s authors excluded 40 series in which the annotator could not segment certain structures because of high ambiguity, for example structures highly distorted by pathology, and showed qualitative test-set examples in which the final model handled broken bones, displaced bowel, a missing spleen and a duplicated kidney (Wasserthal et al., 2023).
• Scanner and protocol shift. In AMOS, models trained on three CT scanners did about as well on two unseen CT scanners, while on MRI the U-Net baseline fell from a mean Dice of 86.05% on scanners seen in training to 64.70% on an unseen one (Table 6 of Ji et al., 2022, in the NeurIPS proceedings, Table 7 in arXiv:2206.08023v3; the text of both gives 64.07%). The authors attribute the difference to CT scans following the same imaging standards, while MRI intensities differed between scanners.
• Annotation cost and variability. Labelling many organs by hand is costly, and recent datasets were labelled with model assistance: TotalSegmentator’s annotators refined the predictions of models retrained after 5, 20 and 100 patients, and AMOS’s radiologists refined coarse labels from networks trained on 50 CT and 20 MRI scans labelled by hand (Wasserthal et al., 2023; Ji et al., 2022). CHAOS measured the variability of its experts directly, as the evaluation section describes.

Approaches

Atlas-based segmentation

An atlas, in this sense, is an image with an expert’s labels. The classical approach treats segmentation as registration: establish a spatial correspondence between the atlas and the new image, then transfer, or propagate, the atlas labels through it (Iglesias & Sabuncu, 2015). With a transform $\varphi$ that maps each point of the new image into the atlas, the propagated label at $\mathbf{x}$ is the atlas label at $\varphi(\mathbf{x})$, read with nearest-neighbour interpolation; medical image registration covers how $\varphi$ is estimated. Thirion (1998) already applied a 3D inter-patient deformation computed with his demons algorithm to a fully labelled reference brain and noted that the inferred labels were mostly correct only where the two subjects had no local morphological differences.

That limitation drove the history. Iglesias and Sabuncu (2015) describe how a single atlas with a deformation model usually cannot capture the range of anatomical variation, and how early atlas-guided segmentation came to be dominated by probabilistic atlases instead: all training images registered into one atlas space, label probabilities precomputed there, and a new image segmented in that space by probabilistic inference. A probabilistic atlas needs only one expensive registration per new image and gives a common frame for population statistics, but it summarizes the training set rather than using each labelled example.

Multi-atlas segmentation and label fusion

In a series of papers in 2003 and 2004, Rohlfing and colleagues proposed an alternative in which the atlases are not summarized in a model but each one remains available for segmenting the new image. Klein et al. (2005), Heckemann et al. (2006) and others followed, and the approach became known as multi-atlas segmentation. In its classical form, each atlas is registered to the new image separately, each propagates its labels, and the propagated labels are combined (Iglesias & Sabuncu, 2015). The survey organizes a multi-atlas method into atlas generation, offline learning, registration, atlas selection, label propagation, online learning, label fusion and post-processing, steps that later methods merge, reorder or connect in feedback loops. Its first application, by Rohlfing and colleagues, was to confocal microscopy images of bee brains.

With $N$ atlases, transforms $\varphi_i$ and propagated labels $L_i(\varphi_i(\mathbf{x}))$, most fusion rules can be written as a weighted vote (our notation):

$$\hat{L}(\mathbf{x}) = \arg\max_{l} \sum_{i=1}^{N} w_i(\mathbf{x}) \, \big[\, L_i(\varphi_i(\mathbf{x})) = l \,\big],$$

where $[\cdot]$ is 1 when the condition holds and 0 otherwise. The rules surveyed by Iglesias and Sabuncu (2015) differ in the weights:

• Best atlas selection keeps a single atlas, usually the one whose registered image best matches the new image, judged for example by the registration cost (sum of squared differences, normalized cross-correlation or mutual information), and discards the others (Rohlfing et al., 2004).
• Majority voting, $w_i = 1$, takes the most frequent label at each voxel. It uses every atlas everywhere but ignores the image intensities (Rohlfing et al., 2004; Klein et al., 2005; Heckemann et al., 2006).
• Global weighted voting gives each atlas one weight from its similarity to the new image; the first such method, by Artaechevarria et al. (2008), used weights proportional to normalized mutual information.
• Local weighted voting lets the weight vary over space, because registration accuracy does: weights inversely proportional to local intensity differences, local cross-correlation, or a Gaussian function of the intensity difference, among many others. The survey notes that these improve on global weights but that which local measure is optimal remains unclear.
• Weights from atlas correlations. Wang et al. (2011, 2013) derived the weights by minimizing the expected segmentation error, exploiting the correlation structure between the atlases so that they are no longer assumed independent, and refined the correspondences by a local patch search. Patch-based fusion compares image patches around each voxel to set the weights (Coupé et al., 2011), or labels whole patches and fuses the overlapping results (Rousseau et al., 2011).
• Probabilistic fusion. Sabuncu et al. (2010) proposed a generative model in which a hidden field records which atlas generated each voxel of the new image; it includes majority voting, best atlas selection and the weighted schemes as special cases. STAPLE (Warfield, Zou & Wells, 2004) was developed to model manual segmentations as noisy observations of a hidden true segmentation, with the noise modelled by a stationary confusion matrix; it was binary at first, was extended to many labels by Rohlfing and colleagues, and a family of fusion methods builds on it.

Atlas selection, using only the most relevant atlases, saves time and can help: it has been observed to improve majority voting but to matter less for weighted fusion. The main cost of the approach is computational, since the work grows at least linearly with the number of atlases and each atlas needs a deformable registration; the survey’s authors saw this demand as one of the main reasons the methods had not yet been widely adopted in clinical applications (Iglesias & Sabuncu, 2015).

BTCV, held at MICCAI 2015, was built around these methods. Its organizers wrote that multi-atlas labeling had been very successful for brain and cranial structures, and less developed outside the head and in soft tissue, so they released abdominal and cervical CT. Each challenge had a “free” competition, in which any registration method could be used, and a “standard” one, in which teams had to use registrations supplied by the organizers, computed with NiftyReg between all pairs of training scans and from every training scan to every test scan. The overall abdominal winners were PATH (standard competition), a method combining label fusion with a sequential learning approach, and an entry by Mattias Heinrich built on the deeds registration with self-similarity context and joint fusion (free competition); a statistical shape model with random regression forest voting won the spleen in the free competition (BTCV pages).

Learned segmentation

A convolutional network trained on labelled volumes learns the mapping from intensities to organ labels directly and segments a new scan in one forward pass, with no registration at test time. The architectures and losses are those of medical image segmentation, and nnU-Net, which configures a plain U-Net from a fingerprint of the training data, is the usual baseline. In CHAOS, nnU-Net was the online winner of the four-organ MRI task, with a mean score of 72.44 out of 100 (Kavur et al., 2021). The AMOS authors trained six networks in the nnU-Net codebase and found that the U-Net beat the more recent Transformer-based and hybrid models on both CT and MRI: 88.87% mean Dice on the CT validation set against 86.37% for Swin-UNETR and 85.63% for nnFormer (Ji et al., 2022). The nnU-Net developers’ larger 2024 comparison, which included BTCV and AMOS, is described in nnU-Net.

Combinations exist: registration can supply a common coordinate frame or a prior for a network, and segmentation networks can guide registration, as in the Learn2Reg winner described in medical image registration.

Whole-body models

TotalSegmentator is an nnU-Net model trained to segment 104 anatomical structures in CT (Wasserthal et al., 2023). Its authors, at University Hospital Basel, sampled 1,368 CT series at random from their hospital’s archive for the years 2012, 2016 and 2020, excluded extremity scans, series with missing sections and ambiguous cases, and kept 1,204, resampled to 1.5 mm isotropic voxels: 1,082 for training, 57 for validation and 65 for testing. The data are deliberately heterogeneous: 8 sites, 16 scanners, several contrast phases and kernels, and 645 patients with pathology. Annotation was model-assisted, as described above, and supervised by two physicians; the final model was trained independently of the intermediate annotation models. The authors changed two nnU-Net defaults: they removed mirroring from the augmentation, because otherwise the model could not tell left from right structures, and raised the training epochs from 1,000 to 4,000. To limit memory, the 104 classes of the 1.5 mm model were split among five networks, while a second, 3 mm model used a single network.

On the 65 test cases the authors reported a Dice of 0.943 and an NSD (at 3 mm) of 0.966 for the 1.5 mm model, and a Dice of 0.840 with the same NSD for the 3 mm model. Against an nnU-Net trained by five-fold cross-validation on the 30 BTCV training scans, on the 13 BTCV structures, their model scored a higher Dice on their own test set (0.932 against 0.871) and on the 30 BTCV scans (0.849 against the cross-validated 0.839) (Wasserthal et al., 2023; the supplement’s list of the BTCV structures is headed “12 structures” and omits the esophagus). This is a retrospective evaluation against model-assisted reference labels from one hospital’s archive, plus one external public dataset.

As of October 2026, the repository’s default CT task segments 117 classes and a separate task 50 classes in MRI, and the README states that the tool is not a medical device and not intended for clinical use (TotalSegmentator repository).

Example: Multi-Atlas Segmentation of Synthetic Abdomens

The script below builds synthetic CT-like volumes of an abdomen-like body with six organs (liver, spleen, both kidneys, aorta and pancreas, plus an unscored vertebra), with intensities in Hounsfield-like units and anatomy that varies with the seed: organ positions and sizes, a global shift and a smooth random deformation of 3 mm rms per axis. Voxels are 3.0 × 1.5 × 1.5 mm. Six volumes serve as atlases and three as targets. Each atlas is registered to each target with the TV-L1 optical flow of scikit-image, a dense deformable registration that assumes the same intensities in both images, which holds here because all volumes come from the same simulated scanner. The atlas labels are then propagated with nearest-neighbour interpolation and fused. The script compares atlas labels used without registration, single atlases after registration (averaged over the six), the best atlas by the sum of squared differences after registration, majority voting, and locally weighted voting with $w_i(\mathbf{x}) = \exp(-D_i(\mathbf{x})/h)$, where $D_i$ is the squared difference between target and warped atlas intensities (scaled to [0, 1]) smoothed by a Gaussian of 2 voxels and $h = 0.002$, a Gaussian intensity-difference weight of the kind the survey lists. It reports Dice and the 95th-percentile Hausdorff distance (HD95) per organ, in millimetres, averaged over the three targets. Boundaries are boundary voxels, unweighted. Run with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2; it takes about 20 s on a laptop CPU. All parameters were fixed before the first run and not changed afterwards; two runs gave identical output.

import numpy as np
from scipy import ndimage as ndi
from skimage.registration import optical_flow_tvl1

SHAPE, SPACING = (40, 64, 64), np.array([3.0, 1.5, 1.5])   # voxels; mm per voxel (z, y, x)
ORGANS = ["liver", "spleen", "kidney_R", "kidney_L", "aorta", "pancreas"]   # labels 1-6
HU = np.array([20, 60, 50, 150, 150, 200, 40, 400])  # soft tissue, organs 1-6, vertebra (7, unscored)
GRID = np.stack(np.meshgrid(*[np.arange(n) for n in SHAPE], indexing="ij")).astype(float)
Z, Y, X = (GRID[i] * SPACING[i] for i in range(3))             # voxel centres in mm

def make_subject(seed):
    """Synthetic abdomen-like CT volume and its label map; anatomy varies with the seed."""
    rng = np.random.default_rng(seed)
    shift = rng.uniform(-3, 3, 3)                                # global position, mm
    ell = lambda c, r: sum(((q - ci - si) / ri) ** 2 for q, ci, si, ri in zip((Z, Y, X), c, shift, r)) <= 1
    lab = np.zeros(SHAPE, np.int64)
    # (centre z, y, x) and radii in mm; painted in this order, later ones overwrite earlier ones
    shapes = [(1, (75, 45, 28), (30, 24, 18)), (2, (80, 60, 74), (18, 12, 9)),
              (3, (50, 66, 30), (18, 9, 8)), (4, (55, 66, 66), (18, 9, 8)),
              (6, (62, 48, 52), (8, 6, 20))]
    for k, c, r in shapes:
        c = np.array(c) + rng.uniform(-4, 4, 3)
        lab[ell(c, np.array(r) * rng.uniform(0.85, 1.15))] = k
    ay, ax = 58 + rng.uniform(-4, 4), 50 + rng.uniform(-4, 4)       # aorta: a tube along z
    lab[(Y - ay - shift[1]) ** 2 + (X - ax - shift[2]) ** 2 <= (6 * rng.uniform(0.85, 1.15)) ** 2] = 5
    lab[(Y - 76 - shift[1]) ** 2 + (X - 48 - shift[2]) ** 2 <= 9 ** 2] = 7     # vertebra
    body = ((Y - 48 - shift[1]) / (36 * rng.uniform(0.92, 1.08))) ** 2 + \
           ((X - 48 - shift[2]) / (44 * rng.uniform(0.92, 1.08))) ** 2 <= 1
    lab[~body] = 0
    # smooth random deformation, 3 mm rms per axis
    disp = [ndi.gaussian_filter(rng.standard_normal(SHAPE), (1.5, 4, 4)) for _ in range(3)]
    disp = [3.0 * d / d.std() / s for d, s in zip(disp, SPACING)]          # mm -> voxels
    lab = ndi.map_coordinates(lab, GRID + np.stack(disp), order=0, mode="nearest")
    body = ndi.map_coordinates(body.astype(np.int64), GRID + np.stack(disp), order=0, mode="nearest") > 0
    img = np.where(body, HU[lab], -1000.0)
    img = ndi.gaussian_filter(img, 0.5) + rng.normal(0, 15, SHAPE)
    return img, lab

norm = lambda img: (np.clip(img, -200, 300) + 200) / 500         # intensities to [0, 1]

def dice(p, g):
    return 2 * (p & g).sum() / (p.sum() + g.sum())

def hd95(p, g):
    """Larger of the two directed 95th-percentile boundary distances, in mm."""
    if not p.any():
        return np.nan
    border = lambda m: m & ~ndi.binary_erosion(m)
    bp, bg = border(p), border(g)
    d_pg = ndi.distance_transform_edt(~bg, sampling=SPACING)[bp]
    d_gp = ndi.distance_transform_edt(~bp, sampling=SPACING)[bg]
    return max(np.percentile(d_pg, 95), np.percentile(d_gp, 95))

def scores(pred, gt):
    return np.array([[dice(pred == k, gt == k), hd95(pred == k, gt == k)] for k in range(1, 7)])

atlases = [make_subject(s) for s in range(6)]
results = {m: [] for m in ["no registration", "single atlas", "best atlas (SSD)",
                           "majority vote", "local weighted vote"]}
for t_seed in (100, 101, 102):
    img, gt = make_subject(t_seed)
    f = norm(img)
    warped_imgs, warped_labs, single, ident = [], [], [], []
    for a_img, a_lab in atlases:
        ident.append(scores(a_lab, gt))                          # atlas labels used as they are
        flow = optical_flow_tvl1(f, norm(a_img))                 # maps target voxels into the atlas
        warped_imgs.append(ndi.map_coordinates(norm(a_img), GRID + flow, order=1, mode="nearest"))
        warped_labs.append(ndi.map_coordinates(a_lab, GRID + flow, order=0, mode="nearest"))
        single.append(scores(warped_labs[-1], gt))
    results["no registration"].append(np.mean(ident, axis=0))
    results["single atlas"].append(np.mean(single, axis=0))
    ssd = [np.mean((f - w) ** 2) for w in warped_imgs]
    results["best atlas (SSD)"].append(single[int(np.argmin(ssd))])
    votes = np.stack([np.stack(warped_labs) == k for k in range(8)]).sum(axis=1)
    results["majority vote"].append(scores(votes.argmax(0), gt))
    w = [np.exp(-ndi.gaussian_filter((f - wi) ** 2, 2) / 0.002) for wi in warped_imgs]
    wvotes = np.stack([sum(wi * (li == k) for wi, li in zip(w, warped_labs)) for k in range(8)])
    results["local weighted vote"].append(scores(wvotes.argmax(0), gt))

print("Dice per organ (mean of 3 targets)")
print(f"{'method':21s}" + "".join(f"{o:>9s}" for o in ORGANS) + "     mean")
for m, r in results.items():
    d = np.mean(r, axis=0)[:, 0]
    print(f"{m:21s}" + "".join(f"{v:9.3f}" for v in d) + f"{d.mean():9.3f}")
print("\nHD95 per organ in mm (mean of 3 targets)")
for m, r in results.items():
    h = np.mean(r, axis=0)[:, 1]
    print(f"{m:21s}" + "".join(f"{v:9.1f}" for v in h) + f"{h.mean():9.1f}")

Output:

Dice per organ (mean of 3 targets)
method                   liver   spleen kidney_R kidney_L    aorta pancreas     mean
no registration          0.692    0.463    0.459    0.474    0.423    0.271    0.464
single atlas             0.738    0.516    0.602    0.622    0.692    0.319    0.581
best atlas (SSD)         0.741    0.570    0.614    0.677    0.751    0.213    0.594
majority vote            0.754    0.600    0.674    0.686    0.786    0.403    0.650
local weighted vote      0.844    0.654    0.819    0.839    0.873    0.446    0.746

HD95 per organ in mm (mean of 3 targets)
no registration           10.4     10.2      9.7      8.3      8.6     10.2      9.6
single atlas               9.0      9.3      7.9      6.6      5.7      9.2      7.9
best atlas (SSD)           9.0      7.8      6.9      5.9      4.4     10.2      7.4
majority vote              8.7      7.5      7.0      5.5      4.6      7.5      6.8
local weighted vote        5.3      7.2      4.4      3.4      2.2      6.8      4.9

Even unregistered atlas labels overlap the target substantially, because the synthetic anatomy, like real anatomy, keeps its arrangement; the large liver overlaps most. Registration helps most for the organs with strong contrast against their surroundings, the aorta (200 against 20) and the kidneys (150), and little for the liver, which already overlapped well, and the pancreas, which differs from soft tissue by 20 units under noise of 15, so that in our reading an intensity-driven registration has little to align there. One registered atlas still leaves most of the error. Choosing the best atlas by a global intensity match improves the mean slightly but makes the pancreas worse, since a criterion summed over the whole volume says little about one small, faint organ. Majority voting improves every organ, which in our reading reflects registration errors that differ between atlases and are outvoted. Local weighting adds nearly as much again as registration did, halves the mean HD95 compared with no registration, and helps the kidneys most and the pancreas least. In our reading, the weight acts as a local intensity check: an atlas that proposes “kidney” where the target shows soft tissue has a large local intensity difference and is down-weighted. That check is strong here because each synthetic organ has a uniform, distinct intensity, more so than in real CT, so the size of the gain should not be expected elsewhere, and it is weakest for the pancreas, as in real data.

Per-organ scores tell more than the mean: the liver has the highest Dice without registration but gains least over the five methods, and its HD95 stays above 5 mm, whereas the aorta, a thin tube, gains most and ends with the highest Dice and an HD95 of about 2 mm. The distances are in millimetres through the spacing, so an error of one slice counts as 3 mm and an in-plane error of one voxel as 1.5 mm. The example has three targets, simple shapes, no affine pre-alignment and default registration parameters; it shows the mechanics of propagation and fusion, not the accuracy of any published method.

Datasets and Benchmarks

BTCV (Multi-Atlas Labeling Beyond the Cranial Vault, MICCAI 2015). The abdominal set has 50 portal venous phase CT scans, selected at random from a colorectal cancer chemotherapy trial and a retrospective ventral hernia study at Vanderbilt University Medical Center, under institutional review board supervision. Two experienced undergraduate students labelled 13 structures (spleen, both kidneys, gallbladder, esophagus, liver, stomach, aorta, inferior vena cava, portal and splenic vein, pancreas, and both adrenal glands), and a radiologist verified the labels. Its protocol (BTCV pages, as of October 2026):

• Splits. 30 training and 20 test scans; the test labels are withheld and used for scoring.
• Registrations. For the standard competition the organizers supplied the 870 pairwise registrations between training scans and the 600 from training to test scans.
• Metrics. Submissions are scored by the Dice similarity coefficient, the symmetrized mean surface distance and the symmetrized Hausdorff distance; a submission may cover all labels or a subset. “Best” algorithms were recognized by their average Dice over the abdomen and cervix data, and the leaderboard is sorted by Dice by default, which the organizers state is not an indication of the winner.
• Access. Data access requires joining the challenge on the Synapse platform.

BTCV became a common benchmark for later networks. The nnU-Net developers found it less suitable than KiTS, AMOS or ACDC for comparing methods, because its statistical noise may exceed the differences between methods (Isensee et al., 2024).

CHAOS (ISBI 2019). Combined (CT-MR) Healthy Abdominal Organ Segmentation provided 40 contrast-enhanced CT scans with the liver labelled and 40 MRI examinations (T1-weighted in-phase and opposed-phase, and T2-weighted SPIR, on a 1.5 T scanner) with the liver, both kidneys and the spleen labelled, from patients without pathology at Dokuz Eylul University Hospital, each split 20/20 into training and test sets (Kavur et al., 2021). Three radiologists labelled every slice and the reference was set by majority voting. Its five tasks were liver segmentation in CT, in MRI and in both with one system, and four-organ segmentation in MRI and in both. The data are on Zenodo under CC BY-SA 4.0, according to the paper. The scoring system is described under evaluation below.

AMOS (MICCAI 2022 challenge; NeurIPS 2022 Datasets and Benchmarks). 500 CT and 100 MRI scans of 600 patients with abdominal tumours or abnormalities, from two hospitals in Shenzhen and eight scanners, with 15 organs labelled: the 13 BTCV structures without the portal and splenic vein, plus the duodenum, bladder and prostate or uterus (Ji et al., 2022). The protocol:

• Splits by scanner. For CT, scans from three scanners were split into 200 training, 100 validation and 78 in-distribution test scans, and 122 scans from two further scanners form an out-of-distribution test set; MRI has 40 training, 20 validation, 11 in-distribution and 29 out-of-distribution test scans.
• Metrics. Dice and normalized surface Dice are computed per organ and averaged over organs (mDice, mNSD); the authors also ask for parameter counts and computational cost.
• Challenge tasks. Task 1 is CT only; Task 2 asks one algorithm to segment both CT and MRI (AMOS challenge website).
• Licence. The data are on Zenodo under CC BY 4.0 (as of October 2026); the appendix of arXiv:2206.08023v3 states CC BY-NC-SA instead.

TotalSegmentator dataset. The CT images and labels behind TotalSegmentator are on Zenodo under CC BY 4.0, with no access request; version 3.0.0 (September 2026) has 1,939 CT images and 117 structures, including 291 paediatric scans, while the 2023 paper describes the first version (Zenodo record; Wasserthal et al., 2023). The README still gives 1,228 subjects, the size of version 2.

Single-organ tasks of the Medical Segmentation Decathlon (liver, pancreas, spleen, among others) and other challenges add further organ data, each with its own label definitions and metrics.

Evaluation Metrics

Organ segmentation is scored per organ with the overlap and boundary metrics of medical image segmentation: the Dice coefficient, surface distances such as HD95 and the average symmetric surface distance in millimetres, and the normalized surface distance (NSD) at a tolerance. Several choices are particular to the multi-organ case.

• Per-organ scores before averages. AMOS reports Dice and NSD per organ and then their mean over organs (Ji et al., 2022). A mean over organs weights the liver and an adrenal gland, about 200 times smaller in AMOS’s labelled voxels, equally, while a score pooled over voxels would be dominated by the liver; the per-organ table shows where a method fails, as in AMOS’s finding that small organs lag.
• Tolerances per organ. NSD ignores boundary errors below its tolerance, which should match what is acceptable for the structure: the Decathlon’s clinicians chose 3 mm for the spleen and 7 mm for the liver (Antonelli et al., 2022), and TotalSegmentator used 3 mm throughout. TotalSegmentator’s two models show why the choice matters: the 3 mm model lost about 0.1 in Dice but kept the NSD of the 1.5 mm model, because its coarser boundaries stayed within 3 mm (Wasserthal et al., 2023).
• Thresholds from observer variability. CHAOS scored each case by four metrics, Dice, relative absolute volume difference, average symmetric surface distance and maximum symmetric surface distance, with distances in millimetres from the DICOM geometry. Each metric was mapped to a score from 0 to 100, with zero beyond a threshold (Dice above 0.8, volume difference below 5%, ASSD below 15 mm, MSSD below 60 mm), and the four scores were averaged; missing results scored zero. The thresholds were set after measuring the experts’ own variability: repeated annotations by one expert reached Dice of 0.97 to 0.98, and annotations by different experts 0.95 to 0.96, with larger relative differences in the distance metrics (Kavur et al., 2021).
• Absent organs. When an organ is missing from the reference, as in BTCV patients without a right kidney or gallbladder, Dice is $0/0$ if nothing is predicted and 0 if anything is, and surface distances need both boundaries; a protocol has to state whether such cases are skipped or scored (tumor segmentation compares the challenge conventions).
• Agreement, not correctness. The reference is one annotation process. With model-assisted labels such as TotalSegmentator’s and AMOS’s, a model’s agreement with the reference may partly reflect the models used to create it (our reading); TotalSegmentator’s authors trained their final model independently of the intermediate annotation models, which they write reduced bias in the test set to a minimum, and argue that fully manual test labels would have introduced a distribution shift (Wasserthal et al., 2023).

Applications

The benchmarks’ authors name the uses that motivate the task. Iglesias and Sabuncu (2015) cite radiotherapy treatment planning as the main driver for head, neck and thoracic CT, where organs at risk must be delineated. CHAOS’s organizers describe liver volume measurement and planning of liver transplant surgery, for both recipient and living donor, and segmentation of the spleen and kidneys, for example for volumetric monitoring of renal tumours (Kavur et al., 2021). TotalSegmentator’s authors name organ volumetry, disease characterization, and surgical and radiotherapy planning, and demonstrated population research: they applied their model to 4,004 whole-body trauma CT scans and reported, among other correlations, that aortic volume rose with age and the attenuation of back muscles fell (Spearman coefficients of 0.64 and −0.74), in a retrospective analysis that, by their account, did not yet correct for confounders (Wasserthal et al., 2023). Organ masks also feed other analyses; TotalSegmentator’s authors suggest, for example, combining their model with a lesion detector to estimate tumour load per body part. This article describes methods and evaluation; a segmentation used in patient care needs clinical validation that benchmark scores do not provide.

Open Problems

• Small and thin structures. Adrenal glands, the duodenum, vessels and thin walls remain the weakest labels in multi-organ benchmarks, and the per-organ mean hides them.
• Shift between scanners, protocols and populations. AMOS’s MRI results show large drops on an unseen scanner, and its authors note that all their data come from two hospitals in one population (Ji et al., 2022); TotalSegmentator’s authors note an over-representation of male patients (Wasserthal et al., 2023).
• Label sets that do not match. Datasets differ in which organs they label and how they define them (whether the inferior vena cava belongs to the liver, as CHAOS had to decide), so combining them requires methods that learn from partial and inconsistent labels.
• Pathology and variants. Organs distorted by tumours, resected or transplanted are rare in training data, and failure on them is what matters most in use.
• References and evaluation. Model-assisted labels, single annotators and benchmarks whose noise exceeds the differences between methods (Isensee et al., 2024) limit what a ranking says; multi-rater references for many organs are rare.
• Uncertainty and quality control. Methods output a mask for every organ; detecting when a mask is wrong, without a reference, remains open. TotalSegmentator’s aging study, for example, discarded segmentations whose volume fell below a plausible lower bound (Wasserthal et al., 2023).