BraTS

BraTS is a series of challenges on brain tumors in MRI, held with the MICCAI conference every year since 2012, and the multi-institutional dataset that grew with it. It began as the Multimodal Brain Tumor Image Segmentation Benchmark, with BRATS spelled in capitals in its early years, at MICCAI 2012 in Nice and 2013 in Nagoya; the 2026 edition was the fifteenth, organized as a cluster of five challenges (CBICA BraTS pages; BraTS 2026 overview). As the challenges came to include histopathology and image synthesis, the organizers renamed the acronym in 2026 from Brain Tumor Segmentation to “Brain TumorS” (BraTS 2026 overview). This article covers the series as a dataset: its editions, its labels and how they changed, access and terms, and the three ways the challenge has scored a segmentation, as the official code computes them. Tumor Segmentation owns the task itself, the 2021 labels and evaluated regions, the case counts, the annotation protocol, the empty-target rule and the ranking scheme, and is linked rather than repeated. The article describes a research benchmark and gives no clinical guidance.

Purpose

The 2026 overview states the aim of the series as advancing brain tumor image analysis by benchmarking algorithms, providing high-quality annotated datasets and setting clinically relevant problems across the course of the disease (BraTS 2026 overview). Up to 2021 the segmentation task was always about gliomas: segmenting them in pre-operative multimodal MRI into nested sub-regions. The 2020 edition, for example, focused on “intrinsically heterogeneous” gliomas and added two tasks that build on the segmentation, prediction of overall survival and the evaluation of uncertainty maps (CBICA, BraTS 2020 overview and tasks). From 2023 the series turned into a cluster of sub-challenges for other tumor entities and populations, each with its own data and evaluation settings.

Dataset Size

The case counts by year, from 50 cases in 2012 to 660 in 2020 and 2,040 in 2021, are given in Tumor Segmentation, from the BraTS 2021 paper (Baid et al., 2021). The 2020 data page describes what each case contains: four MRI volumes, native T1-weighted (T1), post-contrast T1-weighted (T1Gd), T2-weighted (T2) and T2 Fluid Attenuated Inversion Recovery (T2-FLAIR), acquired with different clinical protocols and scanners at 19 institutions; all are NIfTI files, distributed after pre-processing that co-registered them to the same anatomical template, interpolated them to 1 mm³ and skull-stripped them (CBICA, BraTS 2020 data). Every case therefore arrives on the same grid; in our reading, this is why the evaluation settings can give volumes in voxels and millimetres interchangeably (“One voxel = 1 cubic mm”, in the 2023 metrics README’s threshold table).

The data were rebuilt in 2017. The 2020 data page says that the BraTS’14–’16 data, taken from The Cancer Imaging Archive (TCIA), “have been discarded”, because they mixed pre- and post-operative scans and their labels had been produced by fusing the outputs of algorithms that ranked highly in 2012 and 2013. Only the manually annotated BraTS’12–’13 images and labels were kept. For 2017, neuroradiologists assessed the complete TCGA-GBM (n = 262) and TCGA-LGG (n = 199) collections on TCIA and sorted each scan as pre- or post-operative, and all pre-operative scans (135 glioblastoma, 108 lower-grade glioma) were annotated by experts and added; the page links a manuscript for the exact procedure, which this article has not read (CBICA, BraTS 2020 data).

Classes

The label conventions have changed more than once, and every script that reads BraTS files has to know which convention it is reading.

Period	Labels (value: meaning)	Source
2012–2016	a separate non-enhancing tumor label (3) besides necrosis, edema and enhancing tumor	Tumor Segmentation, from Bakas et al. (2018)
2017–2021	1 necrotic and non-enhancing tumor core (NCR/NET), 2 peritumoral edema (ED), 4 GD-enhancing tumor (ET)	CBICA, BraTS 2020 data and tasks
Current configurations (adult glioma; metastases without netc and snfh groups)	1 netc, 2 snfh, 3 et, 4 rc	BraTS_evaluation v0.0.8, config_gli.yaml, config_mets.yaml
Current configuration (pediatric)	1 et, 2 netc, 3 cc, 4 ed	BraTS_evaluation v0.0.8, config_ped.yaml

In 2017 the non-enhancing label was merged into label 1; Tumor Segmentation gives the organizers’ reason. The names in the last two rows are the group names of the evaluation configurations, which give no expansions. Among the readable 2026 pages, a commented-out passage of the METS evaluation page uses “Non-enhancing Tumor Core” and “Resection cavity”. Between the 2020 data and the current configurations the ET label moved from value 4 to value 3, and the pediatric configuration assigns the same values to different structures. The evaluated regions are unions of labels. For the 2017–2021 data they were the enhancing tumor (ET), the tumor core (TC, labels 1 and 4) and the whole tumor (WT, labels 1, 2 and 4) (CBICA, BraTS 2020 tasks). In config_gli.yaml at v0.0.8 they are TC = {1, 3} and WT = {1, 2, 3}, so the resection cavity (label 4) belongs to neither; the pediatric configuration uses TC = {1, 2, 3} and WT = {1, 2, 3, 4}.

Annotation Types

Each case comes with one multi-label volume on the same 1 mm grid as the images. For 2017–2020 the data page says all volumes were segmented manually by one to four raters following one protocol, with the annotations approved by experienced neuroradiologists (CBICA, BraTS 2020 data); Tumor Segmentation describes the protocol and the 2021 change to refining machine-generated labels. The 2020 edition also distributed overall survival in days, age and resection status as a comma-separated file for the survival task (CBICA, BraTS 2020 data).

Splits

Each edition has a training set with labels, a validation set whose labels stay hidden and a test set. In 2020 the validation set was scored by an online portal that allowed multiple submissions, and the test data were released to participants for a 48-hour window only (CBICA, BraTS 2020 data); from 2021 the test was run on submitted containers (Tumor Segmentation). For 2026 the readable GoAT pages state that test sets and validation labels are not released to the public and that the composition of the private test data is not disclosed (BraTS 2026, GoAT data).

Rules on outside data changed. In 2020 additional public or private data were allowed for augmentation if the paper said so and also reported results with BraTS’20 data only (CBICA, BraTS 2020 data). In 2026 GoAT allows only the data provided through its own sub-challenge and states that using any other data, previous BraTS challenges included, will lead to disqualification (BraTS 2026, GoAT data).

Tasks

Segmentation of the tumor sub-regions has been the constant task; others were added and dropped:

• 2020: Task 1 segmentation, Task 2 overall survival prediction (accuracy of a three-way grouping into short, mid and long survivors), Task 3 uncertainty, scored by Dice on voxels kept at several uncertainty thresholds together with the ratios of filtered true positives and negatives. A planned task on pseudoprogression against true recurrence was postponed (CBICA, BraTS 2020 overview and tasks).
• 2023: a cluster of segmentation challenges whose evaluation settings name adult glioma (GLI), meningioma (MEN), sub-Saharan Africa (SSA), pediatric tumors (PED) and metastases (MET) (2023 metrics README, threshold table).
• 2026: five challenges in three computational tasks: segmentation of pre- and post-treatment brain metastases (METS), segmentation of pre- and post-treatment pediatric tumors (PEDS), generalizability across tumors (GoAT), local MRI inpainting, and classification of the histologic landscape of glioma (BraTS-Path) (BraTS 2026 description). GoAT uses pre-operative data collected for earlier BraTS challenges: it trains on the adult glioma training set with labels and on the meningioma and metastasis training sets, each with labels for only a subset, and validates on adult glioma, Africa, meningioma, metastasis and pediatric data (BraTS 2026, GoAT data).

The challenges were presented at MICCAI on 1 October 2026 (BraTS 2026 overview). The inpainting challenge has its own evaluation package, which the BraTS_evaluation README points to and this article does not cover; the registration and uncertainty spin-offs (BraTS-Reg, QU-BraTS) are outside its scope too.

Evaluation

What a BraTS score means depends on the year. The series has computed it in three ways.

Case-wise region scores (through 2021). Each evaluated region of each case is scored by Dice and the 95th-percentile Hausdorff distance (HD95), with sensitivity and specificity added from 2017 (CBICA, BraTS 2020 tasks); the ranking scheme and the convention for cases without enhancing tumor are in Tumor Segmentation. In 2020 the evaluation ran on CBICA’s Image Processing Portal, and the 2020 page promised to publish the scripts it used (CBICA, BraTS 2020 data and tasks). That code was not read for this article, so it is not known here which of the two HD95 forms described below it computed.

Lesion-wise scores (2023). The 2023 challenges introduced lesion-wise Dice and lesion-wise HD95, to penalize missed and false-positive lesions and, as the README explains, to avoid biasing results toward models that capture only large lesions. The README of the repository that published them describes the procedure (2023 metrics README):

1. The ground truth of each region is dilated, and a 26-connected connected-component labelling of the dilated mask decides which ground-truth components count as one lesion; the labels themselves are not changed.
2. The prediction is split into connected components and compared with the ground-truth lesions component by component.
3. Each lesion gets its own Dice and HD95; false positives and false negatives are scored 0, and the mean is taken per case: the sum over the ground-truth lesions divided by TP + FN + FP.
4. Each task has a volume threshold below which lesions are not scored, and a dilation factor, set by radiologists: 50 voxels and dilation 3 for GLI, SSA and PED, 50 and 1 for MEN, 2 and 1 for MET.

The repository has no licence file, so its code was not read; the procedure above comes from its README and figures only. The README also links Sage Bionetworks’ integration of the metrics into the Synapse platform; GitHub’s repository record (read November 2026) shows that repository renamed to brats-infra, which is under the Apache License 2.0, and its 2023 history was not read for this article. They leave three things open that matter for the result: what HD95 a missed or false lesion receives (the text’s “0 score” would reward them under HD95, and no penalty is given), whether the volume threshold applies to ground-truth lesions, predicted components or both, and how a prediction that touches a ground-truth lesion is assigned to it.

panoptica (current, as of November 2026). The current official package, BraTS_evaluation, is a thin layer over panoptica, an instance-wise evaluation library. Its evaluation.py runs a Panoptica_Evaluator loaded from one of six bundled configurations (mets, gli, ped, MenRT, MenPre, GoAT) on each pair of NIfTI files, and metrics_parser.py reduces the result to a CSV (BraTS_evaluation v0.0.8). Sage Bionetworks’ scoring code for the series (brats-infra 4b9a380), whose cohorts are BraTS-MET, BraTS-PED and BraTS-GoAT, pins BraTS_evaluation==0.0.8 in its requirements, which is why this article reads that tag rather than the latest one, v0.0.10. BraTS_evaluation 0.0.8 requires only panoptica >= 2.1.0, so the panoptica version used in scoring is not fixed; this article reads 2.1.7, the latest tag as of November 2026. Read line by line together with panoptica 2.1.7, the segmentation configurations at v0.0.8 compute the following:

• Instances. ConnectedComponentsInstanceApproximator with cca_backend: null, which for 3-D arrays falls back to the cc3d library; cc3d’s connected_components defaults to 26-connectivity at tag 4.1.1. The same code with SciPy’s ndimage.label would use 6-connectivity in 3-D.
• Matching. MaxBipartiteMatching on Dice with threshold 0.000001: a cost matrix of 1 − Dice for every overlapping pair whose Dice is at least the threshold (the comparison is not strict by default), 1 + 10⁻⁶ elsewhere, solved by SciPy’s linear_sum_assignment; an assignment counts as a match only if its cost is below 1. This is the one-to-one matching the Hungarian algorithm solves.
• Detection. TP, FP = predicted instances − TP, FN = reference instances − TP, and the F1 score, which panoptica calls recognition quality (rq), as in panoptic quality.
• Per-instance and global metrics. Dice, normalized surface distance (NSD) and HD95 for each matched pair, and the same three for the binarized region as a whole (“global”). Surface distances are computed with a Euclidean distance transform of the border voxels, and HD95 is np.percentile(np.hstack((sd_ref, sd_pred)), 95): one percentile of both directed distance sets pooled. Medical Image Segmentation describes the other form, the larger of the two directed 95th percentiles; the two can differ widely, as the example below shows.
• NSD tolerance. NSD’s tolerance defaults to the smallest voxel spacing, or to 0.5 when no spacing is given. In our reading of panoptica_result.py, the global metrics are called without a spacing, so global NSD is computed at a tolerance of 0.5 voxels, while the per-instance metrics receive a spacing of (1, 1, 1) and use a tolerance of 1 mm. No configuration sets the tolerance.
• Empty cases. Where reference and prediction do not overlap at all, the configurations return Dice and NSD 0 and HD95 infinite (NaN for all three when both are empty), and the parser replaces infinity by 373.

The parser then reports different things for different tasks. Its seg command, which the scoring container uses for PED and GoAT, writes instance TP, FP, FN and F1 and the global Dice, NSD and HD95 of each region, and no lesion-wise segmentation score. Its mets command, used for METS with a volume threshold of 27 and an overlap threshold of 0.2 (brats-infra 4b9a380), splits reference lesions at the volume threshold: large ones get lesion-wise Dice, HD95 and NSD, with missed lesions and every false positive appended as Dice 0, NSD 0 and HD95 373; small ones are only counted for detection, and a matched lesion of either size whose Dice is below the overlap threshold counts as a miss. Both F1 scores use the total count of false positives. The METS evaluation page names subject-wise Dice and NSD and lesion-wise F1, and applies the segmentation metrics only to lesions larger than 27 mm³, the parser’s threshold; the parser’s test is volume >= vol_threshold, so a lesion of exactly 27 voxels counts as large in the code. The GoAT page ranks on Dice and NSD alone (BraTS 2026, METS and GoAT evaluation).

The sources disagree in three places, and the article reports both sides:

• The penalty. metrics_parser.py writes 373, with the comment “diameter of the cube in SRI space; using the maximum penalty instead of INF”. Tumor Segmentation gives 373.13 for the empty-target case, from Isensee et al. (2020).
• Missing cases. The METS evaluation page states that a case for which an algorithm produces no result is not penalized and “won’t be set to its worst possible value”. The parser’s _handle_missing_data fills every subject in its list of missing cases with 0 and, for HD95 columns, with 373.
• The package licence. BraTS_evaluation’s LICENSE file is the Apache License 2.0, which is also what GitHub reports, while its pyproject.toml declares license = "AGPL-3.0". Both allow reading the code.

Example: one case under the three eras

The script below scores one synthetic case, with exact ground truth, under all three eras. Real BraTS volumes cannot be used here (access requires registration, and the article uses no patient images), and panoptica and BraTS_evaluation cannot run without packages that are not installed (panoptica, cc3d, nibabel, SimpleITK). The script therefore re-types the functions it needs from both, under their Apache licence, and runs them on in-memory arrays; the lesion-wise arm is our own implementation of the 2023 README, so its outcome tests our reading, not the official code. It ran with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2 (one check) in about 2 seconds, and two runs gave identical output; nothing in it is random.

The case is a 96³ label map with 1 mm voxels in the 2023 convention, and only the ET region (label 3) is scored. One large lesion has a necrotic core, an enhancing rim of 2,556 voxels and edema around it; five small lesions are enhancing only. Three details were chosen so that the differences between the eras would show, which makes the example an illustration of how they differ rather than a test: S2 is two balls that touch only along an edge, so it is one lesion under 26-connectivity and two under 6-connectivity; S4 and S5 are four empty voxels apart, so dilation by 3 merges them and dilation by 1 does not; and S3 has 19 voxels, below both GLI’s 50-voxel threshold and METS’s 27 mm³. Three error models stand in for segmentation methods: A misses S1 and S3 and is otherwise perfect, B finds every lesion but erodes each by one voxel, and C is perfect but adds one false-positive lesion of 81 voxels. The reference scored against itself (“ref”) is the ground-truth check. In the 2023 arm, a ground-truth lesion’s prediction is the union of the predicted components that touch it, false positives are predicted components that touch no ground-truth voxel and have at least the threshold’s number of voxels, per-lesion HD95 is pooled as in panoptica, and missed and false lesions get either 373 or 0, both printed. Before the plan for this example was written, the script was run once to time it with its output discarded; it was then run unchanged.

# Three BraTS scoring schemes on one synthetic multifocal case (NumPy, SciPy).
# Parts of this file are ported (modified) from two projects under the Apache License 2.0:
#   BrainLesion/panoptica v2.1.7, "Copyright [2019] [Division of Medical Image Computing,
#   German Cancer Research Center (DKFZ), Heidelberg, Germany]"; licence:
#   https://github.com/BrainLesion/panoptica/blob/3cc264a470b5c02853aace0f30d6a9af21335abd/LICENSE
#   BraTS/BraTS_evaluation v0.0.8 (its LICENSE file fills in no copyright line); licence:
#   https://github.com/BraTS/BraTS_evaluation/blob/88e3e39cd5c4137b0831345c78d16bd393624c3a/LICENSE
# Ported: surface distances, HD95, NSD, maximum bipartite matching, rq, and the metrics
# parser's "seg" and "mets" summaries. Modified: rewritten for in-memory NumPy arrays,
# connected components by scipy.ndimage.label.
import os
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[_v] = "1"
import itertools
import sys

import numpy as np
from scipy import ndimage as ndi
from scipy.optimize import linear_sum_assignment

N = 96                       # 96^3 voxels, 1 mm isotropic
NETC, SNFH, ET = 1, 2, 3     # BraTS 2023+ labels; ET = label 3 (label 4 until 2021)
S26 = np.ones((3, 3, 3), bool)                   # 26-connectivity
S6 = ndi.generate_binary_structure(3, 1)         # 6-connectivity (scipy's default)
PENALTY = 373.0              # BraTS_evaluation's HD95 value for "inf"


def ball(c, r):
    z, y, x = np.ogrid[:N, :N, :N]
    return (z - c[0]) ** 2 + (y - c[1]) ** 2 + (x - c[2]) ** 2 <= r * r


def make_case():
    lab = np.zeros((N, N, N), np.uint8)
    c = (48, 48, 48)                      # large lesion: necrotic core, enhancing rim, edema
    lab[ball(c, 13)] = SNFH
    lab[ball(c, 9)] = ET
    lab[ball(c, 5)] = NETC
    small = {"S1": [((20, 20, 72), 3)],
             "S2": [((76, 24, 24), 2), ((79, 27, 24), 2)],   # two balls touching along an edge
             "S3": [((76, 72, 72), 1.5)],
             "S4": [((20, 72, 24), 2.5)],
             "S5": [((20, 72, 33), 2)]}                       # 4 empty voxels from S4
    masks = {"L0": (lab == ET).copy()}
    for name, parts in small.items():
        m = np.zeros((N, N, N), bool)
        for cc, r in parts:
            m |= ball(cc, r)
        lab[m] = ET
        masks[name] = m
    return lab, masks


def make_predictions(et, masks):
    a = et & ~masks["S1"] & ~masks["S3"]                 # A: misses two small lesions
    b = ndi.binary_erosion(et, S6)                        # B: every lesion 1 voxel too small
    c = et | ball((72, 72, 24), 2.5)                      # C: one false-positive lesion
    return {"ref": et.copy(), "A": a, "B": b, "C": c}


# ---------- surface distances (re-typed from panoptica _surface_distances.py) ----------
def paired_crop(a, b, pad=2):
    idx = np.argwhere(a | b)
    lo = np.maximum(idx.min(0) - pad, 0)
    hi = np.minimum(idx.max(0) + pad + 1, a.shape)
    return tuple(slice(l, h) for l, h in zip(lo, hi))


def surface_pair(ref, pred):
    sl = paired_crop(ref, pred)
    ref, pred = ref[sl], pred[sl]
    fp = ndi.generate_binary_structure(3, 1)
    pb = pred ^ ndi.binary_erosion(pred, structure=fp, iterations=1)
    rb = ref ^ ndi.binary_erosion(ref, structure=fp, iterations=1)
    sd_ref = ndi.distance_transform_edt(~pb)[rb]     # ref border -> nearest pred border
    sd_pred = ndi.distance_transform_edt(~rb)[pb]    # pred border -> nearest ref border
    return sd_ref, sd_pred


def hd95_pooled(sd_ref, sd_pred):                 # panoptica: one percentile of both sets
    return float(np.percentile(np.hstack((sd_ref, sd_pred)), 95))


def hd95_max(sd_ref, sd_pred):                    # larger of the two directed percentiles
    return float(max(np.percentile(sd_ref, 95), np.percentile(sd_pred, 95)))


def nsd(sd_ref, sd_pred, tau):                    # panoptica's NSD formula
    tp_a = np.sum(sd_ref <= tau) / len(sd_ref)
    tp_b = np.sum(sd_pred <= tau) / len(sd_pred)
    fp = np.sum(sd_ref > tau) / len(sd_ref)
    fn = np.sum(sd_pred > tau) / len(sd_pred)
    return float((tp_a + tp_b) / (tp_a + tp_b + fp + fn + sys.float_info.min))


def dice(a, b):
    return 2.0 * np.count_nonzero(a & b) / (np.count_nonzero(a) + np.count_nonzero(b))


# ---------- era 1: case-wise region scores (2017-2021 metrics) ----------
def legacy(ref, pred):
    sr, sp = surface_pair(ref, pred)
    return dice(ref, pred), hd95_pooled(sr, sp), hd95_max(sr, sp)


# ---------- era 2: our analogue of the 2023 lesion-wise README ----------
def lesionwise_2023(ref, pred, thr, dil):
    dlab, nd = ndi.label(ndi.binary_dilation(ref, S26, iterations=dil), S26)
    gts = [ref & (dlab == k) for k in range(1, nd + 1)]
    plab, npred = ndi.label(pred, S26)
    kept = [g for g in gts if g.sum() >= thr]
    d_sum = h_sum = 0.0
    tp = fn = 0
    for g in kept:
        ids = np.unique(plab[g]); ids = ids[ids > 0]
        if ids.size == 0:
            fn += 1
            continue
        p = np.isin(plab, ids)                    # all predicted components touching the lesion
        sr, sp = surface_pair(g, p)
        d_sum += dice(g, p); h_sum += hd95_pooled(sr, sp); tp += 1
    touching = set(np.unique(plab[ref]).tolist()) - {0}
    sizes = ndi.sum_labels(np.ones_like(plab), plab, range(1, npred + 1))
    fp = sum(1 for k in range(1, npred + 1) if k not in touching and sizes[k - 1] >= thr)
    n = tp + fn + fp
    return dict(gt=nd, kept=len(kept), tp=tp, fn=fn, fp=fp, dice=d_sum / n,
                hd_pen=(h_sum + PENALTY * (fn + fp)) / n, hd_lit=h_sum / n)


# ---------- era 3: panoptica pipeline as BraTS_evaluation configures it ----------
def match_max_bipartite(rlab, nr, plab, npr, thr=1e-6):
    cost = np.ones((nr, npr)) * (1.0 + 1e-6)
    pairs = set(zip(rlab[(rlab > 0) & (plab > 0)].tolist(), plab[(rlab > 0) & (plab > 0)].tolist()))
    for r, p in pairs:
        d = dice(rlab == r, plab == p)
        if d >= thr:
            cost[r - 1, p - 1] = 1.0 - d
    rows, cols = linear_sum_assignment(cost)
    return [(r + 1, c + 1) for r, c in zip(rows, cols) if cost[r, c] < 1.0], cost


def panoptica(ref, pred, struct):
    rlab, nr = ndi.label(ref, struct)
    plab, npr = ndi.label(pred, struct)
    matches, cost = match_max_bipartite(rlab, nr, plab, npr)
    tp = len(matches); fp = npr - tp; fn = nr - tp
    rq = (0.0 if nr + npr > 0 else np.nan) if tp == 0 else tp / (tp + 0.5 * fp + 0.5 * fn)
    inst = []
    matched = dict(matches)
    for r in range(1, nr + 1):
        g = rlab == r
        row = dict(volume=float(g.sum()), is_matched=int(r in matched))
        if r in matched:
            p = plab == matched[r]
            sr, sp = surface_pair(g, p)
            row.update(sq_dsc=dice(g, p), sq_hd95=hd95_pooled(sr, sp), sq_nsd=nsd(sr, sp, 1.0))
        inst.append(row)
    if np.count_nonzero(ref & pred) == 0:          # zero-TP edge case of config_gli/mets
        gd, gn, gh = 0.0, 0.0, np.inf
    else:
        sr, sp = surface_pair(ref, pred)          # global path: no voxelspacing -> tau 0.5
        gd, gn, gh = dice(ref, pred), nsd(sr, sp, 0.5), hd95_pooled(sr, sp)
    return dict(nr=nr, npr=npr, tp=tp, fp=fp, fn=fn, rq=rq, inst=inst,
                g_dsc=gd, g_nsd=gn, g_hd95=PENALTY if np.isinf(gh) else gh,
                matches=matches, cost=cost)


def parse_mets(res, vol_threshold=27.0, overlap_threshold=0.2):
    """metrics_parser.parse_mets_results for one region of one subject."""
    l_dsc, l_hd, l_nsd = [], [], []
    det_tp = det_fn = l_tp = l_fn = 0
    small_found = large_present = False
    for les in res["inst"]:
        large = les["volume"] >= vol_threshold
        if large:
            large_present = True
        else:
            small_found = True
        if les["is_matched"] == 1:
            d = les["sq_dsc"]
            if large:
                l_dsc.append(d); l_hd.append(les["sq_hd95"]); l_nsd.append(les["sq_nsd"])
                if d >= overlap_threshold:
                    l_tp += 1
                else:
                    l_fn += 1
            elif d >= overlap_threshold:
                det_tp += 1
            else:
                det_fn += 1
        elif large:
            l_fn += 1; l_dsc.append(0); l_nsd.append(0); l_hd.append(PENALTY)
        else:
            det_fn += 1
    nfp = res["fp"]
    if nfp > 0:
        l_dsc += [0] * nfp; l_nsd += [0] * nfp; l_hd += [PENALTY] * nfp
    den = 2 * l_tp + nfp + l_fn
    out = dict(large_f1=2 * l_tp / den if den > 0 else 0)
    if res["nr"] == 0 or not large_present:
        out.update(lw_dsc=np.nan, lw_hd95=np.nan, lw_nsd=np.nan)
    else:
        valid = [v for v in l_hd if v is not None and not np.isinf(v)]
        out.update(lw_dsc=float(np.mean(l_dsc)), lw_hd95=float(np.mean(valid)) if valid else PENALTY,
                   lw_nsd=float(np.mean(l_nsd)))
    if small_found:
        den = 2 * det_tp + nfp + det_fn
        out["small_f1"] = 2 * det_tp / den if den > 0 else 0
    else:
        out["small_f1"] = np.nan
    return out


# ================================== the case ==================================
lab, masks = make_case()
et = lab == ET                                   # BraTS "ET" region = label 3
preds = make_predictions(et, masks)
print("reference ET lesions (voxels):", {k: int(v.sum()) for k, v in masks.items()})
print("ET components: 26-connected", ndi.label(et, S26)[1], "| 6-connected", ndi.label(et, S6)[1])
for name, thr, dil in (("GLI", 50, 3), ("MET", 2, 1)):
    dlab, nd = ndi.label(ndi.binary_dilation(et, S26, iterations=dil), S26)
    vols = sorted(int((et & (dlab == k)).sum()) for k in range(1, nd + 1))
    print(f"2023 {name}: dilation {dil} -> {nd} lesions {vols}; scored (>= {thr} voxels): {sum(v >= thr for v in vols)}")

# ================================== scores ==================================
print("\n1) case-wise ET region (2017-2021 metrics)")
print("pred   Dice    HD95 pooled  HD95 max-directed")
leg = {}
for k, p in preds.items():
    leg[k] = legacy(et, p)
    print(f"{k:4s} {leg[k][0]:7.4f} {leg[k][1]:11.3f} {leg[k][2]:11.3f}")

print("\n2) lesion-wise, our reading of the 2023 README (FN and FP lesions score Dice 0)")
print("pred  setting  TP FN FP  LW-Dice  LW-HD95 (FN/FP = 373)  LW-HD95 (FN/FP = 0)")
for k, p in preds.items():
    for name, thr, dil in (("GLI", 50, 3), ("MET", 2, 1)):
        r = lesionwise_2023(et, p, thr, dil)
        print(f"{k:4s}  {name}     {r['tp']:2d} {r['fn']:2d} {r['fp']:2d}  {r['dice']:7.4f}  {r['hd_pen']:10.3f}  {r['hd_lit']:16.3f}")

print("\n3) panoptica pipeline (BraTS_evaluation v0.0.8 configs, panoptica 2.1.7)")
print("pred conn  TP FP FN   F1(rq)  | 'seg' parser: global DSC  NSD(tau 0.5)  HD95 | 'mets' parser: LW-DSC  LW-HD95  LW-NSD(tau 1)  large F1  small F1")
pan = {}
for k, p in preds.items():
    for cname, st in (("26", S26), ("6", S6)):
        r = panoptica(et, p, st); m = parse_mets(r)
        pan[k, cname] = r
        print(f"{k:4s} {cname:>4s}  {r['tp']:2d} {r['fp']:2d} {r['fn']:2d}  {r['rq']:7.4f}  |  {r['g_dsc']:21.4f} {r['g_nsd']:13.4f} {r['g_hd95']:7.3f} |"
              f" {m['lw_dsc']:21.4f} {m['lw_hd95']:8.3f} {m['lw_nsd']:13.4f} {m['large_f1']:9.4f} {m['small_f1']:9.4f}")

# ================================== checks ==================================
from scipy.spatial.distance import cdist
from sklearn.metrics import f1_score


def brute_surface(ref, pred):
    rb = np.argwhere(ref ^ ndi.binary_erosion(ref, S6)); pb = np.argwhere(pred ^ ndi.binary_erosion(pred, S6))
    D = cdist(rb, pb)
    return np.sort(D.min(1)), np.sort(D.min(0))


def brute_match(cost):                            # best total DSC over all one-to-one assignments
    nr, npr = cost.shape
    best = 0.0
    for perm in itertools.permutations(range(max(nr, npr)), min(nr, npr)):
        pairs = enumerate(perm) if nr <= npr else ((i, j) for j, i in enumerate(perm))
        best = max(best, sum(1 - cost[i, j] for i, j in pairs if cost[i, j] < 1.0))
    return best


c3 = c4 = c5 = c6 = 0.0
for k, p in preds.items():
    sr, sp = surface_pair(et, p); br, bp = brute_surface(et, p)
    c3 = max(c3, np.abs(np.sort(sr) - br).max(), np.abs(np.sort(sp) - bp).max())
    c4 = max(c4, abs(leg[k][0] - f1_score(et.ravel(), p.ravel())))
    for cname in ("26", "6"):
        r = pan[k, cname]
        c5 = max(c5, abs(sum(1 - r["cost"][a - 1, b - 1] for a, b in r["matches"]) - brute_match(r["cost"])))
        c6 = max(c6, abs(r["g_dsc"] - leg[k][0]), abs(r["g_hd95"] - leg[k][1]))
print(f"\nchecks: EDT vs brute-force distances {c3:.1e} | Dice vs sklearn F1 {c4:.1e} | "
      f"matching vs brute force {c5:.1e} | panoptica global vs case-wise {c6:.1e}")

reference ET lesions (voxels): {'L0': 2556, 'S1': 123, 'S2': 66, 'S3': 19, 'S4': 81, 'S5': 33}
ET components: 26-connected 6 | 6-connected 7
2023 GLI: dilation 3 -> 5 lesions [19, 66, 114, 123, 2556]; scored (>= 50 voxels): 4
2023 MET: dilation 1 -> 6 lesions [19, 33, 66, 81, 123, 2556]; scored (>= 2 voxels): 6

1) case-wise ET region (2017-2021 metrics)
pred   Dice    HD95 pooled  HD95 max-directed
ref   1.0000       0.000       0.000
A     0.9747       0.000      36.447
B     0.7013       1.000       1.414
C     0.9861       0.000       0.000

2) lesion-wise, our reading of the 2023 README (FN and FP lesions score Dice 0)
pred  setting  TP FN FP  LW-Dice  LW-HD95 (FN/FP = 373)  LW-HD95 (FN/FP = 0)
ref   GLI      4  0  0   1.0000       0.000             0.000
ref   MET      6  0  0   1.0000       0.000             0.000
A     GLI      3  1  0   0.7500      93.250             0.000
A     MET      4  2  0   0.6667     124.333             0.000
B     GLI      4  0  0   0.4909       1.311             1.311
B     MET      6  0  0   0.4090       1.276             1.276
C     GLI      4  0  1   0.8000      74.600             0.000
C     MET      6  0  1   0.8571      53.286             0.000

3) panoptica pipeline (BraTS_evaluation v0.0.8 configs, panoptica 2.1.7)
pred conn  TP FP FN   F1(rq)  | 'seg' parser: global DSC  NSD(tau 0.5)  HD95 | 'mets' parser: LW-DSC  LW-HD95  LW-NSD(tau 1)  large F1  small F1
ref    26   6  0  0   1.0000  |                 1.0000        1.0000   0.000 |                1.0000    0.000        1.0000    1.0000    1.0000
ref     6   7  0  0   1.0000  |                 1.0000        1.0000   0.000 |                1.0000    0.000        1.0000    1.0000    1.0000
A      26   4  0  2   0.8000  |                 0.9747        0.9592   0.000 |                0.8000   74.600        0.8000    0.8889    0.0000
A       6   5  0  2   0.8333  |                 0.9747        0.9592   0.000 |                0.8333   62.167        0.8333    0.9091    0.0000
B      26   6  1  0   0.9231  |                 0.7013        0.0000   1.000 |                0.3660   63.805        0.7180    0.8000    0.0000
B       6   7  0  0   1.0000  |                 0.7013        0.0000   1.000 |                0.4507    1.276        0.8879    1.0000    0.0000
C      26   6  1  0   0.9231  |                 0.9861        0.9804   0.000 |                0.8333   62.167        0.8333    0.9091    0.6667
C       6   7  1  0   0.9333  |                 0.9861        0.9804   0.000 |                0.8571   53.286        0.8571    0.9231    0.6667

checks: EDT vs brute-force distances 0.0e+00 | Dice vs sklearn F1 0.0e+00 | matching vs brute force 0.0e+00 | panoptica global vs case-wise 0.0e+00

The first lines confirm the construction: six 26-connected lesions but seven 6-connected ones, five lesions after GLI’s dilation by 3 (S4 and S5 merged into one of 114 voxels) of which four are scored (S3 is too small), and six lesions, all scored, after MET’s dilation by 1. Every “ref” row is perfect, and the four checks at the end are exact: the distance-transform surface distances equal brute-force pairwise distances, Dice equals scikit-learn’s F1 on the flattened masks, the assignment matches a brute-force search over all one-to-one matchings, and panoptica’s global Dice and HD95 equal the case-wise values, as they should, since they are the same quantities.

The plan for the example made eight predictions from the construction before any score of A, B or C was printed, and all eight held: case-wise Dice ranks C above A above B; for A the pooled HD95 is at most 1 mm and the larger directed one at least 10 mm; for C both case-wise HD95 values are 0; the lesion-wise Dice of A and C is 3/4 and 4/6, 4/5 and 6/7, and B, with no missed or false lesion in either setting, scores below A in both; the lesion-wise HD95 of A and C is 0 with a penalty of 0, and 373 times the number of missed and false lesions divided by TP + FN + FP with a penalty of 373; panoptica’s F1 for A, B and C is 0.8000, 0.9231 and 0.9231 under 26-connectivity and 0.8333, 1.0000 and 0.9333 under 6-connectivity, with one false positive for B only under 26-connectivity; the mets parser’s small-lesion F1 is 1, 0, 0 and 0.6667 for ref, A, B and C, with the predicted lesion-wise Dice, HD95 and large-lesion F1 for A and C; and B’s global NSD at tolerance 0.5 is at most 0.05 while its lesion-wise NSD at 1 mm is at least 0.5. Because three details of the case were chosen for these effects, the predictions confirm the construction rather than test the eras.

Case-wise scores hide small lesions. Missing two of six lesions costs A only 0.025 of Dice, because the missed voxels are 142 of 2,878; adding an 81-voxel lesion costs C even less. B, which found everything, scores worst (0.7013); in our reading, because a one-voxel erosion removes a large share of the enhancing rim, which has an inner and an outer surface. The two HD95 forms disagree for A by 36 mm: the missed lesions lie far from any predicted surface, and in our reading their surface voxels make up more than 5% of the reference surface but less than 5% of the pooled set of both surfaces, so the larger directed percentile sees them and the pooled one does not. For C, both forms print 0: case-wise HD95 does not see the false-positive lesion at all.

Lesion-wise scores (our reading of 2023) count lesions. A scores 0.7500 under GLI, where S3 is not scored and its miss costs nothing, but 0.6667 under MET, where it is. C loses one full lesion’s worth in both settings (4/5 and 6/7), for an error that the case-wise Dice barely registered. B has no missed and no false lesion, yet scores lowest (0.4909 and 0.4090), because each small lesion loses a large share of its voxels to the erosion; under MET it is lower still, in our reading because S3 and a separately scored S5, the smallest lesions, lose the largest shares. The HD95 column shows why the open penalty matters: if missed and false lesions add 0, as the README’s text read literally implies, A and C print the same lesion-wise HD95 as the reference, 0.000; with 373 they print 53 to 124 mm.

panoptica separates detection from segmentation. With 26-connectivity, A’s F1 is 0.8000 (two misses) and C’s 0.9231 (one false positive), and B’s is also 0.9231: its erosion splits the edge-connected S2 into two pieces, only one of which can be matched one to one, and the other becomes a false positive. With 6-connectivity S2 is two reference instances from the start and B’s F1 is 1.0000, so the connectivity a re-implementation chooses changes the detection score. The seg parser’s global Dice is the case-wise Dice, and its global NSD gives B 0.0000: every surface voxel of B lies one voxel from the reference surface, outside a tolerance of 0.5, while the mets parser’s lesion-wise NSD at 1 mm gives B 0.7180, a mean that includes an NSD of 0 for its false-positive fragment. In the mets columns, only S3 is a small lesion. B matches it, but the erosion leaves only S3’s centre voxel (a radius-1.5 ball is the centre with its 6 face and 12 edge neighbours, and only the centre has all six face neighbours inside), so its Dice of 2/20 = 0.1 is below the overlap threshold of 0.2 and the match counts as a miss: small F1 0. C finds S3 but still gets a small F1 of 0.6667, because the parser adds every false positive, whatever its size, to the small-lesion F1 as well. B’s lesion-wise HD95 under 26-connectivity, 63.805 against 1.276 under 6-connectivity, includes the 373 given to the fragment of S2 that counts as a false positive.

License

As of November 2026 the readable BraTS pages give terms of use but no named licence. The BraTS 2020 data page, under “Data Usage Agreement / Citations”, says users are “free to use and/or refer to the BraTS datasets in your own research”, provided they cite three papers: Menze et al. (2015), Bakas et al. (2017) and Bakas et al. (2018); where a journal allows data citations it asks for two TCIA citations as well, and states that using the data to create and submit benchmark results to MLPerf.org is considered non-commercial use (CBICA, BraTS 2020 data). Access goes through registration: in 2020 a registration and data request page, and in 2026 the GoAT data page states that data access “is temporarily not available to non-registered participants” (BraTS 2026, GoAT data). No licence text for the data of 2023 and later was found on a page readable without registering, and none was accepted for this article; terms that apply only after registration are unknown here. The evaluation code is open: panoptica is under the Apache License 2.0, and BraTS_evaluation’s licence metadata disagree, as described under Evaluation.

Limitations and Bias

• Scores from different eras are not comparable. A Dice from 2020, a lesion-wise Dice from 2023 and a METS lesion-wise Dice from 2026 measure different things on different label conventions, as the example shows on a single case.
• Image-based, changing definitions. The sub-regions are defined by appearance on particular MRI sequences, and the merging of non-enhancing tumor in 2017 and the change of label values since then changed the ground truth (Tumor Segmentation).
• Unspecified details. The repository that published the 2023 lesion-wise metrics has no licence file, and the Sage Bionetworks integration was not read here, so their HD95 penalty and thresholding rules cannot be checked from the README; the 2017–2021 evaluation code was not read here; and the NSD tolerances of the current pipeline follow from code defaults rather than from a stated choice (our reading).
• Reference quality and data. Single-annotator labels in 2021, retrospective multi-institutional data and the effects Tumor Segmentation lists, such as blood products and post-operative changes, limit what a score says about clinical use.

Common Uses

BraTS data are used to train and compare brain tumor segmentation methods and, through the challenges, to rank them; Tumor Segmentation and nnU-Net describe methods developed on it, and Neuroimaging Analysis its place among brain MRI benchmarks. The 2020 data page names a further use: benchmark results for MLPerf.org (CBICA, BraTS 2020 data). The panoptica pipeline is not specific to brain tumors; BraTS_evaluation only supplies its configurations.

Historical Milestones

• 2012–2013. The first two editions, at MICCAI in Nice and Nagoya; their manually annotated data were the only part of the early data kept after 2016 (CBICA, BraTS 2020 data). Medical Image Segmentation summarizes the 2015 report on them.
• 2014–2016. Boston, Munich and Athens. These editions used TCIA data with labels fused from algorithms, later discarded (CBICA BraTS pages; BraTS 2020 data).
• 2017. The rebuilt dataset with expert labels for pre-operative TCGA scans, the merged non-enhancing label, and sensitivity and specificity added to the metrics (CBICA, BraTS 2020 data and tasks). Tumor Segmentation names the 2017 and 2018 winners.
• 2020. Lima, held online: segmentation, survival and uncertainty tasks. The segmentation task was won by the MIC_DKFZ team (Isensee et al.), whose entry was based on nnU-Net (Tumor Segmentation), followed by NPU_PITT and Radicals, tied for second (CBICA, BraTS 2020 rankings); Tumor Segmentation gives its test-set Dice scores.
• 2021. The RSNA-ASNR-MICCAI edition, with 2,040 cases and a containerized test (Tumor Segmentation).
• 2022–2023. A continuous evaluation (2022, Singapore) and a cluster of challenges (2023, Vancouver), with lesion-wise Dice and HD95 (CBICA BraTS pages; 2023 metrics README).
• 2026. The fifteenth cluster, renamed Brain TumorS (BraTS 2026 overview); Sage Bionetworks’ scoring code for its MET, PED and GoAT cohorts runs panoptica through BraTS_evaluation 0.0.8 (brats-infra 4b9a380).

Related Datasets

The brain task of the Medical Segmentation Decathlon reuses BraTS data (Tumor Segmentation). LiTS and KiTS play the same role for liver and kidney tumors, and Tumor Segmentation compares their protocols with BraTS’s. The TCGA-GBM and TCGA-LGG collections on TCIA are the source of the scans added in 2017, and the 2020 page gives the mapping between BraTS and TCGA file names (CBICA, BraTS 2020 data).

Resources

• CBICA’s BraTS pages, with the editions up to 2023 and links to their proceedings: www.med.upenn.edu/cbica/brats/.
• The 2026 cluster on Synapse: BraTS 2026, the place to look for current results.
• The evaluation package: BraTS/BraTS_evaluation, and the library it configures, BrainLesion/panoptica.
• The 2023 lesion-wise metrics README: rachitsaluja/BraTS-2023-Metrics.