Evaluation & Metrics
Dice Coefficient
Twice the overlap of a predicted and a ground-truth mask divided by their total size, equal to the F1 score over pixels; the standard overlap score of medical image segmentation and, in soft form, a training loss.
beginner
The Dice coefficient, also called the Dice similarity coefficient (DSC) or the Sørensen–Dice coefficient, measures how well a predicted segmentation mask matches the ground truth: twice the number of pixels they share, divided by the total number of pixels in the two masks. It is 1 for a perfect match and 0 when the masks do not overlap. Over pixels it is exactly the F1 score, it is a monotonic function of intersection over union, and it is the usual headline score in medical image segmentation. Written with predicted probabilities in place of a binary mask, it also serves as a training loss.
Formula
For a predicted mask and a ground-truth mask , with counting pixels (or voxels),
where TP, FP and FN count the pixels that are correctly labeled foreground, wrongly labeled foreground, and missed. The second form is the F1 score of precision and recall computed over pixels, . True negatives, the background pixels both masks leave out, do not appear, so enlarging the image does not change the score.
The IoU article derives the conversion between the two overlap scores, with the IoU (Jaccard index):
History. The quotient comes from ecology. Dice (1945) proposed the index for measuring ecological association between species, and the Danish botanist Thorvald Sørensen published the same quotient in 1948 to compare plant communities by their species content, hence the double name. In medical image analysis, Zijdenbos et al. (1994) compared semiautomatic and manual segmentations of white matter lesions in MR images with a similarity measure derived from the kappa statistic, which later segmentation papers commonly cite as the Dice coefficient.
Intuition
Dice compares the overlap with the average size of the two masks: divided by . Like IoU, it penalizes both misses and extra pixels. Because the overlap is counted twice, Dice is never below IoU; the two agree only at 0 and 1. A prediction that covers half of the object and nothing else scores against .
Interpretation
Dice lies in , is symmetric, and, as F1, weights FP and FN equally. Since is increasing, Dice and IoU rank predictions for one mask identically. Averages differ: mean Dice is not , and two methods can swap places depending on which score is averaged, as in the example. In the Metrics Reloaded recommendations, the choice between Dice and IoU is described as typically a matter of user or community preference (Maier-Hein et al., 2024).
What counts as a good value depends on the structure: a Dice of 0.9 may be routine for a large organ and out of reach for a small lesion (see Failure Cases).
When to Use It
- Medical image segmentation. Dice is the standard overlap score for organs, tumors and lesions. The Medical Segmentation Decathlon, which evaluated 19 teams on ten 3D segmentation tasks, ranked them by Dice together with the normalized surface distance (NSD), a boundary-based metric (Antonelli et al., 2022).
- Semantic segmentation in general, as the per-class overlap score, though general computer vision benchmarks usually report mean IoU.
- Choosing a metric. Metrics Reloaded recommends an overlap-based metric, with Dice or IoU as the default, unless the target structures are consistently small relative to the pixel grid and the reference may be noisy, and recommends complementing it with a boundary-based metric (Maier-Hein et al., 2024).
Aggregation. On a test set, Dice can be computed per image and averaged (mean Dice), or from TP, FP and FN pooled over all pixels of all images (global Dice). Pooling lets large structures hide errors on small ones. Metrics Reloaded recommends computing metrics per image first and then aggregating, respecting the hierarchical structure of the data, such as several images per patient (Maier-Hein et al., 2024). Reinke et al. (2024) noted that the TorchMetrics library computed Dice by default as a global average over all pixels. For classes, per-class Dice scores are averaged into a mean Dice, as for mean IoU.
Empty masks. When both masks are empty, . Implementations return 1 (nothing was there, nothing was predicted), return NaN and skip the image, or add a small to numerator and denominator, which also gives 1; Reinke et al. (2024) list empty references and predictions among the pitfalls of metric computation. For the per-image validation of object-level problems such as object detection, Metrics Reloaded recommends excluding undefined cases, except when an empty prediction matches an empty reference, where precision and the F score are set to 1 (Maier-Hein et al., 2024, arXiv version, Supplementary Note 2.4). A predicted lesion in an image with no lesion scores 0, or nearly 0 with , under every convention.
Failure Cases
- Small structures. One pixel of boundary error removes a share of the mask that grows as the mask shrinks. In the example, a 1-pixel dilation costs a 3,268-pixel organ 0.029 Dice and a 52-pixel lesion 0.188. Reinke et al. (2024) illustrate how single-pixel differences can change Dice greatly for small structures, and the Decathlon authors cautioned against relying heavily on Dice for very small objects (Antonelli et al., 2022).
- Boundary and shape errors. Dice counts pixels, not where they are: an error far from the object costs no more than one next to it, and a missed thin spur scores like a compact error of equal size. Boundary-based metrics, such as the Hausdorff distance, the normalized surface distance or Boundary IoU, measure this; Metrics Reloaded lists shape unawareness among the key weaknesses of overlap metrics (Maier-Hein et al., 2024).
- Object count. On a semantic mask, Dice is dominated by the largest object: missing three small lesions while segmenting one large lesion well still gives a high score. Metrics Reloaded illustrates how strongly Dice is biased toward single objects (Maier-Hein et al., 2024, Fig. 1); counting lesions is a detection problem, scored with matched objects.
Variants
Soft Dice loss. Hard Dice gives no useful gradient, so Milletari, Navab and Ahmadi (2016) used softmax foreground probabilities and maximized
over the voxels of prostate MRI volumes with binary ground truth ; the loss is . On binary inputs , so it equals hard Dice; a common variant uses the unsquared denominator . They motivated it by class imbalance: when the anatomy fills a small part of the scan, training tends to favor background, and the Dice loss needs no re-weighting of samples. On the PROMISE 2012 test set they reported an average Dice of 0.869 against 0.739 for the same network trained with a multinomial logistic loss with sample re-weighting (arXiv version, Table 2). Our reading of the gradient: both of its terms are of order , so a small structure gets large per-pixel gradients. Among the Decathlon finalists that described their methods in full, a Dice loss was the most common choice (29%), followed by cross-entropy (21%), and two of the 14 used their sum (Antonelli et al., 2022).
Generalized Dice loss. For several classes, Sudre et al. (2017) proposed as a loss the generalized Dice score of Crum, Camara and Hill (2006),
with the reference and the predicted probability of label at pixel . These weights correct each label’s contribution by the inverse of its volume. In their 2D and 3D comparisons, overlap-based losses were more robust as imbalance grew, and the generalized Dice loss was the most reliable across learning rates and sampling strategies.
Other variants. The score over pixels generalizes Dice () to unequal costs of FP and FN, and the centerline Dice (clDice) scores tubular structures such as vessels; Metrics Reloaded recommends both for those cases (Maier-Hein et al., 2024).
Example
The NumPy code applies the same boundary errors to a large ellipse (an organ) and a small disk (a lesion), and scores two methods on a two-image test set.
import numpy as np
from scipy import ndimage as ndi
H = W = 128
yy, xx = np.mgrid[0:H, 0:W] + 0.5 # pixel centers
def ellipse(cx, cy, a, b):
return ((xx - cx) / a) ** 2 + ((yy - cy) / b) ** 2 <= 1.0
def dice(p, g):
return 2 * (p & g).sum() / (p.sum() + g.sum())
def iou(p, g):
return (p & g).sum() / (p | g).sum()
organ = ellipse(64, 64, 40, 26) # large structure
lesion = ellipse(64, 64, 4, 4) # small disk, radius 4 px
cross = ndi.generate_binary_structure(2, 1) # 4-neighborhood
errors = {
"shift 1 px": lambda g: np.roll(g, 1, axis=1),
"shift 2 px": lambda g: np.roll(g, 2, axis=1),
"dilate 1 px": lambda g: ndi.binary_dilation(g, cross),
"erode 1 px": lambda g: ndi.binary_erosion(g, cross),
}
# 1. The same boundary error on a large and a small structure.
print(f"organ {organ.sum()} px, lesion {lesion.sum()} px")
print("error organ Dice IoU lesion Dice IoU")
gap = 0.0
for name, f in errors.items():
po, pl = f(organ), f(lesion)
print(f"{name:13s} {dice(po, organ):10.3f} {iou(po, organ):.3f} "
f"{dice(pl, lesion):11.3f} {iou(pl, lesion):.3f}")
for p, g in [(po, organ), (pl, lesion)]:
J = iou(p, g)
gap = max(gap, abs(dice(p, g) - 2 * J / (1 + J)))
print(f"largest |Dice - 2J/(1+J)| over the 8 cases: {gap:.1e}")
# 2. Two methods on a two-image test set (image 1: organ, image 2: lesion).
methods = {
"A": [ndi.binary_dilation(organ, cross, iterations=2), ndi.binary_erosion(lesion, cross)],
"B": [organ.copy(), np.roll(lesion, 2, axis=1)],
}
gts = [organ, lesion]
print("\nmethod per-image Dice mean Dice mean IoU 2mIoU/(1+mIoU) global Dice")
for m, preds in methods.items():
D = [dice(p, g) for p, g in zip(preds, gts)]
J = [iou(p, g) for p, g in zip(preds, gts)]
tp = sum((p & g).sum() for p, g in zip(preds, gts))
total = sum(p.sum() + g.sum() for p, g in zip(preds, gts))
mJ = np.mean(J)
print(f"{m:6s} {D[0]:.3f}, {D[1]:.3f} {np.mean(D):9.3f} {mJ:8.3f} "
f"{2 * mJ / (1 + mJ):14.3f} {2 * tp / total:11.3f}")
Output:
organ 3268 px, lesion 52 px
error organ Dice IoU lesion Dice IoU
shift 1 px 0.984 0.969 0.846 0.733
shift 2 px 0.968 0.938 0.692 0.529
dilate 1 px 0.971 0.945 0.812 0.684
erode 1 px 0.970 0.942 0.762 0.615
largest |Dice - 2J/(1+J)| over the 8 cases: 1.1e-16
method per-image Dice mean Dice mean IoU 2mIoU/(1+mIoU) global Dice
A 0.944, 0.762 0.853 0.755 0.860 0.942
B 1.000, 0.692 0.846 0.765 0.867 0.995
A 1-pixel shift leaves the organ at 0.984 but drops the lesion to 0.846, and the identity holds for every mask. On the test set, mean Dice ranks method A first (0.853 against 0.846), while mean IoU ranks method B first (0.765 against 0.755): B’s perfect organ counts for more under IoU, since is convex and so stretches high scores apart. Global Dice is dominated by the organ’s 3,268 pixels, gives B 0.995, and hides that B segments the lesion worse.
The second script evaluates the soft Dice loss on blurry probability maps of masks shifted by one pixel, checks its gradient by finite differences, and compares gradient sizes with mean binary cross-entropy.
import numpy as np
from scipy import ndimage as ndi
H = W = 128
yy, xx = np.mgrid[0:H, 0:W] + 0.5
def disk_like(cx, cy, a, b):
return ((xx - cx) / a) ** 2 + ((yy - cy) / b) ** 2 <= 1.0
def soft_prob(mask, shift=1, width=1.0):
"""A blurry probability map: sigmoid of the signed distance to a shifted mask."""
m = np.roll(mask, shift, axis=1)
sd = ndi.distance_transform_edt(m) - ndi.distance_transform_edt(~m)
return 1 / (1 + np.exp(-sd / width))
def soft_dice_loss(p, g):
"""1 - D with the squared denominator of Milletari et al. (2016)."""
return 1 - 2 * (p * g).sum() / ((p ** 2).sum() + (g ** 2).sum())
def soft_dice_grad(p, g):
I, S = (p * g).sum(), (p ** 2).sum() + (g ** 2).sum()
return -2 * (g * S - 2 * p * I) / S ** 2
organ = disk_like(64, 64, 40, 26).astype(float)
lesion = disk_like(64, 64, 4, 4).astype(float)
# 1. Check the analytic gradient against central finite differences.
p, g = soft_prob(organ > 0), organ
grad = soft_dice_grad(p, g)
h, worst = 1e-5, 0.0
for (i, j) in [(64, 24), (64, 25), (64, 104), (38, 64), (64, 64), (64, 110)]:
e = np.zeros_like(p); e[i, j] = h
fd = (soft_dice_loss(p + e, g) - soft_dice_loss(p - e, g)) / (2 * h)
worst = max(worst, abs(fd - grad[i, j]) / max(abs(fd), 1e-12))
print(f"soft Dice loss {soft_dice_loss(p, g):.4f}, "
f"largest relative gradient error {worst:.1e}")
# 2. Binary input: the soft loss equals 1 - hard Dice (p**2 = p for 0/1 values).
pb = (p > 0.5).astype(float)
hard = 2 * (pb * g).sum() / (pb.sum() + g.sum())
print(f"binary p: soft loss {soft_dice_loss(pb, g):.4f}, 1 - hard Dice {1 - hard:.4f}")
# 3. Gradient size per foreground pixel: Dice vs mean binary cross-entropy.
print("structure fg px Dice loss mean|dL/dp| fg BCE mean|dL/dp| fg")
for name, m in [("organ", organ), ("lesion", lesion)]:
p = soft_prob(m > 0)
fg = m > 0
g_dice = np.abs(soft_dice_grad(p, m))[fg].mean()
g_bce = (np.abs(p - m) / (p * (1 - p)) / m.size)[fg].mean() # d/dp of mean BCE
print(f"{name:9s} {int(m.sum()):5d} {soft_dice_loss(p, m):9.3f} {g_dice:15.2e}"
f" {g_bce:19.2e}")
Output:
soft Dice loss 0.0136, largest relative gradient error 6.4e-06
binary p: soft loss 0.0159, 1 - hard Dice 0.0159
structure fg px Dice loss mean|dL/dp| fg BCE mean|dL/dp| fg
organ 3268 0.014 1.60e-05 6.56e-05
lesion 52 0.132 7.50e-03 9.65e-05
The analytic gradient agrees with finite differences to rounding error. The same 1-pixel shift gives the lesion a soft Dice loss almost ten times the organ’s, and per foreground pixel a Dice gradient about 470 times larger, while the per-pixel cross-entropy gradient barely depends on structure size. That is the rebalancing the Dice loss provides; by the same token, a few uncertain pixels on a small structure produce large gradients.
In PyTorch, autograd differentiates the loss through the sigmoid; the snippet below, run with PyTorch 2.14.1, checks autograd against the formula above times the sigmoid derivative .
import torch
torch.manual_seed(0)
H = W = 64
yy, xx = torch.meshgrid(torch.arange(H) + 0.5, torch.arange(W) + 0.5, indexing="ij")
target = (((xx - 30) / 12) ** 2 + ((yy - 34) / 8) ** 2 <= 1).float() # ground-truth mask
# Logits of a poor initial prediction: a slightly offset, blurry blob plus noise.
logits = (6 - 0.4 * torch.sqrt((xx - 33) ** 2 + (yy - 32) ** 2)
+ 0.5 * torch.randn(H, W)).requires_grad_()
def soft_dice_loss(logits, target, eps=1e-6):
p = torch.sigmoid(logits)
inter = (p * target).sum()
return 1 - 2 * inter / ((p ** 2).sum() + (target ** 2).sum() + eps)
loss = soft_dice_loss(logits, target)
loss.backward()
# Same gradient by hand: dL/dp from the V-Net formula, times sigmoid'(z) = p (1 - p).
with torch.no_grad():
p = torch.sigmoid(logits)
I, S = (p * target).sum(), (p ** 2).sum() + (target ** 2).sum() + 1e-6
manual = -2 * (target * S - 2 * p * I) / S ** 2 * p * (1 - p)
print(f"torch {torch.__version__}: loss {loss.item():.4f}, "
f"max |autograd - manual| {(logits.grad - manual).abs().max().item():.1e}")
Output:
torch 2.14.1: loss 0.3493, max |autograd - manual| 8.7e-11
Related Metrics
Intersection over union determines Dice exactly for a single mask, and mean IoU is the usual semantic segmentation benchmark score. Dice is the pixel-level F1 of precision and recall. Pixel accuracy counts true negatives and is dominated by background. Boundary-based metrics, such as the Hausdorff distance and the normalized surface distance, complement Dice where contours matter.
Related
- Precision and Recall
Two complementary measures of a classifier or detector, the fraction of its positive predictions that are correct and the fraction of true positives it finds, with the F-score, the confusion matrix, and the precision–recall and ROC curves built from them.
- Semantic Segmentation
Labeling every pixel of an image with one of a fixed set of classes, such as road, sky, car or person, without separating individual objects; the dense-prediction counterpart of image classification, also called scene parsing.
- Image Segmentation
Dividing an image into regions by assigning every pixel to a segment, a class, or an object instance, the task behind semantic, instance, panoptic and interactive segmentation.
- Loss Function
The function that scores how far a network's output is from the desired output on one example; training minimizes its average over the training images, and the choice of loss encodes what counts as a good prediction for each vision task.
- Cross-Entropy Loss
The loss that scores a predicted class distribution by the negative log-probability it gives the true label; it is the standard loss for image classification and per-pixel segmentation, the negative log-likelihood of a softmax or sigmoid output, and differs from the KL divergence by a constant.
- Accuracy
The fraction of test examples a classifier labels correctly, with its top-k, balanced, and per-pixel variants, and why it can mislead on imbalanced data or small test sets.
References
- Dice, L. R. (1945). Measures of the Amount of Ecologic Association Between Species. Ecology, 26(3), 297–302.
- Zijdenbos, A. P., Dawant, B. M., Margolin, R. A. & Palmer, A. C. (1994). Morphometric Analysis of White Matter Lesions in MR Images: Method and Validation. IEEE Transactions on Medical Imaging, 13(4), 716–724.
- Milletari, F., Navab, N. & Ahmadi, S.-A. (2016). V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. Fourth International Conference on 3D Vision (3DV), 565–571.
- Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S. & Cardoso, M. J. (2017). Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (DLMIA/ML-CDS), Lecture Notes in Computer Science, 240–248.
- Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B. A., Litjens, G., Menze, B., Ronneberger, O., Summers, R. M., et al. (2022). The Medical Segmentation Decathlon. Nature Communications, 13, 4128.
- Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al. (2024). Metrics Reloaded: Recommendations for Image Analysis Validation. Nature Methods, 21(2), 195–212.
- Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., et al. (2024). Understanding Metric-Related Pitfalls in Image Analysis Validation. Nature Methods, 21(2), 182–194.