Cell Segmentation

Cell segmentation finds every cell, or every cell nucleus, in a microscopy or histology image and gives each its own mask. It is instance segmentation with one or a few classes and many instances: a single fluorescence image can hold hundreds of nuclei, a histology tile thousands, and most of the difficulty lies in separating cells that touch. Nucleus segmentation (also called nuclei segmentation or nuclear segmentation) is the most common form, because the nucleus is a reliable reference for identifying single cells; Caicedo et al. (2019) describe it as often the first step of quantitative image analysis, before counting cells, tracking them, measuring proteins or classifying phenotypes. Whole-cell segmentation outlines the cytoplasm or membrane as well.

Problem Definition

Given an image of cells, predict a label image $L$ in which each pixel carries 0 for background or the identity $k \in \{1, \dots, N\}$ of the cell it belongs to, with $N$ unknown in advance. Many datasets and methods assume that cells do not overlap in the image plane, so a label image (one label per pixel) is enough; this is the panoptic-style partition, without the ranked, possibly overlapping masks of general instance segmentation. Cellpose’s authors note that their method is not expected to work where occlusions require highly overlapping masks (Stringer et al., bioRxiv version). In 3D stacks the label image becomes a label volume.

Unlike natural-image instance segmentation, there is usually a single class: the score depends on separating and delineating instances, not on naming them. It also differs from medical image segmentation of organs and lesions in CT or MRI, where a few large structures are delineated voxel by voxel and the task is usually semantic. Histology methods such as HoVer-Net add a type per nucleus (Graham et al., 2019).

Input and Output

Inputs vary with the imaging:

• Fluorescence microscopy of nuclei stained with DNA dyes such as DAPI or Hoechst, the majority of the 2018 Data Science Bowl images (Caicedo et al., 2019); whole-cell work adds a cytoplasmic or membrane channel, and Cellpose takes the nuclear channel as an optional second input (Stringer et al., bioRxiv version).
• Brightfield images of stained tissue, such as hematoxylin and eosin (H&E) histology, where nuclei are crowded and irregular.
• Phase contrast and differential interference contrast images of unstained cells, and time-lapse sequences in 2D and 3D, as in the Cell Tracking Challenge (Maška et al., 2023).

The output is a label image, sometimes with a class per instance, or a list of polygons that is rasterized into one.

Variants

• Nucleus versus whole-cell segmentation: whole cells have more varied shapes, with protrusions that defeat roundness assumptions.
• Cell detection predicts one point or marker per cell; the Cell Tracking Challenge scores detection separately from segmentation.
• Segmentation for tracking links masks over time into lineages; most Cell Tracking Challenge tracking methods link a prior per-frame segmentation (Maška et al., 2023).
• Joint segmentation and classification of nuclei by cell type in histology (Graham et al., 2019).

Challenges

• Touching and overlapping cells. A pixel classifier sees two touching nuclei as one foreground region, and a few misclassified pixels between them fuse two instances (Schmidt et al., 2018). In the example below, merges are almost the only error of thresholding followed by connected components.
• Uneven illumination and staining. Brightness varies across the field and between cells, so a single global threshold misses dim cells or fuses bright clusters with a bright background.
• Variation between experiments. Classical pipelines must be configured for each modality, scale and condition, and no single parameter choice segments every image correctly (Caicedo et al., 2019). This was the motivation for the Data Science Bowl and for “generalist” models such as Cellpose.
• Annotation cost and disagreement. Annotations are made by hand, object by object. Caicedo et al. call their annotations “target masks” rather than ground truth because each was drawn by one expert and reviewed by others, and a second annotator showed inter-observer variation; the Cell Tracking Challenge’s segmentation gold truth, a majority opinion of three experts, covered 17.8% of cell instances on average (Maška et al., 2023). Stringer et al. (bioRxiv version) estimate that at their median cell diameter of 30 pixels, a cell takes about 30 times longer to annotate in 3D than in one 2D slice.
• Focus, density and size. Out-of-focus signal makes nuclei appear densely packed and partly overlapping (Schmidt et al., 2018), and object size in pixels changes with magnification; Cellpose, for example, rescales images to a common cell size, estimated from the image when not given (Stringer et al., bioRxiv version).

Approaches

Classical pipelines

The classical recipe has three steps: separate foreground from background by thresholding, clean the mask with mathematical morphology (hole filling, opening, removal of small objects), and split touching cells with a marker-controlled watershed, usually flooding the negated distance transform of the mask from its local maxima, or the inverted intensity. The reference segmentations of the 2018 Data Science Bowl were built this way in CellProfiler 3.1.5: one pipeline per image group, with preprocessing to a grayscale image of bright nuclei, segmentation by thresholding, distance transforms and watershed based on the expected nuclear size, and a seeded watershed to revise the result (Caicedo et al., 2019). In the paper’s supplementary note, the authors cite the Cell Tracking Challenge’s finding that 66% of its solutions, including the top ones, were based on thresholding.

Graham et al. (2019) survey the classical variants for nuclei: watershed markers from thresholding or active contours, or an energy landscape from morphological operations; level-set shape models; and splitting clusters at concavities or by ellipse fitting. Each assumes a predefined nuclear geometry and is sensitive to manually chosen parameters. The watershed article explains the flooding itself and why distance-transform splitting assumes roughly convex objects.

Example: thresholding, watershed and instance matching

The NumPy, SciPy and scikit-image code (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2) draws five $256 \times 256$ images of elliptical nuclei with known labels. Each image has 16 clusters of one to four nuclei, placed so that neighbours touch or overlap slightly (centre distances 0.8 to 0.95 times the sum of their mean semi-axes; overlapping pixels go to the nearer nucleus). Brightness varies per nucleus (0.6 to 1.0), illumination falls from 1.0 on the left to 0.4 on the right, an additive haze brightens the top left, and the image is blurred ($\sigma = 1$ pixel) with Gaussian noise ($\sigma = 0.04$). Two foreground masks, a global Otsu threshold and a local threshold (scikit-image’s threshold_local with a 51-pixel window), are each turned into instances by connected components and by a distance-transform watershed.

Scoring matches predicted and true objects one-to-one, as StarDist’s evaluation code does: the assignment maximizes the number of pairs with IoU of at least $t$, with IoU breaking ties. The script reports $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP}+\mathrm{FN})$, which the cell literature calls “average precision” (see Evaluation Metrics), at $t = 0.5, 0.75, 0.9$ and averaged over $t = 0.50, 0.55, \dots, 0.95$, with counts pooled over the five images. It also counts three errors by our own rule: a nucleus is missed if no object holds more than half of it, merged if the object holding most of it also holds most of another nucleus, and split if two or more objects lie mostly inside it.

import sys

import numpy as np
from scipy import ndimage as ndi
from scipy.optimize import linear_sum_assignment
from skimage.feature import peak_local_max
from skimage.filters import threshold_local, threshold_otsu
from skimage.segmentation import watershed

H = W = 256

def synth(seed):
    """Elliptical nuclei, many in touching clusters, under uneven illumination."""
    rng = np.random.default_rng(seed)
    yy, xx = np.mgrid[:H, :W] + 0.5
    nuclei = []                                             # (row, col, a, b, angle)
    for gy in range(4):                                     # one cluster per 64 x 64 cell
        for gx in range(4):
            c = (np.array([gy, gx]) + 0.5) * 64 + rng.uniform(-4, 4, 2)
            n = rng.choice([1, 2, 3, 4], p=[0.2, 0.35, 0.3, 0.15])
            group = [(*c, rng.uniform(7, 10), rng.uniform(5.5, 8), rng.uniform(0, np.pi))]
            for _ in range(100):                            # add touching neighbours
                if len(group) == n:
                    break
                a, b = rng.uniform(7, 10), rng.uniform(5.5, 8)
                q = group[rng.integers(len(group))]
                phi = rng.uniform(0, 2 * np.pi)
                r = rng.uniform(0.8, 0.95) * ((q[2] + q[3]) / 2 + (a + b) / 2)
                p = np.array(q[:2]) + r * np.array([np.sin(phi), np.cos(phi)])
                if all(np.hypot(*(p - np.array(o[:2]))) > 0.8 * ((o[2] + o[3]) / 2 + (a + b) / 2)
                       for o in group) and np.abs(p - c).max() < 20:
                    group.append((*p, a, b, rng.uniform(0, np.pi)))
            nuclei += group
    # Each pixel belongs to the nucleus whose elliptical distance is smallest (< 1).
    dist = np.full((H, W), np.inf)
    gt = np.zeros((H, W), int)
    for i, (cy, cx, a, b, th) in enumerate(nuclei, 1):
        u = (xx - cx) * np.cos(th) + (yy - cy) * np.sin(th)
        v = -(xx - cx) * np.sin(th) + (yy - cy) * np.cos(th)
        d = np.hypot(u / a, v / b)
        win = (d < 1) & (d < dist)
        gt[win], dist[win] = i, d[win]
    gt = np.unique(gt, return_inverse=True)[1].reshape(H, W)   # relabel 1..N
    stain = rng.uniform(0.6, 1.0, gt.max() + 1)             # per-nucleus brightness
    stain[0] = 0.0
    light = 1.0 - 0.6 * xx / W                              # illumination: 1.0 -> 0.4
    glow = 0.3 * np.exp(-((yy - 40) ** 2 + (xx - 60) ** 2) / (2 * 70 ** 2))  # background haze
    img = ndi.gaussian_filter(light * (0.1 + 0.9 * stain[gt]) + glow, 1.0)
    return gt, img + rng.normal(0, 0.04, img.shape)

def overlap(gt, pred):
    """Pixel counts of every (true nucleus, predicted object) pair, background dropped."""
    inter = np.zeros((gt.max() + 1, pred.max() + 1))
    np.add.at(inter, (gt.ravel(), pred.ravel()), 1)
    return inter

def match(inter, t):
    """One-to-one matching maximizing the number of pairs with IoU >= t (as in StarDist)."""
    iou = (inter / (inter.sum(1, keepdims=True) + inter.sum(0, keepdims=True) - inter))[1:, 1:]
    n_g, n_p = iou.shape
    r, c = linear_sum_assignment(-(iou >= t).astype(float) - iou / (2 * min(n_g, n_p)))  # IoU breaks ties
    ok = iou[r, c] >= t                                     # pairs below t do not count
    size = inter[:, 1:].sum(0)[c] / inter[1:, :].sum(1)[r]  # predicted area / true area
    return (int(ok.sum()), n_p - int(ok.sum()), n_g - int(ok.sum())), iou[r, c][ok], size[ok]

def error_masks(inter):
    """Our counts. Missed: no object holds > 50% of the nucleus. Merged: the object
    holding > 50% of it holds > 50% of another nucleus too. Split: > 50% of two or
    more objects lies inside the nucleus."""
    of_gt = inter[1:, 1:] > 0.5 * inter[1:, :].sum(1, keepdims=True)
    of_pred = inter[1:, 1:] > 0.5 * inter[:, 1:].sum(0, keepdims=True)
    shared = of_gt.sum(0) >= 2
    return ~of_gt.any(1), of_gt[:, shared].any(1), of_pred.sum(1) >= 2   # per true nucleus

def components(mask, min_size=30):
    lab, _ = ndi.label(mask)
    sizes = np.bincount(lab.ravel())
    lab[sizes[lab] < min_size] = 0
    return np.unique(lab, return_inverse=True)[1].reshape(mask.shape)

def split_touching(mask):
    """Markers at distance-transform maxima, then flood the negated distance."""
    dt = ndi.distance_transform_edt(mask)
    peaks = peak_local_max(dt, min_distance=5, threshold_abs=3, exclude_border=False,
                           labels=components(mask))
    markers = np.zeros(mask.shape, int)
    markers[tuple(peaks.T)] = np.arange(1, len(peaks) + 1)
    return watershed(-dt, markers, mask=mask)

def segment(img):
    sm = ndi.gaussian_filter(img, 1.0)
    fg = {"global Otsu": sm > threshold_otsu(sm),                      # one threshold
          "local threshold": sm > threshold_local(sm, 51, offset=-0.02)}  # per pixel
    out = {}
    for name, mask in fg.items():
        mask = ndi.binary_opening(ndi.binary_fill_holes(mask))
        out[name + " + components"] = components(mask)
        out[name + " + distance watershed"] = split_touching(mask)
    return out

thresholds = np.round(np.arange(0.5, 1.0, 0.05), 2)       # 0.50, 0.55, ..., 0.95
first = int(sys.argv[1]) if len(sys.argv) > 1 else 0
seeds = range(first, first + 5)
tot, pairs, where, n_true = {}, {}, {}, 0
for seed in seeds:
    gt, img = synth(seed)
    n_true += gt.max()
    idx = np.arange(1, gt.max() + 1)
    cy, cx = (np.array(ndi.mean(g, gt, idx)) for g in np.mgrid[:H, :W])     # centroids
    haze = np.hypot(cy - 40, cx - 60)                       # distance to the haze centre
    for name, pred in segment(img).items():
        pred = np.unique(pred, return_inverse=True)[1].reshape(pred.shape)   # labels 1..M
        inter = overlap(gt, pred)
        a = tot.setdefault(name, np.zeros((len(thresholds) + 1, 4), int))
        res = [match(inter, t) for t in thresholds]
        a[:-1, :3] += [counts for counts, _, _ in res]       # TP, FP, FN per threshold
        missed, merged, split = error_masks(inter)
        a[-1, :3] += missed.sum(), merged.sum(), split.sum()
        w = where.setdefault(name, [[], [], []])
        w[0] += list(cx[missed]); w[1] += list(haze[merged]); w[2] += list(haze)
        a[-1, 3] += pred.max()                               # objects found
        pairs.setdefault(name, []).append(np.c_[res[0][1], res[0][2]])  # matches at 0.5

print(f"images {seeds.start}-{seeds.stop - 1}: {n_true} true nuclei; counts pooled over images")
print(f"{'method':36s} {'found':>5s} {'missed':>6s} {'merged':>6s} {'split':>5s}"
      f"  {'TP/FP/FN @0.5':>13s} {'AP@0.5':>6s} {'AP@0.75':>7s} {'AP@0.9':>6s} {'mean AP':>7s}")
for name, a in tot.items():
    tp, fp, fn = a[:-1, 0], a[:-1, 1], a[:-1, 2]
    ap = tp / (tp + fp + fn)                                 # "AP" = TP / (TP + FP + FN)
    print(f"{name:36s} {a[-1, 3]:5d} {a[-1, 0]:6d} {a[-1, 1]:6d} {a[-1, 2]:5d}"
          f"  {tp[0]:4d}/{fp[0]:3d}/{fn[0]:3d}  {ap[0]:6.3f} {ap[5]:7.3f} {ap[8]:6.3f} {ap.mean():7.3f}")
w = where["global Otsu + distance watershed"]
print("global Otsu + distance watershed: columns of missed nuclei", np.round(w[0]).astype(int),
      "of 256; median distance to haze centre, merged nuclei %.0f px, all nuclei %.0f px"
      % (np.median(w[1]), np.median(w[2])))
m = np.vstack(pairs["local threshold + distance watershed"])
print("local threshold + distance watershed, matches at IoU 0.5: median IoU %.3f, "
      "median predicted/true area %.3f" % tuple(np.median(m, axis=0)))

Output:

images 0-4: 184 true nuclei; counts pooled over images
method                               found missed merged split  TP/FP/FN @0.5 AP@0.5 AP@0.75 AP@0.9 mean AP
global Otsu + components                70      8    161     0    22/ 48/162   0.095   0.041  0.024   0.047
global Otsu + distance watershed       141      8     51     0   125/ 16/ 59   0.625   0.438  0.076   0.388
local threshold + components            81      0    165     0    29/ 52/155   0.123   0.077  0.000   0.057
local threshold + distance watershed   176      0     15     0   172/  4/ 12   0.915   0.837  0.098   0.639
global Otsu + distance watershed: columns of missed nuclei [224 161 220 235 238 223 217 204] of 256; median distance to haze centre, merged nuclei 60 px, all nuclei 152 px
local threshold + distance watershed, matches at IoU 0.5: median IoU 0.861, median predicted/true area 1.105

With connected components, both masks give about 70 to 80 objects for 184 nuclei: 161 and 165 nuclei sit inside merged objects, so fewer than 30 match at IoU 0.5 and the score stays near 0.1. Thresholding finds the foreground; it cannot find the boundaries between cells of the same brightness.

The distance-transform watershed separates the clusters, but how well depends on the mask. On the local-threshold mask it recovers 172 of 184 nuclei at IoU 0.5 with 4 false positives and leaves 15 nuclei in merges, the expected failure where two nuclei overlap deeply enough to leave one distance maximum. On the global Otsu mask it leaves 51 nuclei merged and 8 missed: the misses all lie in the dim right part of the image (columns 161 to 238), where a single threshold set from the whole histogram falls above the faintest nuclei, and the merged nuclei lie much closer to the haze (median 60 pixels from its centre, against 152 for all nuclei), where background pixels pass the threshold and join neighbouring nuclei into one blob with no clean distance maxima. A watershed cannot repair the foreground it is given.

The scores fall steeply with the threshold. For the best method, the median IoU of matched pairs is 0.861 and predicted nuclei are about 10% larger than the true ones, which we attribute to the blur spreading each nucleus so that the threshold falls outside its true edge; at $t = 0.9$ almost every match fails. Low thresholds count instances, high thresholds measure boundary agreement, and averaging over $t$ mixes the two. Running the script with first seeds 5 and 10 (images 5–9 and 10–14) again put the watershed on the local-threshold mask first and the watershed on the Otsu mask second, far above both connected-components methods (whose order swapped for images 5–9), with mean scores of 0.044, 0.394, 0.041, 0.671 and 0.058, 0.417, 0.066, 0.657 for the four methods.

Pixel classifiers with a boundary class

The foreground step is typically done with learned pixel classifiers, such as random forests in ilastik or convolutional networks, before pixels are grouped into instances (Schmidt et al., 2018). A two-class network (cell, background) followed by connected components inherits the merging problem, so the targets themselves were changed to make separation learnable:

• Weighted borders. U-Net (Ronneberger et al., 2015) carved thin background gaps between touching cells in the training masks and weighted those gap pixels heavily in the loss.
• A boundary class. Networks predict background, cell interior and boundary, and instances are the connected components of the interior, as in the three-class U-Net baseline of Schmidt et al. (2018). Kumar et al. (2017), as Graham et al. (2019) describe their method, labelled nuclei and contours and grew regions from them. The winning Data Science Bowl entry used three classes with boundary pixels marked only between touching cells, which its team found gave the best results (Caicedo et al., 2019).
• A contour branch. DCAN (Chen et al., 2016), designed for gland segmentation, predicts the object and its contour as two maps and subtracts the contour.

Distance and direction maps

Instead of a hard boundary, a network can regress a smooth map from which instances are recovered, often by a watershed:

• Distance maps. Naylor et al. (2019) regress each nucleus’s distance map and use it to find better watershed markers, avoiding predictions in regions with indistinct contours (as described by Graham et al., 2019). The second-placed Data Science Bowl entry predicted, for every pixel inside a nucleus, its distance to the boundary in four directions (vertical, horizontal, 45° and 135°), then turned these into boundaries refined with a watershed (Caicedo et al., 2019). A top Cell Tracking Challenge method, KIT-GE, predicts a cell distance map and a neighbour distance map, the inverse normalized distance to the nearest other cell (Maška et al., 2023).
• Deep watershed. The fourth-placed Data Science Bowl entry combined the watershed with networks after the deep watershed transform described in the instance segmentation article: one U-Net predicted direction vectors towards the nucleus boundary and a second the watershed energy levels (Caicedo et al., 2019).
• Horizontal and vertical maps (HoVer-Net). Graham et al. (2019) regress, for every nuclear pixel, its horizontal and vertical distance to the nucleus’s centre of mass, each scaled to $[-1, 1]$. The values change abruptly between neighbouring nuclei, so the Sobel gradients of the two maps, $S_m = \max(H_x(p_x), H_y(p_y))$, are high at boundaries. Thresholding the nuclear probability map and $S_m$ gives markers, and a marker-controlled watershed on an energy built from them gives the instances. The network is a Preact-ResNet50 encoder with a total down-sampling factor of 8 and three decoder branches: nuclear pixels, the HoVer maps and nuclear type. Its loss adds to the mean squared error of the maps a term on the error of their gradients.

Star-convex polygons (StarDist)

Schmidt et al. (2018) argued that axis-aligned boxes represent crowded, slanted nuclei poorly: two touching nuclei can have nearly identical boxes, and non-maximum suppression then deletes one. StarDist represents each nucleus as a star-convex polygon. For every pixel, a U-Net predicts an object probability, defined as the normalized Euclidean distance to the nearest background pixel so that central pixels score highest, and the distances $r^k_{ij}$ from the pixel to the object boundary along $n$ equally spaced rays ($n = 32$ in the paper). Training uses binary cross-entropy for the probabilities and a mean absolute error for the distances, weighted by the true object probability so that background pixels do not contribute. At test time, polygons from pixels above a probability threshold go through greedy non-maximum suppression with polygon intersections. The polygon cannot represent non-star-convex shapes and limits boundary precision: the authors write that StarDist did worse at the largest IoU thresholds on the synthetic datasets, and attribute this to the 32-ray parametric shape. In their table it falls behind the three-class U-Net and Mask R-CNN at 0.85 and 0.9 on both synthetic datasets but still scores above the two-class U-Net.

Flow fields (Cellpose)

Stringer et al. (bioRxiv version) aimed at whole cells of very varied shape. They note that watershed-style methods work when objects are blob-like, which many cells are not, and attribute the failure of earlier methods on their data to low expressive power. Cellpose converts each training mask into a vector field by simulating heat diffusion from the cell’s centre (the pixel closest to the median coordinates) with the mask boundary as a sink; the horizontal and vertical gradients of the resulting distribution point towards the centre, around corners where needed. A U-Net-style network with residual blocks, additive skip connections and a global “style” vector predicts the two flow components and a cell probability. At test time, every pixel with probability above 0.5 follows the predicted flows for 200 steps, and pixels that converge to the same point form one mask. The details in this section are from the bioRxiv version (version 2, posted 1 April 2020); the full text of the Nature Methods version (2021) could not be read for this article. The authors trained a single generalist model on a dataset of 616 images, some of them held out for testing: fluorescent cytoplasm, membranes, brightfield, other microscopy, and 98 non-microscopy images of repeated objects such as fruit and rocks. They extended it to 3D by running the 2D model on XY, XZ and YZ slices and averaging the flows, with no 3D training labels.

Detect then segment

Mask R-CNN predicts a box per object and a mask inside each box. Pretrained on COCO, it was the base of the third-placed Data Science Bowl entry (Caicedo et al., 2019), and it is the usual detector baseline in the papers above. Schmidt et al. (2018) built their synthetic Toy dataset of touching half-ellipse pairs, many slanted, to show its failure mode: suppressed instances whose boxes overlap a neighbour’s.

Reported results

Each paper used its own data, split and matching code, so rows compare only within a paper. “AP” here is $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP}+\mathrm{FN})$ at the stated IoU threshold.

Source	Data and protocol	Reported result
Schmidt et al. (2018)	497 fluorescence images selected from the Data Science Bowl training data, 90/10 split, scored across all test images	AP at 0.5: StarDist 0.864, Mask R-CNN 0.832, U-Net (3 class) 0.806, U-Net (2 class) 0.674; at 0.75: 0.586, 0.597, 0.578, 0.482
Stringer et al. (bioRxiv, 2020)	CellImageLibrary, 100 images of neurons, specialist models	AP at 0.5: Cellpose 0.88, Mask R-CNN 0.80, StarDist 0.76
Stringer et al. (bioRxiv, 2020)	generalist models, test images of all types	AP at 0.5: Cellpose 0.82, Mask R-CNN 0.70, StarDist 0.68
Graham et al. (2019, arXiv v5)	Kumar test set, 14 H&E tiles from 7 organs, scores averaged per image	PQ: HoVer-Net 0.597, Micro-Net 0.519, Mask R-CNN 0.509, U-Net 0.478, CellProfiler 0.300
Graham et al. (2019, arXiv v5)	CoNSeP test set, 14 colorectal adenocarcinoma tiles	PQ: HoVer-Net 0.547, Mask R-CNN 0.460, Micro-Net 0.449, U-Net 0.328
Caicedo et al. (2019)	Data Science Bowl second-stage set, 106 scored images from 15 unseen experiments	competition score: winner 0.6316, CellProfiler reference 0.5281; recall at IoU 0.7: 77.6% and 59.4%

The StarDist row shows the boundary-precision trade-off of the polygon: StarDist leads at 0.5, Mask R-CNN at 0.75. The CellProfiler reference in the last row was configured by an expert for each of five image groups, which the deep-learning entries were not allowed (Caicedo et al., 2019).

Datasets and Benchmarks

2018 Data Science Bowl. A Kaggle competition, with data assembled and annotated by a team at the Broad Institute, to build a nucleus segmenter that works on any 2D light microscopy image of stained nuclei with no human interaction (Caicedo et al., 2019). The dataset has 841 images and 37,333 manually annotated nuclei from more than 30 experiments (31 in the detailed count), covering fluorescence images of cultured cells and brightfield images of stained tissue; it is public as BBBC038 in the Broad Bioimage Benchmark Collection, with images and annotations under a CC0 (public domain) licence. Sixteen experiments supplied the training set (670 images) and the first-stage test set (65 images); 15 experiments not in the training data supplied the second stage. Participants had one week to segment 3,200 second-stage images, of which only 106, unknown to them, had target masks and were scored, a design meant to prevent manual intervention. Training masks were annotated with a superpixel-assisted tool; test masks were drawn pixel by pixel. The competition drew 3,891 teams, and the top three entries were an ensemble of 32 U-Net and FPN networks, a single fully convolutional FPN, and a Mask R-CNN.

Cell Tracking Challenge. Launched in 2013 at the IEEE International Symposium on Biomedical Imaging (ISBI) and run continuously since (Maška et al., 2023). It provides 2D and 3D time-lapse sequences of fluorescent, phase-contrast, differential interference contrast and brightfield microscopy, real and computer-generated, 20 datasets as of the 2023 report. Reference annotations for the training data are public and those of the test data are kept secret; participants submit results with executable versions of their algorithms, which the organizers verify. Real datasets have a gold truth (majority opinion of three experts) and, introduced since the 2017 report, a silver truth fused from high-performing submissions that covers 99.1% of cell instances on average. The Cell Tracking Benchmark scores segmentation (SEG) and tracking (TRA); the Cell Segmentation Benchmark, added in 2019, scores segmentation and detection (DET) without linking. Its overall performance measure averages the two, $\mathrm{OP_{CSB}} = \tfrac{1}{2}(\mathrm{SEG} + \mathrm{DET})$.

Histology nucleus datasets. The Kumar dataset, released with Kumar et al. (2017) and used in the MoNuSeg challenge, has 30 H&E tiles from seven organs; Graham et al. (2019) used 16 tiles from four organs for training and 14 from seven organs for testing. CoNSeP, introduced with HoVer-Net, has 41 H&E tiles of $1{,}000 \times 1{,}000$ pixels at 40× from 16 colorectal adenocarcinoma slides, with 24,319 nuclei, each annotated by one of two pathologists, checked by both, and typed (Graham et al., 2019).

Generalist training sets. Cellpose’s cell dataset combines 100 CellImageLibrary images of neurons with 516 other images, over 70,000 segmented objects; its nucleus dataset of 1,139 images pools BBBC038, BBBC039, MoNuSeg and ISBI 2009 images (Stringer et al., bioRxiv version).

Evaluation Metrics

Cell segmentation is scored at the object level. Predicted and true objects are matched by IoU, and the counts of true positives (matched pairs), false positives (unmatched predictions) and false negatives (unmatched true objects) give the scores. The details differ between benchmarks and change the numbers.

Matching. When predictions and ground truth are each non-overlapping, a match at IoU above 0.5 is unique: no object can have IoU above 0.5 with two disjoint objects (Kirillov et al., 2019; the panoptic segmentation article gives the proof and its strict inequality). Below 0.5, several pairings are possible and a matching rule is needed. StarDist’s code (release 0.9.2) solves a linear assignment that maximizes the number of pairs with IoU at least $t$, using IoU to break ties, the Hungarian-style optimum the example above copies; Cellpose’s evaluation code (v4.2.1) states that it is based on StarDist’s matching. For its error analysis, the Data Science Bowl paper kept, for each target, the prediction with maximum IoU above the threshold and counted remaining pieces as false positives (Caicedo et al., 2019, supplementary note).

“Average precision”. Schmidt et al. (2018) and Stringer et al. (bioRxiv version, which computed it with StarDist’s code) report

$$\mathrm{AP}_t = \frac{\mathrm{TP}_t}{\mathrm{TP}_t + \mathrm{FP}_t + \mathrm{FN}_t},$$

a Jaccard index over object counts at threshold $t$. Despite its name it is not the area under a precision–recall curve of ranked detections, the average precision of COCO-style detection benchmarks; StarDist’s code calls it accuracy and questions the name in a comment. Caicedo et al. also report the F1 score, $2\mathrm{TP}/(2\mathrm{TP} + \mathrm{FP} + \mathrm{FN})$, which is always at least as high.

The Data Science Bowl score averaged $\mathrm{TP}_t/(\mathrm{TP}_t + \mathrm{FP}_t + \mathrm{FN}_t)$ over a set of IoU thresholds and over images, with every image weighted equally whatever its number of nuclei (Caicedo et al., 2019). The paper disagrees with itself on the thresholds: the Methods print $T = \{0.10, 0.15, \dots, 0.95\}$, while the supplementary note calls 0.5 the minimum used in the competition score. The competition’s Kaggle page was not read for this article.

Averaging. Pooling counts over a test set (StarDist’s default) weights crowded images more; averaging per image (the Data Science Bowl, Cellpose’s per-image output, HoVer-Net’s reported values) weights each image equally. Reports should state which.

Panoptic quality and AJI. Graham et al. (2019) argued that the Aggregated Jaccard Index (AJI, from Kumar et al., 2017) and ensemble Dice over-penalize overlaps when a neighbouring nucleus is missed, and adopted panoptic quality, the product of a detection term (an F1 score at IoU 0.5) and a segmentation term (the mean IoU of matches), for nuclear instance segmentation.

Cell Tracking Challenge SEG and DET. SEG uses a different match: a reference cell $R$ and a segmented object $S$ match if $|R \cap S| > 0.5\,|R|$, which allows at most one $S$ per $R$; each reference cell scores the Jaccard index $|R \cap S| / |R \cup S|$ of its match, or 0 if it has none, and SEG is the mean over all reference cells (Maška et al., 2014). Because only reference cells are averaged, false positive objects do not lower SEG; in our reading, that is one reason the Cell Segmentation Benchmark pairs it with DET, one minus a normalized graph-matching cost of turning the detected cells into the reference ones (Maška et al., 2023).

Merges and splits. Counting merged, split, missed and extra objects explains a score: Caicedo et al. used such counts to show that the top models separated merged nuclei that the CellProfiler pipelines left fused, and the example above shows thresholding failing almost entirely by merges.

Applications

Cell segmentation supplies the per-cell measurements of high-content screens of cell phenotypes and the starting point of lineage reconstruction of dividing cells (Schmidt et al., 2018), as well as nuclear counts, shapes and types in histology, which Graham et al. (2019) describe as a prerequisite for quantitative analysis of whole-slide images. Robotic microscopes that find and count nuclei in real time are one of the uses Caicedo et al. (2019) envision.

Open Problems

• Generalization. In the Data Science Bowl, top accuracy fell from 0.90 on the largest image group to 0.55 on the smallest, and the authors found data availability a limit for several image types (Caicedo et al., 2019). Unstained brightfield, electron microscopy and 3D images were outside the challenge.
• Overlapping and irregular cells. Label images cannot represent overlaps, and star-convex or distance-based representations suit round nuclei better than branched cells.
• Boundary precision. Scores at IoU 0.75 to 0.9 remain far below those at 0.5 in every paper above, and part of the gap is annotation noise: Caicedo et al. found that their top model agreed with each of two annotators more often than the annotators agreed with each other.
• 3D and annotation cost. Dense 3D labels are expensive, which is why Cellpose’s 3D mode reuses a 2D model.
• Comparable evaluation. Matching rules, thresholds and averaging differ between benchmarks and codebases, so published numbers are comparable only under the same protocol.