Cellpose

Cellpose is a deep-learning model for cell segmentation, a form of instance segmentation: given a microscopy image, it returns a label image in which every cell or nucleus has its own mask. Carsen Stringer, Tim Wang, Michalis Michaelos and Marius Pachitariu at HHMI Janelia Research Campus posted it as a preprint in February 2020 and published it in Nature Methods (online December 2020, in the January 2021 issue). Its central idea is a representation: instead of predicting boundaries or distances, a U-Net-style convolutional network predicts, at every pixel, a unit vector pointing along the gradient of a heat distribution simulated inside each training mask, and test-time masks are the groups of pixels that converge to the same point when they follow those vectors. Its second idea is generality: one model trained on a varied dataset of cytoplasm- and membrane-labelled cells, other microscopy and even non-biological images of repeated objects, meant to work on new images without retraining (Stringer et al., bioRxiv version). The software has since grown through Cellpose 2.0 (human-in-the-loop training), Cellpose3 (image restoration) and Cellpose-SAM, which replaces the convolutional network with the image encoder of the Segment Anything Model while keeping the flow representation. This article describes the method from the bioRxiv preprint, which was read in full, and the later versions from their abstracts, the official documentation and the source code at pinned releases.

Problem Addressed

For nuclei, the varied images of the 2018 Data Science Bowl had let deep networks generalize far beyond a single lab’s data, but whole cells are harder: their shapes are highly diverse, and in many tissues they are tightly packed (Stringer et al., bioRxiv version). Stringer et al. (bioRxiv version) report that earlier methods trained on their new, varied dataset did not perform well and hypothesized that the reason was low expressive power, an inability to describe diverse cell shapes. They note that the classical watershed works when objects are blob-like, so that they form smooth basins in a topographic map, and that many cells are not.

The paper’s comparison methods were StarDist, which describes each nucleus as a star-convex polygon (Schmidt et al., 2018), and Mask R-CNN, which predicts a box and a mask per object; the Cell Segmentation article compares these families and their failure modes. The goal of Cellpose was a target that a network can predict from the image, that encodes the identity of every pixel’s cell for any shape, and that can be turned back into masks without per-dataset parameters.

Architecture

Cellpose has three parts: a fixed transformation from label images to flow fields, used only to make training targets; a network that predicts the flows and a cell probability from the image; and a deterministic procedure that turns predicted flows back into masks. Only the network has learned parameters. The paper describes the transformation as reversible: masks can be recovered from flows (Stringer et al., bioRxiv version).

Flow Representation

Each training mask is converted into a vector field by a heat-diffusion simulation (Stringer et al., bioRxiv version, Methods). The centre of a cell is the pixel inside it closest to the median of its pixels’ horizontal and vertical coordinates; the authors report that other definitions, such as the 2D medoid, gave similar flows and performance. A heat source adds 1 to the centre pixel at every iteration, and every pixel inside the mask is then set to the mean of the $3 \times 3$ square around it, itself included, with pixels outside the mask set to 0. Writing $\Omega$ for the cell’s pixels, $c$ for its centre and $T_t$ for the heat after $t$ iterations,

$$T_{t+1}(\mathbf{x}) = \frac{1}{9} \sum_{\mathbf{y} \in \mathcal{N}_3(\mathbf{x})} \Big( T_t(\mathbf{y}) + [\mathbf{y} = c] \Big), \quad \mathbf{x} \in \Omega, \qquad T_{t+1}(\mathbf{x}) = 0, \quad \mathbf{x} \notin \Omega,$$

where $\mathcal{N}_3(\mathbf{x})$ is the $3 \times 3$ neighbourhood of $\mathbf{x}$ and $[\cdot]$ is 1 when its argument holds and 0 otherwise. The boundary is “leaky”: heat that reaches it is lost. The simulation runs for $N = 2(\Delta x + \Delta y)$ iterations, where $\Delta x$ and $\Delta y$ are the horizontal and vertical extents of the mask, long enough, in the authors’ words, for heat to reach the furthest corners. The final distribution approaches the equilibrium and is used as an energy function: its horizontal and vertical gradients are the two flow maps, normalized to unit length,

$$\mathbf{u}(\mathbf{x}) = \frac{\nabla T_N(\mathbf{x})}{\lVert \nabla T_N(\mathbf{x}) \rVert}.$$

The flows do not point straight at the centre, which for a curved cell would cross the boundary; locally they move pixels to other pixels of the same cell, and over many steps they lead to the centre, around corners where needed.

Why one fixed point per cell can be expected is our reasoning, not a statement of the paper: at equilibrium, $T$ is a discrete harmonic function everywhere in the cell except at the source and its eight neighbours (which receive part of the added heat), and by the maximum principle a harmonic function has no interior maximum, so the only maximum lies in that $3 \times 3$ block; ascent paths can end only there, or in the degenerate case at a saddle point. A watershed on a distance map has no such guarantee, since a thin, curved or branched cell has a ridge of nearly equal distances and often several maxima. In practice the simulation is stopped after $N$ iterations rather than at equilibrium, and heat at the tips of long arms can be very small, which makes the normalized direction there sensitive to discretization (see the example below).

The implementation has changed in details while keeping the idea. The first public code (release v0.0.1.25, April 2020) took $\log(1 + T)$ before computing central differences, which the preprint does not mention. Release v4.2.1.1 runs the diffusion for all cells at once in PyTorch, uses as centre the pixel nearest to the mask’s rounded mean coordinates (its docstring still says median), runs $2(h + w + 2)$ iterations for all cells, where $h + w$ is the largest sum of bounding-box height and width of any mask in the image, and normalizes the gradients to unit length (cellpose/dynamics.py). Its 3D version averages each voxel with its six face neighbours, starting from the voxel nearest the mean coordinates, for six times the largest sum of a mask’s three bounding-box extents (each plus 1).

Network

The network takes a two-channel image: the cytoplasm or membrane label, and an optional nuclear channel (DAPI in all cases in the paper), replaced by zeros when absent. Intensities are scaled per image so that the 1st and 99th percentiles map to 0 and 1 (Stringer et al., bioRxiv version, Methods).

It follows the U-Net layout of a downsampling pass and a mirror-symmetric upsampling pass (Ronneberger et al., 2015), with three changes:

• Residual blocks. Each of the four spatial scales has two residual blocks of two $3 \times 3$ convolutions each, preceded by batch normalization and ReLU in the pre-activation order; the first block’s skip uses a $1 \times 1$ convolution because the number of feature maps changes. The depth is doubled relative to a plain U-Net, as is usual for residual networks. Downsampling is by max pooling.
• Additive skip connections. Feature maps from the downsampling pass are added, not concatenated, to the upsampling pass, to reduce the number of parameters.
• A style vector. Global average pooling of the smallest feature maps gives a 256-dimensional vector, normalized to unit length per image to account for differences in cell density. After a linear projection, it is added to the last three of the four convolutions at each upsampling scale, broadcast over all positions, so that the network can adjust its computation to the type of image.

Feature maps start at 32 and double at each downsampling step up to 256. On a held-out validation set, a base of 48 did not help and bases of 24 and 16 hurt. A final $1 \times 1$ convolution gives three maps: the vertical and horizontal flows and a logit of the cell probability. The paper’s model was built in MXNet; release v0.6 (December 2020) made PyTorch the default, and the convolutional network of the Cellpose 3 code (CPnet in resnet_torch.py, release v3.1.1.2) keeps the channel widths [32, 64, 128, 256] and a default mean diameter of 30 pixels (17 for the nucleus model).

Mask Recovery

At test time, pixels with cell probability above 0.5 are treated as foreground. From each foreground pixel, a dynamical system is run for 200 iterations: at every step the pixel moves by the predicted flow at the nearest grid location, a finite-difference step of size 1 (Stringer et al., bioRxiv version, Methods). Predicted flows are not renormalized; the training flows had unit norm, so the authors expect predictions on the same scale. After convergence, pixels are grouped by where they ended. For robustness, clusters are grown from high-density points of convergence: neighbouring positions with at least 3 converged pixels are agglomerated until every position around the region has fewer than 3, which handles cases where the network is unsure of the exact centre and predicts a region of low flow magnitude there.

A quality check follows. Nothing forces the network to output flows that correspond to a real shape, so the flows of each recovered mask are recomputed from the mask and compared with the predicted ones; masks whose mean squared difference exceeds a threshold are discarded. The threshold was cross-validated on held-out data at 0.4 (Stringer et al., bioRxiv version), which is still the default flow_threshold in the documentation and in release v4.2.1.1.

The current code differs in its defaults. The cell-probability threshold is applied to the logit, with default cellprob_threshold=0.0 (a probability of 0.5); the dynamics use bilinear interpolation of the flows instead of the nearest grid point, the default since release v0.6; the number of iterations is 200, or $200\,d/30$ when a diameter $d$ is given and the image is resized to the 30-pixel training size; and convergence clusters start from local maxima of the histogram of final positions holding more than 10 pixels and grow by five steps into bins with more than 2 pixels. Masks covering more than 40% of the image and masks smaller than 15 pixels are removed (cellpose/dynamics.py and cellpose/models.py, release v4.2.1.1).

Example: Flows from Masks and Back, Against Watershed

The example re-implements the flow computation and the mask recovery from the bioRxiv Methods in NumPy, SciPy and scikit-image (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2); it does not use the Cellpose package, and no network is trained. It asks a question about the representation alone: if a network predicted a target perfectly, or with the smoothness that a network’s output tends to have, could the cells be recovered from it? Five $256 \times 256$ label images hold about 24 touching cells each, of three shapes: ellipses (blob-like), curved rods (a dilated arc of 150° to 250°) and branched cells (three dilated arms from a common point). Three targets are computed from the true masks, and each is decoded inside a foreground mask:

• the distance transform of the foreground, the classical input to a watershed, which knows nothing of the instances;
• each cell’s distance to its own boundary, the target of distance-regression methods, decoded by a watershed on its negative;
• the Cellpose flows, decoded by following them as described above.

The two distance maps are tried with two marker rules for the watershed: local maxima at least 5 pixels apart, and regional maxima after maxima less than 2 pixels deep are merged by morphological reconstruction (an h-maxima rule with $h = 2$). In the “blurred” condition, every map (the flow components, the per-cell distance map and the foreground indicator) is smoothed with a Gaussian of $\sigma = 1.5$ pixels and the foreground is re-thresholded at 0.5. Objects under 15 pixels are removed, as by Cellpose’s default. Scoring matches predicted and true cells one to one by IoU, as in the Cell Segmentation article’s example, and reports $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP}+\mathrm{FN})$ with counts pooled over the five images, the score the cell literature calls average precision, at IoU 0.5 and averaged over thresholds 0.50 to 0.95, plus, per shape, the cells matched at IoU 0.5 and the cells split into two or more objects.

The script was planned before it was run and changed four times afterwards. The first run stopped with an error in the scoring function (NumPy does not negate a boolean array), fixed by converting the array to floating point. The second run showed both distance-map decoders splitting nearly every curved and branched cell with the 5-pixel peak rule, which we judged too weak a baseline for thin cells, so the second marker rule was added as two new rows, with the original rows kept; its first version used scikit-image’s h_maxima, which returns only the pixels at least $h$ above the reconstructed surface and so can mark several separate pixels along one ridge, and it was replaced by the regional maxima of the reconstruction shown below. An unused import was then removed, with identical output. Apart from the fix in the scoring function, the synthetic data, the flow computation, the flow following and the scoring did not change.

import sys

import numpy as np
from scipy import ndimage as ndi
from scipy.optimize import linear_sum_assignment
from skimage.feature import peak_local_max
from skimage.morphology import local_maxima, reconstruction
from skimage.segmentation import watershed

H = W = 256
SHAPES = ("ellipse", "curved", "branched")


def draw_cell(kind, rng):
    """Boolean mask of one cell in a 61 x 61 patch, centred."""
    yy, xx = np.mgrid[-30:31, -30:31]
    if kind == "ellipse":
        a, b, th = rng.uniform(7, 11), rng.uniform(5, 8), rng.uniform(0, np.pi)
        u = xx * np.cos(th) + yy * np.sin(th)
        v = -xx * np.sin(th) + yy * np.cos(th)
        return (u / a) ** 2 + (v / b) ** 2 < 1
    skel = np.zeros(yy.shape, bool)
    if kind == "curved":                       # arc of a circle, 150-250 degrees
        R, span, th0 = rng.uniform(11, 16), rng.uniform(2.6, 4.4), rng.uniform(0, 2 * np.pi)
        for t in np.linspace(th0, th0 + span, 200):
            skel[int(round(30 + R * np.sin(t))), int(round(30 + R * np.cos(t)))] = True
        r = rng.uniform(3.5, 5)
    else:                                      # three arms from a common point
        th0 = rng.uniform(0, 2 * np.pi)
        for k in range(3):
            t = th0 + k * 2 * np.pi / 3 + rng.uniform(-0.4, 0.4)
            L = rng.uniform(13, 22)
            for s in np.linspace(0, L, 100):
                skel[int(round(30 + s * np.sin(t))), int(round(30 + s * np.cos(t)))] = True
        r = rng.uniform(3, 4.5)
    return ndi.distance_transform_edt(~skel) < r


def synth(seed, n_cells=24):
    rng = np.random.default_rng(seed)
    lab = np.zeros((H, W), int)
    kinds = []
    for _ in range(2000):
        if len(kinds) == n_cells:
            break
        kind = SHAPES[rng.integers(3)]
        m = draw_cell(kind, rng)
        cy, cx = rng.integers(30, H - 30), rng.integers(30, W - 30)
        win = (slice(cy - 30, cy + 31), slice(cx - 30, cx + 31))
        overlap = (lab[win][m] > 0).mean()
        touch = (ndi.binary_dilation(m, iterations=2) & (lab[win] > 0)).any()
        if overlap <= 0.15 and (touch or len(kinds) < 4):     # neighbours touch
            region = lab[win]
            region[m] = len(kinds) + 1                       # newer cell takes overlaps
            kinds.append(kind)
    out = np.zeros_like(lab)                                 # keep the largest piece
    keep = []
    for i in range(1, len(kinds) + 1):
        comp, n = ndi.label(lab == i)
        if n == 0:
            continue
        big = comp == (np.argmax(np.bincount(comp.ravel())[1:]) + 1)
        if big.sum() >= 30:
            keep.append(kinds[i - 1])
            out[big] = len(keep)
    return out, np.array(keep)


def heat_flows(lab):
    """Cellpose flows: heat diffusion from each cell's centre (bioRxiv Methods)."""
    flow = np.zeros((2, H, W))
    for i, sl in enumerate(ndi.find_objects(lab), 1):
        sl = tuple(slice(max(s.start - 1, 0), s.stop + 1) for s in sl)
        m = lab[sl] == i
        ys, xs = np.nonzero(m)
        c = np.argmin((ys - np.median(ys)) ** 2 + (xs - np.median(xs)) ** 2)
        T = np.zeros(m.shape)
        for _ in range(2 * (np.ptp(ys) + np.ptp(xs))):
            T[ys[c], xs[c]] += 1                             # heat source at the centre
            T = ndi.uniform_filter(T, 3, mode="constant") * m  # 3x3 mean, outside = 0
        gy, gx = np.gradient(T)
        flow[0][sl][m], flow[1][sl][m] = gy[m], gx[m]
    return flow / (np.hypot(*flow) + 1e-20)                  # unit vectors


def follow_flows(flow, fg, niter=200):
    """Every foreground pixel takes unit-size steps along the flow at its nearest grid point."""
    p = np.array(np.nonzero(fg), float)
    for _ in range(niter):
        iy, ix = (np.clip(np.rint(p[k]), 0, s - 1).astype(int) for k, s in enumerate((H, W)))
        p += flow[:, iy, ix]
    q = np.clip(np.rint(p), 0, [[H - 1], [W - 1]]).astype(int)
    hist = np.zeros((H, W), int)
    np.add.at(hist, (q[0], q[1]), 1)
    clusters, _ = ndi.label(hist >= 3, structure=np.ones((3, 3)))   # agglomerate bins with >= 3
    out = np.zeros((H, W), int)
    out[fg] = clusters[q[0], q[1]]
    return out


def dist_watershed(height, fg, rule):
    if rule == "peaks":                       # local maxima at least 5 px apart
        markers = np.zeros((H, W), int)
        pk = peak_local_max(height, min_distance=5, labels=ndi.label(fg)[0], exclude_border=False)
        markers[tuple(pk.T)] = np.arange(1, len(pk) + 1)
    else:                                     # maxima less than 2 px deep merged first
        rec = reconstruction(height - 2, height)               # reconstruction by dilation
        markers = ndi.label(local_maxima(rec) & fg, structure=np.ones((3, 3)))[0]
    return watershed(-height, markers, mask=fg)


def clean(pred, min_size=15):
    sizes = np.bincount(pred.ravel())
    pred[sizes[pred] < min_size] = 0
    return np.unique(pred, return_inverse=True)[1].reshape(pred.shape)


def score(gt, pred, ts):
    inter = np.zeros((gt.max() + 1, pred.max() + 1))
    np.add.at(inter, (gt.ravel(), pred.ravel()), 1)
    iou = (inter / (inter.sum(1, keepdims=True) + inter.sum(0, keepdims=True) - inter))[1:, 1:]
    counts = []
    for t in ts:
        r, c = linear_sum_assignment(-(iou >= t).astype(float) - iou / (2 * min(iou.shape)))
        tp = int((iou[r, c] >= t).sum())
        counts.append((tp, iou.shape[1] - tp, iou.shape[0] - tp))
    r, c = linear_sum_assignment(-(iou >= 0.5).astype(float) - iou / (2 * min(iou.shape)))
    matched = np.zeros(iou.shape[0], bool)
    matched[r[iou[r, c] >= 0.5]] = True
    inside = inter[1:, 1:] > 0.5 * inter[:, 1:].sum(0, keepdims=True)   # object mostly in cell
    return np.array(counts), matched, inside.sum(1) >= 2


ts = np.round(np.arange(0.5, 1.0, 0.05), 2)
first = int(sys.argv[1]) if len(sys.argv) > 1 else 0
seeds = range(first, first + 5)
methods = ("foreground distance, peaks", "foreground distance, h-maxima",
           "cell distance map, peaks", "cell distance map, h-maxima", "Cellpose flows")
res = {}
n_cells = {k: 0 for k in SHAPES}
for seed in seeds:
    gt, kinds = synth(seed)
    for k in SHAPES:
        n_cells[k] += (kinds == k).sum()
    fg0 = gt > 0
    flow0 = heat_flows(gt)
    cell_dist0 = np.zeros((H, W))
    for i, sl in enumerate(ndi.find_objects(gt), 1):
        cell_dist0[sl][gt[sl] == i] = ndi.distance_transform_edt(np.pad(gt[sl] == i, 1))[1:-1, 1:-1][gt[sl] == i]
    for cond, sigma in (("exact", 0), ("blurred", 1.5)):
        blur = (lambda a: ndi.gaussian_filter(a, sigma)) if sigma else (lambda a: a)
        fg = blur(fg0.astype(float)) > 0.5
        flow = np.stack([blur(f) for f in flow0])
        fg_dist, cell_dist = ndi.distance_transform_edt(fg), blur(cell_dist0)
        preds = (dist_watershed(fg_dist, fg, "peaks"), dist_watershed(fg_dist, fg, "hmax"),
                 dist_watershed(cell_dist, fg, "peaks"), dist_watershed(cell_dist, fg, "hmax"),
                 follow_flows(flow, fg))
        for name, pred in zip(methods, preds):
            pred = clean(pred)
            counts, matched, split = score(gt, pred, ts)
            r = res.setdefault((cond, name), {"c": 0, "found": 0, "m": [], "s": [], "k": []})
            r["c"] = r["c"] + counts
            r["found"] += pred.max()
            r["m"].append(matched); r["s"].append(split); r["k"].append(kinds)

print(f"images {seeds.start}-{seeds.stop - 1}: cells", ", ".join(f"{k} {n_cells[k]}" for k in SHAPES))
print(f"{'condition':8s} {'decoder':30s} {'found':>5s} {'AP@0.5':>6s} {'mean AP':>7s}  matched at IoU 0.5 (split) by shape")
for (cond, name), r in res.items():
    tp, fp, fn = r["c"][:, 0], r["c"][:, 1], r["c"][:, 2]
    ap = tp / (tp + fp + fn)
    m, s, k = (np.concatenate(r[x]) for x in "msk")
    by = "  ".join(f"{sh} {m[k == sh].sum():2d}/{(k == sh).sum()} ({s[k == sh].sum()})" for sh in SHAPES)
    print(f"{cond:8s} {name:30s} {r['found']:5d} {ap[0]:6.3f} {ap.mean():7.3f}  {by}")

Output:

images 0-4: cells ellipse 22, curved 46, branched 52
condition decoder                        found AP@0.5 mean AP  matched at IoU 0.5 (split) by shape
exact    foreground distance, peaks       506  0.040   0.030  ellipse 21/22 (0)  curved  2/46 (45)  branched  1/52 (52)
exact    foreground distance, h-maxima     60  0.277   0.169  ellipse  9/22 (0)  curved 13/46 (1)  branched 17/52 (0)
exact    cell distance map, peaks         576  0.042   0.036  ellipse 22/22 (0)  curved  1/46 (45)  branched  5/52 (52)
exact    cell distance map, h-maxima      126  0.952   0.906  ellipse 22/22 (0)  curved 46/46 (5)  branched 52/52 (1)
exact    Cellpose flows                   120  1.000   1.000  ellipse 22/22 (0)  curved 46/46 (0)  branched 52/52 (0)
blurred  foreground distance, peaks       430  0.046   0.030  ellipse 20/22 (0)  curved  1/46 (44)  branched  3/52 (51)
blurred  foreground distance, h-maxima     64  0.278   0.133  ellipse  8/22 (0)  curved 13/46 (0)  branched 19/52 (0)
blurred  cell distance map, peaks         269  0.259   0.154  ellipse 22/22 (0)  curved 12/46 (45)  branched 46/52 (38)
blurred  cell distance map, h-maxima       91  0.675   0.524  ellipse 19/22 (0)  curved 33/46 (0)  branched 33/52 (0)
blurred  Cellpose flows                   117  0.975   0.884  ellipse 22/22 (0)  curved 45/46 (0)  branched 50/52 (0)

Following the exact flows returns all 120 cells with no error at any threshold: the transformation is reversible on these shapes, as the paper says it is, because every pixel’s path ends at its own cell’s single source. The 5-pixel peak rule on either distance map fails for the opposite reason: thin cells have long ridges of nearly equal distance, and the rule places a marker every few pixels along them, splitting almost every curved and branched cell while leaving the blob-like ellipses intact. That is the paper’s argument about blob-like shapes in its sharpest form.

With the merged-maxima rule, the per-cell distance map is almost as good as the flows on exact targets (six cells split), so the gap there lies in the decoding rule rather than the information. The foreground distance map is different: it does not know where one cell ends and the next begins, and the necks between touching cells are less than 2 pixels deep, so the same rule merges them into 60 objects. Any instance method has to put the separating information into its target; the two instance-aware targets do, the foreground distance does not.

Blurring separates the two instance-aware targets. A per-cell distance map marks the border between touching cells by a valley that falls to zero; blur fills it, the maxima of neighbours join within 2 pixels, and the watershed merges cells (91 objects for 120 cells, no splits). A flow field marks the same border by a reversal of direction; blur shrinks the vectors near the border but does not turn them, so pixels still travel to the right side. The flows lose about 0.12 in the averaged score, which we attribute to the blurred foreground rounding off the tips and edges of the cells and lowering their IoU, not to errors of identity. Running the script with first seeds 5 and 10 (images 5–9 and 10–14) gave the same picture: 0.992 and 0.992 at IoU 0.5 for exact flows, 0.992 and 0.983 for blurred flows, against 0.952 and 0.952 for the exact per-cell distance map and 0.577 and 0.562 for the blurred one. In each of those two runs one branched cell was split even with exact flows. The output does not show why; in our reading, nearest-grid, unit-step dynamics on a discretized field can stall pixels where neighbouring vectors send them back and forth, one reason the package has used bilinear interpolation of the flows by default since release v0.6.

The example does not show that Cellpose segments real images better than distance-based methods. A real network makes errors other than blur, and the merged-maxima rule, the blur width and the shapes were choices of this example; Stringer et al. compared trained networks, as reported under Performance.

Training

Data. The generalist (“cyto”) dataset has 616 images with over 70,000 segmented objects (Stringer et al., bioRxiv version). 100 are two-channel images of cultured neurons from the CellImageLibrary, whose segmentations the authors corrected where touching neurons were joined and freed of overlaps so that each pixel belongs to at most one mask. The other 516 were segmented by hand by a single annotator: 216 with fluorescent cytoplasmic markers (including the Broad Bioimage Benchmark Collection sets BBBC020 and BBBC007, two-photon images of mouse neurons and images found by web search), 50 brightfield images, 58 membrane-labelled images (including the Micro-Net set, re-segmented on the membrane channel alone), 86 images of other microscopy, and 98 non-microscopy images of repeated objects such as fruit, fish scales, jellyfish and rocks, included in the hope of wider generalization. 69 of the 516 were held out for testing. (The category counts add up to 508, not 516; likewise the Results section’s counts, which give 316 cytoplasm images including the 100 from the CellImageLibrary, add up to 608 of the 616 images. The paper does not explain the difference.) A separate nucleus dataset of 1,139 images, 111 held out, pooled BBBC038 (with unofficial corrected labels by Konstantin Lopuhin), the 157 unique images of BBBC039, MoNuSeg tiles split into nine and inverted so that nuclei are bright, and ISBI 2009. The authors’ annotation interface let operators segment 300 to 600 objects per hour.

Objective. The three output maps are trained jointly: a squared-error loss between the predicted flows and the true flows multiplied by 5, to balance the two terms, and a binary cross-entropy between the sigmoid of the third map and the inside/outside label (Stringer et al., bioRxiv version). Release v4.2.1.1 writes it as

$$\mathcal{L} = \frac{1}{2} \cdot \frac{1}{2P} \sum_{\mathbf{x}} \big\lVert \hat{\mathbf{u}}(\mathbf{x}) - 5\,\mathbf{u}(\mathbf{x}) \big\rVert^2 + \frac{1}{P} \sum_{\mathbf{x}} \mathrm{BCE}\big(\sigma(\hat{z}(\mathbf{x})),\, [\mathbf{x} \in \text{cell}]\big),$$

where $P$ is the number of pixels, $\hat{\mathbf{u}}$ the two predicted flow maps, $\hat{z}$ the predicted logit and $\sigma$ the sigmoid (_loss_fn_seg in cellpose/train.py, with PyTorch’s mean-reduced MSE and BCE-with-logits losses). The flow loss is a mean squared error and the probability loss a cross-entropy; predicted flows therefore come out at 5 times unit length, which is why the code divides them by 5 before running the dynamics and before the quality check.

Procedure. The paper’s networks were trained for 500 epochs with stochastic gradient descent (learning rate 0.2, momentum 0.9, batch size 8, weight decay $10^{-5}$), with a linear warm-up over 10 epochs and the learning rate halved every 10 epochs from epoch 400; the rate was chosen to minimize training loss among 0.01, 0.05, 0.1, 0.2 and 0.4. Every epoch applied random rotations (0° to 360°), scaling and translations, then took a $224 \times 224$ crop from the centre; the flow vectors were rotated with the image. For Cellpose, images were first resized to a common mean cell diameter and then scaled by a random factor between 0.75 and 1.25; StarDist and Mask R-CNN, which cannot resize their predictions, had no common resizing and a wider scale range of 0.5 to 1.5 (data augmentation). Fine-tuning in the current release follows a different recipe: AdamW with learning rate $10^{-5}$, weight decay 0.1, batch size 1 and 100 epochs by default, starting from the built-in model (documentation, Training page; train_seg in release v4.2.1.1).

Inference

The paper’s model is meant to run on images of unknown scale and size, so its inference adds several steps around the network (Stringer et al., bioRxiv version, Methods).

• Size estimation. Because training images were resized to a common cell size, test images must be too. A linear regression from the style vector predicts the median cell diameter; Cellpose is then run at that size and the median diameter of the resulting masks is the refined estimate (median in the caption of figure S2; the Methods text says mean object size), which reached correlations of 0.93 (cytoplasm) and 0.97 (nuclei) with the true diameters of test images. Users may enter the diameter instead. Flows and probabilities are resized back to the original image with bilinear interpolation before the dynamics.
• Tiling and test-time augmentation. The image is cut into four sets of $224 \times 224$ tiles offset by 112 pixels, so almost every pixel is predicted four times. Tile sets are left as they are, flipped horizontally, flipped vertically or flipped both ways; outputs are flipped back, the sign of the flipped flow component is reversed, and the four predictions are averaged with sigmoid tapers of bandwidth 7.5 pixels at tile edges.
• Ensembling. The flows of four separately trained models are averaged.
• 3D without 3D labels. The 2D model is run on all XY, XZ and YZ slices of a volume; each slice orientation gives two of the three flow components, so every voxel gets two estimates of each, which are averaged; the three probability estimates are averaged and thresholded at 0.5, and the dynamics run in 3D. The authors estimate that at their median diameter of 30 pixels, labelling a cell in 3D takes about 30 times longer than in one 2D slice, which is the motivation. Objects smaller than 10% of the median cell volume were discarded.

The current documentation calls the 3D mode “2.5D segmentation” and recommends smoothing the flows with flow3D_smooth when cells are fragmented and, when the XZ and YZ slices are unsuitable, segmenting each plane in 2D and stitching masks between planes whose IoU reaches stitch_threshold; 3D segmentation skips the flow-error check, which the developers found did not help on their test volume, and relies on min_size instead (documentation, 3D segmentation page). The outputs are a label image, the flows and probabilities, and, in the convolutional versions, the style vector; Cellpose-SAM returns a vector of zeros in its place.

Performance

All results below are the authors’ own, on their test splits, scored with StarDist’s matching code and $\mathrm{AP}_t = \mathrm{TP}_t/(\mathrm{TP}_t + \mathrm{FP}_t + \mathrm{FN}_t)$ averaged over test images (the “by image” option), a count-based score that the Cell Segmentation article explains and distinguishes from the average precision of detection benchmarks. StarDist and Mask R-CNN were trained by the authors on the same training sets, with 12% held out for validation, 1,000 epochs, learning rates tuned on training error (0.0007 and 0.001), ImageNet initialization for Mask R-CNN and otherwise default settings. The figures are from the bioRxiv version; the Nature Methods version could not be read.

Training and test data	Result at IoU 0.5 (Stringer et al., bioRxiv version)
Specialist models, trained and tested on the 100 CellImageLibrary neuron images	Cellpose 0.88 (485 of 521 true cells matched, 18 false positives), Mask R-CNN 0.80, StarDist 0.76
Generalist models, all test images of the cell dataset	Cellpose 0.82, Mask R-CNN 0.70, StarDist 0.68
Generalist models, test images other than CellImageLibrary	Cellpose 0.79, StarDist 0.67, Mask R-CNN 0.66
Generalist models, “microscopy: other” and “non-microscopy” test images	Cellpose 0.73, StarDist 0.54, Mask R-CNN 0.55
Generalist 2D model on a 3D test volume of mouse cortical neurons, against an ilastik pipeline tuned for the volume	183 cells annotated; the authors report that Cellpose predicted 172 cells with 17 false positives, and outperformed ilastik at all IoU thresholds

The authors report that Cellpose’s advantage grew at higher IoU thresholds, which measure how closely masks follow cell contours, and that specialist models of all three methods generalized poorly to the full dataset. The generalist Cellpose matched the specialist on the CellImageLibrary images, which they read as spare capacity for more training data. On the nucleus dataset, Cellpose led by smaller margins, which they attribute to the simpler, more uniform nuclear shapes.

For later versions, only the abstracts could be read for this article. The Cellpose 2.0 abstract reports that models pretrained on the Cellpose dataset can be fine-tuned with only 500 to 1,000 user-annotated regions of interest to perform nearly as well as models trained on entire datasets with up to 200,000, and that human-in-the-loop annotation cut the requirement to 100 to 200 (Pachitariu & Stringer, 2022). The Cellpose3 abstract reports substantial out-of-the-box gains in segmentation and image quality for noisy, blurry and undersampled images (Stringer & Pachitariu, 2025). The Cellpose-SAM abstract states that the model substantially outperforms inter-human agreement and approaches a human-consensus bound (Pachitariu et al., 2025); the preprint’s protocol was not read for this article.

An independent comparison comes from the multimodality cell segmentation challenge held at NeurIPS 2022 (Ma et al., 2024, arXiv version). On its 422-image test set of brightfield, fluorescence, phase-contrast and differential interference contrast images, scored by an F1 score at IoU 0.5, the authors report that the winning transformer-based entry’s median F1 exceeded that of the pretrained Cellpose “cyto2” model by 24.4% and that of Cellpose trained from scratch on the challenge data by 35.4%; both Cellpose variants struggled on low-contrast DIC images from new experiments. (The text gives the winner’s median as 89.7% with an interquartile range of 36.7–82.4%, which cannot both be right; an earlier paragraph gives the same median on the same 422 images with an interquartile range of 84.1–94.8%.) On an additional set of 157 yeast and bacteria images, fine-tuned Cellpose beat Cellpose trained from scratch by 12.2% but stayed below the pretrained model, which the authors attribute to catastrophic forgetting during fine-tuning.

Strengths

• A target for any shape. The flow field assigns every pixel to its cell for curved, branched and non-convex cells, where star-convex polygons and distance maps struggle, and the example above shows its reversibility and its tolerance to smoothing of the prediction.
• One model for many image types. The generalist training set and the style vector let one model segment cytoplasm, membrane, brightfield and non-biological images without parameter tuning (nuclei had a separate model); the authors’ comparisons show the largest gains on the images least like the training set.
• Few parameters at test time. The diameter, the cell-probability threshold and the flow-error threshold are the main settings, with documented defaults.
• 3D from 2D. Averaging 2D flows over three slice orientations gives 3D masks without any 3D labels.
• A software ecosystem. The package includes a graphical interface for annotation and correction, a command line and an API, human-in-the-loop training, restoration models (in Cellpose3), a module for larger-than-memory images and export to the BioImage.IO model zoo (release notes and documentation).

Limitations

• No overlaps. A label image gives each pixel one cell, and the authors do not expect Cellpose to work where occlusions require highly overlapping masks, or on images whose objects have different shapes and textures (Stringer et al., bioRxiv version).
• Long and thin cells. In our reading, a diffusion run for a fixed number of iterations from a single centre leaves little heat, and so a poorly defined flow direction, at the tips of long cells, and pixels there must travel far to converge; the documentation advises raising niter, for example to 2,000, for long objects. Omnipose was developed for bacterial cells, including elongated and branched ones, on which, its abstract states, Cellpose produces errors (Cutler et al., 2022).
• Scale. The convolutional versions assume a typical diameter (30 pixels for the cytoplasm models, 17 for nuclei) and rely on a size estimate or the user’s value. The documentation states that the Cellpose-SAM models were trained on diameters from 7.5 to 120 pixels, and it describes the diameter as optional for Cellpose-SAM.
• Image degradation. The Cellpose3 abstract states that generalist methods struggle on images degraded by noise, blur or undersampling, the problem that version addresses.
• Domain shift and fine-tuning. Pretrained models still fail on modalities far from the training data, and fine-tuning can trade general for specific performance (Ma et al., 2024, arXiv version). The Training page warns that retraining from a user’s earlier custom model biases it towards the images it was first trained on.
• 3D quality control. The flow-error check is skipped in 3D, and the documentation mentions ring-like artefacts and fragmented cells, for which it suggests flow smoothing, or plane-by-plane segmentation when the XZ and YZ slices are unsuitable.
• Licensing of models and data. The README of release v4.2.1.1 states that all Cellpose models are trained on data licensed under CC-BY-NC and that the annotated dataset is CC-BY-NC; the code itself is under a three-clause BSD licence (copyright HHMI) in the LICENSE file, which the README’s licence badge, labelled GPL v3, links to.

Historical Importance

Cellpose was presented as a generalist for whole-cell segmentation, trained once on a deliberately varied dataset rather than per experiment, following the example the Data Science Bowl had set for nuclei (Stringer et al., bioRxiv version). Its flow representation was adopted by others: the winning entry of the NeurIPS 2022 challenge predicted cell probabilities and horizontal and vertical gradient flows with gradient-tracking post-processing, which Ma et al. (2024, arXiv version) describe as originally proposed in Cellpose, and Omnipose kept the framework with a different flow. In our reading, its later versions track two shifts in the field: from a fixed pretrained model to models that users adapt with a few hundred annotations (Cellpose 2.0), and from convolutional networks trained from scratch to pretrained transformer backbones (Cellpose-SAM). The Bioimage Analysis article places it, with StarDist, among the deep-learning methods adopted in microscopy.

Variants

Release dates below are the dates of the GitHub releases (MouseLand release notes), as of November 2026.

• Cellpose (1), 2020. Preprint in February 2020, public code with 3D segmentation in release v0.0.1.25 (2 April 2020), PyTorch as default in v0.6 (21 December 2020), version 1.0 in January 2022. The built-in models were “cyto” and “nuclei”, later joined by “cyto2”, which Ma et al. (2024, arXiv version) describe as trained on the Cellpose dataset and user-submitted images.
• Omnipose, 2021–2022. Developed with the Wiggins and Mougous labs and first released inside the Cellpose package (v0.7.2, November 2021), Omnipose targets bacteria and other cells of extreme shape; its abstract names the gradient of the distance field among its outputs (Cutler et al., 2022).
• Cellpose 2.0, 2022. Released in April 2022 (v2.0.4), it added a model zoo of pretrained models and human-in-the-loop training in the graphical interface: run the best zoo model on an image, correct its masks, train, and let the new model segment the next image (documentation, GUI page). Ma et al. (2024, arXiv version) state that the network was the same as in the original Cellpose.
• Cellpose3, 2024–2025. First released in February 2024 (v3.0.1), with the paper in 2025. It trains restoration networks to produce images that a generalist segmentation model segments well while staying perceptually similar to the target images, instead of training them to restore pixel values (Stringer & Pachitariu, 2025, abstract). The documentation lists denoising, deblurring, upsampling and “one-click” models trained on all degradations, each in versions trained on the cyto3, cyto2 and nucleus training sets; it relates to image denoising, deblurring and super-resolution. The Cellpose 3 code (release v3.1.1.2) lists the “cyto3” segmentation model, which it calls a super-generalist, alongside dataset-specific models such as “tissuenet_cp3”, “livecell_cp3” and “bact_phase_cp3”; the README asks users of cyto3 to cite the Cellpose3 paper. Release v3.1.1 (February 2025) added a distributed module for larger-than-memory images. Restoration is available only in Cellpose3, which the documentation says to install as version 3.1.1.2.
• Cellpose-SAM (Cellpose 4), 2025. Released in May 2025 (v4.0.4). The network is the ViT-L image encoder of SAM, a vision transformer, adapted to the Cellpose outputs: in release v4.2.1.1 its patches are $8 \times 8$ pixels instead of $16 \times 16$ (the patch-embedding weights subsampled), its position embeddings are resampled for $256 \times 256$ tiles, every block uses global attention, and a $1 \times 1$ convolution followed by a fixed transposed convolution maps each token back to an $8 \times 8$ patch of the three output maps (CPSAM in cellpose/vit.py). The abstract describes training for robustness to channel shuffling, cell size, shot noise, downsampling and blur (Pachitariu et al., 2025), and the documentation states that the model takes the first three channels in any order. The release removed the Cellpose class and the size model.
• Cellpose 4 models of June 2026. Release v4.2.1.1 (14 June 2026) added “cpdino” and “cpdino-vitb”, built on the ViT-L and ViT-B backbones of DINOv3 with $384 \times 384$ tiles by default, and “cpsam_v2”, a retrained Cellpose-SAM that the README says predicts fewer spurious masks in low-contrast regions; “cpsam_v2” is the default (documentation, Models page; cellpose/models.py).

Implementations

Cellpose is distributed as the Python package cellpose on the Python Package Index, from the MouseLand GitHub repository, with a graphical interface, a command line and an API; the latest release as of November 2026 is 4.2.1.1 (14 June 2026). A minimal call follows the documented API of release 4.2.1.1, which is not run here because the package and its weights were not installed for this article:

from cellpose import models
from cellpose.io import imread

model = models.CellposeModel(gpu=True)            # default pretrained model "cpsam_v2"
img = imread("img.tif")
masks, flows, styles = model.eval(img, flow_threshold=0.4, cellprob_threshold=0.0)

Weights are downloaded on first use. The documentation describes fine-tuning from the command line or with train.train_seg, 3D segmentation with do_3D=True and the restoration models of Cellpose3 in the denoise module of version 3.1.1.2. Cellpose models can also be shared through the BioImage.IO model zoo. The Fluorescence Microscopy and Bioimage Analysis articles describe the imaging and the wider tool ecosystem in which Cellpose is used.