Object Counting

Object counting estimates how many instances of an object category an image contains: people in a crowd, cars in a car park, cells under a microscope, birds on a lake. The answer is a number, and often also a density map that says where the objects are and how many lie in any region. The task is related to object detection, since a detector’s boxes can simply be counted, but counting asks for less than detection and is often solved without finding any individual object. That matters most where objects overlap and are small: in a dense crowd photographed from a distance, the visible part of most people is a few pixels of head, and no detector separates them reliably.

Crowd counting, counting people in still images or video frames, is the variant with the largest literature and the main public benchmarks; when the output is a map rather than a number, it is also described as crowd density estimation. Counting cells and other small objects in microscopy shares the methods and is treated with cell detection. Class-agnostic (or few-shot) counting counts objects of a category the model was not trained on, given a few examples marked in the image.

Problem Definition

Given an image $I$, predict the number $C$ of instances of a target category in it, or in a region of interest. Training data are usually dot annotations: one point per object, typically at the centre of each person’s head in crowd datasets (Ma et al., 2019). Lempitsky and Zisserman (2010) argued that dotting is as cheap for an annotator as counting, because people count large numbers by pointing, while the dots carry spatial information that a bare count does not; dots are also less laborious than boxes or masks.

The usual intermediate output is a density map $D$ over the pixel grid, defined so that its integral over any region $S$ estimates the number of objects in $S$:

$$C(S) = \sum_{p \in S} D(p), \qquad C = \sum_{p \in I} D(p).$$

This is the formulation of Lempitsky and Zisserman (2010). The count of a whole image is the sum of the map, and local counts come from summing over windows. Unlike a detection, a density map does not commit to individual objects: half a person at the edge of a window contributes about half to its count.

Input and Output

The input is a single image (most benchmarks) or a video frame; some early systems used motion from neighbouring frames (Chan, Liang & Vasconcelos, 2008). A region of interest may restrict where objects are counted, as on UCSD and WorldExpo’10. In class-agnostic counting the input also includes a few exemplar boxes drawn around instances of the category to be counted (Ranjan et al., 2021).

The output is a count per image, usually derived from a predicted density map. Some methods also output head locations, from which counts can be read and errors corrected by a person (Idrees et al., 2018).

Variants

• Crowd counting in still images with arbitrary viewpoints and densities, from a few people to over ten thousand per image (Idrees et al., 2018).
• Counting in surveillance video from a fixed camera, where a perspective map of the scene can be measured once and motion segmentation is available (Chan, Liang & Vasconcelos, 2008; Zhang et al., 2015). Zhang et al. (2015) addressed cross-scene counting, in which a model trained on some scenes counts in a new scene without new annotations.
• Other categories. Vehicles in traffic cameras (TRANCOS, evaluated by Li, Zhang & Chen, 2018) and parked cars in drone images (CARPK, evaluated by Ranjan et al., 2021) and cells (Lempitsky & Zisserman, 2010) are counted with the same methods, usually one category per model; the MCNN authors also name animal counts in wildlife sanctuaries (Zhang et al., 2016).
• Localization: predict a point per object as well as the count, scored by matching points (Idrees et al., 2018).

Challenges

• Occlusion and overlap. In dense crowds most people are partly hidden, which, in the MCNN authors’ account, is what limits detection-based counting (Zhang et al., 2016).
• Scale and perspective. The same scene contains heads tens of pixels across near the camera and a few pixels far away. Crowd images from the web come without camera geometry, so perspective must be inferred from the image (Zhang et al., 2016).
• Range of counts. UCF-QNRF images hold 49 to 12,865 people (Idrees et al., 2018), and a model must handle both ends; errors on the densest images dominate the averages.
• Annotation ambiguity. A dot marks one pixel of a person who covers an area, and annotators place it inconsistently (Ma et al., 2019). Where heads become indistinguishable in the distance, annotators stop marking them (Idrees et al., 2018).
• Background and distractors. Models trained only on crowd images can predict people in scenes with none, which led JHU-CROWD to include distractor images without crowds (Sindagi, Yasarla & Patel, 2019).
• Image size. UCF-QNRF images range from 0.08 to 66 megapixels (Ma et al., 2019), far beyond what a network processes at once, so resizing and tiling conventions enter the results.

Approaches

Lempitsky and Zisserman (2010) grouped earlier supervised methods into counting by detection and counting by regression, with counting by segmentation as a hybrid, and proposed density estimation as a third approach; later crowd counting papers use the same three families (Li, Zhang & Chen, 2018; Ma et al., 2019; Wang et al., 2020).

Counting by detection

A detector localizes each instance and the count is the number of detections. Lempitsky and Zisserman (2010) noted that most detectors of their time first produce a real-valued confidence map and then threshold it and apply non-maximum suppression to find peaks, and that detection was far from solved for overlapping instances; in their figure, an SVM detector’s confidence map shows six clear peaks in one window of a cell image and an unclear number in a denser one. Detection-based crowd counting used whole-body detectors with features such as Haar wavelets and HOG and, because most bodies are hidden in crowded scenes, detectors of particular body parts (Li, Zhang & Chen, 2018). Detection also needs box annotation to train, which is laborious and ambiguous in dense crowds (Ma et al., 2019; Wang et al., 2020).

Detection remains a strong choice when objects are separated and the positions are needed. Crowded-scene detectors change the suppression step or the loss to keep overlapping boxes, as described for pedestrian detection. In class-agnostic counting, Ranjan et al. (2021) compared their density-based FamNet with COCO-trained Faster R-CNN, Mask R-CNN and RetinaNet on the FSC-147 images of categories those detectors were trained on (282 test images): MAE 36.20 for Faster R-CNN, 35.56 for Mask R-CNN, 52.67 for RetinaNet and 22.76 for FamNet.

Counting by regression

Regression methods map global or per-segment image features directly to a count, avoiding detection. Chan, Liang and Vasconcelos (2008) segmented a surveillance video into crowd regions moving in different directions with a mixture of dynamic textures, extracted features of each segment’s shape and size, its internal Canny edges and its grey-level co-occurrence texture, weighted pixels by a perspective map measured from a reference person at the front and back of the walkway, and regressed the number of people per segment with a Gaussian process whose kernel combines a linear and an RBF term. They framed the design as privacy preserving: nothing in it detects or tracks individuals. Other systems used ridge regression on local binary patterns or texture features (Zhang et al., 2016). Idrees et al. (2013) combined counts from three sources in image patches, HOG-based head detections, the repetition of texture elements found with SIFT, and frequency-domain analysis, and made the patch counts consistent across neighbourhoods and scales with a Markov random field, for crowds of hundreds to thousands.

Lempitsky and Zisserman (2010) pointed out the weakness of the family: it uses only the total count of each training image and discards the positions of the dots, and so needs many training images. In the deep era, direct count regression was tested against density maps in the same network. On ShanghaiTech Part A, the MCNN architecture trained to regress the count (its loss compares the summed output with the true count) had MAE 245.0, against 110.2 when trained on density maps (Zhang et al., 2016). On UCF-QNRF, a DenseNet-201 regressing counts had MAE 163, against 132 for the same base network with the authors’ density and localization branches (Idrees et al., 2018).

Counting by density maps

Lempitsky and Zisserman (2010) learned the density from dots. The ground-truth density of a training image with dots $P_1, \dots, P_C$ is a sum of normalized Gaussians,

$$D^{\mathrm{gt}}(p) = \sum_{c=1}^{C} \mathcal{N}\!\left(p;\, P_c,\, \sigma^2 \mathbf{1}_{2\times 2}\right),$$

each integrating to one, so the map sums to the count. They modelled the density as linear in per-pixel features, $D(p) = \mathbf{w}^\top \mathbf{x}_p$, and fitted $\mathbf{w}$ with a loss, the MESA distance, that compares the two densities’ sums over all boxes in the image and so tolerates local differences in how the mass is spread; the cell detection article describes the loss and their cell experiments. On the UCSD pedestrian video they reported MAE 1.70 on the “maximal” split of Ryan et al., which trains on every fifth frame from 600 to 1400, competitive with a hybrid segmentation method that needed more detailed annotation.

The Gaussian target resembles the one in heatmap regression, with a difference that matters: a keypoint heatmap has peak value 1 and is read at its maximum, while a density map’s Gaussians have unit mass and the map is read by summing. Overlapping Gaussians add, so a dense region becomes a smooth plateau whose height encodes how many objects it holds, without any need to separate them.

Ground-truth density maps and geometry-adaptive kernels

The spread $\sigma$ decides how the unit mass of each person is spread over the image. With one fixed $\sigma$, a person near the camera and one far away get the same blob, although their heads differ greatly in size. Zhang et al. (2015) used a perspective map $M(p)$, the number of pixels per metre at each location measured for each surveillance scene, to set the kernels, and their density kernel combined a Gaussian for the head with a body term. For web images with no perspective map, Zhang et al. (2016) proposed geometry-adaptive kernels. Assuming the crowd is roughly evenly distributed around each head, the average distance from head $x_i$ to its $k$ nearest neighbours, $\bar d_i$, indicates the local scale, and

$$D^{\mathrm{gt}}(x) = \sum_{i=1}^{N} \delta(x - x_i) * G_{\sigma_i}(x), \qquad \sigma_i = \beta\, \bar d_i ,$$

with $\beta = 0.3$ found best empirically. The MCNN paper does not state $k$; CSRNet’s authors, “following the configuration” of MCNN, give $k = 3$ and call $\sigma_i$ the Gaussian’s standard deviation (Li, Zhang & Chen, 2018), while the MCNN text speaks of a “variance” proportional to $\bar d_i$ before writing $\sigma_i = \beta \bar d_i$. MCNN used geometry-adaptive kernels for ShanghaiTech Part A, a fixed spread for the sparser Part B and for UCSD, and the perspective-based $\sigma = 0.2\,M(x)$ for WorldExpo’10. CSRNet’s table of ground-truth settings uses geometry-adaptive kernels for Part A and UCF_CC_50 and fixed $\sigma = 15$ for Part B, $\sigma = 10$ for TRANCOS and $\sigma = 3$ for WorldExpo’10 and UCSD.

Later work changed the rule. Idrees et al. (2018) set each person’s bandwidth to the distance to the nearest neighbour, capped at a maximum $\tau$, and trained on a series of maps sharpened by $f_k(\sigma) = \sigma^{1/k}$, from a smooth density ($k = 1$) to the dot map itself ($k \to \infty$). Ranjan et al. (2021) used a Gaussian window as large as the mean nearest-neighbour distance in the image, with standard deviation a quarter of the window. These choices are tuning decisions with large effects: in the Bayesian loss paper, the pixel-wise baseline’s MAE on UCF-QNRF varied from 118.4 to 136.2 as its Gaussian’s $\sigma$ changed between 0.1 and 32 (Ma et al., 2019). Two later losses therefore stopped regressing a smoothed target pixel by pixel (below).

Crowd counting networks

Zhang et al. (2015) trained one of the first convolutional networks for crowd counting, alternating between a density-map objective and a count objective, and adapted it to new scenes by fine-tuning on training patches selected by similar perspective and density; they introduced WorldExpo’10, with 108 surveillance scenes and nearly 200,000 annotated heads.

MCNN (Zhang et al., 2016) maps an image of any size to a density map with three parallel columns whose filters have large, medium and small receptive fields, so that each column responds to heads of a different size. Each column has four convolutional layers with two $2 \times 2$ max-pooling steps, fewer filters in the columns with larger kernels, and ReLU activations; the columns’ feature maps are stacked and merged into the density map by a $1 \times 1$ convolution, so the output has a quarter of the input’s width and height and the network has no fully connected layer. The loss is the Euclidean distance between predicted and ground-truth density maps. Each column was first pretrained alone and the whole network then fine-tuned. On ShanghaiTech Part A, single columns reached MAE 141.2 to 160.5 against 110.2 for the full MCNN, and MCNN without column-wise pretraining 122.8 (the paper’s Figure 6). The paper also introduced the ShanghaiTech dataset.

CSRNet (Li, Zhang & Chen, 2018) questioned the multi-column design. Its authors found that MCNN’s three columns gave similar errors across 50 Part A test images, which they read as the columns learning nearly identical features, and showed that a deeper single-column network with fewer parameters (83.84k against 127.68k) did better on Part A. CSRNet uses the first ten convolutional layers of VGG-16, with three pooling layers, as a front end, giving maps at one eighth of the input resolution, and adds a back end of six $3 \times 3$ convolutions with dilation instead of further pooling, followed by a $1 \times 1$ output layer. A dilated convolution with rate $r$ samples its input $r$ pixels apart,

$$y(m, n) = \sum_{i=1}^{M} \sum_{j=1}^{N} x(m + r i,\, n + r j)\, w(i, j),$$

so a $k \times k$ kernel covers $k + (k-1)(r-1)$ pixels per side, enlarging the receptive field without extra parameters or loss of resolution. Of four back ends with the same 16.26 million parameters, dilation 2 throughout (configuration B) did best on Part A, with MAE 68.2, against 69.7 with no dilation, 71.91 with rates 2 and 4 combined and 75.81 with rate 4 throughout. Training used the Euclidean loss, stochastic gradient descent at a fixed learning rate of $10^{-6}$ and nine quarter-size patches per image plus their mirror images. The paper reports results on ShanghaiTech, UCF_CC_50, WorldExpo’10 and UCSD, and vehicle counting on TRANCOS.

Losses for point supervision

The smoothed “ground truth” is not an annotation but a guess about how each person’s mass is spread. Three lines of work loosened the dependence on it.

Idrees et al. (2018) supervised counting, density maps at several sharpness levels and a localization map together, with a composition loss that combines the mean squared errors of the maps and the count error of a regression branch, on a DenseNet-201 base.

The Bayesian loss (Ma et al., 2019) treats each dot as a class label and the Gaussian as a likelihood rather than a target. With likelihood $p(x_m \mid y_n) = \mathcal{N}(x_m; z_n, \sigma^2 \mathbf{1}_{2\times 2})$ around head $z_n$ and equal priors, the posterior that pixel $x_m$ belongs to person $n$ is

$$p(y_n \mid x_m) = \frac{\mathcal{N}(x_m;\, z_n,\, \sigma^2 \mathbf{1}_{2\times 2})}{\sum_{n'=1}^{N} \mathcal{N}(x_m;\, z_{n'},\, \sigma^2 \mathbf{1}_{2\times 2})},$$

and the expected count attributed to person $n$ under the predicted density $D^{\mathrm{est}}$ is $E[c_n] = \sum_m p(y_n \mid x_m)\, D^{\mathrm{est}}(x_m)$. Each person should account for exactly one, so the loss is $\sum_n \lvert 1 - E[c_n] \rvert$ with an $\ell_1$ distance; because the posteriors sum to one over persons at every pixel, the sum of the $E[c_n]$ equals the sum of the predicted map, and no posterior is needed at test time. The enhanced version adds a background label whose likelihood is a Gaussian around a dummy point placed at distance $d$ beyond the nearest head, and asks the background’s expected count to be zero. With a VGG-19 backbone, $\sigma = 8$ and $d$ set to 15% of the image’s shorter side, chosen on 120 UCF-QNRF training images held out for validation, the enhanced Bayesian loss reduced MAE on UCF-QNRF from 106.8 for the same network trained with a pixel-wise loss on geometry-adaptive targets to 88.7. In their ablation, the basic Bayesian loss’s MAE stayed below 98.0 for every $\sigma$ from 0.1 to 32.

DM-Count (Wang et al., 2020) drops the Gaussian altogether. It treats the dot map $\mathbf{z}$ and the prediction $\hat{\mathbf{z}}$ as unnormalized distributions and combines three terms:

$$\ell(\mathbf{z}, \hat{\mathbf{z}}) = \bigl\lvert \lVert \mathbf{z} \rVert_1 - \lVert \hat{\mathbf{z}} \rVert_1 \bigr\rvert + \lambda_1\, W\!\left(\frac{\mathbf{z}}{\lVert \mathbf{z} \rVert_1}, \frac{\hat{\mathbf{z}}}{\lVert \hat{\mathbf{z}} \rVert_1}\right) + \lambda_2\, \lVert \mathbf{z} \rVert_1 \cdot \frac{1}{2}\left\lVert \frac{\mathbf{z}}{\lVert \mathbf{z} \rVert_1} - \frac{\hat{\mathbf{z}}}{\lVert \hat{\mathbf{z}} \rVert_1} \right\rVert_1 ,$$

a counting loss, an optimal transport cost $W$ between the normalized maps with squared Euclidean distance between pixel positions as the transport cost, and a total variation distance between them, scaled by the count. The transport problem is solved approximately with the Sinkhorn algorithm (100 iterations, entropic regularization 10), and the TV term, the authors explain, corrects the approximate transport in sparse regions and stabilizes training; $\lambda_1 = 0.1$ and $\lambda_2 = 0.01$ throughout. The authors argue that the Bayesian loss is underdetermined, since it imposes one constraint per person on a map with many more pixels, so that many maps unlike the annotation give zero loss, and they derive generalization bounds that are tighter without Gaussian smoothing. With the Bayesian-loss network, DM-Count reported MAE 85.6 on UCF-QNRF. The paper’s text says DM-Count beats every other method in its tables except CAN on the NWPU RMSE; its Table 1 also shows CAN with a lower UCF-CC-50 RMSE (243.7 against 291.5), and its Table 2 shows CSRNet and CAN with NWPU test RMSEs (387.8 and 386.5) slightly below DM-Count’s 388.6.

Class-agnostic counting

Counting networks are usually trained for one category. Ranjan et al. (2021) posed counting as a few-shot regression task: given an image and a few exemplar boxes around instances of a category not seen in training, predict a density map of all instances. Their FamNet extracts features with the first four blocks of an ImageNet-pretrained ResNet-50, kept frozen, pools exemplar features from the third and fourth blocks with RoI pooling, and correlates the exemplar features, at scales 0.9, 1.0 and 1.1, with the image features. Only the correlation maps enter the density prediction module, five convolutional blocks with three upsampling layers, so that the module does not depend on the category. At test time the density module is adapted to each image with 100 gradient steps on two losses defined by the exemplar boxes: a min-count loss, which asks the density inside each exemplar box to sum to at least one, and a perturbation loss, which asks the density around each exemplar to look like a Gaussian. Lu and Zisserman’s Generic Matching Network, pretrained on tracking video, was the earlier class-agnostic counter they compared against. The task borrows its protocol, novel classes at test time from a few examples, from few-shot recognition.

Example: Counting by Detection Against Integrating a Density Map

The script draws synthetic crowds of Gaussian blobs (standard deviation 2.5 pixels, amplitudes 0.7 to 1.3, centres uniform over a $128 \times 128$ image, background 0.1 with noise of standard deviation 0.05) at 10 to 400 objects per image, so that for uniformly scattered centres the expected distance to the nearest neighbour falls from about 20 pixels to about 3 (our arithmetic). It counts them in three ways. Detection smooths the image with a matched Gaussian and counts the local maxima found by peak_local_max, which is non-maximum suppression over a response map, with its threshold and minimum distance chosen by grid search on training images. The density method fits a linear map from seven per-pixel features (a constant, the image smoothed at four scales, and the squared image smoothed at two) to a target density of unit-mass Gaussians at the dots, by ridge least squares over all training pixels, in the spirit of Lempitsky and Zisserman (2010) but with a squared pixel loss instead of MESA, and counts by summing the predicted map. Global regression fits the count directly from the image-level means of the same features. Training uses 5 images per density and testing 20, with different seeds.

Two choices are left open by the task, so the script runs each both ways. Overlapping blobs are composited either additively, as transparent objects such as fluorescent cells would be, or by taking the maximum, a crude stand-in for opaque objects hiding each other. The detector is tuned either on the sparsest training images only, as a detector designed on easy data would be, or on training images of all densities. For each density the script prints MAE, RMSE and the mean signed error (bias, negative for undercounting), then repeats the experiment for seeds 0 to 4 and prints the range of MAE at 400 objects. It runs in about a minute and a half with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2.

import numpy as np
from scipy import ndimage as ndi
from skimage.feature import peak_local_max

H = W = 128
SD_OBJ, SD_DOT = 2.5, 2.0
DENSITIES = [10, 25, 50, 100, 200, 400]
N_TRAIN, N_TEST = 5, 20


def make_image(n, rng, mode):
    ys, xs = rng.uniform(0, H, n), rng.uniform(0, W, n)
    amp = rng.uniform(0.7, 1.3, n)
    yy, xx = np.mgrid[0:H, 0:W]
    img = np.zeros((H, W))
    for y, x, a in zip(ys, xs, amp):
        blob = a * np.exp(-((yy - y) ** 2 + (xx - x) ** 2) / (2 * SD_OBJ ** 2))
        img = img + blob if mode == "additive" else np.maximum(img, blob)
    img += 0.1 + rng.normal(0, 0.05, (H, W))
    return img, np.column_stack([ys, xs])


def make_set(seed, mode, per_density):
    rng = np.random.default_rng(seed)
    return [(n,) + make_image(n, rng, mode) for n in DENSITIES for _ in range(per_density)]


def target_density(dots):
    d = np.zeros((H, W))
    iy = np.clip(dots[:, 0].astype(int), 0, H - 1)
    ix = np.clip(dots[:, 1].astype(int), 0, W - 1)
    np.add.at(d, (iy, ix), 1.0)
    return ndi.gaussian_filter(d, SD_DOT, mode="constant")  # unit mass per dot, minus border loss


def features(img):
    f = [np.ones_like(img)]
    f += [ndi.gaussian_filter(img, s, mode="reflect") for s in (1, 2, 4, 8)]
    f += [ndi.gaussian_filter(img ** 2, s, mode="reflect") for s in (2, 4)]
    return np.stack(f, axis=-1).reshape(-1, len(f))


def ridge(X, y, lam):
    return np.linalg.solve(X.T @ X + lam * np.eye(X.shape[1]), X.T @ y)


def detect_count(img, md, thr):
    resp = ndi.gaussian_filter(img, SD_OBJ, mode="reflect")
    return len(peak_local_max(resp, min_distance=md, threshold_abs=thr, exclude_border=False))


def tune_detector(train):
    resp = np.concatenate([ndi.gaussian_filter(im, SD_OBJ, mode="reflect").ravel() for _, im, _ in train])
    thrs = np.percentile(resp, np.linspace(50, 99.9, 25))
    best = None
    for md in range(1, 7):
        for t in thrs:
            mae = np.mean([abs(detect_count(im, md, t) - n) for n, im, _ in train])
            if best is None or mae < best[0]:
                best = (mae, md, t)
    return best[1], best[2]


def run(seed, mode):
    train = make_set(1000 + seed, mode, N_TRAIN)
    test = make_set(seed, mode, N_TEST)
    # density map: per-pixel ridge regression to the Gaussian-smoothed dot density
    X = np.concatenate([features(im) for _, im, _ in train])
    y = np.concatenate([target_density(d).ravel() for _, _, d in train])
    w_den = ridge(X, y, 1e-3 * len(y))
    # global regression: image-level feature means -> count
    G = np.array([features(im).mean(0) for _, im, _ in train])
    c = np.array([n for n, _, _ in train], float)
    w_glob = ridge(G, c, 1e-3)
    # detection, two tuning choices
    sparse = [s for s in train if s[0] == DENSITIES[0]]
    det_params = {"det(sparse)": tune_detector(sparse), "det(all)": tune_detector(train)}
    preds = {k: [] for k in ["det(sparse)", "det(all)", "density", "global"]}
    truth = []
    for n, im, _ in test:
        F = features(im)
        truth.append(n)
        preds["density"].append(float((F @ w_den).sum()))
        preds["global"].append(float(F.mean(0) @ w_glob))
        for k, (md, t) in det_params.items():
            preds[k].append(detect_count(im, md, t))
    return np.array(truth), {k: np.array(v, float) for k, v in preds.items()}, det_params


def stats(truth, pred, n):
    e = pred[truth == n] - n
    return np.abs(e).mean(), np.sqrt((e ** 2).mean()), e.mean()


if __name__ == "__main__":
    for mode in ("additive", "max"):
        truth, preds, dp = run(0, mode)
        print(f"== {mode} compositing, seed 0 ==")
        for k, (md, t) in dp.items():
            print(f"  {k}: min_distance={md}, threshold={t:.3f}")
        print(f"  {'N':>4} " + " ".join(f"{k:>22}" for k in preds))
        print(f"  {'':>4} " + " ".join(f"{'MAE  RMSE  bias':>22}" for _ in preds))
        for n in DENSITIES:
            row = []
            for k in preds:
                mae, rmse, bias = stats(truth, preds[k], n)
                row.append(f"{mae:6.1f} {rmse:6.1f} {bias:+7.1f}")
            print(f"  {n:4d} " + " ".join(f"{r:>22}" for r in row))
    print("== MAE at N = 400 over seeds 0-4 (min-max) ==")
    for mode in ("additive", "max"):
        res = {}
        for seed in range(5):
            truth, preds, _ = run(seed, mode)
            for k in preds:
                res.setdefault(k, []).append(stats(truth, preds[k], 400)[0])
        print(f"  {mode:>8}: " + "  ".join(f"{k} {min(v):.1f}-{max(v):.1f}" for k, v in res.items()))

== additive compositing, seed 0 ==
  det(sparse): min_distance=1, threshold=0.127
  det(all): min_distance=1, threshold=0.201
     N            det(sparse)               det(all)                density                 global
              MAE  RMSE  bias        MAE  RMSE  bias        MAE  RMSE  bias        MAE  RMSE  bias
    10     0.6    1.0    -0.5     0.5    0.9    -0.5     2.3    2.3    -2.3     0.6    0.6    +0.3
    25     2.5    3.1    -2.5     2.8    3.2    -2.8     1.8    2.1    -1.8     0.8    1.0    +0.1
    50    11.7   12.0   -11.7    11.8   12.2   -11.8     1.6    1.9    -1.3     1.1    1.5    -0.4
   100    44.8   45.0   -44.8    44.9   45.0   -44.9     1.5    1.8    -0.7     1.6    1.9    -1.0
   200   131.6  131.6  -131.6   131.6  131.6  -131.6     2.8    3.5    -1.5     2.9    3.6    -1.5
   400   331.4  331.5  -331.4   331.4  331.5  -331.4    12.9   13.2   -12.9     3.2    3.8    -1.2
== max compositing, seed 0 ==
  det(sparse): min_distance=1, threshold=0.127
  det(all): min_distance=1, threshold=0.191
     N            det(sparse)               det(all)                density                 global
              MAE  RMSE  bias        MAE  RMSE  bias        MAE  RMSE  bias        MAE  RMSE  bias
    10     0.5    0.9    -0.4     0.5    0.9    -0.5    11.9   12.0   -11.9     4.0    4.2    -4.0
    25     2.1    2.5    -2.1     2.3    2.7    -2.3     4.8    5.3    -4.8     3.3    4.2    -2.6
    50     9.7   10.1    -9.7     9.8   10.2    -9.8     4.0    4.5    +3.6     3.3    4.1    -1.6
   100    35.3   35.6   -35.3    35.4   35.6   -35.4    14.5   14.9   +14.5     3.9    4.7    +2.5
   200   110.3  110.4  -110.3   110.3  110.4  -110.3    16.1   17.3   +16.1    10.0   11.9    +7.3
   400   293.6  293.7  -293.6   293.6  293.7  -293.6    36.6   37.1   -36.6     9.6   11.9    -8.6
== MAE at N = 400 over seeds 0-4 (min-max) ==
  additive: det(sparse) 329.9-332.4  det(all) 329.9-332.4  density 9.9-14.8  global 2.9-4.6
       max: det(sparse) 292.6-294.8  det(all) 292.6-294.8  density 33.1-41.7  global 8.6-16.1

Detection is nearly exact with 10 isolated blobs and then loses objects at a growing rate: the bias equals the MAE at every density from 25 up, so every error is a miss, and at 400 objects it finds about a sixth of them under additive compositing and about a quarter under max. Blobs closer than about their own width merge into one maximum of the smoothed image, and no threshold or suppression radius can split a peak that is not there (our reasoning); both tuning rules settled on the smallest suppression distance, 1 pixel, and differ only in the threshold, which hardly matters once the misses come from merged peaks. The plan expected that “Tuning on all densities reduces the high-density undercount at the cost of overcounting at low density”; it did neither. Detection loses fewer blobs under max compositing than under additive compositing, which we attribute to the dip that remains between two opaque blobs, where additive blobs sum into a single hump sooner (our reasoning).

The two methods that never separate objects make errors of at most about a seventh of detection’s at 200 and 400 objects, where detection misses most of them, which is the case for density-based counting made by Lempitsky and Zisserman (2010) and the crowd counting papers. They are not free of error. The plan predicted “small error at all densities” for the density map under additive compositing; at seed 0 it undercounts by 2.3 objects at 10 and by 12.9 at 400, and its MAE at 400 ranges from 9.9 to 14.8 over the five seeds. The least squares fit minimizes squared errors per pixel, not count errors per image, so a model that cannot reproduce the target exactly is not constrained to sum correctly, and the target loses part of the mass of dots near the border while the features do not (our reasoning). The global regressor, fitted to the counts themselves, is more accurate here than the density map at most densities, against the plan, which expected it to be “close to density counting in count accuracy”. With a linear per-pixel model the predicted count is a linear function of the summed features, so the two methods use the same information and differ only in how the weights are fitted; this small example, with seven features and 30 training images, does not test Lempitsky and Zisserman’s argument that global regression needs many more training images. Under max compositing both learned counters degrade, and the density map’s bias changes sign with density, because a linear map of these features cannot follow how opaque overlap removes intensity as objects pile up (our reasoning). Real crowd counters use deep networks for that nonlinear relation, and losses with explicit count terms, such as the MESA distance, the Bayesian loss and DM-Count’s counting loss, target the count error that the pixel loss leaves free.

Datasets and Benchmarks

Crowd datasets contain images of people, mostly collected from the web or from surveillance cameras. The terms below are taken from the official pages as of November 2026, where those pages could be read.

UCSD pedestrian dataset (Chan, Liang & Vasconcelos, 2008). The first 2,000 frames of an hour of video from a stationary camera overlooking a walkway at UC San Diego, downsampled to $238 \times 158$ pixels at 10 frames per second. In a region of interest, the visible centre of each pedestrian, and the direction of travel, were annotated every fifth frame and interpolated linearly in between, with cyclists and skateboarders counted as pedestrians: 49,885 pedestrian instances, at most 46 in a frame. The training set is the 800 frames from frame 600 to 1399 and the other 1,200 frames are the test set, a split chosen to test extrapolation rather than interpolation. The original paper counted the two directions of travel separately and reported the mean squared error and the mean absolute error per direction. Later work counts all people in the region, keeps the split (written as frames 601 to 1400 by MCNN and CSRNet) and reports MAE and root mean squared error (Zhang et al., 2016; Li, Zhang & Chen, 2018). Lempitsky and Zisserman (2010) used the four splits of Ryan et al. and two of their own. The project page offers the video and the counting annotations, and the database readme (2008) asks users to cite Chan and Vasconcelos’s 2008 paper on mixtures of dynamic textures; neither states licence terms.

UCF_CC_50 (Idrees et al., 2013). Fifty images of very dense crowds from the web, including Flickr, of concerts, protests, stadiums, marathons and pilgrimages, with 94 to 4,543 people each and about 1,280 on average; one image is a painting and another an abstract depiction of a crowd. The paper’s text gives 63,705 annotations and its abstract 64K, while later dataset tables give 63,974 (Zhang et al., 2016; Idrees et al., 2018); the UCF-QNRF paper gives a range of 96 to 4,633, and the Bayesian loss paper a maximum of 4,532. There is no fixed split; results are five-fold cross-validation over random sets of ten images. The UCF-QNRF authors showed parts of these images where heads were left unannotated or marked wrongly, which they attribute to the limited resolution of crowd images available at the time (Idrees et al., 2018).

ShanghaiTech (Zhang et al., 2016). 1,198 images with 330,165 annotated head centres, in two parts. Part A has 482 images crawled from the Internet, mostly very dense (33 to 3,139 people, mean 501.4), split into 300 training and 182 test images. Part B has 716 images of $768 \times 1024$ pixels taken in busy streets of Shanghai (9 to 578 people, mean 123.6), split into 400 training and 316 test images. The authors state that no two images were taken from the same viewpoint, and the faces in their figure of Part B examples are blurred for privacy. Results are reported for Parts A and B separately. As of November 2026 the dataset is distributed through a GitHub repository of the second author, whose README links download mirrors; the repository carries a BSD 2-Clause licence file (copyright 2017, Desen Zhou), and the README states no separate terms for the images.

UCF-QNRF (Idrees et al., 2018). 1,535 high-resolution images (average $2013 \times 2902$ pixels) with 1,251,642 head annotations, collected from Flickr, web image search and footage of the Hajj, with 49 to 12,865 people per image (median 425, mean 815.4). Flickr images were downloaded only where the original resolution could be downloaded; images with low-density crowds, motion blur or low resolution, cameras at about the height of a person, watermarks or text covering more than 10% of the image were pruned, and regions where heads could not be distinguished were cropped out. Fourteen annotators and four verifiers spent about 2,000 hours in all. The split sorts the images by count and puts every fifth into the test set, giving 1,201 training and 334 test images. The paper proposed counting measures (C-MAE, C-MSE and a normalized C-NAE), per-pixel density measures and histogram intersection, and a localization score: predictions are matched one to one to annotations by greedy association, precision and recall are computed at distance thresholds from 1 to 100 pixels, and the area under the curve is reported (L-AUC); the paper’s localization table averages over four distance thresholds. The dataset’s official page at the University of Central Florida could not be read for this article (its TLS certificate could not be verified), so its terms are not described.

JHU-CROWD and JHU-CROWD++ (Sindagi, Yasarla & Patel, 2019; 2020). The first version (ICCV 2019) has 4,250 images with 1,114,785 head annotations, collected from the web with queries such as “crowd+marathon”, including images degraded by rain, haze and snow and 100 distractor images without crowds, and annotates every head with an occlusion level (unoccluded, partly or fully occluded), a blur flag and a size indicator, plus image-level scene and weather labels; it was split at random into 3,188 training and 1,062 test images. JHU-CROWD++ enlarges it to 4,372 images with 1,515,005 annotations (mean 346, maximum 25,791 per image), 106 distractors and approximate head boxes in place of size indicators, and splits it at random into training, validation and test sets. The technical report’s text gives 2,722 training images, but its figure gives 2,272, which with 500 validation and 1,600 test images sums to the total of 4,372. Results are reported overall and for low (0–50 people), medium (51–500) and high (over 500) density images and weather-degraded images; the official evaluation script draws the boundaries at 50 and 500 slightly differently (fewer than 50, fewer than 500, and 500 or more). As of November 2026 the terms of use restrict the dataset to academic and non-commercial uses, forbid reproducing or exploiting the images or derived data commercially, state that the images were obtained from the Internet and are not the lab’s property, and invite anyone who finds themselves or their belongings in the data to ask for the images to be removed.

FSC-147 (Ranjan et al., 2021). 6,135 images from 147 categories, from kitchen utensils and stationery to vehicles and animals, collected with web image scrapers from Flickr, Google and Bing and kept only if the objects were at least 7, of similar appearance and not severely occluded. Each object is marked with a dot (an occluded instance only if less than 90% is hidden), and three randomly chosen instances per image also get boxes as exemplars. The paper gives the maximum count per image as 3,731 in one section and 3,701 in another; the mean is 56 and the total 343,818. The split is by category, so that no category is shared: 89 categories (3,659 images) for training and 29 each for validation (1,286 images) and test (1,190 images). Results are MAE and RMSE over images, with three exemplars given at test time. The official repository, as of November 2026, links the images and precomputed density maps and carries an MIT licence file; its README states no separate terms for the images.

NWPU-Crowd, with 5,109 images and over two million annotations according to the JHU-CROWD++ report’s comparison table, withholds its test annotations and scores submissions on an evaluation server, according to the DM-Count paper (Wang et al., 2020); WorldExpo’10 (Zhang et al., 2015) is evaluated by MAE in five test scenes with regions of interest (Zhang et al., 2016).

Evaluation Metrics

Counting results are reported as the mean absolute error and a root mean squared error of the per-image counts,

$$\mathrm{MAE} = \frac{1}{N} \sum_{i=1}^{N} \lvert z_i - \hat z_i \rvert, \qquad \mathrm{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (z_i - \hat z_i)^2},$$

over $N$ test images with true counts $z_i$ and predictions $\hat z_i$. Most crowd counting papers call the second quantity “MSE” but define it with the square root: MCNN, CSRNet, the Bayesian loss and the JHU-CROWD++ report all write it so, and the official JHU-CROWD++ evaluation script computes the square root of the mean squared difference (Zhang et al., 2016; Li, Zhang & Chen, 2018; Ma et al., 2019). DM-Count and FamNet call it RMSE. It has the units of a count, not of a squared count, unlike the mean squared error proper, which the UCSD paper reported per direction (Chan, Liang & Vasconcelos, 2008). The MCNN authors read MAE as a measure of accuracy and the RMSE as one of robustness; since RMSE is at least MAE and grows with the spread of the errors, a large gap between them signals a few images with very large errors, typically the densest.

Other measures appear with particular datasets. The normalized absolute error divides each image’s error by its true count, so that sparse images weigh as much as dense ones; Idrees et al. (2013) reported it as NAD, and it reappears as C-NAE on UCF-QNRF and NAE on NWPU. The grid average mean absolute error, used for TRANCOS, splits each image into $4^L$ non-overlapping cells, sums the absolute count errors over the cells and averages over images,

$$\mathrm{GAME}(L) = \frac{1}{N} \sum_{n=1}^{N} \sum_{l=1}^{4^L} \bigl\lvert \hat C_n^l - C_n^l \bigr\rvert ,$$

so that a count right for the image but wrong in its parts is penalized; $\mathrm{GAME}(0)$ is the MAE (Li, Zhang & Chen, 2018). Density maps are sometimes compared by PSNR and SSIM after resizing and normalization (Li, Zhang & Chen, 2018), and localization by precision and recall of matched points (Idrees et al., 2018).

Results tables copied from paper to paper deserve a check against the originals. In MCNN’s UCF_CC_50 table, the “MSE” values for Idrees et al. (2013), Rodriguez et al. and Lempitsky and Zisserman (541.6, 697.8 and 487.1) are the standard deviations of the absolute error in Idrees et al.’s own table, which reported mean and standard deviation, not a root mean squared error. CSRNet’s Table 1 pairs MCNN’s MAE of 110.2 with an MSE of 185.9, which MCNN’s Figure 6 gives for the network without pretraining; the pretrained network’s MSE, which CSRNet’s own Table 5 uses, is 173.2.

Reported results, as printed, with the backbone where stated (MSE columns are root mean squared errors):

Method (paper)	Benchmark, split	MAE	“MSE”	Notes
MCNN (Zhang et al., 2016)	ShanghaiTech A / B, test	110.2 / 26.4	173.2 / 41.3	three columns
CSRNet (Li, Zhang & Chen, 2018)	ShanghaiTech A / B, test	68.2 / 10.6	115.0 / 16.0	VGG-16 front end
Bayesian loss + background (Ma et al., 2019)	ShanghaiTech A / B, test	62.8 / 7.7	101.8 / 12.7	VGG-19
DM-Count (Wang et al., 2020)	ShanghaiTech A / B, test	59.7 / 7.4	95.7 / 11.8	VGG-19, Bayesian-loss network
Composition loss (Idrees et al., 2018)	UCF-QNRF, test (334)	132	191	DenseNet-201
Pixel-wise loss (Ma et al., 2019)	UCF-QNRF, test	106.8	183.7	VGG-19, geometry-adaptive targets
Bayesian loss + background (Ma et al., 2019)	UCF-QNRF, test	88.7	154.8	VGG-19
DM-Count (Wang et al., 2020)	UCF-QNRF, test	85.6	148.3	VGG-19
CSRNet (benchmarked by Sindagi et al., 2020)	JHU-CROWD++, test (1,600)	85.9	309.2	high-density subset MAE 356.2
Bayesian loss (benchmarked by Sindagi et al., 2020)	JHU-CROWD++, test	75.0	299.9	high-density subset MAE 352.0
CG-DRCN-CC (Sindagi et al., 2020)	JHU-CROWD++, test	71.0	278.6	ResNet-101
FSOD few-shot detector (Ranjan et al., 2021)	FSC-147, test	32.53	140.65	adapted for counting
GMN (Ranjan et al., 2021)	FSC-147, test	26.52	124.57	trained on FSC-147
FamNet (Ranjan et al., 2021)	FSC-147, test	22.08	99.54	three exemplars, test-time adaptation

Rows for the same benchmark come from different papers with different training details; the Bayesian-loss and DM-Count rows share a network and training procedure by design, which is what makes their comparison with the pixel-wise loss informative. On JHU-CROWD++, the high-density MAE of each method listed is more than four times its overall MAE.

Applications

The crowd counting papers motivate the task with crowd safety and the planning of public events (Zhang et al., 2016), with the estimation of protest, rally and pilgrimage sizes, where opposing parties often claim very different numbers (Idrees et al., 2018), and with the distribution of people in a scene, which can reveal a dangerously dense region in an image whose total is unremarkable (Idrees et al., 2018). The same methods count cells and bacteria (Lempitsky & Zisserman, 2010; see cell detection) and people in drone footage, a task of the VisDrone benchmarks covered under aerial image analysis. Class-agnostic counting aims at arbitrary categories, such as the food, stationery, building elements and birds among FSC-147’s categories, without a model per category (Ranjan et al., 2021).

Open Problems

• Very dense and very large images. Errors concentrate in JHU-CROWD++’s high-density subset, and resizing or tiling large images changes the apparent size of heads.
• Annotation noise. Dots are placed inconsistently; DM-Count’s authors added uniform noise of up to 5% of the image height to the annotations and saw all losses degrade, the pixel-wise loss most (MAE 144.1, against 108.4 for the Bayesian loss and 105.6 for DM-Count, on a dataset the paper does not name beside that table; Wang et al., 2020).
• Generalization across datasets and scenes. MCNN trained on ShanghaiTech Part A and tested on UCF_CC_50 without fine-tuning had MAE 397.7, against 377.7 when trained on UCF_CC_50 itself (377.6 in the paper’s Table 3), and fine-tuning only its last two layers reached 295.1 (Zhang et al., 2016); domain shift between datasets remains a common problem.
• Counting and localization together. A density map gives counts without positions, and localization read off its peaks depends on post-processing, which the UCF-QNRF authors note can alter results (Idrees et al., 2018).
• Counting new categories. Class-agnostic counters still rely on exemplars that resemble the other instances; FSC-147 deliberately excluded images with large variation in pose and appearance or severe occlusion (Ranjan et al., 2021).