Pedestrian Detection

Pedestrian detection finds every person on foot in an image and returns a box for each. It is object detection restricted to one class, but a class with its own benchmarks, its own metric and its own history: most of the datasets are recorded from cars driving through cities, the people in them are often small, partly hidden or standing in groups, and detectors are compared by how many pedestrians they miss at a fixed number of false alarms per image rather than by average precision. The subject is also called person or human detection when the people are not restricted to upright figures in street scenes; benchmark papers keep “pedestrian” for upright, standing or walking people.

The task shaped detection as a whole. Dalal and Triggs’s histograms of oriented gradients were designed for it, the Caltech benchmark of Dollár, Wojek, Schiele and Perona argued for evaluating complete detectors on full images rather than classifiers on cropped windows, channel-feature detectors were developed for it, and its crowded scenes led to losses and suppression rules for overlapping objects, some of which their authors also tested on general detection benchmarks.

Problem Definition

Given an image, the detector outputs a set of boxes, each with a confidence score, one for every pedestrian in it. The box covers the full extent of the person, including any part hidden behind a car or another person: on Caltech the annotators drew a box for the whole pedestrian and, when part of the person was hidden, a second box for the visible region, and matching uses the full box (Dollár et al., 2009). CityPersons calls these full boxes “amodal” and adds visible boxes derived from segmentation masks (Zhang, Benenson & Schiele, 2017).

Two conventions set pedestrian detection apart from generic detection. First, box width is treated as noise. Dollár et al. (2012) observed that annotated box height follows a person’s size and distance smoothly while width oscillates with the positions of the limbs, and that detectors return boxes of different widths; their benchmark therefore standardizes every ground-truth and detected box to a width of 0.41 times its height (the log-mean aspect ratio on Caltech), keeping height and centre. CityPersons annotates in the same spirit, with the annotator drawing a line from the top of the head to the point between the feet and the box generated from it at aspect ratio 0.41. Second, crowds that cannot be labelled person by person, ambiguous figures and person-like regions such as people on posters are marked as ignore regions, in which detections are neither rewarded nor penalized.

Input and Output

The input is usually a single colour frame from a vehicle-mounted or surveillance camera: Caltech frames are $640 \times 480$ video at 30 Hz (Dollár et al., 2009), CityPersons images are full Cityscapes frames, which Liu, Liao, Ren, Hu and Yu (2019) evaluate at $1024 \times 2048$ pixels, and CrowdHuman collects images of various sizes from the Internet. Some detectors use more than one frame; on Caltech the best detector of the 2012 survey, MultiFtr+Motion, added motion features computed between frames to its appearance features (Dollár et al., 2012). The output is a list of boxes $(x, y, w, h)$ with scores. CrowdHuman annotates three boxes per person, head, visible region and full body, and most of its benchmark results use the full-body box (Liu, Huang & Wang, 2019; Chu et al., 2020).

Variants

• Pedestrians and people in general. Dollár et al. (2009) distinguish “person” datasets, with people in any pose and setting such as PASCAL VOC, from “pedestrian” datasets of upright people seen from restricted viewpoints, often with video and more complete labels. CityPersons keeps both: it labels pedestrians (walking, running or standing), riders, sitting persons and other persons, and evaluates on pedestrians (Zhang et al., 2017).
• Single frames and video. Most benchmarks score single frames; Caltech’s video and its temporal correspondences allow motion cues, and the human baseline of Zhang et al. (2016) was collected without access to neighbouring frames so that it would match single-frame detectors.
• Crowds. CrowdHuman targets scenes with many overlapping people, where suppression of duplicates removes true detections; the crowd methods under Approaches were designed for such scenes.
• 3D and other sensors. In the KITTI benchmark pedestrians are also detected as 3D boxes from LiDAR, cameras or both, which is the subject of sensor fusion for autonomous driving.

Challenges

• Small people. On Caltech, about 68% of pedestrians are 30 to 80 pixels tall (69% in the 2012 paper), 15% are 30 pixels or less, and the median height is 48 pixels (Dollár et al., 2009, 2012). Dollár et al. (2009) relate size to distance: with their camera an 80-pixel person is about 1.5 s away at 55 km/h and a 30-pixel person about 4 s, so a car must detect people at the medium scale, where detectors of the time performed poorly.
• Occlusion. Over 70% of Caltech pedestrians are occluded in at least one frame, and the hidden part is far more often the lower body than the head, contradicting the common assumption that occlusion is uniform (Dollár et al., 2009). In the 2012 evaluation, the best log-average miss rate on pedestrians over 50 pixels under partial occlusion was 73%, against 51–57% for the best detectors on the reasonable subset, and part-based detectors degraded about as much as holistic ones (Dollár et al., 2012).
• Crowds. In the CityPersons validation set, 48.8% of the 3,157 pedestrian annotations overlap another annotated pedestrian with an IoU above 0.1, and 26.4% with an IoU above 0.3 (Wang et al., 2018). A detector’s box for one person drifts to a neighbour, and non-maximum suppression then removes one of two correct boxes.
• Look-alikes and duplicates. Zhang et al. (2016) sorted by hand the 402 false positives that the best Caltech detector of 2015, Checkerboards, made at 0.1 FPPI. Background errors were the most common, mainly vertical structures, tree leaves and traffic lights; localization errors, detections overlapping a person, were dominated by double detections of the same person; the rest were annotation errors. They also measured size, blur and contrast for small pedestrians and concluded that the small number of pixels, not blur or contrast, is the source of difficulty.
• Annotation quality. Caltech’s boxes were drawn on sparse frames and interpolated in between, which leaves them misaligned. Zhang et al. (2016) re-annotated the test set and showed that results depend on which annotations are used, especially at stricter overlap thresholds.

Approaches

Sliding windows with HOG and a linear SVM

Dalal and Triggs (2005) described each $64 \times 128$ window by dense, block-normalized histograms of gradient orientations and scored it with a linear support vector machine (SVMLight with $C = 0.01$), trained on the INRIA person dataset they introduced, with one round of retraining on the false positives found in person-free photos. A detector is the classifier scanned over every position and scale, the procedure that sliding window detection treats in detail, followed by merging of overlapping detections. The HOG paper article covers the paper’s experiments, dataset and evaluation, the HOG article the descriptor, and hard negative mining the retraining step.

The detector proved hard to beat for upright, fully visible people. On Caltech, under the per-image evaluation described below, HOG trained on INRIA had a log-average miss rate of 68% on the reasonable subset in the 2012 evaluation (Dollár et al., 2012), and in the 2009 comparison of seven detectors it performed best on unoccluded near-scale pedestrians (Dollár et al., 2009).

Part models

Deformable part models add higher-resolution part filters that may shift relative to a HOG root filter. The 2012 Caltech evaluation includes two versions, LatSvm-V1 trained on PASCAL VOC and LatSvm-V2 trained on INRIA. LatSvm-V2 was the best detector on INRIA, at a log-average miss rate of 20%, but reached only 63% on the Caltech reasonable subset, and its relative gain grew with pedestrian height, which the authors attribute to parts needing high resolution (Dollár et al., 2012).

Channel features

Integral channel features (ICF) replaced the SVM over a fixed descriptor with boosting over many simple features computed on several image channels (Dollár, Tu, Perona & Belongie, 2009). A channel is any registered, translation-invariant transform of the image; the detector, ChnFtrs, used ten: six gradient-orientation histograms, gradient magnitude and the three LUV colour channels. Each feature is the sum of one channel over a rectangle, computed in constant time from an integral image as in Viola–Jones. The candidate rectangles were generated at random (30,000 for the final detector), and AdaBoost selected 2,000 depth-two decision trees over them, arranged as a soft cascade. The window was scanned with a 4-pixel step and a scale step of $2^{1/10}$, and pairs of detections overlapping by more than 0.6 were suppressed. Trained on INRIA, ChnFtrs found 86% of INRIA pedestrians at 1 false positive per image against 77% for HOG, and on Caltech’s 50-pixel-or-taller, no or partial occlusion pedestrians almost 60% at 1 FPPI, against at most 50% for the other detectors tested, HOG included. The authors’ per-window channel study found the gradient histograms the most informative channels and LUV the best colour space, and reported almost no difference between three boosting algorithms.

Aggregated channel features (ACF) simplified the features further (Dollár, Appel, Belongie & Perona, 2014). The same ten channels are computed on an image smoothed with a $[1\ 2\ 1]/4$ filter, summed over $4 \times 4$ blocks and smoothed again; a feature is a single pixel of the aggregated channels, which gives $128 \cdot 64 \cdot 10 / 16 = 5{,}120$ candidate features in a $128 \times 64$ window, over which AdaBoost trains 2,048 depth-two trees. The paper’s main contribution is the fast feature pyramid. The authors show that, for many channel types $\Omega$, a channel statistic of an image rescaled by $s$ follows a power law,

$$f_\Omega(I_s) \approx f_\Omega(I)\, s^{-\lambda_\Omega},$$

with an exponent $\lambda_\Omega$ estimated once per channel type from natural images. Channels are therefore computed exactly at one scale per octave, and the intermediate scales are obtained by resampling the nearest computed channels and correcting by $(s/s')^{-\lambda_\Omega}$. The detector scanned 8 scales per octave with a 4-pixel step and ran at over 30 frames per second on $640 \times 480$ images, against 12 with exact pyramids. Its log-average miss rates were 17% on INRIA, 45% on Caltech, 52% on TUD-Brussels and 51% on ETH (43%, 50% and 50% on the last three with exact pyramids), against 46%, 68%, 78% and 64% for HOG.

The family’s later members filtered the channels before boosting. In a comparison by Zhang et al. (2016) on the Caltech test set with the original annotations, the log-average miss rate fell from 44.2% for ACF to 24.8% for LDCF and 18.5% for Checkerboards, which filters the HOG and LUV channels with a bank of filters before the boosted forest; the authors concluded that the filter type determines the quality of an ICF detector.

Convolutional detectors

Convolutional networks were slow to win on pedestrians. Hosang, Omran, Benenson and Schiele (2015) noted that earlier networks for the task relied on special designs, such as parts and occlusion models, and that the top detectors on Caltech were still boosted decision forests. Using proposals from a channel-feature detector (SquaresChnFtrs, 34.8% log-average miss rate) and plain networks to rescore them, in the manner of R-CNN, they reached 30.7% with a small network trained only on Caltech and 23.3% with AlexNet pre-trained on ImageNet and fine-tuned on Caltech10x, the training setting that samples ten times as many frames from the training videos as the standard every-30th-frame setting.

The next step adapted Faster R-CNN itself. Zhang, Benenson and Schiele (2017) report that, retrained with default parameters, it underperforms on Caltech because it fails on the dominant small pedestrians of 50 to 70 pixels, as Liliang Zhang, Lin, Liang and He had also reported in 2016 (not the Shanshan Zhang et al. of the same year cited elsewhere here). Five modifications brought the reasonable-subset miss rate on the original annotations from 20.98% to 10.27%: region proposal network scales set to the quantiles of the training heights, the input upsampled by 2, the Adam solver, proposals kept out of ignore regions during training, and a feature stride of 8 instead of 16 pixels, obtained by removing VGG16’s fourth max-pooling layer. Upsampling and the quantized scales gave the largest gains. Pre-training on their new CityPersons dataset and fine-tuning on Caltech gave 5.1% on Zhang et al.’s (2016) new test annotations, against 7.3% for the previous best result.

Anchor-free detectors followed the general trend described in anchor-free object detection. CSP, by Liu, Liao, Ren, Hu and Yu (2019), predicts a heatmap of pedestrian centres and, at each centre, only the height, generating the box at the fixed aspect ratio of 0.41 that the line annotations use; predicting width as well was slightly worse in their ablation. With a ResNet-50 backbone it reported 4.5% on the Caltech reasonable subset with the new annotations, and 3.8% when initialized from a CityPersons model, and 11.0% on the CityPersons validation set at the original image size. The authors write that 4.5% beats the best competitor, RepLoss at 5.0%, by 0.4 points; the difference between the two numbers they print is 0.5.

Occlusion and crowds

Several methods change the training loss so that boxes stay on their own person. Repulsion loss (Wang et al., 2018) adds two terms to the usual smooth L1 attraction of a predicted box $B_P$ to its target:

$$L = L_{\mathrm{Attr}} + \alpha\, L_{\mathrm{RepGT}} + \beta\, L_{\mathrm{RepBox}} .$$

RepGT penalizes the overlap of $B_P$ with $G_{\mathrm{Rep}}$, the ground-truth box that the proposal overlaps most apart from its target, measured as intersection over ground truth, $\mathrm{IoG}(B, G) = |B \cap G| / |G|$, through a smoothed logarithm,

$$\mathrm{Smooth}_{\ln}(x) = \begin{cases} -\ln(1 - x) & x \le \sigma \\[2pt] \dfrac{x - \sigma}{1 - \sigma} - \ln(1 - \sigma) & x > \sigma . \end{cases}$$

The authors chose IoG rather than IoU because a box could reduce its IoU with a neighbour simply by growing. RepBox applies the same function to the IoU between predicted boxes assigned to different targets, so that NMS is less likely to merge them. On the CityPersons validation set, with a ResNet-50 Faster R-CNN and the original image scale, the reasonable miss rate went from 14.6% to 13.2% with both terms ($\alpha = \beta = 0.5$), and to 10.9% with images upscaled by 1.5; on the heavy-occlusion subset from 60.6% to 56.9%. The bounding box regression article covers the attraction losses this builds on.

Occlusion-aware R-CNN (OR-CNN; Zhang, Wen, Bian, Lei & Li, 2018) approaches the same problem from the side of attraction. Its aggregation loss adds to the regression loss a compactness term: the mean of the predictions of all anchors assigned to one person should match that person’s box, so that the proposals cluster tightly. Its part occlusion-aware RoI pooling divides each proposal into five body parts, predicts a visibility score for each from its pooled features, and weights the part features by visibility before summing them element-wise with the whole-proposal feature (the RoI pooling article covers the pooling it builds on). On the CityPersons validation set at $1.3\times$ scale it reported 11.0% on the reasonable subset with a VGG-16 backbone, against 11.6% for repulsion loss at the same scale with ResNet-50, so the backbones differ.

Other methods change the suppression step. Adaptive NMS (Liu, Huang & Wang, 2019) defines the density of an object as its largest IoU with any other ground-truth object, $d_i = \max_{j \ne i} \mathrm{IoU}(b_i, b_j)$, trains a small subnetwork to predict it for each detection, and suppresses around a kept box $M$ with the threshold $N_M = \max(N_t, d_M)$, so that boxes in crowded regions survive larger overlaps while isolated duplicates are removed as before. On the CityPersons validation set the reasonable miss rate of a VGG-16 Faster R-CNN fell from 14.5% with greedy NMS to 12.9%, and on the CrowdHuman validation set that of a ResNet-50 FPN from 52.35% to 49.73%. Chu, Zheng, Zhang and Sun (2020) instead let each proposal predict a set of $K$ boxes ($K = 2$ in their experiments), trained with an earth mover’s distance loss that takes the minimum over the $K!$ assignments of predictions to the proposal’s ground-truth objects, padded with background as needed:

$$L(b_i) = \min_{\pi \in \Pi} \sum_{k=1}^{K} \Big[ L_{\mathrm{cls}}\big(c_i^{(k)}, g_{\pi_k}\big) + L_{\mathrm{reg}}\big(l_i^{(k)}, g_{\pi_k}\big) \Big].$$

Their Set NMS skips suppression between two boxes predicted by the same proposal, since by construction they are different people. On CrowdHuman validation with full-body boxes, their method with an added refinement module raised AP from 85.8% to 90.7%, lowered the log-average miss rate from 42.9% to 41.4% and raised the Jaccard index from 79.8% to 82.3% over their re-implemented ResNet-50 FPN baseline. The NMS article covers Soft-NMS and learned NMS, which were also tried on crowded pedestrians, and label assignment covers the matching rules these losses depend on.

Datasets and Benchmarks

Dataset	Year	Images	People	Protocol
INRIA	2005	614 positive training and 288 positive test images, 1,218 and 453 negative	1,208 training, 566 test	Per-window in the paper; per-image in later evaluations
Caltech Pedestrian	2009	about 250,000 labelled frames of 10 hours of video	about 350,000 boxes, 2,300 individuals	Miss rate vs FPPI, log-average miss rate, reasonable subset
KITTI object	2012	7,481 training, 7,518 test	pedestrians among 8 classes	AP at IoU 0.5, easy/moderate/hard
CityPersons	2017	5,000 (2,975 train, 500 val, 1,575 test as printed)	35,016 persons, 13,172 ignore regions	Log-average miss rate, Caltech protocol
CrowdHuman	2018	15,000 train, 4,370 val, 5,000 test	470,000 in train and val	AP, log-average miss rate, Jaccard index

The INRIA and Caltech counts are from the dataset table of Dollár et al. (2009); the HOG paper article discusses the INRIA counts, which differ slightly between the 2005 paper and later documents. The CityPersons paper’s table prints 1,575 test images, which does not add up to its total of 5,000; the OR-CNN paper gives 1,525, which does (Zhang, Wen et al., 2018).

Caltech Pedestrian. About 10 hours of video were recorded from a car driving through regular traffic in the Los Angeles area (LAX, Santa Monica, Hollywood, Pasadena and Little Tokyo), of which 250,000 frames in 137 segments of about a minute, about 2.3 hours at 30 Hz by our arithmetic, were annotated (later papers describe the dataset as 2.5 hours of video), with 350,000 boxes of 2,300 unique pedestrians; about 400 hours of labelling were reported (Dollár et al., 2009). Boxes are labelled Person, People (groups too dense to label individually) or Person? (ambiguous). The data were split by session, sessions 0–5 for training and 6–10 for testing, and the 2009 paper withheld the test data and asked authors to submit trained detectors. Results are computed on every 30th frame, which gives the 4,024 test frames used in later work (Zhang et al., 2016). Height bins are near (80 pixels or more), medium (30–80) and far (30 or less); occlusion bins are partial (1–35% of the area hidden), heavy (35–80%) and over 80%, which usually means full occlusion. The reasonable setting, used throughout the 2012 evaluation and adopted by later work, keeps pedestrians at least 50 pixels tall under no or partial occlusion (Dollár et al., 2012). The authors’ own demonstration script trains ACF on fully visible pedestrians at least 50 pixels tall and tests with heights from 50 pixels, visible fractions of 0.65 to 1, and boxes within 5 pixels of the image border set to ignore (toolbox, acfDemoCal.m). Later papers call the dataset Caltech-USA; the 2012 evaluation also used Caltech-Japan, a similar dataset the authors recorded in Japan but could not release for legal reasons. Zhang et al. (2016) released corrected annotations, and results are now reported on the original (MR$^O$) or the new (MR$^N$) annotations, which are not comparable. As of November 2026 the dataset, code and published result curves are distributed through a CaltechDATA record under the Creative Commons Attribution 4.0 licence, in two archives totalling about 11.9 GB, the larger holding the data and labels.

KITTI. The KITTI object benchmark evaluates pedestrians with average precision at an IoU threshold of 0.5 and three difficulty levels set by box height, occlusion and truncation, ranking by the moderate level; its article gives the definitions and the licence. CityPersons’ authors note that KITTI and Caltech both average about one person per image and were each recorded in one city (Zhang et al., 2017).

CityPersons. Zhang et al. (2017) annotated persons on the 5,000 finely labelled images of Cityscapes, recorded in 27 cities in Germany and neighbouring countries, using the Cityscapes splits. The training set covers 18 cities and three seasons and averages about 7 persons per image, against 1.4 on Caltech and 0.8 on KITTI. Fewer than 30% of its pedestrians are fully visible, against more than 60% on Caltech. Evaluation follows Caltech: the reported number is the log-average miss rate on the reasonable subset, height at least 50 pixels and occlusion ratio at most 0.35, and cyclists, sitting and other persons and ignore regions do not count. The paper defines the occlusion ratio as the visible box’s area divided by the full box’s, which is the visible fraction; its subset definitions and figures treat it as the hidden fraction. Later papers added occlusion subsets: bare (occlusion at most 10%) and partial (10–35%) within the reasonable set, and heavy (over 35%, outside it) (Wang et al., 2018). Test annotations are withheld and results come from an evaluation server. The annotations are distributed with Cityscapes, whose terms, as of November 2026, require registration by people affiliated with a scientific body, allow non-commercial use only and forbid redistributing the data.

CrowdHuman. Built to evaluate detectors in crowds, CrowdHuman has 15,000 training, 4,370 validation and 5,000 test images collected from the Internet, with about 470,000 human instances in the training and validation sets, about 23 per image, each annotated with head, visible-region and full-body boxes; boxes tagged “mask” mark crowds, reflections and other person-like regions to be ignored (official website). The images were collected from the Internet (Liu, Huang & Wang, 2019). Liu, Huang and Wang (2019) counted 22.64 persons and 2.40 pairs with IoU above 0.5 per image, against 6.47 and 0.32 on CityPersons and 0.32 and 0.02 on Caltech. As of November 2026 the download page restricts the images to non-commercial research and educational use and forbids redistributing them. The dataset paper exists only as an arXiv preprint, which is not used for this article, so the benchmark is described from its website and from papers that evaluate on it.

Evaluation Metrics

Miss rate against false positives per image

Dalal and Triggs (2005) evaluated classifiers on cropped windows, reporting miss rate against false positives per window. Dollár et al. (2009) argued that this methodology is flawed: it ignores errors at wrong positions and scales, false detections on body parts, and the effect of NMS, and classifiers can exploit boundary effects between cropped positives and uncropped negatives, which they observed in two of the detectors they evaluated. Per-window and per-image results on INRIA ranked detectors differently. The benchmark therefore evaluates the final boxes of a complete detector on full images.

A detection matches a ground-truth box if their IoU exceeds 0.5; detections are matched greedily in order of decreasing score, each box at most once. Lowering the score threshold traces a curve of miss rate, $1 - \text{recall}$, against false positives per image (FPPI), plotted on log-log axes. Dollár et al. preferred it to a precision–recall curve for applications such as driving, where there is typically an upper limit on acceptable false positives per image, independent of how many pedestrians there are. In the 2012 evaluation they found it insensitive to the overlap threshold below about 0.6 and kept 0.5.

Log-average miss rate

The 2009 paper compared detectors by the miss rate at 1 FPPI. The 2012 evaluation replaced it with the log-average miss rate, computed from the miss rate at nine FPPI values evenly spaced in log space between $10^{-2}$ and $10^{0}$:

$$\mathrm{MR}_{-2} = \exp\!\Big( \frac{1}{9} \sum_{i=1}^{9} \ln \mathrm{mr}(f_i) \Big), \qquad f_i = 10^{-2 + (i-1)/4} .$$

The 2012 paper’s text describes the computation as averaging the miss rate at the nine points; the name, and the authors’ code, which takes the exponential of the mean logarithm with miss rates clipped at $10^{-10}$, make it a geometric mean (toolbox, acfTest.m). In the code, the miss rate at $f_i$ is the one reached at the largest FPPI not exceeding $f_i$ (bbGt.m, compRoc), and for a curve that ends before $f_i$ the paper uses the lowest miss rate it achieved. The authors present the measure as similar in spirit to average precision, a single number for the whole curve, and note that because curves are nearly linear in this range it is close to the miss rate at $10^{-1}$ FPPI but more stable. Later papers write it as MR or MR$^{-2}$. Zhang et al. (2016) extended the range to $[10^{-4}, 10^{0}]$ (MR$^{-4}$), arguing that the standard range covers the low-precision part of the curve and does not reflect improvements in localization.

Ignore regions, filtering and aspect ratio

Evaluating on a subset, such as the reasonable setting, cannot be done by deleting the other ground truth: detections on the deleted people would count as false positives. Instead, boxes outside the subset are set to ignore (Dollár et al., 2009, 2012). A detection matched to an ignore box is neither a true nor a false positive, an unmatched ignore box is not a miss, matches to normal boxes are preferred, and several detections may match one ignore box. For ignore boxes the overlap is the intersection divided by the detection’s area,

$$a_o = \frac{|B_{\mathrm{dt}} \cap B_{\mathrm{ig}}|}{|B_{\mathrm{dt}}|} ,$$

so that a detection anywhere inside a box labelled People is excused. Boxes under 20 pixels tall, truncated by the image border, or labelled Person? or People are always ignored. Detections are filtered too: under expanded filtering, when the evaluation range is $S_0$ to $S_1$ pixels, detections outside $S_0/r$ to $S_1 r$ with $r = 1.25$ are removed before matching. The authors chose it after showing that removing all detections outside the range under-reports performance, while removing only unmatched ones afterwards over-reports it, and that both can be exploited. Finally, all boxes are standardized to an aspect ratio of 0.41, as described under Problem Definition, which had a large effect on detectors that return narrow boxes.

Other metrics

KITTI ranks pedestrians by AP at IoU 0.5. CrowdHuman results are reported with AP, the log-average miss rate and the Jaccard index, which Chu et al. (2020) compute with the benchmark’s official software; they describe it as measuring how much the detection set at a score threshold overlaps the ground truth, report the best value over thresholds, and observe that the log-average miss rate is very sensitive to high-scoring false positives while AP is more sensitive to recall. The metric articles cover ROC and DET curves and the AP family.

Example: HOG, a linear SVM and the log-average miss rate

The example builds a small Dalal–Triggs-style detector for upright synthetic figures and scores it with the Caltech protocol. Each $144 \times 256$ scene has a brightness ramp, clutter strokes, two filled boxes, four vertical poles as look-alikes, Gaussian noise, and one to four stick figures (head, torso, arms and legs in random poses) whose full box is $0.41h \times h$ with $h$ between 22 and 80 pixels. A figure is lighter or darker than its surroundings, and with probability 0.4 a rectangle hides its lower part, leaving a visible fraction between 0.2 and 1; occlusion is one minus the visible fraction, as on Caltech. Figures do not overlap each other, so the example has occlusion but no crowds.

The detector uses HOG from scikit-image (9 orientations, $6 \times 6$ pixel cells, $2 \times 2$ cell blocks, L2-Hys) on a $48 \times 24$ window holding a 40-pixel figure, computed once per level of a six-level pyramid with scale factors from 1.6 down to 0.5, so that figures of about 25 to 80 pixels reach the template’s size; small figures are found by upsampling, as Caltech’s evaluators did with older detectors. A linear SVM (LinearSVC, $C = 0.01$ as in Dalal and Triggs’s SVMLight setting) is trained on 400 rendered windows of unoccluded figures and their mirror images, and on 10 random windows from each of 60 figure-free scenes, then retrained once with every window of those scenes that it scores above zero. At test time windows scored above $-1$ are merged by greedy NMS at IoU 0.5, and each window reports the figure region inside it, its padding cropped as Caltech crops HOG’s. The evaluation re-implements bbGt.m’s matching with ignore regions and acfTest.m’s nine reference points and geometric mean, with the paper’s expanded filtering at $r = 1.25$, over 100 test scenes, so that one false positive is 0.01 FPPI. Besides the subsets, it prints the reasonable subset with the other figures deleted instead of ignored, and the arithmetic mean of the nine miss rates next to the log-average. The script loops over five data seeds and runs in about a minute on the drafting machine. Versions: NumPy 2.3.5, scikit-image 0.25.2, scikit-learn 1.7.2 and threadpoolctl 3.5.0.

import numpy as np
from threadpoolctl import threadpool_limits
from skimage.feature import hog
from skimage.transform import rescale
from sklearn.svm import LinearSVC

H, W = 144, 256                      # scene size
WIN_H, WIN_W, FIG_H, CELL = 48, 24, 40, 6   # window, figure height inside it, HOG cell (= stride)
ASPECT = 0.41                        # Caltech's standardized width / height
SCALES = 1.6 * 2.0 ** (-np.arange(6) / 3)   # pyramid: figures of about 25-80 px map to 40 px

def capsule(img, p, q, thick, val, box):
    """Draw an antialiased thick segment p-q with grey value val, only inside box (x0, y0, x1, y1)."""
    x0, y0, x1, y1 = [int(v) for v in box]
    x0, y0 = max(x0, 0), max(y0, 0); x1, y1 = min(x1, img.shape[1]), min(y1, img.shape[0])
    if x1 <= x0 or y1 <= y0:
        return
    y, x = np.mgrid[y0:y1, x0:x1].astype(float)
    p, q = np.asarray(p, float), np.asarray(q, float); d = q - p
    t = np.clip(((x - p[0]) * d[0] + (y - p[1]) * d[1]) / max(d @ d, 1e-9), 0, 1)
    dist = np.hypot(x - p[0] - t * d[0], y - p[1] - t * d[1]) - thick / 2
    a = np.clip(0.5 - dist, 0, 1)
    img[y0:y1, x0:x1] = img[y0:y1, x0:x1] * (1 - a) + val * a

def figure(img, x0, y0, h, val, rng):
    """Upright stick figure whose full box is (x0, y0, 0.41 h, h)."""
    cx, box = x0 + ASPECT * h / 2, (x0 - 0.2 * h, y0 - 2, x0 + 0.61 * h, y0 + h + 2)
    hip = (cx, y0 + 0.52 * h)
    for s in (-1, 1):                                   # legs and arms
        foot = (cx + s * rng.uniform(0.02, 0.17) * h, y0 + 0.97 * h)
        capsule(img, hip, foot, 0.08 * h, val, box)
        hand = (cx + s * rng.uniform(0.06, 0.16) * h, y0 + rng.uniform(0.42, 0.52) * h)
        capsule(img, (cx, y0 + 0.21 * h), hand, 0.055 * h, val, box)
    capsule(img, (cx, y0 + 0.2 * h), hip, 0.17 * h, val, box)          # torso
    capsule(img, (cx, y0 + 0.08 * h), (cx, y0 + 0.08 * h), 0.15 * h, val, box)  # head

def background(rng, shape, n_poles, n_boxes, n_strokes):
    h, w = shape
    y, x = np.mgrid[0:h, 0:w] / max(h, w)
    img = rng.uniform(0.3, 0.6) + rng.uniform(-0.15, 0.15) * x + rng.uniform(-0.15, 0.15) * y
    for _ in range(n_strokes):                          # clutter at any angle
        p = rng.uniform(0, 1, 2) * (w, h); a = rng.uniform(0, np.pi); L = rng.uniform(10, 50)
        q = p + L * np.array([np.cos(a), np.sin(a)])
        capsule(img, p, q, rng.uniform(1, 3), rng.uniform(0.1, 0.9), (0, 0, w, h))
    for _ in range(n_boxes):                            # filled boxes
        bw, bh = rng.uniform(10, 50), rng.uniform(8, 30); bx, by = rng.uniform(0, w - bw), rng.uniform(0, h - bh)
        img[int(by):int(by + bh), int(bx):int(bx + bw)] = rng.uniform(0.1, 0.9)
    for _ in range(n_poles):                            # vertical poles: the look-alikes
        px, py, L = rng.uniform(0, w), rng.uniform(0, h - 25), rng.uniform(25, 90)
        capsule(img, (px, py), (px, py + L), rng.uniform(2, 5), rng.uniform(0.05, 0.95), (0, 0, w, h))
    return img

def contrast(bg, rng):
    return float(np.clip(bg + rng.choice([-1, 1]) * rng.uniform(0.2, 0.4), 0, 1))

def scene(rng, n_fig):
    """Scene with n_fig figures; returns image and GT rows (x, y, w, h, occlusion)."""
    img = background(rng, (H, W), n_poles=4, n_boxes=2, n_strokes=10)
    gts, tries = [], 0
    while len(gts) < n_fig and tries < 200:
        tries += 1
        h = rng.uniform(22, 80); w = ASPECT * h
        x0, y0 = rng.uniform(2, W - w - 2), rng.uniform(2, H - h - 2)
        if any(abs(x0 + w / 2 - g[0] - g[2] / 2) < (w + g[2]) / 2 + 4 for g in gts):
            continue                                    # no overlapping people in this example
        figure(img, x0, y0, h, contrast(img[int(y0 + h / 2), int(x0 + w / 2)], rng), rng)
        occ = 0.0
        if rng.random() < 0.4:                          # occluder in front of the lower body
            vis = rng.uniform(0.2, 1.0); occ = 1 - vis
            top = y0 + vis * h
            img[int(top):int(min(y0 + h + 6, H)), int(max(x0 - 6, 0)):int(min(x0 + w + 6, W))] = rng.uniform(0.05, 0.95)
        gts.append((x0, y0, w, h, occ))
    img = img + rng.normal(0, 0.04, img.shape)
    return img, np.array(gts).reshape(-1, 5)

def canonical_positive(rng):
    patch = background(rng, (WIN_H, WIN_W), n_poles=1, n_boxes=0, n_strokes=2)
    h = FIG_H + rng.uniform(-1, 1); x0 = (WIN_W - ASPECT * h) / 2 + rng.uniform(-1, 1)
    figure(patch, x0, 4 + rng.uniform(-1, 1), h, contrast(patch[WIN_H // 2, WIN_W // 2], rng), rng)
    return patch + rng.normal(0, 0.04, patch.shape)

def hog_map(img):
    return hog(img, orientations=9, pixels_per_cell=(CELL, CELL), cells_per_block=(2, 2),
               block_norm="L2-Hys", feature_vector=False)

NBY, NBX = WIN_H // CELL - 1, WIN_W // CELL - 1          # 7 x 3 blocks per window

def windows(img):
    """All windows over the pyramid -> features (n, 756) and figure boxes (x, y, w, h) in image coordinates."""
    feats, boxes = [], []
    for s in SCALES:
        Hm = hog_map(rescale(img, s, order=1, anti_aliasing=s < 1))
        ni, nj = Hm.shape[0] - NBY + 1, Hm.shape[1] - NBX + 1
        if ni < 1 or nj < 1:
            continue
        idx = np.array([(i, j) for i in range(ni) for j in range(nj)])
        V = np.lib.stride_tricks.sliding_window_view(Hm, (NBY, NBX), axis=(0, 1))   # (ni, nj, 2, 2, 9, NBY, NBX)
        feats.append(V.transpose(0, 1, 5, 6, 2, 3, 4).reshape(ni * nj, -1))
        fh = FIG_H / s; fw = ASPECT * fh                 # figure region inside the window (padding cropped)
        cx = (CELL * idx[:, 1] + WIN_W / 2) / s
        boxes.append(np.c_[cx - fw / 2, (CELL * idx[:, 0] + 4) / s, np.full(len(idx), fw), np.full(len(idx), fh)])
    return np.vstack(feats), np.vstack(boxes)

def iou_xywh(a, B):
    ix = np.clip(np.minimum(a[0] + a[2], B[:, 0] + B[:, 2]) - np.maximum(a[0], B[:, 0]), 0, None)
    iy = np.clip(np.minimum(a[1] + a[3], B[:, 1] + B[:, 3]) - np.maximum(a[1], B[:, 1]), 0, None)
    inter = ix * iy
    return inter / (a[2] * a[3] + B[:, 2] * B[:, 3] - inter), inter

def nms(boxes, scores, thr=0.5):
    order, keep = np.argsort(-scores), []
    while order.size:
        i = order[0]; keep.append(i)
        order = order[1:][iou_xywh(boxes[i], boxes[order[1:]])[0] <= thr]
    return np.array(keep, int)

def eval_image(gt, ign, dt):
    """Caltech matching (bbGt evalRes): gt (m,4), ign (m,) bool, dt (n,5) with score.
    Returns the sorted scores and match flags: 1 tp, 0 fp, -1 matched to an ignore region."""
    order = np.argsort(-dt[:, 4], kind="stable"); dt = dt[order]
    g_order = np.argsort(ign, kind="stable"); gt, ign = gt[g_order], ign[g_order]
    gm = np.where(ign, -1, 0); flags = np.zeros(len(dt), int)
    for d in range(len(dt)):
        best, bg, bm = 0.5, -1, 0
        if len(gt):
            iou, inter = iou_xywh(dt[d, :4], gt)
            oa = np.where(ign, inter / (dt[d, 2] * dt[d, 3]), iou)
        for g in range(len(gt)):
            if gm[g] == 1:
                continue
            if bm != 0 and gm[g] == -1:
                break
            if oa[g] < best:
                continue
            best, bg, bm = oa[g], g, (1 if gm[g] == 0 else -1)
        if bm == 1:
            gm[bg] = 1; flags[d] = 1
        elif bm == -1:
            flags[d] = -1
    return dt[:, 4], flags

REFS = 10.0 ** np.arange(-2, 0.01, 0.25)             # nine FPPI points, 1e-2 ... 1e0

def lamr(per_image, n_img):
    """per_image: list of (scores, flags, n_gt). Returns miss rates at REFS, log-average and arithmetic mean."""
    s = np.concatenate([p[0] for p in per_image]); f = np.concatenate([p[1] for p in per_image])
    n_gt = sum(p[2] for p in per_image)
    keep = f != -1; s, f = s[keep], f[keep]
    o = np.argsort(-s, kind="stable"); f = f[o]
    tp = np.cumsum(f == 1); fp = np.cumsum(f != 1)
    xs = np.r_[-np.inf, fp / n_img]; ys = np.r_[0, tp / n_gt]
    miss = np.array([1 - ys[np.flatnonzero(xs <= r)[-1]] for r in REFS])
    return miss, np.exp(np.mean(np.log(np.maximum(1e-10, miss)))), miss.mean()

SUBSETS = {                                           # name: (h0, h1, occ0, occ1)  occlusion = 1 - visible fraction
    "overall (h >= 20)":          (20, np.inf, 0.0, 1.0),
    "reasonable":                 (50, np.inf, 0.0, 0.35),
    "unoccluded, h >= 50":        (50, np.inf, 0.0, 0.0),
    "partial occ., h >= 50":      (50, np.inf, 1e-9, 0.35),
    "heavy occ., h >= 50":        (50, np.inf, 0.35 + 1e-9, 0.8),
    "small 30-50, unoccluded":    (30, 50, 0.0, 0.0),
}

def evaluate(test_dets, test_gts, h0, h1, o0, o1, remove=False):
    per = []
    for dt, gt in zip(test_dets, test_gts):
        insub = (gt[:, 3] >= h0) & (gt[:, 3] <= h1) & (gt[:, 4] >= o0) & (gt[:, 4] <= o1)
        g, ign = gt[:, :4], ~insub
        if remove:
            g, ign = g[insub], ign[insub]
        d = dt[(dt[:, 3] >= h0 / 1.25) & (dt[:, 3] <= h1 * 1.25)]    # expanded filtering, r = 1.25
        sc, fl = eval_image(g, ign, d)
        per.append((sc, fl, int((~ign).sum())))
    return lamr(per, len(test_gts)), sum(p[2] for p in per)

def run(seed, verbose):
    rng = np.random.default_rng(seed)
    P = [canonical_positive(rng) for _ in range(400)]
    P += [p[:, ::-1] for p in P]                       # mirror images
    Xp = np.array([hog_map(p).ravel() for p in P])
    neg_F = [windows(scene(rng, 0)[0])[0] for _ in range(60)]   # all windows of 60 figure-free scenes
    Xn = []
    for F in neg_F:                                     # 10 random windows per figure-free scene
        Xn.append(F[rng.choice(len(F), 10, replace=False)])
    Xn = np.vstack(Xn)
    def fit(Xn):
        X = np.vstack([Xp, Xn]); y = np.r_[np.ones(len(Xp)), np.zeros(len(Xn))]
        return LinearSVC(C=0.01, random_state=0, max_iter=20000).fit(X, y)
    clf = fit(Xn)
    hard = []                                           # one bootstrapping round
    for F in neg_F:
        hard.append(F[clf.decision_function(F) > 0])
    n_hard = sum(len(h) for h in hard)
    clf = fit(np.vstack([Xn] + hard))
    test_gts, test_dets = [], []
    for _ in range(100):
        img, gt = scene(rng, rng.integers(1, 5))
        F, B = windows(img)
        s = clf.decision_function(F)
        cand = np.flatnonzero(s > -1.0)
        k = cand[nms(B[cand], s[cand])]
        test_dets.append(np.c_[B[k], s[k]]); test_gts.append(gt)
    res = {}
    for name, (h0, h1, o0, o1) in SUBSETS.items():
        (miss, lam, am), n = evaluate(test_dets, test_gts, h0, h1, o0, o1)
        res[name] = (lam, am, n, miss)
    (miss_r, lam_r, am_r), n_r = evaluate(test_dets, test_gts, 50, np.inf, 0, 0.35, remove=True)
    res["reasonable, others removed"] = (lam_r, am_r, n_r, miss_r)
    if verbose:
        n_all = sum(len(g) for g in test_gts)
        print(f"seed {seed}: {len(Xp)} positive windows, {len(Xn)} random + {n_hard} hard negatives; "
              f"test: 100 images, {n_all} figures")
        print(f"{'subset':28s}{'figures':>8s}{'log-avg MR':>11s}{'arith. MR':>10s}")
        for name, (lam, am, n, _) in res.items():
            print(f"{name:28s}{n:8d}{lam:11.2f}{am:10.2f}")
        print("reasonable miss rate at FPPI", " ".join(f"{r:.3g}" for r in REFS))
        print("                          ", " ".join(f"{m:.2f}" for m in res["reasonable"][3]))
    return {k: v[0] for k, v in res.items()}

if __name__ == "__main__":
    with threadpool_limits(limits=1):                # one BLAS thread: faster here, and reproducible sums
        rows = [run(s, verbose=(s == 0)) for s in range(5)]
    print("\nlog-average miss rate, seeds 0-4:")
    for name in rows[0]:
        v = np.array([r[name] for r in rows])
        print(f"{name:28s}" + " ".join(f"{x:5.2f}" for x in v) + f"   range {v.min():.2f}-{v.max():.2f}")

seed 0: 800 positive windows, 600 random + 4386 hard negatives; test: 100 images, 260 figures
subset                       figures log-avg MR arith. MR
overall (h >= 20)                260       0.50      0.52
reasonable                       100       0.21      0.26
unoccluded, h >= 50               78       0.18      0.23
partial occ., h >= 50             22       0.25      0.29
heavy occ., h >= 50               26       0.77      0.78
small 30-50, unoccluded           65       0.32      0.37
reasonable, others removed       100       0.39      0.51
reasonable miss rate at FPPI 0.01 0.0178 0.0316 0.0562 0.1 0.178 0.316 0.562 1
                           0.46 0.46 0.37 0.35 0.21 0.17 0.11 0.10 0.08

log-average miss rate, seeds 0-4:
overall (h >= 20)            0.50  0.55  0.59  0.49  0.60   range 0.49-0.60
reasonable                   0.21  0.26  0.36  0.32  0.33   range 0.21-0.36
unoccluded, h >= 50          0.18  0.22  0.27  0.29  0.26   range 0.18-0.29
partial occ., h >= 50        0.25  0.43  0.54  0.43  0.43   range 0.25-0.54
heavy occ., h >= 50          0.77  0.67  0.63  0.57  0.78   range 0.57-0.78
small 30-50, unoccluded      0.32  0.36  0.28  0.29  0.31   range 0.28-0.36
reasonable, others removed   0.39  0.41  0.51  0.41  0.44   range 0.39-0.51

The single number hides a curve. On seed 0 the detector misses 46% of the reasonable figures at 0.01 FPPI, one false positive in the 100 test images, and 8% at 1 FPPI; the first two reference points share a value because 0.0178 FPPI is still below the second false positive. With fewer than 100 test images the lowest reference point would fall before the first false positive could even occur; on Caltech’s 4,024 test frames a single false positive is 0.00025 FPPI.

The subsets order the figures in the way the Caltech evaluations ordered real pedestrians. On every seed the reasonable subset scores better than the overall set, which includes small and heavily occluded figures, and heavy occlusion is the worst subset, at log-average miss rates of 0.57 to 0.78. The template was trained on whole figures, and in our reading an occluder hides the legs that a holistic template scores, much as the lower body is the part most often hidden on Caltech. Partial occlusion scores close to unoccluded figures on seed 0 but 0.43 to 0.54 on the other seeds; seed 0’s partial subset has only 22 figures, and subsets that small give unstable numbers. Figures of 30 to 50 pixels are worse than unoccluded figures of 50 pixels or more on seeds 0, 1 and 4 and about equal on seeds 2 and 3. The pyramid upsamples them to the template’s size, but the upsampled window contains fewer original pixels, the source of difficulty Zhang et al. (2016) identified for small pedestrians on Caltech.

Deleting the out-of-subset figures instead of ignoring them raises the reasonable log-average miss rate on every seed, from 0.21–0.36 to 0.39–0.51. The detector has not changed: its confident detections of small and occluded figures, which ignore regions excuse, become false positives and push the curve to the right, which is the over-reporting Dollár et al. warned against. The arithmetic mean of the nine miss rates is always the larger of the two summaries, since a geometric mean never exceeds an arithmetic one, and the gap is widest for the curve with the high miss rates at low FPPI, 0.51 against 0.39 for the deleted variant on seed 0. The logarithm gives each factor of improvement the same weight, so halving a miss rate from 0.10 to 0.05 at 1 FPPI counts as much as halving it from 0.46 to 0.23 at 0.01 FPPI.

The script was changed twice after its first run, both times for speed. Window features were extracted with a strided view instead of a list comprehension, which gives identical arrays, and the windows of the figure-free training scenes were computed once and reused for sampling and for the bootstrapping scan; this did not shorten the run, so BLAS was then limited to one thread, which brought it from about 2 minutes 20 seconds to one minute. A docstring was also corrected. The data, detector, evaluation and printed output are the same as in the first run.

Applications

Dollár et al. (2009) list surveillance, robotics, assistive technology for the visually impaired, content-based indexing, human–machine interfaces and automotive safety as applications, and single out the automotive case; their 2012 paper cites the figure that nearly 5,000 of the 35,000 annual traffic crash fatalities in the United States involve pedestrians. Detection of pedestrians and cyclists is one of the perception tasks of autonomous driving, alongside 3D detection and tracking. Person detectors are also the first stage of tracking-by-detection in crowded scenes such as those of MOTChallenge: the detector of ByteTrack, for example, was trained on CrowdHuman and CityPersons among other datasets. Top-down human pose estimation estimates joints inside person boxes, so it inherits the detector’s misses and merged boxes in crowds.

Open Problems

• Small and occluded people. Dollár et al. (2012) called performance on far-scale and heavily occluded pedestrians “abysmal”, and CityPersons pre-training later helped most on the harder subsets, cutting the heavy-occlusion miss rate on Caltech, with the original annotations, from 68.3% to 57.7% (Zhang et al., 2017).
• Crowds. Reported log-average miss rates on CrowdHuman’s full-body benchmark stay above 40% for the detectors above (Liu, Huang & Wang, 2019; Chu et al., 2020), and suppression still trades duplicates against neighbours: Adaptive NMS’s authors report that the low-scoring detections Soft-NMS keeps could fall outside the FPPI range of the metric and so need not lower the miss rate.
• Generalization across datasets. Trained on Caltech, an adapted Faster R-CNN reached 10.27% on Caltech but 46.91% on CityPersons and 57.85% on ETH; trained on CityPersons it averaged 20.95% over six test sets against 29.98% for the Caltech-trained model (Zhang et al., 2017). The authors attribute the better transfer to the size and diversity of their data, recorded in many cities, and to the quality of its boxes.
• Localization. With the stricter matching threshold of IoU 0.75, CityPersons’ best Caltech model had a 25.8% miss rate against 5.1% at 0.5 (Zhang et al., 2017); the standard threshold of 0.5 forgives most localization error.
• Measurement. Zhang et al. (2016) found that at about 95% recall the best detectors of 2015 made ten times as many errors as a human baseline on Caltech (5.62% log-average miss rate for the humans against 18.47% for Checkerboards), and that the original annotations contained enough errors to change results. On the new annotations, results reached 4.0% for repulsion loss and 3.8% for CSP, both with CityPersons pre-training (Wang et al., 2018; Liu, Liao et al., 2019); in our reading, so little remains to be measured on the reasonable subset that the harder subsets and datasets carry the comparison.