Caltech Pedestrian

The Caltech Pedestrian Dataset is a collection of about 10 hours of video recorded from a car driving through regular traffic in the Los Angeles area, with about 350,000 boxes around some 2,300 pedestrians in 250,000 annotated frames. Piotr Dollár and Pietro Perona at Caltech, with Christian Wojek and Bernt Schiele at TU Darmstadt, introduced it at CVPR 2009 together with a benchmark that scored complete detectors on full frames rather than classifiers on cropped windows (Dollár et al., 2009). Their 2012 journal paper re-evaluated sixteen detectors under a revised protocol for scoring pedestrian detection: the reasonable setting, the log-average miss rate and a fixed box aspect ratio of 0.41 (Dollár et al., 2012). Zhang et al. (2016) call it the standard Caltech evaluation and the dataset Caltech-USA. As of November 2026 the data, code and published result curves are on CaltechDATA under the Creative Commons Attribution 4.0 licence.

Pedestrian Detection covers the benchmark as part of its task: the subsets, the miss rate against false positives per image, the log-average miss rate, ignore regions and the published results. This article covers the dataset itself (its recording, labels, splits, files and licence), the history of its protocol, and two rules that article mentions only in passing: how detections outside the evaluated height range are filtered, and how box widths are standardized. Both are reproduced on synthetic boxes below.

Purpose

Dollár et al. (2009) argued that pedestrian datasets of the time covered a narrow range of scale, occlusion and pose and were too small to show how detectors would perform in real use. They also argued that the usual per-window evaluation, in which a classifier scores cropped windows, could fail to predict per-image performance. A car-mounted camera fixed the viewpoint and the application: a driver-assistance system must find people at the medium scale, 30 to 80 pixels tall, which with their camera corresponds to roughly 1.5 to 4 seconds ahead at 55 km/h (§2.2). The 2009 paper presented the dataset as two orders of magnitude larger than existing pedestrian sets, and as the first, as far as the authors knew, with both temporal correspondence between boxes and detailed occlusion labels (§2.4). It was built, the authors wrote, not to demonstrate a particular method but as an impartial test bed.

Dataset Size

• Video. About 10 hours of 30 Hz video ($\sim 10^6$ frames) at $640 \times 480$ from a CCD camera, recorded in 11 sessions in five neighbourhoods: LAX, Santa Monica, Hollywood, Pasadena and Little Tokyo (2009, §2.1 and §2.3). The driver was independent of the authors and was told to drive normally through areas with many pedestrians. The video was stabilized to remove the vehicle’s pitching; the 2009 paper describes a differential camera tracker for this, the 2012 paper the inverse compositional algorithm of Baker and Matthews.
• Annotations. 250,000 frames in 137 segments of about a minute, with about 350,000 boxes and 2,300 unique pedestrians (2009, Fig. 3). The 2012 paper calls the labelled part “∼2.3 hours of video”; Zhang et al. (2016) describe the benchmark as 2.5 hours of video. About half of the frames contain no pedestrian and 30% contain two or more; a pedestrian stays visible for about 5 s on average.
• Per split (2009, Table 1, read from the rendered page): training 192k pedestrian boxes, 61k frames without and 67k with pedestrians; testing 155k boxes, 56k and 65k frames. The 10th percentile, median and 90th percentile of pedestrian height are 27, 48 and 97 pixels.
• Evaluated frames. Results are computed on every 30th frame, “starting with the 30th frame” (2012, §3.1). Zhang et al. (2016) give 4,024 test frames and, for the standard “1×” training setting, 4,250 frames with about 1,600 pedestrian crops; some methods train on up to ten times as many frames sampled more densely (“10×”).

Classes

There is one object class, labelled in three ways (2012, §2.1): Person for individuals (about 1,900 instances, counting each labelled sequence as one), People for groups too large or too dense to label person by person (about 300), and Person? when it was unclear whether the object is a pedestrian (about 110). Only Person boxes are scored; People and Person? boxes are ignore regions.

Annotation Types

• Full and visible boxes. In every frame where a pedestrian is visible, annotators drew a box for the full extent of the person, estimating hidden parts (BB-full), and for occluded pedestrians a second box around the visible region (BB-vis). The occlusion fraction is one minus the visible area over the full area (2012, §2.1, §2.2.2). Matching always uses the full box.
• Temporal correspondence. Each box belongs to a labelled track, so a pedestrian’s boxes are linked across frames.
• How the boxes were made. The authors’ labelling tool asks the annotator to label a sparse set of frames and interpolates the boxes in between; the 2012 paper says it used cubic interpolation, applied to each coordinate independently, after tracking-based schemes did worse, and that labelling took about 400 hours including verification (§2.1). Zhang et al. (2016, §3.3) describe the original protocol as linear interpolation that does not follow the up-and-down motion of walking, leaving most frames with shifted boxes. The two sources disagree on the interpolation; we report both.
• Aspect ratio. Annotated box width varies with the position of the limbs. The 2012 paper found the aspect ratio $w/h$ lognormally distributed with a log-average of 0.41 (§2.2.1), the value later used to standardize boxes.

Splits

The 2009 paper split the data by session: sessions 0–5 for training and 6–10 for testing (§2.3). It defined three scenarios, asking authors to use one of them:

• Scenario A: train on any external data, test on sessions 6–10 (used for the paper’s own results, all with pre-trained detectors);
• Scenario B: six-fold cross-validation over sessions 0–5, one session held out at a time;
• Scenario C: train on sessions 0–5, test on sessions 6–10.

In 2009 the test sessions were not released. Authors were asked to submit trained detectors, which the organizers would run, to prevent overfitting and extend the dataset’s lifespan. By 2012 the authors write that they had posted the evaluation code, the ground truth and every detector’s output on the project website (§3). The 2012 evaluation also used Caltech-Training, the training sessions as a test set, and Caltech-Japan, a dataset of the same size and scope recorded in Japan that “cannot be released publicly for legal reasons” (§5.2).

Tasks

The dataset is used for single-frame pedestrian detection, scored by miss rate against false positives per image, as the pedestrian detection article describes. Its labels also support evaluation by subset: by height (near, 80 pixels or more; medium, 30–80; far, 30 or less), by occlusion (partial, 1–35% hidden; heavy, 35–80%) and the reasonable setting of pedestrians at least 50 pixels tall under no or partial occlusion. Because the boxes form tracks over video, detectors that use motion between frames were evaluated alongside single-frame ones; MultiFtr+Motion, one of the three best detectors on the 2012 paper’s reasonable setting, used motion features.

Evaluation

The protocol and its history

Detections are matched greedily to ground-truth boxes at an overlap threshold of 0.5, in order of decreasing score. Boxes outside the evaluated subset become ignore regions, which a detection may match without counting as a true or false positive. The curve of miss rate against false positives per image (FPPI) is summarized by the log-average miss rate over nine FPPI values from $10^{-2}$ to $10^{0}$. The pedestrian detection article derives the log-average, explains ignore regions and their overlap measure $a_o = |B_{dt} \cap B_{ig}| / |B_{dt}|$, and implements both; this article uses that machinery unchanged and does not re-derive it.

The protocol changed substantially between the two papers, which the 2012 authors say makes the evaluation “more accurate and informative” (§3):

	CVPR 2009	TPAMI 2012
Summary number	miss rate at 1 FPPI	log-average miss rate over $[10^{-2}, 10^{0}]$ FPPI
Groups labelled People	split into several overlapping ignore boxes of the same height	one ignore box; a detection may match any subregion ($a_o$ above)
Always ignored	under 20 px, near image borders, Person?, People	under 20 px, truncated by the border, Person?, People
Detections outside the height range	removed before matching (“strict”)	expanded filtering, $r = 1.25$
Box widths	as output, padding cropped	standardized to $0.41\,h$ for ground truth and detections
Detectors	7	16, on six datasets

The 2012 paper names the 2009 rule strict filtering and says that it under-reported performance, as Walk and colleagues had also observed independently (§3.3). Zhang et al. (2016) later extended the FPPI range to $[10^{-4}, 10^{0}]$ and introduced new annotations; they write results on the original annotations as MR$^O$ and on theirs as MR$^N$. The pedestrian detection article explains why the two are not comparable.

Filtering detections and standardizing widths (2012, §3.3–3.4)

To score pedestrians in a height range $[S_0, S_1]$, such as 50 pixels and taller, the ground truth outside the range is set to ignore. The detections must be filtered too, and the 2012 paper compares three ways of doing it:

• Strict filtering: detections outside $[S_0, S_1]$ are removed before matching. A pedestrian inside the range whose only detection falls just outside it becomes a miss, so performance is under-reported.
• Post filtering: every detection may match, and after matching, unmatched detections outside the range are dropped. Their false positives are never counted, so performance is over-reported.
• Expanded filtering: detections outside $[S_0/r, S_1 r]$ are removed before matching. With $r = 1.25$ the paper judged it a good compromise and found detector ranking robust to the exact $r$.

A footnote explains why the paper does not accept strict or post filtering: either can be exploited, on purpose or by accident. Under post filtering, many detections just outside the range can raise the detection rate. Under strict filtering, a detector run in the exact evaluation range keeps all its detections inside it, which can also artificially raise its detection rate; the authors show this with ChnFtrs50, ChnFtrs run only on pedestrians over 50 pixels, whose score is identical under every strategy but whose relative standing is “significantly inflated” under strict filtering. Expanded filtering, they write, cannot be exploited in either way.

Section 3.4 standardizes box widths. Ground-truth width depends on pose, and detectors return boxes of different shapes, from a narrow 0.34 (PLS) to a wide 0.5 (MultiFtr), while the ETH and TUD-Brussels protocols had suggested standardizing to 0.5. The paper sets every box to the Caltech log-mean of 0.41, keeping its height and centre, and states that “the exact constant has only a minor effect” but that detector and ground-truth ratios must match. Standardization had a large positive effect on detectors that return narrow boxes, PLS and LatSvm-V2 among them.

What the code computes

The official evaluation script, dbEval.m, is part of the code in the dataset’s archive on CaltechDATA, which was not downloaded for this article, so it was not read. Dollár’s public toolbox contains the matching and summary functions at commit e873326, under the Simplified BSD licence. Read line by line, they show three things the papers do not:

• The overlap threshold is inclusive. evalRes starts each detection with bstOa = thr and skips a ground-truth box only if oa < bstOa, so an overlap of exactly 0.5 matches. Both papers write that the overlap must “exceed” 50% (Eq. 1). The difference matters only for exact ties; the first line of the example’s output shows one.
• The toolbox’s own evaluation filters no detections. acfTest passes its loading options to bbLoad, whose height range (hRng) only sets out-of-range ground truth to ignore, loads detections unchanged, then calls evalRes and compRoc. Detections of any height therefore count, with no strict, post or expanded rule; expanded filtering lives in dbEval.m, if anywhere in the released code.
• Standardization covers non-ignore ground truth only. bbLoad applies squarify (flag 3: keep height and centre, set $w = 0.41\,h$) to non-ignore boxes, bbs(~ign,:); ignore boxes keep their annotated width. The 2012 paper says all boxes are standardized. The demonstration script acfDemoCal.m tests with 'squarify',{3,.41}, heights from 50 pixels, visible fractions 0.65–1 and boxes within 5 pixels of the border ignored.

Example: filtering strategies and aspect ratios on synthetic boxes

The example scores synthetic detectors on synthetic boxes; there are no images and no people. The filtering strategies are our implementation of the 2012 paper’s description in §3.3, because dbEval.m is unread. Matching, the curve and the log-average are ported from the toolbox (evalRes, compOas, compRoc, the acfTest line and squarify) and carry its licence.

Each of five data seeds draws 100 frames of $640 \times 480$ pixels with Poisson(5) non-overlapping people per frame. Heights are log-uniform between 20 and 120 pixels, and annotated widths are $h \exp(\epsilon)$ with $\epsilon \sim N(\ln 0.41, 0.15^2)$, lognormal around the Caltech log-mean as the 2012 paper describes. The evaluation range is 50 pixels and up ($S_0 = 50$, $S_1 = \infty$), and people outside it are ignore regions, with inclusive bounds as in bbLoad.

The detectors are error models, not trained detectors:

• A, “good”: misses each person with probability 0.1. Otherwise it returns a box whose height is off by up to ±15% and whose centre is off by up to 3% of the height, with score drawn from $N(1, 1)$. It also returns Poisson(0.6) background boxes per frame with scores from $N(0, 1)$.
• B, “range-tuned”: after the paper’s ChnFtrs50, it searches only heights of 50 pixels and up. A’s responses of 42.5–50 pixels come out at 50 pixels, and smaller ones not at all.
• C, “spammer”: A plus a grid of 42- and 46-pixel boxes, 17,160 per frame, all scored below A’s lowest score.

These are design parameters chosen so that the effects can appear, which makes the example an illustration rather than a test of real detectors:

• the background rate of 0.6 per frame makes A’s curve end before 1 FPPI, since low-scored spam can only change the reference points beyond the end of A’s curve;
• the 42.5-pixel cut-off is $50 \times (1 - 0.15)$, so B still returns every in-range person that A finds;
• C’s grid uses no ground-truth positions.

Part 1 standardizes ground truth to 0.41 by bbLoad’s rule and, as the paper describes, every detection too (acfTest leaves detections as loaded), and scores A, B and C under strict filtering, expanded filtering at $r \in \{1.0, 1.1, 1.25, 1.5\}$, post filtering, and no filtering as in acfTest. Part 2 gives the same responses widths of $0.34\,h$ (N), $0.41\,h$ (A) and $0.50\,h$ (W) and scores them under expanded filtering at $r = 1.25$ in four ways:

• raw, without standardization;
• standardized to 0.41 by bbLoad’s rule;
• standardized to 0.41 with the ignore boxes as well, the paper’s “all BBs”;
• standardized to 0.50.

Before the first run we planned these checks:

• hand cases for the port;
• a perfect detector scores the code’s floor of $10^{-10}$ under every strategy;
• expanded filtering at $r = 1$ equals strict filtering;
• B scores the same under every strategy, the paper’s statement about ChnFtrs50, which holds here by construction;
• C equals A under strict filtering;
• N, A and W coincide once standardized.

We also planned these hypotheses from the paper:

• H1: for A, strict > expanded ($r = 1.25$) > post;
• H1b: for A, expanded lies between post and strict for $r = 1.1$ and $1.5$;
• H2: A’s lead over B is largest under strict filtering;
• H3a: C beats A under post filtering;
• H3b: C gains nothing over A under expanded filtering;
• H4a: standardization lowers both N’s and W’s miss rate;
• H4b: N gains more than W;
• H4c: the change from 0.41 to 0.50 is smaller than either gain.

Each outcome had to hold on all five seeds. The design parameters and the hypotheses were fixed before the pilot. The planning scripts that ran earlier printed only the checks; the no-filtering condition was added between two of them, after a reading of acfTest.

The script needs only NumPy (version 2.3.5 here) and runs in about 8 seconds on the drafting machine.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                       # one BLAS thread
import numpy as np

# compOas, evalRes, compRoc and squarify below are ported to NumPy from Piotr's Computer
# Vision Matlab Toolbox, commit e87332637bbe8e8b92dd487c87567d9628404523 (detector/bbGt.m,
# detector/bbApply.m), and the log-average line from detector/acfTest.m. Licence
# (external/bsd.txt at that commit):
#
# Copyright (c) 2012, Piotr Dollar
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this
#    list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice,
#    this list of conditions and the following disclaimer in the documentation
#    and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and documentation are those
# of the authors and should not be interpreted as representing official policies,
# either expressed or implied, of the FreeBSD Project.

REF = 10.0 ** np.arange(-2, 0.01, 0.25)        # acfTest default ref = 10.^(-2:.25:0)

def compOas(dt, gt, ig):
    """bbGt>compOas: IoU, or intersection / area(dt) for ignore gt; 0 if no overlap."""
    w = np.minimum(dt[:, None, 0] + dt[:, None, 2], gt[None, :, 0] + gt[None, :, 2]) - np.maximum(dt[:, None, 0], gt[None, :, 0])
    h = np.minimum(dt[:, None, 1] + dt[:, None, 3], gt[None, :, 1] + gt[None, :, 3]) - np.maximum(dt[:, None, 1], gt[None, :, 1])
    t = w * h
    da = (dt[:, 2] * dt[:, 3])[:, None]
    u = np.where(ig[None, :], da, da + (gt[:, 2] * gt[:, 3])[None, :] - t)
    return np.where((w > 0) & (h > 0), t / u, 0.0)

def evalRes(gt0, dt0, thr=0.5):
    """bbGt>evalRes (mul=0). gt0 rows [x y w h ignore], dt0 rows [x y w h score].
    Returns gt [x y w h match] (-1 ignore, 0 fn, 1 tp) and dt [x y w h score match] (-1, 0 fp, 1 tp)."""
    dt0 = dt0[np.argsort(-dt0[:, 4], kind="stable")]       # highest score first
    gt0 = gt0[np.argsort(gt0[:, 4], kind="stable")]        # ignore last
    gt = gt0.copy(); gt[:, 4] = -gt[:, 4]
    dt = np.c_[dt0, np.zeros(len(dt0))]
    if len(gt) == 0 or len(dt) == 0:
        return gt, dt
    oa = compOas(dt[:, :4], gt[:, :4], gt[:, 4] == -1)
    for d in np.flatnonzero(oa.max(axis=1) >= thr):          # a dt with every oa < thr can match nothing
        bstOa, bstg, bstm = thr, -1, 0
        for g in range(len(gt)):
            m = gt[g, 4]
            if m == 1:
                continue                                       # gt already matched
            if bstm != 0 and m == -1:
                break                                          # matched, and only ignore gt remain
            if oa[d, g] < bstOa:
                continue
            bstOa, bstg, bstm = oa[d, g], g, (1 if m == 0 else -1)
        if bstm == -1:
            dt[d, 5] = -1
        elif bstm == 1:
            gt[bstg, 4] = 1; dt[d, 5] = 1
    return gt, dt

def compRoc(gts, dts, ref=REF):
    """bbGt>compRoc (roc=1): detection rate at each reference FPPI."""
    n_img = len(gts)
    gt = np.concatenate(gts); gt = gt[gt[:, 4] != -1]
    dt = np.concatenate(dts); dt = dt[dt[:, 5] != -1]
    if len(dt) == 0:
        return np.zeros(len(ref))
    tp = dt[np.argsort(-dt[:, 4], kind="stable"), 5]
    fp = np.cumsum(tp != 1); tp = np.cumsum(tp)
    xs1 = np.r_[-np.inf, fp / n_img]; ys1 = np.r_[0.0, tp / len(gt)]
    return np.array([ys1[np.flatnonzero(xs1 <= r)[-1]] for r in ref])

def log_avg_miss(det_rate):
    """acfTest: miss = exp(mean(log(max(1e-10, 1 - miss))))."""
    return float(np.exp(np.mean(np.log(np.maximum(1e-10, 1 - det_rate)))))

def squarify(bb, ar):
    """bbApply>squarify with flag 3: keep height and centre, set width = ar * height."""
    bb = bb.copy(); d = bb[:, 3] * ar - bb[:, 2]
    bb[:, 0] -= d / 2; bb[:, 2] += d
    return bb

# ---------------------------------------------------------------- synthetic frames and detectors
W_IMG, H_IMG, N_FRAMES = 640, 480, 100
S0, S1 = 50.0, np.inf                          # evaluation range: 50 px and taller
AR = 0.41
H_LO, H_HI = 20.0, 120.0
P_MISS, H_ERR, C_ERR, FP_RATE = 0.10, 0.15, 0.03, 0.6
SNAP_LO = 50.0 * (1 - H_ERR)                   # B: boxes of 42.5-50 px snap to 50 px, smaller ones vanish
SPAM_H, SPAM_STEP = (42.0, 46.0), (4, 8)       # C: grid of low-score boxes just under 50 px

def loguniform(rng, lo, hi, size=None):
    return np.exp(rng.uniform(np.log(lo), np.log(hi), size))

def make_frames(rng):
    """Ground truth: [x y w h] per frame; log-uniform heights 20-120 px, annotated widths with
    log-mean aspect 0.41 (log-sd 0.15), inside the frame, no two boxes overlapping."""
    frames = []
    for _ in range(N_FRAMES):
        n, boxes, tries = rng.poisson(5), [], 0
        while len(boxes) < n and tries < 1000:
            tries += 1
            h = loguniform(rng, H_LO, H_HI); w = h * np.exp(rng.normal(np.log(AR), 0.15))
            x, y = rng.uniform(0, W_IMG - w), rng.uniform(0, H_IMG - h)
            if any(x < b[0] + b[2] and b[0] < x + w and y < b[1] + b[3] and b[1] < y + h for b in boxes):
                continue
            boxes.append((x, y, w, h))
        frames.append(np.array(boxes, float).reshape(-1, 4))
    return frames

def detector_core(rng, frames):
    """Error model shared by A, B, N and W: rows [cx cy h score]. Each person is missed with
    probability 0.1, otherwise found with height error within +-15% and centre error within
    +-3% of the height; plus Poisson(0.6) background boxes per frame with log-uniform heights.
    Also returns the largest relative height error of a person's detection (a data check)."""
    out, herr = [], 0.0
    for gt in frames:
        rows = []
        for x, y, w, h in gt:
            if rng.random() < P_MISS:
                continue
            rows.append((x + w / 2 + rng.uniform(-C_ERR, C_ERR) * h, y + h / 2 + rng.uniform(-C_ERR, C_ERR) * h,
                         h * (1 + rng.uniform(-H_ERR, H_ERR)), rng.normal(1.0, 1.0)))
            herr = max(herr, abs(rows[-1][2] / h - 1))
        for _ in range(rng.poisson(FP_RATE)):
            h = loguniform(rng, H_LO, H_HI)
            rows.append((rng.uniform(AR * h / 2, W_IMG - AR * h / 2), rng.uniform(h / 2, H_IMG - h / 2), h, rng.normal(0.0, 1.0)))
        out.append(np.array(rows, float).reshape(-1, 4))
    return out, herr

def to_boxes(core, aspect):
    return [np.c_[c[:, 0] - aspect * c[:, 2] / 2, c[:, 1] - c[:, 2] / 2, aspect * c[:, 2], c[:, 2], c[:, 3]] for c in core]

def range_tuned(core):
    """B: the same responses, but the detector only searches heights of 50 px and up."""
    out = []
    for c in core:
        c = c[c[:, 2] >= SNAP_LO].copy(); c[:, 2] = np.maximum(c[:, 2], S0)
        out.append(c)
    return to_boxes(out, AR)

def spam_grid():
    rows = []
    for h in SPAM_H:
        w = AR * h
        xs, ys = np.arange(0, W_IMG - w, SPAM_STEP[0]), np.arange(0, H_IMG - h, SPAM_STEP[1])
        X, Y = np.meshgrid(xs, ys)
        rows.append(np.c_[X.ravel(), Y.ravel(), np.full(X.size, w), np.full(X.size, h)])
    return np.vstack(rows)

def spammer(rng, a_boxes):
    """C: A plus a grid of boxes at 42 and 46 px in every frame, all scored below A's lowest score."""
    grid, low = spam_grid(), min(b[:, 4].min() for b in a_boxes if len(b)) - 1.0
    return [np.vstack([b, np.c_[grid, low - rng.uniform(0, 1, len(grid))]]) for b in a_boxes]

def exact(frames):
    return [np.c_[squarify(g, AR), np.ones(len(g))] for g in frames]

# ---------------------------------------------------------------- the evaluation with its filters
def evaluate(frames, dets, strategy, r=1.0, std=AR, std_ignore=False, thr=0.5):
    """strategy: 'strict', 'post', 'expanded' (with r) or 'none' (acfTest: detections never filtered). std: aspect ratio for standardization or
    None; non-ignore gt and all detections are standardized (bbLoad's rule), ignore gt too if std_ignore."""
    gts, dts = [], []
    for g_raw, d in zip(frames, dets):
        ign = (g_raw[:, 3] < S0) | (g_raw[:, 3] > S1)          # gt outside the range -> ignore
        g = g_raw.copy(); d = d.copy()
        if std is not None:
            g[~ign] = squarify(g[~ign], std)
            if std_ignore:
                g[ign] = squarify(g[ign], std)
            d[:, :4] = squarify(d[:, :4], std)
        if strategy == "strict":
            d = d[(d[:, 3] >= S0) & (d[:, 3] <= S1)]
        elif strategy == "expanded":
            d = d[(d[:, 3] >= S0 / r) & (d[:, 3] <= S1 * r)]
        gm, dm = evalRes(np.c_[g, ign], d, thr)
        if strategy == "post":                                    # unmatched dt outside the range: removed
            dm[(dm[:, 5] == 0) & ((dm[:, 3] < S0) | (dm[:, 3] > S1)), 5] = -1
        gts.append(gm); dts.append(dm)
    return log_avg_miss(compRoc(gts, dts)), gts, dts

STRATS = [("strict", "strict", 1.0), ("exp r=1.0", "expanded", 1.0), ("exp r=1.1", "expanded", 1.1),
          ("exp r=1.25", "expanded", 1.25), ("exp r=1.5", "expanded", 1.5), ("post", "post", 1.0),
          ("none", "none", 1.0)]
SEEDS = range(5)

def hand_checks():
    E = lambda g, d, thr=0.5: evalRes(np.array(g, float), np.array(d, float), thr)
    g, d = E([[0, 0, 10, 20, 0]], [[0, 0, 10, 10, 1]])
    g2, d2 = E([[0, 0, 10, 20, 0]], [[0, 0, 10, 10, 1]], np.nextafter(0.5, 1))
    print(f"IoU exactly 0.5: bbGt (oa >= thr) dt flag {int(d[0, 5]):+d}, gt flag {int(g[0, 4]):+d};"
          f" paper (> 0.5) dt flag {int(d2[0, 5]):+d}, gt flag {int(g2[0, 4]):+d}")
    g, d = E([[0, 0, 20, 40, 1]], [[5, 5, 10, 10, 0.9], [6, 6, 10, 10, 0.8]])
    print(f"two dt inside one ignore box: dt flags {d[:, 5].astype(int).tolist()}, gt flag {int(g[0, 4]):+d}")
    g, d = E([[0, 0, 40, 40, 1], [0, 0, 10, 20, 0]], [[0, 0, 10, 12, 1]])
    print(f"dt with IoU 0.6 to a person and oa 1.0 to an ignore box: dt flag {int(d[0, 5]):+d}")
    dts = [np.array([[0, 0, 1, 1, .9, 1], [5, 5, 1, 1, .8, 0]]), np.array([[0, 0, 1, 1, .7, 1], [5, 5, 1, 1, .6, 0]])]
    gts = [np.array([[0, 0, 1, 1, 1], [2, 2, 1, 1, 1]]), np.array([[0, 0, 1, 1, 1]])]
    m = log_avg_miss(compRoc(gts, dts))
    print(f"compRoc + log-average on a 2-image toy: {m:.6f} (by hand (2/3)^(7/9) (1/3)^(2/9) = {(2/3)**(7/9) * (1/3)**(2/9):.6f})")
    print(f"squarify [0 0 10 20] to 0.41: {squarify(np.array([[0., 0, 10, 20]]), AR)[0].round(6).tolist()}")

def main():
    hand_checks()
    tab, tab2, checks = {}, {}, []
    for seed in SEEDS:
        rng = np.random.default_rng(seed)
        frames = make_frames(rng)
        core, herr = detector_core(rng, frames)
        A = to_boxes(core, AR); B = range_tuned(core); C = spammer(rng, A)
        allg = np.vstack(frames)
        n_in = int((allg[:, 3] >= S0).sum())
        ov = sum(int((compOas(f, f, np.zeros(len(f), bool)) > 0).sum() - len(f)) for f in frames)
        inside = all(((f[:, :2] >= 0) & (f[:, :2] + f[:, 2:] <= (W_IMG, H_IMG))).all() for f in frames)
        spam = np.vstack([c[len(a):] for a, c in zip(A, C)])
        print(f"\nseed {seed}: {len(allg)} people, {n_in} of 50 px or taller; heights {allg[:, 3].min():.2f}-{allg[:, 3].max():.2f};"
              f" log-mean aspect {np.exp(np.log(allg[:, 2] / allg[:, 3]).mean()):.4f}; overlapping pairs {ov}; inside frame {inside}")
        print(f"  detections: A {sum(len(a) for a in A)} (largest height error of a person's box {herr:.4f}),"
              f" B {sum(len(b) for b in B)} (lowest height {min(b[:, 3].min() for b in B if len(b)):.2f}),"
              f" C {sum(len(c) for c in C)} (spam heights {sorted(set(spam[:, 3].tolist()))},"
              f" highest spam score below A's lowest: {spam[:, 4].max() < min(a[:, 4].min() for a in A if len(a))})")
        for name, dets in (("A", A), ("B", B), ("C", C)):
            for lab, s, r in STRATS:
                tab[(name, lab, seed)] = evaluate(frames, dets, s, r)[0]
        _, _, dpost = evaluate(frames, A, "post")
        print(f"  A under post filtering: {sum(int((d[:, 5] == 0).sum()) for d in dpost)} false positives, final FPPI "
              f"{sum(int((d[:, 5] == 0).sum()) for d in dpost) / N_FRAMES:.2f}")
        ex = [evaluate(frames, exact(frames), s, r)[0] for _, s, r in STRATS]
        checks.append((seed, max(ex),
                       max(abs(tab[(n, "strict", seed)] - tab[(n, "exp r=1.0", seed)]) for n in "ABC"),
                       max(tab[("B", l, seed)] for l, _, _ in STRATS) - min(tab[("B", l, seed)] for l, _, _ in STRATS),
                       abs(tab[("C", "strict", seed)] - tab[("A", "strict", seed)])))
        core_boxes = {"N 0.34": to_boxes(core, 0.34), "A 0.41": A, "W 0.50": to_boxes(core, 0.50)}
        for name, dets in core_boxes.items():
            tab2[(name, "raw", seed)] = evaluate(frames, dets, "expanded", 1.25, std=None)[0]
            tab2[(name, "std 0.41", seed)] = evaluate(frames, dets, "expanded", 1.25, std=AR)[0]
            tab2[(name, "std 0.41 +ign", seed)] = evaluate(frames, dets, "expanded", 1.25, std=AR, std_ignore=True)[0]
            tab2[(name, "std 0.50", seed)] = evaluate(frames, dets, "expanded", 1.25, std=0.50)[0]

    print("\nPart 1: log-average miss rate (%), people of 50 px and up, aspect standardized to 0.41")
    print(f"{'detector':15s}{'filtering':11s}" + "".join(f"{'seed ' + str(s):>9s}" for s in SEEDS))
    for name, desc in (("A", "good"), ("B", "range-tuned"), ("C", "spammer")):
        for lab, _, _ in STRATS:
            print(f"{name + ' ' + desc:15s}{lab:11s}" + "".join(f"{100 * tab[(name, lab, s)]:9.2f}" for s in SEEDS))
    print("\nchecks per seed: exact detector's largest LAMR over strategies; |strict - exp r=1.0| (A, B, C);"
          " B's spread over strategies; |C - A| under strict")
    for s, e, d1, d2, d3 in checks:
        print(f"  seed {s}: {e:.3g}  {d1:.3g}  {d2:.3g}  {d3:.3g}")

    print("\nhypotheses (differences in percentage points, decided unrounded)")
    for s in SEEDS:
        a = lambda l, n="A": 100 * tab[(n, l, s)]
        h1 = a("strict") > a("exp r=1.25") > a("post")
        h1b = all(a("post") <= a(l) <= a("strict") for l in ("exp r=1.1", "exp r=1.5"))
        adv = {l: a(l) - a(l, "B") for l in ("strict", "exp r=1.25", "post")}
        h2 = adv["strict"] > adv["exp r=1.25"] and adv["strict"] > adv["post"]
        g_post, g_exp = a("post") - a("post", "C"), a("exp r=1.25") - a("exp r=1.25", "C")
        h3a, h3b = g_post > 0, g_exp == 0
        print(f"  seed {s}: H1 strict-exp {a('strict') - a('exp r=1.25'):.3f}, exp-post {a('exp r=1.25') - a('post'):.3f} -> {h1};"
              f" H1b {h1b}; H2 A-B strict {adv['strict']:.3f}, exp {adv['exp r=1.25']:.3f}, post {adv['post']:.3f} -> {h2};"
              f" H3 C's gain post {g_post:.3f} -> {h3a}, exp {g_exp:.3g} -> {h3b}")

    print("\nPart 2: log-average miss rate (%), expanded filtering r = 1.25; annotated widths vary (log-mean 0.41)")
    cols = ("raw", "std 0.41", "std 0.41 +ign", "std 0.50")
    for s in SEEDS:
        print(f"  seed {s}: " + " | ".join(f"{n} " + " ".join(f"{100 * tab2[(n, c, s)]:.2f}" for c in cols)
                                        for n in ("N 0.34", "A 0.41", "W 0.50")))
    print(f"  columns: {', '.join(cols)}")
    for s in SEEDS:
        t = lambda n, c: 100 * tab2[(n, c, s)]
        same = max(abs(t(n, c) - t("A 0.41", c)) for n in ("N 0.34", "W 0.50") for c in cols[1:])
        p1 = abs(t("A 0.41", "std 0.41") - 100 * tab[("A", "exp r=1.25", s)])
        gN, gW = t("N 0.34", "raw") - t("N 0.34", "std 0.41"), t("W 0.50", "raw") - t("W 0.50", "std 0.41")
        dc = abs(t("A 0.41", "std 0.50") - t("A 0.41", "std 0.41"))
        print(f"  seed {s}: checks: N, A, W identical once standardized (max diff {same:.3g}); std 0.41 = Part 1 A exp r=1.25 (diff {p1:.3g});"
              f" H4a gains N {gN:.3f}, W {gW:.3f} -> {gN > 0 and gW > 0}; H4b N gains more -> {gN > gW};"
              f" H4c |0.50 - 0.41| {dc:.3f} < min gain -> {dc < min(gN, gW)}")

if __name__ == "__main__":
    main()

IoU exactly 0.5: bbGt (oa >= thr) dt flag +1, gt flag +1; paper (> 0.5) dt flag +0, gt flag +0
two dt inside one ignore box: dt flags [-1, -1], gt flag -1
dt with IoU 0.6 to a person and oa 1.0 to an ignore box: dt flag +1
compRoc + log-average on a 2-image toy: 0.571496 (by hand (2/3)^(7/9) (1/3)^(2/9) = 0.571496)
squarify [0 0 10 20] to 0.41: [0.9, 0.0, 8.2, 20.0]

seed 0: 474 people, 234 of 50 px or taller; heights 20.10-119.32; log-mean aspect 0.4057; overlapping pairs 0; inside frame True
  detections: A 493 (largest height error of a person's box 0.1499), B 294 (lowest height 50.00), C 1716493 (spam heights [42.0, 46.0], highest spam score below A's lowest: True)
  A under post filtering: 35 false positives, final FPPI 0.35

seed 1: 503 people, 248 of 50 px or taller; heights 20.13-119.83; log-mean aspect 0.4104; overlapping pairs 0; inside frame True
  detections: A 534 (largest height error of a person's box 0.1498), B 307 (lowest height 50.00), C 1716534 (spam heights [42.0, 46.0], highest spam score below A's lowest: True)
  A under post filtering: 36 false positives, final FPPI 0.36

seed 2: 499 people, 252 of 50 px or taller; heights 20.11-119.39; log-mean aspect 0.4107; overlapping pairs 0; inside frame True
  detections: A 516 (largest height error of a person's box 0.1500), B 311 (lowest height 50.00), C 1716516 (spam heights [42.0, 46.0], highest spam score below A's lowest: True)
  A under post filtering: 35 false positives, final FPPI 0.35

seed 3: 488 people, 225 of 50 px or taller; heights 20.04-119.57; log-mean aspect 0.4150; overlapping pairs 0; inside frame True
  detections: A 508 (largest height error of a person's box 0.1497), B 277 (lowest height 50.00), C 1716508 (spam heights [42.0, 46.0], highest spam score below A's lowest: True)
  A under post filtering: 37 false positives, final FPPI 0.37

seed 4: 494 people, 230 of 50 px or taller; heights 20.12-119.24; log-mean aspect 0.4141; overlapping pairs 0; inside frame True
  detections: A 493 (largest height error of a person's box 0.1492), B 290 (lowest height 50.00), C 1716493 (spam heights [42.0, 46.0], highest spam score below A's lowest: True)
  A under post filtering: 31 false positives, final FPPI 0.31

Part 1: log-average miss rate (%), people of 50 px and up, aspect standardized to 0.41
detector       filtering     seed 0   seed 1   seed 2   seed 3   seed 4
A good         strict         33.16    29.99    33.75    40.31    36.63
A good         exp r=1.0      33.16    29.99    33.75    40.31    36.63
A good         exp r=1.1      31.82    27.13    31.39    36.58    34.92
A good         exp r=1.25     31.73    29.10    31.51    37.61    35.87
A good         exp r=1.5      31.84    30.91    32.42    41.07    37.21
A good         post           29.64    25.01    28.17    35.66    31.61
A good         none           35.84    37.10    37.01    44.21    39.57
B range-tuned  strict         30.82    26.93    32.21    36.50    34.26
B range-tuned  exp r=1.0      30.82    26.93    32.21    36.50    34.26
B range-tuned  exp r=1.1      30.82    26.93    32.21    36.50    34.26
B range-tuned  exp r=1.25     30.82    26.93    32.21    36.50    34.26
B range-tuned  exp r=1.5      30.82    26.93    32.21    36.50    34.26
B range-tuned  post           30.82    26.93    32.21    36.50    34.26
B range-tuned  none           30.82    26.93    32.21    36.50    34.26
C spammer      strict         33.16    29.99    33.75    40.31    36.63
C spammer      exp r=1.0      33.16    29.99    33.75    40.31    36.63
C spammer      exp r=1.1      31.82    27.13    31.39    36.58    34.92
C spammer      exp r=1.25     31.73    29.10    31.51    37.61    35.87
C spammer      exp r=1.5      31.84    30.91    32.42    41.07    37.21
C spammer      post           27.45    22.85    26.58    34.36    26.88
C spammer      none           35.84    37.10    37.01    44.21    39.57

checks per seed: exact detector's largest LAMR over strategies; |strict - exp r=1.0| (A, B, C); B's spread over strategies; |C - A| under strict
  seed 0: 1e-10  0  0  0
  seed 1: 1e-10  0  0  0
  seed 2: 1e-10  0  0  0
  seed 3: 1e-10  0  0  0
  seed 4: 1e-10  0  0  0

hypotheses (differences in percentage points, decided unrounded)
  seed 0: H1 strict-exp 1.430, exp-post 2.088 -> True; H1b True; H2 A-B strict 2.338, exp 0.908, post -1.180 -> True; H3 C's gain post 2.186 -> True, exp 0 -> True
  seed 1: H1 strict-exp 0.890, exp-post 4.095 -> True; H1b False; H2 A-B strict 3.058, exp 2.168, post -1.927 -> True; H3 C's gain post 2.155 -> True, exp 0 -> True
  seed 2: H1 strict-exp 2.241, exp-post 3.339 -> True; H1b True; H2 A-B strict 1.543, exp -0.697, post -4.036 -> True; H3 C's gain post 1.596 -> True, exp 0 -> True
  seed 3: H1 strict-exp 2.703, exp-post 1.948 -> True; H1b False; H2 A-B strict 3.810, exp 1.107, post -0.841 -> True; H3 C's gain post 1.300 -> True, exp 0 -> True
  seed 4: H1 strict-exp 0.760, exp-post 4.264 -> True; H1b False; H2 A-B strict 2.367, exp 1.608, post -2.657 -> True; H3 C's gain post 4.723 -> True, exp 0 -> True

Part 2: log-average miss rate (%), expanded filtering r = 1.25; annotated widths vary (log-mean 0.41)
  seed 0: N 0.34 33.37 31.73 31.73 34.80 | A 0.41 31.73 31.73 31.73 34.80 | W 0.50 36.91 31.73 31.73 34.80
  seed 1: N 0.34 32.27 29.10 29.10 29.10 | A 0.41 29.10 29.10 29.10 29.10 | W 0.50 30.36 29.10 29.10 29.10
  seed 2: N 0.34 34.45 31.51 31.51 31.51 | A 0.41 31.51 31.51 31.51 31.51 | W 0.50 32.11 31.51 31.51 31.51
  seed 3: N 0.34 39.87 37.61 37.61 37.61 | A 0.41 37.61 37.61 37.61 37.61 | W 0.50 38.67 37.61 37.61 37.61
  seed 4: N 0.34 43.78 35.87 35.87 35.87 | A 0.41 35.87 35.87 35.87 35.87 | W 0.50 36.63 35.87 35.87 35.87
  columns: raw, std 0.41, std 0.41 +ign, std 0.50
  seed 0: checks: N, A, W identical once standardized (max diff 0); std 0.41 = Part 1 A exp r=1.25 (diff 0); H4a gains N 1.640, W 5.179 -> True; H4b N gains more -> False; H4c |0.50 - 0.41| 3.074 < min gain -> False
  seed 1: checks: N, A, W identical once standardized (max diff 0); std 0.41 = Part 1 A exp r=1.25 (diff 0); H4a gains N 3.164, W 1.259 -> True; H4b N gains more -> True; H4c |0.50 - 0.41| 0.000 < min gain -> True
  seed 2: checks: N, A, W identical once standardized (max diff 0); std 0.41 = Part 1 A exp r=1.25 (diff 0); H4a gains N 2.935, W 0.596 -> True; H4b N gains more -> True; H4c |0.50 - 0.41| 0.000 < min gain -> True
  seed 3: checks: N, A, W identical once standardized (max diff 0); std 0.41 = Part 1 A exp r=1.25 (diff 0); H4a gains N 2.256, W 1.063 -> True; H4b N gains more -> True; H4c |0.50 - 0.41| 0.000 < min gain -> True
  seed 4: checks: N, A, W identical once standardized (max diff 0); std 0.41 = Part 1 A exp r=1.25 (diff 0); H4a gains N 7.907, W 0.759 -> True; H4b N gains more -> True; H4c |0.50 - 0.41| 0.000 < min gain -> True

All the planned checks held:

• Hand cases. An overlap of exactly 0.5 is a match under the code’s rule and a miss plus a false positive under the papers’ wording; two detections share one ignore box; a detection prefers a person to an ignore box it overlaps more; the toy curve gives $(2/3)^{7/9}(1/3)^{2/9}$; squarify keeps the centre.
• Perfect detector: $10^{-10}$ under every strategy on every seed.
• Identities: expanded filtering at $r = 1$ reproduces strict filtering exactly for all three detectors; B’s seven rows are identical; C’s strict row equals A’s.
• Part 2: N, A and W print the same three standardized values on every seed.
• Data checks: every printed property is as planned.

Strict, post and expanded (H1 held, H1b failed). For the good detector, strict filtering gave the highest of the three paper strategies and post filtering the lowest on every seed. Strict minus expanded ($r = 1.25$) was 0.76–2.70 points and expanded minus post 1.95–4.26. By construction, strict filtering throws away A’s boxes of 42.5–50 pixels for pedestrians just above 50 pixels, which then count as misses. Post filtering keeps those matches but forgives all of A’s unmatched boxes under 50 pixels, which expanded filtering counts down to 40 pixels.

H1b held for $r = 1.1$, which lay between post and strict on every seed. It failed for $r = 1.5$, which scored above strict on seeds 1, 3 and 4 (30.91 against 29.99, 41.07 against 40.31, 37.21 against 36.63) and between the two on seeds 0 and 2. By construction, A’s box for any person of 50 pixels or more is at least 42.5 pixels tall, so every $r \ge 1/0.85 \approx 1.18$ keeps all of them, and widening from 1.25 to 1.5 adds only boxes of 33–40 pixels, from the background or from people under 50 pixels; whether those outweigh the true positives that strict filtering loses, the construction does not decide. The no-filtering row of acfTest is the highest of all, 35.84–44.21: every unmatched box of any height counts.

The range-tuned detector (H2 held). B printed the same value in all seven rows, as the paper reports for ChnFtrs50. Its standing moved with the strategy:

• under strict filtering it beat A on every seed by 1.54–3.81 points;
• under post filtering it lost to A on every seed, by 0.84–4.04 points;
• under expanded filtering ($r = 1.25$) it was ahead by 0.91–2.17 points on four seeds and behind by 0.70 on seed 2.

B returns a box for every in-range person A finds (those of 42.5–50 pixels raised to 50) and moves A’s other boxes of 42.5–50 pixels into the range as well. It wins most where strict filtering takes true positives away from its rival, which is the inflation the footnote describes.

The spammer (H3a and H3b held). C’s rows equal A’s everywhere except under post filtering. There, its 1.7 million spam boxes lowered the log-average miss rate by 1.30–4.72 points on every seed without adding a false positive: every unmatched spam box lies outside the range and is removed, and in our reading the few that overlap a pedestrian A missed, just above 50 pixels, become true positives at the end of the curve. Under expanded filtering C’s gain was exactly zero on every seed. Every spam box of 40 pixels or more counts there, and by construction fewer than a hundred spam false positives (1 FPPI on 100 frames) carry the curve past the last reference point; the printed zero shows that none of the spam’s true positives arrived before it.

Aspect ratios (H4a held; H4b and H4c failed on seed 0). Without standardization the narrow detector scored 32.27–43.78 and the wide one 30.36–38.67, against 29.10–37.61 for the matched 0.41 detector. Standardization lowered both N and W on every seed, by 1.64–7.91 and 0.60–5.18 points. Because it rewrites all three to the same boxes, their difference disappears entirely, not just mostly.

H4b, that the narrow detector gains more, held on seeds 1–4 and failed on seed 0, where W gained 5.18 points against N’s 1.64. H4c, that the constant matters less than matching it, held on seeds 1–4, where standardizing to 0.50 instead of 0.41 changed nothing at the printed precision. It failed on seed 0, where 0.50 gave 34.80 against 31.73, a change larger than N’s gain on that seed; the printed output does not show which match moved. In this set-up the open choice of whether to standardize ignore boxes made no difference: the two 0.41 columns are identical on every seed and detector.

License

As of November 2026 the CaltechDATA record f6rph-90m20, “Caltech Pedestrians”, gives its rights as Creative Commons Attribution 4.0 International (CC BY 4.0), linking the legal code. Its files are public, with no registration or access request. The record lists the four authors, a publication date of 20 June 2009 (the conference) and the CVPR 2009 paper’s DOI as its identifier. It was created on 11 March 2023 and updated on 13 March 2023, when, according to the record, “data_and_labels” was added. We read the record’s metadata only and downloaded no files.

The toolbox code used for evaluation is under the Simplified (two-clause) BSD licence in external/bsd.txt, copyright 2012 Piotr Dollár; its files carry the line “Licensed under the Simplified BSD License”. Licence terms of the original 2009 project page were not checked for this article.

Limitations and Bias

• One city, one camera. Every frame comes from one car-mounted camera in the Los Angeles area, at $640 \times 480$, with image quality that the authors describe as lower than that of still images. The pedestrian detection article describes later datasets recorded in many cities, such as CityPersons.
• Small people. The median height is 48 pixels, so the benchmark weighs heavily the medium scale, at which the 2009 and 2012 papers found performance poor.
• Annotation alignment and errors. Zhang et al. (2016) found systematic offsets from the interpolation, false and missing annotations, and inconsistent use of ignore regions. A detector trained on INRIA (Roerei) had a median IoU of its true positives of 0.84 against their re-annotated test set and 0.76 against the original boxes (Table 3).
• Sparse evaluation. Scoring every 30th frame keeps evaluation affordable, which the 2012 paper gives as the reason, but it uses a small, correlated sample of the video: 4,024 test frames.
• Protocol sensitivity. As the example shows, the filtering rule and the box standardization can move a detector’s score by several points, a detector tuned to the protocol can move its rank, and the threshold convention decides exact ties. Results are comparable only under the same code. The 2012 paper stresses using “the exact same evaluation code (as opposed to a re-implementation)”. Dataset bias discusses the general problem.

Common Uses

• Benchmarking pedestrian detectors on the reasonable subset and on the scale and occlusion subsets, with the log-average miss rate.
• Training detectors on the 1× or 10× training frames.
• Studying small and occluded pedestrians, using the full and visible boxes.
• Cross-dataset evaluation alongside INRIA, ETH, TUD-Brussels, Daimler, KITTI and CityPersons.

Historical Milestones

• 2009. The dataset and benchmark. Seven pre-trained detectors were scored on full frames. On the whole test set MultiFtr did best, with HOG second, but with a miss rate above 80% at 1 FPPI. On unoccluded pedestrians at the medium scale MultiFtr’s miss rate was 72% at 1 FPPI (2009, §4).
• 2012. The re-evaluation: sixteen detectors on six datasets under the revised protocol. On the reasonable subset of the Caltech test set, MultiFtr+Motion, ChnFtrs and FPDW led with log-average miss rates of 51–57%, and the authors called performance on far-scale and heavily occluded pedestrians abysmal (§5.1). The paper used the reasonable setting for all its results on the other datasets (§5.2).
• 2016. Zhang et al. collected a human baseline and re-annotated the training (1×) and test frames. Against the CVPR 2015 snapshot of the benchmark, in which Checkerboards led at 18.47%, the human baseline reached 5.62%, and the authors wrote that detectors made ten times as many errors as humans at about 95% recall (Fig. 1). The pedestrian detection article follows the later results on the original and new annotations.

Related Datasets

• INRIA Person, the earlier standard set of mostly large, unoccluded people from photographs, described in the HOG paper article. Its log-mean aspect ratio is 0.33 (2012, §3.4).
• ETH, TUD-Brussels and Daimler, the other urban, vehicle- or stroller-mounted sets in the 2012 evaluation.
• Caltech-Japan, the authors’ unreleased companion set recorded in Japan.
• KITTI, which scores pedestrians with average precision instead.
• CityPersons, person annotations on Cityscapes, evaluated with the Caltech protocol (pedestrian detection).
• Caltech-101, an object-recognition set from the same Caltech group, unrelated in content.

Resources

• CaltechDATA record f6rph-90m20: CaltechPedestrians.zip (52,018,026 bytes, which the record describes as including “folders containing the dataset, code, files, and ROC curve results”) and data_and_labels.zip (11,848,898,217 bytes), about 11.9 GB in all.
• Piotr Dollár’s research page: the authors’ copies of the 2009 and 2012 papers, with a link to the original project page on www.vision.caltech.edu, which was not checked for this article.
• Piotr’s Computer Vision Matlab Toolbox at e873326: detector/bbGt.m, bbApply.m, acfTest.m and acfDemoCal.m.
• Zhang et al. (2016) give http://www.mpi-inf.mpg.de/pedestrian_detection_cvpr16 for their new annotations; we did not request it.