Video Surveillance and Security

Video surveillance, in the computer vision literature, is the automatic analysis of video from fixed cameras that watch public or controlled spaces, roads and buildings: finding the people and vehicles in view, following them within and across cameras, describing them, recognizing what they do, and making the recordings searchable. The sources also call it visual surveillance (Ristani & Tomasi, 2018), and its traffic branch, which watches roads rather than people, city-scale video analytics (Tang et al., 2019). Its papers motivate the work by the volume of video: MEVA’s authors cite an estimate of 180 million security cameras shipped worldwide in 2019 against an attention span of about 20 minutes for a human camera operator (Corona et al., 2021).

Multi-Object Tracking and Person Re-Identification describe the two methods this application leans on most. This article covers what is specific to surveillance video: the data, the chain of tasks as the sources describe it, the multi-camera benchmarks and their protocols, how results are scored, the failure modes and accuracy findings the sources document, and the privacy and civil-liberty concerns stated in official documents. It describes methods, evidence and limits as the cited sources report them. It gives no guidance on deploying, configuring or operating surveillance systems, and it attributes every result to its authors with its set-up.

Problem

The task, as the multi-camera literature frames it, is to determine the position of every person at all times from video taken by many cameras (Ristani & Tomasi, 2018), or, for traffic, to follow vehicles across intersections spread over a city (Tang et al., 2019). The cameras are fixed, usually far apart and often without overlapping views, so a target disappears for an unknown time and reappears from another angle under other lighting, and the number of targets is not known in advance (Ristani & Tomasi, 2018). Activity detection adds a second problem: finding instances of defined activities in long, untrimmed, continuous recordings that also contain spontaneous background activity (Corona et al., 2021).

The early systems already had this shape. The Video Surveillance and Monitoring (VSAM) project of 1997–1999, run by Carnegie Mellon University and the Sarnoff Corporation under DARPA funding, combined adaptive background subtraction with three-frame differencing to detect moving objects, classified them as human, human group, car or truck, estimated their positions on a site model, and handed targets between cooperating pan-tilt-zoom cameras, with a single operator watching the results on a map; its stated motivation was battlefield awareness, with security and law enforcement uses listed beside it (Collins et al., 2000). Later work replaced the components with learned detectors, trackers and embeddings, and moved evaluation onto shared benchmarks, but the decomposition into detection, tracking, cross-camera association and higher-level description has stayed.

Visual Data

The data are long, continuous recordings from static cameras, and their properties differ from the short, centred clips of action recognition datasets, as the dataset papers stress.

• Resolution and scale. VIRAT’s ground-camera release was recorded by stationary HD cameras at 1080p or 720p (VIRAT Release 2.0 introduction), and its site describes person heights of 10 to 200 pixels and frame rates of 2 to 30 Hz as part of the design. In MEVA, one actor measured 301, 118, 176 and 89 pixels tall in four views of the same scenario, and annotated boxes had a mean area of 13,559 pixels with a standard deviation of 23,799 (Corona et al., 2021).
• Overlapping and disjoint views. Multi-camera datasets differ in whether fields of view overlap. WILDTRACK’s seven cameras all overlap, while in DukeMTMC only two of eight camera views overlap slightly (Chavdarova et al., 2018), and DukeMTMC was evaluated on disjoint fields of view (Ristani & Tomasi, 2018). CityFlow mixes both: cameras at one intersection sometimes overlap, and the farthest two simultaneous cameras are 2.5 km apart (Tang et al., 2019).
• Lenses and motion. Some CityFlow cameras use fisheye lenses with strong radial distortion, and fast vehicles cause motion blur that can break detection and association (Tang et al., 2019).
• Other modalities. MEVA adds thermal infrared cameras, UAV footage and hand-held or body-worn cameras to its static RGB views (Corona et al., 2021). VSAM used colour pan-tilt-zoom cameras, low-light monochrome cameras and thermal sensors (Collins et al., 2000).
• Faces in video. NIST’s Face in Video Evaluation lists why faces in such footage are harder than portraits: subjects may be absent for long periods and clutter causes false detections, people move so faces must be tracked through blur and occlusion, several people must be localized at once over varying scale and head pose, and detail is lost to motion blur and to the compression used for transmission and storage (Grother, Quinn & Ngan, 2017).
• Calibration and geometry. Multi-camera benchmarks provide camera models so that positions can be compared on the ground. CityFlow gives homographies from each image to GPS coordinates, estimated from 5 to 14 hand-picked landmarks per view with an average reprojection error of 11.52 pixels, which its authors attribute to the limited precision of the map source (Tang et al., 2019); WILDTRACK provides a joint intrinsic and extrinsic calibration of its seven views (Chavdarova et al., 2018).

Vision Tasks Involved

The sources describe a chain of tasks, each of which has its own literature; the outlines below link the articles that cover them.

Detection and background subtraction

With a fixed camera, moving objects can be found by comparing each frame with a model of the empty scene. Background Subtraction covers frame differencing, per-pixel mixtures and the CDnet benchmark; VSAM’s hybrid combined an adaptive background model, which leaves short-lived “holes” of false alarms where a stationary object starts to move, with three-frame differencing, which misses the interior of objects of uniform intensity (Collins et al., 2000). Current systems use trained detectors instead or in addition, as Pedestrian Detection and Object Detection describe: CityFlow’s baselines ran YOLOv3, SSD512 and Faster R-CNN trained on COCO (Tang et al., 2019), and all teams in the 2024 AI City Challenge’s people-tracking track used YOLO-family detectors (Wang et al., 2024). Where cameras overlap, detections can be fused on the ground plane: on WILDTRACK, multi-view detectors outperformed monocular ones, and the rows for a ResNet-18 evaluated view by view, which we read as the paper’s monocular detector, have negative detection accuracy (MODA) in six of the seven views, with recall of 0.71–1.00, mostly near 1, but precision of 0.26–0.47, so that misses and false positives together outnumber the people present (Chavdarova et al., 2018, Table 3).

Single-camera tracking

Detections are linked into trajectories within each camera, usually by tracking-by-detection, as Multi-Object Tracking describes; following one target given only by a box in the first frame is the separate task of Single Object Tracking, while the benchmarks below are multi-target. In CityFlow’s baselines and the AI City Challenge reports, single-camera tracking is a component taken from the general tracking literature: DeepSORT and an offline tracklet-clustering method among CityFlow’s baselines (Tang et al., 2019), and BoT-SORT, StrongSORT and ByteTrack among the 2024 entries (Wang et al., 2024). Ristani and Tomasi (2018) found the detector mattered most at this stage: replacing DPM detections with OpenPose raised single-camera IDF1 on the DukeMTMC validation sequence from 75.0 to 85.5, but multi-camera IDF1 only from 54.98 to 58.24 with their colour-histogram features.

Multi-camera tracking

Multi-target multi-camera (MTMC) tracking links the single-camera trajectories of one target across the network. Tang et al. (2019) describe it as the combination of single-camera tracking, image-based re-identification and spatio-temporal information. The spatio-temporal part constrains which links are plausible: in their comparison, a probabilistic model of transit time between cameras, specified by hand (FVS) or learned online from camera pairs shared between training and test (2WGMMF), worked better than re-ranking appearance matches by the ratio of time difference to distance (PROVID), and both transit-time methods required entry and exit zones to be marked by hand in each view. Ristani and Tomasi (2018) instead cast association as correlation clustering on learned appearance correlations, reasoning hierarchically over one-second tracklets, single-camera trajectories in a 10-second window and multi-camera identities in a 1.5-minute window. They conclude that good features are crucial for multi-camera tracking: learned ResNet features added 10.7 points of multi-camera IDF1 on the validation sequence, where the better detector had added about three. The example below isolates this step.

Person and vehicle re-identification

Re-identification ranks a gallery of crops by appearance similarity to a query crop from another camera; Person Re-Identification covers its descriptors, losses, benchmarks and protocols, including DukeMTMC-reID, which was cut from the multi-camera dataset described below. Vehicles are harder in one respect: many models from different manufacturers look alike, and one vehicle looks different from different angles, so inter-class variation is small and intra-class variation large (Tang et al., 2019; Naphade et al., 2019). On CityFlow-ReID, the best person re-identification model and the best vehicle model both reached 32.0% mAP (Tang et al., 2019, Tables 4 and 6), while the same vehicle models reached up to 67.6% on VeRi-776 and 78.2% on VehicleID (Table 7), a comparison the paper draws across datasets with different test sets and protocols; the authors attribute most failures to viewpoint changes. Tang et al. state that licence plate information should not be used, to preserve drivers’ privacy, and CityFlow’s plates and faces are redacted.

Attribute recognition

Pedestrian attribute recognition labels a person crop with properties such as gender, age, clothing, carried objects or a phone. PA-100K contains 100,000 crops from 598 outdoor surveillance scenes with 26 attributes; its authors discarded people with motion blur or a size below 50 × 100 pixels and split the data 8:1:1, assigning the samples of each tracklet to one set, which they call fairer than the earlier PETA dataset’s random split, made with no consideration of identity (Liu et al., 2017). They score with mean accuracy over attributes (mA) and instance-based accuracy, precision, recall and F1; their HydraPlus-Net reached 74.21% mA on PA-100K. In the AI City Challenge, semantic attributes of vehicles (type, travel direction) were used by the two leading vehicle re-identification teams in 2019, but the organizers note that they need extra annotation and were not widely used (Naphade et al., 2019).

Activity, event and anomaly detection

Activity detection finds and localizes instances of defined activities in untrimmed video. VIRAT Release 2.0 defines 12 event types involving people and vehicles, such as loading an object into a vehicle or getting out of one, each with written start and end rules (VIRAT Release 2.0 introduction), and MEVA defines 37 activity types, from opening a door to “person abandons package”, including activities its authors call threat-based (Corona et al., 2021). Related tasks with their own articles are spatio-temporal action detection and temporal action localization. Anomaly detection, which flags what departs from learned normal activity rather than named classes, is covered with its UCSD, Avenue, ShanghaiTech and UCF-Crime benchmarks in Video Anomaly Detection, including what those benchmarks count as abnormal. The traffic version appears in the 2019 AI City Challenge, whose Track 3 asked teams to detect accidents and stalled vehicles in more than 50 hours of state highway video from Iowa, scored by F1 times one minus the normalized root-mean-square error of the detected start time (Naphade et al., 2019).

Retrieval and search

Search over archives can start from an image (re-identification), from attributes or from text. Li et al. (2017) introduced CUHK-PEDES, 40,206 images of 13,003 people drawn from five re-identification datasets with 80,412 sentences written by crowd workers, and ranked images by their affinity to a sentence; their best model placed the correct person first for 19.05% of queries and within the top ten for 53.64%. In their study with human searchers, sentences located the right person among 20 candidates more often than attribute lists (58.7% against 33.3% top-1). Language also appears in the 2024 AI City Challenge, whose Track 2 asked for captions describing pedestrians and vehicles before and during traffic incidents (Wang et al., 2024). Image Retrieval covers indexing for large galleries.

Face recognition in video

Recognizing faces in surveillance footage is one-to-many identification of people who are not cooperating with the camera. Face Recognition and Biometric Recognition Systems cover the methods and their evaluation; NIST’s measurements on video are summarized under Documented Accuracy and Bias Findings below.

Datasets and Benchmarks

Benchmark	Content	Cameras	Size	Task and main metric
PETS 2009	actor set-up, few distinct individuals	7, overlapping	795 frames, 19 identities	multi-view detection and tracking
VIRAT Release 2.0	people and vehicles, mostly unscripted	stationary HD, 11 scenes, no overlap	about 8.5 hours	event detection, 12 types
DukeMTMC (retracted)	pedestrians on a campus	8, mostly disjoint	85 minutes per camera	MTMC tracking, IDF1
WILDTRACK	unscripted pedestrians in a public area	7, overlapping	7 × 400 annotated frames at 2 fps, 313 identities	multi-view detection (MODA), tracking (IDF1)
CityFlow	vehicles at 10 intersections	40, overlapping and disjoint	3.25 hours, 229,680 boxes, 666 identities	MTMC tracking (IDF1), re-identification (mAP)
MEVA	scripted and spontaneous activities by about 100 actors	38 RGB and thermal IR, plus UAV	over 9,300 hours, 144 annotated	activity detection (ActEV)
AI City 2024 Track 1	synthetic people in six indoor environments	953	212 hours	MTMC tracking in 3D, HOTA

Sources: Chavdarova et al. (2018, Table 1) for PETS 2009, DukeMTMC and WILDTRACK; Ristani and Tomasi (2018) for DukeMTMC’s evaluation; the VIRAT site and Release 2.0 introduction; Tang et al. (2019); Corona et al. (2021); Wang et al. (2024, Section 3.1).

PETS 2009 and WILDTRACK. The PETS 2009 dataset was presented at the Twelfth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (Ferryman & Shahrokni, 2009); that paper was not read for this article. As WILDTRACK’s authors describe it, the S2.L1 sequence was the most widely used multi-camera sequence with overlapping views, but it is only several hundred frames long, its calibration is inaccurate (partly because the scene slopes), and it was recorded in an actor set-up with few distinct individuals, which they argue prevents fair benchmarking of appearance-based methods (Chavdarova et al., 2018). WILDTRACK replaced it with seven synchronized, jointly calibrated 1080p cameras over a public area with unscripted dense groups, more than 40,000 boxes in 400 annotated frames per view. It scores detection on the ground plane: a detection is matched to a ground-truth position by Hungarian matching if it lies within $r = 0.5$ m, roughly the width of a body, and MODA combines the normalized misses and false positives. The best detector reported, Deep-Occlusion with KSP tracking, reached a MODA of 0.752; with tracking evaluated at $r = 1$ m, KSP with a motion-pattern refinement reached IDF1 78.4 (Chavdarova et al., 2018, Tables 2 and 3).

DukeMTMC. DukeMTMC (Ristani et al., 2016; that paper was not read for this article) was recorded by eight 1080p, 60 fps cameras on the Duke University campus, 85 minutes per camera. Ristani and Tomasi (2018) describe 2.8k identities, of which 1.8k are in the training and validation set, while WILDTRACK’s table lists about 2,000; the benchmark’s test-easy (25 minutes) and test-hard (15 minutes, a group of 50 people crossing four cameras) sequences were hosted on MOTChallenge. On them, Ristani and Tomasi’s DeepCC reached multi-camera IDF1 82.0 and 68.5, against single-camera IDF1 89.2 and 79.0 on the same sequences, drops of 7.2 and 10.5 points. The dataset has been withdrawn. Peng, Mathur and Narayanan (2021) report that DukeMTMC was retracted by its creators in April 2019, and MEVA’s comparison table lists its availability as “retracted” (Corona et al., 2021); Person Re-Identification summarizes Peng et al.’s account of the retraction and of the continued use of derived datasets such as DukeMTMC-reID. Results on DukeMTMC remain in the literature and in comparison tables, including those cited here.

CityFlow and the AI City Challenge. CityFlow contains 3.25 hours of synchronized video from 40 cameras at 10 intersections in a mid-sized U.S. city, at least 960p and mostly 10 fps, with 229,680 boxes of 666 vehicles, each of which passes at least two cameras; scenarios 1, 3 and 4 are for training and 2 and 5 for testing (Tang et al., 2019). The 2019 challenge report describes the same Track 1 data as nearly 3 hours of video (Naphade et al., 2019). Annotation started from detector and tracker output that was corrected by hand, and cross-camera links were made by hand with spatio-temporal cues. CityFlow-ReID, its re-identification subset, has 56,277 crops of 666 vehicles split evenly between training and test identities. In the paper’s grid of spatio-temporal, single-camera and re-identification methods, the best MTMC IDF1 was 46.3%, with FVS transit-time modelling, offline tracklet clustering and a batch-all triplet embedding (Tang et al., 2019, Table 9). The 2019 AI City Challenge used CityFlow for Track 1, received 22 team submissions, and the best IDF1 was 0.7059, from a system combining single-camera tracking, deep re-identification features and camera link models; the organizers noted that most algorithms were still far from real-time processing (Naphade et al., 2019).

Later editions changed both the data and the metric. The 2024 people-tracking track was fully synthetic, rendered with NVIDIA’s Omniverse platform in six indoor environments, with 90 subsets (40 training, 20 validation, 30 test), 953 cameras, 2,491 people and 212 hours of 1080p video at 30 fps; the report’s own summary of the track gives roughly 1,300 cameras and about 3,400 people instead (Wang et al., 2024). The 2023 edition had scored with IDF1; in 2024 teams were ranked by HOTA computed on 3D positions, with similarity falling to zero beyond 2 m, and online methods received a 10% bonus. On the leaderboard the best offline entry scored 71.9 HOTA and the best online one 67.2; the report does not say whether the listed scores include the bonus. The organizers list limitations of the entries: separate, computationally heavy models for detection and pose, rule-based rather than learned multi-camera refinement, and teams that assumed the number of people was known (Wang et al., 2024).

VIRAT, MEVA and ActEV. VIRAT Release 2.0 contains about 8.5 hours of stationary ground video from 11 outdoor scenes, with people, vehicles and objects annotated by crowd workers; the release document warns that these annotations may not satisfy its guidelines and that tracks sometimes fragment (VIRAT Release 2.0 introduction). Its site describes the data as natural scenes in which actions by directed actors were minimized, collected at several sites in the United States. MEVA’s comparison table lists 23 activity types and 29 hours for VIRAT, counts that differ from the Release 2.0 document (Corona et al., 2021). MEVA was collected to fill the gap its authors see between focused action datasets and security video: over 9,300 hours of continuous video from 38 RGB and thermal infrared cameras with overlapping and singular views, indoors and outdoors, recorded over about three weeks at an access-controlled facility, with scripted scenarios, periods of unscripted collection and spontaneous background activity. 144 hours are annotated with 37 activity types, of which 122 hours are sequestered for NIST’s Activities in Extended Video (ActEV) evaluation. The authors compared their annotations with high-confidence false alarms from ActEV systems and report a confirmed false negative rate below 0.6%. ActEV defines activity detection without spatial localization, activity and object detection, and activity and object detection and tracking; its leaderboard on MEVA scored activity detection by the probability of missing an activity for at least one second against the time-based false alarm rate, and the authors describe MEVA as very difficult compared with related datasets (Corona et al., 2021).

Attributes and language. PA-100K and CUHK-PEDES, described above, are built from surveillance-style crops; CUHK-PEDES reuses images of earlier re-identification datasets (Li et al., 2017).

Example: Cross-Camera Association Scored by IDF1

The example isolates the cross-camera step of MTMC tracking. Single-camera tracking is assumed perfect, so each appearance of an identity in a camera is one clean tracklet, and the only question is which tracklet in camera B continues which tracklet in camera A. The identities are abstract: each is a 24 × 12 RGB patch of six horizontal bands in three random colours, with a row texture in the middle bands for 30% of them. Each of 80 identities per seed passes through camera A and then camera B with probability 0.6, and is seen only in A or only in B with probability 0.2 each. A tracklet lasts 15–30 frames, and the delay between leaving A and entering B is 20–80 frames. Camera B applies a fixed colour gain and offset. Appearance drifts within each tracklet as a per-channel log-gain random walk whose step size $s$ is the noise level, starting from a random offset of standard deviation $2s$, with pixel noise on top.

The descriptor is the mean colour of each band, averaged over the tracklet and standardized within each camera. Four association rules are compared: nearest neighbour (each B tracklet takes the identity of its closest A tracklet, so several B tracklets can share one), nearest neighbour within a time window (delay between 15 and 90 frames), one-to-one Hungarian assignment, and Hungarian assignment within the window. Each runs with and without a gate that refuses links whose distance exceeds half the median of all A–B distances in that run; the sources give no threshold, so both settings are printed. Unlinked tracklets keep their own identity, and a no-linking baseline gives every tracklet its own identity. IDF1 is computed over all frames of both cameras exactly as py-motmetrics computes it: a square cost matrix with dummy rows and columns for unmatched identities, solved with the Hungarian algorithm, and IDF1 $= 2\,\text{IDTP} / (\text{ground-truth detections} + \text{predicted detections})$ (Heindl et al., py-motmetrics); IDF1 gives the definition and its properties. The script checks the result once against the equivalent maximum-overlap form, runs seeds 0, 1 and 2, and prints the mean and range over seeds.

import numpy as np
from scipy.optimize import linear_sum_assignment

H, W, BANDS = 24, 12, 6
NOISE = [0.0, 0.025, 0.05, 0.1, 0.2]
GAIN_B, OFFSET_B = np.array([0.85, 1.00, 1.15]), 0.03
WINDOW = (15, 90)           # allowed delay from leaving A to entering B, in frames
BIG = 1e6


def make_identities(rng, n=80):
    ids = []
    for k in range(n):
        cols = rng.uniform(0.1, 0.9, (4, 3))
        textured = rng.random() < 0.3
        u = rng.random()
        route = "AB" if u < 0.6 else ("A" if u < 0.8 else "B")
        ids.append(dict(cols=cols, textured=textured, route=route))
    return ids


def render(ident, rng):
    """Clean 24 x 12 patch: two bands per base colour, optional row texture in the middle."""
    patch = np.zeros((H, W, 3))
    rows_per_band = H // BANDS
    for b in range(BANDS):
        patch[b * rows_per_band:(b + 1) * rows_per_band] = ident["cols"][b // 2]
    if ident["textured"]:
        for r in range(2 * rows_per_band, 4 * rows_per_band, 2):
            patch[r] = ident["cols"][3]
    return patch


def schedule(ids, rng):
    """One tracklet per identity per camera it visits: (identity, camera, start, end)."""
    tracklets = []
    for k, ident in enumerate(ids):
        if "A" in ident["route"]:
            a0 = int(rng.integers(0, 601)); a1 = a0 + int(rng.integers(15, 31))
            tracklets.append((k, "A", a0, a1))
            if ident["route"] == "AB":
                b0 = a1 + int(rng.integers(20, 81)); b1 = b0 + int(rng.integers(15, 31))
                tracklets.append((k, "B", b0, b1))
        else:
            b0 = int(rng.integers(20, 681)); b1 = b0 + int(rng.integers(15, 31))
            tracklets.append((k, "B", b0, b1))
    return tracklets


def feature(ident, cam, length, s, rng):
    """Mean band colours over a tracklet whose appearance drifts frame to frame."""
    clean = render(ident, rng)
    if cam == "B":
        clean = clean * GAIN_B + OFFSET_B
    loggain = rng.normal(0, 2 * s, 3)
    feats = []
    for _ in range(length):
        frame = np.clip(clean * np.exp(loggain) + rng.normal(0, 0.03, clean.shape), 0, 1)
        feats.append(frame.reshape(BANDS, -1, 3).mean(axis=1).ravel())
        loggain = loggain + rng.normal(0, s, 3)
    return np.mean(feats, axis=0)


def standardize(F):
    return (F - F.mean(0)) / (F.std(0) + 1e-9)


def associate(FA, FB, delay_ok, method, use_window, gate):
    """Return, for each B tracklet, the index of the linked A tracklet or -1."""
    D = np.linalg.norm(FB[:, None, :] - FA[None, :, :], axis=2)
    tau = 0.5 * np.median(D)
    allowed = delay_ok if use_window else np.ones_like(delay_ok)
    link = -np.ones(len(FB), int)
    if method == "NN":
        Dm = np.where(allowed, D, np.inf)
        for i in range(len(FB)):
            j = int(np.argmin(Dm[i]))
            if np.isfinite(Dm[i, j]):
                link[i] = j
    else:
        rows, cols = linear_sum_assignment(np.where(allowed, D, BIG))
        for i, j in zip(rows, cols):
            if allowed[i, j]:
                link[i] = j
    if gate:
        for i in range(len(FB)):
            if link[i] >= 0 and D[i, link[i]] >= tau:
                link[i] = -1
    return link


def idf1_motmetrics(gt, hyp):
    """IDF1 as py-motmetrics computes it (id_global_assignment, idfp, idfn, idtp, idf1).
    gt, hyp: one label per detection; every detection matches its own box (distance 0)."""
    oids, hids = sorted(set(gt)), sorted(set(hyp))
    oi = {o: i for i, o in enumerate(oids)}; hi = {h: i for i, h in enumerate(hids)}
    no, nh = len(oids), len(hids)
    ocs = np.bincount([oi[o] for o in gt], minlength=no)
    hcs = np.bincount([hi[h] for h in hyp], minlength=nh)
    tps = np.zeros((no, nh))
    np.add.at(tps, ([oi[o] for o in gt], [hi[h] for h in hyp]), 1)
    fp = np.zeros((no + nh, no + nh)); fn = np.zeros((no + nh, no + nh))
    fp[no:, :nh] = np.nan; fn[:no, nh:] = np.nan
    for r in range(no):
        fn[r, :nh] = ocs[r]; fn[r, nh + r] = ocs[r]
    for c in range(nh):
        fp[:no, c] = hcs[c]; fp[c + no, c] = hcs[c]
    fp[:no, :nh] -= tps; fn[:no, :nh] -= tps
    costs = fp + fn
    # forbidden (NaN) cells get motmetrics' large constant 2 r c + 1 (lap.add_expensive_edges)
    valid = np.isfinite(costs)
    large = 2 * min(costs.shape) * (np.abs(costs[valid]).max() + 1) + 1
    rows, cols = linear_sum_assignment(np.where(valid, costs, large))
    idfn = fn[rows, cols].sum(); idfp = fp[rows, cols].sum()
    idtp = len(gt) - idfn
    return 2 * idtp / (len(gt) + len(hyp))


def idf1_overlap(gt, hyp):
    """The same score as a maximum-overlap matching (cross-check)."""
    oids, hids = sorted(set(gt)), sorted(set(hyp))
    oi = {o: i for i, o in enumerate(oids)}; hi = {h: i for i, h in enumerate(hids)}
    tps = np.zeros((len(oids), len(hids)))
    np.add.at(tps, ([oi[o] for o in gt], [hi[h] for h in hyp]), 1)
    r, c = linear_sum_assignment(-tps)
    return 2 * tps[r, c].sum() / (len(gt) + len(hyp))


def run(seed):
    rng = np.random.default_rng(seed)
    ids = make_identities(rng)
    tracklets = schedule(ids, rng)
    A = [t for t in tracklets if t[1] == "A"]; B = [t for t in tracklets if t[1] == "B"]
    delay = np.array([[b[2] - a[3] for a in A] for b in B])
    delay_ok = (delay >= WINDOW[0]) & (delay <= WINDOW[1])
    true_in_window = [delay_ok[i, [a[0] for a in A].index(b[0])]
                      for i, b in enumerate(B) if ids[b[0]]["route"] == "AB"]
    checks = dict(nA=len(A), nB=len(B), cand=delay_ok.sum(1).mean(),
                  true_in=np.mean(true_in_window))
    # ground truth: one label per detection (frame) in both cameras
    gt = [t[0] for t in A for _ in range(t[3] - t[2])] + [t[0] for t in B for _ in range(t[3] - t[2])]
    results = {}
    for s in NOISE:
        FA = standardize(np.array([feature(ids[t[0]], "A", t[3] - t[2], s, rng) for t in A]))
        FB = standardize(np.array([feature(ids[t[0]], "B", t[3] - t[2], s, rng) for t in B]))
        for method in ["NN", "Hungarian"]:
            for use_window in [False, True]:
                for gate in [False, True]:
                    link = associate(FA, FB, delay_ok, method, use_window, gate)
                    hyp_b = [("A", int(link[i])) if link[i] >= 0 else ("B", i) for i in range(len(B))]
                    hyp = [("A", j) for j, t in enumerate(A) for _ in range(t[3] - t[2])] + \
                          [hyp_b[i] for i, t in enumerate(B) for _ in range(t[3] - t[2])]
                    name = method + (" + window" if use_window else "") + (", gated" if gate else "")
                    results[(name, s)] = idf1_motmetrics(gt, hyp)
                    if seed == 0 and s == NOISE[0] and name == "NN":
                        checks["agree"] = np.isclose(results[(name, s)], idf1_overlap(gt, hyp))
        hyp0 = [("A", j) for j, t in enumerate(A) for _ in range(t[3] - t[2])] + \
               [("B", i) for i, t in enumerate(B) for _ in range(t[3] - t[2])]
        results[("no linking", s)] = idf1_motmetrics(gt, hyp0)
    return checks, results


SEEDS = [0, 1, 2]
all_res = []
for seed in SEEDS:
    checks, res = run(seed)
    all_res.append(res)
    line = (f"seed {seed}: {checks['nA']} A and {checks['nB']} B tracklets, "
            f"{checks['cand']:.1f} window candidates per B tracklet, "
            f"true partner inside window: {checks['true_in']:.2f}")
    if "agree" in checks:
        line += f", motmetrics form = overlap form: {checks['agree']}"
    print(line)

names = list(dict.fromkeys(k[0] for k in all_res[0]))
print("\nIDF1, mean over seeds 0-2 (columns: drift noise s)")
print(f"{'method':28s}" + "".join(f"{s:>8}" for s in NOISE))
for n in names:
    print(f"{n:28s}" + "".join(f"{np.mean([r[(n, s)] for r in all_res]):8.3f}" for s in NOISE))
print("\nIDF1, min-max over seeds 0-2")
for n in names:
    print(f"{n:28s}" + "".join(
        f"  {min(r[(n, s)] for r in all_res):.2f}-{max(r[(n, s)] for r in all_res):.2f}" for s in NOISE))

Output:

seed 0: 69 A and 64 B tracklets, 9.2 window candidates per B tracklet, true partner inside window: 1.00, motmetrics form = overlap form: True
seed 1: 58 A and 66 B tracklets, 7.4 window candidates per B tracklet, true partner inside window: 1.00
seed 2: 60 A and 60 B tracklets, 7.6 window candidates per B tracklet, true partner inside window: 1.00

IDF1, mean over seeds 0-2 (columns: drift noise s)
method                           0.0   0.025    0.05     0.1     0.2
NN                             0.864   0.864   0.815   0.692   0.578
NN, gated                      0.958   0.956   0.886   0.739   0.637
NN + window                    0.866   0.866   0.862   0.785   0.654
NN + window, gated             0.994   0.994   0.946   0.796   0.726
Hungarian                      0.894   0.884   0.832   0.717   0.605
Hungarian, gated               0.989   0.979   0.905   0.758   0.669
Hungarian + window             0.897   0.897   0.883   0.826   0.696
Hungarian + window, gated      0.990   0.990   0.939   0.788   0.718
no linking                     0.682   0.682   0.682   0.682   0.682

IDF1, min-max over seeds 0-2
NN                            0.84-0.92  0.84-0.92  0.78-0.87  0.66-0.74  0.56-0.60
NN, gated                     0.93-0.98  0.93-0.97  0.84-0.92  0.70-0.77  0.62-0.64
NN + window                   0.84-0.91  0.84-0.91  0.83-0.91  0.76-0.83  0.64-0.68
NN + window, gated            0.99-1.00  0.99-1.00  0.93-0.97  0.79-0.80  0.72-0.73
Hungarian                     0.85-0.92  0.84-0.92  0.79-0.88  0.67-0.77  0.57-0.63
Hungarian, gated              0.98-0.99  0.97-0.99  0.88-0.92  0.71-0.79  0.66-0.68
Hungarian + window            0.87-0.91  0.87-0.91  0.86-0.90  0.81-0.86  0.68-0.71
Hungarian + window, gated     0.98-1.00  0.98-1.00  0.92-0.95  0.78-0.81  0.71-0.72
no linking                    0.66-0.71  0.66-0.71  0.66-0.71  0.66-0.71  0.66-0.71

The check lines show that the window keeps 7–9 of the roughly 60 A tracklets as candidates for each B tracklet and never excludes the true partner, and that the motmetrics construction and the maximum-overlap form give the same score. Not linking at all already scores 0.68: each identity seen in both cameras is matched to one of its two pieces, and identities seen in one camera are scored perfectly.

Unlinked identities matter as much as noise. Without drift, the gated window rules reach 0.99, but the ungated rules stay at 0.86–0.90. By design about a quarter of the B tracklets belong to identities never seen in A, and an ungated rule links each of them to some A identity whenever a candidate exists; in our reading of the design, these links account for most of the gap, since each adds frames that carry the wrong identity, and the gate, which leaves distant tracklets unlinked, removes them. Under IDF1’s one-to-one matching, two true identities that share a predicted label cannot both be matched to it, so the frames of one of them count both as misses and as false positives.

IDF1 falls with appearance drift, and wrong links can cost more than missing ones. Every rule loses accuracy as $s$ grows. At $s = 0.2$, every rule without the window scores below not linking at all, gated or not (0.58–0.67), as does ungated nearest neighbour in the window (0.65); ungated Hungarian in the window exceeds it by 0.01 on average, and only the gated window rules stay clearly above, at 0.72–0.73. A wrong link leaves the true identity split and puts frames of one identity under another’s label, so an association step that links on weak evidence can score worse than one that does not associate.

The time window helps more as appearance weakens. At $s = 0.1$, the window raises ungated Hungarian assignment from 0.72 to 0.83 and ungated nearest neighbour from 0.69 to 0.79, while at $s = 0$ it changes little. This is the role transit-time models play in the CityFlow comparison (Tang et al., 2019): they remove candidates that appearance alone cannot rule out.

The gate and the assignment rule interact with the noise level. The gate raises the mean IDF1 of every rule at every noise level but one: at $s = 0.1$ it lowers Hungarian assignment in the window from 0.83 to 0.79; the run prints no distances, so it does not show why. The plan written before the first run predicted that “Hungarian (one-to-one) beats NN at moderate and high noise”. One-to-one assignment is ahead in three of the four pairings at every noise level, but with both the window and the gate nearest neighbour is slightly ahead at every noise level, by less than 0.01 on average and within the range over seeds. With only 7–9 candidates per tracklet, the one-to-one constraint has little left to resolve, and one wrong assignment can displace a correct one (our interpretation).

The data are far simpler than real footage: no detection errors, no fragmented tracklets, two cameras and an appearance model chosen for the example. The script was run as planned and not changed after its first run. It runs in about a second with NumPy 2.3.5 and SciPy 1.16.3, and two runs gave identical output.

Evaluation

Identity measures for multi-camera tracking. MTMC benchmarks rank by IDF1, with identification precision (IDP) and recall (IDR) beside it. Ristani and Tomasi (2018) explain the choice: the ID measures report how well a tracker identifies who is where regardless of where or why mistakes occur, whereas MOTA counts how often incorrect decisions are made rather than for how long, and its CLEAR MOT mapping under-reports multi-camera errors, so they report MOTA only for single cameras. In the 2019 AI City Challenge, IDF1 was computed on one video made by concatenating all cameras, ground truth contained only vehicles seen by at least two cameras, and identities appearing in a single camera were removed from the evaluation automatically (Naphade et al., 2019). Because CityFlow annotates only vehicles that cross cameras, false positives are not counted in its single-camera evaluation but are counted in the multi-camera one (Tang et al., 2019). The IDF1 article explains the metric and the criticism that led to HOTA, which the AI City Challenge adopted in 2024, computed on 3D ground positions (Wang et al., 2024). The example above uses the py-motmetrics implementation, whose README lists Ristani et al. (2016) among its references.

Comparing single- and multi-camera scores. Single-camera and multi-camera IDF1 on the same sequences measure different things; Ristani et al. proposed reading the gap between them as a measure of cross-camera association, as the IDF1 article describes, and the DukeMTMC numbers above show its size.

Ground-plane detection. WILDTRACK matches detections to ground truth on the ground plane within a radius, which its authors note is unforgiving of projection errors, unlike overlap in the image plane (Chavdarova et al., 2018).

Re-identification inside surveillance benchmarks. CityFlow-ReID’s evaluation server and the 2019 challenge’s Track 2 used mAP over the top 100 matches per query, which the challenge report calls rank-K mAP; Tang et al. (2019) give limited storage space as the reason, the challenge report the large size of the dataset (Naphade et al., 2019). It is not identical to the full-gallery mAP of the person re-identification benchmarks.

Activity detection. ActEV scores missed activities against false alarms measured in time rather than in counts (Corona et al., 2021); in our reading this suits video in which nothing of interest happens for long periods.

Open-set identification. For face identification in video, NIST argues that most academic work assumes every search has a mate in the gallery and reports rank-based accuracy, whereas in biometrics some subjects have usually not been encountered before, so both the false negative identification rate and the false positive identification rate must be reported (Grother, Quinn & Ngan, 2017). The example above meets a version of the same issue: tracklets with no partner in the other camera.

Challenge protocols. The AI City Challenge showed teams scores on a 50% subset of the test set during the challenge to discourage tuning to it, and revealed full-test-set scores afterwards (Naphade et al., 2019). Leaderboard numbers from different editions use different data and metrics and do not compare.

Failure Modes and Documented Limits

• Low resolution and viewpoint. NIST notes that resolution in video is compromised by optical trade-offs and then by compression, that face recognition algorithms are sensitive to head orientation relative to the camera’s optical axis, and that only one algorithm handled its two lower-resolution datasets well (Grother, Quinn & Ngan, 2017). Viewpoint variation caused most re-identification failures on CityFlow-ReID (Tang et al., 2019).
• Faces that never reach the camera. FIVE notes that some proportion of a non-cooperative population will not be recognized at all, because their faces were never visible to the system or were occluded, for example by taller people in front of them (Grother, Quinn & Ngan, 2017).
• Illumination and camera differences. Changes of lighting between views are among the reasons Ristani and Tomasi (2018) give for the difficulty of MTMC tracking; the AI City organizers list occlusion and lighting changes for vehicles (Naphade et al., 2019).
• Crowding and occlusion. WILDTRACK was built because occlusion between pedestrians defeats monocular detectors in crowds (Chavdarova et al., 2018). In Ristani and Tomasi’s failure analysis, single-camera appearance correlations were poor, and identity recall low, under significant pose change, occlusion or abrupt motion.
• Blind spots and similar appearance. In the same analysis, multi-camera fragmentation came mostly from delays in blind spots and unpredictable motion, and merge errors from people who dressed similarly and whose movement between cameras was plausible (Ristani & Tomasi, 2018). The example reproduces the second kind: wrong links that look plausible to appearance and timing.
• Detection volume. On the same 20 hours of transport-terminus video, face detectors from different developers reported between 8 and 150 faces per minute, with a consensus of 10–15 from the more accurate ones; the true number was not known (Grother, Quinn & Ngan, 2017).
• Domain shift. CityFlow-ReID scored far below older vehicle benchmarks with the same methods, and the 2019 organizers note that teams adding training data from other benchmarks could not neglect the domain gap between datasets (Tang et al., 2019; Naphade et al., 2019); Dataset Bias covers the general problem. Synthetic benchmarks such as the 2024 people-tracking track add a gap between rendered and real video, which the report does not measure.
• Annotation limits. VIRAT’s release warns of crowd-sourced annotations that may not follow the guidelines and of fragmented tracks (VIRAT Release 2.0 introduction). PETA’s identity-blind split let one person appear in both training and test (Liu et al., 2017), and PETS 2009’s calibration was inaccurate enough to make geometric constraints difficult to exploit jointly (Chavdarova et al., 2018).
• Actors and scripts. PETS 2009 was recorded in an actor set-up with few distinct individuals, which WILDTRACK’s authors argue does not allow good generalization or fair benchmarking of appearance-based methods (Chavdarova et al., 2018); MEVA’s activities were scripted, with periods of unscripted collection, and performed by professional actors hired to act scenarios in a natural manner (Corona et al., 2021).
• Offline results. Many challenge systems were offline or far from real-time, and the best 2024 offline entry was about five HOTA points ahead of the best online one (Naphade et al., 2019; Wang et al., 2024).

Documented Accuracy and Bias Findings

Face recognition in video (FIVE). NIST’s Face in Video Evaluation tested 36 algorithms submitted by 16 commercial developers in December 2015 on video searched against galleries of up to 48,000 enrolled people (Grother, Quinn & Ngan, 2017). Its summary describes six video datasets in one place and seven in others. The findings it states include:

• Accuracy varied enormously between algorithms: even on the easiest dataset with a gallery of 480 people, the proportion of searches whose correct identity was not ranked first ranged from below 1% to above 40%, and with professional ceiling-mounted cameras miss rates ranged from 20% to over 90%.
• On about 20 hours of transport-terminus video (roughly 2 million frames with thousands of passers-by), with a threshold set so that only 10 false positives occurred, the most accurate algorithm missed about 20% of the enrolled actors with a gallery of 480 people, 21% with 4,800 and 35% with 48,000; without a threshold, counting a hit at rank one, the miss rates were 14%, 16% and 26%.
• Subjects in video could be identified with error rates approaching those of cooperative portraits only when image quality reached that of engineered systems such as e-Passport gates, which the report calls a very difficult goal.
• Poor image quality and face presentation can make the scores of unrelated people similarly high, which the report says leads to the possibility that a member of the public is falsely matched to someone on a watchlist.

The report lists full-body and gait recognition, behaviour and intent detection, anomaly detection and infrared imagery as outside its scope.

Demographic differentials. NIST’s FRVT Part 3 report found that false positive rates of face recognition algorithms often varied across demographic groups by factors of 10 to beyond 100, in many but not all algorithms tested, while false negatives were more algorithm-specific and often varied by factors below 3; Face Recognition summarizes its datasets, operating points and findings. Two points from it bear on surveillance. The report notes that in identification applications such as surveillance a false positive match could lead to a false accusation, detention or deportation. And it states that it used no images from the Internet or from video surveillance, so it does not capture differentials that may occur in such images (Grother, Ngan & Hanaoka, 2019). None of the sources read for this article (November 2026) measures demographic differentials on surveillance video; the EDPB guidelines cite researchers’ reports that facial identification software performs differently by age, gender and ethnicity (European Data Protection Board, 2020).

Body-based methods. The sources read for this article report no demographic analysis of person re-identification, attribute recognition or multi-camera tracking. PA-100K’s attributes include gender and age (Liu et al., 2017), and MEVA’s actors were scripted to vary in age, sex and ethnicity as the actor pool allowed (Corona et al., 2021), but neither paper reports error rates by group.

Privacy, Legal and Civil-Liberty Concerns

This section reports what official documents and the cited sources state, attributed to them, as of November 2026; it draws no legal conclusions and does not describe the law of any jurisdiction.

European data protection authorities. The European Data Protection Board’s guidelines on video devices open by stating that intensive use of video devices has an impact on citizens’ behaviour, that significant implementation puts additional pressure on individuals to prevent the detection of what might be perceived as anomalies, that these technologies may limit anonymous movement and the possibility of remaining unnoticed, and that data protection implications are massive (European Data Protection Board, 2020). They warn of secondary use of footage for unexpected purposes such as marketing or employee monitoring, note that intelligent video analysis ranges from simple counting algorithms to more intrusive biometric technologies, and state that video surveillance is not by default a necessity when other means can achieve the purpose. They state that the use of biometric data, and in particular facial recognition, entails heightened risks for data subjects’ rights. In their joint opinion on the European Commission’s 2021 proposal for an Artificial Intelligence Act, the EDPB and the European Data Protection Supervisor wrote that remote biometric identification in publicly accessible spaces poses a high risk of intrusion into private lives, might involve processing the data of an indiscriminate and disproportionate number of people to identify only a few, and has an irreversible, severe effect on the expectation of being anonymous in public spaces, with a direct negative effect on freedom of expression, assembly, association and movement; they considered that even identification after the fact, if used in the context of a political protest, is likely to have a significant chilling effect on freedom of assembly and association, and called for a general ban on any use of AI for automated recognition of human features in publicly accessible spaces, in any context, naming faces, gait, fingerprints, DNA, voice and keystrokes (EDPB & EDPS, 2021). The opinion concerns the proposal of 2021; this article does not summarize the regulation as adopted. Biometric Recognition Systems quotes the UK GDPR’s definition of biometric data.

Dataset practice. The research datasets show how their creators handled the people recorded. CityFlow and the AI City Challenge datasets redacted faces and licence plates (Tang et al., 2019; Naphade et al., 2019). MEVA was collected under the oversight of an institutional review board with actors who signed consent forms, at an access-controlled facility chosen to keep non-participants out of view, and released under CC-BY-4.0 (Corona et al., 2021); VIRAT’s ground-camera data are distributed under a data protection agreement (VIRAT site). DukeMTMC, recorded on a university campus, was retracted (Peng et al., 2021), and Peng et al. document that its derivatives remained in use. The 2024 people-tracking track used only synthetic people (Wang et al., 2024); the report does not give privacy as the reason.