Video Anomaly Detection

Video anomaly detection scores each frame of a video, and sometimes each region of a frame, for how far it departs from what normal video of the setting shows. The events to be found are rare and of open-ended kind, so in the classical formulation a model is fitted to training video that contains only normal activity and anything it describes poorly is flagged at test time. Liu et al. (2018, Section 1) give the common definition, the identification of events that do not conform to expected behavior, and Hasan et al. (2016) frame the same idea as learning the temporal regularity of a scene. The literature also calls the task abnormal event detection, irregularity detection and, in the weakly supervised setting built on the UCF-Crime dataset, real-world anomaly detection. Most benchmarks are fixed-camera pedestrian scenes or surveillance footage.

Visual Anomaly Detection covers the same problem for still images, its industrial benchmarks and its image and pixel metrics. Video adds motion: an object can look normal and still be abnormal because of where it goes or how fast. Video also brings protocols of its own, chiefly how frame-level scores are pooled over test videos.

Problem Definition

One-class formulation. Training data are videos $V_1, \ldots, V_N$ of normal activity; test videos contain normal and abnormal events. A model assigns each test frame $t$ of video $v$ a score $s_v(t)$, larger meaning more anomalous (some papers score regularity, larger meaning more normal, and invert it). Frame-level ground truth says whether an abnormal event is visible in each frame. This is the protocol of UCSD Ped1/Ped2, CUHK Avenue and ShanghaiTech Campus, whose training splits contain only normal events (Georgescu et al., 2021, Section 4.1). Papers call it unsupervised (Gong et al., 2019) or one-class classification and outlier detection (Acsintoae et al., 2022, Section 1).

Single scene or many. Ramachandra and Jones (2020, Section 1) define the problem for a single scene: given normal training video from one camera, detect events in test video from the same camera that are significantly different from anything in training. They point out that some papers instead build one model from normal video of many scenes (they name Sultani et al., 2018, and Liu et al., 2018), and argue that this rules out anomalies that depend on location, such as a person walking in an area that is only restricted in one scene. Their view is that the single-scene formulation corresponds to the most common surveillance case. ShanghaiTech Campus has 13 scenes and is used with one model for all of them (Luo, Liu & Gao, 2017; Liu et al., 2018, Section 4.1).

Weakly supervised formulation. Sultani, Chen and Shah (2018) train on both normal and anomalous videos, but labelled only at video level, “anomalous somewhere” or “normal”, and test with frame-level labels. The anomalies are named categories such as fighting, road accidents and burglary. Acsintoae et al. (2022, Section 1) call this a closed-set scenario, since training and test anomalies belong to the same action categories. It assimilates the task to action recognition and does not test detection of unseen anomaly types, which the one-class formulation keeps open.

Supervised open-set formulation. Acsintoae et al. (2022) propose a third setting: abnormal events are annotated at training time, but the anomaly types at test time are disjoint from those seen in training. Their UBnormal benchmark is built from virtual scenes to make this possible.

Input and Output

The input is a video from a fixed camera in the classical benchmarks, or an untrimmed clip of any origin in UCF-Crime and XD-Violence; XD-Violence adds the audio track (Wu et al., 2020). Methods consume short space–time windows rather than single frames: cuboids of $T$ frames in Hasan et al. (2016), four past frames in Liu et al. (2018, Section 3.5), object-centred tubes of seven frames in Georgescu et al. (2021, Section 4.2), or 32 segments per video summarized by pretrained 3D CNN features in the weakly supervised methods (Sultani et al., 2018, Section 5.1; Tian et al., 2021, Section 4.2).

The outputs are:

• a score per frame, from which the frame-level ROC curve is computed;
• optionally, an anomaly map or boxes per frame, for spatial localization (UCSD’s pixel-level criterion, Avenue’s rectangles, Street Scene’s region and track criteria);
• after a threshold, the frames or regions flagged as abnormal.

Scores are often post-processed per video. Hasan et al. (2016, Section 3.4) and Liu et al. (2018, Section 3.6) rescale each test video’s scores with that video’s minimum and maximum, a step with consequences for evaluation (see Evaluation Metrics).

Variants

• Temporal detection only, or detection with localization. Frame-level evaluation asks only when; UCSD’s pixel-level criterion, Avenue’s rectangle criterion and Street Scene’s region-based criterion also ask where (Li, Mahadevan & Vasconcelos, 2014, Section 6.2; Avenue dataset page; Ramachandra & Jones, 2020, Section 4).
• Violence detection. XD-Violence restricts the abnormal class to six kinds of physical violence and adds audio, on the argument that sound separates visually ambiguous events such as explosions (Wu et al., 2020, Sections 1 and 3.1).
• Online detection. Wu et al. (2020, Section 4.3) add an approximator branch that scores snippets without seeing the whole video. It scored 73.67% AP against 78.64% for the offline network on XD-Violence (Table 3).
• Cross-scene generalization. In the weakly supervised setting, Zhong et al. (2019, Section 5.3) observed that UCF-Crime test anomalies can occur in scenes unlike the training scenes of the same class, which they read as a sign that detecting anomalies in generic scenes remains hard.

Challenges

• Abnormality depends on context. A truck on a street is normal and the same truck in a pedestrian area is not (Acsintoae et al., 2022, Section 1). The UCSD project page motivates appearance-based detection with a truck crossing a bridge with a weight limit, which motion alone would not reveal.
• No examples of the target. Abnormal events are open-ended, so gathering all kinds is infeasible (Liu et al., 2018, Section 1). Acsintoae et al. (2022, Section 1) add that some anomaly types cannot ethically be staged.
• Models generalize too well. A high-capacity autoencoder trained on normal data may reconstruct abnormal input well too, so a larger error on anomalies is not guaranteed. Liu et al. (2018, Section 1) argue this against reconstruction methods, and Gong et al. (2019) build their memory module to counter it.
• Rare positives in long videos. In UCF-Crime most frames of an anomalous video are normal. Sultani et al. (2018, Section 5.1) do not use the equal error rate for this reason: it misjudges detection when only a small part of a long video is abnormal.
• Crowds, occlusion and perspective. On UCSD Ped1, optical-flow descriptors produced spurious detections near the camera, where perspective makes motion large (Mahadevan et al., 2010, Section 8). Li et al. (2014, Section 6.4) found that optical flow led to too many false positives in regions of complex motion and occlusion.
• Nuisance changes. UCF-Crime failure cases include a dark night scene, insects in front of the lens and a crowd gathering to watch a race (Sultani et al., 2018, Section 5.2). Avenue’s test video 2 has camera shake, a few outliers are present in the training data, and some normal patterns rarely appear in training (Avenue dataset page).

Approaches

Hand-crafted normality models

Early methods fit a statistical model to local motion features from normal video and flag low-likelihood observations. Liu et al. (2018, Section 2.1) survey them: trajectories of tracked objects, which fail in crowded scenes; local histograms of optical flow modelled by an exponential distribution; mixtures of probabilistic PCA on local flow (MPPCA); and Markov random fields. Mahadevan et al. (2010) argued that methods based on optical flow ignore appearance. They modelled local space–time patches by mixtures of dynamic textures (MDT), a joint model of appearance and dynamics. Low-probability patches gave temporal anomalies, and discriminant saliency against the surrounding region gave spatial anomalies. On UCSD Ped1 and Ped2, MDT had a frame-level equal error rate of 25% on both, against 30–42% for their implementations of social force, MPPCA and flow monitoring (Table 1). The journal version adds multiple scales and a conditional random field (Li et al., 2014). Mahadevan et al. reported training times of about 2 hours and about 25 seconds per test frame (Section 8).

Sparse coding learns a dictionary of normal patches and flags patches with a large reconstruction error. Lu, Shi and Jia (2013) replaced the sparse optimization with a small set of learned combinations of dictionary atoms, each tested by a cheap least-squares fit. They reported 140–150 frames per second in MATLAB on a desktop CPU (abstract), against seconds per frame for the methods they compared on UCSD Ped1 (Table 5). Luo, Liu and Gao (2017) made sparse coding temporally coherent across neighbouring frames and mapped its optimization onto a stacked recurrent network (sRNN).

Reconstruction

Hasan et al. (2016) trained a fully convolutional autoencoder (Conv-AE) on cuboids of $T$ consecutive grey frames from normal video. The reconstruction error of frame $t$ is the sum of its pixel errors, $e(t) = \sum_{x,y} e(x, y, t)$, and the regularity score is

$$s(t) = 1 - \frac{e(t) - \min_t e(t)}{\max_t e(t)},$$

with the minimum and maximum taken over the test video, as printed in their Section 3.4. They trained one model on several datasets together (about 1 hour 50 minutes of video) and found that adding optical flow channels gave insignificant improvement. Their frame-level AUC was 70.2% on Avenue, 81.0% on Ped1 and 90.0% on Ped2 (Table 1). They note that their model flags any departure from regularity, including events the ground truth calls normal: running at the Subway Exit was detected as irregular but is not annotated as abnormal (Section 4.5).

Gong et al. (2019) put a memory between encoder and decoder. Each encoding is replaced by a sparse combination of a fixed set of memory items learned from normal data, so the decoder can only produce normal-looking output and errors on anomalies grow. Their 3D-convolutional MemAE reported frame-level AUC of 0.941 on Ped2, 0.833 on Avenue and 0.712 on ShanghaiTech, against 0.917, 0.810 and 0.697 for the same autoencoder without memory (Table 2). They normalize each test video’s errors to $[0, 1]$, as Hasan et al. and Luo et al. do.

Future frame prediction

Liu, Luo, Lian and Gao (2018) argued that comparing a frame with its expectation fits the definition of anomaly better than reconstructing it. Their generator, a U-Net without cropping, predicts frame $\hat I_{t+1}$ from the four previous frames. Training combines four terms: an intensity loss $\|\hat I - I\|_2^2$; a gradient loss on differences between neighbouring pixels; a motion loss $\|f(\hat I_{t+1}, I_t) - f(I_{t+1}, I_t)\|_1$, where $f$ is a pretrained, frozen FlowNet (optical flow estimation); and a least-squares adversarial loss with a patch discriminator (Sections 3.1–3.5). At test time, each frame’s PSNR between prediction and truth is computed. The PSNR values of each test video are rescaled to $[0, 1]$ with that video’s minimum and maximum to give a regularity score (Section 3.6).

They reported frame-level AUC of 85.1% on Avenue, 83.1% on Ped1, 95.4% on Ped2 and 72.8% on ShanghaiTech (Table 1). Removing the motion loss lowered Ped1 to 81.8% and Ped2 to 93.5% (Table 3), and their prediction network separated normal and abnormal frames by a larger mean score gap than Conv-AE on Ped1, Ped2 and Avenue (Fig. 7). They trained one model on all 13 ShanghaiTech scenes, following Luo et al. (Section 4.1).

Object-centric and self-supervised methods

Ionescu, Khan, Georgescu and Shao (2019) first run a pretrained single-shot object detector, then learn appearance and motion features per object with convolutional autoencoders. They cluster the normal training objects with k-means and train one-versus-rest classifiers in which the other clusters act as “dummy anomalies”. An object is abnormal when every classifier rejects it. Georgescu, Barbalau, Ionescu, Khan, Popescu and Shah (2021) keep the object-centric pipeline (YOLOv3 detections; see YOLO) and train one 3D CNN on four proxy tasks, learned from normal video only:

• arrow of time, whether an object tube plays forwards or backwards;
• motion irregularity, whether its frames are consecutive or intermittent;
• middle bounding box prediction, reconstructing the appearance of the middle frame;
• knowledge distillation from pretrained classification and detection networks.

Failure on these tasks signals anomaly. In their ablation, arrow of time alone was the best single task on Avenue (83.6% frame-level AUC), which they attribute to Avenue’s anomalies being unusual actions. Box prediction and distillation suited Ped2, whose anomalies are unusual objects (Section 4.4, Table 4). With late fusion of object-level and frame-level scores they reported 92.8% on Avenue, 90.2% on ShanghaiTech and 99.8% on Ped2 (Table 2); these are macro-averaged values (see Evaluation Metrics). The approach rests on self-supervised learning and an object detector: an anomaly the detector misses cannot be scored at object level.

Weakly supervised multiple-instance learning

Sultani, Chen and Shah (2018) treat each training video as a bag of 32 temporal segments. An anomalous video is a positive bag, of which at least one segment is abnormal, and a normal video is a negative bag. Segment features are C3D activations averaged over 16-frame clips, and a three-layer network scores each segment. The training signal is a ranking hinge on the highest-scoring segment of each bag,

$$\ell(\mathcal{B}_a, \mathcal{B}_n) = \max\!\Big(0,\; 1 - \max_{i \in \mathcal{B}_a} f(V_a^i) + \max_{i \in \mathcal{B}_n} f(V_n^i)\Big) + \lambda_1 \sum_i \big(f(V_a^i) - f(V_a^{i+1})\big)^2 + \lambda_2 \sum_i f(V_a^i),$$

with a temporal smoothness and a sparsity term on the anomalous bag ($\lambda_1 = \lambda_2 = 8 \times 10^{-5}$; Sections 3.2 and 5.1). On their UCF-Crime dataset the method reached 75.41% frame-level AUC, against 74.44% without the two constraints, 65.51% for the sparse combinations of Lu et al., 50.6% for Hasan et al.’s autoencoder, and 50.0% for a binary video classifier (Table 3). On normal test videos at a threshold of 0.5, its false alarm rate was 1.9%, against 27.2% and 3.1% for the two unsupervised methods (Table 4).

Zhong et al. (2019) recast the weak labels as noisy segment labels. A graph convolutional network cleans them using feature similarity and temporal consistency, so that an ordinary action classifier can be trained on the cleaned labels; they reported 82.12% AUC on UCF-Crime (abstract). Tian et al. (2021) observed that the top-scoring segment of an anomalous video is often a normal one. Their RTFM (robust temporal feature magnitude learning) instead ranks the mean feature norm of the $k = 3$ largest-norm snippets of each video, under the assumption that abnormal snippets have larger expected feature magnitude. It adds dilated temporal convolutions and self-attention over snippets (Sections 3 and 4.2). With I3D RGB features they reported 84.30% AUC on UCF-Crime and 97.21% on the weakly supervised ShanghaiTech split. On XD-Violence, Table 3 gives 77.81% for I3D features and 75.89% for C3D features as AP; Section 4.5 calls the C3D value an AUC (Tables 1–3). These are RGB-only results: the same table lists Wu et al. at 75.41%, the value of their RGB-only input (Wu et al., 2020, Section 5.3), not the 78.64% of their audio-visual network.

Datasets and Benchmarks

Dataset	Source	Training	Test	Abnormal events	Labels for evaluation
UCSD Ped1	one walkway, camera elevated	34 clips, normal	36 clips	40	per-frame flags; pixel masks on a subset
UCSD Ped2	a second walkway	16 clips, normal	12 clips	12	per-frame flags; pixel masks
CUHK Avenue	one campus avenue	16 videos, normal	21 videos	47	rectangles per frame
ShanghaiTech Campus	13 campus scenes	274,515 frames, normal	42,883 frames	130	per-frame and pixel-level
UCF-Crime	1,900 web surveillance videos	800 normal, 810 anomalous (video labels)	150 normal, 140 anomalous	13 classes	start and end frames
XD-Violence	4,754 videos with audio	3,954 (video labels)	500 violent, 300 non-violent	6 classes	start and end frames

Sources: UCSD dataset page and Li et al. (2014, Section 6.1) for UCSD; Luo et al. (2017, Section 4) and Liu et al. (2018, Section 4.1) for event counts on UCSD and Avenue; the Avenue dataset page; Luo et al. (2017, Table 1) for ShanghaiTech; Sultani et al. (2018, Section 4.2); Wu et al. (2020, Section 3.2).

UCSD Ped1 and Ped2

Mahadevan et al. (2010, Section 6) recorded two pedestrian walkways with a stationary, elevated camera, crowd density varying from sparse to very crowded. Normal video contains only pedestrians. Anomalies are non-pedestrian entities on the walkway (bikers, skaters, small carts, a few wheelchairs) or anomalous pedestrian motion such as walking across a walkway or on the grass. All occurred naturally and none was staged. Ped1 shows people walking towards and away from the camera with some perspective distortion, at 158 × 238 pixels; Ped2 shows movement parallel to the image plane at 240 × 360 pixels. Clips are 120–200 frames long, and the test sets hold around 5,500 normal and 3,400 abnormal frames (Li et al., 2014, Section 6.1).

The ground truth changed after the conference paper: the journal version’s “full annotation”, including Ped1 ground truth contributed by Antić and Ommer, supersedes the earlier “partial annotation” (Li et al., 2014, Section 6.1). The journal paper describes pixel masks for every test clip; the dataset page lists them for 10 Ped1 clips and all 12 Ped2 clips. The conference paper lists 14 test clips for Ped2 (Section 6.1); the journal paper and the dataset page list 12. Event counts depend on how events are delimited: 40 and 12 abnormal events in Luo et al. (2017) and Liu et al. (2018), 54 and 23 in Ramachandra and Jones (2020, Table 1). Ramachandra and Jones (2020, Section 2) also note that every UCSD anomaly can be detected from a single frame.

CUHK Avenue

The dataset page describes 16 training and 21 test clips filmed on the CUHK campus avenue, 30,652 frames in all (15,328 for training, 15,324 for testing). Its three abnormal examples are a strange action, walking in the wrong direction and an abnormal object. The ICCV paper that introduced it describes an earlier version: 15 sequences, 35,240 frames, 14 unusual events, 4 training videos (Lu et al., 2013, Section 3.3). Hasan et al. (2016, Section 4.5) note that the results reported in that paper are on the old version, which they call smaller than the current one, although its stated frame count exceeds the dataset page’s. The page’s localization criterion is PASCAL-style: a detection counts when its IoU with an annotated rectangle exceeds a threshold, reported at thresholds 0.2 to 0.8. Sultani et al. (2018, Section 4.1) point out that Avenue’s anomalies are staged, filmed at one location, and sometimes unrealistic, such as throwing paper.

ShanghaiTech Campus

Luo, Liu and Gao (2017, Section 4 and Table 1) built ShanghaiTech Campus for scene diversity: 13 scenes with complex lighting and camera angles, 317,398 frames (274,515 training, 42,883 test), 130 abnormal events, and pixel-level ground truth. Ramachandra and Jones (2020, Section 2) attribute the dataset to Liu et al. (2018) instead. Its anomalies include sudden motion such as chasing and brawling, which earlier datasets lacked. Georgescu et al. (2021, Section 4.1) count 330 training and 107 test videos, with robbing, jumping, fighting and riding bikes in pedestrian areas among the anomalies. The name is shared with an unrelated crowd counting dataset used in object counting.

Zhong et al. (2019, Section 5.1) reorganized the dataset for weak supervision: randomly selected anomalous test videos moved into training and training videos into test, so that both splits cover all 13 scenes. They give the split’s details in an appendix, which was not read for this article. Tian et al. (2021, Sections 4.1 and 4.7) use the same split, with 63 anomalous training videos. Results on this split and on the original one-class split measure different tasks and are not comparable. RTFM’s 97.21% (weak split) and the 90.2% of Georgescu et al. (one-class split, macro-averaged) are both “frame-level AUC on ShanghaiTech”.

UCF-Crime

Sultani et al. (2018, Section 4.2) collected 1,900 untrimmed real-world surveillance videos, 128 hours in total, from YouTube and LiveLeak through text queries in several languages. They removed edited, prank, hand-held, news and compilation videos and videos in which the anomaly is unclear. The 13 anomaly classes are abuse, arrest, arson, assault, road accident, burglary, explosion, fighting, robbery, shooting, stealing, shoplifting and vandalism, chosen for their impact on public safety. Training videos carry video-level labels. For the test videos, several annotators marked the start and end of each anomaly, and their annotations were averaged. In the test set, normal videos contain no anomaly at all (Fig. 5).

XD-Violence

Wu et al. (2020, Section 3) collected 4,754 untrimmed videos (2,405 violent, 2,349 non-violent), 217 hours in total, from 91 movies and from YouTube. Sources range from sports, games and news to CCTV, hand-held and car recorders. Six physically violent classes are labelled: abuse, car accident, explosion, fighting, riot and shooting. They collected non-violent videos with backgrounds matching the violent ones, so that a model could not separate the classes by scene. The test set has 500 violent and 300 non-violent videos, with start and end frames averaged over annotators. The dataset is scored by frame-level AP rather than AUC (see Evaluation Metrics).

Other benchmarks

Subway Entrance and Exit are two long videos with wrong-direction and fare-dodging anomalies. Luo et al. (2017, Section 5.3) did not use them because different works annotated different ground truth, and Ramachandra and Jones (2020, Section 2) list the frame rate and train/test frames as ambiguous. UMN’s anomalies are staged: in each clip everyone suddenly runs away (Ramachandra & Jones, 2020, Section 2), and several methods achieve near-perfect detection on it (Li et al., 2014, Section 6.8). Street Scene has 46 training and 35 test sequences of a two-way street at 1280 × 720 pixels, with 205 anomalous events of 17 types, such as jaywalking, bikers outside the bike lane and car u-turns, and bounding-box tracks as ground truth (Ramachandra & Jones, 2020, Section 3). UBnormal has 29 virtual scenes rendered in Cinema4D on real backgrounds, 236,902 frames, pixel-level labels of abnormal events in training, and anomaly types in test that are disjoint from those in training (Acsintoae et al., 2022).

Evaluation Metrics

Frame-level and pixel-level criteria

The criteria used on most benchmarks come from the UCSD papers (Mahadevan et al., 2010, Section 6.1; Li et al., 2014, Section 6.2).

• Frame-level criterion. A frame is predicted abnormal if any pixel of its thresholded anomaly map is abnormal, and is compared with the frame’s ground-truth flag. Sweeping the threshold traces an ROC curve of frames.
• Pixel-level criterion. An abnormal frame is a true positive only if at least 40% of its ground-truth anomalous pixels are detected, and a normal frame with any detected pixel is a false positive.

Mahadevan et al. reported the equal error rate (EER) of the frame-level curve. Later papers report the area under it, the frame-level AUC, as the main metric (Georgescu et al., 2021, Section 4.2).

The authors of the criteria warned about the frame-level one. It allows “lucky co-occurrences” of a wrong detection and a true anomaly: a detector that marks one random pixel in every abnormal frame scores perfectly. They argued that the pixel-level criterion should be the main one (Li et al., 2014, Section 6.2). Ramachandra and Jones (2020, Section 2.1) showed that the pixel-level criterion has a loophole of its own. If every pixel of any frame with a detection is marked abnormal, the pixel-level result equals the frame-level one: all ground-truth pixels are covered, and a normal frame already counts as one false positive. The criterion neither rewards tight localization nor counts multiple false regions.

Region- and track-based criteria

Ramachandra and Jones (2020, Section 4) count regions instead of frames.

• Region-based detection criterion (RBDC). A ground-truth region counts as detected when a detected region overlaps it with IoU of at least $\beta$. The detection rate is taken over all ground-truth regions in all test frames.
• Track-based detection criterion (TBDC). A ground-truth event track counts as detected when at least a fraction $\alpha$ of its regions are detected.
• False positives. Every detected region whose IoU with every ground-truth region is below $\beta$ is a false positive, so a frame can hold several. Both criteria are plotted against false positives per frame and summarized by the area for false-positive rates from 0 to 1; they use $\alpha = \beta = 0.1$.

Georgescu et al. (2021) report RBDC and TBDC on UCSD Ped2 (72.8% and 91.2%, Table 3). Tian et al. (2021, Section 4.1) note that the criteria cannot be applied in the weakly supervised setting, where training gives no spatial labels.

Micro and macro frame-level AUC

A frame-level AUC over several test videos can be computed in two ways. The README of Georgescu et al.’s official repository (as of November 2026) names them:

• Micro-averaged AUC concatenates the frames of all test videos and computes one ROC curve;
• Macro-averaged AUC computes the AUC of each test video and averages them,

$$\mathrm{AUC}_{\mathrm{micro}} = \mathrm{AUC}\big(\textstyle\bigcup_v \{(s_v(t), y_v(t))\}\big), \qquad \mathrm{AUC}_{\mathrm{macro}} = \frac{1}{|\mathcal{V}'|} \sum_{v \in \mathcal{V}'} \mathrm{AUC}\big(\{(s_v(t), y_v(t))\}_t\big),$$

where $\mathcal{V}'$ is the set of test videos that contain both normal and abnormal frames. The README states that many earlier works do not say which they computed, and that its paper reported the macro version. It then gives both versions at object level: micro 91.5%, 82.4% and 97.5% against macro 91.9%, 89.3% and 99.8% on Avenue, ShanghaiTech and Ped2. The ShanghaiTech gap of 6.9 points is larger than the 5.3-point margin by which the paper’s best result led the previous best method there (Section 4.3).

The two versions measure different things (our reading of the pairwise definition in ROC AUC, whose pooling effect this is). Micro AUC also compares abnormal frames of one video with normal frames of every other video, so an offset in a video’s overall score level counts, whether it comes from lighting, crowd density or a different scene. Macro AUC compares frames only within a video, which removes such offsets, gives each video equal weight whatever its length, and is undefined for a video with no anomaly. Macro AUC therefore cannot use the 150 normal test videos of UCF-Crime or the 300 non-violent ones of XD-Violence. On a one-class benchmark the two can differ in either direction. In Acsintoae et al.’s Table 3, Liu et al.’s method has micro 85.1% and macro 81.7% on Avenue, but micro 72.8% and macro 80.6% on ShanghaiTech; the macro values are taken from a journal paper by Georgescu et al. on a background-agnostic framework (TPAMI, 2021), which was not read for this article and whose values are cited here through Acsintoae et al. The same table ranks two methods differently by the two versions on Avenue: that background-agnostic framework leads by micro AUC (92.3% against 91.5%), and the multi-task method of Georgescu et al. (2021) by macro AUC (91.9% against 90.4%).

Per-video normalization. A third variant is common in code. Liu et al.’s official evaluate.py (commit 9e0f161) rescales the PSNR values of each test video to $[0, 1]$ with that video’s own minimum and maximum, drops the first four frames of each video, and then concatenates all videos into one ROC curve: a micro AUC of per-video-normalized scores. Hasan et al. and Gong et al. also normalize per video, with slightly different formulas. Two consequences follow from the definition (our reasoning). First, the normalization removes much of the offset between videos that hurts raw micro AUC. Second, a frame’s score depends on the minimum and maximum of the whole test video, including frames after it, so the score could not be computed by a detector running online. When given a directory of result files, the same script reports the highest AUC among them; Liu et al.’s paper does not say how the reported model was chosen.

Average precision

Wu et al. (2020, Section 5.1) score XD-Violence by frame-level average precision, the area under the precision–recall curve. They argue that AUC is optimistic under class imbalance and that AP focuses on the positive class; Precision and Recall discusses why.

Comparing Reported Numbers

Reported numbers on the same benchmark are often not comparable:

• the AUC is micro, macro or per-video-normalized micro, often unstated;
• ShanghaiTech is used in its one-class split or in Zhong et al.’s weakly supervised split;
• Avenue appears in its old or current version, and UCSD with partial or full annotation;
• the backbone features differ: C3D, I3D RGB, I3D with flow, or TSN.

The same method also carries different numbers in different papers. Hasan et al.’s Conv-AE reported 70.2% frame-level AUC on Avenue and 90.0% on Ped2 (Table 1). Luo et al. (2017, Table 2) list it at 74.5% and 81.1%, and Liu et al. (2018, Table 1) at 80.0% and 85.0%. Neither later table says where its values come from.

The trend on each benchmark is still clear, though pooling and features differ between the numbers below. Frame-level AUC on Ped2 rose from 82.9% for MDT (as listed by Liu et al., 2018) to 95.4% for future frame prediction (per-video-normalized micro in its official code) and 99.8% (macro) for Georgescu et al. (2021). Georgescu et al. count 23 works above 90% on Ped2 (Section 4.3), and Acsintoae et al. (2022, Section 2.2) call UCSD Ped and UMN saturated. Georgescu et al.’s background-agnostic framework, reported at 98.7% micro AUC on Ped2, reached 59.3% on UBnormal’s test set (Acsintoae et al., 2022, Section 2.2 and Table 2). With weak supervision, AUC on UCF-Crime rose from 75.41% (Sultani et al., C3D) to 82.12% (Zhong et al.) and 84.30% (RTFM, I3D RGB). Benchmark overfitting discusses why gains of a fraction of a point on saturated benchmarks carry little information.

Example: Prediction, Reconstruction and the Pooling of AUC

The example builds a synthetic surveillance-style scene, not real footage. Grey 32 × 32 frames show a fixed textured background, and Gaussian blobs (“pedestrians”) walk left to right along three lanes at about one pixel per frame. Each video has its own crowd density, illumination gain and sensor noise. Twenty normal training videos come from one range of these settings. Twelve test videos come from a wider range, as the videos of a multi-session or multi-scene benchmark differ. Ten test videos contain two or three abnormal tracks: a blob walking in the wrong direction, a blob moving three times too fast, or an unusual object, a square of the same total brightness as a blob, moving normally. Two test videos contain no anomaly. As on UCSD and Avenue, a frame is abnormal if an abnormal object is visible in it.

Three scorers are compared:

• the energy of the frame difference, with no learning;
• a linear version of Hasan et al.’s reconstruction: PCA with 24 components fitted to normal cuboids of three frames;
• a linear version of Liu et al.’s prediction: ridge regression from frames $t-2$ and $t-1$ to frame $t$.

Each is scored by micro AUC, by micro AUC after per-video min–max normalization (as in Liu et al.’s evaluate.py), and by macro AUC over the test videos that contain both classes. Because the sources leave the pooling open, the code prints all three. It runs in under a minute (10 to 40 seconds in our runs) with NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2, most of it in the PCA fits.

import sys
import numpy as np
import scipy.linalg
from sklearn.metrics import roc_auc_score

H = W = 32
LANES = (8, 16, 24)
yy, xx = np.mgrid[0:H, 0:W].astype(float)
# fixed scene seen by the camera: a gradient with a little texture
BACKGROUND = 0.2 + 0.2 * xx / W + 0.05 * np.sin(yy / 3.0) * np.cos(xx / 5.0)
DISC_PEAK, SIGMA = 0.5, 1.5
SQUARE_SIDE = 5
SQUARE_LEVEL = DISC_PEAK * 2 * np.pi * SIGMA**2 / SQUARE_SIDE**2  # same total brightness


def render(obj):
    if obj["kind"] == "square":
        half = SQUARE_SIDE / 2
        inside = (np.abs(xx - obj["x"]) < half) & (np.abs(yy - obj["y"]) < half)
        return SQUARE_LEVEL * inside
    return DISC_PEAK * np.exp(-((xx - obj["x"]) ** 2 + (yy - obj["y"]) ** 2) / (2 * SIGMA**2))


def make_video(rng, T, density, gain, noise, events=()):
    """events: list of (start frame, type) with type in 'wrong', 'fast', 'square'."""
    objects, frames, labels, kinds = [], [], [], []
    pre = 40
    starts = {s + pre: k for s, k in events}
    for t in range(T + pre):
        for lane in LANES:
            if rng.random() < density:
                objects.append(dict(x=-3.0, y=lane, v=1 + 0.05 * rng.standard_normal(),
                                    kind="disc", abnormal=None))
        if t in starts:
            k = starts[t]
            lane = LANES[rng.integers(3)]
            if k == "wrong":
                objects.append(dict(x=W + 2.0, y=lane, v=-1.0, kind="disc", abnormal=k))
            elif k == "fast":
                objects.append(dict(x=-3.0, y=lane, v=3.0, kind="disc", abnormal=k))
            else:
                objects.append(dict(x=-3.0, y=lane, v=1.0, kind="square", abnormal=k))
        if t >= pre:
            frame = BACKGROUND + sum((render(o) for o in objects), np.zeros((H, W)))
            frames.append(gain * frame + noise * rng.standard_normal((H, W)))
            present = {o["abnormal"] for o in objects if o["abnormal"] and 0 <= o["x"] <= W - 1}
            labels.append(int(bool(present)))
            kinds.append(present)
        for o in objects:
            o["x"] += o["v"]
        objects = [o for o in objects if -4 <= o["x"] <= W + 3]
    return np.array(frames), np.array(labels), kinds


def cuboids(v):   # last three frames, flattened: shape (T-2, 3*H*W), row i ends at frame i+2
    return np.hstack([v[:-2], v[1:-1], v[2:]])


def flat(v):
    return v.reshape(len(v), -1)


class Reconstruction:   # linear autoencoder: PCA of 3-frame cuboids
    def __init__(self, train, k=24):
        X = np.vstack([cuboids(flat(v)) for v in train])
        self.mean = X.mean(0)
        Xc = X - self.mean
        d = Xc.shape[1]   # top-k eigenvectors of the covariance = top-k principal axes
        _, V = scipy.linalg.eigh(Xc.T @ Xc, subset_by_index=[d - k, d - 1])
        self.P = V.T

    def score(self, v):  # one score per frame t >= 2
        X = cuboids(flat(v)) - self.mean
        R = X - (X @ self.P.T) @ self.P
        return (R**2).mean(1)


class Prediction:   # linear predictor of frame t from frames t-2, t-1 (ridge regression)
    def __init__(self, train, lam=1.0):
        X = np.vstack([np.hstack([flat(v)[:-2], flat(v)[1:-1]]) for v in train])
        Y = np.vstack([flat(v)[2:] for v in train])
        self.mx, self.my = X.mean(0), Y.mean(0)
        Xc, Yc = X - self.mx, Y - self.my
        self.B = np.linalg.solve(Xc.T @ Xc + lam * np.eye(X.shape[1]), Xc.T @ Yc)

    def score(self, v):
        X = np.hstack([flat(v)[:-2], flat(v)[1:-1]]) - self.mx
        return (((X @ self.B + self.my) - flat(v)[2:]) ** 2).mean(1)


def frame_difference(v):   # no learning: energy of the change from frame t-1 to t
    return ((flat(v)[2:] - flat(v)[1:-1]) ** 2).mean(1)


def minmax(s):
    return (s - s.min()) / (s.max() - s.min())


def aucs(scores, labels):
    micro = roc_auc_score(np.concatenate(labels), np.concatenate(scores))
    micro_norm = roc_auc_score(np.concatenate(labels), np.concatenate([minmax(s) for s in scores]))
    per_video = [roc_auc_score(l, s) for s, l in zip(scores, labels) if 0 < l.sum() < len(l)]
    return micro, micro_norm, np.mean(per_video), len(per_video)


def run(seed, verbose=False):
    rng = np.random.default_rng(seed)
    T = 200
    train = [make_video(rng, T, rng.uniform(0.02, 0.05), rng.uniform(0.9, 1.1),
                        rng.uniform(0.01, 0.02))[0] for _ in range(20)]
    held_out = make_video(rng, T, rng.uniform(0.02, 0.05), rng.uniform(0.9, 1.1),
                          rng.uniform(0.01, 0.02))[0]
    tests = []
    for i in range(12):
        events = []
        if i < 10:
            n = rng.integers(2, 4)
            starts = np.sort(rng.choice(np.arange(5, T - 40), n, replace=False))
            events = [(int(s), str(rng.choice(["wrong", "fast", "square"]))) for s in starts]
        density, gain, noise = rng.uniform(0.01, 0.08), rng.uniform(0.8, 1.2), rng.uniform(0.005, 0.03)
        v, lab, kinds = make_video(rng, T, density, gain, noise, events)
        tests.append((v, lab[2:], kinds[2:], density, noise))

    rec, pred = Reconstruction(train), Prediction(train)
    scorers = {"frame difference": frame_difference, "reconstruction": rec.score,
               "prediction": pred.score}
    labels = [t[1] for t in tests]
    results = {}
    if verbose:
        print("label-free check, mean error (training videos / held-out normal video):")
        for name, f in [("reconstruction", rec.score), ("prediction", pred.score)]:
            tr = np.mean([f(v).mean() for v in train])
            print(f"  {name:15s} {tr:.2e} / {f(held_out).mean():.2e}")
        print(f"abnormal frames: {sum(l.sum() for l in labels)} of {sum(len(l) for l in labels)}")
        print()
        print(f"{'scorer':18s}{'micro':>8s}{'micro, video-normalized':>25s}{'macro':>8s}")
    for name, f in scorers.items():
        scores = [f(t[0]) for t in tests]
        micro, micro_norm, macro, n_videos = aucs(scores, labels)
        results[name] = (micro, micro_norm, macro)
        if verbose:
            print(f"{name:18s}{micro:8.3f}{micro_norm:25.3f}{macro:8.3f}")
    if verbose:
        print(f"(macro averages the {n_videos} test videos that contain both classes)")
        print()
        print("micro AUC by anomaly type (frames with that type against all normal frames):")
        all_kinds = [k for t in tests for k in t[2]]
        y = np.concatenate(labels)
        for name, f in scorers.items():
            s = np.concatenate([f(t[0]) for t in tests])
            row = []
            for kind in ["wrong", "fast", "square"]:
                m = np.array([kind in k for k in all_kinds]) | (y == 0)
                row.append(roc_auc_score(y[m], s[m]))
            print(f"  {name:18s}" + "".join(f"{kind} {a:.3f}   " for kind, a in
                                              zip(["wrong direction", "fast", "square"], row)))
    return results


if __name__ == "__main__":
    seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
    shown = run(seed, verbose=True)
    print()
    print("range over seeds 0-4 (micro / micro, video-normalized / macro):")
    allres = [shown if s == seed else run(s) for s in range(5)]
    for name in allres[0]:
        cols = np.array([r[name] for r in allres])
        print(f"  {name:18s}" + "   ".join(f"{c.min():.3f}-{c.max():.3f}" for c in cols.T))

label-free check, mean error (training videos / held-out normal video):
  reconstruction  9.69e-04 / 8.40e-04
  prediction      1.43e-04 / 1.98e-04
abnormal frames: 531 of 2376

scorer               micro  micro, video-normalized   macro
frame difference     0.619                    0.729   0.812
reconstruction       0.631                    0.795   0.820
prediction           0.926                    0.934   0.992
(macro averages the 10 test videos that contain both classes)

micro AUC by anomaly type (frames with that type against all normal frames):
  frame difference  wrong direction 0.449   fast 0.833   square 0.662   
  reconstruction    wrong direction 0.560   fast 0.670   square 0.687   
  prediction        wrong direction 0.982   fast 0.991   square 0.859   

range over seeds 0-4 (micro / micro, video-normalized / macro):
  frame difference  0.619-0.777   0.657-0.754   0.713-0.822
  reconstruction    0.631-0.766   0.727-0.795   0.753-0.820
  prediction        0.925-0.983   0.934-0.952   0.989-0.994

The first lines check, without labels, what the two fitted models learned about normal video. On a held-out normal video, the predictor’s mean error is within the range of the sensor-noise variance used for training video ($10^{-4}$ to $4 \times 10^{-4}$), which no predictor can remove. The 24-component reconstruction leaves an error several times larger, so this linear “autoencoder” is a weak model of the normal scene. The number of components was fixed before the first run, and the script and output are shown as first run, without later changes.

The pooling dominates the headline numbers. In the seed-0 run, for every scorer, raw micro AUC is the lowest of the three and macro AUC the highest, and over seeds 0–4 the ranges of raw micro and macro for the prediction scorer do not overlap. By construction, test videos with more blobs or more noise have higher errors in every frame, so in one pooled ROC curve the normal frames of a noisy video can outrank the abnormal frames of a quiet one; macro AUC compares frames only within each video and is immune to such offsets. Per-video min–max normalization recovers part of the gap for the two weaker scorers and little for prediction; for the frame difference, the highest normalized value over the five seeds (0.754) is below the highest raw value (0.777), so in at least one seed normalization lowered the AUC. Normalization is a per-video rescaling, not a guarantee. The order of the three scorers is the same under every pooling in the seed-0 run, so the example shows the values disagreeing rather than the ranking; the published Avenue numbers above show a ranking change.

The breakdown by anomaly type shows what each scorer sees. Frame difference responds mainly to fast motion; it gives no useful signal for the wrong-direction blob (0.449), as expected for an object that changes the frame as much as a normal blob does. Prediction separates wrong direction and speed almost perfectly, presumably because a linear map learned from rightward motion at one pixel per frame cannot extrapolate other motion. Against the plan’s hypothesis, the cuboid reconstruction was not the better detector of the unusual object (0.687 against 0.859 for prediction). A plausible reading, not tested here, is that the predictor reconstructs a blob’s shape as part of its motion, so a square leaves residual error too, while the weak PCA model spreads its error over everything. Georgescu et al.’s ablation, in which the arrow of time suited Avenue’s action anomalies and appearance tasks suited Ped2’s object anomalies, is a similar observation at benchmark scale: what a scorer models determines which anomalies it can see.

What the Benchmarks Call Abnormal

Each benchmark operationalizes “abnormal” as its own short list of event types, from bikes on a walkway (UCSD) and staged throwing (Avenue) to crimes chosen for their impact on public safety (UCF-Crime) and six classes of violence (XD-Violence), as described under Datasets and Benchmarks. Wu et al. (2020, Section 3.1) state the underlying difficulty for their case: no generic definition of violence exists, so they restricted the classes to reduce annotators’ ambiguity. Several documented consequences follow.

• Unusual and abnormal are not the same. A model of regularity flags events that are rare in training but annotated as normal, such as Hasan et al.’s running at the Subway Exit and Sultani et al.’s crowd watching a race. Wu et al. (2020, Section 5.5) found their model flagging the second half of a test video that is not annotated as violent, and traced it to an overturned car left after the accident had ended.
• Labels can change. Beyond the revised UCSD ground truth and Subway’s differing annotations, UCF-Crime and XD-Violence average temporal boundaries over annotators, and Ramachandra and Jones (2020, Section 4) note ambiguities in when an anomalous track begins and ends.
• Closed sets, pooled scenes and easy anomalies. The weakly supervised abnormal class is a union of categories seen in training, pooling scenes loses scene-specific normality, and UCSD and UMN anomalies are easy to see, as described above.

A frame-level AUC on these benchmarks therefore measures agreement with one dataset’s list of events under one pooling convention. It is not a rate of correct alarms for “anomalies” in general.

Applications

The papers cited here motivate the task by surveillance video, in which almost all footage is unremarkable and unusual segments could be flagged for human inspection (Lu et al., 2013, Section 1; Sultani et al., 2018, Section 1; Ramachandra & Jones, 2020, Section 1). Other uses they name include finding meaningful moments in long videos for annotation, summarization and indexing (Hasan et al., 2016, Section 1), violence detection in movies and online video (Wu et al., 2020), and crowd analysis (Mahadevan et al., 2010). The methods connect to temporal action localization, which detects named action classes in untrimmed video, and to video object segmentation, whose masks or tracks an object-centric detector could in principle score. Video Understanding places the task among the other video tasks.

Open Problems

• Evaluation that reflects detection. Frame-level AUC permits lucky co-occurrences and depends on the pooling; region- and track-based criteria need spatial labels, which UCF-Crime and XD-Violence lack, and results that omit the pooling, normalization, split and dataset version cannot be compared.
• Open-set detection with supervision. Using abnormal training data without reducing the task to recognizing known categories is the aim of UBnormal (Acsintoae et al., 2022). Whether models trained on virtual anomalies transfer to real scenes is open, though the authors report gains on Avenue and ShanghaiTech.
• Generalizing normality and saturation. Reconstruction models can reconstruct anomalies too, weakly supervised models face anomalies in unseen scenes, and saturated benchmarks such as Ped2 no longer separate methods.