AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions

“AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions” is a paper by Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid and Jitendra Malik, all of Google Research, with Schmid also at Inria Grenoble and Malik at UC Berkeley. It was presented at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) in 2018 and posted in four versions on arXiv between May 2017 and April 2018. The paper introduced AVA: keyframes sampled once per second from the 15th to the 30th minute of 437 movies, in which every person has a box and one or more labels from a vocabulary of 80 “atomic” actions, 1.59 million labels in all. It also built a detector from an I3D video network and Faster R-CNN that set new results on two older benchmarks but reached only 15.8% frame-mAP on AVA. This article follows the proceedings version and notes where the arXiv versions differ. The dataset as released and used today, its versions and its official evaluation code are described in the AVA article, and the task in spatio-temporal action detection.

Problem

The authors sorted the action datasets of 2017 into four groups (§2). Classification sets such as KTH, Weizmann, Hollywood-2, HMDB and UCF101 hold short clips trimmed around one action. Large sets such as Sports-1M, YouTube-8M, Something-Something, Moments in Time and Kinetics label whole clips, often automatically. ActivityNet, THUMOS, MultiTHUMOS and Charades localize actions in time but not in space. The sets with boxes on people, such as UCF Sports and J-HMDB with short clips, and the extensions of UCF101, DALY and Hollywood2Tubes to untrimmed videos, have few classes, their actions are composite, such as pole-vaulting, and they do not label everyone: in UCF101’s BasketballDunk only the dunking player is annotated.

Behind this the authors saw a vocabulary problem. Citing Barker and Wright’s 1954 study of everyday behaviour in a Kansas town, they argued that activity is hierarchical: at the finest level it consists of body movements and object manipulations, while coarser levels are described by intentions and goals, which cannot be listed exhaustively (§1). At a fine time scale actions are physical and visually distinct, so a short, fixed context can label them. They also pointed to the THUMOS observation that action boundaries are fuzzy and annotators disagree on them, and argued that labelling a keyframe from three seconds of video avoids marking boundaries while still localizing an action to about ±0.5 s.

Finally, they suspected that high scores on UCF101 and J-HMDB gave an “artificially rosy picture” (§5): a single person swimming in a pool is easy to tell from a person running, whereas real scenes have several small actors whose actions differ subtly, such as touching versus holding an object.

Contribution

The abstract lists five characteristics of the dataset:

1. Atomic actions rather than composite ones.
2. Precise spatio-temporal annotation, with possibly several labels for each person.
3. Exhaustive annotation of these actions over 15-minute clips.
4. People linked across consecutive keyframes.
5. Movies as the source, for a varied set of action representations.

Beyond the data, the paper contributed a baseline detector that classifies short tubes with I3D features (§5), analyses of the label statistics and of how actions follow one another (§4), and experiments meant to show where AVA’s difficulty lies (§6).

Method

Building the dataset

Section 3 describes five stages. The AVA article gives the outcome; here are the design decisions as the paper states them.

• Vocabulary. Three principles: generality (daily-life actions rather than activities tied to a setting such as a basketball court), atomicity (actions that look the same whatever the object, such as “hold”) and exhaustivity. The list began from earlier datasets and was revised over several rounds until it covered about 99% of the actions annotators labelled, ending with 14 pose, 49 person–object and 17 person–person classes (§3.1).
• Movies. For each name on a list of top actors of many nationalities, the authors took up to 2,000 YouTube search results and kept films and television programmes that passed filters on length, age, views and content. They deliberately did not filter by action keywords or action classifiers, nor force a uniform label distribution, and they note that film depicts action with biases, for example by gender (§3.2). Each movie contributes the same 15 minutes, a length chosen to fit more movies, and so more diversity, into a fixed annotation budget.
• Segments. Each 15-minute clip is cut into 900 overlapping three-second segments with a stride of one second; the label applies to the middle frame (§3.2).
• Boxes and links. A Faster R-CNN person detector set for high precision proposed boxes, annotators added the boxes it missed, which were only 5% more, and wrong boxes were removed at the action stage (§3.3). A person embedding (PersonNet) and the Hungarian algorithm linked boxes between adjacent keyframes, and annotators verified every match, giving 81,000 tracklets (§3.4).
• Actions. The annotation interface showed the keyframe with the box above the segment playing as a loop, with fields for one required pose label and up to three person–object and three person–person labels, an “other action” checkbox and flags for blocked or inappropriate content and bad boxes (§3.5, Figure 3). Because annotators facing 80 classes inevitably missed correct actions, the authors split labelling into a proposal stage and a verification stage, following Sigurdsson et al. (2016). Proposing took 22 seconds per segment on average and verifying 19.7 seconds. Three annotators labelled each clip, and a label became ground truth when at least two verified it (§3.5).
• Splits. 239 training, 64 validation and 134 test videos, about 55:15:30, giving 215k, 57k and 120k segments (§3.6).

The paper says that the recall gain of the two-stage design is analysed in the supplementary material. The CVF open-access page offers no supplementary file (as of October 2026). arXiv v3 and v4 carry an appendix with that analysis: a per-class bar chart of how many labels came from each stage, whose caption states that more than half of the classes doubled their recall through verification, and in which small classes such as “kick”, “exit” and “extract” were the ones most often missed at the proposal stage (v4 appendix §8, Figure 11).

The baseline detector

The detector follows Peng and Schmid’s (2016) end-to-end Faster R-CNN for actions, which, the authors note, loses temporal information at its first layer, where the channels of several frames are concatenated. The AVA model instead keeps time throughout with I3D, the inflated Inception network of Carreira and Zisserman (2017), pretrained on Kinetics (§5, Figure 7):

• Proposals come from a region proposal network on a 2D ResNet-50 that sees only the keyframe, so that proposal quality does not depend on the clip length fed to I3D.
• Features are I3D’s 3D feature maps at the Mixed_4e layer, whose spatial stride of 16 matches ResNet’s conv4 block.
• 3D RoI pooling applies ordinary 2D RoI pooling at the same box location in every time step.
• Two streams. An RGB I3D and an optical flow I3D, with flow from FlowNet 2.0, are fused by averaging their feature maps.
• Loss. Because AVA’s labels are not mutually exclusive, the softmax is replaced by an independent sigmoid loss per class on AVA; the other datasets keep the softmax.
• Training. Asynchronous SGD for 600K to 1M iterations, the count set on a validation set, at an input of 320 by 400 pixels, with other parameters at values tuned for object detection; frame-level detections pass through non-maximum suppression at 0.6.

A 2D baseline, a variant of Peng and Schmid’s model with a ResNet-50 Faster R-CNN, one RGB frame and five stacked flow frames, measures what the 3D model adds.

For video-level results, detections are linked into tubes with the algorithm of Singh et al. (2017) without its temporal labelling step. Because AVA has labels only once per second and a tube may carry several actions, the paper’s AVA video-mAP is an upper bound: detections are linked through the ground-truth person links, and the IoU for a class counts only the parts of a tube labelled with that class (§5).

Analyses of the labels

Section 4 characterizes the annotations. Class sizes roughly follow Zipf’s law (Figure 5), which the authors present as the way it should be: models need to cope with realistic long-tailed distributions, and exhaustive labelling also avoids a query bias, since a door opening that someone tagged on YouTube is unusual in a way that ordinary door openings in films are not (§1). Box sizes vary with zoom and pose: boxes labelled “enter” average 30% of the image width and 72% of its height, while “lie/sleep” boxes are close to square (§4.2, Figure 6). Every box has a pose label, 28% have at least one person–object label and 67% at least one person–person label.

The person links let the authors measure which actions follow one another. They rank pairs by normalized pointwise mutual information,

$$\mathrm{NPMI}(x, y) = \frac{\ln \dfrac{p(x, y)}{p(x)\,p(y)}}{-\ln p(x, y)},$$

which tends to −1 for pairs that never co-occur, is 0 for independent pairs and 1 for pairs that always do (§4.3). For the same person in consecutive seconds, the top pairs after removing identity transitions include ride → drive (0.68), open → close (0.59), text on or look at a cellphone → answer phone (0.53) and fall down → lie/sleep (0.46) (Table 1). For different people at the same time, they include ride ↔ drive (0.60), play musical instrument ↔ listen (0.57) and take ↔ give/serve (0.51) (Table 2). The authors offer these transitions as data for models of longer activities.

Results

The benchmark kept the classes with at least 25 instances in both validation and test, 60 classes with 214,622 training, 57,472 validation and 120,332 test examples; results are trained on the training set and reported on validation, with 10% of the training data held out for tuning. Frame-mAP follows the PASCAL VOC protocol at an IoU of 0.5, and video-mAP uses 3D IoU at 0.5 (§6.1). How the official code later turned this into a benchmark is described in the AVA article.

Older benchmarks. On J-HMDB (928 clips, 21 classes, averaged over three splits) and UCF101-24 (3,207 videos, split 1), the two-stream I3D model reached 73.3% and 76.3% frame-mAP and 78.6% and 59.9% video-mAP at IoU 0.5 (Table 3). The best earlier entries in the table were ACT, with 65.7% and 69.5% frame-mAP and 73.7% and 51.4% video-mAP, and T-CNN, with 76.9% video-mAP on J-HMDB.

AVA. The same model reached 15.8% frame-mAP on validation, 12.3% video-mAP at IoU 0.5 and 17.9% at 0.2, and 14.7% frame-mAP on test (§6.2). The authors attribute the gap to the vocabulary, in which context and object cues discriminate less, so that fine detail and temporal modelling matter more.

Temporal context. Table 4 varies the number of frames per stream:

Model	Input	J-HMDB	UCF101-24	AVA
2D	1 RGB + 5 flow	52.1	60.1	14.2
3D	5 RGB + 5 flow	67.9	76.1	13.6
3D	10 RGB + 10 flow	73.4	78.0	14.2
3D	20 RGB + 20 flow	76.4	78.3	14.8
3D	40 RGB + 40 flow	76.7	76.0	15.8
3D	50 RGB + 50 flow	–	73.2	15.7
3D	20 RGB	73.2	77.0	14.6
3D	20 flow	67.0	71.3	10.1

Frame-mAP at IoU 0.5 in percent; J-HMDB split 1, UCF101-24 split 1 evaluated on a random subset of 20,000 frames, AVA validation (Gu et al., 2018, Table 4).

Every 3D model beat the 2D baseline on J-HMDB and UCF101-24, but on AVA the 3D model needed more than ten frames to do so. AVA kept improving up to 40 frames per stream, whereas the other two datasets saturated at about 20, and the authors read this, together with the transitions of Table 1, as a sign that longer temporal context could help further. Combining RGB and flow beat either alone.

Localization or recognition. Table 5 compares action detection with class-agnostic actor detection on the same data: 76.7% against 92.8% on J-HMDB, 76.3% against 84.8% on UCF101-24, and 15.8% against 75.3% on AVA. The authors conclude that AVA’s difficulty lies in classifying actions rather than finding people. Their text puts the gap at “nearly 60%” on AVA and “less than 15%” on J-HMDB and UCF101; by the table, the J-HMDB gap is 16.1 points. Table 5 also does not say which configuration it reports: its J-HMDB and AVA numbers equal the 40 + 40 row of Table 4, while its UCF101-24 number equals Table 3 rather than any row of Table 4 (our reading). arXiv v3 printed 78.3% there, the 20 + 20 row of its Table 4; v4 has 76.3%, as the proceedings do.

Per class. Figure 8 plots per-class AP against the number of training examples. More data generally helped, but “swim”, one of the smallest classes, scored among the highest because the scene gives it away, and “fall down”, a class of low diversity, also scored high, while frequent classes such as “touch” and “smoke” scored low. Figure 9 shows high-scoring false alarms for smoking, which the authors take as a sign that recognition fails on fine detail. They conclude that atomic actions will need not only large datasets but also rich models of motion and interaction.

Example: what 3D RoI pooling sees of a moving actor

The baseline pools features inside the keyframe’s box at every time step. The sketch renders a person as a head and body on an otherwise empty frame of 320 by 400 pixels, reduces each frame to a stride-16 grid of actor coverage as a stand-in for the Mixed_4e feature map, and applies the paper’s 3D RoI pooling: the keyframe’s box replicated over a clip centred on the keyframe, pooled per frame and averaged over time. The output is the pooled actor response relative to a 2D RoI on the keyframe alone, for clip lengths from Table 4 and actors moving sideways at constant speed. Speeds are in pixels per frame, because the paper does not state the frame rate of the clips fed to I3D. A last run lets the actor drift in a random walk instead.

import numpy as np

H, W, STRIDE = 320, 400, 16            # input size and Mixed_4e stride (Gu et al., §5)
yy, xx = np.mgrid[0:H, 0:W]


def render(cx, cy=170):
    """One frame: a rendered person (head disc + body box) on an empty background."""
    head = (xx - cx) ** 2 + (yy - (cy - 70)) ** 2 <= 14 ** 2
    body = (abs(xx - cx) <= 22) & (yy >= cy - 55) & (yy <= cy + 75)
    return (head | body).astype(float)


def feature_map(frame):
    """Stand-in for a stride-16 feature map: mean actor coverage per 16x16 cell."""
    return frame.reshape(H // STRIDE, STRIDE, W // STRIDE, STRIDE).mean(axis=(1, 3))


def roi_pool_3d(features, box):
    """The paper's 3D RoI pooling: the keyframe's 2D box, replicated at every
    time step, pooled in each frame, then averaged over time."""
    x0, y0, x1, y1 = (int(v) // STRIDE for v in box)
    per_frame = [f[y0:y1 + 1, x0:x1 + 1].mean() for f in features]
    return np.mean(per_frame)


def actor_signal(n_frames, speed, jitter=0.0, seed=0):
    """Pooled actor response for a clip of n_frames centred on the keyframe,
    relative to a single-frame (2D) RoI on the keyframe itself."""
    rng = np.random.default_rng(seed)
    t = np.arange(n_frames) - n_frames // 2           # keyframe at t = 0
    xs = 200 + speed * t + jitter * rng.standard_normal(n_frames).cumsum()
    feats = [feature_map(render(x)) for x in xs]
    key = render(200)
    ys, xs_ = np.nonzero(key)
    box = (xs_.min(), ys.min(), xs_.max(), ys.max())   # ground-truth keyframe box
    return roi_pool_3d(feats, box) / roi_pool_3d([feature_map(key)], box)


lengths = [5, 10, 20, 40, 50]                         # frames per stream, as in Table 4
print("speed (px/frame) | relative actor signal for T =", lengths)
for speed in [0, 1, 2, 4]:
    row = [actor_signal(T, speed) for T in lengths]
    print(f"{speed:16} | " + "  ".join(f"{r:.2f}" for r in row))

print("\nrandom-walk actor (jitter 3 px/frame), T = 40, seeds 0-9:")
vals = [actor_signal(40, 0, jitter=3.0, seed=s) for s in range(10)]
print(f"mean {np.mean(vals):.2f}, range {min(vals):.2f}-{max(vals):.2f}")

speed (px/frame) | relative actor signal for T = [5, 10, 20, 40, 50]
               0 | 1.00  1.00  1.00  1.00  1.00
               1 | 1.00  0.98  0.93  0.82  0.76
               2 | 0.98  0.93  0.82  0.59  0.48
               4 | 0.93  0.82  0.59  0.30  0.24

random-walk actor (jitter 3 px/frame), T = 40, seeds 0-9:
mean 0.82, range 0.54-0.96

For a still actor the replicated box loses nothing, however long the clip. Once the actor moves, the share of the pooled response that comes from the actor falls with the product of speed and clip length: at 4 pixels per frame a 40-frame tube keeps less than a third of it, and frames far from the keyframe contribute mostly background. A random walk does the same to a different degree for each seed. In our reading, this is a cost the paper’s design accepts in exchange for longer context. AVA still gained up to 40 frames per stream; one possible reason, which the paper does not test, is that its most frequent classes, such as stand, sit, talk to and listen to, involve little movement across the frame. Later work took up the fixed-location assumption directly, as described below.

Versions

The arXiv record went through four versions before and after the conference, and the dataset, the benchmark classes and the baseline changed between them. The dataset counts of each release are in the AVA article; the papers differ as follows.

Version	Authors	Data described	Benchmark classes	Baseline	AVA frame-mAP
arXiv v1, May 2017	11	214 movies, 64k three-second segments, 197k labels; 170 training and 44 test videos	40 (≥ 26 test instances)	Faster R-CNN with ResNet-101, RGB + stacked flow	16.6
arXiv v2, July 2017	11	192 movies, 57.6k segments, 210k labels; 154 / 38 videos	44 (≥ 25 test instances)	same	18.4
arXiv v3, Nov. 2017	12	192 movies at 1 Hz, 740k labels; 154 / 38 videos	63 (≥ 25 test instances)	I3D on Faster R-CNN	16.2
CVPR 2018	12	437 movies, 1.59M labels; 239 / 64 / 134 videos	60 (≥ 25 in val and test)	same	15.8 (val)
arXiv v4, Apr. 2018	12	430 movies, 1.58M labels; 235 / 64 / 131 videos	60	same	15.6 (val)

Sources: the abstract, §3 and §5 or §6 of each version.

The first two versions had 11 authors in different orders and described a dataset of non-overlapping three-second segments. In v1 each training segment was labelled by a single annotator and each test segment by three; in v2 every segment had three annotators, and labels given by only one were re-checked in a second round. Their baseline was a frame-level two-stream Faster R-CNN after Peng and Schmid with ResNet-101 in place of VGG and inputs of 600 by 800 pixels, comparing TVL-1 flow with a learned flow network. The best result fused RGB with five stacked learned-flow frames (v2 Table 4). These versions also measured annotator consistency, as the share of a test segment’s raw labels that only one annotator gave (§4.1, Figure 7); that analysis was dropped from v3 onward, although the proceedings’ Table 5 caption still describes the annotators as consistent.

Version 3 added Carl Vondrick as an author and introduced what the proceedings describe: annotation at 1 Hz, person links, the I3D model and the UCF101-24 comparison. v1 and v2 had already compared with earlier results on J-HMDB and ranked label pairs by NPMI, both within a segment and across consecutive segments. The v3 introduction states that a preliminary release, v1.0, annotated at 1/3 Hz without person links, was already online and that the full version, v2.0, “as described in this paper”, would follow. The proceedings name no version and replace that sentence with a statement that the dataset has been released. By its counts the CVPR paper describes the v2.0 release of March 2018 (our reading, as in the AVA article, although the archived March 2018 page gives 1.65M labels rather than 1.59M), while v4 describes v2.1, which, according to the official download page, differs from v2.0 only by the removal of a small number of duplicate movies.

Version 4 is the proceedings text with the v2.1 counts, “nearly 900” keyframes and 897 segments per movie, and new AVA values in Tables 4 and 5: 15.6% for the 40 + 40 model, 14.5% for 20 RGB frames and 9.9% for 20 flow frames. It prints the same video-mAP (12.3% and 17.9%), test frame-mAP (14.7%) and actor detection (75.3%) as the proceedings. It also carries an appendix, as every arXiv version did and the CVF copy does not: the box-annotation interface, the recall analysis of the two-stage labelling, per-class instance counts for train and validation (Tables 6–7, with the 20 classes left out of the benchmark marked) and further examples of detections.

Impact

AVA became the standard benchmark for keyframe-based spatio-temporal action detection. The ActivityNet challenge at CVPR 2018 ran an AVA task on v2.1, scored by frame-mAP at IoU 0.5 over 60 classes, and pointed participants to the arXiv paper for baseline results (official task page). Challenges on AVA and its extensions followed until 2022; the AVA article records them.

Later papers quote the v2.1 numbers of arXiv v4 even when they cite the proceedings. The SlowFast paper lists the AVA baseline as I3D at 14.5% without flow and 15.6% with flow on v2.1 and takes the difference as a mild gain from optical flow (Feichtenhofer et al., 2019, Table 7, §5.1). In our reading, that difference also mixes clip lengths, since v4’s 14.5% is a 20-frame RGB model and 15.6% a 40 + 40 two-stream model. SlowFast also kept the paper’s design: its detector extends each 2D RoI into a 3D one by replicating it along time, “similar to” Gu et al., uses a per-class sigmoid classifier, and restates the conclusion of Table 5 that the difficulty of AVA lies in action detection rather than actor localization (Feichtenhofer et al., 2019, §5). Its detectors, with Kinetics pretraining and no optical flow, reached 26.3% to 28.2% on v2.1 validation.

The annotation protocol outlived the AVA movies. The AVA-Kinetics paper calls AVA an influential example of a video task that is expensive to annotate, and labelled one keyframe in each of over 230k Kinetics-700 clips (a subset of the training clips and all validation and test clips) with AVA’s boxes and 80 classes through a propose-and-verify process (Li et al., 2020, abstract, §1, §1.2). The AVA project also added speaking and speech labels on the same movies, as the AVA article describes.

Limitations

The authors name several limitations themselves:

• Movies are an approximation. Behaviour “in the wild” was not available, and films carry the bias of storytelling and film grammar; the depiction of action in film is biased, for example by gender, so the label distribution does not reflect the true distribution of human activity (§1, §3.2).
• Missing labels. With 80 classes, annotators inevitably miss correct actions, which motivated the two-stage labelling (§3.5).
• Video-level evaluation. Labels at 1 Hz allow video-mAP only as an upper bound computed through the ground-truth links (§5).

Further limitations come from the publication itself:

• A release that changed within months. The proceedings report v2.0, but v2.1 removed duplicate movies, and v2.2 later added 2.5% more labels and corrected boxes in a few videos (official download page). The headline 15.8% is therefore a result on data that is no longer distributed, and the comparable figure in later work is v4’s 15.6% on v2.1.
• Benchmark details outside the paper. The paper defines the measure only as PASCAL VOC AP at IoU 0.5 over 60 classes. Which keyframes count, how many detections per keyframe are scored and how missing actions are handled were fixed by the official evaluation code, as the AVA article describes.
• Supporting analyses outside the proceedings. The recall analysis of the two-stage labelling and the per-class counts are in the arXiv appendix but not in the CVF copy. The proceedings give no measure of annotator agreement for Table 5’s statement that annotators are consistent; only arXiv v1 and v2 measured it, on the earlier data.
• Text and tables disagree. The text’s “less than 15%” gap for J-HMDB is 16.1 points in Table 5.

What Came After

Most later AVA detectors kept the paper’s two-stage structure: a person detector proposes boxes on the keyframe, a video backbone computes features over a short clip, and RoI features are pooled from the replicated box. SlowFast replaced the two-stream I3D with its own two-pathway network and showed large gains without optical flow. TubeR later argued that methods which repeat 2D proposals over time, citing this paper among them, assume that the action stays at a fixed location, and predicted a box per frame with a transformer instead (Zhao et al., 2022, §1, §4.3). The spatio-temporal action detection article surveys these methods. The dataset itself moved on to v2.2 and to AVA-Kinetics, which adds keyframes from Kinetics clips to the AVA videos (Li et al., 2020).