AVA

AVA (Atomic Visual Actions) is a video dataset from Google Research in which every person in keyframes sampled once per second has a bounding box and one or more action labels from a vocabulary of 80 “atomic” actions, such as stand, sit, hold an object, talk to a person or answer a phone. The footage is the 15th to 30th minute of feature films found on YouTube. Chunhui Gu, Chen Sun, David Ross and colleagues first posted the paper on arXiv in May 2017, offered v1.0 for download by November 2017 and presented the dataset at CVPR 2018; the current release, v2.2, dates from 2019. AVA defined the keyframe form of spatio-temporal action detection: find each person, box them, and give all their simultaneous actions. Its score, frame-level mean average precision at an IoU of 0.5 over 60 classes, is defined by an official evaluation script. From 2018 to 2022 AVA and its extension AVA-Kinetics were a task of the ActivityNet challenge.

Purpose

The authors wanted a dataset where actions are short, physical and visually defined, rather than composite activities such as pole-vaulting. Each action is judged from a three-second segment centred on a keyframe, which they argue is long enough to tell picking up from putting down but avoids having to mark fuzzy action boundaries (Gu et al., 2018, §1). Three other choices follow from that aim (§1):

• Person-centric labels. Each person is labelled separately, so a frame with two people can carry “talk to” for one and “listen to” for the other.
• Exhaustive labelling. Every action of every person in every keyframe is labelled, rather than collecting clips by searching for each action. The result is a long-tailed class distribution, which the authors kept on purpose.
• Movies. Films serve as an approximation of behaviour “in the wild”, with an expected bias from storytelling and film grammar.

Dataset Size

The counts changed with each version, and papers often quote whichever version they used. Each row gives its source:

Version	Source	Movies	Annotated unit	Labels	Train / val / test videos
v1.0	arXiv v2 (July 2017)	192	57.6k three-second segments, 300 per movie	210k	154 / – / 38
(interim)	arXiv v3 (November 2017)	192	keyframes at 1 Hz, 900 per movie	740k	154 / – / 38
v2.0	CVPR 2018 paper; official page, March 2018	437 (paper)	keyframes at 1 Hz, 900 per movie (paper); 351k clips (page)	1.59M (paper); 1.65M (page)	239 / 64 / 134
v2.1	arXiv v4 (April 2018); official page, July 2018	430	keyframes at 1 Hz	1.58M	235 / 64 / 131
v2.2	official page, from March 2019	430	keyframes at 1 Hz	1.62M	235 / 64 / 131

v1.0 annotated non-overlapping three-second segments, so its keyframes were three seconds apart (arXiv v2). The January 2018 page announced that v2.0 would add “more densely sampled annotations and person links”. As of October 2026, the official download page says that v2.1 differs from v2.0 only by the removal of a few movies found to be duplicates, and that v2.2 added a rating round that inserted missing labels (2.5% more annotations) and corrected boxes in a few videos with aspect ratios much wider than 16:9. By its counts, the CVPR paper describes v2.0, before the duplicate movies were removed (our reading; the paper names no version), although its figures differ from the March 2018 page’s: 437 movies, 215k, 57k and 120k segments in its three splits, and 214,622, 57,472 and 120,332 benchmark examples (§3.6, §6.1). arXiv v4 gives the v2.1 figures: 211k, 57k and 118k segments, and 210,634, 57,371 and 117,441 examples. The AVA-Kinetics paper lists the same three numbers as unique annotated frames of AVA in Table 1 (Li et al., 2020). Later papers sometimes mix versions: the SlowFast paper describes AVA as taken from 437 movies while evaluating on v2.1 (Feichtenhofer et al., 2019, §5).

The counts of v1.0 videos also differ between sources. The AVA-Speech paper says v1.0 had 192 videos, 7 of which were no longer on YouTube (Chaudhuri et al., 2018, §3), while the official page and the AVA-ActiveSpeaker paper refer to an original list of 188 videos (Roth et al., 2020).

Classes

The vocabulary has 80 classes in three groups: 14 pose classes (stand, sit, walk, lie/sleep, crouch/kneel, …), 49 person–object interactions (carry/hold, drink, open, answer phone, …) and 17 person–person interactions (talk to, listen to, watch, hug, hand shake, …) (Gu et al., 2018, §3.1). The authors built it to be generic, atomic (independent of the object: “hold” without saying what is held) and exhaustive, iterating until it covered about 99% of the actions annotators labelled. The official page states that the class list and label map have been unchanged since v1.0.

Each person box has exactly one pose label and may have up to three person–object and three person–person labels (§3.5). Per the paper, 28% of boxes have at least one person–object label and 67% at least one person–person label (§4.2), and class sizes roughly follow Zipf’s law (Figure 5).

The benchmark scores only 60 of the 80 classes. The class tables in the appendix of arXiv v4 (Tables 6–7) mark the 20 classes left out of the benchmark, which leaves 13 pose classes, 32 person–object and 15 person–person interactions (our count).

Annotation Types

• Person boxes on each keyframe. A Faster R-CNN person detector, set to high precision, proposed boxes and annotators added the ones it missed, which were 5% more (Gu et al., 2018, §3.3). Wrong boxes were removed during action labelling.
• Person links. Boxes in adjacent keyframes were matched by a person embedding and the Hungarian algorithm, and annotators verified each match, giving 81,000 tracklets of a few seconds to a few minutes (§3.4).
• Action labels. Annotators saw the keyframe with the box and the three-second segment as a looping video. Several annotators first proposed actions, then others verified them. Each segment had three annotators, and a label was kept if at least two verified it (§3.5). Annotators could also flag inappropriate content and wrong boxes.

The released CSV files have one row per (person, action) pair: video ID, keyframe timestamp in seconds, the box corners normalized to the frame size, the action ID and a person ID that links the box to the same person in adjacent keyframes. A person with three actions therefore appears in three rows with the same box. Keyframes are generally at seconds 902 to 1798 of each video. The official page lists two special cases:

• Included keyframes are scored even when raters found no action in them.
• Excluded keyframes were not annotated because raters marked the clip inappropriate, and are not scored. Separate files list them for each split.

Splits

Splits are by video, so all keyframes of a movie are in one split (Gu et al., 2018, §3.6). The v2.1 and v2.2 splits have 235 training, 64 validation and 131 test videos. Training and validation labels are public. Test labels were withheld for the ActivityNet challenge, whose evaluation server scored test predictions. As of October 2026, the download page announces a release of the ground truth of the challenge test sets of AVA Actions, AVA-Kinetics and AVA-ActiveSpeaker, but its link points to an internal Google address (x20web.corp.google.com) rather than a public one. Most papers report validation results. The paper held out 10% of the training data for tuning (§6.1).

Tasks

• Spatio-temporal action detection on keyframes: boxes plus a score for every class, the official benchmark.
• Action classification with given boxes: scoring the actions of ground-truth person boxes, which separates recognition from person detection. The AVA-Kinetics paper reports its baselines both with ground-truth boxes and with detected ones (Li et al., 2020, §3).
• Video-level detection with linked tubes: the 2018 paper reports a video-mAP only as an upper bound, linking detections through the ground-truth person links, because labels exist only once per second (§5).
• Active speaker detection and speech activity detection on the companion datasets AVA-ActiveSpeaker and AVA-Speech, described under Related Datasets.

Evaluation

The paper defines the measure as PASCAL VOC average precision per class on keyframes at IoU 0.5, averaged over classes (Gu et al., 2018, §6.1). The spatio-temporal action detection article compares it with video-mAP, and average precision covers the metric itself. The benchmark is what the official script computes: get_ava_performance.py in the ActivityNet repository. Its ava/ folder holds a copy of the TensorFlow Object Detection API’s evaluation modules (the files carry the TensorFlow Authors’ copyright). This section follows the script at commit af461c3 (May 2020), the last commit that changed it; the example below runs it. The paper says only that the measure follows PASCAL VOC; the detection cap, the AP variant and the handling of empty and excluded keyframes below are details of the code.

Frame-mAP as the official code computes it

The script takes three files: the ground-truth CSV, a detection CSV in the same format with a score in place of the person ID, and a label map listing the classes to score. For AVA this is the 60-class list.

1. Keys. A keyframe is identified by its video ID and timestamp. Since May 2020 timestamps may carry fractions of a second.
2. Class filter. Rows whose action is not in the label map are dropped from both files. Each remaining ground-truth row is one instance, so a person with two scored actions counts once in each class.
3. Included keyframes. Every key that appears in the ground-truth file is included, including keys given as two-column rows (video ID and timestamp only), which mark keyframes without actions. Detections on any other key are ignored, so excluded timestamps are handled by leaving them out of the ground truth.
4. Top 50 per keyframe. After the class filter, each keyframe keeps only its 50 highest-scoring (box, class) detections.
5. Matching. Detections with zero or negative width or height are dropped. For each class and keyframe, the rest are taken in descending score order. Each one is compared with the ground-truth boxes of that class in that keyframe and takes the one with the highest IoU. It is a true positive if that IoU is at least 0.5 and the box has not been claimed; otherwise it is a false positive, even if another unclaimed box also overlaps by 0.5. No box is marked difficult.
6. Average precision. For each class, the detections of all keyframes are ranked by score, and AP is the all-point interpolated area under the precision–recall curve, which the code’s docstring attributes to the VOC development kit; it is the method of VOC 2010 and later. Classes with no ground-truth instance are skipped, and frame-mAP is the mean over the rest. The script prints it as PascalBoxes_Precision/mAP@0.5IOU, with the AP of each class.

The code has changed in ways that matter for a careful reproduction:

• Excluded and empty keyframes. From April 2018 to March 2019 (commits be7854e to 3fee089) there were no two-column rows. The script instead took an optional file of excluded timestamps (-e), and detections on a keyframe absent from the ground truth were scored against an empty ground truth, so all of them were false positives. Since April 2020 the ground-truth file itself lists the keyframes to score. Apart from the detection cap below, both versions give the same result when used as intended, but the same files can give different numbers: a ground-truth file without two-column rows ignores false positives on empty keyframes in the current code, and counts them in the older one.
• Detection cap. The limit of 50 detections per keyframe was added in March 2019 (commit 3fee089); the 2018 versions scored every detection.
• Non-maximum suppression. The April 2018 version called per-class non-maximum suppression inside the evaluation, with an IoU threshold of 1.0 and a limit of 10,000 boxes. At a threshold of 1.0 the suppression routine returns early, so it only sorted the boxes and capped them at 10,000 per class and keyframe, and the step was removed in March 2019. Deduplicating boxes is left to the detector: the 2018 baseline applied non-maximum suppression at 0.6 to its own output (Gu et al., 2018, §5).
• Ties. The code sorts scores with NumPy’s default sort and reverses the order, so detections with equal scores are taken in an order that is not specified. With many tied scores, a reimplementation that breaks ties differently can differ slightly.

The 60 classes. The CVPR paper selects classes with at least 25 instances in both validation and test (§6.1). The ActivityNet task pages of 2018 to 2020 describe the same list as “the top 60 most common action classes” and link the label maps for v2.1 and v2.2. The selection was redone as the data grew: arXiv v2 benchmarked 44 classes with at least 25 test instances, and arXiv v3 benchmarked 63. Since then the list has been fixed: the challenge server and papers such as SlowFast score the 60 classes of the label map rather than re-deriving a subset.

Example: the protocol on synthetic keyframes

The example renders keyframes as label images, 8 videos × 25 keyframes at “1 Hz”, with zero to four non-overlapping people each, and reads every ground-truth box off its pixels. Each person gets one of 3 pose labels plus independent draws of 9 interaction labels whose frequencies fall off as in a long tail. Two keyframes per video are flagged as excluded and never labelled. A simulated detector returns a jittered box with a sigmoid score for every class on every person, including the unlabelled people in excluded keyframes, and hallucinates an extra person in 30% of keyframes. Weaker separation for rarer classes mimics their difficulty. The NumPy function implements the six steps above. If the environment variable AVA_EVAL_DIR points to the repository’s Evaluation folder at commit af461c3, the same rows are written to CSV files and scored by the official get_ava_performance.py. The official code needs one shim to run under NumPy 2: it uses np.float, which NumPy removed in version 1.24. The benchmark subset is chosen as AVA’s was, from the classes with at least 25 instances.

import os, sys, csv, tempfile
import numpy as np
from scipy import ndimage

# A 12-class toy vocabulary: one pose label per person, interactions optional.
POSE = {1: "stand", 2: "sit", 3: "walk"}
INTER = {4: "talk to", 5: "listen to", 6: "watch (person)", 7: "carry/hold",
         8: "answer phone", 9: "drink", 10: "hug", 11: "dance", 12: "swim"}
P_POSE = [0.5, 0.35, 0.15]
P_INTER = [0.45, 0.35, 0.25, 0.30, 0.06, 0.05, 0.03, 0.015, 0.008]   # long tail
H, W, N_VID, N_KEY = 72, 128, 8, 25            # 8 videos x 25 keyframes ("1 Hz")


def draw_labels(rng):
    """One pose label, then each interaction independently (multi-label)."""
    return [1 + rng.choice(3, p=P_POSE)] + [c for c, p in zip(INTER, P_INTER)
                                            if rng.random() < p]


def make_scene(rng):
    """Render each keyframe as a label image with 0-4 people, read every
    ground-truth box off its pixels, and draw the labels. A few keyframes per
    video are flagged as excluded (as AVA raters flag inappropriate clips)."""
    gt, keys, excluded = [], [], []
    pid = 0
    for v in range(N_VID):
        flagged = set(rng.choice(N_KEY, 2, replace=False))
        for k in range(N_KEY):
            key = (f"vid{v}", f"{902 + k:04d}")
            img = np.zeros((H, W), int)
            n = rng.choice(5, p=[0.12, 0.33, 0.30, 0.15, 0.10])
            slots = rng.choice(4, n, replace=False)          # people do not overlap
            for i, s in enumerate(slots, 1):
                w, h = rng.integers(10, 30), rng.integers(30, 70)
                x0 = s * 32 + rng.integers(0, 32 - w)
                y0 = rng.integers(0, H - h + 1)
                img[y0:y0 + h, x0:x0 + w] = i
            if k in flagged:
                excluded.append(key); continue                # never labelled
            keys.append(key)
            for sl in ndimage.find_objects(img):
                box = (sl[1].start / W, sl[0].start / H, sl[1].stop / W, sl[0].stop / H)
                pid += 1
                gt += [(*key, *np.round(box, 3), c, pid) for c in draw_labels(rng)]
    return gt, keys, excluded


def detect(gt, keys, excluded, rng, sep=2.0):
    """Simulated detector: a jittered box with a sigmoid score for every class
    on every person, also on the people in excluded keyframes, whose actions
    nobody labelled; in 30% of keyframes it also hallucinates a person and
    scores it like one."""
    people = {}
    for vid, ts, *box, c, pid in gt:
        people.setdefault(((vid, ts), pid), [tuple(box), set()])[1].add(c)
    for key in excluded:
        for j in range(1, rng.integers(2, 4)):
            x, y = rng.uniform(0, 0.75), rng.uniform(0, 0.4)
            people[(key, -j)] = [(x, y, x + 0.15, y + 0.6), set(draw_labels(rng))]
    for key in keys + excluded:
        if rng.random() < 0.3:
            x, y = rng.uniform(0, 0.8), rng.uniform(0, 0.5)
            people[(key, 0)] = [(x, y, x + 0.12, y + 0.4), set(draw_labels(rng))]
    dets = []
    for (key, pid), (box, labels) in people.items():
        b = np.clip(np.array(box) + rng.normal(0, 0.02, 4), 0, 1)
        for c in list(POSE) + list(INTER):
            s = sep if c in POSE else sep * (0.4 + 0.6 * P_INTER[c - 4] ** 0.25)
            logit = (s if c in labels else -s) + rng.normal(0, 1.0)
            dets.append((*key, *np.round(b, 3), c, round(1 / (1 + np.exp(-logit)), 6)))
    return dets


def iou(b, g):
    """IoU of one box against many, boxes as (x1, y1, x2, y2)."""
    w = np.clip(np.minimum(b[2], g[:, 2]) - np.maximum(b[0], g[:, 0]), 0, None)
    h = np.clip(np.minimum(b[3], g[:, 3]) - np.maximum(b[1], g[:, 1]), 0, None)
    inter = w * h
    area = lambda r: (r[..., 2] - r[..., 0]) * (r[..., 3] - r[..., 1])
    return inter / (area(b) + area(g) - inter)


def voc_ap(scores, tp, n_gt):
    """All-point interpolated AP, as in the evaluation code's metrics.py."""
    order = np.argsort(-np.asarray(scores), kind="stable")
    tp = np.asarray(tp, float)[order]
    rec = np.r_[0, np.cumsum(tp) / n_gt, 1]
    prec = np.r_[0, np.cumsum(tp) / np.arange(1, len(tp) + 1), 0]
    prec = np.maximum.accumulate(prec[::-1])[::-1]
    i = np.flatnonzero(rec[1:] != rec[:-1]) + 1
    return np.sum((rec[i] - rec[i - 1]) * prec[i])


def frame_map(gt_rows, det_rows, classes, cap=50):
    """AVA frame-mAP@0.5: a keyframe counts if the ground truth lists it (a
    2-column row marks one with no actions); labels outside `classes` are
    dropped; each keyframe keeps its `cap` highest-scoring (box, class) pairs;
    per class and keyframe, detections by descending score take their
    highest-IoU box, a TP if IoU >= 0.5 and that box is not yet taken."""
    included = {tuple(r[:2]) for r in gt_rows}
    gt = {}
    for r in gt_rows:
        if len(r) > 2 and r[6] in classes:
            gt.setdefault((tuple(r[:2]), r[6]), []).append(r[2:6])
    per_key = {}
    for r in det_rows:
        if r[6] in classes and tuple(r[:2]) in included:
            per_key.setdefault(tuple(r[:2]), []).append(r)
    aps = {}
    for c in classes:
        n_gt = sum(len(b) for (k, cc), b in gt.items() if cc == c)
        if n_gt == 0:
            continue                                         # class not scored
        scores, tps = [], []
        for key, rows in per_key.items():
            rows = sorted(rows, key=lambda r: -r[7])[:cap]
            g = np.array(gt.get((key, c), []), float).reshape(-1, 4)
            taken = np.zeros(len(g), bool)
            for r in rows:
                if r[6] != c:
                    continue
                hit = False
                if len(g):
                    ov = iou(np.array(r[2:6]), g); j = np.argmax(ov)
                    hit = ov[j] >= 0.5 and not taken[j]
                    taken[j] |= hit
                scores.append(r[7]); tps.append(hit)
        aps[c] = voc_ap(scores, tps, n_gt) if scores else 0.0
    return np.mean(list(aps.values())), aps


def official(gt_rows, det_rows, classes):
    """Run the ActivityNet get_ava_performance.py (commit af461c3) on the same
    rows written as CSV files, if AVA_EVAL_DIR points to its Evaluation folder."""
    root = os.environ.get("AVA_EVAL_DIR")
    if not root:
        return None
    if root not in sys.path:
        sys.path[:0] = [root, os.path.join(root, "ava")]
    import logging
    np.float = float                           # shim: NumPy 1.24 removed np.float
    import get_ava_performance as gap
    out = {}
    gap.pprint.pprint = lambda m, indent=None: out.update(m)
    gap.logging.disable(logging.INFO)
    with tempfile.TemporaryDirectory() as d:
        names = {**POSE, **INTER}
        paths = [os.path.join(d, f) for f in ("labels.pbtxt", "gt.csv", "det.csv")]
        with open(paths[0], "w") as f:
            for c in classes:
                f.write(f'item {{\n  name: "{names[c]}"\n  id: {c}\n}}\n')
        for p, rows in zip(paths[1:], (gt_rows, det_rows)):
            with open(p, "w", newline="") as f:
                csv.writer(f).writerows(rows)
        with open(paths[0]) as l, open(paths[1]) as g, open(paths[2]) as dd:
            gap.run_evaluation(l, g, dd)
    return out["PascalBoxes_Precision/mAP@0.5IOU"]


scene_rng = np.random.default_rng(7)
gt, keys, excluded = make_scene(scene_rng)
labelled = {tuple(r[:2]) for r in gt}
empty = [k for k in keys if k not in labelled]
gt_full = gt + [list(k) for k in empty]                   # 2-column rows
counts = {c: sum(r[6] == c for r in gt) for c in list(POSE) + list(INTER)}
subset = [c for c, n in counts.items() if n >= 25]
print(f"{len(keys)} keyframes ({len(empty)} with no one, {len(excluded)} excluded),"
      f" {len({r[7] for r in gt})} person boxes, {len(gt)} (box, action) labels")
print("instances per class:", counts)
print("benchmark subset (>= 25 instances):", subset)

variants = [
    ("subset, empty keyframes listed", gt_full, subset),
    ("all 12 classes", gt_full, list(counts)),
    ("subset, empty keyframes omitted", gt, subset),
    ("subset, excluded scored as empty", gt_full + [list(k) for k in excluded], subset),
]
print(f"\n{'variant':34} | {'frame-mAP, seeds 0-9':20} | official, max |diff|")
for name, g, cls in variants:
    ours, diff = [], []
    for seed in range(10):
        dets = detect(gt, keys, excluded, np.random.default_rng(seed))
        m, aps = frame_map(g, dets, cls)
        ours.append(m)
        o = official(g, dets, cls)
        diff.append(abs(o - m) if o is not None else np.nan)
    check = f"{np.nanmax(diff):.1e}" if not np.isnan(diff).all() else "not run"
    print(f"{name:34} | {np.mean(ours):.3f} [{min(ours):.3f}-{max(ours):.3f}] | {check}")

dets = detect(gt, keys, excluded, np.random.default_rng(0))
_, aps = frame_map(gt_full, dets, list(counts))
print("\nseed 0, AP per class:", {c: round(float(a), 2) for c, a in aps.items()})

184 keyframes (31 with no one, 16 excluded), 316 person boxes, 772 (box, action) labels
instances per class: {1: 165, 2: 104, 3: 47, 4: 138, 5: 106, 6: 78, 7: 84, 8: 22, 9: 18, 10: 7, 11: 1, 12: 2}
benchmark subset (>= 25 instances): [1, 2, 3, 4, 5, 6, 7]

variant                            | frame-mAP, seeds 0-9 | official, max |diff|
subset, empty keyframes listed     | 0.825 [0.779-0.863] | 0.0e+00
all 12 classes                     | 0.725 [0.665-0.794] | 0.0e+00
subset, empty keyframes omitted    | 0.840 [0.799-0.878] | 0.0e+00
subset, excluded scored as empty   | 0.773 [0.712-0.811] | 0.0e+00

seed 0, AP per class: {1: 0.84, 2: 0.86, 3: 0.81, 4: 0.82, 5: 0.8, 6: 0.83, 7: 0.85, 8: 0.81, 9: 0.79, 10: 0.67, 11: 0.33, 12: 0.5}

The output was produced with the official code available. The NumPy reimplementation and the official script agree exactly in all 40 evaluations, so the six steps describe what the script computes. Each row gives the mean over detector seeds 0–9 and their range.

• Class subset. Scoring all 12 classes instead of the 7 with at least 25 instances lowers the mean by about 0.10 and widens the spread across seeds. The rare classes are scored on one to 22 instances, where a single detection moves the AP: “dance” has one instance and an AP of 0.33 under seed 0. This is the instability the AVA authors gave as their reason for the threshold.
• Empty keyframes. Leaving the two-column rows out of the ground truth raises the score, because the current code then ignores the hallucinated people on empty keyframes instead of counting them as false positives.
• Excluded keyframes. Scoring the excluded keyframes as if they had no actions lowers the score sharply, because the detector correctly finds the people in them, who simply have no labels.

The last two rows differ only in which keyframes the evaluation is told about, not in the detections. Our reasoning is that a submission’s score therefore depends on using the ground-truth files and timestamp lists of the right version, as well as on the detector.

License

As of October 2026, the official download page states that all datasets listed there, namely AVA-Kinetics, AVA Actions, AVA-ActiveSpeaker and AVA-Speech, are made available by Google under a Creative Commons Attribution 4.0 International (CC BY 4.0) licence. What the page distributes are annotation files (CSV files, label maps and timestamp lists) keyed by YouTube video ID, not the videos. It states no terms for the films themselves, feature films that the authors found on YouTube (Gu et al., 2018, §3.2), so the terms for the video content are unclear. The Common Visual Data Foundation hosts copies of the videos, and the ActivityNet task pages point to its download instructions. Its README (commit 7034062) states no licence or terms of use for those copies. Secondary sources misstate the annotation licence: the VideoMAE paper lists AVA as CC BY-NC 4.0 while linking the CC BY 4.0 deed (Tong et al., 2022, supplementary §F “License of Data”).

Limitations and Bias

• Movies are not everyday life. The authors expect biases from storytelling and film grammar and note that the depiction of action in film is biased, for example by gender, so the class distribution does not reflect the true distribution of human activity (Gu et al., 2018, §1, §3.2).
• Selection. Movies were found by searching YouTube for the names of top actors of many nationalities, keeping results with a “film” or “television” topic, longer than 30 minutes, at least one year since upload and at least 1,000 views. Black-and-white, low-resolution, animated, cartoon and gaming videos and mature content were excluded (§3.2). Taking minutes 15 to 30 of each film skips titles and trailers; in our observation it also fixes the sample to the same early part of every story.
• The people in the videos are those who appear in the films. As of October 2026, the AVA papers (CVPR 2018 and arXiv v1–v4), the AVA-Kinetics, AVA-ActiveSpeaker and AVA-Speech papers, and the official pages make no statement about consent or a process for removing a video or a person, and report no demographic composition; the two speech papers only describe the speakers as demographically diverse. The only content controls they describe are the exclusion of mature content at selection, raters’ flags for inappropriate content, and the excluded timestamps that result.
• Long tail and the 60-class benchmark. Twenty of the 80 classes are not scored at all, and the mean over 60 classes weights “stand” and “swim” equally. The paper found per-class AP only loosely related to the number of training examples: “swim” scored high because the scene gives it away, while frequent classes such as “touch” and “smoke” scored low (§6.3, Figure 8).
• Missing labels. With 80 classes, annotators miss correct actions. The propose-and-verify design raised recall (§3.5), yet the v2.2 round still added 2.5% more labels (official page). A correct detection of an unlabelled action counts as a false positive.
• Availability. Videos are referenced by YouTube ID, and some disappear: by 2018 seven of the 192 v1.0 videos were gone (Chaudhuri et al., 2018), and AVA-ActiveSpeaker labels 160 of the 188 videos on the original list that were still available (Roth et al., 2020). The third-party CVDF mirror hosts copies of the videos.

Common Uses

AVA v2.1 and v2.2 validation are the standard test for person-level action detectors, which often pair a separately trained person detector with a video backbone, as in SlowFast’s AVA detector. They are also a standard downstream task for video representations: self-supervised video models such as VideoMAE report frame-mAP on AVA v2.2 next to their Kinetics results. AVA-Kinetics is used to train and test the same detectors on more varied scenes. The companion datasets serve audio-visual work: active speaker detection on AVA-ActiveSpeaker and speech activity detection on AVA-Speech.

Historical Milestones

All values are frame-mAP at IoU 0.5 over the 60 classes, as printed. Versions, splits and test-time settings differ, so rows are not directly comparable:

Year	Result	Data and protocol	Source
2018	Baseline: I3D two-stream on Faster R-CNN proposals, 15.8	v2.0 validation (CVPR version); 15.6 on v2.1 validation in arXiv v4; 14.7 on test	Gu et al., 2018 (CVPR Table 4 and §6.2; arXiv v4)
2018	First ActivityNet AVA challenge won by Tsinghua/Megvii, 21.08 and 20.99 in the page’s Task 1 and Task 2 columns	v2.1 test server; vision-only and any-input rankings per the 2018 task page; 16 teams, 35 submissions	official Challenges page
2019	SlowFast 8×8 R101, 26.3 (Kinetics-400 pretraining, one clip); 16×8 R101+NL, 29.8 (Kinetics-600 pretraining)	v2.1 and v2.2 validation	Feichtenhofer et al., 2019, Tables 7–8
2019	SlowFast 7-model ensemble with multi-scale and flip testing wins the challenge, 34.25 (34.3 in the paper)	v2.2 test server; 30 teams, 79 submissions	official Challenges page; Feichtenhofer et al., 2019, Table 8
2020	AVA-Kinetics challenge won by CUHK-SenseTime (Actor-Context-Actor Relation Network), 39.62	AVA-Kinetics test server (AVA and Kinetics keyframes)	official Challenges page
2021	AVA-Kinetics challenge won by Alibaba and Tsinghua University, 40.67	AVA-Kinetics test server	official Challenges page
2022	VideoMAE ViT-H, self-supervised on Kinetics-400 then fine-tuned with its labels, 39.5	v2.2 validation, one model	Tong et al., 2022, Table 5

In 2018 the paper’s analysis had placed the difficulty in recognition rather than localization: on validation, class-agnostic actor detection reached 75.3 while action detection reached 15.8 (Gu et al., 2018, Table 5). The SlowFast paper later attributed much of its gain over earlier two-stream detectors to its Fast pathway rather than to optical flow. The 2019 winning entry was an ensemble with test-time augmentation, against 30.7 on validation for a single SlowFast model with the same test-time augmentation (Feichtenhofer et al., 2019, Table 8), and from 2020 the challenge moved to AVA-Kinetics, whose test set mixes AVA and Kinetics keyframes. The official page’s “+5.4 mAP” for 2020 therefore compares results on different test sets (our reading). The 2022 challenge ran AVA-Kinetics and active speaker tasks, but as of October 2026 the official page lists results for the active speaker task only.

Related Datasets

• AVA-Kinetics (Li et al., 2020) adds AVA labels to one keyframe of each of 238k Kinetics-700 clips. The keyframe is the frame with the most confident person detection, at least one second from either end of the clip, and is labelled with a two-second clip centred on it under the same propose-and-verify protocol. Combined with the AVA v2.2 videos, it has 352,091 training, 89,882 validation and 182,457 test frames (Table 1). Training clips were sampled to favour 27 AVA classes that had performed poorly, through 115 related Kinetics classes; validation and test clips were annotated in full. Its paper reports a gain of 5.26 mAP on AVA validation from adding the Kinetics clips to training, with ground-truth boxes. It is scored with the same 60 classes and the same code.
• AVA-ActiveSpeaker (Roth et al., 2020) labels whether each visible face is speaking and audible on the v1.0 movies: about 3.65 million frames in about 38,500 face tracks per the paper, which elsewhere gives about 40K tracks from 160 videos (about 39K per the official page). Its labels are SPEAKING_AND_AUDIBLE, SPEAKING_BUT_NOT_AUDIBLE and NOT_SPEAKING.
• AVA-Speech (Chaudhuri et al., 2018) labels speech activity densely over the same 15-minute segments as no speech, clean speech, speech with music or speech with noise: 185 movies and 45 hours per the paper; about 46K labelled segments and 45 hours per the official home page, whose download section says the labels cover 160 videos. The third-party mirror’s README gives about 40K segments and 40 hours.
• UCF101-24 and J-HMDB are the tube-based detection benchmarks that preceded AVA. The AVA authors contrast them as having few, composite actions in short clips (Gu et al., 2018, §2).
• Kinetics and ActivityNet label clips or temporal segments rather than people. Many AVA methods start from Kinetics pretraining, as the 2020 challenge page notes.
• Charades, whose videos were acted out by crowdsourced workers (Gu et al., 2018, §2), and the egocentric EPIC-KITCHENS label actions in time but have no per-person boxes like AVA’s.

Resources

• The paper: Gu et al. (2018), on the annotation design, the baseline model and how the arXiv versions differ.
• Official pages: home, download (CSV files, label maps, and included and excluded timestamps for v2.2, archives of v2.1 and v1.0, the AVA-Kinetics archive) and challenge results 2018–2022. As of October 2026 the download and challenge addresses redirect to sites.research.google/gr/ava/, while the home address still serves an older copy of the home page with the same counts.
• Evaluation code: get_ava_performance.py in the ActivityNet repository (official; commit af461c3).
• Challenge task pages: 2018, 2019 and 2020, with the submission format.
• Videos: the Common Visual Data Foundation mirror, a third-party copy.
• Baseline: the download page offers a Faster R-CNN ResNet-101 keyframe detector trained with the TensorFlow Object Detection API, which it reports at 11.25 mAP on the 60 classes of v2.1 validation, together with its validation predictions.