ActivityNet

ActivityNet is a benchmark of untrimmed YouTube videos of everyday human activities, introduced at CVPR 2015 by Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem and Juan Carlos Niebles (Universidad del Norte and KAUST). Videos carry activity classes from a taxonomy of daily life and the start and end of every activity instance, supporting both classification and temporal action localization. The 200-class release 1.3 (2016) became a standard test set for localization and the base of an annual challenge held at CVPR.

Purpose

Heilbron et al. (2015) found existing action datasets too narrow: mostly sports, cooking or simple movements, about 100 classes of about 100 short, hand-trimmed clips each, with taxonomies two levels deep. They wanted classes that reflect how people spend their days, noting from the American Time Use Survey (ATUS) that Americans spend 1.7 hours a day on household activities and 18 minutes on sports, exercise or recreation. They kept the videos untrimmed, because web videos rarely contain only the activity: a “Preparing pasta” search also returns material about the chef.

Dataset Size

The paper describes the first release: 203 activity classes, on average 137 untrimmed videos per class and 1.41 activity instances per video, 849 video hours in total. Its benchmark sections give 27,801 videos and 193 trimmed instances per class on average. The download page lists releases 1.2 and 1.3; release 1.1 survives only in a commented-out block of its source (as of October 2026):

Release (date on page)	Classes	Training	Validation	Testing
1.1 (06-2015), “Raw ActivityNet”	203	20,565 videos	6,862 videos	–
1.2 (10-2015), “ActivityNet 100”	100	4,819 videos (7,151 instances)	2,383 videos (3,582 instances)	2,480 videos (labels withheld)
1.3 (03-2016), “ActivityNet 200”	200	10,024 videos (15,410 instances)	4,926 videos (7,654 instances)	5,044 videos (labels withheld)

Release 1.1 holds post-processed crowdsourced annotations and release 1.2 in-house curated ones. The 1.1 subsets add up to 27,427 videos, 374 fewer than the paper’s 27,801, unexplained. The home page summarizes release 1.3 as 200 classes, 100 untrimmed videos per class, 1.54 instances per video and 648 video hours; by our arithmetic the training and validation counts above give 23,064 instances in 14,950 videos, 1.54 per video.

The search favored videos under 20 minutes; about half are HD (1280 × 720) and most run at 30 fps.

Classes

The class list comes from the ATUS coding lexicon, built by the US Department of Labor, which organizes more than 2,000 activities into 18 top-level categories, each with two further levels. The authors chose the 203 classes by hand from 7 top-level categories: personal care, eating and drinking, household, caring and helping, working, socializing and leisure, and sports and exercises. ActivityNet keeps the ATUS path above each class, so the hierarchy is at least four levels deep (four, in Section 3.1); the paper’s example places “Filing nails” under “Washing, dressing and grooming”, under “Grooming”, under “Personal care”.

Annotation Types

The collection had three stages (Heilbron et al., 2015):

1. Search. Text queries for each class, expanded with WordNet synonyms, hypernyms and hyponyms, retrieved candidate YouTube videos.
2. Verification. Workers on Amazon Mechanical Turk checked whether each video contains the intended activity, with verifiable test questions; unrelated videos were discarded.
3. Temporal annotation. Several workers marked the start and end of every instance in each video, and their annotations were clustered into agreed segments, each with exactly one class. A video may contain several instances, of more than one class.

The released files list, for each YouTube ID, the duration, resolution, subset and segments, each a label with start and end times in seconds (download page).

Splits

The paper split the videos at random 50/25/25 into training, validation and testing, keeping all instances of a video in one subset. From release 1.2 on, test labels are withheld for the evaluation server, so papers train on the training set and report on validation, which ActionFormer calls common practice for release 1.3 (Zhang et al., 2022).

Tasks

The paper defined three benchmarks and ran one baseline on all of them: improved trajectory descriptors (HOG, HOF, MBH), SIFT and AlexNet fc6–fc8 activations, each encoded with Fisher vectors or averaged over frames, with one-vs-all linear SVMs.

• Untrimmed video classification of whole videos: all features combined gave 42.2% mAP on the test set (Table 1).
• Trimmed activity classification of single instances: 50.2% test mAP with all features (Table 1), or 45.9% accuracy (Table 4).
• Activity detection of each instance’s start, end and class, with sliding windows of 7 lengths from 25 to 250 frames and non-maximum suppression. The combined features reached 12.5% mAP at a temporal IoU threshold of 0.1 and 9.7% at 0.5 (Table 2); the paper reported several thresholds because activity boundaries are ambiguous.

By top-level category, trimmed classification ranged from 33.9% test mAP for household activities to 66.1% for sports and exercises (Table 3), which the authors attributed to the defined temporal ordering of sports.

Evaluation

Localization on release 1.3 is scored by mAP averaged over temporal IoU thresholds 0.5:0.05:0.95, and proposals by average recall; temporal action localization defines both and describes the official code. Untrimmed classification in the 2016 challenge was scored by mAP and top-3 accuracy (2016 guidelines).

Video-level classifiers in reported numbers. Because most release 1.3 videos contain a single class (Xu et al., 2017), many localizers produce class-agnostic segments and take the class from a separate video-level classifier. ActionFormer, for example, gives every detected segment the top two classes of external video-level scores (from a 2017 challenge entry), multiplying the scores, so each detection becomes two (Zhang et al., 2022, supplementary material). The example below shows when fusion helps.

It builds 60 untrimmed $T \times H \times W = 300 \times 20 \times 20$ clips per test set, each with one dominant class: a square moves 2 or 3 pixels per frame in one of four directions (8 classes) during 1 to 3 segments and stands still in between, in Gaussian noise. A motion detector correlates each frame with its predecessor shifted by every candidate motion; its actionness gives class-agnostic proposals. Each proposal is labeled either by the detector’s mean class probabilities inside it (per-segment classification) or by a simulated video-level classifier whose top-1 label is right with probability $a$.

import numpy as np
from scipy.ndimage import uniform_filter1d

S, T, V = 20, 300, 60                      # frame size, frames per video, videos per test set
SHIFTS = [(0, 0)] + [(dy * v, dx * v) for v in (2, 3) for dy, dx in ((0, -1), (0, 1), (-1, 0), (1, 0))]
C = len(SHIFTS) - 1                        # 8 classes: left/right/up/down at 2 or 3 pixels per frame

def make_video(rng, noise):
    """An untrimmed clip of a 3x3 bright square in noise. During an action of class c the square
    moves by SHIFTS[c + 1] per frame; in the background it stays still. One dominant class per
    video, in 1-3 segments of 15-90 frames."""
    c = int(rng.integers(C))
    gt, t = [], int(rng.integers(5, 60))
    for _ in range(int(rng.integers(1, 4))):
        d = int(rng.integers(15, 91))
        if t + d > T - 5:
            break
        gt.append((t, t + d, c))
        t += d + int(rng.integers(10, 60))
    step = np.zeros((T, 2), int)
    for s, e, k in gt:
        step[s:e] = SHIFTS[k + 1]
    pos = (rng.integers(0, S, 2) + np.cumsum(step, axis=0)) % S
    frames = rng.normal(0, noise, (T, S, S))
    for f, (y, x) in zip(frames, pos):
        f[np.ix_((y + np.arange(-1, 2)) % S, (x + np.arange(-1, 2)) % S)] += 1.0
    return np.array(gt, float), c, frames

def motion_scores(frames):
    """Per frame pair: correlation of frame t+1 with frame t shifted by each candidate motion.
    Returns actionness (best moving shift minus the still shift) and class probabilities."""
    corr = np.stack([(np.roll(frames[:-1], sh, axis=(1, 2)) * frames[1:]).sum(axis=(1, 2))
                     for sh in SHIFTS], axis=1)
    corr = uniform_filter1d(corr, 9, axis=0)                       # smooth over 9 frame pairs
    act = 1 / (1 + np.exp(-(corr[:, 1:].max(1) - corr[:, 0]) / 3))
    z = corr[:, 1:] / 3
    p = np.exp(z - z.max(1, keepdims=True))
    return act, p / p.sum(1, keepdims=True)

def tiou(seg, segs):
    inter = np.clip(np.minimum(seg[1], segs[:, 1]) - np.maximum(seg[0], segs[:, 0]), 0, None)
    return inter / ((seg[1] - seg[0]) + (segs[:, 1] - segs[:, 0]) - inter)

def proposals(act, ths=(0.3, 0.5, 0.7), nms_thr=0.5):
    """Class-agnostic proposals: runs of frames above each actionness threshold, scored by mean
    actionness, then 1D NMS."""
    segs = []
    for th in ths:
        d = np.diff(np.r_[0, (act > th).astype(int), 0])
        segs += [(s, e) for s, e in zip(np.flatnonzero(d == 1), np.flatnonzero(d == -1)) if e - s >= 5]
    if not segs:
        return np.zeros((0, 2)), np.zeros(0)
    segs = np.array(segs, float)
    sc = np.array([act[int(s):int(e)].mean() for s, e in segs])
    order, keep = np.argsort(-sc), []
    while order.size:
        i = order[0]
        keep.append(i)
        order = order[1:][tiou(segs[i], segs[order[1:]]) <= nms_thr]
    return segs[keep], sc[keep]

def average_precision(dets, gts, c, thr):
    """AP of class c at tIoU thr over all videos: greedy matching by score to the unclaimed
    ground truth with the highest tIoU, then the area under the precision envelope."""
    D = sorted(((q, v, s, e) for v, dv in enumerate(dets) for s, e, k, q in dv if k == c), reverse=True)
    G = [g[g[:, 2] == c, :2] for g in gts]
    npos = sum(len(g) for g in G)
    if npos == 0:
        return np.nan
    used = [np.zeros(len(g), bool) for g in G]
    tp = np.zeros(len(D))
    for i, (q, v, s, e) in enumerate(D):
        if len(G[v]):
            o = np.where(used[v], -1.0, tiou((s, e), G[v]))
            j = int(np.argmax(o))
            if o[j] >= thr:
                tp[i], used[v][j] = 1, True
    rec = np.r_[0, np.cumsum(tp) / npos, 1]
    prec = np.maximum.accumulate(np.r_[0, np.cumsum(tp) / np.arange(1, len(D) + 1), 0][::-1])[::-1]
    i = np.flatnonzero(rec[1:] != rec[:-1]) + 1
    return float(np.sum((rec[i] - rec[i - 1]) * prec[i]))

def avg_map(dets, gts):
    """ActivityNet's average mAP: mAP at tIoU 0.5, 0.55, ..., 0.95, averaged."""
    return np.mean([np.nanmean([average_precision(dets, gts, c, t) for c in range(C)])
                    for t in np.linspace(0.5, 0.95, 10)])

def run(seed, noise, accs):
    rng = np.random.default_rng(seed)
    vids = [make_video(rng, noise) for _ in range(V)]
    gts = [g for g, _, _ in vids]
    per_seg, fused, seg_ok, n_seg = [], {a: [] for a in accs}, 0, 0
    u = rng.uniform(size=V)                                   # one draw per video, shared by all accuracies
    wrong = (np.array([c for _, c, _ in vids]) + rng.integers(1, C, V)) % C
    for v, (g, c, frames) in enumerate(vids):
        act, p = motion_scores(frames)
        segs, sc = proposals(act)
        # (a) per-segment classification: mean class probabilities inside each proposal
        lab = [int(np.argmax(p[int(s):int(e)].mean(0))) for s, e in segs]
        per_seg.append([(s, e, k, q) for (s, e), k, q in zip(segs, lab, sc)])
        for (s, e), k in zip(segs, lab):                      # label accuracy of well-placed proposals
            if len(g) and tiou((s, e), g[:, :2]).max() >= 0.5:
                seg_ok, n_seg = seg_ok + (k == c), n_seg + 1
        # (b) fusion: every proposal takes the video-level classifier's top-1 label, which is
        #     right with probability a (simulated: the classifier itself is not modeled)
        for a in accs:
            k = c if u[v] < a else int(wrong[v])
            fused[a].append([(s, e, k, q) for (s, e), q in zip(segs, sc)])
    return avg_map(per_seg, gts), {a: avg_map(fused[a], gts) for a in accs}, seg_ok / n_seg

accs = (0.6, 0.7, 0.8, 0.9, 1.0)
rng = np.random.default_rng(0)
g, c, frames = make_video(rng, 0.8)
act, p = motion_scores(frames)
print(f"video 0: class {c} (shift {SHIFTS[c + 1]}), segments {g[:, :2].astype(int).tolist()}, "
      f"mean actionness inside {act[[t for s, e, _ in g for t in range(int(s), int(e) - 1)]].mean():.2f}, "
      f"outside {np.delete(act, [t for s, e, _ in g for t in range(int(s), int(e) - 1)]).mean():.2f}")
chk = [make_video(rng, 0.8)[0] for _ in range(20)]
print("check, ground truth as detections: avg mAP", avg_map([[(s, e, k, 1.0) for s, e, k in g] for g in chk], chk))
for noise in (0.6, 0.8, 1.0):
    rows = [run(seed, noise, accs) for seed in range(5)]
    ps = np.array([r[0] for r in rows])
    print(f"\nnoise {noise}: per-segment labels correct (proposals at tIoU >= 0.5) "
          f"{min(r[2] for r in rows):.2f}-{max(r[2] for r in rows):.2f}; avg mAP, mean (range) over seeds 0-4")
    print(f"  per-segment classification      avg mAP {ps.mean():.3f} ({ps.min():.3f}-{ps.max():.3f})")
    for a in accs:
        f = np.array([r[1][a] for r in rows])
        print(f"  fusion, video classifier {a:4.0%}   avg mAP {f.mean():.3f} ({f.min():.3f}-{f.max():.3f})")

video 0: class 6 (shift (-3, 0)), segments [[40, 75], [100, 118]], mean actionness inside 0.94, outside 0.43
check, ground truth as detections: avg mAP 1.0

noise 0.6: per-segment labels correct (proposals at tIoU >= 0.5) 0.91-0.97; avg mAP, mean (range) over seeds 0-4
  per-segment classification      avg mAP 0.411 (0.338-0.442)
  fusion, video classifier  60%   avg mAP 0.213 (0.161-0.263)
  fusion, video classifier  70%   avg mAP 0.257 (0.236-0.283)
  fusion, video classifier  80%   avg mAP 0.316 (0.285-0.347)
  fusion, video classifier  90%   avg mAP 0.356 (0.321-0.406)
  fusion, video classifier 100%   avg mAP 0.443 (0.402-0.468)

noise 0.8: per-segment labels correct (proposals at tIoU >= 0.5) 0.75-0.88; avg mAP, mean (range) over seeds 0-4
  per-segment classification      avg mAP 0.169 (0.143-0.205)
  fusion, video classifier  60%   avg mAP 0.099 (0.064-0.137)
  fusion, video classifier  70%   avg mAP 0.124 (0.100-0.160)
  fusion, video classifier  80%   avg mAP 0.156 (0.139-0.201)
  fusion, video classifier  90%   avg mAP 0.173 (0.153-0.227)
  fusion, video classifier 100%   avg mAP 0.203 (0.185-0.243)

noise 1.0: per-segment labels correct (proposals at tIoU >= 0.5) 0.61-0.68; avg mAP, mean (range) over seeds 0-4
  per-segment classification      avg mAP 0.054 (0.037-0.071)
  fusion, video classifier  60%   avg mAP 0.043 (0.022-0.082)
  fusion, video classifier  70%   avg mAP 0.051 (0.032-0.088)
  fusion, video classifier  80%   avg mAP 0.063 (0.048-0.104)
  fusion, video classifier  90%   avg mAP 0.071 (0.056-0.118)
  fusion, video classifier 100%   avg mAP 0.094 (0.077-0.128)

With a perfect video-level label, fusion beats per-segment classification at every noise level (0.443 against 0.411 at noise 0.6, 0.094 against 0.054 at noise 1.0, means over seeds 0–4). Otherwise the break-even accuracy depends on how well segments can be labeled alone: when per-segment labels are 91–97% right, even a 90% video classifier loses (0.356 against 0.411); at 75–88% the two cross near 90%; at 61–68% a 70–80% classifier matches or wins. In our reading, a wrong video-level label costs every segment of its video at once, while per-segment errors are spread out. The simulation assumes one class per video, hiding the cost of fusion on videos with several classes. It runs in about 5 seconds and gave identical output twice.

License

As of October 2026, the home, download and about pages state no licence for the annotations or the videos, which are third-party YouTube uploads. The MIT licence of the ActivityNet GitHub repository covers its code.

Availability and Dataset Decay

ActivityNet is distributed as annotation files with YouTube IDs, and the official crawler downloads the videos with youtube-dl (Crawler README, commit 82304fa), so videos vanish when uploaders remove them. On 20 March 2018 a maintainer wrote in the repository’s issue 18 that about 1,000 videos were unavailable on YouTube and about 19,000 still online, and named country blocking as another cause of failed downloads. As of October 2026 the download page offers a full copy of the videos on Google and Baidu drives through a request form, with 7-day access, and the evaluation code skips a server-side list of blocked videos.

Limitations and Bias

• One dominant class per video. Release 1.3 averages about 1.5 instances per video, and most videos contain instances of a single class (Xu et al., 2017), so localization results partly measure video classification, as the example shows.
• Background and coverage. The paper reports that 68.8 of its 849 hours contain activities, yet R-C3D’s authors describe release 1.3 instances as covering a great deal of each video, and ActivityNet Captions paragraphs cover 94.6% of their videos on average (Krishna et al., 2017). The figures come from different releases and annotations. The paper’s own figures also disagree: by our arithmetic, 849 hours over 27,801 videos is under two minutes per video, while the text says a large proportion of videos last 5 to 10 minutes.
• Ambiguous boundaries. The paper reported several IoU thresholds because temporal boundaries are ambiguous; the evidence on annotator agreement in other datasets is summarized in temporal action localization.
• Static cues. ActivityNet is among the datasets with strong static bias summarized in video understanding.

Common Uses

• Temporal localization and proposals, trained on release 1.3 training and evaluated on validation. Many methods use precomputed features from TSN-style two-stream networks or two-stream I3D. ActionFormer extracted I3D features at a 16-frame stride and interpolated each video’s sequence to a fixed length, 160 in the main text and 128 in the supplementary material (Zhang et al., 2022); its comparison table spans TSN, I3D, P3D, SlowFast and R(2+1)D features, so rows rarely share inputs.
• Untrimmed classification, as in TSN’s ablations on release 1.2.

Historical Milestones

• 2016. The first ActivityNet challenge, at CVPR 2016, used release 1.3 for untrimmed classification and detection; the CUHK team won classification, and the challenge page links its released code to the TSN repository.
• 2017. The challenge added temporal action proposals, dense captioning on ActivityNet Captions, and trimmed recognition on Kinetics (2017 guidelines).
• 2018–2022. Guest tasks on Kinetics-600, AVA and Moments in Time (2018 summary), then EPIC-Kitchens and ActEV (2019), ActivityNet Entities and HACS (2020), and twelve challenges in 2021, including Kinetics-700-2020 and AVA-Kinetics. As of October 2026 the site lists challenges up to 2022.

Related Datasets

ActivityNet Captions (Krishna et al., 2017) describes 20k ActivityNet videos with 100k temporally localized sentences, as printed: 3.65 per video and 13.48 words per sentence on average, each sentence covering on average 36 seconds and 31% of its video, with 10% of the descriptions overlapping in time. Two annotators described each of the 4,926 validation and 5,044 test videos; on average each sentence overlapped the best-matching combination of the other annotator’s sentences at 70.2% temporal IoU. ActivityNet Entities (Zhou et al., 2019) adds 158k bounding boxes, each grounding a noun phrase of ActivityNet Captions, over about 10k, 2.5k and 2.5k videos for training, validation and testing (README, as of October 2026). Kinetics has trimmed 10-second clips instead of untrimmed videos; THUMOS14 is the other standard localization benchmark.

Resources

• ActivityNet download page: annotation files of releases 1.2 and 1.3, the format, and the video request form (as of October 2026).
• ActivityNet challenge pages: tasks and results of the 2016–2022 challenges.
• activitynet/ActivityNet: crawler and evaluation code.
• ActivityNet Entities: annotations and challenge code.