Temporal Action Localization

Temporal action localization (TAL), also called temporal action detection, finds every action instance in an untrimmed video and returns its start time, end time and class. Unlike action recognition on trimmed clips, it must find where each action begins and ends, in videos that are mostly background. It is the counterpart of object detection along the time axis; video understanding gives the overview of video tasks.

Problem Definition

A video of $T$ snippets (short groups of frames) is annotated with instances $(t_s^i, t_e^i, c^i)$, each with a start, an end and one of $C$ classes, their number varying between videos (Zhang et al., 2022). A method outputs scored segments $(\hat t_s, \hat t_e, \hat c, q)$, compared with the ground truth by the temporal intersection over union (tIoU), the 1D Jaccard index:

$$\operatorname{tIoU}(a, b) = \frac{\max\big(0, \min(a_e, b_e) - \max(a_s, b_s)\big)}{(a_e - a_s) + (b_e - b_s) - \max\big(0, \min(a_e, b_e) - \max(a_s, b_s)\big)}.$$

The detection analogy. With intervals in place of boxes, the machinery of detection carries over: proposals, anchors, boundary regression, non-maximum suppression and average precision at an overlap threshold. S-CNN was inspired by R-CNN (Shou et al., 2016), and BSN splits the task, as in detection, into proposals and classification (Lin et al., 2018).

Where the analogy breaks.

• Boundaries are ambiguous. An object’s extent is visible; an action’s start and end are a judgment, “generally vague and subjective” in the THUMOS organizers’ words (Idrees et al., 2017, read in arXiv v1 and the authors’ HAL preprint of January 2017). Crowd workers re-annotating actions agreed with the original segments at 72.5% IoU on average on Charades and 58.7% on MultiTHUMOS; on Charades, ends varied more than starts (Sigurdsson et al., 2017).
• Durations span two orders of magnitude. 99.5% of THUMOS14 validation instances last under 30 seconds, while ActivityNet instances can fill most of a long video; R-C3D’s anchors spanned 0.64–5.12 s on the first and 2.7–170 s on the second (Xu et al., 2017).
• Density varies. ActivityNet-1.3 averages 1.5 instances per video and EPIC-Kitchens 100 averages 128 (Zhang et al., 2022); Charades activities overlap, some with identical boundaries (Xu et al., 2017).

Input and Output

Precomputed features. Most methods do not read pixels: a pretrained recognition network runs over the video once, giving a $T \times D$ feature sequence. On THUMOS14, ActionFormer fed 16-frame windows at stride 4 to a Kinetics-pretrained two-stream I3D and concatenated the streams’ 1024-dimensional features (Zhang et al., 2022, supplementary material). BSN and BMN used a two-stream network in the style of temporal segment networks (ResNet on RGB, BN-Inception on optical flow) pretrained on ActivityNet-1.3, one snippet every 5 frames on THUMOS14 and every 16 on ActivityNet, where each sequence was rescaled to 100 positions by linear interpolation (Lin et al., 2018; Lin et al., 2019).

End-to-end training. R-C3D trained C3D from frames with its detection heads, on 768-frame buffers at 25 fps on THUMOS14 and 3 fps on ActivityNet (Xu et al., 2017).

Approaches

Sliding windows and proposals plus classification

S-CNN (Shou et al., 2016) slid windows of 16 to 512 frames with 75% overlap and passed each, sampled to 16 frames, through three 3D CNNs: a proposal network (action or background), a classifier, and a localization network initialized from it. A classifier can score a window highly even when it barely overlaps the action, so that NMS keeps the wrong window; the localization loss therefore ties the confidence to the window’s tIoU with the ground truth. Boundary regression did not work well for the authors. BSN’s authors later argued that windows of predefined durations are imprecise and cover widely varying durations poorly (Lin et al., 2018).

Boundary-based proposals

BSN (Lin et al., 2018) predicts, for every snippet, the probabilities that an action starts, ends or is in progress. Snippets whose start or end probability exceeds 0.9 or peaks locally become boundaries, start–end pairs within the dataset’s range of durations become proposals, and a small network scores each from the “actionness” inside and around it. BMN (Lin et al., 2019) scores all proposals at once: a boundary-matching layer samples features for every pair of start and duration, giving a 2D confidence map with one entry per proposal. Both proposal generators are class-agnostic.

Anchor-based detectors

R-C3D (Xu et al., 2017) is a temporal Faster R-CNN. At each position of the shared C3D feature map it places $K$ multi-scale anchor segments ($K = 10$ on THUMOS14, 20 on ActivityNet) and predicts an actionness score and center and length offsets for each; proposals surviving NMS at 0.7 are pooled to fixed size by 3D RoI pooling, classified and refined.

Graph-based context

G-TAD (Xu et al., 2020) makes snippets the nodes of a graph, with temporal edges between neighbors and semantic edges to each node’s nearest neighbors in feature space, recomputed at every layer, so that a snippet draws context from similar, distant snippets. Graph convolutions update the features before candidate start–end pairs are scored.

One-stage anchor-free transformers

ActionFormer (Zhang et al., 2022) treats every position of a temporal feature pyramid as a candidate. A transformer encoder with local self-attention (a 19-snippet window on THUMOS14) and downsampling builds the pyramid, and 1D convolutional heads predict at each position the class probabilities and the distances to the action’s start and end, as anchor-free image detectors do. TriDet (Shi et al., 2023) predicts each boundary as a probability distribution over nearby positions rather than one distance, motivated by ambiguous boundaries, and replaces self-attention with convolutional and pooling branches.

Query-based detectors

TadTR (Liu et al., 2022) adapts DETR: a decoder turns a few learned action queries into segments, matched one-to-one to the ground truth with the Hungarian algorithm in training, so NMS is unnecessary within a window. On THUMOS14, processed in overlapping windows, it still merges windows’ detections with NMS.

Post-Processing

NMS and Soft-NMS in 1D. NMS and Soft-NMS carry over with tIoU in place of box IoU. S-CNN and R-C3D applied greedy NMS at 0.1 below the evaluation threshold. BSN used Gaussian Soft-NMS with threshold 0.8 on ActivityNet-1.3 and 0.65 on THUMOS14; G-TAD, ActionFormer and TriDet also use Soft-NMS.

Fusion with video-level classifiers. Most ActivityNet videos contain one class (Xu et al., 2017), so many methods take the class of class-agnostic segments from a video-level classifier. BSN gave every proposal the top-1 class of a challenge entry’s video classifier on ActivityNet, and on THUMOS14 multiplied proposal scores with UntrimmedNet’s top two video-level class scores (Lin et al., 2018); G-TAD, ActionFormer and TadTR also use external classifiers. The classifier moves the numbers: TadTR’s average mAP on ActivityNet-1.3 rose from 29.90 as a self-contained detector to 34.64 with an ensemble of action classifiers, on the same two-stream features (Liu et al., 2022, Table III, read in arXiv v4).

Datasets and Benchmarks

• THUMOS14. The temporal detection task of the THUMOS 2014 challenge covers 20 classes, with trimmed clips for training and untrimmed videos for testing (THUMOS 2014 page). The organizers chose classes with short, well-defined boundaries and marked unclear instances as ambiguous (Idrees et al., 2017). Papers train on the 200 annotated validation videos and test on the 213 test videos that contain the actions (Xu et al., 2017; Zhang et al., 2022), of 1,574 in the test set (Shou et al., 2016).
• ActivityNet v1.3. 200 activity classes in 10,024 training, 4,926 validation and 5,044 test videos, with test labels withheld for the challenge server (Xu et al., 2017), so results are usually reported on validation. The dataset (Heilbron et al., 2015) has its own article, ActivityNet.
• Others. Charades is scored by multi-label mAP on 25 equally spaced frames per video (as R-C3D describes); EPIC-Kitchens 100 localizes verbs and nouns in kitchen video, with mAP averaged over tIoU 0.1–0.5 (Zhang et al., 2022); Ego4D’s Moments Queries seek every window of 110 activities, scored by mAP over tIoU thresholds and recall (Grauman et al., 2022).

Evaluation Metrics

Detection: mAP at tIoU thresholds. As in average precision for boxes, each class’s detections are ranked over all videos and each instance can be matched once (Idrees et al., 2017). ActivityNet’s code matches each detection to the unclaimed instance with the highest tIoU and integrates the interpolated precision–recall curve, the VOC 2010–2012 convention; it also skips videos on a server-side list of blocked videos (ActivityNet evaluation code). Thresholds differ by benchmark:

Benchmark	Thresholds as printed	Summary
THUMOS 2014–2015 challenges	0.1, 0.2, 0.3, 0.4, 0.5	mAP at each (Idrees et al., 2017)
THUMOS14 in recent papers	0.3, 0.4, 0.5, 0.6, 0.7	mAP at each; ActionFormer and TriDet also average them
ActivityNet v1.3	0.5, 0.75, 0.95; 0.5:0.05:0.95	average mAP over the 10 thresholds (BSN; code default)

R-C3D’s authors date this average to the 2017 ActivityNet challenge. An “average mAP” on THUMOS14 (over 0.3–0.7) and one on ActivityNet (over 0.5–0.95) are different quantities.

Proposals: AR@AN and AUC. Class-agnostic proposals are scored by average recall (AR), recall averaged over tIoU thresholds (0.5:0.05:0.95 on ActivityNet-1.3, 0.5:0.05:1.0 on THUMOS14 in BSN and BMN), as a function of the average number of proposals per video (AN); AR@100 uses 100 per video. On ActivityNet, the area under the AR–AN curve for AN from 0 to 100 (AUC) summarizes it (Lin et al., 2018; ActivityNet evaluation code).

Representative Results

Printed results on THUMOS14 (test set, mAP at tIoU) and ActivityNet-1.3 (validation set, average mAP over 0.5:0.05:0.95). Rows differ in features and fusion; none is a current best result.

Method (year)	Features as printed	THUMOS14 0.3 / 0.5 / 0.7	ActivityNet-1.3 avg mAP
S-CNN (2016)	C3D-based networks	36.3 / 19.0 / – (Table 2)	–
R-C3D (2017)	C3D, end to end	44.8 / 28.9 / – (Table 1)	12.7 (text, trained on train only)
BSN + classifier (2018)	two-stream	53.5 / 36.9 / 20.0 (Table 6, UntrimmedNet)	33.72 (Table 5, UTS classifier)
BMN + classifier (2019)	two-stream	56.0 / 38.8 / 20.5 (Table 7, UntrimmedNet)	33.85 (Table 6, CUHK 2017 classifier)
G-TAD (2020)	TSN (THUMOS14); two-stream (ActivityNet)	54.5 / 40.2 / 23.4 (Table 2)	34.09 (Table 1)
ActionFormer (2022)	two-stream I3D	82.1 / 71.0 / 43.9 (Table 1)	35.6 (Table 1, I3D, external classifier)
TriDet (2023)	I3D (THUMOS14); TSP R(2+1)D (ActivityNet)	83.6 / 72.9 / 47.4 (Table 1)	36.8 (Table 4)

Comparing numbers. BSN’s own table gives 33.72 average mAP on ActivityNet validation with the UTS classifier; BMN’s lists BSN at 30.03 with the CUHK 2017 classifier. G-TAD’s 40.2 at 0.5 appears as 40.3 in ActionFormer’s and TriDet’s tables. TadTR’s mAP at 0.5 on THUMOS14 is 46.6 in arXiv v1 and 60.1 in the journal version (read in arXiv v4; with focal loss and integrity-based instance filtering); ActionFormer’s table lists 49.2. Features matter: with TSP-pretrained R(2+1)D features instead of I3D, ActionFormer fell from 71.0 to 59.1 at 0.5 on THUMOS14 but rose from 35.6 to 36.6 on ActivityNet (Zhang et al., 2022, Table 1). S-CNN’s authors re-scored all THUMOS 2014 entries with the organizers’ corrected toolkit.

Example: Localizers and mAP on Simulated Snippet Scores

The example simulates a snippet classifier’s output on 20 untrimmed videos of 600 snippets. Actions of three classes last 6 to 150 snippets; inside each, the true class scores high, with boundaries blurred and jittered in proportion to the duration, plus temporally correlated noise. The localizers threshold and group snippets, or slide windows of 8 to 128 snippets scored by their mean or by the mean minus that of a flanking context, each followed by 1D NMS; mAP follows ActivityNet’s matching and interpolation.

import numpy as np
from scipy.ndimage import gaussian_filter1d

C, T, V = 3, 600, 20                       # classes, snippets per video, videos per test set

def tiou(seg, segs):
    """Temporal IoU of one segment [s, e) with an array of segments, shape (n, 2)."""
    inter = np.clip(np.minimum(seg[1], segs[:, 1]) - np.maximum(seg[0], segs[:, 0]), 0, None)
    return inter / ((seg[1] - seg[0]) + (segs[:, 1] - segs[:, 0]) - inter)

def make_video(rng, noise, jitter):
    """Ground-truth segments (start, end, class) and the per-snippet class scores of a simulated
    snippet classifier: high inside each action, with blurred and jittered boundaries and
    temporally correlated noise."""
    gt, t = [], int(rng.integers(5, 40))
    while True:
        d = int(np.exp(rng.uniform(np.log(6), np.log(150))))       # long-tailed durations: 6-150 snippets
        if t + d >= T:
            break
        gt.append((t, t + d, int(rng.integers(C))))
        t += d + int(rng.integers(10, 80))                         # background gap before the next action
    act = np.zeros((T, C))
    for s, e, c in gt:
        js, je = np.round(rng.normal(0, jitter * (e - s), 2)).astype(int)
        act[max(s + js, 0):min(e + je, T), c] = 1.0                # where the classifier "sees" the action
    z = -2.5 + 5.0 * gaussian_filter1d(act, 1.5, axis=0)
    n = gaussian_filter1d(rng.normal(0, 1, (T, C)), 2.0, axis=0)    # noise correlated over ~2 snippets
    z += noise * n / n.std()
    return np.array(gt, float).reshape(-1, 3), 1 / (1 + np.exp(-z))

def nms_1d(segs, scores, thr):
    """Greedy NMS on segments of one class: keep the best, drop those with tIoU > thr, repeat."""
    order, keep = np.argsort(-scores), []
    while order.size:
        i = order[0]
        keep.append(i)
        order = order[1:][tiou(segs[i], segs[order[1:]]) <= thr]
    return np.array(keep, int)

def runs(mask):
    """Start and end (exclusive) of each run of True values."""
    d = np.diff(np.r_[0, mask.astype(int), 0])
    return np.flatnonzero(d == 1), np.flatnonzero(d == -1)

def threshold_localizer(p, thresholds=(0.3, 0.5, 0.7), nms_thr=0.4):
    """Group consecutive snippets above each threshold into segments scored by their mean score."""
    dets = []
    for c in range(C):
        segs = np.array([(s, e) for th in thresholds for s, e in zip(*runs(p[:, c] > th)) if e - s >= 2], float)
        if len(segs) == 0:
            continue
        sc = np.array([p[int(s):int(e), c].mean() for s, e in segs])
        k = nms_1d(segs, sc, nms_thr)
        dets += [(s, e, c, q) for (s, e), q in zip(segs[k], sc[k])]
    return dets

def window_localizer(p, scales=(8, 16, 32, 64, 128), contrast=True, nms_thr=0.4):
    """Multi-scale sliding windows with 75% overlap, scored by the mean score inside, minus
    (contrast=True) the mean score in a context of half the window length on each side."""
    cs = np.r_[np.zeros((1, C)), np.cumsum(p, axis=0)]
    mean = lambda a, b: (cs[b] - cs[a]) / np.maximum(b - a, 1)[:, None]
    dets = []
    for L in scales:
        s = np.arange(0, T - L + 1, L // 4); e = s + L
        score = mean(s, e)
        if contrast:
            a, b = np.maximum(s - L // 2, 0), np.minimum(e + L // 2, T)
            ctx = (cs[b] - cs[e] + cs[s] - cs[a]) / np.maximum((b - e) + (s - a), 1)[:, None]
            score = score - ctx
        dets += [(si, ei, c, score[i, c]) for i, (si, ei) in enumerate(zip(s, e)) for c in range(C)]
    dets = np.array(dets, float)
    out = []
    for c in range(C):
        d = dets[dets[:, 2] == c]
        out += [tuple(r) for r in d[nms_1d(d[:, :2], d[:, 3], nms_thr)][:100]]
    return out

def average_precision(dets, gts, c, thr):
    """AP of class c at tIoU thr, over all videos. Detections are ranked by score; each matches the
    unclaimed ground truth of its video and class with the highest tIoU >= thr. The precision
    envelope is integrated over recall (all-point interpolated AP, as in ActivityNet's code)."""
    D = sorted(((q, v, s, e) for v, dv in enumerate(dets) for s, e, cc, q in dv if cc == c), reverse=True)
    G = [g[g[:, 2] == c, :2] for g in gts]
    npos = sum(len(g) for g in G)
    used = [np.zeros(len(g), bool) for g in G]
    tp = np.zeros(len(D))
    for i, (q, v, s, e) in enumerate(D):
        if len(G[v]) == 0:
            continue
        o = np.where(used[v], -1.0, tiou((s, e), G[v]))
        j = int(np.argmax(o))
        if o[j] >= thr:
            tp[i], used[v][j] = 1, True
    rec = np.cumsum(tp) / npos
    prec = np.cumsum(tp) / np.arange(1, len(D) + 1)
    mrec, mprec = np.r_[0, rec, 1], np.maximum.accumulate(np.r_[0, prec, 0][::-1])[::-1]
    i = np.flatnonzero(mrec[1:] != mrec[:-1]) + 1
    return float(np.sum((mrec[i] - mrec[i - 1]) * mprec[i]))

def mAP(dets, gts, thr):
    return np.mean([average_precision(dets, gts, c, thr) for c in range(C)])

AVG = np.linspace(0.5, 0.95, 10)           # ActivityNet: tIoU 0.5:0.05:0.95
def report(dets, gts):
    return [mAP(dets, gts, t) for t in (0.3, 0.5, 0.7)] + [np.mean([mAP(dets, gts, t) for t in AVG])]

# --- one test set: ground truth, a perfect detector, and the three localizers
rng = np.random.default_rng(4)                    # seed 4; the seed loop below covers seeds 0-4
data = [make_video(rng, noise=1.0, jitter=0.05) for _ in range(V)]
gts = [g for g, _ in data]
dur = np.concatenate([g[:, 1] - g[:, 0] for g in gts])
print(f"{V} videos x {T} snippets, {len(dur)} actions, durations {dur.min():.0f}-{dur.max():.0f} snippets "
      f"(median {np.median(dur):.0f}), {np.mean([sum(g[:, 1] - g[:, 0]) for g in gts]) / T:.0%} of snippets inside actions")
print("tIoU of [10, 30) with [20, 40), [10, 30), [0, 100):", tiou((10, 30), np.array([[20, 40], [10, 30], [0, 100]])).round(3))
print(f"{'':28s} mAP@0.3  mAP@0.5  mAP@0.7  avg mAP 0.5:0.05:0.95")
loc = {"ground truth as detections": lambda p, g: [(s, e, c, 1.0) for s, e, c in g],
       "threshold + grouping": lambda p, g: threshold_localizer(p),
       "sliding windows, mean": lambda p, g: window_localizer(p, contrast=False),
       "sliding windows, contrast": lambda p, g: window_localizer(p)}
for name, f in loc.items():
    print(f"{name:28s}" + "".join(f"{m:9.3f}" for m in report([f(p, g) for g, p in data], gts)))
p0 = data[0][1]
raw = sum(int(np.sum(e - s >= 2)) for c in range(C) for th in (0.3, 0.5, 0.7) for s, e in [runs(p0[:, c] > th)])
print(f"threshold + grouping, video 0: {raw} segments -> {len(threshold_localizer(p0))} after NMS, "
      f"{len(gts[0])} ground-truth actions")
avg = {name: [] for name in list(loc)[1:]}
for seed in range(5):
    r = np.random.default_rng(seed)
    d = [make_video(r, noise=1.0, jitter=0.05) for _ in range(V)]
    for name in avg:
        avg[name].append(report([loc[name](p, g) for g, p in d], [g for g, _ in d])[3])
print("avg mAP over seeds 0-4: " + ", ".join(f"{n} {min(a):.3f}-{max(a):.3f}" for n, a in avg.items()))

# --- noise and boundary jitter over seeds 0-4
print(f"\n{'noise':>5s} {'jitter':>6s}   threshold + grouping: mAP@0.3 / mAP@0.7 / avg (mean over seeds 0-4, range)")
for noise in (0.5, 1.5):
    for jitter in (0.0, 0.15):
        res = []
        for seed in range(5):
            r = np.random.default_rng(seed)
            data = [make_video(r, noise, jitter) for _ in range(V)]
            res.append(report([threshold_localizer(p) for _, p in data], [g for g, _ in data]))
        res = np.array(res)[:, [0, 2, 3]]
        print(f"{noise:5.1f} {jitter:6.2f}   " + "   ".join(
            f"{m:.3f} ({lo:.3f}-{hi:.3f})" for m, lo, hi in zip(res.mean(0), res.min(0), res.max(0))))

20 videos x 600 snippets, 134 actions, durations 6-144 snippets (median 28), 47% of snippets inside actions
tIoU of [10, 30) with [20, 40), [10, 30), [0, 100): [0.333 1.    0.2  ]
                             mAP@0.3  mAP@0.5  mAP@0.7  avg mAP 0.5:0.05:0.95
ground truth as detections      1.000    1.000    1.000    1.000
threshold + grouping            0.786    0.625    0.426    0.381
sliding windows, mean           0.102    0.077    0.022    0.025
sliding windows, contrast       0.977    0.963    0.816    0.588
threshold + grouping, video 0: 37 segments -> 22 after NMS, 7 ground-truth actions
avg mAP over seeds 0-4: threshold + grouping 0.375-0.440, sliding windows, mean 0.023-0.032, sliding windows, contrast 0.575-0.592

noise jitter   threshold + grouping: mAP@0.3 / mAP@0.7 / avg (mean over seeds 0-4, range)
  0.5   0.00   0.997 (0.989-1.000)   0.938 (0.902-0.963)   0.859 (0.819-0.896)
  0.5   0.15   0.992 (0.975-1.000)   0.606 (0.514-0.666)   0.482 (0.468-0.501)
  1.5   0.00   0.625 (0.556-0.715)   0.198 (0.131-0.254)   0.200 (0.165-0.249)
  1.5   0.15   0.564 (0.477-0.639)   0.121 (0.095-0.149)   0.124 (0.106-0.148)

The ground truth scores 1.000, a check of the evaluation. Windows scored by their mean fail (average mAP 0.023–0.032 over seeds 0–4), as S-CNN’s observation predicts: in our reading, each short window inside a long action scores as high as the one that fits, survives NMS because it overlaps it little, and becomes a high-ranked false positive. Subtracting the context lifts the same windows to 0.575–0.592. By our arithmetic, with lengths a factor of two apart, a 24-snippet action is matched at best at tIoU 0.75 (by a 32-snippet window), so windows alone fail the strictest thresholds: their average mAP (0.588 for seed 4) trails their mAP at 0.7 (0.816). In the seed loop, boundary jitter of 15% of the duration barely changes mAP at tIoU 0.3 (0.997 to 0.992 at low noise) but cuts mAP at 0.7 from 0.938 to 0.606; stronger noise lowers both. Two runs gave identical output, each in under 10 seconds.

Challenges and Open Problems

In our assessment, the main open questions as of October 2026 are these.

• Boundary ambiguity. With annotators agreeing at 72.5% IoU on Charades, the strictest thresholds partly measure agreement with one annotator, although ignoring ambiguous boundary regions did not change the order of methods there (Sigurdsson et al., 2017).
• Long-tailed durations. Pyramids, multi-scale anchors and rescaled sequences are compromises for durations from instants to minutes.
• Feature-extractor dependence. Most localizers inherit frozen backbones trained for classification, and end-to-end training on long videos remains costly.
• Online detection. The methods above see the whole video; detecting actions as they happen, from past frames only, is a separate setting.

Applications

Uses include video search and highlights, sports analytics, surveillance review and the indexing of egocentric video, as in Ego4D’s episodic-memory queries.