Spatio-Temporal Action Detection

Spatio-temporal action detection finds the people in a video, boxes each of them over time, and labels what each one is doing. It combines action recognition, which names the action of a whole clip, with object detection: in a scene where one person walks while talking and another sits, a single clip label cannot say who does what. Temporal action localization finds when actions happen; this task also finds where and by whom. The task is also called spatio-temporal action localization or video action detection. Its methods adapt image detectors to video, most often two-stage detectors.

Problem Definition

Keyframe detection. On AVA, every person in keyframes sampled once per second has a box and one or more action labels, and a detector outputs, for each keyframe, boxes with a score for every class (Gu et al., 2018). Labels are not mutually exclusive, so the classifier uses one sigmoid per class rather than a softmax. Temporal context is still needed: annotators saw 1.5 seconds on either side of the keyframe, because actions such as picking up and putting down cannot be told apart in a still frame.

Action tubes. On UCF101-24 and J-HMDB, each action instance has a box in every frame, and a detection is an action tube: boxes linked over consecutive frames, with one class and one score, starting and ending with the action. Gkioxari and Malik (2015) named action tubes and introduced frame-level and video-level average precision for the task.

Challenges

• Long tail. AVA labels everything every person does, so standing and sitting dwarf rare actions; the authors describe the distribution as roughly following Zipf’s law and kept it unbalanced on purpose (Gu et al., 2018).
• Recognition, not localization. With one model, AVA frame-mAP@0.5 was 75.3% for class-agnostic actor detection but 15.8% for action detection (Gu et al., 2018, Table 5).
• Interactions. Labels such as “talk to”, “hold” or “answer phone” depend on objects and people outside the actor’s box.
• Context versus actor. “Swim” scored high despite few training examples because the scene gives it away, while frequent classes such as “touch” and “smoke” scored low (Gu et al., 2018, Figure 8).
• Label noise and ambiguity. Annotators miss correct actions in an 80-class vocabulary; AVA keeps a label only if two of three annotators verify it, which still leaves missing and ambiguous labels.
• Compute. Each evaluated frame needs a person detector and a 3D backbone over a clip.

Approaches

Frame-level detection and linking

Gkioxari and Malik (2015) adapted R-CNN: about 2,000 selective-search regions per frame, filtered by optical-flow magnitude, were classified by an RGB CNN and a flow CNN, as in two-stream networks, feeding SVMs. A link between regions in consecutive frames scored both regions’ class scores plus a weighted IoU; the Viterbi algorithm found the best path through the video, its regions were removed, and the search repeated. Linking could fail: on UCF Sports, Running reached 54.9% frame-AP but 11.7% video-AP because tubes jumped between runners side by side.

Later linkers let tubes start and stop. Weinzaepfel et al. (2015) followed proposals with a tracking-by-detection approach. Singh et al. (2017) grow tubes greedily: tubes sorted by mean score each take the highest-scoring unclaimed detection whose IoU with their last box exceeds $\lambda = 0.1$, end once more than $k = 5$ frames pass without a match, and are trimmed by a two-state Viterbi labelling of each box as action or background. This is multi-object tracking with class scores in the association.

Tubelet detectors

The ACT detector (Kalogeiton et al., 2017) extended the SSD detector with anchor cuboids, anchor boxes repeated over $K$ frames, so that features stacked over the sequence score and regress a whole tubelet ($K = 6$ worked well); overlapping tubelets were linked with a variant of Singh et al.’s algorithm. On UCF101-24 (split 1) it reported 51.4% video-mAP@0.5, against 46.3% for Singh et al. in the same table.

Actor detection plus clip features

The AVA baseline (Gu et al., 2018) set the pattern most AVA methods follow. A Faster R-CNN region proposal network on a 2D ResNet-50 proposes actors on the keyframe; an I3D network computes features over the surrounding clip; and RoI pooling is extended to 3D by pooling the same 2D box at every time step. With RGB and flow streams it reached 15.8% frame-mAP@0.5 on AVA validation, and 76.3% frame-mAP and 59.9% video-mAP on UCF101-24.

Later systems, such as Long-Term Feature Banks and SlowFast, take proposals from a separately trained Faster R-CNN person detector and pool the replicated box with RoIAlign; the SlowFast article describes that detector, and VideoMAE follows its architecture.

Actor–context relations

The Actor-Centric Relation Network (Sun et al., 2018) pairs the actor feature with every cell of the clip’s feature map, computes a relation feature per pair with a $1 \times 1$ convolution, aggregates neighbouring relations with further convolutions and pooling, and classifies the actor from both. It needs only actor labels, no object annotation. On AVA v2.1 it raised frame-mAP@0.5 from 15.5% (its base model) to 17.4%, with the largest gains on interaction classes such as “answer phone” and “dance”.

Long-Term Feature Banks (Wu et al., 2019) relate the actor to the rest of the movie. A 3D CNN fine-tuned on AVA runs once per second over the whole video, and the RoI features of the detected people form a time-indexed bank; stacked modified non-local (self-attention) blocks let each actor’s short-term feature attend to the bank within a window. Lengthening the clip by a larger frame stride lowered frame-mAP@0.5 from 22.1% (2 s) to 15.7% (60 s), whereas a 60 s bank raised it to 25.5% (Table 1a). The bank’s networks are trained first and frozen, since back-propagating through the whole video was infeasible.

End-to-end query-based detectors

TubeR (Zhao et al., 2022) drops the external person detector, arguing that two-stage methods assume the action stays at a fixed location over the clip. Adapting DETR, it learns tubelet queries, one embedding per output frame, that regress a box per frame with action scores; an “action switch” marks frames where the actor is not performing the action, and a context head adds whole-clip features and a memory of neighbouring clips. In its AVA v2.1 ablation, the short-term context raised frame-mAP@0.5 from 23.4% to 27.7%, and long-term context to 28.8% (Table 1d).

Pretrained video transformers

Recent gains came largely from the backbone. VideoMAE fine-tuned its self-supervised video transformers for AVA v2.2: a ViT-B pretrained without labels on Kinetics-400 reached 26.7%, 31.8% after supervised fine-tuning on Kinetics-400, and ViT-H 39.5% (Tong et al., 2022, Table 5). VideoMAE V2 reported 42.6% on AVA (Wang et al., 2023, Table 6e), in a table that names neither the AVA version nor the model size.

Datasets and Benchmarks

AVA (Gu et al., 2018) annotates the 15th to 30th minute of 437 movies at 1 Hz, 900 keyframes each, with 1.59 million labels from 80 atomic actions: 14 pose classes, 49 person–object and 17 person–person interactions. Each box has one pose label and up to three of each interaction type. Boxes came from a high-precision Faster R-CNN person detector plus manual additions (5% more boxes), and people were linked into 81,000 tracklets. The paper split 239/64/134 videos and benchmarked the 60 classes with at least 25 instances in both validation and test.

As of October 2026, the official page offers AVA v2.2: 430 videos split 235/64/131 and 1.62 million labels, evaluated on 60 classes. It describes v2.2 as adding a rating round that inserted missing labels (2.5% more annotations) and fixing boxes in a few videos, and v2.1, current until March 2019, as removing duplicate movies from v2.0; the class list is unchanged since v1.0. Versions are not interchangeable: SlowFast’s 28.2% on v2.1 became 29.0% on v2.2 (Feichtenhofer et al., 2019). AVA-Kinetics v1.0 adds AVA labels on one frame of each of 238k Kinetics-700 clips to the AVA v2.2 videos (official page).

UCF101-24 is the 24-class subset of UCF101 with per-frame boxes, 3,207 videos (Singh et al., 2017, call them untrimmed; Weinzaepfel et al., 2015, note most actions span nearly the whole video), on split 1; J-HMDB has 928 trimmed clips of 21 classes and three splits (Kalogeiton et al., 2017).

Evaluation Metrics

Frame-mAP. Boxes are matched to ground truth in the same frame as in PASCAL VOC average precision: per class, by descending score, at IoU 0.5 or more, each ground-truth box claimed once (Gu et al., 2018). The official AVA code treats every (box, action) pair as a detection and every labelled pair as a ground-truth instance, keeps the 50 highest-scoring pairs per keyframe, computes all-point interpolated AP per class and averages over the 60 classes. Frame-mAP ignores whether boxes form consistent tubes.

Video-mAP matches tubes. Weinzaepfel et al. (2015) define the spatio-temporal IoU of a detected tube $D$ over frames $\theta_D$ and a ground-truth tube $G$ over $\theta_G$ as the temporal IoU times the mean box IoU over shared frames:

$$\mathrm{IoU}_{\text{tube}}(D, G) = \frac{|\theta_D \cap \theta_G|}{|\theta_D \cup \theta_G|} \cdot \frac{1}{|\theta_D \cap \theta_G|} \sum_{t \in \theta_D \cap \theta_G} \mathrm{IoU}(D_t, G_t).$$

Gkioxari and Malik (2015) used the mean per-frame IoU, which for trimmed clips, where tubes span the whole clip, is the same quantity (our reading). Thresholds of 0.2, 0.5 and 0.75 are common, and ACT also averaged over 0.5:0.05:0.95. A tube that switches actors, starts late or breaks in two loses overlap even if every frame is classified correctly. On AVA, sparse labels allow video-mAP only as an upper bound with ground-truth links (Gu et al., 2018), so AVA results are frame-mAP.

The long tail in the mean. mAP weights all 60 classes equally, and per-class AP on AVA varies widely (Gu et al., 2018, Figure 8), so the mean says little about any single action.

Example

The NumPy example renders four 80-frame label videos of three moving rectangles and reads each actor’s box from its pixels. Each actor is labelled per frame “walk” or “stand” from its speed over ±3 frames, which one frame cannot show, plus “talk” when another actor is within 24 pixels, so a box can carry two labels. A simulated detector outputs every actor’s box and one background box per frame with a score per class, under box jitter and random score flips. Frame-mAP@0.5 is computed on keyframes every 10 frames; tubes are linked as by Singh et al. (2017), without their non-maximum suppression since the simulated detector emits no duplicates, trimmed by the two-state Viterbi labelling and scored by video-mAP@0.5 with the tube IoU above.

import numpy as np
from scipy import ndimage

H = W = 96; T = 80; KEY = 10            # 80 frames; a keyframe every 10 frames ("1 Hz")
CLASSES = ["stand", "walk", "talk"]     # pose labels plus a person-person label
BW, BH = 12, 24                         # actor size in pixels


def trajectories(v):
    """Known paths of 3 actors in video v: waypoints joined at 1.5 px/frame,
    with pauses. Returns centers (3, T, 2) as (x, y)."""
    rng = np.random.default_rng(100 + v)          # the scene is fixed per video
    paths = np.zeros((3, T, 2))
    for a in range(3):
        p = rng.uniform([12, 16], [84, 80]); t = 0
        while t < T:
            pause = rng.integers(8, 25)
            paths[a, t:t + pause] = p; t += pause
            q = rng.uniform([12, 16], [84, 80])
            n = int(np.ceil(np.linalg.norm(q - p) / 1.5))
            steps = p + (q - p) * np.arange(1, n + 1)[:, None] / n
            m = max(0, min(n, T - t)); paths[a, t:t + m] = steps[:m]; t += m; p = q
    return paths


def render(paths):
    """Draw the actors into a label video and read each ground-truth box off
    its own pixels; labels: walk if speed over +-3 frames > 0.5 px/frame,
    else stand; talk if another actor's center is within 24 px."""
    video = np.zeros((T, H, W), np.uint8)
    boxes = np.zeros((3, T, 4)); labels = np.zeros((3, T, 3), bool)
    for t in range(T):
        for a in range(3):
            x, y = np.round(paths[a, t]).astype(int)
            mask = np.zeros((H, W), bool)
            mask[max(y - BH // 2, 0):y + BH // 2, max(x - BW // 2, 0):x + BW // 2] = True
            video[t][mask] = a + 1
            sl = ndimage.find_objects(mask.astype(int))[0]
            boxes[a, t] = sl[1].start, sl[0].start, sl[1].stop, sl[0].stop
            t0, t1 = max(t - 3, 0), min(t + 3, T - 1)
            speed = np.linalg.norm(paths[a, t1] - paths[a, t0]) / (t1 - t0)
            labels[a, t, 1] = speed > 0.5; labels[a, t, 0] = not labels[a, t, 1]
            d = np.linalg.norm(paths[:, t] - paths[a, t], axis=1)
            labels[a, t, 2] = np.sum(d < 24) > 1
    return video, boxes, labels


def iou(a, b):
    w = max(0., min(a[2], b[2]) - max(a[0], b[0]))
    h = max(0., min(a[3], b[3]) - max(a[1], b[1]))
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return w * h / (area(a) + area(b) - w * h)


def detect(boxes, labels, rng, sigma, p_flip):
    """Simulated per-frame actor detector: every actor plus one background box
    per frame, boxes jittered by sigma px, a score per class; each score is
    flipped (true <-> false) with probability p_flip."""
    dets = []                                   # per frame: list of (box, scores)
    for t in range(T):
        out = []
        for a in range(3):
            box = boxes[a, t] + rng.normal(0, sigma, 4)
            truth = labels[a, t] ^ (rng.random(3) < p_flip)
            out.append((box, np.clip(np.where(truth, 0.8, 0.2) + rng.normal(0, 0.1, 3), 0, 1)))
        x, y = rng.uniform(0, W - BW), rng.uniform(0, H - BH)
        out.append((np.array([x, y, x + BW, y + BH]), rng.uniform(0, 0.6, 3)))
        dets.append(out)
    return dets


def voc_ap(scores, tp, n_gt):
    """All-point interpolated AP (VOC 2010+, as in the AVA evaluation code)."""
    order = np.argsort(-np.asarray(scores), kind="stable")
    tp = np.asarray(tp, float)[order]
    rec = np.cumsum(tp) / n_gt; prec = np.cumsum(tp) / np.arange(1, len(tp) + 1)
    rec = np.r_[0, rec, 1]; prec = np.r_[0, prec, 0]
    prec = np.maximum.accumulate(prec[::-1])[::-1]
    return np.sum((rec[1:] - rec[:-1]) * prec[1:])


def match(cands, gts, overlap, thr=0.5):
    """VOC rule: each candidate (by score) takes its highest-overlap ground
    truth in the same unit; a TP if >= thr and unclaimed, else a FP."""
    used, scores, tp = set(), [], []
    for unit, obj, s in sorted(cands, key=lambda c: -c[2]):
        ov = [(overlap(obj, g), j) for j, (u, g) in enumerate(gts) if u == unit]
        o, j = max(ov, default=(0., None))
        hit = o >= thr and j not in used
        if hit:
            used.add(j)
        scores.append(s); tp.append(hit)
    return scores, tp


def frame_map(all_dets, all_gt):
    """AVA-style frame-mAP@0.5 on keyframes; every (box, class) is scored."""
    aps = []
    for c in range(3):
        cands = [((v, t), box, sc[c]) for v, dets in enumerate(all_dets)
                 for t in range(0, T, KEY) for box, sc in dets[t]]
        gts = [((v, t), boxes[a, t]) for v, (boxes, labels) in enumerate(all_gt)
               for t in range(0, T, KEY) for a in range(3) if labels[a, t, c]]
        aps.append(voc_ap(*match(cands, gts, iou), len(gts)))
    return np.mean(aps)


def link(dets, c, lam=0.1, k=5):
    """Greedy online linking for class c (Singh et al., 2017): tubes sorted by
    mean score each take the best-scoring unclaimed box with IoU > lam with
    their last box; a tube ends once more than k frames pass without a match."""
    tubes = []                                           # dicts: frames, boxes, scores, miss
    for t, frame in enumerate(dets):
        free = list(range(len(frame)))
        live = [u for u in tubes if u["miss"] <= k]
        for u in sorted(live, key=lambda u: -np.mean(u["scores"])):
            ok = [i for i in free if iou(u["boxes"][-1], frame[i][0]) > lam]
            if ok:
                i = max(ok, key=lambda i: frame[i][1][c]); free.remove(i)
                u["frames"].append(t); u["boxes"].append(frame[i][0])
                u["scores"].append(frame[i][1][c]); u["miss"] = 0
            else:
                u["miss"] += 1
        for i in free:
            tubes.append(dict(frames=[t], boxes=[frame[i][0]],
                              scores=[frame[i][1][c]], miss=0))
    return tubes


def trim(scores, alpha=1.0):
    """Two-state Viterbi labelling (action / background) of a tube's boxes,
    maximising sum of s or 1 - s minus alpha per label switch."""
    s = np.asarray(scores); unary = np.stack([1 - s, s], 1)
    acc, back = unary[0].copy(), []
    for r in range(1, len(s)):
        stay, switch = acc, acc[::-1] - alpha
        back.append(stay < switch)
        acc = np.maximum(stay, switch) + unary[r]
    lab = [int(np.argmax(acc))]
    for b in back[::-1]:
        lab.append(lab[-1] ^ int(b[lab[-1]]))
    return np.array(lab[::-1])


def tube_iou(a, b):
    """Spatio-temporal IoU (Weinzaepfel et al., 2015): temporal IoU times the
    mean box IoU over the frames both tubes cover."""
    fa, fb = dict(zip(*a)), dict(zip(*b))
    both = sorted(set(fa) & set(fb))
    if not both:
        return 0.
    t_iou = len(both) / len(set(fa) | set(fb))
    return t_iou * np.mean([iou(fa[t], fb[t]) for t in both])


def video_map(all_dets, all_gt):
    aps = []
    for c in range(3):
        cands, gts = [], []
        for v, (dets, (boxes, labels)) in enumerate(zip(all_dets, all_gt)):
            for u in link(dets, c):
                lab = trim(u["scores"])
                runs = np.split(np.arange(len(lab)), np.flatnonzero(np.diff(lab)) + 1)
                for r in runs:
                    if lab[r[0]] == 1:
                        tube = ([u["frames"][i] for i in r], [u["boxes"][i] for i in r])
                        cands.append((v, tube, np.mean([u["scores"][i] for i in r])))
            for a in range(3):                    # ground-truth tubes: label runs
                on = labels[a, :, c].astype(int)
                for r in np.split(np.arange(T), np.flatnonzero(np.diff(on)) + 1):
                    if on[r[0]]:
                        gts.append((v, (list(r), list(boxes[a, r]))))
        aps.append(voc_ap(*match(cands, gts, tube_iou), len(gts)))
    return np.mean(aps)


scenes = [render(trajectories(v)) for v in range(4)]
gt = [(b, l) for _, b, l in scenes]
video, boxes, labels = scenes[0]
n_inst = sum(l[:, ::KEY].sum() for _, l in gt)
n_tubes = sum(np.sum(np.diff(np.r_[0, l[a, :, c].astype(int)]) == 1)
              for _, l in gt for a in range(3) for c in range(3))
print("label video", video.shape, "| keyframe (box, action) pairs:", n_inst,
      "| ground-truth tubes:", n_tubes)
print("video 0, frame 40:", [[CLASSES[c] for c in range(3) if labels[a, 40, c]]
                             for a in range(3)])

print(f"{'jitter px':>9} {'flip p':>6} | {'frame-mAP (10 seeds)':>22} | {'video-mAP (10 seeds)':>22}")
for sigma, p in [(0, 0), (2, 0), (4, 0), (0, 0.1), (0, 0.2), (2, 0.1)]:
    fm, vm = [], []
    for seed in range(10):
        rng = np.random.default_rng(seed)
        dets = [detect(b, l, rng, sigma, p) for b, l in gt]
        fm.append(frame_map(dets, gt)); vm.append(video_map(dets, gt))
    print(f"{sigma:>9} {p:>6} | {np.mean(fm):.3f} [{min(fm):.3f}-{max(fm):.3f}] |"
          f" {np.mean(vm):.3f} [{min(vm):.3f}-{max(vm):.3f}]")

label video (80, 96, 96) | keyframe (box, action) pairs: 155 | ground-truth tubes: 72
video 0, frame 40: [['walk', 'talk'], ['stand'], ['stand', 'talk']]
jitter px flip p |   frame-mAP (10 seeds) |   video-mAP (10 seeds)
        0      0 | 0.966 [0.962-0.972] | 0.837 [0.777-0.899]
        2      0 | 0.868 [0.779-0.916] | 0.554 [0.461-0.622]
        4      0 | 0.223 [0.147-0.302] | 0.012 [0.002-0.029]
        0    0.1 | 0.868 [0.830-0.893] | 0.729 [0.655-0.777]
        0    0.2 | 0.776 [0.730-0.816] | 0.586 [0.531-0.662]
        2    0.1 | 0.779 [0.709-0.836] | 0.460 [0.401-0.540]

Each row averages seeds 0–9 and prints their range. With exact boxes and labels, frame-mAP is near 1 but video-mAP is 0.837: some linked and trimmed tubes do not cover the ground-truth interval well enough. Box jitter separates the metrics most: two pixels per coordinate on a 12 × 24 box cost frame-mAP about 0.10 and video-mAP about 0.28, and at four pixels both collapse. In our reading, the tube IoU multiplies a mean box IoU, lowered by every jittered frame, by a temporal IoU below 1, which leaves little margin above 0.5. Flipping 10% of the class scores costs both metrics about 0.10 here.

Results on AVA

Validation frame-mAP@0.5 on the 60 classes, as printed; versions differ, so rows on different versions are not directly comparable:

Method	AVA version	Backbone	Pretraining	Frame-mAP	Source
AVA baseline, RGB + flow	CVPR 2018 release	I3D	Kinetics	15.8	Gu et al., Table 4
ACRN	v2.1	S3D-G, RGB + flow	Kinetics	17.4	Sun et al., Table 3b
LFB, 3-crop, flips, 3 scales	235/64 videos (version not named)	R101-I3D-NL	Kinetics-400	27.7	Wu et al., Table 1g
SlowFast 8×8	v2.1	R101	Kinetics-400	26.3	Feichtenhofer et al., Table 7
SlowFast 16×8, +NL	v2.2	R101	Kinetics-600	29.8	Feichtenhofer et al., Table 8
TubeR, 1 view	v2.2	CSN-152	IG-65M, Kinetics-400	33.4	Zhao et al., Table 3
VideoMAE, with labels	v2.2	ViT-H	Kinetics-400	39.5	Tong et al., Table 5

LFB (Table 1h) and SlowFast (Table 7) list the AVA baseline at 15.6%. On test sets, the AVA baseline reported 14.7%, LFB 27.2% and SlowFast 27.1% (v2.1). Views differ too: LFB’s row uses multi-crop testing (26.8% with one center crop), the SlowFast and TubeR rows one clip.

Applications

Per-person action labels serve video search, sports analytics and robotics. The AVA authors see atomic actions as building blocks of longer activities.

Open Problems

Rare actions and interactions remain far below the frequent pose classes; two-stage systems depend on an external person detector; context over minutes enters through frozen feature banks or memories rather than end-to-end learning; and incomplete multi-label annotation turns some correct detections into false positives.