Action Recognition

Action recognition is the task of naming the action that a video shows: “playing violin”, “pushing something from left to right”, “a back dive with one and a half somersaults”. It is also called action classification, human activity recognition or, when the labels are not restricted to people, video classification, and it is a central task of video understanding. It extends image classification in one decisive way: the label can depend on how the frames change, not only on what each frame contains. How models combine frames, and the vocabulary of clips, strides and views, belong to temporal modeling; the history of the field belongs to the hub.

Problem Definition

In the standard setting, a trimmed video, cut so that it contains one instance of one action, receives one label from a fixed set of $K$ classes. Most benchmarks assume that the classes are mutually exclusive and that every test video contains exactly one of them, though both assumptions are often only approximately true.

What counts as an action depends on the dataset, and the definitions differ in granularity. The AVA authors cite Barker and Wright’s classic study of “behavior episodes” in the daily lives of a small Kansas town’s residents, which noted that activity is hierarchical: at the finest level, actions consist of atomic body movements or object manipulation, while at coarser levels they are most naturally described by intentions and goals. At fine time scales, AVA’s authors argue, actions are physical and have clear visual signatures (Gu et al., 2018). Datasets sit at different levels:

• Atomic actions. AVA labels 80 atomic actions, typically independent of the object involved (“hold” without saying what is held), on every person in movie keyframes sampled at 1 Hz (Gu et al., 2018).
• Actions. Kinetics is “focused on human actions (rather than activities or events)” (Carreira & Zisserman, 2017, §3), in three groups: actions of one person (drawing, laughing), person–person actions (hugging, shaking hands) and person–object actions (mowing a lawn, washing dishes). Its clips last about 10 seconds (Kay et al., 2017).
• Activities. ActivityNet takes its 203 activity classes from the taxonomy of the American Time Use Survey and keeps its hierarchy, at least four levels deep, from top-level categories such as household activities down to leaves such as polishing shoes (Heilbron et al., 2015).
• Templates. Something-Something defines 174 classes as caption templates with placeholders for objects, such as “Putting something onto something” (from the official class list), which crowd workers act out with objects of their choice (Goyal et al., 2017).
• Codified sequences. In Diving48, each of the 48 classes is a dive as standardized by the international federation FINA: a sequence of take-off, somersaults, twists and flight position (Li, Li & Vasconcelos, 2018).

Single-label and multi-label. A clip can show several actions at once. Kinetics clips are labeled with one class even when another Kinetics action also occurs (“texting” while “driving a car”), so the annotation is not exhaustive; for that reason its authors consider top-5 accuracy more suitable than top-1 (Kay et al., 2017). Multi-label datasets such as Charades, with 157 classes over activities that last about 30 seconds on average, assign several labels per video and are scored by mean average precision (Feichtenhofer et al., 2019).

Input and Output

A video is a sequence of $F$ frames. A model reads clips: a clip of $T$ frames starting at frame $s$ with temporal stride $\tau$ is $\mathbf{x}_s = (I_s, I_{s+\tau}, \dots, I_{s+(T-1)\tau})$, cropped to a tensor of shape $T \times H \times W \times C$ (PyTorch’s Conv3d takes a batch as $(N, C, T, H, W)$), and spanning $T\tau$ source frames. A network $f_\theta$ maps a clip to $K$ class scores. Because the clip is shorter than the video, the video-level prediction averages over a set $\mathcal{V}$ of test-time views, each a clip $s$ with a spatial crop $c$:

$$\hat{\mathbf{p}}(V) = \frac{1}{|\mathcal{V}|} \sum_{(s, c) \in \mathcal{V}} \operatorname{softmax}\big(f_\theta(\operatorname{crop}_c(\mathbf{x}_s))\big), \qquad \hat{y} = \arg\max_k \hat{p}_k .$$

Training usually minimizes the cross-entropy loss of a single randomly sampled clip and crop, labeled with the label of the whole video:

$$\min_\theta \; \mathbb{E}_{(V, y)} \, \mathbb{E}_{s, c} \Big[ -\log \operatorname{softmax}\big(f_\theta(\operatorname{crop}_c(\mathbf{x}_s))\big)_y \Big].$$

In our reading, this objective treats every clip as if it showed the action, so when a clip misses the action, its label is noise; the example below shows the effect. For multi-label data, $f_\theta$ ends in one sigmoid per class and the loss is binary cross-entropy.

Variants

Trimmed and untrimmed video. ActivityNet distinguishes three tasks: classifying an untrimmed video that may contain long stretches of unrelated content, classifying trimmed activity instances, and detecting the start and end of each instance (Heilbron et al., 2015). The last is temporal action localization, the video counterpart of object detection along the time axis, and labeling every person with a box over time is spatio-temporal action detection; both have their own articles.

Fine-grained and temporal-reasoning datasets. Something-Something groups similar templates, such as putting something on top of, next to or behind something, and adds pretended actions, so that a network cannot shortcut the task by reading the object type from a frame or the action from hand position or camera shake (Goyal et al., 2017). Diving48 was built to minimize the static cues of earlier datasets: the scenes (a board, a pool, spectators) and the divers look alike in every class, and dives 16 and 18 differ only in the number of somersaults (Li, Li & Vasconcelos, 2018).

First-person (egocentric) recognition. In head-mounted video, the camera wearer’s hands and the objects they handle dominate the view. EPIC-KITCHENS recorded 55 hours from 32 participants in their own kitchens and defines an action as a pair of a verb class and a noun class, with 125 verb and 331 noun classes, so that an action is correct only if both are (Damen et al., 2018). Its test sets separate kitchens seen during training from unseen ones; the authors’ TSN baseline reached 20.5% top-1 action accuracy on seen and 10.9% on unseen kitchens.

Skeleton-based recognition. Instead of pixels, the input is a sequence of body-joint positions, estimated by a pose estimator or measured by a depth camera. NTU RGB+D, recorded with Microsoft Kinect v2, has 56,880 samples of 60 classes from 40 subjects, with the 3D positions of 25 joints, and tests generalization to new subjects (cross-subject) and to a new camera (cross-view) (Shahroudy et al., 2016). Skeleton-based action recognition has its own article.

Zero-shot and open-vocabulary recognition. Classifiers with a fixed output layer cannot name unseen classes. Video–language models embed clips and class names in a shared space, in the manner of CLIP, and classify by comparing a clip with the text of each class name (see vision–language pretraining). X-CLIP, which adapts a pretrained CLIP to video and trains it on Kinetics-400, reached 65.2% top-1 on Kinetics-600 classes absent from Kinetics-400, averaged over three draws of 160 of those 220 classes (Ni et al., 2022, Table 4; the ECCV version defers the protocol to its supplementary material, and the arXiv v1 appendix describes it).

Approaches

The families below are summarized, not taught; the linked articles and the hub cover them.

Family	Representative methods	What it adds
Hand-crafted local features	Dense trajectories, improved dense trajectories (iDT)	Descriptors of shape, appearance and motion along tracked points, pooled with Fisher vectors
Frame-level 2D networks	Per-frame CNN with averaged scores	ImageNet pretraining; order-blind (temporal modeling)
Two-stream networks	Simonyan & Zisserman; convolutional fusion	A second network on stacked optical flow supplies explicit motion
3D convolutional networks	C3D, I3D, R(2+1)D, X3D	Kernels that span time learn motion from pixels
Sparse sampling and cheap temporal mixing	TSN, TSM	Snippets spread over the whole video; temporal shifts at no FLOP cost
Two-pathway networks	SlowFast	A slow, wide pathway for appearance and a fast, thin one for motion
Video transformers	TimeSformer, ViViT, MViT, Video Swin	Self-attention over space-time patches
Self-supervised and video–language pretraining	VideoMAE; CLIP-based models	Pretraining from unlabeled video (self-supervised learning) or paired text

Training Practice

Initialization. 3D networks can start from an image network by inflation, as I3D does from an ImageNet model (Carreira & Zisserman, 2017; see 3D convolutional networks). Video transformers start from image transformers: TimeSformer reached 75.8% top-1 on Kinetics-400 from ImageNet-1K weights and 78.0% from ImageNet-21K, but 59.5% on Something-Something V2 either way (Bertasius et al., 2021, Table 3). Image pretraining is not always needed: SlowFast was trained on Kinetics from scratch, and its authors report that ImageNet pretraining performed similarly (±0.3%) (Feichtenhofer et al., 2019).

Pretraining on Kinetics, fine-tuning on small datasets. UCF101 and HMDB51 have on the order of 10k videos, too few to train large video models. Carreira and Zisserman (2017, Table 3) pretrained on miniKinetics, an early 213-class version, and compared three settings on split 1: training on the target data alone, training only the last layer on frozen features, and fine-tuning end to end. For two-stream I3D, full fine-tuning raised accuracy from 93.4% to 96.5% on UCF101 and from 66.4% to 75.9% on HMDB51; for a C3D-like network, from 49.2% to 79.9% and from 24.3% to 49.4% (the paper’s Table 2 gives 51.6%, not 49.2%, for the same network trained and tested on UCF101 split 1 alone; we report both as printed). Full-Kinetics results are in the table below; see transfer learning.

Augmentation. Spatial augmentation follows image practice. I3D resized the smaller side of each video to 256 pixels, cropped a random $224 \times 224$ patch, chose a random starting frame and flipped videos left–right at random, consistently for all frames of a video (Carreira & Zisserman, 2017). TSN added corner cropping, taking crops only from the corners or the center so that the network does not focus implicitly on the image center, and scale jittering: from frames resized to $256 \times 340$, crops of width and height drawn independently from $\{256, 224, 192, 168\}$ are resized to $224 \times 224$, which also varies the aspect ratio (Wang et al., 2016).

When a flip changes the label. A horizontal flip turns a clip of “pushing something from left to right” into a valid clip of the opposite class, so applying it with the original label teaches the model that the two are the same; in our reasoning, the flip must either be left out or be paired with a label swap. VideoMAE’s settings, for example, use flips on Kinetics-400 but none on Something-Something V2, in pretraining or fine-tuning (Tong et al., 2022, NeurIPS supplementary material, Tables 9 and 10; the arXiv v3 appendix has the same tables), though the paper does not give the reason. The same reasoning implies that flipping an optical-flow input requires negating its horizontal component.

Clip sampling at training and test time. Training usually samples one clip at a random position per video and epoch, as I3D and SlowFast do, or one snippet per segment, as TSN does (temporal modeling covers the sampling schemes). Testing covers the video more systematically: I3D applied its network convolutionally over the whole video with a $224 \times 224$ center crop (Carreira & Zisserman, 2017); TSN followed the two-stream protocol of 25 frames or flow stacks, each with five crops and their flips (Wang et al., 2016); SlowFast averaged 10 uniformly spaced clips, each with 3 crops of $256 \times 256$, larger than its $224 \times 224$ training crops (Feichtenhofer et al., 2019).

Cost. In TimeSformer’s Table 2, a SlowFast R50 initialized from ImageNet needed 3,840 Tesla V100 GPU hours to reach 75.6% on Kinetics-400 (70.0% when limited to 448 hours), while TimeSformer reached 75.8% in 416 hours (Bertasius et al., 2021). Mixed-precision training stores weights, activations and gradients in 16-bit floating point while keeping a 32-bit master copy of the weights, which nearly halves memory (Micikevicius et al., 2018; see neural network training); for video, in our reasoning, the saving goes mainly to longer clips or larger batches.

Evaluation

Kinetics: top-1 and top-5. Models are scored by top-1 and top-5 accuracy on the validation set (as TimeSformer states) or the held-out test set (as I3D’s arXiv v3 does). Top-5 compensates for the non-exhaustive labels described above (Kay et al., 2017).

UCF101 and HMDB51: three splits. Both datasets come with three train/test splits, and papers report accuracy averaged over them (Kuehne et al., 2011; Carreira & Zisserman, 2017); the UCF101 page asks for results on its “Three Train/Test Splits”. HMDB51’s splits put 70 clips per class in training and 30 in testing, with no clips from the same source video on both sides (Kuehne et al., 2011). UCF101’s clips of each class come in 25 groups that share background or actors (Soomro et al., 2012), and the dataset page warns that groups must not be split between training and testing, since clips of a group come from one long video (UCF101 dataset page, archived January 2025). Because every HMDB51 test set has 30 clips per class, overall accuracy and mean per-class accuracy (balanced accuracy) coincide there. For UCF101, whose classes have different numbers of clips, the papers we read average over the three splits without saying whether accuracy within a split is computed over clips or averaged over classes. Ablations often use split 1 only, as I3D’s architecture comparison does (Table 2), which is not the same number as the three-split mean.

Other protocols. Multi-label datasets such as Charades use mean average precision (Feichtenhofer et al., 2019); EPIC-KITCHENS reports top-1 and top-5 accuracy for verbs, nouns and actions, plus precision and recall averaged over classes with more than 100 training samples (Damen et al., 2018).

Views and cost. Because predictions average several views, accuracy depends on how many were used, and cost is reported as GFLOPs per view × number of views (see video understanding): a SlowFast 16×8 R101+NL costs 234 GFLOPs × 30 views (Feichtenhofer et al., 2019, Table 2). TimeSformer plotted Kinetics-400 accuracy against the number of test clips (1, 3, 5, 10) and found that X3D and SlowFast need at least 5 clips to approach their best, while its long-range variant, which covers about 12 seconds per clip, does not (Bertasius et al., 2021, Figure 6). FLOPs do not settle speed either: on the 20k Kinetics-400 validation videos and 8 V100 GPUs, SlowFast took 14.88 hours and TimeSformer-L 2.6 hours, although the two have comparable TFLOPs (Bertasius et al., 2021).

Pitfalls in comparing numbers.

• Different views and different FLOP counts. A result at 1 clip × 3 crops and one at 10 clips × 3 crops differ tenfold in test cost at equal cost per view. Even the cost of the same model varies between papers. TimeSformer’s Table 5 gives TimeSformer-L an inference cost of 7.14 TFLOPs; ViViT, MViT and Video Swin read this as the total for its 3 views (see video transformers), as do we (the same table’s 7.0 TFLOPs for SlowFast matches 234 GFLOPs × 30 views), or about 2,380 GFLOPs per view. VideoMAE’s Table 7 lists the same model at 8,353 GFLOPs × 1 × 3, about 25 TFLOPs in total. VideoMAE V2, from the same group, lists 7.14 TFLOPs for 1 × 3 views (VideoMAE); we did not find the source of the difference.
• Different Kinetics. Kinetics is distributed as YouTube links that disappear over time, so a result depends on the copy used (see video understanding). Validation and test numbers also get mixed. I3D’s arXiv v3 reports two-stream I3D at 74.2% top-1 on the Kinetics test set (Table 3); the Non-local paper lists it at 75.7% on the validation set and 74.2% on the test set (Wang et al., 2018, Table 3). SlowFast’s Table 2 gives it 75.7%, the validation figure, but lists the I3D trained without ImageNet at 71.6%, its test figure in I3D’s arXiv v3, and names no split.
• Different pretraining data. ImageNet-1K, ImageNet-21K, JFT-300M, other video datasets or none at all: VideoMAE’s tables carry an “extra data” column for this reason. Datasets can also overlap: the Kinetics-600 validation set overlaps the Kinetics-400 training set, so SlowFast’s authors did not pretrain on Kinetics-400 for their Kinetics-600 results (Feichtenhofer et al., 2019).
• Different numbers under one name. Both papers cite Wang and Schmid (2013) for improved dense trajectories, which appear at 85.9% on UCF101 and 57.2% on HMDB51 in the two-stream paper’s table (Simonyan & Zisserman, 2014, Table 4; it also cites the authors’ THUMOS submission) and at 86.4% and 61.7% in I3D’s (Carreira & Zisserman, 2017, Table 4). The ICCV paper itself reports 57.2% on HMDB51 and no UCF101 result; we did not find the source of I3D’s figures.

Representative Results

The table collects printed results from each era, with the conditions the papers state. The rows are not directly comparable: they differ in dataset, split, pretraining, input and number of views, and none is a current best result.

Method (year)	Dataset and split	Pretraining	Test views	Result as printed
iDT, Fisher vectors, human detection (2013)	HMDB51, mean over 3 splits	none (hand-crafted)	whole video	57.2% accuracy (Wang & Schmid, Table 4)
Two-stream, SVM fusion (2014)	UCF101 / HMDB51, mean over 3 splits	ImageNet (spatial stream)	25 frames × 10 inputs (5 crops and their flips)	88.0% / 59.4% (Simonyan & Zisserman, Table 4)
C3D, 3 nets + linear SVM (2015)	UCF101, 3-split setting	Sports-1M (and I380K)	16-frame clips, 8-frame overlap, features averaged	85.2% (Tran et al., Table 3)
Two-stream I3D (2017)	Kinetics test	ImageNet	whole video, center crop	74.2% top-1, 91.3% top-5 (arXiv v3, Table 3)
Two-stream I3D (2017)	UCF101 / HMDB51, mean over 3 splits	ImageNet + Kinetics	whole video, center crop	97.9% / 80.2% (CVPR, Table 4)
SlowFast 16×8, R101+NL (2019)	Kinetics-400 (split not named in the table)	none	234 GFLOPs × 30 views (10 × 3)	79.8% top-1, 93.9% top-5 (Table 2)
TimeSformer-L (2021)	Kinetics-400 validation	ImageNet-21K	1 clip × 3 crops; 7.14 TFLOPs as printed (see Pitfalls)	80.7% top-1, 94.7% top-5 (Table 5)
TimeSformer-L (2021)	Something-Something V2	ImageNet-1K	1 clip × 3 crops	62.4% top-1 (Table 7)
VideoMAE ViT-L, 32 frames (2022)	Something-Something V2	self-supervised on Something-Something V2, no extra data	1,436 GFLOPs × 1 × 3	75.4% top-1, 95.2% top-5 (Table 6)
VideoMAE ViT-H, $320^2$ (2022)	Kinetics-400	self-supervised on Kinetics-400, no extra data	7,397 GFLOPs × 4 × 3	87.4% top-1, 97.6% top-5 (Table 7)

Example: Appearance Classes, Motion Classes, Views and Flips

The example builds a six-class dataset of 40-frame, $16 \times 16$ grayscale videos. In “square” and “ring” a shape drifts in a random direction throughout, so one frame identifies the class. In the other four classes the same plus sign is still except during one 10-frame event at a random time, in which it moves left, moves right, grows or shrinks; in our reasoning their frames have nearly the same distribution, and only frame order separates them. The frame wraps around at its edges, so position carries no information. Models read clips of $T = 8$ frames at stride $\tau = 2$. The NumPy part generates the data, defines the views and checks what a horizontal flip does.

import numpy as np

F, S, E = 40, 16, 10           # frames per video, frame size in pixels, event length in frames
T, TAU = 8, 2                  # a clip: T = 8 frames at stride tau = 2
SPAN = (T - 1) * TAU + 1       # 15 source frames from first to last sampled frame
CLASSES = ["square", "ring", "left", "right", "grow", "shrink"]
SWAP = np.array([0, 1, 3, 2, 4, 5])          # label after a horizontal flip
yy, xx = np.mgrid[0:S, 0:S]

def wrap(d):
    """Signed distance on a circle of S pixels (the frame wraps around at its edges)."""
    return (d + S / 2) % S - S / 2

def draw(kind, cx, cy, r):
    dx, dy = np.abs(wrap(xx - cx)), np.abs(wrap(yy - cy))
    if kind == "square":
        return np.maximum(dx, dy) <= r
    if kind == "ring":
        return (dx**2 + dy**2 <= r**2) & (dx**2 + dy**2 >= (r - 1.2)**2)
    return ((dx <= r) & (dy <= 1)) | ((dy <= r) & (dx <= 1))       # a plus sign of arm length r

def make_video(label, rng):
    """One video of shape (F, S, S): a bright shape on a noisy background."""
    cx, cy = rng.uniform(0, S, 2)
    video = np.zeros((F, S, S))
    if label < 2:              # square or ring, drifting in a random direction all the time
        a = rng.uniform(0, 2 * np.pi)
        for t in range(F):
            video[t] = draw(CLASSES[label], cx + np.cos(a) * t, cy + np.sin(a) * t, 3)
    else:                      # a plus sign, still except during one event of E frames
        start = rng.integers(0, F - E + 1)
        for t in range(F):
            p = np.clip((t - start) / (E - 1), 0, 1)            # 0 before the event, 1 after it
            x = cx + {2: -0.8, 3: 0.8}.get(label, 0.0) * (E - 1) * p
            r = {4: 1.5 + 4 * p, 5: 5.5 - 4 * p}.get(label, 3.5)
            video[t] = draw("plus", x, cy, r)
    return video + rng.normal(0, 0.2, video.shape)

def make_dataset(n_per_class, rng):
    y = np.repeat(np.arange(6), n_per_class)
    return np.stack([make_video(c, rng) for c in y]).astype(np.float32), y

def view_starts(n_views):
    """Test-time clips: the central one, or n_views evenly spaced over the video."""
    if n_views == 1:
        return np.array([(F - SPAN) // 2])
    return np.linspace(0, F - SPAN, n_views).round().astype(int)

def take_clips(videos, starts):
    """videos (N, F, S, S), starts (N, V) -> clips (N, V, T, S, S)."""
    idx = starts[..., None] + TAU * np.arange(T)
    return videos[np.arange(len(videos))[:, None, None], idx]

def per_class_accuracy(pred, y):
    return np.array([np.mean(pred[y == c] == c) for c in range(6)])

def centroid_x(frames):
    """Horizontal position of the bright shape per frame (circular mean)."""
    w = np.clip(frames - 0.5, 0, None).sum(axis=-2)
    ang = 2 * np.pi * np.arange(S) / S
    return np.angle((w * np.exp(1j * ang)).sum(-1)) * S / (2 * np.pi)

rng = np.random.default_rng(0)
X, y = make_dataset(50, rng)
print("videos", X.shape, "| clip: T =", T, "frames at stride", TAU, "spanning", SPAN, "of", F)
print("test clip starts, 1 view:", view_starts(1), " 5 views:", view_starts(5))
for c in range(2, 6):                                  # the four plus-sign classes
    v = X[y == c]
    dx = wrap(np.diff(centroid_x(v), axis=1)).sum(1)                 # summed frame-to-frame shift
    dx_flip = wrap(np.diff(centroid_x(v[..., ::-1]), axis=1)).sum(1) # the same, mirrored video
    size = np.clip(v - 0.5, 0, None).sum((2, 3))                     # bright mass per frame
    print(f"{CLASSES[c]:6s} x-shift {dx.mean():+5.2f} px, mirrored {dx_flip.mean():+5.2f} px | "
          f"size first {size[:, 0].mean():4.1f}, last {size[:, -1].mean():4.1f}")
event_starts = np.arange(F - E + 1)                    # every possible event start
def sees_event(clip_start):                            # do the clip's first and last frames differ?
    return (clip_start < event_starts + E - 1) & (clip_start + SPAN - 1 > event_starts)
print("event starts whose motion the central clip sees:", sees_event(view_starts(1)[0]).sum(),
      "of", len(event_starts), "| at least one of 5 views:",
      np.any([sees_event(s) for s in view_starts(5)], axis=0).sum())

videos (300, 40, 16, 16) | clip: T = 8 frames at stride 2 spanning 15 of 40
test clip starts, 1 view: [12]  5 views: [ 0  6 12 19 25]
left   x-shift -7.23 px, mirrored +7.23 px | size first 12.3, last 11.9
right  x-shift +7.29 px, mirrored -7.29 px | size first 12.1, last 12.2
grow   x-shift +0.08 px, mirrored -0.08 px | size first  4.1, last 20.4
shrink x-shift -0.02 px, mirrored +0.02 px | size first 20.5, last  4.1
event starts whose motion the central clip sees: 22 of 31 | at least one of 5 views: 31

Mirroring reverses the 7.2-pixel shift of “left” and “right” (0.8 pixels per frame over 9 steps) but leaves the size change of “grow” and “shrink” alone: a flip turns one class into another for two classes only. The central test clip misses the event for 9 of the 31 possible start times, while five evenly spaced clips always include it.

The PyTorch part, run after the NumPy part with PyTorch 2.14.1, trains two networks with the same layers. With temporal kernel depth $k_t = 1$ the network is a 2D CNN applied to each frame, and its global average pooling followed by a linear layer equals averaging per-frame class scores; with $k_t = 3$ the kernels are $3 \times 5 \times 5$ and $3 \times 3 \times 3$ and span neighboring frames. The 3D network is trained three ways: without flips, with random flips that keep the label, and with flips that swap “left” and “right”. Each model is tested with the central clip (1 view) and with 5 clips, averaging their softmax scores, for seeds 0 to 2 (each seed draws new training and test data and a new initialization).

import torch
import torch.nn as nn

torch.set_num_threads(4)
print("torch", torch.__version__)

class VideoCNN(nn.Module):
    """Two conv layers, global average pooling, a linear classifier. kt = 3: 3D kernels that
    span neighbouring frames. kt = 1: the same network applied to each frame separately (a 2D
    CNN); with a linear head, its pooled output equals the average of per-frame class scores."""
    def __init__(self, kt):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv3d(1, 8, (kt, 5, 5), stride=(1, 2, 2), padding=(kt // 2, 2, 2)),
            nn.BatchNorm3d(8), nn.ReLU(),
            nn.Conv3d(8, 16, (kt, 3, 3), padding=(kt // 2, 1, 1)), nn.BatchNorm3d(16), nn.ReLU(),
            nn.AdaptiveAvgPool3d(1), nn.Flatten(), nn.Linear(16, 6))
    def forward(self, x):                                   # x: (N, C, T, H, W)
        return self.net(x)

def train(kt, X, y, flip, seed, epochs=30, batch=50):
    """One random clip per video per epoch. flip: None, 'flip' or 'flip+swap'."""
    torch.manual_seed(seed)
    rng = np.random.default_rng(seed)
    model = VideoCNN(kt)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    sched = torch.optim.lr_scheduler.OneCycleLR(opt, 3e-3, total_steps=epochs * -(-len(X) // batch))
    for epoch in range(epochs):
        starts = rng.integers(0, F - SPAN + 1, size=(len(X), 1))
        clips, labels = take_clips(X, starts)[:, 0], y.copy()      # (N, T, S, S)
        if flip:
            f = rng.random(len(X)) < 0.5
            clips[f] = clips[f][..., ::-1]                         # mirror left-right
            if flip == "flip+swap":
                labels[f] = SWAP[labels[f]]                        # left <-> right
        order = rng.permutation(len(X))
        for i in range(0, len(X), batch):
            b = order[i:i + batch]
            xb = torch.from_numpy(np.ascontiguousarray(clips[b][:, None]))   # (N, 1, T, S, S)
            loss = nn.functional.cross_entropy(model(xb), torch.from_numpy(labels[b]))
            opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    return model

def predict(model, X, n_views):
    """Average the softmax scores of n_views clips per video."""
    model.eval()
    clips = take_clips(X, np.tile(view_starts(n_views), (len(X), 1)))   # (N, V, T, S, S)
    with torch.no_grad():
        p = model(torch.from_numpy(clips.reshape(-1, 1, T, S, S))).softmax(1)
    return p.view(len(X), n_views, 6).mean(1).argmax(1).numpy()

configs = [("2D frame average", 1, None), ("3D conv", 3, None),
           ("3D conv, flip", 3, "flip"), ("3D conv, flip+swap", 3, "flip+swap")]
top1 = {(name, v): [] for name, _, _ in configs for v in (1, 5)}
for seed in range(3):
    Xtr, ytr = make_dataset(120, np.random.default_rng(10 + seed))
    Xte, yte = make_dataset(100, np.random.default_rng(20 + seed))
    for name, kt, flip in configs:
        model = train(kt, Xtr, ytr, flip, seed)
        for v in (1, 5):
            pred = predict(model, Xte, v)
            top1[name, v].append(np.mean(pred == yte))
            if seed == 0:
                acc = " ".join(f"{a:.2f}" for a in per_class_accuracy(pred, yte))
                print(f"seed 0, {v} view{'s' if v > 1 else ' '}, per class: {name:18s} {acc}")
print("top-1 accuracy, seeds 0-2 (chance 0.167):")
for name, _, _ in configs:
    cells = [f"{v} view{'s' if v > 1 else ' '} " + " ".join(f"{a:.3f}" for a in top1[name, v])
             for v in (1, 5)]
    print(f"  {name:18s} " + "   ".join(cells))

torch 2.14.1
seed 0, 1 view , per class: 2D frame average   1.00 1.00 0.42 0.41 0.29 0.58
seed 0, 5 views, per class: 2D frame average   1.00 1.00 0.39 0.43 0.24 0.63
seed 0, 1 view , per class: 3D conv            1.00 1.00 0.84 0.75 0.76 0.83
seed 0, 5 views, per class: 3D conv            1.00 1.00 0.97 0.97 0.94 0.98
seed 0, 1 view , per class: 3D conv, flip      1.00 1.00 0.71 0.25 0.87 0.79
seed 0, 5 views, per class: 3D conv, flip      1.00 1.00 0.70 0.21 0.98 0.92
seed 0, 1 view , per class: 3D conv, flip+swap 1.00 1.00 0.88 0.68 0.78 0.77
seed 0, 5 views, per class: 3D conv, flip+swap 1.00 1.00 0.97 0.98 0.94 0.93
top-1 accuracy, seeds 0-2 (chance 0.167):
  2D frame average   1 view  0.617 0.652 0.583   5 views 0.615 0.652 0.625
  3D conv            1 view  0.863 0.838 0.813   5 views 0.977 0.972 0.947
  3D conv, flip      1 view  0.770 0.723 0.695   5 views 0.802 0.797 0.785
  3D conv, flip+swap 1 view  0.852 0.818 0.812   5 views 0.970 0.978 0.975

The per-class rows (columns in the order square, ring, left, right, grow, shrink) show the structure the dataset was built with. The frame-averaging network gets both appearance classes right and is near the pairwise chance level of 0.5 within left/right and grow/shrink, so its top-1 accuracy stays between 0.583 and 0.652 over seeds 0 to 2 whatever the number of views. The 3D network separates all six classes; with one view it reaches 0.813–0.863, and five views raise it to 0.947–0.977, the gain coming from the four event classes, because the central clip sometimes misses the event. Flipping without changing the label costs the 3D network most of its left/right distinction (0.70 and 0.21 for seed 0 with five views) and leaves grow and shrink intact, as the NumPy check predicts; overall it falls to 0.785–0.802. In our reading, half of the training clips of each direction class then show the opposite motion under the same label, so direction carries almost no information about the label. Flipping with a label swap recovers 0.970–0.978, close to the 0.947–0.977 without flips; in our reading the synthetic data has no other left–right asymmetry for a flip to remove, so it adds little here. Two runs gave identical output, in about 40 seconds.

Challenges and Open Problems

The following are, in our assessment, the main open questions as of October 2026.

• Scene and static bias. Benchmarks reward whatever cue separates their classes, and that cue is often static. Already in the HMDB51 paper, scene descriptors (gist) classified UCF Sports at 60.0%, above the 58.6% of spatio-temporal HOG/HOF features, whereas on HMDB51 gist reached 13.4% against 20.2% (Kuehne et al., 2011, Table 2). RESOUND’s measurements with frozen per-frame features, summarized in the hub, found Diving48 about as unbiased as the six-class KTH (Li, Li & Vasconcelos, 2018, Table 1). Results on Kinetics, UCF101 or HMDB51 alone say little about temporal modeling.
• Temporal reasoning. On Diving48, the best model in RESOUND’s experiments, a C3D on 64 frames, reached 27.60%, up from 11.51% with 8 frames (Li, Li & Vasconcelos, 2018, Table 3); TimeSformer-L later reported 81.0%, but its authors note an issue with a previous version of the Diving48 labels and compare only with a SlowFast they reproduced (Bertasius et al., 2021, Table 7), so the two numbers may not measure the same thing. Gains like this come with larger pretraining and many other changes, so how much reflects better reasoning about time is hard to isolate.
• Long-range context. Models typically read a few seconds at a time; activities defined over minutes, such as the steps of a procedure, need memory beyond the clip.
• Compute. Accuracy has been bought with views, resolution and pretraining (the hub gives the arithmetic); fair comparison needs cost per view, number of views and wall-clock time.

Applications

Action recognition serves video search and indexing, sports analysis, safety monitoring and assistive systems, and supplies the pretrained backbones of temporal action localization, spatio-temporal action detection and video–language models.