Video Understanding

Video understanding is the part of computer vision that recognizes, locates and describes what happens in video: which action a clip shows, when an activity starts and ends, who is doing what and where, and how a video relates to a sentence that describes it. It is also called video recognition or video analysis, and its best-known task, naming the action in a clip, is called action recognition or activity recognition. Where image classification asks what a picture shows, video understanding asks what is happening, which depends on how the picture changes.

This article orients the reader across the field: what makes video different from a stack of images, the tasks, the main model eras, the datasets and how results are measured. How a model represents time (pooling frame features, recurrence, 3D convolution, attention or optical flow) and how frames are sampled from a video are the subject of temporal modeling; action recognition, the model families and the tasks beyond classification each have their own article.

Definition

Video understanding is the family of tasks that map a video, a sequence of frames ordered in time, to a description of its content: one or more labels for the whole video or clip, labels with start and end times, labels attached to people or objects with their boxes over time, or text. What sets it apart from image understanding is that the answer may depend on the order and change of the frames, not only on what any single frame contains.

Intuition

Why a single frame is not enough

Something-Something, a dataset of people handling objects, has the classes “Pushing something from left to right” and “Pushing something from right to left” (Goyal et al., 2017; names from the official class list). Played backwards, a clip of one shows the object moving the other way, from exactly the same frames in a different order. It follows, in our reasoning, that if each clip of one class is as likely as its reversal in the other, no classifier that looks at one frame, or at an unordered set of frames, can tell the classes apart better than chance: the evidence lives in the relation between frames. The same holds for picking up and putting down, opening and closing, or pretending to do something versus doing it, a distinction that dataset was designed to require. The AVA authors made the point for annotation: they showed annotators about 1.5 seconds on either side of each labeled frame because actions such as picking up and putting down cannot be told apart in a static frame (Gu et al., 2018).

Early evidence that motion carries information that static pose lacks came from HMDB51. On 10 of its classes, features from manually annotated joint positions in single frames reached 35% accuracy (chance 10%), against 54% for motion features computed over the whole clip (Kuehne et al., 2011).

But often a single frame is enough

The opposite observation is just as important. Many action classes can be guessed from the scene or the objects alone: a swimming pool, a piano, a basketball court. On Sports-1M, Karpathy et al. (2014, Table 1) found that a network seeing single frames reached 59.3% video-level hit@1, against 60.9% for their best network with temporal connections, and noted that the single-frame model was already strong. A good model must therefore use appearance where it suffices and motion where it does not, and a good benchmark must contain classes of the second kind.

Formal Definition

A video is a function $I(x, y, t)$ of two spatial coordinates and time, sampled into frames. A clip of $T$ frames of height $H$, width $W$ and $C$ color channels is a tensor of shape $T \times H \times W \times C$; deep-learning code for 3D convolutions usually stores a batch of $N$ clips as $(N, C, T, H, W)$. A clip of $T$ frames taken with temporal stride $\tau$ spans $T\tau$ frames of the source video. Kernel and patch sizes are written $t \times h \times w$, time first.

The main tasks differ in their output:

Task	Input	Output
Clip or video classification	Trimmed clip or whole video	Class scores for the whole input
Temporal action localization	Untrimmed video	Segments $(t_{\text{start}}, t_{\text{end}}, \text{class}, \text{score})$
Spatio-temporal action detection	Video	Boxes per frame (or tubes over time) with action labels per person
Temporal action segmentation	Video	One class label per frame
Video–text retrieval	Video and a set of sentences (or the reverse)	A ranking

What Makes Video Different

Time is a dimension. A video has one more axis than an image, and it behaves differently from the two spatial axes. Feichtenhofer et al. (2019) start from this observation: natural images are close to isotropic in $x$ and $y$, but in $(x, y, t)$ slow motions are much more likely than fast ones, so time need not be treated like space. Models that do treat it symmetrically, such as 3D convolutions with $3 \times 3 \times 3$ kernels (Tran et al., 2015), and models that treat it separately, such as two-pathway networks or attention factorized into time and space, are both common.

Neighboring frames are redundant. At 25 or 30 frames per second, consecutive frames are nearly identical, so most of the pixels in a clip can be predicted from the others. Models exploit this by sampling a few frames spread over the video instead of every frame, and self-supervised methods exploit it by hiding most of the input: VideoMAE masks 90% to 95% of space-time patches during pretraining, a ratio its authors attribute to temporal redundancy (Tong et al., 2022).

Motion and appearance are complementary cues. Appearance (objects, scene, people) comes from single frames; motion comes from their changes, explicitly as optical flow or implicitly through layers that span several frames. In the two-stream network of Simonyan and Zisserman (2014, Table 4), averaged over the three splits of UCF101, a network on single RGB frames reached 73.0% accuracy, one on stacked optical flow fields 83.7%, and their SVM fusion 88.0%: on UCF101 at least, motion alone was the stronger cue, and the two together beat either.

Scale. A 10-second clip at 25 frames per second has 250 frames. Karpathy et al. (2014) trained their Sports-1M networks for a month with 10 to 50 model replicas, and evaluated each test video by averaging the predictions of 20 random clips, each seen in 4 crops and flips. Labels are expensive too: marking when an action starts and ends, or boxing every person once per second for 15 minutes of film, costs far more than one label per image.

The Task Landscape

• Clip and video classification, usually called action recognition: one label for a trimmed clip, or for a whole video by aggregating clips. It is the task most models are first designed and benchmarked on, and the pretraining task whose features the others reuse.
• Temporal action localization (temporal action detection): find the start, end and class of every action instance in an untrimmed video, the video counterpart of object detection along the time axis. ActivityNet benchmarked it alongside untrimmed and trimmed classification (Heilbron et al., 2015).
• Spatio-temporal action detection: box each person in each annotated frame and label their actions, possibly several at once. AVA defines the task on movies, with labels at one frame per second (Gu et al., 2018); detectors are usually an object detector with a video backbone, as in SlowFast’s AVA detector, which its authors describe as similar to Faster R-CNN (Feichtenhofer et al., 2019).
• Skeleton-based action recognition: classify actions from sequences of body joint positions estimated by a pose estimator or captured by a depth sensor, instead of from pixels.
• Temporal action segmentation: label every frame of a long video, typically a procedure such as cooking, with the step being performed.
• Video retrieval and video–language tasks: find the video that matches a text query or the reverse, caption a video, or answer questions about it. These rely on models that embed video and text in a shared space or connect a video encoder to a language model, the video extension of vision–language models.
• Anticipation: predict an action before it happens from the frames that precede it, which matters for robots and assistive systems that must act in time.
• Egocentric video: video from a head-worn camera, where the camera wearer’s hands and the objects they handle dominate the view. Ego4D organizes its benchmarks around the wearer’s past (querying an episodic memory), present (hand–object manipulation, conversation, social interaction) and future (forecasting activities) (Grauman et al., 2022).

The Model Eras

Hand-crafted space-time features (2004–2013)

Before deep learning, the dominant approach represented a video as a bag of local space-time features, by analogy with local image descriptors. Laptev (2005) extended the Harris and Förstner interest point operators from images to space-time, detecting points where the image values vary significantly in both space and time, with scales chosen in space and in time. Schuldt, Laptev and Caputo (2004) classified videos represented by such local space-time features with support vector machines and introduced the KTH dataset to evaluate them. Blank et al. (2005) took a global view instead, treating a person’s silhouettes over time as a 3D shape in the space-time volume.

Dense trajectories (Wang et al., 2011) replaced sparse interest points by points sampled densely and tracked with optical flow, described by shape, appearance and motion histograms along each track; Wang and Schmid (2013) improved them by estimating and removing camera motion. Improved dense trajectories were the hand-crafted baseline that early deep video networks were measured against.

Convolutional networks (2014–2020)

Single frames and early fusion (2014). Karpathy et al. (2014) trained CNNs on Sports-1M, one million YouTube videos, and compared a single-frame network with networks that fuse frames early (in the first layer), late (two single-frame towers combined at the top) or slowly (temporal convolutions over the first layers). Slow fusion was best but only slightly better than single frames; fine-tuning the top three layers of the Sports-1M network gave 65.4% three-fold accuracy on UCF101, against 41.3% for the same network trained from scratch (their Table 3).

Two-stream networks (2014). Simonyan and Zisserman (2014) split the problem into an appearance stream and an optical-flow motion stream, described above; with explicit motion as input, ConvNets trained on the small datasets of the time became competitive with improved dense trajectories (their Table 4).

3D convolutional networks (2015). C3D (Tran et al., 2015) learned motion directly from pixels with 3D convolutions over 16-frame clips, using $3 \times 3 \times 3$ kernels in all layers, the best of the temporal kernel depths the authors compared on UCF101 split 1, and trained on Sports-1M to produce generic video features.

Temporal segment networks (2016). Wang et al. (2016) addressed the short reach of networks that see only a few consecutive frames: a temporal segment network divides the video into segments, takes a short snippet from each, and combines the snippet predictions, so that sparse sampling covers the whole video at the cost of a few frames.

I3D and Kinetics (2017). Carreira and Zisserman (2017) evaluated architectures on the new Kinetics dataset (Kay et al., 2017), which they describe as two orders of magnitude larger than HMDB51 and UCF101, and introduced I3D, which inflates the 2D filters and pooling kernels of an ImageNet network into 3D so that a 3D network can be initialized from image weights. With Kinetics pretraining, the two-stream I3D reached 97.9% on UCF101 and 80.2% on HMDB51, averaged over three splits (their Table 4). Pretraining on Kinetics and fine-tuning elsewhere became a standard recipe, as ImageNet pretraining had for images (transfer learning).

SlowFast (2019). Feichtenhofer et al. (2019) used two 3D-convolutional pathways: a Slow pathway at a low frame rate for spatial semantics and a lightweight Fast pathway at a frame rate typically 8 times higher, for motion, taking about 20% of the computation. On Kinetics-400, their SlowFast 8×8 ResNet-50 reached 77.0% top-1 at 65.7 GFLOPs per view × 30 views (10 clips × 3 crops) (their Table 2, which does not name the evaluation split).

Transformers (2021–)

After the vision transformer, video models replaced convolutions by self-attention over space-time patches. The difficulty is the number of tokens: the cost of attention between all space-time patches grows with the square of their number. TimeSformer (Bertasius, Wang & Torresani, 2021) compared several attention schemes and found “divided” attention, a temporal attention step followed by a spatial one in each block, the most accurate of those it tried. ViViT (Arnab et al., 2021) embedded non-overlapping space-time tubelets, linear projections equivalent to a 3D convolution, and proposed variants that factorize space and time. Both started from image transformers pretrained on ImageNet or larger image datasets, since transformers were known to need large training sets (Arnab et al., 2021).

Self-supervised and foundation-scale pretraining (2022–)

Labels limit how far supervised pretraining can scale. VideoMAE (Tong et al., 2022) applied the masked autoencoder to video with “tube” masking, in which the same spatial patches are hidden in every frame, so that, for content that barely moves, the model cannot copy a hidden patch from a neighboring frame. Pretrained only on the videos of the target dataset, without labels or extra data, a ViT-L reached 75.4% top-1 on Something-Something V2 at 1,436 GFLOPs per view × 1 clip × 3 crops (their Table 6), and a ViT-H fine-tuned at $320 \times 320$ reached 87.4% top-1 on Kinetics-400 at 7,397 GFLOPs per view × 4 clips × 3 crops (their Table 7). Other self-supervised and contrastive methods learn from the video itself or from paired text: video–language models pretrained on clips with captions, narrations or speech transcripts, in the manner of CLIP and vision–language pretraining, are used to retrieve videos from text and to recognize actions named only at test time. Ego4D, for example, provides dense, timestamped narrations of the camera wearer’s activity (Grauman et al., 2022).

Datasets

The datasets grew in size, realism and temporal difficulty. Sizes are as the papers print them.

Dataset	Year	Source	Content and size as printed
KTH	2004	Recorded	6 actions, 25 people, 4 scenarios, 2,391 sequences
Weizmann	2005	Recorded	9 actions by 9 people, 81 sequences at $180 \times 144$, 25 fps
HMDB51	2011	Movies, web video	51 actions, 6,766 clips, at least 101 per class
UCF101	2012	YouTube	101 actions, 13,320 clips, 27 hours
Sports-1M	2014	YouTube	1 million videos, 487 sports classes, labels from video metadata
ActivityNet	2015	YouTube	203 activities, 137 untrimmed videos per class on average, 849 hours
Kinetics-400	2017	YouTube	400 actions, 306,245 clips of about 10 s, each from a different video
Something-Something (first release)	2017	Crowd-acted	174 template classes, 108,499 videos, 4.03 s on average
AVA	2018	Movies	80 atomic actions, 437 15-minute clips, 1.59 million labels at 1 Hz
Ego4D	2022	Head-worn cameras	3,670 hours from 931 camera wearers, 74 locations, 9 countries

KTH and Weizmann were recorded in controlled conditions (Li, Li & Vasconcelos, 2018). HMDB51 and UCF101 brought realistic footage and remained standard benchmarks into the deep-learning era. Sports-1M was large enough to train deep networks but weakly labeled: its labels come automatically from text metadata, so some are wrong, and many frames of a correctly labeled video show something else (Karpathy et al., 2014). Kinetics (Kay et al., 2017) combined size with labels confirmed by several crowd workers per clip and became a standard pretraining and benchmark set, later extended to 600 and 700 classes (as summarized by Smaira et al., 2020; see Kinetics). Something-Something was collected so that classes differ in what happens rather than in the objects or scenes, including pretended actions. ActivityNet moved to untrimmed videos and temporal localization; AVA to dense, per-person, multi-label annotation; Ego4D to long, first-person recordings of daily life.

Datasets distributed as lists of YouTube links shrink as videos are deleted or made private. Smaira et al. (2020, Table 2) report that, as of 14 October 2020, 220,033 of the 246,245 Kinetics-400 training clips (89%) were still available, and 35,400 of the 40,000 test clips. Results on such a dataset depend on which copy a group downloaded and when.

Evaluation

• Classification is scored by top-1 and top-5 accuracy: the fraction of test videos whose true class is the highest-scoring class or among the five highest. Sports-1M, which has multiple labels per video, used hit@k, the fraction of videos with at least one true label among the top $k$ predictions (Karpathy et al., 2014). UCF101 and HMDB51 report the mean accuracy over their three splits, and multi-label datasets report mean average precision.
• Temporal localization is scored by mean average precision at a temporal IoU threshold (mAP@tIoU): a predicted segment counts as correct if its intersection over union with a ground-truth segment along the time axis exceeds the threshold. ActivityNet reported several thresholds, from 0.1 to 0.5, because the boundaries of an activity are ambiguous (Heilbron et al., 2015).
• Spatio-temporal detection on AVA is scored by frame-mAP: average precision per class over annotated keyframes, with a box IoU threshold of 0.5 following the PASCAL VOC protocol (Gu et al., 2018).

Test-time views and cost. A video-level prediction usually averages several views, each a temporal clip with a spatial crop. Papers differ in how many they use, so accuracy alone does not compare models; SlowFast’s authors therefore reported cost as GFLOPs per view × number of views (Feichtenhofer et al., 2019), a convention that later tables such as VideoMAE’s follow (temporal modeling covers how views are sampled).

Examples

A frame-level classifier cannot see direction. The example generates $24 \times 24$ clips of 8 frames with a bright disc on a noisy background in three classes: moving left, moving right (2 pixels per frame) and still. The disc wraps around the frame edges and starts at a uniformly random position, so in every frame its position is uniformly distributed for all three classes: the distribution of a single frame is identical across classes by construction. A linear classifier (ridge regression onto one-hot labels) is trained on individual frames and tested on one random frame per clip; a single temporal feature, the mean frame-to-frame displacement of the disc’s horizontal centroid, is classified with two fixed thresholds.

import numpy as np

T, S, R, SPEED = 8, 24, 3.0, 2          # frames, frame size, disc radius, pixels/frame
CLASSES = ["left", "right", "still"]
VEL = {0: -SPEED, 1: SPEED, 2: 0}

def make_clips(n_per_class, rng):
    """Clips of a bright disc on a noisy background, shape (N, T, S, S).
    The disc wraps around the frame edges and starts at a uniformly random
    position, so in every frame its position is uniform for all classes."""
    yy, xx = np.mgrid[0:S, 0:S]
    clips, labels = [], []
    for c in range(3):
        for _ in range(n_per_class):
            x0, y0 = rng.uniform(0, S, 2)
            clip = np.empty((T, S, S))
            for t in range(T):
                cx = (x0 + VEL[c] * t) % S
                dx = (xx - cx + S / 2) % S - S / 2          # wrapped distance
                dy = (yy - y0 + S / 2) % S - S / 2
                clip[t] = (dx**2 + dy**2 <= R**2).astype(float)
            clips.append(clip + rng.normal(0, 0.2, clip.shape))
            labels.append(c)
    return np.array(clips), np.array(labels)

def ridge_fit_predict(Xtr, ytr, Xte, lam=10.0):
    """Linear classifier: ridge regression onto one-hot labels, argmax."""
    mu = Xtr.mean(0)
    A, B = Xtr - mu, np.eye(3)[ytr]
    W = np.linalg.solve(A.T @ A + lam * np.eye(A.shape[1]), A.T @ (B - B.mean(0)))
    return np.argmax((Xte - mu) @ W, axis=1)

def centroid_x(frames):
    """Horizontal disc position per frame as a circular mean (handles wrapping)."""
    w = np.clip(frames - 0.5, 0, None).sum(axis=-2)              # column mass
    ang = 2 * np.pi * np.arange(S) / S
    return np.angle((w * np.exp(1j * ang)).sum(-1)) * S / (2 * np.pi)

def displacement(clips):
    """Mean frame-to-frame horizontal displacement, wrapped to [-S/2, S/2)."""
    d = np.diff(centroid_x(clips), axis=1)
    return ((d + S / 2) % S - S / 2).mean(axis=1)

def run(seed):
    rng = np.random.default_rng(seed)
    Xtr, ytr = make_clips(200, rng)
    Xte, yte = make_clips(200, rng)
    acc = {}
    # 1. a frame-level classifier: train on every frame, test on random frames
    t = rng.integers(0, T, len(Xte))
    pred = ridge_fit_predict(Xtr.reshape(-1, S * S), np.repeat(ytr, T),
                             Xte[np.arange(len(Xte)), t].reshape(-1, S * S))
    acc["single frame, linear"] = np.mean(pred == yte)
    # 2. one temporal feature: mean centroid displacement, two thresholds
    d = displacement(Xte)
    pred = np.where(d < -SPEED / 2, 0, np.where(d > SPEED / 2, 1, 2))
    acc["centroid displacement"] = np.mean(pred == yte)
    return acc, Xte, yte

acc, Xte, yte = run(0)
# Per-frame statistics by class: brightness and disc position look the same;
# only the displacement between frames differs.
for c, name in enumerate(CLASSES):
    f = Xte[yte == c]
    x = centroid_x(f) % S
    print(f"{name:6s} mean pixel {f.mean():.3f}  disc x: mean {x.mean():5.2f}, "
          f"std {x.std():.2f}  displacement {displacement(f).mean():+.2f} px/frame")
print("seed 0 test accuracy (chance = 0.333):")
for k, v in acc.items():
    print(f"  {k:24s} {v:.3f}")
runs = [run(s)[0] for s in range(10)]
print("seeds 0-9, min-max:")
for k in acc:
    vals = [r[k] for r in runs]
    print(f"  {k:24s} {min(vals):.3f}-{max(vals):.3f}")

Output:

left   mean pixel 0.049  disc x: mean 12.15, std 6.86  displacement -2.00 px/frame
right  mean pixel 0.049  disc x: mean 11.73, std 6.96  displacement +2.00 px/frame
still  mean pixel 0.049  disc x: mean 12.42, std 6.80  displacement +0.00 px/frame
seed 0 test accuracy (chance = 0.333):
  single frame, linear     0.345
  centroid displacement    1.000
seeds 0-9, min-max:
  single frame, linear     0.303-0.375
  centroid displacement    1.000-1.000

Per frame, the three classes look the same: equal mean brightness, and a disc position with mean near 12 and standard deviation near $24/\sqrt{12} \approx 6.93$, the values of a uniform position on a 24-pixel circle. The frame-level classifier is at chance, between 30.3% and 37.5% over seeds 0 to 9, because there is nothing in a single frame to learn; in our reasoning, a deeper frame-level network would do no better, since the frame distributions themselves are identical. One number computed from consecutive frames separates the classes perfectly in every seed. Real datasets sit between this example and its opposite: Kinetics classes are often recognizable from one frame, while Something-Something classes such as moving something left or right resemble this example. The script runs in a few seconds.

Challenges and Open Problems

The following are, in our assessment, the main open questions as of October 2026.

• Temporal reasoning versus scene bias. Benchmarks reward whatever cue separates their classes, and in many datasets that cue is static. Li, Li and Vasconcelos (2018, Table 1) measured the representation bias of action datasets with a linear classifier on frozen, per-frame features of networks trained for objects, scenes or people, and found strong static biases in HMDB51, UCF101, ActivityNet and Kinetics, with Kinetics, the largest, the most biased toward objects; KTH, recorded in controlled conditions, had much lower static bias. TimeSformer gives a direct measurement on the validation sets of two datasets, with its default 1 clip × 3 crops (Bertasius et al., 2021, Table 1): with attention over space only, which the authors describe as having no temporal modeling, it reached 76.9% on Kinetics-400 against 78.0% with divided space-time attention, but only 36.6% on Something-Something V2 against 59.5%. A high Kinetics score is therefore weak evidence of temporal modeling.
• Long videos. Most models classify short clips: Kinetics clips last about 10 seconds, and TimeSformer’s default input is 8 frames “sampled at a rate of 1/32” (Bertasius et al., 2021), which we read as a stride of 32, spanning 256 source frames. Understanding a cooking procedure, a film or hours of egocentric recording such as Ego4D’s requires memory and reasoning over minutes to hours, where both the cost of attention and the scarcity of long annotated videos bite.
• Compute. The cost of a prediction is the cost per view times the number of views. In our arithmetic from the printed tables, SlowFast 16×8 ResNet-101 with non-local blocks costs 234 GFLOPs per view × 30 views = 7,020 GFLOPs per Kinetics-400 video (Feichtenhofer et al., 2019, Table 2), and the VideoMAE ViT-H at $320 \times 320$ costs 7,397 GFLOPs per view × 12 views = 88,764 GFLOPs (Tong et al., 2022, Table 7). Efficient architectures, fewer views and frame selection are active responses.
• Annotation. Temporal boundaries are ambiguous, per-person labels are expensive, and labels derived from metadata are noisy. Self-supervised pretraining and narrations reduce the need for labels but do not remove the need for well-designed evaluation sets.

Common Misconceptions

• “A video model is an image model applied to every frame.” Averaging per-frame predictions can do well on scene-biased datasets, but it cannot separate classes that differ only in order or direction, as the example and Something-Something show.
• “Accuracies from different papers are directly comparable.” They depend on the number of test-time views, input resolution, pretraining data and, for YouTube datasets, which surviving copy of the dataset was used.

Where It Is Used

Video understanding is applied to content search and recommendation, sports analysis, surveillance and safety monitoring, human–robot interaction and assistive systems, and the analysis of instructional and egocentric video. It shares methods with object tracking, multi-object tracking and motion estimation.