Temporal Modeling

Temporal modeling is the part of a video model that combines information across frames. An image network applied to each frame sees appearance; telling an opening door from a closing one needs their order. Video models pool per-frame features, carry a recurrent state, convolve over time, take optical flow as input, attend across frames, or shift features between neighboring frames. This article also defines the frame-sampling vocabulary that the model articles use.

Definition

A temporal model maps a sequence of frames or per-frame features to an output that can depend on how the frames relate in time. Three notions describe it:

• Temporal order. A model is order-invariant if permuting the frames leaves its output unchanged, and order-sensitive otherwise. Average and max pooling over frames are invariant; recurrence, temporal convolution, and attention with a positional encoding are sensitive.
• Temporal receptive field. The set of input frames on which a unit depends, the time axis of the receptive field. A unit cannot relate events farther apart than this field.
• Temporal resolution. The spacing of the frames the model reads. Motion that completes between two sampled frames is invisible to it.

Intuition

A clip of a growing square, played backwards, shows a shrinking square. Both contain exactly the same frames, so any model that summarizes a bag of frame features, however good, gives both the same output. Telling them apart requires comparing frames in sequence: subtracting neighbors, carrying a state, or tagging frames with their time.

Formal Definition

A clip is a tensor $T \times H \times W \times C$ ($T$ frames); PyTorch’s Conv3d takes it as $(N, C, T, H, W)$, time being the axis its documentation calls depth $D$. A 2D convolutional network $\phi$ maps frame $t$ to a feature vector $\mathbf{f}_t = \phi(\mathbf{x}_t) \in \mathbb{R}^d$ at a cost of $c$ operations per frame.

Frame pooling computes

$$\mathbf{z} = \frac{1}{T} \sum_{t=1}^{T} \mathbf{f}_t \quad \text{or} \quad z_j = \max_t f_{t,j},$$

and for every permutation $\pi$ of $\{1, \dots, T\}$, $\mathbf{z}(\mathbf{f}_{\pi(1)}, \dots, \mathbf{f}_{\pi(T)}) = \mathbf{z}(\mathbf{f}_1, \dots, \mathbf{f}_T)$.

Temporal convolution with kernel size $k = 2r + 1$ computes $\mathbf{y}_t = \sum_{j=-r}^{r} W_j \mathbf{f}_{t+j}$ with $W_j \in \mathbb{R}^{d' \times d}$. Reversing time is equivalent to flipping the kernel to $W_{-j}$, so with a symmetric kernel ($W_j = W_{-j}$), stride 1 and padding that treats both ends alike (none, zeros or circular), the reversed clip yields the reversed outputs, which average pooling cannot tell apart (our derivation; the example shows it). Such a kernel is still sensitive to other permutations. $L$ stacked layers with stride 1 and no dilation give a temporal receptive field of $1 + L(k - 1)$ frames (our derivation). A 3D convolution applies a $t \times h \times w$ kernel to the clip, with $t$ times the weights and operations per output position of an $h \times w$ kernel.

Recurrence updates a state, $\mathbf{h}_t = \sigma(A \mathbf{h}_{t-1} + B \mathbf{f}_t)$, that depends on all frames up to $t$, in order; an LSTM adds gates.

Attention over time over frame or patch tokens is permutation-equivariant without position information (see self-attention), so, by our reasoning, its pooled output is invariant until an embedding per time step is added.

The Approaches

Frame pooling. Run an image network on each frame and average or max-pool the features or predictions over time (see pooling). It adds no parameters and reuses image pretraining but, as Carreira and Zisserman (2017) note, ignores temporal structure.

Fusion in early layers. Karpathy et al. (2014) fused 10 frames in the first-layer filters (early fusion), two frames 15 apart in the first fully connected layer (late fusion), or 10 frames gradually over the first three convolutional layers (slow fusion). Tran et al. (2015) point out that a 2D convolution over stacked frames collapses time after the first layer, whereas 3D convolution keeps a time axis.

Recurrence. Long-term recurrent convolutional networks (LRCN; Donahue et al., 2015) feed per-frame CNN features into a single-layer LSTM with 256 hidden units and average its per-step predictions. The state is causal with no fixed horizon, but frames are processed one after another, and I3D’s authors argued that an LSTM on top-layer features may miss fine low-level motion.

Temporal and 3D convolution. A 1D convolution over per-frame features adds local, order-sensitive mixing cheaply. 3D convolution mixes time on pixels and at every layer; C3D (Tran et al., 2015) learned it from scratch, and I3D inflates a pretrained 2D network’s filters (the 3D convolutional networks article covers both).

Optical flow as an input. The two-stream network (Simonyan & Zisserman, 2014) adds a second CNN whose input stacks the horizontal and vertical components of $L = 10$ consecutive optical-flow fields, $2L$ channels. Flow hands the network motion directly, but a separate estimator must compute it first, a cost FLOP counts can leave out (TSM’s Table 2 gives no FLOPs for its RGB+flow entries).

Space-time attention. TimeSformer (Bertasius et al., 2021) applies the transformer blocks of the Vision Transformer to the patches of all frames. For $T$ frames of $N$ patches (the paper writes $F$ for $T$), joint attention compares each patch with all $NT + 1$ tokens, the class token included; divided attention attends first across frames at the same patch position, then within the frame, with $N + T + 2$ comparisons.

Temporal shift. The temporal shift module (TSM; Lin et al., 2019) moves 1/8 of the channels of each frame’s features one step forward in time and 1/8 one step back (zero-filled at the ends) inside the residual branches of a 2D ResNet; the next 2D convolution then mixes neighboring frames. It adds no parameters or FLOPs, and each shifted block widens the temporal receptive field by 2 frames, like a temporal convolution of kernel size 3. Moving the data still adds some latency.

Approach	Extra parameters	Computation for $T$ frames	Temporal receptive field	Order
Frame pooling	none	$Tc$	whole clip, at the end only	invariant
Recurrence (CNN + LSTM)	$4h(d + h + 1)$ for an LSTM of width $h$	$Tc$ plus $T$ sequential state updates	all earlier frames	sensitive, causal
2D convolution on stacked frames (early fusion)	first layer × $T$	about $c$	the $T$ stacked frames, collapsed after layer 1	sensitive, fixed $T$
1D temporal convolution on features	$k d d'$ per layer	$Tc + T k d d'$	$1 + L(k-1)$ frames	sensitive; blind to reversal if the kernel is symmetric
3D convolution	$t$ × a 2D layer’s	about $t$ × a 2D layer’s per output frame	grows with depth and temporal stride	sensitive, local
Optical-flow stream	a second network	flow estimation plus a second network	$L$ flow fields per input	local direction of motion
Space-time attention	independent of $T$, except position embeddings	joint $O((NT)^2 D)$, divided $O(NT(N + T)D)$ per layer	whole clip in one layer	invariant without position embeddings
Temporal shift (TSM)	none	$Tc$ (no extra FLOPs; some data movement)	+2 frames per shifted block	sensitive

The table is our summary of the mechanisms. For printed figures, I3D’s Table 1 lists 9M parameters for its ConvNet+LSTM, 79M for its C3D-style 3D ConvNet, 12M for the two-stream network and 25M for two-stream I3D (Carreira & Zisserman, 2017).

Frame Sampling

A video holds far more frames than a model can read, so training and testing sample from it.

• Dense clips. $T$ frames at temporal stride $\tau$ span $T\tau$ source frames (the first and last are $(T-1)\tau$ apart). C3D’s kernel-depth study used non-overlapping 16-frame clips ($T = 16$, $\tau = 1$). I3D trained on 64-frame snippets of 25 fps video (2.56 s). TimeSformer samples 8 frames “at a rate of 1/32”, which we read as $\tau = 32$, spanning 256 frames. A small $\tau$ resolves fast motion; a large one covers more time.
• Sparse segment sampling. Temporal segment networks (TSN; Wang et al., 2016) divide a video into $K$ segments of equal duration ($K = 3$ in the paper), train on one short snippet drawn at random from each (an RGB frame or a stack of flow fields), and combine the snippets’ class scores by a consensus function; averaging gave the best two-stream accuracy on UCF101 split 1 (their Table 3). $K$ snippets cover the whole video, but detail between them is lost, and averaging discards their order.
• Test-time views. Models average predictions over several clips and spatial crops per video, the views (clips × crops), so the cost per video is GFLOPs per view × views. The two-stream network averaged 25 frames, each with the four corner and center crops and their flips (Simonyan & Zisserman, 2014); TimeSformer uses 1 clip × 3 crops, whereas, its authors note, most previous methods used 10 × 3 = 30 views.

SlowFast combines two frame rates in one network, a slow and a fast pathway (Feichtenhofer et al., 2019).

Examples

The NumPy code below builds noisy clips of a square growing from side 4 to 16 pixels over 12 frames; the other class is each clip played backwards, so both classes contain identical frames. A frozen random 2D network (eight $5 \times 5$ filters, ReLU, global average pooling) gives 8 features per frame, and a linear classifier is fitted on each temporal model’s output.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

T, S = 12, 24                                     # frames per clip, frame size in pixels
rng = np.random.default_rng(0)
FILTERS = rng.standard_normal((25, 8))            # a frozen random 2D "CNN": eight 5x5 filters

def frame_features(frames):
    """Per-frame features: 5x5 convolution, ReLU, global average pooling -> 8 numbers per frame."""
    patches = sliding_window_view(frames, (5, 5), axis=(-2, -1))
    patches = patches.reshape(*frames.shape[:-2], -1, 25)
    return np.maximum(patches @ FILTERS, 0).mean(axis=-2)

def square_clip(rng):
    """A bright square that grows from side 4 to 16 pixels over T frames, plus pixel noise."""
    cy, cx = rng.integers(9, S - 9, size=2)
    clip = 0.1 * rng.standard_normal((T, S, S))
    for t, side in enumerate(np.linspace(4, 16, T).round().astype(int)):
        y0, x0 = cy - side // 2, cx - side // 2
        clip[t, max(y0, 0):y0 + side, max(x0, 0):x0 + side] += 1.0
    return clip

def dataset(n_pairs, rng):
    """Class 0: the square grows. Class 1: the same clip played backwards (it shrinks)."""
    grow = np.stack([square_clip(rng) for _ in range(n_pairs)])
    clips = np.concatenate([grow, grow[:, ::-1]])         # identical frames, only the order differs
    labels = np.repeat([0, 1], n_pairs)
    return frame_features(clips), labels                  # (2 n_pairs, T, 8) feature sequences

def ridge_accuracy(f_train, y_train, f_test, y_test, lam=1e-3):
    """Least-squares linear classifier on standardized features; returns test accuracy."""
    mu, sd = f_train.mean(0), f_train.std(0) + 1e-9
    A = np.c_[(f_train - mu) / sd, np.ones(len(f_train))]
    w = np.linalg.solve(A.T @ A + lam * np.eye(A.shape[1]), A.T @ (2 * y_train - 1))
    pred = np.c_[(f_test - mu) / sd, np.ones(len(f_test))] @ w > 0
    return np.mean(pred == y_test)

def softmax(z):
    e = np.exp(z - z.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)

d, h = 8, 32
W_q, W_k, W_v = (rng.standard_normal((d, d)) / np.sqrt(d) for _ in range(3))
A_rec = rng.standard_normal((h, h)); A_rec *= 0.9 / np.abs(np.linalg.eigvals(A_rec)).max()
B_rec = rng.standard_normal((h, d))

def attention_pool(F):
    """One self-attention layer over the T frame tokens (random weights, no position), average-pooled."""
    X = (F - F.mean((0, 1))) / F.std((0, 1))
    att = softmax((X @ W_q) @ (X @ W_k).swapaxes(1, 2) / np.sqrt(d))
    return (att @ (X @ W_v)).mean(axis=1)

def recurrent(F):
    """Untrained recurrent update h_t = tanh(A h_{t-1} + B f_t); the readout sees h_T."""
    X = (F - F.mean((0, 1))) / F.std((0, 1))
    state = np.zeros((len(F), h))
    for t in range(T):
        state = np.tanh(state @ A_rec.T + X[:, t] @ B_rec.T)
    return state

models = {
    "average pooling":                 lambda F: F.mean(1),
    "max pooling":                     lambda F: F.max(1),
    "temporal conv [1, 1] + avg pool": lambda F: (F[:, 1:] + F[:, :-1]).mean(1),
    "temporal conv [-1, 1] + avg pool": lambda F: (F[:, 1:] - F[:, :-1]).mean(1),
    "attention, no position":          attention_pool,
    "recurrent state h_T":             recurrent,
}
F_train, y_train = dataset(200, np.random.default_rng(1))
F_test, y_test = dataset(200, np.random.default_rng(2))
n = len(F_test) // 2
for name, g in models.items():
    z_train, z_test = g(F_train), g(F_test)
    gap = np.abs(z_test[:n] - z_test[n:]).max()          # clip vs its own reversal
    acc = ridge_accuracy(z_train, y_train, z_test, y_test)
    print(f"{name:33s} max |z(clip) - z(reversed)| = {gap:.3f}   test accuracy {acc:.3f}")

average pooling                   max |z(clip) - z(reversed)| = 0.000   test accuracy 0.500
max pooling                       max |z(clip) - z(reversed)| = 0.000   test accuracy 0.500
temporal conv [1, 1] + avg pool   max |z(clip) - z(reversed)| = 0.000   test accuracy 0.500
temporal conv [-1, 1] + avg pool  max |z(clip) - z(reversed)| = 0.637   test accuracy 1.000
attention, no position            max |z(clip) - z(reversed)| = 0.000   test accuracy 0.500
recurrent state h_T               max |z(clip) - z(reversed)| = 2.000   test accuracy 1.000

Pooling gives each clip and its reversal the same vector, up to rounding, so each test pair gets one prediction: exactly 50%, whatever the features. The symmetric kernel $[1, 1]$ and attention without position are just as blind. The difference kernel $[-1, 1]$ followed by averaging yields $(\mathbf{f}_T - \mathbf{f}_1)/(T-1)$, which changes sign under reversal, and even an untrained recurrent state separates the classes once a readout is fitted.

The second part, run after the first, hides a brief event in a long video: in 96 frames of noise, a square appears, grows (or shrinks) over 8 frames, and vanishes. Every sampling strategy feeds one model: a kernel-2 temporal convolution with 64 random channels over consecutive samples within each clip or snippet, ReLU, averaging and a fitted readout.

N, E = 96, 8                                      # video length and event length, in frames
W_pair = np.random.default_rng(3).standard_normal((16, 64)) / 4   # temporal conv, kernel 2, 64 channels

def long_video(rng, grow):
    """Noise for N frames; once, for E frames, a square appears, grows (or shrinks), and vanishes."""
    video = 0.1 * rng.standard_normal((N, S, S))
    start = rng.integers(0, N - E + 1)
    cy, cx = rng.integers(9, S - 9, size=2)
    sides = np.linspace(4, 16, E).round().astype(int)
    for t, side in enumerate(sides if grow else sides[::-1]):
        y0, x0 = cy - side // 2, cx - side // 2
        video[start + t, max(y0, 0):y0 + side, max(x0, 0):x0 + side] += 1.0
    return video

def clip_model(F, groups):
    """Temporal conv (kernel 2) over consecutive samples inside each group, ReLU, average pooling."""
    out = []
    for g in groups:                              # g: sorted frame indices of one clip or snippet
        pairs = np.c_[F[g[:-1]], F[g[1:]] - F[g[:-1]]]
        out.append(np.maximum(pairs @ W_pair, 0))
    return np.concatenate(out).mean(0)

def dense(rng, T=8, tau=1, views=1):
    """`views` clips of T frames at stride tau, random start or evenly spaced."""
    span = (T - 1) * tau + 1
    starts = [rng.integers(0, N - span + 1)] if views == 1 else np.linspace(0, N - span, views).astype(int)
    return [s + tau * np.arange(T) for s in starts]

def tsn(rng, K=4, L=2):
    """TSN-style: K equal segments, one random snippet of L consecutive frames from each."""
    seg = N // K
    return [k * seg + rng.integers(0, seg - L + 1) + np.arange(L) for k in range(K)]

strategies = {                                    # name: (sampler, frames read per video)
    "dense clip T=8, tau=1":       (lambda r: dense(r), 8),
    "strided clip T=8, tau=12":    (lambda r: dense(r, tau=12), 8),
    "TSN K=4 snippets of 2":       (lambda r: tsn(r), 8),
    "TSN K=8 snippets of 2":       (lambda r: tsn(r, K=8), 16),
    "dense T=8, tau=1, 10 views":  (lambda r: dense(r, views=10), 80),
}
results = {name: [] for name in strategies}
for seed in range(5):
    rng = np.random.default_rng(100 + seed)
    labels = np.tile([0, 1], 300)                 # 600 videos: grow (0) / shrink (1), alternating
    feats = [frame_features(long_video(rng, grow=(y == 0))) for y in labels]
    for name, (sampler, _) in strategies.items():
        z = np.array([clip_model(F, sampler(rng)) for F in feats])
        results[name].append(ridge_accuracy(z[:300], labels[:300], z[300:], labels[300:]))
for name, (_, cost) in strategies.items():
    acc = np.array(results[name])
    print(f"{name:27s} frames/video {cost:3d}   accuracy over seeds 100-104: "
          f"mean {acc.mean():.3f}, range {acc.min():.3f}-{acc.max():.3f}")

dense clip T=8, tau=1       frames/video   8   accuracy over seeds 100-104: mean 0.578, range 0.563-0.600
strided clip T=8, tau=12    frames/video   8   accuracy over seeds 100-104: mean 0.525, range 0.510-0.563
TSN K=4 snippets of 2       frames/video   8   accuracy over seeds 100-104: mean 0.667, range 0.650-0.687
TSN K=8 snippets of 2       frames/video  16   accuracy over seeds 100-104: mean 0.857, range 0.823-0.893
dense T=8, tau=1, 10 views  frames/video  80   accuracy over seeds 100-104: mean 1.000, range 1.000-1.000

With 8 frames, the dense clip resolves the event but usually misses it: by our count it touches the event in at most 15 of its 89 start positions, about 16% of videos on average, which caps accuracy near 58%. The strided clip’s first and last samples are 84 frames apart, but its samples are 12 frames apart, more than the event lasts, so it never sees two event frames and stays near chance. Four 2-frame snippets combine coverage with local detail and do best for 8 frames; eight reach 0.86. Ten evenly spaced dense views always catch the event, at ten times the cost.

The last part, run with PyTorch 2.14.1 after the first block, trains a small attention model on the reversal dataset’s frame features, with and without a learned position embedding.

import torch
import torch.nn as nn

torch.manual_seed(0)
torch.set_num_threads(1)
print("torch", torch.__version__)

class FrameAttention(nn.Module):
    """Frame features -> embedding (+ learned position embedding) -> one self-attention block -> mean -> class."""
    def __init__(self, d_in=8, d=32, T=12, position=True):
        super().__init__()
        self.embed = nn.Linear(d_in, d)
        self.pos = nn.Parameter(0.02 * torch.randn(T, d)) if position else None
        self.attn = nn.MultiheadAttention(d, num_heads=4, batch_first=True)
        self.mlp = nn.Sequential(nn.LayerNorm(d), nn.Linear(d, d), nn.GELU())
        self.head = nn.Linear(d, 2)

    def forward(self, f):                                 # f: (batch, T, d_in)
        x = self.embed(f)
        if self.pos is not None:
            x = x + self.pos
        x = x + self.attn(x, x, x, need_weights=False)[0]
        x = x + self.mlp(x)
        return self.head(x.mean(dim=1))

mu, sd = F_train.mean((0, 1)), F_train.std((0, 1))
to_t = lambda F: torch.tensor((F - mu) / sd, dtype=torch.float32)
Xtr, Xte = to_t(F_train), to_t(F_test)
ytr, yte = torch.tensor(y_train), torch.tensor(y_test)
for position in (False, True):
    torch.manual_seed(0)
    model = FrameAttention(position=position)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    for step in range(300):                               # full-batch training on 400 clips
        loss = nn.functional.cross_entropy(model(Xtr), ytr)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        logits = model(Xte)
        acc = (logits.argmax(1) == yte).float().mean().item()
        gap = (logits[:n] - logits[n:]).abs().max().item()   # clip vs its reversal
    print(f"position embedding {str(position):5s}: train loss {loss.item():.4f}, "
          f"max |logit(clip) - logit(reversed)| = {gap:.3f}, test accuracy {acc:.3f}")

torch 2.14.1
position embedding False: train loss 0.6931, max |logit(clip) - logit(reversed)| = 0.000, test accuracy 0.500
position embedding True : train loss 0.0000, max |logit(clip) - logit(reversed)| = 14.343, test accuracy 1.000

Without position, training loss stays at $\ln 2 \approx 0.6931$, a coin flip; with a learned embedding per time step the model fits and generalizes. Two runs gave identical output.

Properties

Order-blind models fail on order-defined classes. By the invariance above, no classifier on pooled frame features separates classes that differ only in order; TSN’s average consensus is likewise order-blind across segments, though flow snippets carry local direction.

Static and scene bias. Many action classes can be recognized from one frame, so benchmarks reward appearance. TSM’s authors group Something-Something with datasets focused on temporal relationships, and Kinetics with those less sensitive to them. In TSM’s Table 1 (ResNet-50, 8 frames, 10 clips per video on Kinetics and 2 on Something-Something), adding temporal shifts to a TSN baseline raised Kinetics top-1 accuracy from 70.6% to 74.1%, but Something-Something V1 from 20.5% to 47.3%. TimeSformer with space-only attention reached 76.9% on Kinetics-400 but 36.6% on Something-Something V2 (validation, video-level accuracy), against 78.0% and 59.5% with divided space-time attention (Bertasius et al., 2021, Table 1). High scores on appearance-heavy datasets are weak evidence of temporal modeling.

Common Misconceptions

• “More frames means more temporal modeling.” Averaging 64 frames still yields a bag of frames.
• “Attention models time by itself.” Without position embeddings, TimeSformer fell from 59.5% to 45.8% on Something-Something V2, but only from 78.0% to 75.4% on Kinetics-400 (Bertasius et al., 2021, Table 4).
• “A 3D network sees the whole clip.” Its temporal receptive field grows with depth and temporal stride and can be shorter than the clip; no model sees beyond the sampled span.

Where It Is Used

Two-stream networks, 3D convolutional networks, temporal segment networks, SlowFast and video transformers each combine the mechanisms above, and action recognition, temporal action localization and spatio-temporal action detection build on them (see video understanding).