Video Transformers

A video transformer classifies a clip by cutting it into space-time patches, treating each patch as a token and passing the tokens through transformer blocks, as the Vision Transformer (ViT) does for one image. In one layer, attention can relate a patch in the first frame to one in the last, whereas a 3D convolution widens its temporal reach a few frames per layer. Four models published in 2021–2022 show the main patterns: TimeSformer factorizes attention into a time step and a space step, ViViT compares four factorizations and a 3D patch embedding, MViT pools the tokens into a multiscale hierarchy, and Video Swin Transformer restricts attention to local 3D windows. Temporal modeling places attention among the other ways of combining frames, and action recognition covers the task and its evaluation pitfalls.

Core Idea

Tokens. A clip $T \times H \times W \times C$ becomes a sequence in one of two ways. Frame patches cut each frame into $N = HW/P^2$ patches of $P \times P$ pixels, as in ViT, giving $NT$ tokens. Tubelets cut the clip into non-overlapping $t \times h \times w$ boxes and project each to a token, giving $\lfloor T/t \rfloor \lfloor H/h \rfloor \lfloor W/w \rfloor$ tokens, so a tubelet already mixes $t$ frames before any attention. Both are linear maps of non-overlapping blocks, so each equals a convolution whose stride equals its kernel, with no padding: $1 \times P \times P$ for frame patches and $t \times h \times w$ for tubelets, the condition the 3D convolutional networks article states. The first example below checks it.

Position. Self-attention is permutation-equivariant, so each token must carry where and when it is. TimeSformer adds a learned embedding per space-time position; ViViT initializes its video embeddings by repeating an image model’s spatial embeddings over time; Video Swin adds a learned relative position bias inside each window (see positional encoding).

Cost. Joint attention over all $NT$ tokens compares every pair, $O((NT)^2 D)$ operations per layer for width $D$ , and storing the attention weights takes $(NT)^2$ numbers per head. With ViT-B on $224 \times 224$ frames ( $N = 196$ ), 8 frames give 1,568 tokens and 96 frames 18,816.

Data and initialization. Transformers lack the locality built into convolutions, and ViT trailed CNNs unless pretrained on large image sets. Video datasets are smaller: ViViT’s authors note that even Kinetics has several orders of magnitude fewer labelled examples than the image datasets ViT was trained on (Arnab et al., 2021, §3.4). TimeSformer, ViViT and Video Swin therefore start from image models pretrained on ImageNet-1K, ImageNet-21K or JFT, and add or reshape only the parts that video needs. MViT showed that a multiscale design could instead be trained from scratch on Kinetics.

TimeSformer

TimeSformer (Bertasius, Wang & Torresani, 2021) applies ViT-Base to frame patches of $16 \times 16$ pixels and compares five attention schemes, drawn as blocks in its Figure 1 and as attention neighborhoods in Figure 2: space-only (each patch attends within its own frame), joint space-time (all patches of all frames), divided space-time (time, then space), sparse local-global (a local $T \times H/2 \times W/2$ neighborhood, then a global one with a stride of 2 patches in time and space) and axial (time, width, height in turn).

Divided attention. Writing $T$ for the paper’s $F$ , patch $(p, t)$ is embedded as $\mathbf{z}^{(0)}_{(p,t)} = E\mathbf{x}_{(p,t)} + \mathbf{e}^{pos}_{(p,t)}$ (Eq. 1), and a class token $\mathbf{z}^{(0)}_{(0,0)}$ is prepended. In block $\ell$ , head $a$ , joint attention (Eq. 5) gives patch $(p, t)$ weights over the class token and all $NT$ patches. Divided attention first compares it with the patches at the same position in every frame (Eq. 11):

\boldsymbol{\alpha}^{(\ell,a)\,\text{time}}_{(p,t)} = \operatorname{SM}\!\left( \frac{\mathbf{q}^{(\ell,a)\top}_{(p,t)}}{\sqrt{D_h}} \cdot \left[ \mathbf{k}^{(\ell,a)}_{(0,0)} \;\; \left\{ \mathbf{k}^{(\ell,a)}_{(p,t')} \right\}_{t'=1,\dots,T} \right] \right),

forms the weighted sum of values (Eq. 7) and adds it to the input through the output projection (Eq. 8), $\mathbf{z}'^{(\ell)}_{(p,t)} = W_O [\mathbf{s}^{(\ell,1)}_{(p,t)}; \dots; \mathbf{s}^{(\ell,A)}_{(p,t)}] + \mathbf{z}^{(\ell-1)}_{(p,t)}$ . That result then computes new queries, keys and values for spatial attention within frame $t$ (Eq. 6):

\boldsymbol{\alpha}^{(\ell,a)\,\text{space}}_{(p,t)} = \operatorname{SM}\!\left( \frac{\mathbf{q}^{(\ell,a)\top}_{(p,t)}}{\sqrt{D_h}} \cdot \left[ \mathbf{k}^{(\ell,a)}_{(0,0)} \;\; \left\{ \mathbf{k}^{(\ell,a)}_{(p',t)} \right\}_{p'=1,\dots,N} \right] \right),

and only then the MLP (Eq. 9), $\mathbf{z}^{(\ell)} = \operatorname{MLP}(\operatorname{LN}(\mathbf{z}'^{(\ell)})) + \mathbf{z}'^{(\ell)}$ . Time and space have separate query, key and value matrices. Each patch makes $N + T + 2$ comparisons instead of $NT + 1$ (§3); the temporal modeling article derives the resulting costs, and the second example tabulates them. The authors’ released code differs (timesformer/models/vit.py at commit ddb4cf3 of 4 May 2021, the file’s latest version as of October 2026): temporal attention runs over the $T$ patch tokens without the class token, so each patch makes $N + T + 1$ comparisons, and the temporal output passes through an extra linear layer initialized to zero in every block except the first.

Results. All Table 1 models start from ViT-B pretrained on ImageNet-21K and use 8 frames of $224 \times 224$ with 1 clip × 3 crops at test time:

Attention (Table 1)	Parameters	Kinetics-400	Something-Something V2
Space	85.9M	76.9	36.6
Joint space-time	85.9M	77.4	58.5
Divided space-time	121.4M	78.0	59.5
Sparse local-global	121.4M	75.9	56.3
Axial	156.8M	73.5	56.2

Divided attention was the most accurate on both validation sets, which the authors attribute partly to its separate temporal parameters, and it scaled: in Figure 3, joint attention ran out of GPU memory at a $448 \times 448$ crop or at 32 frames, while divided attention reached 96 frames.

Training from scratch on Kinetics-400 reached only 64.8%, so all later experiments start from ImageNet. Two larger variants follow: TimeSformer-HR reads 16 frames at $448 \times 448$ , and TimeSformer-L reads 96 frames at $224 \times 224$ sampled at a rate of 1/4, about 12 seconds of video. Table 5 gives Kinetics-400 top-1 accuracy of 78.0%, 79.7% and 80.7% and inference costs of 0.59, 5.11 and 7.14 TFLOPs for TimeSformer, -HR and -L, with 3 views. The table does not say whether these costs cover all 3 views. MViT (Table 4, and “7140G” in its text) and Video Swin (Table 1) list TimeSformer-L at 2,380 GFLOPs with 1 × 3 views, and ViViT’s Table 5a, whose costs cover all views, lists it at 7.14 TFLOPs, so these papers read the figure as the total for 3 views. We read it the same way: the same Table 5 lists SlowFast at 7.0 TFLOPs, which is 234 GFLOPs × 30 views, and Figure 3 plots divided attention at about 0.2 TFLOPs for 8 frames and about 2.4 for 96, a third of 0.59 and 7.14. Our count in the second example, 553 billion multiply-adds for 3 views of the default model, agrees. The action recognition article notes that VideoMAE lists a different cost, and discusses TimeSformer’s training-time comparison with SlowFast (Table 2).

On long instructional videos (a 1,059-task subset of HowTo100M), TimeSformer beat SlowFast at every clip length, reaching 62.6% with 96-frame clips spanning 102.4 seconds each (Table 8). The 1/32 rate there gives 8.5 seconds for 8 frames, which matches a stride of 32 frames at 30 frames per second (our arithmetic: $8 \times 32 / 30 = 8.53$ , $96 \times 32 / 30 = 102.4$ ), supporting the reading of the default “rate of 1/32” in temporal modeling.

ViViT

ViViT (Arnab et al., 2021) studies both tokenizations and four ways to factorize the encoder. Uniform frame sampling takes $n_t$ frames and embeds each with ViT’s 2D patch embedding. Tubelet embedding projects non-overlapping $t \times h \times w$ tubes, which the authors note corresponds to a 3D convolution. The paper names models by their tubelet size in the order $h \times w \times t$ : ViViT-B/16x2 is a ViT-Base with $2 \times 16 \times 16$ tubelets in this article’s time-first notation.

A pretrained image model supplies a 2D filter $E_\text{image}$ , and the 3D filter needs initial values. Inflation replicates it over the $t$ frames and divides by $t$ (Eq. 8), as I3D does for convolutions; central frame initialization puts $E_\text{image}$ at the central position $\lfloor t/2 \rfloor$ and zeros elsewhere (Eq. 9), so that the tubelet embedding starts as uniform frame sampling of every tube’s central frame and can learn to use the other frames. The first example confirms both readings. With ViViT-B and spatio-temporal attention on Kinetics-400 (Table 1), random initialization gave 73.2%, inflation 77.6%, uniform frame sampling 78.5% and central frame initialization 79.2%.

Four models. ViViT’s Figure 1 and §3.3 define:

Spatio-temporal attention: all tokens in one encoder, $O((n_t n_h n_w)^2)$ , the same as TimeSformer’s joint attention.
Factorized encoder: a spatial encoder of $L_s$ layers per time index, whose class tokens (or averages) feed a temporal encoder of $L_t$ layers ( $L_t = 4$ in the experiments). The authors call it a late fusion and give its cost as $O((n_h n_w)^2 + n_t^2)$ .
Factorized self-attention: each layer applies spatial and then temporal multi-head attention, like TimeSformer’s divided attention, with no class token. The temporal attention weights start at zero, so the layer begins as a residual connection; with this initialization the order of the two steps made no difference.
Factorized dot-product attention: half the heads attend over the token’s spatial neighbors and half over its temporal ones, and their outputs are concatenated, keeping Model 1’s parameter count at Model 3’s cost.

With ViViT-B and 16×2 tubelets (Table 2):

Model	Kinetics-400 top-1	Epic Kitchens action	GFLOPs	Parameters	TPU-v3 runtime (ms)
1. Spatio-temporal	80.0	43.1	455.2	88.9M	58.9
2. Factorized encoder	78.8	43.7	284.4	115.1M	17.4
3. Factorized self-attention	77.4	39.1	372.3	117.3M	31.7
4. Factorized dot-product	76.3	39.5	277.1	88.9M	22.9
Model 2 with averaging, no temporal encoder	75.8	38.8	283.9	86.7M	17.3

Model 1 was the most accurate on Kinetics but overfit the smaller Epic Kitchens, where the factorized encoder did best; the averaging baseline lost most on Epic Kitchens, which the authors read as that dataset needing more temporal modeling. Model 1 with a Large backbone ran out of memory beyond 48 frames, while the factorized encoder took 128 frames, enough to cover a 250-frame Kinetics video in one view at stride 2.

Regularization. The authors saw overfitting on Epic Kitchens and Something-Something V2 even from ImageNet weights, and added regularization cumulatively: Kinetics-400 initialization, stochastic depth, RandAugment, label smoothing and mixup. On Epic Kitchens this raised top-1 action accuracy from 38.4% to 43.7% (Table 3), and on Something-Something V2 from 60.4% to 65.4%; Kinetics and Moments in Time were trained without them.

Results. Models start from ImageNet-21K unless JFT is named. On Kinetics-400 (Table 5a), ViViT-L/16x2 with the factorized encoder reached 80.6% with 1 view and 81.7% with 1 clip × 3 crops, at 3.98 and 11.94 TFLOPs for all views; from JFT, it reached 83.5%, and ViViT-H/16x2 84.9% with 4 × 3 views and 47.77 TFLOPs. On Something-Something V2 the factorized encoder reached 65.9% (Table 5e). These are the ICCV numbers. The arXiv v1 table (Table 6a), which MViT and Video Swin compare against, printed different ViViT rows, among them 81.3% for ViViT-L/16x2 at 320 pixels with 4 × 3 views and 84.8% for ViViT-H.

MViT

Multiscale Vision Transformers (MViT; Fan et al., 2021) keep global attention but shrink the sequence it runs on. Where ViT holds width and token grid constant through the network, MViT has scale stages: the first operates on many tokens with few channels, and each later stage reduces the space-time resolution while widening the channels, like a convolutional feature pyramid.

Pooling attention. Multi-head pooling attention projects the input $X \in \mathbb{R}^{L \times D}$ to $\hat{Q}, \hat{K}, \hat{V}$ as usual, then reshapes each to its $T \times H \times W$ grid and pools it with a kernel, stride and padding $\Theta = (k, s, p)$ before attending:

\operatorname{PA}(\cdot) = \operatorname{Softmax}\!\left( \frac{\mathcal{P}(\hat{Q}; \Theta_Q)\, \mathcal{P}(\hat{K}; \Theta_K)^\top}{\sqrt{d}} \right) \mathcal{P}(\hat{V}; \Theta_V).

Pooling the keys and values (with equal strides) shortens what each query compares against; pooling the queries shortens the output, so it changes the resolution. MViT pools queries only in the first block of each stage, with stride $(1, 2, 2)$ , and pools keys and values in every block, with stride $(1, 8, 8)$ in the first stage, decreasing in later stages. The residual path is pooled the same way as the query. With reduction factors $f_Q$ and $f_K$ , the authors give the per-head cost as $O(THW D/h \,(D + THW/f_Q f_K))$ . When the resolution falls 4× at a stage transition, the channels double, which roughly preserves computation per stage. The best pooling was a learned channel-wise convolution (Table 10).

MViT-B. The 16-frame input (stride 4) is projected by overlapping $3 \times 7 \times 7$ cubes with stride $2 \times 4 \times 4$ to 96 channels: $8 \times 56 \times 56 = 25{,}088$ tokens plus a class token. Four stages follow with 96, 192, 384 and 768 channels on grids of $56^2$ , $28^2$ , $14^2$ and $7^2$ per time step, ending with 393 tokens. MViT-B needs 70.5 GFLOPs and 36.5M parameters per clip, against 179.6 GFLOPs and 87.2M for a ViT-B on 8 frames (Table 3).

Results without pretraining. MViT is trained from scratch on Kinetics for 200 epochs. On Kinetics-400 (Table 4), MViT-B 32×3 (32 frames at stride 3) reached 80.2% at 170 GFLOPs × 5 clips × 1 crop, and a 64-frame version 81.2% at 455 GFLOPs × 3 × 3; a ViT-B trained from scratch with MViT’s recipe reached 68.5%, and 79.3% when fine-tuned from ImageNet-21K. Shuffling the test frames lowered MViT-B from 77.2% to 70.1% but ViT-B by only 0.1 points (Table 9), which the authors read as ViT-B ignoring temporal information. On Something-Something V2 (Table 6), MViT-B-24 pretrained on Kinetics-600 reached 68.7%.

MViTv2 (Li et al., 2022) adds decomposed relative position embeddings and a residual pooling connection that adds the pooled query to the attention output. Trained from scratch, MViTv2-B 32×3 reached 82.9% on Kinetics-400 at 225 GFLOPs × 5 clips; MViTv2-L, pretrained on ImageNet-21K and run on 40 frames at $312 \times 312$ , reached 86.1% at 2,828 GFLOPs × 5 clips × 3 crops (Table 10).

Video Swin Transformer

Video Swin Transformer (Liu et al., 2022) argues for locality instead: pixels close in space and time are more correlated, so full attention can be approximated by attention within local windows. It extends the image Swin Transformer (Liu et al., 2021; see vision transformer) to three dimensions. Patches of $2 \times 4 \times 4$ pixels become tokens, so a 32-frame $224 \times 224$ clip gives $16 \times 56 \times 56$ tokens. Four stages follow, with $2 \times$ spatial patch merging between them and no temporal downsampling.

Each block attends within non-overlapping 3D windows of $P \times M \times M$ tokens ( $8 \times 7 \times 7$ by default). The next block shifts the windows by $(P/2, M/2, M/2)$ tokens, so information crosses window borders; for $8 \times 8 \times 8$ tokens and $4 \times 4 \times 4$ windows, the shift turns 8 windows into 27, which the image Swin’s batched computation still processes as 8 (Figure 3). Attention adds a learned 3D relative position bias, $\operatorname{SoftMax}(QK^\top/\sqrt{d} + B)V$ (Eq. 2). Each token’s attention costs a fixed $PM^2 = 392$ comparisons, so cost grows linearly with the number of tokens (the 2D version’s cost formula is in the self-attention article).

From image Swin. Only two parameter tensors change shape. The patch embedding, now spanning 2 frames, takes the image weights duplicated and multiplied by 0.5, an inflation; the relative position bias is duplicated $2P - 1$ times along time. Central initialization, which helped ViViT, gave the same 78.8% for Swin-T (Table 8), a difference the authors left to analyze. A backbone learning rate one tenth of the head’s improved accuracy, more so from ImageNet-21K (82.6% against 82.0% for Swin-B, Table 7).

Results. With ImageNet-21K pretraining and 4 clips × 3 crops, Swin-B reached 82.7% on Kinetics-400 at 282 GFLOPs per view and 88.1M parameters, and Swin-L at $384 \times 384$ reached 84.9% at 2,107 GFLOPs per view, with 10 clips × 5 crops as Table 1 prints for this row (the text’s default is 4 × 3). Swin-B initialized from Kinetics-400 reached 69.6% on Something-Something V2 with 1 × 3 views (Table 3). Ablations with Swin-T on Kinetics-400 (Table 4) found joint attention within windows (78.8%, 88 GFLOPs, 28.2M parameters) better than a factorized version with temporal attention after each spatial layer (78.5%, 95 GFLOPs, 36.5M) or a split version with two temporal layers on top (76.4%). Removing all 3D shifting lowered Something-Something V2 accuracy from 64.7% to 61.2% (Table 6), and shuffling the test frames cost 3.3 points on Kinetics-400 but 49.9 on Something-Something V2 (Table 10).

The paper’s abstract and text say Swin-L’s 84.9% surpasses ViViT by 0.1 points; that comparison is against the 84.8% of ViViT’s arXiv v1, while ViViT’s ICCV table prints 84.9%.

Comparison

	TimeSformer	ViViT	MViT	Video Swin
Tokens	$1 \times 16 \times 16$ frame patches	$2 \times 16 \times 16$ tubelets (or frame patches)	$3 \times 7 \times 7$ overlapping cubes, stride $2 \times 4 \times 4$	$2 \times 4 \times 4$ patches
Attention	divided: time, then space, global along each	four variants; factorized encoder preferred	global, on pooled keys, values and queries; multiscale stages	local 3D windows, shifted every other block; four stages
Attention cost per layer ( $N$ spatial, $T$ temporal tokens)	$O(NT(N + T)D)$	Model 1 $O((NT)^2 D)$ ; Model 2 $O(TN^2D + T^2D)$	$O((NT)^2 D / f_Q f_K)$ , $N$ falling 4× per stage	$O(NT \cdot PM^2 D)$ , linear in $NT$
Pretraining	ImageNet-1K or -21K ViT-B	ImageNet-21K or JFT ViT	none (Kinetics from scratch)	ImageNet-1K or -21K Swin
Kinetics-400 top-1 as printed	TimeSformer-L 80.7%, 1 × 3 views, 7.14 TFLOPs (Table 5)	ViViT-L/16x2 FE 81.7%, 1 × 3 views, 11.94 TFLOPs for all views (Table 5a)	MViT-B 32×3 80.2%, 170 GFLOPs × 5 clips × 1 crop (Table 4)	Swin-B (ImageNet-21K) 82.7%, 282 GFLOPs × 4 clips × 3 crops (Table 1)

The complexity entries are our summary of each paper’s design, counting the attention products only. The accuracy rows differ in pretraining, input and views and are not a ranking.

Training Considerations

Pretraining. Starting from an image model is the default for the factorized and windowed designs, and how much the image data matters depends on the target: TimeSformer gained 2.2 points on Kinetics-400 from ImageNet-21K instead of ImageNet-1K but none on Something-Something V2 (Table 3). JFT is Google’s in-house dataset, so ViViT’s JFT rows cannot be reproduced from public data. MViT and MViTv2 show that a hierarchical design trains from scratch on Kinetics, at the cost of long schedules: Video Swin’s authors estimate MViT-style training from scratch at about 13.3 times their fine-tuning time.

Fewer tokens. Tokens trade directly against accuracy: larger tubelets lowered ViViT’s accuracy and cost (Figure 6), and raising the crop from 224 to 320 pixels raised ViViT-L from 80.3% to 81.0% and its cost from 1,446 to 3,992 GFLOPs (Table 4). Key-value pooling and windows cut comparisons rather than tokens.

Long videos. A single view rarely covers a whole video, so models average several, and fewer views of longer clips can replace many short ones: TimeSformer-L needed one clip where X3D and SlowFast needed five or more to approach their best (Bertasius et al., 2021, Figure 6), and ViViT’s 128-frame model needed one view where its 32-frame model needed four, at about the same total GFLOPs (Arnab et al., 2021, Figure 7). HowTo100M’s videos, about 7 minutes long on average, still took TimeSformer four 96-frame clips each on average (Table 8).

When convolution still wins. With little video, the missing inductive bias shows. Trained on 25% of the Something-Something V2 videos, TimeSformer fell below SlowFast and I3D, all three from ImageNet-1K, and it was the strongest only with 75% or more (Figure 4). Video Swin restores locality and hierarchy and MViT a resolution pyramid; in their tables both match or beat 3D CNNs with less pretraining (Video Swin) or none (MViT). In our reading, for a small labelled dataset without a suitable pretrained video transformer, a pretrained 3D CNN or a self-supervised video transformer is the safer start.

Self-supervised pretraining. VideoMAE (Tong et al., 2022) pretrains the plain ViT on unlabeled video with the masked autoencoder objective, removing the dependence on large labelled image sets; it has its own article.

Examples

The first example builds a tubelet embedding in NumPy and compares it with a direct 3D convolution, then checks what ViViT’s two initializations compute.

import numpy as np

rng = np.random.default_rng(0)
T, H, W, C = 16, 32, 32, 3                  # clip T x H x W x C
t, h, w, D = 2, 8, 8, 64                     # tubelet t x h x w, embedding width D
clip = rng.standard_normal((T, H, W, C))
E = rng.standard_normal((t * h * w * C, D)) / np.sqrt(t * h * w * C)   # learned projection
b = rng.standard_normal(D)

def tubelet_embed(clip, E, b):
    """Cut the clip into non-overlapping t x h x w tubes, flatten each, project with E."""
    nt, nh, nw = T // t, H // h, W // w
    tubes = clip.reshape(nt, t, nh, h, nw, w, C).transpose(0, 2, 4, 1, 3, 5, 6)
    return tubes.reshape(nt, nh, nw, t * h * w * C) @ E + b     # (nt, nh, nw, D) tokens

def conv3d(x, K, b, stride):
    """Direct 3D cross-correlation, no padding: x (T, H, W, C), K (kt, kh, kw, C, D)."""
    kt, kh, kw = K.shape[:3]
    st, sh, sw = stride
    To, Ho, Wo = (x.shape[0] - kt) // st + 1, (x.shape[1] - kh) // sh + 1, (x.shape[2] - kw) // sw + 1
    out = np.tile(b, (To, Ho, Wo, 1))
    for i in range(kt):
        for j in range(kh):
            for k in range(kw):
                window = x[i:i + st * To:st, j:j + sh * Ho:sh, k:k + sw * Wo:sw]   # (To, Ho, Wo, C)
                out += window @ K[i, j, k]
    return out

tokens = tubelet_embed(clip, E, b)
conv = conv3d(clip, E.reshape(t, h, w, C, D), b, stride=(t, h, w))
print("tokens", tokens.shape, " conv3d stride = kernel", conv.shape,
      " max |difference| =", f"{np.abs(tokens - conv).max():.1e}")
overlap = conv3d(clip, E.reshape(t, h, w, C, D), b, stride=(1, h, w))
print("conv3d with temporal stride 1 instead:", overlap.shape, "-> overlapping tubes, not a tubelet embedding")

# ViViT's two initializations of E from a 2D patch embedding E_image (h x w x C -> D)
E_img = rng.standard_normal((h * w * C, D)) / np.sqrt(h * w * C)
E_inflate = np.concatenate([E_img] * t) / t                                   # Eq. 8: replicate, average
E_central = np.concatenate([E_img if i == t // 2 else 0 * E_img for i in range(t)])   # Eq. 9
centre = clip[t // 2::t]                                                      # frame t//2 of every tube
mean = clip.reshape(T // t, t, H, W, C).mean(1)                               # temporal mean of every tube

def patch(frames):
    """ViT's 2D patch embedding of each frame with E_image."""
    p = frames.reshape(len(frames), H // h, h, W // w, w, C).transpose(0, 1, 3, 2, 4, 5)
    return p.reshape(len(frames), H // h, W // w, -1) @ E_img

print("central-frame init = 2D patch embedding of each tube's frame t//2:",
      np.allclose(tubelet_embed(clip, E_central, 0), patch(centre)))
print("inflation init     = 2D patch embedding of each tube's mean frame: ",
      np.allclose(tubelet_embed(clip, E_inflate, 0), patch(mean)))

tokens (8, 4, 4, 64)  conv3d stride = kernel (8, 4, 4, 64)  max |difference| = 4.9e-15
conv3d with temporal stride 1 instead: (15, 4, 4, 64) -> overlapping tubes, not a tubelet embedding
central-frame init = 2D patch embedding of each tube's frame t//2: True
inflation init     = 2D patch embedding of each tube's mean frame:  True

The two agree to rounding error only with stride equal to the kernel and no padding; a smaller temporal stride gives 15 overlapping tubes in time instead of 8 tokens. Central-frame initialization embeds each tube’s frame $\lfloor t/2 \rfloor$ and ignores the other, while inflation embeds the tube’s average frame, which blurs anything that moves within the tube.

The second example counts the multiply-adds of the attention products ( $QK^\top$ and the weighted sum of values) per layer, and the attention weights a 12-layer ViT-B stores for backpropagation, at a fixed $14 \times 14$ grid as the number of frames grows. “Window” is a Video Swin-style $8 \times 7 \times 7$ window on the same grid, for comparison.

N, D, heads, layers = 14 * 14, 768, 12, 12        # ViT-B on 224 x 224 frames, 16 x 16 patches

def attention_cost(T, scheme, P=8, M=7):
    """Multiply-adds of QK^T and AV per layer, and the attention weights it stores (all heads, float32)."""
    n = N * T                                      # patch tokens (class token ignored)
    if scheme == "joint":                          # every token attends to all NT tokens
        keys, stored = n, n * n
    elif scheme == "divided":                      # T keys in time, then N keys in space
        keys, stored = T + N, n * (T + N)
    elif scheme == "window":                       # 3D windows of P x M x M tokens (Swin-style)
        keys, stored = P * M * M, n * P * M * M
    macs = 2 * n * keys * D                        # QK^T and AV, summed over heads
    return macs, stored * heads * 4               # bytes for one layer's attention weights

print(f"{'T':>3} {'tokens':>6} | {'GMACs/layer: joint':>18} {'divided':>8} {'window':>7} |"
      f" {'weights GB, 12 layers: joint':>28} {'divided':>8} {'window':>7}")
for T in (8, 16, 32, 64, 96):
    row = {s: attention_cost(T, s) for s in ("joint", "divided", "window")}
    print(f"{T:3d} {N * T:6d} | {row['joint'][0] / 1e9:18.1f} {row['divided'][0] / 1e9:8.1f}"
          f" {row['window'][0] / 1e9:7.1f} | {layers * row['joint'][1] / 1e9:28.1f}"
          f" {layers * row['divided'][1] / 1e9:8.2f} {layers * row['window'][1] / 1e9:7.2f}")

def vit_b_macs(T, scheme):
    """Whole ViT-B per clip: patch embedding + per layer 12 D^2 per token (QKV, output, 4x MLP), + attention."""
    n = N * T
    per_token = 16 * D * D if scheme == "divided" else 12 * D * D    # divided: own Q, K, V, output in time
    return (n * 16 * 16 * 3 * D + layers * (n * per_token + attention_cost(T, scheme)[0])) / 1e9

print(f"whole ViT-B, T = 8: joint {vit_b_macs(8, 'joint'):.1f} GMACs, divided {vit_b_macs(8, 'divided'):.1f} GMACs")

  T tokens | GMACs/layer: joint  divided  window | weights GB, 12 layers: joint  divided  window
  8   1568 |                3.8      0.5     0.9 |                          1.4     0.18    0.35
 16   3136 |               15.1      1.0     1.9 |                          5.7     0.38    0.71
 32   6272 |               60.4      2.2     3.8 |                         22.7     0.82    1.42
 64  12544 |              241.7      5.0     7.6 |                         90.6     1.88    2.83
 96  18816 |              543.8      8.4    11.3 |                        203.9     3.16    4.25
whole ViT-B, T = 8: joint 179.4 GMACs, divided 184.4 GMACs

Going from 8 to 96 frames multiplies joint attention by 144 and divided attention by about 17. At 32 frames the joint attention weights alone take 22.7 GB per clip, before activations or a batch, which fits TimeSformer’s running out of memory at 32 frames (our arithmetic, not the authors’). At 8 frames, attention is a small part of either model, and divided attention’s extra temporal projections make it slightly more expensive. The joint count matches the 179.6 GFLOPs MViT’s Table 3 gives for ViT-B on 8 frames, so those FLOPs count multiply-adds, as MViT’s Figure 2 states; three times the divided count, 553 GMACs, is close to TimeSformer’s 0.59 TFLOPs for 3 views.

The last example, run with PyTorch 2.14.1, trains a tiny TimeSformer-style model on clips of a $3 \times 3$ square moving one pixel per frame right, left, down or up through 6 noisy $16 \times 16$ frames. The space-only model attends within frames and has no time embedding, as in TimeSformer’s released code, so it is blind to frame order, and a clip played backwards is a valid clip of the opposite class. The divided model adds temporal attention and a time embedding.

import torch
import torch.nn as nn

torch.set_num_threads(4)
print("torch", torch.__version__)
T, S, P, D = 6, 16, 4, 32                    # frames, frame side, patch side, width
N = (S // P) ** 2                            # 16 patches per frame
DIRS = torch.tensor([[0, 1], [0, -1], [1, 0], [-1, 0]])   # right, left, down, up (dy, dx per frame)

def make_clips(n, g):
    """A 3 x 3 bright square moving 1 pixel per frame in one of four directions, plus noise."""
    y = torch.randint(0, 4, (n,), generator=g)
    clips = 0.1 * torch.randn(n, T, S, S, generator=g)
    lo = torch.randint(0, S - 3 - T + 2, (n, 2), generator=g)          # start where the path fits
    hi = torch.randint(0, S - 2, (n, 2), generator=g)
    for i in range(n):
        d = DIRS[y[i]]
        start = torch.where(d == 0, hi[i], torch.where(d > 0, lo[i], lo[i] + T - 1))
        for f in range(T):
            r, c = (start + f * d).tolist()
            clips[i, f, r:r + 3, c:c + 3] += 1.0
    return clips, y

class Block(nn.Module):
    """Pre-norm block: [temporal attention], spatial attention, MLP (TimeSformer-style)."""
    def __init__(self, divided):
        super().__init__()
        self.divided = divided
        self.norm_t, self.norm_s, self.norm_m = nn.LayerNorm(D), nn.LayerNorm(D), nn.LayerNorm(D)
        self.attn_t = nn.MultiheadAttention(D, 2, batch_first=True) if divided else None
        self.attn_s = nn.MultiheadAttention(D, 2, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(D, 2 * D), nn.GELU(), nn.Linear(2 * D, D))

    def forward(self, x):                                  # x: (B, T, N, D)
        B = x.shape[0]
        if self.divided:                                   # each patch position attends over the T frames
            xt = self.norm_t(x).transpose(1, 2).reshape(B * N, T, D)
            x = x + self.attn_t(xt, xt, xt, need_weights=False)[0].reshape(B, N, T, D).transpose(1, 2)
        xs = self.norm_s(x).reshape(B * T, N, D)           # each frame attends within itself
        x = x + self.attn_s(xs, xs, xs, need_weights=False)[0].reshape(B, T, N, D)
        return x + self.mlp(self.norm_m(x))

class TinyVideoTransformer(nn.Module):
    def __init__(self, divided):
        super().__init__()
        self.embed = nn.Linear(P * P, D)                   # 1 x 4 x 4 patches (grayscale)
        self.pos_space = nn.Parameter(0.02 * torch.randn(1, 1, N, D))
        self.pos_time = nn.Parameter(0.02 * torch.randn(1, T, 1, D)) if divided else None
        self.blocks = nn.Sequential(Block(divided), Block(divided))
        self.head = nn.Sequential(nn.LayerNorm(D), nn.Linear(D, 4))

    def forward(self, clips):                              # clips: (B, T, S, S)
        B = clips.shape[0]
        p = clips.reshape(B, T, S // P, P, S // P, P).permute(0, 1, 2, 4, 3, 5).reshape(B, T, N, P * P)
        x = self.embed(p) + self.pos_space
        if self.pos_time is not None:                      # space-only has no time embedding
            x = x + self.pos_time
        return self.head(self.blocks(x).mean(dim=(1, 2)))  # average over frames and patches

def run(divided, seed):
    g = torch.Generator().manual_seed(seed)
    x_tr, y_tr = make_clips(1000, g)
    x_te, y_te = make_clips(500, g)
    torch.manual_seed(seed)
    model = TinyVideoTransformer(divided)
    opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=0.01)
    for step in range(300):
        idx = torch.randint(0, len(x_tr), (64,), generator=g)
        loss = nn.functional.cross_entropy(model(x_tr[idx]), y_tr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        logits = model(x_te)
        gap = (logits - model(x_te.flip(1))).abs().max().item()   # clip vs the clip played backwards
    return (logits.argmax(1) == y_te).float().mean().item(), gap

for divided in (False, True):
    res = [run(divided, seed) for seed in range(5)]
    print(f"{'divided space-time' if divided else 'space only        '}: test accuracy, seeds 0-4: "
          + " ".join(f"{a:.3f}" for a, _ in res)
          + " | max |logits(clip) - logits(reversed)|: " + " ".join(f"{g:.1e}" for _, g in res))

torch 2.14.1
space only        : test accuracy, seeds 0-4: 0.446 0.396 0.362 0.430 0.446 | max |logits(clip) - logits(reversed)|: 7.2e-07 9.5e-07 7.2e-07 7.2e-07 4.8e-07
divided space-time: test accuracy, seeds 0-4: 1.000 1.000 1.000 0.990 1.000 | max |logits(clip) - logits(reversed)|: 7.9e+00 9.4e+00 7.7e+00 6.9e+00 7.6e+00

The space-only model gives a clip and its reversal the same logits up to rounding, so it cannot tell right from left or down from up, and its accuracy stays below 50% over all five seeds. The divided model separates the four directions in every seed. Two runs gave identical output.

Limitations

Cost grows with tokens. Factorized, pooled or windowed, attention still grows at least linearly with $NT$ , and joint attention quadratically. FLOPs are also a poor proxy for runtime: TimeSformer-L and SlowFast have comparable TFLOPs, yet took 2.6 and 14.88 hours to process 20K Kinetics-400 validation videos on 8 V100 GPUs (Bertasius et al., 2021, §4.5), and ViViT’s Model 4 needs fewer FLOPs than Model 2 but runs slower (Table 2).
Dependence on pretraining. The factorized ViT designs rely on ImageNet-21K or private JFT weights, and their advantage on motion-heavy datasets is smaller than on Kinetics.
Appearance shortcuts. On Kinetics-400, TimeSformer without temporal attention lost only 1.1 points and ViT-B barely noticed shuffled frames (see temporal modeling).
Short clips. All four classify clips of a few seconds and average views; longer context needs more frames, memory or a second model over clip features.