TimeSformer

TimeSformer (from “Time-Space Transformer”) is a video classification model by Gedas Bertasius, Heng Wang and Lorenzo Torresani of Facebook AI and Dartmouth College, presented at ICML 2021. It applies the Vision Transformer (ViT-Base) to the $16 \times 16$ patches of every frame of a clip, with no convolution beyond the linear patch embedding, and replaces each block’s single attention over all patches by a divided space–time attention: a temporal step that compares each patch with the patches at the same position in the other frames, then a spatial step within its frame. Its paper compares this design with four other attention schemes on Kinetics-400 and Something-Something V2, and adds two larger configurations, TimeSformer-HR (higher resolution) and TimeSformer-L (96-frame clips). The video transformers article places TimeSformer beside ViViT, MViT and Video Swin and derives its attention equations; this article covers the model itself: how the paper and the released code build each block, what each scheme costs in parameters and operations, how the model was trained and evaluated, and what its results show.

Problem Addressed

In 2021, video classifiers were built from 2D or 3D convolutions, with self-attention at most added on top of convolutional features, as in Non-local Networks. The authors argued that convolutions limit video models in three ways: their inductive biases may restrict expressivity when data are plentiful, their receptive field reaches long-range dependencies only through deep stacks, and 3D networks are costly to train on long or high-resolution clips (§1). ViT had shown that an image cut into patch tokens could be classified by a plain transformer. The obstacle for video is the token count: $T$ frames of $N$ patches give $NT$ tokens, and attention between all pairs grows with $(NT)^2$. The paper’s question, as its title puts it, is whether attention over space and time can replace convolution, and which way of restricting it is both affordable and accurate.

Architecture

The input is a clip $X \in \mathbb{R}^{H \times W \times 3 \times F}$ of $F$ RGB frames; this article writes $T$ for the paper’s $F$, as the video transformers and temporal modeling articles do. Each frame is cut into $N = HW/P^2$ non-overlapping patches with $P = 16$, so a $224 \times 224$ frame gives a $14 \times 14$ grid of $N = 196$ patches. Unless noted otherwise, experiments use the “Base” ViT (supplementary §1): 12 blocks, width $D = 768$, 12 heads of $D_h = 64$ (vit.py). A class token is prepended and, after the last block, its LayerNorm output is classified. The paper describes the classifier as a 1-hidden-layer MLP on top of that output (after Eq. 10); the released code uses a single linear layer.

Patch and Position Embeddings

Each flattened patch $\mathbf{x}_{(p,t)} \in \mathbb{R}^{3P^2}$ is projected by a learned matrix $E \in \mathbb{R}^{D \times 3P^2}$ and given a learned position embedding (Eq. 1):

$$\mathbf{z}^{(0)}_{(p,t)} = E\,\mathbf{x}_{(p,t)} + \mathbf{e}^{pos}_{(p,t)}, \qquad p = 1, \dots, N, \quad t = 1, \dots, T.$$

The equation gives one vector per space–time position. The released code builds it from two tables instead: ViT’s spatial table of $N + 1$ vectors (class token included) and a separate temporal table of $T$ vectors, added together, so $\mathbf{e}^{pos}_{(p,t)} = \mathbf{e}^{space}_p + \mathbf{e}^{time}_t$ ($N + T$ vectors instead of $NT$). Motionformer’s authors list TimeSformer’s positional encoding as “Separate” (Patrick et al., 2021, Table 1). When a checkpoint is applied to another crop size or clip length, the code resizes both tables by nearest-neighbour interpolation.

The embedding matters most where order matters. With no position embedding, accuracy fell from 78.0% to 75.4% on Kinetics-400 but from 59.5% to 45.8% on Something-Something V2; with spatial embeddings only, the two were 77.8% and 52.5% (Table 4). The temporal modeling article cites the first pair; the second shows that it is the temporal half that Something-Something needs. The space-only attention model in the code has no temporal table at all; since it also processes frames separately and averages them, it cannot tell a clip from the same frames shuffled.

Divided Space–Time Attention

A block follows ViT’s pre-norm layout: queries, keys and values come from the LayerNorm of the block input (Eqs. 2–4), each head forms softmax weights from scaled dot products (Eq. 5), the weighted values of all heads are concatenated and projected by $W_O$ and added to the input (Eqs. 7–8), and an MLP with its own residual follows (Eq. 9). Divided attention inserts a second attention step before the MLP. The first step compares patch $(p, t)$ with the class token and the patches $(p, t')$ for $t' = 1, \dots, T$ (Eq. 11); its output, after the residual of Eq. 8, supplies new queries, keys and values for attention over the class token and the patches $(p', t)$ of frame $t$ (Eq. 6), and only then the MLP. Each step has its own $W_Q$, $W_K$, $W_V$, and a patch makes $N + T + 2$ comparisons per block instead of the $NT + 1$ of joint attention (§3). The video transformers article writes both equations out.

The released code differs from the text in three details:

• Class token. Temporal attention runs over the $T$ patch tokens only. For the spatial step the class token is copied into every frame, attends with that frame’s $N$ patches, and the $T$ outputs are averaged into one update. A patch therefore makes $N + T + 1$ comparisons.
• An extra projection. The temporal step’s output passes through one more linear layer, temporal_fc, before it is added to the residual stream. The paper does not mention it, but the parameter count in Table 1 includes it, as the first example below shows.
• Initialization. The supplementary material says that the temporal and spatial attention of each block start from the same weights, those of the corresponding ViT attention layer (supplementary §1). The code’s loader does this, copying ViT’s attention and its LayerNorm into the temporal branch; in addition, the model’s constructor sets temporal_fc to zero in every block except the first, so blocks 2–12 begin as the image model’s spatial blocks and the temporal path opens during training.

The order of the two steps was chosen experimentally: applying spatial attention first lowered accuracy by 0.5 points on both Kinetics-400 and Something-Something V2, and computing the two in parallel by 0.4 points (supplementary §2).

The Other Attention Schemes

The paper draws five blocks in Figure 1 and their neighbourhoods in Figure 2. All except space-only and joint attention add attention steps to the block, and each step has its own query, key and value matrices (§3).

• Space (S). Attention within the query’s frame only (Eq. 6). In the code every frame keeps its own class token and the final class tokens are averaged over frames, so the model is an image ViT applied frame by frame and averaged at the end.
• Joint space–time (ST). One attention over all $NT$ patches and the class token (Eq. 5), the full space–time attention that ViViT calls its Model 1.
• Divided space–time (T+S). Time, then space, as above.
• Sparse local–global (L+G). A local attention over the neighbouring $T \times H/2 \times W/2$ patches (half the grid in each spatial direction, all frames), then a global attention over the whole clip with a stride of 2 patches in time and in both spatial directions. The authors describe it as a faster approximation of full attention with a sparsity pattern like that of Child et al. (2019).
• Axial (T+W+H). Three steps: along time, along the width and along the height, extending two-axis image attention (Ho et al., 2019; Huang et al., 2019; Wang et al., 2020) to a third axis.

Counting keys per query from these definitions (our arithmetic, class token ignored): space $N$, joint $NT$, divided $N + T$, sparse local–global $TN/4 + TN/8 = 3TN/8$, and axial $T + 2\sqrt{N}$. With every patch a query, the attention products of one block cost

$$2 \cdot NT \cdot k \cdot D$$

multiply-adds for $k$ keys per query, one $NT \times k$ product for $QK^\top$ and one for the weighted sum of values. Sparse local–global still grows with $(NT)^2$, at three-eighths of joint attention’s rate; divided and axial attention grow with $NT(N + T)$ and $NT(T + 2\sqrt{N})$. The released code implements only space, joint and divided attention: sparse local–global and axial attention exist only as descriptions in the paper.

Cost

The first example counts the parameters of each scheme and the multiply-adds of a whole ViT-B forward pass (patch embedding, projections, MLPs and attention products), and compares them with the paper’s Table 1 and Figure 3. One added attention step costs a LayerNorm ($2D$), query, key, value and output matrices with biases ($4D^2 + 4D$) and, if the code’s temporal_fc is counted, $D^2 + D$ more, so the source left open whether the paper’s counts include it. The code prints both.

D, L, P = 768, 12, 16                          # ViT-Base: width, blocks, patch side

def params(steps, temporal_fc, classes, time_embed=True, N=196, T=8):
    """Parameters of a TimeSformer with `steps` attention steps per block (1 = space or joint)."""
    embed = 3 * P * P * D + D + D + (N + 1) * D + (T * D if time_embed else 0)  # patches, cls, pos
    block = 2 * D + 4 * D * D + 4 * D + 2 * D + 8 * D * D + 5 * D                # LN, QKV + out, LN, MLP
    extra = 2 * D + 4 * D * D + 4 * D + ((D * D + D) if temporal_fc else 0)      # one more step
    return embed + L * (block + (steps - 1) * extra) + 2 * D + D * classes + classes

print("Parameters (millions); Table 1 prints 85.9 / 85.9 / 121.4 / 121.4 / 156.8")
for name, steps, te in [("space", 1, False), ("joint", 1, True), ("divided", 2, True),
                        ("local+global", 2, True), ("axial", 3, True)]:
    row = [params(steps, fc, c, te) / 1e6 for fc in (False, True) for c in (400, 174)]
    print(f"  {name:12s} no extra linear: {row[0]:6.1f} (400 classes) {row[1]:6.1f} (174)"
          f" | with temporal_fc: {row[2]:6.1f} (400) {row[3]:6.1f} (174)")

def keys_per_query(scheme, N, T):
    """Keys each patch query compares with, summed over a block's attention steps (class token ignored)."""
    side = int(N ** 0.5)
    return {"space": N, "joint": N * T, "divided": T + N,
            "local+global": T * N // 4 + (T // 2) * (N // 4),   # T x H/2 x W/2, then stride 2 in t, h, w
            "axial": T + 2 * side}[scheme]

print("\nKeys per query at 8 x 224 x 224 (N = 196, T = 8):",
      {s: keys_per_query(s, 196, 8) for s in ("space", "joint", "divided", "local+global", "axial")})

def gmacs(scheme, crop=224, T=8):
    """Whole-model multiply-adds per view: patch embedding, projections and MLPs, attention products."""
    N = (crop // P) ** 2
    n = N * T
    steps = 2 if scheme == "divided" else 1
    linear = n * 3 * P * P * D + L * n * (12 * D * D + (steps - 1) * 5 * D * D)
    attention = L * 2 * n * keys_per_query(scheme, N, T) * D       # Q K^T and A V
    return (linear + attention) / 1e9

print("\nGMACs per view   crop:", "  ".join(f"{c:>5d}" for c in (224, 336, 448, 560)),
      "|  frames:", "  ".join(f"{t:>5d}" for t in (8, 32, 64, 96)))
for s in ("joint", "divided"):
    print(f"  {s:8s}             " + "  ".join(f"{gmacs(s, crop=c):5.0f}" for c in (224, 336, 448, 560))
          + " |          " + "  ".join(f"{gmacs(s, T=t):5.0f}" for t in (8, 32, 64, 96)))

Parameters (millions); Table 1 prints 85.9 / 85.9 / 121.4 / 121.4 / 156.8
  space        no extra linear:   86.1 (400 classes)   85.9 (174) | with temporal_fc:   86.1 (400)   85.9 (174)
  joint        no extra linear:   86.1 (400 classes)   85.9 (174) | with temporal_fc:   86.1 (400)   85.9 (174)
  divided      no extra linear:  114.5 (400 classes)  114.3 (174) | with temporal_fc:  121.6 (400)  121.4 (174)
  local+global no extra linear:  114.5 (400 classes)  114.3 (174) | with temporal_fc:  121.6 (400)  121.4 (174)
  axial        no extra linear:  142.8 (400 classes)  142.7 (174) | with temporal_fc:  157.0 (400)  156.8 (174)

Keys per query at 8 x 224 x 224 (N = 196, T = 8): {'space': 196, 'joint': 1568, 'divided': 204, 'local+global': 588, 'axial': 36}

GMACs per view   crop:   224    336    448    560 |  frames:     8     32     64     96
  joint                  179    531   1261   2608 |            179   1261   3973   8135
  divided                195    456    850   1408 |            195    785   1577   2376

Two choices reproduce Table 1’s rounded counts: an extra linear layer per added attention step, $5D^2 + 7D = 2{,}954{,}496$ parameters per block, and a classifier with 174 outputs, Something-Something V2’s class count. Without temporal_fc, divided attention would have 114.3M parameters, not 121.4M. With a 400-class Kinetics head, the counts are 86.1M, 121.6M and 157.0M; Table 2 lists the Kinetics-400 model at 121.4M, so the table appears to give one count for both datasets. The 174-class head is our inference from the arithmetic, not something the paper states: it does not say which head it counted, and any head of roughly 120 to 180 classes rounds to the same three numbers, so the arithmetic rules out the 400-class head but does not single out 174. The counts also leave no room for a hidden layer in the classifier, which matches the code’s linear head rather than the MLP described after Eq. 10.

The operation counts match Figure 3, read off its plotted points: divided attention at about 0.2, 0.45, 0.85 and 1.4 TFLOPs for crops of 224, 336, 448 and 560 pixels, and about 2.4 at 96 frames; joint attention at about 2.6 for a 560-pixel crop and about 8 for 96 frames. So Figure 3’s “TFLOPs” count multiply-adds for one view, and the 0.59 TFLOPs of Table 5 is three views of 195 billion, about 585 billion, which supports the reading that Table 5’s costs cover all 3 views (see video transformers and action recognition). At 8 frames, attention is a small share of the total and divided attention costs slightly more than joint attention because of its extra projections; joint attention’s quadratic term dominates from 32 frames or a 448-pixel crop, where the authors report that it ran out of GPU memory (§4.1). The points in Figure 3’s grey “out of memory” region are therefore computed costs, not runs.

Training

Recipe. All models start from ImageNet weights (§4.2). The supplementary material gives one schedule for every model: 15 epochs of synchronized SGD across 32 GPUs, momentum 0.9, weight decay $10^{-4}$, an initial learning rate of 0.005 divided by 10 at epochs 11 and 14, and a batch size of 16. Training clips are sampled from the full videos “with a frame rate of 1/32”, the shorter side is resized to a random value in $[256, 320]$ ($[448, 512]$ for TimeSformer-HR) and a $224 \times 224$ ($448 \times 448$) crop is taken. The loss is the cross-entropy over classes (LOSS_FUNC in the released configurations). The released configurations repeat the schedule and set the clip sampling per variant: 8 frames with a sampling rate of 32 for the default model, 96 frames at rate 4 for TimeSformer-L and, although the supplementary material gives 1/32 for all models, 16 frames at rate 16 for TimeSformer-HR on Kinetics. The rate-32 setting supports reading “1/32” as a stride of 32 source frames, as temporal modeling does.

Something-Something V2. The Something-Something configurations sample differently: the video is split into $T$ equal segments and one frame is taken from each, at random during training and at the segment’s middle during testing (datasets/ssv2.py), and random horizontal flipping is turned off. Flipping would swap classes such as “Pushing something from left to right”, a pitfall the Something-Something article demonstrates. The Something-Something models use 8, 16 and 64 frames; the README’s TimeSformer-L for this dataset reads 64 frames, not the 96 of the Kinetics model.

Image pretraining. Training from scratch on Kinetics-400 needed a longer schedule and more augmentation and reached only 64.8% (§4.2). ImageNet-21K pretraining helped on Kinetics-400 in every configuration (75.8% to 78.0% for the default model, 77.8% to 79.7% for HR, 78.1% to 80.7% for L) but not on Something-Something V2 (59.5% to 59.5%, 62.2% to 62.5%, 62.4% to 62.3%; Table 3). The authors explain the difference by Kinetics’ bias toward scene appearance. Model size was also tested against this data: ViT-Large gave results about 1% worse on both datasets and ViT-Small about 5% worse, and $32 \times 32$ patches about 3% worse than $16 \times 16$, while smaller patches were not tried because of their cost (supplementary §2).

Training cost. Table 2 gives Kinetics-400 training time in Tesla V100 GPU hours, from ImageNet-1K: TimeSformer reached 75.8% in 416 hours, SlowFast R50 75.6% in 3,840 hours (70.0% when limited to 448) and I3D R50 73.4% in 1,440 hours (71.0% at 444). The SlowFast baseline trained from scratch took 6,336 hours for 76.4%. The baselines used their original protocols, 118 epochs for I3D and the ImageNet-initialized SlowFast, 196 for SlowFast from scratch (supplementary §1), so the comparison sets a short fine-tuning schedule from a strong image model against long schedules, as the authors note. With fewer training videos the order changes on Something-Something V2, where TimeSformer was the most accurate of the three only with 75% or more of the data (Figure 4; see video transformers).

Example: Divided Against Joint Attention on Moving Squares

Two later papers argue that divided attention’s temporal step is weakened by motion, because it compares only patches at the same position: when an object or the camera moves, the patch at that position in another frame shows something else (Bulat et al., 2021, §3; Patrick et al., 2021, whose trajectory attention follows motion instead). The second example tests this on a synthetic task. Each clip has 8 noisy $32 \times 32$ frames with a $4 \times 4$ square moving in one of 8 directions, the class, and two static squares as distractors. With $8 \times 8$ patches ($N = 16$, 128 tokens), a square moving 1 pixel per frame stays within about one patch, and one moving 3 pixels per frame, 21 pixels over the clip, crosses two or three patch borders. Two tiny models of 2 blocks and width 64 differ only in attention: joint attention over all 128 tokens, or divided attention built like the released code (temporal step, temporal_fc initialized to zero after the first block, spatial step). Both use the code’s factorized position embeddings and, unlike TimeSformer, average all tokens instead of using a class token. Two label-free checks come first: a rule that reads the direction from the brightest and darkest $4 \times 4$ regions of the last-minus-first frame difference shows the task is solvable, and the time-averaged frame shows that order matters. The second check holds by construction, since averaging over time discards order: a clip played backwards is a valid clip of the opposite direction with the same frames, so a model blind to frame order can be right on at most half of the clips (our reasoning; the printed True confirms only the identity). We expected both models to learn the slow task and joint attention to do better on the fast one.

import sys
import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(4)
T, S, P, D, HEADS = 8, 32, 8, 64, 4          # frames, frame side, patch side, width, heads
N = (S // P) ** 2                            # 16 patches per frame, 128 tokens per clip
DIRS = [(dy, dx) for dy in (-1, 0, 1) for dx in (-1, 0, 1) if (dy, dx) != (0, 0)]   # 8 classes
STEPS = int(sys.argv[1]) if len(sys.argv) > 1 else 400

def make_clips(n, speed, rng):
    """A 4 x 4 square moving `speed` px/frame in one of 8 directions, two static squares, noise."""
    y = rng.integers(0, 8, n)
    clips = 0.3 * rng.standard_normal((n, T, S, S)).astype(np.float32)
    span = speed * (T - 1)
    for i in range(n):
        dy, dx = DIRS[y[i]]
        start = [rng.integers(span if d < 0 else 0, S - 4 - (span if d > 0 else 0) + 1) for d in (dy, dx)]
        for _ in range(2):                                          # static distractors
            r, c = rng.integers(0, S - 3, 2)
            clips[i, :, r:r + 4, c:c + 4] += 1.0
        for f in range(T):
            r, c = start[0] + f * speed * dy, start[1] + f * speed * dx
            clips[i, f, r:r + 4, c:c + 4] += 1.0
    return clips, y

def label_free_checks(clips, y, speed):
    """Oracle from first/last frames, and the time-mean frame's blindness to frame order."""
    diff = clips[:, -1] - clips[:, 0]                               # static squares cancel
    c = np.cumsum(np.cumsum(np.pad(diff, ((0, 0), (1, 0), (1, 0))), 1), 2)
    box = c[:, 4:, 4:] - c[:, :-4, 4:] - c[:, 4:, :-4] + c[:, :-4, :-4]   # 4 x 4 box sums
    hi = np.array(np.unravel_index(box.reshape(len(y), -1).argmax(1), box.shape[1:])).T
    lo = np.array(np.unravel_index(box.reshape(len(y), -1).argmin(1), box.shape[1:])).T
    move = np.round((hi - lo) / (speed * (T - 1))).clip(-1, 1).astype(int)   # expected step +-1 or 0
    pred = np.array([DIRS.index(tuple(m)) if tuple(m) in DIRS else -1 for m in move])
    same = np.allclose(clips.mean(1), clips[:, ::-1].mean(1), atol=1e-5)
    return (pred == y).mean(), same

class Block(nn.Module):
    def __init__(self, divided, first):
        super().__init__()
        self.divided = divided
        self.norm1, self.norm2 = nn.LayerNorm(D), nn.LayerNorm(D)
        self.attn = nn.MultiheadAttention(D, HEADS, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
        if divided:                                                 # as in the released code
            self.temporal_norm = nn.LayerNorm(D)
            self.temporal_attn = nn.MultiheadAttention(D, HEADS, batch_first=True)
            self.temporal_fc = nn.Linear(D, D)
            if not first:
                nn.init.zeros_(self.temporal_fc.weight); nn.init.zeros_(self.temporal_fc.bias)

    def forward(self, x):                                           # x: (B, T, N, D)
        B = x.shape[0]
        if self.divided:
            xt = self.temporal_norm(x).transpose(1, 2).reshape(B * N, T, D)   # same patch, all frames
            xt = self.temporal_attn(xt, xt, xt, need_weights=False)[0]
            x = x + self.temporal_fc(xt.reshape(B, N, T, D).transpose(1, 2))
            xs = self.norm1(x).reshape(B * T, N, D)                           # each frame alone
            x = x + self.attn(xs, xs, xs, need_weights=False)[0].reshape(B, T, N, D)
        else:
            xj = self.norm1(x).reshape(B, T * N, D)                           # all 128 tokens
            x = x + self.attn(xj, xj, xj, need_weights=False)[0].reshape(B, T, N, D)
        return x + self.mlp(self.norm2(x))

class TinyTimeSformer(nn.Module):
    def __init__(self, divided):
        super().__init__()
        self.embed = nn.Linear(P * P, D)
        self.pos_space = nn.Parameter(0.02 * torch.randn(1, 1, N, D))     # factorized, as in the code
        self.pos_time = nn.Parameter(0.02 * torch.randn(1, T, 1, D))
        self.blocks = nn.Sequential(Block(divided, True), Block(divided, False))
        self.head = nn.Sequential(nn.LayerNorm(D), nn.Linear(D, 8))

    def forward(self, clips):                                       # (B, T, S, S)
        B = clips.shape[0]
        p = clips.reshape(B, T, S // P, P, S // P, P).permute(0, 1, 2, 4, 3, 5).reshape(B, T, N, P * P)
        x = self.embed(p) + self.pos_space + self.pos_time
        return self.head(self.blocks(x).mean(dim=(1, 2)))           # mean over tokens

def attention_macs(divided):
    keys = (T + N) if divided else T * N
    return 2 * 2 * T * N * keys * D                                 # 2 blocks x (QK^T + AV)

def run(divided, speed, seed, data):
    (x_tr, y_tr), (x_te, y_te) = data
    torch.manual_seed(seed)
    model = TinyTimeSformer(divided)
    opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for step in range(STEPS):
        idx = torch.randint(0, len(x_tr), (64,), generator=g)
        loss = nn.functional.cross_entropy(model(x_tr[idx]), y_tr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        acc = (model(x_te).argmax(1) == y_te).float().mean().item()
    return acc, sum(p.numel() for p in model.parameters())

print("torch", torch.__version__)
for speed in (1, 3):
    rng = np.random.default_rng(speed)
    tr, te = make_clips(4000, speed, rng), make_clips(1000, speed, rng)
    oracle, same = label_free_checks(*te, speed)
    print(f"speed {speed} px/frame: first/last-frame oracle {oracle:.3f}; "
          f"time-mean frame equal for clip and reversal: {same}")
    data = tuple((torch.from_numpy(x), torch.from_numpy(y)) for x, y in (tr, te))
    for divided in (False, True):
        res = [run(divided, speed, seed, data) for seed in (0, 1, 2)]
        print(f"  {'divided' if divided else 'joint  '}: params {res[0][1]:,}, attention MACs/clip "
              f"{attention_macs(divided) / 1e6:.2f}M, test accuracy seeds 0-2: "
              + " ".join(f"{r[0]:.3f}" for r in res))

torch 2.14.1
speed 1 px/frame: first/last-frame oracle 1.000; time-mean frame equal for clip and reversal: True
  joint  : params 106,312, attention MACs/clip 4.19M, test accuracy seeds 0-2: 0.113 0.131 0.145
  divided: params 148,168, attention MACs/clip 0.79M, test accuracy seeds 0-2: 0.453 0.447 0.514
speed 3 px/frame: first/last-frame oracle 1.000; time-mean frame equal for clip and reversal: True
  joint  : params 106,312, attention MACs/clip 4.19M, test accuracy seeds 0-2: 0.257 0.222 0.277
  divided: params 148,168, attention MACs/clip 0.79M, test accuracy seeds 0-2: 0.457 0.492 0.496

The result goes against our expectation. After 400 steps, divided attention was ahead in both conditions: at about 0.45–0.51 accuracy against chance at 0.125, while joint attention was barely above chance with slow motion and at 0.22–0.28 with fast motion. Neither approaches the label-free rule’s 1.000, so both models are under-trained, and the comparison is of how quickly each learns at a fixed budget, not of what each could reach; we did not train longer. Divided attention has more parameters, and its first block’s temporal step relates every patch to the same position in all frames from the start, which may be an easier signal to pick up than one among 128 keys; this is our interpretation, not something the run isolates. Faster motion did not hurt divided attention here, possibly because one divided block already passes information from $(p', t')$ to $(p, t)$ in two hops, time then space. Joint attention needs 5.3 times the attention multiply-adds of divided attention at this size, but at 128 tokens the attention products are a small part of the work: counting the projections and MLPs as the first example does, the divided model’s forward pass has more multiply-adds in total, as ViT-B’s does at 8 frames (our arithmetic). The saving appears only with longer clips or larger frames. The whole example takes 6 to 8 minutes on a CPU with 4 threads (PyTorch 2.14.1). The plan was 800 steps; a 20-step timing run before the first full run, whose accuracies were still at chance, showed steps three to four times slower than estimated, and the step count was halved to 400 before the first full run. The label-free rule was also rewritten after that timing run, since its first version, a centroid of thresholded differences, was dominated by noise pixels. The first full run also printed milliseconds per step, which differ from run to run; in fact-checking that measurement was removed from the printed output and nothing else was changed, so two runs now print identical output.

Inference

At test time a single temporal clip is taken from the middle of the video; the shorter side is scaled to 224 pixels (448 for HR), three crops (top-left, centre, bottom-right) cover the frame, and the softmax scores of the three are averaged (§4; supplementary §1). Most earlier models averaged 10 clips × 3 crops (§4.5). With more clips, each with one crop, accuracy rose for TimeSformer, SlowFast and X3D in Figure 6, and X3D and SlowFast needed at least 5 clips to approach their best, while TimeSformer-L, whose clip covers about 12 seconds of a Kinetics video, did not need multiple clips: its plotted accuracy is flat from 1 to 10 clips (§4.5).

The authors also timed inference on the 20K Kinetics-400 validation videos on 8 Tesla V100 GPUs: 36 minutes for TimeSformer, 1.06 hours for TimeSformer-HR, 2.6 hours for TimeSformer-L and 14.88 hours for SlowFast, although TimeSformer-L and SlowFast have similar TFLOPs in Table 5 (§4.5).

Performance

Attention schemes. With ViT-B from ImageNet-21K, 8 frames and 1 × 3 views, divided attention was the most accurate scheme on both validation sets, 78.0% on Kinetics-400 and 59.5% on Something-Something V2, against 77.4% and 58.5% for joint attention and 76.9% and 36.6% for space-only attention (Table 1). Sparse local–global (75.9%, 56.3%) and axial attention (73.5%, 56.2%) were less accurate than joint attention despite having more parameters. The authors attribute divided attention’s lead over joint attention to its larger capacity, the separate temporal parameters, and its lead in cost to scaling (§4.1). Space-only attention losing only 1.1 points on Kinetics but 22.9 on Something-Something is the paper’s measurement of how much each dataset needs temporal modeling.

Kinetics. Results are top-1 video-level accuracy on the validation sets with 1 clip × 3 crops; the paper states ImageNet-21K pretraining for the Kinetics-400 rows and does not restate it for Kinetics-600:

Model (input)	Kinetics-400 top-1 / top-5 (Table 5)	TFLOPs as printed	Kinetics-600 top-1 / top-5 (Table 6)
TimeSformer ($8 \times 224^2$)	78.0 / 93.7	0.59	79.1 / 94.4
TimeSformer-HR ($16 \times 448^2$)	79.7 / 94.4	5.11	81.8 / 95.8
TimeSformer-L ($96 \times 224^2$)	80.7 / 94.7	7.14	82.2 / 95.6

On Kinetics-400, TimeSformer-L was above every model in the table, including X3D-XXL at 80.4% and SlowFast at 79.8%, while, the authors note, most earlier methods used 30 views; on Kinetics-600 it was above X3D-XL’s 81.9%. The abstract’s claim of the best reported accuracy on both refers to these rows. Clip-level accuracy on Kinetics-400 rose steadily with the number of frames, from about 66% with 8 to about 78% with 96, and with the crop size up to 448 pixels, after which it fell (Figure 5, as plotted; the authors write that resolution helps “up to a certain point”).

Something-Something V2 and Diving-48. These results start from ImageNet-1K, since ImageNet-21K did not help on Something-Something V2 (Table 7). TimeSformer reached 59.5%, HR 62.2% and L 62.4%, below SlowFast (61.7%) for the default model and below TSM, STM, MSNet, TEA and bLVNet (63.4–65.2%) for all three; the authors present the results as promising for a design so different, not as the state of the art. On Diving-48 (16K training and 3K test videos of 48 dive types), TimeSformer-L reached 81.0% against 77.6% for a SlowFast 16×8 R101 the authors reproduced; because of a recently found problem with an earlier version of Diving-48’s labels, they compared with no published result.

Comparisons in later papers. Later tables do not always copy the same TimeSformer row. ViViT’s Something-Something table lists TimeSformer-HR at 62.5%, which matches the ImageNet-21K number in Table 3 rather than the 62.2% of Table 7, and its text puts its own lead at 2.9 points, while its table gives 65.9 against 62.5, a difference of 3.4 (Arnab et al., 2021, Table 5e and §4.3). X-ViT’s Kinetics-600 table lists TimeSformer-HR at 82.4% top-1 and 96.0% top-5 (Bulat et al., 2021), not the 81.8% and 95.8% of the ICML Table 6; the source of the difference is not stated. ViViT and X-ViT both cite TimeSformer’s arXiv version, not the ICML paper.

Strengths

• Simplicity. The model is ViT-B with one extra attention step per block, so ImageNet checkpoints load directly and the temporal branch can start from the spatial one.
• Scaling with clip length. Attention cost grows with $NT(N+T)$, which made 96-frame clips feasible on the authors’ GPUs where joint attention failed at 32 (§4.1, §4.3).
• Short fine-tuning. 15 epochs from an image model reached the accuracy of 3D networks trained several times longer (Table 2).
• Few views. One clip × 3 crops, and for TimeSformer-L one clip of about 12 seconds, replaced the 30 views common for 3D networks.

Limitations

• Dependence on image pretraining. From scratch the model reached 64.8% on Kinetics-400, and with less video it fell behind 3D CNNs on Something-Something V2 (Figure 4).
• Motion-heavy benchmarks. On Something-Something V2 every TimeSformer variant trailed the best convolutional models of the time (Table 7).
• Fixed-position temporal attention. The temporal step compares a patch only with the same position in other frames; X-ViT’s and Motionformer’s authors cite this as the reason to mix neighbouring frames or follow trajectories.
• Cost still grows quickly. TimeSformer-L costs 7.14 TFLOPs for its three views, and the 96-frame limit came from GPU memory (§4.3).
• Paper and code differ. The class token’s place in temporal attention, the temporal_fc layer, the factorized position embeddings and the linear head are described differently in the text, and the README’s checkpoints were retrained “using a slightly different implementation”, with accuracies that differ slightly from the paper (77.9% against 78.0% for the default model on Kinetics-400, for example).

Historical Importance

TimeSformer was first posted on arXiv in February 2021 (arXiv:2102.05095, the copy the repository’s README links) and published at ICML in July 2021. Its authors write that they knew of no earlier video recognition model built only from self-attention (§2), and ViViT, published at ICCV the same year, calls TimeSformer concurrent work. Together they established the recipe that the video transformers article describes: frame or tubelet patches, an image transformer as the starting point, and factorized attention. Factorized spatial and temporal attention also appears in diffusion-based video generation models, which use it to keep attention over many frames within memory.

Variants

TimeSformer-HR. 16 frames at $448 \times 448$, so $N = 784$ patches per frame and 12,544 tokens per clip, 5.11 TFLOPs for 3 views (Table 5). The Kinetics configuration samples every 16th frame.

TimeSformer-L and long clips. 96 frames at $224 \times 224$, sampled every 4th frame, so one clip spans about 12 seconds of a Kinetics video (18,816 tokens). The longest-range experiment uses HowTo100M (Miech et al., 2019): from about a million instructional videos of over 23K tasks, averaging about 7 minutes, the authors kept the 1,059 tasks with at least 100 videos, 120K videos split at random into 85K for training and 35K for testing (§4.6). Clips of 8, 32, 64 and 96 frames at rate 1/32 cover 8.5, 34.1, 68.3 and 102.4 seconds; at test time a video is covered by non-overlapping clips, 48 to 4 on average, and their predictions are averaged. Pretrained on Kinetics-400, TimeSformer reached 56.8%, 61.2%, 62.2% and 62.6%, against 48.2–51.5% for SlowFast R101 on the same clips, a margin of 8–11 points that the video transformers article also cites (Table 8). The authors also fine-tuned directly from ImageNet, skipping Kinetics: from ImageNet-1K, accuracy rose from 52.8% at 8 frames to 59.4% at 96, and from ImageNet-21K from 56.0% to 62.1% (§4.6). In the Table 8 comparison, the text says, the TimeSformer models start from an ImageNet-21K ViT and all models are trained on Kinetics-400 before fine-tuning on HowTo100M. The abstract describes the clips as “over one minute long”, which the 64- and 96-frame clips are.

Successors that change the attention. Motionformer (Patrick et al., 2021) keeps TimeSformer’s separate position embeddings, takes ViViT’s $2 \times 16 \times 16$ tubelets and replaces divided attention by trajectory attention, which aggregates information along implicitly determined motion paths across frames, at about twice the cost of joint attention (369.5 against 180.6 GFLOPs, Table 4). In its ablation with ViT-B from ImageNet-21K and 16-frame clips, joint attention reached 79.2% on Kinetics-400 and 64.0% on Something-Something V2, divided attention 78.5% and 64.2%, and trajectory attention 79.7% and 66.5%. In that setting divided attention did not beat joint attention on Kinetics-400, the reverse of TimeSformer’s Table 1, which used frame patches and 8 frames. Motionformer’s Table 2b also found the separate space and time embeddings better on Something-Something V2 than a joint embedding (64.0% against 60.8% with joint attention). X-ViT (Bulat et al., 2021) keeps the cost of space-only attention, $O(TS^2)$ in its notation with $S$ patches per frame, by letting each query in a spatial attention compare with keys whose channels are taken from neighbouring frames in a local temporal window (a “shift trick” that costs no extra projection). ViViT’s Model 3 (factorized self-attention) is the closest parallel to divided attention; the video transformers article compares it, and VideoMAE later removed the need for labelled image pretraining for the plain ViT.

Implementations

The official PyTorch code, facebookresearch/TimeSformer, is built on PySlowFast and pytorch-image-models. As of November 2026 the repository is archived, its last commit is from 28 May 2021, and the model file timesformer/models/vit.py last changed on 4 May 2021. It implements three attention types, space_only, joint_space_time and divided_space_time, with configurations for Kinetics (default, HR, L, joint and space-only) and Something-Something V2 (8, 16 and 64 frames), and the README lists checkpoints for Kinetics-400, Kinetics-600, Something-Something V2 and HowTo100M. The README notes that the HR and L configurations need GPUs with about 32 GB of memory and that the authors trained on four nodes of 8 Tesla V100 GPUs. Most of the code is licensed under Creative Commons Attribution-NonCommercial 4.0 (the README writes “CC-NC 4.0”), with the PySlowFast and pytorch-image-models portions under Apache 2.0, so the official code is not licensed for commercial use; the README states no separate licence for the checkpoints.