VideoMAE

VideoMAE (video masked autoencoder; Tong et al., NeurIPS 2022) pretrains a plain Vision Transformer on unlabeled video by hiding 90–95% of its space-time patches and reconstructing their pixels. It carries the masked autoencoder (MAE) of He et al. (2022) over to video transformers with two changes: a higher masking ratio and tube masking, in which every frame hides the same spatial positions. Its main finding is data efficiency: pretrained only on the target dataset’s unlabeled videos, even 3,500 clips, the ViT fine-tuned well for action recognition. VideoMAE V2 (Wang et al., 2023) also masks the decoder and scales the encoder to a billion parameters. Table and figure numbers below are the NeurIPS paper’s (the arXiv v3 matches) unless the supplementary material is named.

Problem Addressed

Video transformers of the time, such as TimeSformer and ViViT, started from image models pretrained on ImageNet-21K or JFT; training them from scratch, the authors note, had given poor results, MViT excepted.

Image masking does not transfer directly, the authors argue, because of two properties of video (see video understanding):

• Temporal redundancy. Frames are captured densely and their content changes slowly, so at the usual masking ratios of 50–75%, missing pixels can be recovered with little understanding of the scene.
• Temporal correlation. Neighboring frames show the same surfaces, so under random or frame masking a hidden cube often has a visible copy at the same place in an adjacent frame. The authors call this information leakage: a shortcut through low-level correspondence instead of reasoning about content.

Architecture

Input and Tubelets

A clip of $T = 16$ frames is sampled with stride $\tau = 4$ on Kinetics-400 and $\tau = 2$ on Something-Something V2 (§3.3 and supplementary Table 8), at $224 \times 224$ pixels. A cube embedding, the tubelet embedding of video transformers, projects each non-overlapping $2 \times 16 \times 16$ cube to one token, giving $8 \times 14 \times 14 = 1{,}568$ tokens for ViT-B, with fixed sine–cosine position embeddings by default in the official code.

Tube Masking

One random spatial mask of ratio $\rho$ is drawn per clip and repeated in all 8 temporal slots, so a masked cube’s temporal neighbors are masked too. With $\rho = 90\%$ the official generator hides $\lfloor 0.9 \times 196 \rfloor = 176$ of 196 positions per slot, leaving 160 visible tokens. The authors present this as relieving the leak for cubes with little or no motion; in our reading, moving content can still appear at a visible position in another frame.

The NumPy script below measures the shortcut on 100 synthetic 16-frame clips, a striped disc drifting half a pixel per frame over a static textured background. It “reconstructs” each masked $2 \times 8 \times 8$ cube by copying the visible cube at the same position from the nearest time slot, or, if there is none, the mean of that slot’s visible cubes.

import numpy as np

T, S, t, p = 16, 64, 2, 8            # 16 frames of 64x64; cubes of t x p x p = 2 x 8 x 8
nt, ns = T // t, (S // p) ** 2       # 8 time slots x 64 spatial positions = 512 cubes

def make_clip(rng):
    """Static textured background plus a striped disc drifting about 0.5 px per frame."""
    yy, xx = np.mgrid[0:S, 0:S].astype(float)
    bg = 0.4 + 0.25 * np.sin(xx / rng.uniform(2, 4) + rng.uniform(0, 6)) \
             * np.cos(yy / rng.uniform(2, 4) + rng.uniform(0, 6))
    cy, cx = rng.uniform(16, 48, 2)
    vy, vx = 0.5 * rng.choice([-1, 1], 2)          # px per frame
    r, f = rng.uniform(10, 16), rng.uniform(0.6, 1.2)
    clip = np.empty((T, S, S))
    for k in range(T):
        dy, dx = yy - (cy + vy * k), xx - (cx + vx * k)
        inside = dy ** 2 + dx ** 2 < r ** 2
        clip[k] = np.where(inside, 0.7 + 0.25 * np.sin(f * (dx + dy)), bg)
    return clip

def cubes(clip):
    """(T, S, S) -> (nt, ns, t*p*p): non-overlapping 2x8x8 cubes (tubelets)."""
    g = S // p
    return clip.reshape(nt, t, g, p, g, p).transpose(0, 2, 4, 1, 3, 5).reshape(nt, ns, -1)

def make_mask(kind, ratio, rng):
    """Boolean (nt, ns) mask, True = masked (hidden from the encoder)."""
    if kind == "random":                            # independent per cube
        m = np.zeros(nt * ns, bool)
        m[rng.permutation(nt * ns)[: int(round(ratio * nt * ns))]] = True
        return m.reshape(nt, ns)
    if kind == "tube":                              # one spatial map shared by all time slots
        m = np.zeros(ns, bool)
        m[rng.permutation(ns)[: int(round(ratio * ns))]] = True
        return np.tile(m, (nt, 1))
    if kind == "frame":                             # whole time slots hidden
        m = np.zeros((nt, ns), bool)
        m[rng.permutation(nt)[: int(round(ratio * nt))]] = True
        return m

def copy_baseline(x, mask):
    """Fill each masked cube with the visible cube at the same position in the nearest
    time slot. Where there is none, use the mean of the visible cubes of that time slot,
    or of the whole clip. Returns the MSE over masked cubes and the share that found a copy."""
    pred, copied = x.copy(), np.zeros_like(mask)
    clip_mean = x[~mask].mean(0)
    for i, j in zip(*np.nonzero(mask)):
        vis_t = np.nonzero(~mask[:, j])[0]
        if len(vis_t):
            pred[i, j] = x[vis_t[np.argmin(abs(vis_t - i))], j]
            copied[i, j] = True
        else:
            pred[i, j] = x[i][~mask[i]].mean(0) if (~mask[i]).any() else clip_mean
    err = ((pred - x) ** 2).mean(-1)                       # per-cube MSE
    return err[mask].sum(), err[copied].sum(), copied.sum(), mask.sum()

rng = np.random.default_rng(0)
clips = [cubes(make_clip(rng)) for _ in range(100)]
print("cube-level copy baseline over 100 clips (MSE on masked cubes, pixel values in [0, 1])")
print(f"{'masking':<8}{'ratio':>7}{'copy found':>12}{'MSE all':>10}{'MSE copied':>12}{'MSE rest':>10}")
for kind, ratio in [("random", 0.75), ("random", 0.90), ("random", 0.95),
                    ("frame", 0.875), ("tube", 0.75), ("tube", 0.90), ("tube", 0.95)]:
    e_all, e_cop, n_cop, n_all = np.array([copy_baseline(x, make_mask(kind, ratio, rng))
                                           for x in clips]).sum(0)
    rest = f"{(e_all - e_cop) / (n_all - n_cop):>10.4f}" if n_all > n_cop else f"{'-':>10}"
    cop = f"{e_cop / n_cop:>12.4f}" if n_cop else f"{'-':>12}"
    print(f"{kind:<8}{ratio:>7.3f}{n_cop / n_all:>11.0%}{e_all / n_all:>10.4f}{cop}{rest}")

cube-level copy baseline over 100 clips (MSE on masked cubes, pixel values in [0, 1])
masking   ratio  copy found   MSE all  MSE copied  MSE rest
random    0.750        87%    0.0120      0.0092    0.0300
random    0.900        53%    0.0214      0.0104    0.0336
random    0.950        30%    0.0309      0.0108    0.0397
frame     0.875       100%    0.0110      0.0110         -
tube      0.750         0%    0.0302           -    0.0302
tube      0.900         0%    0.0335           -    0.0335
tube      0.950         0%    0.0375           -    0.0375

Copied cubes cost about a third of the error of cubes with no copy (0.009–0.011 against 0.030–0.040). Raising the random-masking ratio from 75% to 95% cuts the share with a copy from 87% to 30%; frame masking, despite hiding 87.5%, leaves a copy for every cube; tube masking leaves none. In our reading, a network trained with random or frame masks can lower its loss by learning to copy, and with tube masks it cannot.

Encoder

The encoder is a vanilla ViT with joint space-time attention, whose quadratic cost is offset in pretraining because it sees only about a tenth of the tokens.

Decoder and Target

A linear projection maps the encoded tokens to the decoder width; learnable mask tokens fill the masked positions, and a 4-block decoder of width 384 (half of ViT-B’s 768) predicts the $3 \times 2 \times 16 \times 16 = 1{,}536$ pixel values of each cube (supplementary Table 8). The loss is the mean squared error on masked cubes only (Eq. 1), against pixels normalized per cube, as the captions of Tables 6 and 7 state; the official code uses each cube’s mean and standard deviation per color channel. Unlike in MAE, decoder depth mattered (Table 1a).

Training

Ablations. Top-1 of a 16-frame ViT-B pretrained for 800 epochs at $\rho = 90\%$, fine-tuned, with 2 clips × 3 crops on Something-Something V2 (SSV2) and 5 × 3 on Kinetics-400 (K400) (Table 1):

Factor	Setting	SSV2	K400
Mask sampling	tube, 75%	68.0	79.8
	tube, 90% (default)	69.6	80.0
	random, 90%	68.3	79.5
	frame, 87.5% (14 of 16 frames)	61.5	76.5
Decoder depth	1 / 2 / 4 / 8 blocks	68.5 / 69.2 / 69.6 / 69.3	79.0 / 79.2 / 80.0 / 79.7
Target	center frame only	63.0	79.3
Loss	L1 / MSE / smooth L1	69.1 / 69.6 / 68.9	79.7 / 80.0 / 79.6
Pretraining	none (from scratch)	32.6	68.8
	ImageNet-21K supervised	61.8	78.9
	VideoMAE on the target dataset	69.6	80.0

Over masking ratios from 50% to 98%, accuracy peaked at 90% (69.6% SSV2, 80.0% K400) and stayed close at 95% (69.3%, 79.8%; Figure 3). Masking schemes differ less on K400, whose videos the authors call mostly stationary and scene-related.

Recipe. Pretraining used AdamW ($\beta_2 = 0.95$), base learning rate $1.5 \times 10^{-4}$, batch 1,024 and multi-scale cropping, with horizontal flips on K400 but not on SSV2 (supplementary Table 9; the paper gives no reason, see action recognition). Fine-tuning added layer-wise learning-rate decay, RandAugment, mixup, CutMix and label smoothing, again without flips on SSV2 (supplementary Table 10, captioned for Tables 6 and 7; the text points to Table 12, which is the AVA setting), with segment sampling on SSV2 and dense sampling on K400 (see temporal modeling).

Data efficiency. With ViT-B pretrained and fine-tuned on each dataset’s own training videos (Table 2):

Dataset	Training videos	From scratch	MoCo v3	VideoMAE
K400	240k	68.8	74.2	80.0
SSV2	169k	32.6	54.2	69.6
UCF101	9.5k	51.4	81.7	91.3
HMDB51	3.5k	18.0	39.2	62.6

The authors write that the gap widens as training sets shrink; over the contrastive MoCo v3 it is largest on HMDB51 (23.4 points) but smaller on UCF101 than on SSV2. For HMDB51 the text says “around 61%”. The UCF101 and HMDB51 numbers match supplementary Table 13, whose models used a 75% ratio for 3,200 and 4,800 epochs. On SSV2, VideoMAE pretrained in 19.5 hours (800 epochs) against MoCo v3’s 61.7 (300 epochs) on 64 V100 GPUs (Table 3).

Quality versus quantity. K400 pretraining gave 96.1% on UCF101 and 73.3% on HMDB51, above in-domain pretraining, but 68.5% on SSV2, below it (Table 4). A quarter of SSV2 (42k videos), pretrained for as many iterations, reached 68.7%, above the 240k-video K400 model (Figure 4). The authors conclude that domain shift matters and data quality outweighs quantity.

A Tiny VideoMAE

The PyTorch script below (version 2.14.1, 4 threads, about 55 seconds, identical output on two runs) builds a miniature VideoMAE for 8-frame, $32 \times 32$ clips of a striped disc moving left, right, up or down: 64 tubelets of $2 \times 8 \times 8$, a two-block encoder that sees 8 of them, and a one-block decoder. It reconstructs raw pixels, our departure from the paper’s normalized target: in our reasoning, per-cube normalization would amplify the faint background texture. For seeds 0–3, it pretrains for 800 steps on 3,000 unlabeled clips with tube or random masking, then classifies motion direction from 200 labelled clips by a linear probe and by brief fine-tuning, against the untrained encoder.

import math
import numpy as np
import torch
from torch import nn
import torch.nn.functional as F

torch.set_num_threads(4)
print(f"torch {torch.__version__}")

T, S, t, p = 8, 32, 2, 8                 # 8 frames of 32x32, tubelets of 2x8x8
nt, ns = T // t, (S // p) ** 2           # 4 time slots x 16 positions = 64 tokens
VIS = 2                                  # visible tokens per time slot: 8 of 64 (87.5% masked)
D, DD = 64, 32                           # encoder and decoder widths

def make_clips(n, rng):
    """A striped disc moves 1 px per frame left, right, up or down (the label) over a faintly
    textured static background. Start positions are random, so one frame cannot tell the class."""
    yy, xx = np.mgrid[0:S, 0:S].astype(np.float32)
    X = np.empty((n, 1, T, S, S), np.float32)
    y = rng.integers(0, 4, n)
    step = np.array([[0, -1], [0, 1], [-1, 0], [1, 0]], np.float32)       # (dy, dx) per frame
    for i in range(n):
        a, b = rng.uniform(0.15, 0.4, 2)
        bg = 0.4 + 0.05 * np.sin(a * xx + rng.uniform(0, 6)) * np.cos(b * yy + rng.uniform(0, 6))
        c0, r, f = rng.uniform(10, 22, 2), rng.uniform(5, 8), rng.uniform(0.8, 1.5)
        for k in range(T):
            dy, dx = yy - (c0[0] + step[y[i], 0] * k), xx - (c0[1] + step[y[i], 1] * k)
            X[i, 0, k] = np.where(dy ** 2 + dx ** 2 < r ** 2, 0.7 + 0.25 * np.sin(f * (dx - dy)), bg)
    return torch.from_numpy(X), torch.from_numpy(y)

def tubelets(x):
    """(B, 1, T, S, S) -> (B, nt*ns, t*p*p), tokens in time-major order."""
    B, g = x.shape[0], S // p
    return x.reshape(B, nt, t, g, p, g, p).permute(0, 1, 3, 5, 2, 4, 6).reshape(B, nt * ns, -1)

def sincos(n, d):
    """Fixed 1D sine-cosine embedding of the flattened token index."""
    pos = torch.arange(n).float()[:, None]
    om = 1.0 / 10000 ** (torch.arange(0, d, 2).float() / d)
    return torch.cat([torch.sin(pos * om), torch.cos(pos * om)], 1)

def visible_ids(B, kind):
    """Indices of the visible tokens, (B, nt*VIS)."""
    if kind == "tube":                       # the same spatial positions in every time slot
        pos = torch.argsort(torch.rand(B, ns), 1)[:, :VIS]
        ids = (torch.arange(nt)[None, :, None] * ns + pos[:, None, :]).reshape(B, -1)
    else:                                    # "random": any nt*VIS of the 64 tokens
        ids = torch.argsort(torch.rand(B, nt * ns), 1)[:, : nt * VIS]
    return ids.sort(1).values

class Block(nn.Module):
    def __init__(self, d, heads=4):
        super().__init__()
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))
    def forward(self, z):
        h = self.n1(z)
        z = z + self.attn(h, h, h, need_weights=False)[0]   # joint space-time attention
        return z + self.mlp(self.n2(z))

class Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Linear(t * p * p, D)                 # tubelet embedding
        self.register_buffer("pos", sincos(nt * ns, D))
        self.blocks = nn.Sequential(Block(D), Block(D))
        self.norm = nn.LayerNorm(D)
    def forward(self, tok, ids=None):
        z = self.embed(tok) + self.pos
        if ids is not None:                                  # pretraining: visible tokens only
            z = torch.gather(z, 1, ids[..., None].expand(-1, -1, D))
        return self.norm(self.blocks(z))
    def features(self, x):
        return self(tubelets(x)).mean(1)                     # all tokens, average-pooled

class TinyVideoMAE(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = Encoder()
        self.proj = nn.Linear(D, DD)
        self.mask_token = nn.Parameter(torch.zeros(1, 1, DD))
        self.register_buffer("dpos", sincos(nt * ns, DD))
        self.decoder, self.dnorm = Block(DD), nn.LayerNorm(DD)
        self.head = nn.Linear(DD, t * p * p)                 # pixels of one tubelet
    def forward(self, x, kind):
        tok = tubelets(x)
        B, N, _ = tok.shape
        ids = visible_ids(B, kind)
        z = self.proj(self.encoder(tok, ids))
        full = self.mask_token.expand(B, N, DD).clone()
        full.scatter_(1, ids[..., None].expand(-1, -1, DD), z)   # visible tokens back in place
        pred = self.head(self.dnorm(self.decoder(full + self.dpos)))
        masked = torch.ones(B, N, dtype=torch.bool)
        masked.scatter_(1, ids, False)
        return ((pred - tok) ** 2)[masked].mean()            # MSE on masked tokens (raw pixels)

def pretrain(model, X, kind, steps=800, bs=32, lr=2e-3):
    opt = torch.optim.AdamW(model.parameters(), lr=lr, betas=(0.9, 0.95), weight_decay=0.05)
    sched = torch.optim.lr_scheduler.LambdaLR(
        opt, lambda s: min((s + 1) / 30, 0.5 * (1 + math.cos(math.pi * s / steps))))
    for _ in range(steps):
        loss = model(X[torch.randint(0, len(X), (bs,))], kind)
        opt.zero_grad(); loss.backward(); opt.step(); sched.step()

def linear_probe(enc, Xtr, ytr, Xte, yte, steps=300):
    """Logistic regression on frozen, standardized features."""
    with torch.no_grad():
        ftr, fte = enc.features(Xtr), enc.features(Xte)
    mu, sd = ftr.mean(0), ftr.std(0) + 1e-6
    head = nn.Linear(D, 4)
    opt = torch.optim.Adam(head.parameters(), lr=0.01, weight_decay=1e-3)
    for _ in range(steps):
        loss = F.cross_entropy(head((ftr - mu) / sd), ytr)
        opt.zero_grad(); loss.backward(); opt.step()
    return (head((fte - mu) / sd).argmax(1) == yte).float().mean().item()

def fine_tune(enc, Xtr, ytr, Xte, yte, steps=200, bs=32):
    """Encoder and linear head trained end to end on the labelled clips."""
    head = nn.Linear(D, 4)
    opt = torch.optim.AdamW(list(enc.parameters()) + list(head.parameters()), lr=1e-3, weight_decay=0.05)
    for _ in range(steps):
        i = torch.randint(0, len(ytr), (bs,))
        loss = F.cross_entropy(head(enc.features(Xtr[i])), ytr[i])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (head(enc.features(Xte)).argmax(1) == yte).float().mean().item()

Xte, yte = make_clips(1000, np.random.default_rng(1000))      # fixed test set
print("motion direction, 200 labelled clips, 1,000 test clips (chance 0.25)")
print("seed   linear probe: init / tube / random     fine-tune: init / tube / random")
res = []
for seed in range(4):
    rng = np.random.default_rng(seed)
    X_unlab, _ = make_clips(3000, rng)                        # labels discarded
    Xtr, ytr = make_clips(200, rng)
    row = []
    for kind in [None, "tube", "random"]:
        torch.manual_seed(seed)                               # same initial weights for all three
        model = TinyVideoMAE()
        if kind:
            pretrain(model, X_unlab, kind)
        state = {k: v.clone() for k, v in model.encoder.state_dict().items()}
        torch.manual_seed(seed + 100)
        lin = linear_probe(model.encoder, Xtr, ytr, Xte, yte)
        model.encoder.load_state_dict(state)
        torch.manual_seed(seed + 100)
        row.append((lin, fine_tune(model.encoder, Xtr, ytr, Xte, yte)))
    res.append(row)
    print(f"  {seed}        {row[0][0]:.3f} / {row[1][0]:.3f} / {row[2][0]:.3f}"
          f"            {row[0][1]:.3f} / {row[1][1]:.3f} / {row[2][1]:.3f}")
m = np.array(res).mean(0)
print(f"mean       {m[0, 0]:.3f} / {m[1, 0]:.3f} / {m[2, 0]:.3f}"
      f"            {m[0, 1]:.3f} / {m[1, 1]:.3f} / {m[2, 1]:.3f}")

torch 2.14.1
motion direction, 200 labelled clips, 1,000 test clips (chance 0.25)
seed   linear probe: init / tube / random     fine-tune: init / tube / random
  0        0.679 / 0.822 / 0.620            0.678 / 0.791 / 0.730
  1        0.708 / 0.740 / 0.749            0.250 / 0.738 / 0.795
  2        0.728 / 0.861 / 0.818            0.454 / 0.826 / 0.585
  3        0.619 / 0.746 / 0.864            0.343 / 0.813 / 0.811
mean       0.684 / 0.792 / 0.763            0.431 / 0.792 / 0.730

On all four seeds, the tube-masked encoder beat the untrained one under both protocols (0.740–0.861 against 0.619–0.728 linear; 0.738–0.826 against 0.250–0.678 fine-tuned). Random-masked pretraining helped too, except for seed 0’s linear probe. Tube masking had the higher means but lost to random masking on two seeds in linear probing and one in fine-tuning, so this toy does not separate the two schemes; the paper’s 1.3-point gap on SSV2 comes from far larger models and data.

Inference

The decoder is discarded; the encoder, with a classification head, sees all 1,568 tokens of each view, 180 GFLOPs per view for ViT-B (Table 7), and predictions are averaged over several clips × crops.

Performance

All without extra data or labels beyond the target dataset, top-1 (Tables 6 and 7; cost as GFLOPs per view × clips × crops):

Model	Dataset	Frames, resolution	Cost	Top-1
ViT-B (87M)	SSV2	16, $224^2$	180 × 2 × 3	70.8%
ViT-L (305M)	SSV2	32, $224^2$	1,436 × 1 × 3	75.4%
ViT-B	K400	16, $224^2$	180 × 5 × 3	81.5%
ViT-H (633M)	K400	16, $224^2$	1,192 × 5 × 3	86.6%
ViT-H	K400	32, $320^2$	7,397 × 4 × 3	87.4%

Pretraining lasted 2,400 epochs for SSV2 and 1,600 for K400; the $320^2$ model was fine-tuned from its $224^2$ counterpart. On AVA v2.2 action detection (average precision at IoU 0.5), a K400-pretrained ViT-B scored 26.7 mAP, 31.8 after supervised fine-tuning on K400 labels, and ViT-H with labels 39.5 (Table 5).

Strengths

• No image pretraining: a plain ViT trained from the target videos alone, transferring to small datasets and AVA detection.
• Simple, cheap pretraining: no tokenizer, negatives or momentum encoder; 3.2 times faster than MoCo v3 on SSV2 (Table 3).

Limitations

• Compute. Pretraining runs for 800–4,800 epochs; the V2 authors report over two weeks on 64 A100 GPUs for ViT-g with the original method.
• Small motions. The authors suspect that all tokens holding a very small object’s motion can be masked (supplementary §D).
• Pixel targets. Other methods predict features instead: MaskFeat, which VideoMAE’s related work describes as reconstructing HOG features, and V-JEPA (Bardes et al., 2024), whose arXiv abstract describes a feature-prediction objective without pixel reconstruction.

Historical Importance

VideoMAE (2022) showed that a plain ViT could be pretrained on video alone, and its tube masking at 90% carried over unchanged into VideoMAE V2’s encoder; the concurrent MAE-ST is summarized in masked autoencoder. Its finding that a few thousand in-domain videos suffice for self-supervised pretraining, in our reading, qualifies the habit in transfer learning of starting from the largest generic dataset.

Variants

VideoMAE V2 (CVPR 2023)

Dual masking. The encoder keeps tube masking at 90%; the decoder is masked too, reconstructing only a subset of cubes chosen by “running cell” masking (50% by default), which spreads the kept cubes over the clip. The loss covers only decoder-visible cubes that were hidden from the encoder (Eq. 2). With ViT-B pretrained on SSV2, SSV2 top-1 was 70.15% against 70.28% without decoder masking (Table 1), at 25.87 instead of 35.48 GFLOPs; random decoder masking scored 64.87%. For 1,200 epochs on 64 GPUs, pretraining time fell from 28.4 to 15.9 hours for ViT-B and from 356 hours, estimated from 5 epochs, to 241 hours for ViT-g (Table 2).

Data and progressive training. Pretraining used UnlabeledHybrid, about 1.35M clips from Kinetics, SSV2, AVA, WebVid2M and Instagram. A supervised post-pretraining stage on K710 (710 classes, 0.66M clips merged from Kinetics-400, -600 and -700; 658k training videos per the supplementary) preceded fine-tuning on the target. It raised K400 accuracy; on SSV2, the text reports, it lowered it.

ViT-g. The giant encoder has patch size 14, 40 blocks, width 1,408 and 1,011M parameters; the decoder stays at 4 blocks of width 512 (supplementary §A; the main text gives 512 for ViT-g and 384 for ViT-B).

Results. Without post-pretraining, ViT-g reached 87.2% on K400 (5 × 3 views) and 77.0% on SSV2 (2 × 3) (Tables 3 and 4); the authors suggest saturation without extra labels. With K710 post-pretraining, ViT-g reached 88.5% on K400 (Table 5); Table 6a lists 90.0% (64 frames at $266^2$, 2 × 3 views, 160.30 TFLOPs total). As ActionFormer features it reached 69.6 average mAP on THUMOS14 (Table 6g); for AVA, with VideoMAE’s detection pipeline, Table 6e lists 42.6 mAP without naming the version or model size. A ViT-B distilled from ViT-g reached 87.1% on K400 (supplementary Table 11).

Implementations

As of October 2026, the authors’ PyTorch code and checkpoints are in MCG-NJU/VideoMAE, mostly under CC BY-NC 4.0, and its README notes integration into Hugging Face Transformers. In OpenGVLab/VideoMAEv2, its MODEL_ZOO.md says, ViT-g weights require a request form; distilled ViT-S and ViT-B checkpoints download directly.