Video Foundation Models

A video foundation model is a large video encoder, almost always a vision transformer over space-time patches, that is pretrained once on broad video data with little or no human labelling and then reused, frozen or fine-tuned, for many downstream tasks: action recognition, temporal action localization, spatio-temporal action detection, video–text retrieval and video question answering. Papers use several abbreviations for the term: VFM in the Unmasked Teacher paper (Li et al., 2023), ViFM in VideoPrism (Zhao et al., 2024) and InternVideo2 (Y. Wang, K. Li, X. Li et al., 2024). The family is defined less by an architecture than by its training objectives, of which four recur: reconstructing masked pixels, predicting the masked features of a frozen teacher, aligning video with text by contrastive learning, and predicting the representation of hidden parts of a clip in a learned latent space. This article covers those objectives, the models that combine them, how such encoders are evaluated, and the data and compute their papers report. The VideoMAE article covers that model in detail, and foundation models covers the general concept and its disputes.

Definition

VideoMAE V2’s related-work section takes the term from the foundation-model report of Bommasani et al. (2021): models “pre-trained on broad data” that can be adapted to a wide range of downstream tasks (L. Wang, Huang, Zhao et al., 2023). Applied to video, three properties separate a foundation model from an ordinary pretrained action classifier, as these papers describe their goals:

• Breadth of pretraining data. The pretraining set is not the target dataset’s labelled training split but a mix of unlabelled clips (1.35 million in VideoMAE V2’s UnlabeledHybrid), video with noisy text such as speech transcripts (582 million clips in VideoPrism), or both.
• Breadth of downstream use. One checkpoint is evaluated on many tasks: VideoMAE V2 on recognition, spatial and temporal action detection; VideoPrism on 33 benchmarks in four task groups; InternVideo2 on more than 60 video and audio tasks.
• Adaptation without retraining from scratch. The encoder is kept and only a head, a probe or the whole network with a small learning rate is trained per task.

The term is applied loosely. VideoMAE V2 calls its self-supervised encoders video foundation models; UMT argues that image foundation models neglect motion and object interactions and calls “true” video foundation models underexplored; VideoPrism emphasizes a single frozen model across all tasks.

Intuition

A labelled video dataset is small compared with the video that exists: Kinetics-400 has about 240,000 training videos and Something-Something V2 about 169,000, as VideoMAE’s Table 2 counts them, against the more than 600 million clips VideoPrism pretrains on. A video foundation model tries to turn that unlabelled video into a representation that already separates what downstream tasks need, so that each task needs few labels and little compute.

Video makes this harder than images in two ways that shape every objective below. First, frames are redundant and correlated: neighbouring frames show almost the same pixels, so a pretext task that hides some patches can often be solved by copying from a neighbouring frame rather than by understanding the scene. VideoMAE calls this risk information leakage, ties it to temporal correlation, and answers it with tube masking (VideoMAE measures the shortcut). Second, many video labels can be read from appearance alone. V-JEPA’s authors, citing earlier work, note that many Kinetics classes can be inferred from the objects present, while Something-Something classes are decoupled from object identity and need motion (Bardes et al., 2024, §3.4). An encoder can therefore score well on appearance-heavy benchmarks without modelling motion, and papers in this family report both kinds of benchmark side by side for that reason.

Text attached to video, such as titles, speech transcripts or generated captions, supplies semantics cheaply but describes mostly appearance; VideoPrism’s authors cite prior work that captions mainly reveal appearance cues, while “contextual self-supervision” from masking helps learn motion (Zhao et al., 2024, §1). Much of the field’s development, described under Examples, consists of combining the two signals.

Formal Definition

Let a clip $x \in \mathbb{R}^{T \times H \times W \times 3}$ be cut into $N$ space-time tokens (tubelets of, typically, $2 \times 16 \times 16$ pixels, as in video transformers). A mask splits the token indices into a visible set $\mathcal{V}$ and a masked set $\mathcal{M}$. $E_\theta$ is the encoder that becomes the foundation model; everything else is discarded after pretraining.

Masked pixel reconstruction (VideoMAE, MAE-ST, VideoMAE V2, Hiera). A light decoder $D$ sees the encoded visible tokens plus learnable mask tokens and regresses the pixels of the hidden ones:

$$\mathcal{L}_{\text{pix}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \big\| D\big(E_\theta(x_{\mathcal{V}})\big)_i - \bar{x}_i \big\|_2^2 ,$$

where $\bar{x}_i$ is the token’s pixels, usually normalized per patch. The encoder sees only $|\mathcal{V}| \approx 0.1N$ tokens at a 90% masking ratio, which is where the training speed comes from (MAE-ST measured a 4.1-fold wall-clock speedup for ViT-L, Table 1).

Masked feature distillation (MVD, UMT, InternVideo2 stage 1, VideoPrism stage 2). The target is the output of a frozen teacher $g$, an image or video model pretrained earlier:

$$\mathcal{L}_{\text{distill}} = \frac{1}{|\mathcal{S}|} \sum_{i \in \mathcal{S}} \big\| h\big(E_\theta(x_{\mathcal{V}})\big)_i - g(x)_i \big\|_2^2 .$$

The methods differ in the set $\mathcal{S}$ and the head $h$. MVD predicts teacher features of masked tokens with a decoder; UMT aligns only the unmasked tokens with a linear projection to CLIP’s features, with no decoder ($\mathcal{S} = \mathcal{V}$); InternVideo2’s first stage aligns unmasked tokens with two teachers, InternViT-6B and VideoMAE V2’s ViT-g, as a weighted sum of two such terms (its Eq. 1); VideoPrism’s second stage predicts the first-stage model’s embeddings of all tokens, by cosine distance, plus a global embedding.

Contrastive video–text alignment (VideoCLIP, ViCLIP, VideoPrism stage 1, InternVideo2 stage 2). With pooled video and text embeddings $f^V_i, f^T_i$ for the $i$-th pair in a batch of $B$, cosine similarity $s$ and temperature $\tau$, the symmetric InfoNCE loss is

$$\mathcal{L}_{\text{con}} = -\sum_{i=1}^{B} \left[ \log \frac{e^{s(f^V_i, f^T_i)/\tau}}{\sum_{j} e^{s(f^V_i, f^T_j)/\tau}} + \log \frac{e^{s(f^T_i, f^V_i)/\tau}}{\sum_{j} e^{s(f^T_i, f^V_j)/\tau}} \right],$$

the CLIP objective with clips in place of images (contrastive learning derives it). It is the form printed in the InternVid paper (Eq. 1) and in InternVideo2 (Eq. 3).

Prediction in latent space (V-JEPA, V-JEPA 2). A predictor $P_\phi$ maps the encoded visible tokens and the positions $\Delta_y$ of the masked ones to the representation that a target encoder $E_{\bar\theta}$, an exponential moving average of $E_\theta$, gives the full clip:

$$\mathcal{L}_{\text{JEPA}} = \frac{1}{|\mathcal{M}|} \sum_{k \in \mathcal{M}} \big\| P_\phi\big(E_\theta(x_{\mathcal{V}}), \Delta_y\big)_k - \operatorname{sg}\big(E_{\bar\theta}(x)\big)_k \big\|_1 ,$$

with $\operatorname{sg}$ a stop-gradient (Bardes et al., 2024, Eqs. 1–2). Without the stop-gradient and moving average, the authors note, a constant encoder would solve the regression trivially; they chose the L1 distance as “more stable” than L2.

Evaluation by probing. After pretraining, $E_\theta$ is frozen and gives token features $s_1, \dots, s_N$. A linear probe trains a classifier on their average $\tfrac{1}{N}\sum_i s_i$. An attentive probe learns a query $q$ and pools by cross-attention,

$$z = q + \sum_{i=1}^{N} \frac{\exp(q^\top W_k s_i)}{\sum_j \exp(q^\top W_k s_j)} \, W_v s_i ,$$

followed in V-JEPA by a two-layer MLP, a LayerNorm and a linear classifier (Appendix D.1); only $q$, $W_k$, $W_v$ and the head are trained. V-JEPA 2 uses four transformer blocks, the last with the cross-attention (§5). Fine-tuning updates $E_\theta$ itself, usually with layer-wise learning-rate decay, as in fine-tuning.

Properties

The masking strategy depends on the target

With pixel targets, the two papers of 2022 disagree about masking. VideoMAE’s ablation favoured tube masking over random masking at 90% for ViT-B (69.6% against 68.3% on Something-Something V2 and 80.0% against 79.5% on Kinetics-400, fine-tuned; Tong et al., 2022, Table 1). MAE-ST, with ViT-L pretrained for 800 epochs on Kinetics-400, found the opposite: spacetime-agnostic random masking at 90% reached 84.4% top-1 after fine-tuning, space-only (“tube”) masking at 90% 83.5%, block masking at 75% 83.2% and time-only masking at 75% 79.1% (Feichtenhofer et al., 2022, Table 2a). MAE-ST’s authors argue that structured masks keep less information at a very high ratio and make the task too hard; VideoMAE’s argue that random masks leak. The settings differ (model size, schedule, dataset), so the tables do not settle which argument is right.

Other targets shift the balance again. With a frozen CLIP teacher and unmasked-token alignment, UMT measured tube and random masking as equal on Something-Something V2 (70.2% each) and random slightly ahead on Kinetics-400 (84.6% against 84.3%), with attention-guided “semantic” masking best on Kinetics (84.9%) (Li et al., 2023, Table 2). With latent prediction, for a ViT-L pretrained on Kinetics-710 and Something-Something V2, V-JEPA’s random-tube masking at 90% gave frozen attentive-probe accuracies of 51.5% (Kinetics-400), 46.4% (Something-Something V2) and 55.6% (ImageNet), far below its multi-block masking at 72.9%, 67.4% and 72.8%, which hides a union of spatial blocks (eight small or two large) repeated over all frames (Bardes et al., 2024, Table 4). VideoPrism adds a different guard: it shuffles the encoder’s output tokens before the decoder so that the decoder cannot simply copy visible tokens through (Zhao et al., 2024, §2.3.2). In our reading, the common thread is that each objective needs its own way of removing trivial solutions, and the mask is one of the tools.

Feature targets against pixel targets

V-JEPA compared the two objectives with everything else fixed: ViT-L, VideoMix2M, 90,000 iterations at batch 3,072, multi-block masks, a learning-rate sweep for both. Predicting features instead of normalized pixels raised frozen attentive-probe accuracy (single centre view) from 68.6% to 73.7% on Kinetics-400 and from 73.3% to 74.8% on ImageNet, left Something-Something V2 almost unchanged (66.0% and 66.2%) and fine-tuned Kinetics-400 accuracy nearly unchanged (85.4% and 85.6%) (Table 1). UMT found pixel reconstruction weaker than unmasked-token alignment to CLIP for ViT-B (67.1% against 70.2% on Something-Something V2, 78.8% against 84.9% on Kinetics-400), at 44.0 against 16.0 GB of memory, and adding the pixel loss to the alignment lowered Kinetics and MSR-VTT retrieval results, which the authors read as a conflict between low-level reconstruction and high-level alignment (Table 1).

Frozen and fine-tuned evaluation rank models differently

Fine-tuning hides much of what probing reveals. In V-JEPA’s comparison of ViT-L and Hiera-L models, VideoMAE, Hiera-L and V-JEPA were within 2 points of each other after fine-tuning on Kinetics-400 (85.4%, 87.3%, 85.6%) but 5 points apart under the frozen attentive probe (77.8%, 75.5%, 80.8%) (Table 5). VideoMAE V2’s billion-parameter ViT-g, the strongest fine-tuned model of its paper, gave 71.2% on Kinetics-400 with a frozen attentive probe in V-JEPA’s evaluation, below the 630-million-parameter VideoMAE ViT-H at 79.8% (Table 6).

The probe matters as much as the protocol. Attentive pooling raised V-JEPA ViT-L from 56.7% to 73.7% on Kinetics-400 and from 50.1% to 66.2% on Something-Something V2, single view, over average pooling with a linear classifier (Table 3; the caption gives the Kinetics gain as 17.3 points, the table’s numbers as 17.0), and VideoMAE ViT-L from 52.5% to 77.8% on Kinetics-400 (Appendix Table 12). That table’s views are not consistent with the rest of the paper: its caption gives 16 frames × 5 clips × 3 crops for Kinetics and 16 × 2 × 2 for Something-Something V2, but its V-JEPA linear entries equal Table 3’s single-view average-pooling scores and its Kinetics attentive entries equal Table 5’s, whose caption gives 16 × 8 × 3; and its attentive score for VideoMAE on Something-Something V2, 61.2%, differs from the 65.5% of Table 5 (16 × 2 × 3). The authors explain that the feature-prediction loss is unnormalized, so there is no a priori reason for a linearly separable subspace (§4.3). The same frozen model is also reported differently by different papers: VideoMAE V2-g scores 71.2% on Kinetics-400 in V-JEPA (V-JEPA’s attentive probe, 16 frames × 8 clips × 3 crops at $224^2$; Table 6), 82.1% in VideoPrism’s VideoGLUE evaluation (a multi-head attention pooling probe; the table does not give frames or views; Table 2) and 82.8% among the results “reported in the literature” that V-JEPA 2 lists, which its authors describe as obtained with a similar frozen protocol but possibly a different attentive head (Table 4). V-JEPA 2 does not say where its figure comes from; in our observation, its Something-Something V2 entry for the same model, 56.1%, equals VideoPrism’s, while its Kinetics entry differs. V-JEPA’s own ViT-H scores 71.4% on Something-Something V2 in its paper (one cross-attention layer, $224^2$, 16 × 2 × 3) and 74.3% when V-JEPA 2 re-evaluated it with a four-block probe at $256^2$, also 16 × 2 × 3 (V-JEPA 2, Table 4). Comparisons are meaningful only within one protocol, as V-JEPA 2’s authors state when they separate results “reported in the literature” from encoders evaluated with their own protocol.

Appearance and motion pull in different directions

Text supervision and image initialization favour appearance; masked prediction from video favours motion. Under V-JEPA’s frozen protocol, the image model DINOv2 ViT-g reached 83.4% on Kinetics-400 but 50.6% on Something-Something V2, while V-JEPA ViT-H reached 82.0% and 71.4% (Table 6). In V-JEPA 2’s common protocol, the contrastively trained InternVideo2-s2-1B led on Kinetics-400 (89.4% against 86.6%), COIN and ImageNet, while V-JEPA 2 ViT-g led on Something-Something V2 (75.3% against 69.7%), Diving-48 and Jester (Table 4). VideoPrism’s two stages are designed around this: the authors found that the second, masked stage alone lost accuracy on appearance-heavy tasks relative to the first, contrastive stage, which they attribute to catastrophic forgetting, and added the global distillation loss to keep it (§2.3.2).

Data composition and quality

Several papers find that a larger pretraining set helps on average but not on every task. In V-JEPA’s ablation at a fixed 90,000 iterations, the average over Kinetics-400, Something-Something V2 and ImageNet rose from 70.9 (Kinetics-710, 700,000 clips) to 71.5 (VideoMix2M, 2 million), but the best Kinetics score came from Kinetics-710 alone and the best Something-Something score from Kinetics-710 plus Something-Something V2 (Table 2). For contrastive video–text learning, ViCLIP trained on a 10-million-clip subset of InternVid filtered for caption–frame similarity and diversity (InternVid-10M-FLT) gave higher zero-shot Kinetics-400 accuracy than ViCLIP trained on 200 million clips; the authors conjecture that many clips from the same video act as false negatives in the contrastive loss. After fine-tuning, the order reversed: the 200-million-clip model was more accurate on Kinetics-400 and Something-Something V2 (Y. Wang, He, Li et al., 2024, InternVid paper, §5.1, Tables 2–3). V-JEPA 2 reports a 1.4-point average gain for a ViT-L from curating YT-Temporal-1B by retrieval over training on it uncurated (§2.3).

Compute, as reported

The papers report compute in different units, and few give totals; the figures below are as printed.

Model	Encoder	Pretraining compute as reported
VideoCLIP (2021)	6-layer video transformer on frozen S3D features	8 V100 GPUs, 25 epochs, about 1 day
MAE-ST (2022)	ViT-L	128 A100 GPUs; 35.8 hours load and compute for 800 epochs
VideoMAE V2 (2023)	ViT-g, 1,011M parameters	241 hours for 1,200 epochs on 64 GPUs with dual masking (356 hours estimated without)
UMT (2023)	ViT-L	32 A100 GPUs, within 6 days for both stages
ViCLIP (2024)	ViT-L	64 A100 GPUs, 3 days, 50M video–text pairs
InternVideo2 (2024)	6B parameters	256 A100 GPUs for 18 days (stage 1) and 14 days (stage 2), 64 for 3 days (stage 3)
V-JEPA (2024)	ViT-H	90,000 iterations at batch 3,072 on A100 80 GB GPUs; the text gives no total, and Figure 5 plots wall-clock times extrapolated from one GPU
V-JEPA 2 (2025)	ViT-g, 1B	about 60 GPU-years estimated for full-resolution training at 64 × $384^2$; progressive resolution reported as an 8.4-fold reduction

UMT’s authors compare their 32 A100 GPUs for 6 days with the 2,048 TPU v4 chips for 5 days used by CoCa and report a 70-fold reduction in carbon emissions, counting the cost of the CLIP teacher as zero because it was publicly available (§1, Figure 1). Teacher-based methods move part of their compute into the teacher; V-JEPA’s authors make the same point about MVD, which processes more samples because of its teacher’s pre-pretraining (Appendix E.2).

Examples

How the family developed

Video–text contrastive models (2021). VideoCLIP (Xu et al., EMNLP 2021) trained video and text transformers with InfoNCE on HowTo100M, 1.1 million instructional videos with speech transcripts. Video features came from a frozen S3D network at one token per second, so the model saw up to 32 seconds of video; two changes over earlier work were positives built from loosely overlapping (rather than exactly aligned) clip–transcript pairs and batches formed from clusters of similar videos to obtain hard negatives. Without target labels it reached 22.7 R@1 on YouCook2 text-to-video retrieval, above the 15.1 of MIL-NCE (Table 1).

Masked video modelling (2022). VideoMAE and MAE-ST (Feichtenhofer et al., NeurIPS 2022) carried the masked autoencoder to video at 90% masking, with tube and random masks respectively, and showed that a plain ViT could be pretrained on video alone. MAE-ST also pretrained on 1 million random, uncurated Instagram videos and, with ViT-L pretrained for 800 epochs, reached 84.4% on Kinetics-400 after fine-tuning, the same as with Kinetics-400 itself (Table 4).

Combining the two (2022–2023). InternVideo, released as a technical report in December 2022, is described in its repository as coordinating a masked video encoder and a video–language contrastive model “in a learnable manner”; the README lists VideoMAE-B, -L and -H pretrained on a 1-million-clip UnlabeledHybrid set and a multimodal InternVideo-MM-L-14 trained on “WebVid10M+Self-collected (14M)”, and reports 91.1% top-1 on Kinetics-400 and 77.2% on Something-Something V2 (the InternVideo2 paper lists the Kinetics result as an ensemble). The report itself, on arXiv, was not read for this article. UMT criticized InternVideo’s dependence on CLIP pretraining and its training cost, and trained a ViT from scratch whose unmasked tokens are aligned to a frozen CLIP teacher (stage 1, Kinetics-710 video only), then added video–text contrastive, matching and masked-language losses on 5 to 25 million image– and video–text pairs (stage 2) (Li et al., 2023, §3–4). MVD (R. Wang, Chen, Wu et al., CVPR 2023, a different Wang et al. from VideoMAE V2’s) distils from MAE-pretrained image and video teachers at once; the authors report that video teachers help on temporally heavy tasks and image teachers on spatially heavy ones.

Scaling (2023). VideoMAE V2 (L. Wang, Huang, Zhao et al., CVPR 2023) masks the decoder as well as the encoder to make a billion-parameter ViT-g trainable, pretrains on 1.35 million unlabelled clips, and adds a supervised post-pretraining stage on Kinetics-710; its details, ablations and results are in VideoMAE. Hiera (Ryali et al., ICML 2023) argued that a hierarchical transformer pretrained with MAE does not need the specialized modules of MViTv2; it masks coarse “mask units” ($2 \times 32 \times 32$ pixels in video) so that hierarchy and sparse MAE are compatible, and its Hiera-L reached 87.3% on Kinetics-400 fine-tuned, against 85.2% for MAE-pretrained ViT-L (Table 4).

Data built for video–text learning (2023–2024). InternVid (Y. Wang, He, Li et al., ICLR 2024) collected over 7 million YouTube videos, nearly 760,000 hours, cut into 234 million clips with captions generated by a multi-scale captioning pipeline. Its ViCLIP, a CLIP ViT-L with spatiotemporal attention trained contrastively with random patch masking and a short unmasked phase at the end, reached 64.8% top-1 zero-shot on Kinetics-400 when trained on InternVid-10M-FLT (Table 2).

Frozen general-purpose encoders (2024). VideoPrism (Zhao et al., ICML 2024) pretrains a factorized ViViT-style encoder (ViT-g spatial encoder) first contrastively on 36 million manually captioned clips and 582 million clips with noisy text, then by masked modelling of the first stage’s token embeddings with token shuffling and global distillation, and evaluates one frozen checkpoint on all tasks; its authors report state-of-the-art results on 31 of 33 benchmarks. InternVideo2 (Y. Wang, K. Li, X. Li et al., ECCV 2024, by an overlapping team) scales the encoder to 6 billion parameters in three stages, unmasked-token distillation from two teachers, contrastive alignment with text, audio and speech, and next-token prediction through a language model, on 402 million entries in total. With a frozen attentive probe it reports 88.8% on Kinetics-400 and 67.7% on Something-Something V2 for the 6B model after stage 2 (Table 3); fine-tuned after stage 1, the 6B model reaches 92.1% on Kinetics-400 with 16 frames at $224^2$ (Table 2).

Latent prediction without teachers or text (2024–2025). V-JEPA (Bardes et al., published in TMLR in August 2024; its tables are cited here from the February 2024 version on Meta’s research site, as the TMLR copy was not read) uses no pretrained image encoder, no text, no negatives and no pixel reconstruction. Trained on VideoMix2M (HowTo100M, Kinetics-400/600/700 and Something-Something V2, about 2 million videos after removing validation overlap) for 90,000 iterations, its ViT-H/16 at $384^2$ reached 81.9% on Kinetics-400, 72.2% on Something-Something V2 and 77.4% on ImageNet with a frozen one-layer attentive probe (Table 6; the abstract’s 77.9% on ImageNet uses a two-layer probe, §5.2). V-JEPA 2 (Assran et al., 2025) keeps the objective, replaces absolute position embeddings with 3D rotary embeddings, and scales data to 22 million samples (VideoMix22M, over 1 million hours, including a curated YT-Temporal-1B and 1 million ImageNet images), the encoder to a 1-billion-parameter ViT-g and training to 252,000 iterations, with a final phase at higher resolution and up to 64 frames (the text counts the 12,000-iteration cool-down within the 252,000; the caption of Figure 5 describes 252,000 iterations followed by 12,000 more). Its ViT-g at $384^2$ reached 77.3% on Something-Something V2 with a frozen four-block attentive probe and 64 frames × 2 clips × 3 crops (Table 4). The report then trains an action-conditioned predictor on robot video for planning, a use beyond the encoder this article covers.

Common Misconceptions

• “A bigger fine-tuned number means better features.” Fine-tuning re-learns much of the network. VideoMAE V2-g leads its paper’s fine-tuning tables yet trails smaller models under V-JEPA’s frozen probe; which model is “better” depends on whether it will be frozen.
• “Linear-probe accuracy measures the representation.” It measures linear separability of the average token. Attentive pooling changed V-JEPA’s Kinetics-400 accuracy by 17 points, and InternVideo2’s table lists 25.8% for VideoMAE V2-H with a linear probe on Kinetics-400 (Table 4), against 82.0% for its own stage-1 6B model in the same table.
• “Tube masking is always better than random masking.” VideoMAE’s and MAE-ST’s ablations point in opposite directions, and with distillation targets UMT found the two equal on Something-Something V2. The answer depends on the target and the setting.
• “Kinetics accuracy shows motion understanding.” An image model applied frame by frame, DINOv2 ViT-g, scored 83.4% on Kinetics-400 with frozen features and 50.6% on Something-Something V2 (V-JEPA, Table 6). A video foundation model’s motion modelling shows on motion-heavy benchmarks.
• “More pretraining video always helps.” ViCLIP’s zero-shot accuracy was higher with a filtered 10 million clips than with 200 million, and V-JEPA’s best per-task scores came from task-matched subsets, not the largest mix.

Where It Is Used

Video foundation models serve as backbones or frozen feature extractors for action recognition on Kinetics, Something-Something, UCF101 and HMDB51; for temporal action localization, where VideoMAE V2 and InternVideo2 provide features to ActionFormer; for spatio-temporal action detection on AVA; and for action anticipation on EPIC-KITCHENS-100, where V-JEPA 2 reports a mean-class recall@5 of 39.7 for actions on the validation set with its frozen ViT-g at $384^2$ and an attentive probe (Table 5). Encoders trained with text, such as ViCLIP and the CLIP-style InternVideo2, also support zero-shot classification and video–text retrieval, and InternVideo2’s third stage connects the encoder to a language model as the vision component of a vision–language model for video question answering. InternVideo2’s paper also uses its stage-1 encoder as a Mask2Former backbone for video instance segmentation (Table 8), a task related to video object segmentation.

As of November 2026, the V-JEPA repository releases ViT-L and ViT-H checkpoints and attentive probes under CC BY-NC 4.0; the V-JEPA 2 repository, mostly under the MIT licence, releases V-JEPA 2 checkpoints from ViT-L to ViT-g and, since March 2026, V-JEPA 2.1 models; and the InternVideo repository (Apache 2.0) links InternVideo2 checkpoints and the later InternVideo2.5, InternVideo3 and InternVideo-Next releases.