Video Generation

Video generation is the task of synthesizing a video: a sequence of frames that looks like real footage both frame by frame and over time. The input may be nothing (unconditional generation), a class label, a text prompt (text-to-video), a still image (image-to-video, or image animation), or the first frames of a clip (video prediction, also called future frame prediction). It extends image generation along the time axis, and it borrows most of its machinery from text-to-image generation: since 2022 most systems have been diffusion models, usually latent diffusion models whose image backbone is extended with temporal layers. The other half of the background is video understanding, whose datasets, 3D networks and video transformers generative models reuse as building blocks and as evaluators.

This article is the overview: it defines the task, explains what time adds to the difficulty, follows the model families from the first video GANs of 2016 to the diffusion transformers of 2024, and covers evaluation, data and risks.

Problem Definition

A video clip is an array $x \in \mathbb{R}^{T \times H \times W \times 3}$ of $T$ frames, sampled at some frame rate. Training data are clips, possibly paired with captions or labels, drawn from an unknown distribution. The task is to sample from $p(x)$, or from a conditional distribution $p(x \mid c)$ where $c$ is a caption, an image, a set of past frames, or a combination. As in text-to-image generation, a condition underspecifies its video (a caption leaves the camera path, the timing and most of the motion open), so a good model returns varied clips for one input.

Three numbers define a clip’s size: its length, its resolution and its frame rate. They multiply. Imagen Video’s highest-resolution output, 128 frames of $1280 \times 768$ at 24 frames per second (about 5.3 seconds), contains about 126 million pixels (Ho et al., 2022b, §2), 160 times as many as one $1024 \times 768$ image (our arithmetic); much of the design described below is about managing that cost.

Generated video is judged on three properties that pull against each other:

• Frame quality: each frame on its own should look like a natural image.
• Temporal consistency and motion: objects keep their identity, shape and texture from frame to frame, and they move plausibly; a clip that is consistent because nothing moves fails this too.
• Condition fidelity: the clip shows what the prompt, image or past frames ask for, including actions and how they unfold over time.

Input and Output

The output is a fixed or variable number of frames at a fixed resolution and frame rate, often produced in stages. Imagen Video’s cascade starts from 16 frames of $40 \times 24$ at 3 frames per second and alternates spatial and temporal super-resolution models until it reaches 128 frames of $1280 \times 768$ at 24 frames per second (Ho et al., 2022b, Fig. 6). Make-A-Video adds a frame-interpolation network to raise the frame rate (Singer et al., 2023, §3.3). Later systems generate directly in a compressed spacetime latent and decode once: Movie Gen Video produces videos of up to 16 seconds at 16 frames per second at $768 \times 768$ pixels (scaled for other aspect ratios), and a separate spatial upsampler converts them to 1080p (Polyak et al., 2024, §3, §3.1.5), and OpenAI’s report says Sora can generate up to a minute of high-definition video and sample sizes from widescreen $1920 \times 1080$ to vertical $1080 \times 1920$ (Brooks et al., 2024).

The inputs by variant:

• Text is encoded by a frozen language or text–image model and read by the generator through cross-attention, as in text-to-image models; Imagen Video uses a frozen T5-XXL encoder (Ho et al., 2022b, §2).
• An image is usually given as the first frame. Stable Video Diffusion’s image-to-video model replaces the text embedding with the CLIP image embedding of the conditioning frame and concatenates a noise-augmented copy of that frame, repeated across the frames, to the denoiser’s input (Blattmann et al., 2023b, §4.3).
• Past frames are given as context, for video prediction or to extend a clip, by masking: the context frames are fed in alongside a binary mask marking which frames to generate (Blattmann et al., 2023a, §3.2).

Variants

• Unconditional and class-conditional generation, the usual benchmark setting of the GAN and early diffusion papers, for example on UCF101 at $64 \times 64$ or $128 \times 128$ (Ho et al., 2022a, §4.1).
• Text-to-video, the setting of most of the systems described below from 2022 on.
• Image-to-video, which animates a still image; Generative Image Dynamics (Li et al., 2024) turns a single photograph into a looping video of oscillating motion, such as trees and flowers swaying in the wind, and won a CVPR 2024 Best Paper award (CVPR 2024 awards page).
• Video prediction, which continues given frames, and world models, which predict future frames given actions (see Video Prediction and World Models below).
• Video extension and interpolation: generating frames before, after or between given frames. Sora’s report shows clips extended backward and forward in time and interpolations between two input videos (Brooks et al., 2024).
• Story or time-varying prompts: Phenaki generates a clip whose prompt changes over time, which its authors present as the first study of this setting (Villegas et al., 2023).
• Video editing and stylization, which change an existing clip according to a prompt. The image versions of these methods are covered in image-to-image translation and image editing; Sora’s report applies SDEdit to video (Brooks et al., 2024).

Challenges

Temporal consistency. An image generator run once per frame produces frames that are each plausible and jointly incoherent. Make-A-Video’s authors point out that their network, initialized from an image model with identity temporal layers, generates frames that each match the text but lack temporal coherence (Singer et al., 2023, §3.2.1). Even the autoencoder of a latent model matters: Blattmann et al. (2023a, §3.1.1) found that an image-trained decoder causes flickering when it decodes a coherent latent sequence, and fine-tuned its decoder on video with a 3D-convolutional discriminator.

Motion. Plausible motion is harder to learn than plausible appearance, and the easy failure is too little of it. Stable Video Diffusion’s authors removed near-static clips from their training data by optical flow and still note that their model sometimes generates too little motion (Blattmann et al., 2023b, §3.1 and Appendix A). Sora’s report lists physical failures such as glass that does not shatter correctly, eating that does not always change an object’s state, and incoherence that develops in long samples (Brooks et al., 2024).

Compute. Cost grows with the number of frames on top of the cost of each frame. Video Diffusion Models trained on 16-frame segments of videos that are hundreds or thousands of frames long, and generated longer videos by extension (Ho et al., 2022a, §3.1); Harvey et al. (2022, §1) describe photorealistic image models as already close to the memory limits of current hardware, with a long video needing at least the resources of all its frames. Stable Video Diffusion’s authors list slow sampling and high GPU memory requirements among its limitations (Blattmann et al., 2023b, Appendix A).

Data. Paired video and text are scarce compared with image and text. Phenaki’s authors contrast billions of image–text pairs in LAION-5B with about 10 million videos in WebVid (Villegas et al., 2023, §1), and Make-A-Video was designed to learn text–visual correspondence from images and motion from unlabeled video, so that it needs no paired text–video data (Singer et al., 2023, §1). Web video brings its own problems: Stable Video Diffusion’s curation pipeline detects scene cuts so that cuts and fades do not leak into generated videos, and doing so yielded about four times as many clips as the unprocessed data, many more cuts than the metadata indicated (Blattmann et al., 2023b, §3.1, Fig. 2).

Approaches

The field moved through three families, roughly in the order of image generation itself: generative adversarial networks from 2016, autoregressive models over discrete tokens from 2021, and diffusion models from 2022, first in pixel space and then in a compressed latent space, with transformer denoisers since 2024. In our reading, each transition followed the corresponding result for images and was driven by scale: the later families could reuse image models and image data, and train on larger and more varied video.

GANs (2016–2020)

VGAN (Vondrick, Pirsiavash & Torralba, 2016) extended the DCGAN recipe to video with fractionally strided spatio-temporal convolutions, generating 32 frames of $64 \times 64$, a little over a second. Its two-stream generator composes a moving foreground, produced by 3D convolutions, with a static background image replicated over time, through a learned mask, $G(z) = m(z) \odot f(z) + (1 - m(z)) \odot b(z)$ (§2.2), which builds the assumption of a stationary camera into the model. It was trained on scene-filtered subsets of over two million unlabeled Flickr videos (§3.1); a variant with an image encoder in front of the generator, trained to start from a given image, produced futures that were rarely correct but often had plausible motion (§3.4). TGAN (Saito, Matsumoto & Saito, 2017) split the generator in two: a temporal generator maps one latent vector to a sequence of per-frame latents, and an image generator renders each frame; it was trained as a Wasserstein GAN with singular value clipping for stability. MoCoGAN (Tulyakov et al., 2018) decomposed the latent code into a content part held fixed for the clip and a motion part produced by a recurrent network (a GRU) at every step, and trained with two discriminators, one judging single frames and one judging clips; because motion is a trajectory in latent space, the same content could be shown with different motions, and clips of different lengths generated (§1–§3).

These models stayed at small resolutions and short clips, on narrow datasets such as UCF101, golf scenes, moving digits or faces (Saito et al., 2017, §6.1; Tulyakov et al., 2018, §4). The GAN article describes the training instabilities they inherited.

Autoregressive and token models (2021–2024)

Following DALL·E’s recipe for images, these models compress video into a grid of discrete tokens and train a transformer to generate the tokens. VideoGPT (Yan et al., 2021) learns its tokens with a VQ-VAE (variational autoencoder with a learned discrete codebook) built from 3D convolutions and axial attention, and models them with a GPT-like transformer; its authors offered it as a minimal, reproducible reference and reported an FVD of 103.3 on BAIR robot pushing (Table 1). Phenaki (Villegas et al., 2023) introduced C-ViViT, a tokenizer with causal attention in time so that each frame’s tokens depend only on earlier frames, which lets it encode videos of any length; a bidirectional masked transformer in the style of MaskGIT generates the tokens from text, and clips are extended by re-encoding the last frames and generating more, optionally under a new prompt (§2). It was trained jointly on image–text pairs and a smaller set of video–text pairs, so that concepts seen only in images transfer to video (§2.2).

VideoPoet (Kondratyuk et al., 2024) treated video generation as language modeling: a decoder-only transformer over a single vocabulary of MAGVIT-v2 visual tokens and SoundStream audio tokens, with text given as T5 embeddings, pretrained on a mixture of tasks (unconditional generation, text-to-video, image-to-video, future prediction, inpainting and outpainting, stylization, audio-to-video, video-to-audio and audio–video continuation) and then adapted to specific ones (§3, §4.1). Its tokenizer encodes 17 frames of $128 \times 128$ video at 8 frames per second, 2.125 seconds, into a $5 \times 16 \times 16$ latent grid, 1,280 tokens (§3.1). The paper received an ICML 2024 Best Paper award (ICML 2024 awards page).

Diffusion in pixel space (2022)

Video Diffusion Models (Ho et al., 2022a) carried the image diffusion U-Net over to video with two changes: each 2D convolution becomes a space-only 3D convolution ($1 \times 3 \times 3$), and each spatial attention block is followed by a temporal attention block that attends along the frame axis, with relative position embeddings (§3). Masking the temporal attention turns the network back into an image model, so it can be trained jointly on videos and independent images, which the authors found important for sample quality (§3, §4.3.1). They also introduced reconstruction guidance for extending a sample in time or resolution (see Long Video below), and reported the first results on a large text-conditioned video task, trained on 10 million captioned videos (§4.3).

Imagen Video (Ho et al., 2022b) scaled the idea into a cascade of seven video diffusion models, a 5.6-billion-parameter base model with three temporal and three spatial super-resolution models, all conditioned on a frozen T5-XXL encoder and trained on 14 million internal video–text pairs, 60 million internal image–text pairs and LAION-400M (§2, §3). The base model uses temporal attention; the super-resolution models use cheaper temporal convolutions, since temporal attention brought no significant improvement there in initial experiments, which the authors attribute to the strong temporal correlation already present in their conditioning input (§2.3). They apply classifier-free guidance and progressive distillation to 8 sampling steps per stage (§2.7, Fig. 14), and did not release the model, citing problematic training data and remaining social biases (§4).

Make-A-Video (Singer et al., 2023) started from a text-to-image model and learned motion from unlabeled video. Its pseudo-3D convolution follows each pretrained 2D convolution with a new 1D temporal convolution, and its pseudo-3D attention follows each pretrained spatial attention layer with a temporal one; the new layers are initialized to the identity so that training starts from the image model (§3.2). A spatio-temporal decoder, a frame-interpolation network and two super-resolution models complete the pipeline (§3).

Latent video diffusion (2023)

Align your Latents, or Video LDM (Blattmann et al., 2023a), applied the latent diffusion model to video: pretrain an image LDM, insert temporal layers after its spatial layers, and train only those on encoded video, with a learned mixing factor per layer so that skipping the temporal layers recovers the image model exactly (§3.1, Fig. 4). Because video training touches only the temporal layers (after a brief fine-tuning of the spatial layers on video frames), the authors could turn the public Stable Diffusion into a text-to-video model, at up to $1280 \times 2048$ with a video fine-tuned upsampler (§4.2), and reuse the same temporal layers with fine-tuned (personalized) image checkpoints. On real driving scenes at $512 \times 1024$ they also trained prediction models conditioned on context frames, and generated driving videos of up to five minutes (§3.2, §4.1).

Stable Video Diffusion (Blattmann et al., 2023b) kept that architecture but fine-tuned the whole network, and focused on data and training stages: text-to-image pretraining (Stable Diffusion 2.1), video pretraining on a large curated set, and high-quality video fine-tuning at higher resolution (§3). The initial Large Video Dataset held 577 million clips, 212 years of footage (Table 1; the text rounds to 580 million); curation removed clips with scene cuts, little motion by optical flow, much written text by OCR, or low aesthetic and text–image scores (§3.1, Table 1). The image-to-video models generate 14 or 25 frames at $576 \times 1024$ from one image (§4.3, Fig. 5), and the code and weights were released (abstract).

Diffusion transformers and large systems (2024–)

By 2024 the largest systems had replaced the U-Net by a transformer over spacetime patches of a compressed latent, the video form of the Diffusion Transformer (DiT) described in the latent diffusion model article. What is known about the largest systems comes from their developers’ reports, which state what their authors chose to disclose.

• Sora (OpenAI, February 2024). The technical report (Brooks et al., 2024) describes a video compression network that compresses both time and space, a diffusion transformer trained on spacetime latent patches of videos and images at their native durations, resolutions and aspect ratios, captions written by a trained captioning model, and prompting with images or videos for animation, extension and editing. It states that model and implementation details are not included. Its claims about emerging simulation abilities, such as 3D consistency and long-range coherence (which it says Sora achieves “often, though not always”), are the authors’ qualitative observations, and the report lists failures alongside them.
• Movie Gen (Meta, October 2024). The report (Polyak et al., 2024) is a detailed paper. A Temporal Autoencoder (TAE) compresses video 8 times along each of time, height and width into 16 latent channels; it inflates an image autoencoder with a 1D temporal convolution after each spatial convolution and 1D temporal attention after each spatial attention (§3.1.1). A 30-billion-parameter transformer trained with flow matching generates up to 16 seconds at 16 frames per second; at 768 pixels, 256 frames make about 73,000 tokens after the TAE and $2 \times 2$ patchification (§3.1). Its authors evaluated by human raters on a new 1,003-prompt benchmark, stating that FVD and Inception Score did not correlate with human judgments of video quality (§3.5).

As of October 2026, several companies offer video generation services; the descriptions here are limited to what the cited reports and pages state.

Architectures

Three ideas recur across these families.

Factorized space-time layers. Full attention over every patch of every frame costs quadratically in $T \cdot h \cdot w$ tokens. Video Diffusion Models, Imagen Video and Make-A-Video all factorize it into spatial attention within each frame followed by temporal attention across frames at each position, the scheme video transformers such as TimeSformer use for classification; Make-A-Video’s authors write that adding the temporal dimension to full attention would be infeasible in memory (Singer et al., 2023, §3.2.2). Phenaki’s C-ViViT likewise applies a spatial and then a causal temporal transformer (Villegas et al., 2023, §2.1). Sora’s report describes its transformer only as operating on spacetime patches (Brooks et al., 2024).

Temporal layers inflated from an image model. Instead of training from scratch, most diffusion video models start from a pretrained image model and insert new temporal layers, initialized to the identity (Make-A-Video) or blended in by a learned factor (Video LDM), echoing how 3D convolutional networks such as I3D were inflated from image classifiers. The image weights bring appearance and text alignment learned from billions of images; the temporal layers learn motion from less video data. The variants differ in whether the image layers are frozen during video training (Video LDM) or fine-tuned with the rest of the network (Stable Video Diffusion).

Compressed spacetime latents. Latent diffusion compresses each frame with an image autoencoder; later systems compress time as well. VideoPoet’s MAGVIT-v2 tokenizer reduces 17 frames to 5 latent frames, keeping the first frame separate so that images and videos share one representation (Kondratyuk et al., 2024, §3.1); Movie Gen’s TAE encodes $T'$ frames into $\lceil T'/8 \rceil$ latent frames and discards the extra decoded frames (Polyak et al., 2024, Fig. 4). Movie Gen’s authors write that compressing time lets them generate at native frame rates without the separate frame-interpolation models of earlier cascades (Polyak et al., 2024, §3.1.1).

Conditioning

Text. Text enters through cross-attention as in text-to-image models (see Input and Output; the text-to-image generation article discusses the choice of text encoder). Two video-specific practices recur. Captions are often generated: Stable Video Diffusion annotates each clip with an image captioner on the middle frame, a video captioner and a language-model summary of the two (Blattmann et al., 2023b, §3.1), and Sora’s report describes training on highly descriptive captions from a trained captioner and expanding short user prompts with GPT (Brooks et al., 2024). Classifier-free guidance is used throughout; Stable Video Diffusion’s authors found that with a constant guidance scale, too little guidance left the video inconsistent with the conditioning frame and too much oversaturated it, and increased the scale linearly along the frame axis instead (Blattmann et al., 2023b, §4.3).

Motion. Some conditions control the motion itself. Stable Video Diffusion is conditioned on the frame rate, and its image-to-video model was given camera-motion LoRA modules in its temporal attention blocks for horizontal movement, zooming and a static camera (Blattmann et al., 2023b, §2, §4.3.1). Generative Image Dynamics predicts the motion first: a diffusion model generates a spectral volume, the Fourier coefficients of every pixel’s trajectory, from one image, and an image-based renderer moves the pixels accordingly; the same representation lets a user drag a point and see a simulated oscillating response (Li et al., 2024, abstract and §1).

Long Video and Consistency

Models are trained on short windows, so long videos are assembled from several samples, and errors accumulate at each step.

• Autoregressive extension. Generate a window, then generate the next one conditioned on the last few frames. Phenaki re-encodes the last frames and continues, possibly under a new prompt (Villegas et al., 2023, §2.2); Video LDM trains its denoiser with 0, 1 or 2 context frames and uses classifier-free guidance on the context to stabilize repeated extension (Blattmann et al., 2023a, §3.2).
• Conditioning an unconditional model. Video Diffusion Models showed that simply replacing the known frames by noised copies at each step (the replacement method) gives continuations that look good in isolation but are often not coherent with the known frames, and proposed reconstruction guidance, which adds the gradient of the error in reconstructing the known frames to the denoising direction (Ho et al., 2022a, §3.1, Eq. 7, Fig. 4).
• Hierarchical and sparse schedules. The Flexible Diffusion Model (Harvey et al., 2022) trains one model to generate any subset of frames given any other subset, so that keyframes far apart can be generated first and the gaps filled later; its Hierarchy-2 schedule, for example, first samples frames spread over the whole video and then fills in between them, and its authors sampled coherent videos of over 25 minutes, on CARLA driving-simulator and MineRL data (abstract, Fig. 1, §4).
• Cascades in time. Imagen Video and Make-A-Video generate few frames at a low frame rate and interpolate (Ho et al., 2022b; Singer et al., 2023).

Stable Video Diffusion’s authors note that generating many keyframes at once remains expensive and suggest coarse cascades or dedicated video tokenizers for long videos (Blattmann et al., 2023b, Appendix A).

Video Prediction and World Models

Video prediction predicts future frames from past ones, and is older than generation from noise. Mathieu, Couprie and LeCun (2016) trained a convolutional network to predict the next frames and found that the mean-squared-error loss gives blurry predictions, since it averages over possible futures; they added a multi-scale architecture, an adversarial loss and an image gradient difference loss to sharpen them. That averaging is the general argument for treating prediction as sampling from a distribution rather than regressing one answer. Benchmarks such as BAIR robot pushing (condition on 1 frame, generate 15) and Kinetics-600 (condition on 5, generate 11) evaluate it with FVD (Ho et al., 2022a, §4.2).

A world model predicts how an environment evolves in response to actions, so that an agent can plan or learn inside it. Ha and Schmidhuber (2018) compressed game frames with a VAE, predicted the next latent with a recurrent mixture-density network, and trained a controller entirely inside the model’s own generated rollouts. Genie (Bruce et al., 2024) learns an 11-billion-parameter generative interactive environment from unlabeled Internet videos, with a video tokenizer, an autoregressive dynamics model and a latent action model, so that users can act frame by frame in generated environments without any action labels (abstract). Sora’s report frames large video generators as a possible path toward general simulators of the physical world (Brooks et al., 2024); the failures it lists show the gap between plausible video and a correct simulation.

Datasets and Benchmarks

Benchmarks from video understanding. UCF101 (13,320 clips of 101 actions), which Video Diffusion Models’ and Make-A-Video’s authors call a popular benchmark, served for unconditional and class-conditional generation; Video Diffusion Models trained on all 13,320 videos, 16 frames at $64 \times 64$ (Ho et al., 2022a, §4.1), and Make-A-Video reported zero-shot results by writing one prompt per class (§4.1). Kinetics-600 serves for video prediction (Ho et al., 2022a, §4.2), and Phenaki compared text-to-video models on Kinetics-400 (Table 1). These datasets were built for recognition, and their videos are third-party YouTube uploads whose licensing is unclear; their articles give the details.

Video–text training data. WebVid (Bain et al., 2021) was collected for text–video retrieval: WebVid-2M has 2.5 million video–text pairs scraped from the web, and WebVid-10M, which Make-A-Video, Video LDM and many others trained on, was described as 10 million pairs from stock footage sites (WebVid README). Its clips carry stock-site watermarks, which Stable Video Diffusion’s authors list as a drawback (§2) and which the README warns can bias features computed on them. As of October 2026 the README states that the URLs and captions are no longer distributed (since 23 February 2024, per its FAQ), following a cease-and-desist request from Shutterstock, while saying that results on WebVid may still be reported for benchmarking and non-commercial purposes; its authors also warn that web data may carry social biases and graphic content. The largest systems train on internal collections: Imagen Video on 14 million internal video–text pairs (§3), Stable Video Diffusion’s LVD on 577 million clips from an unnamed source (§3.1, Table 1), Movie Gen on $O(100)$ million video–text pairs whose filtering and captioning its report describes but whose sources it does not name (§3.2.1), and Sora on data its report does not describe.

Prompt suites. Text-to-video models are compared on prompt lists with human raters: Make-A-Video’s 300 prompts collected from crowd workers (§4.1), Movie Gen Video Bench’s 1,003 prompts (Polyak et al., 2024, §3.5.2), and VBench’s prompts per evaluation dimension (below).

Evaluation Metrics

Fréchet Video Distance (FVD). Unterthiner et al. (2018) proposed FVD as the video counterpart of the Fréchet Inception Distance: fit Gaussians to features of real and generated clips and compute the Fréchet distance between them, with the features taken from the logits of an I3D network trained on Kinetics-400, among the best-ranked configurations in their study of injected spatial and temporal noise (§2, §4.1). In human comparisons of video prediction models drawn from over 3,000 trained on BAIR robot pushing, FVD agreed with raters more often than SSIM, PSNR or frame-averaged FID (§4.3, Table 1). A shorter version appeared at an ICLR 2019 workshop (Unterthiner et al., 2019). Its known problems:

• Content bias. Ge et al. (2024) found that FVD barely rises under severe temporal corruption, that it can be lowered substantially by choosing generated videos without motion, and attributed this to features from a supervised classifier trained on a dataset they describe as content-biased, Kinetics; FVD computed on features of self-supervised VideoMAE-v2 models was less biased toward per-frame quality (abstract, §3–§4).
• Short clips. Make-A-Video’s authors note that the FVD evaluation model expects 16-frame clips, about half a second, too short for practical generation (§4.2).
• Inconsistent protocols. Sample counts, the real reference set, resolution, preprocessing and the feature network differ between papers, as for FID. Video Diffusion Models’ authors note inconsistent preprocessing across papers and that sampling the reference set with or without replacement moved their Kinetics-600 FVD from 16.2 to 16.9 (§4.1–§4.2). For UCF101, papers computed Fréchet distances with the C3D network instead, and the values in Video Diffusion Models’ comparison range from about 300 to about 27,000 (Ho et al., 2022a, Table 1). Video Diffusion Models’ own Table 1 changed between arXiv v1 (FID 330, Inception Score 57.8 ± 1.3) and v2 (295 ± 3 and 57 ± 0.62), and Make-A-Video’s comparison lists the v1 Inception Score; numbers copied between papers need their version.
• Disagreement with people for modern models. Movie Gen’s authors found that FVD and Inception Score did not correlate with human judgments of video quality and relied on human evaluation instead (Polyak et al., 2024, §3.5.3).

Inception Score, computed with a video classifier (C3D on UCF101), was reported alongside FVD in the GAN and early diffusion papers.

CLIP-based scores. Prompt alignment is measured per frame and averaged: CLIPSIM, the average CLIP similarity between the frames and the text, on MSR-VTT captions (Singer et al., 2023, §4.1), or CLIP score and CLIP R-Precision over all frames (Ho et al., 2022b, §3.4). Like CLIP score for images, they inherit CLIP’s weaknesses described in the text-to-image article, and, being averages over frames, they ignore order and motion by construction: a clip whose frames are shuffled scores the same.

VBench (Huang et al., 2024) decomposes quality into 16 dimensions, each with its own prompts and an automatic measure, and validated each against human preference annotations. Temporal dimensions include subject consistency (DINO feature similarity across frames), background consistency (CLIP feature similarity), temporal flickering (mean absolute difference between frames of static scenes), motion smoothness (using a frame-interpolation model’s motion prior) and dynamic degree, measured with RAFT optical flow, which exists because a completely static video can score well on the other temporal dimensions (§3.1). Frame-wise quality uses aesthetic and image-quality predictors, and video–condition consistency uses detectors, captioners and an action recognizer for objects, actions, colors and spatial relations (§3.1).

Human evaluation remains the reference: raters compare two videos for the same prompt for visual quality, motion and text alignment. Movie Gen’s authors describe what made their human studies reliable: detailed guidelines and examples, raters giving reasons for their choices, a large prompt set, majority votes over three or six annotations, and audits (Polyak et al., 2024, §3.5.3).

Example: Per-Frame Metrics Miss Flicker

The data are $48 \times 48$ grayscale clips of 16 frames: a textured $12 \times 12$ square moves 2 pixels down and 1 right per frame over a smooth textured background, with a little independent sensor noise in each frame. Because the clips are rendered, the motion (the optical flow) is known exactly. Two hundred such clips serve as the “real” videos, and three generators are compared against them:

• consistent draws more clips from the same process;
• flicker keeps the object’s path but draws every frame independently from the same per-frame distribution, with a new background and object texture each time, as an image model applied frame by frame would;
• frozen is the consistent process with the object standing still.

Four measures are computed. Frame FD is a Fréchet distance between Gaussians fitted to features of individual frames (the means of a $4 \times 4$ grid of blocks), an FID-like score that sees frames one at a time. Clip FD is the same distance on clip-level features, the block means of the average frame and of the average absolute change between frames, a crude stand-in for an FVD feature that sees time. Warp error is the mean absolute difference between each frame and the previous frame moved by the known motion, with uncovered pixels excluded. Motion is the object’s mean displacement per frame, in pixels summed over the two axes.

import numpy as np
from scipy import linalg, ndimage

H = W = 48; T = 16; S = 12          # frame size, frames per clip, object size
V = np.array([2, 1])                # object velocity (rows, cols) in pixels per frame

def texture(rng, shape, smooth):
    """Smooth random texture in [0, 1]."""
    t = ndimage.gaussian_filter(rng.normal(size=shape), smooth)
    return (t - t.min()) / (t.max() - t.min())

def clip(rng, kind):
    """One clip of a textured square moving over a textured background (static camera).
    consistent: one background, one object texture, small per-frame sensor noise.
    flicker:    each frame drawn independently from the same per-frame distribution
                (new background and object texture every frame), object path unchanged.
    frozen:     a consistent clip whose object never moves."""
    bg, obj = texture(rng, (H, W), 3), 0.5 + 0.5 * texture(rng, (S, S), 1.5)
    p0 = rng.integers(2, 6, 2)
    frames, pos = [], []
    for t in range(T):
        if kind == "flicker":
            bg, obj = texture(rng, (H, W), 3), 0.5 + 0.5 * texture(rng, (S, S), 1.5)
        p = p0 + (V * t if kind != "frozen" else 0)
        f = bg.copy(); f[p[0]:p[0] + S, p[1]:p[1] + S] = obj
        frames.append(f + rng.normal(0, 0.02, (H, W))); pos.append(p)
    return np.array(frames), np.array(pos)

def warp_error(frames, pos):
    """Mean |I_{t+1} - warp(I_t)| using the known motion: the object patch moves with it,
    the background stays; pixels uncovered by the move are excluded."""
    errs = []
    for t in range(T - 1):
        (r0, c0), (r1, c1) = pos[t], pos[t + 1]
        pred = frames[t].copy(); valid = np.ones((H, W), bool)
        valid[r0:r0 + S, c0:c0 + S] = False                       # uncovered background
        pred[r1:r1 + S, c1:c1 + S] = frames[t][r0:r0 + S, c0:c0 + S]
        valid[r1:r1 + S, c1:c1 + S] = True
        errs.append(np.abs(frames[t + 1] - pred)[valid].mean())
    return np.mean(errs)

def blocks(x):                       # 4x4 grid of block means: a tiny "feature extractor"
    return x.reshape(4, H // 4, 4, W // 4).mean((1, 3)).ravel()

def frechet(a, b):
    """Frechet distance between Gaussians fitted to two feature sets (as in FID/FVD)."""
    ma, mb = a.mean(0), b.mean(0)
    ca, cb = np.cov(a, rowvar=False), np.cov(b, rowvar=False)
    covmean = linalg.sqrtm(ca @ cb).real
    return float(((ma - mb) ** 2).sum() + np.trace(ca + cb - 2 * covmean))

def features(clips):
    frame_f = np.array([blocks(f) for c, _ in clips for f in c])                 # per frame
    clip_f = np.array([np.concatenate([blocks(c.mean(0)),                         # per clip:
                                       blocks(np.abs(np.diff(c, axis=0)).mean(0))])  # mean frame,
                       for c, _ in clips])                                        # mean |change|
    return frame_f, clip_f

N = 200                                                     # clips per set
KINDS = ["consistent", "flicker", "frozen"]
rows = {k: [] for k in KINDS}
for seed in range(5):
    rng = np.random.default_rng(seed)
    ref = [clip(rng, "consistent") for _ in range(N)]       # "real" videos
    rf, rc = features(ref)
    for k in KINDS:
        gen = [clip(rng, k) for _ in range(N)]
        gf, gc = features(gen)
        motion = np.mean([np.abs(np.diff(p, axis=0)).sum(1).mean() for _, p in gen])
        rows[k].append((frechet(rf, gf), frechet(rc, gc),
                        np.mean([warp_error(c, p) for c, p in gen]), motion))

print("set          frame FD   clip FD   warp error   motion (px/frame)   [seed 0]")
for k in KINDS:
    print(f"{k:11s}{rows[k][0][0]:10.4f}{rows[k][0][1]:10.4f}{rows[k][0][2]:12.4f}{rows[k][0][3]:12.1f}")
print("\nranges over seeds 0-4")
for k in KINDS:
    r = np.array(rows[k])
    print(f"{k:11s}" + "".join(f"  {r[:, j].min():.4f}-{r[:, j].max():.4f}" for j in range(3)))

Output (NumPy 2.3.5, SciPy 1.16.3, about 3 seconds):

set          frame FD   clip FD   warp error   motion (px/frame)   [seed 0]
consistent     0.0063    0.0070      0.0226         3.0
flicker        0.0033    0.6026      0.2004         3.0
frozen         0.0354    0.0411      0.0226         0.0

ranges over seeds 0-4
consistent   0.0062-0.0083  0.0068-0.0089  0.0226-0.0226
flicker      0.0033-0.0042  0.5947-0.6026  0.1995-0.2008
frozen       0.0306-0.0377  0.0348-0.0425  0.0226-0.0226

The frame-level score cannot see the flicker at all: every flickering frame is a valid draw from the real per-frame distribution, so its frame FD is as small as the consistent generator’s on every seed, in fact smaller. Our reading is that 16 frames of one real clip share a background and are correlated, while the flickering frames are all independent, so they estimate the frame distribution with less noise. The two scores that look across frames separate it clearly: on every seed its clip FD is near 0.6 while the consistent generator’s stays below 0.01, and its warp error is about nine times larger. The consistent generator’s warp error of 0.0226 is the floor set by sensor noise alone, the expected absolute difference of two independent noise values with standard deviation 0.02 ($0.02\sqrt{2}\sqrt{2/\pi} \approx 0.0226$, our calculation).

The frozen generator shows the opposite failure. Its warp error, computed with its own (zero) motion, equals the noise floor, as good as possible, because a clip with no motion is perfectly consistent with itself; only the motion column reveals it. Frame FD notices it only indirectly, because the object stays near its starting corner and the frames’ layouts differ from real ones, and the crude clip FD rates it far closer to real than the flicker. That a feature-distance score can rate motionless video close to real is what Ge et al. (2024) found for FVD, and it is the reason VBench pairs its consistency measures with a dynamic-degree measure. The block-mean features here are much weaker than I3D’s or VideoMAE’s, so the example illustrates what each kind of measure can and cannot see, not the size of the effect in real evaluations.

Applications

The developers of these systems name creative content production as the main use: Video LDM’s authors focus on creative content creation and on simulation of driving data (Blattmann et al., 2023a, abstract), and Imagen Video’s authors expect generative video to augment human creativity (§4). Generative Image Dynamics targets looping videos and interactive animation of still photographs (Li et al., 2024). Stable Video Diffusion’s authors fine-tuned it into a multi-view generator that produces orbital views of an object from one image, a form of novel view synthesis (Blattmann et al., 2023b, §4.5). World models generate environments for training agents (Ha & Schmidhuber, 2018; Bruce et al., 2024), and VGAN’s authors already expected generative video models to serve simulation and forecasting (Vondrick et al., 2016, §1).

Risks

Deepfakes and misinformation. Realistic generated video can show events that never happened and people saying or doing things they did not. Imagen Video’s authors list fake, hateful, explicit or harmful content among the possible misuses and withheld the model (Ho et al., 2022b, §4); Stable Video Diffusion’s authors call reducing the potential for misinformation and harm crucial before real-world deployment (Blattmann et al., 2023b, Appendix A). Detecting manipulated faces is a research area of its own; the ROC-AUC article describes how one deepfake detection benchmark scores detectors.

Consent and training data. The training sets described above were collected from YouTube, stock-footage sites and other web sources, and none of the papers cited here reports asking the people shown for consent. The recognition datasets used as benchmarks consist of third-party YouTube uploads with unclear licences (see Kinetics and UCF101), and WebVid’s distribution stopped after a rights holder’s request (WebVid README, as of October 2026). The text-to-image generation article summarizes the copyright questions that the same practices raise for images, including official reports and a court judgment.

Bias. Imagen Video’s authors write that, while filtering removed much explicit and violent content in internal testing, social biases and stereotypes remain hard to detect and filter (§4).

Provenance and watermarking. Two complementary mechanisms mark generated media, each with a limited scope.

• Content Credentials (C2PA). The C2PA specification (version 2.5, October 2026) defines a manifest of signed assertions about an asset’s origin and edits, attached to the file and cryptographically bound to its content by a hash (a hard binding), so that copying a manifest to another asset fails validation (§2.3, §11.1). A manifest can be removed from its asset; the specification therefore defines soft bindings, fingerprints or invisible watermarks, through which such an asset can be matched to a copy of its manifest kept in a provenance store (§2.3, §12.2.3). The specification states that it makes no value judgment about whether provenance data is good or bad, only whether it can be validated (§1.2); a credential records who signed what, not whether the content is true, and it helps only where creators and tools attach credentials.
• Invisible watermarks. Google DeepMind’s SynthID page (as of October 2026) states that SynthID embeds an imperceptible watermark in images and video generated by Google’s tools, designed to withstand cropping, filters, frame-rate changes and lossy compression, and that its detector checks content from Google and named partners. A watermark identifies output only from generators that embed it, and its robustness claims are the developer’s.

Open Problems

• Physics and long-range coherence. Incorrect physical interactions, incoherence in long samples and objects that appear spontaneously are listed failures in Sora’s own report (Brooks et al., 2024).
• Long videos at reasonable cost. Training windows are seconds long; extension and hierarchical sampling accumulate errors or cost (Blattmann et al., 2023b, Appendix A; Harvey et al., 2022).
• Motion versus stillness. Models trade temporal consistency for too little motion, and metrics can reward stillness unless motion is measured separately (Ge et al., 2024; Huang et al., 2024).
• Evaluation. FVD is content-biased and protocol-dependent, CLIP-based scores ignore time, benchmark suites depend on the detectors and captioners they use, and the largest systems are evaluated by human studies on their developers’ own prompt sets.
• Data. Openly licensed, captioned, high-quality video at scale remains scarce, and the largest training sets are not public.
• Provenance at scale. Watermarks and credentials cover only cooperating generators and platforms.