Video Prediction

Video prediction is the task of generating the frames that follow a given stretch of video. It is also called future frame prediction or video forecasting. The input is a few context frames, sometimes with the actions an agent will take. The output is the next frames, as pixels. It is the setting of video generation conditioned on past frames; that article surveys the whole field. Prediction was studied as a way to learn video representations without labels before generation from noise became practical (Srivastava, Mansimov & Salakhutdinov, 2015). It is also used to plan robot actions (Ebert et al., 2017). An action-conditioned predictor is, in our reading, the simplest form of the learned world models that embodied agents use to plan.

The defining difficulty is that the future is uncertain. A ball that hits the ground may bounce off in many directions, and a predictor that must give one answer will hedge. Denton and Fergus (2018, §1) put it as the ball “blurring to accommodate multiple possible futures”. Much of the history of the task is a series of responses to that blur:

• losses other than squared error;
• predictors that move existing pixels instead of painting new ones;
• latent-variable models whose samples are sharp and differ from one another;
• autoregressive, token and diffusion generators conditioned on the past.

This article covers the task and its methods in that order, then action-conditioned prediction, the benchmarks, and how stochastic predictions are evaluated.

Problem Definition

Let $x_{1:T}$ be a clip of $T$ frames. A predictor sees the first $C$ frames $x_{1:C}$, and in the action-conditioned setting also actions $a_{1:T}$. It produces the remaining frames $\hat x_{C+1:T}$. Benchmarks fix $C$ and the horizon $T - C$. Examples include 10 context frames and 10 predicted on Moving MNIST (Kalchbrenner et al., 2017, §5) and 2 and 28 on BAIR robot pushing in Denton and Fergus’s evaluation (2018, §4.4).

There are two formulations:

• Deterministic prediction learns a function $\hat x_{C+1:T} = f(x_{1:C})$, usually trained with a pixel loss such as mean squared error.
• Stochastic prediction learns the conditional distribution $p(x_{C+1:T} \mid x_{1:C})$ and returns samples from it, each a different possible future.

The difference matters because of what squared error rewards. The function that minimizes the expected squared error $\mathbb{E}\,\lVert x_{C+1:T} - f(x_{1:C}) \rVert^2$ is the conditional mean $f^\star(x_{1:C}) = \mathbb{E}[x_{C+1:T} \mid x_{1:C}]$, a standard result of least squares. Suppose an object is equally likely to move left or right. Then the optimal prediction shows it at half intensity in both places, an image that is not a possible future at all. The blur is therefore the correct answer to the question the loss asks (our reasoning). Srivastava et al. (2015, §2.3) gave this example in their own words: if a ball is equally likely to move left or right, a model should predict one or the other, “not a ball in the middle plus Gaussian noise”. They suggested that variational autoencoders were a promising way to add the needed stochasticity. A larger network did not remove the blur in their experiments. On natural $32 \times 32$ patches from UCF101, their squared-error model’s future predictions “quickly blur out”, and still did so when they doubled the LSTM to 4,096 units (§3.3).

Uncertainty is not spread evenly in time. In Denton and Fergus’s bouncing-ball example, a deterministic model suffices for most of the trajectory, and uncertainty matters only at the moment of contact (2018, §1). A good stochastic predictor is therefore nearly deterministic most of the time and spreads its samples when something unpredictable happens.

Input and Output

The frames of the standard benchmarks are small: $64 \times 64$ for Moving MNIST, KTH and BAIR in the papers cited here (Denton & Fergus, 2018, §4; Franceschi et al., 2020, Appendix B). MAGVIT generates $128 \times 128$ clips, except on BAIR, where it works at $64 \times 64$ (Yu et al., 2023, §4.1). Context and horizon depend on the benchmark and differ between papers on the same data (see Datasets and Benchmarks). Models are often trained on shorter horizons than they are tested on. Denton and Fergus trained on BAIR to predict 10 frames and tested on 28, to measure how far the model generalizes beyond its training length (§4.4, Fig. 7).

Extra inputs define the variants:

• Actions. On the Robotic Pushing data of Finn et al. (2016), each frame comes with the robot’s state and the desired action to reach the next frame. The 18 future frames are generated from 2 context frames together with the 18 future actions (Kalchbrenner et al., 2017, §6).
• Class labels. MAGVIT evaluates class-conditional frame prediction as one of its tasks (Yu et al., 2023, §3).
• Masks over frames. Video LDM marks with a binary mask which frames are given and which are to be generated, and its separate interpolation models use the same masking mechanism (Blattmann et al., 2023, §3.2).

Variants

• Deterministic and stochastic prediction, as defined above.
• Action-conditioned prediction, where the future depends on what an agent does. It is the basis of planning with a learned model; see Action-Conditioned Prediction below.
• Long-term prediction, over horizons where small errors compound. Villegas et al. (2017, abstract) write that small errors in pixel space grow exponentially when a model observes its own predictions, and they avoid feeding back pixels altogether (see Approaches).
• Extrapolation and interpolation. Predicting the next frames and synthesizing frames between known ones are closely related. Deep Voxel Flow handles both with one architecture, trained on any video by dropping frames and learning to predict them (Liu et al., 2017, abstract).
• Prediction in other representations, such as human pose (Villegas et al., 2017) or optical flow (Liang et al., 2017). These representations serve as intermediate steps toward pixels in the methods below.

Forecasting the trajectories of road users from tracked positions is a separate task with its own benchmarks. Motion forecasting describes it, including why a single regressed trajectory averages the possible futures. The same averaging, at the level of pixels, is the subject of this article.

Challenges

Uncertainty. Several futures are consistent with the context, and the loss and the model must allow for that.

Appearance against motion. Most of the next frame is already in the current one. Vondrick and Torralba (2017, §1) argue that a network generating pixels from scratch must store low-level details of the past, which complicates learning the higher-level semantics needed for prediction. Their proposed remedy is to transform past pixels instead.

Occlusion and object permanence. When a robot arm passes in front of an object, a predictor that only transforms the previous frame loses the object. Ebert et al. (2017, §4) found this to be one of the major causes of planning failures with an earlier transformation model.

Compounding error. Predictors that feed their own outputs back as inputs accumulate errors: Villegas et al. (2017, abstract) write that small errors in pixel space amplify exponentially as predictions go further into the future. In our reading, part of the cause is that such a predictor sees its own imperfect frames at test time, whereas training often feeds it real frames, as in Srivastava et al.’s conditional decoder (2015, §2.3).

Evaluation. A good stochastic model’s sample need not resemble the one future that happened. Per-frame comparisons with the ground truth therefore penalize correct uncertainty and reward blur (see Evaluation Metrics).

Approaches

Deterministic pixel predictors

Srivastava et al. (2015) encoded a sequence of frames with an LSTM and decoded it with a second LSTM that either reconstructed the input or predicted the future (§2). Their future predictor came in two forms. The unconditioned decoder does not see its own previous output. The conditional decoder is fed the generated frame at test time and the ground-truth frame in training (§2.3). They argued that conditioning on the last output helps a model represent several modes, but only if some randomness enters the generation. Their conditional decoder stayed deterministic.

Deterministic predictors trained with squared error remained a main line of work, judged on the deterministic Moving MNIST benchmark where the future is fully determined by the past (see Datasets and Benchmarks). SimVP’s authors sort these models by architecture (§1, Fig. 1):

• stacked recurrent networks, such as ConvLSTM and PredRNN;
• convolutional encoders around a recurrent core;
• convolutional encoders around a vision transformer;
• networks with no recurrence at all.

Their SimVP is of the last kind: a convolutional encoder, a convolutional translator and a convolutional decoder, trained end to end with squared error and no extra techniques (abstract, §3). In their comparison on Moving MNIST (10 frames in, 10 out), MSE fell from 103.3 for ConvLSTM and 56.8 for PredRNN to 24.4 for PhyDNet, 22.3 for CrevNet and 23.8 for SimVP (Table 3, values taken from the original papers). The text claims state-of-the-art MSE and SSIM for SimVP on this benchmark, but its own table shows CrevNet slightly ahead on both: MSE 22.3 against 23.8, and SSIM 0.949 against 0.948.

Motion and transformation-based predictors

Instead of painting every pixel, these models predict how to move pixels that are already there. This reuses the appearance of the context frames and needs no motion labels.

• Neural advection (DNA and CDNA). As Ebert et al. (2017, §3.2) describe the model of Finn et al. (2016), it predicts a set of transformations of the previous frame, applied as convolutions, together with masks that composite the transformed images into the next frame. Intuitively, each transformation corresponds to one moving object and each mask to its extent. When the kernels are normalized, they can be read as transition probabilities for where each pixel goes.
• Skip connections in time. Ebert et al. (2017, §4) extended this to copy pixels from earlier frames, so that an occluded object can reappear. In their simplified model, the next frame is a masked mixture of the transformed previous frame and the first, real frame of the sequence. This rests on their assumption that occluded objects usually stay still during the predicted motion.
• Voxel flow. Deep Voxel Flow predicts, for each output pixel, a flow vector into the input frames and a temporal weight, and samples the inputs by trilinear interpolation (Liu et al., 2017, §3.1). The authors present this as combining the advantages of optical-flow-based methods, which fail where flow is hard to estimate, and networks that hallucinate pixels, which often blur; the output is a warping of the inputs, and training needs no human supervision (abstract).
• Adversarial transformers. Vondrick and Torralba (2017, §4.1) generate a transformation that maps the past into the future, applied by a differentiable transformer, and train it with an adversarial loss (§4.2). In two-alternative choices on Mechanical Turk, workers preferred videos from the adversarially trained transformation model over those from a regression-trained one 55.6% of the time (§5.3, Table 2).

In our reading, transformations make the copied content sharp, but they do not resolve which way the motion goes. That uncertainty remains for the loss or the latent variables to handle.

Predicting structure first

Villegas et al. (2017) predict human pose rather than pixels. A pose estimator extracts joint positions from the context frames, and an LSTM predicts the future poses. An image generator then renders each future frame from the last observed frame and the predicted pose, by visual-structure analogy (§3). Because the generator never looks at its own predicted frames, pixel errors cannot propagate, which is their remedy for long-term prediction (abstract). They evaluated on Human3.6M and Penn Action. Denton and Fergus (2018, §2) note that this approach needs annotated poses during training.

Other models decompose the video without labels. Denton and Fergus (2018, §2) summarize earlier models that split content from pose or motion, and contrast them with their own model, which splits a video into deterministic and stochastic parts instead. MOSO separates motion, scene and object into distinct groups of discrete tokens. A transformer predicts the object and scene tokens of the next clip and adds motion at the token level (Sun et al., 2023, abstract). The authors argue that separating objects from scenes avoids the blurred, distorted appearance they attribute to content–motion splits (Fig. 1).

Adversarial losses

An adversarial loss replaces “close to the true future” with “indistinguishable from real futures”, which a blurred average is not. Mathieu, Couprie and LeCun (2016), as Denton and Fergus (2018, §2) summarize them, showed that a loss based on generative adversarial networks produces sharper generations than squared error. The Video Generation article describes their multi-scale model. Vondrick and Torralba’s transformer and the Dual Motion GAN followed. The Dual Motion GAN predicts future frames and future optical flow in two coupled branches, makes both look real with adversarial training, and handles motion uncertainty with a VAE-based probabilistic motion encoder (Liang et al., 2017, abstract and §1).

Denton and Fergus (2018, §1–§2) give the limitation of this route: training difficulties and mode collapse often mean that the full distribution of futures is not captured. An adversarial predictor without a latent input returns one sharp, plausible future rather than a distribution (our reasoning). SAVP (Lee et al., 2018), as Franceschi et al. (2020, §2) describe it, combined a ConvLSTM predictor and a learned prior with an adversarial loss that sharpens predictions at the cost of some diversity. In their comparison they credit its adversarial loss with modelling small objects better (§4).

Latent-variable predictors

Stochastic predictors add a latent variable $z$ that carries the information the past does not determine. They are trained, as a conditional variational autoencoder is, by maximizing an evidence lower bound. Different draws of $z$ give different futures, each as sharp as the decoder can make it.

SV2P. Babaeizadeh et al. (2018) proposed stochastic variational video prediction. As Denton and Fergus describe it (2018, §3.1), its inference network encodes the whole sequence with a feed-forward network into $q(z \mid x_{1:T})$. Its time-invariant version samples one $z$ per video, and its time-variant version samples a new $z_t$ at every step from the same distribution. In the same account, its authors reported that naive training led the model to ignore the latents, and trained in three phases: first the deterministic part, then with latents but no KL term, then with the full loss.

SVG. Denton and Fergus (2018) built stochastic video generation around a recurrent frame predictor $p_\theta(x_t \mid x_{1:t-1}, z_{1:t})$, a fixed-variance Gaussian whose mean $\mu_\theta$ is used as the frame. A recurrent inference network $q_\phi(z_t \mid x_{1:t})$ gives a separate posterior at every step (§3). With a learned prior $p_\psi(z_t \mid x_{1:t-1})$ (the SVG-LP model), training maximizes

$$\mathcal{L}_{\theta,\phi,\psi}(x_{1:T}) = \sum_{t=1}^{T} \Big( \mathbb{E}_{q_\phi(z_{1:t} \mid x_{1:t})} \log p_\theta(x_t \mid x_{1:t-1}, z_{1:t}) - \beta\, D_{\mathrm{KL}}\big(q_\phi(z_t \mid x_{1:t}) \,\Vert\, p_\psi(z_t \mid x_{1:t-1})\big) \Big).$$

With a Gaussian likelihood, the first term is a squared error between $\hat x_t$ and $x_t$, estimated with one reparameterized sample. The KL term keeps $z_t$ from simply copying the target frame, forcing it to carry only information not present in earlier frames (§3). SVG-FP uses a fixed $\mathcal{N}(0, I)$ prior instead. Its drawback is that latents drawn independently at each step ignore the temporal dependencies between frames (§3).

At test time, $z_t$ is drawn from the prior, the frame is decoded, and generated frames are fed back into both the predictor and the learned prior (§3, Fig. 2c). The three networks are LSTMs fed by a shared convolutional frame encoder, and a convolutional decoder maps the predictor’s output back to pixels (§3). Skip connections run from the encoder at the last ground-truth frame to the decoder, so that static background can be copied (§4.1).

The weight $\beta$ is the central trade-off (§3):

• If $\beta$ is too small, the inference network may learn to copy the target frame. Training error is then low, but test performance is poor because the posterior and the prior no longer match.
• If $\beta$ is too large, the model may ignore $z_t$ and reduce to a deterministic predictor.

The authors found $\beta$ easy to tune, particularly for SVG-LP. Section 4.1 gives $\beta = 10^{-4}$ for KTH and BAIR and, in the same sentence, $\beta = 10^{-6}$ for KTH again, so the sentence is inconsistent. The official README trains SVG-LP on both Stochastic Moving MNIST and BAIR with $\beta = 10^{-4}$, and its KTH command trains SVG-FP at $64 \times 64$ with $\beta = 10^{-6}$; the README also says the paper’s KTH results used $128 \times 128$ frames, whereas §4.3 gives $64 \times 64$. The SVG-LP training script averages the squared error over pixels and sums the KL over latent dimensions, averaged over the batch (official code, train_svg_lp.py at commit 3f19f0b).

The learned prior is what lets SVG-LP be deterministic where the data are. On Stochastic Moving MNIST, where digits bounce off walls in random directions, five hundred test sequences with different digits but synchronized trajectories were fed through the prior. The mean of its predicted standard deviation rose at the moments when a digit hit a wall (Fig. 5). Sampling many futures reproduced the true distribution of post-bounce velocities, which broadens after each collision and is reshaped by later ones (Fig. 4). The deterministic baseline, with the same encoder, decoder and LSTM but no latents, was sharp for several steps and then degraded rapidly once a digit hit a wall (Fig. 3, §4.2). The authors hypothesize that SVG-LP’s advantage over SVG-FP is due to the mix of deterministic and stochastic motion, since the prior network can take over the burden of predicting when uncertainty arises (§4.2). On KTH the two variants performed comparably (§4.3). They also argue that their per-step posterior made training much easier than SV2P’s three-phase schedule (§3.1).

Later latent models. Franceschi et al. (2020) moved the dynamics entirely into latent space. Their stochastic latent residual model, SRVP, updates a latent state by a residual rule motivated by the discretization of differential equations, and decodes each frame from the latent state and a content variable computed from the context frames. The temporal model is fully latent, which the authors describe as untying frame synthesis from temporal dynamics (abstract, §3). Because the update is an Euler step, they could refine the time step at test time and generated frames at double and quadruple frame rates with the same model (§4).

Autoregressive and token predictors

Video Pixel Networks model the joint distribution of raw pixel values exactly, as a four-dimensional dependency chain over time, height, width and colour. Resolution-preserving convolutional encoders and a convolutional LSTM summarize past frames, and PixelCNN decoders generate each frame pixel by pixel (Kalchbrenner et al., 2017, §1–§3; see autoregressive image models). On Moving MNIST they reached a cross-entropy of 87.6 nats per frame. The authors compute a lower bound of 86.3, the loss obtained when the predictions equal the targets, which is not zero because the grayscale targets are read as probabilities. The previous best they list is 179.8 (Table 1, §5). Ebert et al. (2017, §2) note that such autoregressive models often produce sharp predictions but are extremely demanding computationally, and had not been applied to real-world robotic control.

Token models predict discrete codes of a learned tokenizer, as in VQ-VAE and its successors. MAGVIT quantizes a clip into $4 \times 16 \times 16$ tokens with a 3D VQ model and fills in masked tokens with a bidirectional transformer, treating frame prediction as one of several masked-generation tasks (Yu et al., 2023, §3, §4.1). In frame prediction its large model, MAGVIT-L, reported FVD 62 on BAIR (1 context frame) and 9.9 on Kinetics-600 (5 context frames). The best earlier values in its table were 84 for RaMViD and 16.2 for Video Diffusion Models (Table 2). MOSO, described above, is also a token model.

Diffusion predictors

Diffusion models handle uncertainty by construction: they sample from a conditional distribution rather than regress a mean. Video Diffusion Models (Ho et al., 2022), which the Video Generation article describes, appear in MAGVIT’s Kinetics-600 comparison above. MAGVIT’s Table 2 also lists two diffusion predictors with BAIR FVDs of 90 for MCVD (Voleti, Jolicoeur-Martineau & Pal, 2022) and 84 for RaMViD (Höppe et al., 2022); those papers were not read for this article.

Video LDM trains its latent video diffusion model as a prediction model by masking. The context frames are encoded, multiplied by a binary mask, and concatenated with the mask as input to the temporal layers. Training samples 0, 1 or 2 context frames, so the network learns an unconditional mode as well. Long videos are generated by reusing the latest predictions as new context. To keep this iteration stable, the authors apply classifier-free guidance with respect to the context frames, which they call context guidance (Blattmann et al., 2023, §3.2). On real driving scenes, the model generated several different plausible futures from one initial frame (§4.1.2).

Action-Conditioned Prediction

When an agent’s actions are among the inputs, a predictor answers “what will I see if I do this?”, which is what a planner needs. Denton and Fergus (2018, §2) cite Oh et al. (2015) and Chiappa et al. (2017) for action-conditioned prediction in video games, which they describe as producing accurate long-range predictions. Video Pixel Networks generated the future frames of the Robotic Pushing dataset conditioned on the robot’s desired actions. The authors note that the transitions are non-deterministic even given the actions, because the arm may not reach the desired state, due to occlusion by objects on its trajectory (Kalchbrenner et al., 2017, §6).

Ebert et al. (2017) used such a model for control, an approach called visual model-predictive control. A user marks pixels on an object in the first image and goal positions for them. The transformation model predicts how the probability distribution of each marked pixel moves under a candidate action sequence. The cross-entropy method samples action sequences, keeps those with the lowest predicted cost, refits a Gaussian, and repeats. The first action of the best plan is executed and the plan is recomputed at the next step (§3.1, Appendix). The predictor was trained on 44,000 trajectories of 30 frames (Appendix), recorded while the robot made random pushing motions (§8), so the data needed no human supervision (abstract).

In their benchmark of 20 pushing tasks, their expected-distance cost substantially outperformed the negative log-probability cost of the earlier work of Finn and Levine (2017) (§7, Table 1). With the new cost, the occlusion-aware model and the earlier DNA model performed comparably on this task, which involves no occlusions. DNA’s mean final distance on seen objects was slightly smaller (9.2 against 13.0 pixels in the $64 \times 64$ image), a difference the authors judge to be within the standard deviation. A second task, pushing one object around another that the gripper may occlude, tested occlusion handling (§7).

The connection to learned models of environments for embodied agents and to visuomotor policies is direct: an action-conditioned predictor in a learned latent space is a world model.

Datasets and Benchmarks

• Moving MNIST (Srivastava et al., 2015, §3.3). Clips of 20 frames of $64 \times 64$ in which two MNIST digits move with random constant velocities, bounce off the edges and may overlap. Clips are generated on the fly, so the training set is effectively unlimited. In this original form the motion is deterministic. Common protocols condition on 10 frames and predict 10, with a fixed test set of 10,000 sequences of test digits (Kalchbrenner et al., 2017, §5; Gao et al., 2022, Table 2). Results are reported as binary cross-entropy in nats per frame (Video Pixel Networks) or as MSE and SSIM (SimVP).
• Stochastic Moving MNIST (Denton & Fergus, 2018, §4.2). Digits move at constant velocity until they hit a wall, then bounce off with a random speed and direction. The data therefore mix deterministic segments with moments of uncertainty. Initial velocities are drawn from $[-4, 4]^2$ pixels per frame, and models condition on 5 frames and train to predict 10. Franceschi et al. (2020, Appendix B.2) later adapted it so that trajectories no longer depend on the frame rate, and retrained SVG on their corrected version.
• KTH actions (Schuldt, Laptev & Caputo, 2004), a recognition dataset also described in video understanding. Franceschi et al. (2020, Appendix B.3) describe it as $64 \times 64$ grayscale videos of 25 people performing six actions (walking, jogging, running, boxing, hand waving and hand clapping). Following Denton and Fergus, they train on the first 20 people and test on the other five, conditioning on 10 frames and predicting 30. Denton and Fergus note that the motion is fairly regular but the exact positions of limbs remain uncertain (§4.3).
• BAIR robot pushing (Ebert et al., 2017). Videos of a Sawyer arm pushing objects on a table. Denton and Fergus call the arm’s movements highly stochastic and discard the actions included with the data (§4.4). Protocols differ:
  • SVG and SRVP condition on 2 frames and predict 28 at $64 \times 64$, on 256 test sequences (Denton & Fergus, 2018, §4.4; Franceschi et al., 2020, Appendix B.5).
  • MAGVIT and MOSO condition on 1 frame and predict 15, reported by FVD (Yu et al., 2023, §4.1; Sun et al., 2023, §4.1).
• Robotic Pushing (Finn et al., 2016), an earlier robot dataset described here from Video Pixel Networks’ account (Kalchbrenner et al., 2017, §6). Sequences of 20 frames of $64 \times 64$ with the robot’s state and action for each frame: 50,000 training sequences and two test sets of 1,500, one with objects seen in training and one with novel objects.
• Human3.6M and Penn Action, videos of people, used for structure-based prediction (Villegas et al., 2017). Human3.6M also serves latent models (Franceschi et al., 2020, Appendix B.4) and SimVP (Gao et al., 2022, §4.3).
• Kinetics-600, natural YouTube video of human actions, evaluated with 5 context frames by FVD (Yu et al., 2023, §4.1). MOSO also reports results on RoboNet, KITTI and UCF101 (Sun et al., 2023, §4).

Numbers from different protocols on the same data are not comparable: the context length, the horizon, the number of samples per test clip and the FVD implementation all vary.

Evaluation Metrics

Frame-wise comparison with the ground truth. Mean squared error, PSNR and SSIM compare each predicted frame with the real one and average over time. Later papers add the learned LPIPS distance (Franceschi et al., 2020, §4). These measures are fair for deterministic data such as Moving MNIST. For uncertain futures they reward the blurred mean, since the mean minimizes squared error (see Problem Definition). Denton and Fergus write that PSNR is biased toward overly smooth results (§4.4). Franceschi et al. (2020, §4) add that PSNR may favour blurry predictions, that SSIM compares local patches but loses some information about dynamics, and that LPIPS has been shown to correlate better with human judgment on real images.

Best of K samples. For stochastic models the usual protocol draws K futures per test clip and reports the score of the one closest to the ground truth: K = 100 for SVG with SSIM and PSNR (Denton & Fergus, 2018, §4.2–§4.4), and for SRVP with PSNR, SSIM and LPIPS (Franceschi et al., 2020, §4). The protocol asks whether the true future is among the model’s samples. It does not check whether the other samples are plausible; in our reading, a model that produces many diverse but poor samples can score well. MOSO used 100 trials on KTH, KITTI and RoboNet but one trial on BAIR (Sun et al., 2023, §4), so even this protocol varies.

The disagreement between measures shows in Franceschi et al.’s BAIR results (Appendix D, Table 5, best of 100 samples, 2 context frames, 28 predicted, averaged over time steps):

Model	PSNR (higher is better)	SSIM (higher is better)	LPIPS (lower is better)
SV2P	20.39	0.8169	0.0912
SAVP	18.44	0.7887	0.0634
SVG	18.95	0.8058	0.0609
SRVP	19.59	0.8196	0.0574

SV2P has the best PSNR and the worst LPIPS. The authors attribute this to its very blurry samples (§4). By FVD (Table 1, intervals over five sets of samples), SAVP scored 152, SRVP 163, SVG 255 and SV2P 965.

Distribution-level measures. The Fréchet Video Distance (FVD) compares features of predicted and real clips, as FID does for images, and is MAGVIT’s primary measure (Yu et al., 2023, §4.1). Its content bias and inconsistent protocols are described in the Video Generation article. Prediction papers add their own variations:

• MAGVIT also reports a “debiased” FVD on BAIR to correct for the small validation set (Table 2).
• MOSO computes FVD with the StyleGAN-V implementation, which its authors describe as a precise reimplementation of the official one (§4).

Likelihood. Exact likelihood models report negative log-likelihood, for example Video Pixel Networks’ nats per frame on Moving MNIST and nats per dimension on Robotic Pushing (Kalchbrenner et al., 2017, Tables 1–3).

Coverage on synthetic data. Real data show one future per clip, so whether a model captures the whole distribution can be checked only where the distribution is known. On single-digit Stochastic Moving MNIST, Denton and Fergus recovered each sample’s digit trajectory by cross-correlating frames with the digit. This gave the empirical distribution of velocities, which they compared with the true one (§4.2, Fig. 4).

Two published examples on this site show related effects without a stochastic model. Video Generation’s example shows a frame-level Fréchet distance missing temporal flicker. Video anomaly detection’s example uses a linear future-frame predictor as an anomaly scorer.

Applications

• Robot planning. Action-conditioned predictors trained on data the robot collected on its own, without human supervision, are used to choose actions for robotic manipulation (Ebert et al., 2017).
• Representation learning. Predicting the future was proposed as a training signal for video features without labels (Srivastava et al., 2015). Denton and Fergus (2018, §2) review further work of this kind.
• Anomaly detection. The error of a future-frame predictor flags events that a model of normal scenes cannot anticipate; video anomaly detection describes this approach and its benchmarks.
• Simulation and other domains. Video LDM’s authors explore driving-scene prediction as a possible simulation engine (Blattmann et al., 2023, §4.1.2). SimVP is evaluated on traffic flows from taxi GPS data (TrafficBJ) and on Human3.6M as well as on Moving MNIST (Gao et al., 2022, §4.3).

Open Problems

• Evaluating distributions. Best-of-K scores reward coverage but not plausibility, frame-wise scores reward blur, and FVD depends on its features and protocol. A real clip shows one future, so the distribution a model should match is never observed.
• Long horizons. Feeding back predictions compounds error (Villegas et al., 2017), while long latent rollouts must stay consistent with the scene. Models are routinely tested beyond the horizon they were trained on (Denton & Fergus, 2018).
• The β trade-off and latent use. Latent-variable predictors can ignore their latents or let them copy the target, depending on the weight of the KL term (Denton & Fergus, 2018, §3), and SV2P needed a staged schedule to use them at all (as described by Denton & Fergus, 2018, §3.1).
• Objects, occlusion and physics. Keeping objects that disappear behind others, and predicting contact, remain difficult for pixel-level models (Ebert et al., 2017). Structured and decomposed representations are partial answers (Villegas et al., 2017; Sun et al., 2023).
• Comparable benchmarks. Contexts, horizons, sample counts and FVD code differ between papers on the same datasets, and SimVP’s authors list disparate metrics, datasets with distinct protocols, and different code frameworks as the obstacles to a fair comparison (Gao et al., 2022, §4.1).