Video Diffusion Model

A video diffusion model is a diffusion model whose denoising network takes a whole clip, a stack of frames, as its input and output. The noise process, the training loss and the samplers are those of image diffusion; what changes is the network that predicts the clean clip from a noisy one. Ho, Salimans and colleagues introduced the video U-Net (a 3D U-Net factorized over space and time) in Video Diffusion Models (Ho et al., 2022a in the Video Generation article’s labels; NeurIPS 2022), which presents what its authors call first results on video generation with diffusion models (§1). The later systems described below keep its central idea: process each frame with image layers, and mix information across frames with separate temporal layers.

The Video Generation article describes the task and the history of the systems built on this network, from Video Diffusion Models and Imagen Video to Stable Video Diffusion, Sora and Movie Gen, with their data, evaluation and risks. This article describes the denoiser itself: how it is assembled from an image network, the shapes it works on, how one network learns from both images and videos, how a model trained without conditions is made to continue or fill in given frames, and the design variants of the latent, cascaded and transformer-based models. Section, equation and table numbers refer to the NeurIPS copy of Video Diffusion Models and its Appendix A, and to the copies of the other papers named in the references.

Core Idea

A clip $x \in \mathbb{R}^{T \times H \times W \times C}$ is just a larger array than an image, so a diffusion model can treat it as one sample: the forward process noises all frames together, $z_t = \alpha_t x + \sigma_t \epsilon$, and a network $\hat x_\theta(z_t)$ is trained to recover $x$ (§2, Eq. 1–2). Ho et al. state that their models use “essentially the standard formulation” of Gaussian diffusion, with architectural changes to fit video into accelerator memory (§1).

The architectural change is a 3D U-Net factorized over space and time (§3). In our reading, the alternative, a network whose every layer mixes all frames, would cost more and could not reuse an image network’s layers as they are. The video U-Net instead keeps the image U-Net’s operations per frame and adds a small number of operations that run along the time axis only:

• every $3 \times 3$ convolution becomes a $1 \times 3 \times 3$ convolution, with the axes ordered frames, height, width, so it still sees one frame at a time (§3);
• every spatial attention block still attends over the positions of one frame, with the frame axis treated as a batch axis (§3);
• after each spatial attention block a temporal attention block attends over the frames at each spatial position, with the spatial axes treated as batch axes, and uses relative position embeddings so that the network knows the order of frames without an absolute notion of time (§3).

This has two consequences that recur in the later designs. Temporal mixing is cheap compared with joint space-time attention, the reason the same factorization is used in video transformers such as TimeSformer (§3 cites it for this reason). And because the temporal blocks are the only place where frames interact, switching them off turns the video network into an image network with the same weights, which lets one network train on images and videos together.

Building Blocks

Residual blocks with space-only convolutions. The image U-Net’s residual blocks, in the Wide ResNet style, are kept; only their kernels gain a time axis of length 1 (§3). Conditioning, such as the noise level $\lambda_t$ (the log signal-to-noise ratio) and a text or class embedding, is passed through several MLP layers and added into each residual block, as in image models (§3). In Appendix A the UCF101 and text-to-video models use 4 MLP layers and a 1,024-dimensional conditioning embedding.

Spatial attention. Multi-head self-attention over the $h \times w$ positions of each frame, run only at the coarser resolutions: in all the configurations of Appendix A, at $32 \times 32$, $16 \times 16$ and $8 \times 8$, with heads of dimension 64 (128 in the $9 \times 128 \times 128$ text-to-video model).

Temporal attention. Self-attention over the $T$ frames at each spatial position, inserted after each spatial attention block (§3). Its sequences are short ($T$ tokens), and there are $h \cdot w$ of them per clip. Relative position embeddings (Shaw et al., cited in §3) encode the offset between two frames rather than their absolute index; in our reading this suits a model that is later applied to windows taken from anywhere in a longer video.

Temporal convolution. Imagen Video’s U-Net offers a second temporal operation: a convolution along the frame axis (Fig. 7). In the paper’s words, temporal attention lets its base model capture long-term temporal dependencies, while the temporal convolutions of its super-resolution models keep local temporal consistency during upsampling, at lower memory and compute cost where frame rates and resolutions are high (Imagen Video, §2.3). Video LDM uses both kinds, temporal attention and residual blocks of 3D convolutions, and reports that on its real driving scenes its 3D-convolutional temporal layers, which also let context frames enter the network spatially, gave lower FVD and FID than an attention-only model with the same spatial layers and number of trainable parameters (Blattmann et al., 2023a, §3.1, §4.1.1, Table 1).

Skip connections and the U shape. Downsampling and upsampling act on height and width only: the frame count is the same at every level of the network (Fig. 1 of Ho et al. halves height and width at each of its $K$ levels and labels each block by its spatial size and channel multiplier). Upsampling blocks receive the matching downsampling activations by concatenation, as in the U-Net (Fig. 1).

How Data Flows

Appendix A’s UCF101 model makes the shapes concrete. It generates 16 frames of $64 \times 64$ RGB, has a base width of 256 channels with multipliers 1, 2, 4, 8 and two residual blocks per resolution. Following Fig. 1 (our arithmetic from these settings), the activations at the four levels are:

Level	Tensor (frames × height × width × channels)	Spatial attention	Temporal attention
1	$16 \times 64 \times 64 \times 256$	no	no
2	$16 \times 32 \times 32 \times 512$	16 sequences of 1,024 positions	1,024 sequences of 16 frames
3	$16 \times 16 \times 16 \times 1024$	16 sequences of 256	256 sequences of 16
4	$16 \times 8 \times 8 \times 2048$	16 sequences of 64	64 sequences of 16

At level 2, factorized attention forms $16 \times 1024^2 + 1024 \times 16^2 \approx 17.0$ million query–key pairs, where attention over all $16 \times 1024$ positions jointly would form about 268 million (our arithmetic); the TimeSformer and Video Transformers articles work through these costs for classification networks.

In code, the factorization is a pair of reshapes around each layer. Stable Video Diffusion’s open implementation (VideoResBlock in video_model.py of Stability AI’s generative-models at commit e8cd657) keeps the batch and frame axes merged, (b t) c h w, for the spatial residual block, reshapes to b c t h w for a 3D residual block whose kernel is [3, 1, 1] (time first, as set by video_kernel_size in svd.yaml), and reshapes back; Video LDM describes the same reshapes in einops notation (Blattmann et al., 2023a, §3.1).

Mathematical Formulation

Training objective and parameterizations

Ho et al. write the forward process in continuous time with $q(z_t \mid x) = \mathcal{N}(\alpha_t x, \sigma_t^2 I)$, log signal-to-noise ratio $\lambda_t = \log(\alpha_t^2/\sigma_t^2)$ decreasing in $t$, and train $\hat x_\theta$ with a weighted squared error $\mathbb{E}_{\epsilon,t}\, w(\lambda_t)\, \lVert \hat x_\theta(z_t) - x \rVert_2^2$ (§2, Eq. 1–2). In practice they predict $\epsilon$, with $\hat x_\theta = (z_t - \sigma_t \epsilon_\theta)/\alpha_t$ and a cosine schedule, and for some models $v$ (§2); Appendix A lists $\epsilon$ for the UCF101 and text-to-video models and $v$ for BAIR and Kinetics, with log-SNR clipped to $[-20, 20]$ in every configuration.

The v-parameterization, $v_t = \alpha_t \epsilon - \sigma_t x$ (introduced by Salimans and Ho and described in the diffusion model article), has a benefit specific to video. Imagen Video uses it for all its models and reports that at high resolution it avoids colour-shifting artifacts, and in video the temporal colour shifting that sometimes appeared with $\epsilon$-prediction; it also converged faster on sample-quality metrics (Imagen Video, §2.4, §3.3). In our reading, the reason is that $\hat x = (z_t - \sigma_t \hat\epsilon)/\alpha_t$ divides by a small $\alpha_t$ at high noise, so small errors in $\hat\epsilon$ become large errors in the predicted clip, whereas $\hat x = \alpha_t z_t - \sigma_t \hat v$ has no such division.

Sampling

Video Diffusion Models uses the ancestral sampler

$$z_s = \tilde\mu_{s|t}\big(z_t, \hat x_\theta(z_t)\big) + \sqrt{(\tilde\sigma^2_{s|t})^{1-\gamma}(\sigma^2_{t|s})^{\gamma}}\;\epsilon, \qquad \tilde\mu_{s|t}(z_t, x) = e^{\lambda_t - \lambda_s}\frac{\alpha_s}{\alpha_t} z_t + \big(1 - e^{\lambda_t - \lambda_s}\big)\alpha_s x,$$

with $\tilde\sigma^2_{s|t} = (1 - e^{\lambda_t - \lambda_s})\sigma_s^2$ and $\sigma^2_{t|s} = (1 - e^{\lambda_t - \lambda_s})\sigma_t^2$, where $\gamma$ interpolates between the two variance bounds (§2, Eq. 3–4), and a predictor–corrector variant that adds a Langevin step $z_s \leftarrow z_s - \tfrac12 \delta \sigma_s \epsilon_\theta(z_s) + \sqrt{\delta}\,\sigma_s \epsilon'$ with $\delta = 0.1$ (Eq. 5). Appendix A gives 256 sampling steps for most models, 128 plus 128 Langevin corrections for Kinetics and 256 plus 256 for BAIR, and $\gamma$ between 0.0 and 0.3.

Conditioning on known frames: replacement and reconstruction guidance

A model trained on 16-frame windows can produce longer or smoother video only if it can generate frames $x^b$ given other frames $x^a$: the next window given the last frames (extension), or the frames between given ones (temporal interpolation). Ho et al. derive this conditional from an unconditionally trained model rather than training a separate one (§3.1).

The replacement method, which they attribute to Song et al. (2021), samples the joint latent $z_s = [z_s^a, z_s^b]$ as usual but overwrites the known part at every step with a sample of the forward process, $z_s^a \sim q(z_s^a \mid x^a)$, or with one from $q(z_s^a \mid x^a, z_t^a)$, which also has the right conditional distribution; Ho et al. call both forms the replacement method (§3.1). Its premise is that $z^b$ will conform to $z^a$ through the denoiser, which sees both.

Ho et al. report that in their video models the replaced samples “looked good in isolation” but were often not coherent with $x^a$, and identify the reason (§3.1). The update of $z^b$ follows $\hat x^b_\theta(z_t) \approx \mathbb{E}_q[x^b \mid z_t]$, whereas exact conditional sampling needs

$$\mathbb{E}_q[x^b \mid z_t, x^a] = \mathbb{E}_q[x^b \mid z_t] + \frac{\sigma_t^2}{\alpha_t}\,\nabla_{z_t^b}\log q(x^a \mid z_t).$$

The second term, missing under replacement, measures how a change of $z^b$ would change the evidence for the known frames. It has no closed form, so they approximate $q(x^a \mid z_t) \approx \mathcal{N}\big(\hat x^a_\theta(z_t), (\sigma_t^2/\alpha_t^2) I\big)$, the model’s own reconstruction of the known frames with variance $\sigma_t^2/\alpha_t^2$, an approximation they note becomes exact as $t \to 0$ for a perfect model (§3.1). With a weight $w_r$, this gives reconstruction guidance (Eq. 7):

$$\tilde x^b_\theta(z_t) = \hat x^b_\theta(z_t) - \frac{w_r \alpha_t}{2}\,\nabla_{z_t^b}\big\lVert x^a - \hat x^a_\theta(z_t)\big\rVert_2^2 .$$

Substituting the Gaussian approximation into the exact expression, $\frac{\sigma_t^2}{\alpha_t}\cdot\big(-\frac{\alpha_t^2}{2\sigma_t^2}\big) = -\frac{\alpha_t}{2}$, so $w_r = 1$ is the weight the approximation itself implies (our derivation, checked against Eq. 7). The paper finds larger weights, $w_r > 1$, tend to improve sample quality, and that the method works especially well with the predictor–corrector sampler (§3.1); Appendix A lists $w_r = 50$ for BAIR and $w_r = 9$ for Kinetics. The gradient runs through the whole denoiser, so each guided step costs a backward pass as well as a forward pass (our reading of Eq. 7).

The same construction gives spatial super-resolution by an unconditional high-resolution model: the squared error is taken between a low-resolution video $x^a$ and a differentiable downsampling (for example bilinear) of the model’s high-resolution prediction (§3.1, Eq. 8). Ho et al. combine both: a $16 \times 64 \times 64$ text-to-video model at frameskip 4 produces a coarse clip, and a $9 \times 128 \times 128$ model extends and upsamples it to $64 \times 128 \times 128$ at frameskip 1 using reconstruction guidance for both conditions (§3.1, Fig. 2).

In their experiments on 64-frame text-to-video extension from a 16-frame model, reconstruction guidance gave an FVD of 136.22 against the replacement method’s 451.45 at the table’s guidance weight 2.0, and 133.92 against 456.24 at 5.0 (training-set reference; Table 6), and their Fig. 4 shows replacement samples whose 16-frame blocks look like unrelated clips. The first-frame scores in Table 6 are almost equal for the two methods, as expected when the first block is sampled before any conditioning is applied (our reading).

Example: Reconstruction Guidance with an Exact Denoiser

Ho et al.‘s argument for reconstruction guidance is that replacement leaves out the term $\nabla_{z^b}\log q(x^a \mid z_t)$, so its samples look valid but drift from the given frames. With a trained network one cannot tell whether a failure comes from the conditioning method or from the network. The script below removes the network: the data are a Gaussian mixture, so the denoiser $\mathbb{E}[x \mid z_t]$ and the gradient in Eq. 7 are exact, and the exact conditional distribution of the missing frames is known in closed form. Any incoherence that remains is due to the conditioning method alone. The outcome therefore tests our exact-mixture analogue, not Video Diffusion Models’ trained networks on real video.

Clips. Each clip has 8 frames of $12 \times 12$ pixels: background $-1$ and a $3 \times 3$ square at $+1$ whose top-left corner follows one of 32 trajectories, and pixel noise of standard deviation 0.1, so the clip distribution is an equal-weight mixture of 32 Gaussians around rendered templates. The trajectories come in 8 groups of 4 that share frames 0 and 1 and then continue as 4 different random walks (steps of $-1$, 0 or $+1$ pixel per axis per frame). Frames are flattened in the order frame, row, column.

Tasks. predict gives frames 0–1 and asks for frames 2–7; since four trajectories share every pair of starting frames, the exact conditional has four equally likely continuations. interp gives frames 0, 2, 4, 6 and asks for 1, 3, 5, 7; here the given frames identify a single trajectory. Each task uses 64 given clips with 8 samples each, 512 samples per method.

Methods. All diffusion samplers use VDM’s ancestral sampler (Eq. 3–4) with $\gamma = 0$, 256 steps and a cosine log-SNR schedule clipped to $[-20, 20]$, as in Appendix A:

• reference: samples drawn directly from the closed-form conditional $p(x^b \mid x^a)$;
• exact-cond: the diffusion sampler run on the missing frames only, with the exact conditional denoiser; this checks the sampler;
• replace fresh and replace post: the replacement method in both forms §3.1 names, $z_s^a \sim q(z_s^a \mid x^a)$ and $z_s^a \sim q(z_s^a \mid x^a, z_t^a)$, with the unconditional exact denoiser on the whole clip;
• RG fresh and RG post: each replacement form with $\hat x^b$ replaced by Eq. 7 at $w_r \in \{1, 2, 5, 10\}$. The gradient is the exact vector–Jacobian product of the mixture denoiser; each RG run uses the same random stream as the replacement run of its form.

Measures. coherent counts the samples whose generated frames are nearest (in Euclidean distance) to a template that has the same given frames as the true clip. isolation RMS is the per-pixel RMS distance from the generated frames to their nearest template, whatever its given frames: it asks whether a sample “looks good in isolation”. error vs $\mathbb{E}[x^b \mid x^a]$ is the per-pixel RMS distance to the exact conditional mean, averaged over samples; it is not zero for a correct sampler, because a sample is not a mean.

Checks and hypotheses, fixed before any replacement or guided sampler ran:

• C1: the templates are distinct, groups share frames 0–1 exactly and only within a group; C2: the gradient agrees with central differences to a relative error of at most $10^{-4}$; C3: exact-cond has at least 507 of 512 coherent samples and an error within 0.010 of reference’s; C4: RG with $w_r = 0$ (the guidance code run with weight 0) reproduces replace fresh exactly; C5: reference is fully coherent and the given frames of each clip are nearest to their own template.
• H1, from VDM’s claim: for each task and form, RG at each $w_r$ is coherent more often than replacement (16 cases per seed).
• H2, VDM’s “looked good in isolation”: replacement’s isolation RMS is within 0.010 of exact-cond’s (4 cases per seed).
• H3, from “$w_r > 1$ tends to improve sample quality”: the coherent count at $w_r = 2$, 5 and 10 is at least that at $w_r = 1$ (12 cases per seed).
• H4: RG’s error against the conditional mean is lower than replacement’s in the same form (16 cases per seed).

The tolerance 0.010 in C3 and H2 was set after a planning run of the reference and exact-cond rows only, with the same data and random streams as the output below, had printed those rows; C3 therefore checks values that were known when its bar was set, while no replacement or guided sampler had run when H2 was fixed. The script below is the planned script, run once as a pilot on seed 0 and then, unchanged, on seeds 0, 1 and 2 for the output shown.

# Video Diffusion Model example: reconstruction guidance (Ho et al., 2022a, Eq. 7) against
# the replacement method, with an exact Gaussian-mixture denoiser on 8-frame 12x12 clips.
# Usage: python pilot.py SEED [SEED ...]
import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"; os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"; os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
import sys
import numpy as np

T, H, W, S = 8, 12, 12, 3          # frames, height, width, square side
G, C = 8, 4                        # 8 groups sharing frames 0-1, 4 continuations each
K = G * C                          # 32 trajectories = 32 mixture components
S0 = 0.1                           # pixel noise of the data (clip = template + S0 * noise)
STEPS = 256                        # ancestral sampler (VDM Eq. 4, gamma = 0)
M, R = 64, 8                       # conditioning clips per task, samples per clip
TASKS = {"predict": [0, 1], "interp": [0, 2, 4, 6]}   # known frames x^a
WR = [1, 2, 5, 10]                 # reconstruction-guidance weights

def trajectories(seed=1234):
    """K paths of the square's top-left corner: G distinct (frame 0, frame 1) prefixes,
    each continued by C distinct random walks (steps in {-1, 0, 1} per axis)."""
    rng = np.random.default_rng(seed)
    paths, prefixes = [], set()
    while len(prefixes) < G:
        p0 = rng.integers(1, H - S, 2); p1 = np.clip(p0 + rng.integers(-1, 2, 2), 0, H - S)
        key = (tuple(p0), tuple(p1))
        if key in prefixes: continue
        conts = set()
        while len(conts) < C:
            p, path = p1.copy(), [p0, p1]
            for t in range(2, T):
                p = np.clip(p + rng.integers(-1, 2, 2), 0, H - S); path.append(p.copy())
            k2 = tuple(map(tuple, path[2:]))
            if k2 in conts: continue
            conts.add(k2); paths.append(np.array(path))
        prefixes.add(key)
    return np.array(paths)

def render(path):
    clip = -np.ones((T, H, W))
    for t, (r, c) in enumerate(path): clip[t, r:r + S, c:c + S] = 1.0
    return clip

PATHS = trajectories()
MU = np.array([render(p) for p in PATHS]).reshape(K, -1)     # (K, T*H*W), values +-1

def idx(frames):
    return np.concatenate([np.arange(f * H * W, (f + 1) * H * W) for f in frames])

LAM = np.clip(-2 * np.log(np.tan(np.pi * np.linspace(1, 0, STEPS + 1) / 2) + 1e-300), -20, 20)
def a_s(lam): return np.sqrt(1 / (1 + np.exp(-lam))), np.sqrt(1 / (1 + np.exp(lam)))

def denoise(z, mu, al, si, logw=None):
    """Exact E[x | z_t] for x ~ (1/K) sum_k N(mu_k, S0^2 I), z_t = al x + si eps,
    optionally with prior log weights logw (N, K). Returns xhat, responsibilities r, c."""
    v = al**2 * S0**2 + si**2
    d2 = (z**2).sum(1)[:, None] - 2 * al * z @ mu.T + al**2 * (mu**2).sum(1)[None]
    lr = -d2 / (2 * v) + (0 if logw is None else logw)
    lr -= lr.max(1, keepdims=True); r = np.exp(lr); r /= r.sum(1, keepdims=True)
    c = al * S0**2 / v
    return (1 - c * al) * (r @ mu) + c * z, r, c

def recon_grad(z, xa, xhat, r, c, ia, ib, al, si):
    """Exact gradient wrt z^b of ||x^a - xhat^a(z)||^2 (vector-Jacobian product)."""
    v = al**2 * S0**2 + si**2
    e = xa - xhat[:, ia]
    ma = (1 - c * al) * MU[:, ia][None] + c * z[:, ia][:, None, :]   # per-component means m_k^a
    s = np.einsum("nkd,nd->nk", ma, e) * r
    return -2 * (al / v) * (s @ MU[:, ib] - s.sum(1, keepdims=True) * (r @ MU[:, ib]))

def sample(rng, xa, ia, ib, method, wr=None, logw=None):
    """Ancestral sampling (VDM Eq. 3-4, gamma = 0). method: 'exact' (exact conditional
    denoiser on z^b only), 'fresh' or 'post' (replacement method, z^a from q(z_s^a|x^a) or
    q(z_s^a|x^a, z_t^a)), with reconstruction guidance (Eq. 7) when wr is given (wr = 0 runs the guidance code
    and adds zero times its gradient)."""
    N = len(xa)
    if method == "exact":
        z = rng.normal(size=(N, len(ib))); mu = MU[:, ib]
    else:
        z = rng.normal(size=(N, T * H * W)); mu = MU
    for i in range(STEPS):
        al_t, si_t = a_s(LAM[i]); al_s, si_s = a_s(LAM[i + 1])
        xhat, r, c = denoise(z, mu, al_t, si_t, logw if method == "exact" else None)
        if method != "exact" and wr is not None:
            xhat = xhat.copy()
            xhat[:, ib] -= wr * al_t / 2 * recon_grad(z, xa, xhat, r, c, ia, ib, al_t, si_t)
        e = np.exp(LAM[i] - LAM[i + 1])
        za_t = None if method == "exact" else z[:, ia].copy()
        z = e * (al_s / al_t) * z + (1 - e) * al_s * xhat \
            + np.sqrt((1 - e) * si_s**2) * rng.normal(size=z.shape)
        if method == "fresh":                       # z_s^a ~ q(z_s^a | x^a)
            z[:, ia] = al_s * xa + si_s * rng.normal(size=xa.shape)
        elif method == "post":                      # z_s^a ~ q(z_s^a | x^a, z_t^a)
            z[:, ia] = e * (al_s / al_t) * za_t + (1 - e) * al_s * xa \
                + np.sqrt((1 - e) * si_s**2) * rng.normal(size=xa.shape)
    out = z / a_s(LAM[-1])[0]
    return out if method == "exact" else out[:, ib]

def measures(xb, kstar, ia, ib, w):
    """coherent: nearest template of the generated frames shares the known frames' template;
    isolation: RMS distance of the generated frames to their nearest template;
    err: RMS distance to the exact conditional mean E[x^b | x^a]."""
    A, B = MU[:, ia], MU[:, ib]
    d2 = ((xb[:, None] - B[None]) ** 2).sum(2); kh = d2.argmin(1)
    coh = (np.abs(A[kh] - A[kstar]).max(1) == 0)
    iso = np.sqrt(d2[np.arange(len(xb)), kh] / B.shape[1])
    err = np.sqrt(((xb - w @ B) ** 2).mean(1))
    return int(coh.sum()), iso.mean(), err.mean()

def data_checks():
    pre = MU[:, :2 * H * W]
    same = np.abs(pre[:, None] - pre[None]).max(2) == 0
    groups = all(same[i, j] == (i // C == j // C) for i in range(K) for j in range(K))
    d = np.sqrt(((MU[:, None] - MU[None]) ** 2).sum(2)); np.fill_diagonal(d, np.inf)
    print(f"C1 data: {K} templates, distinct: {bool(d.min() > 0)}, groups of {C} share frames 0-1 "
          f"only within group: {groups}, inside grid: {bool(PATHS.min() >= 0 and PATHS.max() <= H - S)}")
    for name, fa in TASKS.items():
        A = MU[:, idx(fa)]
        sz = [int((np.abs(A - A[k]).max(1) == 0).sum()) for k in range(K)]
        print(f"   {name}: known frames {fa}, consistent templates per clip {min(sz)}-{max(sz)}")

def fd_check(rng):
    out = []
    for fa in TASKS.values():
        ia = idx(fa); ib = idx([f for f in range(T) if f not in fa])
        for lam in (-6.0, -4.0, -2.0):
            al, si = a_s(lam)
            i, j = rng.choice(K, 2, replace=False)
            xa = MU[i, ia][None] + S0 * rng.normal(size=(1, len(ia)))
            z = al * 0.5 * (MU[i] + MU[j])[None] + si * rng.normal(size=(1, T * H * W))
            xhat, r, c = denoise(z, MU, al, si)
            g = recon_grad(z, xa, xhat, r, c, ia, ib, al, si)[0]
            L = lambda zz: ((xa[0] - denoise(zz[None], MU, al, si)[0][0, ia]) ** 2).sum()
            for _ in range(5):
                u = rng.normal(size=len(ib)); zp = z[0].copy(); zm = z[0].copy()
                zp[ib] += 1e-4 * u; zm[ib] -= 1e-4 * u
                fd = (L(zp) - L(zm)) / 2e-4; an = g @ u
                out.append(abs(fd - an) / max(abs(fd), abs(an)))
    print(f"C2 gradient vs central differences: max relative error {max(out):.3e} ({len(out)} directions)")

ROWS = ["reference", "exact-cond", "replace fresh", "replace post"] + \
       [f"RG fresh w={w}" for w in WR] + [f"RG post w={w}" for w in WR]

def run(seed):
    res = {}
    for ti, (name, fa) in enumerate(TASKS.items()):
        ia = idx(fa); ib = idx([f for f in range(T) if f not in fa])
        rng = np.random.default_rng([seed, ti])
        kstar = rng.integers(K, size=M)
        xa = MU[kstar][:, ia] + S0 * rng.normal(size=(M, len(ia)))
        logw = -((xa[:, None] - MU[None, :, ia]) ** 2).sum(2) / (2 * S0**2)
        logw -= logw.max(1, keepdims=True); w = np.exp(logw); w /= w.sum(1, keepdims=True)
        A = MU[:, ia]; ka = ((xa[:, None] - A[None]) ** 2).sum(2).argmin(1)
        a_ok = bool((np.abs(A[ka] - A[kstar]).max(1) == 0).all())
        rep = np.repeat(np.arange(M), R); xa_r, k_r, w_r, lw_r = xa[rep], kstar[rep], w[rep], logw[rep]
        ks = np.array([rng.choice(K, p=p) for p in w_r])
        x = {"reference": MU[ks][:, ib] + S0 * rng.normal(size=(M * R, len(ib)))}
        g = lambda j: np.random.default_rng([seed, ti, j])     # same noise stream per method index
        x["exact-cond"] = sample(g(1), xa_r, ia, ib, "exact", logw=lw_r)
        x["replace fresh"] = sample(g(2), xa_r, ia, ib, "fresh")
        x["replace post"] = sample(g(3), xa_r, ia, ib, "post")
        rg0 = sample(g(2), xa_r, ia, ib, "fresh", wr=0.0)       # C4: w_r = 0 must equal replacement
        for wv in WR:
            x[f"RG fresh w={wv}"] = sample(g(2), xa_r, ia, ib, "fresh", wr=wv)
            x[f"RG post w={wv}"] = sample(g(3), xa_r, ia, ib, "post", wr=wv)
        print(f"seed {seed}, {name} (known frames {fa}): x^a nearest template consistent: {a_ok}; "
              f"C4 max|RG(w=0) - replace fresh| = {np.abs(rg0 - x['replace fresh']).max():.1e}")
        print(f"  {'method':15s} coherent/{M*R}  isolation RMS  error vs E[x^b|x^a]")
        for k in ROWS:
            c_, i_, e_ = measures(x[k], k_r, ia, ib, w_r); res[(name, k)] = (c_, i_, e_)
            print(f"  {k:15s} {c_:9d}  {i_:13.4f}  {e_:18.4f}")
    return res

if __name__ == "__main__":
    seeds = [int(s) for s in sys.argv[1:]] or [0]
    data_checks(); fd_check(np.random.default_rng(99))
    allres = [run(s) for s in seeds]
    if len(seeds) > 1:
        print(f"\nranges over seeds {seeds}")
        for name in TASKS:
            for k in ROWS:
                v = np.array([r[(name, k)] for r in allres])
                print(f"  {name:8s} {k:15s} coherent {int(v[:,0].min())}-{int(v[:,0].max())}  "
                      f"isolation {v[:,1].min():.4f}-{v[:,1].max():.4f}  error {v[:,2].min():.4f}-{v[:,2].max():.4f}")

Output (Python with NumPy 2.3.5, BLAS threads set to 1; about 7 minutes on a laptop CPU, approximate since other jobs shared the machine; a second run gave identical output):

C1 data: 32 templates, distinct: True, groups of 4 share frames 0-1 only within group: True, inside grid: True
   predict: known frames [0, 1], consistent templates per clip 4-4
   interp: known frames [0, 2, 4, 6], consistent templates per clip 1-1
C2 gradient vs central differences: max relative error 1.071e-06 (30 directions)
seed 0, predict (known frames [0, 1]): x^a nearest template consistent: True; C4 max|RG(w=0) - replace fresh| = 0.0e+00
  method          coherent/512  isolation RMS  error vs E[x^b|x^a]
  reference             512         0.1001              0.3805
  exact-cond            512         0.0941              0.3811
  replace fresh         308         0.0942              0.4561
  replace post          232         0.0941              0.4834
  RG fresh w=1          433         0.0942              0.4096
  RG fresh w=2          467         0.0942              0.3970
  RG fresh w=5          483         0.0942              0.3906
  RG fresh w=10         496         0.0942              0.3865
  RG post w=1           376         0.0941              0.4330
  RG post w=2           425         0.0941              0.4143
  RG post w=5           471         0.0941              0.3971
  RG post w=10          488         0.0941              0.3897
seed 0, interp (known frames [0, 2, 4, 6]): x^a nearest template consistent: True; C4 max|RG(w=0) - replace fresh| = 0.0e+00
  method          coherent/512  isolation RMS  error vs E[x^b|x^a]
  reference             512         0.1001              0.1001
  exact-cond            512         0.0941              0.0941
  replace fresh         502         0.0943              0.1036
  replace post          466         0.0952              0.1417
  RG fresh w=1          511         0.0943              0.0953
  RG fresh w=2          511         0.0943              0.0953
  RG fresh w=5          512         0.0943              0.0943
  RG fresh w=10         511         0.0943              0.0953
  RG post w=1           496         0.0950              0.1099
  RG post w=2           505         0.0943              0.1012
  RG post w=5           507         0.0942              0.0994
  RG post w=10          512         0.0942              0.0942
seed 1, predict (known frames [0, 1]): x^a nearest template consistent: True; C4 max|RG(w=0) - replace fresh| = 0.0e+00
  method          coherent/512  isolation RMS  error vs E[x^b|x^a]
  reference             512         0.0999              0.3848
  exact-cond            512         0.0942              0.3851
  replace fresh         315         0.0943              0.4546
  replace post          239         0.0943              0.4834
  RG fresh w=1          447         0.0943              0.4080
  RG fresh w=2          474         0.0943              0.3984
  RG fresh w=5          498         0.0943              0.3912
  RG fresh w=10         500         0.0943              0.3892
  RG post w=1           388         0.0943              0.4307
  RG post w=2           427         0.0943              0.4164
  RG post w=5           472         0.0943              0.4011
  RG post w=10          493         0.0943              0.3928
seed 1, interp (known frames [0, 2, 4, 6]): x^a nearest template consistent: True; C4 max|RG(w=0) - replace fresh| = 0.0e+00
  method          coherent/512  isolation RMS  error vs E[x^b|x^a]
  reference             512         0.1000              0.1000
  exact-cond            512         0.0942              0.0942
  replace fresh         505         0.0941              0.1009
  replace post          466         0.0961              0.1412
  RG fresh w=1          512         0.0941              0.0941
  RG fresh w=2          512         0.0941              0.0941
  RG fresh w=5          511         0.0941              0.0950
  RG fresh w=10         512         0.0941              0.0941
  RG post w=1           492         0.0946              0.1132
  RG post w=2           496         0.0942              0.1098
  RG post w=5           502         0.0940              0.1036
  RG post w=10          505         0.0940              0.1008
seed 2, predict (known frames [0, 1]): x^a nearest template consistent: True; C4 max|RG(w=0) - replace fresh| = 0.0e+00
  method          coherent/512  isolation RMS  error vs E[x^b|x^a]
  reference             512         0.1000              0.3828
  exact-cond            512         0.0941              0.3801
  replace fresh         313         0.0940              0.4508
  replace post          239         0.0942              0.4816
  RG fresh w=1          440         0.0940              0.4043
  RG fresh w=2          467         0.0940              0.3962
  RG fresh w=5          491         0.0940              0.3880
  RG fresh w=10         498         0.0940              0.3853
  RG post w=1           367         0.0942              0.4335
  RG post w=2           410         0.0942              0.4174
  RG post w=5           460         0.0942              0.3978
  RG post w=10          488         0.0942              0.3890
seed 2, interp (known frames [0, 2, 4, 6]): x^a nearest template consistent: True; C4 max|RG(w=0) - replace fresh| = 0.0e+00
  method          coherent/512  isolation RMS  error vs E[x^b|x^a]
  reference             512         0.0999              0.0999
  exact-cond            512         0.0942              0.0942
  replace fresh         498         0.0940              0.1070
  replace post          457         0.0972              0.1514
  RG fresh w=1          511         0.0940              0.0949
  RG fresh w=2          511         0.0940              0.0949
  RG fresh w=5          512         0.0940              0.0940
  RG fresh w=10         512         0.0940              0.0940
  RG post w=1           494         0.0942              0.1116
  RG post w=2           499         0.0942              0.1068
  RG post w=5           506         0.0942              0.1002
  RG post w=10          509         0.0942              0.0973

ranges over seeds [0, 1, 2]
  predict  reference       coherent 512-512  isolation 0.0999-0.1001  error 0.3805-0.3848
  predict  exact-cond      coherent 512-512  isolation 0.0941-0.0942  error 0.3801-0.3851
  predict  replace fresh   coherent 308-315  isolation 0.0940-0.0943  error 0.4508-0.4561
  predict  replace post    coherent 232-239  isolation 0.0941-0.0943  error 0.4816-0.4834
  predict  RG fresh w=1    coherent 433-447  isolation 0.0940-0.0943  error 0.4043-0.4096
  predict  RG fresh w=2    coherent 467-474  isolation 0.0940-0.0943  error 0.3962-0.3984
  predict  RG fresh w=5    coherent 483-498  isolation 0.0940-0.0943  error 0.3880-0.3912
  predict  RG fresh w=10   coherent 496-500  isolation 0.0940-0.0943  error 0.3853-0.3892
  predict  RG post w=1     coherent 367-388  isolation 0.0941-0.0943  error 0.4307-0.4335
  predict  RG post w=2     coherent 410-427  isolation 0.0941-0.0943  error 0.4143-0.4174
  predict  RG post w=5     coherent 460-472  isolation 0.0941-0.0943  error 0.3971-0.4011
  predict  RG post w=10    coherent 488-493  isolation 0.0941-0.0943  error 0.3890-0.3928
  interp   reference       coherent 512-512  isolation 0.0999-0.1001  error 0.0999-0.1001
  interp   exact-cond      coherent 512-512  isolation 0.0941-0.0942  error 0.0941-0.0942
  interp   replace fresh   coherent 498-505  isolation 0.0940-0.0943  error 0.1009-0.1070
  interp   replace post    coherent 457-466  isolation 0.0952-0.0972  error 0.1412-0.1514
  interp   RG fresh w=1    coherent 511-512  isolation 0.0940-0.0943  error 0.0941-0.0953
  interp   RG fresh w=2    coherent 511-512  isolation 0.0940-0.0943  error 0.0941-0.0953
  interp   RG fresh w=5    coherent 511-512  isolation 0.0940-0.0943  error 0.0940-0.0950
  interp   RG fresh w=10   coherent 511-512  isolation 0.0940-0.0943  error 0.0940-0.0953
  interp   RG post w=1     coherent 492-496  isolation 0.0942-0.0950  error 0.1099-0.1132
  interp   RG post w=2     coherent 496-505  isolation 0.0942-0.0943  error 0.1012-0.1098
  interp   RG post w=5     coherent 502-507  isolation 0.0940-0.0942  error 0.0994-0.1036
  interp   RG post w=10    coherent 505-512  isolation 0.0940-0.0942  error 0.0942-0.1008

All five checks held on every seed. The gradient matched central differences to a relative error of $1.071 \times 10^{-6}$ (C2), running the guidance code with weight 0 changed nothing (C4), and the exact conditional sampler produced 512 coherent samples of 512 in both tasks on every seed, with errors within 0.006 of the direct reference (C3). Its isolation RMS, 0.094 against the reference’s 0.100, is lower in every row; in our reading this is because the $\gamma = 0$ step leaves the uncertainty about $x$ given $z_t$ out of the step’s variance, so diffusion samples sit slightly closer to the templates. For that reason the diffusion rows are compared with exact-cond, not with reference.

When frames are continued (predict), the replacement method is often incoherent, with 308–315 of 512 samples coherent in the fresh form and 232–239 in the posterior form, while its isolation RMS (0.0940–0.0943) is that of the exact sampler (0.0941–0.0942). Its samples are valid clips of the mixture that continue some other pair of starting frames, which is VDM’s observation that replaced samples look good in isolation but are not coherent with the given frames. H2 held in all 12 cases (the largest difference, 0.0030, is replace post in interp on seed 2), and H1 held in all 48: reconstruction guidance was coherent more often than replacement in every task, form, weight and seed. In predict its coherence rises with $w_r$, from 433–447 at $w_r = 1$ to 496–500 at $w_r = 10$ in the fresh form and from 367–388 to 488–493 in the posterior form, without reaching the exact sampler’s 512 at the weights tried; the paper’s own models used $w_r = 9$ and 50 (Appendix A). H4 held in all 48 cases: with guidance the samples were always closer to the exact conditional mean than with replacement, and approached the exact sampler’s error (0.38 in predict, 0.094 in interp).

Interpolation is easier for replacement, with 498–505 (fresh) and 457–466 (posterior) of 512 coherent, and the fresh form with guidance reached 511 or 512 at every weight. H3 held in 35 of its 36 cases and failed in one: in interp with the fresh form on seed 1, $w_r = 5$ gave 511 coherent samples against 512 at $w_r = 1$, both within one sample of the ceiling. In our reading, the difference between the tasks follows from how the exact denoiser weighs the evidence: its responsibilities depend on all pixels equally, and in predict the six missing frames outweigh the two given ones, so the missing frames can settle on a continuation while the given frames are still too noisy to steer them; in interp half the clip is given. The output does not say why the posterior form of replacement does worse than the fresh one, and we do not offer a reason.

In our reading, an exact denoiser is the most favourable case for replacement, since nothing in the denoiser is learned imperfectly, and the replacement method still lost coherence; the size of the effect, though, belongs to this toy mixture and says nothing about the FVD gap in VDM’s Table 6.

Training Considerations

Joint image and video training. Because masking the temporal attention turns the video U-Net into an image model, Ho et al. append independent images, drawn from random videos of the same dataset, to the end of each training clip and mask the temporal attention so that no information flows between them and the video frames (§4.3.1); for images the attention matrix is fixed to the identity, each frame attending only to itself (§3). On text-to-video at $16 \times 64 \times 64$, adding 4 and 8 such images per clip reduced FVD from 202.28 to 68.11 and 57.84 (training-set reference; Table 4). The authors interpret the images as reducing the variance of the gradient at the cost of some bias for the video objective, a memory optimization that fits more independent examples in a batch (§4.3.1). Imagen Video extends the scheme to temporal convolutions, bypassing them by masking their computation path for images, and its authors find that styles learned from images, such as paintings, carry over into generated video (§2.6, §3.1, Fig. 8).

Starting from an image model. Later models start from a pretrained text-to-image model and add temporal layers, which the Video Generation article traces through Make-A-Video (Singer et al., 2023, not read here), Video LDM and Stable Video Diffusion. Video LDM mixes each temporal layer’s output $z'$ with the spatial output $z$ as $\alpha z + (1 - \alpha) z'$ with a learned $\alpha \in [0, 1]$, so that $\alpha = 1$ recovers the image model exactly, and trains only the temporal layers with the image backbone frozen (Blattmann et al., 2023a, §3.1, Fig. 4). Stability AI’s code implements this blend in AlphaBlender (util.py at e8cd657): $\alpha$ is $\operatorname{sigmoid}$ of a learned scalar, and with the learned_with_images strategy that svd.yaml selects, $\alpha$ is forced to 1 for samples flagged as images, so images pass through the spatial path alone. Emu Video, too, freezes its 2.7B spatial parameters and trains 1.7B temporal ones, initialized as identity operations (identity convolution kernels and a zeroed final layer in temporal attention), which in its preliminary experiments made convergence twice as fast (Girdhar et al., 2024, §3.3).

The latent autoencoder needs temporal layers too. In a latent model the image autoencoder decodes each frame independently. Video LDM’s authors found that an image-trained decoder causes flickering when decoding a temporally coherent latent sequence, and added temporal layers to the decoder, fine-tuned on video with a patch-wise 3D-convolutional discriminator while the encoder stays unchanged so that the latent space of the image model is kept (§3.1.1, Fig. 3). svd.yaml at e8cd657 likewise pairs an image encoder with a VideoDecoder whose temporal kernels are [3, 1, 1].

Noise schedules shift toward more noise. Image schedules leave some signal even at their noisiest step, and Emu Video’s authors write that this residual signal is higher for high-resolution video frames, whose pixels are redundant across space and time (Girdhar et al., 2024, §3.2). Stable Video Diffusion switches Stable Diffusion 2.1’s discrete schedule to EDM-style continuous noise with $\log\sigma \sim \mathcal{N}(P_\text{mean}, P_\text{std}^2)$ and preconditioning equivalent to EDM with $\sigma_\text{data} = 1$, starting from $P_\text{mean} = -1.2$, $P_\text{std} = 1$ for the image model and shifting $P_\text{mean}$ to 0 at $320 \times 576$, to 0.5 (with $P_\text{std} = 1.4$) for the $576 \times 1024$ text-to-video model, and to 0.7 and then 1.0 ($P_\text{std} = 1.6$) for image-to-video, which its authors call essential for high-resolution fine-tuning (Blattmann et al., 2023b, §2, §4.1, App. D.2–D.4). The code’s VScalingWithEDMcNoise (denoiser_scaling.py) gives the same preconditioning: $c_\text{skip} = 1/(\sigma^2 + 1)$, $c_\text{out} = -\sigma/\sqrt{\sigma^2 + 1}$, $c_\text{in} = 1/\sqrt{\sigma^2 + 1}$ and $c_\text{noise} = \tfrac14 \log\sigma$. Emu Video makes the terminal step pure noise, rescaling its schedule so that $\alpha_N = 0$ (zero terminal SNR), a choice its authors find critical for high-resolution video generation (§3.2).

Correlated noise priors. PYoCo (Ge et al., 2023) observed that when real frames of one video are inverted to noise by an image diffusion model, their noise maps are correlated (average pairwise cosine similarity 0.206 against 0.001 across different videos; Table 1), so an i.i.d. noise prior forces a fine-tuned model to unlearn that correlation (§4). It proposes two priors with a parameter $\alpha$: mixed, $\epsilon^i = \epsilon_\text{shared} + \epsilon^i_\text{ind}$ with variances $\alpha^2/(1+\alpha^2)$ and $1/(1+\alpha^2)$, and progressive, $\epsilon^i = \frac{\alpha}{\sqrt{1+\alpha^2}}\epsilon^{i-1} + \epsilon^i_\text{ind}$ with $\epsilon^i_\text{ind}$ of variance $1/(1+\alpha^2)$ (Eq. 3–4). Each frame’s noise still has unit variance, and the correlation between frames is $\alpha^2/(1+\alpha^2)$ for every pair under the mixed prior and $(\alpha/\sqrt{1+\alpha^2})^{|i-j|}$ under the progressive one (our arithmetic). $\alpha = 0$ is i.i.d. noise and $\alpha \to \infty$ gives identical noise and frozen video (§4). On UCF101, both priors improved fine-tuning from an image model over i.i.d. noise (§5.3, Table 5), and in an ablation over $\alpha$ with a small model the best FVD came at $\alpha = 1$ (mixed) and $\alpha = 2$ (progressive) (§5.3, Fig. 7).

Micro-conditioning. Stable Video Diffusion conditions its denoiser on the frame rate and on a motion score so that the amount of motion can be varied at inference (Blattmann et al., 2023b, App. D.2). In svd.yaml at e8cd657, three scalars, fps_id, motion_bucket_id and cond_aug (the noise level added to the conditioning frame), each have an embedder with outdim: 256, and the network’s vector input has adm_in_channels: 768, three times 256 (our arithmetic); video_model.py adds that vector’s embedding to the timestep embedding (emb = emb + self.label_emb(y)). The image-to-video model’s conditioning frame is concatenated channel-wise to the noisy latent as a noise-augmented copy encoded by the autoencoder, with augmentation noise $\log\sigma \sim \mathcal{N}(-3.0, 0.5^2)$ during training (App. D.4); in our reading this is why the configuration has in_channels: 8 for a 4-channel latent.

Conditioning augmentation in cascades. Cascaded models condition each super-resolution stage on the stage below. Imagen Video applies the noise conditioning augmentation of Cascaded Diffusion Models to all its temporal and spatial super-resolution models: during training the conditioning video is noised at a random signal-to-noise ratio that the model is told, and at sampling a fixed ratio such as 3 or 5 removes some artifacts of the previous stage while keeping most of its structure (Imagen Video, §2.5). The Imagen article describes the image version.

Variants

Cascades in pixel space. Imagen Video chains a base video U-Net with three spatial and three temporal super-resolution models, all generating whole blocks of frames; the super-resolution models condition by channel-wise concatenation of the upsampled input, bilinearly resized in space and repeated or blank-filled in time (§2.2–§2.3). Video Diffusion Models’ own two-stage pipeline did the same job with reconstruction guidance and no conditional training. The Video Generation article describes Imagen Video’s stages and results.

Latent video diffusion. Video LDM and Stable Video Diffusion run the same factorized U-Net on the latents of an image autoencoder, with the α-blend, the temporal decoder and the schedule shifts described above. Video LDM conditions prediction and interpolation models by a binary frame mask concatenated with the masked context latents (§3.2–§3.3), the trained alternative to reconstruction guidance; Stable Video Diffusion’s sampler in svd.yaml is an Euler sampler on an EDM discretization with $\sigma_\text{max} = 700$, and its LinearPredictionGuider raises the classifier-free guidance scale linearly across frames, from 1.0 to 2.5 for the 14-frame model and from 1.5 to 3.0 for the 25-frame model (svd_xt.yaml), the technique its report introduces in App. D.4.1.

Factorizing by an image. Emu Video splits text-to-video into two steps: generate the first frame with the text-to-image model (the same network with its temporal layers skipped), then generate the clip conditioned on the text and that frame, concatenated with the noisy input together with a mask marking its position (Girdhar et al., 2024, §3.2). With the zero-terminal-SNR schedule and multi-stage training, its authors generate $512 \times 512$ video directly, without a deep cascade (§3.2–§3.3).

Transformer denoisers. W.A.L.T. (Gupta et al., 2024) replaces the U-Net with a transformer over the latents of a causal 3D encoder that encodes the first frame on its own, so that an image is a one-frame video (§4.1). Its blocks alternate spatial window attention, over the tokens of one latent frame, and attention within non-overlapping spatiotemporal windows, with an identity attention mask for images (§4.2): the same factorization and the same image masking as the video U-Net, in transformer form. The Diffusion Transformer article covers this family, with video in outline.

Inflating an image model without video training. Tune-A-Video fine-tunes a text-to-image model on a single video: the spatial self-attention is replaced by a sparse attention in which each frame queries the first and the previous frame, a temporal attention layer is added, and only some projections are updated; at inference the source video is inverted with DDIM to give the starting noise (Wu et al., 2023, §3.2–§3.3). Text2Video-Zero trains nothing: it builds correlated starting latents by warping one frame’s latent with a global translation and re-noising, and replaces self-attention by cross-frame attention on the first frame to keep an object’s appearance (Khachatryan et al., 2023, §3.3).

Use in Computer Vision

Video diffusion models are the backbone of text-to-video, image-to-video and video prediction systems, which the Video Generation and Video Prediction articles describe, and of video world models for driving and robotics (World Models). Their denoisers are also used as priors outside generation: Stable Video Diffusion’s authors fine-tune theirs into a multi-view generator (Blattmann et al., 2023b), and the 3D Generation article follows that line. Reconstruction guidance conditions an unconditionally trained diffusion model on a known part or a downsampled copy of its output (§3.1, Eq. 8); in our reading the same construction applies to image inpainting and super-resolution.

Limitations

• Short windows. The denoiser models a fixed number of frames, 16 in most of Video Diffusion Models’ experiments, and longer video is assembled by extension or interpolation, whose conditioning (replacement, guidance or masks) can lose coherence over many windows (§3.1; Video Generation, Long Video and Consistency).
• Cost. Every denoising step processes all frames; reconstruction guidance adds a backward pass through the network per step, and the predictor–corrector sampler that helps it doubles the number of network evaluations (§2, §3.1; our reading of the cost).
• Factorization limits. Temporal attention at a fixed spatial position compares the same pixel location across frames, so large motion must be carried by the coarser levels of the U-Net, where a position covers more of the image (our reading; Imagen Video’s description of temporal attention as capturing long-term and convolution as local dependencies is the closest statement in the sources, §2.3).
• Latent decoders. Latent models inherit the flicker of a per-frame autoencoder unless the decoder is made temporal (Blattmann et al., 2023a, §3.1.1).
• Schedules tuned to resolution. The noise distribution has to be shifted as resolution or conditioning strength changes, and the papers set it by experiment (Blattmann et al., 2023b, App. D; Girdhar et al., 2024, §3.2).

Important Models

• Video Diffusion Models (Ho et al., NeurIPS 2022): the video U-Net, joint image–video training and reconstruction guidance.
• Imagen Video (Ho, Chan, Saharia et al., 2022): a pixel-space cascade of seven video U-Nets with temporal attention in the base model, temporal convolution in the super-resolution models, v-prediction and conditioning augmentation.
• Video LDM (Blattmann et al., CVPR 2023): temporal layers blended into a frozen latent image model, and a temporally fine-tuned decoder.
• PYoCo (Ge et al., ICCV 2023): correlated noise priors for fine-tuning image models to video.
• Stable Video Diffusion (Blattmann et al., 2023): a fully fine-tuned latent video U-Net with shifted EDM schedules, frame-rate and motion conditioning, released with code (as of November 2026, Stability AI’s generative-models repository holds the code under an MIT licence and the weights under separate model licences).
• Emu Video (Girdhar et al., ECCV 2024): image-conditioned factorization with zero terminal SNR.
• W.A.L.T. (Gupta et al., ECCV 2024): a window-attention transformer denoiser on a causal video latent.