Video–Language Model

A video–language model is a vision–language model whose visual input is a video. It samples frames, turns each sampled frame into visual tokens with a pretrained image encoder, compresses and orders those tokens, and passes them with a text prompt to a large language model (LLM) that answers questions about the video, describes it or converses about it. Papers call such systems video MLLMs (multimodal large language models; the term of Video-MME and MLVU) or LMMs (large multimodal models; the LLaVA team’s term). The image-side design, a vision encoder, a connector and an LLM, carries over unchanged. What is new is a budget problem: a 10-second clip at 30 frames per second has 300 frames, which at LLaVA-1.5’s 576 tokens per image would be 172,800 tokens, against the 4,096-token sequences LLaVA-NeXT was trained with. Every video–language model therefore decides which frames to read, how many tokens each frame may keep, and how the model learns when each frame occurred. This article covers those decisions, the tasks and benchmarks used to judge them, and the evidence that many benchmarks can be answered from a single frame. Video captioning covers description as a task, and video foundation models cover the video encoders that some of these systems use.

Core Idea

An image–language model spends $n$ tokens on one image. A video of $T$ sampled frames at the same resolution would cost $Tn$ tokens, and the language model’s attention cost grows with that length (the Vision–Language Model article counts the cost per transformer block). Video–language models keep the image encoder and the language model and add a stage between them that reduces $Tn$ to an affordable number $m$, while keeping enough of the order of the frames for questions about when and how often. Three choices define a model: the sampling rule that picks $T$ frames out of $N$, the compression that maps $T$ frames of $n$ tokens to $m$ tokens, and the signal that tells the language model the time of each token.

Building Blocks

Frame sampling

Uniform sampling of a fixed number of frames is the rule of most models in this article. Video-LLaMA’s configuration samples 8 frames, VideoChat’s 8, Video-ChatGPT’s code 100, and LLaVA-NeXT-Video’s demonstration 32 (official repositories at the commits cited). The word “uniform” hides a detail that matters for short events: the implementations place the samples differently. Video-LLaMA’s loader takes the first frame of each of $T$ equal segments, np.arange(0, N, N/T); Video-ChatGPT takes the middle of each of $T$ segments of length $(N-1)/T$; the loader in LLaVA-Video’s documentation spaces $T$ samples from the first frame to the last, np.linspace(0, N - 1, T), so its spacing is $(N-1)/(T-1)$ rather than $N/T$. The example below measures the difference.

Sampling at a fixed rate keeps the time between samples constant instead of their number. Qwen2-VL’s preprocessing samples 2 frames per second, with at least 4 and at most 768 frames (qwen-vl-utils at commit 35ba6e1), and LLaMA-VID extracts frames at 1 frame per second (Li, Wang & Jia, 2024). Flamingo trained on 8 frames at 1 frame per second and read 30 frames at 3 frames per second at inference. A cap on the number of frames turns rate sampling back into count sampling for long videos: by our arithmetic, 768 frames spread over an hour are one frame every 4.7 seconds. The Video-MME authors list sparse sampling as one of three reasons their models lose accuracy on long videos: open-source models fix the number of input frames, for example 8, so the information per second falls as videos get longer, whereas sampling at a fixed rate would keep it constant (Fu et al., 2025, §4.3).

Dense clips of consecutive frames are what pre-LLM video models read. The NExT-QA baselines, for example, sampled 16 clips of 16 consecutive frames per video, with appearance features from a 2D ResNet-101 and motion features from a 3D ResNeXt-101 (Xiao et al., 2021, §4). Temporal modeling covers dense clips, sparse segment sampling and test-time views for those models.

Keyframe or content-aware sampling spends samples where something happens. The ActivityNet-QA authors compared a fixed stepsize with a dynamic stepsize (their terms) that places samples where an external temporal action-proposal model finds actions; the dynamic stepsize improved all their baselines by at least 1 point, and the gap narrowed as the number of sampled units rose from 20 to 60 (Yu et al., 2019, Figure 4). Buch et al. (2022) train a small selector, the atemporal probe described under Limitations, that picks one frame from a sparse random set.

Per-frame encoders and temporal information

The models covered here mostly encode each sampled frame separately with an image encoder pretrained on image–text pairs: CLIP ViT-L/14 taken from LLaVA in Video-ChatGPT (Maaz et al., 2024, §3.1), EVA-CLIP’s ViT-G/14 followed by BLIP-2’s frozen Q-Former in Video-LLaMA (Zhang, Li & Bing, 2023, §2.1.1), and SigLIP SO400M in LLaVA-OneVision (LLaVA-NeXT repository documentation). Features computed this way contain no information about time, as Video-LLaMA’s authors note. Two designs mix frames inside the encoder. In VideoChat’s official code, added layers named gmhra in eva_vit.py update a global token from the features of all frames at several depths of the EVA-CLIP encoder, and the Q-Former then reads the tokens of all frames, with that global token, at once. Qwen2-VL’s image processor sets a temporal patch size of 2, so in our reading of the configuration each patch spans two consecutive frames, and two frames share one grid of tokens (model card configuration).

Time then reaches the language model in one or more of these ways:

• Learned frame position embeddings added before a video-level compressor: Video-LLaMA adds one per frame position, for up to 32 positions in its configuration (max_frame_pos), before its Video Q-Former.
• Sequence order. When frame tokens are concatenated in temporal order, the LLM’s own position encoding orders them; Video-ChatGPT’s temporal tokens are the per-frame averages in frame order. Separator tokens can mark frame boundaries: the LLaVA-NeXT code can append a learned token after every frame or every row of a frame’s grid (mm_newline_position).
• Timestamps as text. The usage example for LLaVA-Video tells the model in words how long the video lasts, how many frames were sampled and at which seconds; Video-LLaMA’s loader can produce a similar message.
• Rotary positions with a time axis. Qwen2-VL’s multimodal rotary position embedding (M-RoPE) splits positions into parts for text, 2D image and 3D video positions, and Qwen2.5-VL aligns the time component with absolute time and samples at dynamic frame rates, which its README says lets the model learn speed and locate moments (official READMEs).

Without one of these signals the frames form an unordered set, and no amount of compression can recover the order; the Temporal Modeling article shows the invariance for pooling and for attention without position.

Token compression

• Spatial pooling per frame. LLaVA-NeXT-Video average-pools each frame’s $24 \times 24$ grid with stride 2 to $12 \times 12 = 144$ tokens (LLaVA-NeXT blog; demonstration settings in the repository). LLaVA-OneVision represents each video frame with 196 tokens (repository documentation); by our reading, that count matches the code’s bilinear option in get_2dPool, which resizes the $27 \times 27$ SigLIP grid to $14 \times 14$, whereas stride-2 average pooling would give $13 \times 13$.
• Pooling over space and over time. Video-ChatGPT averages the frame features over space to get $T$ temporal tokens and over time to get $n$ spatial tokens (the paper writes $N$ for the patches per frame), and feeds the $T + n$ tokens to the LLM through a linear layer (Maaz et al., 2024, Eq. 1–2). With 100 frames of $16 \times 16$ patches that is 356 tokens, the video_token_len in the official code.
• Learned queries. BLIP-2’s Q-Former, described in the BLIP article, returns a fixed number of query outputs per image. Video-LLaMA applies the frozen image Q-Former to every frame (32 outputs each), adds frame position embeddings, and lets a randomly initialized two-layer Video Q-Former with 32 queries attend to all $8 \times 32$ frame outputs; a linear layer maps its 32 outputs into the LLM (paper §2.1.1; model code). VideoChat’s code uses 32 + 64 = 96 queries over the tokens of all 8 frames.
• Text-conditioned tokens. LLaMA-VID represents each frame by two tokens: a context token, the mean over text-derived queries of their attention-weighted visual features, and a content token, the frame’s features pooled down to one token for long videos (Li, Wang & Jia, 2024, §3.2, Eq. 1).
• Slow and fast frames. The LLaVA-NeXT code has an option (add_faster_video) that keeps the stride-$s$ pooled tokens for every $k$-th frame and pools the other frames at stride $2s$, four times fewer tokens, appending a learned token after each frame (llava_arch.py).
• Merging in the patch embedding. Qwen2-VL merges two frames per temporal patch and then each $2 \times 2$ group of neighbouring tokens (model card configuration: temporal_patch_size 2, merge_size 2), so a frame pair of $H \times W$ pixels costs $(H/28)(W/28)$ tokens; the Vision–Language Model article covers the merge for images.

The LLaVA-NeXT team measured the trade-off between tokens per frame and frames on their image model, using the blog’s Video Detailed Description score. Within a 4,096-token limit, 16 frames of $12 \times 12$ tokens scored 2.76 and 4 frames of $24 \times 24$ scored 2.71, both 2,304 tokens; 32 frames of $6 \times 6$ scored 2.73, and 64 frames of $6 \times 6$ scored 2.24. Raising the limit to 8,192 by linear scaling of the rotary position embedding, 32 frames of $12 \times 12$ scored 3.12 against 3.08 for 8 frames of $24 \times 24$ (LLaVA-NeXT blog, configuration table). The authors chose $12 \times 12$ tokens per frame.

How Data Flows

1. Decode and sample. Read $N$ frames and keep $T$ of them by count or by rate; resize them to the encoder’s input size.
2. Encode each frame. A frozen or lightly tuned image encoder produces $n$ patch tokens per frame (576 for a $336 \times 336$ frame with $14 \times 14$ patches).
3. Add time and compress. Add frame positions or separators, then pool, merge or query the $Tn$ tokens down to $m$.
4. Project. A linear layer or MLP maps the $m$ tokens to the LLM’s embedding width.
5. Generate. The LLM reads the visual tokens together with the prompt, which may include timestamps or subtitles, and generates the answer token by token.

Example: the visual token budget of published settings

The script collects the settings of the models above from their code, configurations and blog posts and prints how many visual tokens each sends to the LLM. For Qwen2-VL it ports the frame-count and resizing rules of the official preprocessing (smart_nframes, smart_resize) at two commits of the repository, and applies them to a $1280 \times 720$ video at 30 frames per second. Special start and end tokens are left out. It runs instantly with Python 3.13 and the standard library.

import math

# 1. Fixed settings from official code, configs and blog posts (see text for sources).
rows = [
    # name, frames, tokens per frame before compression, LLM visual tokens, how
    ("Video-LLaMA", 8, 32, 32, "image Q-Former 32/frame -> video Q-Former 32"),
    ("VideoChat", 8, 16 * 16 + 1, 32 + 64, "ViT tokens of all frames -> Q-Former 96"),
    ("Video-ChatGPT", 100, 16 * 16, 100 + 256, "mean over space (100) + over time (256)"),
    ("LLaVA-NeXT-Video", 32, 24 * 24, 32 * 12 * 12, "2x2 average pooling per frame"),
    ("LLaVA-OneVision", 32, 27 * 27, 32 * 14 * 14, "bilinear 27x27 -> 14x14 per frame"),
]
print(f"{'model':17s} {'frames':>6s} {'tok/frame in':>12s} {'unpooled':>9s} {'to LLM':>7s}  how")
for name, f, per, llm, how in rows:
    print(f"{name:17s} {f:6d} {per:12d} {f * per:9d} {llm:7d}  {how}")

# 2. LLaVA-NeXT-Video blog: configurations at three inference budgets (12x12 = stride 2).
print("\nLLaVA-NeXT-Image-7B configs (pooling stride, frames): visual tokens")
for budget, cfgs in [(4096, [(4, 32), (4, 64), (2, 16), (1, 4)]),
                     (8192, [(2, 32), (1, 8)]), (16384, [(2, 64), (1, 16)])]:
    print(f"  max {budget:5d}: " + ", ".join(
        f"s{s} x {f} -> {f * (24 // s) ** 2}" for s, f in cfgs))

# 3. LLaMA-VID: 2 tokens per frame at 1 fps.
for hours in [1, 3]:
    print(f"\nLLaMA-VID, {hours} h at 1 fps x 2 tokens: {hours * 3600 * 2} tokens", end="")
print()

# 4. Qwen2-VL frame and token count, ported from qwen-vl-utils (smart_nframes, smart_resize).
def by(x, f, op):
    return op(x / f) * f

def smart_resize(h, w, min_px, max_px, factor=28):
    hb, wb = max(factor, by(h, factor, round)), max(factor, by(w, factor, round))
    if hb * wb > max_px:
        beta = math.sqrt(h * w / max_px)
        hb, wb = by(h / beta, factor, math.floor), by(w / beta, factor, math.floor)
    elif hb * wb < min_px:
        beta = math.sqrt(min_px / (h * w))
        hb, wb = by(h * beta, factor, math.ceil), by(w * beta, factor, math.ceil)
    return hb, wb

def qwen_video_tokens(seconds, total_tokens, nframes_round, src_fps=30, h=720, w=1280):
    total = seconds * src_fps
    nf = min(max(seconds * 2.0, 4), by(min(768, total), 2, math.floor))   # 2 fps, 4-768 frames
    nf = by(nf, 2, nframes_round)
    min_px = 128 * 28 * 28
    max_px = max(min(768 * 28 * 28, total_tokens * 28 * 28 / nf * 2), int(min_px * 1.05))
    hb, wb = smart_resize(h, w, min_px, max_px)
    per_pair = (hb // 28) * (wb // 28)          # 14-px patches, 2 frames x 2x2 patches merged
    return nf, hb, wb, per_pair, per_pair * nf // 2

for label, budget, rnd in [("commit 35ba6e1 (Qwen2-VL README)", 24576, round),
                           ("commit 6e98a0a (Qwen2.5-VL README)", int(128000 * 0.9), math.floor)]:
    print(f"\nqwen-vl-utils {label}, 1280x720 at 30 fps, total budget {budget} tokens")
    for sec in [10, 60, 600, 3600]:
        nf, hb, wb, pp, tot = qwen_video_tokens(sec, budget, rnd)
        print(f"  {sec:5d} s: {int(nf):3d} frames at {hb}x{wb}, {pp:3d} tokens per frame pair, "
              f"{tot:6d} visual tokens")

model             frames tok/frame in  unpooled  to LLM  how
Video-LLaMA            8           32       256      32  image Q-Former 32/frame -> video Q-Former 32
VideoChat              8          257      2056      96  ViT tokens of all frames -> Q-Former 96
Video-ChatGPT        100          256     25600     356  mean over space (100) + over time (256)
LLaVA-NeXT-Video      32          576     18432    4608  2x2 average pooling per frame
LLaVA-OneVision       32          729     23328    6272  bilinear 27x27 -> 14x14 per frame

LLaVA-NeXT-Image-7B configs (pooling stride, frames): visual tokens
  max  4096: s4 x 32 -> 1152, s4 x 64 -> 2304, s2 x 16 -> 2304, s1 x 4 -> 2304
  max  8192: s2 x 32 -> 4608, s1 x 8 -> 4608
  max 16384: s2 x 64 -> 9216, s1 x 16 -> 9216

LLaMA-VID, 1 h at 1 fps x 2 tokens: 7200 tokens
LLaMA-VID, 3 h at 1 fps x 2 tokens: 21600 tokens

qwen-vl-utils commit 35ba6e1 (Qwen2-VL README), 1280x720 at 30 fps, total budget 24576 tokens
     10 s:  20 frames at 560x1008, 720 tokens per frame pair,   7200 visual tokens
     60 s: 120 frames at 420x728, 390 tokens per frame pair,  23400 visual tokens
    600 s: 768 frames at 224x420, 120 tokens per frame pair,  46080 visual tokens
   3600 s: 768 frames at 224x420, 120 tokens per frame pair,  46080 visual tokens

qwen-vl-utils commit 6e98a0a (Qwen2.5-VL README), 1280x720 at 30 fps, total budget 115200 tokens
     10 s:  20 frames at 560x1008, 720 tokens per frame pair,   7200 visual tokens
     60 s: 120 frames at 560x1008, 720 tokens per frame pair,  43200 visual tokens
    600 s: 768 frames at 336x644, 276 tokens per frame pair, 105984 visual tokens
   3600 s: 768 frames at 336x644, 276 tokens per frame pair, 105984 visual tokens

The fixed-count designs span two orders of magnitude, from 32 tokens for a whole video (Video-LLaMA) to 6,272 (32 OneVision frames). The compression ratios differ just as much: Video-ChatGPT keeps 356 of 25,600 tokens, a factor of 72, while LLaVA-NeXT-Video’s pooling keeps a quarter; the 32-frame OneVision total equals the lower end of the token budgets quoted in the LLaVA article. The LLaVA-NeXT configurations show the trade-off of the previous section as arithmetic: each budget admits either more frames or more tokens per frame. Qwen2-VL’s rule adapts both. For short videos it keeps 2 frames per second at high resolution; under the first commit’s budget the resolution per frame is already cut for a one-minute video, and once the 768-frame cap is reached, at 384 seconds, longer videos are sampled ever more sparsely. At commit 35ba6e1 the 10-minute and 1-hour videos exceed the nominal 24,576 tokens, because the code never lets the pixel limit per frame fall below 1.05 times its minimum of $128 \times 28 \times 28$ pixels; in our reading of the ported code, that floor overrides the total budget once 768 frames would leave fewer pixels per frame. The later default budget of 115,200 tokens keeps 2.3 times as many tokens per frame pair for long videos.

Mathematical Formulation

Let a video have frames $x_1, \dots, x_N$ and let a sampler choose indices $\tau_1 < \dots < \tau_T$. The image encoder $f$ gives $Z_t = f(x_{\tau_t}) \in \mathbb{R}^{n \times d_v}$, a compressor $g$ with time signal $e(\tau_t)$ gives

$$H = g\big(Z_1 + e(\tau_1), \dots, Z_T + e(\tau_T)\big) \in \mathbb{R}^{m \times d},$$

and the language model generates the answer $y$ by $p(y \mid H, q) = \prod_\ell p_\theta(y_\ell \mid y_{<\ell}, H, q)$ for a question $q$, trained with the token-level negative log-likelihood of the Vision–Language Model article. The designs above differ in $m$:

$$m_{\text{concat}} = Tn, \qquad m_{\text{pool}, s} = \frac{Tn}{s^2}, \qquad m_{\text{Video-ChatGPT}} = T + n, \qquad m_{\text{Video Q-Former}} = k, \qquad m_{\text{LLaMA-VID}} = 2T,$$

with $s$ the spatial pooling stride and $k$ the number of video-level queries.

What a uniform sampler can see. Our derivation: let the samples be spaced $\Delta = N/T$ frames apart, and let an event last $d$ frames, starting at a position uniformly distributed relative to the sampling grid. The event contains a sample with probability

$$p(d, \Delta) = \min\!\left(1, \frac{d}{\Delta}\right).$$

A question about the order of two such events, answered by guessing when either is missed, is answered correctly with probability $p^2 + (1 - p^2)/2$ if the events are independent. A question about how many events occurred needs every event and every gap between events to contain a sample; when events and gaps last at least $d$ frames, $\Delta \le d$ guarantees this. A question about something visible throughout the video needs $T = 1$.

Why It Works

In our reading of the sources, the architecture works for the same reason image VLMs do: a contrastively pretrained image encoder already recognizes objects, scenes and actions frame by frame, and a large language model can combine what it is shown with what it knows. Much of what people ask about a video, which objects, where, what kind of activity, is visible in most of its frames, so a few sampled frames carry the answer, and a model that began as an image model can transfer to video. The LLaVA-NeXT team reported that their image-only model, given frames as a grid of images, outperformed open-source models trained on video in zero-shot video benchmarks (LLaVA-NeXT blog). The sampling formula marks where this stops working: answers that depend on short events, or on how often something happens, are only as good as the sampling rate.

Example: when sampled frames miss the event

The script generates synthetic videos of 300 frames of $32 \times 32$ pixels (10 seconds at 30 frames per second). Each video contains a square of one of three colours that is visible throughout; a red and a blue square that each appear once, for $d$ frames, at random non-overlapping times; and a green square that flashes 1 to 4 times, each flash lasting $d$ frames, with gaps of at least $d$ frames. Three questions follow: the colour of the persistent square (any single frame suffices), whether red appeared before blue, and how many times green flashed. A rule-based reader, with no learning and no language model, sees only the $T$ sampled frames in order: it detects colours by thresholds on the pixel values, compares the first sampled frames in which red and blue appear (guessing “red first” if either is never seen), and counts the runs of consecutive sampled frames that contain green. The reader is a stand-in for a model with perfect per-frame recognition, so its errors come from sampling alone. The three sampling placements of the official loaders are run side by side, because they are a modelling choice the sources leave open. It runs in about 25 seconds with NumPy 2.3.5.

Before the first run, reading all 300 frames recovered all three answers in 100 of 100 test videos for each $d$, with no colour detected in any frame where it had not been painted, so the labels are present in the pixels; with one frame, order and count were near chance. In the first run, the column of events seen counted the videos in which at least one of the two order events was seen, because adding two NumPy booleans returns their logical or; the column was changed to count events, and the accuracy columns were unchanged.

import numpy as np

N, S, NOISE = 300, 32, 0.05            # frames ("10 s at 30 fps"), frame size, pixel noise
DURATIONS = [4, 12, 36]                # event length d in frames
T_LIST = [1, 2, 4, 8, 16, 32, 64, 300] # number of sampled frames
N_VIDEOS = 600                         # per duration
COLORS = {"red": (1, 0, 0), "green": (0, 1, 0), "blue": (0, 0, 1),
          "yellow": (1, 1, 0), "magenta": (1, 0, 1), "cyan": (0, 1, 1)}
STATIC = ["yellow", "magenta", "cyan"]
BOXES = {"static": (1, 9, 1, 9), "red": (12, 17, 4, 9),
         "blue": (12, 17, 20, 25), "green": (24, 29, 12, 17)}   # rows, cols


def place(k, d, rng):
    """Start frames of k events of length d with gaps >= d, uniform over valid layouts."""
    slack = N - k * d - (k - 1) * d
    bars = np.sort(rng.choice(slack + k, size=k, replace=False))
    extra = np.diff(np.concatenate([[-1], bars])) - 1         # slack before each event
    starts, t = [], 0
    for e in extra:
        t += e
        starts.append(t)
        t += 2 * d                                            # event, then minimum gap
    return starts


def make_video(i, d, rng):
    """One synthetic video and its three answers."""
    static = STATIC[i % 3]
    k = 1 + i % 4                                             # number of green flashes
    red_first = (i // 4) % 2 == 0
    v = 0.5 + NOISE * rng.standard_normal((N, S, S, 3))
    def paint(name, color, t0, t1):
        r0, r1, c0, c1 = BOXES[name]
        v[t0:t1, r0:r1, c0:c1] = np.array(COLORS[color]) + NOISE * rng.standard_normal(
            (t1 - t0, r1 - r0, c1 - c0, 3))
    paint("static", static, 0, N)
    first, second = place(2, d, rng)
    paint("red", "red", *((first, first + d) if red_first else (second, second + d)))
    paint("blue", "blue", *((second, second + d) if red_first else (first, first + d)))
    for t0 in place(k, d, rng):
        paint("green", "green", t0, t0 + d)
    return v, (STATIC.index(static), int(red_first), k)


def detect(v):
    """Pixels of each colour in every frame, by colour thresholds only (no positions)."""
    hi, lo = v > 0.7, v < 0.3
    counts = {}
    for name, (r, g, b) in COLORS.items():
        m = np.ones(v.shape[:3], bool)
        for ch, on in enumerate((r, g, b)):
            m &= hi[..., ch] if on else lo[..., ch]
        counts[name] = m.sum(axis=(1, 2))
    return counts


def sample(T, how):
    """Frame indices of T uniformly sampled frames, three conventions from official code."""
    if how == "start":                                        # Video-LLaMA load_video
        idx = np.arange(0, N, N / T).astype(int)
    elif how == "middle":                                     # Video-ChatGPT get_seq_frames
        seg = (N - 1) / T
        idx = np.array([(int(np.round(seg * i)) + int(np.round(seg * (i + 1)))) // 2
                        for i in range(T)])
    else:                                                     # LLaVA-Video load_video
        idx = np.linspace(0, N - 1, T, dtype=int)
    assert len(idx) == T
    return idx


def read(counts, idx):
    """Rule-based reader that sees only the sampled frames, in order."""
    seen = {c: counts[c][idx] >= 6 for c in COLORS}
    static = int(np.argmax([counts[c][idx].sum() for c in STATIC]))
    if seen["red"].any() and seen["blue"].any():
        red_first = int(np.argmax(seen["red"]) < np.argmax(seen["blue"]))
        both = True
    else:
        red_first, both = 1, False                           # fixed guess
    g = seen["green"].astype(int)
    runs = int(g[0] + np.sum(np.diff(g) == 1))
    return (static, red_first, int(np.clip(runs, 1, 4))), both


# ---- experiment ----
rng = np.random.default_rng(0)
HOWS = ["start", "middle", "linspace"]
print("columns: T, stride N/T, predicted catch min(1, d/stride), measured catch (middle),")
print("         accuracy static | order start/middle/linspace | count start/middle/linspace")
for d in DURATIONS:
    videos = []
    for i in range(N_VIDEOS):
        v, labels = make_video(i, d, rng)
        videos.append((detect(v), labels))
    print(f"\nd = {d} frames")
    for T in T_LIST:
        acc = {h: np.zeros(3) for h in HOWS}
        caught = 0
        for counts, labels in videos:
            for h in HOWS:
                ans, both = read(counts, sample(T, h))
                acc[h] += np.equal(ans, labels)
                if h == "middle":
                    idx = sample(T, h)
                    caught += int((counts["red"][idx] >= 6).any()) + int((counts["blue"][idx] >= 6).any())
        acc = {h: a / N_VIDEOS for h, a in acc.items()}
        stride = N / T
        print(f"T={T:3d}  stride {stride:5.1f}  pred {min(1, d / stride):.2f}  "
              f"seen {caught / (2 * N_VIDEOS):.2f}  static {acc['middle'][0]:.2f} | order "
              + "/".join(f"{acc[h][1]:.2f}" for h in HOWS) + " | count "
              + "/".join(f"{acc[h][2]:.2f}" for h in HOWS))

columns: T, stride N/T, predicted catch min(1, d/stride), measured catch (middle),
         accuracy static | order start/middle/linspace | count start/middle/linspace

d = 4 frames
T=  1  stride 300.0  pred 0.01  seen 0.02  static 1.00 | order 0.50/0.50/0.50 | count 0.25/0.25/0.25
T=  2  stride 150.0  pred 0.03  seen 0.02  static 1.00 | order 0.50/0.50/0.50 | count 0.25/0.25/0.25
T=  4  stride  75.0  pred 0.05  seen 0.05  static 1.00 | order 0.50/0.50/0.50 | count 0.25/0.25/0.25
T=  8  stride  37.5  pred 0.11  seen 0.11  static 1.00 | order 0.51/0.51/0.50 | count 0.25/0.25/0.25
T= 16  stride  18.8  pred 0.21  seen 0.23  static 1.00 | order 0.53/0.52/0.52 | count 0.26/0.26/0.26
T= 32  stride   9.4  pred 0.43  seen 0.42  static 1.00 | order 0.61/0.59/0.58 | count 0.32/0.32/0.31
T= 64  stride   4.7  pred 0.85  seen 0.85  static 1.00 | order 0.86/0.86/0.84 | count 0.72/0.73/0.71
T=300  stride   1.0  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00

d = 12 frames
T=  1  stride 300.0  pred 0.04  seen 0.05  static 1.00 | order 0.50/0.50/0.50 | count 0.25/0.25/0.25
T=  2  stride 150.0  pred 0.08  seen 0.09  static 1.00 | order 0.50/0.51/0.50 | count 0.25/0.25/0.25
T=  4  stride  75.0  pred 0.16  seen 0.19  static 1.00 | order 0.51/0.52/0.51 | count 0.26/0.26/0.25
T=  8  stride  37.5  pred 0.32  seen 0.32  static 1.00 | order 0.56/0.56/0.54 | count 0.28/0.27/0.26
T= 16  stride  18.8  pred 0.64  seen 0.64  static 1.00 | order 0.71/0.70/0.69 | count 0.46/0.46/0.40
T= 32  stride   9.4  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00
T= 64  stride   4.7  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00
T=300  stride   1.0  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00

d = 36 frames
T=  1  stride 300.0  pred 0.12  seen 0.12  static 1.00 | order 0.50/0.50/0.50 | count 0.25/0.25/0.25
T=  2  stride 150.0  pred 0.24  seen 0.25  static 1.00 | order 0.50/0.54/0.50 | count 0.25/0.25/0.25
T=  4  stride  75.0  pred 0.48  seen 0.57  static 1.00 | order 0.58/0.67/0.53 | count 0.28/0.30/0.25
T=  8  stride  37.5  pred 0.96  seen 0.97  static 1.00 | order 0.95/0.96/0.84 | count 0.86/0.93/0.56
T= 16  stride  18.8  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00
T= 32  stride   9.4  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00
T= 64  stride   4.7  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00
T=300  stride   1.0  pred 1.00  seen 1.00  static 1.00 | order 1.00/1.00/1.00 | count 1.00/1.00/1.00

The persistent colour is answered perfectly from a single frame, while order and count stay at chance (0.50 and 0.25) until the samples come close to the event length. The fraction of events seen follows $\min(1, d/\Delta)$ within 0.03 in all but one setting ($d = 36$, $T = 4$: 0.57 against 0.48), and order accuracy follows $p^2 + (1-p^2)/2$: at $d = 4$ and $T = 64$, $p = 0.85$ predicts 0.86, as printed. Counting needs more: it reaches 1.00 exactly where the spacing first drops to $d$ or below, at $T = 16$ for $d = 36$, $T = 32$ for $d = 12$, and only with all 300 frames for $d = 4$, where 64 frames still give 0.72. In time units, a 4-frame event lasts 0.13 seconds; 8 frames over these 10 seconds (Video-LLaMA’s count) see it 11% of the time, and 2 frames per second, Qwen2-VL’s rate, are a spacing of 15 frames. The placement convention, which we expected to matter little, changes count accuracy from 0.93 (middle) to 0.56 (linspace) at $d = 36$ and $T = 8$. By our reading of the indices, linspace spreads 8 samples over 299 frames, 42.7 frames apart, wider than the 36-frame flashes, while the start and middle conventions space their samples 37 or 38 frames apart, so they miss only a flash that falls almost exactly between two samples; the start convention also leaves the last 37 frames unsampled, and the middle convention only about 18 at each end. Near the threshold where sampling just suffices, which of the three conventions a loader uses changes the answer.

Training Considerations

The models covered here are trained in the stages used for image VLMs: alignment on captioned data, then instruction tuning.

• Video-LLaMA froze the image encoder, the image Q-Former and the LLM, and trained the frame position embeddings, the Video Q-Former and the projection. Stage 1 generated captions for WebVid-2M videos and the CC595k image subset (an image is a one-frame video); stage 2 tuned on image instructions from MiniGPT-4 and LLaVA and video instructions from VideoChat. The authors note that many WebVid captions describe only part of their video (Zhang, Li & Bing, 2023, §2.2.1). The repository README names Webvid-2M in its overview but describes the pretraining data of its checkpoints as WebVid with 2.5M video–caption pairs, against the paper’s WebVid-2M.
• Video-ChatGPT started from LLaVA and trained only the linear layer, for 3 epochs at learning rate $2 \times 10^{-5}$ and batch size 32, about 3 hours on 8 A100 40GB GPUs. Its 100,000 video instructions came from human annotators enriching ActivityNet captions (30%) and from a semi-automatic pipeline (70%) in which BLIP-2 and GRiT captioned keyframes, Tag2Text tags filtered those captions, and GPT-3.5 merged the results and wrote question–answer pairs; the combination had the best average score in their ablation, although the automatic data alone scored slightly higher on consistency (Maaz et al., 2024, §4–5, Table 3).
• LLaVA-NeXT-Video began from an image model that, given frames as an image grid, already transferred to video. Fine-tuning on video helped only when image and video samples were mixed within batches (3.32 description score, against 3.02 for continued tuning on video alone and 3.00 for batches of one type), and direct preference optimization with AI feedback improved the scores further (LLaVA-NeXT blog). The rotary scaling that extended inference to 8,192 tokens let the model read up to 56 frames of 144 tokens.
• LLaVA-OneVision adds a stage of 4M high-quality samples between alignment and instruction tuning, then tunes on 3.2M single-image samples and finally 1.6M single-image, multi-image and video samples; the maximum visual token counts of the three scenarios are designed to be similar so that skills transfer between them (repository documentation and blog; the LLaVA article covers the paper).
• LLaMA-VID adds a long-video stage of 15K question–answer pairs, 9K built from more than 400 MovieNet movies with their scripts and 6K sampled from LongLoRA’s data for token expanding, and reports inputs of 64K tokens covering more than three hours of video with subtitles (Li, Wang & Jia, 2024, §3.3).

Video-ChatGPT’s and LLaMA-VID’s instruction data were written by language models from captions, keyframe descriptions, synopses or scripts; by our reasoning, such answers can be only as temporal as the text they were written from.

Variants

• Audio. Video-LLaMA adds an audio branch: ImageBind encodes 2-second audio segments, an Audio Q-Former with learnable position embeddings compresses them, and a linear layer maps the result into the LLM. Because audio–text data were scarce, the branch was trained on visual–text data through ImageBind’s shared embedding space, yet answered questions about sound at inference (Zhang, Li & Bing, 2023, §2.1.2, §2.2.2).
• Subtitles and speech. Video-MME and LongVideoBench supply subtitles; LongVideoBench interleaves each subtitle between the frames before and after its middle timestamp (Wu et al., 2024, §3.2). Adding subtitles raised Gemini 1.5 Pro’s overall Video-MME accuracy from 75.0% to 81.3%, by 2.8 points on short videos and 10.1 on long ones (Fu et al., 2025, Table 5).
• Long context instead of compression. Rather than shrink each frame, some systems extend the language model’s context: LLaVA-NeXT-Video by rotary scaling, LLaMA-VID with two tokens per frame and a 64K context, and Qwen2.5-VL, whose README extends dynamic resolution to time and claims understanding of videos lasting hours (sources above).

Use in Computer Vision

Video question answering

Video question answering pairs a video with a question and expects an answer, either free text (open-ended) or one of several options (multiple choice). It inherits the scoring and the language priors of image visual question answering.

Benchmark	Videos	Questions	Format	Source of questions
MSVD-QA	1,970	50,505	open-ended	generated from captions
MSRVTT-QA	10,000	243,690	open-ended	generated from captions
ActivityNet-QA	5,800 (about 3 min each)	58,000	open-ended	crowd workers
NExT-QA	5,440 (44 s on average)	52,044 (47,692 multiple choice)	5 options or open-ended	trained annotators
EgoSchema	3-minute clips, over 250 h	5,063	5 options	LLM from Ego4D narrations, then human curation

Counts for MSVD-QA, MSRVTT-QA and ActivityNet-QA as listed in NExT-QA’s Table 2; the others from their own papers. The MSVD-QA and MSRVTT-QA paper (Xu et al., 2017) was not read; NExT-QA and LongVideoBench describe these question sets as generated automatically. Test-set sizes differ between sources: MSVD-QA’s test set has 504 videos in Video-MME’s Table 1 and 520 in LongVideoBench’s, both with 13,157 questions.

• ActivityNet-QA asks three template types per video, about motion, spatial relations in one frame and temporal relations, plus free questions; its baselines scored lowest on temporal relations, 2.5% accuracy at best (Yu et al., 2019, Table 3).
• NExT-QA groups questions into causal (48%), temporal (29%) and descriptive (23%) and scores multiple choice by accuracy and open-ended answers by WUPS; humans answered 88.4% of validation multiple-choice questions correctly, and a BERT model fine-tuned on question and options alone 43.8% (Xiao et al., 2021, Table 3).
• EgoSchema introduced the temporal certificate, the shortest set of subclips a human verifier must watch to be convinced of an answer, and estimated a median certificate of about 100 seconds for its questions. Zero-shot, the best model it tested scored 32% at 90 frames (InternVideo), against 20% chance and 75.1% for humans without constraints; humans watching 1 frame per second reached 67.2%, and mPLUG-Owl scored 30.2% with 5 frames but 20.0% with 30 (Mangalam, Akshulakov & Malik, 2023; the text calls these results Tables 6 and 7, their captions Figures 6 and 7). The paper is inconsistent about its margin over other datasets: its abstract and §4.1 say 5.7 times longer than the next dataset, while Figure 1 and the introduction say 5 times, and its human “no constraint” accuracy is 75.1% in the table and 75.0% in the text.

The LLM-based models changed the scoring. Video-ChatGPT evaluated open-ended answers zero-shot by asking GPT-3.5 whether each prediction was correct and to rate it from 1 to 5:

Model (7B)	MSVD-QA	MSRVTT-QA	TGIF-QA	ActivityNet-QA
Video-LLaMA	51.6 / 2.5	29.6 / 1.8	—	12.4 / 1.1
VideoChat	56.3 / 2.8	45.0 / 2.5	34.4 / 2.3	26.5 / 2.2
Video-ChatGPT	64.9 / 3.3	49.3 / 2.8	51.4 / 3.0	35.2 / 2.8

GPT-judged accuracy (%) / score, from Maaz et al. (2024, Table 2). These numbers are not comparable with exact-match accuracies such as BLIP’s zero-shot results in the BLIP article. The LLaVA-NeXT blog and LLaMA-VID’s results table list Video-ChatGPT on ActivityNet-QA as 35.2/2.7, against 2.8 in the ACL paper’s table; the blog lists LLaVA-NeXT-Video-DPO (7B) at 60.2/3.5.

Text–video retrieval

Text-to-video retrieval ranks a collection of videos by their match to a sentence, scored by recall at $K$ as in image–text retrieval. On MSR-VTT, whose 10,000 clips each have 20 captions (Xu et al., 2016), a common protocol trains on 7,000 train and validation videos and tests on a 1,000-video test set; DiDeMo and ActivityNet Captions are used for paragraph-to-video retrieval, with all of a video’s captions concatenated (Lei, Berg & Bansal, 2023, §4.1). In our reading of the sources cited here, retrieval is mostly addressed by dual encoders that embed video and text separately, such as the contrastive models of the Video Foundation Models article, since a collection can then be embedded once; the generative models above report question answering and description instead, and retrieval enters this article mainly through the bias studies below.

Long-video understanding

Benchmark	Videos	Questions	Duration	Notes
Video-MME	900	2,700, 4 options	11 s to 1 h; short < 2 min, medium 4–15 min, long 30–60 min	subtitles and audio; questions filtered so that text alone fails
LongVideoBench	3,763	6,678, 17 categories	8 s to 1 h	referring reasoning: a query points to a moment, a question asks about it; subtitles interleaved
MLVU	1,730	3,102, 9 tasks	3 min to 2 h, about 15 min on average	needle questions, action order and count, summaries

Video-MME’s annotators watched each whole video and wrote three questions; questions that Gemini 1.5 Pro answered from text alone were revised, leaving it under 15% accuracy without the video. Median certificate lengths were 26.0, 164.7 and 890.7 seconds for the short, medium and long subsets. Without subtitles, Gemini 1.5 Pro scored 75.0% overall (81.7% short, 67.4% long), GPT-4o 71.9%, and the best open-source model, VILA-1.5 with a 34B LLM, 59.0%; LLaVA-NeXT-Video-34B scored 52.0% (Fu et al., 2025, Tables 3–4). The Qwen READMEs and the Qwen2-VL-7B model card at the cited revisions report Video-MME accuracy without and with subtitles of 63.3/69.0 for Qwen2-VL-7B, 71.2/77.8 for Qwen2-VL-72B and 73.3/79.1 for Qwen2.5-VL-72B, and EgoSchema test accuracy of 66.7 and 77.9 for the two Qwen2-VL sizes; these are the developers’ own evaluations, and the Gemini 1.5 Pro and GPT-4o entries in the Qwen2-VL table match Video-MME’s Table 4.

LongVideoBench was built so that more frames should help. On its validation set, GPT-4o rose from 41.7% with 1 frame to 66.7% with 256, and Gemini 1.5 Pro from 38.6% to 64.0%, while open-source models did not improve beyond 16 frames; Idefics2 fell from 49.7% with 16 frames to 30.9% with 64 (Wu et al., 2024, Table 5). In MLVU’s test set, GPT-4o sampling at 0.5 frames per second led the multiple-choice average with 54.5%, LLaVA-OneVision with 32 frames trailed by 2.8 points, and GPT-4o answered 42.9% of the needle questions, whose answer lies in a short clip inserted into a long video (Zhou et al., 2025, Table 2). Video-MME, LongVideoBench and MLVU all report accuracy falling as videos get longer.

Limitations

The single-frame bias

Several studies argue that popular benchmarks reward appearance more than time, the video–language form of the static bias described in video understanding.

• Lei, Berg and Bansal (2023) trained SINGULARITY on one randomly chosen frame per video, in pretraining and fine-tuning, and at inference concatenated the features of several uniformly sampled frames (early fusion), which beat averaging per-frame predictions (late fusion). Pretrained on 17M images and videos, it reached 41.5 R@1 on MSR-VTT text-to-video retrieval with one training frame, against 42.0 for CLIP4Clip with 12 training frames and CLIP’s 400M pretraining pairs, and 43.5% on MSRVTT-QA (Tables 1–2). The authors call this a static appearance bias. On two retrieval tasks they built from Something-Something v2, where a template such as “Throwing [something] in the air and catching it” leaves only the action, the single-frame model pretrained on 5M scored 42.0 R@1 on template retrieval against 77.0 for its 4-frame variant with a two-layer temporal encoder (Table 3).
• Buch et al. (2022) proposed the atemporal probe (ATP): a small transformer without positional encodings that sees an unordered set of frozen CLIP frame embeddings and must pass exactly one of them to the task. On MSR-VTT multiple choice, ATP choosing 1 of 16 frames scored 93.2%, above the 92.1% of VideoCLIP; on NExT-QA it scored 49.2%, against 49.7% for the temporal reasoning model HGA, with descriptive, temporal and causal accuracies of 58.9, 46.7 and 48.3 (Tables 1 and 3; the table does not name the split). Using the confidence and agreement of an ensemble of ATP models, the authors selected a harder subset of causal and temporal questions on which ATP scored 20.2% and HGA 44.1%.
• Later benchmarks were designed around this finding. LongVideoBench’s authors write that existing benchmarks show strong single-frame bias, with results that do not improve when models read more frames (Wu et al., 2024, §1); MLVU’s authors note that many earlier questions concern a single frame (Zhou et al., 2025, §1). Video-MME found image models given multiple frames, Qwen-VL-Max and InternVL-Chat-V1.5, close to the video model LLaVA-NeXT-Video, which its authors read as image understanding being the basis of video understanding (Fu et al., 2025, §4.2).

The example above shows both sides: a question about a persistent object is answered from one frame, and a question about short events is answered reliably only when the samples are spaced no wider than the events.

Other limitations

• Short events and counting. Sparse sampling cannot see events shorter than the spacing between samples. Video-MME’s authors name counting as a joint bottleneck of the models they tested (Fu et al., 2025, Figure 3), MLVU’s models mostly failed its action order and count tasks, and Qwen2-VL’s model card lists insufficient counting accuracy as a known limitation.
• Long videos. Accuracy falls with duration on every long-video benchmark above, and Video-ChatGPT’s authors report difficulty with subtle temporal relationships in videos longer than two minutes (Maaz et al., 2024, §7).
• Hallucination. Video-LLaMA’s authors list hallucination inherited from the frozen LLM among its limitations, with limited perception and long videos (Zhang, Li & Bing, 2023, §6); multimodal hallucination covers the problem for images.
• Evaluation by language models. GPT-judged accuracy and 1–5 scores depend on the judging model and its prompt; Video-ChatGPT repeated its benchmark with Vicuna-1.5 (13B) as the judge and found similar trends, with different numbers (Maaz et al., 2024, Table 4).
• Cost. Each visual token passes through every LLM layer, so the budget, not the encoder, limits frames and resolution, as the token arithmetic shows.

Important Models

Model	Source	Frames	Vision side	Visual tokens to the LLM	Language model
Video-LLaMA	EMNLP 2023 demo; repository	8, segment starts	EVA-CLIP ViT-G/14 + frozen BLIP-2 Q-Former, frame positions, Video Q-Former; ImageBind audio branch	32	Vicuna or LLaMA-2-Chat, frozen
VideoChat	repository (paper not read)	8	EVA-CLIP ViT-g with added gmhra layers, Q-Former	96	Vicuna-7B or 13B
Video-ChatGPT	ACL 2024; repository	100, segment middles	CLIP ViT-L/14 from LLaVA; mean over space and over time	356	Vicuna-v1.1 7B from LLaVA, frozen
LLaVA-NeXT-Video	blog, April 2024	32 (demo)	LLaVA-NeXT encoder, $2 \times 2$ average pooling	144 per frame	7B and 34B
LLaVA-OneVision	blog and repository, August 2024	e.g. 32	SigLIP SO400M, bilinear to $14 \times 14$	196 per frame	Qwen2 0.5B, 7B, 72B
LLaMA-VID	ECCV 2024	1 fps	context token + content token per frame	2 per frame	Vicuna-7B, 13B
Qwen2-VL	README and model card, 2024	2 fps, 4–768	dynamic-resolution ViT, 2 frames per temporal patch, $2 \times 2$ merge, M-RoPE	variable	models of 2B, 7B, 72B
Qwen2.5-VL	README, 2025	dynamic fps	window attention in the ViT, absolute-time M-RoPE	variable	models of 3B, 7B, 32B, 72B

Frame counts are those of the official configurations or demonstrations cited in the text; models are often evaluated with other counts. The LLMs of LLaVA-NeXT-Video are given by size, because the blog’s model card describes the image model; LLaMA-VID’s language model is from its paper. As of November 2026, the Video-LLaMA README describes Video-LLaMA-2 with LLaMA-2-Chat, the Video-ChatGPT and Ask-Anything repositories remain online, and the Qwen2-VL repository has been renamed QwenLM/Qwen3-VL.