Video Captioning

Video captioning, also called video description or video-to-text, is the task of generating natural language that describes a video. In its basic form a model reads a short trimmed clip and writes one sentence, such as “a man is cutting a bottle”; in dense video captioning it finds every event in a long untrimmed video and writes a timestamped sentence for each. The task inherits the encoder–decoder recipe and the n-gram metrics of image captioning, and adds what video understanding adds to image recognition: motion, temporal order, many frames per example and events that start and end.

Problem Definition

Clip captioning. Given frames $x_1, \dots, x_n$ of a trimmed clip, produce a word sequence $y_1, \dots, y_m$ that people would accept as describing it. As in image captioning, neural models factorize the sentence probability word by word and are trained with the cross-entropy of each reference word given the reference prefix:

$$p(y_1, \dots, y_m \mid x_1, \dots, x_n) = \prod_{t=1}^{m} p(y_t \mid y_{<t}, x_1, \dots, x_n),$$

which is how Venugopalan et al. (2015, ICCV) state the problem for S2VT. The input and output lengths differ, and there are usually many more frames than words.

Dense video captioning. Given an untrimmed video, produce a set of events $\{(t^{s}_j, t^{e}_j, S_j)\}_{j=1}^{N}$, each a start time, an end time and a sentence, where the number of events $N$ is unknown. Krishna et al. (2017) introduced the task as detecting and describing events together, with events that may overlap; PDVC states it as predicting such a set, including its size (Wang et al., 2021). Localization alone is temporal action localization, with free-form sentences in place of class labels.

Input and Output

The input is a sequence of frames, almost never all of them. S2VT sampled every tenth frame of MSVD clips (Venugopalan et al., 2015, ICCV); Yao et al. (2015) took 26 equally spaced frames from the first 240 of each clip; SwinBERT samples $T$ frames uniformly, up to 64 (Lin et al., 2022); Vid2Seq samples at 1 frame per second and pads or subsamples to 100 frames (Yang et al., 2023). SwinBERT’s authors call learning from features extracted offline the canonical approach (Lin et al., 2022): an ImageNet-trained 2D CNN per frame, optical flow images, or clip features from a 3D convolutional network such as C3D. Some benchmarks keep the audio track (MSR-VTT) or subtitles (TVC), and Vid2Seq also reads the transcribed speech.

The output of clip captioning is a sentence over a fixed vocabulary, ended by an end-of-sentence token. The output of dense captioning is a list of segments with one sentence each, which can be read in start-time order as a paragraph.

Variants

• Clip captioning on trimmed clips of up to about 30 seconds (MSVD, MSR-VTT, VATEX).
• Dense video captioning: localize and describe all events in minutes-long videos (ActivityNet Captions, YouCook2).
• Paragraph captioning: write a coherent paragraph for an untrimmed video, often given the event segments; PDVC calls it a simplified version of dense captioning that needs no temporal locations (Wang et al., 2021).
• Movie description: clips of feature films described by audio description for blind viewers or by scripts, as in MPII-MD and M-VAD, with about one sentence per clip (Venugopalan et al., 2015, ICCV).
• Multilingual captioning: VATEX pairs every clip with English and Chinese captions (Wang et al., 2019).
• Captioning with speech or subtitles: TVC captions may describe what the subtitles say as well as what is seen (Lin et al., 2022); Vid2Seq uses transcribed speech as input and, in pretraining, as a target (Yang et al., 2023).

Challenges

• Temporal structure. Yao et al. (2015) point out that a single vector averaged over frames cannot tell the order in which two objects appear. Captions such as “she cracks the egg then whisks it” depend on order; many benchmark captions do not.
• Static bias. On MSVD, S2VT trained on randomly reordered frames reached METEOR 28.2 against 29.2 for the same model on ordered frames (Table 2); the authors call the drop considerable, but it is one point. On MSR-VTT, soft attention over VGG-19 frame features scored BLEU-4 37.6 and METEOR 28.9 against 37.3 and 28.7 for mean pooling (Xu et al., 2016, Table 5). By our reading, these benchmarks reward recognizing objects, scenes and actions far more than order.
• Cost. A 10-second clip at 30 frames per second has 300 frames. SwinBERT’s authors describe end-to-end training of feature extractors on densely sampled frames as computationally intensive or infeasible, and train end to end with an efficient video transformer (Lin et al., 2022).
• Event boundaries. Annotators of ActivityNet Captions agreed on temporal extent only partly: each sentence overlapped the best-matching combination of another annotator’s sentences at 70.2% temporal IoU on average (Krishna et al., 2017).
• Many valid descriptions. MSVD has about 41 English descriptions per clip and MSR-VTT 20, while dense benchmarks have one sentence per event, so references cover a small part of what could be said.
• Noisy supervision at scale. Narration is not always about what is visible and is often temporally misaligned with it (Yang et al., 2023).

Approaches

Templates

Early video description systems recognized a subject, verb, object and sometimes scene with separate classifiers, combined their confidences with a language model in a graphical model, and filled a sentence template; S2VT’s authors describe this two-stage pipeline and compare against one such factor graph model (Venugopalan et al., 2015, ICCV). The approach separates content selection from sentence generation but needs a fixed set of objects and actions.

Mean-pooled frame features

Venugopalan et al. (2015, NAACL) applied an image captioner to video: AlexNet fc7 features of each frame are averaged over the whole clip, and the mean vector is fed at every step to an LSTM that generates the sentence. Because MSVD has only 1,970 clips, they pretrained the captioner on Flickr30k and COCO images and fine-tuned it on video; on the MSVD test set, BLEU-4 rose from 31.19 to 33.29 and METEOR from 26.87 to 29.07 with COCO pretraining (Table 3). Averaging makes the representation independent of frame order, which temporal modeling treats in general.

Sequence to sequence: S2VT

S2VT (Venugopalan et al., 2015, ICCV) reads the frames in order and then writes, following sequence-to-sequence machine translation. A stack of two LSTMs with 1,000 hidden units runs over the whole sequence: during the encoding steps the top LSTM receives a 500-dimensional embedding of each frame’s CNN feature while the second receives the top LSTM’s output with zero padding in place of words, and after the last frame a beginning-of-sentence token starts decoding, with zeros as visual input. One network, with the same weights, both encodes and decodes, and the loss is computed only on the words. Inputs were VGG-16 or AlexNet RGB features or AlexNet features of optical flow images, and the final model combined the word probabilities of an RGB and a flow model in a weighted sum whose weight was tuned on the validation set. The authors describe it, to their knowledge, as the first video description model built on a general sequence-to-sequence model.

Model on MSVD (test)	METEOR
Factor graph model (template)	23.9
Mean pool, AlexNet	26.9
Mean pool, AlexNet, COCO-pretrained	29.1
Temporal attention, GoogLeNet + 3D-CNN	29.6
S2VT, flow (AlexNet)	24.3
S2VT, RGB (VGG), random frame order	28.2
S2VT, RGB (VGG)	29.2
S2VT, RGB (VGG) + flow (AlexNet)	29.8

Rows from S2VT’s Table 2, which collects its own results and those of earlier papers; METEOR 1.5 from the COCO evaluation code. On the movie datasets, with one reference per clip, S2VT scored METEOR 7.1 on MPII-MD and 6.7 on M-VAD (Tables 4 and 5). In Table 3, 42.9% of its MSVD captions were identical to a training sentence and another 38.3% were one word edit away.

Temporal attention

Yao et al. (2015) let the decoder choose frames. The encoder gives a set of feature vectors $V = \{v_1, \dots, v_n\}$, one per sampled frame, and at each word $t$ the decoder’s visual input is a weighted sum computed from its previous state $h_{t-1}$, the soft attention of machine translation and of Show, Attend and Tell applied to time:

$$e^{(t)}_i = w^\top \tanh(W_a h_{t-1} + U_a v_i + b_a), \qquad \alpha^{(t)}_i = \frac{\exp e^{(t)}_i}{\sum_{j=1}^{n} \exp e^{(t)}_j}, \qquad \varphi_t(V) = \sum_{i=1}^{n} \alpha^{(t)}_i v_i .$$

For local motion they concatenated GoogLeNet frame features with features from a 3D CNN that runs on a grid of HoG, HoF and MBH descriptors and was pretrained for action recognition. On MSVD, which the paper calls Youtube2Text, the full model scored BLEU 41.92, METEOR 29.60 and CIDEr 51.67, against 38.69, 28.68 and 44.78 for the model with neither attention nor 3D-CNN features (Table 1). The text describes temporal structure as generally beneficial; in the table, adding the 3D-CNN features alone lowered METEOR from 28.68 to 28.32, while both additions together improved every metric on both datasets.

The paper argues that attending to subsets of frames in sequence lets the model exploit temporal ordering. By our reading of the equation, the scores $e^{(t)}_i$ contain no time index, so $\varphi_t(V)$ is unchanged when the $v_i$ are permuted, the invariance that temporal modeling shows for pooling and for attention without position. Attention selects which frames matter for each word, but it can tell the order of two events only if the features themselves carry their position in time, which neither the frame features nor the local 3D-CNN features do explicitly.

Feature and pooling comparisons on MSR-VTT

Xu et al. (2016) compared encoders on their new dataset with the same LSTM decoder. With mean pooling, C3D clip features scored BLEU-4 39.9 and METEOR 29.3, against 37.3 and 28.7 for VGG-19 frame features; combining C3D and VGG-19 with soft attention reached 40.5 and 29.9 (Tables 3 and 4). Soft attention beat mean pooling for every feature, by at most 0.9 METEOR, and a single middle frame scored 32.4 and 22.6 (Table 5).

An experiment on event order

The script below (PyTorch 2.14.1 on CPU, with NumPy 2.5.4; 5 to 7 minutes on a laptop) trains four small captioners on synthetic videos whose captions depend on event order: a coloured shape moves in one direction, then in another (“the red square moves left then up”). Each video is two independent 4-frame shots of $20 \times 20$ pixels, and each model sees six snippets, the consecutive frame pairs inside each shot, stacked as 6 channels so that a snippet shows short-range motion, as flow or 3D-convolution features do. Because the shots are generated independently, the snippets of “left then up” have exactly the same distribution as those of “up then left”. The models share a small CNN per snippet and a GRU decoder and differ in how they combine the six snippet features:

• mean: the average, as in the mean-pooled captioner;
• attn: Yao-style soft attention recomputed at every word, with no time index;
• attn+time: the same attention after adding a learned embedding for each snippet position;
• gru: a recurrent encoder that reads the snippets in order, closer in spirit to S2VT, whose final state starts the decoder.

Training targets are drawn at random from four templates, one of which reverses the word order of the events (“the red square moves up after moving left”). Before the first run, a check on the pixels alone (the centroid of bright pixels in each shot) recovered both motions and their order in 2,000 of 2,000 videos, so the labels are present in the data; swapping the two shots changed the logits of untrained mean and attn models by at most $1.2 \times 10^{-7}$, float rounding, and those of the other two models by much more. The scoring functions come from the next section’s script, saved as caption_metrics.py. The model and batch were made smaller than first planned, after timing the first training steps and before any run, and the runtime was kept rather than cutting seeds.

"""Mean pooling against temporal models for captioning synthetic two-shot videos of a moving shape.
PyTorch (CPU). Uses the coco-caption port in caption_metrics.py."""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import numpy as np
import torch
from torch import nn
import torch.nn.functional as F
from caption_metrics import bleu, rouge_l, cider_d, references, COLORS, SHAPES, DIRS, TEMPLATES

torch.set_num_threads(2)
STEPS = int(sys.argv[1]) if len(sys.argv) > 1 else 800
SEEDS = [0, 1, 2]
VOCAB = ["<pad>", "<bos>", "<eos>", "the", "a", "moves", "goes", "then", "and", "first", "after", "moving",
         *COLORS, *SHAPES, *DIRS]
IDX = {w: i for i, w in enumerate(VOCAB)}
RGB = {"red": (1.0, 0.2, 0.2), "green": (0.2, 1.0, 0.2), "blue": (0.2, 0.2, 1.0)}
STEP = {"left": (0, -2), "right": (0, 2), "up": (-2, 0), "down": (2, 0)}      # (dy, dx) per frame, pixels
H = 20
yy, xx = np.mgrid[0:H, 0:H] + 0.5

def draw(color, shape, cy, cx, r=3.2):
    y, x = yy - cy, xx - cx
    if shape == "circle":
        m = x**2 + y**2 < r**2
    elif shape == "square":
        m = (np.abs(x) < 0.85 * r) & (np.abs(y) < 0.85 * r)
    else:
        m = (y < 0.8 * r) & (y > -r + 2 * np.abs(x))
    img = np.full((3, H, H), 0.05)
    img[:, m] = np.array(RGB[color])[:, None]
    return img

def shot(color, shape, d, rng):
    """4 frames: the shape moves 3 steps of 2 px in direction d from a random start."""
    dy, dx = STEP[d]
    lo_y, hi_y = 4 - min(0, 3 * dy), H - 4 - max(0, 3 * dy)
    lo_x, hi_x = 4 - min(0, 3 * dx), H - 4 - max(0, 3 * dx)
    cy, cx = rng.uniform(lo_y, hi_y), rng.uniform(lo_x, hi_x)
    return [draw(color, shape, cy + t * dy, cx + t * dx) for t in range(4)]

def make_video(rng):
    """Two independent shots (motion a, then motion b != a); returns 6 snippets (frame pairs) and labels."""
    c, s = COLORS[rng.integers(3)], SHAPES[rng.integers(3)]
    a, b = (DIRS[i] for i in rng.choice(4, size=2, replace=False))
    frames = shot(c, s, a, rng) + shot(c, s, b, rng)
    frames = [f + 0.05 * rng.standard_normal(f.shape) for f in frames]
    snips = [np.concatenate([frames[k + t], frames[k + t + 1]]) for k in (0, 4) for t in range(3)]
    return np.stack(snips).astype(np.float32), (c, s, a, b)

def batch(n, rng):
    vids, labels = zip(*[make_video(rng) for _ in range(n)])
    tok = torch.zeros(n, 11, dtype=torch.long)
    for i, lab in enumerate(labels):        # training target: one of the four templates at random
        cap = TEMPLATES[rng.integers(4)].format(c=lab[0], s=lab[1], a=lab[2], b=lab[3]).split()
        ids = [IDX["<bos>"]] + [IDX[w] for w in cap] + [IDX["<eos>"]]
        tok[i, :len(ids)] = torch.tensor(ids)
    return torch.from_numpy(np.stack(vids)), tok, list(labels)

class Captioner(nn.Module):
    """kind: 'mean' (mean pooling), 'attn' (soft temporal attention, no time index),
    'attn+time' (attention over snippet features plus learned time embeddings), 'gru' (recurrent encoder)."""
    def __init__(self, kind, d=128):
        super().__init__()
        self.kind = kind
        self.cnn = nn.Sequential(nn.Conv2d(6, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                                 nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                                 nn.Flatten(), nn.Linear(32 * 5 * 5, d), nn.ReLU())
        self.time = nn.Parameter(0.1 * torch.randn(6, d))       # used only by 'attn+time'
        self.enc = nn.GRU(d, d, batch_first=True)               # used only by 'gru'
        self.att_h, self.att_v, self.att_w = nn.Linear(d, d), nn.Linear(d, d), nn.Linear(d, 1)
        self.init = nn.Linear(d, d)
        self.emb = nn.Embedding(len(VOCAB), d)
        self.dec = nn.GRUCell(2 * d, d)
        self.out = nn.Linear(d, len(VOCAB))

    def encode(self, x):                                        # x: (B, 6 snippets, 6, 20, 20)
        e = self.cnn(x.flatten(0, 1)).view(x.shape[0], 6, -1)   # one feature per snippet
        if self.kind == "attn+time":
            e = e + self.time
        if self.kind == "gru":
            h = self.enc(e)[1][0]                               # reads the snippets in order
            return e, h, h
        m = e.mean(1)                                           # symmetric in the snippets
        return e, torch.tanh(self.init(m)), m

    def context(self, e, h, fixed):
        if self.kind in ("mean", "gru"):
            return fixed
        score = self.att_w(torch.tanh(self.att_h(h)[:, None] + self.att_v(e))).squeeze(-1)
        return (F.softmax(score, -1)[..., None] * e).sum(1)     # Yao et al.'s soft attention, per word

    def forward(self, x, tokens):                               # teacher forcing
        e, h, fixed = self.encode(x)
        logits = []
        for t in range(tokens.shape[1]):
            h = self.dec(torch.cat([self.emb(tokens[:, t]), self.context(e, h, fixed)], -1), h)
            logits.append(self.out(h))
        return torch.stack(logits, 1)

    @torch.no_grad()
    def greedy(self, x, max_len=10):
        e, h, fixed = self.encode(x)
        tok, out = torch.full((x.shape[0],), IDX["<bos>"]), []
        for _ in range(max_len):
            h = self.dec(torch.cat([self.emb(tok), self.context(e, h, fixed)], -1), h)
            tok = self.out(h).argmax(-1)
            out.append(tok)
        caps = []
        for row in torch.stack(out, 1).tolist():
            words = []
            for i in row:
                if i == IDX["<eos>"]:
                    break
                words.append(VOCAB[i])
            caps.append(" ".join(words))
        return caps

def train(kind, seed, steps=STEPS, bs=32):
    torch.manual_seed(seed)
    rng = np.random.default_rng(1000 + seed)
    model = Captioner(kind)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    for _ in range(steps):
        x, tok, _ = batch(bs, rng)
        logits = model(x, tok[:, :-1])
        loss = F.cross_entropy(logits.reshape(-1, len(VOCAB)), tok[:, 1:].reshape(-1), ignore_index=0)
        opt.zero_grad(); loss.backward(); opt.step()
    return model, loss.item()

def judge(cap, lab):
    """Order right? Direction words right (ignoring order)? Colour and shape right?"""
    w = cap.split()
    ds = [x for x in w if x in DIRS]
    events = ds[::-1] if "after" in w else ds                   # "moves b after moving a"
    col = next((x for x in w if x in COLORS), None) == lab[0]
    shp = next((x for x in w if x in SHAPES), None) == lab[1]
    return events == [lab[2], lab[3]], sorted(ds) == sorted(lab[2:]), col and shp

if __name__ == "__main__":
    x_test, _, labels = batch(600, np.random.default_rng(999))  # same 600 test videos for every model
    refs = [references(*lab) for lab in labels]
    print(f"{'model':9s} seed  loss  order  dirs  col+shape  BLEU-4 sent-B4 ROUGE-L CIDEr-D")
    summary = {}
    for kind in ["mean", "attn", "attn+time", "gru"]:
        for seed in SEEDS:
            model, loss = train(kind, seed)
            model.eval()
            caps = model.greedy(x_test)
            acc = np.mean([judge(c, l) for c, l in zip(caps, labels)], axis=0)
            (b, per), r, cd = bleu(caps, refs), rouge_l(caps, refs)[0], cider_d(caps, refs)[0]
            summary.setdefault(kind, []).append((acc[0], cd))
            print(f"{kind:9s} {seed:4d} {loss:5.3f} {acc[0]:6.3f} {acc[1]:5.3f} {acc[2]:9.3f} "
                  f"{b[3]:7.3f} {per[:, 3].mean():7.3f} {r:7.3f} {cd:7.3f}")
        if kind == "mean":
            print("  e.g.", caps[0], "| truth:", refs[0][0])
    for kind, v in summary.items():
        o, c = np.array(v).T
        print(f"{kind:9s} order accuracy {o.min():.3f}-{o.max():.3f}, CIDEr-D {c.min():.3f}-{c.max():.3f}")

model     seed  loss  order  dirs  col+shape  BLEU-4 sent-B4 ROUGE-L CIDEr-D
mean         0 0.469  0.137 0.263     0.863   0.608   0.577   0.811   4.733
mean         1 0.428  0.202 0.433     0.940   0.695   0.674   0.819   5.148
mean         2 0.463  0.097 0.225     0.965   0.631   0.614   0.794   4.966
  e.g. the blue triangle moves down then right | truth: the blue triangle moves down then left
attn         0 0.421  0.163 0.312     0.927   0.622   0.596   0.823   4.963
attn         1 0.376  0.383 0.702     0.942   0.787   0.774   0.863   5.509
attn         2 0.438  0.175 0.347     0.945   0.643   0.622   0.806   4.982
attn+time    0 0.196  1.000 1.000     0.885   0.952   0.948   0.985   5.876
attn+time    1 0.187  1.000 1.000     0.862   0.941   0.933   0.981   5.807
attn+time    2 0.210  0.997 0.997     0.942   0.972   0.969   0.991   6.296
gru          0 0.352  0.408 0.417     0.737   0.789   0.775   0.890   4.736
gru          1 0.347  0.663 0.798     0.785   0.775   0.743   0.897   5.064
gru          2 0.271  0.827 0.835     0.830   0.854   0.837   0.941   5.388
mean      order accuracy 0.097-0.202, CIDEr-D 4.733-5.148
attn      order accuracy 0.163-0.383, CIDEr-D 4.963-5.509
attn+time order accuracy 0.997-1.000, CIDEr-D 5.807-6.296
gru       order accuracy 0.408-0.827, CIDEr-D 4.736-5.388

Two runs printed identical output. In all six runs of mean and attn, order accuracy is between 0.43 and 0.55 times the direction accuracy: among captions with both directions right, the order is a coin flip, as the symmetry predicts. Neither model learned the directions reliably in 800 steps, so their order accuracy is far below 50%. With time embeddings, attention got the order right in at least 99.7% of test videos for every seed. The recurrent encoder learned order, but more slowly: 0.41 to 0.83 over seeds, with weaker colour and shape accuracy as well.

The metrics separate the models less clearly than the order column does. BLEU-4 puts attn+time first in every seed. CIDEr-D ranks attn with seed 1, which gets the order right for 38% of videos, above gru with seed 2, which gets it right for 83%, and gives gru seed 0 almost the same score as mean seed 0, whose order accuracy is a third of gru’s. By our reading, the order error costs few n-grams, while colour and shape words carry high IDF weight, which the next section’s script shows directly. Mean sentence BLEU-4 and corpus BLEU-4 (the open choice discussed below) differ by at most 0.032 and rank the twelve runs in the same order.

Dense captioning: localize, then describe

Krishna et al. (2017) built the first dense video captioner as two modules. An event proposal module, a variant of DAPs, runs an LSTM over C3D features sampled at strides 1, 2, 4 and 8 so that it can propose both short events and events lasting minutes, and keeps overlapping proposals instead of suppressing them. A captioning LSTM then describes each proposal, with context from the other events: it attends separately over events that ended earlier (“past”) and the rest (“future”), and concatenates both summaries with the event’s own representation. With ground-truth segments on ActivityNet Captions, context raised METEOR from 7.85 to 9.46 and CIDEr from 20.97 to 24.56; with the top 1,000 learned proposals the full model scored BLEU-4 2.20, METEOR 4.82 and CIDEr 17.29 (Table 1). The paper averaged its scores over tIoU thresholds 0.3, 0.5 and 0.7.

Early dense captioners typically output about ten times more proposal–caption pairs than there are ground-truth events, for high recall (Wang et al., 2021). By our reading, the localization stage of such two-stage systems borrows anchors, boundary scores and non-maximum suppression from temporal action localization.

Dense captioning as set prediction: PDVC

PDVC (Wang et al., 2021) removes the proposal stage. Frame features from a pretrained action network (C3D or TSN) pass through a deformable transformer encoder at four temporal scales; a decoder turns $N$ learned event queries (10 for ActivityNet Captions, 100 for YouCook2) into event features, and two heads read each query in parallel: a localization head predicts a segment and a confidence, and a captioning head, an LSTM with or without deformable soft attention, writes the sentence. An event counter predicts how many events to keep, up to 10 on ActivityNet Captions. As in DETR, training matches predictions to ground-truth events one to one with the Hungarian algorithm, with a matching cost from generalized IoU and a focal classification loss, so no non-maximum suppression is needed. At test time the top events are ranked by localization confidence plus a length-modulated sum of the caption’s word log-probabilities.

Dense captioning as one sequence: Vid2Seq

Vid2Seq (Yang et al., 2023) writes the whole dense caption as a single token sequence: each event becomes its start-time token, its end-time token and its sentence, the events are ordered by start time, and the time tokens are 100 extra vocabulary entries that quantize the video’s duration. A frozen CLIP ViT-L/14 encodes each frame, a transformer adds temporal context with learned temporal position embeddings, a text encoder reads the transcribed speech (itself turned into timestamped sentences), and a T5-Base decoder, 314M trainable parameters in all, generates the event sequence. Pretraining uses unlabelled narrated videos from YT-Temporal-1B, which the paper describes as 18 million videos (its Table 4 lists the default pretraining set as 15M), with speech sentences as pseudo-events: a generative objective predicts the transcript from the frames, and a denoising objective recovers masked spans of it. In the ablations, YouCook2 validation CIDEr rose from 18.0 without pretraining to 47.1 with pretraining on untrimmed videos with time tokens (Table 1), and pretraining on 150K videos already gave 40.1 (Table 4).

Dense captioning, validation sets	Features	YouCook2 SODA$_c$ / CIDEr / METEOR	ActivityNet Captions SODA$_c$ / CIDEr / METEOR
PDVC (Wang et al., 2021)	C3D	—	5.26 / 25.87 / 7.50
PDVC (Wang et al., 2021)	TSN	4.42 / 22.71 / 4.74	5.44 / 28.96 / 7.96
Vid2Seq (Yang et al., 2023)	CLIP, speech	7.9 / 47.1 / 9.3	5.8 / 30.1 / 8.5

Scores with predicted events, from PDVC’s Tables 2 and 3 and Vid2Seq’s Table 5; CIDEr and METEOR are from the ActivityNet evaluation code averaged over tIoU 0.3, 0.5, 0.7 and 0.9. Vid2Seq’s table lists PDVC at the same values, rounded. Vid2Seq used only videos still on YouTube, 10–20% fewer than the original datasets, and PDVC’s I3D+VGGish features (not shown) miss 9% of the videos, so the rows are not on identical test sets. On event localization alone, Vid2Seq trailed PDVC on ActivityNet Captions (precision 53.9 against 58.1, Table 6), which its authors attribute to building in less task-specific knowledge.

End-to-end video transformers and pretrained models

SwinBERT (Lin et al., 2022) trains on raw frames instead of precomputed features. A Video Swin Transformer initialized from Kinetics-600 turns $T$ frames of $H \times W$ pixels into $\tfrac{T}{2} \times \tfrac{H}{32} \times \tfrac{W}{32}$ video tokens; a multimodal transformer encoder takes those tokens and up to 50 word tokens, with a causal mask so that each word sees only earlier words and all video tokens, and is trained by masked language modelling; at test time it generates word by word. A learnable attention mask over the video tokens, pushed towards sparsity by an $L_1$ penalty, regularizes attention over long inputs. Without that mask, MSR-VTT CIDEr rose from 36.6 with 2 frames to 55.3 with 64 (Table 4a). With 64 frames and the mask, SwinBERT reported, on the MSR-VTT test split, BLEU-4 45.4, METEOR 30.6, ROUGE-L 64.1 and CIDEr 55.9, and on MSVD 66.3, 42.4, 80.9 and 149.4 (Table 2). The paper’s Table 1 lists higher numbers, CIDEr 160.0 on MSVD, which match none of the settings in Table 2 or Table 5; the closest is 160.3 for a model first trained on VATEX (Table 6). Vid2Seq’s Table 8 lists SwinBERT at CIDEr 53.8 on MSR-VTT and 120.6 on MSVD, without saying where these values come from.

Vision–language pretraining made captioning a fine-tuning task for video as for images. Vid2Seq’s clip-captioning results show the effect of pretraining data with the same model: CIDEr on MSR-VTT test rose from 57.2 without pretraining to 61.5 with HowTo100M and 64.6 with YT-Temporal-1B, and on MSVD from 120.3 to 140.6 and 146.2 (Table 8). Generative image–text models such as GIT (Wang et al., 2022) and audio–visual pretraining models such as VALOR (Liu et al., 2025) report video captioning among their tasks. BLIP was transferred zero-shot to video retrieval and question answering by concatenating the features of 8 or 16 frames, which its authors note ignores all temporal information (Li et al., 2022). Video–language models, which connect a video encoder to a large language model, treat captioning as one instruction among many.

Training and decoding

Training is teacher-forced cross-entropy, as in image captioning; S2VT decoded greedily and Krishna et al. used beam search of size 5. PDVC compares only methods trained with cross-entropy, noting that reinforcement learning on the metric tends to produce longer sentences with repeated phrases (Wang et al., 2021); Vid2Seq likewise leaves out methods that directly optimize the test metric. Image Captioning covers teacher forcing, beam search and metric optimization.

Datasets and Benchmarks

Dataset	Videos	Sentences	Split (videos)	Task
MSVD	1,970 clips used	about 41 per clip	1,200 / 100 / 670	clip
MSR-VTT	10,000 clips	20 per clip, 200K	6,513 / 497 / 2,990	clip
VATEX	41,250 clips	10 English + 10 Chinese per clip	25,991 / 3,000 / 6,000 public test / 6,278 secret test	clip, multilingual
ActivityNet Captions	20K videos	3.65 events per video, 100K	about 10,000 / 4,900 / 5,044	dense
YouCook2	2,000 videos	3–16 segments per video	67% / 23% / 10% by recipe	dense

Splits are train / validation / test.

• MSVD (Microsoft Research Video Description Corpus; Chen and Dolan, 2011) was collected on Mechanical Turk in 2010 for paraphrase research: workers chose short YouTube segments showing a single unambiguous event, and others described them in any language, with audio muted. Collection yielded 2,089 segments and 85,550 English descriptions, about 41 per video. Captioning papers use 1,970 clips with the 1,200 / 100 / 670 split of earlier work (Venugopalan et al., 2015, NAACL). Sources disagree on clip length: “generally 4–10 seconds” in the collection paper, 10 to 25 seconds in the NAACL paper, 10.2 seconds on average in S2VT’s Table 1.
• MSR-VTT (Xu et al., 2016) has 10,000 clips of 10 to 30 seconds, 41.2 hours in total, from 7,180 videos returned for 257 popular queries to a commercial video search engine, in 20 categories, with audio kept. Each clip has 20 sentences after duplicates and short sentences were removed; the vocabulary has 29,316 words. The abstract counts 1,327 annotators and the text 1,317. VATEX’s authors measured a 66.0% duplicated-sentence rate within MSR-VTT videos, against none in VATEX (Wang et al., 2019, Table 3).
• VATEX (Wang et al., 2019) captions clips of about 10 seconds from the Kinetics-600 validation and held-out test sets, covering its 600 human activities. Every clip has 10 English captions, for which workers were asked to write at least 10 words, from crowd workers in English-speaking countries, and 10 Chinese captions, five of them translations of English ones, giving about 206,000 English–Chinese pairs for video-guided translation. The abstract counts over 41,250 videos and 825,000 captions; the text gives 41,269 clips. The secret-test captions are withheld for a challenge. English captions average 15.23 words against 9.28 for MSR-VTT.
• ActivityNet Captions (Krishna et al., 2017) describes ActivityNet videos, 849 hours in all, with temporally localized sentences averaging 13.48 words; sentences cover 36 seconds and 31% of their video on average, and 10% overlap. Validation and test videos have two independent annotations; the evaluation code scores against both validation files. Papers give different split sizes: 10,009 / 4,925 / 5,044 (Wang et al., 2021) and 10,024 / 4,915 / 5,044 (Fujita et al., 2020), and the dataset paper counts 4,926 validation videos; test captions are on the challenge server only. The ActivityNet article describes the parent dataset and the challenge, which added dense captioning in 2017.
• YouCook2 (Zhou et al., 2018) has 2,000 untrimmed cooking videos of up to 10 minutes, 176 hours in all, nearly evenly spread over 89 recipes from four cuisine regions. Each video has 3–16 procedure segments, temporally localized and described by one imperative English sentence (e.g. “grill the tomatoes in a pan”); segments last 19.6 seconds on average and the vocabulary is about 2,600 words. Annotators had audio and subtitles but wrote their own descriptions, and each video was annotated once. The split is 67% / 23% / 10% within each recipe; PDVC uses 1,333 / 457 / 210 videos and, like Vid2Seq, reports on validation. Clip captioning on YouCook2 uses the ground-truth segments as clips: SwinBERT counts 15.4K clips with one caption each.

Other benchmarks include the movie description corpora MPII-MD and M-VAD, TVC with subtitles and ViTT with short tags for instructional videos.

Results on clip captioning.

Model	Year	Input	MSR-VTT B4 / M / C	MSVD B4 / M / C
Mean pool + LSTM, COCO-pretrained	2015	AlexNet frames	—	33.3 / 29.1 / —
Temporal attention + 3D-CNN	2015	GoogLeNet + 3D-CNN	—	41.9 / 29.6 / 51.7
SA-LSTM	2016	C3D + VGG-19	40.5 / 29.9 / —	—
SwinBERT	2022	Video Swin, 64 frames, end to end	45.4 / 30.6 / 55.9	66.3 / 42.4 / 149.4
Vid2Seq, YT-Temporal-1B pretraining	2023	CLIP frames, speech	— / 30.8 / 64.6	— / 45.3 / 146.2

Test-set results as printed: B4 is BLEU-4, M METEOR, C CIDEr (all ×100). Sources: Venugopalan et al. (2015, NAACL) Table 3, Yao et al. (2015) Table 1 (whose “BLEU” column is not labelled with an n-gram order), Xu et al. (2016) Table 4, Lin et al. (2022) Table 2, Yang et al. (2023) Table 8. Vid2Seq evaluated on the videos still available, so its rows are not on the full test sets. On VATEX’s public test set, the dataset authors’ baseline scored BLEU-4 28.4 and CIDEr 45.1, and SwinBERT 38.7 and 73.0 (Lin et al., 2022, Table 3).

Evaluation Metrics

Clip captioning reuses the image-captioning metrics, BLEU-4, METEOR, ROUGE-L and CIDEr, which Image Captioning defines; Yao et al. (2015), S2VT’s authors and VATEX’s authors computed them with the COCO caption evaluation code. Details of that code change the numbers.

• Tokenization. Captions are tokenized with the Stanford PTB tokenizer, lower-cased, and stripped of punctuation before scoring; METEOR runs as the Java METEOR 1.5 program.
• BLEU is a corpus score. Matched n-grams, candidate n-grams, candidate lengths and reference lengths are summed over the whole test set before precisions and the brevity penalty are computed, as Papineni et al. (2002) define it to avoid punishing short sentences harshly. For each candidate the reference length is the “best match length”, the closest reference length; the code breaks ties towards the shorter reference, which the paper does not specify. Small constants ($10^{-15}$ and $10^{-9}$) keep a candidate with no matching 4-gram from scoring exactly zero.
• “CIDEr” in the code is CIDEr-D. The scorer called CIDEr implements the clipped, length-penalized CIDEr-D with $\sigma = 6$ and the factor 10 (Vedantam et al., 2015). It also weights raw n-gram counts by IDF rather than the length-normalized term frequency of the paper’s Eq. 1; by our reading, the normalization cancels in a cosine but changes the clipped numerator.
• CIDEr depends on the test set. Document frequencies are counted over the reference sets of the videos being scored, so a video’s score changes with the other videos evaluated alongside it, and a model scored on a subset of a benchmark (videos removed from YouTube) is scored with different IDF weights.
• More references help consensus metrics. In the CIDEr paper’s experiments on images, the accuracy of unigram CIDEr and ROUGE at predicting human consensus kept improving up to about 50 references in a pilot study, while BLEU-1 got worse with more references because it can match a poor one (Vedantam et al., 2015). By that finding, a 20-reference MSR-VTT clip supports consensus scoring better than an ActivityNet event with one sentence. S2VT chose METEOR as its only metric because the movie datasets have one reference per clip and CIDEr’s authors had found METEOR better with few references.

Dense captioning metrics

The ActivityNet Captions evaluation code (evaluate.py in Krishna’s densevid_eval repository) scores dense captions as follows, for each tIoU threshold $\tau$:

1. Keep at most 1,000 predicted events per video.
2. Pair every prediction with every ground-truth event, in either annotation file, whose temporal IoU with it is at least $\tau$; a prediction that overlaps none is paired with a random string, so it scores about zero.
3. Score each video’s pairs with the COCO scorers, each pair as one “image” with one reference, and average over videos; a video without predictions scores zero.
4. Average over $\tau \in \{0.3, 0.5, 0.7, 0.9\}$, the code’s default; the 2017 paper names 0.3, 0.5 and 0.7.

METEOR is the challenge’s main metric: without the verbose flag, the code computes only METEOR. By our reading, step 3 makes CIDEr’s IDF and BLEU’s corpus sums local to one video’s pairs. The code changed after the paper: SODA’s authors used a version from November 2017 that fixed an overestimate in counting pairs (Fujita et al., 2020), a May 2018 commit fixed which unmatched predictions were paired with the dummy reference, and a June 2019 commit replaced the fixed dummy string “abc123!@#” with a random one, because submissions could add that string to their own captions. PDVC reports with the 2018 challenge’s version (Wang et al., 2021). Numbers from different versions or threshold sets are not comparable: Krishna et al.’s 4.82 METEOR used three thresholds and predates these changes.

SODA. Fujita et al. (2020) argued that the official score ignores the order of events and rewards redundancy: a prediction can match several references and a reference several predictions, and the average over matched pairs does not depend on how many captions are produced. SODA first finds a one-to-one, order-preserving matching between predictions and references, both sorted by start time, by dynamic programming that maximizes the summed IoU (counting only pairs with IoU at least $\tau$). It then computes METEOR for the matched pairs and turns the sum into precision (divided by the number of predicted captions $|P|$) and recall (divided by the number of reference captions $|G|$), reporting their F-measure:

$$\text{Precision} = \frac{\sum_{g \in G} f(g, p_{a(g)})}{|P|}, \qquad \text{Recall} = \frac{\sum_{g \in G} f(g, p_{a(g)})}{|G|}, \qquad F = \frac{2\, \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}},$$

where $a(g)$ is the prediction matched to reference $g$ and $f$ is METEOR. SODA$_c$, the variant later papers report, uses the product of IoU and METEOR as the matching score. On ActivityNet Captions validation, sampling more of a transformer model’s outputs barely changed the official score (4.10 with as many captions as references, 4.19 with all, about 228 per video) while SODA$_c$ fell from 4.02 to 0.63 (Table 1). Shuffling the order of oracle captions lowered the official score by 33.1% and SODA$_c$ by 57.0% for the same model (Table 3), On 50 videos judged by crowd workers, SODA agreed with human preferences between two systems’ oracle captions in 76% of cases against 66% for the official score, and between swapped and reversed reference captions in 94% against 72% (Section 6.3).

What the n-gram metrics reward

The NumPy script below (NumPy 2.3.5) ports the COCO code’s BLEU, ROUGE-L and CIDEr-D scorers and scores six systematic edits of the correct caption on 200 synthetic two-motion videos, each with the four reference templates of the experiment above. The edits are the same on every video, so the scores show what each metric rewards rather than noise. A check run beside it (not shown) found that the port’s per-sentence BLEU agrees with Papineni’s formula to $4 \times 10^{-10}$ and its CIDEr-D with Image Captioning’s simpler implementation to $3 \times 10^{-15}$. The original is Python 2 code; every division in it is on floats, and the port keeps Python floats and float64 arrays, so neither Python 3 nor NumPy 2’s promotion rules change its results.

"""NumPy port of the coco-caption BLEU, ROUGE-L and CIDEr-D scorers (tylin/coco-caption, commit 3a9afb2),
applied to captions of synthetic moving-shape videos whose meaning depends on event order."""
import numpy as np
from collections import Counter

def ngrams(words, n=4):
    """All 1..n-grams of a token list with their counts (precook in coco-caption)."""
    return Counter(tuple(words[i:i + k]) for k in range(1, n + 1) for i in range(len(words) - k + 1))

def _bleu_from(correct, guess, testlen, reflen, n):
    tiny, small = 1e-15, 1e-9                      # coco-caption's guards: zero matches give ~0, not 0
    p = (correct + tiny) / (guess + small)         # modified n-gram precisions p_1..p_n
    b = np.cumprod(p) ** (1.0 / np.arange(1, n + 1))   # BLEU-k = geometric mean of p_1..p_k
    ratio = (testlen + tiny) / (reflen + small)
    return b * np.exp(1 - 1 / ratio) if ratio < 1 else b   # brevity penalty only if shorter

def bleu(cands, refs, n=4):
    """Corpus BLEU-1..n (as reported by coco-caption) and the per-sentence scores it also computes."""
    tot_c, tot_g, tot_t, tot_r, per_sent = np.zeros(n), np.zeros(n), 0.0, 0.0, []
    for c, rs in zip(cands, refs):
        cw, rws = c.split(), [r.split() for r in rs]
        maxref = Counter()                          # clip each n-gram at its count in any ONE reference
        for rw in rws:
            for g, k in ngrams(rw, n).items():
                maxref[g] = max(maxref[g], k)
        correct = np.zeros(n)
        for g, k in ngrams(cw, n).items():
            correct[len(g) - 1] += min(maxref[g], k)
        guess = np.array([max(0, len(cw) - k + 1) for k in range(1, n + 1)], dtype=float)
        reflen = min((abs(len(rw) - len(cw)), len(rw)) for rw in rws)[1]   # closest; ties -> shorter
        tot_c, tot_g, tot_t, tot_r = tot_c + correct, tot_g + guess, tot_t + len(cw), tot_r + reflen
        per_sent.append(_bleu_from(correct, guess, len(cw), reflen, n))
    return _bleu_from(tot_c, tot_g, tot_t, tot_r, n), np.array(per_sent)

def rouge_l(cands, refs, beta=1.2):
    """ROUGE-L: LCS precision and recall, each maximized over references, combined as an F-measure."""
    def lcs(a, b):
        L = np.zeros((len(a) + 1, len(b) + 1), dtype=int)
        for i in range(1, len(a) + 1):
            for j in range(1, len(b) + 1):
                L[i, j] = L[i - 1, j - 1] + 1 if a[i - 1] == b[j - 1] else max(L[i - 1, j], L[i, j - 1])
        return L[-1, -1]
    scores = []
    for c, rs in zip(cands, refs):
        tc = c.split(" ")
        l = [lcs(r.split(" "), tc) for r in rs]
        P = max(x / len(tc) for x in l)
        R = max(x / len(r.split(" ")) for x, r in zip(l, rs))
        scores.append((1 + beta**2) * P * R / (R + beta**2 * P) if P > 0 and R > 0 else 0.0)
    return float(np.mean(scores)), np.array(scores)

def cider_d(cands, refs, n=4, sigma=6.0):
    """coco-caption's "CIDEr", which is CIDEr-D: IDF over the reference sets of the evaluated videos,
    raw counts x IDF, clipped cosine per n, Gaussian length penalty, x10, mean over n and references."""
    crefs = [[ngrams(r.split(), n) for r in rs] for rs in refs]
    df = Counter(g for rs in crefs for g in {g for r in rs for g in r})   # videos whose refs contain g
    log_n = np.log(float(len(crefs)))
    def vec(cnts):
        v, norm, length = [{} for _ in range(n)], np.zeros(n), 0
        for g, tf in cnts.items():
            k = len(g) - 1
            v[k][g] = float(tf) * (log_n - np.log(max(1.0, df[g])))
            norm[k] += v[k][g] ** 2
            if k == 1:
                length += tf                         # coco counts bigrams: same length difference
        return v, np.sqrt(norm), length
    scores = []
    for c, rs in zip(cands, crefs):
        vc, nc, lc = vec(ngrams(c.split(), n))
        s = np.zeros(n)
        for r in rs:
            vr, nr, lr = vec(r)
            for k in range(n):
                val = sum(min(x, vr[k].get(g, 0.0)) * vr[k].get(g, 0.0) for g, x in vc[k].items())
                if nc[k] != 0 and nr[k] != 0:
                    val /= nc[k] * nr[k]
                s[k] += val * np.e ** (-(float(lc - lr) ** 2) / (2 * sigma**2))
        scores.append(10.0 * s.mean() / len(rs))
    return float(np.mean(scores)), np.array(scores)

COLORS, SHAPES, DIRS = ["red", "green", "blue"], ["circle", "square", "triangle"], ["left", "right", "up", "down"]
TEMPLATES = ["the {c} {s} moves {a} then {b}", "a {c} {s} moves {a} and then {b}",
             "the {c} {s} goes {a} first and then {b}", "the {c} {s} moves {b} after moving {a}"]

def references(c, s, a, b):
    return [t.format(c=c, s=s, a=a, b=b) for t in TEMPLATES]

if __name__ == "__main__":
    rng = np.random.default_rng(0)
    videos = []
    for _ in range(200):                            # invented labels: colour, shape, first and second motion
        a, b = rng.choice(4, size=2, replace=False)
        videos.append((COLORS[rng.integers(3)], SHAPES[rng.integers(3)], DIRS[a], DIRS[b]))
    refs = [references(*v) for v in videos]
    other = lambda a, b: next(d for d in DIRS if d not in (a, b))
    edits = {
        "correct":   lambda c, s, a, b: f"a {c} {s} moves {a} then {b}",
        "swapped":   lambda c, s, a, b: f"a {c} {s} moves {b} then {a}",
        "wrong dir": lambda c, s, a, b: f"a {c} {s} moves {a} then {other(a, b)}",
        "wrong col": lambda c, s, a, b: f"a {COLORS[(COLORS.index(c) + 1) % 3]} {s} moves {a} then {b}",
        "after":     lambda c, s, a, b: f"a {c} {s} moves {b} after moving {a}",
        "vague":     lambda c, s, a, b: f"a {c} {s} moves",
    }
    print(f"{'candidate':10s} {'BLEU-1':>6s} {'BLEU-4':>6s} {'sent-B4':>7s} {'ROUGE-L':>7s} {'CIDEr-D':>7s}")
    for name, edit in edits.items():
        cands = [edit(*v) for v in videos]
        (b, per_sent), r, cd = bleu(cands, refs), rouge_l(cands, refs)[0], cider_d(cands, refs)[0]
        print(f"{name:10s} {b[0]:6.3f} {b[3]:6.3f} {per_sent[:, 3].mean():7.3f} {r:7.3f} {cd:7.3f}")
    df = Counter(g for rs in refs for g in {g for r in rs for g in ngrams(r.split())})
    idf = {" ".join(g): np.log(200) - np.log(max(1, df[g])) for g in
           [("then",), ("moves",), ("left",), ("red",), ("moves", "left"), ("left", "then")]}
    print("IDF:", ", ".join(f"'{k}' {v:.2f}" for k, v in idf.items()))

candidate  BLEU-1 BLEU-4 sent-B4 ROUGE-L CIDEr-D
correct     1.000  1.000   1.000   0.922   6.007
swapped     1.000  0.669   0.669   0.659   4.705
wrong dir   0.857  0.809   0.809   0.790   4.921
wrong col   0.857  0.643   0.643   0.790   2.937
after       1.000  1.000   1.000   0.875   5.553
vague       0.472  0.472   0.472   0.629   3.919
IDF: 'then' 0.00, 'moves' 0.00, 'left' 0.77, 'red' 1.04, 'moves left' 0.77, 'left then' 1.22

Swapping the two events keeps every word, so BLEU-1 stays at 1.0, and BLEU-4 keeps two thirds of the correct caption’s score and CIDEr-D 78%; replacing one direction with a wrong one scores higher on both. A caption that says the opposite of what happened is scored about like one with a single wrong word. ROUGE-L, whose longest common subsequence can keep only one of two swapped words in order, also scores the swap below the wrong direction. The “after” phrasing reverses the word order of the events but is correct, and scores BLEU-4 1.0 because it matches the fourth reference: n-gram order is not event order. The wrong colour costs the most CIDEr-D, since “red” has more IDF weight than “left”, and “then” and “moves”, which occur in every video’s references, have none. The vague caption keeps 65% of the correct caption’s CIDEr-D while saying nothing about the motion. Corpus and mean sentence BLEU-4 coincide here only because every candidate of a row has the same length and match pattern; for the trained models above, with mixed outputs, they differ. Image Captioning’s example shows the same mechanics for a single image and sentence-level scores.

Applications

• Accessibility. Describing video for blind and visually impaired people, with speech synthesis, is named as a motivation by S2VT’s and Yao et al.’s authors; movie audio description is itself the source of MPII-MD and M-VAD.
• Indexing and search. Captions make video searchable by text, a use named by Yao et al. (2015); the same datasets serve text–video retrieval.
• Instructional video. Segmenting and describing the steps of procedures such as recipes was YouCook2’s motivation (Zhou et al., 2018).
• Video-guided translation. VATEX’s paired English and Chinese captions support translating a sentence with the video as context (Wang et al., 2019).
• Human–robot interaction, named as an application by S2VT’s authors.

Open Problems

• Evaluating order and story. Clip-captioning metrics barely register event order, and the official dense-captioning score ignores it; SODA addresses order and redundancy but still scores each matched sentence with METEOR.
• Static bias in benchmarks. S2VT’s one-point loss with shuffled frames suggests that MSVD captions rarely depend on order; benchmarks whose captions need temporal reasoning are needed to measure temporal models.
• Single references for events. Dense benchmarks have one or two sentences per event, too few for consensus metrics by the CIDEr paper’s findings.
• Data that disappears. Benchmarks distribute YouTube identifiers, and videos are removed over time; Vid2Seq evaluated on 10–20% fewer videos than the original datasets, which by our reasoning makes comparisons across years less exact.
• Noisy, large-scale supervision. Narration scales where human captions do not, but it is not always visually grounded and is often temporally misaligned (Yang et al., 2023); the image counterpart of describing absent content is covered under multimodal hallucination.
• Long videos. The models above sample tens to a hundred frames; in our view, describing hour-long video within a fixed visual token budget is an open problem that video–language models inherit.