Video Summarization

Video summarization shortens a video to the parts that matter, so that a viewer can grasp it without watching all of it. The result is either a handful of still frames, called keyframes or a static summary, or a much shorter video stitched from selected clips, called a skim, a dynamic summary or a keyshot summary. De Avila et al. (2011), citing a review by Truong and Venkatesh (2007), list further names: representative frames, still-image abstracts or a static storyboard for the first form, and a moving-image abstract or moving storyboard for the second; the static task is also called keyframe extraction and the dynamic one video skimming. It belongs to video understanding, but unlike classification or localization it has no single correct answer: what to keep depends on the viewer, and that subjectivity shapes its methods, its benchmarks and the long-running argument about how to evaluate it.

Most current methods follow one pipeline. The video is cut into short segments, each frame or segment gets an importance score, and segments are chosen to maximize total importance within a length budget, usually 15% of the video. The scores are where most of the modelling happens, using the frame-aggregation tools of temporal modeling; the segmentation and the selection step look like fixed pre- and post-processing, yet Otani et al. (2019) showed that they decide most of the benchmark score.

Problem Definition

Given a video of $T$ frames, a summary is a subset of its frames, chosen so that it conveys the main content and is much shorter than the original. Two criteria recur in the literature, under several names. Importance (also interestingness or relevance) asks whether a part is worth showing at all. Representativeness and diversity (or coverage and non-redundancy) ask whether the selected parts together cover the video without repeating themselves. Zhang et al. (2016) group the criteria used by earlier methods into relevance, representativeness or importance, and diversity or coverage.

Most recent methods, as Otani et al. (2019) describe them, write the dynamic case as a selection problem over segments. Let a video be cut into segments $S_1, \dots, S_n$ with lengths $|S_i|$ and importance scores $v_i$. The summary is the indicator vector $x \in \{0, 1\}^n$ that solves

$$\max_x \sum_{i=1}^{n} x_i v_i \quad \text{subject to} \quad \sum_{i=1}^{n} x_i |S_i| \le L,$$

with the budget $L$ a fraction of the video length. This is the 0/1 knapsack problem, which dynamic programming solves exactly in $O(nL)$ time when lengths are integers (Gygli et al., 2014, Eq. 5; Song et al., 2015, Eq. 6). Gygli et al. took segment scores as sums of frame scores; TVSum and the deep-learning methods that followed use the mean, and that choice matters, as the section on the evaluation critique explains. Diversity does not fit this form, because it is a property of the selected set rather than of each segment; methods that model it use set functions such as determinantal point processes or submodular objectives instead, or add it to a learning objective.

Input and Output

The input is the video, sometimes with side information: its title (Song et al., 2015), its category (Potapov et al., 2014) or a user’s query (Sharghi et al., 2017). Methods work on per-frame features, usually sampled sparsely; Zhang et al. (2016) and Zhou et al. (2018) subsample videos to 2 frames per second and use 1,024-dimensional GoogLeNet pool5 features from an ImageNet-trained Inception network.

The output takes one of three forms, which Zhang et al. (2016, supplement, Table 1) illustrate on a six-frame example: selected keyframes, such as frames 2 and 6; interval-based keyshots, such as frames 1–2 and 5–6; and frame-level importance scores, such as $[0.5, 0.9, 0.1, 0.2, 0.7, 0.8]$. Their supplement converts between the forms: keyframes become keyshots by marking each segment that contains one and selecting segments by keyframes per unit duration; keyshots become keyframes by taking a frame of each shot; scores become keyshots by averaging them within segments and solving the knapsack problem. Gygli et al. (2014) argue for skims over keyframes for user videos, since keyframes lose all motion, which makes them less useful for watching or for finding, for example, a panning shot.

Variants

• Generic summarization keeps what is important in the video as a whole and is what SumMe and TVSum measure.
• Category-specific summarization learns importance from videos of the same category. Potapov et al. (2014) trained one linear support vector machine per TRECVID multimedia-event category on whole videos and applied it to segments, defining importance as evidence that a segment belongs to the event.
• Query-focused summarization conditions the summary on a user’s query, so that two users of the same video get different summaries. Sharghi, Laurel and Gong (2017) describe the task as recently proposed by Sharghi, Gong and Shah at ECCV 2016 (that paper was not read here) and built a dataset for it.
• Highlight detection scores short clips by how good a highlight they would make and keeps the top-scoring ones (Lei et al., 2021). Datasets differ in whether highlights depend on a query: most are query-agnostic, while QVHighlights annotates highlight saliency for each natural-language query (Lei et al., 2021).
• Egocentric summarization condenses hours of first-person recording; the UT Egocentric videos used by Gygli et al. (2015) and by Sharghi et al. (2017) last 3–5 hours each.

Neighbouring problems produce different outputs. Temporal action localization returns labelled segments for every action instance, and temporal action segmentation labels every frame; neither has a length budget or a notion of redundancy. Mahasseni et al. (2017) separate summarization from video synopsis, which packs tracked object tubes into a smaller space-time volume, and from montages that merge keyframes into one image, neither of which needs to keep the temporal layout of motions in the video; they also list hyperlapses, which speed up a video while changing the viewpoint.

Challenges

Subjectivity. Annotators agree only partly. On SumMe, the mean pairwise F-measure between one person’s summary and the others’ is 0.31 (range 0.18–0.51 over videos) and the mean Cronbach’s α is 0.74 (Gygli et al., 2014). TVSum’s crowd scores reach a mean pairwise F1 of 0.36 and a Cronbach’s α of 0.81 (Song et al., 2015, Table 1). Otani et al. (2019, §5.2) found a TVSum video in which two annotators marked almost the opposite parts from the rest.

Order effects. Song et al. (2015) observed a chronological bias: annotators who score shots in temporal order give higher scores to shots that appear earlier, even when later shots show the same content. They avoided it by clustering the shots with k-means and presenting each cluster’s shots in random order, with the option to re-watch the video.

Interest against coverage. A summary can show the most interesting moments or a balanced overview, and which is better depends on the video. Gygli et al. (2014) found that enforcing diversity by clustering superframes before selection brought no significant improvement on SumMe and concluded that their participants chose interesting over representative parts. On hours-long egocentric video, Gygli et al. (2015, Table 2) found that uniform sampling beat all earlier methods, which they attributed to slow-paced videos with few highlights.

Small benchmarks. The standard generic benchmarks have 25 and 50 videos, so supervised methods are trained on a few dozen examples. Zhang et al. (2016, Table 4) found that their LSTM model trailed simpler multilayer perceptrons when trained on one dataset and overtook them only when trained with more data from other datasets (their Augmented and Transfer settings, described below).

Evaluation. User studies are expensive and cannot be repeated exactly; comparisons with reference summaries are repeatable but, as Otani et al. showed, can reward properties unrelated to content. This is treated in detail below.

Approaches

Temporal segmentation

Selection works on segments, so the video must first be cut. Edited video can be split at shot boundaries; user video often has a single continuous shot.

• Uniform segments cut the video every few seconds; TVSum’s annotators scored 2-second shots (Song et al., 2015).
• Superframes (Gygli et al., 2014) are segments whose boundaries fall at frames with little motion, where editing guidelines say a cut is least disruptive, named by analogy with superpixels. They start uniform, at the most common length of the human-selected segments, and their boundaries are moved by hill climbing on an energy that combines a cut cost (motion magnitude at the boundaries, from KLT point tracks) with a log-normal length prior fitted to those segments.
• Kernel temporal segmentation (KTS; Potapov et al., 2014) is a change-point detector that compares all pairs of frames rather than neighbouring ones, so it finds gradual changes as well as cuts. With a kernel $K$ and its feature map $\phi$, it chooses $m$ change points $t_0 < \dots < t_{m-1}$ to minimize the within-segment variance plus a penalty on the number of segments,

$$J_{m,n} = \sum_{i=0}^{m} \sum_{t=t_{i-1}}^{t_i - 1} \left\| \phi(x_t) - \mu_i \right\|^2 + C\, m \left( \log \frac{n}{m} + 1 \right),$$

where $\mu_i$ is the mean of segment $i$ in feature space, $n$ the number of frames, $t_{-1} = 0$ and $t_m = n$. Cumulative sums of the Gram matrix give each segment’s variance in constant time, and dynamic programming finds the optimum for every $m$ up to $m_{\max}$ in $O(m_{\max} n^2)$; the penalty, a BIC-type term, then picks $m$, with $C$ cross-validated on annotated videos. KTS became the default: Zhang et al. (2016) segment all datasets with it, getting segments of about 5 seconds on average, and most methods compared on SumMe and TVSum use it (Otani et al., 2019, Table 1).

• Change-point detection by group LASSO. TVSum finds a piecewise-constant approximation of the feature sequence by penalizing the $\ell_{2,1}$ norm of its differences, solved by group LARS, which avoids the quadratic cost of dynamic programming (Song et al., 2015, §3.1).

Unsupervised methods

Clustering. Clustering-based keyframe extraction groups similar frames and keeps one per group. VSUMM (de Avila et al., 2011) samples one frame per second, describes each by a 16-bin histogram of the hue channel in HSV space, removes meaningless frames (near-monochromatic frames from fades, detected by the low spread of their histogram), and groups the rest by k-means. The number of clusters is estimated by counting how often the distance between consecutive sampled frames exceeds a threshold of 0.5; in the variant VSUMM2, clusters smaller than half the average size are dropped. The frame nearest each cluster centre becomes a keyframe, keyframes whose histograms are too similar are removed, and the rest are shown in temporal order. Clustering also appears as a baseline in later work, for example k-means and spectral clustering of frames in TVSum and k-medoids in DR-DSN.

Interestingness. Gygli et al. (2014) score each frame by a combination of features found to relate to interest: attention from spatial and temporal saliency, aesthetics (colourfulness, contrast, edge distribution), landmark detection, face and person areas, and a classifier for a camera that follows an object. The weights of a linear model with pairwise terms are fitted by least squares to the fraction of annotators who selected each frame, so the importance model is supervised although the features are hand-designed; superframe scores are sums of frame scores, and the knapsack step selects them.

Web priors. TVSum (Song et al., 2015) uses a video’s title as a description of its topic. Images found by searching the title (with query expansion) are factorized jointly with the frames by co-archetypal analysis, which learns latent patterns shared by both sets rather than patterns found in only one of them; a frame’s importance is its contribution to reconstructing the video through these shared patterns.

Representativeness and diversity as learning signals. Two deep methods train a frame selector without labels. SUM-GAN (Mahasseni et al., 2017) passes the selector’s weighted frames through a variational autoencoder built from LSTMs and trains a discriminator LSTM to tell the original feature sequence from the reconstruction, in the spirit of a generative adversarial network; a summary that lets the reconstruction fool the discriminator is taken to represent the video. A sparsity term keeps the mean selection score near a chosen fraction $\sigma$, and variants add a determinantal point process (SUM-GANdpp) or a repelling regularizer as diversity terms. DR-DSN (Zhou et al., 2018) instead writes the criteria directly as a reward. A bidirectional LSTM outputs a probability $p_t$ per frame, frames are sampled from Bernoulli($p_t$), and the selected set $\mathcal{Y}$ earns

$$R_{\text{div}} = \frac{1}{|\mathcal{Y}|(|\mathcal{Y}|-1)} \sum_{t \in \mathcal{Y}} \sum_{t' \in \mathcal{Y}, t' \ne t} d(x_t, x_{t'}), \qquad R_{\text{rep}} = \exp\!\left( -\frac{1}{T} \sum_{t=1}^{T} \min_{t' \in \mathcal{Y}} \| x_t - x_{t'} \|^2 \right),$$

where $d$ is one minus cosine similarity, set to 1 for frames more than $\lambda = 20$ sampled frames apart so that distant repetitions, which may carry the story, are not penalized. $R_{\text{rep}}$ is a k-medoids criterion: it rewards selections close to every frame. The policy is trained with REINFORCE and a moving-average baseline, plus a term that keeps the mean of $p_t$ near $\epsilon = 0.5$. The paper adds the two rewards; the public PyTorch code averages them, which only rescales the reward.

Supervised methods

Supervised methods learn importance from annotations. Gygli et al. (2015) learned the weights of a mixture of submodular objectives (interestingness, representativeness, uniformity) from reference summaries, and found on egocentric video that the learned weights shifted from interestingness towards representativeness and uniformity as summaries grew longer. Zhang et al. (2016) introduced vsLSTM, a bidirectional LSTM over frame features whose hidden states and features feed a multilayer perceptron that predicts each frame’s importance, and wrote that, to their knowledge, it was the first use of LSTMs for video summarization. dppLSTM adds a determinantal point process, which assigns a subset $z$ of frames the probability

$$P(z) = \frac{\det(L_z)}{\det(L + I)}, \qquad L_{tt'} = y_t\, y_{t'}\, \phi_t^\top \phi_{t'},$$

where $y_t$ is the predicted importance and $\phi_t$ a learned similarity feature; near-duplicate frames make $L_z$ nearly singular, so diverse subsets are more probable. The LSTM is first trained on importance scores with a squared loss and then fine-tuned by maximizing the likelihood of the annotated keyframes. Because the DPP’s selection usually covers only about 10% of the video, the summary is topped up from the importance scores to reach the 15% budget (Zhang et al., 2016, supplement §3.2). Zhang et al. also showed that adding the keyframe datasets Open Video Project (OVP) and YouTube from de Avila et al. (2011) to the training data improved results on both benchmarks, defining the Canonical (train and test on splits of one dataset), Augmented (other datasets added to training) and Transfer (train only on the other datasets) settings that later papers report. SUM-GAN and DR-DSN both have supervised variants, SUM-GANsup and DR-DSNsup, which add a keyframe loss.

Query-focused summarization and highlight detection

Sharghi et al. (2017) built a query-focused benchmark on the four UT Egocentric videos of Lee, Ghosh and Grauman (CVPR 2012). Each 5-second shot was tagged with the concepts it shows, from a lexicon of 48, as the union of three crowd workers’ tags. Each video has 46 queries of two or three concepts, covering four cases: concepts that appear together in some shot (15 queries), concepts that all appear but never together (15), queries where only one concept appears (15) and a query with none present (1). Three trained volunteers summarized every query and video. Their model is a sequential determinantal point process whose kernel is parameterized by a memory network that attends from the query to the shots.

QVHighlights (Lei et al., 2021) joins moment retrieval and highlight detection. Its 10,148 videos are 150-second pieces of YouTube lifestyle vlogs and news footage, with 10,310 human-written queries and 18,367 relevant moments; each query can have several disjoint moments, and every 2-second clip inside them was rated by three workers on a five-point scale from “Very Bad” to “Very Good”. Moment-DETR treats moment retrieval as set prediction, in the manner of DETR, and adds a saliency head for highlights. Moment retrieval connects this line of work to temporal action localization, with natural-language queries in place of class labels.

Datasets and Benchmarks

SumMe (Gygli et al., 2014) has 25 raw or minimally edited user videos of holidays, events and sports, from 39 seconds to about 6.5 minutes (2 minutes 40 seconds on average) in the paper’s Table 1, whose text says about 1 to 6 minutes, filmed by egocentric, moving or static cameras. In a controlled experiment, 41 participants (19 men and 22 women, aged 19 to 39) cut a summary of each video they were shown, between 5% and 15% of its length, with the audio removed; each video has 15 to 18 summaries. The paper reports F-measures computed from per-frame precision and recall at a 15% summary length. The dataset page named in the paper, which also hosted the evaluation code, now redirects to the first author’s home page without it (checked November 2026).

TVSum (Song et al., 2015; the paper calls it TVSum50) has 50 YouTube videos, five for each of ten TRECVID multimedia-event categories such as changing a vehicle tire, parkour and dog shows, chosen to be under a Creative Commons licence, 2 to 10 minutes long, with more than one shot and a title that describes the visual topic; in total 352,356 frames and about 3.5 hours. Twenty Amazon Mechanical Turk workers per video read the title, watched the whole video without sound, and scored every 2-second shot from 1 to 5, within target frequencies that keep high scores rare. The paper’s list of these targets names score 4 and score 5 twice, so the printed ranges for the two lowest scores are evidently misnumbered. Unlike SumMe, TVSum supplies scores, not summaries: reference summaries are built from them for each annotator, with the same segments as the method under test (Otani et al., 2019, §3.2; Zhang et al., 2016, supplement §2.2).

Later papers describe the two datasets inconsistently: Mahasseni et al. (2017) give TVSum video lengths as 1 to 5 minutes and say that SumMe provides frame-level scores, while the dataset papers give 2 to 10 minutes and say that SumMe’s annotations are summaries.

Dataset	Videos	Content	Annotation	Used for
SumMe	25	user videos, 39 s–6.5 min	15–18 user summaries per video (5–15% long)	generic skims
TVSum	50	YouTube, 10 categories, 2–10 min	20 importance scores (1–5) per 2-s shot	generic skims
OVP (VSUMM)	50	Open Video Project, 1–4 min	5 user keyframe sets per video	keyframes; extra training data
YouTube (VSUMM)	50	web videos, 1–10 min	user keyframe sets	keyframes; extra training data
UTE query-focused	4	egocentric, 3–5 h each	48-concept tags per 5-s shot; 3 summaries per query	query-focused
QVHighlights	10,148	vlogs and news, 150 s	queries, moments, 5-point clip saliency	moment retrieval, highlights

The OVP and YouTube rows are from de Avila et al. (2011); Zhang et al. (2016) list 39 YouTube videos, which Zhou et al. (2018) describe as the collection without its cartoons. Potapov et al. (2014) released MED-Summaries, 100 test and 60 validation videos from TRECVID 2011 with segment-level importance on a 0–3 scale, for category-specific summarization.

Evaluation Metrics

The F-score protocol

The common protocol compares a generated keyshot summary $A$ with each reference summary $B$ by temporal overlap. With $y_i$ and $y_i^*$ the frame indicators of the two summaries,

$$P = \frac{\sum_i y_i y_i^*}{\sum_i y_i}, \qquad R = \frac{\sum_i y_i y_i^*}{\sum_i y_i^*}, \qquad F = \frac{2PR}{P + R},$$

which is the frame-level F1 score built from precision and recall (Zhang et al., 2016, Eq. 5–6; Otani et al., 2019, Eq. 1–2). The summary must be at most 15% of the video, a limit that Song et al. (2015) and later work took from SumMe. In the evaluation code of Zhou et al., the generated summary is built from frame scores by averaging them within segments (KTS segments, according to the DR-DSN paper) and solving the knapsack problem with a capacity of $\lfloor 0.15\,T \rfloor$ frames; the code also offers a greedy alternative that goes through the segments in score order and adds each one that still fits (vsum_tools.py, generate_summary, at the pinned commit).

With several reference summaries, the per-reference F-scores are combined in one of two ways, and the two give very different numbers:

• Average. SumMe’s own measure is the mean pairwise F-measure (Gygli et al., 2014, Eq. 6), and TVSum reports the average pairwise F1 over its gold-standard summaries (Song et al., 2015, Eq. 7).
• Maximum. Keep the best match, on the argument that every annotator’s summary is a reasonable one (Otani et al., 2019, §3.3). Otani et al. (2019, Table 1) list most SumMe results from Gygli et al. (2015) onwards as maxima; one 2016 result is listed as an average. The difference shows in the two Gygli papers: the 2014 paper reports 0.234 for its method and 0.139 for random summaries, while the 2015 paper, which evaluated against the nearest-neighbour reference summary (the maximum) with the code from the dataset page, reports 39.34% for the 2014 method and 28.55% for random summaries (Gygli et al., 2015, §5 and Table 3). Our reading is that the change of aggregation accounts for most of that gap; the 2015 paper does not compare the two.

In Zhou et al.’s code the choice is made by dataset: the evaluation uses the average for TVSum and the maximum for SumMe (main.py, evaluate). Zhang et al. (2016) say only that they follow earlier work for multiple references, and Mahasseni et al. (2017) state that they merge multiple annotations into a single ground-truth summary. Published SumMe numbers are therefore comparable only when the aggregation is known.

Keyframe summaries are scored differently. VSUMM’s Comparison of User Summaries (CUS) matches each automatic keyframe to a user keyframe when the Manhattan distance between their colour histograms is below 0.5, removing matched pairs, and reports an accuracy rate (matched automatic keyframes over user keyframes) and an error rate (unmatched automatic keyframes over user keyframes), for each user separately (de Avila et al., 2011). Sharghi et al. (2017) score query-focused summaries semantically: two shots are as similar as the intersection over union of their concept sets, and a maximum-weight bipartite matching between the two summaries gives precision, recall and F1. QVHighlights counts clips rated “Very Good” as positive and scores highlights by mean average precision and by HIT@1, the hit rate of the top-scored clip, against each of the three annotators and then averaged (Lei et al., 2021, §5.1).

Otani et al.’s critique and rank correlation

Otani et al. (2019) ran a randomization test: frame scores drawn uniformly from $[0, 1]$, averaged within segments, and passed to the knapsack step, with segments from KTS, from uniform 60-frame cuts, or from random lengths drawn from one Poisson distribution ($\lambda = 60$ frames) or a mixture of two ($\lambda = 30$ and $90$). Over 100 trials, random scores with KTS segments reached an F-score of 0.41 (maximum) on SumMe and 0.57 (average) on TVSum (Otani et al., 2019, Table 1). In the same table, the published methods using KTS reached 0.41–0.45 on SumMe (maximum) and 0.58–0.60 on TVSum (average), and human summaries in a leave-one-out test 0.54 and 0.54. On TVSum, random scores therefore scored above the human leave-one-out average. With random two-peak segments, which ignore the content entirely, random scores reached 0.58 on TVSum. On SumMe, whose references are fixed summaries rather than scores, random scores with KTS did about as well as DR-DSN’s importance scores with KTS, and the segmentation method made a larger difference than the scoring (Otani et al., 2019, §4.1 and Fig. 3).

Their explanation concerns the knapsack step. A long segment is selected only if no combination of shorter segments with at most its total length has at least as much value; with average-pooled scores this rarely holds for long segments, so summaries are made mostly of the shortest segments, whatever the scores. When segment lengths vary widely, as with KTS or the two-peak segmentation, generated and reference summaries then share most of their segments. With equal-length segments the scores matter more, and summing frame scores instead of averaging them would weaken the effect, but averaging is used, according to Otani et al., because it raises TVSum F-scores sharply (§4.3). On SumMe, KTS has a further advantage: it groups visually similar frames, so long KTS segments tend to be static and redundant, the kind of footage annotators leave out.

As an alternative for methods that output frame scores, Otani et al. propose comparing the ranking of frames by predicted importance with each annotator’s ranking by Kendall’s $\tau$ and Spearman’s $\rho$, averaged over annotators. On TVSum (Table 2), dppLSTM scored $\tau = 0.042$ and $\rho = 0.055$, DR-DSN 0.020 and 0.026, random scores 0 by construction, and human annotators in leave-one-out 0.177 and 0.204. The measure separates the methods from chance and from each other, but its values are low even for humans, and it does not apply to clustering methods that select segments near cluster centres without scoring frames (§6). They also propose a curve that accumulates the mean human score over frames sorted by predicted importance, which shows at a glance how predictions compare with random scores and with each annotator.

Representative Results

The results below are as reported, with the 15% budget; the learned methods use GoogLeNet features and the Canonical setting unless marked (80% of the videos for training and validation, the rest for testing). The SumMe F-scores of methods evaluated as by Zhang et al. are listed by Otani et al. as maxima over references, and TVSum F-scores as averages. Values are percentages.

Method	Supervision	SumMe F	TVSum F	Source
Uniform sampling	none	29.3	15.5	Zhou et al. (2018), Table 2
VSUMM	none	33.7	—	Zhou et al. (2018), Table 2
TVSum (co-archetypal)	none (web images)	—	50.0	Song et al. (2015), Table 2
SUM-GANdpp	none	39.1	51.7	Mahasseni et al. (2017), Table 1
DR-DSN	none	41.4	57.6	Zhou et al. (2018), Table 2
vsLSTM	importance scores	37.6	54.2	Zhang et al. (2016), Table 3
dppLSTM	scores, then keyframes	38.6	54.7	Zhang et al. (2016), Table 3
dppLSTM, Augmented	plus OVP, YouTube and the other benchmark	42.9	59.6	Zhang et al. (2016), Table 3
SUM-GANsup	keyframes	41.7	56.3	Mahasseni et al. (2017), Table 1
DR-DSNsup	keyframes	42.1	58.1	Zhou et al. (2018), Table 3
Random scores, KTS	none	41	57	Otani et al. (2019), Table 1

Three cautions apply. First, Otani et al.’s Table 1 quotes dppLSTM at 0.43 and 0.60, which match Zhang et al.‘s Augmented results rather than the Canonical ones. Second, baselines depend on the protocol: Zhou et al. report uniform sampling at 15.5% on TVSum, while Song et al. report their uniform-sampling baseline at 0.36 in their own protocol, with their own change-point segments (the protocol of their 50.0 as well), and Otani et al. report random scores at 0.57. Third, SUM-GAN’s evaluation merges annotations into one reference, so its numbers may not be directly comparable with the others. On QVHighlights’ test split, Moment-DETR reached 35.69 highlight mAP and 55.60 HIT@1, against 31.30 and 61.04 for a CLIP baseline that scores each clip’s centre frame against the query; with weakly supervised pretraining on speech-recognition captions it reached 37.43 and 60.17 (Lei et al., 2021, Table 3).

Example: Keyframe Methods, Random Scores and the F-Score Protocol

The example builds synthetic videos from images bundled with scikit-image, so that the importance of every part is known by construction, and scores several summarizers with the F-score protocol of the previous section. Each video has 20–30 scenes of log-normally distributed length (median 4 seconds, 1–15 seconds, at 25 frames per second), about two minutes in all. A scene shows one of twelve images through a $48 \times 48$ window that drifts across a $96 \times 96$ copy of it, with brightness flicker and pixel noise; about a quarter of the scenes are fast “action” scenes, and images can recur within a video. Each image has a fixed interest value, the same in every video. Fifteen synthetic annotators score every 2-second chunk from 1 to 5, TVSum-style, as the image’s interest plus a bonus for action, plus the annotator’s own liking for that image and noise. Frames are described at about 2 frames per second by an $8 \times 8$ thumbnail, a 16-bin histogram and a frame-difference energy.

Five summarizers produce frame scores: uniformly spaced keyframes; k-means keyframes (the frame nearest each centre); keyframes chosen greedily to maximize DR-DSN’s reward $\tfrac{1}{2}(R_{\text{div}} + R_{\text{rep}})$, the objective rather than the trained policy; random scores, 100 trials per video as in Otani et al.’s test; and a small supervised model, ridge regression from features to the mean annotator score, trained on other videos in five folds. The three keyframe methods select 15% of the sampled frames, and a keyframe scores 1 and other frames 0, so after averaging within segments a segment’s value is its keyframe density, the conversion of Zhang et al.’s supplement. Segments come from a NumPy implementation of KTS with a linear kernel, its penalty $C$ set without labels on five separate videos so that segments last about 5 seconds, or from uniform 2-second cuts, since the choice of segmentation is open. Summaries and F-scores follow ports of knapsack_dp, generate_summary and evaluate_summary from Zhou et al.’s code; reference summaries are built from each annotator’s scores with the same segments, as for TVSum. The script also prints the average and maximum aggregations, the human leave-one-out score, Kendall’s $\tau$ and Spearman’s $\rho$ against each annotator at the sampled frames, and the mean length of the segments the knapsack picks.

import os
import numpy as np
import skimage
from skimage import io, transform, color
from sklearn.cluster import KMeans
from sklearn.linear_model import Ridge
from scipy.stats import kendalltau, spearmanr

FPS, STEP, CHUNK, W = 25, 12, 50, 48          # 25 fps, features at ~2 fps, annotation every 2 s, 48x48 frames
DATA = os.path.join(os.path.dirname(skimage.__file__), "data")
FILES = ["astronaut.png", "camera.png", "coffee.png", "chelsea.png", "rocket.jpg", "coins.png",
         "page.png", "text.png", "horse.png", "moon.png", "brick.png", "grass.png"]

def load(f):
    im = io.imread(os.path.join(DATA, f))     # bundled files only, read from disk
    if im.ndim == 3:
        im = color.rgb2gray(im[..., :3])
    return transform.resize(im.astype(float) / (255 if im.max() > 1 else 1), (96, 96), anti_aliasing=True)

IMAGES = [load(f) for f in FILES]
BASE = np.random.default_rng(2026).uniform(1.0, 4.0, len(FILES))   # interest of each image, same in every video

# ---------------------------------------------------------------- synthetic videos and annotators
def make_video(rng):
    """Scenes of different lengths, each a moving 48x48 window over one image; some fast 'action' scenes."""
    n_sc = rng.integers(20, 31)
    img = rng.choice(len(FILES), n_sc)                                  # images may recur in a video
    length = np.clip(np.exp(rng.normal(np.log(4 * FPS), 0.6, n_sc)), FPS, 15 * FPS).astype(int)
    action = rng.random(n_sc) < 0.25
    frames, scene = [], []
    for s in range(n_sc):
        pos = rng.uniform(0, 48, 2)
        vel = rng.normal(0, 1.5 if action[s] else 0.3, 2)
        for t in range(length[s]):
            pos = pos + vel
            for k in range(2):                                          # bounce at the image border
                if not 0 <= pos[k] <= 48:
                    vel[k] = -vel[k]
                    pos[k] = np.clip(pos[k], 0, 48)
            y, x = pos.astype(int)
            f = IMAGES[img[s]][y:y + W, x:x + W] * rng.normal(1, 0.03)  # brightness flicker
            frames.append(f + rng.normal(0, 0.05, (W, W)))              # sensor noise
            scene.append(s)
    return np.clip(np.array(frames), 0, 1), np.array(scene), img, action

def annotate(scene, img, action, rng, n_ann=15):
    """Each annotator scores every 2-s chunk from 1 to 5 (TVSum-style): content interest + own taste + noise."""
    T = len(scene)
    chunks = np.arange(0, T, CHUNK)
    pref = rng.normal(0, 0.7, (n_ann, len(FILES)))                      # per-annotator liking of each image
    out = np.zeros((n_ann, T))
    for c0 in chunks:
        s = np.bincount(scene[c0:c0 + CHUNK]).argmax()                  # dominant scene in the chunk
        raw = BASE[img[s]] + 1.5 * action[s] + pref[:, img[s]] + rng.normal(0, 0.5, n_ann)
        out[:, c0:c0 + CHUNK] = np.clip(np.round(raw), 1, 5)[:, None]
    return out

def features(frames):
    """Per sampled frame: 8x8 thumbnail, 16-bin histogram, frame-difference energy."""
    pos = np.arange(0, len(frames), STEP)
    f = frames[pos]
    thumb = f.reshape(len(pos), 8, 6, 8, 6).mean((2, 4)).reshape(len(pos), -1)
    hist = np.stack([np.histogram(x, 16, (0, 1))[0] / x.size for x in f])
    mot = np.abs(frames[pos] - frames[np.maximum(pos - 1, 0)]).mean((1, 2))[:, None]
    return np.hstack([thumb, hist, 10 * mot]), pos

def zscore(X):
    return (X - X.mean(0)) / (X.std(0) + 1e-8)

# ---------------------------------------------------------------- KTS (Potapov et al., 2014, Algorithm 1)
def kts(X, C, m_max=80):
    n = len(X)
    K = X @ X.T                                                         # linear kernel
    S = np.zeros((n + 1, n + 1)); S[1:, 1:] = K.cumsum(0).cumsum(1)
    d = np.concatenate([[0], np.cumsum(np.diag(K))])
    s, e = np.meshgrid(np.arange(n + 1), np.arange(n + 1), indexing="ij")
    with np.errstate(divide="ignore", invalid="ignore"):
        V = (d[e] - d[s]) - (S[e, e] - S[s, e] - S[e, s] + S[s, s]) / (e - s)
    V[e <= s] = np.inf                                                  # V[t, j]: variance of positions t..j-1
    L = np.full((m_max + 1, n + 1), np.inf); L[0] = V[0]
    arg = np.zeros((m_max + 1, n + 1), int)
    for i in range(1, m_max + 1):
        M = L[i - 1][:, None] + V
        L[i], arg[i] = M.min(0), M.argmin(0)
    m = np.arange(m_max + 1)
    g = np.where(m > 0, m * (np.log(n / np.maximum(m, 1)) + 1), 0)
    m_star = int(np.argmin(L[:, n] + C * g))
    cps, j = [], n
    for i in range(m_star, 0, -1):
        j = arg[i, j]; cps.append(j)
    return sorted(cps)                                                  # change points, in sampled positions

def segments_from_cps(cps, T):
    b = [0] + [c * STEP for c in cps] + [T]
    return np.array([[b[i], b[i + 1] - 1] for i in range(len(b) - 1)])  # inclusive ends, as generate_summary expects

def uniform_segments(T, L=2 * FPS):
    b = list(range(0, T, L)) + [T]
    return np.array([[b[i], b[i + 1] - 1] for i in range(len(b) - 1)])

# ---------------------------------------------------------------- protocol (ports of vsum_tools.py / knapsack.py @ 041606f)
def knapsack_dp(values, weights, capacity):
    n = len(values)
    table = np.zeros((n + 1, capacity + 1), np.float32)
    keep = np.zeros((n + 1, capacity + 1), bool)
    w_all = np.arange(capacity + 1)
    for i in range(1, n + 1):
        wi, vi = weights[i - 1], values[i - 1]
        cand = np.full(capacity + 1, -np.inf)                           # compared in float64, as the original's
        ok = w_all >= wi                                                # float + float32 sum is under NumPy 1.x
        cand[ok] = vi + table[i - 1, w_all[ok] - wi].astype(float)
        keep[i] = cand > table[i - 1]
        table[i] = np.where(keep[i], cand, table[i - 1])
    picks, k = [], capacity
    for i in range(n, 0, -1):
        if keep[i, k]:
            picks.append(i - 1); k -= weights[i - 1]
    return sorted(picks)

def expand(ypred, positions, T):
    """Sampled-position scores to frame scores, as generate_summary does."""
    fs = np.zeros(T, np.float32)
    pos = np.concatenate([positions, [T]])
    for i in range(len(pos) - 1):
        fs[pos[i]:pos[i + 1]] = ypred[i]
    return fs

def summary_from_frame_scores(fs, segs, proportion=0.15):
    T = len(fs)
    nfps = [int(e - s + 1) for s, e in segs]
    seg_score = [float(fs[s:e + 1].mean()) for s, e in segs]
    picks = knapsack_dp(seg_score, nfps, int(np.floor(T * proportion)))
    summ = np.zeros(T)
    for p in picks:
        summ[segs[p, 0]:segs[p, 1] + 1] = 1
    return summ, picks

def evaluate_summary(machine, users):
    f = []
    for u in users:
        ov = (machine * u).sum()
        p, r = ov / (machine.sum() + 1e-8), ov / (u.sum() + 1e-8)
        f.append(0.0 if p == 0 and r == 0 else 2 * p * r / (p + r))
    return np.mean(f), np.max(f)

# ---------------------------------------------------------------- keyframe selectors
def kf_uniform(X, K):
    return ((np.arange(K) + 0.5) * len(X) / K).astype(int)

def kf_kmeans(X, K):
    km = KMeans(K, n_init=10, random_state=0).fit(X)
    kf = []
    for c in range(K):
        idx = np.flatnonzero(km.labels_ == c)
        kf.append(idx[np.argmin(((X[idx] - km.cluster_centers_[c]) ** 2).sum(1))])
    return np.array(sorted(kf))

def kf_diversity(X, K, lam=20):
    """Greedy maximisation of DR-DSN's reward (Eq. 3-5; 0.5 * (R_div + R_rep) as in rewards.py).
    R_div ignores pairs more than lam sampled frames apart; computed incrementally."""
    Xs = X / np.sqrt(X.shape[1])                                        # keeps squared distances of order 1
    Xn = X / np.linalg.norm(X, axis=1, keepdims=True)
    n = len(X)
    D = 1 - Xn @ Xn.T                                                   # cosine dissimilarity
    D[np.abs(np.arange(n)[:, None] - np.arange(n)[None, :]) > lam] = 1
    dist = ((Xs[:, None, :] - Xs[None, :, :]) ** 2).sum(-1)
    sel, cur_min, d_sum, pair = [], np.full(n, np.inf), np.zeros(n), 0.0
    for _ in range(K):
        m = len(sel) + 1
        r_div = (pair + 2 * d_sum) / (m * (m - 1)) if m > 1 else np.zeros(n)
        r_rep = np.exp(-np.minimum(cur_min[:, None], dist).mean(0))
        r = 0.5 * (r_div + r_rep)
        r[sel] = -np.inf
        c = int(np.argmax(r))
        pair += 2 * d_sum[c]; d_sum += D[:, c]
        cur_min = np.minimum(cur_min, dist[:, c]); sel.append(c)
    return np.array(sorted(sel))

# ---------------------------------------------------------------- experiment
def calibrate_C():
    """Label-free: choose KTS's C on five separate videos so that segments last about 5 s on average."""
    rng = np.random.default_rng(999)
    vids = [zscore(features(make_video(rng)[0])[0]) for _ in range(5)]
    grid = np.logspace(0, 4, 41)
    dur = sum(len(X) for X in vids) * STEP / FPS
    mean_len = np.array([dur / sum(len(kts(X, C)) + 1 for X in vids) for C in grid])
    i = int(np.argmin(np.abs(mean_len - 5)))
    return grid[i], mean_len[i]

def rank_corr(y, A):
    """Kendall's tau and Spearman's rho against each annotator, averaged (Otani et al., 2019, Sec. 5.1)."""
    return np.mean([[kendalltau(y, a)[0], spearmanr(y, a)[0]] for a in A], 0)

KEYFRAME = {"uniform keyframes": kf_uniform, "k-means keyframes": kf_kmeans,
            "diversity (DR reward, greedy)": kf_diversity}
ORDER = list(KEYFRAME) + ["random scores (100 trials)", "supervised ridge", "human leave-one-out"]

def run(seed, C, n_videos=20, n_random=100):
    rng, rr = np.random.default_rng(seed), np.random.default_rng(seed + 100)
    vids = []
    for _ in range(n_videos):
        frames, scene, img, action = make_video(rng)
        X, pos = features(frames)
        vids.append((len(frames), X, pos, annotate(scene, img, action, rng)))
    sup = {}
    for fold in range(5):                                               # supervised: 5-fold over videos
        tr = [v for v in range(n_videos) if v % 5 != fold]
        Xtr = np.vstack([vids[v][1] for v in tr])
        ytr = np.concatenate([vids[v][3].mean(0)[vids[v][2]] for v in tr])
        mu, sd = Xtr.mean(0), Xtr.std(0) + 1e-8
        model = Ridge(alpha=1.0).fit((Xtr - mu) / sd, ytr)
        for v in range(fold, n_videos, 5):
            sup[v] = model.predict((vids[v][1] - mu) / sd)
    F = {m: {"kts": [], "2s": []} for m in ORDER}
    R = {m: [] for m in ORDER[3:]}
    seg_len = {"all": [], "random": [], "supervised": []}
    n_seg, n_empty = [], 0
    for v, (T, X, pos, ann) in enumerate(vids):
        Xz, K = zscore(X), round(0.15 * len(X))
        scores = {}
        for name, fn in KEYFRAME.items():
            y = np.zeros(len(X)); y[fn(Xz, K)] = 1
            scores[name] = expand(y, pos, T)
        scores["supervised ridge"] = expand(sup[v], pos, T)
        for sg, segs in (("kts", segments_from_cps(kts(Xz, C), T)), ("2s", uniform_segments(T))):
            lens = (segs[:, 1] - segs[:, 0] + 1) / FPS
            refs = np.array([summary_from_frame_scores(a, segs)[0] for a in ann])
            for name, fs in scores.items():
                summ, picks = summary_from_frame_scores(fs, segs)
                F[name][sg].append(evaluate_summary(summ, refs))
                if sg == "kts" and name == "supervised ridge" and picks:
                    seg_len["supervised"].append(lens[picks].mean())
            trials = []
            for _ in range(n_random):
                summ, picks = summary_from_frame_scores(rr.random(T), segs)
                trials.append(evaluate_summary(summ, refs))
                if sg == "kts":
                    if picks:
                        seg_len["random"].append(lens[picks].mean())
                    else:
                        n_empty += 1
            F["random scores (100 trials)"][sg].append(np.mean(trials, 0))
            F["human leave-one-out"][sg].append(np.mean(
                [evaluate_summary(refs[a], np.delete(refs, a, 0)) for a in range(len(refs))], 0))
            if sg == "kts":
                seg_len["all"].append(lens.mean()); n_seg.append(len(segs))
        A = ann[:, pos]                                                 # annotator scores at the sampled frames
        R["supervised ridge"].append(rank_corr(sup[v], A))
        R["random scores (100 trials)"].append(np.mean([rank_corr(rr.random(len(pos)), A) for _ in range(n_random)], 0))
        R["human leave-one-out"].append(np.mean(
            [rank_corr(A[a], np.delete(A, a, 0)) for a in range(len(A))], 0))
    out = {m: [100 * np.mean(F[m]["kts"], 0), 100 * np.mean(F[m]["2s"], 0),
               np.mean(R[m], 0) if m in R else None] for m in ORDER}
    info = (np.mean([v[0] for v in vids]) / FPS, np.mean(n_seg),
            np.mean(seg_len["all"]), np.mean(seg_len["random"]), np.mean(seg_len["supervised"]), n_empty)
    return out, info

C, mlen = calibrate_C()
print(f"KTS penalty C = {C:.1f} (mean segment {mlen:.1f} s on five calibration videos)")
res, info = run(0, C)
print(f"seed 0: 20 videos of {info[0]:.1f} s on average; KTS gives {info[1]:.1f} segments per video")
print(f"{'method':32s} {'KTS avg':>8s} {'KTS max':>8s} {'2s avg':>8s} {'2s max':>8s} {'tau':>7s} {'rho':>7s}")
for m in ORDER:
    k, u, r = res[m]
    rc = f"{r[0]:7.3f} {r[1]:7.3f}" if r is not None else f"{'-':>7s} {'-':>7s}"
    print(f"{m:32s} {k[0]:8.1f} {k[1]:8.1f} {u[0]:8.1f} {u[1]:8.1f} {rc}")
print(f"mean length of KTS segments: all {info[2]:.1f} s, picked with random scores {info[3]:.1f} s, "
      f"picked with supervised scores {info[4]:.1f} s; empty random summaries: {info[5]}")
allres = [res] + [run(s, C)[0] for s in (1, 2)]
print("\nseeds 0-2, mean (min-max) of avg F-score: KTS | 2-s segments")
for m in ORDER:
    k = np.array([r[m][0][0] for r in allres]); u = np.array([r[m][1][0] for r in allres])
    print(f"{m:32s} {k.mean():5.1f} ({k.min():.1f}-{k.max():.1f}) | {u.mean():5.1f} ({u.min():.1f}-{u.max():.1f})")

KTS penalty C = 79.4 (mean segment 4.2 s on five calibration videos)
seed 0: 20 videos of 114.8 s on average; KTS gives 25.6 segments per video
method                            KTS avg  KTS max   2s avg   2s max     tau     rho
uniform keyframes                    45.4     63.3     18.7     37.4       -       -
k-means keyframes                    61.1     78.7     12.2     32.1       -       -
diversity (DR reward, greedy)        60.3     77.3     16.4     35.8       -       -
random scores (100 trials)           67.1     85.0     15.7     31.6  -0.001  -0.001
supervised ridge                     72.6     91.0     31.4     56.4   0.312   0.412
human leave-one-out                  71.8     91.8     45.8     69.6   0.482   0.566
mean length of KTS segments: all 4.6 s, picked with random scores 1.7 s, picked with supervised scores 1.7 s; empty random summaries: 0

seeds 0-2, mean (min-max) of avg F-score: KTS | 2-s segments
uniform keyframes                 45.9 (44.8-47.6) |  22.0 (18.7-24.6)
k-means keyframes                 59.5 (58.5-61.1) |  12.9 (11.5-15.1)
diversity (DR reward, greedy)     60.4 (60.3-60.5) |  15.1 (14.2-16.4)
random scores (100 trials)        68.8 (67.1-71.6) |  15.5 (15.4-15.7)
supervised ridge                  74.4 (72.2-78.3) |  33.1 (29.5-38.3)
human leave-one-out               74.9 (71.8-77.9) |  46.4 (45.8-46.8)

With KTS segments, random scores beat all three keyframe methods in every seed and came within about six points of the supervised model and the human leave-one-out score, which agree closely with each other. A reader who saw only the KTS column would conclude that the unsupervised methods are worse than chance and that a ridge regression on thumbnails summarizes as well as people do. The diagnostic line shows the mechanism Otani et al. describe: KTS segments last 4.6 seconds on average, but the knapsack picks segments of 1.7 seconds on average, for random and supervised scores alike. Reference summaries are built by the same knapsack on the same segments, so they also favour the shortest segments, and in Otani et al.’s account any score that does not strongly prefer long segments agrees with them. The keyframe methods lose this advantage, in our reading, because a segment without a keyframe has value zero and is never chosen; many of the short segments that the references contain hold no keyframe. The run does not separate this cause from others.

With uniform 2-second segments, where every segment weighs the same, the scores matter: random falls to about 15, the supervised model reaches about 33 and humans about 46. Uniform keyframes now beat k-means and the diversity objective, which on average score below random. These keyframe numbers should be read with a property of the selection step in mind: the knapsack port keeps an item only when it is strictly better, as the original does, so among segments of equal value the earliest ones usually win (rounding can break a tie), and with equal-length segments most segments that hold one keyframe have the same value. Uniform keyframes fall in about four times as many 2-second segments as the budget admits, so it follows from the code that their summaries lean towards the start of the video; reference summaries, built from whole-number chunk scores on the same segments, contain ties too. The script does not measure these effects, and the order among the three keyframe methods is left unexplained. The maximum over annotators raises every method by about 16 to 25 points in seed 0, random included, which is why SumMe results reported as maxima cannot be compared with averages. Rank correlation behaves as Otani et al. intended: random scores sit at zero, the supervised scores at $\tau = 0.31$ and humans at $0.48$. The synthetic annotators agree more than TVSum’s real ones, whose leave-one-out $\tau$ was 0.177. The keyframe methods have no frame ranking and get no correlation, the limitation Otani et al. note for clustering methods.

The synthetic videos are simpler than real ones: interest is a fixed function of the image, which a regression on raw thumbnails can learn, and there is no story or semantics to understand. The example’s point is the protocol’s behaviour, which in our reading does not depend on those details, and it agrees with Otani et al.’s finding on TVSum that random and human-annotated scores score alike under KTS and differ under equal-length segments.

The script uses NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2, gave identical output in two runs and takes about 40 seconds. In its first run, the videos had 8 to 13 scenes (about 50 seconds) and $C$ was calibrated on one video. On the test videos KTS segments then averaged longer than the 15% budget, so in some videos no segment fitted the knapsack and summaries were empty, which showed as an undefined segment-length diagnostic. The videos were then lengthened to 20–30 scenes so that the budget holds several segments, as it does on SumMe and TVSum; $C$ was calibrated on five videos; $m_{\max}$ was raised from 40 to 80; the diagnostic was made to skip empty summaries; and the greedy diversity selection was rewritten to update its sums incrementally, after a check that it selects the same keyframes as the first version on four test videos. An unused import was later removed and a code comment reworded, without changing the output. During review, the knapsack port was corrected to compare values in double precision, as the original does under the NumPy 1.x of its Python 2 code.

Applications

• Browsing and search. Song et al. (2015) motivate summaries as a better preview than thumbnails and titles for choosing what to watch online, and Potapov et al. (2014) as a way to skim long, unedited uploads.
• User and egocentric video. Gygli et al. (2014) target personal recordings, which are long, redundant and partly of poor quality; egocentric summarization condenses whole days of first-person recording (Gygli et al., 2015; Sharghi et al., 2017).
• Sports and highlights. Otani et al. (2019) give as an example compressing the footage of a sports event into a few minutes of goals and penalty kicks, and query-based highlight detection lets a user ask for the moments they care about (Lei et al., 2021).
• Indexing. Keyframes serve as visual indexes for retrieval in video archives (Gygli et al., 2014; de Avila et al., 2011).

Open Problems

In our assessment, the main open questions as of November 2026 are these.

• Evaluation. Otani et al. showed that the F-score protocol on SumMe and TVSum mostly measures segment lengths. Rank correlation tests importance scores but not the final summary, does not apply to methods without frame scores, and gives low values even for humans; semantic measures such as Sharghi et al.’s concept matching need dense annotation. No measure is yet agreed for whether a summary tells the story of a video.
• Data. The generic benchmarks are small and old, and Otani et al. called for substantially larger datasets; QVHighlights is larger but annotates 150-second clips rather than whole videos.
• Subjectivity. Summaries differ legitimately between people. Query-focused and highlight formulations make preferences explicit, and averaging over annotators, keeping the best match, or modelling each annotator remain different answers to the same problem.
• Long video. Hours-long recordings require understanding events and their order, not only visual novelty; the structure of a story is what Zhang et al. (2016) cite as the reason for sequence models, and it is not what the standard benchmarks reward.