HMDB: A Large Video Database for Human Motion Recognition

“HMDB: A Large Video Database for Human Motion Recognition” is a paper by H. Kuehne (Karlsruhe Institute of Technology), H. Jhuang, E. Garrote and T. Poggio (MIT) and T. Serre (Brown University), presented at the IEEE International Conference on Computer Vision (ICCV) in Barcelona in November 2011. It introduced the Human Motion DataBase, HMDB51: 51 action classes of at least 101 clips each, cut from digitized movies and web video, with meta-labels for how each clip was filmed and three fixed train/test splits. The paper argued that existing action benchmarks were either near ceiling or classifiable from appearance alone, tested that claim by classifying six datasets with low-level colour and scene features, and evaluated two systems on the new data, a bag of HOG/HOF words and a model of the motion pathway of the visual cortex. Both reached a little over 20% against a chance level of 2%. The PAMI Technical Committee awarded the paper the Helmholtz Prize in 2021. This article covers the 2011 publication; the dataset as distributed today, its split files and protocol, its licence and the results that followed are described in the HMDB51 article.

The paper was read in the authors’ copy linked from the earlier official page and archived from the Serre lab website in November 2021: 8 pages, matching the proceedings pages 2556–2563, in a PDF certified by IEEE’s PDF eXpress service on 10 August 2011, which in our reading is the camera-ready file. The IEEE Xplore copy was not compared. Section, table and figure numbers below are those of this copy.

Problem

The paper opens with scale: billions of online videos and, by its figure, about 24 hours of video uploaded to YouTube every minute, against action datasets that had not grown the way image datasets had (it cites LabelMe, 80 Million Tiny Images, ImageNet, PASCAL VOC and SUN on the image side). The most used benchmarks, KTH, Weizmann and IXMAS, held 6 to 11 actions each, typically performed by one staged actor with no occlusion, little clutter and limited variation in lighting and camera position. Results on them were near ceiling: in a survey the authors cite (Weinland, Ronfard & Boyer), 12 of 21 systems scored above 90% on KTH, and on Weizmann 14 of 16 systems reached 90% or more, 8 exceeded 95% and 3 scored 100%.

More realistic datasets existed. Hollywood (Laptev et al., 2008) and Hollywood2 used clips from feature films, and the paper describes UCF Sports, UCF50 (which extends UCF YouTube’s 11 classes) and Olympic Sports as mostly sports videos from YouTube; Table 1 lists them with their numbers of actions and clips per action. The authors’ objection to the YouTube collections was different: because the clips were found by searching for specific titles, the actions were unambiguous and, they argued, distinguishable from shape alone. Their evidence was a small experiment. Crowd workers on Amazon Mechanical Turk marked 14 joint positions in every frame of 5 random clips from each of the 9 UCF Sports classes, and features of the joint positions in single frames, classified with leave-one-clip-out testing, recognized more than 98% (chance 11%). The authors read this as saying that static posture sufficed and joint kinematics were unnecessary, which they considered unlikely to hold for real-world actions and incompatible with Johansson’s demonstrations that kinematics are critical for perceiving biological motion.

The same kind of annotation on HMDB51, for 10 classes similar to UCF50’s in over 1,100 random clips, gave 35% from single-frame joints (chance 10%), against 54% for the motion features of §4.1 computed over the whole clip; the same motion features scored 66% on the corresponding 10 UCF50 classes. The paper concludes that HMDB51’s classes differ mainly in motion rather than static pose, which makes the dataset useful both as a benchmark and for studying the contributions of motion and shape cues, a question it ties to research on biological motion perception.

Contribution

• A dataset of 51 classes, each with at least 101 clips, 6,766 clips in total according to the contributions paragraph, from digitized movies, public archives and web video. The authors call it the largest and perhaps most realistic action dataset available at the time. Each clip was validated by at least two human observers.
• Meta-labels for every clip: visible body parts, camera motion, viewpoint, number of people and video quality, meant to allow experiments on chosen subsets and analyses of robustness.
• Three train/test splits with 70 training and 30 test clips per class, balanced across meta-labels, that keep clips of one source video on one side.
• A stabilized copy of every clip with the camera motion removed.
• A comparison of datasets by how well low-level colour and scene features classify them relative to spatio-temporal features.
• Two baselines and an analysis of their robustness to nuisance factors and of shape versus motion cues.

Method

Collecting and annotating the clips

Students watched digitized movies, public databases such as the Prelinger archive, other internet video and YouTube and Google videos, and marked any segment showing a single unambiguous human action (§2.1). They were asked for one action per clip, a main actor at least 60 pixels tall, a minimum contrast, at least one second of clip and acceptable compression artifacts. The first pass produced more than 60 categories; keeping those with at least 101 clips left 51, which the paper groups into five types: facial actions, facial actions with object manipulation, body movements, body movements with object interaction, and body movements with human interaction. The class list and the distributed class names are in the HMDB51 article.

Each clip was annotated with visible body parts (head, upper body, lower body or full body), camera motion (moving or static), viewpoint relative to the actor (front, back, left or right) and the number of people (one, two or more) (§2.2). A quality label graded what could be seen through most of the clip: high when details such as the actor’s fingers and eyes were identifiable, medium when the larger limbs were, low when even those were lost to motion blur and compression. Figure 2 gives the distribution of four of the labels over the whole dataset; the HMDB51 article reproduces the proportions.

The three splits (§2.3) were chosen from a very large number of random splits: first the one with the most balanced meta-label distribution, then greedily the two least correlated with those already chosen, by normalized Hamming distance. Every split puts 70 clips per class in training and 30 in testing, and clips from the same video file are never used for both. The authors note that the source-video constraint does not always allow a perfect balance of meta-labels. How the split files encode this, and what it implies for evaluation, is described in the HMDB51 article.

Normalization and stabilization

All clips were rescaled to a height of 240 pixels with the aspect ratio kept, converted to 30 frames per second and compressed with the DivX 5.0 codec through ffmpeg (§2.4). Because camera motion was expected to interfere with local motion features, every clip was also stabilized (§2.5). Salient features were detected and matched between adjacent frames using a distance that combines absolute pixel differences with what the paper calls the “Euler distance” of the points (presumably the Euclidean distance between their positions); RANSAC estimated the geometric transformation for each pair of neighbouring frames independently, and the frames were warped and combined into a stabilized clip, as in image stitching. The authors report inspecting many stabilized clips and finding that the method worked surprisingly well; Figure 3 shows the 1st, 30th and 50th frames of one clip before and after. The systems’ overall and shape-versus-motion results (Tables 3 and 6) are given for both versions.

Comparing datasets with low-level features

Section 3 compares HMDB51 with Hollywood, Hollywood2, UCF Sports, UCF YouTube and UCF50 by asking how well features that know nothing about actions can classify them. The reasoning is stated directly: in an unbiased action dataset, low-level features such as colour should not predict the action. Three kinds of features were used:

• Colour and grey values: the mean colour in HSV space on a 12 × 16 grid of each frame, combined with grey values and reduced by PCA (“Color+Gray+PCA” in Table 2).
• Gist, the coarse orientation-based scene descriptor of Oliva and Torralba, computed on three frames of each clip with the authors’ code.
• HOG/HOF spatio-temporal interest-point descriptors in a bag of visual words, with 2,000, 3,000, 4,000 and 5,000 words, reporting the best.

Each dataset was scored with its own splits where it had them and with 3- or 5-fold cross-validation otherwise. The “percent drop” in Table 2 is the relative loss from HOG/HOF to each low-level feature, $(a_{\mathrm{HOG/HOF}} - a_{\mathrm{low}}) / a_{\mathrm{HOG/HOF}}$ (the printed values agree with this formula to one decimal); a small drop means the low-level feature does nearly as well as the motion feature.

Dataset	Classes	Color+Gray+PCA	Drop	Gist	Drop	HOG/HOF
Hollywood	8	26.9%	16.7%	27.4%	15.2%	32.3%
UCF Sports	9	47.7%	18.6%	60.0%	−2.4%	58.6%
UCF YouTube	11	38.3%	35.0%	53.8%	8.7%	58.9%
Hollywood2	12	16.2%	68.7%	21.8%	57.8%	51.7%
UCF50	50	41.3%	13.8%	38.8%	19.0%	47.9%
HMDB51	51	8.8%	56.4%	13.4%	33.7%	20.2%

On UCF Sports, gist beat HOG/HOF, so the dataset could be classified by scene rather than action. The authors conjecture that sports are tied to places (ball games on grass, swimming in water, skiing on snow), and that YouTube videos in general carry low-level biases such as the vantage points amateur directors prefer, since colour and gist were also fairly predictive on UCF YouTube and UCF50. On the movie datasets the low-level features did worse relative to HOG/HOF, with Hollywood the exception, which they attribute to its small number of source movies, 12, whose clips share scenes. HMDB51 had the second-largest drops after Hollywood2. In our reading, the test is an early form of the measurements described in the dataset bias article.

The low-level test on synthetic clips

The script below repeats the logic of Table 2 on rendered clips. A grey disc moves right, left, up or down (the action) over a smooth coloured background (the place). In one dataset each action happens in its own kind of place, as sports do; in the other the place is drawn independently of the action, as the authors expected of clips from many films. Colour features follow the paper’s grid (mean HSV and grey per cell on a 12 × 16 grid, averaged over frames, then PCA); the motion feature is a global Lucas–Kanade translation estimate per frame pair, standing in for HOF. Both feed an RBF-kernel SVM trained on 70 clips per class and tested on 30, for five seeds.

import numpy as np
from scipy.ndimage import gaussian_filter
from matplotlib.colors import hsv_to_rgb, rgb_to_hsv
from sklearn.decomposition import PCA
from sklearn.svm import SVC

T, H, W = 10, 48, 64                       # frames; 48 x 64 pixels = a 12 x 16 grid of 4 x 4 cells
CLASSES = ["right", "left", "up", "down"]  # the action is the direction of motion
N_PER_CLASS, N_TRAIN = 100, 70             # 70 training and 30 test clips per class, as in HMDB51
yy, xx = np.mgrid[:H, :W]

def render_clip(rng, c, scene_hue):
    """A grey disc moving in direction c over a smooth coloured background whose hue is scene_hue."""
    hue = (scene_hue + 0.5 * gaussian_filter(rng.normal(size=(H, W)), 4)) % 1.0
    sat = np.clip(0.6 + 2.0 * gaussian_filter(rng.normal(size=(H, W)), 4), 0, 1)
    bg = hsv_to_rgb(np.stack([hue, sat, np.full((H, W), 0.7)], -1))
    dy, dx = {"right": (0, 1), "left": (0, -1), "up": (-1, 0), "down": (1, 0)}[CLASSES[c]]
    v = rng.uniform(1.0, 2.0)
    y0, x0 = rng.uniform(18, 30), rng.uniform(22, 42)
    grey = rng.uniform(0.1, 0.9)
    clip = np.empty((T, H, W, 3))
    for t in range(T):
        disc = (yy - y0 - dy * v * t) ** 2 + (xx - x0 - dx * v * t) ** 2 <= 6 ** 2
        clip[t] = np.where(disc[..., None], grey, bg) + 0.02 * rng.normal(size=(H, W, 3))
    return np.clip(clip, 0, 1)

def make_dataset(rng, scene_tied):
    """scene_tied: each class is filmed in its own kind of place (hue), as sports are;
    otherwise the place is drawn independently of the action."""
    clips, labels = [], []
    for c in range(len(CLASSES)):
        for _ in range(N_PER_CLASS):
            hue = (c / len(CLASSES) + rng.normal(0, 0.06)) if scene_tied else rng.uniform()
            clips.append(render_clip(rng, c, hue)); labels.append(c)
    return np.array(clips), np.array(labels)

def colour_features(clips):
    """Mean HSV and grey value on a 12 x 16 grid per frame, averaged over the clip (Table 2's
    'Color+Gray' before PCA; how the paper pooled frames is not stated)."""
    hsv = rgb_to_hsv(clips)
    grey = clips.mean(-1, keepdims=True)
    f = np.concatenate([hsv, grey], -1).reshape(len(clips), T, 12, 4, 16, 4, 4).mean((3, 5))
    return f.mean(1).reshape(len(clips), -1)

def motion_features(clips):
    """Global Lucas-Kanade translation between consecutive grey frames: a minimal motion cue."""
    g = clips.mean(-1)
    It = np.diff(g, axis=1)
    Iy, Ix = np.gradient(g[:, :-1], axis=(2, 3))
    A = np.stack([(Ix * Ix).sum((2, 3)), (Ix * Iy).sum((2, 3)), (Iy * Iy).sum((2, 3))], -1)
    b = np.stack([-(Ix * It).sum((2, 3)), -(Iy * It).sum((2, 3))], -1)
    det = A[..., 0] * A[..., 2] - A[..., 1] ** 2
    u = (A[..., 2] * b[..., 0] - A[..., 1] * b[..., 1]) / det
    w = (A[..., 0] * b[..., 1] - A[..., 1] * b[..., 0]) / det
    return np.concatenate([u, w], 1) * 1e3

def accuracy(rng, X, y, pca=False):
    test = np.zeros(len(y), bool)
    for c in np.unique(y):
        test[rng.permutation(np.flatnonzero(y == c))[N_TRAIN:]] = True
    Xtr, Xte = X[~test], X[test]
    if pca:
        p = PCA(n_components=20, svd_solver="full").fit(Xtr); Xtr, Xte = p.transform(Xtr), p.transform(Xte)
    mu, sd = Xtr.mean(0), Xtr.std(0) + 1e-9
    clf = SVC(kernel="rbf", C=10).fit((Xtr - mu) / sd, y[~test])
    return np.mean(clf.predict((Xte - mu) / sd) == y[test])

print(f"chance = {1 / len(CLASSES):.2f}")
for scene_tied in (True, False):
    res = []
    for seed in range(5):
        rng = np.random.default_rng(seed)
        clips, y = make_dataset(rng, scene_tied)
        a_col = accuracy(np.random.default_rng(100 + seed), colour_features(clips), y, pca=True)
        a_mot = accuracy(np.random.default_rng(100 + seed), motion_features(clips), y)
        res.append((a_col, a_mot, 100 * (a_mot - a_col) / a_mot))   # Table 2's percent drop
    r = np.array(res)
    name = "place tied to action" if scene_tied else "place independent  "
    print(f"{name}  colour+grey+PCA {r[:, 0].mean():.2f} ({r[:, 0].min():.2f}-{r[:, 0].max():.2f})  "
          f"motion {r[:, 1].mean():.2f} ({r[:, 1].min():.2f}-{r[:, 1].max():.2f})  "
          f"percent drop {r[:, 2].mean():.1f} ({r[:, 2].min():.1f} to {r[:, 2].max():.1f})")

Output:

chance = 0.25
place tied to action  colour+grey+PCA 0.84 (0.79-0.89)  motion 0.99 (0.98-1.00)  percent drop 15.4 (10.8 to 20.8)
place independent    colour+grey+PCA 0.46 (0.42-0.48)  motion 0.99 (0.97-1.00)  percent drop 53.4 (50.4 to 57.5)

The motion feature solves both datasets, so any difference comes from the colour features. When places are tied to actions, colour alone reaches 0.79 to 0.89 and the drop is 11 to 21%, the regime of UCF Sports and UCF50 in Table 2. When places are independent of actions, the drop rises to 50 to 58%, close to HMDB51’s 56.4% for the same kind of feature. Colour does not fall to chance, though: averaging the frames leaves a smear where the disc passed, and its position and shape give the direction away in part. In our reading this may also explain part of why Color+Gray+PCA scored 8.8% on HMDB51 rather than the 2% of chance: pooled appearance features still carry a trace of where and how the actor moved. The percent drop therefore separates datasets only relative to each other, and it depends on how strong the motion feature is: a weak motion baseline would shrink every drop.

The two baseline systems

The paper restricts itself to methods built on local space-time features, which it calls the dominant trend of the previous five years (§4). Wang et al. (BMVC 2009) had evaluated detectors and descriptors, grouped into six types, on KTH, UCF Sports and Hollywood2, and Laptev’s HOG/HOF had done best on the two datasets most like HMDB51’s sources, Hollywood2 and UCF Sports, so it became the first baseline. The second came from the authors’ group.

HOG/HOF (§4.1). With Laptev’s binaries, 3D Harris corners were detected in every clip (Harris corner detector extended to space-time) and described by concatenated HOG and HOF histograms. 100,000 descriptors sampled from the training set were clustered by k-means into $k$ = 2,000, 4,000, 6,000 or 8,000 visual words, each clip became a $k$-bin histogram of its nearest words, and an SVM with the RBF kernel $K(u, v) = \exp(-\gamma \lVert u - v \rVert^2)$ classified the histograms, with $C$ and $\gamma$ chosen by a greedy search under 5-fold cross-validation on the training set. The best vocabulary was 8,000 words for the original clips and 2,000 for the stabilized ones. To check the reimplementation, the authors report reproducing the HOG (81.4%) and HOF (90.7%) results that Wang et al. had reported on KTH.

C2 features (§4.2). The second system is a hierarchical model of the visual cortex in the HMAX family. Its ventral-stream version (Serre et al., 2007), intended for shape and object recognition, applies a pyramid of Gabor filters at several orientations and scales (S1, modelling simple cells), pools them locally over position and scale (C1, complex cells), compares the C1 output with a dictionary of $n$ prototypes learned from training images (S2, keeping the similarity between 0 and 1 rather than a quantized index, unlike a bag of words), and takes the maximum of each S2 response over all positions and scales to form an $n$-dimensional C2 vector. The dorsal-stream version of Jhuang et al. (2007) replaces the S1 filters with spatio-temporal ones whose receptive fields are not separable in space and time and which respond to directions of motion; its S2 units are tuned to optic-flow patterns rather than shape patterns. For HMDB51 the authors computed both kinds of C2 features and concatenated them. The paper motivates this system by a recent result of the same group, which it describes as matching human performance at recognizing mouse behaviour in the home cage, a different task; no human performance was measured on HMDB51.

The paper does not describe how C2 features were turned into clip labels. The authors’ public code, read for the HMDB51 article, classifies per-frame C2 vectors and gives each clip the majority label of its frames.

Results

Overall accuracy

Averaged over the three splits, both systems were slightly above 20% with chance at 2% (§5.1, Table 3):

System	Original clips	Stabilized clips
HOG/HOF	20.44%	21.96%
C2	22.83%	23.18%

The confusion matrices of Figure 4 show errors with no apparent pattern across classes. The authors found it most surprising that stabilization helped only marginally.

To ask whether the low scores reflected harder videos or simply more classes, the authors matched 10 HMDB51 classes to UCF50 classes (UCF50’s name first: basketball with shoot ball, biking with ride bike, diving with dive, golf swing with golf, horse riding with ride horse, pull ups, push-ups, rock climbing indoor with climb, walking with dog with walk, and fencing with a class the list calls “stab”) and ran HOG/HOF on both. Accuracy fell only from 66.3% on UCF50 to 54.3% on HMDB51, with chance at 10%, comparable to the same system’s 51.7% on Hollywood2’s 12 classes and 58.9% on UCF YouTube’s 11. The paper concludes that the number of classes, more than the difficulty of the clips, explains the low overall accuracy.

Robustness to nuisance factors

Section 5.2 breaks accuracy down by occlusion, camera motion, viewpoint and quality. Table 4 gives the breakdown for camera motion and quality:

System	Camera motion	No camera motion	Low quality	Medium	High
HOG/HOF	19.84%	19.99%	17.18%	18.68%	27.90%
C2	25.20%	19.13%	17.54%	23.10%	28.62%

Quality mattered most: both systems lost about 10 points from high- to low-quality clips. Occlusion and viewpoint, by the text, did not seem to influence performance. Camera motion separated the systems: HOG/HOF was unaffected, while C2 did better on clips with camera motion. The authors offer two explanations, that camera motion may increase the response of the S1 motion detectors or that camera motion may itself correlate with the action; against the second, a classifier given only the mean frame-to-frame motion parameters from the stabilization recognized 5.29% of clips.

A logistic regression then predicted whether each system’s answer on a clip was correct from the meta-labels (Table 5). Viewpoint was coded 0 for front and back and 1 for the two side views; occlusion 0 for a full-body view and 1 otherwise; quality as two binary variables with low quality as the reference. The coding of camera motion is not stated.

Coefficient	HOG/HOF $\beta$	$p$	Odds ratio	C2 $\beta$	$p$	Odds ratio
Intercept	−1.60	0.000	0.20	−1.52	0.000	0.22
Occluders	0.07	0.427	1.06	−0.22	0.007	0.81
Camera motion	−0.12	0.132	0.88	−0.43	0.000	0.65
Viewpoint	0.09	0.267	1.09	0.19	0.009	1.21
Medium quality	0.11	0.254	1.12	0.47	0.000	1.60
High quality	0.65	0.000	1.91	0.97	0.000	2.65

The text reads the table as confirming quality as by far the strongest factor, with systems “nearly twice as likely to be correct” on high-quality clips; as confirming that camera motion improves classification, significantly only for C2; and as showing no significant influence of occlusion or viewpoint on either system. Several of these readings do not match the table (see Limitations). In the conclusion, the authors summarize the analysis as showing both systems fairly robust to camera position, camera motion and occlusion, but limited in their power to represent the complexity of human actions. Logistic regression covers how coefficients map to odds ratios.

Shape versus motion

Section 5.3 splits each system into its shape and motion parts (Table 6):

Features	Original clips	Stabilized clips
HOG/HOF (shape + motion)	20.44%	21.96%
HOG (shape)	15.01%	15.47%
HOF (motion)	17.95%	22.48%
C2 motion + shape	22.83%	23.18%
C2 shape	13.40%	13.44%
C2 motion	21.96%	22.73%

Shape alone was much worse than motion alone in both systems, and combining them helped only moderately, mainly on the original clips; on the stabilized clips HOF alone beat HOG/HOF. The authors use this to qualify Schindler and Van Gool’s (2008) suggestion that local shape and flow in a single frame suffice to recognize actions: true, perhaps, for simple actions such as KTH’s, but not for HMDB51’s.

The conclusion places the roughly 23% best accuracy next to object recognition on Caltech-101, where performance had started around 16%, and calls HMDB51 a good place to start, though still far from the richness of video in films or online.

Impact

HMDB51 became, with UCF101, the standard pair of benchmarks for action recognition through the move from hand-crafted features to deep networks, and the protocol the paper set up, three fixed splits scored by mean accuracy, is the one later papers used. The earlier official page, first published on 29 July 2011 according to its change log, distributed the clips, the stabilized clips, the splits and the STIP features, asked users to cite this paper, and kept an evaluation table that was updated until November 2013.

Within two years the paper’s baselines were far behind. Wang and Schmid (2013) reached 57.2% with improved dense trajectories, following the original protocol on the original rather than the stabilized videos. In our reading, compensating camera motion inside the feature, as they did, succeeded where the paper’s stabilization of whole clips had brought only marginal gains. Simonyan and Zisserman (2014) used HMDB51 as the second benchmark of the two-stream network and listed the paper’s C2 system in their comparison as a spatio-temporal HMAX network at 22.8%. Their related-work section credits this paper with combining a spatial HMAX model with Jhuang et al.’s temporal one into ventral-like and dorsal-like streams, hand-crafted and shallow, and their §3.2 describes their own ConvNets as a learnable generalisation of HMAX. Jhuang and colleagues built J-HMDB on the dataset (Jhuang et al., 2013): 21 classes and 928 clips with every frame annotated by a 2D puppet model, with splits it describes as generated as in this paper (random groupings close to 70:30 that keep the clips of one video together). Carreira and Zisserman (2017) wrote that HMDB51 was purposefully built to be hard, giving the example of draw sword clips taken from the same videos as sword and sword exercise clips; after Kinetics pretraining, their two-stream I3D reached 80.2%. The HMDB51 article follows these milestones in more detail.

The Helmholtz Prize, presented by the PAMI Technical Committee at each ICCV and described by the committee as recognizing ICCV papers from ten years earlier with significant impact on computer vision research, went to this paper in 2021, together with the ORB and DTAM papers; the improved dense trajectories paper received it in 2023 (committee’s list of awardees, accessed October 2026).

Limitations

The paper states some limitations itself. Its conclusion calls HMDB51 far from capturing the richness and complexity of video in films or online, and §2.3 acknowledges that the source-video constraint does not always allow a perfect balance of meta-labels across the splits.

Other limitations come from the text, the tables and later work (our reading, except where a source is named):

• Three clip counts. The contributions paragraph gives 6,766 clips, the abstract “around 7,000” and the conclusion “just under 7,000”; the official pages have given 6,849 since at least 2014. The HMDB51 article compares the sources; no source we found explains the difference.
• Text and figures disagree on camera motion. Section 2.5 puts camera motion at about 2/3 of the clips, “as shown in Figure 2”, but Figure 2 gives 59.9%.
• Two HOG/HOF settings without a comparison. Section 3 used vocabularies of 2,000 to 5,000 words and reports 20.2% on HMDB51 (Table 2); Section 4.1 used 2,000 to 8,000 words and reports 20.44% (Table 3). The paper does not say whether the Table 2 figure is a three-split mean.
• The regression text does not match Table 5. The text says occlusion and viewpoint had no significant influence on either system, but C2’s coefficients have $p = 0.007$ and $0.009$. It reads the table as confirming that camera motion improves C2’s accuracy, but C2’s camera-motion coefficient is negative (odds ratio 0.65); if camera motion was coded 1 when present, as viewpoint and occlusion are coded 1 for the condition named, the table says camera motion lowered C2’s odds of being correct, against Table 4, and the paper does not state the coding that would settle it. “Nearly twice as likely to be correct on high vs. medium quality” compares odds, not probabilities, and the odds ratios of 1.91 and 2.65 are against low quality; high against medium is about 1.7 by the coefficients.
• Table 4 against Table 3 for HOG/HOF. By our arithmetic, weighting Table 4’s rows with the Figure 2 proportions gives about 19.9% for HOG/HOF in both breakdowns, below the 20.44% of Table 3, whereas C2’s rows average about 22.8% to 22.9%, in line with its 22.83%. Both camera-motion figures for HOG/HOF lie below the overall figure, which a pooled average over the same test clips could not produce; the paper does not say how Table 4 was computed, so the HOG/HOF rows may come from a different run or a different averaging (for example over classes within each condition).
• The joint experiment’s comparison. The 54% that single-frame joints are compared with comes from the “motion features” of §4.1, that is, HOG/HOF, which includes the HOG shape descriptor. The classes of the two experiments are also described differently: page 2 gives climb, climb stairs, run, walk and jump as examples, while §5.1 lists ten matched classes that include neither climb stairs, run nor jump. The 54% and 54.3% figures suggest the same experiment, but the paper does not say.
• A class name outside the list. The matched pair “fencing / stab” names a class that is not among the 51; HMDB51 has a fencing class.
• The Hollywood explanation. The paper attributes Hollywood’s small drops to its 12 source movies. In Laptev et al. (2008), the 12 movies supplied the training clips and the test clips came from 20 different movies, so a classifier of scenes would not find the training films in the test set; the explanation is at least incomplete.
• Cross-validation on grouped data. For datasets without official splits, Table 2 used 3- or 5-fold cross-validation, and the paper does not say which datasets those were or whether folds respected groups of clips. The paper itself notes (p. 2) that UCF50’s clips come in 25 groups per class that share features such as background and camera position, and the UCF50 page asks for leave-one-group-out testing (UCF101 article). If the folds ignored the groups, UCF50’s low-level scores would be inflated by near-duplicate scenes, the kind of leakage the paper’s own splits were designed to avoid.
• No human baseline. The paper evaluates two machine systems and validates labels with two observers per clip, but does not measure how accurately people classify the clips; the human comparison it mentions concerns mouse behaviour in another paper.
• Same scenes, different classes. Section 2.3 states the source-video rule once in general terms and once inside the per-class selection of 70 and 30 clips. Carreira and Zisserman (2017) noted that clips of different classes come from the same videos; whether such clips can fall on opposite sides of a split, the paper does not say.
• Reference list. Reference [18] prints its authors as “J. M. S. Maji, L. Bourdev” (the CVPR 2011 paper is by Maji, Bourdev and Malik), and [23] prints the volume and issue of Torralba et al. (2008) as 11(30) instead of 30(11).

What Came After

The dataset was described again in a chapter titled “HMDB51: A Large Video Database for Human Motion Recognition” in High Performance Computing in Science and Engineering ’12 (Kuehne, Jhuang, Stiefelhagen & Serre; published online October 2012, in the 2013 volume), with Rainer Stiefelhagen in place of Garrote and Poggio. Its abstract repeats the ICCV abstract except for the last sentence, which describes the dataset as a tool for evaluation instead of announcing the two-system evaluation; only the abstract was read for this article.

The paper’s low-level test anticipated a line of work on static bias in action datasets. RESOUND (Li, Li & Vasconcelos, 2018) measured the representation bias of several datasets from the accuracy of frozen per-frame object, scene and people features, and found HMDB51 less biased than UCF101 and Kinetics but far more than Diving48, a dataset the RESOUND authors assembled to have little such bias; Something-Something took the same direction by making classes differ in motion rather than objects or places. The dataset bias article covers these measurements.

The paper’s diagnosis that the number of classes, more than clip difficulty, limited accuracy was followed by larger datasets: UCF101 in 2012 with 101 classes, then Kinetics with hundreds, which turned HMDB51 from a training benchmark into a transfer target. In the later papers we read, the splits are what carried over; the meta-labels and the stabilized clips, central to this paper’s analysis, do not appear in their evaluations, and the improved trajectories paper states that it used the original clips. The results traced in the HMDB51 article are reported that way. The licence and the current form of the official page are also described there.