UCF101

UCF101 is a dataset of 13,320 short video clips of 101 human actions, collected from YouTube by Khurram Soomro, Amir Roshan Zamir and Mubarak Shah at the Center for Research in Computer Vision (CRCV) of the University of Central Florida and described in a technical report of November 2012. With HMDB51, it was the standard benchmark for action recognition through the years in which deep video models emerged, from two-stream networks to I3D. Its clips come in groups cut from the same source video, and its three official splits keep each group on one side: the rule that matters most when using it.

Purpose

The report names two weaknesses of earlier action datasets: few classes (KTH 6, Weizmann 9, UCF Sports 9, IXMAS 11), and unrealistic recording, either staged by actors or filmed by professional crews. Web-video datasets had addressed realism, but the largest, HMDB51 (6,766 clips, 51 actions) and UCF50 (6,681 clips, 50 actions), still had about 50 classes. UCF101 roughly doubled both counts with user-uploaded footage showing camera motion, varied lighting, partial occlusion and low-quality frames.

Dataset Size

The report’s summary table (Table 1):

Property	Value
Actions	101
Clips	13,320
Groups per action	25
Clips per group	4–7
Clip length	mean 7.21 s, min 1.06 s, max 71.04 s
Total duration	1,600 min (the abstract says 27 hours)
Frame rate, resolution	25 fps, 320 × 240
Audio	yes, for the 51 new actions

Clips are .avi files compressed with the DivX codec. The official page distributes them as a single RAR archive. Counting the official split lists, we find 100 to 167 clips per class.

Classes

The authors divided the classes into five types: human–object interaction (applying lipstick, mopping the floor), body motion only (push-ups, tai chi), human–human interaction (haircut, head massage), playing musical instruments (cello, tabla), and sports (pole vault, rafting). Fifty classes come from UCF50, the other 51 are new. Each file name encodes the class, group and clip: v_ApplyEyeMakeup_g03_c04.avi is clip 4 of group 3.

Annotation Types

• One action label per clip. Clips are trimmed to the action.
• Group membership, in the file name. Clips of a group can share background or viewpoint, and they come from one long video (official page, as of August 2026).
• Human bounding boxes for 24 classes. Released for the THUMOS 2013 challenge as frame-by-frame boxes for humans; the THUMOS 2013 download page lists the file as updated on 3 September 2015. The UCF101 page’s “Revised Annotations” link points to the THUMOS 2015 download page, which also lists the boxes.
• Precomputed features: STIP features on the dataset page; STIP, SIFT and dense-trajectory bag-of-words features, and class-level attributes, on the THUMOS 2013 page.

Splits

The official page provides three train/test splits for recognition and asks that published results use them, so that experiments are comparable. Our count of the split lists: split 1 has 9,537 training and 3,783 test clips, split 2 has 9,586 and 3,734, split 3 has 9,624 and 3,696. Each test set is a block of seven groups in every class (groups 1–7, 8–14 and 15–21); groups 22–25 are always in training. The page explains why groups must not be split: a group’s clips come from a single long video, so placing some in training and others in testing inflates accuracy.

A separate set of three detection splits covers the 24 annotated classes, 3,207 videos, with the same group blocks (our count). There is no official validation set.

The report itself recommended a different protocol, 25-fold leave-one-group-out cross-validation, and computed its baseline that way. The official page lists the same baseline at 43.90% under the three splits, while the report prints 44.5%.

Tasks

• Action recognition: 101-way classification of trimmed clips, the main task.
• Spatio-temporal action localization on the 24-class subset, usually called UCF101-24: a box around the actor in every frame where the action happens, and the action’s temporal extent. THUMOS 2013 included this task (Idrees et al., 2017, arXiv v1); see spatio-temporal action detection.
• Training data for temporal action localization: THUMOS 2014 used UCF101’s trimmed clips as its training set and untrimmed videos for validation and testing (temporal action localization).

Evaluation

Recognition is scored by accuracy, the mean over the three splits (Simonyan & Zisserman, 2014); ablations often report split 1 alone. Action recognition discusses how accuracy is averaged and other comparison pitfalls.

For UCF101-24, Weinzaepfel, Harchaoui and Schmid (2015) evaluated on split 1 only, scoring spatio-temporal detections by mean average precision at several IoU thresholds; most of the 24 classes, they note, cover almost the whole video.

Simulating group leakage

The code builds a synthetic dataset with UCF101’s structure: 8 classes × 25 groups × 4–7 clips. Each clip is 8 frames of 32 × 32 pixels in which a bright square moves in one of eight directions (the class), at a random speed from a random start. All clips of a group share a textured background, the group’s fingerprint, as a scene or viewpoint does. A 1-nearest-neighbour classifier is tested two ways: with groups 1–7 held out, as in UCF101’s split 1, and with the same number of test clips drawn at random. It runs on raw pixels and on a motion feature, the mean displacement of the square estimated from frame differences, in which the static background cancels.

import numpy as np
from scipy.ndimage import gaussian_filter

T, H, W = 8, 32, 32                       # frames, height, width of each tiny clip
DIRS = np.array([(0, 1), (1, 1), (1, 0), (1, -1), (0, -1), (-1, -1), (-1, 0), (-1, 1)])  # (dy, dx)
N_CLASSES, N_GROUPS = len(DIRS), 25       # class = direction in which a square moves
TEST_GROUPS = range(1, 8)                 # like UCF101 split 1: groups 1-7 are the test set

def make_dataset(rng):
    """Clips of a bright square moving over a textured background. All clips of one
    (class, group) share the background: the group's 'fingerprint' (same scene)."""
    clips, labels, groups = [], [], []
    for c in range(N_CLASSES):
        for g in range(1, N_GROUPS + 1):
            background = gaussian_filter(rng.normal(size=(H, W)), 2.0)
            background = 0.5 * background / background.std()
            for _ in range(rng.integers(4, 8)):            # 4-7 clips per group
                v = DIRS[c] * rng.uniform(0.8, 1.6)         # speed varies per clip
                y0, x0 = rng.uniform(12, 20, size=2)         # start position varies per clip
                clip = np.repeat(background[None], T, axis=0) + 0.2 * rng.normal(size=(T, H, W))
                for t in range(T):
                    y, x = np.round([y0 + v[0] * t, x0 + v[1] * t]).astype(int)
                    clip[t, max(y - 1, 0):y + 2, max(x - 1, 0):x + 2] += 1.5   # 3x3 square
                clips.append(clip); labels.append(c); groups.append(g)
    return np.array(clips), np.array(labels), np.array(groups)

def motion_feature(clips):
    """Mean displacement of the square: centroid of where intensity rose minus centroid
    of where it fell, between consecutive frames. The static background cancels out."""
    d = np.diff(clips, axis=1)
    yy, xx = np.mgrid[:H, :W]
    def centroid(w):
        s = w.sum(axis=(2, 3)) + 1e-9
        return np.stack([(w * yy).sum(axis=(2, 3)) / s, (w * xx).sum(axis=(2, 3)) / s], -1)
    pos, neg = np.clip(d - 1.0, 0, None), np.clip(-d - 1.0, 0, None)   # ignore pixel noise
    return (centroid(pos) - centroid(neg)).mean(axis=1)                 # (N, 2): (vy, vx)

def nn_accuracy(X, y, test):
    """1-nearest-neighbour (Euclidean) accuracy on the test clips."""
    Xtr, ytr, Xte, yte = X[~test], y[~test], X[test], y[test]
    d2 = (Xte ** 2).sum(1)[:, None] - 2 * Xte @ Xtr.T + (Xtr ** 2).sum(1)[None]
    return np.mean(ytr[d2.argmin(1)] == yte)

results = []
for seed in range(10):
    rng = np.random.default_rng(seed)
    clips, y, g = make_dataset(rng)
    group_test = np.isin(g, TEST_GROUPS)                       # whole groups held out
    random_test = np.zeros(len(y), bool)                       # same number of test clips,
    random_test[rng.permutation(len(y))[:group_test.sum()]] = True   # drawn clip by clip
    pixels, motion = clips.reshape(len(y), -1), motion_feature(clips)
    row = [nn_accuracy(F, y, s) for F in (pixels, motion) for s in (random_test, group_test)]
    results.append(row)
    if seed == 0:
        print(f"{len(y)} clips, {N_CLASSES} classes x {N_GROUPS} groups, "
              f"{group_test.sum()} test clips; chance = {1 / N_CLASSES:.3f}")

r = np.array(results)
names = ["pixels, random clip split", "pixels, group split      ",
         "motion, random clip split", "motion, group split      "]
for name, col in zip(names, r.T):
    print(f"{name}  mean {col.mean():.3f}  range {col.min():.3f}-{col.max():.3f}  (seeds 0-9)")

Output:

1139 clips, 8 classes x 25 groups, 324 test clips; chance = 0.125
pixels, random clip split  mean 0.995  range 0.963-1.000  (seeds 0-9)
pixels, group split        mean 0.164  range 0.101-0.251  (seeds 0-9)
motion, random clip split  mean 0.971  range 0.954-0.987  (seeds 0-9)
motion, group split        mean 0.974  range 0.964-0.984  (seeds 0-9)

On raw pixels, the random split scores 0.995 on average over seeds 0–9 and the group split 0.164, barely above the chance level of 0.125. With a random split, almost every test clip has a sibling from its group in the training set, and the nearest neighbour is that sibling because the shared background dominates the pixel distance; the classifier recognizes the scene, not the motion. Holding out whole groups removes the siblings and exposes the failure. The motion feature scores the same under both splits (0.971 and 0.974), because it contains no background to memorize. In our reading, the gap between a random and a group split measures how much of a score comes from group-specific appearance.

The real case looks the same. On UCF50, whose clips are grouped the same way, Sadanand and Corso (2012, Table 4) reported 76.4% with 10-fold video-wise cross-validation and 57.9% with 5-fold group-wise cross-validation for the same Action Bank features, and the UCF50 page lists both with a note that video-wise splits put similar clips on both sides.

License

Neither official UCF101 page (archived copies of August and October 2026) states a license or terms of use; the earlier page asks users to cite the technical report. The clips are from YouTube. The license is therefore unclear; check with the authors before uses other than research.

Limitations and Bias

• Small. About 9.5K training clips per split are too few to train large networks from scratch. On split 1, Simonyan and Zisserman’s spatial network reached 52.3% trained from scratch and 72.7% when ImageNet-pretrained with only the last layer retrained (Table 1); I3D and Kinetics moved the field to video pretraining for this reason.
• Saturated. Accuracy rose from the report’s 44.5% to 85.9% for the best THUMOS 2013 entry (three splits) and to 97.9% for two-stream I3D pretrained on Kinetics (Carreira & Zisserman, 2017, Table 4; 98.0% with ImageNet and Kinetics pretraining in arXiv v3, Table 5). With about 2% error left, differences between strong models are small.
• Groups and near-duplicates. Clips of a group come from one long video, so 13,320 clips are 2,525 groups, and the number of independent samples is closer to the second figure (our reasoning). Any split other than the official one, including a validation set carved at random from training data, can leak groups, as the simulation shows.
• Scene and object bias. Li, Li and Vasconcelos (2018) illustrate static bias with UCF101 classes: playing piano is the only class with pianos (object bias), and basketball dunk and soccer juggling can be told apart by the court or field (scene bias). In their measurements with frozen per-frame features (Table 1), UCF101 has the second-largest average bias of the eight datasets compared and is the most biased toward scene features. The report already attributed the higher baseline accuracy of sports classes partly to less cluttered backgrounds.
• Format. Trimmed, single-label clips at 320 × 240 do not test finding actions in long videos or recognizing several at once, which THUMOS 2014 and later datasets addressed.

Common Uses

• Fine-tuning benchmark after pretraining. With miniKinetics pretraining, two-stream I3D (ImageNet-initialized) on split 1 rose from 93.4% trained on UCF101 only to 96.5% fine-tuned end to end (Carreira & Zisserman, 2017, Table 3); see transfer learning.
• Feature evaluation. C3D, trained on Sports-1M, was evaluated with a linear SVM on its features: 82.3% for one network and 85.2% for three over the three splits (Tran et al., 2015; 3D convolutional networks).
• Self-supervised pretraining on small data, as in VideoMAE.
• Training set of THUMOS 2014 and 2015, and the source of UCF101-24 for spatio-temporal action detection.

Historical Milestones

• 2012. The report and a bag-of-words baseline: Harris3D interest points with HOG/HOF descriptors, a 4,000-word codebook and a histogram-intersection SVM, at 44.5% overall (25-fold leave-one-group-out).
• 2013. THUMOS 2013 at ICCV used UCF101’s three splits; the best of 16 entries reached 85.9% (THUMOS 2013 page). Idrees et al. (2017, arXiv v1) summarize the rise as from 45% to almost 90%.
• 2014. Two-stream networks reached 88.0% (three splits, SVM fusion; Simonyan & Zisserman, 2014, Table 4), comparable to hand-crafted dense trajectories.
• 2017. Two-stream I3D with Kinetics pretraining reached 97.9% (Carreira & Zisserman, 2017, Table 4).

Related Datasets

UCF101 is the last of four UCF action datasets, each including its precursor: UCF Sports, UCF11, UCF50 and UCF101 (Soomro et al., 2012). UCF Sports (Rodriguez, Ahmed & Shah, 2008) collected sports actions of the kind shown on channels such as the BBC and ESPN, at 720 × 480; the paper lists nine actions in “over 200” sequences and scored them by leave-one-out cross-validation, the dataset page (archived August 2026) lists 10 actions and 150 sequences with actor bounding boxes, and the UCF101 report counts 9 actions and 182 clips. UCF50 has 50 YouTube action classes whose clips are grouped as in UCF101 (25 groups of more than four clips per class), and its page asks for leave-one-group-out cross-validation. HMDB51 (6,766 clips, 51 actions) was the other benchmark of the period. Kinetics replaced both as the large-scale benchmark and source of pretraining; Something-Something targets the motion that UCF101’s scene bias lets models skip.

Resources

• UCF101 dataset page: videos, recognition and detection splits, the report, and the group rule.
• THUMOS 2013 downloads: the 24-class bounding boxes, features and attributes.
• UCF101 technical report on arXiv.