Kinetics

Kinetics is a series of human action video datasets built at DeepMind. Will Kay, João Carreira, Andrew Zisserman and colleagues released the first version in May 2017 with 400 action classes, each with at least 400 clips of about 10 seconds according to its abstract, every clip cut from a different YouTube video. Larger editions followed: Kinetics-600 (2018), Kinetics-700 (2019) and a replenished Kinetics-700-2020. The first version, now called Kinetics-400, became the standard benchmark for action recognition and the usual pretraining set for video models, as ImageNet is for images. Kinetics is distributed as YouTube links, not video files, so the downloadable copy shrinks over time.

Purpose

Kay et al. (2017) built Kinetics because HMDB51 and UCF101 were too small and too little varied to train and compare deep video models, and took ImageNet, where networks are trained for classification and then reused, as the model. Their stated aim was “a large scale high quality dataset, covering a diverse range of human actions”, for classification rather than temporal localization, so it contains only short clips around the action. One clip per video was a design goal; UCF101’s 13,320 clips come from about 2,500 videos.

Dataset Size

Split totals are as listed in the Kinetics-700-2020 note (Smaira et al., 2020, Table 2).

Version (report)	Classes	Training clips per class	Training	Validation	Test
Kinetics-400 (Kay et al., 2017)	400	250–1,000	246,245	20,000	40,000
Kinetics-600 (Carreira et al., 2018)	600	450–1,000	392,622	30,000	60,000
Kinetics-700 (Carreira et al., 2019)	700	450–1,000	545,317	35,000	70,000
Kinetics-700-2020 (Smaira et al., 2020)	700	at least 700 clips per class in total	545,793	34,256	67,858

In the first three versions, validation and test sets have 50 and 100 clips per class. Kinetics-400 totals 306,245 clips, Kinetics-600 495,547 and Kinetics-700 650,317. The per-class counts in the Kinetics-400 appendix sum to 306,245 and range from 378 (“taking a shower”) to 1,150, with five classes below 400; the abstract says at least 400 clips per class and the text 400–1,150. Clips have variable resolution and frame rate and keep their sound, which annotators did not hear.

Classes

Kay et al. (2017) group the human actions into actions of one person (drawing, laughing), person–person actions (hugging, shaking hands) and person–object actions (mowing lawn, washing dishes). Some classes need motion to tell apart (swimming strokes), others the object (which wind instrument). There is no deep hierarchy, only 38 non-exclusive groupings in the appendix, such as music, cooking and dancing.

The later versions are approximate supersets. Kinetics-600 kept 368 of the 400 classes unchanged and renamed, split or removed the others, for example splitting “hugging” into “hugging baby” and “hugging (not baby)” (Carreira et al., 2018). Kinetics-700 kept 597 of the 600 and added classes taken partly from the verb lists of EPIC-KITCHENS and AVA (Carreira et al., 2019). Kinetics-700-2020 kept the classes and topped up the 123 rarest (Smaira et al., 2020).

Annotation Types

Each clip carries one action label and a time interval in its source video. The Kinetics-400 pipeline (Kay et al., 2017):

1. Candidate videos. YouTube videos whose titles matched a class name.
2. Clip position. Image classifiers for the action, trained from Google Image Search feedback, were run on the frames, and a 10-second clip was cut around each of the two strongest responses.
3. Verification. Workers on Amazon Mechanical Turk answered whether a human performs the named action; a clip needed at least 3 positive answers out of up to 5, and two known clips in each task of 20 measured worker accuracy.
4. Cleaning. One clip was kept per video; near-duplicates across videos, such as compilations, were removed by clustering Inception-v1 features, a further 15% of approved clips; classes that two-stream classifiers confused were merged, split or removed; a final manual pass reviewed the least confident clips.

Kinetics-600 also sourced classes from Google’s Knowledge Graph and YouTube autocomplete, and matched videos with several queries per class, in English and Portuguese, scored against weighted n-grams of the metadata (Carreira et al., 2018). Kinetics-700 machine-translated the queries into French, Portuguese and Spanish and crowdsourced the final class cleaning, previously done by the authors (Carreira et al., 2019).

Splits

Kinetics-400 test labels were at first withheld and scored by an external server; they were released in August 2018 with the Kinetics-600 note, whose dataset adds a held-out test set for the ActivityNet challenge (Carreira et al., 2018). Kinetics-700 released validation labels and held out its test set, and asks for results on validation (Carreira et al., 2019).

The versions overlap. Some Kinetics-400 validation clips became Kinetics-600 test clips and some training clips its validation clips, so evaluating a Kinetics-400-pretrained model on Kinetics-600 is “not ideal”, though the authors expect almost no difference in practice (Carreira et al., 2018). The revised Kinetics-700 note (arXiv v2, 2022) adds that many Kinetics-600 validation and test clips are in the Kinetics-700 training set, so training on Kinetics-700 and evaluating on Kinetics-600 is not safe.

Availability and Dataset Decay

The official releases contain YouTube IDs with start and end times, and users download the clips themselves, so clips vanish when uploaders delete videos or make them private. The Kinetics-700 note estimates removals at about 3% per year (Carreira et al., 2019), and Smaira et al. (2020, Table 2) counted what remained on 14 October 2020: 220,033 of the 246,245 Kinetics-400 training clips (89%), 18,059 of 20,000 validation clips and 35,400 of 40,000 test clips; Kinetics-600 and Kinetics-700 had kept 95–98% of their training clips. On that date the smallest Kinetics-400 class had 303 clips (Table 1).

Papers train on different copies: the Non-local paper describes about 246k training videos (Wang et al., 2018), SlowFast about 240k (Feichtenhofer et al., 2019), and X3D about 240k training, 20k validation and 35k test videos (Feichtenhofer, 2020).

The example below simulates decay on synthetic video. A dataset of 1,000 clips, $8 \times 16 \times 16$ frames of a bright square moving left, right, up or down in noise, is classified by a fixed motion detector that correlates each frame with its predecessor shifted one pixel in each direction. Each clip has a difficulty $d \in [0, 1]$ that sets its noise level, and disappears after an exponential waiting time whose rate is set by the class (1–8% per year on average) and grows with $d$, so that noisier clips vanish sooner. The control keeps the same class rates without the dependence on difficulty.

import numpy as np

T, S = 8, 16                                   # frames per clip, frame size (pixels)
CLASSES = ["moving left", "moving right", "moving up", "moving down"]
STEP = {0: (0, -1), 1: (0, 1), 2: (-1, 0), 3: (1, 0)}      # (dy, dx) per frame
BASE_RATE = np.array([0.01, 0.02, 0.04, 0.08])  # removal hazard per year, by class
YEARS = [0, 1, 2, 3, 5]

def make_dataset(n_per_class, rng):
    """Clips of a bright 3x3 square moving one pixel per frame in noise.
    Each clip has a difficulty d in [0, 1] that sets its noise level."""
    labels = np.repeat(np.arange(4), n_per_class)
    d = rng.uniform(0, 1, labels.size)
    clips = np.zeros((labels.size, T, S, S))
    for i, (c, di) in enumerate(zip(labels, d)):
        y, x = rng.integers(4, S - 4, 2)
        dy, dx = STEP[c]
        for t in range(T):
            yy, xx = (y + dy * (t - T // 2)) % S, (x + dx * (t - T // 2)) % S
            for oy in (-1, 0, 1):
                for ox in (-1, 0, 1):
                    clips[i, t, (yy + oy) % S, (xx + ox) % S] = 1.0
        clips[i] += rng.normal(0, 0.3 + 1.0 * di, (T, S, S))
    return clips, labels, d

def classify(clips):
    """A fixed motion classifier: correlate each frame with the next one shifted
    by one pixel in each direction, and pick the direction that matches best."""
    scores = np.zeros((len(clips), 4))
    for c, (dy, dx) in STEP.items():
        shifted = np.roll(clips[:, :-1], (dy, dx), axis=(2, 3))
        scores[:, c] = (shifted * clips[:, 1:]).sum(axis=(1, 2, 3))
    return scores.argmax(1)

def removal_times(labels, d, rng, coupled=True):
    """Year in which each clip disappears: exponential, with a hazard that depends
    on the class and, if coupled, grows with difficulty (noisier clips vanish sooner)."""
    hazard = BASE_RATE[labels] * ((0.25 + 1.5 * d) if coupled else 1.0)
    return rng.exponential(1 / hazard)

def run(seed, coupled=True, report=False):
    rng = np.random.default_rng(seed)
    clips, labels, d = make_dataset(250, rng)
    correct = classify(clips) == labels
    gone = removal_times(labels, d, rng, coupled)
    acc = []
    for year in YEARS:
        alive = gone > year
        acc.append(correct[alive].mean())
        if report:
            frac = " ".join(f"{alive[labels == c].mean():.2f}" for c in range(4))
            print(f"year {year}: {alive.sum():4d} clips ({alive.mean():.3f}), per class {frac}, "
                  f"hard half {(d[alive] > 0.5).mean():.3f}, accuracy {acc[-1]:.3f}")
    return np.array(acc)

print("classes:", ", ".join(CLASSES), "| removal per year:", BASE_RATE)
run(0, report=True)
for coupled in (True, False):
    acc = np.array([run(seed, coupled) for seed in range(10)])
    shift = acc[:, -1] - acc[:, 0]
    print(f"removal {'tied to' if coupled else 'independent of'} difficulty, seeds 0-9, "
          f"accuracy at year 5 minus year 0: {shift.min():+.3f} to {shift.max():+.3f}")

Output:

classes: moving left, moving right, moving up, moving down | removal per year: [0.01 0.02 0.04 0.08]
year 0: 1000 clips (1.000), per class 1.00 1.00 1.00 1.00, hard half 0.527, accuracy 0.680
year 1:  954 clips (0.954), per class 0.97 0.98 0.95 0.92, hard half 0.522, accuracy 0.686
year 2:  926 clips (0.926), per class 0.97 0.96 0.91 0.86, hard half 0.521, accuracy 0.692
year 3:  898 clips (0.898), per class 0.97 0.94 0.88 0.80, hard half 0.512, accuracy 0.697
year 5:  853 clips (0.853), per class 0.95 0.92 0.83 0.72, hard half 0.504, accuracy 0.699
removal tied to difficulty, seeds 0-9, accuracy at year 5 minus year 0: +0.009 to +0.026
removal independent of difficulty, seeds 0-9, accuracy at year 5 minus year 0: -0.007 to +0.006

After three years 89.8% of the clips remain, close to the Kinetics-400 figure, but “moving down” has lost 20% and “moving left” 3%, so the class balance shifts too. The fixed classifier’s accuracy rises from 68.0% on the full set to 69.9% on the copy available after five years, because the hard clips leave faster; a group downloading in year 1 would measure 68.6%. Over seeds 0–9 the gain is 0.9 to 2.6 points, against −0.7 to +0.6 when removal ignores difficulty. We found no measurement of whether removal correlates with difficulty in real Kinetics; the example only shows that a shift of one to three points can come from the copy rather than the model. In our reasoning, the remedy is to report the date and the clip count of the copy used, or to evaluate on a shared list of clip IDs.

Evaluation

Results are top-1 and top-5 accuracy. Because a clip has one label even when it shows another Kinetics action (“texting” while “driving a car”), Kay et al. (2017) consider top-5 more suitable. Papers such as Non-local Networks report on the validation set (Wang et al., 2018), while I3D’s own Kinetics-400 numbers are on the test set (Carreira & Zisserman, arXiv v3); action recognition covers this pitfall and test-time views. ActivityNet’s Kinetics tasks scored mean top-1 and top-5 error (task pages, 2018–2020).

License

The archived DeepMind page (21 September 2023) states that the dataset is licensed by Google Inc. under a Creative Commons Attribution 4.0 International License; it does not say how this applies to the videos, which are third-party YouTube uploads. As of October 2026 the page’s addresses redirect to the Google DeepMind home page, while the annotation archives it linked still respond.

Limitations and Bias

• Demographic balance. Kay et al. (2017) report a preliminary study: in 340 of the 400 classes no single gender dominates or gender cannot be determined, while classes such as “shaving beard” (mostly male) and “cheerleading” (mostly female) are imbalanced. They found little evidence of classifier bias by gender, but some by age: many videos of non-babies “crying” were misclassified.
• Geography. Of the about 90% of videos with an upload location, 59.0% of Kinetics-400 came from North America and 0.8% from Africa; Latin America rose from 3.4% to 7.7% in Kinetics-700-2020, which the authors link to Portuguese and Spanish queries (Smaira et al., 2020, Table 3).
• Static and object bias. Li, Li and Vasconcelos (2018, Table 1) trained linear classifiers on frozen per-frame features of networks trained for objects, scenes or people attributes. Kinetics had the highest object and people bias, the log of accuracy over chance, of the eight datasets measured (4.51 for objects, against 1.47 for KTH), and the authors note that its size did not reduce the bias.
• Label noise and ambiguity. Labels are not exhaustive, and classes overlap: the two-stream baseline confused “riding mule” with “riding or walking with horse” in 40% of cases (Kay et al., 2017, Table 5). The Kinetics-700 authors suggest that crowdsourced cleaning may have left classes less homogeneous, one possible reason why I3D’s test top-1 fell to 57.3%, from 68.4% on Kinetics-400 (Carreira et al., 2019).
• Short, continuous actions. The collection favors actions that last the whole clip, such as juggling, over actions with a beginning and end, such as getting out of a car (Carreira et al., 2019).

Common Uses

• Benchmarking action recognition models, from I3D and SlowFast to video transformers.
• Pretraining before fine-tuning on smaller datasets such as UCF101 and HMDB51, as I3D showed (transfer learning), or without labels, as in VideoMAE.
• Further annotation. The Kinetics page also distributes the annotations of AVA-Kinetics and Countix (Smaira et al., 2020), which the CVDF README describes as using Kinetics-700 videos.

Historical Milestones

• 2017. Baselines trained from scratch on Kinetics-400 reached 61.0% top-1 and 81.3% top-5 on the test set with a two-stream ResNet-50 (Kay et al., 2017, Table 4).
• 2017–2019. The ActivityNet Kinetics challenge winners’ error fell from 12.4% on Kinetics-400 (2017) to 11.0% on Kinetics-600 (2018); in 2019, on Kinetics-700, the best team reached 17.9% (Carreira et al., 2019).

Related Datasets

UCF101 and HMDB51 are the smaller predecessors; Something-Something (classes that differ in motion) and AVA (per-person labels in movies) are compared with Kinetics in the video understanding hub.

Resources

• Kinetics page (archived): DeepMind’s page, with the annotation archives of all versions and the licence.
• cvdfoundation/kinetics-dataset: video files for Kinetics-400 and Kinetics-700-2020 hosted by the Common Visual Data Foundation, a non-profit, with download scripts (as of October 2026). Its Kinetics-700-2020 readme lists 536,699 of 545,793 training clips, downloaded mostly in September 2020. This is a copy made by a third party; the archived DeepMind page does not link to it.