Something-Something

Something-Something is a video dataset of people performing simple actions with everyday objects, each video labeled with a caption template such as “Pushing [something] from left to right”. It was collected by the company Twenty Billion Neurons (TwentyBN) and introduced by Goyal et al. at ICCV 2017 with 108,499 videos in 174 classes. The second version, released in 2018 with 220,847 videos, is usually called Something-Something V2 (SSv2) and is now distributed by Qualcomm. Because its classes differ in how hands and objects move rather than in what is visible, it became the standard benchmark for the temporal side of action recognition.

Purpose

Goyal et al. (2017) argued that popular action datasets label high-level activities (sports, “getting out of a car”) that features from a still-image network often classify well, so they do not push a model to learn motion, object properties or intuitive physics. They wanted labels for fine-grained physical situations, and instead of labeling existing footage they asked crowd workers to film themselves acting out given labels (“crowd-acting”), an approach Charades had used before.

Dataset Size

Version	Videos	Train / validation / test	Classes	Frames as distributed
V1 (2017)	108,499	86,017 / 11,522 / 10,960	174	JPG images, 100 px high, extracted at 12 frames per second
V2 (2018)	220,847	168,913 / 24,777 / 27,157	174	WebM (VP9) videos, 240 px high

The V1 counts are from the paper and the archived V1 page. The paper gives clips of 2 to 6 seconds (4.03 on average) from 1,133 crowd workers, and 77 to 986 videos per class, 620 on average.

The V2 counts are from TwentyBN’s archived page of 2018 and Qualcomm’s current page (as of October 2026); the archived per-class counts, 115 to 4,081 videos, sum to 220,847. Qualcomm’s page credits more than 1,300 crowd actors; its summary table still lists “Quality 100px” and 12 frames per second, while its text describes 240-pixel files, so the table may describe V1. Test labels are withheld in both versions.

Classes

The 174 classes are caption templates in which “something” stands for an object: “Putting something into something”, “Tearing something into two pieces”, “Something falling like a rock”. V2 keeps V1’s categories (Mahdisoltani et al., 2018); the two official class lists contain the same 174 names.

Two design choices make the classes hard to tell apart from appearance:

• Groups of similar actions. Members of an action group differ in small visual details, such as putting something on top of, next to or behind something, or poking something so lightly that it barely moves, so that it slightly moves, or so that it falls over; workers were encouraged to perform several actions of a group with the same object. The aim was to stop a network from “cheating” by reading the object type from a frame, or the action from cues such as hand position, overall velocity or camera shake (Goyal et al., 2017). Mahdisoltani et al. (2018) count 50 groups.
• Pretending. Some groups pair an action with a pretended version, such as “Putting something behind something” and “Pretending to put something behind something”. Telling them apart requires watching the object, not the hand. In the V2 class list, 22 of the 174 classes begin with “Pretending”.

Some templates are mirror pairs, such as “Pushing something from left to right” and “… from right to left”, or opposites, such as “Covering something with something” and “Uncovering something”; a few move the camera instead of the object (“Approaching something with your camera”).

Annotation Types

Each video has one template label. When uploading, workers typed the object words for its placeholders, so “Picking [something] up” becomes, for example, “Picking a shoe up” (Goyal et al., 2017). The V1 paper counts 23,137 distinct object names, variants of case and wording included, and estimates at least a few thousand distinct objects.

For the V2 training and validation videos, the official pages state 318,572 object annotations involving 30,408 unique objects. To reduce label noise, five crowd workers checked each V2 video against its label, and only videos all five accepted were kept.

Something-Else (Materzynska et al., 2020) added boxes: hands and the objects involved, in 8,183,381 frames of 180,049 SSv2 videos, 16,963,135 boxes in total.

Splits

V1 was split 8:1:1 into training, validation and test sets, with all videos from one worker in the same split (Goyal et al., 2017). The official V2 pages do not describe the split; Materzynska et al. (2020) report that it also keeps each person’s videos together.

Tasks

• Template classification into 174 classes, the standard use.
• Captioning: generating the filled-in template (Mahdisoltani et al., 2018).
• Compositional and few-shot recognition on the Something-Else splits.

Evaluation

Results are top-1 and top-5 accuracy over the 174 classes. Since the test labels are withheld, papers report the validation set; TwentyBN ran a test-set leaderboard, visible on the archived 2018 page, and Qualcomm’s current page links none (as of October 2026). Compare results at equal test-time views (action recognition explains views and lists SSv2 results).

The paper’s baselines. Goyal et al. (2017) reported error rates. On easier 10- and 40-class subsets, concatenated 2D and 3D CNN features had top-1 errors of 44.9% and 63.8% (Table 4). On all 174 classes, a 3D CNN pretrained on Sports-1M and on the 40-class subset had 88.5% top-1 error, that is, 11.5% accuracy, and a top-5 error of 70.0% in Table 4; the text gives 70.3%. Ten people classifying about 700 test videos informally reached about 60%.

Why Order and Motion Matter

TSM’s authors place Something-Something, with Jester and Charades, among datasets focused on temporal relationships, and Kinetics, UCF101 and HMDB51 among those less sensitive to them. In their Table 1 (ResNet-50, 8 frames; 10 clips per video for Kinetics, 2 for the others), adding the temporal shift to a TSN baseline at no extra computation raised SSv2 top-1 accuracy from 30.4% to 61.7%, against a gain from 70.6% to 74.1% on Kinetics (Lin et al., 2019). TimeSformer with attention over space only reached 36.6% on SSv2, against 59.5% with divided space-time attention, while the two differed by about one point on Kinetics-400 (Bertasius et al., 2021, Table 1). Temporal modeling and video transformers discuss these comparisons.

Flipping. A horizontal flip turns “Pushing something from left to right” into a valid video of the opposite class (action recognition gives the reasoning). TSM’s official code turns flipping off for RGB training on Something-Something and Jester. Several of MMAction2’s SSv2 configurations, for TSM, TSN and TRN, flip with a label map that swaps three pairs of classes: pushing and pulling from left to right versus right to left, and turning the camera left versus right. The ICCV paper itself tried flipping only for classes it calls invariant, without significant gains (Goyal et al., 2017).

A synthetic template dataset

The example renders 16-frame, $24 \times 48$ grayscale clips of a bright disc (the object) and a grey bar (the hand) in four simplified templates: “moving left” and “moving right” (the hand grips the disc from above and carries it), “pretending to move” (the hand moves, the disc stays) and “pushing so it falls” (the hand pushes the disc past a table edge). Each frame is reduced to the centroids of disc and hand, found by intensity. The frame-level model is a logistic regression on single frames whose log-probabilities are averaged over the clip, like an image classifier averaged over a video; the trajectory model sees whole-clip motion of disc and hand. The trajectory model is then retrained with mirrored training clips added, with labels unchanged or with left and right swapped.

import numpy as np

T, H, W = 16, 24, 48                      # frames, height, width
TABLE = (6, 41)                           # table spans these columns; beyond them a disc falls
CLASSES = ["moving left", "moving right", "pretending to move", "pushing so it falls"]
SWAP = np.array([1, 0, 2, 3])             # label of the mirrored clip
yy, xx = np.mgrid[:H, :W]

def render(disc, hand, rng):
    """disc, hand: (T, 2) arrays of (x, y) centres. Disc = bright circle, hand = grey bar."""
    v = rng.normal(0, 0.05, (T, H, W))
    for t in range(T):
        dx, dy = disc[t]
        hx, hy = hand[t]
        v[t][(xx - dx) ** 2 + (yy - dy) ** 2 <= 9] += 1.0                 # radius-3 disc
        v[t][(np.abs(xx - hx) <= 1) & (yy >= hy - 4) & (yy <= hy + 4)] += 0.5  # 3 x 9 bar
    return v

def make_clip(c, rng):
    a, b = rng.integers(3, 6), rng.integers(10, 13)     # move phase: frames a..b
    D = rng.uniform(8, 14)                               # distance moved
    s = np.clip((np.arange(T) - a) / (b - a), 0, 1)      # 0 before, ramps, 1 after
    down = np.clip(np.arange(T) / a, 0, 1)               # hand descends in frames 0..a
    up = np.clip((np.arange(T) - b) / (T - 1 - b), 0, 1) # and rises after b
    if c in (0, 1, 2):
        sign = (-1, 1)[c] if c < 2 else rng.choice([-1, 1])
        x0 = rng.uniform(TABLE[0] + 4 + D * (sign < 0), TABLE[1] - 4 - D * (sign > 0))
        hx = x0 + sign * D * s
        hy = -4 + 13 * down - 13 * up                     # from above the frame to y = 9 and back
        dxs = hx if c < 2 else np.full(T, x0)            # the disc follows the hand only if real
        disc = np.stack([dxs, np.full(T, 16.0)], 1)
    else:                                                # push from the side until it falls
        sign = rng.choice([-1, 1])
        edge = TABLE[1] if sign > 0 else TABLE[0]
        x0 = edge - sign * rng.uniform(2, D - 2)         # the push carries it past the edge
        dxs = x0 + sign * D * s
        fall = np.maximum(0, sign * (dxs - edge))        # distance past the edge
        disc = np.stack([np.where(fall > 0, edge + sign * np.minimum(fall, 3), dxs),
                         np.minimum(16 + 1.5 * fall, 20)], 1)   # drops to the floor, stays in view
        t = np.arange(T)                                 # hand: approach, push, withdraw
        hx = np.where(t < a, dxs[0] - sign * (5 + 2 * (a - t)), dxs - sign * 5)
        hx = np.where(sign * (hx - edge) > 0, edge, hx)  # the hand stops at the edge
        hx = np.where(t > b, hx[b] - sign * 2 * (t - b), hx)
        hy = np.full(T, 16.0)
    hand = np.stack([hx, hy], 1)
    return render(disc, hand, rng)

def dataset(n_per_class, rng):
    X = np.stack([make_clip(c, rng) for c in range(4) for _ in range(n_per_class)])
    return X, np.repeat(np.arange(4), n_per_class)

def tracks(v):
    """Per-frame centroids of the disc (bright) and the hand (grey) by intensity bands."""
    disc_m = v > 0.75
    hand_m = (v > 0.3) & (v <= 0.75)
    def cen(m):
        n = m.sum((-1, -2)) + 1e-9
        return np.stack([(m * xx).sum((-1, -2)) / n, (m * yy).sum((-1, -2)) / n,
                         (n > 1).astype(float)], -1)
    return cen(disc_m), cen(hand_m)       # (..., T, 3): x, y, visible

def frame_features(v):
    """One descriptor per frame: positions of disc and hand and their offset."""
    d, h = tracks(v)
    f = np.concatenate([d, h, d[..., :2] - h[..., :2]], -1)        # (N, T, 8)
    q = np.einsum("nti,ntj->ntij", f, f).reshape(*f.shape[:2], -1)  # quadratic terms
    return np.concatenate([f, q], -1)

def trajectory_features(v):
    """Whole-clip motion: net and total displacements of disc and hand."""
    d, h = tracks(v)
    step_d, step_h = np.diff(d[..., :2], axis=1), np.diff(h[..., :2], axis=1)
    return np.concatenate([d[:, -1, :2] - d[:, 0, :2],            # disc net dx, dy
                           np.abs(step_d).sum(1),                 # disc path length
                           (step_h[..., 0] * (h[:, 1:, 1] > 6)).sum(1, keepdims=True),  # hand dx while low
                           (step_d[..., 0] * step_h[..., 0]).sum(1, keepdims=True)], 1)

def fit(F, y, k=4, lam=1e-2, iters=300, lr=0.5):
    """Multinomial logistic regression by gradient descent on standardized features."""
    mu, sd = F.mean(0), F.std(0) + 1e-6
    Z = np.c_[(F - mu) / sd, np.ones(len(F))]
    Wt = np.zeros((Z.shape[1], k))
    Y = np.eye(k)[y]
    for _ in range(iters):
        P = np.exp(Z @ Wt - (Z @ Wt).max(1, keepdims=True)); P /= P.sum(1, keepdims=True)
        Wt -= lr * (Z.T @ (P - Y) / len(F) + lam * Wt)
    return lambda G: np.c_[(G - mu) / sd, np.ones(len(G))] @ Wt

def per_class(pred, y):
    return np.array([np.mean(pred[y == c] == c) for c in range(4)])

acc = []
for seed in range(5):
    rng = np.random.default_rng(seed)
    Xtr, ytr = dataset(150, rng)
    Xte, yte = dataset(100, rng)
    # Frame-level: train on single frames with the clip label, average log-scores over frames.
    Ftr, Fte = frame_features(Xtr), frame_features(Xte)
    g = fit(Ftr.reshape(-1, Ftr.shape[-1]), np.repeat(ytr, T))
    s = g(Fte.reshape(-1, Fte.shape[-1])).reshape(len(Xte), T, 4)
    p_frame = (s - np.log(np.exp(s).sum(-1, keepdims=True))).mean(1).argmax(1)
    # Trajectory: features of the hand and disc tracks over the whole clip.
    h = fit(trajectory_features(Xtr), ytr)
    p_traj = h(trajectory_features(Xte)).argmax(1)
    # Flip augmentation: add mirrored training clips, with or without swapping labels.
    Xfl = Xtr[..., ::-1]
    F2 = trajectory_features(np.concatenate([Xtr, Xfl]))
    p_noswap = fit(F2, np.r_[ytr, ytr])(trajectory_features(Xte)).argmax(1)
    p_swap = fit(F2, np.r_[ytr, SWAP[ytr]])(trajectory_features(Xte)).argmax(1)
    preds = [p_frame, p_traj, p_noswap, p_swap]
    acc.append([np.mean(p == yte) for p in preds] + [per_class(p, yte)[:2].mean() for p in preds])
    if seed == 0:
        d0, d1 = trajectory_features(Xtr[ytr == 1])[:, 0], trajectory_features(Xfl[ytr == 1])[:, 0]
        print(f"'moving right', disc net dx: {d0.mean():+.1f} px, mirrored {d1.mean():+.1f} px")
        for name, p in zip(["frame-level", "trajectory", "flip, no swap", "flip + swap"], preds):
            print(f"seed 0 per class (L, R, pretend, fall) {name:14s}", np.round(per_class(p, yte), 2))
acc = np.array(acc)
for j, name in enumerate(["frame-level", "trajectory", "flip, no swap", "flip + swap"]):
    print(f"seeds 0-4 {name:14s} top-1 {acc[:, j].min():.3f}-{acc[:, j].max():.3f}"
          f"   left/right classes {acc[:, 4 + j].min():.3f}-{acc[:, 4 + j].max():.3f}")

Output:

'moving right', disc net dx: +11.2 px, mirrored -11.2 px
seed 0 per class (L, R, pretend, fall) frame-level    [0.42 0.56 1.   1.  ]
seed 0 per class (L, R, pretend, fall) trajectory     [1. 1. 1. 1.]
seed 0 per class (L, R, pretend, fall) flip, no swap  [0.44 0.54 1.   1.  ]
seed 0 per class (L, R, pretend, fall) flip + swap    [1. 1. 1. 1.]
seeds 0-4 frame-level    top-1 0.735-0.757   left/right classes 0.470-0.515
seeds 0-4 trajectory     top-1 1.000-1.000   left/right classes 1.000-1.000
seeds 0-4 flip, no swap  top-1 0.745-0.775   left/right classes 0.490-0.550
seeds 0-4 flip + swap    top-1 1.000-1.000   left/right classes 1.000-1.000

Over seeds 0 to 4, the frame-level model’s mean accuracy on the two moving classes stays between 47.0% and 51.5%, a coin toss between them: a frame of “moving left” shows a hand gripping a disc somewhere on the table, like a frame of “moving right”, and only the order differs. In seed 0 it still gets the other two templates right, because in this toy single frames give them away: a pretending hand is seen off the disc, a pushed disc below the table. Real pretended actions need not be that visible in any one frame (our reading). The trajectory model is perfect on every seed. Mirroring reverses the disc’s displacement (+11.2 to −11.2 pixels); with mirrored clips under unchanged labels, the trajectory model falls to 49.0% to 55.0% on the moving pair, and with swapped labels it stays perfect.

License

As of October 2026, Qualcomm’s page states that the dataset is available for research purposes, under the “Data License Agreement – Research Use” of Qualcomm Technologies, Inc. (dated 28 July 2022), which covers Jester and Something-Something. It allows internal processing solely for non-profit research, excludes commercial use and forbids redistribution. TwentyBN’s archived pages offered the data free of charge for academic research with commercial licences on request; the V1 page named a Creative Commons NonCommercial-NoDerivatives 4.0 licence, although its wording also says “Attribution 4.0 International”.

Limitations and Bias

• Staged video. The clips are acted for the camera and last seconds; in our reading they differ from long, unscripted footage, and workers may act templates in stereotyped ways.
• Object labels. The object words are free text, so the same object appears under many names (23,137 names in V1). Something-Else counts 12,554 object descriptions in V2, against 30,408 unique objects on the official page; the counting methods are not described, so the two numbers are not comparable.
• Ambiguous classes. The authors themselves point to label ambiguities, and, in our reading, some templates overlap, such as “Putting something on a surface” and “Putting something onto something”.
• Objects seen in training. Because train and test share verb–object combinations, Materzynska et al. (2020) built a compositional split in which they do not: I3D fell from 61.7% top-1 on a random (“shuffled”) split of the same videos to 46.8% (Table 3a).

Common Uses

SSv2 is the main benchmark for temporal modeling, reported alongside Kinetics-400 by temporal shift (Lin et al., 2019), video transformers (Bertasius et al., 2021) and VideoMAE, which also pretrains on its videos without labels.

Historical Milestones

• 2017. V1 and the ICCV paper, with 11.5% top-1 accuracy for the best 174-class baseline.
• 2018. V2: twice the videos, higher resolution, object annotations, five-worker verification.
• 2019. TSM reaches 61.7% on V2 with a 2D network and temporal shift.
• 2020. Something-Else’s boxes and compositional splits.

Related Datasets

Jester (Materzynska et al., 2019), also crowd-acted and from the same company, has 148,092 videos in 27 classes (25 hand gestures and two non-gesture classes) performed toward the camera by 1,376 actors; TSM’s Table 1 lists it with the temporal datasets (TSN 83.9%, TSM 97.0%). Something-Else is a re-split and box annotation of SSv2, not new video. Kinetics is the appearance-heavy counterpart, and the video understanding hub compares the main action datasets.

Resources

• Something-Something v. 2 Dataset: the official page, with the licence and downloads.
• TwentyBN V2 page (archived): the original release notes, class counts and test leaderboard.
• Something-Else project page: the box annotations and compositional splits.