Temporal Segment Networks

A temporal segment network (TSN) trains a frame-level ConvNet on a whole video rather than one short clip. It divides the video into $K$ segments of equal duration, draws one short snippet from each, scores every snippet with the same network, combines the $K$ score vectors by a segmental consensus function, and computes the loss on that consensus. Wang et al. introduced it at ECCV 2016 on top of two-stream networks, with a study of practices for training deep networks on small action recognition datasets; a journal version (IEEE TPAMI, 2019) extended it to untrimmed video. Its segment-based sampling, defined in temporal modeling, was reused by later 2D video networks.

Problem Addressed

Two-stream networks trained on single frames and short flow stacks, each covering a fraction of a second. Wang et al. (2016) named two obstacles. Actions such as sports moves span several stages, and the long-range models of the time read dense frame sequences of fixed length, which is costly and misses content beyond that length. And UCF101 (13,320 clips) and HMDB51 (6,766 clips) are small, so very deep networks risk overfitting. Their premise: consecutive frames are highly redundant, so sparse samples spread over the video suffice.

Architecture

TSN is a training and inference framework, not a new layer: each stream is an unchanged 2D ConvNet, in the ECCV version BN-Inception, the Inception network with batch normalization of Ioffe and Szegedy (2015).

Segments and Snippets

A video $V$ is divided into $K$ segments $\{S_1, \dots, S_K\}$ of equal duration, and a snippet $x_k$ is sampled at random from segment $S_k$ (the paper’s $T_k$; $T$ here counts frames). A snippet is one RGB frame for the spatial stream, or a short stack of flow fields or RGB differences for the motion streams. With $F(x; W)$ the ConvNet’s class scores, the network computes

$$\mathrm{TSN}(x_1, \dots, x_K) = H\big(G(F(x_1; W), \dots, F(x_K; W))\big),$$

where $H$ is the softmax and all $K$ copies of $F$ share the weights $W$. With $K$ fixed, the cost is independent of the video’s length. The ECCV version sets $K = 3$, following earlier models of temporal structure (Gaidon et al.’s actoms; a latent hierarchical model by Wang et al.).

Segmental Consensus

The consensus works class by class: $G_i = g(F_i(x_1), \dots, F_i(x_K))$. The ECCV paper compared three aggregation functions $g$ on UCF101 split 1 with RGB and optical flow and $K = 3$ (Table 3):

Consensus $g$	Spatial	Temporal	Two-stream
Max	85.0%	86.0%	91.6%
Average	85.7%	87.9%	93.5%
Weighted average	86.2%	87.7%	92.4%

The text says average pooling performs best and adopts it; the table bears that out for the temporal and two-stream columns, but weighted averaging is best for the spatial stream.

Input Modalities

Besides RGB frames and stacked optical flow, Wang et al. tested stacked RGB differences of consecutive frames, which show where appearance changes, and warped optical flow, which borrows the camera-motion compensation of improved dense trajectories: a homography is estimated and the camera motion removed, so the flow concentrates on the actor. On UCF101 split 1, with all their training practices (Table 2):

Modality	Accuracy
RGB image	84.5%
RGB difference	83.8%
RGB image + RGB difference	87.3%
Optical flow	87.2%
Warped flow	86.9%
Optical flow + warped flow	87.8%
Optical flow + warped flow + RGB	92.3%
All modalities	91.7%

Adding RGB difference to the other three lowered accuracy; the authors conjecture it describes motion less stably, though it is a fast, low-quality substitute for flow.

Training

The loss is the categorical cross-entropy of the softmax of the consensus. With $C$ classes and ground-truth label vector $y$ (ECCV Eq. 2),

$$L(y, G) = -\sum_{i=1}^{C} y_i \Big( G_i - \log \sum_{j=1}^{C} \exp G_j \Big),$$

and back-propagation sends the gradient through the consensus to every snippet (Eq. 3):

$$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial G} \sum_{k=1}^{K} \frac{\partial G}{\partial F(x_k)} \frac{\partial F(x_k)}{\partial W}.$$

For a one-hot $y$, $\partial L / \partial G_i = \mathrm{softmax}(G)_i - y_i$ (the standard softmax–cross-entropy gradient). The journal version writes out $\partial G / \partial F$: average pooling gives $1/K$ to every snippet, max pooling gives 1 to the highest-scoring snippet for each class and 0 to the others. Under averaging, every update uses all $K$ snippets, and the label supervises the video, not a snippet that may miss the action.

Good Practices

All from the ECCV version:

• Cross-modality pretraining of the motion streams from the ImageNet model (see two-stream networks).
• Partial BN. After pretrained initialization, the mean and variance of every batch-normalization layer except the first are frozen, since small datasets give biased estimates; the first is re-estimated because flow is distributed differently from RGB.
• Extra dropout after global pooling: 0.8 in the spatial, 0.7 in the temporal stream.
• Corner cropping and scale jittering, described in action recognition.

Training used SGD with batch size 256 and momentum 0.9, and TV-L1 flow; on UCF101 it took about 2 hours for the spatial and 9 hours for the temporal network on four TITAN X GPUs. The ablations on UCF101 split 1 (Tables 1, 4 and 5):

Setting (BN-Inception unless stated)	Spatial	Temporal	Two-stream
Original two-stream network (Clarifai)	72.7%	81.0%	87.0%
From scratch	48.7%	81.7%	82.9%
ImageNet-pretrained spatial stream only	84.1%	81.7%	90.0%
+ cross-modality pretraining	84.1%	86.6%	91.5%
+ partial BN with dropout	84.5%	87.2%	92.0%
+ TSN, $K = 3$	85.7%	87.9%	93.5%
GoogLeNet, good practices	77.1%	83.9%	89.0%
VGGNet-16, good practices	79.8%	85.7%	90.9%

Table 5 repeats the two-stream chain from 90.0% to 93.5%: the deeper network and the practices give most of the gain over the original network, the segments 1.5 points.

Inference

Since the snippet networks share weights, a trained TSN is tested like an ordinary two-stream network. The ECCV version followed the 2014 protocol: 25 RGB frames or flow stacks per video, each with the four corner crops, the center crop and their horizontal flips. Snippet and stream scores are fused before the softmax, as in training. Because TSN narrowed the gap between the streams, the weights were 1 for spatial and 1.5 for temporal; with warped flow, the temporal weight was split into 1 for flow and 0.5 for warped flow.

Performance

ECCV Table 6 reports the final systems on the three splits of each dataset:

Method	HMDB51	UCF101
TSN, RGB + flow	68.5%	94.0%
TSN, RGB + flow + warped flow	69.4%	94.2%

The best previous entries in the table were 65.5% (MPR) and 93.1% (KVMF); the original two-stream network had 59.4% and 88.0%.

Example: Segments and Consensus

The script makes videos of $T \times H \times W = 120 \times 20 \times 20$ noisy frames with two 16-frame events at random positions: a $4 \times 4$ square moving right or left by one pixel per frame. The class is the unordered pair of directions (right+right, right+left, left+left), so a model must find both events; a snippet of two consecutive frames shows one only if it falls inside it. Snippets pass through a frozen random “ConvNet” (32 filters of $3 \times 3$ over both frames, ReLU, global average pooling), and a linear snippet classifier $F$ is trained on the loss of the consensus, with new snippets every step, as in Eq. 3. Testing uses each method’s own sampling and, as in the paper, 25 evenly spaced snippets.

import numpy as np

N, S, E = 120, 20, 16                 # frames per video, frame size, event length (frames)
CLASSES = ["right+right", "right+left", "left+left"]
FILT = np.random.default_rng(0).standard_normal((2, 9, 32)) / 3   # frozen 3x3 filters on (frame t, frame t+1)
SQUARE = np.pad(np.ones((4, 4)), ((0, S - 4), (0, S - 4)))

def video(rng, label):
    """N noisy frames with two events, anywhere: a square moving right (+1) or left (-1) for E frames."""
    v = 0.2 * rng.standard_normal((N, S, S))
    first = rng.integers(0, N - 2 * E + 1)                       # two non-overlapping events
    second = rng.integers(first + E, N - E + 1)
    for start, d in zip((first, second), rng.permutation([[1, 1], [1, -1], [-1, -1]][label])):
        y, x = rng.integers(0, S, size=2)
        for t in range(E):                                       # wraps around: position is uninformative
            v[start + t] += np.roll(SQUARE, (y, x + d * t), axis=(0, 1))
    return v

def pair_features(v):
    """Frozen 'ConvNet' on each 2-frame snippet: 3x3 conv over both frames, ReLU, global average pool."""
    p = np.lib.stride_tricks.sliding_window_view(v, (3, 3), axis=(1, 2)).reshape(N, -1, 9)
    r0, r1 = p @ FILT[0], p @ FILT[1]
    return np.maximum(r0[:-1] + r1[1:], 0).mean(1)               # (N-1, 32): snippet starting at each frame

def softmax(z):
    e = np.exp(z - z.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)

def consensus(f, kind):
    """Snippet scores f (videos, K, classes) -> consensus G (videos, classes) and dG/df."""
    if kind == "average":
        return f.mean(1), np.full(f.shape, 1 / f.shape[1])
    G = f.max(1)
    return G, (f == G[:, None]).astype(float)                    # gradient only to the arg-max snippet

def tsn_sampler(K):
    """K equal segments of the N-1 snippet positions, one random snippet per segment."""
    bounds = np.arange(K + 1) * (N - 1) // K
    return lambda rng, n: bounds[:-1] + (rng.random((n, K)) * np.diff(bounds)).astype(int)

def dense_sampler(T=8):
    """One clip of T consecutive frames at a random position = its T-1 consecutive snippets."""
    return lambda rng, n: rng.integers(0, N - T + 1, size=(n, 1)) + np.arange(T - 1)

def train(F, y, sampler, kind, rng, steps=600, lr=0.1):
    """Linear snippet classifier F(x; W) trained on the loss of the consensus, new snippets every step."""
    W, b, Y = np.zeros((F.shape[-1], 3)), np.zeros(3), np.eye(3)[y]
    for _ in range(steps):
        X = F[np.arange(len(F))[:, None], sampler(rng, len(F))]  # (videos, K, 32)
        G, dG = consensus(X @ W + b, kind)
        dL_df = dG * ((softmax(G) - Y) / len(F))[:, None, :]     # Eq. (3): dL/dG spread over snippets
        W -= lr * np.einsum("nkd,nkc->dc", X, dL_df)
        b -= lr * dL_df.sum((0, 1))
    return W, b

def predict(F, W, b, idx, kind):
    return consensus(F[np.arange(len(F))[:, None], idx] @ W + b, kind)[0].argmax(1)

methods = [("dense clip, T=8", dense_sampler(), "average", 8)]
methods += [(f"TSN K={K}, average", tsn_sampler(K), "average", 2 * K) for K in (1, 3, 5, 7)]
methods += [(f"TSN K={K}, max", tsn_sampler(K), "max", 2 * K) for K in (3, 7)]
grid = np.tile(np.linspace(0, N - 2, 25).round().astype(int), (300, 1))   # 25 evenly spaced snippets
res = {name: [] for name, *_ in methods}
for seed in range(5):
    rng = np.random.default_rng(seed)
    y = np.tile([0, 1, 2], 200)                                  # 300 training and 300 test videos
    F = np.stack([pair_features(video(rng, c)) for c in y])
    F = (F - F[:300].mean((0, 1))) / F[:300].std((0, 1))
    for name, sampler, kind, _ in methods:
        W, b = train(F[:300], y[:300], sampler, kind, np.random.default_rng(seed))
        own = predict(F[300:], W, b, sampler(np.random.default_rng(99), 300), kind)
        p25 = predict(F[300:], W, b, grid, kind)
        res[name].append([np.mean(own == y[300:]), np.mean(p25 == y[300:]), np.mean(p25 == 1)])
print("accuracy over seeds 0-4: mean (min-max)")
print(f"{'':20s} frames  {'own sampling':20s} {'25 snippets':20s} predicted right+left")
for name, _, _, frames in methods:
    r = np.array(res[name])
    print(f"{name:20s} {frames:5d}   " + "   ".join(f"{r[:, j].mean():.3f} ({r[:, j].min():.3f}-{r[:, j].max():.3f})" for j in (0, 1))
          + f"   {r[:, 2].mean():.3f}")

# Where does the gradient of one video's loss go?  (last seed, K = 7)
X = F[:1, tsn_sampler(7)(np.random.default_rng(5), 1)[0]]       # one right+right training video, 7 snippets
for kind in ("average", "max"):
    W, b = train(F[:300], y[:300], tsn_sampler(7), kind, np.random.default_rng(4))
    G, dG = consensus(X @ W + b, kind)
    grad = dG * (softmax(G) - np.eye(3)[y[:1]])[:, None, :]
    print(f"{kind:7s} |dL/dF(x_k)| for k = 1..7:", np.abs(grad[0]).sum(1).round(3))

accuracy over seeds 0-4: mean (min-max)
                     frames  own sampling         25 snippets          predicted right+left
dense clip, T=8          8   0.440 (0.380-0.473)   0.877 (0.837-0.900)   0.356
TSN K=1, average         2   0.400 (0.357-0.447)   0.905 (0.870-0.930)   0.322
TSN K=3, average         6   0.528 (0.480-0.597)   0.906 (0.853-0.933)   0.310
TSN K=5, average        10   0.613 (0.600-0.630)   0.881 (0.853-0.917)   0.280
TSN K=7, average        14   0.644 (0.603-0.670)   0.867 (0.830-0.900)   0.272
TSN K=3, max             6   0.487 (0.437-0.550)   0.564 (0.380-0.667)   0.080
TSN K=7, max            14   0.616 (0.537-0.643)   0.662 (0.643-0.670)   0.071
average |dL/dF(x_k)| for k = 1..7: [0.149 0.149 0.149 0.149 0.149 0.149 0.149]
max     |dL/dF(x_k)| for k = 1..7: [0.    0.    0.    0.418 0.418 0.    0.   ]

With its own sampling, accuracy rises with $K$, from 0.40 for one snippet to 0.64 for seven (chance is 1/3), since each extra segment raises the chance of catching both events; three snippets (6 frames) beat the 8-frame dense clip, whose consecutive frames rarely span both. With 25 test snippets every average-consensus model has a mean of 0.87–0.91, with overlapping ranges: in this toy (a linear classifier on frozen features) the training $K$ hardly matters once testing covers the video. The paper’s TSN gain (92.0% to 93.5%, both tested on 25 snippets) came from training a deep network end to end, which the example does not reproduce.

Max consensus fails on the mixed class: with 25 snippets it predicts right+left for 7–8% of test videos instead of a third, and stays at about 2/3 accuracy or below. By our reasoning, the max makes each class score depend on one snippet, so it can express “some snippet shows right” but not “one snippet shows right and another left”; the average adds evidence across snippets. The last lines show Eq. 3: averaging sends the same gradient, $(\mathrm{softmax}(G) - y)/K$, to all seven snippets; max sends it only to the snippets that won a class.

Strengths

• Video-level supervision at constant cost: $K$ snippets per video, whatever its length.
• Any 2D backbone and input, image-pretrained and unchanged.
• Practices for small datasets that made very deep networks trainable on UCF101-sized data (Table 1).

Limitations

• Order-agnostic consensus. Average and max discard the order of the snippets, so classes that differ only in the order of their stages cannot be told apart (the temporal modeling example shows this for pooling); only the motion streams see local direction within a snippet. The example’s classes are unordered pairs for this reason.
• Sparse snippets. Content between snippets is not read.
• Optical flow cost. The best results need TV-L1 flow; warped flow adds a homography and a second flow computation. RGB difference trades accuracy for speed.
• Weak on temporal relations. On data that depend on them, TSN-style 2D networks do poorly: in TSM’s Table 1, a TSN ResNet-50 baseline reached 20.5% top-1 on Something-Something V1.

Historical Importance

TSN (2016) outperformed every method in its comparison on UCF101 and HMDB51 with 2D networks, shortly before Kinetics and I3D (2017) moved the field to large-scale pretraining and 3D networks. What lasted is segment-based sparse sampling, supervision of the consensus of several snippets, and the training recipe; TSM, for example, used TSN as its 2D baseline.

Variants

Journal version. The TPAMI extension (Wang et al., 2019) adds:

• More segments. With average consensus, two-stream accuracy over the three UCF101 splits (1:1 fusion) rose from 92.4% at $K = 1$, which the authors equate with the plain two-stream network, to 94.2% at $K = 3$ and 94.9% at $K = 7$, with no further gain at $K = 9$; $K = 7$ became the default (Table 4).
• More consensus functions. Top-$k$ pooling (the mean of the $k$ highest-scoring snippets per class; the paper writes $\mathcal{K}$) and attention weighting, a learned softmax weight per snippet computed from its features. With $K = 7$, averaging stayed best on UCF101 (94.9%), but on ActivityNet v1.2 validation, top-$k$ (88.1%) and attention (88.2%) beat averaging (87.8%) (Table 5).
• Untrimmed video. Multi-scale temporal window integration scores one snippet per second, max-pools sliding windows of 1, 2, 4, 8 and 16 snippets, pools each window size by top-$k$ and averages the sizes.
• ActivityNet Challenge 2016. The authors’ entry (“CES”), an ensemble of TSN models (Inception V3 and ResNet-152) plus audio models, ranked first of 24 in untrimmed video classification with 93.23% mAP on the ActivityNet v1.3 test set (Table 8). The official challenge page lists the CUHK team as classification winners, with a link to the TSN code.

TSN as a sampling scheme. TSM (Lin et al., 2019) adds temporal shifts to a 2D ResNet and compares against TSN under the same protocol: each of $T$ sampled frames passes through the 2D network and the logits are averaged. Its official code, built on the TSN codebase, samples one frame per segment by default, with I3D-style dense sampling as an option; the README says the paper’s Kinetics results in Tables 1 and 4 used dense sampling (as of October 2026). Dense-clip models read consecutive frames from one window instead: SlowFast (Feichtenhofer et al., 2019) trains on one randomly placed clip (for the Slow pathway of its Table 1 instantiation, 4 frames at stride 16 from 64 raw frames) and tests on 10 uniformly spaced clips × 3 crops.

Implementations

The ECCV paper points to the authors’ Caffe code and models (github.com/yjxiong/temporal-segment-networks). As of October 2026, its README recommends the authors’ PyTorch toolbox MMAction instead.