Two-Stream Convolutional Networks for Action Recognition in Videos

“Two-Stream Convolutional Networks for Action Recognition in Videos” by Karen Simonyan and Andrew Zisserman of the Visual Geometry Group, University of Oxford, appeared in Advances in Neural Information Processing Systems 27 (NIPS 2014), after a first arXiv version in June 2014. It recognized actions with two convolutional networks, one on single RGB frames and one on a stack of precomputed optical flow fields, fused at the level of class scores. On UCF101 and HMDB51 the combination was competitive with hand-crafted improved dense trajectories, which earlier deep video networks had trailed by a wide margin. This article covers the publication; two-stream networks explains the architecture, its training and its variants.

Problem

Hand-crafted features led. In 2014 the best action recognition systems were shallow pipelines built on dense trajectories: points tracked through dense optical flow, described by histograms of gradients, of flow (HOF) and of flow gradients (motion boundary histograms, MBH), and encoded with Fisher vectors. The paper names MBH as the pipeline’s best descriptor and camera-motion compensation and Fisher vectors as its recent gains; improved dense trajectories (iDT; Wang & Schmid, 2013) were the reference in its results table.

Deep networks had not caught up. The closest deep competitor was Karpathy et al. (2014), who trained networks on stacks of RGB frames from Sports-1M, about a million YouTube sports videos. Simonyan and Zisserman drew two lessons from it: a single-frame network performed about as well as networks on frame stacks, suggesting that the learned features did not capture motion well, and the representation fine-tuned on UCF101 was 20% less accurate than the best trajectory-based features. On the Sports-1M test set, Karpathy et al.’s best temporal model, slow fusion, reached 60.9% video-level hit@1 against 59.3% for a single frame (their Table 1), a gain they called surprisingly modest. On UCF101, fine-tuning the top three layers of the Sports-1M slow-fusion network gave 65.4% three-fold accuracy, against 41.3% when trained from scratch (their Table 3; their abstract quotes 63.3%).

Datasets were small. UCF101 has 13K videos in 101 classes and HMDB51 6.8K videos in 51; one training split holds 9.5K and 3.7K videos respectively. Pretraining on ImageNet could help an appearance network, but not a network whose input is motion. Sports-1M, a footnote says, was very challenging to process at that scale and was left for future work.

Contribution

The abstract lists three contributions:

• A two-stream architecture. A spatial ConvNet on still frames and a temporal ConvNet on motion, combined by late fusion. Decoupling the streams lets the spatial one be pretrained on ImageNet.
• A ConvNet trained on multi-frame dense optical flow. The temporal stream reaches good accuracy despite the limited training data, because flow describes motion explicitly and the network “does not need to estimate motion implicitly” (§3).
• Multi-task learning across datasets. Training one network with a classification head for UCF101 and another for HMDB51 increases the effective amount of training data and improves results on both.

Relation to trajectory features. The abstract states the aim of generalizing the best hand-crafted features in a learned framework, and Section 3.2 argues that the temporal ConvNet generalizes the flow-based descriptors of the trajectory pipeline. HOF and MBH, histograms of the orientations of flow and of its gradient, could be computed from stacked flow by one layer of orientation-sensitive filters followed by rectification and pooling; Jain et al.’s divergence, curl and shear also derive from the flow gradient; and the trajectory descriptor corresponds to the trajectory-stacking input. As evidence, the NIPS supplementary material (Section 3.3 of arXiv v2) shows first-layer temporal filters that compute spatial derivatives of the flow, as MBH does, and others that compute temporal derivatives. The conclusion names what the network still lacked: pooling over spatio-temporal tubes that follow the trajectories, as iDT does, and explicit camera-motion handling beyond subtracting the mean flow.

Relation to biology and earlier deep models. The authors relate the design to the two-streams hypothesis (Goodale & Milner, 1992), in which, in their summary, a ventral pathway recognizes objects and a dorsal pathway motion, but do not investigate the connection further. Nor was the idea new to video: the paper credits Kuehne et al.’s HMDB51 paper with ventral-like and dorsal-like streams built from hand-crafted, three-layer HMAX models, and presents its ConvNets as a deeper, learned generalization.

Method

Both streams use the CNN-M-2048 configuration (five convolutional and two fully connected layers). The spatial stream sees a $224 \times 224$ RGB crop, the temporal stream a $224 \times 224 \times 2L$ stack of the horizontal and vertical components of $L$ flow fields computed with Brox et al.’s method, in several input variants. Softmax scores are fused by averaging or a linear SVM; joint fully connected layers over both networks were, the authors report, not feasible because of overfitting. Testing averages 25 equally spaced frames, each with ten crops and flips. Two-stream networks gives the layers and training details.

Apart from one dropped normalization layer, the streams differ only in input depth, so the first convolution carries the change. The snippet below (PyTorch 2.14.1) builds that layer for both inputs, without padding, which the paper does not specify:

import torch
import torch.nn as nn

L = 10                                    # flow fields per input stack
for name, c in [("spatial", 3), ("temporal", 2 * L)]:
    conv1 = nn.Conv2d(c, 96, kernel_size=7, stride=2)      # CNN-M-2048 conv1: 7x7x96, stride 2
    x = torch.zeros(1, c, 224, 224)                        # one 224 x 224 crop, (N, C, H, W)
    n_params = sum(p.numel() for p in conv1.parameters())  # weights + biases
    print(f"{name:8s} input {tuple(x.shape)}: {x.numel():>9,} values | conv1 "
          f"{tuple(conv1.weight.shape)}: {n_params:>6,} parameters | output {tuple(conv1(x).shape)}")

# Frames spanned by one temporal-stream input for frame tau (Sect. 3.1).
uni = range(0, L + 1)                     # forward flows tau -> tau+1, ..., tau+L-1 -> tau+L
bi = range(-L // 2, L // 2 + 1)           # L/2 backward flows before tau, L/2 forward flows after
print(f"uni-directional: frames tau+{uni[0]}..tau+{uni[-1]} ({len(uni)} frames)")
print(f"bi-directional:  frames tau{bi[0]}..tau+{bi[-1]} ({len(bi)} frames)")
print("torch", torch.__version__)

spatial  input (1, 3, 224, 224):   150,528 values | conv1 (96, 3, 7, 7): 14,208 parameters | output (1, 96, 109, 109)
temporal input (1, 20, 224, 224): 1,003,520 values | conv1 (96, 20, 7, 7): 94,176 parameters | output (1, 96, 109, 109)
uni-directional: frames tau+0..tau+10 (11 frames)
bi-directional:  frames tau-5..tau+5 (11 frames)
torch 2.14.1

The temporal input holds 6.7 times as many values and its first layer 6.6 times as many parameters, but from the output of conv1 onward the two networks have identical shapes. Either flow stack covers 11 frames, which at UCF101’s 25 frames per second is under half a second (our arithmetic).

Results

Ablations use UCF101 split 1, the final comparison three splits; tables are from the NIPS main text unless noted. The two-stream networks article has the temporal-input, HMDB51 multi-task and final comparison tables; the figures below complement them.

• Pretraining and dropout (Table 1a). From scratch, the spatial network reached 42.5% with dropout 0.5 and 52.3% with 0.9, overfitting either way. With ImageNet pretraining, training only the last layer gave 72.7% at 0.5 but 59.9% at 0.9, and full fine-tuning 70.8% and 72.8%; the authors kept the last-layer setting. Their ImageNet model reached 13.5% top-5 error on the ILSVRC-2012 validation set, against 16.0% reported by Zeiler and Fergus for a similar network, which they attribute to sampling crops from the whole image.
• Multi-task learning. Beyond the HMDB51 gains, adding all HMDB51 data as a second task raised the UCF101 temporal stream from 81.0% to 81.5%.
• Fusion (Table 3). Bi-directional flow, slightly better for the temporal stream alone (81.2% against 81.0%), was worse after averaging (85.6% against 85.9%), so the final model used forward flow. Fusion beat the temporal stream by about 6 points and the spatial by 14, the authors summarize.
• Earlier deep models (Table 4, mean over three splits). Omitted from the two-stream networks table, they trailed even the single streams (73.0% and 40.5% for the spatial one): Karpathy et al.’s slow-fusion network had 65.4% on UCF101 and the spatio-temporal HMAX network 22.8% on HMDB51. The authors called the SVM-fused model’s 88.0% and 59.4% comparable to the hand-crafted state of the art.
• Supplementary results. Without the HMDB51 data, the UCF101 three-split means were 82.5% for the temporal stream and 86.5% and 87.1% for averaging and SVM fusion (supplementary Table 1). The supplement’s text calls this 0.9% worse than with the extra data, which matches the SVM rows (88.0% against 87.1%) but not averaging (86.9% against 86.5%) or the temporal stream (83.7% against 82.5%). The supplement also traces the worst UCF101 class, Hammering, to the spatial network confusing it with HeadMassage (faces in both) and the temporal network with BrushingTeeth (up-and-down hand motion). arXiv v2 moved this analysis (§6) and the filter visualization into the main text, but not the table.

A single temporal network trained for one day on a machine with four NVIDIA Titan GPUs, 3.2 times faster than on one GPU, using a multi-GPU modification of Caffe.

Impact

Follow-up work on action recognition kept the two streams and changed what the 2014 paper had fixed: shallow networks, score-level fusion and short temporal context.

• Fusion (2016). Feichtenhofer, Pinz and Zisserman (CVPR 2016) argued that the original could not register what moves with where, nor follow how these cues evolve, and fused the streams at a convolutional layer and over time. The same year, Feichtenhofer, Pinz and Wildes (NIPS 2016) added residual connections between the appearance and motion pathways.
• Temporal segment networks (2016). Wang et al. (ECCV 2016) built on the two-stream architecture, naming its limited temporal context and relatively shallow network as the problems; their TSN kept the 2014 testing protocol.
• Two-stream I3D (2017). Carreira and Zisserman’s I3D paper reimplemented the two-stream design on ImageNet-pretrained Inception-v1 as a baseline. Its §2.3 describes the 2014 model as two replicas of an ImageNet-pretrained ConvNet, although the 2014 temporal stream was trained from scratch on video. Their new 3D network still kept a separate flow stream, because, they suggest, flow algorithms are in some sense recurrent while a 3D ConvNet is feedforward.
• SlowFast (2019). Feichtenhofer et al. called the two-stream method a foundation of many competitive results but methodologically unsatisfactory, since flow is hand-designed and two-stream methods are often not trained end to end with it. SlowFast computes no flow, yet fuses its pathways with lateral connections, which, its authors note, had been used to fuse two-stream networks, citing both 2016 papers above.

Limitations

• Flow precomputation. The OpenCV GPU implementation of Brox et al.’s method took 0.06 s per frame pair, still a bottleneck during training, so flow was precomputed. Tran et al. (2015), who read the 0.06 s as excluding I/O, measured that implementation at 0.85–0.9 s per pair including I/O on one K40 GPU: 607.8 hours for UCF101, against 2.2 hours for their C3D features (their Table 6).
• Short temporal extent. Each input spans at most 11 frames; longer structure comes only from test-time averaging.
• Small-dataset training. Training the temporal stream from scratch on small datasets shaped the design: dropout 0.9, multi-task heads, and fusion at the softmax scores.
• What the flow stream learns. The paper presents the temporal stream as a learned motion descriptor; later probes by Sevilla-Lara et al. (2019), summarized in two-stream networks, credit much of flow’s value to its invariance to appearance.

What Came After

The authors’ conclusion pointed to training on large video collections and, in Section 3.2, to building flow’s constancy and smoothness assumptions into end-to-end networks. Kinetics (2017) supplied the scale, and I3D pretrained on it reached 97.9% on UCF101 and 80.2% on HMDB51 with two streams (CVPR version, Table 4). 3D convolutional networks, SlowFast and video transformers then learned motion from raw frames, dropping the precomputed flow; temporal modeling compares the approaches.