Two-Stream Networks

A two-stream network recognizes actions in video with two convolutional neural networks: a spatial stream that sees appearance in single RGB frames and a temporal stream that sees motion in a stack of optical flow fields, fused into one prediction. Simonyan and Zisserman introduced the design in 2014, and many of the best UCF101 and HMDB51 results of the next three years came from two-stream networks or models built on them (see the comparison tables of Feichtenhofer et al., 2016, and Carreira & Zisserman, 2017). Temporal modeling compares flow input with other ways of using time.

Core Idea

Earlier deep video models fed stacked frames to one network. Simonyan and Zisserman note that in Karpathy et al.’s (2014) experiments such a network did about as well as a single-frame one, suggesting that it captured motion poorly. They split the problem instead: an image classifier handles appearance, and motion is handed to a second network as dense flow from a classical optical flow estimation method, so the network “does not need to estimate motion implicitly” (§3). The spatial stream can also be pretrained on ImageNet, with far more labelled data than UCF101 or HMDB51.

The authors relate the design to the two-streams hypothesis of the visual cortex (Goodale and Milner, 1992), with a ventral pathway that, in their summary, performs object recognition and a dorsal pathway that recognizes motion, but state that they do not investigate the connection further.

Building Blocks

Spatial stream. A 2D CNN classifies one frame. The original averaged test scores over 25 equally spaced frames, each with five crops and their flips.

Temporal stream. A CNN with the same layers takes a $224 \times 224 \times 2L$ input: the horizontal and vertical flow of $L$ consecutive frame pairs, stacked as channels. The 2014 paper compared:

• Optical flow stacking: the flow vectors at the same pixel in each of the $L$ fields.
• Trajectory stacking: the flow vectors sampled along the trajectory starting at the pixel, as in dense trajectory descriptors.
• Bi-directional flow: $L/2$ forward fields after the frame and $L/2$ backward fields before it, keeping $2L$ channels.
• Mean flow subtraction: subtracting each field’s mean vector, a simple camera-motion compensation.

Fusion. The original fused softmax scores late, by averaging or by a linear SVM on the stacked, L2-normalized scores; later variants fused feature maps (see multimodal fusion for fusion in general).

How Data Flows

Both streams in the 2014 paper use the CNN-M-2048 layer configuration, which its Figure 1 prints as:

Layer	Spatial stream	Temporal stream
conv1	$7 \times 7 \times 96$, stride 2, norm., pool	same
conv2	$5 \times 5 \times 256$, stride 2, norm., pool	same, without norm.
conv3–conv5	$3 \times 3 \times 512$, stride 1; pool after conv5	same
full6, full7	4096 and 2048 units, dropout	same
output	softmax	softmax

The figure labels the pooling layers “pool 2x2”, while the text specifies max pooling over $3 \times 3$ windows with stride 2. Dropping the second normalization layer (to save memory) is the only difference between the streams; with $L = 10$ the temporal stream’s first layer takes 20 channels instead of 3.

Mathematical Formulation

Let $d_t(u, v) = (d^x_t, d^y_t)$ be the flow at pixel $(u, v)$ from frame $t$ to $t + 1$. Optical flow stacking builds the input for frame $s$ (the paper’s $\tau$) as

$$I_s(u, v, 2k - 1) = d^x_{s + k - 1}(u, v), \qquad I_s(u, v, 2k) = d^y_{s + k - 1}(u, v), \qquad k = 1, \dots, L.$$

Trajectory stacking samples the same components at the points $p_k$ of a trajectory instead,

$$p_1 = (u, v), \qquad p_k = p_{k-1} + d_{s + k - 2}(p_{k-1}), \quad k > 1.$$

With class probabilities $\mathbf{p}_s$ and $\mathbf{p}_t$ from the spatial and temporal streams, fusion by averaging predicts $\arg\max_c \tfrac{1}{2}(p_{s,c} + p_{t,c})$.

Feichtenhofer, Pinz and Zisserman (2016) fused two feature maps $\mathbf{x}^a, \mathbf{x}^b \in \mathbb{R}^{H \times W \times D}$ of equal resolution. Sum and max fusion pair channel $d$ with channel $d$, an arbitrary correspondence; concatenation stacks them into $2D$ channels; conv fusion adds a learned $1 \times 1$ convolution back to $D$ channels,

$$\mathbf{y}^{\text{conv}} = \mathbf{y}^{\text{cat}} * \mathbf{f} + \mathbf{b}, \qquad \mathbf{f} \in \mathbb{R}^{1 \times 1 \times 2D \times D},$$

which can learn the correspondence; bilinear fusion sums the outer products $\mathbf{x}^{a}_{ij} \mathbf{x}^{b\top}_{ij}$ over all locations, giving $D^2$ values that discard spatial layout. Over time, fused maps from $T$ inputs $\tau$ frames apart form an $H \times W \times T \times D$ volume for 3D pooling, optionally after a 3D convolution.

Why It Works

In the 2014 ablations on UCF101 split 1, the spatial stream trained from scratch overfitted (52.3%); ImageNet pretraining with only the last layer retrained gave 72.7%, full fine-tuning 72.8%. The temporal stream, trained from scratch, did better:

Temporal stream input (UCF101 split 1)	No mean subtraction	Mean subtraction
Single flow field ($L = 1$)	–	73.9%
Optical flow stacking, $L = 5$	–	80.4%
Optical flow stacking, $L = 10$	79.9%	81.0%
Trajectory stacking, $L = 10$	79.6%	80.2%
Optical flow stacking, $L = 10$, bi-directional	–	81.2%

A network on 11 stacked RGB frames trained from scratch reached 56.4%. On split 1, fusion gave 86.2% by averaging and 87.0% by SVM (with the multi-task temporal stream below). Mean accuracy over the three splits (Table 4):

Method	UCF101	HMDB51
Improved dense trajectories (IDT)	85.9%	57.2%
IDT with higher-dimensional encodings	87.9%	61.1%
IDT with stacked Fisher encoding	–	66.8%
Spatial stream	73.0%	40.5%
Temporal stream	83.7%	54.6%
Two-stream, fusion by averaging	86.9%	58.0%
Two-stream, fusion by SVM	88.0%	59.4%

It matched hand-crafted dense-trajectory features on UCF101 but not the best of them on HMDB51; the authors call the results comparable.

Probing a TSN model’s temporal stream on UCF101 (split not stated), Sevilla-Lara et al. (2019) found that shuffling the flow fields within each five-field block at test time lowered accuracy only from 86.85% to 78.64%, and flow computed from shuffled frames still gave 59.55%, far above chance. Recolouring the frames with a random colormap dropped the RGB stream from 85.43% to 34.23% but the flow stream only to 84.30%. The authors conclude that much of flow’s value is its invariance to appearance rather than motion trajectories, and that a flow method’s endpoint error correlates poorly with the recognition accuracy it yields.

Training Considerations

Small video datasets. A UCF101 split has about 9.5K training videos and an HMDB51 split 3.7K. Simonyan and Zisserman therefore gave the temporal stream two softmax heads, one per dataset, and summed their losses (multi-task learning). On HMDB51 split 1 it reached 46.6% alone, 49.0% fine-tuned from UCF101, 52.8% with 78 non-overlapping UCF101 classes added, and 55.4% multi-task.

The cost of flow. The 2014 paper used Brox et al.’s (2004) method on a GPU (OpenCV), 0.06 s per frame pair, still a training bottleneck, so flow was precomputed. TSN and I3D used TV-L1 flow (Zach, Pock & Bischof, 2007); the fusion paper’s CVPR version cites Brox et al., and its arXiv v2 adds TV-L1 for the VGG-16 models.

Storing flow as images. Simonyan and Zisserman rescaled each flow component linearly to $[0, 255]$ and compressed it with JPEG, rescaling back after decoding; for UCF101 this cut 1.5 TB to 27 GB. Feichtenhofer et al. also stored JPEG flow, clipping displacements above 20 pixels.

ImageNet initialization of the temporal stream. The original temporal stream was trained from scratch; Feichtenhofer et al. initialized it from an ImageNet model, reporting faster training at no loss of accuracy. Wang et al.’s (2016) cross-modality pretraining discretizes flow to $[0, 255]$ like an image, averages the ImageNet model’s first-layer weights over the RGB channels and replicates the average across the $2L$ flow channels (transfer learning across modalities). With BN-Inception on UCF101 split 1 it raised the temporal stream from 81.7% to 86.6% (Table 1).

Variants

Convolutional fusion (Feichtenhofer, Pinz & Zisserman, 2016). Late fusion cannot relate what moves to where: in the authors’ example, brushing teeth and brushing hair share a periodic hand motion at different places. Two VGG-M-2048 streams, UCF101 split 1 (Table 1):

Fusion	Layer	Accuracy	Parameters
Sum (averaging, their run)	softmax	85.94%	181.42M
Max	ReLU5	82.70%	97.31M
Concatenation	ReLU5	83.53%	172.81M
Bilinear	ReLU5	85.05%	6.61M + SVM
Sum	ReLU5	85.20%	97.31M
Conv	ReLU5	85.96%	97.58M

Conv fusion after the last convolutional layer matched late fusion with about half the parameters, as one fully connected tower is dropped; earlier fusion was worse (82.25% at ReLU2). With a VGG-16 spatial and a VGG-M temporal stream, both towers kept, 3D conv fusion plus 3D pooling reached 90.40% on UCF101 and 58.63% on HMDB51 (split 1), against 89.35% and 56.93% for 2D conv fusion and 2D pooling. With VGG-16 in both streams the final model reached 92.5% and 65.4% (mean over three splits), and 93.5% and 69.2% averaged with IDT scores. The authors caution that, with small or noisy datasets, some conclusions are tentative.

Temporal segment networks (Wang et al., 2016). TSN trains each stream on $K = 3$ snippets from equal segments of the video, averaging their scores before the loss, and fuses the streams with weights 1 (RGB) and 1.5 (flow): 94.0% on UCF101 and 68.5% on HMDB51 (mean over three splits).

Two-stream I3D (Carreira & Zisserman, 2017). I3D inflates ImageNet Inception-v1 into a 3D network, yet trains one copy on RGB and one on TV-L1 flow separately and averages their predictions; the authors suggest flow helps because flow algorithms are in some sense recurrent, a 3D CNN purely feedforward. In the CVPR version, two-stream I3D with Kinetics pretraining reached 97.9% on UCF101 and 80.2% on HMDB51 (mean over three splits); arXiv v3 reports 97.8–98.0% and 80.7–80.9% for its two pretraining settings.

Example: Each Stream Solves Different Classes

The script below makes clips of $T \times H \times W \times C = 11 \times 32 \times 32 \times 3$ with one moving $8 \times 8$ square on a static textured background. In “red” and “green” the square moves along a random diagonal, so only colour identifies the class; in “right”, “left”, “up” and “down” it is the same white square, so only direction does. The image wraps around at the borders, so position is uninformative.

Flow is computed, not taken from the generator, by exhaustive block matching: at each pixel, the displacement of up to 2 pixels minimizing the colour difference over a $5 \times 5$ window, the shortest on ties. Each stream is a linear softmax classifier on features pooled to a $2 \times 2$ grid: one RGB frame, or the ten flow fields ($L = 10$) split into positive and negative parts. Scores are fused by averaging.

import numpy as np
from scipy.ndimage import uniform_filter

H = W = 32; T = 11; S = 8          # 11 frames -> L = 10 flow fields; an 8x8 square
CLASSES = ["red", "green", "right", "left", "up", "down"]
COLOR = {"red": (1.0, 0.2, 0.2), "green": (0.2, 1.0, 0.2), "white": (0.9, 0.9, 0.9)}
CARDINAL = {"right": (0, 1), "left": (0, -1), "up": (-1, 0), "down": (1, 0)}   # (dy, dx)
DIAGONAL = [(-1, -1), (-1, 1), (1, -1), (1, 1)]

def make_clip(label, rng):
    """A static textured background and one moving 8x8 square. 'red' and 'green'
    differ only in colour (each moves along a random diagonal); the other four are
    the same white square and differ only in direction. The image wraps around at
    the borders, so position says nothing about the class. Returns the clip
    (T, H, W, 3) and the generator's exact flow (T-1, H, W, 2) with channels (u, v)."""
    name = CLASSES[label]
    if name in COLOR:
        color, (dy, dx) = COLOR[name], DIAGONAL[rng.integers(4)]
    else:
        color, (dy, dx) = COLOR["white"], CARDINAL[name]
    speed = rng.integers(1, 3)                       # 1 or 2 pixels per frame
    dy, dx = dy * speed, dx * speed
    y0, x0 = rng.integers(0, H), rng.integers(0, W)
    bg = rng.uniform(0.0, 0.4, (H, W, 1)) * np.ones(3)
    clip, exact = np.empty((T, H, W, 3)), np.zeros((T - 1, H, W, 2))
    for t in range(T):
        mask = np.zeros((H, W), bool)
        mask[:S, :S] = True
        mask = np.roll(mask, (y0 + t * dy, x0 + t * dx), axis=(0, 1))
        f = bg.copy()
        f[mask] = color
        clip[t] = f + rng.normal(0, 0.03, (H, W, 3))
        if t < T - 1:
            exact[t][mask] = (dx, dy)
    return clip, exact

def block_matching_flow(clips, r=2, win=5):
    """Forward flow between consecutive frames by exhaustive block matching: at every
    pixel, the displacement (dx, dy) with |dx|, |dy| <= r that minimizes the colour SSD
    over a win x win window, with a tiny penalty that prefers short vectors on ties.
    clips: (N, T, H, W, 3). Returns (N, T-1, H, W, 2) with channels (u, v) = (dx, dy)."""
    f0, f1 = clips[:, :-1], clips[:, 1:]
    pad = np.pad(f1, ((0, 0), (0, 0), (r, r), (r, r), (0, 0)), mode="wrap")
    best = np.full(f0.shape[:4], np.inf); flow = np.zeros(f0.shape[:4] + (2,))
    for dy in range(-r, r + 1):
        for dx in range(-r, r + 1):
            moved = pad[:, :, r + dy:r + dy + H, r + dx:r + dx + W]      # f1(x + d)
            ssd = uniform_filter(((f0 - moved) ** 2).sum(-1), size=(1, 1, win, win))
            cost = ssd + 1e-4 * (dx * dx + dy * dy)
            better = cost < best
            best[better] = cost[better]
            flow[better] = (dx, dy)
    return flow

def pool(x, p=16):
    """p x p average pooling of (..., H, W, C) maps (here to a 2 x 2 grid), a crude
    stand-in for a CNN's downsampling that keeps the linear streams from memorizing noise."""
    return x.reshape(*x.shape[:-3], H // p, p, W // p, p, x.shape[-1]).mean(axis=(-4, -2))

def softmax(Z):
    E = np.exp(Z - Z.max(1, keepdims=True))
    return E / E.sum(1, keepdims=True)

def train_stream(Xtr, ytr, k, iters=300, lr=0.5, wd=1e-3):
    """One 'stream': multinomial logistic regression on standardized features."""
    mu, sd = Xtr.mean(0), Xtr.std(0) + 1e-6
    X = (Xtr - mu) / sd
    Wt, b, Y = np.zeros((X.shape[1], k)), np.zeros(k), np.eye(k)[ytr]
    for _ in range(iters):
        G = (softmax(X @ Wt + b) - Y) / len(X)
        Wt -= lr * (X.T @ G + wd * Wt); b -= lr * G.sum(0)
    return lambda Xte: softmax(((Xte - mu) / sd) @ Wt + b)

def run(seed, n_train=100, n_test=50, verbose=False):
    rng = np.random.default_rng(seed)
    k = len(CLASSES)
    y_tr, y_te = np.repeat(np.arange(k), n_train), np.repeat(np.arange(k), n_test)
    clips_tr = np.stack([make_clip(c, rng)[0] for c in y_tr])
    te = [make_clip(c, rng) for c in y_te]
    clips_te, exact_te = np.stack([c for c, _ in te]), np.stack([e for _, e in te])

    # Spatial stream: one RGB frame (a random frame in training, the middle one at test).
    rgb_tr = pool(clips_tr[np.arange(len(y_tr)), rng.integers(0, T, len(y_tr))])
    rgb_te = pool(clips_te[:, T // 2])
    # Temporal stream: L = 10 stacked flow fields (2L = 20 channels), each split into
    # positive and negative parts (a ReLU), so a linear model can tell "diagonal" apart.
    flow_tr, flow_te = block_matching_flow(clips_tr), block_matching_flow(clips_te)
    if verbose:
        agree = np.all(flow_te == exact_te, axis=-1)
        on_obj = np.any(exact_te != 0, axis=-1)
        print(f"block matching = exact flow: {agree[on_obj].mean():.2f} of square pixels, "
              f"{agree[~on_obj].mean():.2f} of background pixels")
    rect = lambda f: pool(np.concatenate([np.maximum(f, 0), np.maximum(-f, 0)], -1))

    predict = {"spatial": train_stream(rgb_tr.reshape(len(y_tr), -1), y_tr, k),
               "temporal": train_stream(rect(flow_tr).reshape(len(y_tr), -1), y_tr, k)}
    scores = {"spatial": predict["spatial"](rgb_te.reshape(len(y_te), -1)),
              "temporal": predict["temporal"](rect(flow_te).reshape(len(y_te), -1))}
    scores["fused"] = (scores["spatial"] + scores["temporal"]) / 2    # late fusion: average

    correct = {m: s.argmax(1) == y_te for m, s in scores.items()}
    if verbose:
        print(f"{'class':>6} {'spatial':>8} {'temporal':>9} {'fused':>6}")
        for c in range(k):
            sel = y_te == c
            print(f"{CLASSES[c]:>6} {correct['spatial'][sel].mean():8.2f} "
                  f"{correct['temporal'][sel].mean():9.2f} {correct['fused'][sel].mean():6.2f}")
    return {m: c.mean() for m, c in correct.items()}

for seed in range(5):
    acc = run(seed, verbose=(seed == 0))
    print(f"seed {seed}: " + "  ".join(f"{m} {a:.3f}" for m, a in acc.items()))

block matching = exact flow: 0.42 of square pixels, 0.92 of background pixels
 class  spatial  temporal  fused
   red     1.00      0.40   1.00
 green     1.00      0.58   1.00
 right     0.20      1.00   1.00
  left     0.32      1.00   1.00
    up     0.12      1.00   1.00
  down     0.32      1.00   1.00
seed 0: spatial 0.493  temporal 0.830  fused 1.000
seed 1: spatial 0.497  temporal 0.820  fused 0.997
seed 2: spatial 0.543  temporal 0.847  fused 0.997
seed 3: spatial 0.490  temporal 0.870  fused 1.000
seed 4: spatial 0.530  temporal 0.833  fused 1.000

The spatial stream solves the colour classes and is near chance (0.25) among the directions; the temporal stream solves the directions and is near chance (0.5) between red and green, since flow ignores colour. Averaging gives 0.997–1.000 over seeds 0–4, against at most 0.543 and 0.870 for the single streams. Block matching matches the exact flow on only 42% of the square’s pixels: inside the uniform square any displacement fits about equally well, and along a straight edge only the component across the edge is determined (the aperture problem).

Two choices matter (our reasoning). Coarse pooling stops each linear stream fitting noise, which would make it overconfident and let it dominate the average. Splitting flow into positive and negative parts supplies a CNN’s ReLU nonlinearity: signed flow averages to zero over the four diagonals and over the four cardinal directions alike, so “any diagonal” is not linearly separable from them.

Limitations

• Flow precomputation. Flow must be computed, and usually stored, before training, a cost outside the network that its FLOP counts omit.
• Not end to end. The flow method is fixed and tuned for endpoint error; Sevilla-Lara et al. found that fine-tuning flow networks for classification improved recognition.
• Short temporal extent. The original streams see one frame and ten flow fields, and longer structure comes only from test-time averaging, which TSN and 3D fusion address.

Successors. 3D convolutional networks such as C3D learn spatio-temporal filters from RGB frames; pretrained on Kinetics, I3D’s RGB network alone beat all earlier published UCF101 and HMDB51 results, though adding flow still helped. SlowFast networks (Feichtenhofer et al., 2019) keep two pathways, both on raw frames: a slow one at a low frame rate and a lightweight fast one at a high frame rate. They compute no optical flow and train end to end; their lateral connections between pathways had earlier been used to fuse two-stream networks.