SlowFast Networks for Video Recognition

“SlowFast Networks for Video Recognition” by Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik and Kaiming He of Facebook AI Research (FAIR) appeared on arXiv in December 2018 and was presented at the IEEE/CVF International Conference on Computer Vision (ICCV) in Seoul in 2019. It proposed reading one clip at two frame rates: a Slow pathway with few frames and many channels, and a Fast pathway with many frames and few channels, fused by lateral connections. Trained on Kinetics without ImageNet pretraining, it reported state-of-the-art results on Kinetics-400, Kinetics-600, Charades and AVA. The third arXiv version (October 2019) has the same text and tables as the proceedings, which this article follows.

Problem

By 2018, deep action recognition had two main families. Two-stream networks (Simonyan & Zisserman, 2014) added a second network fed with optical flow; the paper objects that flow is a hand-designed representation, often not learned end to end with the recognizer. 3D convolutional networks learned motion from frames, but handled space and time alike, with the same kind of kernels along each axis.

The paper’s opening argument targets that symmetry. Treating the two image axes alike is justified, it says, because natural image statistics are approximately isotropic: all orientations are about equally likely. Motion is the spatiotemporal counterpart of orientation, but spatiotemporal orientations are not equally likely: slow motions are more common than fast ones, and most of the world is at rest at any moment. The authors cite Bayesian accounts of human motion perception, in which a prior favouring slow motion explains the aperture-problem percept. If orientations in space-time are not equally likely, they conclude, nothing requires treating space and time symmetrically.

They pair this with a recognition argument. Categories change slowly (waving hands remain hands; a person who starts running is still a person), while the motion itself (clapping, waving, jumping) can change much faster. Appearance can therefore be refreshed at a low frame rate and motion should be sampled at a high one.

Contribution

The authors present SlowFast as a generic concept rather than one network. The paper has no numbered contribution list; its introduction and Section 3 put forward:

1. Two pathways at two speeds. A Slow pathway with a large temporal stride for spatial semantics, and a Fast pathway that samples the same clip $\alpha$ times more densely (typically 8) and keeps that temporal resolution through every layer. The paper treats $\alpha$ as the core of the concept.
2. A lightweight Fast pathway. Its channels are a fraction $\beta$ of the Slow pathway’s (typically $1/8$), so it takes, by the authors’ estimate, about 20% of the total computation. They read the low channel capacity as a deliberately weak spatial model, and tested weaker inputs (half resolution, gray-scale, frame differences) on the same reasoning.
3. Lateral connections from Fast to Slow, one per stage, as in earlier two-stream fusion networks and feature pyramid detectors; bidirectional fusion gave similar results.
4. No optical flow and no ImageNet. Both pathways learn from raw frames end to end, and the Kinetics models were trained from random initialization.

The paper also contrasts itself with two-stream networks on three points: those networks do not exploit different temporal speeds, use the same backbone for both streams, and depend on flow.

Biological inspiration. The authors say the design was partly inspired by studies of primate retinal ganglion cells, while admitting the analogy is “rough and premature”. In their summary, about 80% of these cells are parvocellular (fine spatial detail and colour, slow responses) and 15–20% magnocellular (fast temporal changes, insensitive to detail and colour). The Fast pathway mirrors the M-cells in three ways: it runs at high temporal resolution, models little spatial detail, and is a small share of the network.

Method

The SlowFast article covers the published design: the sampling ($T$ Slow frames at stride $\tau$, $\alpha T$ Fast frames, $\beta$ times the channels), the two-pathway 3D ResNet-50 of Table 1, the three lateral variants, the training recipe, and the AVA detector, which the authors describe as similar to Faster R-CNN. Models are named $T \times \tau$ plus backbone.

Results

All results are as printed. Kinetics inference used 10 clips × 3 crops of $256 \times 256$, and because papers differ in how many views they average, the paper reports cost as GFLOPs per view × views.

Kinetics-400 (Table 2). The best model, SlowFast 16×8 R101+NL, reached 79.8% top-1. The authors highlight two margins: 2.1 points over Non-local R101 (77.7%, ImageNet-pretrained; Wang et al., 2018) and 5.9 points over the best earlier result without ImageNet pretraining, R(2+1)D two-stream at 73.9%. They also stress cost: their models use 30 views, while R(2+1)D used 115 and ARTNet 250, and for several earlier models the number was not available. The same architectures reached 78.8% to 81.8% on Kinetics-600 (Table 3), against 79.0% for the best single model of the ActivityNet Challenge 2018, which was pretrained on ImageNet and Kinetics-400.

Charades (Table 4). Fine-tuned from Kinetics-400, the 16×8 R101 Slow-only baseline scored 39.0 mAP; adding the Fast pathway gave 42.1, non-local blocks 42.5, and Kinetics-600 pretraining 45.2. The previous best, STRG, had 39.7 at 630 GFLOPs × 30 views, against 234 × 30 for the best SlowFast.

Accuracy against compute (Figure 2). Six Slow-only models, from 2×32 R50 to 16×8 R101, each gain from the Fast pathway: +3.3, +3.0 and +3.4 points for the models with 2 or 4 Slow frames, +2.1 and +2.0 for those with 8, and +1.7 for 16×8 R101. In our reading, the gain shrinks as the Slow pathway itself becomes denser in time, though it never disappears. The figure’s red arrow marks SlowFast 8×8 R101, which is more accurate than Slow-only 16×8 R101 at lower cost.

Ablations (Table 5, 4×16 R50 on Kinetics-400). The Fast pathway alone reaches only 51.7%, yet adds up to 3.0 points to Slow-only’s 72.6%. Concatenating pooled outputs without laterals gave just 0.9 points, which suggests that most of the gain comes from fusion inside the network (our reading). Gray-scale Fast input matched RGB within 0.1 point, consistent, the authors note, with M-cells’ insensitivity to colour; optical flow as Fast input gave 73.8%, below RGB’s 75.6%. The channel ratio sweep (Table 5b) shows how cheaply that accuracy comes. The code below fits the printed extra cost over Slow-only with one term quadratic in $\beta$ (most Fast layers, whose input and output widths both scale with $\beta$) and one linear (the laterals and the Slow-layer inputs they widen). It uses only the table’s numbers:

import numpy as np

# Table 5b (SlowFast 4x16, R50, alpha = 8, one 256^2 clip): beta -> GFLOPs, top-1
slow_only = 27.3
beta = np.array([1/4, 1/6, 1/8, 1/12, 1/16, 1/32])
gflops = np.array([54.5, 41.8, 36.1, 32.8, 30.6, 28.6])
top1 = np.array([75.6, 75.8, 75.6, 75.2, 75.1, 74.2])

# Extra cost over Slow-only = Fast pathway (most layers scale with beta^2)
# + laterals and the Slow layers that read their channels (scale with beta)
extra = gflops - slow_only
A = np.stack([beta**2, beta], axis=1)
(a, b), *_ = np.linalg.lstsq(A, extra, rcond=None)
print(f"fit: extra GFLOPs = {a:.1f} * beta^2 + {b:.1f} * beta")
print(" beta  printed  fitted  extra share  gain over Slow-only")
for bt, g, e, t in zip(beta, gflops, A @ np.array([a, b]), top1):
    print(f" 1/{round(1/bt):<3d} {g:6.1f}  {slow_only + e:6.1f}   {(g - slow_only) / g:6.1%}       {t - 72.6:+.1f}")

fit: extra GFLOPs = 284.9 * beta^2 + 37.8 * beta
 beta  printed  fitted  extra share  gain over Slow-only
 1/4     54.5    54.6    49.9%       +3.0
 1/6     41.8    41.5    34.7%       +3.2
 1/8     36.1    36.5    24.4%       +3.0
 1/12    32.8    32.4    16.8%       +2.6
 1/16    30.6    30.8    10.8%       +2.5
 1/32    28.6    28.8     4.5%       +1.6

The two-term model, our interpretation of where the cost goes, matches every printed cost within 0.4 GFLOPs, and halving $\beta$ from 1/4 to 1/8 cuts the added share from half the network to about a quarter with no loss of accuracy. At the default, everything added to Slow-only takes 24%, more than the text’s “∼20%”, which fits the Fast pathway alone (6.4 of 36.1 GFLOPs, about 18%; our arithmetic, consistent with the layer-by-layer count in SlowFast).

Training from scratch (Table 6). To check that training from scratch did not handicap their system, the authors retrained the Non-local paper’s 3D ResNet-50 from scratch. With their recipe it reached 73.5%, against 73.4% for the ImageNet-initialized original; with the Non-local paper’s own recipe, training from scratch reached only 69.4%. For SlowFast itself, they report that ImageNet-pretrained and from-scratch versions performed similarly (±0.3%), without a table.

AVA (Tables 7–9). On AVA v2.1 validation (frame-mAP at IoU 0.5, 60 classes; spatio-temporal action detection defines the task), SlowFast 8×8 R101 pretrained on Kinetics-400 reached 26.3 mAP. By Table 7’s entries, that is 4.6 points above the best earlier Kinetics-400 single model (ATR with flow, 21.7) and 6.3 above the best without flow (ATR, 20.0; our arithmetic); the text gives 5.6 and 7.3. With Kinetics-600 pretraining and non-local blocks, the model reached 27.3 on validation and 27.1 on the test server. Training also on predicted proposals that overlap ground truth raised validation to 28.2; on AVA v2.2 the same setup gave 29.0, and the 16×8 model 29.8 (Table 8). In the ablation (Table 9), the Fast pathway lifted a 4×16 R50 detector from 19.0 to 24.2 mAP, far more than optical flow’s +1.1 and +1.7 in earlier AVA detectors.

Impact

According to the paper, a 7-model SlowFast ensemble with multi-scale and flip testing reached 34.3 mAP on the AVA v2.2 test set and ranked first in the AVA action detection task of the ActivityNet Challenge 2019 (Table 8; the official 2019 task page confirms that the challenge used AVA v2.2 and frame-mAP at IoU 0.5). The authors released their code as PySlowFast, which its README says was released with their ICCV 2019 tutorial; as of October 2026 it also implements Slow, C2D, I3D, Non-local networks, X3D and MViT.

Limitations

The paper states few limitations of its own: it calls the biological analogy rough and notes that early temporal kernels hurt the sparse Slow pathway. Others were identified later or follow from the design:

• Cost and runtime. TimeSformer’s authors reported 14.88 hours for SlowFast to process the 20k Kinetics-400 validation videos on 8 V100 GPUs, against 2.6 hours for TimeSformer-L, and found that SlowFast needed at least 5 test clips to approach its best accuracy (Bertasius et al., 2021; see action recognition).
• Reporting gaps. The paper does not print $\alpha$ for its 8×8 and 16×8 models; the ±0.3% ImageNet comparison has no table; and two AVA margins in the text disagree with Table 7.
• Fixed rates and external proposals. In our reading, $\tau$ and $\alpha$ are fixed per model rather than adapted to content, and the AVA detector relies on a separately trained person detector, as earlier AVA systems did.

What Came After

Two follow-ups came from the same group. X3D (Feichtenhofer, 2020) noted that the Fast pathway is very thin but weak on its own, and that the paper’s exploration kept the costly, image-derived Slow pathway fixed; it asked whether that pathway was needed at all. Its single-pathway networks are thin but high in spatiotemporal resolution; X3D-XL came within 0.7 points of SlowFast 16×8 R101+NL on Kinetics-400 with 4.8 times fewer multiply-adds, by X3D’s own comparison (3D convolutional networks covers X3D). MViT (Fan et al., 2021), a multiscale video transformer trained from scratch on Kinetics, was released in the same repository and compared with SlowFast in its Kinetics tables.