Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

“Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset”, by João Carreira of DeepMind and Andrew Zisserman of DeepMind and the University of Oxford, was posted to arXiv on 22 May 2017 and published at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in July 2017. It asked whether pretraining on a large video dataset helps video models as ImageNet pretraining helps image models, answered with a controlled comparison of five architectures on the new Kinetics dataset, and introduced the Two-Stream Inflated 3D ConvNet, I3D. The paper exists in four versions with different numbers: arXiv v1, the CVPR proceedings, arXiv v2 (20 July 2017) and arXiv v3 (12 February 2018). Results below are labelled by version; the model itself is covered in the I3D article.

Problem

The title comes from Figure 1, a still from the 1951 film Quo Vadis in which a single frame cannot show whether two actors are about to kiss or have just kissed; the caption then asks where action recognition is going. The authors blamed the small size of the existing datasets. UCF101 and HMDB51, the standard benchmarks, had on the order of 10k videos, and most methods scored similarly on them, so they could not single out good video architectures. The best-performing video models stayed close to single-image analysis built on ImageNet classifiers.

Whether a large video dataset would give a boost on other video datasets, as ImageNet had for images, was open (§1). And 3D ConvNets, which model time directly, had stayed relatively shallow (up to 8 layers) and been trained from scratch, because they have many parameters and seemed to rule out ImageNet pretraining (§2). The Discussion returns to the opening question, phrased as “is there a benefit in transfer learning from videos?” (§6).

Contribution

The abstract names two:

• An evaluation on Kinetics. The paper re-evaluates state-of-the-art architectures on the Kinetics Human Action Video Dataset, which it describes as two orders of magnitude larger than HMDB51 and UCF101, with 400 classes and over 400 clips per class from YouTube. It measures how much pretraining on Kinetics helps on the small benchmarks. Despite the title, the dataset was introduced in a companion report by Kay et al. (2017), posted three days earlier with both authors as co-authors; §3 summarizes it and refers to that report.
• I3D. Very deep image classifiers can be inflated into 3D networks by giving their filters and pooling kernels a time dimension, and their ImageNet weights can initialize the result. §2 calls this the main technical contribution and adds that a two-stream configuration is still useful.

The comparison’s design is the method: all models except the C3D-like network share one backbone, Inception-v1 with batch normalization, pretrained on ImageNet, so that differences can be traced to how each model handles time.

Method

The five architectures (Figure 2) cover the main choices of 2017: 2D or 3D kernels, RGB only or also optical flow, and, for 2D networks, recurrence or aggregation over time. Table 1, identical in all four versions, gives their parameter counts and temporal interfaces:

Model	Parameters	Training input	Training footprint	Test input	Test footprint
ConvNet+LSTM	9M	25 RGB frames	5 s	50 RGB	10 s
3D-ConvNet	79M	16 RGB	0.64 s	240 RGB	9.6 s
Two-Stream	12M	1 RGB, 10 flow	0.4 s	25 RGB, 250 flow	10 s
3D-Fused	39M	5 RGB, 50 flow	2 s	25 RGB, 250 flow	10 s
Two-Stream I3D	25M	64 RGB, 64 flow	2.56 s	250 RGB, 250 flow	10 s

• ConvNet+LSTM (“The Old I”). A batch-normalized LSTM with 512 hidden units after Inception-v1’s last average pool, on frames subsampled one in five from 25 fps video; only the last step’s output is used at test time.
• 3D-ConvNet (“The Old II”). A variant of C3D with 8 convolutional, 5 pooling and 2 fully connected layers, on 16-frame $112 \times 112$ clips, trained from scratch. The authors added batch normalization after every layer and a temporal stride of 2 in the first pooling layer, for larger batches (3D convolutional networks covers C3D).
• Two-Stream (“The Old III”). Simonyan and Zisserman’s design with Inception-v1: one RGB frame and a stack of 10 flow frames, predictions averaged (two-stream networks).
• 3D-Fused. An approximation of Feichtenhofer et al.‘s fusion: 5 RGB frames sampled 10 frames apart with their flow, the two streams’ $5 \times 7 \times 7$ feature grids fused by a $3 \times 3 \times 3$ convolution with 512 outputs and a $3 \times 3 \times 3$ max-pool. Both two-stream baselines were trained end to end.
• Two-Stream I3D (“The New”). Inflated Inception-v1 on 64-frame RGB and TV-L1 flow snippets, the two networks trained separately and averaged at test time (I3D explains inflation and initialization).

All models shared one training and testing protocol (§2.5; I3D gives it). The CVPR version and v1–v2 train for up to 35k steps on miniKinetics and 110k on Kinetics; v3 mentions only the 110k Kinetics steps.

miniKinetics. Most CVPR-version experiments used miniKinetics, an early version of Kinetics with 213 classes and 120k clips: 150–1,000 training clips per class, 25 for validation and 75 for testing (§3). The CVPR text gives Kinetics as 400 classes of at least 400 clips with 100 test clips per class; v2 and v3 add a total of 240k training videos.

Checking Table 1

The paper gives no breakdown of its parameter counts. The script below recounts them: Inception-v1 with the module widths and kernel sizes of the released I3D code, kept 2D for the 2D models and inflated for I3D, with 400 Kinetics classes. The C3D widths come from Tran et al.’s Figure 3.

# Inception-v1 modules as in the released I3D code: (1x1 | 1x1 -> 3x3 | 1x1 -> 3x3 | pool -> 1x1) widths.
MIXED = [(64, 96, 128, 16, 32, 32), (128, 128, 192, 32, 96, 64),                        # 3b, 3c
         (192, 96, 208, 16, 48, 64), (160, 112, 224, 24, 64, 64), (128, 128, 256, 24, 64, 64),
         (112, 144, 288, 32, 64, 64), (256, 160, 320, 32, 128, 128),                     # 4b-4f
         (256, 160, 320, 32, 128, 128), (384, 192, 384, 48, 128, 128)]                   # 5b, 5c

def conv(c_in, c_out, k, t):
    """Weights of a convolution with a k x k spatial kernel and temporal extent t (no bias;
    each convolution is followed by batch norm, which we leave out of the count)."""
    return c_in * c_out * t * k * k

def inception_v1(c_in, t3=1):
    """Inception-v1 trunk up to the final average pool. t3 = 1: the 2D network;
    t3 = None: inflated, every N x N filter becomes N x N x N. Returns (weights, output channels)."""
    t = (lambda k: 1) if t3 == 1 else (lambda k: k)
    n = conv(c_in, 64, 7, t(7)) + conv(64, 64, 1, 1) + conv(64, 192, 3, t(3))
    c = 192
    for b0, b1a, b1, b2a, b2, b3 in MIXED:
        n += (conv(c, b0, 1, 1) + conv(c, b1a, 1, 1) + conv(b1a, b1, 3, t(3))
              + conv(c, b2a, 1, 1) + conv(b2a, b2, 3, t(3)) + conv(c, b3, 1, 1))
        c = b0 + b1 + b2 + b3
    return n, c

K = 400                                          # Kinetics classes
fc = lambda a, b: a * b + b                      # fully connected (or 1x1x1) layer with bias

i3d_rgb = inception_v1(3, None)[0] + fc(1024, K)
i3d_flow = inception_v1(2, None)[0] + fc(1024, K)
ts_rgb = inception_v1(3)[0] + fc(1024, K)
ts_flow = inception_v1(20)[0] + fc(1024, K)     # a stack of 10 flow fields = 20 input channels
lstm = inception_v1(3)[0] + 4 * (1024 + 512 + 1) * 512 + fc(512, K)
fused = inception_v1(3)[0] + inception_v1(20)[0] + conv(2 * 1024, 512, 3, 3) + 512 + fc(512, K)
c3d_convs = [(3, 64), (64, 128), (128, 256), (256, 256), (256, 512), (512, 512), (512, 512), (512, 512)]
c3d = sum(conv(a, b, 3, 3) + b for a, b in c3d_convs) + fc(4096, 4096) + fc(4096, K)
c3d_4x4, c3d_3x3 = c3d + fc(512 * 4 * 4, 4096), c3d + fc(512 * 3 * 3, 4096)  # fc6 input: pool5 grid

rows = [("ConvNet+LSTM", lstm, "9M"), ("3D-ConvNet, pool5 1x4x4", c3d_4x4, "79M"),
        ("3D-ConvNet, pool5 1x3x3", c3d_3x3, "79M"), ("Two-Stream", ts_rgb + ts_flow, "12M"),
        ("3D-Fused", fused, "39M"), ("Two-Stream I3D", i3d_rgb + i3d_flow, "25M")]
for name, n, printed in rows:
    print(f"{name:24s} {n / 1e6:6.2f}M   (Table 1: {printed})")
print(f"RGB I3D alone {i3d_rgb / 1e6:.2f}M, flow I3D alone {i3d_flow / 1e6:.2f}M")

ConvNet+LSTM               8.94M   (Table 1: 9M)
3D-ConvNet, pool5 1x4x4   79.63M   (Table 1: 79M)
3D-ConvNet, pool5 1x3x3   64.95M   (Table 1: 79M)
Two-Stream                12.04M   (Table 1: 12M)
3D-Fused                  39.74M   (Table 1: 39M)
Two-Stream I3D            25.34M   (Table 1: 25M)
RGB I3D alone 12.68M, flow I3D alone 12.66M

Every printed count is matched within 1M, under assumptions that are ours: the 3D-Fused network concatenates the two 1024-channel streams before its fusion convolution, and the C3D-like network’s last pooling layer leaves a $4 \times 4$ grid, so that its first fully connected layer has $512 \times 16$ inputs; with a $3 \times 3$ grid it would have about 65M parameters. Each I3D stream, about 12.7M, is twice the size of 2D Inception-v1, while the fully connected layers alone hold about 52M of the C3D-like network’s parameters, consistent with the paper’s remark that I3D is much deeper yet has far fewer parameters (§5).

Results

Training on one dataset (Table 2)

On split 1 of UCF101 and HMDB51, all four versions print the same numbers:

Architecture	UCF101 RGB	Flow	RGB + Flow	HMDB51 RGB	Flow	RGB + Flow
ConvNet+LSTM	81.0	–	–	36.0	–	–
3D-ConvNet	51.6	–	–	24.3	–	–
Two-Stream	83.6	85.6	91.2	43.2	56.3	58.3
3D-Fused	83.2	85.8	89.3	49.2	55.5	56.8
Two-Stream I3D	84.5	90.6	93.4	49.8	61.9	66.4

The third dataset differs. The CVPR version, v1 and v2 train and test on miniKinetics; arXiv v3 replaces it with Kinetics (held-out test set, top-1 accuracy):

Architecture	miniKinetics (CVPR) RGB	Flow	RGB + Flow	Kinetics (v3) RGB	Flow	RGB + Flow
ConvNet+LSTM	69.9	–	–	63.3	–	–
3D-ConvNet	60.0	–	–	56.1	–	–
Two-Stream	70.1	58.4	72.9	62.2	52.4	65.6
3D-Fused	71.4	61.0	74.0	–	–	67.2
Two-Stream I3D	74.1	69.6	78.7	71.1	63.4	74.2

I3D was best in every column, evidence to the authors that ImageNet pretraining extends to 3D networks. Flow was worth more than RGB on UCF101 and much more on HMDB51, but less on Kinetics, where the authors suspected camera motion; they attributed I3D’s better use of flow to its 64-frame temporal receptive field against 10 frames. The CVPR version (and v1–v2) also calls the LSTM and the 3D-ConvNet data-hungry, being much more competitive on miniKinetics; v3 drops that remark. arXiv v2 added a Table 3 of I3D on the full Kinetics test set: two-stream I3D reached 74.2% top-1 and 91.3% top-5 from ImageNet, 71.6% and 90.0% from scratch. v3’s Table 3 extends this comparison to the other models and finds that ImageNet helped every model, slightly more for RGB.

Pretraining, then fine-tuning (CVPR and v1 Table 3, v2 Table 4)

The CVPR version’s Table 3 pretrains on miniKinetics and compares training on the target data alone (Original), training only the last layer on frozen features (Fixed) and fine-tuning end to end (Full-FT), on split 1:

Architecture	UCF101 Original	Fixed	Full-FT	Δ	HMDB51 Original	Fixed	Full-FT	Δ
ConvNet+LSTM	81.0	81.6	82.1	−6%	36.0	46.6	46.4	−10.6%
3D-ConvNet	49.2	76.0	79.9	−60.5%	24.3	47.5	49.4	−25.1%
Two-Stream	91.2	90.3	91.5	−3.4%	58.3	64.0	58.7	−5.7%
3D-Fused	89.3	88.5	90.1	−7.5%	56.8	59.0	61.4	−4.6%
Two-Stream I3D	93.4	95.7	96.5	−47.0%	66.4	74.3	75.9	−9.5%

Every architecture gained from pretraining, I3D and the 3D-ConvNet most; for I3D even the frozen features beat training on the target data, which the authors attribute to its high temporal resolution.

The table disagrees with itself in two places. The 3D-ConvNet’s Original UCF101 entry is 49.2, while Table 2 gives 51.6 for the same setting; arXiv v1 also prints 49.2, and v2 (Table 4) changes it to 51.6 with Δ recomputed as −58.5%. And the caption defines Δ as the relative change in misclassification. The UCF101 column fits (for I3D, error falls from 6.6% to 3.5%, −47.0%), but the HMDB51 entries are differences in accuracy points: 75.9 − 66.4 = 9.5, where the relative reduction is 28.3%, the value arXiv v1 and v2 print in that column. arXiv v3’s Table 4 repeats the comparison after full Kinetics pretraining, with and without ImageNet weights and without the Δ column: two-stream I3D reached 98.0% (UCF101) and 81.2% (HMDB51) fine-tuned from ImageNet and Kinetics, and 97.6% and 81.3% from Kinetics alone.

Comparison with earlier work (CVPR and v1 Table 4, v2 and v3 Table 5)

Averaged over the three splits, the best earlier result was ST-ResNet with IDT, 94.6% on UCF101 and 70.3% on HMDB51. The I3D rows changed with every version:

Model (pretraining)	CVPR	arXiv v1 and v2	arXiv v3
RGB-I3D (miniKinetics)	91.8 / 66.4	91.8 / 66.4	–
Flow-I3D (miniKinetics)	94.7 / 72.4	94.7 / 72.4	–
Two-Stream I3D (miniKinetics)	96.9 / 76.3	96.9 / 76.3	–
RGB-I3D (Kinetics; v3: ImageNet + Kinetics)	95.4 / 74.5	95.6 / 74.8	95.6 / 74.8
Flow-I3D (same)	95.4 / 74.6	96.7 / 77.1	96.7 / 77.1
Two-Stream I3D (same)	97.9 / 80.2	98.0 / 80.7	98.0 / 80.7
RGB-I3D (Kinetics only, v3)	–	–	95.1 / 74.3
Flow-I3D (Kinetics only, v3)	–	–	96.5 / 77.3
Two-Stream I3D (Kinetics only, v3)	–	–	97.8 / 80.9

In every version, either stream alone beat every earlier result in the table once pretrained on Kinetics. The abstracts quote the two-stream figures: 97.9% and 80.2% in the CVPR version, 98.0% and 80.7% in v1 and v2, and 98.0% and 80.9% in v3, which takes each dataset’s better row. The CVPR text calls its figures 57% and 33% reductions in misclassification against ST-ResNet with IDT. By our arithmetic, UCF101 error falls from 5.4% to 2.1%, a 61% reduction (33% checks for HMDB51); v1–v3 print 63% and 35%, which fit their 98.0% and 80.7%.

Impact

At CVPR 2017, a team of Carreira, Brian Zhang and Zisserman won the Charades Challenge, held with the Workshop on Visual Understanding Across Modalities, in both its classification and localization tracks. Its leaderboard entries came from a single fine-tuned RGB I3D with dense per-frame outputs: 0.3441 mAP for classification against 0.2974 for the runner-up, and 0.2072 against 0.1803 for localization (challenge page, archived copy). The official I3D repository notes the same win and released Kinetics-trained RGB and flow checkpoints, as the paper had promised (README, as of October 2026).

I3D was reused as a backbone and as a frozen feature extractor, for instance by the Fréchet Video Distance and by temporal action localization methods (I3D gives the details). Kinetics pretraining followed by fine-tuning (transfer learning) became a standard recipe for action recognition, and the dataset grew into Kinetics-600 and -700 (Kinetics). At 98.0%, I3D left about 2% error on UCF101.

Limitations

The authors named some limits themselves: no flipped test views or photometric augmentation (§2.5); no action tubes, attention or the other recent representations they surveyed in §6; and no evidence yet that Kinetics pretraining helps tasks other than classification, such as video segmentation, detection or optical flow (§6). Their 3D network also still benefited from a flow stream (§2.4).

The model’s cost, precomputed flow and 2.56 s training context are discussed in I3D. Specific to the publication:

• Version drift. Headline numbers differ between versions, and most CVPR-version experiments rest on miniKinetics. Quoting “I3D” needs the version, pretraining and split; action recognition shows how later papers mixed them.

What Came After

The CVPR version ended by promising to repeat every experiment on full Kinetics, with and without ImageNet, and to inflate other image networks (§6). arXiv v2 added full-Kinetics test results for I3D, and v3 reran the comparison. Others took up inflation: Non-local Networks inflated ResNets and S3D made I3D’s convolutions separable (I3D covers both). SlowFast kept two pathways but dropped optical flow and ImageNet initialization, and video transformers replaced convolution with attention.