I3D

I3D, the Two-Stream Inflated 3D ConvNet, is a video classification model introduced by Carreira and Zisserman at CVPR 2017. It takes an image convolutional neural network, Inception-v1 with batch normalization, and inflates it: every 2D filter and pooling kernel gains a time dimension, and the 3D filters can be initialized from the network’s ImageNet weights. One inflated network reads RGB frames and a second reads optical flow, as in two-stream networks. Trained on Kinetics, I3D became a widely used action recognition model, backbone and video feature extractor. The mathematics of 3D convolution is in 3D convolutional networks.

Problem Addressed

Before 2017, 3D networks such as C3D were relatively shallow (up to 8 layers) and trained from scratch: a 3D network seemed to rule out ImageNet pretraining, and benchmarks of the order of 10k videos were too small for many parameters. The best UCF101 and HMDB51 results came from 2D two-stream networks built on ImageNet models. I3D’s authors proposed reusing both the design and the weights of proven image networks rather than designing video architectures by trial and error, with the much larger Kinetics dataset for training.

Architecture

Inflated Inception-v1

The backbone is Inception-v1 with every square $N \times N$ filter made cubic, $N \times N \times N$, and the pooling kernels given a temporal extent. The paper’s Figure 3 gives the layers, with kernel sizes as $t \times h \times w$ (time first) and strides of 1 where none is printed:

• a $7 \times 7 \times 7$ convolution with stride 2 (in time too, as in the released code);
• a $1 \times 3 \times 3$ max-pool with stride $1, 2, 2$, a $1 \times 1 \times 1$ convolution, a $3 \times 3 \times 3$ convolution and another $1 \times 3 \times 3$ max-pool with stride $1, 2, 2$;
• two Inception modules, a $3 \times 3 \times 3$ max-pool with stride 2, five Inception modules, a $2 \times 2 \times 2$ max-pool with stride 2, and two more Inception modules;
• a $2 \times 7 \times 7$ average pool and a $1 \times 1 \times 1$ convolution that outputs class scores.

Each Inception module concatenates four branches: a $1 \times 1 \times 1$ convolution, two branches of a $1 \times 1 \times 1$ convolution followed by a $3 \times 3 \times 3$ one, and a $3 \times 3 \times 3$ max-pool followed by a $1 \times 1 \times 1$ convolution. Every convolution except the last is followed by batch normalization and a ReLU.

Training input is a 64-frame snippet at 25 frames per second, 2.56 s (temporal modeling defines the sampling terms). At this frame rate, the authors found it helpful not to pool in time in the first two max-pools. Their argument concerns pacing the receptive field: the right growth rate in time depends on frame rate and image size; too fast may merge edges of different objects, too slow may miss scene dynamics. The printed theoretical receptive fields grow from 7 frames and $11 \times 11$ pixels after the first max-pool to 99 frames and $539 \times 539$ pixels at the final average pool, more than the 64-frame training snippet. By our arithmetic, these values fit four middle Inception modules; with the five drawn (as in the released code), the last would be 107 frames and $571 \times 571$ pixels.

Table 1 of the paper lists 25M parameters for the two-stream I3D; Xie et al. (2018) count 12.06M parameters and 107.89 GFLOPs for a single RGB I3D on 64 frames of $224 \times 224$.

Bootstrapping from ImageNet

An image can be turned into a “boring” video by copying it into every frame. The authors ask that the 3D network satisfy the boring-video fixed point: on a boring video, its pooled activations should equal those of the 2D network on the image. Repeating each 2D filter $W$ $N$ times along time and dividing by $N$ achieves this. For a clip $x_s = I$ at every frame $s$,

$$\sum_{a=0}^{N-1} \sum_{u,v} \frac{W[u,v]}{N}\, x_{s+a}[i+u,\, j+v] = \sum_{u,v} W[u,v]\, I[i+u,\, j+v],$$

so each inflated convolution reproduces the 2D response, with the bias unchanged. The result is constant in time, so pointwise nonlinearities and average or max pooling leave it unchanged, and every layer matches the image network. Batch normalization, which the paper does not discuss, is a per-channel affine map in inference mode, so in our reading it preserves the identity too. Only linearity is used, so any temporal extent $N$ works.

In a finite clip with zero padding in time, though, frames within $\lfloor N/2 \rfloor$ of each end see padded zeros, and the error spreads inward by that amount at each layer. The output equals the 2D network’s only away from the temporal borders, and a global average over time includes the border frames.

Two Streams

A 3D network can learn motion from RGB, yet the authors still found a flow stream useful, suggesting that a feedforward network lacks the iterative optimization of optical flow estimation. One I3D reads RGB, the other TV-L1 flow; they are trained separately and their predictions averaged at test time. The released code combines the streams by adding their logits with equal weights (README).

Training

Training used SGD with momentum 0.9, synchronized across 64 GPUs for “the 3D ConvNets” (I3D included, in our reading), with a tenfold learning-rate reduction when validation loss saturated: 110k steps on Kinetics, and up to 5k steps of fine-tuning on UCF101 and HMDB51. Augmentation resized the shorter side to 256 pixels and cropped a random $224 \times 224$ patch, chose a random start frame (looping short videos) and flipped videos left–right, consistently across frames. Most CVPR-version experiments used miniKinetics (213 classes); arXiv v3 reran them on full Kinetics.

For the released checkpoints, the README adds: 64 replicas per stream, minibatches of 6, dropout 0.5, weight decay $10^{-7}$, and 115k steps for RGB but 155k for flow, which needed more training.

Inference

At test time I3D runs convolutionally in time over the whole video (250 frames per stream for a 10-second clip, Table 1) on the $224 \times 224$ center crop and averages the predictions over time. Spatially convolutional testing on $256 \times 256$ frames did not help, and flipped test views were not used.

Inflation in Code

3D convolutional networks checks the identity for a single filter. The NumPy code below follows the border error through a three-layer network inflated with temporal extents 3 and 5, pooling in time only at the second max-pool as I3D does, on a 24-frame static clip of a synthetic image.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

def conv3d(x, W, b, pad_t):
    """x: (C, T, H, W) clip; W: (C_out, C, t, h, w); 'same' spatial zero padding,
    pad_t frames of zero padding in time; stride 1."""
    _, _, t, h, w = W.shape
    xp = np.pad(x, ((0, 0), (pad_t, pad_t), (h // 2, h // 2), (w // 2, w // 2)))
    win = sliding_window_view(xp, (t, h, w), axis=(1, 2, 3))
    return np.einsum("cTHWijk,ocijk->oTHW", win, W) + b[:, None, None, None]

def maxpool(x, kt, ks):
    """Max pooling with kernel = stride = (kt, ks, ks), no padding."""
    C, T, H, W = x.shape
    x = x[:, :T // kt * kt, :H // ks * ks, :W // ks * ks]
    return x.reshape(C, T // kt, kt, H // ks, ks, W // ks, ks).max(axis=(2, 4, 6))

def inflate(W2, n):
    """2D kernels (C_out, C, h, w) -> 3D kernels (C_out, C, n, h, w): repeat n times along time, divide by n."""
    return np.repeat(W2[:, :, None], n, axis=2) / n

def net(x, params, pad_t, pool_t):
    """conv-ReLU-maxpool, conv-ReLU-maxpool, conv-ReLU; returns the last feature map and the
    globally average-pooled feature vector. A 2D net is the same code on a 1-frame clip with t = 1."""
    (W1, b1), (W2, b2), (W3, b3) = params
    x = maxpool(np.maximum(conv3d(x, W1, b1, pad_t), 0), pool_t[0], 2)
    x = maxpool(np.maximum(conv3d(x, W2, b2, pad_t), 0), pool_t[1], 2)
    x = np.maximum(conv3d(x, W3, b3, pad_t), 0)
    return x, x.mean(axis=(1, 2, 3))

rng = np.random.default_rng(0)
widths = [3, 8, 16, 16]
params2d = [(rng.standard_normal((o, i, 3, 3)) / np.sqrt(9 * i), 0.1 * rng.standard_normal(o))
            for i, o in zip(widths[:-1], widths[1:])]
# One RGB image (C, 1, H, W): a gradient background with a red disk and a blue square.
yy, xx = np.mgrid[:32, :32]
img = np.stack([xx / 31, yy / 31, np.full((32, 32), 0.5)])
img[:, (yy - 10) ** 2 + (xx - 10) ** 2 < 25] = np.array([1.0, 0.2, 0.2])[:, None]
img[:, 18:27, 16:25] = np.array([0.1, 0.3, 1.0])[:, None, None]
frame = img[:, None]
T = 24
boring = np.repeat(frame, T, axis=1)                     # the "boring video": the image copied 24 times

# 2D network: kernels as 1 x 3 x 3 (time first), applied to the single frame.
f2, v2 = net(frame, [(W[:, :, None], b) for W, b in params2d], pad_t=0, pool_t=(1, 1))
print("2D feature map", f2.shape)

for n in (3, 5):
    params3d = [(inflate(W, n), b) for W, b in params2d]
    # (a) zero padding in time, as in 'same' 3D convolution; I3D-style pooling: no temporal pooling
    #     in the first max-pool, temporal pooling 2 in the second
    f3, v3 = net(boring, params3d, pad_t=n // 2, pool_t=(1, 2))
    per_frame = np.abs(f3 - f2).max(axis=(0, 2, 3))      # max |difference| at each output time step
    print(f"\nN = {n}: 3D feature map {f3.shape}")
    print("  max |3D - 2D| per output time step:", " ".join(f"{d:.0e}" for d in per_frame))
    print(f"  pooled feature vector, max |3D - 2D| = {np.abs(v3 - v2).max():.2e}"
          f"  (relative {np.abs(v3 - v2).max() / np.abs(v2).max():.1%})")
    # (b) no temporal padding: every output frame sees N real frames
    f3v, v3v = net(boring, params3d, pad_t=0, pool_t=(1, 2))
    print(f"  no temporal padding: feature map {f3v.shape}, max |3D - 2D| = {np.abs(f3v - f2).max():.0e},"
          f" pooled {np.abs(v3v - v2).max():.0e}")

2D feature map (16, 1, 8, 8)

N = 3: 3D feature map (16, 12, 8, 8)
  max |3D - 2D| per output time step: 2e-01 2e-02 7e-16 7e-16 7e-16 7e-16 7e-16 7e-16 7e-16 7e-16 2e-02 2e-01
  pooled feature vector, max |3D - 2D| = 1.69e-02  (relative 7.8%)
  no temporal padding: feature map (16, 8, 8, 8), max |3D - 2D| = 7e-16, pooled 6e-17

N = 5: 3D feature map (16, 12, 8, 8)
  max |3D - 2D| per output time step: 3e-01 2e-01 5e-02 6e-03 9e-16 9e-16 9e-16 9e-16 6e-03 5e-02 2e-01 3e-01
  pooled feature vector, max |3D - 2D| = 3.37e-02  (relative 15.5%)
  no temporal padding: feature map (16, 4, 8, 8), max |3D - 2D| = 9e-16, pooled 4e-17

Away from the borders the inflated network reproduces the 2D feature map to rounding, through ReLUs and pooling. With zero padding, two output steps at each end differ for $N = 3$ and four for $N = 5$, and the pooled vector is off by up to 7.8% and 15.5% of its largest entry. Without temporal padding the identity holds to rounding everywhere, at the price of a shorter output.

Does the Initialization Help?

The PyTorch snippet below (PyTorch 2.14.1) mimics I3D’s recipe at toy scale. A 2D network first learns to tell a square, a ring and a cross apart in single noisy frames, standing in for ImageNet. The target task has six classes: each shape moving right or left over 8 frames, a left clip being a right clip reversed, so direction is invisible in any single frame. One 3D network is trained with 4 or 16 clips per class, either inflated from the 2D network (with a new classifier) or randomly initialized, over seeds 0–4.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(4)
print("torch", torch.__version__)
T, S = 8, 20                                         # frames, frame size
SHAPES = ["square", "ring", "cross"]

def draw(name, y, x):
    """A 7 x 7 binary shape at (y, x) in an S x S frame."""
    m = torch.zeros(S, S)
    if name == "square":
        m[y:y + 7, x:x + 7] = 1
    elif name == "ring":
        m[y:y + 7, x:x + 7] = 1
        m[y + 2:y + 5, x + 2:x + 5] = 0
    else:
        m[y + 3, x:x + 7] = 1
        m[y:y + 7, x + 3] = 1
    return m

def make_frames(n, gen, noise=0.8):
    """Appearance task for the 2D network: which shape is in a single noisy frame?"""
    labels = torch.randint(0, 3, (n,), generator=gen)
    frames = noise * torch.randn(n, 1, S, S, generator=gen)
    for i, c in enumerate(labels.tolist()):
        y, x = torch.randint(0, S - 7, (2,), generator=gen).tolist()
        frames[i, 0] += draw(SHAPES[c], y, x)
    return frames, labels

def make_clips(n_per_class, gen, noise=0.8):
    """Motion task: 6 classes = 3 shapes x 2 directions (moving right or left, 1 px per frame).
    Clips (N, 1, T, S, S); a 'left' clip shows the frames of a 'right' clip in reverse order."""
    clips, labels = [], []
    for c in range(6):
        shape, left = SHAPES[c // 2], c % 2
        for _ in range(n_per_class):
            y = torch.randint(0, S - 7, (1,), generator=gen).item()
            x0 = torch.randint(0, S - 7 - (T - 1), (1,), generator=gen).item()
            clip = noise * torch.randn(1, T, S, S, generator=gen)
            for t in range(T):
                step = T - 1 - t if left else t
                clip[0, t] += draw(shape, y, x0 + step)
            clips.append(clip)
            labels.append(c)
    return torch.stack(clips), torch.tensor(labels)

class Net(nn.Module):
    """Three conv-BN-ReLU layers, max pooling, global average pooling, linear classifier.
    t = 1 gives the 2D network (kernels 1 x 3 x 3); t = 3 the 3D one. As in I3D,
    the first max-pool does not pool in time; the second does."""
    def __init__(self, t, n_classes, width=12):
        super().__init__()
        self.t = t
        self.convs = nn.ModuleList(nn.Conv3d(i, o, (t, 3, 3), padding=(t // 2, 1, 1), bias=False)
                                   for i, o in [(1, width), (width, width), (width, width)])
        self.bns = nn.ModuleList(nn.BatchNorm3d(width) for _ in range(3))
        self.fc = nn.Linear(width, n_classes)

    def features(self, x):                           # x: (N, C, T, H, W)
        x = F.max_pool3d(F.relu(self.bns[0](self.convs[0](x))), (1, 2, 2))
        x = F.max_pool3d(F.relu(self.bns[1](self.convs[1](x))), (2 if self.t > 1 else 1, 2, 2))
        return F.relu(self.bns[2](self.convs[2](x)))

    def forward(self, x):
        return self.fc(self.features(x).mean(dim=(2, 3, 4)))

def train(net, X, y, steps, lr=2e-3, batch=16):
    net.train()
    opt = torch.optim.Adam(net.parameters(), lr=lr)
    for _ in range(steps):
        idx = torch.randint(0, len(y), (batch,))
        loss = F.cross_entropy(net(X[idx]), y[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    return net

def accuracy(net, X, y):
    net.eval()
    with torch.no_grad():
        return (net(X).argmax(1) == y).float().mean().item()

def inflate_from(net2d, n_classes, n=3):
    """Build a 3D net whose conv kernels are the 2D kernels repeated n times in time, divided by n.
    The classifier is new (different classes), as when fine-tuning on a new dataset."""
    net3d = Net(n, n_classes)
    with torch.no_grad():
        for c2, c3 in zip(net2d.convs, net3d.convs):
            c3.weight.copy_(c2.weight.repeat(1, 1, n, 1, 1) / n)
        net3d.bns.load_state_dict(net2d.bns.state_dict())      # batch norm carries over unchanged
    return net3d

# 1. "ImageNet" stage: train the 2D network on single frames of the appearance task.
torch.manual_seed(0)
gen = torch.Generator().manual_seed(0)
Xf, yf = make_frames(2000, gen)
Xf_te, yf_te = make_frames(600, gen)
net2d = train(Net(1, 3), Xf.unsqueeze(2), yf, steps=300, batch=32)
print(f"2D network, shape accuracy on held-out single frames: {accuracy(net2d, Xf_te.unsqueeze(2), yf_te):.3f}")

# Boring-video check: one test frame repeated 16 times, every layer in evaluation mode.
net3d = inflate_from(net2d, 3)
net3d.fc.load_state_dict(net2d.fc.state_dict())     # same classifier, to compare outputs
frame = Xf_te[:1].unsqueeze(2)                       # (1, 1, 1, S, S)
net2d.eval(); net3d.eval()
with torch.no_grad():
    f2, f3 = net2d.features(frame), net3d.features(frame.repeat(1, 1, 16, 1, 1))
    diff = (f3 - f2).abs().amax(dim=(0, 1, 3, 4))
    p2, p3 = net2d(frame).softmax(1), net3d(frame.repeat(1, 1, 16, 1, 1)).softmax(1)
print("last feature map, max |3D - 2D| per output time step:", " ".join(f"{d:.0e}" for d in diff.tolist()))
print(f"class probabilities, 2D on the frame {[round(v, 3) for v in p2[0].tolist()]},"
      f" inflated 3D on the static clip {[round(v, 3) for v in p3[0].tolist()]}")

# 2. Motion task with few training clips: inflated initialization vs random initialization.
print("\nmotion task (6 classes, chance 0.167), test accuracy over seeds 0-4")
for n_per_class in (4, 16):
    rows = {"inflated": [], "random": []}
    for seed in range(5):
        g = torch.Generator().manual_seed(100 + seed)
        X, y = make_clips(n_per_class, g)
        Xte, yte = make_clips(30, g)
        for init in rows:
            torch.manual_seed(seed)
            net = inflate_from(net2d, 6) if init == "inflated" else Net(3, 6)
            rows[init].append(accuracy(train(net, X, y, steps=150), Xte, yte))
    for init, accs in rows.items():
        print(f"{n_per_class:2d} clips/class, {init:8s}: " + " ".join(f"{a:.3f}" for a in accs)
              + f" | mean {sum(accs) / len(accs):.3f}")

torch 2.14.1
2D network, shape accuracy on held-out single frames: 0.865
last feature map, max |3D - 2D| per output time step: 1e+00 4e-01 4e-06 4e-06 4e-06 4e-06 4e-01 1e+00
class probabilities, 2D on the frame [0.09, 0.45, 0.46], inflated 3D on the static clip [0.085, 0.337, 0.579]

motion task (6 classes, chance 0.167), test accuracy over seeds 0-4
 4 clips/class, inflated: 0.511 0.700 0.444 0.417 0.767 | mean 0.568
 4 clips/class, random  : 0.317 0.361 0.261 0.439 0.278 | mean 0.331
16 clips/class, inflated: 0.639 0.861 0.628 0.750 0.789 | mean 0.733
16 clips/class, random  : 0.600 0.356 0.394 0.706 0.500 | mean 0.511

The static check holds to float32 rounding in the four interior output steps, with batch normalization in inference mode, but the border steps enter the global average and move this ambiguous frame’s top-class probability from 0.46 to 0.579. On the motion task, the inflated start averaged 56.8% with 4 clips per class against 33.1% from random initialization, and 73.3% against 51.1% with 16. It won all five seed pairs at 16 clips but four of five at 4 clips (seed 3: 41.7% against 43.9%). Two runs gave identical output. Freshly inflated filters are symmetric in time, so direction had to be learned from the few clips; the 2D stage supplied, in our reading, the shape features. The toy says nothing about the effect at Kinetics scale.

Performance

Selected results as printed:

• Kinetics test set (arXiv v3, Table 3), top-1 / top-5: RGB-I3D 71.1% / 89.3%, Flow-I3D 63.4% / 84.9%, two-stream I3D 74.2% / 91.3%, all initialized from ImageNet; without ImageNet, 68.4%, 61.5% and 71.6% top-1. The README reports the same numbers for the released checkpoints. The 75.7% often quoted for two-stream I3D is a Kinetics validation figure that appears in later papers (Wang et al., 2018, Table 3), not in I3D’s.
• UCF101 / HMDB51, mean over three splits: in the CVPR version (Table 4), two-stream I3D pretrained on Kinetics reached 97.9% / 80.2%, with 95.4% / 74.5% for RGB alone and 95.4% / 74.6% for flow alone. In arXiv v3 (Table 5), two-stream I3D reached 98.0% / 80.7% with ImageNet and Kinetics pretraining and 97.8% / 80.9% with Kinetics alone; its text quotes 98.0% and 80.9%, the better row for each dataset.

Strengths

• Reuses a proven image architecture and its ImageNet weights.
• High temporal resolution: 64 consecutive frames in training, every frame at test time, which the authors offer as one explanation for why I3D features transferred better than those of models with sparser inputs.

Limitations

• Cost. About 108 GFLOPs per 64-frame RGB clip in Xie et al.’s count, multiplied by whole-video testing.
• The flow stream. TV-L1 flow must be precomputed for every frame, and a second network must be trained and run.
• Limited context. Training clips cover 2.56 s; longer structure comes only from averaging predictions over time.

Historical Importance

I3D (2017) showed that a 3D network could benefit from ImageNet, and, with Kinetics, that pretraining on a large video dataset before fine-tuning gave large gains, as ImageNet had for images (transfer learning). The README states that Kinetics-pretrained I3D models placed first in the CVPR 2017 Charades challenge.

Variants

• I3D on ResNets. Non-local networks (Wang et al., 2018) inflate a ResNet: in one of every two residual blocks either the $3 \times 3$ kernel becomes $3 \times 3 \times 3$ or the first $1 \times 1$ becomes $3 \times 1 \times 1$, and conv1 becomes $5 \times 7 \times 7$, each plane initialized with the 2D weights scaled by $1/t$. With ResNet-101 on Kinetics validation, these reached 74.1% and 74.4% top-1 against 73.1% for the 2D baseline, and 76.0% with non-local blocks added to the $3 \times 1 \times 1$ version (their Tables 2e–f).
• S3D and S3D-G. Xie et al. (2018) replaced I3D’s 3D convolutions by separable spatial and temporal ones; see 3D convolutional networks.
• I3D as a feature extractor. The Fréchet Video Distance (Unterthiner et al., 2018), a video counterpart of the Fréchet Inception Distance, embeds videos with an I3D trained on Kinetics-400, using its logits layer. Temporal action localization methods often work on precomputed I3D features, as about half the entries in ActionFormer’s THUMOS14 comparison do; ActionFormer (Zhang et al., 2022) itself fed 16-frame windows at stride 4 to a Kinetics-pretrained two-stream I3D and concatenated the streams’ 1024-dimensional features (supplementary material).

Implementations

DeepMind’s kinetics-i3d repository (TensorFlow and Sonnet) provides the Inception-v1 I3D model and four checkpoints, RGB and flow, each trained on Kinetics either from scratch or bootstrapped from ImageNet, plus an RGB checkpoint trained from scratch on Kinetics-600 (README as of October 2026, last commit 12 September 2019).