3D Convolutional Networks

A 3D convolutional network is a convolutional neural network whose kernels extend over time as well as over the two image axes. Each filter slides through a clip of frames and responds to patterns in space and time at once, so motion can be learned directly from pixels at every layer, without optical flow. Since C3D (Tran et al., 2015) showed that a deep 3D network trained on a million sports videos gives strong generic video features, the family has grown into factorized, mixed, separable, inflated and efficiency-oriented designs that served as video backbones for action recognition and detection. Temporal modeling compares 3D convolution with frame pooling, recurrence, optical flow and attention, and defines the frame-sampling vocabulary used here.

Core Idea

A 2D convolution applied to a stack of frames treats the frames as input channels, so the output has no time axis left after the first layer (Tran et al., 2015; see temporal modeling). A 3D convolution instead outputs a volume that keeps a time axis, so temporal structure can propagate through every layer. Stacking 3D convolutions and 3D pooling builds a hierarchy of spatio-temporal features in the same way that stacking 2D layers builds edges, textures and parts.

The price is one more kernel dimension: more weights per filter and computation on $T$ frames instead of one. Most designs after C3D try to keep the benefit while paying less, by splitting the kernel into spatial and temporal parts, using 3D convolution only in some layers, borrowing weights from image networks, or making layers thin.

Mathematical Formulation

The operation. The input to a layer is a tensor of $C$ channels, each a $T \times H \times W$ volume (for the first layer, a clip of $T$ RGB frames, $C = 3$). PyTorch’s Conv3d takes a batch as $(N, C, T, H, W)$, the time axis being the one its documentation calls depth $D$, and stores weights as $(C', C, t, h, w)$ (PyTorch 2.14 documentation, as of October 2026). With $C'$ filters, strides $(s_t, s_h, s_w)$ and the input already zero-padded, output channel $o$ at time $\ell$ and position $(i, j)$ is

$$y_o[\ell, i, j] = b_o + \sum_{c=1}^{C} \sum_{a=0}^{t-1} \sum_{u=0}^{h-1} \sum_{v=0}^{w-1} w_{o,c}[a, u, v]\; x_c[s_t \ell + a,\; s_h i + u,\; s_w j + v],$$

a cross-correlation, as in 2D.

Output shape. Each axis follows the 2D rule. With padding $p_t$ in time,

$$T' = \left\lfloor \frac{T + 2p_t - t}{s_t} \right\rfloor + 1,$$

and likewise for $H'$ and $W'$. A $3 \times 3 \times 3$ kernel with padding 1 and stride 1 preserves the clip’s length; temporal stride 2 halves it; no temporal padding shortens it by $t - 1$ frames. A $1 \times h \times w$ kernel is a 2D convolution applied to each frame separately with shared weights.

Parameters and cost. The layer has $C' C\, t h w$ weights plus $C'$ biases, and computes

$$C' C\, t h w \cdot T' H' W'$$

multiply-adds, the unit that X3D (Feichtenhofer, 2020) and CSN (Tran et al., 2019) report as FLOPs. Relative to the 2D layer with the same spatial kernel, both the parameters and the cost per output position grow by the factor $t$, and the cost grows again in proportion to the number of output frames $T'$. Parameters do not depend on $T$; computation does.

Temporal receptive field. Stacking layers widens the span of frames a unit depends on, by the receptive field recursion applied to the time axis:

$$r_L = 1 + \sum_{l=1}^{L} (t_l - 1) \prod_{m=1}^{l-1} s_{t,m}.$$

Eight $3 \times 3 \times 3$ layers at temporal stride 1 see 17 frames; temporal pooling multiplies the growth of every later layer. A unit’s theoretical field can exceed the clip, in which case the excess falls on padding: the network never sees beyond the $T\tau$ source frames it was given.

Computing a 3D Convolution

The NumPy code below implements a 3D convolution, checks it against SciPy on an 8-frame clip of a moving square, prints output shapes, counts parameters and multiply-adds for a full $3 \times 3 \times 3$ layer, a 2D layer and the factorized forms discussed under Variants, computes C3D’s temporal receptive field, and tests I3D’s inflation identity.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view
from scipy.signal import correlate

def conv3d(x, W, stride=(1, 1, 1), pad=(0, 0, 0)):
    """3D convolution (cross-correlation, as in deep learning), no bias.
    x: (C_in, T, H, W) clip; W: (C_out, C_in, t, h, w) kernels, time first."""
    _, _, t, h, w = W.shape
    xp = np.pad(x, ((0, 0), (pad[0],) * 2, (pad[1],) * 2, (pad[2],) * 2))   # zero padding
    win = sliding_window_view(xp, (t, h, w), axis=(1, 2, 3))                 # every t x h x w block
    win = win[:, ::stride[0], ::stride[1], ::stride[2]]                     # keep every s-th position
    return np.einsum("cTHWijk,ocijk->oTHW", win, W)                         # one dot product per filter

def out_size(n, k, s, p):
    return (n + 2 * p - k) // s + 1

# A 3-channel clip (C, T, H, W) = (3, 8, 20, 20): a square moving 2 pixels per frame to the right.
T, S = 8, 20
clip = np.zeros((3, T, S, S))
for t in range(T):
    clip[:, t, 7:13, 2 + 2 * t:8 + 2 * t] = np.array([1.0, 0.6, 0.3])[:, None, None]
rng = np.random.default_rng(0)
W = rng.standard_normal((4, 3, 3, 3, 3))                                    # 4 kernels of 3x3x3 over 3 channels

# Check against SciPy: sum over input channels of 3D correlations, then subsample by the stride.
y = conv3d(clip, W, stride=(2, 1, 1), pad=(1, 1, 1))
xp = np.pad(clip, ((0, 0), (1, 1), (1, 1), (1, 1)))
ref = np.stack([sum(correlate(xp[c], W[o, c], mode="valid") for c in range(3)) for o in range(4)])
print("matches scipy.signal.correlate:", np.allclose(y, ref[:, ::2]))

print("input", clip.shape)
for k, s, p in [((3, 3, 3), (1, 1, 1), (1, 1, 1)), ((3, 3, 3), (2, 1, 1), (1, 1, 1)),
                ((3, 3, 3), (1, 1, 1), (0, 0, 0)), ((1, 3, 3), (1, 1, 1), (0, 1, 1)),
                ((3, 7, 7), (1, 2, 2), (1, 3, 3))]:
    shape = conv3d(clip, np.zeros((4, 3) + k), s, p).shape
    formula = tuple(out_size(n, kk, ss, pp) for n, kk, ss, pp in zip(clip.shape[1:], k, s, p))
    print(f"kernel {k} stride {s} pad {p}: output {shape}, formula {formula}")

# Parameters and multiply-adds of one layer, N_in -> N_out channels, output T x 56 x 56.
def m_i(n_in, n_out, t=3, d=3):
    """R(2+1)D's intermediate width: matches the parameters of a t x d x d 3D convolution."""
    return (t * d * d * n_in * n_out) // (d * d * n_in + t * n_out)

n_in = n_out = 64
M = m_i(n_in, n_out)
p3d = 27 * n_in * n_out
p21 = 9 * n_in * M + 3 * M * n_out
p2d = 9 * n_in * n_out
print(f"\n64 -> 64 channels: 3x3x3 conv {p3d:,} parameters; 1x3x3 (2D) conv {p2d:,};"
      f" (2+1)D with M_i = {M}: {p21:,}; (2+1)D with M_i = 64: {9 * 64 * 64 + 3 * 64 * 64:,}")
for T_out in (8, 16, 32):
    pos = T_out * 56 * 56                                    # output positions (stride 1, same padding)
    print(f"T = {T_out:2d}: multiply-adds 3D {p3d * pos / 1e9:.2f} G, (2+1)D {p21 * pos / 1e9:.2f} G,"
          f" 2D {p2d * pos / 1e9:.2f} G")
print("M_i for R(2+1)D-18 layer widths:",
      {f"{a}->{b}": m_i(a, b) for a, b in [(64, 64), (64, 128), (128, 128), (128, 256),
                                          (256, 256), (256, 512), (512, 512)]})

# Temporal receptive field of C3D's stack: r += (k - 1) * jump; the jump multiplies by the stride.
layers = [("conv1a", 3, 1), ("pool1", 1, 1), ("conv2a", 3, 1), ("pool2", 2, 2),
          ("conv3a", 3, 1), ("conv3b", 3, 1), ("pool3", 2, 2), ("conv4a", 3, 1), ("conv4b", 3, 1),
          ("pool4", 2, 2), ("conv5a", 3, 1), ("conv5b", 3, 1), ("pool5", 2, 2)]
r, j, trace = 1, 1, []
for name, k, s in layers:
    r, j = r + (k - 1) * j, j * s
    trace.append(f"{name} {r}")
print("\nC3D temporal receptive field (frames):", ", ".join(trace))

# Inflation: repeat a 2D filter N = 3 times along time and divide by N.
W2 = rng.standard_normal((4, 3, 1, 3, 3))                    # a 2D filter bank as a 1x3x3 kernel
W3 = np.repeat(W2, 3, axis=2) / 3                            # inflated to 3x3x3
frame = rng.standard_normal((3, 1, S, S))
boring = np.repeat(frame, T, axis=1)                         # a static clip: one frame copied T times
y2 = conv3d(frame, W2, pad=(0, 1, 1))                        # 2D response on the single frame
y3 = conv3d(boring, W3, pad=(1, 1, 1))                       # inflated filter, zero padding in time
y3v = conv3d(boring, W3, pad=(0, 1, 1))                      # no temporal padding
print("\ninflated filter on a static clip vs 2D filter on the frame, max |difference|:")
print(f"  interior frames {np.abs(y3[:, 1:-1] - y2).max():.1e}, all frames without temporal padding"
      f" {np.abs(y3v - y2).max():.1e}")
print(f"  first and last frames with zero temporal padding: response x"
      f" {np.median(y3[:, 0] / y2[:, 0]):.4f}, x {np.median(y3[:, -1] / y2[:, 0]):.4f}")
y_fwd = conv3d(clip, np.repeat(W2, 3, axis=2) / 3, pad=(1, 1, 1))
y_rev = conv3d(clip[:, ::-1], np.repeat(W2, 3, axis=2) / 3, pad=(1, 1, 1))
print(f"  moving clip vs its reversal (output reversed), max |difference|: "
      f"{np.abs(y_rev[:, ::-1] - y_fwd).max():.1e}")

matches scipy.signal.correlate: True
input (3, 8, 20, 20)
kernel (3, 3, 3) stride (1, 1, 1) pad (1, 1, 1): output (4, 8, 20, 20), formula (8, 20, 20)
kernel (3, 3, 3) stride (2, 1, 1) pad (1, 1, 1): output (4, 4, 20, 20), formula (4, 20, 20)
kernel (3, 3, 3) stride (1, 1, 1) pad (0, 0, 0): output (4, 6, 18, 18), formula (6, 18, 18)
kernel (1, 3, 3) stride (1, 1, 1) pad (0, 1, 1): output (4, 8, 20, 20), formula (8, 20, 20)
kernel (3, 7, 7) stride (1, 2, 2) pad (1, 3, 3): output (4, 8, 10, 10), formula (8, 10, 10)

64 -> 64 channels: 3x3x3 conv 110,592 parameters; 1x3x3 (2D) conv 36,864; (2+1)D with M_i = 144: 110,592; (2+1)D with M_i = 64: 49,152
T =  8: multiply-adds 3D 2.77 G, (2+1)D 2.77 G, 2D 0.92 G
T = 16: multiply-adds 3D 5.55 G, (2+1)D 5.55 G, 2D 1.85 G
T = 32: multiply-adds 3D 11.10 G, (2+1)D 11.10 G, 2D 3.70 G
M_i for R(2+1)D-18 layer widths: {'64->64': 144, '64->128': 230, '128->128': 288, '128->256': 460, '256->256': 576, '256->512': 921, '512->512': 1152}

C3D temporal receptive field (frames): conv1a 3, pool1 3, conv2a 5, pool2 6, conv3a 10, conv3b 14, pool3 16, conv4a 24, conv4b 32, pool4 36, conv5a 52, conv5b 68, pool5 76

inflated filter on a static clip vs 2D filter on the frame, max |difference|:
  interior frames 7.1e-15, all frames without temporal padding 7.1e-15
  first and last frames with zero temporal padding: response x 0.6667, x 0.6667
  moving clip vs its reversal (output reversed), max |difference|: 1.3e-15

The shapes match the formula. A $3 \times 3 \times 3$ layer from 64 to 64 channels has three times the weights of its $1 \times 3 \times 3$ counterpart, and its cost grows linearly with the clip, from 2.77 billion multiply-adds at 8 frames to 11.10 billion at 32 on $56 \times 56$ maps. In C3D (16-frame input), units after pool3 already span 16 frames, the whole clip; the deeper fields of 68 and 76 frames mostly cover padding.

How Data Flows

C3D is the plain form of the architecture (Tran et al., 2015). Frames are resized to $128 \times 171$, and training takes random $16 \times 112 \times 112$ crops of clips, so the input is $3 \times 16 \times 112 \times 112$ in the paper’s $c \times l \times h \times w$ notation. Eight convolution layers, all $3 \times 3 \times 3$ with stride 1, carry 64, 128, 256, 256, 512, 512, 512 and 512 filters, interleaved with five max-pooling layers. Pool1 is $1 \times 2 \times 2$, halving only the spatial size so as not to merge temporal information too early; the other pools are $2 \times 2 \times 2$ with stride 2. Two fully connected layers of 4096 units and a softmax follow.

Most later 3D networks follow the ResNet pattern: residual blocks of 3D convolutions with batch normalization, strided stages, and global spatio-temporal average pooling before a linear classifier. R3D-18 (Tran et al., 2018) takes $L$ frames of $112 \times 112$, downsamples only in space at conv1 ($3 \times 7 \times 7$, stride $1 \times 2 \times 2$), then in space and time at the start of each of the last three stages, ending at $\frac{L}{8} \times 7 \times 7$ with 512 channels.

Variants

Early 3D CNNs

Ji et al. (2010) proposed 3D convolution for human action recognition, extended in a journal version (Ji et al., 2013); the details here come from the 2010 conference paper. On TRECVID airport surveillance video, a human detector and tracker located each person, and the network took the person’s bounding box from 7 frames at stride 2, scaled to $60 \times 40$ pixels. A hardwired layer computed gray, gradient and optical-flow channels, and three convolution layers, the first two 3D with kernels such as $3 \times 7 \times 7$ (written $7 \times 7 \times 3$ in the paper), were applied to each channel separately before a fully connected layer. The model was also evaluated on KTH. C3D’s authors contrast this reliance on preprocessing with their own use of full frames.

C3D

Tran et al. (2015) asked which temporal kernel depth works best. Training 5-layer networks from scratch on split 1 of UCF101, with spatial kernels fixed at $3 \times 3$, they compared homogeneous temporal depths of 1, 3, 5 and 7 and two varying schedules (3-3-5-5-7 and 7-5-5-3-3); the networks differed by less than 0.3% of their 17.5 million parameters. Depth 3 gave the best clip accuracy and depth 1, a 2D network applied to each frame, the worst. The authors phrase the conclusion with care: $3 \times 3 \times 3$ was among the best of the limited set of architectures explored.

C3D was trained from scratch on Sports-1M (1.1 million videos, 487 sports classes), on five random 2-second clips per training video. As a feature extractor, it splits a video into 16-frame clips overlapping by 8, averages the clips’ fc6 activations into a 4096-dimensional descriptor, and L2-normalizes it. With a linear SVM on these features, the authors reported 82.3% accuracy on UCF101 (three splits) for one network, 85.2% for three networks concatenated, and 90.4% combined with improved dense trajectories (iDT), against 76.2% for iDT with bag-of-words and 68.8% for averaged ImageNet features under the same linear SVM. On Sports-1M itself, their Table 2 gives 46.1% clip hit@1, 61.1% video hit@1 and 85.2% video hit@5 for the network pretrained on an internal dataset (I380K) and fine-tuned, from 10 clips per video; the text gives 85.5% for that hit@5. Feature extraction ran at 313.9 frames per second on one K40 GPU, about 91 times faster than improved dense trajectories on CPU (their Table 6, a comparison the authors note is not fair to the CPU method).

Mixed convolutions

Tran et al. (2018) studied where 3D convolution is needed within an 18-layer residual network. MC$x$ (mixed convolutions) keeps 3D convolutions in the early groups and uses 2D convolutions from group $x$ onward; rMC$x$ (reversed) uses 2D convolutions in the groups before $x$ and 3D from $x$ on. Their motivating hypothesis for MC was that motion modeling is a low- or mid-level operation. On the Kinetics validation set, trained from scratch on 8-frame clips, MC3 reached 50.7% clip and 62.9% video top-1 with 11.7M parameters, against 49.4% and 61.8% for the full 3D network R3D with 33.4M and 48.1% and 59.4% for the per-frame 2D network f-R2D; the rMC variants reached 49.4–49.9% clip top-1 (their Table 2).

Factorized (2+1)D and pseudo-3D convolutions

P3D (Qiu, Yao & Mei, 2017) simulates a $3 \times 3 \times 3$ convolution with a $1 \times 3 \times 3$ spatial convolution $S$ and a $3 \times 1 \times 1$ temporal convolution $T$ inside a ResNet bottleneck block, between the two $1 \times 1 \times 1$ convolutions. Three blocks differ in how $S$ and $T$ connect, with $\mathbf{x}$ the block input:

• P3D-A, cascaded: $\mathbf{x} + T(S(\mathbf{x}))$;
• P3D-B, parallel: $\mathbf{x} + S(\mathbf{x}) + T(\mathbf{x})$;
• P3D-C, cascaded with a shortcut from $S$: $\mathbf{x} + S(\mathbf{x}) + T(S(\mathbf{x}))$.

The spatial filters can be initialized from a 2D ResNet. Replacing all blocks of ResNet-50 by one type raised UCF101 split-1 accuracy from 80.8% to 83.7%, 82.8% and 83.0% for A, B and C, and alternating A, B and C through the network gave 84.2% (their Table 1). The full P3D ResNet, built on a 152-layer ResNet with ImageNet initialization, reached 66.4% video hit@1 on Sports-1M, against 64.6% for ResNet-152 on single frames and 61.1% for C3D (their Table 2).

R(2+1)D (Tran et al., 2018) replaces each 3D convolution of a plain residual network by a $(2+1)$D block: $M_i$ spatial filters of size $1 \times d \times d$, then batch normalization and a ReLU, then $N_i$ temporal filters of size $t \times 1 \times 1$. The intermediate width is chosen so that the block has about as many parameters as the $N_i$ full filters of size $N_{i-1} \times t \times d \times d$ it replaces:

$$M_i = \left\lfloor \frac{t d^2 N_{i-1} N_i}{d^2 N_{i-1} + t N_i} \right\rfloor, \qquad d^2 N_{i-1} M_i + t M_i N_i \approx t d^2 N_{i-1} N_i .$$

For $t = d = 3$ and 64 channels in and out, $M_i = 144$ and the parameters match exactly (110,592, as the NumPy example prints); with stride 1, the multiply-adds match too. A strided 3D convolution is factorized into the corresponding spatial and temporal strides. The authors give two reasons the factorization helps. It doubles the number of nonlinearities for the same number of parameters, as VGG networks gain by replacing large filters with stacks of small ones, so the network can represent more complex functions. And it appears to make optimization easier: R(2+1)D reached lower training error as well as lower test error than R3D of the same depth, with the gap largest at 34 layers (their Figure 3).

On Kinetics validation (18 layers, from scratch, their Table 2), R(2+1)D reached 52.8% clip and 64.8% video top-1 with 8-frame clips and 56.8% and 68.0% with 16-frame clips, against 49.4% and 61.8%, and 52.5% and 64.2%, for R3D at almost the same parameter count (33.3M against 33.4M). The 34-layer R(2+1)D trained from scratch on RGB reached 72.0% top-1 on Kinetics validation, against 67.5% for I3D trained from scratch on RGB; with Sports-1M pretraining it reached 74.3%, and its two-stream version, with Farnebäck optical flow, 75.4%, slightly below two-stream I3D’s 75.7% (their Table 5).

P3D-A is close to a $(2+1)$D block, but R(2+1)D’s authors stress the differences: P3D uses bottlenecks, mixes three block types and does not match parameters, whereas R(2+1)D uses one block type throughout.

Separable 3D and top-heavy networks

Xie et al. (2018) applied “network surgery” to I3D. Replacing every $k_t \times k \times k$ convolution by a $1 \times k \times k$ convolution followed by a $k_t \times 1 \times 1$ convolution gave S3D (separable 3D). Unlike R(2+1)D, the factorization here is not width-matched, so it removes parameters: with equal widths a $3 \times 3 \times 3$ layer shrinks by a factor of 2.25 (49,152 against 110,592 weights in the example). On Kinetics-Full validation with RGB input, S3D reached 72.2% top-1 with 8.77M parameters and 66.38 GFLOPs, against 71.1%, 12.06M and 107.89 GFLOPs for I3D, the FLOPs counted for 64 frames of $224 \times 224$ (their Table 2). Adding spatio-temporal feature gating, $\mathbf{Y} = \sigma(W \operatorname{pool}(\mathbf{X}) + \mathbf{b}) \odot \mathbf{X}$ after each temporal convolution with pooling over space and time, gave S3D-G at 74.7%.

The same paper varied where 3D convolutions sit. Bottom-heavy networks keep 3D convolutions in the lower layers and 2D in the upper ones; top-heavy networks do the opposite. On Mini-Kinetics-200 and Something-Something, top-heavy models were often clearly more accurate at the same FLOPs, which the authors read as evidence that temporal patterns are most useful among high-level features; they note that this contradicts the mixed-convolution conclusion of Tran et al. (2018), whose differences were small and confounded with cost. A top-heavy separable model, Fast-S3D, came close to I3D on Mini-Kinetics-200 (78.0% against 78.4%) at 43.47 against 107.9 GFLOPs. Similarly, the Slow pathway of SlowFast’s example network uses temporal kernels larger than 1 only in res4 and res5, because earlier temporal convolutions degraded accuracy in the authors’ experiments (Feichtenhofer et al., 2019, Table 1).

Inflation of image networks

Carreira and Zisserman (2017) built I3D by inflating a 2D image network, Inception-v1, into 3D by giving its filters and pooling kernels a time dimension: square $N \times N$ filters become cubic $N \times N \times N$. To start from ImageNet weights, they bootstrap each 3D filter by repeating the 2D filter $N$ times along time and dividing by $N$. On a “boring” video, a single image repeated in time, the inflated filter then gives the same response as the 2D filter on the image, so pointwise nonlinearities, pooling and the whole network reproduce the image network’s output. The NumPy example confirms the identity to rounding (difference $7 \times 10^{-15}$), with one caveat that follows from zero padding: at the first and last frames, a $3$-frame kernel sees one padded frame, and the response is $2/3$ of the 2D one. It also shows that a freshly inflated filter is symmetric in time: it answers a reversed clip with the time-reversed response, so order sensitivity must come from training. The I3D article covers the rest of the model, including its two streams.

Efficient designs

CSN (channel-separated networks; Tran et al., 2019) separates channel interactions from spatio-temporal ones: apart from conv1, every convolution is either a $1 \times 1 \times 1$ convolution, which mixes channels, or a depthwise $k \times k \times k$ convolution, one filter per channel, which does not. In the interaction-preserved form (ip-CSN) a $3 \times 3 \times 3$ convolution in a bottleneck becomes a $1 \times 1 \times 1$ plus a depthwise $3 \times 3 \times 3$; the interaction-reduced form (ir-CSN) drops the extra $1 \times 1 \times 1$. The authors found that channel separation acts as a regularizer, giving higher training error but lower test error than a full 3D ResNet; ip-CSN-152 trained from scratch reached 77.8% video top-1 on Kinetics validation at 108.8 GFLOPs × 30 crops (their Table 5, which counts the 30 views as crops).

X3D (Feichtenhofer, 2020) starts from X2D, a tiny image network that reads one $112 \times 112$ frame, with 20.67M FLOPs and 1.63M parameters, whose bottleneck blocks use channel-wise (depthwise) $3 \times 3 \times 3$ convolutions in the style of MobileNet. It then expands one axis at a time, roughly doubling the cost per step: temporal duration $\gamma_t$, frame rate $\gamma_\tau$, spatial resolution $\gamma_s$, width $\gamma_w$, bottleneck width $\gamma_b$ and depth $\gamma_d$. At each step it trains one model per axis and keeps the expansion with the best accuracy for its cost, a form of coordinate descent; a backward contraction meets a target budget. There is no temporal downsampling anywhere before global pooling. The first step chosen was bottleneck width, the second the number of frames (from one to two), and global width was expanded only at the last step, so X3D models are thin but have high spatio-temporal resolution. Trained from scratch on Kinetics-400, X3D-M reached 76.0% top-1 at 6.2 GFLOPs × 30 views with 3.8M parameters and X3D-XL 79.1% at 48.4 × 30 with 11.0M, against 79.8% at 234 × 30 with 59.9M for SlowFast 16×8 R101+NL (X3D’s Table 4, validation set, 10 clips × 3 crops).

Example: Full 3D Against (2+1)D

The PyTorch code below (run with PyTorch 2.14.1) trains three tiny networks on noisy clips whose class is the direction in which a square moves: right, left, down or up. A “left” clip visits the same positions as a “right” clip in reverse order, so as unordered sets their frames are distributed alike. The networks share one layout and differ only in the convolution: full $3 \times 3 \times 3$ (C3D-style), $(2+1)$D with R(2+1)D’s $M_i$, or $1 \times 3 \times 3$ on each frame with no temporal pooling. Each is trained with five seeds.

import torch
import torch.nn as nn

torch.set_num_threads(4)
print("torch", torch.__version__)
T, S, SIDE = 8, 24, 5                                   # frames, frame size, square side (pixels)
CLASSES = ["right", "left", "down", "up"]

def make_clips(n_per_class, gen):
    """Clips (N, 1, T, S, S) of a bright square moving one pixel per frame, plus noise.
    A "left" clip visits the same positions as a "right" clip, in reverse order."""
    clips, labels = [], []
    for label, name in enumerate(CLASSES):
        for _ in range(n_per_class):
            y0, x0 = torch.randint(2, S - SIDE - T, (2,), generator=gen).tolist()
            clip = 0.3 * torch.randn(1, T, S, S, generator=gen)
            for t in range(T):
                step = t if name in ("right", "down") else T - 1 - t
                y, x = (y0, x0 + step) if name in ("right", "left") else (y0 + step, x0)
                clip[0, t, y:y + SIDE, x:x + SIDE] += 1.0
            clips.append(clip)
            labels.append(label)
    return torch.stack(clips), torch.tensor(labels)

def m_i(n_in, n_out, t=3, d=3):
    return (t * d * d * n_in * n_out) // (d * d * n_in + t * n_out)

def conv(n_in, n_out, kind):
    """One conv -> BN -> ReLU unit: full 3x3x3, (2+1)D with R(2+1)D's M_i, or per-frame 1x3x3."""
    if kind == "3d":
        layers = [nn.Conv3d(n_in, n_out, 3, padding=1, bias=False)]
    elif kind == "2+1d":
        m = m_i(n_in, n_out)
        layers = [nn.Conv3d(n_in, m, (1, 3, 3), padding=(0, 1, 1), bias=False),
                  nn.BatchNorm3d(m), nn.ReLU(),                          # the extra nonlinearity
                  nn.Conv3d(m, n_out, (3, 1, 1), padding=(1, 0, 0), bias=False)]
    else:
        layers = [nn.Conv3d(n_in, n_out, (1, 3, 3), padding=(0, 1, 1), bias=False)]
    return layers + [nn.BatchNorm3d(n_out), nn.ReLU()]

def tiny_net(kind, width=16):
    pool_t = 1 if kind == "2d" else 2                # the per-frame model never mixes frames
    return nn.Sequential(*conv(1, width, kind), nn.MaxPool3d((1, 2, 2)),
                         *conv(width, width, kind), nn.MaxPool3d((pool_t, 2, 2)),
                         *conv(width, width, kind),
                         nn.AdaptiveAvgPool3d(1), nn.Flatten(), nn.Linear(width, len(CLASSES)))

def run(kind, seed, X, y, Xte, yte, epochs=12):
    torch.manual_seed(seed)
    net = tiny_net(kind)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for epoch in range(epochs):
        net.train()
        perm = torch.randperm(len(y))
        total = 0.0
        for i in range(0, len(y), 32):
            idx = perm[i:i + 32]
            loss = nn.functional.cross_entropy(net(X[idx]), y[idx])
            opt.zero_grad(); loss.backward(); opt.step()
            total += loss.item() * len(idx)
    net.eval()
    with torch.no_grad():
        acc = (net(Xte).argmax(1) == yte).float().mean().item()
    return sum(p.numel() for p in net.parameters()), total / len(y), acc

results = {}
for seed in range(5):
    gen = torch.Generator().manual_seed(100 + seed)
    X, y = make_clips(50, gen)                        # 200 training clips
    Xte, yte = make_clips(100, gen)                   # 400 test clips
    for kind in ("2d", "3d", "2+1d"):
        results.setdefault(kind, []).append(run(kind, seed, X, y, Xte, yte))
for kind, rows in results.items():
    losses = " ".join(f"{r[1]:.3f}" for r in rows)
    accs = " ".join(f"{r[2]:.3f}" for r in rows)
    print(f"{kind:5s} {rows[0][0]:5d} params | final-epoch train loss {losses} | test acc {accs}")

torch 2.14.1
2d     4916 params | final-epoch train loss 1.124 1.075 1.105 1.096 1.121 | test acc 0.287 0.385 0.298 0.250 0.300
3d    14420 params | final-epoch train loss 0.138 0.157 0.108 0.138 0.161 | test acc 1.000 1.000 1.000 1.000 1.000
2+1d  14545 params | final-epoch train loss 0.100 0.118 0.105 0.118 0.107 | test acc 1.000 1.000 1.000 1.000 1.000

The per-frame network scores 25.0–38.5% over the five seeds, near the chance level of 25%: left and right clips (and up and down clips) have identically distributed frames as sets, so no per-frame model can exceed 50% here. Both temporal networks classify every test clip correctly. Their convolution weights match within 0.3% (14,223 against 14,256; the intermediate batch normalization adds 158 parameters), and the $(2+1)$D network ends with the lower training loss in all five seeds (0.100–0.118 against 0.108–0.161), the direction Tran et al. (2018) report at scale, though a toy task this easy says nothing about accuracy. Two runs gave identical output.

Training Considerations

Data. 3D networks have more parameters per layer than their 2D counterparts and, trained from scratch, cannot use image pretraining, so they need large video datasets. In I3D’s comparison, a C3D-like network with 79M parameters, trained from scratch, reached only 51.6% on UCF101 split 1 and 24.3% on HMDB51 split 1 with RGB input, while the ImageNet-pretrained models, I3D included, reached 81.0–84.5% and 36.0–49.8% (Carreira & Zisserman, 2017, Tables 1 and 2). Hara, Kataoka and Satoh (2018) trained 3D ResNets of growing depth and found that ResNet-18 overfit on UCF101, HMDB51 and ActivityNet but not on Kinetics, and that accuracy on Kinetics validation rose with depth up to 152 layers and then levelled off (ResNet-200), much as for 2D ResNets on ImageNet. A Kinetics-pretrained ResNeXt-101 on 64-frame clips, fine-tuned, reached 94.5% on UCF101 and 70.2% on HMDB51 (only conv5_x and the classifier fine-tuned; three splits; their Table 5; the abstract gives these numbers without naming the 64-frame variant). Pretraining on Kinetics and fine-tuning on smaller datasets became the standard recipe for 3D networks, the video analogue of ImageNet transfer learning.

Initialization. Inflation lets a 3D network start from ImageNet weights, and P3D initializes its spatial filters from a 2D ResNet; R(2+1)D, CSN and X3D used no image pretraining, only large video datasets.

Clip length. Tran et al. (2018) found that a model trained on 8-frame clips lost 5.8 points of video accuracy when tested on 32-frame clips, whereas fine-tuning it on 32-frame clips nearly matched training on 32 frames from scratch (59.8% against 60.1% clip accuracy) at about a third of the training time (their Table 3). Test-time accuracy is raised by averaging several views per video (see temporal modeling).

Comparison with Alternatives

Two-stream networks hand motion to a second network as precomputed optical flow; 3D networks learn it from frames. Flow still helped 3D backbones (two-stream I3D and R(2+1)D beat their RGB versions), at the cost of computing it: C3D’s authors timed Brox’s GPU flow at 1.2 frames per second against C3D’s 313.9. SlowFast (Feichtenhofer et al., 2019) keeps two pathways but both are 3D networks on raw frames: a slow pathway at a low frame rate and a lightweight fast pathway at a high frame rate with a fraction of the channels (for example 1/8); the SlowFast article covers it.

Video transformers apply self-attention across space-time patches, as the vision transformer does across image patches. Attention relates distant frames within one layer, whereas a 3D convolution’s temporal receptive field grows a few frames per layer; a linear embedding of non-overlapping space-time tubelets is, in our reading, a 3D convolution with bias whose stride equals its kernel size, without padding. Temporal modeling tabulates the trade-offs among these approaches, and video understanding places them in the history of the field.

Limitations

• Cost grows with the clip. Computation is linear in the number of frames, and with $t \times$ the work per position, 3D networks are much heavier than image networks; X3D’s author notes that an image ResNet can use about 27 times fewer multiply-adds than a temporally extended video variant. This limits clip length and resolution and pushes the field toward factorized and depthwise designs.
• Short temporal extent. A clip of 8 to 64 frames covers a few seconds. Longer structure comes from averaging clip predictions or from other models, and the receptive field cannot exceed the sampled span.
• Appearance can dominate. On Kinetics, I3D scored 71.1% top-1 whether the test frames were in normal or reversed order, while on Something-Something it fell from 45.8% to 15.2% (Xie et al., 2018, Table 1). A 3D network does not have to use motion when the benchmark does not demand it.

Important Models

Model	Year	Key idea	Reported result
Ji et al. 3D CNN	2010	3D convolution over hardwired gray, gradient and flow channels of a tracked person	Evaluated on TRECVID 2008 and KTH
C3D	2015	Homogeneous $3 \times 3 \times 3$ kernels, trained on Sports-1M; generic features	UCF101 82.3% (1 net + linear SVM), 85.2% (3 nets), 90.4% with iDT
I3D	2017	Inflated Inception-v1 bootstrapped from ImageNet; RGB and flow streams	Covered in the I3D article
P3D ResNet	2017	Mixed pseudo-3D bottleneck blocks (A, B, C), 199 layers	Sports-1M video hit@1 66.4%
R(2+1)D	2018	Parameter-matched spatial + temporal factorization; MC and rMC study	Kinetics val top-1 72.0% (RGB, from scratch); 75.4% (two-stream, Sports-1M pretrained)
S3D / S3D-G	2018	Separable 3D in I3D; top-heavy placement; feature gating	Kinetics-Full val top-1 72.2% (S3D), 74.7% (S3D-G), RGB
CSN	2019	Channel-separated: $1 \times 1 \times 1$ plus depthwise $3 \times 3 \times 3$	Kinetics val top-1 77.8% (ip-CSN-152, from scratch, 108.8 GFLOPs × 30 crops)
X3D	2020	Stepwise expansion of a tiny 2D network along six axes	Kinetics-400 val top-1 79.1% (X3D-XL, 48.4 GFLOPs × 30 views, 11.0M parameters)

All results are as reported by the authors, on different datasets, splits, inputs and test-time views, so they are not directly comparable.