Depthwise Separable Convolution

A depthwise separable convolution replaces one standard convolutional layer with two cheaper ones: a depthwise convolution that filters each input channel separately with its own small spatial kernel, and a pointwise $1 \times 1$ convolution that forms new channels as weighted sums of the filtered ones. A standard convolutional layer does both jobs, spatial filtering and channel mixing, in one step with one large kernel. Splitting them cuts the parameters and multiply-adds by a factor close to $k^2$ for a $k \times k$ kernel, about 8 to 9 times for $3 \times 3$. That saving made the factorization the basic layer of Xception, the MobileNets and EfficientNet, and of the atrous separable convolutions in DeepLabv3+.

Definition

Take an input with $C_\text{in}$ channels, in PyTorch’s $(N, C, H, W)$ layout, and a desired output with $C_\text{out}$ channels.

• A standard convolution has $C_\text{out}$ filters, each of shape $C_\text{in} \times k \times k$; every output feature map reads every input channel.
• A depthwise convolution has one $k \times k$ filter per input channel. Filter $c$ is applied to channel $c$ only, so the output has $C_\text{in}$ maps and no channel sees another.
• A pointwise convolution is a $1 \times 1$ convolution: at each position it multiplies the vector of $C_\text{in}$ channel values by a $C_\text{out} \times C_\text{in}$ matrix. It mixes channels and does no spatial filtering.

A depthwise separable convolution is a depthwise convolution followed by a pointwise one. Howard et al. (2017) describe it as splitting a standard convolution into a separate layer for filtering and a separate layer for combining. A variant has a depth multiplier $K$: the depthwise stage applies $K$ filters to each input channel, giving $K C_\text{in}$ intermediate maps (Sifre, 2014).

Intuition

Consider the first layer of a network looking at a colour image. A standard $3 \times 3$ layer with 16 outputs learns 16 kernels, each a $3 \times 3 \times 3$ block that can look for, say, a red-to-green edge at one orientation. If several of them apply similar spatial patterns, edges or blobs, and differ mainly in how they weight red, green and blue, much of the layer is redundant; Sifre (2014) observed such redundancy in trained networks (see History).

The separable version makes that structure explicit. First it asks one spatial question per channel: where are vertical edges in red, horizontal edges in green, blobs in blue? Then, at each pixel, it builds each of the 16 outputs as a different mix of those three answers. The spatial work is done once per input channel instead of once per input-output pair, which is where the saving comes from. The price is that every output must reuse the same spatial filter on a given input channel, scaled differently. A layer that needs a vertical-edge detector on red for one output and a horizontal-edge detector on red for another cannot be expressed with one depthwise filter per channel.

Formal Definition

For one image, write $x_c$ for input channel $c$ and $\star$ for the cross-correlation computed by deep-learning libraries. Leaving out biases, a standard convolution computes

$$y_o = \sum_{c=1}^{C_\text{in}} W_{o,c} \star x_c, \qquad o = 1, \dots, C_\text{out},$$

with $C_\text{out} C_\text{in}$ spatial kernels $W_{o,c}$ of size $k \times k$. The depthwise stage computes $z_c = D_c \star x_c$ with one kernel $D_c$ per channel, and the pointwise stage computes $y_o = \sum_c P_{o,c}\, z_c$ with a matrix $P$ of size $C_\text{out} \times C_\text{in}$. Because convolution is linear,

$$y_o = \sum_{c=1}^{C_\text{in}} P_{o,c}\,(D_c \star x_c) = \sum_{c=1}^{C_\text{in}} \left(P_{o,c} D_c\right) \star x_c .$$

The separable block is therefore exactly a standard convolution whose kernel factorizes as $W_{o,c} = P_{o,c} D_c$. Stack, for one input channel $c$, the $C_\text{out}$ flattened kernels $W_{1,c}, \dots, W_{C_\text{out},c}$ as the rows of a $C_\text{out} \times k^2$ matrix: for a separable layer that matrix is the outer product $\mathbf{p}_c \mathbf{d}_c^\top$ of the column $\mathbf{p}_c$ of $P$ and the flattened filter $\mathbf{d}_c$, a matrix of rank at most 1. With depth multiplier $K$ the rank is at most $K$. This is the precise sense in which the factorization restricts the kernel.

Cost. With stride 1 and padding chosen so that the output has $H \times W$ positions, the counts are

	Weights	Multiply-adds
Standard	$k^2 C_\text{in} C_\text{out}$	$k^2 C_\text{in} C_\text{out}\, H W$
Depthwise	$k^2 C_\text{in}$	$k^2 C_\text{in}\, H W$
Pointwise	$C_\text{in} C_\text{out}$	$C_\text{in} C_\text{out}\, H W$
Depthwise separable	$C_\text{in}(k^2 + C_\text{out})$	$C_\text{in}(k^2 + C_\text{out})\, H W$

plus $C_\text{out}$ biases for the standard layer and $C_\text{in} + C_\text{out}$ for the separable pair if both stages have them. The ratio of separable to standard cost, the same for weights and multiply-adds, is

$$\frac{k^2 C_\text{in} + C_\text{in} C_\text{out}}{k^2 C_\text{in} C_\text{out}} = \frac{1}{C_\text{out}} + \frac{1}{k^2} .$$

Howard et al. (2017, §3.1) derive this in their notation, with $M$ input channels, $N$ output channels, kernel size $D_K$ and a square $D_F \times D_F$ feature map, and print the result as $1/N + 1/D_K^2$. Since layers in the middle of a network have hundreds of channels, the first term is small and the saving is close to $k^2$: they state that their $3 \times 3$ separable layers use 8 to 9 times less computation than standard ones. With a depth multiplier $K$, Sifre (2014) gives the weight count as $K C_\text{in}(k^2 + C_\text{out})$, in his notation $K_m D_m (Q_m + D_{m+1})$.

Grouped convolution is the general case. A convolution with $g$ groups splits the input channels into $g$ equal groups and the output channels into $g$ groups, and convolves each input group only with its own output group. It has $k^2 C_\text{in} C_\text{out} / g$ weights. PyTorch’s torch.nn.Conv2d exposes this as the groups argument: the weight tensor has shape $(C_\text{out}, C_\text{in}/g, k, k)$; groups=1 is a standard convolution, groups=2 is equivalent to two half-width layers side by side whose outputs are concatenated, and with groups=in_channels and out_channels a multiple $K$ of in_channels the documentation calls the operation a depthwise convolution. A depthwise separable layer in PyTorch is therefore Conv2d(C_in, K*C_in, k, groups=C_in) followed by Conv2d(K*C_in, C_out, 1).

Properties

Most of the cost moves into the pointwise layer. Once the spatial filtering is cheap, the $1 \times 1$ convolution dominates. In MobileNet, Howard et al. report that $1 \times 1$ convolutions account for 94.86% of the multiply-adds and 74.59% of the parameters, against 3.06% and 1.06% for the $3 \times 3$ depthwise layers (their Table 2); most of the remaining parameters are in the final fully connected layer. Zhang et al. (2018) make the same point about grouped designs: in a ResNeXt unit with 32 groups, the pointwise convolutions take 93.4% of the multiply-adds, which led them to group the $1 \times 1$ layers too in ShuffleNet. A corollary that Howard et al. draw: factorizing the depthwise $3 \times 3$ further, into $1 \times 3$ and $3 \times 1$, saves little, since so little computation is spent there.

Fewer multiply-adds do not mean proportionally less time. Howard et al. stress that a low multiply-add count is not enough and that the operations must be efficient to implement; they note that $1 \times 1$ convolutions map directly onto highly optimized matrix multiplication (GEMM) without the im2col reordering that GEMM-based $k \times k$ convolutions need. Depthwise convolution is the weak point. Zhang et al. (2018) found it hard to implement efficiently on low-power mobile devices despite its low theoretical complexity and suggest that this may come from a worse ratio of computation to memory access than dense operations; they therefore used depthwise layers only on narrow bottleneck maps. Chollet (2017) reported that Xception, with about the same number of parameters as Inception V3, trained slightly slower, at 28 against 31 steps per second on 60 GPUs (his Table 3), and expected optimized depthwise kernels to make Xception the faster of the two. A rough count (our estimate, in the example below) makes the ratio concrete: for MobileNets’ $14 \times 14 \times 512$ layer, a depthwise $3 \times 3$ convolution performs about 4 multiply-adds per tensor element it must read or write, against about 111 for the pointwise layer and 181 for a standard $3 \times 3$ convolution, which suggests that the depthwise stage is limited by memory bandwidth rather than arithmetic.

Memory for activations. The intermediate maps between the two stages must be stored for the backward pass. Ioffe and Szegedy (2015, supplementary material of the ICML paper, also the appendix of arXiv v3) used a separable first layer with depth multiplier 8 in their batch-normalized Inception and note that it reduced computation while increasing memory consumption at training time. With $K = 8$ on a three-channel image the pair stores an extra 24-channel intermediate tensor (our reading of why; the paper does not give the reason).

Nonlinearity between the stages is a design choice. MobileNet puts batch normalization and ReLU after both the depthwise and the pointwise layer. Chollet (2017, §4.7) tried ReLU and ELU between the two stages of Xception’s separable layers and found that no intermediate nonlinearity gave both faster convergence and better final accuracy on ImageNet, the opposite of what Szegedy et al. (2016) had reported for their factorized Inception modules, where linear activations in the first factor were always worse. Chollet’s suggested explanation, which he offers as a possibility, is that a nonlinearity helps on deep intermediate feature spaces but loses information on the one-channel spaces of a depthwise convolution. ShuffleNet follows Xception and has no ReLU after its depthwise layer, while DeepLabv3+ adds batch normalization and ReLU after each depthwise convolution as MobileNet does.

A spectrum from standard to depthwise. Chollet (2017) reads the number of groups as a dial: one segment is a regular convolution, one segment per channel is a depthwise convolution, and Inception modules, which split a few hundred channels into 3 or 4 branches, lie in between. Xception’s name, “Extreme Inception”, comes from taking that dial to the end, under the hypothesis that cross-channel and spatial correlations can be mapped entirely separately. He notes two differences from an extreme Inception module, the order of the $1 \times 1$ and spatial steps and the nonlinearity between them, and argues that the order hardly matters once layers are stacked.

Larger kernels become cheap. In a separable layer the kernel size enters only the depthwise term, $k^2 C_\text{in}$, not the much larger $C_\text{in} C_\text{out}$ term, so a wider receptive field costs little. Liu et al. (2022) used this in ConvNeXt: raising the depthwise kernel from $3 \times 3$ to $7 \times 7$ increased ImageNet-1K top-1 accuracy from 79.9% to 80.6% while the FLOPs stayed roughly the same, with no further gain beyond $7 \times 7$.

Regularization of tiny filters. Howard et al. found it important to put very little or no weight decay on the depthwise filters, since they have so few parameters.

History

Laurent Sifre developed the factorization during an internship at Google Brain in 2013, according to Chollet (2017), who adds that an overview was presented at ICLR 2014. Sifre’s PhD thesis (2014, §6.2) starts from an observation: in the first two layers of an AlexNet-like network, the learned filters belonging to one output are highly redundant across input channels. He presents the factorization as inspired by the separable convolutions on the rigid-motion group from his scattering-transform work with Stéphane Mallat: spatial convolutions with $K$ filters per input channel, then a pointwise matrix multiplication across the $K C_\text{in}$ intermediate channels. He replaced the dense first two layers of the network of Zeiler and Fergus, an AlexNet-style ImageNet classifier, with separable ones in Google’s DistBelief system; for the first layer ($7 \times 7$, 3 to 96 channels) the weights fell from 14,112 to 1,740 with $K = 4$ (his Table 6.1). On ImageNet ILSVRC2012 the separable version needed 20% fewer steps to converge to an identical or slightly better accuracy, with less time per step. Chollet summarizes this work as having been done in AlexNet; the thesis names the Zeiler–Fergus network.

The Inception line used depthwise separable layers only at the input, in the batch-normalized Inception of Ioffe and Szegedy (2015), and otherwise factorized convolutions spatially, an $n \times n$ kernel into $1 \times n$ and $n \times 1$; Inception covers both, and Chollet’s claim that Inception V1 also had a separable first layer. Spatial separability is a different factorization from the depthwise kind.

Chollet credits the operation’s wider adoption to its inclusion in TensorFlow in 2016. Two papers published in 2017 built whole networks from it: Xception (Chollet, 2017), which replaced Inception modules with depthwise separable layers, and MobileNets (Howard et al., 2017), which used them for small, low-latency models. Both cite Sifre’s thesis as the origin.

Grouped convolution is older: AlexNet’s two-GPU layout split three of its layers into two groups (AlexNet), and ResNeXt made the number of groups, its cardinality, a design dimension (ResNet). With as many groups as channels, a grouped convolution is the depthwise stage of a separable one.

Examples

The PyTorch code below (output from PyTorch 2.14.1, double precision, one thread) uses a $3 \times 32 \times 32$ synthetic colour image whose red channel holds both vertical and horizontal lines. It measures costs with PyTorch’s FLOP counter, checks the factorization $W_{o,c} = P_{o,c} D_c$, fits separable blocks to a target layer that applies two different line detectors to the same red channel, and estimates arithmetic intensity.

import torch
import torch.nn as nn
from torch.utils.flop_counter import FlopCounterMode

torch.manual_seed(0)
torch.set_num_threads(1)
torch.set_default_dtype(torch.float64)

# A 3x32x32 colour image, layout (N, C, H, W). Red: vertical lines in the left
# half, horizontal lines in the right half. Green: horizontal lines. Blue: a square.
x = torch.zeros(1, 3, 32, 32)
x[0, 0, :, 0:16:4] = 1.0         # red vertical lines, columns 0, 4, 8, 12
x[0, 0, 2::4, 16:] = 1.0         # red horizontal lines, rows 2, 6, ..., 30
x[0, 1, ::4, :] = 1.0            # green horizontal lines every 4 pixels
x[0, 2, 8:24, 8:24] = 1.0        # blue: a 16x16 square

def separable(c_in, c_out, k, mult=1):
    return nn.Sequential(
        nn.Conv2d(c_in, mult * c_in, k, padding=k // 2, groups=c_in, bias=False),  # depthwise
        nn.Conv2d(mult * c_in, c_out, 1, bias=False))                             # pointwise

def n_params(m):
    return sum(p.numel() for p in m.parameters())

def mult_adds(m, inp):
    with FlopCounterMode(display=False) as fc:
        m(inp)
    return fc.get_total_flops() // 2      # PyTorch counts one multiply-add as 2 FLOPs

# 1. Cost: the colour layer, and MobileNets' Table 3 layer (14x14x512 -> 512, k=3)
for c_in, c_out, k, size in [(3, 16, 3, 32), (512, 512, 3, 14)]:
    inp = torch.zeros(1, c_in, size, size)
    std = nn.Conv2d(c_in, c_out, k, padding=k // 2, bias=False)
    sep = separable(c_in, c_out, k)
    ps, pd = n_params(std), n_params(sep)
    ms, md = mult_adds(std, inp), mult_adds(sep, inp)
    print(f"C_in={c_in} C_out={c_out} k={k} {size}x{size}:")
    print(f"  standard : {ps:>9,} weights {ms:>13,} mult-adds")
    print(f"  separable: {pd:>9,} weights {md:>13,} mult-adds")
    print(f"  ratio {md / ms:.4f}   1/C_out + 1/k^2 = {1 / c_out + 1 / k**2:.4f}")

# 2. A separable block is a standard conv whose kernel factorizes:
#    W[o, c, :, :] = P[o, c] * D[c, :, :]
sep = separable(3, 16, 3)
D = sep[0].weight[:, 0]                      # (3, 3, 3): one 3x3 filter per input channel
P = sep[1].weight[:, :, 0, 0]                # (16, 3): channel-mixing matrix
std = nn.Conv2d(3, 16, 3, padding=1, bias=False)
with torch.no_grad():
    std.weight.copy_(P[:, :, None, None] * D[None])
    diff = (std(x) - sep(x)).abs().max().item()
print(f"rank-1 standard conv vs separable block: max difference {diff:.1e}")

# 3. The converse fails: for each input channel, the C_out spatial filters of a
#    separable layer are multiples of one filter (rank 1). A layer that needs two
#    different filters on the red channel has rank 2 there.
vert = torch.tensor([[-1., 2., -1.]] * 3)    # vertical-line detector
horiz = vert.T                               # horizontal-line detector
target = nn.Conv2d(3, 2, 3, padding=1, bias=False)
with torch.no_grad():
    target.weight.zero_()
    target.weight[0, 0] = vert               # output 0: vertical lines in red
    target.weight[1, 0] = horiz              # output 1: horizontal lines in red
    y = target(x)
sv = torch.linalg.svdvals(target.weight[:, 0].reshape(2, 9))
print(f"red-channel filters, singular values: {sv[0]:.3f} {sv[1]:.3f}")
print(f"target response: output 0 max {y[0, 0].max():.0f}, output 1 max {y[0, 1].max():.0f}")
print(f"target mean square {(y ** 2).mean():.3f}")

# Fit separable blocks with depth multiplier 1 and 2 to the target on this image
for mult in (1, 2):
    losses = []
    for seed in range(5):
        torch.manual_seed(seed)
        model = separable(3, 2, 3, mult)
        opt = torch.optim.Adam(model.parameters(), lr=0.05)
        for step in range(1500):
            opt.zero_grad()
            loss = ((model(x) - y) ** 2).mean()
            loss.backward()
            opt.step()
        losses.append(loss.item())
    print(f"depth multiplier {mult}: {n_params(model)} weights, "
          f"final MSE over seeds 0-4: {min(losses):.2e} to {max(losses):.2e}")

# 4. Arithmetic intensity (our estimate): multiply-adds per tensor element that
#    must be read or written at least once (input, output, weights), MobileNets layer
c, k, n = 512, 3, 14 * 14              # channels, kernel, positions
dw = k * k * c * n, 2 * c * n + k * k * c
pw = c * c * n, 2 * c * n + c * c
full = k * k * c * c * n, 2 * c * n + k * k * c * c
for name, (macs, elems) in [("depthwise", dw), ("pointwise", pw), ("standard", full)]:
    print(f"{name:9s}: {macs / elems:6.1f} mult-adds per element moved")

Output:

C_in=3 C_out=16 k=3 32x32:
  standard :       432 weights       442,368 mult-adds
  separable:        75 weights        76,800 mult-adds
  ratio 0.1736   1/C_out + 1/k^2 = 0.1736
C_in=512 C_out=512 k=3 14x14:
  standard : 2,359,296 weights   462,422,016 mult-adds
  separable:   266,752 weights    52,283,392 mult-adds
  ratio 0.1131   1/C_out + 1/k^2 = 0.1131
rank-1 standard conv vs separable block: max difference 1.1e-16
red-channel filters, singular values: 4.243 4.243
target response: output 0 max 6, output 1 max 6
target mean square 6.332
depth multiplier 1: 33 weights, final MSE over seeds 0-4: 2.54e+00 to 2.54e+00
depth multiplier 2: 66 weights, final MSE over seeds 0-4: 8.51e-09 to 2.81e-07
depthwise:    4.4 mult-adds per element moved
pointwise:  111.0 mult-adds per element moved
standard :  180.6 mult-adds per element moved

On the colour layer the saving is only about 5.8 times, because with 16 outputs the $1/C_\text{out}$ term is not small; the 512-channel counts are those of MobileNets’ Table 3 (462 million against 52.3 million multiply-adds, 2.36 million against 0.27 million weights). The small saving at three input channels may be one reason MobileNet keeps its first layer as a standard convolution (our reading; the paper gives no reason).

The target layer’s two red-channel filters have two equal nonzero singular values, so no rank-1 kernel can reproduce it: with one depthwise filter per channel, about 40% of the target’s energy (2.54 of 6.33) stays out of reach for every seed, while depth multiplier 2 can hold both filters. In a network, more channels or more layers make up for this restriction.

The intensity figures assume each input, output and weight is moved once, a lower bound on data movement rather than a hardware measurement; the depthwise layer’s figure is about $k^2/2$.

Common Misconceptions

• “A separable layer approximates any standard layer.” It equals a standard layer exactly only when, for each input channel, all output channels use multiples of one spatial filter. Otherwise it is a different, smaller model class: a separable network is its own architecture, not a compressed copy of a standard one.
• “It is the same as a separable filter.” A separable filter in image processing factors a 2D kernel into a row filter and a column filter, as for the Gaussian (see convolution); Chollet (2017) calls this spatially separable and warns against confusing the two. A depthwise separable convolution factors over space and channels, and each depthwise filter is a full $k \times k$ kernel.
• “The depthwise convolution is the whole layer.” On its own it never combines channels, so the output channels cannot represent anything that depends on two input channels together. Howard et al. note that it only filters input channels and that the $1 \times 1$ layer is needed to create new features.

Where It Is Used

• MobileNet. Howard et al. (2017) built MobileNet from $3 \times 3$ depthwise separable layers everywhere except a standard first layer, with downsampling by stride-2 depthwise convolutions. Against the same architecture with standard convolutions, it reached 70.6% against 71.7% ImageNet accuracy with 569 million against 4,866 million multiply-adds and 4.2 million against 29.3 million parameters (their Table 4; the paper does not specify the evaluation protocol). MobileNetV2 (Sandler et al., 2018) keeps the depthwise layer inside an inverted residual block with a $1 \times 1$ expansion before it and a linear $1 \times 1$ projection after it. MobileNet covers the family.
• Xception. Chollet (2017) built a 36-layer network from depthwise separable layers with residual connections. With 22.9 million parameters against Inception V3’s 23.6 million, it reached 79.0% against 78.2% top-1 and 94.5% against 94.1% top-5 accuracy on the ImageNet validation set (single crop, single model; Inception V3 without its auxiliary tower, his Table 1), and 6.70 against 6.36 MAP@100 on Google’s internal JFT dataset (his Table 2). Inception covers the family it comes from.
• EfficientNet. The main block of EfficientNet-B0 is MobileNetV2’s mobile inverted bottleneck (MBConv) with squeeze-and-excitation added, using $3 \times 3$ and $5 \times 5$ depthwise kernels (Tan & Le, 2019, Table 1).
• Segmentation. DeepLabv3+ uses atrous separable convolution, a depthwise separable convolution whose depthwise stage is dilated, so that the receptive field grows without extra weights. Chen et al. (2018) use it throughout their Xception backbone; in the ASPP and decoder modules it cut multiply-adds by 33% to 41% at similar mIOU on the PASCAL VOC 2012 validation set (their Table 5).
• Modern convolutional networks. ConvNeXt (Liu et al., 2022) uses a $7 \times 7$ depthwise convolution in every block, which the authors relate to vision transformers: depthwise and $1 \times 1$ layers separate spatial mixing from channel mixing, and in a vision transformer, too, each operation mixes information across either space or channels but not both.
• Video. Channel-separated networks factor 3D convolutions into $1 \times 1 \times 1$ and depthwise $k \times k \times k$ layers, and X3D uses depthwise $3 \times 3 \times 3$ convolutions (see 3D convolutional networks).