Dilated Convolution

A dilated convolution, also called an atrous convolution, is a convolution whose filter taps are spaced $r$ pixels apart instead of being adjacent. A $3 \times 3$ filter with rate $r = 2$ reads a $5 \times 5$ window, skipping every other pixel, with the same nine weights. In a convolutional neural network this widens the receptive field of later layers without adding weights and without downsampling, which is why dense prediction networks use it to keep feature maps at high resolution. Its cost is that stacked dilated layers can see the input on a sparse lattice, a defect called gridding.

This article covers the operator, its growth of the receptive field, its history in wavelet analysis, gridding and the degridding of Dilated Residual Networks, and its uses outside segmentation. The way DeepLab converts a classification backbone to output stride 8 or 16, and its atrous spatial pyramid pooling (ASPP), are covered in that article.

Definition

A dilated convolution with rate (or dilation factor) $r$ applies a filter of $k$ taps to input samples that are $r$ positions apart. For $r = 1$ it is an ordinary convolution. It changes neither the number of weights nor the number of multiplications per output; it changes which inputs those weights see.

Dilation is defined for any dimension: 1-D for audio and frame sequences, 2-D for images, 3-D for volumes and video. In deep-learning frameworks it is a parameter of the ordinary convolution layer (dilation in PyTorch’s nn.Conv2d), usually used with stride 1.

The word is unrelated to dilation in mathematical morphology, which grows bright regions by taking a local maximum.

Intuition

Think of the filter as a stencil of nine holes over the image. An ordinary $3 \times 3$ stencil has its holes touching. Spread the holes so that there are $r - 1$ pixels between neighbours and the stencil covers a wider area with the same nine holes, ignoring the pixels in between.

Two ways to grow a receptive field are available without dilation. More layers grow it linearly. Downsampling, by strided convolution or pooling, grows it faster but coarsens the output grid, and a segmentation network then has to recover the lost resolution. Dilation spreads later filters over the same full-resolution grid instead. Removing a stride-2 step and dilating every later filter by 2 leaves the receptive fields where they were and doubles the output resolution, the conversion that Dilated Residual Networks (DRN) and DeepLab apply to ResNet.

The pixels between the taps are the price. One dilated layer that follows ordinary layers leaves no holes in the field, because the earlier layers have already mixed each pixel with its neighbours. A stack of dilated layers with the same rate never fills the holes, and its output depends on only some of the input pixels.

Formal Definition

For a 1-D input $x$ and a filter $w$ with taps $k = -K, \dots, K$, the dilated convolution with rate $r$ is

$$y[i] = \sum_{k=-K}^{K} x[i + r k]\, w[k],$$

the form DeepLabv3+ writes as its Equation 1 (Chen et al., 2018), where the rate “determines the stride with which we sample the input signal”. As usual in deep learning the filter is not mirrored, so this is a cross-correlation. In 2-D the same offset $r$ applies along both axes. Equivalently, $y$ is an ordinary convolution with the upsampled filter $w_r$ obtained by inserting $r - 1$ zeros between taps:

$$w_r[m] = \begin{cases} w[m / r] & \text{if } r \text{ divides } m, \\ 0 & \text{otherwise.} \end{cases}$$

A filter of size $k$ then spans $k + (k - 1)(r - 1)$ samples. With stride 1 and “same” padding of $r(k - 1)/2$ on each side, the output has the size of the input.

Receptive field of a stack. With stride 1 everywhere, $L$ layers of size-$k$ filters with rates $r_1, \dots, r_L$ have receptive field

$$s = 1 + (k - 1) \sum_{l=1}^{L} r_l .$$

This is the recurrence in the Receptive Field article with all jumps equal to 1. Doubling rates $1, 2, 4, \dots, 2^{L-1}$ with $k = 3$ give $s = 2^{L+1} - 1$: the field grows exponentially with depth while the weights grow linearly. MS-TCN states this doubling case for its kernel-size-3 temporal layers (Abu Farha and Gall, 2019, Eq. 3).

Composite weights. A stack of linear dilated layers is one convolution whose kernel is the convolution of the upsampled filters. If every filter is all ones, the composite kernel counts, for each input offset, the number of tap paths that reach it. That count shows which pixels the stack can see and how strongly. For $3 \times 3$ all-ones filters, each layer adds an independent offset of $-r_l$, 0 or $+r_l$ along each axis with equal weight, so the spread of the normalized composite weights along one axis is, by our derivation,

$$\sigma = \sqrt{\tfrac{2}{3} \textstyle\sum_l r_l^2 } .$$

Polyphase view. If every rate in a stack is a multiple of $s$, an output at $(i, j)$ depends only on inputs at $(i + s a, j + s b)$. The stack, nonlinearities included, then splits into $s^2$ independent copies, each with its rates divided by $s$, running on the $s \times s$ subsampled images, with their outputs interleaved. Long, Shelhamer and Darrell (2015) describe the converse: a network that downsamples by $f$ can be made dense either by running it on $f^2$ shifted inputs and interlacing the outputs (shift-and-stitch) or, equivalently, by removing the subsampling and enlarging the later filters with zeros, which they call filter rarefaction. They present this as a well-known trick for producing the same results efficiently, known to the wavelet community as the à trous algorithm.

Frequency view. In our reading, the same fact explains gridding in the frequency domain. The upsampled filter’s frequency response is $W_r(\omega) = W(r\omega)$, periodic with period $2\pi/r$. A rate-2 filter therefore responds to a pattern at frequency $\omega + \pi$ with the same gain as to one at $\omega$: content that alternates from pixel to pixel passes it as strongly as smooth content. DRN puts the condition in words: gridding artefacts occur “when a feature map has higher-frequency content than the sampling rate of the dilated convolution” (Yu, Koltun and Funkhouser, 2017, Sec. 4).

Origins and Names

The operator comes from signal processing. The algorithme à trous (“algorithm with holes”) computes an undecimated wavelet transform with filters upsampled by inserting zeros, the holes that give the name, as the DeepLab article describes; DeepLabv3+ cites Holschneider et al. (1989), among other works, for atrous convolution. FCN cites the wavelet literature for the name and earlier networks for the dense-evaluation trick (Long et al., 2015). DeepLab’s papers adopted the name atrous convolution, and the DeepLab article traces how its versions used it. Yu and Koltun (2016), whose paper could not be read here, named the operator dilated convolution in the title of their paper on multi-scale context aggregation; DRN cites that paper for dilated convolutions, and the Receptive Field article gives the receptive-field sizes of their doubling-rate stack. Both names are in use: DeepLab says “atrous”, DRN and PyTorch’s layer argument say “dilated”, and PWC-Net speaks of “dilation constants”.

Properties

Resolution and weights. Dilation keeps the output grid at the input’s resolution and the weights at $k^2$ per input–output channel pair, for any rate. A trained classification network can therefore be converted without new weights: DRN’s converted networks have “the same number of layers and parameters” as the ResNets they come from, and on a $224 \times 224$ input give a $28 \times 28$ final map instead of $7 \times 7$.

Compute and memory. Every layer after a removed stride runs on a map with four times as many positions. DRN notes that memory grows by the same factor. DeepLabv3+ counts the layers affected in ResNet-101: the last 3 residual blocks (9 layers) for output stride 16, and 26 blocks (78 layers) for output stride 8, which is why it pairs output stride 16 with a decoder (Chen et al., 2018). The DeepLab article measures the multiply-accumulates of ResNet-50 at output strides 32, 16 and 8.

Gridding. A stack whose rates share a common factor $g > 1$ reaches only inputs on a lattice of spacing $g$, whatever the weights, since every path moves in multiples of $g$. DRN shows the didactic case of a single active pixel: a 2-dilated convolution turns it into a grid pattern (Yu et al., 2017, Fig. 4). In trained networks it shows up in activation maps and segmentation outputs (their Figs. 3 and 8). DRN cites Wang et al. (2018), whose paper could not be read here, as concurrent work that observed the artefacts; the DeepLab article credits them with the name gridding. That article’s example measures the share of the field that stacks such as rates (2, 2, 2) and (2, 4, 8) reach, and it describes Wang et al.’s hybrid dilated convolution, which uses rates without a common factor such as 1, 2, 5.

Uneven weights. Filling the field is not the same as using it evenly. Even with rates that cover every pixel, the composite weights are a sum over paths, concentrated in the centre and uneven between pixels. The example below measures this. It is related to the effective receptive field of Luo et al., whose spread grows only as the square root of depth, as the Receptive Field article describes.

Large rates on small maps. When the rate approaches the size of the feature map, most taps of a $3 \times 3$ filter fall on padding and the filter degenerates towards its centre tap. DeepLabv3 added an image-level pooling branch to ASPP for this reason, as the DeepLab article describes and measures.

Degridding. DRN removes gridding from a dilated ResNet in three steps (Yu et al., 2017, Sec. 4):

• removing max pooling: the max pooling after the first $7 \times 7$ convolution produced high-amplitude, high-frequency activations that could propagate to later layers and exacerbate gridding, so it was replaced by convolutional layers;
• adding layers: after the last 4-dilated layer, a 2-dilated residual block and then a 1-dilated block were added (levels 7 and 8), a step the authors liken to removing aliasing with filters of the appropriate frequency;
• removing residual connections in levels 7 and 8, because the shortcuts could carry the gridding from level 6 past the new layers.

The result, DRN-C, is deeper than the DRN-A it comes from: DRN-C-26, derived from the 18-layer network, has 21.1 million parameters against 11.7 million. On the ImageNet 2012 validation set (single centre crop) DRN-C-26 has a top-1 error of 24.86%, against 28.00% for DRN-A-18, 24.81% for the deeper DRN-A-34 and 30.43% for ResNet-18 (their Table 1). On Cityscapes validation, used without extra modules and trained with only cropping and mirroring, DRN-C-26 reaches a mean IoU of 68.0 and DRN-C-42 70.9, against 67.3 for DRN-A-50 (Table 3).

Examples

The example works in NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2, with no training. It runs in about 7 seconds.

• Part 1 computes the exact composite kernels of linear stacks of $3 \times 3$ all-ones filters by convolving the upsampled filters, and checks them against a brute-force enumeration of tap paths. For each stack it prints the field size, the share of the field with nonzero weight (the coverage that DeepLab’s example prints for such stacks), the spread $\sigma$ against the closed form above, and the effective coverage $1 / (N \sum_i p_i^2)$, where $p_i$ are the normalized weights over the field’s $N$ pixels. Effective coverage is 100% only when every pixel weighs the same, and equals the covered share when the covered pixels weigh the same.
• Part 2 applies the same analysis to three tails on the stride-8 feature map, following the per-level dilations of DRN’s Figure 5. Tail A has residual blocks at rates 2, 2, 4, 4 (two convolutions each, as in levels 5 and 6 of DRN-A-18). Tail B adds DRN-B’s residual blocks at rates 2 and 1 (levels 7 and 8). Tail C adds the same four layers without shortcuts, as DRN-C does. A linear residual block is $x + c \cdot \text{branch}(x)$, and the share of the identity path depends on the branch gain $c$, which a linear model leaves open. The code prints both an all-ones branch ($c = 1$) and a branch normalized to unit sum ($c = 1/81$). The extra column is the share of the composite weight off the even lattice, which is 0 for a gridded stack. A field with uniform weights would put about three-quarters of its weight there, the share of positions off the lattice (by our arithmetic, $1 - 31^2/61^2 \approx 74\%$ for a $61 \times 61$ field).
• Part 3 builds the three tails as random-weight ReLU networks: 8 channels, He-normal weights, no bias, zero padding, and a $1 \times 1$ stem from the grayscale input. Their input is scikit-image’s camera image subsampled to $256 \times 256$ and standardized. The code checks its dilated convolution against SciPy, then checks the polyphase view: tail A, all of whose rates are even, equals four copies with halved rates run on the four $2\times$-subsampled images and interleaved. DeepLab’s example checks the single-layer linear case; this check covers a whole residual ReLU stack. Finally it adds 1 to one pixel, DRN’s single active pixel, and records which outputs in the field change and how much of the summed change lands on the even lattice, over seeds 0–4.

import itertools
import numpy as np
from scipy.signal import convolve2d, correlate2d
from skimage import data

# ---------- Part 1: exact composite kernels of linear stacks (all-ones 3x3 kernels) ----------
def upsampled_ones(r):
    """All-ones 3x3 kernel with r - 1 zeros between taps."""
    w = np.zeros((2 * r + 1, 2 * r + 1)); w[::r, ::r] = 1.0
    return w

def composite(rates):
    """Weight of every input offset in the stack's output: the number of tap paths reaching it."""
    K = np.ones((1, 1))
    for r in rates:
        K = convolve2d(K, upsampled_ones(r))
    return K

def residual_block(K, rates, c):
    """Linear residual block x + c * branch(x), the branch being plain layers with these rates."""
    B = c * composite(rates)
    B[B.shape[0] // 2, B.shape[1] // 2] += 1.0          # identity path
    return convolve2d(K, B)

def describe(K):
    """Field size, coverage, spread sigma, effective coverage, weight share off the even lattice."""
    R = K.shape[0] // 2
    yy, xx = np.mgrid[-R:R + 1, -R:R + 1]
    p = K / K.sum()
    sigma = np.sqrt((p * xx ** 2).sum())                 # spread along one axis, in pixels
    eff = 1.0 / (K.size * (p ** 2).sum())                # 1 if every pixel of the field weighs the same
    off = p[(yy % 2 != 0) | (xx % 2 != 0)].sum()
    return K.shape[0], (K > 0).mean(), sigma, eff, off

def brute_force(rates):
    """Independent check: enumerate every combination of taps and count where it lands."""
    R = sum(rates); K = np.zeros((2 * R + 1, 2 * R + 1))
    for taps in itertools.product(*[[(a * r, b * r) for a in (-1, 0, 1) for b in (-1, 0, 1)] for r in rates]):
        K[R + sum(t[0] for t in taps), R + sum(t[1] for t in taps)] += 1
    return K

print("rates        field  covered  sigma (closed form)  effective coverage")
for rates in [(2, 2, 2), (1, 2, 4), (1, 2, 5), (1, 2, 9)]:
    K = composite(rates)
    assert np.array_equal(K, brute_force(rates))
    n, cov, sigma, eff, _ = describe(K)
    print(f"{str(rates):12s} {n:3d}x{n:<3d} {cov:6.1%}  {sigma:5.2f} ({np.sqrt(2 / 3 * sum(r * r for r in rates)):5.2f})"
          f"        {eff:5.1%}")

# ---------- Part 2: DRN-like tails on the stride-8 feature map ----------
# A = residual blocks at rates 2, 2, 4, 4 (two convs
# each); B adds residual blocks at rates 2 and 1; C adds the same four convs without shortcuts.
# The share of the identity path depends on the branch's gain c, which the linear model leaves open.
print("\nDRN-like tails                field  covered  sigma  effective coverage  weight off the lattice")
for label, c in (("branch gain 1 (all-ones)", 1.0), ("branch gain 1/81 (unit sum)", 1 / 81)):
    KA = np.ones((1, 1))
    for r in (2, 2, 4, 4):
        KA = residual_block(KA, (r, r), c)
    KB = residual_block(residual_block(KA, (2, 2), c), (1, 1), c)
    KC = convolve2d(KA, composite((2, 2, 1, 1)))
    print(label)
    for name, K in (("A", KA), ("B", KB), ("C", KC)):
        n, cov, sigma, eff, off = describe(K)
        print(f"  {name}                          {n}x{n}  {cov:6.1%}  {sigma:5.2f}  {eff:17.1%}  {off:21.1%}")

# ---------- Part 3: random-weight ReLU stacks (8 channels, He initialization, no bias) ----------
def conv(x, W, r):
    """x: (C_in, H, W); W: (C_out, C_in, 3, 3); dilated cross-correlation with zero padding."""
    _, H, Wd = x.shape
    xp = np.pad(x, ((0, 0), (r, r), (r, r)))
    return sum(np.einsum("oc,chw->ohw", W[:, :, u, v], xp[:, r * u:r * u + H, r * v:r * v + Wd])
               for u in range(3) for v in range(3))

relu = lambda z: np.maximum(z, 0)

def make_net(kind, seed, C=8):
    rng = np.random.default_rng(seed)
    he = lambda cin, k: rng.standard_normal((C, cin, k, k)) * np.sqrt(2 / (cin * k * k))
    net = [("stem", 1, he(1, 1))] + [("res", r, he(C, 3), he(C, 3)) for r in (2, 2, 4, 4)]
    if kind == "B":
        net += [("res", 2, he(C, 3), he(C, 3)), ("res", 1, he(C, 3), he(C, 3))]
    if kind == "C":
        net += [("plain", r, he(C, 3)) for r in (2, 2, 1, 1)]
    return net

def run(net, img, rate_divisor=1):
    x = img[None]
    for kind, r, *W in net:
        if kind == "stem":
            x = relu(np.einsum("oc,chw->ohw", W[0][:, :, 0, 0], x))
        elif kind == "res":
            r //= rate_divisor
            x = relu(x + conv(relu(conv(x, W[0], r)), W[1], r))
        else:
            x = relu(conv(x, W[0], r // rate_divisor))
    return x

# check the convolution against SciPy on a zero-upsampled kernel
rng = np.random.default_rng(1)
xt, Wt = rng.standard_normal((3, 20, 20)), rng.standard_normal((2, 3, 3, 3))
ref = np.array([sum(correlate2d(xt[c], np.kron(Wt[o, c], [[1, 0], [0, 0]])[:5, :5], mode="same")
                    for c in range(3)) for o in range(2)])
print(f"\ndilated conv vs SciPy, max |difference|: {np.abs(conv(xt, Wt, 2) - ref).max():.0e}")

img = data.camera()[::2, ::2].astype(float)            # 256 x 256
img = (img - img.mean()) / img.std()
print(f"input {img.shape}, mean {abs(img.mean()):.1f}, std {img.std():.1f}")

# A's rates are all even: it equals four rate-halved copies on the 2x-subsampled images, interleaved
net = make_net("A", 0)
y = run(net, img)
y4 = np.zeros_like(y)
for a in (0, 1):
    for b in (0, 1):
        y4[:, a::2, b::2] = run(net, img[a::2, b::2], rate_divisor=2)
print(f"A vs four interleaved half-rate copies, max |difference|: {np.abs(y - y4).max():.0e}")

# one active pixel (DRN's didactic case) through deep stacks: which outputs change, and where
print("\nimpulse at (128, 128), seeds 0-4   outputs changed in the field   share of |change| on the lattice")
i0 = 128
for kind, R in (("A", 24), ("B", 30), ("C", 30)):
    changed, on_lattice = [], []
    for seed in range(5):
        net = make_net(kind, seed)
        x2 = img.copy(); x2[i0, i0] += 1.0
        d = np.abs(run(net, x2) - run(net, img)).sum(0)[i0 - R:i0 + R + 1, i0 - R:i0 + R + 1]
        yy, xx = np.mgrid[-R:R + 1, -R:R + 1]
        changed.append((d > 1e-12).mean())
        on_lattice.append(d[(yy % 2 == 0) & (xx % 2 == 0)].sum() / d.sum())
    print(f"  {kind} ({2 * R + 1}x{2 * R + 1} field)                 {min(changed):6.1%} to {max(changed):6.1%}"
          f"             {min(on_lattice):6.1%} to {max(on_lattice):6.1%}")

Output:

rates        field  covered  sigma (closed form)  effective coverage
(2, 2, 2)     13x13   29.0%   2.83 ( 2.83)        15.8%
(1, 2, 4)     15x15  100.0%   3.74 ( 3.74)        67.9%
(1, 2, 5)     17x17  100.0%   4.47 ( 4.47)        83.2%
(1, 2, 9)     25x25   70.6%   7.57 ( 7.57)        55.9%

DRN-like tails                field  covered  sigma  effective coverage  weight off the lattice
branch gain 1 (all-ones)
  A                          49x49   26.0%   7.26               7.1%                   0.0%
  B                          61x61  100.0%   7.70              19.9%                  68.3%
  C                          61x61  100.0%   7.70              20.1%                  69.1%
branch gain 1/81 (unit sum)
  A                          49x49   26.0%   5.16               1.9%                   0.0%
  B                          61x61  100.0%   5.48               4.1%                  34.6%
  C                          61x61  100.0%   5.77               9.9%                  69.1%

dilated conv vs SciPy, max |difference|: 4e-15
input (256, 256), mean 0.0, std 1.0
A vs four interleaved half-rate copies, max |difference|: 0e+00

impulse at (128, 128), seeds 0-4   outputs changed in the field   share of |change| on the lattice
  A (49x49 field)                  26.0% to  26.0%             100.0% to 100.0%
  B (61x61 field)                  99.9% to 100.0%              36.9% to  41.4%
  C (61x61 field)                  99.9% to 100.0%              27.3% to  32.1%

Coverage is not uniform use. The (2, 2, 2) row reproduces the 49 of 169 pixels that DeepLab’s example reports, and (1, 2, 9) leaves holes, as in that example and as hybrid dilated convolution’s choice of rates predicts. Rates (1, 2, 4) and (1, 2, 5) reach every pixel. Even so, their effective coverage is 68% and 83%, because offsets reachable along several tap paths count several times. The spread $\sigma$ matches the closed form in every row: it depends only on the sum of squared rates, not on whether the rates share a factor.

Degridding in the tails. Tail A sees a quarter of its $49 \times 49$ field (26%, the lattice of even offsets within it) and puts no weight off the lattice. Adding layers at rates 2 and 1 fills the $61 \times 61$ field in both B and C. The shortcuts matter only through their gain. With all-ones branches, the identity path’s single unit of weight is small against the branch’s 81, and B is almost the same as C: 68.3% against 69.1% of the weight off the lattice. With unit-sum branches, half of B’s weight bypasses the rate-1 block, and B keeps only 34.6% off the lattice against C’s 69.1%. This is DRN’s argument for removing the residual connections in levels 7 and 8, in a linear model: whether a shortcut carries the gridding past the degridding layers depends on how strong the branch is relative to the identity. Neither tail reaches the 74% of a uniform field. The even-rate layers move no weight off the lattice, and along each axis two all-ones rate-1 layers land on an even offset with probability $(1/3)^2 + (2/3)^2 = 5/9$, so C’s share is $1 - (5/9)^2 \approx 69.1\%$ at either gain, by our arithmetic. The weights also concentrate in the centre: effective coverage stays at about 20% or below.

Through nonlinear networks. The polyphase identity holds to the last bit (a difference of 0): a stack whose rates are all even is four networks that never exchange information, so neighbouring outputs come from disjoint sets of pixels. Accordingly, the single active pixel changes exactly the 26% of A’s field on the lattice, for every seed, and all of the change lands there. In B and C it changes practically every output in the field (99.9% to 100%), but in B 36.9% to 41.4% of the change still falls on the even lattice, against 27.3% to 32.1% in C, which has no shortcuts in the added layers. These are untrained networks; the example shows where the weight can go, not what training puts there.

Tails and the real DRN. The tails follow the per-level dilations in DRN’s Figure 5, whose caption gives every layer of a level the same dilation. Section 2 of the paper, and its Figure 1, specify the conversion per layer differently: the first convolution of group 4, from which only the stride is removed, keeps rate 1, the rest of group 4 and the first convolution of group 5 are 2-dilated, and the rest of group 5 is 4-dilated. A real DRN-A therefore has a rate-1 convolution inside its first dilated residual block. By our reasoning, which the example does not compute and which the DeepLab article also gives for a rate-1 layer in a converted stage, that layer fills the lattice in the same way as the rate-1 layers of tails B and C, while the block’s shortcut, which bypasses it, keeps part of the weight on the lattice, as in tail B. This fits DRN’s observation that DRN-A still shows gridding artefacts (its Fig. 3).

Common Misconceptions

• “A dilated filter has more weights.” It has exactly as many as the undilated filter; only its span grows. The extra pixels inside the span are skipped, not weighted.
• “Dilation is the same as downsampling.” A rate-$r$ layer computes the same values as filtering the $r \times r$ subsampled images and interleaving the results (Long et al., 2015), but it does so at every position. The output stays dense; the filter itself sees no finer detail than its taps allow.
• “Dilation is transposed convolution.” A transposed convolution with stride $s$ can be written as inserting zeros between the input samples and convolving, which upsamples the map. A dilated convolution inserts zeros between the filter taps and keeps the map’s size.
• “A field that covers the whole image uses the whole image.” The theoretical field of a dilated stack can have holes (gridding), and even without holes its weights are uneven, as the effective coverage above shows. Training changes the weights but not the lattice of a gridded stack.
• “Adding undilated layers at the end always removes gridding.” DRN found that the shortcuts of residual blocks can carry the artefacts past the added layers, and removed them; in the linear tails above, how much they carry depends on the branch gain.

Where It Is Used

Dense prediction backbones. Semantic segmentation networks dilate the last stages of a classification backbone to output stride 8 or 16: DeepLab, PSPNet and DRN, whose class activation maps at $28 \times 28$ also served for weakly supervised localization on ImageNet without fine-tuning. DeepLabv3+ makes the depthwise stage of a depthwise separable convolution dilated, an atrous separable convolution, which it applies in its ASPP and decoder modules and in a modified Xception backbone whose max poolings are replaced by strided depthwise separable convolutions (Chen et al., 2018).

Optical flow. PWC-Net’s context network, which refines the estimated flow, is a stack of seven $3 \times 3$ convolutions with dilation constants 1, 2, 4, 8, 16, 1 and 1 from bottom to top (Sun et al., 2018). The two rate-1 layers at the top play the part of DRN’s degridding layers, in our reading. Removing the context network raised the average endpoint error, for a model trained on FlyingChairs, from 2.00 to 2.23 on Chairs and from 3.33 to 3.47 on Sintel Clean (their Table 5c; the PWC-Net article notes that the supplement prints the “No context” and “No DenseNet” rows the other way round).

Tracking. SiamRPN++ sets the strides of conv4 and conv5 of its ResNet-50 to 1 and dilates them, so that the last blocks have an effective stride of 8 instead of 16 and 32 for Siamese tracking (Li et al., 2019, Sec. 3.2). Its paper cites FCN for the dilated convolutions.

Temporal models. MS-TCN, a temporal action segmentation network, stacks ten acausal 1-D convolutions of kernel size 3 with dilations doubling from 1 to 512, each in a residual layer, so that by its Equation 3 ten layers cover 2,047 frames at full temporal resolution. Its authors say the doubling was inspired by WaveNet, a generative model for raw audio by van den Oord et al. (2016), which used causal dilated convolutions, so that each output depends only on the past. On 50Salads, going from 6 to 10 layers per stage raised frame-wise accuracy from 63.7% to 80.7%, which the authors attribute mainly to the larger receptive field (their Table 7). ByteNet (Kalchbrenner et al., 2016) applied dilated convolutions to machine translation; as the Attention Is All You Need article describes, the Transformer paper noted that in ByteNet’s dilated convolutions the number of operations relating two positions grows logarithmically with their distance. Neither the WaveNet nor the ByteNet paper could be read here.