DeepLab

DeepLab is a family of semantic segmentation networks by Liang-Chieh Chen, George Papandreou and colleagues, first at UCLA and Google and later at Google, released in four versions between 2014 and 2018. Like FCN, each version runs an ImageNet-pretrained CNN fully convolutionally. Unlike FCN, it replaces the last downsampling steps with atrous (dilated) convolution, computing features every 8 or 16 pixels instead of every 32 with the same weights. The versions then add, in turn, a fully connected conditional random field (CRF) for boundaries (v1, ICLR 2015), atrous spatial pyramid pooling, ASPP, for multi-scale context (v2, IEEE TPAMI 2018), an improved ASPP with batch normalization and image-level features, without the CRF (v3, 2017), and a light decoder with depthwise separable convolutions (v3+, ECCV 2018).

Problem Addressed

The DeepLab journal paper names three obstacles to dense labeling with classification networks: pooling and striding reduce resolution, typically by 32; objects appear at many scales; and the invariance that helps classification blurs localization. DeepLab computes deep features more densely, upsamples the smooth class scores bilinearly, and recovers boundaries separately, first with a CRF and later with a decoder.

Architecture

Atrous convolution

For a 1-D input $x$ and a filter $w$ of length $K$, atrous convolution with rate $r$ is

$$y[i] = \sum_{k=1}^{K} x[i + r \cdot k]\, w[k],$$

with non-mirrored filters, as is usual in deep learning; $r = 1$ is ordinary convolution, and 2-D is the same along each axis. Equivalently, the filter is upsampled by inserting $r - 1$ zeros between taps, the “holes” (trous) of the algorithme à trous for the undecimated wavelet transform, from which the name comes. A $k \times k$ filter then spans

$$k_e = k + (k - 1)(r - 1)$$

pixels, which is the dilated kernel span in the receptive field recurrences, while its parameters and multiplications per output stay at $k^2$. The ICLR paper called it the “hole algorithm”; the TPAMI paper says the authors first used the term atrous convolution in Papandreou, Kokkinos and Savalle (2015). Yu and Koltun (2016) named the same operator dilated convolution, noting that the algorithme à trous uses it rather than being it.

The example below implements the 2-D operator on a synthetic image (a disk on stripes), checks it against ordinary convolution with the zero-upsampled filter, and shows impulse responses for rates 1, 2 and 4. It then reproduces the TPAMI paper’s dense-versus-sparse illustration: filtering a 2× subsampled image gives the rate-2 response at every other pixel only. Parts 3 and 4 belong to Limitations.

import numpy as np
from scipy.signal import convolve2d

def atrous2d(x, w, r):
    """y[i, j] = sum_{u, v} x[i + r*u, j + r*v] w[u, v] with u, v centered on 0 and zero
    padding: correlation with non-mirrored filters, as in the DeepLab papers."""
    k = w.shape[0]; pad = r * (k // 2)
    xp = np.pad(x, pad)
    H, W = x.shape
    y = np.zeros((H, W))
    for u in range(k):
        for v in range(k):
            y += w[u, v] * xp[r * u : r * u + H, r * v : r * v + W]
    return y

def upsample_filter(w, r):
    """Insert r - 1 zeros between taps: a k x k filter becomes k + (k-1)(r-1) wide."""
    k = w.shape[0]; ke = k + (k - 1) * (r - 1)
    we = np.zeros((ke, ke)); we[::r, ::r] = w
    return we

# A synthetic image: a bright disk on a striped background, plus noise.
rng = np.random.default_rng(0)
yy, xx = np.mgrid[0:64, 0:64]
img = 0.3 + 0.1 * np.sin(yy / 2.0) + 0.5 * ((yy - 30) ** 2 + (xx - 34) ** 2 < 15 ** 2)
img += 0.02 * rng.standard_normal(img.shape)
w = rng.standard_normal((3, 3))                                    # any 3 x 3 filter: 9 weights

# 1. Impulse response, parameters, and equivalence with the zero-upsampled filter.
imp = np.zeros((15, 15)); imp[7, 7] = 1.0
for r in (1, 2, 4):
    resp = atrous2d(imp, np.ones((3, 3)), r) > 0                   # outputs the impulse reaches
    rows = np.flatnonzero(resp.any(axis=1)).tolist()
    ref = convolve2d(img, upsample_filter(w, r)[::-1, ::-1], mode="same")
    print(f"rate {r}: impulse reaches {resp.sum()} outputs in rows {rows} "
          f"(span {rows[-1] - rows[0] + 1}), weights {w.size}, "
          f"max |atrous - upsampled filter| {np.abs(atrous2d(img, w, r) - ref).max():.0e}")

# 2. Dense vs sparse extraction: a 3 x 3 filter on the 2x-subsampled image gives the
#    rate-2 atrous response at every other pixel; atrous convolution gives it at all pixels.
sparse = atrous2d(img[::2, ::2], w, 1)
dense = atrous2d(img, w, 2)
print("subsample then convolve:", sparse.shape, "| atrous, rate 2:", dense.shape,
      "| max difference on the even grid:", np.abs(dense[::2, ::2] - sparse).max())

# 3. Gridding: which input pixels can reach one output of three stacked 3 x 3 layers?
def stack_support(rates):
    s = np.ones((1, 1))
    for r in rates:
        s = convolve2d(s, upsample_filter(np.ones((3, 3)), r)) > 0
    return s

for rates in ((2, 2, 2), (2, 4, 8), (1, 2, 3), (1, 2, 5), (1, 2, 9)):
    s = stack_support(rates)
    print(f"rates {rates}: field {s.shape[0]}x{s.shape[1]}, reached {s.sum():3d} of {s.size:3d}"
          f" pixels ({100 * s.mean():.0f}%)")
print("rates (2, 2, 2), center 9 x 9 of the field:\n", stack_support((2, 2, 2))[2:11, 2:11].astype(int))

# 4. Large rates on a small map: on a 65 x 65 map, how many of a 3 x 3 filter's taps
#    fall inside the map rather than on zero padding?
n = 65
i = np.arange(n)
for r in (1, 6, 12, 18, 24, 36, 64):
    inside = [((i + r * u >= 0) & (i + r * u < n)) for u in (-1, 0, 1)]   # per axis
    taps = sum(np.outer(a, b) for a in inside for b in inside)             # valid taps per output
    print(f"rate {r:2d}: outputs using all 9 taps {np.mean(taps == 9):.2f}, "
          f"only the center tap {np.mean(taps == 1):.2f}")

Output:

rate 1: impulse reaches 9 outputs in rows [6, 7, 8] (span 3), weights 9, max |atrous - upsampled filter| 1e-15
rate 2: impulse reaches 9 outputs in rows [5, 7, 9] (span 5), weights 9, max |atrous - upsampled filter| 1e-15
rate 4: impulse reaches 9 outputs in rows [3, 7, 11] (span 9), weights 9, max |atrous - upsampled filter| 1e-15
subsample then convolve: (32, 32) | atrous, rate 2: (64, 64) | max difference on the even grid: 0.0
rates (2, 2, 2): field 13x13, reached  49 of 169 pixels (29%)
rates (2, 4, 8): field 29x29, reached 225 of 841 pixels (27%)
rates (1, 2, 3): field 13x13, reached 169 of 169 pixels (100%)
rates (1, 2, 5): field 17x17, reached 289 of 289 pixels (100%)
rates (1, 2, 9): field 25x25, reached 441 of 625 pixels (71%)
rates (2, 2, 2), center 9 x 9 of the field:
 [[1 0 1 0 1 0 1 0 1]
 [0 0 0 0 0 0 0 0 0]
 [1 0 1 0 1 0 1 0 1]
 [0 0 0 0 0 0 0 0 0]
 [1 0 1 0 1 0 1 0 1]
 [0 0 0 0 0 0 0 0 0]
 [1 0 1 0 1 0 1 0 1]
 [0 0 0 0 0 0 0 0 0]
 [1 0 1 0 1 0 1 0 1]]
rate  1: outputs using all 9 taps 0.94, only the center tap 0.00
rate  6: outputs using all 9 taps 0.66, only the center tap 0.00
rate 12: outputs using all 9 taps 0.40, only the center tap 0.00
rate 18: outputs using all 9 taps 0.20, only the center tap 0.00
rate 24: outputs using all 9 taps 0.07, only the center tap 0.00
rate 36: outputs using all 9 taps 0.00, only the center tap 0.01
rate 64: outputs using all 9 taps 0.00, only the center tap 0.94

The impulse reaches nine outputs at every rate, over 3, 5 and 9 rows: same weights, wider span.

Backbone at output stride 8 or 16

The output stride is the ratio of input to final feature resolution: 32 for VGG-16 and ResNet classifiers. To halve it, the network sets the stride of the last downsampling layer to 1 and gives every later convolution rate 2; doing this twice gives output stride 8 with rate 4 at the end. DeepLab v1 did this to VGG-16’s last two max-pooling layers (rate 2 for conv5, 4 for fc6), keeping the pooling but not its striding, as Yu and Koltun (2016) note. No new weights are needed, so the ImageNet initialization still applies and the output stride can differ between training and testing.

The price is compute: each halving of the output stride quadruples the positions of every later layer. The PyTorch example (run with PyTorch 2.14.1, torch only) builds ResNet-50’s bottleneck stages, converting stride to dilation with torchvision’s rule (the first block of a converted stage keeps the previous rate), and counts the multiply-accumulates of all convolutions for a $513 \times 513$ input, DeepLab’s crop size for PASCAL VOC. It then builds DeepLabv3’s ASPP head for both output strides.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

class Bottleneck(nn.Module):
    """ResNet-50 bottleneck: 1x1 reduce, 3x3 (strided or dilated), 1x1 expand, + shortcut."""
    def __init__(self, cin, width, stride, dilation):
        super().__init__()
        cout = 4 * width
        self.conv1 = nn.Conv2d(cin, width, 1, bias=False); self.bn1 = nn.BatchNorm2d(width)
        self.conv2 = nn.Conv2d(width, width, 3, stride, padding=dilation, dilation=dilation, bias=False)
        self.bn2 = nn.BatchNorm2d(width)
        self.conv3 = nn.Conv2d(width, cout, 1, bias=False); self.bn3 = nn.BatchNorm2d(cout)
        self.short = None
        if stride != 1 or cin != cout:
            self.short = nn.Sequential(nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))

    def forward(self, x):
        y = F.relu(self.bn1(self.conv1(x)))
        y = F.relu(self.bn2(self.conv2(y)))
        y = self.bn3(self.conv3(y))
        return F.relu(y + (x if self.short is None else self.short(x)))

def resnet50_backbone(output_stride):
    """Stages of 3, 4, 6, 3 blocks. Below the target output stride, a stage's stride 2 is
    replaced by dilation: its first block keeps the previous rate, later blocks double it."""
    layers = [nn.Conv2d(3, 64, 7, 2, 3, bias=False), nn.BatchNorm2d(64), nn.ReLU(),
              nn.MaxPool2d(3, 2, 1)]                                   # stride 4
    cin, stride_now, rate = 64, 4, 1
    for i, (width, blocks) in enumerate(((64, 3), (128, 4), (256, 6), (512, 3))):
        s = 1 if i == 0 else 2
        prev_rate = rate
        if s == 2 and stride_now * 2 > output_stride:
            rate, s = rate * 2, 1                                      # dilate instead of stride
        stride_now *= s
        for b in range(blocks):
            layers.append(Bottleneck(cin, width, s if b == 0 else 1, prev_rate if b == 0 else rate))
            cin = 4 * width
    return nn.Sequential(*layers)

class ASPP(nn.Module):
    """DeepLabv3 ASPP: 1x1 conv, three 3x3 atrous convs, image pooling; 256 channels each,
    concatenated and projected by a 1x1 conv (all with batch norm and ReLU)."""
    def __init__(self, cin, rates, c=256):
        super().__init__()
        def cbr(k, r=1):
            return nn.Sequential(nn.Conv2d(cin, c, k, padding=r * (k // 2), dilation=r, bias=False),
                                 nn.BatchNorm2d(c), nn.ReLU())
        self.branches = nn.ModuleList([cbr(1)] + [cbr(3, r) for r in rates])
        self.pool = nn.Sequential(nn.AdaptiveAvgPool2d(1), cbr(1))
        self.project = nn.Sequential(nn.Conv2d(c * (len(rates) + 2), c, 1, bias=False),
                                     nn.BatchNorm2d(c), nn.ReLU())

    def forward(self, x):
        outs = [b(x) for b in self.branches]
        outs.append(F.interpolate(self.pool(x), size=x.shape[-2:], mode="bilinear", align_corners=False))
        return self.project(torch.cat(outs, dim=1))

def conv_macs(model, x):
    """Multiply-accumulates of all convolutions, counted with forward hooks."""
    total = [0]
    def hook(m, inp, out):
        total[0] += out.numel() // out.shape[0] * (m.in_channels // m.groups) * m.kernel_size[0] * m.kernel_size[1]
    hs = [m.register_forward_hook(hook) for m in model.modules() if isinstance(m, nn.Conv2d)]
    with torch.no_grad():
        out = model(x)
    for h in hs:
        h.remove()
    return out, total[0]

nparams = lambda m: sum(p.numel() for p in m.parameters())
x = torch.randn(1, 3, 513, 513)                     # DeepLab's crop size for PASCAL VOC
backbone_macs = {}
for os_ in (32, 16, 8):
    net = resnet50_backbone(os_).eval()
    rates = [m.dilation[0] for m in net.modules() if isinstance(m, nn.Conv2d) and m.kernel_size == (3, 3)]
    feat, macs = conv_macs(net, x)
    backbone_macs[os_] = macs
    print(f"output stride {os_:2d}: features {tuple(feat.shape)}, {macs / 1e9:6.1f} GMACs, "
          f"{nparams(net):,} parameters, 3x3 rates in last 9 blocks {rates[-9:]}")

for os_, rates in ((16, (6, 12, 18)), (8, (12, 24, 36))):
    feat = torch.randn(1, 2048, (513 - 1) // os_ + 1, (513 - 1) // os_ + 1)
    aspp = ASPP(2048, rates).eval()
    out, macs = conv_macs(aspp, feat)
    per_branch = [nparams(b) for b in aspp.branches]
    print(f"ASPP rates {rates} at OS {os_}: {tuple(feat.shape)} -> {tuple(out.shape)}, "
          f"{macs / 1e9:.1f} GMACs, {nparams(aspp):,} parameters, per branch {per_branch}")
    print(f"  backbone + ASPP at OS {os_}: {(backbone_macs[os_] + macs) / 1e9:.1f} GMACs")

# torchvision's deeplabv3_resnet50 = OS-8 backbone + ASPP(12, 24, 36) + 3x3 conv head and
# 1x1 classifier (21 classes) + auxiliary FCN head on the stride-8 layer3 features.
cbr3 = lambda cin, c: [nn.Conv2d(cin, c, 3, padding=1, bias=False), nn.BatchNorm2d(c), nn.ReLU()]
head = nn.Sequential(*cbr3(256, 256), nn.Conv2d(256, 21, 1))
aux = nn.Sequential(*cbr3(1024, 256), nn.Dropout(0.1), nn.Conv2d(256, 21, 1))
total = nparams(resnet50_backbone(8)) + nparams(ASPP(2048, (12, 24, 36))) + nparams(head) + nparams(aux)
print(f"backbone + ASPP + head + auxiliary head: {total:,} parameters")

Output:

output stride 32: features (1, 2048, 17, 17),   22.6 GMACs, 23,508,032 parameters, 3x3 rates in last 9 blocks [1, 1, 1, 1, 1, 1, 1, 1, 1]
output stride 16: features (1, 2048, 33, 33),   34.1 GMACs, 23,508,032 parameters, 3x3 rates in last 9 blocks [1, 1, 1, 1, 1, 1, 1, 2, 2]
output stride  8: features (1, 2048, 65, 65),  102.7 GMACs, 23,508,032 parameters, 3x3 rates in last 9 blocks [1, 2, 2, 2, 2, 2, 2, 4, 4]
ASPP rates (6, 12, 18) at OS 16: (1, 2048, 33, 33) -> (1, 256, 33, 33), 16.3 GMACs, 15,535,104 parameters, per branch [524800, 4719104, 4719104, 4719104]
  backbone + ASPP at OS 16: 50.4 GMACs
ASPP rates (12, 24, 36) at OS 8: (1, 2048, 65, 65) -> (1, 256, 65, 65), 63.4 GMACs, 15,535,104 parameters, per branch [524800, 4719104, 4719104, 4719104]
  backbone + ASPP at OS 8: 166.1 GMACs
backbone + ASPP + head + auxiliary head: 42,004,074 parameters

The parameter count, the same at all three output strides, is torchvision’s 25,557,032 for ResNet-50 minus its 2,049,000-parameter classifier. Going from output stride 16 to 8 triples the backbone’s multiply-accumulates (34.1 to 102.7 G). The DeepLabv3+ paper reports the same pattern for ResNet-101 DeepLabv3, 81.02 billion multiply-adds when evaluated at output stride 16 and 276.18 billion at 8, for 77.21% and 78.51% mean IoU on VOC 2012 val (trained at 16).

Atrous spatial pyramid pooling

DeepLab v2’s ASPP classifies each position from several fields of view at once: parallel branches with different rates read the same feature map, and their class scores are summed. In the VGG-16 version, ASPP-L, four branches on pool5 (output stride 8) each apply a $3 \times 3$ convolution with rate 6, 12, 18 or 24 followed by two $1 \times 1$ layers; the ResNet-101 model puts ASPP on top of the backbone. It generalizes v1’s DeepLab-LargeFOV, a single $3 \times 3$ rate-12 fc6 branch, with fc6 and fc7 cut from 4,096 to 1,024 channels.

DeepLabv3 rebuilt the module on features, with 256 channels and batch normalization in every branch, concatenated and fused by a $1 \times 1$ convolution before the classifier:

$$z = W_{\text{fuse}} \ast \big[\, B_1(x),\; B_{6}(x),\; B_{12}(x),\; B_{18}(x),\; U\big(B_{\text{pool}}(x)\big) \,\big],$$

where $B_1$ is a $1 \times 1$ convolution, $B_r$ a $3 \times 3$ convolution with rate $r$ (rates 6, 12 and 18 at output stride 16, doubled at 8), $B_{\text{pool}}$ global average pooling followed by a $1 \times 1$ convolution, and $U$ bilinear upsampling back to the map size.

The image-level branch fixes a problem DeepLabv3 identified: as the rate nears the feature-map size, taps fall on padding and a $3 \times 3$ filter degenerates to $1 \times 1$. In the output above, each $3 \times 3$ branch has 4,719,104 parameters whatever its rate. DeepLabv3 also studied cascaded copies of ResNet’s last block with doubling rates and “multi-grid” unit rates such as (1, 2, 1) within each block, and chose ASPP after it scored higher (79.77% against 79.35% on VOC val with test-time augmentation).

Fully connected CRF

DeepLab v1 and v2 upsample the class scores bilinearly by 8 and refine the labeling $\mathbf{x}$ with the fully connected CRF of Krähenbühl and Koltun (2011), whose energy is

$$E(\mathbf{x}) = \sum_i -\log P(x_i) + \sum_{i<j} \mu(x_i, x_j)\left[ w_1 \exp\!\left( -\frac{\lVert p_i - p_j \rVert^2}{2\sigma_\alpha^2} - \frac{\lVert I_i - I_j \rVert^2}{2\sigma_\beta^2} \right) + w_2 \exp\!\left( -\frac{\lVert p_i - p_j \rVert^2}{2\sigma_\gamma^2} \right) \right],$$

where $P(x_i)$ is the network’s softmax probability at pixel $i$, $p$ is position, $I$ is RGB color, and $\mu$ is the Potts penalty, 1 for different labels and 0 otherwise. The first, bilateral kernel asks pixels close in position and color to share a label; the second only enforces spatial smoothness. Every pair of pixels is connected, unlike the grid energies minimized by graph cuts. Mean-field inference turns each message-passing step into Gaussian filtering in the feature space, which the permutohedral lattice makes linear in the number of pixels; DeepLab runs 10 iterations.

The CRF was trained piecewise: the network was fixed, $w_2 = 3$ and $\sigma_\gamma = 3$ were set to defaults, and $w_1$, $\sigma_\alpha$, $\sigma_\beta$ were found by coarse-to-fine search on 100 validation images. It raised VOC 2012 val mean IoU by 3 to 5 points across v1’s variants, LargeFOV’s from 62.25 to 67.64.

Decoder (DeepLabv3+)

DeepLabv3+ uses DeepLabv3 as the encoder of an encoder–decoder: its features at output stride 16 are upsampled by 4, concatenated with stride-4 backbone features reduced to 48 channels, refined by two $3 \times 3$ convolutions and upsampled by 4 again. The backbone is a modified aligned Xception: deeper, with every max pooling replaced by a strided depthwise separable convolution so that atrous convolution can be applied anywhere, and batch normalization and ReLU after each depthwise convolution. Depthwise separable convolution in ASPP and the decoder cut multiply-adds by 33% to 41% at similar accuracy.

Training

Each version minimizes the per-pixel cross-entropy, ignoring unlabeled pixels:

$$\ell(\theta) = -\sum_{i\,:\,y_i \neq \text{void}} \log \operatorname{softmax}\big(s_i(\theta)\big)_{y_i}.$$

In v1 and v2 the targets are the ground truth subsampled by 8; DeepLabv3 instead upsamples the logits.

• v1. SGD with 20-image batches, learning rate 0.001 (0.01 for the classifier) divided by 10 every 2,000 iterations, on the 10,582-image augmented VOC training set (trainaug); fine-tuning took about 10 hours.
• v2. The “poly” schedule multiplies the base rate by $(1 - t/T)^{0.9}$ at iteration $t$ of $T$; with batch size 10 and 20,000 iterations it raised LargeFOV by 3.63 points on val over step decay at similar training time. The ResNet-101 model added multi-scale inputs (scales 0.5, 0.75 and 1, fused by taking the maximum score), COCO pretraining and random rescaling between 0.5 and 1.5.
• v3. $513 \times 513$ crops, because large rates need large inputs; batch normalization fine-tuned with 16 images per batch at output stride 16, then frozen for a second stage at output stride 8 on VOC trainval; random scaling (0.5 to 2.0) and flips. An appendix added in the third arXiv version reports VOC val mean IoU of 77.21% with this protocol, 76.01% with downsampled labels and 67.22% with $321 \times 321$ crops. For the test submission, images with hard classes such as bicycle and chair were duplicated.

Inference

One forward pass gives scores at output stride 8 or 16, upsampled bilinearly or by the v3+ decoder; v1 and v2 then run the CRF. The TPAMI paper reports 8 frames per second for its network on an NVIDIA Titan X and 0.5 s for the CRF on a CPU. The best DeepLabv3 val figures add output stride 8 at test time and average probabilities over flips and scales from 0.5 to 1.75, which for ResNet-101 DeepLabv3 costs 4,870.59 billion multiply-adds against 81.02 billion for one pass at output stride 16 (DeepLabv3+ paper).

Performance

PASCAL VOC 2012 test results, as reported in each paper:

Version	Year	Backbone	Key additions	VOC 2012 test mIoU
DeepLab v1	2015	VGG-16, ImageNet	Atrous conv. at output stride 8, dense CRF, LargeFOV, multi-scale features	71.6
DeepLab v2	2016 (journal 2018)	ResNet-101, ImageNet + COCO	ASPP (rates 6–24), multi-scale inputs, poly schedule, dense CRF	79.7
DeepLabv3	2017	ResNet-101, ImageNet + COCO / + JFT-300M	ASPP with batch norm and image pooling, multi-grid, no CRF	85.7 / 86.9
DeepLabv3+	2018	Modified aligned Xception; without / with JFT-300M	Decoder, atrous separable conv.	87.8 / 89.0

The v1 figure is DeepLab-MSc-CRF-LargeFOV trained on trainval; LargeFOV with the CRF alone scored 70.3. The JFT-300M result first appears in the second arXiv version of the DeepLabv3 paper. The DeepLabv3+ paper describes COCO pretraining as its setup for comparisons and names only JFT as the difference between its two test entries.

On other benchmarks, the TPAMI paper reports 45.7% mean IoU on PASCAL-Context val (59 classes plus background), 64.94% on PASCAL-Person-Part val, and 70.4% on the Cityscapes test set, trained on the train set only. DeepLabv3 reports 81.3% on the Cityscapes test set with fine train+val and the coarse annotations, in an appendix added in the paper’s third arXiv version; DeepLabv3+ reports 82.1% with coarse annotations.

Strengths

• Resolution without new weights. Any pretrained classification backbone can produce denser features.
• Adjustable output stride. One trained model can run at output stride 16 for speed or 8 for accuracy.
• Multi-scale context in one pass. ASPP probes several fields of view on one feature map instead of an image pyramid.

Limitations

• Gridding. A stack of atrous layers with a common factor in their rates samples the input on a lattice. Wang et al. (2018), who called this gridding, illustrate it with three rate-2 layers, whose top unit sees the input as a checkerboard; in the NumPy output they reach 49 of the 169 pixels in their field, and rates 2, 4, 8 reach 27%. Their hybrid dilated convolution (HDC) uses rates without a common factor, chosen so that the field has no holes: 1, 2, 3 or 1, 2, 5, but not 1, 2, 9. By our reasoning, residual shortcuts do not remove the lattice, since every path still moves in multiples of the common factor, but one rate-1 $3 \times 3$ layer in the chain, such as the first block of a converted stage under torchvision’s rule, fills it.
• Large rates on small maps. DeepLabv3 found that the effective taps of a $3 \times 3$ filter shrink as the rate grows; in the NumPy example, on a $65 \times 65$ map only 7% of outputs use all nine taps at rate 24, and at rate 64 most use only the center tap.
• Cost of dense features. Output stride 8 roughly triples the backbone’s cost relative to 16; DeepLabv3 trained at 16 because only 6 images per batch fit at 8, too few for batch norm.
• Decoupled CRF. In v1 and v2 the CRF was tuned separately, added CPU time, and could not restore what the network missed: the TPAMI paper shows bicycles and chairs whose delicate boundaries are lost even after the CRF, and suggests an encoder–decoder, which v3+ adopted.

Historical Importance

The TPAMI paper observed that most top PASCAL VOC methods of the time used atrous convolution, the dense CRF or both. PSPNet (Zhao et al., 2017) builds its features on a ResNet adapted by this “dilated network strategy”, citing DeepLab and Yu and Koltun.

Variants

Besides the four versions (v1 preprint December 2014, v2 preprint 2016), v1’s multi-scale features, two-layer heads on the input and first four pooling layers, added about 1.5 points on VOC val, less than the CRF. The TensorFlow repository adds MobileNetV2 and MobileNetV3 backbones and the architecture-searched Auto-DeepLab.

Implementations

• TensorFlow (tensorflow/models, research/deeplab): the authors’ implementation (first released March 2018), with ResNet, Xception, MobileNet and searched backbones. Its README directs users to the newer TensorFlow 2 codebase, DeepLab2.
• torchvision deeplabv3_resnet50, deeplabv3_resnet101 and deeplabv3_mobilenet_v3_large implement DeepLabv3 without the v3+ decoder. The ResNet models dilate the last two stages (output stride 8) and use ASPP rates 12, 24 and 36, a $3 \times 3$ convolution, a $1 \times 1$ classifier and an optional auxiliary FCN head; with 21 classes, the PyTorch example’s parts add up to the documented 42,004,074 parameters of deeplabv3_resnet50. Its weights were trained on a COCO subset with the 20 VOC categories and score 66.4 mean IoU on COCO val2017 with VOC labels.
• Original Caffe code for v1 and v2 was released with the papers.