Encoder–Decoder

An encoder–decoder is a network in two halves. The encoder turns the input into a compact internal representation; the decoder turns that representation into the output. In vision the encoder is usually a convolutional neural network that shrinks an image to a coarse grid of many-channel features, and the decoder grows the grid back, often to the input’s own resolution, so that it can output one prediction per pixel: a class label in semantic segmentation, a depth, a flow vector, a clean pixel value, a joint-location heatmap. Without the spatial grid, the same split underlies sequence-to-sequence models and the original transformer.

Core Idea

A per-pixel prediction needs two things that pull in opposite directions. Deciding what a pixel shows needs context from a large receptive field, which a CNN builds by downsampling with strided convolutions and pooling. Deciding where a boundary runs needs full resolution. The encoder–decoder buys context by downsampling and then pays the resolution back in the decoder.

The narrowest point between the two is the bottleneck. In an autoencoder it is deliberately small, so the network must learn a compressed code. In a segmentation network it is typically 1/16 or 1/32 of the input’s width and height, with hundreds of channels: context is rich there, but exact positions are blurred. Most of the design choices below decide how to get spatial detail back past the bottleneck.

Building Blocks

Encoder. Usually a classification backbone with its classifier removed (VGG, ResNet, or a hierarchical transformer), pretrained on ImageNet. Its stages produce feature maps at strides 2, 4, 8, 16 and 32.

Upsampling in the decoder. Three choices are common:

• Transposed convolution, a learned upsampling layer. FCN uses it, initialized to bilinear interpolation. U-Net describes its “up-convolution” only as an upsampling followed by a $2 \times 2$ convolution; a $2 \times 2$, stride-2 transposed convolution is one way to implement it.
• Unpooling with pooling indices. SegNet’s encoder stores, for every $2 \times 2$ max-pooling window, which position held the maximum; the decoder places each value back at that position and fills the rest with zeros, then densifies the sparse map with trainable convolutions (Badrinarayanan, Kendall & Cipolla, 2017). The upsampling itself has no parameters, and the indices fit in 2 bits per window.
• Fixed interpolation followed by convolution. Nearest-neighbor or bilinear resizing, then a stride-1 convolution. The hourglass network, FPN and DeepLabv3+ upsample this way.

Skip connections. Paths that carry encoder features directly to the decoder stage of the same resolution, bypassing the bottleneck. They differ in what is passed and how it is merged:

• Concatenation of features (U-Net): the decoder’s upsampled map and the encoder’s map are stacked along the channel axis, and the following convolutions learn how to combine them.
• Summation of predictions (FCN): Long, Shelhamer and Darrell (2015) predict class scores from pool4 with a $1 \times 1$ convolution and add them to a 2× upsampled copy of the stride-32 scores (FCN-16s), then repeat from pool3 (FCN-8s).
• Lateral addition of projected features (FPN): a $1 \times 1$ convolution projects each encoder stage to a common width and adds it to the upsampled coarser map. Unlike U-Net, the feature pyramid network predicts from every level, not only the finest.

Skip connections in this sense are not residual connections: they cross between the two halves to restore spatial detail, not to ease optimization.

How Data Flows

The PyTorch snippet below (run with PyTorch 2.14.1 on a CPU, about 25 s) builds a tiny encoder–decoder: a full-resolution convolution, two stride-2 stages down to an $8 \times 8$ bottleneck, and two $2 \times 2$ stride-2 transposed convolutions back to $32 \times 32$. With skips=True it concatenates the encoder features of matching resolution, as U-Net does; otherwise the decoder sees only the bottleneck. Each model trains for 600 steps on synthetic $32 \times 32$ images that contain two one-pixel-wide line segments, four single-pixel dots and two bright disks, plus Gaussian noise. The target mask is the lines and dots but not the disks, so the network needs context (is this bright pixel part of a large disk?) and pixel precision (exactly which pixel is the dot?).

import sys
import torch
from torch import nn
import torch.nn.functional as F

torch.set_num_threads(1)
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0

def make_batch(n, g):
    """n noisy 32x32 images, each with two 1-pixel-wide line segments, four single-pixel
    dots and two bright disks of radius 3-5. Target: the lines and dots, not the disks."""
    flat = torch.zeros(2, n, 32 * 32)                       # channel 0: lines, channel 1: dots
    t = torch.linspace(0, 1, 64).view(1, 1, 64, 1)
    p0, p1 = torch.rand(2, n, 2, 1, 2, generator=g) * 31
    lines = (p0 + t * (p1 - p0)).round().long().view(n, -1, 2)
    dots = torch.randint(0, 32, (n, 4, 2), generator=g)
    flat[0].scatter_(1, lines[..., 0] * 32 + lines[..., 1], 1.0)
    flat[1].scatter_(1, dots[..., 0] * 32 + dots[..., 1], 1.0)
    yy, xx = torch.meshgrid(torch.arange(32.), torch.arange(32.), indexing="ij")
    c = torch.randint(4, 28, (n, 2, 2, 1, 1), generator=g).float()
    r = 3 + 2 * torch.rand(n, 2, 1, 1, generator=g)
    disks = ((yy - c[:, :, 0]) ** 2 + (xx - c[:, :, 1]) ** 2 <= r ** 2).any(1)
    thin = flat.view(2, n, 32, 32) * ~disks                # pixels on a disk are not targets
    img = torch.maximum(thin.amax(0), disks.float()) + 0.3 * torch.randn(n, 32, 32, generator=g)
    return img[:, None], thin.amax(0)[:, None], thin[1][:, None]   # image, mask, dots only

def block(cin, cout, stride=1):
    return nn.Sequential(nn.Conv2d(cin, cout, 3, stride, padding=1), nn.ReLU())

class EncoderDecoder(nn.Module):
    def __init__(self, skips):
        super().__init__()
        self.skips = skips
        self.enc0 = block(1, 8)                                   # full resolution
        self.enc1 = nn.Sequential(block(8, 16, 2), block(16, 16)) # 1/2
        self.enc2 = nn.Sequential(block(16, 32, 2), block(32, 32))# 1/4: the bottleneck
        self.up1 = nn.ConvTranspose2d(32, 16, 2, stride=2)        # 1/4 -> 1/2
        self.dec1 = block(32 if skips else 16, 16)
        self.up0 = nn.ConvTranspose2d(16, 8, 2, stride=2)         # 1/2 -> full
        self.dec0 = block(16 if skips else 8, 8)
        self.head = nn.Conv2d(8, 1, 1)                            # one logit per pixel

    def forward(self, x, log=None):
        e0 = self.enc0(x)
        e1 = self.enc1(e0)
        b = self.enc2(e1)
        u1 = self.up1(b)
        if self.skips:
            u1 = torch.cat([u1, e1], dim=1)                       # concatenate encoder features
        d1 = self.dec1(u1)
        u0 = self.up0(d1)
        if self.skips:
            u0 = torch.cat([u0, e0], dim=1)
        out = self.head(self.dec0(u0))
        if log is not None:
            log += [("enc0", e0), ("enc1", e1), ("bottleneck", b),
                    ("dec1 input", u1), ("dec0 input", u0), ("logits", out)]
        return out

g = torch.Generator().manual_seed(1234)
x_test, m_test, dot_test = make_batch(500, g)             # the same test set for every run

for skips in (False, True):
    torch.manual_seed(SEED)
    g = torch.Generator().manual_seed(SEED)
    net = EncoderDecoder(skips)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for step in range(600):
        x, m, _ = make_batch(32, g)
        loss = F.binary_cross_entropy_with_logits(net(x), m)
        opt.zero_grad()
        loss.backward()
        opt.step()
    with torch.no_grad():
        log = []
        pred = net(x_test, log) > 0
    truth = m_test > 0.5
    iou = (pred & truth).sum() / (pred | truth).sum()
    dot_recall = pred[dot_test > 0.5].float().mean()
    line_recall = pred[truth & (dot_test < 0.5)].float().mean()
    n_params = sum(p.numel() for p in net.parameters())
    print(f"{'with skips' if skips else 'no skips'}: {n_params:,} parameters")
    print("  " + ", ".join(f"{k} {tuple(v.shape[1:])}" for k, v in log))
    print(f"  test IoU {iou:.3f} | recall: line pixels {line_recall:.3f}, dots {dot_recall:.3f}")

Output (seed 0):

no skips: 22,953 parameters
  enc0 (8, 32, 32), enc1 (16, 16, 16), bottleneck (32, 8, 8), dec1 input (16, 16, 16), dec0 input (8, 32, 32), logits (1, 32, 32)
  test IoU 0.651 | recall: line pixels 0.792, dots 0.160
with skips: 25,833 parameters
  enc0 (8, 32, 32), enc1 (16, 16, 16), bottleneck (32, 8, 8), dec1 input (32, 16, 16), dec0 input (16, 32, 32), logits (1, 32, 32)
  test IoU 0.801 | recall: line pixels 0.909, dots 0.516

Concatenation doubles the channels entering each decoder convolution, hence the 2,880 extra parameters. Over seeds 0 to 9, test IoU ranged from 0.604 to 0.670 without skips and from 0.773 to 0.814 with them; seed 0 is typical. The single-pixel dots show the difference most clearly: the network without skips found 12–22% of them across the ten seeds, the one with skips 36–52%. In our reading, the bottleneck keeps only a blurred record of which of the 16 pixels under each bottleneck cell held a dot, while the skipped full-resolution features keep it exactly.

Mathematical Formulation

Write the encoder as $E$ and the decoder as $D$, with encoder stages $e_0, \dots, e_L$ at strides $1, 2, \dots, 2^L$. A plain encoder–decoder computes $\hat{y} = D(E(x))$, so the output depends on $x$ only through the bottleneck $z = e_L$. With skips, the decoder stage at stride $2^l$ computes

$$d_l = f_l\big(\,[\,U(d_{l+1}) \,;\, e_l\,]\,\big),$$

where $U$ is a 2× upsampling, $[\,\cdot\,;\,\cdot\,]$ concatenates channels and $f_l$ is a few convolutions. FPN replaces the concatenation with $U(d_{l+1}) + W_l e_l$ for a $1 \times 1$ projection $W_l$; FCN applies the sum to class scores rather than features. Training averages a per-pixel loss over the output grid: cross-entropy for labels, an $L^1$ or $L^2$ error for regression.

Why It Works

Downsampling makes context cheap: at stride 16 the taps of a $3 \times 3$ convolution span $48 \times 48$ input pixels, at 1/256 of the positions. Skips then supply what the bottleneck has lost. In pix2pix, a U-Net lets low-level information “shortcut across the network” (Isola et al., 2017): input and output share edges and layout, so there is no reason to squeeze them through the bottleneck.

Training Considerations

• Memory for skip features. Every skipped map must be kept until the decoder uses it. In Badrinarayanan et al.’s (2017) example, 64 first-layer maps at $180 \times 240$ in 32-bit floats take 11 MB, SegNet’s pooling indices about 0.17 MB. Their ablations still favored storing encoder maps in full, with compressed forms such as indices for when inference memory is constrained.
• Resolution alignment. Stride-2 layers map two input sizes to one output size, so a decoder may return a map one pixel off. U-Net’s unpadded convolutions shrink each map, so its encoder features are cropped before concatenation and a $572 \times 572$ input tile yields a $388 \times 388$ output map (Ronneberger et al., 2015). In practice (our note), padded convolutions and inputs whose sides are multiples of $2^L$ avoid both problems.
• Upsampling artifacts. Transposed convolutions whose kernel size is not divisible by the stride can produce checkerboard patterns; resize-then-convolve avoids them (transposed convolution).
• Normalization at the bottleneck. In pix2pix (arXiv v3 appendix), batch normalization with batch size 1 zeroed the bottleneck activations; the U-Net could bypass them through its skips, the plain encoder–decoder could not.

Variants

Symmetric and asymmetric decoders. U-Net keeps many feature channels in its expanding path so that context can reach the higher resolutions, which makes it roughly symmetric to the contracting path (Ronneberger et al., 2015). Newell, Yang and Deng (2016) designed the hourglass to be symmetric, contrasting it with FCN’s heavy bottom-up and light top-down processing. DeepLabv3+ is deliberately asymmetric (Chen et al., 2018): its atrous encoder produces features at output stride 16; the decoder upsamples them bilinearly by 4, concatenates stride-4 backbone features reduced to 48 channels, refines with two $3 \times 3$ convolutions, and upsamples by 4 again. Adding a stride-8 skip as well gave no significant improvement. SegFormer’s decoder is lighter still (Xie et al., 2021): linear layers bring the four encoder stages to one width, all are upsampled to 1/4 resolution and concatenated, and two more linear layers fuse them and predict classes. With the smallest encoder (MiT-B0) it has 0.4M parameters; the authors credit their transformer encoder’s large effective receptive field.

Autoencoders. The output space is the input space. Hinton and Salakhutdinov (2006) trained deep autoencoders with a small central layer, initialized layer by layer; from 30-dimensional codes of MNIST digits they reconstructed much better than principal component analysis. A denoising autoencoder (Vincent et al., 2008) is trained to reconstruct the clean input from a corrupted copy, in their experiments one with a fixed number of randomly chosen inputs set to 0. Vincent et al. stacked them to initialize deep networks; convolutional encoder–decoders now also serve image denoising directly.

Sequence-to-sequence models. Sutskever, Vinyals and Le (2014) mapped an input sentence to a vector of fixed dimension with a multilayer LSTM and decoded the translation from it with another; Cho et al. (2014) named the two-RNN design the RNN Encoder–Decoder. The fixed-size vector is a bottleneck in the same sense as above. The transformer keeps the encoder–decoder split but lets every decoder layer attend to all encoder outputs.

Use in Computer Vision

• Segmentation. FCN, SegNet, U-Net, DeepLabv3+ and SegFormer; see semantic segmentation and the image segmentation hub.
• Image-to-image translation. pix2pix (Isola et al., 2017) uses a U-Net generator with skips between layers $i$ and $n - i$. Without them, the encoder–decoder could not learn to generate realistic Cityscapes photos from label maps (arXiv v3, batch size 10); the U-Net also won with an $L^1$ loss alone.
• Monocular depth. Eigen, Puhrsch and Fergus (2014), an early network for depth estimation, predicted depth at 1/4 resolution from the two fully connected layers of a coarse network, then refined it with a fine-scale network that also sees the image: a bottleneck without a convolutional decoder.
• Optical flow. FlowNet (Dosovitskiy et al., 2015) compresses a pair of frames in a contracting part; each expanding step applies an “upconvolution” (unpooling and a convolution) and concatenates the result with the matching contracting-part features and an upsampled coarser flow estimate. After four steps the flow is at 1/4 resolution and is upsampled bilinearly.
• Human pose. The stacked hourglass network (Newell et al., 2016) pools down to $4 \times 4$, upsamples by nearest neighbor and adds a branch from each resolution on the way up, and outputs one $64 \times 64$ heatmap per joint for a $256 \times 256$ input. Eight hourglasses are stacked, with a loss on the heatmaps after each.

Limitations

• The bottleneck loses detail. Without skips, fine structure must survive downsampling as channels; the dots in the example mostly did not.
• Cost at high resolution. Each 2× decoder step quadruples the number of positions, so layers near full resolution, and the skipped maps they consume, are costly on large images. Many designs stop decoding at 1/4 of the input resolution: DeepLabv3+ and FlowNet then interpolate bilinearly, SegFormer predicts at 1/4, and the hourglass outputs its heatmaps there.

Important Models

Model	Year	Task	Encoder	Decoder upsampling	Skip type
Deep autoencoder (Hinton & Salakhutdinov)	2006	Dimensionality reduction	Fully connected layers to a 30-D code	Mirror of the encoder	None
Seq2seq (Sutskever et al.)	2014	Machine translation	Multilayer LSTM	Multilayer LSTM	None (fixed-size vector)
FCN	2015	Semantic segmentation	VGG-16	Transposed convolution, bilinear-initialized	Sum of class scores
U-Net	2015	Biomedical segmentation	Contracting path, unpadded $3 \times 3$ convolutions	$2 \times 2$ up-convolution	Concatenation of cropped features
FlowNet	2015	Optical flow	Contracting CNN on two frames	Unpooling + convolution	Concatenation of features and coarse flow
Stacked hourglass	2016	Human pose estimation	Residual modules, max pooling to $4 \times 4$	Nearest-neighbor	Elementwise addition
SegNet	2017	Scene segmentation	VGG-16 convolutional layers	Unpooling with max-pooling indices	Pooling indices only
pix2pix	2017	Image-to-image translation	$4 \times 4$ stride-2 convolutions	$4 \times 4$ convolutions upsampling by 2	Concatenation (layer $i$ to $n - i$)
FPN	2017	Detection, segmentation	ResNet	Nearest-neighbor	Lateral $1 \times 1$ projection, addition
DeepLabv3+	2018	Semantic segmentation	Atrous ResNet or Xception with ASPP	Bilinear ×4, twice	One concatenation (stride 4, 48 channels)
SegFormer	2021	Semantic segmentation	Hierarchical transformer	Upsampling to 1/4, all-MLP fusion	All stages concatenated

Years follow the cited publications; SegNet’s TPAMI paper appeared in 2017 after arXiv versions from 2015.