Transposed Convolution

A strided convolution shrinks a feature map: a $3 \times 3$ kernel with stride 2 turns a $64 \times 64$ map into a $32 \times 32$ one. A transposed convolution goes the other way, from the small map to the large one, while keeping the same pattern of connections between input and output positions. It is the learned upsampling layer of segmentation networks, which must return per-pixel predictions at the input resolution, and of image generators.

Definition

Any convolutional layer is linear, so with the input and output images flattened into vectors it is a matrix product $\mathbf{y} = C\mathbf{x}$ , where $C$ is a sparse matrix whose nonzero entries are the kernel weights. The transposed convolution with the same kernel, stride and padding is the linear map whose matrix is $C^\top$ : it takes a vector the size of the convolution’s output to one the size of its input (Dumoulin & Visin, 2016).

Since backpropagation carries a gradient from a convolution’s output to its input by multiplying with $C^\top$ , a transposed convolution is the gradient of a convolution with respect to its input: it swaps the convolution’s forward and backward passes (Dumoulin & Visin, 2016), as PyTorch’s documentation of ConvTranspose2d also states.

Intuition

A convolution gathers: each output pixel is a weighted sum of a $k \times k$ window of input pixels. A transposed convolution scatters: each input pixel stamps a copy of the kernel, scaled by its value, onto the output, and overlapping stamps add up. With stride $s$ , consecutive stamps are placed $s$ pixels apart in the output, so an $n \times n$ input produces an output roughly $s$ times larger. Odena, Dumoulin and Olah (2016) describe this as letting every point of the small image paint a square in the larger one.

The same result is an ordinary convolution of a stretched input, hence the name fractionally strided convolution: insert $s - 1$ zeros between neighboring input pixels, pad, and slide the kernel with stride 1, so it moves $1/s$ of an input pixel per step (Dumoulin & Visin, 2016).

Formal Definition

For a convolution with kernel size $k$ , stride $s$ and padding $p$ on an input of size $i$ , the output size is $\lfloor (i + 2p - k)/s \rfloor + 1$ . The associated transposed convolution, applied to an input of size $i'$ , is equivalent to a stride-1 convolution of the stretched input (with $s - 1$ zeros inserted between units) padded by $p' = k - p - 1$ , and its output size is (Dumoulin & Visin, 2016, Relationship 14)

o' = s\,(i' - 1) + a + k - 2p,

where $a = (i + 2p - k) \bmod s$ adds $a$ extra rows and columns at the bottom and right. The term $a$ exists because a stride- $s$ convolution maps $s$ input sizes to the same output size, so one must be chosen. PyTorch calls $a$ output_padding and includes the dilation $d$ of the kernel:

o' = (i' - 1)\,s - 2p + d\,(k - 1) + a + 1,

which equals the formula above for $d = 1$ . In PyTorch, padding is the padding of the forward convolution: the layer pads its stretched input by $d(k-1) - p$ , so that a Conv2d and a ConvTranspose2d built with the same arguments have mirrored input and output shapes.

Written as a direct (cross-correlation) convolution, the transposed layer applies the kernel rotated by 180 degrees, because transposing $C$ reverses the order of the taps; the code below checks this.

Properties

Shape, not values. $C^\top C \neq I$ in general: the transpose restores the input’s shape and connectivity, not the input (Dumoulin & Visin, 2016). A transposed convolution layer learns its own kernel, untied to the encoder it mirrors.
Bilinear upsampling is a special case. Bilinear interpolation by an integer factor $f$ spreads each input pixel with the same weights everywhere, so away from the border it is a transposed convolution with stride $f$ and a fixed kernel. Long, Shelhamer and Darrell (2015) note that such a filter need not be fixed to bilinear upsampling but can be learned.

Examples

The NumPy code below builds the matrix $C$ of a $3 \times 3$ convolution on a $4 \times 4$ image (stride 1 without padding, and stride 2 with padding 1, both giving $2 \times 2$ ), compares $C^\top$ with the zero-insertion construction, checks the gradient by finite differences, upsamples a ramp image with a bilinear kernel, and counts kernel overlaps for $k = 3$ and $k = 4$ with stride 2.

import numpy as np
from scipy import signal

def conv_matrix(i, w, s=1, p=0):
    """Matrix C of a strided, zero-padded cross-correlation (a CNN 'convolution')
    acting on an i x i image flattened row by row."""
    k = w.shape[0]
    o = (i + 2*p - k) // s + 1
    C = np.zeros((o*o, i*i))
    for r in range(o):
        for c in range(o):
            for u in range(k):
                for v in range(k):
                    y, x = s*r + u - p, s*c + v - p      # input pixel under tap (u, v)
                    if 0 <= y < i and 0 <= x < i:        # taps on the zero padding drop out
                        C[r*o + c, y*i + x] = w[u, v]
    return C, o

def conv_transpose(z, w, s=1, p=0, a=0):
    """Transposed convolution as a direct convolution: insert s-1 zeros between inputs,
    pad by k-p-1 (plus a extra rows/columns at the bottom and right), then
    cross-correlate with the kernel rotated by 180 degrees."""
    k = w.shape[0]
    n = z.shape[0]
    up = np.zeros((s*(n - 1) + 1,) * 2)
    up[::s, ::s] = z                                     # zero insertion ("fractional stride")
    q = k - p - 1
    up = np.pad(up, ((q, q + a), (q, q + a)))
    return signal.correlate2d(up, w[::-1, ::-1], mode="valid")

# A 4x4 image: a bright 2x2 square in the top-left corner on a dark background.
img = np.array([[0, 0, 0, 0],
                [0, 9, 9, 0],
                [0, 9, 9, 0],
                [0, 0, 0, 0]], float)
w = np.array([[1, 2, 0],
              [0, 1, 0],
              [0, 0, -1]], float)                        # asymmetric, so the flip matters

for s, p in [(1, 0), (2, 1)]:
    C, o = conv_matrix(4, w, s, p)
    y = (C @ img.ravel()).reshape(o, o)
    ref = signal.correlate2d(np.pad(img, p), w, mode="valid")[::s, ::s]
    print(f"stride {s}, padding {p}: C is {C.shape[0]}x{C.shape[1]}, output {o}x{o},"
          f" matches correlate2d: {np.allclose(y, ref)}")
    z = np.array([[1., 2.], [3., 4.]])                   # any 2x2 map, e.g. coarse class scores
    a = (4 + 2*p - 3) % s                                # extra row/column (output_padding)
    up_matrix = (C.T @ z.ravel()).reshape(4, 4)
    up_direct = conv_transpose(z, w, s, p, a)
    print(f"  C^T maps 2x2 -> {up_matrix.shape[0]}x{up_matrix.shape[1]};"
          f" zero-insertion + flipped kernel (a={a}) gives the same: {np.allclose(up_matrix, up_direct)}")
    print("  using the unflipped kernel instead:", np.allclose(up_matrix, conv_transpose(z, w[::-1, ::-1], s, p, a)))
print("stride-2 transposed convolution of [[1, 2], [3, 4]]:")
print(up_matrix.astype(int))

# C^T is the gradient of the convolution with respect to its input:
# for L = sum(g * conv(x)), dL/dx = C^T g. Check by central finite differences.
C, o = conv_matrix(4, w, 2, 1)
g = np.array([[0.5, -1.0], [2.0, 0.25]])                 # an upstream gradient on the 2x2 output
L = lambda x: np.sum(g.ravel() * (C @ x.ravel()))
eps = 1e-6
fd = np.zeros(16)
for j in range(16):
    e = np.zeros(16); e[j] = eps
    fd[j] = (L(img.ravel() + e) - L(img.ravel() - e)) / (2 * eps)
print("finite-difference gradient == C^T g:", np.allclose(fd, C.T @ g.ravel(), atol=1e-6))
print("C C^T == identity (is C^T an inverse?):", np.allclose(C @ C.T, np.eye(4)))
print("conv then transpose recovers the image?", np.allclose(C.T @ C @ img.ravel(), img.ravel()))

# Bilinear 2x upsampling is a fixed transposed convolution: k=4, s=2, p=1.
b1 = np.array([0.25, 0.75, 0.75, 0.25])
B = np.outer(b1, b1)
ramp = np.add.outer(np.arange(4.0), 2 * np.arange(4.0))  # 4x4 ramp image: value = row + 2*col
up = conv_transpose(ramp, B, s=2, p=1, a=0)              # 4x4 -> 8x8
# Bilinear interpolation reproduces a linear ramp exactly, so the answer is the ramp
# evaluated at the output pixel centers, mapped back as source = (x + 0.5)/2 - 0.5.
src = (np.arange(8) + 0.5) / 2 - 0.5
exact = np.add.outer(src, 2 * src)
print("bilinear kernel:", B.shape, "| output", up.shape)
print("interior (rows/cols 1..6) equals the ramp:", np.allclose(up[1:7, 1:7], exact[1:7, 1:7]))
print("row 3, transposed conv:", up[3])
print("row 3, ramp:           ", exact[3])

# Checkerboard: transposed convolution of a constant 6x6 map with an all-ones kernel
# counts how many inputs reach each output pixel.
const = np.ones((6, 6))
for k in (3, 4):
    out = conv_transpose(const, np.ones((k, k)), s=2, p=1)
    print(f"k={k}, s=2: output {out.shape[0]}x{out.shape[1]}, central 4x4 overlap counts:")
    c = out.shape[0] // 2
    print(out[c-2:c+2, c-2:c+2].astype(int))

Output:

stride 1, padding 0: C is 4x16, output 2x2, matches correlate2d: True
  C^T maps 2x2 -> 4x4; zero-insertion + flipped kernel (a=0) gives the same: True
  using the unflipped kernel instead: False
stride 2, padding 1: C is 4x16, output 2x2, matches correlate2d: True
  C^T maps 2x2 -> 4x4; zero-insertion + flipped kernel (a=1) gives the same: True
  using the unflipped kernel instead: False
stride-2 transposed convolution of [[1, 2], [3, 4]]:
[[ 1  0  2  0]
 [ 6  3  8 -2]
 [ 3  0  4  0]
 [ 0 -3  0 -4]]
finite-difference gradient == C^T g: True
C C^T == identity (is C^T an inverse?): False
conv then transpose recovers the image? False
bilinear kernel: (4, 4) | output (8, 8)
interior (rows/cols 1..6) equals the ramp: True
row 3, transposed conv: [0.9375 1.75   2.75   3.75   4.75   5.75   6.75   5.4375]
row 3, ramp:            [0.75 1.75 2.75 3.75 4.75 5.75 6.75 7.75]
k=3, s=2: output 11x11, central 4x4 overlap counts:
[[4 2 4 2]
 [2 1 2 1]
 [4 2 4 2]
 [2 1 2 1]]
k=4, s=2: output 12x12, central 4x4 overlap counts:
[[4 4 4 4]
 [4 4 4 4]
 [4 4 4 4]
 [4 4 4 4]]

For stride 2 with padding 1, $a = (4 + 2 - 3) \bmod 2 = 1$ ; without it the zero-insertion construction would give $3 \times 3$ . In the stride-2 result, even positions hold the input values (the kernel’s center weight is 1) and the others sum off-center weights of neighboring stamps. $C^\top$ matches the finite-difference gradient but does not undo the convolution.

The bilinear kernel, the outer product of $(\tfrac14, \tfrac34, \tfrac34, \tfrac14)$ with itself, reproduces the ramp exactly except at the border, where the missing neighbors outside the image count as zeros. With a constant input and an all-ones kernel, the output counts the stamps reaching each pixel: 1, 2 and 4 for $k = 3$ , a uniform 4 for $k = 4$ .

Common Misconceptions

“Deconvolution.” The layer is often called a deconvolution layer, including by Long et al. (2015), but it does not invert a convolution. Dumoulin and Visin (2016) advise against the name for that reason, and Radford, Metz and Chintala (2016) call it wrong. In image processing, deconvolution means undoing a blur, estimating a sharp image from one convolved with a known point spread function, as Wiener filtering and Richardson–Lucy deconvolution do. The deconvolutional networks of Zeiler, Krishnan, Taylor and Fergus (2010) are a different model again: an unsupervised sparse decomposition that generates an image as a sum of feature maps convolved with learned filters, so inferring the maps is a true deconvolution problem. The “deconvnet” visualizations of Zeiler and Fergus (2014) project activations back to pixels with transposed versions of the trained filters, which the arXiv version (arXiv:1311.2901) explains as flipping each filter vertically and horizontally, plus unpooling and rectification.

“The zero-insertion picture is how the layer is computed.” Implementations skip the multiplications by zero and usually compute the layer as the gradient of a convolution (Dumoulin & Visin, 2016).

Checkerboard Artifacts

Odena, Dumoulin and Olah (2016) traced the checkerboard pattern in many generated images to uneven overlap: when the kernel size is not divisible by the stride, some output pixels receive more stamps than others. In one dimension, a stride-2, size-3 transposed convolution gives some outputs twice as many inputs as others; in two dimensions the unevenness multiplies to a factor of four, exactly the 1, 2, 4 pattern of the example. A network could in principle learn weights that balance the overlap, but the authors report that models struggle to, that the artifacts appear in generators even before training, and that stacked layers compound them at several scales.

In signal-processing terms (our reading), zero insertion creates spectral copies of the input and the kernel is the interpolation filter that should remove them. Each output phase (position modulo $s$ ) uses its own subset of taps; if the subsets sum differently, a flat input comes out periodic, an aliasing of reconstruction rather than of sampling.

The remedies follow:

Resize, then convolve. Upsample with a fixed interpolation (nearest-neighbor or bilinear, as in image warping), then apply a stride-1 convolution. Odena et al. (2016) had their best results with nearest-neighbor resizing and reported difficulty with bilinear resizing.
Make the kernel size divisible by the stride, such as $k = 4$ or $k = 2$ for $s = 2$ . This removes the uneven overlap, but Odena et al. (2016) found that even-overlap layers can still learn artifact-producing kernels.
Sub-pixel convolution. Shi et al. (2016) compute a stride-1 convolution in the low-resolution space with $r^2 C$ output channels and then rearrange them with a periodic shuffle into an $r$ times larger image with $C$ channels (PyTorch’s PixelShuffle). They state that this equals a convolution with fractional stride $1/r$ whose kernel size is a multiple of $r$ , computed without any convolution in the high-resolution space; Odena et al. (2016) equate the divisible-kernel choice with it.

In PyTorch

The snippet below (run with PyTorch 2.14.1) loads the NumPy example’s kernel into nn.ConvTranspose2d, compares it with autograd’s gradient of conv2d, and checks the output-size formula.

import torch
from torch import nn
import torch.nn.functional as F

w = torch.tensor([[1., 2., 0.], [0., 1., 0.], [0., 0., -1.]])
z = torch.tensor([[1., 2.], [3., 4.]]).view(1, 1, 2, 2)

# The same layer as the NumPy example: k=3, stride 2, padding 1, output_padding 1.
up = nn.ConvTranspose2d(1, 1, kernel_size=3, stride=2, padding=1, output_padding=1, bias=False)
with torch.no_grad():
    up.weight.copy_(w.view(1, 1, 3, 3))   # weight shape is (in_channels, out_channels, k, k)
print(up(z)[0, 0].int())

# It is the gradient of Conv2d with the same weight with respect to the conv's input.
x = torch.zeros(1, 1, 4, 4, requires_grad=True)
y = F.conv2d(x, w.view(1, 1, 3, 3), stride=2, padding=1)       # 4x4 -> 2x2
(grad,) = torch.autograd.grad(y, x, grad_outputs=z)
print("equals the gradient of conv2d:", torch.equal(grad, up(z)))

# Output size: o = (i - 1)s - 2p + d(k - 1) + output_padding + 1
def formula(i, k, s, p, op=0, d=1):
    return (i - 1) * s - 2 * p + d * (k - 1) + op + 1

for k, s, p, op, d in [(3, 2, 1, 0, 1), (3, 2, 1, 1, 1), (4, 2, 1, 0, 1), (2, 2, 0, 0, 1), (3, 1, 1, 0, 2)]:
    layer = nn.ConvTranspose2d(1, 1, k, stride=s, padding=p, output_padding=op, dilation=d)
    o = layer(torch.zeros(1, 1, 16, 16)).shape[-1]
    print(f"k={k} s={s} p={p} output_padding={op} dilation={d}: 16 -> {o} (formula {formula(16, k, s, p, op, d)})")

# A stride-2 Conv2d maps both 31x31 and 32x32 inputs to 16x16; output_size picks one.
down = nn.Conv2d(1, 1, 3, stride=2, padding=1)
up = nn.ConvTranspose2d(1, 1, 3, stride=2, padding=1)
for n in (31, 32):
    h = down(torch.zeros(1, 1, n, n))
    print(f"{n}x{n} -> {h.shape[-1]}x{h.shape[-1]} -> back to", up(h, output_size=[n, n]).shape[-2:])