Residual Connection

A residual connection (also called a skip or shortcut connection) adds the input of a block of layers to the block’s output, so the block computes $\mathbf{y} = \mathbf{x} + \mathcal{F}(\mathbf{x})$ instead of $\mathbf{y} = \mathcal{F}(\mathbf{x})$. Proposed as residual learning by He, Zhang, Ren and Sun in ResNet (2016), it let them train convolutional networks of 152 layers on ImageNet and over 1,000 layers on CIFAR-10, where plain networks of a few dozen layers already trained worse than shallower ones. Transformers, in language and vision, rely on it too.

Definition

A residual block passes its input $\mathbf{x}$ through a few layers that compute the residual function $\mathcal{F}$ and adds $\mathbf{x}$ back, unchanged, through the shortcut. The layers no longer learn the desired mapping $\mathcal{H}(\mathbf{x})$ itself but its difference from the identity, $\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x}$ (He et al., 2016). When the identity is already a good mapping, the block can approach it by driving $\mathcal{F}$ toward zero.

Intuition

He et al. (2016) started from an observation they called the degradation problem. On CIFAR-10, a 56-layer plain network (a stack of convolutions without shortcuts) had higher training error than a 20-layer one (their Fig. 1), and on ImageNet a 34-layer plain network had higher training error than an 18-layer one throughout training. Rising training error rules out overfitting, and capacity is not the limit either: the deeper network could copy the shallower one and set its extra layers to the identity. The optimizer fails to find them.

A residual block makes the identity the default: with its layers’ output at zero, the block passes its input through unchanged. The layers need only learn what to change in the incoming feature maps, for example to sharpen an edge response or add a part detector. He et al. called this a preconditioning of the problem, and supported it by measuring that the residual branches of trained ResNets have smaller responses than the corresponding layers of plain networks, and smaller still in deeper ResNets.

Formal Definition

He et al. (2016) define a block as

$$\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x},$$

where for two layers $\mathcal{F} = W_2\,\sigma(W_1 \mathbf{x})$ with ReLU $\sigma$ (biases omitted), and a second ReLU follows the addition. The shortcut adds no parameters, so a residual network has the same size as its plain counterpart. When $\mathcal{F}$ changes the number of channels or the resolution, the shortcut needs a linear projection $W_s$:

$$\mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + W_s \mathbf{x}.$$

Identity or projection. When ResNet halves the resolution and doubles the channels, $\mathbf{x}$ and $\mathcal{F}(\mathbf{x})$ no longer match. He et al. (2016) compared three options on a 34-layer network: (A) identity shortcuts everywhere, with extra channels padded with zeros; (B) $1 \times 1$ convolution projections only where dimensions increase; (C) projections on every shortcut. All three clearly beat the plain network (ImageNet validation top-1 error 25.03%, 24.52% and 24.19%, against 28.54%, 10-crop testing). The authors attributed C’s small gain to the extra parameters of its thirteen projections and concluded that projections are not essential for the degradation problem.

Gradient flow. The Jacobian of one block follows from the sum rule: $\partial \mathbf{y}/\partial \mathbf{x} = I + \partial \mathcal{F}/\partial \mathbf{x}$. Their follow-up paper (He et al., 2016b) carries this through a whole stack. If nothing follows the addition, so that $\mathbf{x}_{l+1} = \mathbf{x}_l + \mathcal{F}(\mathbf{x}_l, W_l)$, unrolling the recursion from block $l$ to a deeper block $L$ gives

$$\mathbf{x}_L = \mathbf{x}_l + \sum_{i=l}^{L-1} \mathcal{F}(\mathbf{x}_i, W_i),$$

and the chain rule gives the gradient of a loss $E$ as

$$\frac{\partial E}{\partial \mathbf{x}_l} = \frac{\partial E}{\partial \mathbf{x}_L} \left( I + \frac{\partial}{\partial \mathbf{x}_l} \sum_{i=l}^{L-1} \mathcal{F}(\mathbf{x}_i, W_i) \right).$$

The first term carries the gradient from block $L$ back to block $l$ without passing through any weight layer. In a plain network, backpropagation multiplies the error signal by one Jacobian per layer, and such a product can shrink or grow exponentially. He et al. (2016b) argue that the second term is unlikely to cancel the first for every example of a mini-batch, so the gradient does not vanish even when the weights are arbitrarily small. Replacing the identity shortcut by a scaling, $\mathbf{x}_{l+1} = \lambda \mathbf{x}_l + \mathcal{F}(\mathbf{x}_l)$, turns the first term into a product $\prod_i \lambda_i$, which again vanishes or explodes with depth.

Properties

Keeping the shortcut clean. In the original block a ReLU follows the addition, so the shortcut path is not a pure identity and the derivation above holds only approximately. He et al. (2016b) moved batch normalization and ReLU in front of each convolution (BN → ReLU → conv, twice) and left the sum untouched. On CIFAR-10, a 1001-layer ResNet with the original block reduced its training error slowly at first and reached 7.61% test error; with pre-activation blocks it reached 4.92%. Shortcuts that scale, gate, apply a $1 \times 1$ convolution or apply dropout to $\mathbf{x}$ all gave a 110-layer network higher error than the identity (6.61%), or failed to train. Goyal et al. (2017) initialize the scale $\gamma$ of the last batch-normalization layer in each residual block to 0, so each block starts as the identity and signals initially pass through the shortcuts. They report that this eases optimization early in training and improves results at mini-batch sizes 256 and 8,192.

Highway networks, the gated precursor. Srivastava, Greff and Schmidhuber (2015) proposed

$$\mathbf{y} = H(\mathbf{x})\,T(\mathbf{x}) + \mathbf{x}\,\bigl(1 - T(\mathbf{x})\bigr), \qquad T(\mathbf{x}) = \sigma(W_T \mathbf{x} + \mathbf{b}_T),$$

an elementwise sigmoid gate inspired by LSTM recurrent networks that interpolates between transforming and carrying the input. Initializing $\mathbf{b}_T$ negative (for example −1 or −3) biases the network toward carrying, and in preliminary experiments the authors optimized highway networks of up to 900 layers with SGD and momentum; their NeurIPS paper of the same year extends this work. He et al. (2016), who describe highway networks as concurrent work, point to the difference: gates have parameters and depend on the data, and a closed gate removes the shortcut, whereas an identity shortcut is never closed. In the general form of Srivastava et al., $\mathbf{y} = H(\mathbf{x})\,T(\mathbf{x}) + \mathbf{x}\,C(\mathbf{x})$ with a separate carry gate $C$, a residual block corresponds to both gates fixed at 1; that comparison is ours.

An ensemble of paths. Veit, Wilber and Belongie (2016) unroll a residual network into paths: each block is either entered or skipped, so $n$ blocks create $2^n$ paths of different lengths. They deleted single residual blocks from a trained 110-layer CIFAR-10 ResNet at test time: apart from the downsampling blocks, no deletion noticeably changed the error, whereas deleting any layer of a 15-layer VGG-style network dropped it to chance. Deleting several blocks raised the error smoothly, as in an ensemble. Almost all of the gradient during training came from paths 5 to 17 blocks long, 0.45% of all paths.

A smoother loss landscape. Li et al. (2018) plotted the loss around trained minimizers along filter-normalized random directions. Without shortcuts, the landscape went from nearly convex at 20 layers to chaotic at 56 and 110 layers; with shortcuts, it stayed smooth as depth grew. For 20-layer networks the shortcuts made little visible difference.

Examples

The bottleneck block. For ResNet-50, -101 and -152, He et al. (2016) replaced the two $3 \times 3$ layers by three layers: a $1 \times 1$ convolution that reduces the channels (256 to 64 in the first stage), a $3 \times 3$ convolution on the reduced maps, and a $1 \times 1$ convolution that restores them, with option B shortcuts. The identity shortcut matters here: a projection between the two 256-channel ends would, the authors note, double the block’s time complexity and model size.

Degradation on digit images. The PyTorch example below (run with PyTorch 2.14.1) trains plain and residual CNNs of 8 and 26 weight layers on 1,437 of scikit-learn’s 8×8 digit images. Both versions have identical parameters; the residual one adds each block’s input before its last ReLU. It reports training loss and accuracy averaged over the sixth and last epoch, and the first convolution’s gradient norm at step 1.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X = torch.tensor(digits.images / 16.0, dtype=torch.float32).unsqueeze(1)   # (1797, 1, 8, 8)
y = torch.tensor(digits.target)
idx_tr, _ = train_test_split(np.arange(len(y)), test_size=0.2, random_state=0, stratify=digits.target)
X_tr, y_tr = X[idx_tr], y[idx_tr]                                          # 1,437 training images

class Block(nn.Module):
    """Two 3x3 conv-BN layers; with residual=True the input is added before the last ReLU."""
    def __init__(self, c, residual):
        super().__init__()
        self.residual = residual
        self.f = nn.Sequential(nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c), nn.ReLU(),
                               nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c))
    def forward(self, x):
        return torch.relu(x + self.f(x)) if self.residual else torch.relu(self.f(x))

def make_net(n_blocks, residual, c=16):     # depth = 2 * n_blocks + 2 weight layers
    return nn.Sequential(nn.Conv2d(1, c, 3, padding=1, bias=False), nn.BatchNorm2d(c), nn.ReLU(),
                         *[Block(c, residual) for _ in range(n_blocks)],
                         nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(c, 10))

def train(net, epochs=6):
    opt = torch.optim.SGD(net.parameters(), lr=0.05, momentum=0.9)
    first_grad = None
    for epoch in range(epochs):
        total_loss = correct = 0.0
        for batch in torch.randperm(len(y_tr)).split(64):
            out = net(X_tr[batch])
            loss = nn.functional.cross_entropy(out, y_tr[batch])
            opt.zero_grad(); loss.backward()
            if first_grad is None:                       # gradient norm of the first conv layer, step 1
                first_grad = net[0].weight.grad.norm().item()
            opt.step()
            total_loss += loss.item() * len(batch)
            correct += (out.argmax(1) == y_tr[batch]).sum().item()
    return total_loss / len(y_tr), correct / len(y_tr), first_grad

for n_blocks in (3, 12):
    for residual in (False, True):
        torch.manual_seed(6)
        loss, acc, g = train(make_net(n_blocks, residual))
        print(f"{'residual' if residual else 'plain   '} {2 * n_blocks + 2:2d} layers: last-epoch training loss"
              f" {loss:.3f}, training accuracy {acc:.3f}, first-layer gradient norm at step 1 {g:.2f}")

Output:

plain     8 layers: last-epoch training loss 0.065, training accuracy 0.986, first-layer gradient norm at step 1 0.21
residual  8 layers: last-epoch training loss 0.032, training accuracy 0.992, first-layer gradient norm at step 1 0.26
plain    26 layers: last-epoch training loss 0.977, training accuracy 0.589, first-layer gradient norm at step 1 2.65
residual 26 layers: last-epoch training loss 0.066, training accuracy 0.980, first-layer gradient norm at step 1 0.61

At 8 layers both versions fit the training digits; at 26 the plain network degrades to 58.9% training accuracy, against 98.0% with shortcuts. Over 10 seeds, the 26-layer plain network ended at 39.5–83.4% training accuracy (median 56%) and the residual one at 96.6–98.8% (median 98%); seed 6 is typical. At step 1, the 26-layer plain network’s first-layer gradient is not smaller but about four times the residual network’s (1.9–3.2 against 0.4–0.7 over the seeds). In our reading, with batch normalization the plain network’s failure does not start from a vanishing gradient, consistent with He et al.’s (2016) report of healthy gradient norms.

The gradient identity. The NumPy code below applies a residual branch, conv → ReLU → conv, to an 8×8 digit image, computes its Jacobian by the chain rule, and checks $\partial \mathbf{y}/\partial \mathbf{x} = I + \partial \mathcal{F}/\partial \mathbf{x}$ against finite differences. It then chains 30 blocks with and without shortcuts and prints the largest gain (spectral norm) of the input-to-output Jacobian.

import numpy as np
from scipy import signal
from sklearn.datasets import load_digits

rng = np.random.default_rng(0)
x0 = load_digits().images[3] / 16.0                    # one 8x8 digit image (a "3")
x0 = x0 + rng.normal(0, 0.01, x0.shape)                # faint noise: no pre-activation sits exactly at ReLU's kink

def conv(x, k):                                        # 3x3 convolution, zero padding, same size
    return signal.correlate2d(x, k, mode="same")

def conv_matrix(k):                                    # the 64x64 matrix C with C @ x.ravel() = conv(x, k).ravel()
    return np.stack([conv(e.reshape(8, 8), k).ravel() for e in np.eye(64)], axis=1)

def F(x, k1, k2):                                      # residual branch: conv -> ReLU -> conv
    return conv(np.maximum(conv(x, k1), 0), k2)

def jac_F(x, k1, k2):                                  # chain rule: C2 @ diag(ReLU'(conv(x, k1))) @ C1
    mask = (conv(x, k1).ravel() > 0).astype(float)
    return conv_matrix(k2) @ (mask[:, None] * conv_matrix(k1))

# One block: check the Jacobian of y = x + F(x) against finite differences.
k1, k2 = rng.normal(0, 0.3, (2, 3, 3))
eps, J_fd = 1e-6, np.zeros((64, 64))
for i in range(64):
    d = np.zeros(64); d[i] = eps; d = d.reshape(8, 8)
    J_fd[:, i] = ((x0 + d + F(x0 + d, k1, k2)) - (x0 - d + F(x0 - d, k1, k2))).ravel() / (2 * eps)
print("one block: finite-difference dy/dx == I + dF/dx:",
      np.allclose(J_fd, np.eye(64) + jac_F(x0, k1, k2), atol=1e-6))

# 30 blocks with the same random kernels, with and without the identity shortcut.
for std in (0.1, 0.3):                                 # small and larger branch weights
    kernels = rng.normal(0, std, (30, 2, 3, 3))
    for residual in (False, True):
        x, J = x0, np.eye(64)
        for k1, k2 in kernels:                         # multiply the block Jacobians along the forward pass
            Jb = jac_F(x, k1, k2) + (np.eye(64) if residual else 0)
            x, J = (x + F(x, k1, k2) if residual else F(x, k1, k2)), Jb @ J
        print(f"kernel std {std}: {'residual' if residual else 'plain   '} 30 blocks,"
              f" largest gain of d(output)/d(input): {np.linalg.norm(J, 2):.1e}")

Output:

one block: finite-difference dy/dx == I + dF/dx: True
kernel std 0.1: plain    30 blocks, largest gain of d(output)/d(input): 2.4e-39
kernel std 0.1: residual 30 blocks, largest gain of d(output)/d(input): 1.7e+00
kernel std 0.3: plain    30 blocks, largest gain of d(output)/d(input): 1.0e-15
kernel std 0.3: residual 30 blocks, largest gain of d(output)/d(input): 8.1e+01

With small kernels, 60 plain convolutions shrink every input direction by about 39 orders of magnitude, while the residual stack keeps the gain near 1, as the term $I$ predicts. With larger kernels the residual product grows instead, to about 80: each factor $I + \partial \mathcal{F}/\partial \mathbf{x}$ is close to the identity only when the branch is small. In our reading, this is why normalization inside the branch and initializations that start each branch near zero, such as Goyal et al.’s, matter in deep residual networks. (Over seeds 1–9: plain gains of $10^{-39}$ or less and $10^{-13}$ to $10^{-10}$; residual gains 1.3–1.9 and 23–280.)

Common Misconceptions

• “Residual connections exist to fix vanishing gradients.” He et al. (2016) addressed the degradation of plain networks that had batch normalization and healthy gradients, as in the digit example. The gradient identity helps keep very deep stacks trainable, but the motivating problem was optimization difficulty.
• “A skip connection is a skip connection.” U-Net (Ronneberger, Fischer & Brox, 2015) also has skip connections, but they concatenate feature maps from the contracting path, cropped to size, with upsampled maps in the expanding path. They carry fine spatial detail to a segmentation decoder rather than adding a residual. DenseNet (Huang et al., 2017) also concatenates rather than sums: each layer receives the feature maps of all preceding layers in its block, $L(L+1)/2$ direct connections for $L$ layers.

Where It Is Used

Residual blocks are the building unit of ResNet, which also serves as a backbone for other tasks: He et al. (2016) reported higher detection accuracy on PASCAL VOC and MS COCO when they replaced VGG-16 with ResNet-101 in Faster R-CNN. More blocks also enlarge the receptive field of each unit. The transformer of Vaswani et al. (2017) wraps every sub-layer, such as self-attention or the feed-forward network, in a residual connection followed by layer normalization, $\mathrm{LayerNorm}(\mathbf{x} + \mathrm{Sublayer}(\mathbf{x}))$. The Vision Transformer (Dosovitskiy et al., 2021) applies layer normalization before every block and a residual connection after it, which in our reading keeps the shortcut path clean, as in the pre-activation block.