Receptive Field

The receptive field of a unit in a convolutional neural network is the region of the input image that can change its value. A first-layer unit with a $3 \times 3$ kernel sees nine pixels; a deep unit may see hundreds of pixels across. The receptive field limits what a unit can recognize: a stereo network cannot match a pixel on a blank wall if the unit sees only blank wall.

Definition

The theoretical receptive field of a unit is the set of input pixels on which the unit’s value depends: changing any pixel outside it leaves the unit unchanged, whatever the weights. For a CNN built from convolutions and pooling it is an $r \times r$ square, and $r$ is called the receptive field size.

The term comes from neurophysiology, where a neuron’s receptive field is the part of the sensory surface in which a stimulus changes its firing. Hubel and Wiesel mapped the receptive fields of single neurons in the cat’s visual cortex (1959, 1962), and Fukushima (1980) modeled the cells of the neocognitron, a forerunner of CNNs, on their simple and complex cells, with receptive fields that grow from stage to stage.

The effective receptive field (ERF), a notion introduced by Luo et al. (2016), is the part of the theoretical field where pixels have a non-negligible influence on the unit, measured by the gradient $\partial y / \partial x_{ij}$.

Intuition

A second-layer unit of two stacked $3 \times 3$ convolutions reads a $3 \times 3$ window of first-layer units, each of which reads its own $3 \times 3$ window of pixels; together they cover $5 \times 5$ pixels. Each further $3 \times 3$ layer adds one pixel on every side.

After a stride-2 layer, neighboring units are two input pixels apart, so the next $3 \times 3$ layer adds four pixels instead of two, and every later layer inherits the coarser spacing.

Not every pixel counts equally: a central pixel reaches the unit along many spatial paths through the intermediate layers, a corner pixel along only one. Influence concentrates near the center, in a soft blob much smaller than the square.

Formal Definition

Describe layer $l$ by its kernel size $k_l$, stride $s_l$, padding $p_l$ and dilation $d_l$. A dilated kernel spans $\hat{k}_l = d_l(k_l - 1) + 1$ input positions. Track three quantities for each layer, with the input image as layer 0 ($r_0 = 1$, $j_0 = 1$, $\text{start}_0 = 0$):

• $r_l$, the receptive field size in input pixels;
• $j_l$, the jump: the distance in input pixels between neighboring units of layer $l$ (also called the effective stride);
• $\text{start}_l$, the input coordinate of the center of the receptive field of unit 0.

Unit $i$ of layer $l$ is centered at $\text{start}_l + i\, j_l$ in the input. It reads the units $s_l i - p_l + d_l m$ of layer $l-1$, for $m = 0, \dots, k_l - 1$. Those units are centered at $\text{start}_{l-1} + (s_l i - p_l + d_l m)\, j_{l-1}$, so the outermost two are $(\hat{k}_l - 1)\, j_{l-1}$ apart, and each brings its own field of size $r_{l-1}$. Their midpoint, at $m = (k_l - 1)/2$, is the new center. This gives

$$r_l = r_{l-1} + (\hat{k}_l - 1)\, j_{l-1}, \qquad j_l = j_{l-1}\, s_l, \qquad \text{start}_l = \text{start}_{l-1} + \left( \frac{\hat{k}_l - 1}{2} - p_l \right) j_{l-1}.$$

Unrolling the first recurrence gives the closed form

$$r_L = 1 + \sum_{l=1}^{L} (\hat{k}_l - 1) \prod_{i=1}^{l-1} s_i ,$$

which Araujo, Norris and Sim (2019) derive from the equivalent recurrence run from the output back to the input, together with the field’s center, which equals $\text{start}_L + i\, j_L$ above. With “same” padding, $p_l = (\hat{k}_l - 1)/2$, the start stays at 0, field centers sit at input pixels $0, j_L, 2j_L, \dots$, and border units’ fields extend into the zero padding.

Pooling enters like a convolution with its window and stride; elementwise operations have $k = 1$. For networks with several paths, as in ResNet, Araujo et al. take the largest receptive field over the paths, provided the paths are aligned (same effective stride, coinciding centers).

Properties

Depth grows it linearly. With stride 1 everywhere, $n$ layers of $k \times k$ kernels give $r = n(k-1) + 1$. Simonyan and Zisserman (2015) built VGG on this: three stacked $3 \times 3$ layers cover $7 \times 7$ pixels, like one $7 \times 7$ layer but with fewer weights.

Stride and pooling grow it multiplicatively. Every downsampling, by a strided convolution or pooling, multiplies the jump, and with it the growth of every later layer: VGG-16’s last convolution sees 196 pixels, against 27 for its 13 convolutions without the pooling layers. The cost is resolution, since the jump is also the spacing of the output grid.

Dilation grows it exponentially at full resolution. Yu and Koltun (2016) stacked $3 \times 3$ convolutions with dilations $1, 2, 4, \dots$; the layer with dilation $2^i$ has a receptive field of $(2^{i+2} - 1) \times (2^{i+2} - 1)$, while the parameters grow only linearly.

The effective receptive field is Gaussian-like and grows as $\sqrt{n}$. Backpropagation convolves the gradient map with one kernel per layer, so by a central-limit argument the gradient of a unit with respect to the input is distributed like a Gaussian; Luo et al. (2016) showed this for linear networks with uniform or random weights and, under simplifying assumptions, for ReLU networks. Its standard deviation grows as $O(\sqrt{n})$ while the theoretical field grows as $O(n)$, so the ERF’s share shrinks as $O(1/\sqrt{n})$. In their analysis and experiments, subsampling and dilation enlarged the ERF and skip connections made it smaller; in a CamVid segmentation network whose theoretical field was $505 \times 505$, they report an ERF diameter of about 100 pixels at initialization and about 150 after training.

Examples

The NumPy calculator below applies the recurrences to three $3 \times 3$ convolutions versus one $7 \times 7$; VGG-16 up to its last convolution, plus the $3 \times 3$ sliding window of Faster R-CNN’s region proposal network; the ResNet-18 stem (stride-2 $7 \times 7$ convolution and $3 \times 3$ max pooling) and the four $3 \times 3$ convolutions of its first stage (He et al., 2016), assuming “same” padding, which the paper does not state; and a dilated stack.

import numpy as np

def receptive_fields(layers):
    """layers: list of (name, kernel k, stride s, padding p, dilation d).
    Returns per layer: receptive field r, jump j, and start (input-pixel
    coordinate of the centre of the first output unit's receptive field)."""
    r, j, start = 1, 1, 0.0                # the input image: one pixel, unit spacing
    rows = []
    for name, k, s, p, d in layers:
        k_eff = d * (k - 1) + 1            # a dilated kernel spans d(k-1)+1 pixels
        r = r + (k_eff - 1) * j            # widen by (k_eff-1) steps of the input grid
        start = start + ((k_eff - 1) / 2 - p) * j
        j = j * s                          # output units are s input units apart
        rows.append((name, r, j, start))
    return rows

def show(title, layers):
    print(title)
    for name, r, j, start in receptive_fields(layers):
        print(f"  {name:<10} r = {r:4d}   jump = {j:3d}   start = {start:6.1f}")

conv3 = lambda n, d=1: (n, 3, 1, d, d)         # 3x3, stride 1, "same" padding
show("three 3x3 convolutions", [conv3("conv1"), conv3("conv2"), conv3("conv3")])
show("one 7x7 convolution", [("conv7x7", 7, 1, 3, 1)])

vgg = []                                       # VGG-16 up to conv5_3: 13 convs, 4 pools
for stage, n in enumerate([2, 2, 3, 3, 3], start=1):
    vgg += [conv3(f"conv{stage}_{i}") for i in range(1, n + 1)]
    if stage < 5:
        vgg.append((f"pool{stage}", 2, 2, 0, 1))
vgg.append(("rpn3x3", 3, 1, 1, 1))             # Faster R-CNN's 3x3 sliding window
rows = receptive_fields(vgg)
print("VGG-16:")
for name, r, j, start in rows:
    if name.startswith("pool") or name in ("conv5_3", "rpn3x3"):
        print(f"  {name:<10} r = {r:4d}   jump = {j:3d}   start = {start:6.1f}")

show("ResNet-style stem and two basic blocks", [
    ("conv7x7/2", 7, 2, 3, 1), ("maxpool/2", 3, 2, 1, 1),
    conv3("block1a"), conv3("block1b"), conv3("block2a"), conv3("block2b")])

show("dilated 3x3 stack (Yu & Koltun)", [conv3(f"d={d}", d) for d in (1, 2, 4, 8, 16)])

# Closed form (Araujo et al., 2019): r = 1 + sum_l (k_l - 1) * prod_{i<l} s_i
ks = [k for _, k, s, p, d in vgg]; ss = [s for _, k, s, p, d in vgg]
print("closed form for VGG-16 + RPN:", 1 + sum((k - 1) * int(np.prod(ss[:l])) for l, k in enumerate(ks)))

Output:

three 3x3 convolutions
  conv1      r =    3   jump =   1   start =    0.0
  conv2      r =    5   jump =   1   start =    0.0
  conv3      r =    7   jump =   1   start =    0.0
one 7x7 convolution
  conv7x7    r =    7   jump =   1   start =    0.0
VGG-16:
  pool1      r =    6   jump =   2   start =    0.5
  pool2      r =   16   jump =   4   start =    1.5
  pool3      r =   44   jump =   8   start =    3.5
  pool4      r =  100   jump =  16   start =    7.5
  conv5_3    r =  196   jump =  16   start =    7.5
  rpn3x3     r =  228   jump =  16   start =    7.5
ResNet-style stem and two basic blocks
  conv7x7/2  r =    7   jump =   2   start =    0.0
  maxpool/2  r =   11   jump =   4   start =    0.0
  block1a    r =   19   jump =   4   start =    0.0
  block1b    r =   27   jump =   4   start =    0.0
  block2a    r =   35   jump =   4   start =    0.0
  block2b    r =   43   jump =   4   start =    0.0
dilated 3x3 stack (Yu & Koltun)
  d=1        r =    3   jump =   1   start =    0.0
  d=2        r =    7   jump =   1   start =    0.0
  d=4        r =   15   jump =   1   start =    0.0
  d=8        r =   31   jump =   1   start =    0.0
  d=16       r =   63   jump =   1   start =    0.0
closed form for VGG-16 + RPN: 228

The 228 pixels match the figure Ren et al. (2017) give for their VGG-based proposal network, and the jump of 16 is its total stride. VGG’s unpadded $2 \times 2$ pools put its field centers between pixels. The dilated stack reproduces Yu and Koltun’s sizes 3, 7, 15.

The PyTorch example below (run with PyTorch 2.14.1) checks these numbers on random He-initialized ReLU networks: it feeds a $128 \times 128$ image (a scikit-learn digit enlarged twelve times, plus faint noise), backpropagates from the center unit of one output channel, and records which pixels get a nonzero gradient. For plain $3 \times 3$ stacks of growing depth, the mean gradient magnitude over 20 seeds gives the ERF: its spread $\sigma$ along one axis and the smallest centered square holding 90% of it.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits

torch.set_default_dtype(torch.float64)
digits = load_digits().images / 16.0
SIZE = 128

def digit_image(seed):
    """A digit enlarged 12x (8x8 -> 96x96) on a 128x128 canvas, plus faint noise."""
    rng = np.random.default_rng(seed)
    img = np.zeros((SIZE, SIZE))
    img[16:112, 16:112] = np.kron(digits[seed], np.ones((12, 12)))
    img += 0.05 * rng.standard_normal(img.shape)
    return torch.tensor(img)[None, None]                  # (1, 1, 128, 128)

def build(spec, width=16):
    """spec: list of (kind, k, s, p, d); random He-initialized convs with ReLU."""
    layers, c = [], 1
    for kind, k, s, p, d in spec:
        if kind == "conv":
            conv = nn.Conv2d(c, width, k, stride=s, padding=p, dilation=d)
            nn.init.kaiming_normal_(conv.weight, nonlinearity="relu")
            nn.init.zeros_(conv.bias)
            layers += [conv, nn.ReLU()]
            c = width
        else:
            layers.append(nn.MaxPool2d(k, stride=s, padding=p))
    return nn.Sequential(*layers)

def input_gradient(spec, seed):
    """|d y / d x| for the centre unit y of output channel 0."""
    torch.manual_seed(seed)
    net = build(spec)
    x = digit_image(seed).requires_grad_(True)
    y = net(x)
    i = y.shape[-1] // 2
    y[0, 0, i, i].backward()
    return x.grad[0, 0].abs().numpy(), i

def theoretical(spec):
    r, j, start = 1, 1, 0.0
    for _, k, s, p, d in spec:
        k_eff = d * (k - 1) + 1
        r, start, j = r + (k_eff - 1) * j, start + ((k_eff - 1) / 2 - p) * j, j * s
    return r, j, start

SEEDS = range(20)
c3 = lambda d=1: ("conv", 3, 1, d, d)
stacks = {
    "ResNet-style stem + 4 convs": [("conv", 7, 2, 3, 1), ("pool", 3, 2, 1, 1)] + [c3()] * 4,
    "dilated 3x3, d = 1,2,4,8,16":  [c3(d) for d in (1, 2, 4, 8, 16)],
}
print("Theoretical receptive field: predicted vs nonzero-gradient region (union over 20 seeds)")
for name, spec in stacks.items():
    r, j, start = theoretical(spec)
    grads = [input_gradient(spec, s) for s in SEEDS]
    i = grads[0][1]
    support = np.any([g > 0 for g, _ in grads], axis=0)
    rows, cols = np.nonzero(support)
    centre = start + i * j
    print(f"  {name}: predicted r = {r}, rows {centre - (r-1)/2:.0f}..{centre + (r-1)/2:.0f};"
          f" measured {rows.max() - rows.min() + 1} x {cols.max() - cols.min() + 1},"
          f" rows {rows.min()}..{rows.max()}")
    one = np.mean([np.count_nonzero(g) for g, _ in grads]) / r**2
    print(f"    a single seed reaches {one:.0%} of the r x r square on average")

print("Effective receptive field of plain 3x3 stacks (ReLU, mean |gradient| over 20 seeds)")
ns, sigmas = [4, 8, 16, 32], []
for n in ns:
    spec = [c3()] * n
    r = theoretical(spec)[0]
    g = np.mean([input_gradient(spec, s)[0] for s in SEEDS], axis=0)
    w = g / g.sum()                                     # gradient magnitude as a distribution
    coords = np.arange(SIZE)
    py, px = w.sum(1), w.sum(0)                         # marginals
    sigma = np.sqrt(((coords - (px * coords).sum())**2 * px).sum())
    # side of the smallest centred square holding 90% of the gradient mass
    c = SIZE // 2
    side = next(2*h + 1 for h in range(SIZE) if w[c-h:c+h+1, c-h:c+h+1].sum() >= 0.9)
    sigmas.append(sigma)
    print(f"  {n:2d} layers: theoretical r = {r:2d}   ERF sigma = {sigma:4.2f}"
          f"   90% of |grad| within {side:2d} x {side:<2d} ({side**2 / r**2:.0%} of the r x r area)")
slope = np.polyfit(np.log(ns), np.log(sigmas), 1)[0]
print(f"  log-log slope of sigma vs depth: {slope:.2f} (sqrt(n) growth gives 0.5)")

Output:

Theoretical receptive field: predicted vs nonzero-gradient region (union over 20 seeds)
  ResNet-style stem + 4 convs: predicted r = 43, rows 43..85; measured 43 x 43, rows 43..85
    a single seed reaches 59% of the r x r square on average
  dilated 3x3, d = 1,2,4,8,16: predicted r = 63, rows 33..95; measured 63 x 63, rows 33..95
    a single seed reaches 50% of the r x r square on average
Effective receptive field of plain 3x3 stacks (ReLU, mean |gradient| over 20 seeds)
   4 layers: theoretical r =  9   ERF sigma = 2.02   90% of |grad| within  9 x 9  (100% of the r x r area)
   8 layers: theoretical r = 17   ERF sigma = 3.02   90% of |grad| within 13 x 13 (58% of the r x r area)
  16 layers: theoretical r = 33   ERF sigma = 4.47   90% of |grad| within 19 x 19 (33% of the r x r area)
  32 layers: theoretical r = 65   ERF sigma = 6.44   90% of |grad| within 25 x 25 (15% of the r x r area)
  log-log slope of sigma vs depth: 0.56 (sqrt(n) growth gives 0.5)

The nonzero gradients fill exactly the predicted square, in the predicted place, for the strided stem with max pooling and for the dilated stack. A single network reaches only 50–60% of it, because inactive ReLUs and max-pooling windows whose maximum lies elsewhere block paths; the theoretical field bounds what can matter.

From 4 to 32 layers the theoretical field grows from 9 to 65 pixels, but $\sigma$ only from 2.0 to 6.4, and the 90% square shrinks from the whole theoretical field to 15% of its area. The log-log slope of 0.56 is close to the $\sqrt{n}$ law (other sets of 20 seeds give 0.54–0.56). These networks are untrained; training enlarges the ERF (Luo et al., 2016).

Common Misconceptions

• “Effective receptive field” always means the gradient-weighted field. Some papers use it for the theoretical size: Simonyan and Zisserman (2015) write that three $3 \times 3$ layers have a $7 \times 7$ effective receptive field, and Ren et al. (2017) so describe the 228 pixels above.
• “Padding enlarges the receptive field.” It shifts the field, so border units see zero padding, but it does not change $r$.
• “A field covering the image means the unit uses the whole image.” Araujo et al. note that the receptive fields of recent classification networks usually cover the entire input. Given the small effective field, this does not mean that every pixel contributes much.
• “Normalization layers have no receptive field.” Araujo et al. point out that batch normalization is elementwise at inference time but, during training, uses statistics over the whole layer, so its receptive field is the whole image.

Where It Is Used

Dense prediction. Luo et al. (2016) single out semantic segmentation, stereo and optical flow as tasks in which every output pixel needs a large receptive field. In stereo matching, a pixel on a textureless wall can be matched only by seeing past the wall; in our reading, the 3D convolutions that regularize a cost volume serve this purpose, as aggregation windows did in classical stereo. In optical flow estimation, RAFT’s update operator uses $1 \times 5$ and $5 \times 1$ convolutions to enlarge its receptive field cheaply. Yu and Koltun (2016) built their dilated context module because dense prediction needs multi-scale context together with full-resolution output, which pooling trades against each other.

Object detection. A detector’s feature-map stride (the jump) sets the spacing of its predictions, and the receptive field limits the context of each. Faster R-CNN places anchors (reference boxes) at every position of a stride-16 feature map; its authors note that boxes larger than the receptive field can still be predicted, since an object’s extent can be roughly inferred from its visible middle.

Global receptive fields. Every output of self-attention is a weighted average over all tokens, so one layer already has a receptive field covering the image. Dosovitskiy et al. (2021) measured a Vision Transformer’s mean “attention distance”, which they call analogous to receptive field size: some heads attend to most of the image already in the lowest layers, others stay local, and the distance increases with depth. In feature matching, LoFTR’s authors credit the transformer’s global receptive field for matches in low-texture regions.