Pooling

Pooling replaces each small window of a feature map with one number, usually its maximum or its average. In a convolutional neural network it is the classic way to downsample between stages, and it has no learned parameters. Its variants include global average pooling, the usual classification head, and the region pooling of object detectors.

Definition

A pooling layer slides a $k \times k$ window over each channel with stride $s$ and outputs the window’s maximum (max pooling) or mean (average pooling). Channels are pooled separately, so their number is unchanged. With $k = s$ the windows tile the map without overlap; with $k > s$ they overlap. Global pooling uses one window that covers the whole map, giving one number per channel.

Intuition

A channel of a convolutional layer maps how strongly one pattern, such as a vertical edge, is present at each position. Max pooling keeps the strongest response in each neighborhood and drops where in the window it was: an edge is somewhere in this $2 \times 2$ block. Average pooling reports how much of the pattern the block contains. A slight shift of the input often leaves a max-pooled value unchanged, because the maximum stays inside the same window.

Max pooling with a flat $k \times k$ window is a grayscale dilation from mathematical morphology followed by subsampling. Average pooling is a box filter followed by subsampling, a crude step of an image pyramid.

Formal Definition

For one channel $x$ and output position $(i, j)$:

$$y_{\max}[i, j] = \max_{0 \le u, v < k} x[s\,i + u,\; s\,j + v], \qquad y_{\text{avg}}[i, j] = \frac{1}{k^2} \sum_{u=0}^{k-1} \sum_{v=0}^{k-1} x[s\,i + u,\; s\,j + v].$$

For an input of size $n$ with padding $p$ the output size is the same as for a convolution,

$$o = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1.$$

PyTorch pads max pooling with $-\infty$, so padding never wins the maximum. Its ceil_mode=True uses the ceiling instead of the floor, keeping a last window that starts inside the input but runs past its edge (PyTorch documentation); the example below shows it.

Global average pooling of channel $c$ over an $H \times W$ map is $z_c = \frac{1}{HW} \sum_{a, b} x_c[a, b]$.

Backward pass. In backpropagation, max pooling passes each output’s gradient $\partial L / \partial y[i, j]$ to the input that was its window’s maximum; the others get nothing from that window. Average pooling gives each input in the window $\frac{1}{k^2} \partial L / \partial y[i, j]$. With overlapping windows an input sums the contributions of every window it belongs to, as Girshick (2015) writes out for RoI pooling.

Properties

What pooling buys:

• Smaller maps. A $2 \times 2$, stride-2 pooling cuts the positions, and the cost of the layers that follow, by four. VGG uses five such max-pooling layers (Simonyan & Zisserman, 2015).
• A larger receptive field. After each halving, a $3 \times 3$ convolution covers an input region twice as wide.
• Tolerance of small shifts. LeCun et al. (1998) used subsampling layers to encode feature positions less precisely and so reduce the output’s sensitivity to shifts and distortions. Each LeNet-5 subsampling unit averaged a non-overlapping $2 \times 2$ window, then applied a trainable coefficient and bias and a sigmoid.

What it costs:

• Resolution. Positions within each window are lost, which hurts precise localization unless skip connections or upsampling restore detail.
• Aliasing. Pooling with stride $s$ subsamples without an adequate low-pass prefilter, so detail finer than the new grid becomes false coarse patterns, and outputs jump when the input moves by one pixel (see sampling and aliasing). Zhang (2019) inserted a low-pass filter between the dense, stride-1 max and the subsampling (MaxBlurPool). He noted that blurred downsampling with a box filter is average pooling, and that a stronger filter gives better shift-equivariance. With blur filters of length 2, 3 and 5 (the last the binomial $[1, 4, 6, 4, 1]$ of Laplacian pyramids), he reported that ResNet-50 became more consistent under shifts and gained 0.7 to 0.9 points of ImageNet classification accuracy.

Examples

Overlapping pooling. AlexNet used $3 \times 3$ max pooling with stride 2 throughout. Krizhevsky et al. (2012) reported 0.4 and 0.3 points lower top-1 and top-5 error than with $2 \times 2$, stride-2 pooling, which gives outputs of the same size, and found such models slightly harder to overfit. ResNet’s stem uses the same pooling (He et al., 2016).

Global average pooling. Lin, Chen and Yan (2014) averaged each final feature map and fed the vector to the softmax in place of fully connected layers: a fixed, block-diagonal linear map with no parameters to overfit. On CIFAR-10 their network had 10.41% test error with it, against 11.59% with a fully connected layer and 10.88% with one plus dropout.

Strided convolution instead of pooling. Springenberg et al. (2015) found that max pooling can be replaced by a stride-2 convolution without loss in accuracy on several benchmarks. On CIFAR-10 without augmentation, their all-convolutional model C had 9.08% error against 9.74% for base model C, which max-pools; merely removing the pooling and striding the previous convolution gave 10.19%. They caution that one learning-rate schedule served all models. ResNet also downsamples with stride-2 convolutions (He et al., 2016); Zhang (2019) lists strided convolution with max and average pooling as breaking shift-equivariance.

Spatial pyramid pooling (SPP). SPP-net (He et al.; ECCV 2014, details here from the 2015 journal version) replaces the last pooling layer with max pooling over bins at several levels, such as $6 \times 6$, $3 \times 3$, $2 \times 2$ and $1 \times 1$ (50 bins). Because the bins scale with the map, any input size yields a vector of the same length. For detection, SPP-net computes the feature maps once per image and pools about 2,000 candidate windows from them; the authors reported 24 to 102 times faster testing than R-CNN with comparable or better accuracy on PASCAL VOC 2007.

RoI pooling. Fast R-CNN’s region-of-interest (RoI) pooling divides each $h \times w$ region into an $H \times W$ grid of cells of about $h/H \times w/W$ and max-pools each cell, to $7 \times 7$ for VGG16; Girshick (2015) describes it as one-level SPP.

The NumPy code below pools a scikit-learn $8 \times 8$ digit and its Sobel edge map, checks both backward passes by finite differences, measures a one-pixel shift over all 1,797 digits, and pools thin vertical lines with and without Zhang’s blur.

import numpy as np
from scipy import ndimage
from sklearn.datasets import load_digits

def pool2d(x, k=2, s=2, mode="max"):
    """Max or average pooling of a 2-D map with a k x k window and stride s (no padding)."""
    H, W = x.shape
    y = np.zeros(((H - k) // s + 1, (W - k) // s + 1))
    for i in range(y.shape[0]):
        for j in range(y.shape[1]):
            win = x[i*s:i*s+k, j*s:j*s+k]
            y[i, j] = win.max() if mode == "max" else win.mean()
    return y

def pool2d_backward(x, g, k=2, s=2, mode="max"):
    """Gradient of the loss w.r.t. x, given g = gradient w.r.t. the pooled output."""
    dx = np.zeros_like(x)
    for i in range(g.shape[0]):
        for j in range(g.shape[1]):
            win = x[i*s:i*s+k, j*s:j*s+k]
            if mode == "max":                          # route g to the window's argmax
                u, v = np.unravel_index(win.argmax(), win.shape)
                dx[i*s + u, j*s + v] += g[i, j]
            else:                                      # spread g evenly over the window
                dx[i*s:i*s+k, j*s:j*s+k] += g[i, j] / k**2
    return dx

digits = load_digits().images / 16.0                   # 1,797 digits, 8x8, values in [0, 1]
digit = digits[0]                                      # a "0"
print("digit (x 16):\n", np.round(16 * digit).astype(int))
print("2x2 max pool, stride 2 (x 16):\n", np.round(16 * pool2d(digit)).astype(int))
print("2x2 average pool, stride 2 (x 16):\n", np.round(16 * pool2d(digit, mode="avg")).astype(int))
print("3x3 max pool, stride 2 (overlapping): output", pool2d(digit, 3, 2).shape)

# Edge map: Sobel gradient magnitude of the digit on a 12x12 canvas.
def canvas(d):
    c = np.zeros((12, 12)); c[2:10, 2:10] = d
    return c
def edges(c):
    return np.hypot(ndimage.sobel(c, axis=1), ndimage.sobel(c, axis=0))
e = edges(canvas(digit))
print("edge map 12x12 -> 2x2 max pool 6x6 (x 10):\n", np.round(10 * pool2d(e)).astype(int))

# Backward pass on a 4x4 corner of the digit, upstream gradient [[1, 2], [3, 4]].
patch, g = digit[:4, :4], np.array([[1.0, 2.0], [3.0, 4.0]])
print("patch (x 16):\n", np.round(16 * patch).astype(int))
print("max pool backward:\n", pool2d_backward(patch, g))
print("average pool backward:\n", pool2d_backward(patch, g, mode="avg"))

# Finite-difference check on the edge map (loss = sum(G * pool(x)), random G). The max is not
# differentiable where a window has tied maxima (e.g. all zeros), so tiny noise breaks the ties.
rng = np.random.default_rng(0)
e_check = e + 1e-3 * rng.random(e.shape)
for k, s, mode in [(2, 2, "max"), (3, 2, "max"), (2, 2, "avg")]:
    G = rng.standard_normal(pool2d(e_check, k, s).shape)
    num = np.zeros_like(e_check)
    for idx in np.ndindex(e_check.shape):
        d = np.zeros_like(e_check); d[idx] = 1e-6
        num[idx] = (np.sum(G * pool2d(e_check + d, k, s, mode))
                    - np.sum(G * pool2d(e_check - d, k, s, mode))) / 2e-6
    print(f"{mode} k={k} s={s}: max |analytic - finite difference| =",
          f"{np.abs(pool2d_backward(e_check, G, k, s, mode) - num).max():.1e}")

# Translation tolerance: shift every digit one pixel right; relative change before and after pooling.
def rel_change(a, b):
    return np.linalg.norm(b - a) / np.linalg.norm(a)
for name, prep in [("digit", lambda c: c), ("edge map", edges)]:
    r_in, r_max, r_avg = [], [], []
    for d in digits:
        x, xs = prep(canvas(d)), prep(np.roll(canvas(d), 1, axis=1))   # canvas border is zero
        r_in.append(rel_change(x, xs))
        r_max.append(rel_change(pool2d(x), pool2d(xs)))
        r_avg.append(rel_change(pool2d(x, mode="avg"), pool2d(xs, mode="avg")))
    print(f"{name}: median relative change, input {np.median(r_in):.2f}, max-pooled {np.median(r_max):.2f},"
          f" avg-pooled {np.median(r_avg):.2f}; max-pooled changes less in"
          f" {np.mean(np.array(r_max) < np.array(r_in)):.1%} of digits")

# Aliasing: 1-pixel-wide vertical lines every `period` pixels, pooled with stride 2.
def max_blur_pool(x, taps=(1, 4, 6, 4, 1)):
    """Zhang (2019): dense 2x2 max (stride 1), then a binomial blur, then subsample by 2."""
    dense = ndimage.maximum_filter(x, size=2, origin=-1, mode="wrap")   # max over x[i:i+2, j:j+2]
    b = np.outer(taps, taps).astype(float); b /= b.sum()
    return ndimage.convolve(dense, b, mode="wrap")[::2, ::2]
for period in (3, 5):
    lines = np.zeros((30, 30)); lines[:, ::period] = 1.0
    print(f"lines every {period} px, one input row: {lines[0, :12].astype(int)} ...")
    for name, f in [("max pool", pool2d), ("average pool", lambda x: pool2d(x, mode="avg")),
                    ("max-blur pool", max_blur_pool)]:
        y, y_shift = f(lines), f(np.roll(lines, 1, axis=1))
        print(f" {name:13s} row: {np.round(y[0, :9], 2)} ...  ripple {np.ptp(y[0]):.2f},"
              f" change under a 1-px shift {np.abs(y_shift - y).max():.2f}")

Output:

digit (x 16):
 [[ 0  0  5 13  9  1  0  0]
 [ 0  0 13 15 10 15  5  0]
 [ 0  3 15  2  0 11  8  0]
 [ 0  4 12  0  0  8  8  0]
 [ 0  5  8  0  0  9  8  0]
 [ 0  4 11  0  1 12  7  0]
 [ 0  2 14  5 10 12  0  0]
 [ 0  0  6 13 10  0  0  0]]
2x2 max pool, stride 2 (x 16):
 [[ 0 15 15  5]
 [ 4 15 11  8]
 [ 5 11 12  8]
 [ 2 14 12  0]]
2x2 average pool, stride 2 (x 16):
 [[ 0 12  9  1]
 [ 2  7  5  4]
 [ 2  5  6  4]
 [ 0 10  8  0]]
3x3 max pool, stride 2 (overlapping): output (3, 3)
edge map 12x12 -> 2x2 max pool 6x6 (x 10):
 [[ 0  4 25 21  1  0]
 [ 0 30 36 35 31  0]
 [ 0 35 38 31 28  0]
 [ 0 28 26 31 29  0]
 [ 0 29 28 28 28  0]
 [ 0  5 26 22  0  0]]
patch (x 16):
 [[ 0  0  5 13]
 [ 0  0 13 15]
 [ 0  3 15  2]
 [ 0  4 12  0]]
max pool backward:
 [[1. 0. 0. 0.]
 [0. 0. 0. 2.]
 [0. 0. 4. 0.]
 [0. 3. 0. 0.]]
average pool backward:
 [[0.25 0.25 0.5  0.5 ]
 [0.25 0.25 0.5  0.5 ]
 [0.75 0.75 1.   1.  ]
 [0.75 0.75 1.   1.  ]]
max k=2 s=2: max |analytic - finite difference| = 4.0e-10
max k=3 s=2: max |analytic - finite difference| = 2.0e-09
avg k=2 s=2: max |analytic - finite difference| = 4.5e-10
digit: median relative change, input 0.82, max-pooled 0.54, avg-pooled 0.58; max-pooled changes less in 99.3% of digits
edge map: median relative change, input 0.68, max-pooled 0.46, avg-pooled 0.46; max-pooled changes less in 100.0% of digits
lines every 3 px, one input row: [1 0 0 1 0 0 1 0 0 1 0 0] ...
 max pool      row: [1. 1. 0. 1. 1. 0. 1. 1. 0.] ...  ripple 1.00, change under a 1-px shift 1.00
 average pool  row: [0.5 0.5 0.  0.5 0.5 0.  0.5 0.5 0. ] ...  ripple 0.50, change under a 1-px shift 0.50
 max-blur pool row: [0.69 0.69 0.62 0.69 0.69 0.62 0.69 0.69 0.62] ...  ripple 0.06, change under a 1-px shift 0.06
lines every 5 px, one input row: [1 0 0 0 0 1 0 0 0 0 1 0] ...
 max pool      row: [1. 0. 1. 0. 0. 1. 0. 1. 0.] ...  ripple 1.00, change under a 1-px shift 1.00
 average pool  row: [0.5 0.  0.5 0.  0.  0.5 0.  0.5 0. ] ...  ripple 0.50, change under a 1-px shift 0.50
 max-blur pool row: [0.62 0.12 0.62 0.31 0.31 0.62 0.12 0.62 0.31] ...  ripple 0.50, change under a 1-px shift 0.31

Max pooling keeps the stroke of the “0” at full strength wherever it touches a window, and the edge map’s ring of strong edges at half the resolution; average pooling dims the stroke. At $4 \times 4$ both lose the hole of the “0”.

Each max-pooling gradient lands on one pixel: 2 on the 15 in the top-right window, 4 on the 15 in the bottom-right, 3 on the 4 in the bottom-left. The all-zero top-left window is a tie, and its gradient goes to the first element. With ties broken by tiny noise, both poolings agree with finite differences, also for overlapping windows.

A one-pixel shift changes a digit by a median 82% of its norm but its max-pooled map by 54% (edge maps: 68% and 46%), and the pooled map changes less for 99.3% of digits. That is tolerance, not invariance: the pooled maps still change by about half. Average pooling behaves much the same.

Lines every 3 pixels repeat faster than a stride-2 grid can represent (at most one cycle per 4 input pixels). Max pooling turns them into a false pattern repeating every 6 input pixels, which a one-pixel shift rearranges, flipping outputs between 0 and 1; average pooling halves it. The max-blur pool leaves a ripple of 0.06, a nearly uniform gray. Lines every 5 pixels fit the new grid, and the blur keeps them (ripple 0.50); their output changes under the shift because the pattern really moves by half an output pixel.

The second part, run with PyTorch 2.14.1 after the code above, checks the NumPy functions against torch.nn.functional and shows ceil mode, global, adaptive and pyramid pooling.

import math
import torch
import torch.nn.functional as F

torch.manual_seed(0)
print("torch", torch.__version__)
x = torch.tensor(e, dtype=torch.float64)[None, None].requires_grad_()   # edge map, shape (1, 1, 12, 12)
for k, s, mode in [(2, 2, "max"), (3, 2, "max"), (2, 2, "avg")]:
    y = F.max_pool2d(x, k, s) if mode == "max" else F.avg_pool2d(x, k, s)
    G = torch.randn_like(y)
    (grad,) = torch.autograd.grad(y, x, G)
    print(f"{mode} k={k} s={s}: forward matches {torch.allclose(y[0, 0], torch.tensor(pool2d(e, k, s, mode)))},"
          f" gradient matches {torch.allclose(grad[0, 0], torch.tensor(pool2d_backward(e, G[0, 0].numpy(), k, s, mode)))}")

d = torch.tensor(digit)[None, None]                                       # the 8x8 digit
print("8x8, k=3, s=2: floor mode", tuple(F.max_pool2d(d, 3, 2).shape[2:]),
      "| ceil mode", tuple(F.max_pool2d(d, 3, 2, ceil_mode=True).shape[2:]))

# Global average pooling = adaptive average pooling to 1x1 = the mean of each map.
print("global average pooling equals the mean:", torch.allclose(F.adaptive_avg_pool2d(x, 1).flatten(), x.mean()))

# Adaptive pooling to n x n bins: bin i covers rows floor(i*H/n) .. ceil((i+1)*H/n) - 1.
def adaptive_avg(a, n):
    H, W = a.shape
    return [[a[math.floor(i*H/n):math.ceil((i+1)*H/n), math.floor(j*W/n):math.ceil((j+1)*W/n)].mean()
             for j in range(n)] for i in range(n)]
print("8x8 -> 3x3 adaptive average matches the bin formula:",
      torch.allclose(F.adaptive_avg_pool2d(d, 3)[0, 0], torch.tensor(adaptive_avg(digit, 3))))

# Spatial pyramid pooling: max over 1x1, 2x2 and 4x4 bins gives 21 numbers per channel for any map size.
def spp(fmap, levels=(1, 2, 4)):
    return torch.cat([F.adaptive_max_pool2d(fmap, n).flatten(1) for n in levels], dim=1)
for H, W in [(12, 12), (9, 15), (30, 20)]:
    print(f"feature map {H}x{W} with 16 channels -> SPP vector of length", spp(torch.rand(1, 16, H, W)).shape[1])

Output:

torch 2.14.1
max k=2 s=2: forward matches True, gradient matches True
max k=3 s=2: forward matches True, gradient matches True
avg k=2 s=2: forward matches True, gradient matches True
8x8, k=3, s=2: floor mode (3, 3) | ceil mode (4, 4)
global average pooling equals the mean: True
8x8 -> 3x3 adaptive average matches the bin formula: True
feature map 12x12 with 16 channels -> SPP vector of length 336
feature map 9x15 with 16 channels -> SPP vector of length 336
feature map 30x20 with 16 channels -> SPP vector of length 336

PyTorch’s gradients match the NumPy routing even on the edge map’s tied all-zero windows, so both evidently pick the first maximum. Adaptive pooling, PyTorch’s form of SPP bins, overlaps bins when the map size is not a multiple of the bin count; the bin formula is the one in PyTorch’s C++ source (AdaptivePooling.h), and the comparison confirms it.

Common Misconceptions

• “Pooling makes a CNN translation invariant.” Its subsampling breaks exact equivariance, so a one-pixel shift can change the output; only global pooling discards position entirely.
• “Average pooling is anti-aliased.” A box filter is a weak low-pass filter; in the example half the false pattern survives.
• “Max pooling blocks the gradient.” Only for inputs that were not the maximum; each output passes its full gradient to one input.

Where It Is Used

Max pooling between stages shaped classification backbones such as AlexNet and VGG; global average pooling followed by one linear layer heads GoogLeNet and ResNet (Szegedy et al., 2015; He et al., 2016); SPP and RoI pooling feed detection heads. Anti-aliased pooling brings back the Gaussian or binomial prefilter of classical sampling.