Convolutional Neural Network

A convolutional neural network (CNN, ConvNet) is a neural network whose main layers are convolutions: each layer slides a bank of small learned filters over its input and records every filter’s response at every position. Stacked with nonlinearities and downsampling, they turn an image into maps of edges, then textures and parts, and finally class scores or per-pixel predictions. CNNs have long been the standard architecture for classification, detection, segmentation, and dense matching, and remain the main alternative to vision transformers.

Core Idea

A fully connected layer links every unit to every pixel (about 150,000 weights per unit for a $224 \times 224$ color image), and nothing in it says that neighboring pixels belong together or that an edge means the same thing wherever it appears. A convolutional layer builds both facts in:

• Local connectivity. Each output unit sees only a small window of the input, typically $3 \times 3$.
• Weight sharing. All units in one output map use the same filter, so a pattern detector learned at one position works at every position.
• Translation equivariance. Because the same filter is applied everywhere, shifting the input shifts the output map by the same amount (exactly for stride 1, up to borders).

LeCun et al. (1998) name local receptive fields, shared weights, and spatial subsampling as the three ideas that give a convolutional network some degree of shift, scale, and distortion invariance. In the vocabulary of image filtering, a convolutional layer is a learned filter bank.

Building Blocks

The convolutional layer

The input is a stack of $C_{\text{in}}$ channels (three for a color image, or the previous layer’s maps). Each of the layer’s $C_{\text{out}}$ filters has size $C_{\text{in}} \times k \times k$: it spans all input channels, sums the per-channel responses, and adds a bias, producing one output channel. Padding adds $p$ rows and columns of zeros around the input; stride $s$ evaluates the filters only at every $s$-th position.

As the convolution article explains, deep-learning libraries compute a cross-correlation (no kernel flip), as PyTorch’s documentation of Conv2d states. For learned filters this is immaterial, but a designed kernel such as a Sobel template must be entered as the template, not the flipped kernel.

Nonlinearity and normalization

Each convolution is followed by an activation function, usually ReLU; without it, a stack of convolutions collapses into one convolution. Many CNNs since 2015 also insert batch normalization between convolution and activation, as ResNet does after every convolution (He et al., 2016).

Downsampling

Stages are separated by downsampling, either pooling (typically a max or average over $2 \times 2$ windows with stride 2, covered in the pooling article) or a strided convolution. ResNet doubles the number of filters whenever the feature map size is halved, to preserve the computation per layer (He et al., 2016).

1×1 convolutions

A $1 \times 1$ convolution mixes channels at each pixel without looking at neighbors, like a fully connected layer applied at every position. Lin, Chen and Yan (2014) built Network in Network from small per-pixel multilayer perceptrons, which they note are equivalent to $1 \times 1$ convolutions. GoogLeNet used $1 \times 1$ convolutions mainly to reduce channels before expensive $3 \times 3$ and $5 \times 5$ convolutions (Szegedy et al., 2015).

The head

Early CNNs such as AlexNet fed the flattened last feature maps to fully connected layers. Lin et al. (2014) proposed global average pooling instead: average each final channel over all positions and classify the resulting vector. It has no parameters, and the authors argue that it acts as a structural regularizer and is more robust to translations. GoogLeNet reported about 0.6% higher top-1 accuracy from the switch, and ResNet also uses it.

How Data Flows

A typical classification CNN is a stem, stages of conv → BN → ReLU blocks separated by downsampling, and a head, with resolution falling and channels rising from stage to stage: ResNet-34’s first stage works on $56 \times 56$ maps with 64 channels, its last on $7 \times 7$ maps with 512 (He et al., 2016).

The NumPy code below implements a convolutional layer’s forward pass and applies it to one of scikit-learn’s $8 \times 8$ digit images on a $12 \times 12$ canvas, with two hand-set Sobel templates as filters, so the output channels are edge maps. It then checks output sizes and parameter counts and tests translation equivariance.

import numpy as np
from scipy import signal
from sklearn.datasets import load_digits

def conv2d(x, W, b, stride=1, pad=0):
    """Convolutional layer forward pass (cross-correlation, as in deep learning).
    x: (C_in, H, W) input; W: (C_out, C_in, k, k) filters; b: (C_out,) biases."""
    C_out, C_in, k, _ = W.shape
    x = np.pad(x, ((0, 0), (pad, pad), (pad, pad)))          # zero padding
    H_out = (x.shape[1] - k) // stride + 1
    W_out = (x.shape[2] - k) // stride + 1
    y = np.zeros((C_out, H_out, W_out))
    for i in range(H_out):
        for j in range(W_out):
            patch = x[:, i*stride:i*stride+k, j*stride:j*stride+k]   # (C_in, k, k)
            y[:, i, j] = np.tensordot(W, patch, axes=3) + b          # one dot product per filter
    return y

digit = load_digits().images[0] / 16.0        # an 8x8 "0", values in [0, 1]
img = np.zeros((1, 12, 12))
img[0, 2:10, 2:10] = digit                    # place it on a 12x12 canvas (C_in = 1)

# Two hand-set 3x3 filters: Sobel templates for d/dx and d/dy (C_out = 2, C_in = 1).
sx = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], float)
W = np.stack([sx, sx.T])[:, None]             # shape (2, 1, 3, 3)
b = np.zeros(2)

y = conv2d(img, W, b, stride=1, pad=1)
print("input", img.shape, "-> output", y.shape, "| parameters:", W.size + b.size)
print("matches scipy correlate2d:",
      all(np.allclose(y[c], signal.correlate2d(img[0], W[c, 0], mode="same")) for c in range(2)))
print("digit (x 16):")
print(np.round(16 * digit).astype(int))
print("horizontal-derivative channel over the digit (x 10):")
print(np.round(10 * y[0, 2:10, 2:10]).astype(int))

# Output size o = floor((i + 2p - k) / s) + 1
for k, s, p in [(3, 1, 1), (3, 1, 0), (3, 2, 1), (5, 2, 2)]:
    o = conv2d(img, np.zeros((2, 1, k, k)), b, stride=s, pad=p).shape[1]
    print(f"k={k} s={s} p={p}: output {o}x{o}, formula {(12 + 2*p - k)//s + 1}")

# Parameter count of a layer with 64 filters of size 3x3 over 64 channels, vs a fully connected map.
C_in, C_out, k, H = 64, 64, 3, 56
print(f"conv 64->64, 3x3: {C_out*C_in*k*k + C_out:,} parameters;"
      f" fully connected on the same 56x56x64 maps: {(H*H*C_in)**2 + H*H*C_in:,}")

# Translation equivariance: shift the digit by (1, 2) pixels.
shifted = np.roll(img, (1, 2), axis=(1, 2))                       # canvas border is zero, nothing wraps
y_shift = conv2d(shifted, W, b, stride=1, pad=1)
print("stride 1: conv(shift(x)) == shift(conv(x)):",
      np.allclose(y_shift, np.roll(y, (1, 2), axis=(1, 2))))
# With stride 2 the output grid is subsampled: a 1-pixel input shift has no exact counterpart.
y2 = conv2d(img, W, b, stride=2, pad=1)
y2_shift = conv2d(np.roll(img, (0, 1), axis=(1, 2)), W, b, stride=2, pad=1)
y2_shift2 = conv2d(np.roll(img, (0, 2), axis=(1, 2)), W, b, stride=2, pad=1)
print("stride 2, input shifted 2 pixels: output shifted 1:",
      np.allclose(y2_shift2, np.roll(y2, (0, 1), axis=(1, 2))))
best = min(np.abs(y2_shift - np.roll(y2, (0, d), axis=(1, 2))).max() for d in (0, 1))
print(f"stride 2, input shifted 1 pixel: best-matching output shift still differs by up to {best:.2f}"
      f" (largest |output| {np.abs(y2).max():.2f})")

Output:

input (1, 12, 12) -> output (2, 12, 12) | parameters: 20
matches scipy correlate2d: True
digit (x 16):
[[ 0  0  5 13  9  1  0  0]
 [ 0  0 13 15 10 15  5  0]
 [ 0  3 15  2  0 11  8  0]
 [ 0  4 12  0  0  8  8  0]
 [ 0  5  8  0  0  9  8  0]
 [ 0  4 11  0  1 12  7  0]
 [ 0  2 14  5 10 12  0  0]
 [ 0  0  6 13 10  0  0  0]]
horizontal-derivative channel over the digit (x 10):
[[  0  14  26   3 -15 -14 -11  -3]
 [  2  29  26 -11  -2  -7 -26 -11]
 [  6  34   6 -28  16  12 -28 -18]
 [ 10  29  -9 -29  21  20 -22 -20]
 [ 11  24 -11 -24  24  19 -24 -19]
 [  9  28  -6 -20  25   6 -28 -14]
 [  5  28   9  -9   8 -15 -22  -4]
 [  1  16  18   2 -12 -19  -8   0]]
k=3 s=1 p=1: output 12x12, formula 12
k=3 s=1 p=0: output 10x10, formula 10
k=3 s=2 p=1: output 6x6, formula 6
k=5 s=2 p=2: output 6x6, formula 6
conv 64->64, 3x3: 36,928 parameters; fully connected on the same 56x56x64 maps: 40,282,296,320
stride 1: conv(shift(x)) == shift(conv(x)): True
stride 2, input shifted 2 pixels: output shifted 1: True
stride 2, input shifted 1 pixel: best-matching output shift still differs by up to 3.38 (largest |output| 3.12)

The horizontal-derivative channel is positive where the stroke of the “0” begins (dark to bright, left to right) and negative where it ends. The layer has 20 parameters (two $3 \times 3$ filters and two biases) whatever the image size; a trained first layer learns filters of this kind on its own. With stride 1, shifting the digit shifts both edge maps exactly. With stride 2, a two-pixel shift moves the output by one, but a one-pixel shift has no counterpart on the coarser grid (the best match differs by more than the largest response).

Mathematical Formulation

For input $x$ with channels $c = 1, \dots, C_{\text{in}}$, filters $w$ and biases $b$, output channel $o$ at position $(i, j)$ is

$$y_{o}[i, j] = b_o + \sum_{c=1}^{C_{\text{in}}} \sum_{u=0}^{k-1} \sum_{v=0}^{k-1} w_{o,c}[u, v]\; x_c[s\,i + u,\; s\,j + v],$$

with $x$ already zero-padded by $p$ on each side. For an input of size $i$ the output size is (Dumoulin & Visin, 2016)

$$o = \left\lfloor \frac{i + 2p - k}{s} \right\rfloor + 1,$$

so $p = (k-1)/2$ with $s = 1$ preserves the size for odd $k$, and $s = 2$ halves it. The layer has

$$C_{\text{out}} \cdot C_{\text{in}} \cdot k^2 + C_{\text{out}}$$

parameters, independent of the image size. A fully connected layer mapping the same $H \times W \times C_{\text{in}}$ input to an $H \times W \times C_{\text{out}}$ output would need $(HWC_{\text{in}})(HWC_{\text{out}})$ weights: about 40.3 billion for $56 \times 56 \times 64$ maps, against 36,928 for a $3 \times 3$ convolution. A convolutional layer is in fact a fully connected layer with most weights forced to zero (locality) and the rest tied (sharing).

Equivariance. Writing $T_{\mathbf{d}}$ for a shift by $\mathbf{d}$ pixels, a stride-1 layer $f$ satisfies $f(T_{\mathbf{d}} x) = T_{\mathbf{d}} f(x)$ away from the borders, and elementwise activations preserve this; with stride $s$, only shifts by multiples of $s$ commute with the layer. Invariance, $g(T_{\mathbf{d}} x) = g(x)$, is a different property, approached only at the end by pooling over all positions.

Receptive field. Stacking layers widens the input region that affects a unit: three $3 \times 3$ layers see $7 \times 7$ pixels with $27C^2$ weights for $C$ channels, against $49C^2$ for one $7 \times 7$ layer (Simonyan & Zisserman, 2015). The receptive field article covers how depth and downsampling make it grow.

Why It Works

A prior that fits images. Krizhevsky et al. (2012) argue that CNNs make strong and mostly correct assumptions about images, namely stationarity of statistics and locality of pixel dependencies, and so have far fewer connections and parameters than standard feedforward networks with similarly sized layers. LeCun et al. (1998) report 340,908 connections but only 60,000 trainable parameters in LeNet-5, thanks to weight sharing.

A hierarchy of features. Krizhevsky et al. (2012) describe AlexNet’s first-layer kernels as frequency- and orientation-selective filters and colored blobs. Zeiler and Fergus (2014) projected deeper activations back to pixel space with a deconvolutional network: in their visualizations the second layer responds to corners and other edge and color conjunctions, the third to textures such as mesh patterns, the fourth to class-specific parts such as dog faces, and the fifth to entire objects in varied poses.

Invariance is only approximate. Zeiler and Fergus (2014) found their network’s output stable to translation and scaling, though not to rotation. But subsampling breaks exact equivariance: Azulay and Weiss (2019) showed that a one-pixel shift can change a modern CNN’s prediction, attributed this to architectures that ignore the sampling theorem, and found that data augmentation teaches invariance only for images similar to typical training images. Zhang (2019) low-pass filtered before each subsampling step (for max pooling, between a dense max and the subsampling), the classical remedy of sampling and aliasing, and reported more stable outputs and higher ImageNet accuracy for several architectures.

Training Considerations

A CNN is trained like any network (see training a neural network): a cross-entropy loss, gradients by backpropagation, where each shared weight sums the gradients from every position it is used at, and an optimizer such as SGD or Adam. Batch normalization permits higher learning rates, dropout is used mainly in fully connected heads (as in AlexNet), and data augmentation (crops, flips, color changes) supplies variability the architecture does not build in.

The PyTorch example below (run with PyTorch 2.14.1) trains a small CNN (two conv → BN → ReLU → max-pool blocks, global average pooling, and a linear layer) on scikit-learn’s 1,797 digit images, compares it with a fully connected network (MLP) of about the same size, and tests both on test images shifted one pixel to the right.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

SEED = 14
torch.manual_seed(SEED)
digits = load_digits()
X = torch.tensor(digits.images / 16.0, dtype=torch.float32).unsqueeze(1)   # (1797, 1, 8, 8)
y = torch.tensor(digits.target)
idx_tr, idx_te = train_test_split(np.arange(len(y)), test_size=0.2, random_state=0, stratify=digits.target)
X_tr, y_tr, X_te, y_te = X[idx_tr], y[idx_tr], X[idx_te], y[idx_te]

def block(c_in, c_out):                       # conv -> BN -> ReLU -> 2x2 max pooling
    return nn.Sequential(nn.Conv2d(c_in, c_out, 3, padding=1, bias=False),
                         nn.BatchNorm2d(c_out), nn.ReLU(), nn.MaxPool2d(2))

cnn = nn.Sequential(block(1, 16),             # 1x8x8  -> 16x4x4
                    block(16, 32),            # 16x4x4 -> 32x2x2
                    nn.AdaptiveAvgPool2d(1),  # global average pooling -> 32x1x1
                    nn.Flatten(), nn.Linear(32, 10))
mlp = nn.Sequential(nn.Flatten(), nn.Linear(64, 68), nn.ReLU(), nn.Linear(68, 10))

def train(net, epochs=30):
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for epoch in range(epochs):
        net.train()
        for batch in torch.randperm(len(y_tr)).split(64):
            loss = nn.functional.cross_entropy(net(X_tr[batch]), y_tr[batch])
            opt.zero_grad(); loss.backward(); opt.step()
    net.eval()

def accuracy(net, X, y):
    with torch.no_grad():
        return (net(X).argmax(1) == y).float().mean().item()

shift = torch.roll(X_te, 1, dims=3)           # every test digit moved one pixel to the right
shift[..., 0] = 0                             # (the wrapped column is cleared)
for name, net in [("CNN", cnn), ("MLP", mlp)]:
    train(net)
    n = sum(p.numel() for p in net.parameters())
    print(f"{name}: {n:,} parameters, test accuracy {accuracy(net, X_te, y_te):.3f},"
          f" shifted test images {accuracy(net, shift, y_te):.3f}")

Output:

CNN: 5,178 parameters, test accuracy 0.992, shifted test images 0.872
MLP: 5,110 parameters, test accuracy 0.972, shifted test images 0.442

The convolutions have no bias because batch normalization’s shift replaces it. With about 5,000 parameters each, the CNN classifies 99.2% of the 360 test digits, the MLP 97.2% (over 20 seeds, medians 99.3% and 97.2%; seed 14 is typical). The one-pixel shift separates them: the MLP, whose weights are tied to pixel positions, falls to 44.2% (median 44.3%), while the CNN keeps 87.2% (median 87.9%, range 79.7–91.9%). In our reading, the CNN loses accuracy at all because its stride-2 pools are not shift-equivariant, the effect seen in the NumPy example.

Variants

• Dilated (atrous) convolution spaces the filter taps $d$ pixels apart. Yu and Koltun (2016) stacked $3 \times 3$ convolutions with dilations 1, 2, 4, …, so the receptive field grows exponentially with depth while the parameters grow linearly and resolution is kept, for dense prediction such as semantic segmentation.
• Depthwise separable convolution factors a layer into a depthwise $k \times k$ convolution, one filter per input channel, and a pointwise $1 \times 1$ convolution that mixes channels, cutting the cost from $C_{\text{in}} C_{\text{out}} k^2$ to $C_{\text{in}} k^2 + C_{\text{in}} C_{\text{out}}$ per position. With $3 \times 3$ filters it needs 8 to 9 times less computation than a standard convolution, the basis of MobileNets (Howard et al., 2017); Chollet (2017) replaced Inception modules with it in Xception.
• Residual connections add a block’s input to its output, which let He et al. (2016) train networks of 152 layers.
• Fully convolutional networks have no fully connected layers on flattened maps, so the network accepts any image size and can output a map rather than a vector, as dense prediction requires.

Use in Computer Vision

CNNs long served as the backbone of vision systems, a network pretrained on ImageNet classification supplying feature maps for detection, segmentation, and depth. They also compute learned feature descriptors, the matching costs of MC-CNN in stereo matching and cost volumes, the frame encoder of the optical-flow network RAFT, the appearance descriptor in DeepSORT tracking, and the regressors of monocular depth estimation. Their translation equivariance suits sliding-window and dense tasks, a point Liu et al. (2022) make in arguing for convolutions over plain transformers.

Limitations

• Local context. A unit sees only its receptive field, so relating distant parts of an image needs depth, downsampling, dilation, or an added attention layer.
• Limited built-in symmetry. Only translation is built in, and subsampling weakens it; rotation and scale must be learned.
• Competition from transformers. Vision transformers apply self-attention over image patches; Dosovitskiy et al. (2021) note that they lack CNN inductive biases such as translation equivariance and locality and generalize worse on insufficient data, but that with 14M–300M training images, scale outweighs the inductive bias. Liu et al. (2022) modernized a ResNet step by step toward a transformer’s design (patchify stem, $7 \times 7$ depthwise convolutions, fewer activations and normalizations, LayerNorm, GELU) and found the resulting pure CNN, ConvNeXt, competitive with Swin Transformers. Part of the gap was training: their transformer-style recipe alone raised ResNet-50 from 76.1% to 78.8% ImageNet top-1 accuracy.

Important Models

Fukushima’s neocognitron (1980) introduced the structure: alternating layers of feature-extracting S-cells and shift-tolerant C-cells, modeled on Hubel and Wiesel’s simple and complex cells, with all cells of one plane sharing the same input synapses, and trained without a teacher; as LeCun et al. (1998) note, no globally supervised procedure such as backpropagation was available then. LeCun et al. (1989) trained a convolutional network end to end by backpropagation to read handwritten zip codes, and LeNet-5 (LeCun et al., 1998) refined the design for document recognition.

AlexNet (Krizhevsky et al., 2012) brought CNNs to large natural images, combining a bigger network with ReLUs, dropout, and GPU training; it won ILSVRC-2012 with a top-5 test error of 15.3%, against 26.2% for the second-best entry. VGG (Simonyan & Zisserman, 2015) showed that depth built from uniform $3 \times 3$ stacks pays off. GoogLeNet (Szegedy et al., 2015) grew depth and width at a fixed compute budget with Inception modules and $1 \times 1$ reductions, using 12 times fewer parameters than AlexNet. ResNet (He et al., 2016) used residual connections against the degradation problem, in which deeper plain networks had higher training error.

Model	Year	Depth (weight layers)	Key idea	Reported result
LeNet-5	1998	7	Convolution + subsampling trained end to end	MNIST test error 0.95% (0.8% with distorted training data)
AlexNet	2012	8 (5 conv + 3 FC), 60M parameters	ReLU, dropout, GPU training at ImageNet scale	ILSVRC-2010 test top-1/top-5 error 37.5% / 17.0% (single net); ILSVRC-2012 top-5 test error 15.3% (ensemble of 7 nets, 2 pretrained on extra ImageNet data)
VGG	2015	16–19	Stacks of $3 \times 3$ convolutions	ILSVRC-2014 top-5 test error 7.0% (single net), 6.8% (2 nets), both after the challenge
GoogLeNet	2015	22	Inception modules, $1 \times 1$ reductions, average-pooling head	ILSVRC-2014 top-5 test error 6.67% (7 nets, 144 crops)
ResNet	2016	up to 152	Residual connections, BN after every convolution	ImageNet top-5 error 4.49% (single model, validation); ILSVRC-2015 3.57% (6-model ensemble, test)
ConvNeXt	2022	ResNet-like stages	Transformer-inspired modernization of ResNet	ImageNet-1K top-1 accuracy 82.1% (ConvNeXt-T); 87.8% (ConvNeXt-XL, ImageNet-22K pretraining, 384² input)

All results are as reported by the authors; the evaluation protocols (test or validation set, crops, ensembles, extra training data) differ, so the numbers are not directly comparable.