Multilayer Perceptron

A multilayer perceptron (MLP) is a neural network made of fully connected layers stacked one after another, with an activation function after every layer except the last. Every unit of a layer reads every output of the layer before, so the network has no built-in idea of which inputs are neighbors. That makes it the most general feedforward design and, for images, an expensive and position-blind one. Vision models rarely run an MLP on raw pixels; instead they use small MLPs inside larger designs: as the classifier head of a convolutional network, as the second half of every transformer block, and as the whole model in coordinate networks such as NeRF.

Core Idea

An MLP computes a function by alternating two operations: an affine map that mixes all its inputs, and an elementwise nonlinearity. The affine maps hold all the learned parameters; the nonlinearities keep the stack from collapsing into a single linear map, as the prerequisite articles show. Goodfellow, Bengio and Courville (2016, Ch. 6) treat “multilayer perceptron” as another name for a deep feedforward network, one in which information flows from input to output with no feedback connections.

The name is historical: it extends Rosenblatt’s (1958) perceptron, whose story and XOR limitation the neural network article tells. Goodfellow et al. (2016, §6.6) note that learning nonlinear functions needed both a multilayer perceptron and a way to compute gradients through it. Back-propagation supplied the second, and Rumelhart, Hinton and Williams (1986) singled out its ability to create useful new features in the hidden units as what distinguished it from the earlier perceptron-convergence procedure. Since gradients must pass through the units, modern MLPs use differentiable activations such as the sigmoid, ReLU or GELU rather than the perceptron’s hard threshold, so the “perceptron” in the name refers to the lineage rather than the unit (our observation).

Building Blocks

• Fully connected (linear) layer. A weight matrix and a bias vector, $\mathbf{h} \mapsto \mathbf{W}\mathbf{h} + \mathbf{b}$, connecting every input to every output. In PyTorch this is nn.Linear.
• Activation function. Applied elementwise after each hidden layer: ReLU in most convolutional-network heads, GELU in Vision Transformer blocks.
• Output layer. A final linear layer with no activation, whose outputs are the logits; for classification, softmax turns them into class probabilities.
• Optional extras. Dropout between hidden layers, normalization layers, and residual connections are added in deeper or larger MLPs; none changes the fully connected structure.

The number of hidden layers is the depth and the number of units per layer the width; an MLP is specified by its list of widths $[n_0, n_1, \dots, n_L]$, from the input size $n_0$ to the number of outputs $n_L$.

How Data Flows

An MLP accepts a vector, so an image batch of shape $(N, C, H, W)$ is first flattened to $(N, CHW)$: a $16 \times 16$ grayscale image becomes 256 numbers, a $224 \times 224$ RGB image 150,528. Flattening fixes an order for the pixels, and from then on the network sees only that list. Each layer then maps the $(N, n_{\ell-1})$ batch to $(N, n_\ell)$ with one matrix product.

Because nn.Linear acts on the last dimension of its input, the same MLP can also be applied to every position of a feature map or every token of a sequence at once, with shared weights: on a $(N, T, D)$ tensor of $T$ tokens it transforms each $D$-dimensional token separately. Used this way an MLP does not flatten the image; it mixes channels at each position and never mixes positions. This per-position form is what a transformer block uses, and on a convolutional feature map it is the same as a stack of $1 \times 1$ convolutions, as the convolutional neural network article notes for Network in Network.

Mathematical Formulation

With $\mathbf{h}^{(0)} = \mathbf{x}$, an MLP with $L$ layers computes

$$\mathbf{h}^{(\ell)} = \phi\big(\mathbf{W}^{(\ell)} \mathbf{h}^{(\ell-1)} + \mathbf{b}^{(\ell)}\big), \quad \ell = 1, \dots, L-1, \qquad \mathbf{s} = \mathbf{W}^{(L)} \mathbf{h}^{(L-1)} + \mathbf{b}^{(L)},$$

where $\mathbf{W}^{(\ell)} \in \mathbb{R}^{n_\ell \times n_{\ell-1}}$. Layer $\ell$ has $n_{\ell-1} n_\ell$ weights and $n_\ell$ biases, so the whole network has

$$P = \sum_{\ell=1}^{L} (n_{\ell-1} + 1)\, n_\ell$$

parameters, and one forward pass costs $\sum_\ell n_{\ell-1} n_\ell$ multiply-adds per image: one per weight, since each weight is used exactly once.

The first term, $(n_0 + 1) n_1$, grows with the number of pixels. The script below counts the parameters of a one-hidden-layer MLP with 1,024 hidden units at several image sizes, then splits the parameters of VGG-16 (configuration D of Simonyan and Zisserman, 2015) into its 13 convolutional layers and its MLP head. The paper describes the head as three fully connected layers of 4,096, 4,096 and 1,000 channels on the last feature maps, which for a $224 \times 224$ input are $7 \times 7 \times 512$ after five $2 \times 2$ max-poolings.

def mlp_params(widths):
    """Weights plus biases of an MLP with layer widths [n_0, n_1, ..., n_L]."""
    return sum((a + 1) * b for a, b in zip(widths[:-1], widths[1:]))

# One hidden layer of 1,024 units and 10 outputs, on square RGB images of growing size.
for side in (32, 64, 128, 224):
    n_in = side * side * 3
    print(f"{side:3d}x{side:<3d} RGB: {n_in:7,} inputs -> {mlp_params([n_in, 1024, 10]):12,} parameters")

# VGG-16 (configuration D): 13 3x3 conv layers, then the MLP head on 7x7x512 feature maps.
convs = [(3, 64), (64, 64), (64, 128), (128, 128), (128, 256), (256, 256), (256, 256),
         (256, 512), (512, 512), (512, 512), (512, 512), (512, 512), (512, 512)]
conv = sum(9 * c_in * c_out + c_out for c_in, c_out in convs)
head = mlp_params([7 * 7 * 512, 4096, 4096, 1000])
print(f"VGG-16: conv layers {conv:,}, MLP head {head:,}, total {conv + head:,}; "
      f"head share {head / (conv + head):.1%}")

Output:

 32x32  RGB:   3,072 inputs ->    3,157,002 parameters
 64x64  RGB:  12,288 inputs ->   12,594,186 parameters
128x128 RGB:  49,152 inputs ->   50,342,922 parameters
224x224 RGB: 150,528 inputs ->  154,151,946 parameters
VGG-16: conv layers 14,714,688, MLP head 123,642,856, total 138,357,544; head share 89.4%

The parameter count grows almost exactly in proportion to the number of pixels, so doubling the image side roughly quadruples it. The total for VGG-16 agrees with the 138 million in the paper’s Table 2, and by our arithmetic its three fully connected layers hold 89.4% of the weights, 102.8 million of them in the first layer alone, even though the thirteen convolutional layers do almost all of the spatial processing.

Why It Works

Universal approximation. Cybenko (1989) and Hornik, Stinchcombe and White (1989) proved that an MLP with one hidden layer of sigmoid-like (“squashing”) units and a linear output is a universal approximator, and Leshno et al. (1993), as Goodfellow et al. (2016, §6.4.1) report, extended the result to non-polynomial activations including ReLU. The neural network article states the theorems exactly and what they leave open.

For a one-dimensional input the idea can be made concrete with ReLUs. The script below takes the intensity profile across one row of an image of a blurred bright bar and builds, by hand, a network whose hidden unit $j$ computes $\operatorname{ReLU}(x - t_j)$ for evenly spaced knots $t_j$; choosing each output weight as the change of slope at its knot makes the output the piecewise-linear interpolant of the profile.

import numpy as np

# Target: the intensity profile across a blurred bright bar, one row of an image, x in [0, 1].
f = lambda x: np.exp(-((x - 0.5) / 0.08) ** 2)
x = np.linspace(0, 1, 10001)

def relu_net(n_units):
    """One hidden layer of ReLUs, set by hand so the output interpolates f at n_units + 1 knots."""
    knots = np.linspace(0, 1, n_units + 1)
    slopes = np.diff(f(knots)) / np.diff(knots)               # slope of each linear piece
    a = np.concatenate([[slopes[0]], np.diff(slopes)])         # output weight = change of slope
    hidden = np.maximum(0, x[:, None] - knots[None, :-1])      # unit j: ReLU(x - knot_j)
    return f(0) + hidden @ a

for n in (10, 20, 40, 80, 160, 320):
    print(f"{n:4d} hidden units: max error {np.abs(relu_net(n) - f(x)).max():.5f}")

Output:

  10 hidden units: max error 0.10608
  20 hidden units: max error 0.06946
  40 hidden units: max error 0.02242
  80 hidden units: max error 0.00597
 160 hidden units: max error 0.00152
 320 hidden units: max error 0.00038

Any accuracy can be reached, but only by adding units: from 40 units on, each doubling cuts the worst-case error by about four. This construction places one unit per piece along one input axis; it is an illustration, not the proof the theorems use.

What the theorem does not promise. It is an existence result: it says neither how many hidden units are needed nor whether training will find the weights. Goodfellow et al. (2016, §6.4.1) add that in the worst case the number of hidden units can be exponential in the input dimension. For images, our reading adds a further gap: the theorem treats the input as a point in $\mathbb{R}^n$ and says nothing about which functions are cheap to represent or learn, so it gives no reason to expect a network to treat a shifted pattern like the original. NeRF’s authors make the practical point directly: although networks are universal approximators, an MLP fed raw coordinates rendered high-frequency colour and geometry poorly, which they addressed with a positional encoding of the input (Mildenhall et al., 2020, §5.1).

Training Considerations

An MLP is trained like any network: a loss function such as cross-entropy, gradients from backpropagation, and an optimizer such as stochastic gradient descent or Adam; the training article covers the loop. Points specific to MLPs:

• Overfitting in wide layers. Large fully connected layers hold most of a classic CNN’s parameters and are where it overfits; VGG applied dropout with ratio 0.5 to its first two fully connected layers only (Simonyan & Zisserman, 2015).
• Initialization and depth. The scale of the initial weights relative to each layer’s fan-in decides whether activations and gradients keep their size through the stack; weight initialization and vanishing and exploding gradients are the subjects of their own articles. Deep MLPs in modern models add normalization and residual connections, as transformers do.
• Input scaling. The first layer weights pixels directly, so inputs are normalized (for example to zero mean and unit variance); the activation function article shows how unscaled 8-bit intensities saturate a sigmoid.

Use in Computer Vision

• Classifier heads. Classic CNNs such as AlexNet and VGG end in an MLP of two 4,096-unit layers and a class layer. Later networks largely replaced it with global average pooling and a single linear layer (see convolutional neural network). In ViT, the classification head is an MLP with one hidden layer during pre-training and a single linear layer for fine-tuning (Dosovitskiy et al., 2021).
• Transformer blocks. Every block of a transformer or Vision Transformer applies a two-layer MLP with a GELU to each token separately, after self-attention has mixed the tokens. With the usual fourfold expansion it holds about two thirds of a block’s weights, as counted in the transformer article.
• All-MLP vision models. MLP-Mixer (Tolstikhin et al., 2021) replaces attention by a second MLP: a token-mixing MLP applied to each channel across the patches, interleaved with a channel-mixing MLP applied to each patch across channels, both with GELUs, skip connections and layer normalization. The authors reported 87.94% ImageNet validation top-1 accuracy for Mixer-H/14 pre-trained on JFT-300M and fine-tuned at resolution 448 (their Table 2), and 76.4% for Mixer-B/16 trained from scratch on ImageNet at resolution 224, where it tended to overfit.
• Point sets. PointNet (Qi et al., 2017) applies the same MLP to every point of a 3D point cloud and combines the results by max pooling, a symmetric function, so the output does not depend on the order of the points.
• Coordinate networks. A neural implicit representation is an MLP that maps a coordinate to a value instead of an image to a label. NeRF’s MLP maps an encoded 3D position through 8 fully connected ReLU layers of 256 channels to a density and a feature vector, and with the viewing direction through one more 128-channel layer to a colour (Mildenhall et al., 2020).

Limitations

Parameters scale with resolution. As computed above, the first layer of an MLP on pixels grows with the image area, and its multiply-adds with it. A convolution’s parameter count does not depend on the image size at all.

No notion of position or neighborhood. A first-layer weight belongs to one pixel. A pattern learned at one position is unrelated, for the network, to the same pattern a few pixels away, and a fixed permutation of all pixels gives an equivalent problem. The convolutional neural network article measures the effect of a one-pixel shift on real digits; the PyTorch script below (run with PyTorch 2.14.1) makes both properties exact on synthetic images. Each $16 \times 16$ image contains one $5 \times 5$ glyph, a plus or a diagonal cross, both with nine lit pixels; the training set has every glyph at every position in the left half (columns 0–7), and the test set every position in the right half (columns 8–15), which no training image ever lights. An MLP (256 → 64 → 2) is compared with a convolutional model of 8 shared $5 \times 5$ filters with wrap-around padding, a ReLU, a global max over positions and a linear classifier. The script then checks the MLP over 20 seeds, and trains it once on pixel-shuffled images.

import torch
from torch import nn

torch.set_num_threads(1)
torch.set_default_dtype(torch.float64)

# Two 5x5 glyphs with the same number of lit pixels (9): a plus and a diagonal cross.
plus = torch.zeros(5, 5); plus[2, :] = 1; plus[:, 2] = 1
cross = torch.eye(5); cross = torch.maximum(cross, cross.flip(1))

def make_set(columns):
    """Every glyph at every top-left corner (row 0..11, column in `columns`) on a 16x16 canvas."""
    X, y = [], []
    for label, g in enumerate([plus, cross]):
        for r in range(12):
            for c in columns:
                img = torch.zeros(16, 16); img[r:r + 5, c:c + 5] = g
                X.append(img); y.append(label)
    return torch.stack(X).unsqueeze(1), torch.tensor(y)   # (N, 1, 16, 16)

X_left, y_left = make_set(range(0, 4))     # glyphs lie in columns 0..7
X_right, y_right = make_set(range(8, 12))  # glyphs lie in columns 8..15, never lit in training
print("train", tuple(X_left.shape), "| unseen positions", tuple(X_right.shape))

def mlp():
    return nn.Sequential(nn.Flatten(), nn.Linear(256, 64), nn.ReLU(), nn.Linear(64, 2))

def convnet():  # 8 shared 5x5 filters (wrap-around padding), ReLU, global max, linear classifier
    return nn.Sequential(nn.Conv2d(1, 8, 5, padding=2, padding_mode="circular"), nn.ReLU(), nn.AdaptiveMaxPool2d(1),
                         nn.Flatten(), nn.Linear(8, 2))

def train(net, X, y, steps=300):
    opt = torch.optim.Adam(net.parameters(), lr=1e-2)   # no weight decay
    for _ in range(steps):
        opt.zero_grad()
        loss = nn.functional.cross_entropy(net(X), y)
        loss.backward()
        opt.step()
    return loss.item()

def acc(net, X, y):
    with torch.no_grad():
        return (net(X).argmax(1) == y).double().mean().item()

for name, make in [("MLP", mlp), ("conv", convnet)]:
    torch.manual_seed(0)
    net = make()
    W0 = [p.detach().clone() for p in net.parameters()]
    loss = train(net, X_left, y_left)
    n = sum(p.numel() for p in net.parameters())
    print(f"{name:4s} {n:6,} parameters | train loss {loss:.4f} | "
          f"accuracy: trained positions {acc(net, X_left, y_left):.3f}, "
          f"unseen positions {acc(net, X_right, y_right):.3f}")
    if name == "MLP":
        W1 = net[1].weight.view(64, 16, 16)
        change = (W1 - W0[0].view(64, 16, 16)).abs()
        print(f"     first-layer weight change: columns 0-7 max {change[:, :, :8].max():.3f}, "
              f"columns 8-15 max {change[:, :, 8:].max():.1e}")
    else:
        feat = lambda X: net[:4](X)
        with torch.no_grad():
            F = torch.stack([feat(X_left).view(2, 48, 8), feat(X_right).view(2, 48, 8)], 1).view(2, 96, 8)
            d = (F - F[:, :1]).abs().max()   # each glyph at all 96 positions vs its first position
        print(f"     max difference between pooled features of a glyph at any two positions: {d:.1e}")

# Spread over seeds: MLP accuracy on unseen positions.
accs = []
for seed in range(20):
    torch.manual_seed(seed)
    net = mlp(); train(net, X_left, y_left)
    accs.append(acc(net, X_right, y_right))
accs = torch.tensor(accs)
print(f"MLP, seeds 0-19, unseen positions: median {accs.median():.3f}, "
      f"range {accs.min():.3f}-{accs.max():.3f}")

# The MLP ignores pixel layout: shuffle the pixels of every image with one fixed permutation
# and permute the first-layer weights the same way; training follows the same path.
perm = torch.randperm(256, generator=torch.Generator().manual_seed(1))
torch.manual_seed(0); a = mlp()
torch.manual_seed(0); b = mlp()
with torch.no_grad():
    b[1].weight.copy_(a[1].weight[:, perm])
X_shuf = X_left.view(-1, 256)[:, perm].view(-1, 1, 16, 16)
la, lb = train(a, X_left, y_left), train(b, X_shuf, y_left)
with torch.no_grad():
    gap = (a(X_right) - b(X_right.view(-1, 256)[:, perm].view(-1, 1, 16, 16))).abs().max()
print(f"original vs shuffled pixels: final losses {la:.6f} / {lb:.6f}, max output difference {gap:.1e}")

Output:

train (96, 1, 16, 16) | unseen positions (96, 1, 16, 16)
MLP  16,578 parameters | train loss 0.0000 | accuracy: trained positions 1.000, unseen positions 0.500
     first-layer weight change: columns 0-7 max 0.526, columns 8-15 max 0.0e+00
conv    226 parameters | train loss 0.0002 | accuracy: trained positions 1.000, unseen positions 1.000
     max difference between pooled features of a glyph at any two positions: 0.0e+00
MLP, seeds 0-19, unseen positions: median 0.500, range 0.427-0.521
original vs shuffled pixels: final losses 0.000026 / 0.000026, max output difference 5.6e-16

Both models classify every training image. On the unseen right half the MLP is at chance, 50.0% (over seeds 0–19, between 42.7% and 52.1%), and the weight check shows why: the gradient of a first-layer weight is the error at its hidden unit times its input pixel, and the right-half pixels are zero in every training image, so their weights never moved from their random initial values. Whatever the MLP learned about pluses and crosses is attached to the left-half pixels. The convolutional model, with 73 times fewer parameters, classifies every unseen image, and not by luck: with shared filters and wrap-around padding, shifting a glyph shifts the filter responses, the global max discards the position, and the pooled features of a glyph are identical wherever it sits. Training the MLP on pixel-shuffled images, with its first-layer weights permuted to match, follows the same path to within rounding error, so the spatial arrangement of the pixels carries no information for it.

Data augmentation can show an MLP each pattern at every position, but it then has to learn a separate detector per position from more data, which weight sharing provides for free (our reasoning). Tolstikhin et al. (2021) ran a related test at scale: trained on JFT-300M with one fixed shuffle applied to every image, Mixer-B/16 was, as expected, invariant to shuffling the order of the $16 \times 16$ patches and the pixels within them, whereas ResNet-50 lost much of its accuracy; when pixels were shuffled globally, Mixer’s linear 5-shot ImageNet accuracy dropped by about 45% and ResNet’s by about 75%.

Not a fit for variable-size inputs. A flattened MLP fixes the number of inputs, so it accepts one image size. VGG worked around this at test time by converting its fully connected layers into convolutions, the first into a $7 \times 7$ convolution and the last two into $1 \times 1$ convolutions, so that the network could be applied to whole images of any size (Simonyan & Zisserman, 2015).

Variants

• Gated MLPs multiply two linear projections, one passed through a nonlinearity, as in the SwiGLU blocks described in the activation function article.
• Deep MLPs with skip connections concatenate or add the input to a middle layer, as NeRF’s MLP does at its fifth layer in the arXiv version (Mildenhall et al., 2020, arXiv v2, Appendix A), and normalized residual MLPs are the basis of Mixer.