Dropout

Dropout is a regularization technique for neural networks: during training, each unit’s output is set to zero at random, independently for every training example, and at test time the full network is used with its activations rescaled. It was introduced by Hinton et al. (2012), described in full by Srivastava et al. (2014), and used in the AlexNet image classifier.

Purpose

Srivastava et al. argue that hidden units co-adapt: a unit becomes useful only together with particular other units, an arrangement that fits the training set but not new data. If any unit may vanish on any example, no unit can rely on specific partners. A second motivation is model averaging: ensembles of separately trained networks reduce test error but are expensive, while dropout trains a large family of weight-sharing networks inside one network and averages them approximately in one forward pass.

When to Use It

Dropout helps a network that overfits because it has many parameters relative to its training data (see training a neural network); it does not help a model that underfits. It is applied mostly to fully connected layers, such as a classifier head. Ghiasi et al. (2018) observed that recent convolutional architectures rarely use dropout, and then mainly in fully connected layers; spatial correlation and batch normalization, discussed below, help explain this.

How It Works

For a layer with activations $\mathbf{h}$, in training a mask $\mathbf{r}$ is drawn with independent entries $r_j \sim \text{Bernoulli}(p)$, where $p$ is the retention probability, and the next layer receives

$$\tilde{\mathbf{h}} = \mathbf{r} \odot \mathbf{h}.$$

A new mask is drawn for every example and forward pass, and backpropagation runs through the thinned network only. Since $\mathbb{E}[\tilde h_j] = p\,h_j$, the next layer’s inputs are on average $p$ times smaller in training than with all units present. Two conventions remove this mismatch:

• Test-time weight scaling (the original formulation): train with $\tilde{\mathbf{h}} = \mathbf{r} \odot \mathbf{h}$ and, at test time, multiply the outgoing weights of each unit by $p$. With $p = 0.5$, Hinton et al. (2012) halved the outgoing weights of their “mean network”, and AlexNet multiplied the dropout layers’ outputs by 0.5.
• Inverted dropout: scale the survivors up during training, $\tilde{\mathbf{h}} = \mathbf{r} \odot \mathbf{h} / p$, so that $\mathbb{E}[\tilde h_j] = h_j$, and leave the test-time network unchanged. Srivastava et al. (§10) note that the two are equivalent with appropriately scaled learning rates and initial weights.

PyTorch implements inverted dropout, with the opposite parameter convention: nn.Dropout(p) takes the probability that an element is zeroed (default 0.5), scales outputs by $1/(1-p)$ in training, and is the identity in evaluation mode, so Srivastava et al.’s input retention $p = 0.8$ is nn.Dropout(0.2). Tompson et al. (2015) also write a drop probability, while DropBlock’s keep_prob is a retention probability.

An exponential ensemble. A network with $n$ droppable units contains $2^n$ thinned subnetworks sharing weights; each training example trains one, and most are rarely or never sampled. The rescaled full network approximates their average. For one hidden layer followed by a softmax, Hinton et al. (2012) point out that it is exactly equivalent to the geometric mean of the label distributions of all $2^n$ subnetworks; for deeper networks it is an approximation. On MNIST, Srivastava et al. (§7.5) found that averaging $k$ sampled thinned networks matched it at about $k = 50$ and was only slightly better beyond.

How to Apply It

The first script, in NumPy, checks that inverted dropout preserves a digit image’s expected activation, then compares pixel-wise dropout with dropping 2×2 blocks (DropBlock’s recipe, under Variants) on a smooth feature map: the digit upsampled to 16×16 and blurred, standing in for an early convolutional layer.

import numpy as np
from sklearn.datasets import load_digits

rng = np.random.default_rng(2)
digits = load_digits()
img = digits.images[0] / 16.0                       # an 8x8 "0", values in [0, 1]

# 1. Inverted dropout: keep each unit with probability p, scale survivors by 1/p.
def inverted_dropout(a, p, rng):
    mask = rng.random(a.shape) < p                  # Bernoulli(p) keep mask
    return a * mask / p

p = 0.8                                             # retention probability
one = inverted_dropout(img, p, rng)
print(f"one pass: {np.sum(one == 0) - np.sum(img == 0)} of {np.sum(img > 0)} ink pixels dropped, "
      f"survivors x{1 / p:.2f}; mean intensity {img.mean():.3f} -> {one.mean():.3f}")
avg = np.mean([inverted_dropout(img, p, rng) for _ in range(10000)], axis=0)
print(f"mean over 10,000 passes: max |E[out] - img| = {np.abs(avg - img).max():.3f}")
std_dropout = np.mean([img * (rng.random(img.shape) < p) for _ in range(10000)], axis=0)
print(f"without the 1/p scaling, E[out]/img = {std_dropout.sum() / img.sum():.3f} (= p)")

# 2. Pixel-wise dropout vs 2x2 block dropout on a smooth 16x16 feature map.
fmap = np.kron(img, np.ones((2, 2)))                # upsample to 16x16
pad = np.pad(fmap, 1, mode="edge")                  # 3x3 box blur: smooth, like
fmap = sum(pad[i:i + 16, j:j + 16] for i in range(3) for j in range(3)) / 9  # an early conv map

def block_mask(shape, keep_prob, bs, rng):
    """DropBlock-style mask: seeds ~ Bernoulli(gamma), each grows into a bs x bs zero block."""
    n = shape[0]
    gamma = (1 - keep_prob) / bs**2 * n**2 / (n - bs + 1) ** 2   # Ghiasi et al., Eq. 1
    seeds = rng.random((n - bs + 1, n - bs + 1)) < gamma         # valid seed region only
    mask = np.ones(shape, dtype=bool)
    for i, j in zip(*np.nonzero(seeds)):
        mask[i:i + bs, j:j + bs] = False
    return mask

def leak(fmap, mask):
    """How much of a dropped unit's information is still available nearby?"""
    m = np.pad(mask.astype(float), 1); f = np.pad(fmap * mask, 1)
    n = sum(m[i:i + 16, j:j + 16] for i in range(3) for j in range(3))
    s = sum(f[i:i + 16, j:j + 16] for i in range(3) for j in range(3))
    dropped = ~mask
    est = np.where(n > 0, s / np.maximum(n, 1), 0)[dropped]
    err = np.sqrt(np.mean((est - fmap[dropped]) ** 2))        # estimate from 3x3 survivors
    erased = ~(mask[:-1, :-1] | mask[1:, :-1] | mask[:-1, 1:] | mask[1:, 1:])  # all 4 dropped
    return dropped.mean(), erased.mean(), err

keep = 0.75
for name, make in [("pixel-wise", lambda: rng.random(fmap.shape) < keep),
                   ("2x2 blocks", lambda: block_mask(fmap.shape, keep, 2, rng))]:
    r = np.array([leak(fmap, make()) for _ in range(2000)]).mean(axis=0)
    print(f"{name:10s}: dropped {r[0]:.3f}, 2x2 windows fully erased {r[1]:.3f}, "
          f"RMS error of 3x3-neighbor estimate {r[2]:.3f}")
print(f"(feature-map RMS value {np.sqrt(np.mean(fmap**2)):.3f})")

Output:

one pass: 8 of 35 ink pixels dropped, survivors x1.25; mean intensity 0.287 -> 0.282
mean over 10,000 passes: max |E[out] - img| = 0.006
without the 1/p scaling, E[out]/img = 0.800 (= p)
pixel-wise: dropped 0.250, 2x2 windows fully erased 0.004, RMS error of 3x3-neighbor estimate 0.069
2x2 blocks: dropped 0.226, 2x2 windows fully erased 0.081, RMS error of 3x3-neighbor estimate 0.107
(feature-map RMS value 0.377)

Averaged over passes, every pixel returns to within 0.006 of its value; without the $1/p$ factor the average is 80% of it, the shrinkage that test-time weight scaling corrects. Pixel-wise dropout fully erases only 0.4% ($0.25^4$) of the 2×2 windows, so the next layer nearly always still sees part of every small region, and a dropped value can be estimated from its surviving neighbors with an RMS error of 0.069. Dropping 2×2 blocks erases 8.1% of the windows and raises the error to 0.107. (Overlapping blocks drop 0.226 rather than 0.25 of the units, as Ghiasi et al. note.)

The second script trains a fully connected network on 300 of the 1,797 digit images and tests on the other 1,497, with and without dropout (drop probabilities 0.2 for input pixels and 0.5 for hidden units), and runs Monte Carlo dropout (under Variants). It was run with PyTorch 2.14.1 and takes about ten seconds on a CPU.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

print("torch", torch.__version__)

# PyTorch's p is the DROP probability; survivors are scaled by 1/(1-p) in training.
drop = nn.Dropout(p=0.5)
x = torch.ones(1, 8)
torch.manual_seed(0)
print("train:", drop.train()(x).tolist()[0])
print("eval: ", drop.eval()(x).tolist()[0])

X, y = load_digits(return_X_y=True)                  # 1797 8x8 images, 64 pixels each
X = ((X - X.mean()) / X.std()).astype(np.float32)

def make_mlp(p_in, p_hid, width=256):
    return nn.Sequential(
        nn.Dropout(p_in),                              # drops input pixels
        nn.Linear(64, width), nn.ReLU(), nn.Dropout(p_hid),
        nn.Linear(width, width), nn.ReLU(), nn.Dropout(p_hid),
        nn.Linear(width, 10))

def run(seed, p_in, p_hid, n_train=300, epochs=100):
    Xtr, Xte, ytr, yte = (torch.tensor(a) for a in train_test_split(
        X, y, train_size=n_train, random_state=seed, stratify=y))
    torch.manual_seed(seed)
    model = make_mlp(p_in, p_hid)
    opt = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
    for _ in range(epochs):
        model.train()                                  # dropout on
        for idx in torch.randperm(n_train).split(20):
            loss = F.cross_entropy(model(Xtr[idx]), ytr[idx])
            opt.zero_grad(); loss.backward(); opt.step()
    model.eval()                                       # dropout off: identity
    with torch.no_grad():
        stats = [(F.cross_entropy(model(A), b).item(), (model(A).argmax(1) == b).float().mean().item())
                 for A, b in ((Xtr, ytr), (Xte, yte))]
    return model, Xte, yte, stats

SEED = 8
for name, p_in, p_hid in [("no dropout", 0.0, 0.0), ("dropout", 0.2, 0.5)]:
    model, Xte, yte, ((trl, tra), (tel, tea)) = run(SEED, p_in, p_hid)
    print(f"{name:10s}  train loss {trl:.3f} acc {tra:.3f}   test loss {tel:.3f} acc {tea:.3f}")

# Monte Carlo dropout: keep dropout on at test time and average T softmax outputs.
model.train()
with torch.no_grad():
    probs = torch.stack([F.softmax(model(Xte), 1) for _ in range(50)])   # (T, N, 10)
mean = probs.mean(0)
right = mean.argmax(1) == yte
spread = probs.std(0).gather(1, mean.argmax(1, keepdim=True)).squeeze(1)
print(f"MC dropout (T=50): test acc {right.float().mean():.3f}; std of top-class probability "
      f"{spread[right].mean():.3f} on correct vs {spread[~right].mean():.3f} on wrong images")

# Over ten splits and seeds, does dropout help?
acc = np.array([[run(s, pi, ph)[3][1][1] for pi, ph in ((0.0, 0.0), (0.2, 0.5))] for s in range(10)])
d = acc[:, 1] - acc[:, 0]
print(f"10 seeds: test acc {acc[:, 0].mean():.3f} -> {acc[:, 1].mean():.3f}; "
      f"gain min {d.min():+.3f}, median {np.median(d):+.3f}, max {d.max():+.3f}")

Output:

torch 2.14.1
train: [0.0, 0.0, 2.0, 0.0, 0.0, 0.0, 2.0, 2.0]
eval:  [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
no dropout  train loss 0.002 acc 1.000   test loss 0.163 acc 0.953
dropout     train loss 0.004 acc 1.000   test loss 0.112 acc 0.961
MC dropout (T=50): test acc 0.953; std of top-class probability 0.156 on correct vs 0.353 on wrong images
10 seeds: test acc 0.946 -> 0.957; gain min +0.002, median +0.009, max +0.023

In training mode nn.Dropout(0.5) turns each 1 into 0 or 2. Both networks fit the training images perfectly, but dropout raises test accuracy from 95.3% to 96.1% and lowers the test cross-entropy from 0.163 to 0.112, consistent with less overconfident predictions. Over ten splits dropout helped every time, by 0.2 to 2.3 percentage points (median 0.9): consistent but modest on this easy task. Averaging 50 stochastic passes gave 95.3% here, slightly below the rescaled network, and the spread of the predicted probability across passes was more than twice as large on misclassified images.

Variants

• SpatialDropout (Tompson et al., 2015) draws one Bernoulli variable per channel and drops entire feature maps. In the authors’ fully convolutional human pose network, standard dropout lengthened training without preventing overfitting, which they attributed to strongly correlated neighboring activations; channel-wise dropout improved results, especially on the small FLIC dataset. PyTorch implements it as nn.Dropout2d.
• DropBlock (Ghiasi, Lin and Le, 2018) drops contiguous block_size × block_size regions of each feature map. Block seeds are sampled at a rate $\gamma$ derived from the target keep_prob, and survivors are rescaled by the ratio of total to kept units. It resembles dropout with block_size 1 and SpatialDropout with a block covering the whole map. A fixed keep_prob hurt early learning, so the authors decreased it linearly from 1 to its target during training.
• DropConnect (Wan et al., 2013) zeroes individual weights of fully connected layers instead of activations; at inference it approximates each unit’s input by a moment-matched Gaussian and averages sampled activations.
• Stochastic depth (Huang et al., 2016) skips whole residual blocks for each mini-batch, making the network shorter rather than thinner. Survival probabilities decrease linearly with depth, to 0.5 for the last block in the authors’ experiments, and residual branches are scaled by them at test time.
• Monte Carlo dropout (Gal and Ghahramani, 2016) keeps dropout on at test time and uses the mean of $T$ stochastic forward passes as the prediction and their spread as a measure of model uncertainty, justified by casting dropout training as approximate Bayesian inference in deep Gaussian processes. Shown a digit 1 rotated in steps, their dropout LeNet’s sampled outputs for competing classes overlapped at the intermediate rotations where its prediction switched to 5 and then 7.

Best Practices

• Rates. Srivastava et al. found retention $p = 0.5$ close to optimal for hidden units across many tasks, give 0.5 to 0.8 as the typical range, and suggest about 0.8 for real-valued inputs such as image patches (PyTorch drop probabilities 0.5, 0.2 to 0.5, and 0.2). DropBlock’s authors used keep_prob between 0.75 and 0.95.
• Compensate for the noise. Srivastava et al. recommend widening a layer from $n$ to about $n/p$ units, a 10 to 100 times larger learning rate or momentum of 0.95 to 0.99, and a max-norm constraint (typically 3 to 4) on each unit’s incoming weights. A dropout network typically takes 2 to 3 times longer to train, they report; AlexNet’s authors found that it roughly doubled the iterations to converge.
• Use structured dropout on feature maps (SpatialDropout, DropBlock) and plain dropout on fully connected layers.
• Keep dropout after batch normalization. Li et al. (2019) advise against dropout before batch normalization layers inside the network, recommend drop rates below 0.5, and note that a single dropout layer after the network’s last batch normalization layer does not cause the variance shift described under Pitfalls.

Pitfalls

• Forgetting evaluation mode. A model left in training mode keeps dropping units; in PyTorch, model.eval() switches dropout off.
• Mixing up the conventions. “Dropout 0.8” may mean keeping or dropping 80% of units; a retention probability of 0.8 passed to nn.Dropout drops 80% of them.
• Variance shift with batch normalization. Li et al. (2019) explain why the two often combine badly: dropout changes a unit’s variance between training and test, even with inverted scaling, while batch normalization applies at test time the running variance accumulated in training, so the normalized activations shift at inference. In their example, a DenseNet on CIFAR-100 reached 68.55% test accuracy with dropout 0.5 in each bottleneck and 77.42% without. Huang et al. (2016) likewise found dropout gave practically no improvement on 110-layer residual networks with batch normalization on CIFAR-10.

Evidence and Impact

Srivastava et al. (2014) reported 1.35% MNIST error for a dropout network against about 1.60% for the best earlier networks without dropout or unsupervised pretraining. On SVHN, dropout in the fully connected layers of a convolutional network lowered the error from 3.95% to 3.02%, and adding it to the convolutional layers lowered it to 2.55%; on CIFAR-10 the corresponding errors were 14.98%, 14.32% and 12.61%, and on CIFAR-100 dropout lowered it from 43.48% to 37.20%. The authors attribute the gain from dropout in convolutional layers to the noisier inputs it gives the fully connected layers above.

AlexNet (Krizhevsky et al., 2012) applied dropout with probability 0.5 to its first two fully connected layers, without which, its authors reported, it overfit substantially; Srivastava et al. list the resulting ILSVRC-2012 winning entry at 16.4% top-5 test error against 26.2% for the next best method. DropBlock raised the ImageNet top-1 validation accuracy of ResNet-50 from 76.51% to 78.13%, against 76.80% for the best dropout setting and 77.41% for SpatialDropout (averages over three runs), and RetinaNet’s COCO detection AP from 36.8 to 38.4. Li et al. reported that one dropout layer with drop rate 0.2 after the last batch normalization layer reduced the ImageNet top-1 error of ResNet-200 from 21.70% to 21.48%.