Activation Function

An activation function, or nonlinearity, is the fixed function $\phi$ that a neural network applies to each unit’s weighted sum $z = \mathbf{w}^\top \mathbf{x} + b$. Without it, any number of layers computes the same kind of function as one. The choice also decides how well gradients pass back through the layers during training, which is why the switch from the sigmoid to the rectified linear unit (ReLU) was one of the changes that made deep image classifiers trainable.

Definition

An activation function is a scalar function $\phi: \mathbb{R} \to \mathbb{R}$, applied elementwise to the pre-activations of a layer: a vector of weighted sums in a fully connected layer, or a feature map of filter responses in a convolutional layer. It has no learned parameters, with a few exceptions such as PReLU. It must be nonlinear, and for gradient-based training it must have a usable derivative almost everywhere.

Intuition

Consider the first layer of an image network as a bank of filters. A Sobel filter for horizontal gradients gives a positive response where the image brightens from left to right, a negative one where it darkens, and something near zero in flat regions. An activation function decides what the next layer sees of that response. ReLU passes positive responses unchanged and replaces negative ones with zero, so the output is a sparse map that marks one edge polarity only. A sigmoid squashes every response into $(0, 1)$: weak and strong edges become hard to tell apart once both are near the top of the curve.

The second reason for the nonlinearity is that linear operations compose into linear operations. Blurring an image and then applying a Sobel filter is the same as applying one combined $5 \times 5$ filter, since convolution is associative. A network of convolutions without activations, however deep, is therefore one big convolution. Putting a ReLU between the two filters breaks this; the example below checks both facts numerically.

Formal Definition

For a layer with weights $\mathbf{W}$ and biases $\mathbf{b}$, the activation acts elementwise:

$$\mathbf{h} = \phi(\mathbf{z}), \qquad \mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}, \qquad h_i = \phi(z_i).$$

Sigmoid and tanh. The logistic sigmoid and the hyperbolic tangent were the standard choices before ReLU:

$$\sigma(z) = \frac{1}{1 + e^{-z}}, \quad \sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big); \qquad \tanh(z) = 2\sigma(2z) - 1, \quad \tanh'(z) = 1 - \tanh^2(z).$$

The rectifier family. ReLU and its variants differ only in what they do with negative inputs:

$$\text{ReLU}(z) = \max(0, z), \qquad \text{PReLU}(z) = \max(0, z) + a \min(0, z), \qquad \text{ELU}(z) = \begin{cases} z & z > 0 \\ \alpha (e^{z} - 1) & z \le 0. \end{cases}$$

ReLU’s derivative is 1 for $z > 0$ and 0 for $z < 0$. Leaky ReLU fixes $a$ at a small value such as $0.01$; PReLU learns it. ELU’s derivative for $z \le 0$ is $\alpha e^{z}$, which Clevert et al. (2016) write as $\text{ELU}(z) + \alpha$.

Smooth gated units. GELU and SiLU multiply the input by a smooth gate between 0 and 1, the standard normal CDF $\Phi$ or the sigmoid:

$$\text{GELU}(z) = z\,\Phi(z), \quad \text{GELU}'(z) = \Phi(z) + z\,\varphi(z); \qquad \text{SiLU}(z) = z\,\sigma(z), \quad \text{SiLU}'(z) = \sigma(z)\big(1 + z(1 - \sigma(z))\big),$$

where $\varphi$ is the standard normal density.

Function	Range	Derivative	Typical use in vision
Sigmoid $\sigma(z)$	$(0, 1)$	$\sigma(1 - \sigma)$, at most $1/4$	Binary or multi-label outputs, gates
$\tanh(z)$	$(-1, 1)$	$1 - \tanh^2$, at most $1$	Early networks, recurrent models
ReLU $\max(0, z)$	$[0, \infty)$	0 or 1	Default in CNNs since AlexNet
Leaky ReLU / PReLU	$\mathbb{R}$	$a$ or 1	CNNs, PReLU in He et al. (2015)
ELU ($\alpha = 1$)	$(-1, \infty)$	$e^{z}$ or 1	CNN classifiers (Clevert et al., 2016)
GELU $z\Phi(z)$	about $[-0.17, \infty)$	$\Phi + z\varphi$	Vision Transformer MLP blocks
SiLU / Swish $z\sigma(z)$	about $[-0.28, \infty)$	$\sigma(1 + z(1 - \sigma))$	EfficientNet

The minima of GELU and SiLU, near $z = -0.75$ and $z = -1.28$, were computed numerically.

Properties

Saturation and vanishing gradients. The sigmoid and tanh saturate on both sides: once $|z|$ exceeds a few units the derivative is close to zero. Goodfellow, Bengio and Courville (2016, §6.3.2) note that this can make gradient-based learning very difficult and discourage sigmoidal hidden units in feedforward networks. During backpropagation, the gradient that reaches an early layer contains one factor $\phi'(z)$ for every layer it passes through. The following bound is our own illustration: since $\sigma'(z) \le 1/4$, ten sigmoid layers multiply the gradient by at most $(1/4)^{10} \approx 10^{-6}$ from the activations alone, before the weights are taken into account. The same section notes that tanh typically performs better than the sigmoid because it resembles the identity more closely: $\tanh(0) = 0$, whereas $\sigma(0) = 1/2$.

Rectifiers do not saturate for positive inputs. Nair and Hinton (2010) used rectified linear units in restricted Boltzmann machines, and Krizhevsky et al. (2012) took the name ReLU from them. ReLU’s derivative is exactly 1 wherever the unit is active, so Glorot, Bordes and Bengio (2011) argue that gradients flow well along the active paths, with no vanishing effect from the activations there. They also point out that ReLU produces true zeros, a sparse representation, and measured an average of 72.0% zeros in the hidden layers of their CIFAR-10 networks, which were trained with an L1 penalty that encourages sparsity. Krizhevsky, Sutskever and Hinton (2012) reported that a four-layer convolutional network with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh units, adding that the size of the effect varies with the architecture. Goodfellow et al. (2016, §6.3) call ReLU an excellent default.

Dead units. The flip side is that ReLU passes no gradient for $z < 0$. Maas, Hannun and Ng (2013) point out that a unit that is inactive for every input never has its weights adjusted. The leaky rectifier was their remedy: its slope of $0.01$ for negative inputs keeps a small gradient, although in their speech-recognition experiments the leaky and standard rectifiers performed about the same.

Learning the negative slope. He et al. (2015) made the slope a learned parameter (PReLU), one per channel or one shared by all channels of a layer, initialized at $0.25$. On a 14-layer ImageNet model the per-channel version lowered the top-1 error from 33.82% to 32.64%. The learned slopes in the first convolutional layer were about 0.6 to 0.7, which the authors read as the network keeping both polarities of its edge-like filters’ responses; deeper layers learned smaller slopes. PReLU was also part of the combination of models with which they reported 4.94% top-5 error on the ImageNet 2012 test set.

Mean activation. ReLU outputs are never negative, so their mean is positive. Clevert et al. (2016) argue that pushing mean activations toward zero speeds up learning, and designed the ELU to take negative values that saturate at $-\alpha$ for very negative inputs.

Smoothness. GELU and SiLU are smooth and non-monotonic: they dip slightly below zero for negative inputs before returning to zero. Hendrycks and Gimpel (2016) proposed GELU, which weights the input by $\Phi(z)$ instead of gating it by its sign, and reported gains over ReLU and ELU on vision, language and speech tasks. The function $z\sigma(z)$ appears in three places: Elfwing, Uchibe and Doya proposed it in 2017 for reinforcement learning (published 2018, where it is called SiLU); Ramachandran, Zoph and Le (2017) found $z\sigma(\beta z)$ in an automated search over activation functions and named it Swish, noting that with $\beta = 1$ it is Elfwing et al.’s unit; and later versions (from 2018) of Hendrycks and Gimpel’s paper discuss it as the SiLU and state that they introduced it.

Gated linear units. Some layers multiply two linear projections, one passed through a nonlinearity, for example $\text{SwiGLU}(\mathbf{x}) = \text{SiLU}(\mathbf{x}\mathbf{W}) \odot \mathbf{x}\mathbf{V}$. Shazeer (2020) tested such GLU variants in the feed-forward sublayers of a transformer and found that some of them improved quality over ReLU and GELU.

Examples

The code below uses the 8×8 digit images bundled with scikit-learn. It checks that two filter layers without an activation equal one filter, passes the Sobel responses of all 1,797 digits through each activation (counting zero and negative outputs and units with slope below 0.01, where almost no gradient gets through), shows sigmoid saturation on raw pixel values, and checks each derivative against central finite differences.

import numpy as np
from scipy.signal import convolve2d
from scipy.special import erf
from sklearn.datasets import load_digits

images = load_digits().images / 16.0           # 1,797 digit images, 8x8, values in [0, 1]
sobel_x = np.array([[1, 0, -1], [2, 0, -2], [1, 0, -1]])   # responds to vertical edges
blur = np.array([[1, 2, 1], [2, 4, 2], [1, 2, 1]]) / 16.0  # 3x3 binomial smoothing

# 1. Without a nonlinearity, two filter layers collapse into one.
img = images[0]
two = convolve2d(convolve2d(img, sobel_x), blur)            # Sobel layer, then blur layer
one = convolve2d(img, convolve2d(sobel_x, blur))            # one combined 5x5 filter
print("Sobel then blur == one 5x5 filter:", np.allclose(two, one))
relu_between = convolve2d(np.maximum(0, convolve2d(img, sobel_x)), blur)
print(f"with ReLU between them, max difference: {np.abs(relu_between - one).max():.2f}")

# 2. Activations and their derivatives.
sig = lambda z: 1 / (1 + np.exp(-z))
Phi = lambda z: 0.5 * (1 + erf(z / np.sqrt(2)))             # standard normal CDF
acts = {  # name: (f, f')
    "sigmoid": (sig, lambda z: sig(z) * (1 - sig(z))),
    "tanh":    (np.tanh, lambda z: 1 - np.tanh(z) ** 2),
    "ReLU":    (lambda z: np.maximum(0, z), lambda z: (z > 0) * 1.0),
    "leaky":   (lambda z: np.where(z > 0, z, 0.01 * z), lambda z: np.where(z > 0, 1, 0.01)),
    "ELU":     (lambda z: np.where(z > 0, z, np.expm1(np.minimum(z, 0))),
                lambda z: np.where(z > 0, 1, np.exp(np.minimum(z, 0)))),
    "GELU":    (lambda z: z * Phi(z),
                lambda z: Phi(z) + z * np.exp(-z ** 2 / 2) / np.sqrt(2 * np.pi)),
    "SiLU":    (lambda z: z * sig(z), lambda z: sig(z) * (1 + z * (1 - sig(z)))),
}

# 3. Apply each activation to the Sobel responses of all 1,797 digits.
Z = np.stack([convolve2d(im, sobel_x, mode="valid") for im in images])  # (1797, 6, 6)
print(f"\nSobel responses: {Z.size} values in [{Z.min():.2f}, {Z.max():.2f}], "
      f"{(Z > 0).mean():.0%} positive, {(Z == 0).mean():.0%} exactly zero")
print("activation  zeros  negative   mean  |slope|<0.01")
for name, (f, df) in acts.items():
    A, D = f(Z), df(Z)
    print(f"{name:10s} {(A == 0).mean():6.0%} {(A < 0).mean():9.0%} {A.mean():6.2f} "
          f"{(np.abs(D) < 0.01).mean():13.1%}")

# 4. Sigmoid saturation on raw 8-bit intensities versus standardized ones.
pix = images.ravel() * 255                                  # pixel values 0..255
standardized = (pix - pix.mean()) / pix.std()
for label, x in [("raw 0..255", pix), ("standardized", standardized)]:
    d = acts["sigmoid"][1](x)
    print(f"sigmoid on {label:12s}: median slope {np.median(d):.2g}, "
          f"slope < 0.01 for {(d < 0.01).mean():.0%} of pixels")

# 5. Check every derivative against central finite differences.
z = np.linspace(-4, 4, 801)
z = z[np.abs(z) > 1e-3]                                     # avoid the kink at 0
h = 1e-5
for name, (f, df) in acts.items():
    fd = (f(z + h) - f(z - h)) / (2 * h)
    print(f"{name:8s} max |analytic - numeric| = {np.abs(fd - df(z)).max():.1e}")

Output:

Sobel then blur == one 5x5 filter: True
with ReLU between them, max difference: 1.78

Sobel responses: 64692 values in [-4.00, 4.00], 51% positive, 3% exactly zero
activation  zeros  negative   mean  |slope|<0.01
sigmoid        0%        0%   0.51          0.0%
tanh           3%       46%   0.03         22.0%
ReLU          49%        0%   0.95         49.3%
leaky          3%       46%   0.94          0.0%
ELU            3%       46%   0.60          0.0%
GELU           3%       46%   0.88         10.6%
SiLU           3%       46%   0.77          1.4%
sigmoid on raw 0..255  : median slope 1.2e-07, slope < 0.01 for 51% of pixels
sigmoid on standardized: median slope 0.21, slope < 0.01 for 0% of pixels
sigmoid  max |analytic - numeric| = 1.4e-11
tanh     max |analytic - numeric| = 3.3e-11
ReLU     max |analytic - numeric| = 1.6e-11
leaky    max |analytic - numeric| = 1.6e-11
ELU      max |analytic - numeric| = 1.6e-11
GELU     max |analytic - numeric| = 4.5e-11
SiLU     max |analytic - numeric| = 5.1e-11

ReLU zeroes 49% of the Sobel responses, the 46% negative responses plus the 3% that were already zero, and the same 49% of units pass no gradient: every edge that darkens from left to right is gone. Leaky ReLU and ELU keep the negative responses, and on these responses their slope never drops below 0.01. Tanh flattens the strongest edges, where $|z| > 3$, so 22% of its units have almost no slope. GELU’s slope is below 0.01 for strongly negative responses and in a narrow band around its minimum near $z = -0.75$, where the slope crosses zero. Fed intensities on the 8-bit scale 0 to 255, the sigmoid is flat for any pixel brighter than about 5 out of 255, which covers every non-black pixel, half of the total; the black background sits at $z = 0$, where the slope is 0.25. After standardizing the pixels to zero mean and unit variance, no pixel is in the flat region. Every derivative formula agrees with finite differences to about $10^{-11}$.

Common Misconceptions

• “ReLU’s kink breaks gradient descent.” ReLU is not differentiable at $z = 0$, but Goodfellow et al. (2016, §6.3) explain that software returns one of the one-sided derivatives there, and that in practice the nondifferentiability can be safely disregarded.
• “ReLU has no vanishing-gradient problem at all.” Its gradient is zero for negative inputs, and vanishing gradients also depend on the weights, initialization and architecture.

Where It Is Used

In a vision network, activations follow the convolutional and hidden fully connected layers, while the layer that produces class scores has none; a softmax, which normalizes the whole score vector rather than acting elementwise, may then turn them into probabilities. ReLU was used throughout AlexNet (Krizhevsky et al., 2012), whose authors note that, unlike saturating units, ReLUs do not need normalized inputs to avoid saturation, and it remains the usual choice in convolutional networks. The Vision Transformer of Dosovitskiy et al. (2021) uses GELU in the two-layer MLP of each block, as BERT (Devlin et al., 2019) did for language; EfficientNet (Tan & Le, 2019) uses SiLU. The derivatives in the table are what backpropagation multiplies through each layer when gradient descent trains a network; normalization layers and residual connections also affect how well gradients pass through many layers.