Neural Network

A neural network is a function assembled from many simple units. Each unit takes a weighted sum of its inputs, adds a bias, and passes the result through a fixed nonlinear function; units are arranged in layers, and each layer feeds the next. The weights and biases are learned from examples rather than designed. For an image classifier, the input is the image’s pixels and the output is one score per class. Almost every modern vision model, from convolutional networks to vision transformers, is a neural network in this sense.

Core Idea

A classical vision pipeline computes hand-designed features, such as gradient histograms or feature descriptors, and feeds them to a classifier. A neural network replaces both stages with one parameterized function

$$f(\mathbf{x}; \theta) = \text{scores},$$

where $\mathbf{x}$ holds the pixels and $\theta$ collects all the weights and biases. Training searches for the $\theta$ that makes the scores agree with labeled examples. LeCun, Bengio and Hinton (2015) describe the result for images: the first learned layer typically responds to oriented edges at particular positions, later layers to arrangements of edges and then to object parts, none of it engineered by hand.

Building Blocks

The artificial neuron. A neuron, or unit, receives inputs $x_1, \dots, x_n$ and computes

$$a = \phi\Big(\sum_{i=1}^{n} w_i x_i + b\Big) = \phi(\mathbf{w}^\top \mathbf{x} + b),$$

with one weight $w_i$ per input, a bias $b$, and an activation function $\phi$. The weighted sum $z = \mathbf{w}^\top \mathbf{x} + b$ measures how well the input matches the pattern $\mathbf{w}$, which for an image input can be displayed as an image, a template; thresholding $z$ gives a linear classifier. The bias shifts the threshold.

The activation function. The nonlinearity $\phi$ is applied elementwise. LeCun et al. (2015) name the rectified linear unit (ReLU), $\phi(z) = \max(0, z)$, as the most popular choice, replacing the logistic sigmoid and $\tanh$ of earlier networks. The activation function article compares the options.

Layers. A layer is a set of units that all read the same input. A fully connected (or dense) layer connects every unit to every input. Layers between the input and the output are hidden layers, and a network of fully connected layers is often called a multilayer perceptron (MLP). The number of units in a layer is its width; the number of layers is the network’s depth.

How Data Flows

A fully connected network first flattens the image: an $8 \times 8$ grayscale image becomes a vector of 64 numbers, and a $224 \times 224$ color image becomes a vector of $224 \cdot 224 \cdot 3 = 150{,}528$ numbers. The vector passes through each layer in turn, and the last layer produces one score per class. Computing the output from the input this way is the forward pass.

To read the scores as probabilities, the softmax function exponentiates and normalizes them,

$$p_k = \frac{\exp(s_k)}{\sum_{j} \exp(s_j)},$$

so that the $p_k$ are positive and sum to one; the predicted class is the one with the largest score.

The example below runs the forward pass of a two-layer network (64 inputs, 32 hidden ReLU units, 10 outputs) on the 1,797 handwritten-digit images bundled with scikit-learn, each $8 \times 8$ pixels with values from 0 to 16. The weights are random, so the network has learned nothing yet.

import numpy as np
from sklearn.datasets import load_digits

digits = load_digits()                      # 1,797 grayscale images, 8x8 pixels, values 0..16
X = digits.images.reshape(-1, 64) / 16.0    # flatten each image to a 64-vector in [0, 1]
y = digits.target                           # labels 0..9
print("images:", digits.images.shape, "-> inputs:", X.shape)

rng = np.random.default_rng(1)
n_in, n_hidden, n_out = 64, 32, 10
W1 = rng.normal(0, 1 / np.sqrt(n_in), (n_hidden, n_in))      # layer 1: 64 -> 32
b1 = np.zeros(n_hidden)
W2 = rng.normal(0, 1 / np.sqrt(n_hidden), (n_out, n_hidden))  # layer 2: 32 -> 10
b2 = np.zeros(n_out)
n_params = W1.size + b1.size + W2.size + b2.size
print(f"parameters: {W1.size} + {b1.size} + {W2.size} + {b2.size} = {n_params}")

def forward(X):
    H = np.maximum(0, X @ W1.T + b1)        # hidden layer: weighted sums, then ReLU
    return H @ W2.T + b2                    # output layer: one score per class

def softmax(S):
    E = np.exp(S - S.max(axis=1, keepdims=True))
    return E / E.sum(axis=1, keepdims=True)

S = forward(X)
P = softmax(S)
print("scores:", S.shape)
np.set_printoptions(precision=2, suppress=True)
for i in range(3):
    print(f"image {i} (label {y[i]}): probabilities {P[i]} -> predicts {P[i].argmax()}")
print(f"accuracy of the untrained network: {(S.argmax(axis=1) == y).mean():.3f}")

Output:

images: (1797, 8, 8) -> inputs: (1797, 64)
parameters: 2048 + 32 + 320 + 10 = 2410
scores: (1797, 10)
image 0 (label 0): probabilities [0.08 0.13 0.11 0.09 0.08 0.11 0.12 0.08 0.12 0.09] -> predicts 1
image 1 (label 1): probabilities [0.07 0.1  0.1  0.08 0.07 0.19 0.16 0.05 0.14 0.04] -> predicts 5
image 2 (label 2): probabilities [0.06 0.11 0.08 0.08 0.05 0.25 0.14 0.06 0.12 0.05] -> predicts 5
accuracy of the untrained network: 0.100

Each row of X is an image and each row of S its ten scores, so the whole dataset passes through in one matrix product per layer. With random weights the probabilities are spread over all classes and the accuracy is 10%, chance level for ten classes (over 200 random seeds the median was 10.0%, with 80% of seeds between 6% and 15%). Everything useful must come from training the 2,410 parameters.

The same network in PyTorch (run with PyTorch 2.14.1) is three modules in a torch.nn.Sequential. A torch.nn.Linear layer stores its weight with shape (outputs, inputs), the same layout as W1 and W2 above, so the NumPy arrays can be loaded directly. Continuing from the code above:

import torch
from torch import nn

net = nn.Sequential(
    nn.Linear(64, 32),   # weight shape (32, 64), like W1
    nn.ReLU(),
    nn.Linear(32, 10),   # weight shape (10, 32), like W2
).double()               # float64, to match NumPy
print("parameters:", sum(p.numel() for p in net.parameters()))

state = {"0.weight": W1, "0.bias": b1, "2.weight": W2, "2.bias": b2}
net.load_state_dict({k: torch.from_numpy(v) for k, v in state.items()})

with torch.no_grad():
    S_torch = net(torch.from_numpy(X)).numpy()
print(f"max |PyTorch - NumPy| score difference: {np.abs(S_torch - S).max():.1e}")

Output:

parameters: 2410
max |PyTorch - NumPy| score difference: 0.0e+00

The two compute the same function (in general, expect agreement to rounding error rather than exact zeros). The framework adds automatic gradients and GPU execution, not new mathematics.

Mathematical Formulation

Stacking the weights of a layer’s $m$ units as the rows of a matrix $\mathbf{W} \in \mathbb{R}^{m \times n}$ and their biases in $\mathbf{b} \in \mathbb{R}^m$, a fully connected layer computes all its units at once:

$$\mathbf{h} = \phi(\mathbf{W}\mathbf{x} + \mathbf{b}).$$

A network with $L$ layers repeats this, starting from $\mathbf{h}^{(0)} = \mathbf{x}$:

$$\mathbf{h}^{(\ell)} = \phi\big(\mathbf{W}^{(\ell)} \mathbf{h}^{(\ell-1)} + \mathbf{b}^{(\ell)}\big), \quad \ell = 1, \dots, L-1, \qquad \mathbf{s} = \mathbf{W}^{(L)} \mathbf{h}^{(L-1)} + \mathbf{b}^{(L)}.$$

The output layer has no activation; it produces the class scores $\mathbf{s}$. For images stored as the rows of $\mathbf{X}$, as in the code, a layer computes $\phi(\mathbf{X}\mathbf{W}^\top + \mathbf{b}^\top)$.

Why the nonlinearity matters. Without $\phi$, two layers collapse into one: $\mathbf{W}_2(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 = (\mathbf{W}_2\mathbf{W}_1)\mathbf{x} + (\mathbf{W}_2\mathbf{b}_1 + \mathbf{b}_2)$, an affine function. By induction, a network of any depth without nonlinearities is no more expressive than a linear classifier.

Counting parameters. A fully connected layer from $n$ inputs to $m$ units has $mn$ weights and $m$ biases. In the digit network, the first layer has $32 \cdot 64 + 32 = 2{,}080$ parameters and the second $10 \cdot 32 + 10 = 330$. The count grows with the image size: one fully connected layer of 1,000 units on a $224 \times 224 \times 3$ image needs $150{,}528 \cdot 1{,}000 = 150{,}528{,}000$ weights before any further layers. By comparison, a convolution layer with 64 filters of size $3 \times 3$ over three color channels has $64 \cdot 27 + 64 = 1{,}792$ parameters, whatever the image size, because each filter’s weights are reused at every position. This is one reason image networks are built from convolutions.

Why It Works

Universal approximation. Cybenko (1989) proved that for any continuous sigmoidal $\sigma$ (one that tends to 1 as $t \to +\infty$ and to 0 as $t \to -\infty$), finite sums

$$G(\mathbf{x}) = \sum_{j=1}^{N} \alpha_j\, \sigma(\mathbf{y}_j^\top \mathbf{x} + \theta_j)$$

are dense in the continuous functions on the unit hypercube $[0, 1]^n$: for any continuous $f$ and any $\varepsilon > 0$ there is such a sum with $|G(\mathbf{x}) - f(\mathbf{x})| < \varepsilon$ everywhere on the cube. This $G$ is exactly a network with one hidden layer of $N$ sigmoid units and a linear output. Hornik, Stinchcombe and White (1989, Theorem 2.4) independently proved uniform approximation on any compact set for any “squashing” function (any nondecreasing function with limits 0 and 1), and approximation of any Borel measurable function in a weaker sense: for any probability distribution of the inputs, some network is off by more than $\varepsilon$ only on inputs of total probability below $\varepsilon$. Goodfellow, Bengio and Courville (2016, §6.4.1) note that Leshno et al. (1993) extended universal approximation to a broad family of non-polynomial activations, including the ReLU.

The theorems prove that a good approximation exists; they do not bound $N$, which Cybenko suspected would be astronomically large for most problems, and say nothing about whether training finds the weights or whether the network generalizes to new images.

Depth versus width. A wide single hidden layer is enough in principle, but depth is often more efficient. Goodfellow et al. (2016, §6.4–6.4.1) review results showing that some functions a deep network represents compactly need exponentially many units in a shallow one, and note that deeper networks often need far fewer parameters and frequently generalize better, but tend to be harder to optimize. For images, depth matches the edges-to-parts-to-objects hierarchy described above. A network with too little capacity (informally, the range of functions it can represent) cannot fit the data; one with too much can memorize the training images instead of generalizing, the subject of regularization.

Training Considerations

Training chooses $\theta$ so that the outputs match labeled examples. A loss function measures the mismatch for one image, for example the cross-entropy between the softmax probabilities and the true label; gradient descent repeatedly moves all parameters a small step against the gradient of the average loss; and backpropagation computes that gradient efficiently by applying the chain rule backward through the layers. Each has its own article. The example draws weights with standard deviation $1/\sqrt{n}$ for $n$ inputs. Our reasoning for this simple choice: variances of independent terms add, so with weight variance $1/n$ the weighted sum has variance equal to the average squared input, a scale that does not grow with the number of inputs.

Variants

The fully connected network is the simplest feedforward network: information flows from input to output without loops. Convolutional networks connect each unit to a small image neighborhood and share weights across positions. Transformers split an image into patches that exchange information through self-attention. Recurrent networks feed outputs back as inputs, mainly for sequences.

Use in Computer Vision

Fully connected layers are still used on compact inputs, for example as the final classifier on features computed by a convolutional backbone, but rarely on the raw pixels of realistic images. Besides their parameter count, they treat the input as an unordered list: shuffling the pixels of every training and test image by the same permutation gives the network an equivalent problem, so it gets no built-in notion that neighboring pixels are related or that a pattern means the same thing wherever it appears. Convolutional networks build both assumptions in.

History. McCulloch and Pitts (1943) modeled neurons as all-or-none threshold units and showed that networks of them could be described by propositional logic; their connections were fixed, not learned. Rosenblatt (1958) proposed the perceptron, in which a “retina” of sensory units feeds threshold units whose influence on the output responses is strengthened or weakened by reinforcement, so that the system learns to classify optical patterns. Goodfellow et al. (2016, §1.2.1) call it the first model that could learn its weights from examples. Such linear models cannot learn even the XOR of two binary inputs, and the authors note that criticism of these limits by Minsky and Papert (1969) contributed to the first decline in neural network research. Rumelhart, Hinton and Williams (1986) showed that back-propagation could train networks with hidden units, which came to represent useful features of the task; Goodfellow et al. credit it, among others, with popularizing the method. Deep learning reached mainstream computer vision in 2012: according to LeCun et al. (2015), a deep convolutional network (AlexNet) trained on about a million ImageNet images almost halved the error rates of the best competing approaches in the ImageNet competition.

Limitations

• Data and compute. Many parameters need many labeled images and much computation; with too few images, the network overfits.
• Opacity. The learned function is spread over many parameters, so individual predictions are hard to explain.
• Sensitivity. Szegedy et al. (2014) showed that a trained network can be made to misclassify an image by adding a perturbation too small to notice, chosen to maximize the network’s prediction error.