Fully Connected Layer

A fully connected layer, also called a dense layer, a linear layer or an inner-product layer, computes every output as a weighted sum of every input plus a bias. It is the building block of the multilayer perceptron, and for years it ended the standard image classifier: the convolutional layers turned an image into a stack of feature maps, and one or more fully connected layers turned those maps into class scores. In networks such as AlexNet and VGG those last layers held most of the parameters. Later classifiers replaced them with global average pooling and a single linear layer, and the layer’s equivalence to a convolution let fully convolutional networks reuse classifiers for dense prediction. It has not disappeared: every transformer block applies a small stack of fully connected layers to each token, and classifiers such as ResNet still end in one.

Definition

A fully connected layer with $n$ inputs and $m$ outputs is the affine map

$$\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}, \qquad \mathbf{W} \in \mathbb{R}^{m \times n},\ \mathbf{b} \in \mathbb{R}^{m},$$

where every entry $W_{ij}$ is a learned weight connecting input $j$ to output $i$. “Fully connected” means that no entry of $\mathbf{W}$ is fixed to zero and no two entries are tied to each other. The layer usually has an activation function after it, except when it produces the network’s final logits.

Intuition

A fully connected layer makes no assumption about how its inputs are arranged. Each output unit is a template the size of the whole input: the unit responds strongly when the input looks like its row of $\mathbf{W}$. When the input is a flattened image or feature map, each weight belongs to one position and one channel, so a pattern learned at one position says nothing about the same pattern elsewhere. The neural network and multilayer perceptron articles show the consequences on pixels.

Convolution makes the opposite assumption. Goodfellow, Bengio and Courville (2016, §9.2) present it through three ideas, sparse interactions, parameter sharing and equivariant representations (for convolution, equivariance to translation), the first two set against the matrix multiplication of a traditional layer. In a fully connected layer each weight is used exactly once per forward pass. In a convolutional layer one kernel weight is reused at every position. They also note (§9.1) that a convolution can itself be written as multiplication by a matrix, a very sparse one with many entries repeated. The fully connected layer is the case with no zeros and no repetition, the most general linear map.

Formal Definition

For a batch of $N$ inputs stored as the rows of $\mathbf{X} \in \mathbb{R}^{N \times n}$, the layer computes $\mathbf{Y} = \mathbf{X}\mathbf{W}^\top + \mathbf{1}\mathbf{b}^\top$. PyTorch’s nn.Linear(in_features, out_features) states its operation in this form, $y = xA^\top + b$. It stores the weight with shape (out_features, in_features) and initializes weight and bias from $\mathcal{U}(-1/\sqrt{n}, 1/\sqrt{n})$ unless told otherwise (PyTorch 2.14.1 source). Its input may have any number of leading dimensions, $(*, n)$. The same weights are then applied to every vector along the last axis, which is how a transformer applies one layer to every token at once.

Parameters and cost. The layer has $mn$ weights and $m$ biases, so $(n + 1)m$ parameters. One forward pass costs $mn$ multiply-adds per input, one per weight, and the memory needed for the weights grows with the product of the two widths. Goodfellow et al. (2016, §9.2) give the runtime of the matrix multiplication as $O(m \times n)$ per example. This is the reverse of a convolution’s balance. A convolution has few parameters, but each one is used at every position, so it costs many operations.

Gradients. For a loss $L$ and $\boldsymbol{\delta} = \partial L / \partial \mathbf{y}$, the gradient of the weights is the outer product $\boldsymbol{\delta}\mathbf{x}^\top$. The gradient passed back to the input is $\mathbf{W}^\top \boldsymbol{\delta}$. The backpropagation article derives these for stacks of layers.

Flattening. A feature map of shape $C \times H \times W$ must be flattened before a fully connected layer can read it, so $n = CHW$. The order of flattening is part of the layer’s definition. PyTorch’s channels-first tensors flatten channel by channel, while a channels-last layout flattens position by position. Weights trained under one order give a different function under the other unless their columns are permuted to match.

Properties

Parameters grow with the size of the input. A layer that reads the last feature map of a classifier has $CHW \cdot m$ weights. In AlexNet, the first fully connected layer, fc6, reads a $6 \times 6 \times 256$ map, 9,216 values, and has 4,096 outputs. The AlexNet article counts 96.2% of the network’s 61.0 million parameters in its three fully connected layers and 61.9% in fc6 alone. In VGG-16, fc6 is a $25{,}088 \times 4{,}096$ matrix (Girshick, 2015, §4.4), and the multilayer perceptron article finds 89.4% of the 138.4 million parameters in the three fully connected layers.

Few operations for those parameters. Each weight is used once per image, so in a classifier the head is cheap to run. By the VGG article’s count, VGG-16’s fully connected layers take 0.12 billion of its 15.47 billion multiply-adds, under 1%. The balance changes when the head runs many times per image. In Fast R-CNN, the fully connected layers are evaluated once per region of interest. Girshick (2015, §3.1) found that nearly half of the detection forward pass was spent in them, and compressed them with a truncated singular value decomposition. That replaces a $u \times v$ weight matrix by two thinner layers with no nonlinearity between them, cutting $uv$ parameters to $t(u + v)$. Keeping the top 1,024 singular values of VGG16’s fc6 and the top 256 of its $4{,}096 \times 4{,}096$ fc7 cut detection time by more than 30% for a 0.3-point drop in mAP on PASCAL VOC 2007, without fine-tuning again (§4.4, Table 4). The Fast R-CNN and model compression articles reproduce this at small scale.

A fixed input size. Because $n = CHW$ is fixed, a network with a fully connected head accepts one input size. Long, Shelhamer and Darrell (2015, §3.1) put it this way: the fully connected layers of LeNet, AlexNet and their successors have fixed dimensions and throw away spatial coordinates.

Equivalence to convolutions. Two equivalences follow from the definition:

• A fully connected layer that reads a whole $k \times k \times C$ map is a $k \times k$ convolution with $m$ filters and no padding, evaluated at the single position where the kernel fits. Its weight matrix, reshaped to $m \times k \times k \times C$ in the flattening order, is the kernel. Long et al. (2015, §3.1) describe fully connected layers as convolutions with kernels that cover their entire input regions.
• A fully connected layer applied separately to the $C$-vector at every position of a feature map is a $1 \times 1$ convolution with $m$ filters. It mixes channels and never mixes positions.

Rewritten this way, a classifier accepts larger inputs and returns a grid of class scores. Each score equals the original network’s output on one window, and the windows are spaced by the network’s total stride. Both the VGG article, for VGG’s dense evaluation, and the FCN article work through this conversion. The FCN article’s code checks it on every window of a feature map.

Examples

Classifier heads up to 2014. The heads of the classic ImageNet classifiers differ in depth but share the pattern. The LeNet article tabulates LeNet-5’s fully connected layer F6, 84 units with 10,164 parameters, followed by an output layer of radial basis function units rather than another linear layer. AlexNet and VGG end in three fully connected layers of 4,096, 4,096 and 1,000 units, with dropout on the first two, as their articles describe. Szegedy et al. (2015, §2) summarize the standard design of the time as stacked convolutional layers followed by one or more fully connected layers.

Global average pooling. Network in Network (Lin, Chen & Yan, 2014) avoided the fully connected head. As Zhou et al. (2016) summarize it, it used global average pooling, which averages each final channel over all positions, and this acts as a structural regularizer that prevents overfitting. GoogLeNet based its average pooling before the classifier on Network in Network, but its authors note that their implementation has an additional linear layer, used mainly so that the network can be adapted to other label sets. Szegedy et al. (2015, §5) found that moving from fully connected layers to average pooling improved top-1 accuracy by about 0.6%, and that dropout remained essential even without the fully connected layers. Their Table 1 lists a $7 \times 7$ average pool producing $1 \times 1 \times 1024$, then a linear layer with 1,000K parameters. GoogLeNet as a whole used 12 times fewer parameters than Krizhevsky et al.’s winning network of two years earlier (§1). ResNet ends the same way, with global average pooling and a 1,000-way fully connected layer (He et al., 2016, §3.3).

On a $C$-channel map, global average pooling plus a linear layer has $1{,}000C + 1{,}000$ parameters for 1,000 classes: 257,000 for AlexNet’s 256 channels and 513,000 for VGG’s 512. Its size does not depend on the map’s height and width. Average pooling commutes with the linear layer, because both are linear. Applying the class weights at every position first, as a $1 \times 1$ convolution, and averaging afterwards gives the same logits. Zhou et al. (2016, §2) used this to define the class activation map: the per-position class scores before averaging, which show where the evidence for a class lies. Class activation maps and their gradient-based successor, Grad-CAM, belong to neural network interpretability.

The NumPy script below (NumPy 2.3.5, scikit-image 0.25.2; it runs in under a second) checks both parts of the global-average-pooling story. It computes 64-channel feature maps of scikit-image’s bundled astronaut photograph with a fixed, untrained network: three stages of $3 \times 3$ convolution, ReLU and $2 \times 2$ average pooling, all with seeded random weights. First, it compares a global-average-pooling head with the class map of its own weights at two input sizes, and with Zhou et al.’s summed form of the pooling. Second, it checks Zhou et al.’s statement (§3.1) that removing the fully connected layers gives “90% less parameters for VGGnet”. It builds their GAP networks as described there: layers after conv5 (AlexNet) or conv5-3 (VGG) removed, a $3 \times 3$ convolution with 1,024 units added, then global average pooling and a 1,000-way fully connected layer. The paper does not say which VGG it used, or whether its AlexNet kept the original two-GPU grouping, so the script counts VGG-16 and VGG-19, and AlexNet with and without the grouping of conv2, conv4 and conv5.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view
from skimage import data
from skimage.transform import resize

rng = np.random.default_rng(0)
relu = lambda z: np.maximum(z, 0)

def conv(x, K, b, pad=0):
    """x: h x w x C_in, K: C_out x k x k x C_in -> h' x w' x C_out (stride 1)."""
    k = K.shape[1]
    if pad:
        x = np.pad(x, ((pad, pad), (pad, pad), (0, 0)))
    win = sliding_window_view(x, (k, k), axis=(0, 1))
    return np.einsum("ijcuv,ouvc->ijo", win, K) + b

# A fixed, untrained feature extractor: three stages of 3x3 conv, ReLU, 2x2 average pool.
widths = [3, 16, 32, 64]
stages = [(rng.standard_normal((co, 3, 3, ci)) * np.sqrt(2 / (9 * ci)), np.zeros(co))
          for ci, co in zip(widths[:-1], widths[1:])]

def features(img):
    x = img
    for K, b in stages:
        x = relu(conv(x, K, b, pad=1))
        h, w, c = x.shape
        x = x.reshape(h // 2, 2, w // 2, 2, c).mean((1, 3))
    return x

# 1. GAP + linear == 1x1-conv class map, then GAP (Zhou et al., eq. 1-2)
V = rng.standard_normal((10, 64)) / 8; c = rng.standard_normal(10) * 0.1
photo = data.astronaut() / 255.0
for side in (224, 320):
    F = features(resize(photo, (side, side)))
    logits = V @ F.mean((0, 1)) + c                     # GAP, then the linear layer
    M = F @ V.T + c                                     # the linear layer at every position (1x1 conv)
    gap_sum = V @ F.sum((0, 1))                         # Zhou's sum form, no bias
    print(f"E: {side}x{side} -> features {F.shape}, positive {np.mean(F > 0):.2f}; "
          f"|logits - mean of class map| {np.abs(logits - M.mean((0, 1))).max():.1e}; "
          f"|sum-form / (h*w) - mean-form| {np.abs(gap_sum / (F.shape[0] * F.shape[1]) - (logits - c)).max():.1e}")

# 2. Whole networks vs their GAP versions as Zhou et al. (2016, §3.1) build them
conv_p = lambda k, ci, co, g=1: k * k * ci // g * co + co
fc = lambda i, o: i * o + o
fc_head = lambda s, ch: fc(s * s * ch, 4096) + fc(4096, 4096) + fc(4096, 1000)
gap_head = lambda ch: conv_p(3, ch, 1024) + fc(1024, 1000)   # added 3x3 conv, 1024 units; GAP; 1000-way FC
vgg = {"VGG-16": [(3,64),(64,64),(64,128),(128,128),(128,256),(256,256),(256,256),(256,512),
                  (512,512),(512,512),(512,512),(512,512),(512,512)]}
vgg["VGG-19"] = vgg["VGG-16"][:7] + [(256,256)] + vgg["VGG-16"][7:10] + [(512,512)] + vgg["VGG-16"][10:] + [(512,512)]
convs = {name: sum(conv_p(3, a, b) for a, b in L) for name, L in vgg.items()}
for g, name in [(2, "AlexNet two-tower"), (1, "AlexNet ungrouped")]:
    convs[name] = (conv_p(11, 3, 96) + conv_p(5, 96, 256, g) + conv_p(3, 256, 384)
                   + conv_p(3, 384, 384, g) + conv_p(3, 384, 256, g))
for name, cv in convs.items():
    s = 6 if name.startswith("Alex") else 7
    orig, gapnet = cv + fc_head(s, 256 if s == 6 else 512), cv + gap_head(256 if s == 6 else 512)
    print(f"F: {name:18s} original {orig:12,}  GAP version {gapnet:11,}  "
          f"fewer parameters {1 - gapnet / orig:.1%}")

# Added in fact-checking: the same networks without the added 3x3 conv (GAP, then one C -> 1000 layer)
for name, cv in convs.items():
    s, ch = (6, 256) if name.startswith("Alex") else (7, 512)
    orig, fc_only = cv + fc_head(s, ch), cv + fc(ch, 1000)
    print(f"G: {name:18s} without the added conv {fc_only:11,}  fewer parameters {1 - fc_only / orig:.1%}")

Output:

E: 224x224 -> features (28, 28, 64), positive 0.62; |logits - mean of class map| 4.4e-16; |sum-form / (h*w) - mean-form| 1.1e-16
E: 320x320 -> features (40, 40, 64), positive 0.61; |logits - mean of class map| 8.9e-16; |sum-form / (h*w) - mean-form| 2.2e-16
F: VGG-16             original  138,357,544  GAP version  20,459,304  fewer parameters 85.2%
F: VGG-19             original  143,667,240  GAP version  25,769,000  fewer parameters 82.1%
F: AlexNet two-tower  original   60,965,224  GAP version   5,719,400  fewer parameters 90.6%
F: AlexNet ungrouped  original   62,378,344  GAP version   7,132,520  fewer parameters 88.6%
G: VGG-16             without the added conv  15,227,688  fewer parameters 89.0%
G: VGG-19             without the added conv  20,537,384  fewer parameters 85.7%
G: AlexNet two-tower  without the added conv   2,591,080  fewer parameters 95.7%
G: AlexNet ungrouped  without the added conv   4,004,200  fewer parameters 93.6%

The E lines show the identity exactly, up to floating-point rounding. A head made of global average pooling and a linear layer gives the same logits as its $1 \times 1$-convolution class map averaged over positions. The same ten-class head runs unchanged on a $28 \times 28$ and a $40 \times 40$ map, because nothing in it depends on the number of positions. The weights are random, so the class maps here show no object. The check concerns the arithmetic, which holds for any weights.

The second number on each E line concerns a wording difference in the source. Zhou et al.’s text says that global average pooling outputs the spatial average of each map, but their formula defines the pooled value as the sum $F_k = \sum_{x,y} f_k(x, y)$ (§2). The two differ only by the factor $hw$, the number of positions, as the check confirms. With a fixed input size, that factor can be absorbed into the linear layer’s weights. With varying input sizes, the summed form scales the logits with the area of the map and the averaged form does not.

The F lines reproduce the published totals for the original networks: 138,357,544 for VGG-16, 143,667,240 for VGG-19, and 60,965,224 for AlexNet with its two-GPU grouping, the counts the VGG and AlexNet articles give. For the networks as Zhou et al. built them, the “90%” is reached or nearly reached for AlexNet (90.6% with the grouping, 88.6% without) but for neither VGG reading: 85.2% for VGG-16 and 82.1% for VGG-19. In our reading of §3.1, the paper’s sentence comes before its description of the $3 \times 3$, 1,024-unit convolution added before the pooling, so it may refer to removing the fully connected layers alone. That convolution has $3 \cdot 3 \cdot 512 \cdot 1{,}024 + 1{,}024 = 4{,}719{,}616$ parameters on VGG’s 512 channels. The G lines count that second reading, with global average pooling and a single 1,000-way layer directly after the original convolutions; this case was added during fact-checking. Under it, VGG-16 comes close to the figure (89.0%), VGG-19 still falls short (85.7%), and both AlexNet readings exceed it. The paper gives the figure in passing (“e.g.”), with no count behind it. The removal also cost accuracy. On the ILSVRC validation set, Zhou et al.’s Table 1 gives 33.4% top-1 and 12.2% top-5 error for VGGnet-GAP against 31.2% and 11.4% for VGGnet. AlexNet-GAP lost most, 51.1% against 42.6% top-1, and adding two convolutional layers before the pooling (AlexNet*-GAP) brought it to 44.9%.

Fully convolutional networks. Long et al. (2015, §4.1) decapitated AlexNet, VGG-16 and GoogLeNet by discarding the final classifier layer, converted all fully connected layers to convolutions, and appended a $1 \times 1$ convolution with 21 outputs to score the PASCAL classes at every coarse position. The convolutionalized AlexNet produced a $10 \times 10$ grid of outputs from a $500 \times 500$ image in 22 ms, against 1.2 ms for one $227 \times 227$ image. They put this at more than five times faster than classifying the patches one by one, because overlapping windows share computation (§3.1). The FCN article covers the rest of the architecture.

Transformer blocks. Each block of a transformer or Vision Transformer contains a position-wise MLP: two fully connected layers with a nonlinearity between them, applied to every token with the same weights. With the usual fourfold expansion it holds about two thirds of a block’s weights, as the transformer article counts. On a grid of image patch tokens this is again a stack of $1 \times 1$ convolutions. All mixing between tokens happens in self-attention.

Common Misconceptions

• “Global average pooling removes the fully connected layer.” In GoogLeNet, ResNet and most later classifiers, a fully connected layer still maps the pooled vector to the logits. What disappeared is the large layer that reads every position of the final map. GoogLeNet’s authors describe that remaining linear layer as an addition to Network in Network’s design.
• “A dense layer and a convolution are different operations.” A dense layer on a whole $k \times k$ map is a $k \times k$ convolution at one position, and a dense layer applied per position is a $1 \times 1$ convolution. The difference lies in which weights exist and which are shared, not in the arithmetic of a single output.
• “Most of the parameters means most of the compute.” In AlexNet and VGG the fully connected layers hold most of the parameters but take a small fraction of the multiply-adds. The share of compute only grows when the head runs many times per image, as with Fast R-CNN’s regions of interest.
• “Flattening order does not matter.” It fixes which weight multiplies which channel at which position. Moving weights between channels-first and channels-last layouts, or reshaping them into a convolution kernel, requires the same order.

Where It Is Used

• Classifier heads on top of a backbone network, usually after global average pooling. In transfer learning and fine-tuning this last layer is the part replaced for a new label set, the convenience Szegedy et al. (2015) gave for keeping it.
• Multilayer perceptrons, where fully connected layers make up the whole network, including the coordinate networks described in the multilayer perceptron article.
• Transformer blocks, as the position-wise MLP and as the query, key, value and output projections of attention, which are fully connected layers applied to each token.
• Detection and segmentation heads. These include the per-region heads of Fast R-CNN and the $1 \times 1$ convolutions that score classes at every position in FCN.
• Shared per-element maps. Other architectures apply one fully connected layer with shared weights to many elements: PointNet’s per-point MLP, the channel gates of squeeze-and-excitation blocks, and the node updates of graph neural networks.