Feature Map

A feature map is the output of one filter of a convolutional layer: a two-dimensional grid with one value per position, recording how strongly the pattern that filter detects is present there. A layer with 64 filters produces 64 feature maps, stacked as the channels of its output, and the next layer reads all of them. The maps of a trained network’s intermediate layers are the image features that detectors and segmenters are built on.

Definition

A convolutional layer with $C_\text{out}$ filters turns an input of shape $(N, C_\text{in}, H_\text{in}, W_\text{in})$ into an output of shape $(N, C_\text{out}, H_\text{out}, W_\text{out})$, in the layout PyTorch’s Conv2d documentation uses: batch, channels, height, width. A feature map is one channel of that output for one image, the $H_\text{out} \times W_\text{out}$ slice $y[n, c, :, :]$. It is also called an activation map.

Usage varies. Zeiler and Fergus (2014) use the term for single channels: their network’s first layer produces “96 different 55 by 55 element feature maps”. Lin et al. (2017), in the feature pyramid network paper, call a whole multi-channel stage output a feature map. The context usually makes clear which is meant; this article uses the first sense and calls the whole output the layer’s feature maps.

Intuition

Slide a small template for a thin vertical line over an image and record, at each position, how well the patch matches: the result is a map of where vertical lines are. A template for horizontal lines gives a second map of the same image. Each filter asks one question at every position, and its feature map holds the answers.

The next layer’s filters span all channels of their input, so they combine maps: a filter that responds where the vertical-line and horizontal-line maps are both active near each other detects corners. Stacking layers builds detectors for larger and more specific patterns from the maps below.

Formal Definition

For one image, filter $c$ of a layer with weights $w_c$ (shape $C_\text{in} \times k \times k$) and bias $b_c$ produces the feature map

$$y_c = \phi\!\left(b_c + \sum_{c'=1}^{C_\text{in}} w_{c,c'} \star x_{c'}\right),$$

where $\star$ is the cross-correlation computed by deep-learning libraries (see convolution), $x_{c'}$ are the input’s channels, and $\phi$ is the activation function when the map is taken after it.

Its spatial size is set by the kernel size $k$, stride $s$, padding $p$ and dilation $d$. PyTorch’s documentation gives, for the height (the width is analogous),

$$H_\text{out} = \left\lfloor \frac{H_\text{in} + 2p - d\,(k - 1) - 1}{s} \right\rfloor + 1 .$$

With $d = 1$ this is $\lfloor (H_\text{in} + 2p - k)/s \rfloor + 1$, the form given in the convolutional neural network article. The number of maps is the number of filters, $C_\text{out}$, a design choice independent of the image.

Properties

Each map is one detector applied everywhere. All positions share one filter, so a map responds to its pattern wherever it occurs.

Maps become more abstract with depth. Zeiler and Fergus (2014) projected the top nine activations of individual feature maps of an ImageNet-trained network back to pixel space. In their visualizations the second layer responds to corners and other edge and color conjunctions, the third to textures such as mesh patterns and text, the fourth to class-specific parts such as dog faces and bird legs, and the fifth to whole objects such as keyboards and dogs across varied poses. They also found that the lower layers’ maps settled within a few epochs of training, while the upper layers took 40 to 50 epochs to develop. To measure what each depth encodes, they trained classifiers on features taken after different layers: on Caltech-101 (30 training images per class) a linear SVM reached 44.8% accuracy with layer-1 features and 86.2% with layer-5 features (their Table 6). The text says the best results came from using all layers; the table agrees for Caltech-256, but on Caltech-101 the layer-7 SVM scored 85.5%, slightly below layer 5.

Resolution falls as depth rises. Each stride-2 layer or pooling step roughly halves a map’s height and width, so deep maps are coarse: in Zeiler and Fergus’s eight-layer network, layer 1 outputs 96 maps of $55 \times 55$ after pooling, and layer 5 outputs 256 maps of $13 \times 13$, pooled to $6 \times 6$ (9,216 numbers) for the fully connected layers. A map’s stride relative to the input, 4, 8, 16 or 32 pixels for the four stages of a ResNet that FPN uses, is the spacing of its cells on the image. Networks usually add channels as they lose resolution; how stride also widens what each cell sees is covered under receptive field.

Depth and resolution pull in opposite directions. Lin et al. (2017) describe the trade-off directly: high-resolution maps carry low-level features that harm their capacity for object recognition, while a deep map, subsampled more times, is semantically stronger but less precisely localized. Small objects may cover only a cell or two of a stride-32 map.

Examples

The NumPy code below builds a $12 \times 12$ image with two one-pixel-wide vertical bars (at columns 4 and 7) and one horizontal bar (row 8), and applies a layer of two $3 \times 3$ filters, one for vertical and one for horizontal bright lines, followed by ReLU. It prints the shapes in $(N, C, H, W)$ layout, both feature maps, the output sizes for other strides and paddings, and the maps at stride 2.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

def conv2d(x, w, stride=1, pad=0):
    """x: (N, C_in, H, W); w: (C_out, C_in, k, k). Cross-correlation, no bias."""
    x = np.pad(x, ((0, 0), (0, 0), (pad, pad), (pad, pad)))
    win = sliding_window_view(x, w.shape[2:], axis=(2, 3))[:, :, ::stride, ::stride]
    return np.einsum("nchwij,ocij->nohw", win, w)   # (N, C_out, H_out, W_out)

# A 12x12 image with three one-pixel-wide bright bars on a dark background.
img = np.zeros((1, 1, 12, 12))                       # (N, C, H, W)
img[0, 0, 1:6, 4] = 1                                # vertical bar, column 4 (even)
img[0, 0, 1:6, 7] = 1                                # vertical bar, column 7 (odd)
img[0, 0, 8, 2:10] = 1                               # horizontal bar, row 8

line = np.array([-1.0, 2.0, -1.0])                   # bright centre, dark sides
w_vert = np.tile(line, (3, 1))                       # responds to vertical lines
w = np.stack([w_vert, w_vert.T])[:, None]            # (C_out=2, C_in=1, 3, 3)

def show(fmap):
    for row in fmap.astype(int):
        print("   " + " ".join("." if v == 0 else str(v) for v in row))

y = np.maximum(conv2d(img, w, stride=1, pad=1), 0)   # convolution + ReLU
print("input", img.shape, "weights", w.shape, "-> feature maps", y.shape)
for c, name in enumerate(["vertical-line detector", "horizontal-line detector"]):
    print(f"channel {c} ({name}):")
    show(y[0, c])

# Output size: floor((H + 2p - k) / s) + 1, with dilation 1
for k, s, p in [(3, 1, 1), (3, 1, 0), (3, 2, 1)]:
    out = conv2d(img, w, stride=s, pad=p)
    print(f"k={k} s={s} p={p}: maps {out.shape}, formula {(12 + 2*p - k)//s + 1}")

y2 = np.maximum(conv2d(img, w, stride=2, pad=1), 0)  # same filters, stride 2
print("stride 2, channel 0 (vertical-line detector):")
show(y2[0, 0])
print("stride 2, channel 1 (horizontal-line detector):")
show(y2[0, 1])

Output:

input (1, 1, 12, 12) weights (2, 1, 3, 3) -> feature maps (1, 2, 12, 12)
channel 0 (vertical-line detector):
   . . . . 2 . . 2 . . . .
   . . . . 4 . . 4 . . . .
   . . . . 6 . . 6 . . . .
   . . . . 6 . . 6 . . . .
   . . . . 6 . . 6 . . . .
   . . . . 4 . . 4 . . . .
   . . . . 2 . . 2 . . . .
   . . 1 . . . . . . 1 . .
   . . 1 . . . . . . 1 . .
   . . 1 . . . . . . 1 . .
   . . . . . . . . . . . .
   . . . . . . . . . . . .
channel 1 (horizontal-line detector):
   . . . . . . . . . . . .
   . . . 1 1 1 1 1 1 . . .
   . . . . . . . . . . . .
   . . . . . . . . . . . .
   . . . . . . . . . . . .
   . . . 1 1 1 1 1 1 . . .
   . . . . . . . . . . . .
   . . . . . . . . . . . .
   . 2 4 6 6 6 6 6 6 4 2 .
   . . . . . . . . . . . .
   . . . . . . . . . . . .
   . . . . . . . . . . . .
k=3 s=1 p=1: maps (1, 2, 12, 12), formula 12
k=3 s=1 p=0: maps (1, 2, 10, 10), formula 10
k=3 s=2 p=1: maps (1, 2, 6, 6), formula 6
stride 2, channel 0 (vertical-line detector):
   . . 2 . . .
   . . 6 . . .
   . . 6 . . .
   . . 2 . . .
   . 1 . . . .
   . . . . . .
stride 2, channel 1 (horizontal-line detector):
   . . . . . .
   . . . . . .
   . . . . . .
   . . . . . .
   . 4 6 6 6 2
   . . . . . .

One image and two filters give two feature maps of the same size. Channel 0 peaks (6) along both vertical bars and channel 1 along the horizontal bar; each gives only a weak response (1) at the ends of the other orientation’s bars, where its template partly fits. Padding 1 keeps $12 \times 12$, no padding gives $10 \times 10$, and stride 2 gives $6 \times 6$, as the formula predicts.

The stride-2 maps show what resolution costs. A stride-2 layer evaluates the filter only at even input positions, so the one-pixel bar in odd column 7 always falls on the filter’s negative side weights: it vanishes from channel 0, while the bar in even column 4 survives. The effect is extreme here because the bars are one pixel wide and the filters sharp, but the principle holds: a coarse map keeps fewer positions, and detail finer than its stride can be lost or misplaced (see sampling and aliasing).

Common Misconceptions

• “Each feature map detects one nameable object part.” Some maps do, but not all. Zeiler and Fergus (2014) show a layer-5 map whose top image patches seemed to have little in common until the visualization revealed that it responded to the grass in the background, not the foreground objects. In their occlusion experiments the strongest layer-5 map in one image picked out faces, while the classifier was sensitive to the dog, because it uses many maps.
• “A feature map is a feature descriptor.” A feature descriptor is a vector describing one location. The values of all channels at one position of a layer form such a vector; a single feature map is one filter’s response across all positions.

Where It Is Used

In a classifier, the last feature maps feed the head through pooling or fully connected layers. Other tasks reuse maps from several depths:

• Multi-scale detection. A feature pyramid network takes a ResNet’s stage outputs at strides 4 to 32 and combines low-resolution, semantically strong maps with high-resolution, semantically weak ones through a top-down pathway, so that a detector such as Faster R-CNN predicts small objects on fine maps and large ones on coarse maps (Lin et al., 2017).
• Segmentation. U-Net concatenates each upsampled map of its expanding path with the corresponding map of its contracting path, so that the high-resolution features localize the output (Ronneberger et al., 2015); fully convolutional networks fuse maps of several depths for the same reason.
• Transfer. Feature maps from a network pretrained on one dataset serve as features for another task, as transfer learning describes.
• Interpretation. Visualizing which inputs drive a map, as Zeiler and Fergus did, is one way to inspect what a network has learned.