Feature Pyramid Network

A feature pyramid network (FPN) builds a multi-scale set of feature maps inside a convolutional neural network, one map per resolution, each with strong semantic features. It takes the maps a backbone already computes at strides 4 to 32 and runs a cheap top-down pathway that upsamples the deep, coarse maps and adds them, through lateral connections, to the finer ones. Lin et al. introduced it at CVPR 2017 for object detection, where it replaced image pyramids at little extra cost; it has since served as the multi-scale stage of two-stage and one-stage detectors and of segmentation models.

Core Idea

Objects in one image appear at very different sizes, and a classifier or box regressor looking through a fixed-size window handles only a narrow range of them. Lin et al. compare four ways of covering the full range (their Fig. 1):

1. Featurized image pyramid. Compute features separately on every level of an image pyramid. Every level is semantically strong, and a change of object size becomes a change of level. But the authors cite a fourfold increase in inference time, and end-to-end training on all levels does not fit in memory, so image pyramids were used only at test time, inconsistently with training.
2. Single feature map. Predict from the network’s last map only, as Fast and Faster R-CNN did by default: fast, but a stride-16 or stride-32 map covers small objects with very few cells.
3. Pyramidal feature hierarchy. Predict from several of the backbone’s own maps, as SSD (Liu et al., 2016) did. The shallow, high-resolution maps carry low-level features, poor for recognition. The authors note that SSD avoided them by starting its pyramid high in the network, at VGG’s conv4_3, and adding new layers, which left out the finer maps they found important for small objects.
4. Feature Pyramid Network. Keep the backbone’s maps at every resolution, enrich each with semantics passed down from coarser levels, and predict independently on every level, as on a featurized image pyramid, at a cost close to designs 2 and 3.

Predicting on every level separates FPN from earlier top-down designs with skip connections, such as U-Net and SharpMask, which predict only from the finest map (the paper’s Fig. 2).

Building Blocks

Bottom-up pathway. The ordinary forward pass of the backbone. Layers producing maps of one size form a stage, and the pyramid takes the output of each stage’s last layer, which should carry its strongest features. For a ResNet (He et al., 2016), these are the outputs of the last residual block of conv2 to conv5, named $C_2, C_3, C_4, C_5$, with strides of 4, 8, 16 and 32 pixels. The conv1 output is left out because of its large memory footprint.

Lateral connections. A $1 \times 1$ convolution maps each $C_k$ to $d = 256$ channels; all levels need the same width because they share prediction heads.

Top-down pathway. Starting from the projection of $C_5$, each coarser map is upsampled by 2 with nearest-neighbor interpolation (chosen for simplicity) and added element-wise to the lateral projection of the next finer stage, down to stride 4.

Output convolutions. A $3 \times 3$ convolution on each merged map, meant to reduce the aliasing of upsampling, produces $P_2$ to $P_5$, the same sizes as $C_2$ to $C_5$. For the region proposal network (RPN), $P_6$ is a stride-2 subsampling of $P_5$, added only for the largest anchors.

The extra layers have no nonlinearities, which the authors found to matter little; residual blocks in place of single convolutions were only marginally better.

How Data Flows

Written out, the merged maps $M_k$ and outputs $P_k$ are

$$M_5 = L_5(C_5), \qquad M_k = L_k(C_k) + \operatorname{Up}_2(M_{k+1}) \quad (k = 4, 3, 2), \qquad P_k = \operatorname{Conv}_{3\times 3}(M_k),$$

where $L_k$ is the lateral convolution and $\operatorname{Up}_2$ is 2× nearest-neighbor upsampling. $P_2$ thus contains contributions from all four stages: top-down terms from units with very large receptive fields, and a lateral term whose activations are better localized because they were subsampled fewer times.

The NumPy example imitates this on a $256 \times 256$ image with a large $64 \times 64$ and a small $8 \times 8$ square. In this toy, “semantic strength” is just the edge evidence a unit has gathered over its window: the bottom-up pathway applies 2×2 average pooling and a $3 \times 3$ averaging layer per level to a Sobel edge map, the lateral convolutions become scalar normalizations, and the output convolutions $3 \times 3$ averages.

import numpy as np
from scipy import ndimage

# A 256 x 256 image with a large (64 x 64) and a small (8 x 8) bright square.
img = np.zeros((256, 256))
img[40:104, 40:104] = 1.0
img[176:184, 184:192] = 1.0
large, small = (slice(40, 104), slice(40, 104)), (slice(176, 184), slice(184, 192))
inner = (slice(56, 88), slice(56, 88))      # the large square's interior, away from its edges

# Low-level features: Sobel gradient magnitude (an edge map) at stride 1.
edges = np.hypot(ndimage.sobel(img, 0), ndimage.sobel(img, 1))

def avgpool2(x):
    return x.reshape(x.shape[0] // 2, 2, x.shape[1] // 2, 2).mean(axis=(1, 3))

def box3(x):
    """A 3 x 3 averaging layer: each unit gathers evidence from a wider window."""
    return ndimage.uniform_filter(x, 3, mode="constant")

def region(F, k, sl):
    """Values of the stride-2^k map F over the image region sl (pixel slices)."""
    s = 2 ** k
    return F[sl[0].start // s:(sl[0].stop - 1) // s + 1, sl[1].start // s:(sl[1].stop - 1) // s + 1]

def fires(F, k, sl):
    """A toy detector fires where the response reaches half the level's maximum."""
    return bool(region(F, k, sl).max() >= 0.5 * F.max())

# Toy bottom-up pathway: C2..C5 at strides 4, 8, 16, 32; a unit of level k sees about 3 * 2^k pixels.
C, x = {}, edges
for k in range(1, 6):
    x = avgpool2(x)
    if k >= 2:
        C[k] = box3(x)

print("level  stride  small peak  large peak  small/large")
for k in range(2, 6):
    ps, pl = region(C[k], k, small).max(), region(C[k], k, large).max()
    print(f"  C{k}    {2**k:>4}   {ps:9.3f}  {pl:10.3f}  {ps / pl:10.2f}")
print("C5 fires on large:", fires(C[5], 5, large), "| on small:", fires(C[5], 5, small))
print(f"C2 mean inside the large square: {abs(region(C[2], 2, inner).mean()):.3f}")

# Top-down pathway with lateral connections: scale each level (the 1x1 lateral conv),
# add the 2x nearest-neighbor upsampled coarser map, smooth the sum (the 3x3 output conv).
def up2(x):
    return x.repeat(2, axis=0).repeat(2, axis=1)

lateral = {k: C[k] / C[k].max() for k in C}
merged = lateral[5]
P = {5: box3(merged)}
for k in (4, 3, 2):
    merged = lateral[k] + up2(merged)
    P[k] = box3(merged)

far = ndimage.binary_dilation(img > 0, iterations=24)[::4, ::4]   # stride-4 mask, objects plus margin
print(f"P2 small peak {region(P[2], 2, small).max():.3f} | large interior mean "
      f"{region(P[2], 2, inner).mean():.3f} | background mean {P[2][~far].mean():.3f}")
print("P2 fires on large:", fires(P[2], 2, large), "| on small:", fires(P[2], 2, small))

Output:

level  stride  small peak  large peak  small/large
  C2       4       1.216       0.861        1.41
  C3       8       0.417       0.437        0.95
  C4      16       0.104       0.276        0.38
  C5      32       0.026       0.220        0.12
C5 fires on large: True | on small: False
C2 mean inside the large square: 0.000
P2 small peak 2.105 | large interior mean 1.464 | background mean 0.094
P2 fires on large: True | on small: True

Going up, the small square’s response falls from 1.41 times the large square’s to 0.12 times, and at $C_5$ it no longer reaches the threshold. $C_2$ has the opposite weakness: it responds only at edges, and is zero inside the large square. After the top-down merge, $P_2$ responds strongly inside the large square (1.46, against 0.09 for the background) and keeps the small square’s sharp peak, so both objects are found on one fine map.

In a real network the lateral and output convolutions are learned. The module below, run with PyTorch 2.14.1, applies them to random tensors with the channel counts of ResNet-50’s stages (256, 512, 1024, 2048) for an $800 \times 1216$ input.

import torch
import torch.nn as nn
import torch.nn.functional as F

class FPN(nn.Module):
    """Top-down pathway with lateral connections over backbone maps C2..C5 (Lin et al., 2017)."""
    def __init__(self, in_channels=(256, 512, 1024, 2048), d=256):
        super().__init__()
        self.lateral = nn.ModuleList(nn.Conv2d(c, d, 1) for c in in_channels)
        self.output = nn.ModuleList(nn.Conv2d(d, d, 3, padding=1) for _ in in_channels)

    def forward(self, feats):                       # feats = [C2, C3, C4, C5], finest first
        merged = self.lateral[-1](feats[-1])        # start from C5
        outs = [self.output[-1](merged)]            # P5
        for i in range(len(feats) - 2, -1, -1):     # C4, C3, C2
            top_down = F.interpolate(merged, size=feats[i].shape[-2:], mode="nearest")
            merged = self.lateral[i](feats[i]) + top_down
            outs.insert(0, self.output[i](merged))  # 3x3 conv reduces upsampling aliasing
        p6 = F.max_pool2d(outs[-1], kernel_size=1, stride=2)   # stride-2 subsampling of P5
        return outs + [p6]                          # [P2, P3, P4, P5, P6]

torch.manual_seed(0)
H, W = 800, 1216                                    # shorter side 800, as in the paper's training
C = [torch.randn(1, c, H // s, W // s) for c, s in zip((256, 512, 1024, 2048), (4, 8, 16, 32))]
fpn = FPN()
with torch.no_grad():
    P = fpn(C)
print("torch", torch.__version__)
for name, t in zip(["C2", "C3", "C4", "C5"], C):
    print(name, tuple(t.shape))
for k, t in enumerate(P, start=2):
    print(f"P{k}", tuple(t.shape))
print("FPN parameters:", sum(p.numel() for p in fpn.parameters()))

Output:

torch 2.14.1
C2 (1, 256, 200, 304)
C3 (1, 512, 100, 152)
C4 (1, 1024, 50, 76)
C5 (1, 2048, 25, 38)
P2 (1, 256, 200, 304)
P3 (1, 256, 100, 152)
P4 (1, 256, 50, 76)
P5 (1, 256, 25, 38)
P6 (1, 256, 13, 19)
FPN parameters: 3344384

Every level has 256 channels at its stage’s resolution; the added layers hold 3.3 million parameters, 2.4 million of them in the $3 \times 3$ convolutions.

Mathematical Formulation

Heads on every level. The RPN attaches one head, a $3 \times 3$ convolution with two sibling $1 \times 1$ convolutions, to every level from $P_2$ to $P_6$, each with anchors of a single scale labeled by IoU as in Faster R-CNN (Ren et al., 2015); ground-truth boxes reach a level only through its anchors. Shared head parameters did about as well as separate ones, which the authors read as evidence that all levels have similar semantic strength.

Assigning regions to levels. A Fast R-CNN detector pools features for each region of interest (RoI), so with a pyramid it must choose a level. Treating the feature pyramid as if it came from an image pyramid, Lin et al. assign an RoI of width $w$ and height $h$ (input-image pixels) to level

$$k = \left\lfloor k_0 + \log_2\!\left(\frac{\sqrt{wh}}{224}\right) \right\rfloor, \qquad k_0 = 4.$$

Here 224 is the canonical ImageNet pre-training size, and $k_0 = 4$ sends a $224 \times 224$ RoI to $P_4$, matching the $C_4$ map of ResNet-based Faster R-CNN; halving an RoI’s size moves it one level finer. RoI pooling then extracts $7 \times 7$ features for two shared 1,024-dimensional fully connected layers, a lighter head than the ResNet baseline’s conv5 stage, which FPN already uses in the pyramid.

import numpy as np

def fpn_level(w, h, k0=4, canonical=224, k_min=2, k_max=5):
    """Pyramid level for an RoI of w x h pixels, Eq. (1) of Lin et al. (2017), clipped to P2..P5."""
    k = np.floor(k0 + np.log2(np.sqrt(w * h) / canonical))
    return np.clip(k, k_min, k_max).astype(int)

boxes = np.array([[20, 20], [60, 40], [110, 110], [112, 112], [120, 120], [100, 400],
                  [224, 224], [300, 200], [448, 448], [640, 480]], float)
levels = fpn_level(boxes[:, 0], boxes[:, 1])
for (w, h), k in zip(boxes, levels):
    print(f"{w:4.0f} x {h:<4.0f} sqrt(wh) = {np.sqrt(w * h):6.1f} -> P{k} (stride {2 ** k:2d}, "
          f"RoI spans {np.sqrt(w * h) / 2 ** k:5.1f} cells)")

Output:

  20 x 20   sqrt(wh) =   20.0 -> P2 (stride  4, RoI spans   5.0 cells)
  60 x 40   sqrt(wh) =   49.0 -> P2 (stride  4, RoI spans  12.2 cells)
 110 x 110  sqrt(wh) =  110.0 -> P2 (stride  4, RoI spans  27.5 cells)
 112 x 112  sqrt(wh) =  112.0 -> P3 (stride  8, RoI spans  14.0 cells)
 120 x 120  sqrt(wh) =  120.0 -> P3 (stride  8, RoI spans  15.0 cells)
 100 x 400  sqrt(wh) =  200.0 -> P3 (stride  8, RoI spans  25.0 cells)
 224 x 224  sqrt(wh) =  224.0 -> P4 (stride 16, RoI spans  14.0 cells)
 300 x 200  sqrt(wh) =  244.9 -> P4 (stride 16, RoI spans  15.3 cells)
 448 x 448  sqrt(wh) =  448.0 -> P5 (stride 32, RoI spans  14.0 cells)
 640 x 480  sqrt(wh) =  554.3 -> P5 (stride 32, RoI spans  17.3 cells)

The clipping is ours, not stated in the paper: the detector uses only $P_2$ to $P_5$. Between the ends, an RoI spans 14 to 28 cells of its level whatever its size, which is the point of the rule. But the boundaries are hard: the nearly identical 110- and 120-pixel squares are pooled from different levels.

Why It Works

Lin et al. ran their ablations with ResNet-50 on COCO minival, training on trainval35k at a shorter side of 800 pixels.

• Proposals. FPN raised the RPN’s average recall at 1,000 proposals from 48.3 (single-scale $C_4$ baseline) to 56.3, and by 12.9 points on small objects (32.0 to 44.9).
• Ablations. Without the top-down pathway (an SSD-style bottom-up pyramid), average recall fell to 49.5, on par with the baseline, and Fast R-CNN AP on fixed proposals fell from 33.9 to 24.9; the authors conjecture that semantic gaps between levels are the cause. Without lateral connections, recall was 46.1, which they attribute to locations blurred by repeated down- and upsampling. Using only $P_2$, with all anchors on it (750k instead of 200k), gave 51.3: more anchors alone did not close the gap.
• Detection. With Faster R-CNN, AP rose from 31.6 to 33.9 (+2.3) and AP@0.5 from 53.1 to 56.9 (+3.8) over the authors’ own $C_4$ baseline.

With ResNet-101, the minival AP was 35.0, or 35.2 when the RPN and detector shared features. Training the detector twice as long raised it to 35.6 (without shared features). That model, submitted to the COCO evaluation server, scored 36.2 AP and 59.1 AP@0.5 on test-dev (18.2 AP on small objects), and 35.8 AP on test-std. It used a single input scale, without image pyramids, and surpassed the single-model entries of the 2015 and 2016 COCO competition winners listed in the paper. With shared features, inference took 0.165 s per image (ResNet-50) and 0.19 s (ResNet-101) on an NVIDIA M40 GPU, against 0.32 s for the single-scale ResNet-50 baseline; the arXiv revision (v2) updates these to 0.148 s and 0.172 s.

Training Considerations

Because the pyramid comes from one image scale, the system trains end-to-end and runs identically at training and test time. The backbone is pretrained on ImageNet and fine-tuned; the detector’s fully connected layers start from random weights, since ResNets have no pretrained fc layers. Unlike Faster R-CNN, the RPN experiments kept anchors crossing the image border.

Variants

• RetinaNet (Lin et al., 2017, focal loss) uses levels $P_3$ to $P_7$. It drops $P_2$ for computational reasons, computes $P_6$ by a $3 \times 3$ stride-2 convolution on $C_5$, and adds $P_7$ (ReLU, then another stride-2 convolution) for large objects.
• PANet (Liu et al., 2018) adds a bottom-up path augmentation after FPN. From $N_2 = P_2$, each new level adds a stride-2 $3 \times 3$ convolution of the finer one to the matching $P$ level and applies another $3 \times 3$ convolution, giving low-level localization signals a path of fewer than 10 layers to the top instead of the backbone’s 100+. It also pools each proposal from all levels (adaptive feature pooling) instead of using the hard rule above.
• NAS-FPN (Ghiasi et al., 2019) searches the cross-scale connections with neural architecture search, using a reinforcement-learning controller, inside RetinaNet. The resulting cell mixes top-down and bottom-up connections and can be stacked repeatedly.
• BiFPN (Tan et al., 2020, EfficientDet) simplifies a PANet-like bidirectional network. It removes single-input nodes, adds same-level input-to-output edges, repeats the bidirectional block, and replaces plain addition with learned, normalized nonnegative weights, because inputs at different resolutions contribute unequally.

Use in Computer Vision

FPN is a component of other models. In Faster R-CNN it serves both the RPN and the detector. Mask R-CNN (He et al., 2017) offers a ResNet-FPN backbone, reported to give gains in both accuracy and speed; on COCO minival, ResNet-50-FPN reached 33.6 mask AP against 30.3 for ResNet-50-C4. RetinaNet predicts densely on every level. Panoptic FPN (Kirillov et al., 2019) adds to Mask R-CNN’s shared FPN a semantic segmentation branch that merges all levels at 1/4 resolution. The original paper also generated segment proposals with FPN ($d = 128$), with over 8.3 points higher average recall than the image-pyramid methods DeepMask and SharpMask, and faster.

Limitations

• Hard level assignment. PANet’s authors note that two proposals differing by 10 pixels can be pooled from different levels, and that high-level context could help small proposals too.
• One-way, equal fusion. Information flows only top-down, and inputs are summed with equal weight; PANet, NAS-FPN and BiFPN revisit these choices.
• Cost of the finest level. $P_2$ has 16 times as many positions as $P_4$; RetinaNet drops it.