Backbone Network

A backbone network is the part of a vision model that turns the input image into feature maps: a deep network, usually designed and pretrained for image classification, with its classifier removed. Detectors, segmenters, pose estimators and depth networks put small task-specific modules on top of it. It is also called the feature extractor, base network, trunk or, in encoder–decoder models, the encoder. Choosing one is an early design decision, and improvements in classification architectures have often reached detection and segmentation by a swap of backbone.

Definition

A backbone is the shared, task-agnostic first part of a network: it maps an image to one or more feature maps and carries no task-specific output. The modules after it are the head, which produces the task’s output (class scores, boxes, masks, per-pixel labels), and, in many detectors, a neck between the two that combines the backbone’s maps at several resolutions. In practice the backbone is a known architecture (ResNet-50, VGG-16, a vision transformer) initialized with pretrained weights and fine-tuned with the rest of the model, as described under transfer learning.

Where the Term Comes From

The split itself is older than the word. LeCun et al. (1998) describe a classical pattern recognizer as two modules, a hand-designed feature extractor and a general-purpose trainable classifier, and argue that the extractor too should be learned from pixels. Once CNNs trained on ImageNet learned such extractors, detection papers reused them. R-CNN fine-tuned an ImageNet network as its feature extractor, and its tech report (v5) showed the effect of swapping it: replacing Krizhevsky et al.’s network (“T-Net”) with the 16-layer Oxford network (“O-Net”) raised VOC 2007 test mAP from 58.5% to 66.0%, at about seven times the forward-pass time.

Names varied. Faster R-CNN (2015) speaks of shareable convolutional layers, SSD (Liu et al., 2016) of a truncated base network, and Huang et al. (2017) of feature extractors combined with detection meta-architectures. “Backbone” was in use in detection papers by 2016: R-FCN (Dai et al., 2016) has a paragraph headed “Backbone architecture” and describes adopting fully convolutional image-classifier backbones such as ResNets. Feature Pyramid Networks (Lin et al., 2017) and Mask R-CNN (He et al., 2017) use it throughout. Mask R-CNN separates the convolutional backbone that extracts features over the entire image from the network head applied to each region of interest, and names backbones by network, depth and features: ResNet-50-C4 or ResNet-101-FPN. We have not traced who first used the word; the papers above show usage, not origin.

In the papers checked here, the neck appears later. The MMDetection toolbox paper (Chen et al., 2019) decomposes detectors into a backbone that turns the image into feature maps, a neck that connects backbone and heads by refining or reconfiguring those maps (for example an FPN), and dense or region-of-interest heads. YOLOv4 (Bochkovskiy, Wang & Liao, 2020) uses the same three terms, calling the layers that collect feature maps from different stages between backbone and head “the neck of an object detector”.

Intuition

A network trained to classify ImageNet must already locate edges, textures, parts and objects to do its job. Its intermediate maps are a general description of the image, and only the last layer is specific to the 1,000 ImageNet classes. Cutting that layer off leaves a machine that describes any image, and each new task needs only a small reader for the description: a box regressor, a mask predictor, a per-pixel classifier. Several readers can share one description, which is how a single backbone in Mask R-CNN serves box, mask and keypoint heads at once.

The description comes at several resolutions. Early stages see small neighbourhoods at fine resolution; late stages see most of the image at coarse resolution. A classifier reads only the last stage, but a detector of small objects or a segmenter that must place boundaries needs the fine stages too, which is why necks exist.

Formal Definition

Write the model as a composition

$$f = h \circ n \circ b ,$$

with backbone $b$, neck $n$ (possibly the identity) and head $h$. A convolutional backbone is organized in stages: runs of layers that keep one spatial size, each stage starting with a stride-2 convolution or pooling. For an input of shape $(N, 3, H, W)$, stage $\ell$ outputs

$$C_\ell \in \mathbb{R}^{N \times c_\ell \times \lceil H / 2^\ell \rceil \times \lceil W / 2^\ell \rceil}, \qquad \ell = 2, \dots, 5,$$

up to rounding conventions, so its output stride (the ratio of input to feature resolution) is $s_\ell = 2^\ell$. In ResNet (He et al., 2016), stages conv2_x to conv5_x produce $56 \times 56$, $28 \times 28$, $14 \times 14$ and $7 \times 7$ maps for a $224 \times 224$ input; the FPN paper names their outputs $C_2$ to $C_5$, at strides 4 to 32 (the feature pyramid network article gives the details). A classification head reads only $C_5$, for example $h(C_5) = W \,\mathrm{GAP}(C_5) + \mathbf{b}$ with global average pooling. A neck maps $\{C_\ell\}$ to maps $\{P_\ell\}$ of a common width, and a dense head runs on each $P_\ell$ or on a merged map.

Backbone, Neck and Head in Practice

Faster R-CNN with ResNet-50-FPN. ResNet-50 supplies $C_2$ to $C_5$; the FPN merges them top-down into a pyramid of 256-channel maps $P_2$ to $P_5$; the region proposal network and the Fast R-CNN box head read the pyramid, and Mask R-CNN adds a mask branch to that head. Where the backbone ends is a convention:

Component	Mask R-CNN’s terms	MMDetection’s and YOLOv4’s terms
ResNet-50, conv1 to conv5	backbone (ResNet-50-FPN)	backbone
FPN top-down pathway	part of the backbone	neck
RPN, box and mask branches	RPN and network head	dense head and RoI heads (MMDetection); head (YOLOv4)

The boundary moves with the design, too. In Mask R-CNN’s ResNet-C4 variant the backbone stops after conv4 (stride 16), and conv5 (res5) runs on each pooled region as part of the head, which makes that head compute-intensive; with FPN, conv5 belongs to the backbone and the head is lighter.

YOLOv4. Bochkovskiy et al. list the parts of an ordinary detector as backbones (VGG16, ResNet-50, EfficientNet, CSPDarknet53 and others), necks (additional blocks such as SPP and ASPP, and path-aggregation blocks such as FPN, PAN and BiFPN) and heads (dense, one-stage prediction or sparse, two-stage prediction). YOLOv4 itself is a CSPDarknet53 backbone, an SPP block and a PAN neck, and YOLOv3’s anchor-based head. PAN (Liu et al., 2018) adds a bottom-up path after FPN’s top-down one (see the YOLO and feature pyramid network articles).

Segmentation. In FCN and DeepLab the backbone is a classification network run fully convolutionally, and the head predicts a label at every position. U-Net-style models call the backbone the encoder and add a decoder that plays the role of the neck, upsampling and merging stages through skip connections.

Properties

Several stages are tapped for dense tasks. Deep stages are semantically strong but coarse; shallow ones are fine but semantically weak. Detectors with FPN or PAN necks and U-Net decoders read more than one stage, and the example below shows why on a toy problem.

Output stride can be traded for compute. A classification backbone ends at output stride 32, too coarse for many dense tasks. DeepLabv3 runs ResNet at output stride 16 or 8 by replacing the last strides with dilation, keeping the pretrained weights (Chen et al., 2017); the DeepLab article explains the conversion and its cost, fourfold in the affected layers for each halving. Huang et al. (2017) reported the same trade-off in detection: in Faster R-CNN and R-FCN, a stride-8 instead of stride-16 ResNet-101 or Inception-ResNet feature extractor improved COCO mAP by 5% (relative) but increased running time by 63%.

Classification accuracy predicts detection accuracy only roughly. In Huang et al.’s unified comparison on COCO, detection mAP correlated with the feature extractor’s ImageNet top-1 accuracy overall, but significantly only for Faster R-CNN and R-FCN; SSD depended less on its feature extractor. Bochkovskiy et al. found CSPResNeXt-50 the better ImageNet classifier (79.8% against 78.7% top-1 for CSPDarknet-53 with their best training recipes, on ILSVRC 2012 val; crop protocol not stated) but CSPDarknet53 the better detector backbone (43.0 COCO AP at $512 \times 512$, against at most 42.4 for CSPResNeXt50, on test-dev 2017, the split the paper names for its detector experiments). They argued that a detector backbone needs higher input resolution, more layers for a larger receptive field, and more parameters than a classifier.

Input resolution is a second accuracy–cost knob. Huang et al. found that halving the input resolution in both dimensions lowered accuracy by 15.88% on average and reduced inference time by a relative 27.4%, with small objects affected most. Convolutional layers accept any input size, so a backbone pretrained at $224 \times 224$ can run on much larger detection inputs: FPN’s ImageNet-pretrained ResNets were fine-tuned on images resized to a shorter side of 800 pixels.

Pretraining is not strictly required for detection. He, Girshick and Dollár (2019) trained Mask R-CNN on COCO from random initialization and matched ImageNet-pretrained models given a longer schedule; the transfer learning article reports their main results. The two changes they found necessary were normalization that works with small per-GPU batches (group normalization or synchronized batch normalization) and the longer schedule. Even a ResNeXt-152 backbone with group normalization, about four times ResNet-101’s FLOPs, showed no noticeable overfitting from scratch: with cascade heads and test-time augmentation it reached 50.9 box AP and 43.2 mask AP on COCO val2017, against 50.3 and 42.5 for the same model pretrained on ImageNet.

Examples

The PyTorch example below (PyTorch 2.14.1, torch only, about 35 seconds on one CPU thread) builds a small convolutional backbone with a stride-2 stem and four stages $C_2$ to $C_5$ at strides 4 to 32. It trains the backbone with a classification head on 3,000 noisy $64 \times 64$ images, each containing one shape (disk, square, ring or plus) at a random position and size. It then freezes the backbone and reuses its features under a second, dense head that labels every pixel as background or one of the four shapes. The dense head’s neck projects the chosen stages to 32 channels, upsamples them to stride 4 and sums them, an FPN reduced to its simplest form; it is trained on three stage choices with three seeds each.

import numpy as np
import torch
from torch import nn
F = nn.functional
torch.set_num_threads(1)                       # single thread: identical output on every run

CLASSES = ["disk", "square", "ring", "plus"]

def make_images(n, seed, size=64):
    """Noisy images with one shape each; returns images, class labels and per-pixel masks."""
    rng = np.random.default_rng(seed)
    yy, xx = np.mgrid[:size, :size]
    X, y, M = [], [], []
    for _ in range(n):
        c = int(rng.integers(len(CLASSES)))
        r = rng.uniform(8, 20)                                    # half-size in pixels
        cy, cx = rng.uniform(r + 1, size - 1 - r, size=2)
        dy, dx = yy - cy, xx - cx
        w = max(2.5, r / 3)                                       # stroke width
        m = [dy**2 + dx**2 <= r**2,
             np.maximum(abs(dy), abs(dx)) <= r,
             abs(np.hypot(dy, dx) - r + w / 2) <= w / 2,
             ((abs(dy) <= w / 2) & (abs(dx) <= r)) | ((abs(dx) <= w / 2) & (abs(dy) <= r))][c]
        fg, bg = rng.uniform(0.6, 1.0), rng.uniform(0.0, 0.3)
        X.append(bg + (fg - bg) * m + rng.normal(0, 0.2, m.shape))
        y.append(c)
        M.append(np.where(m, c + 1, 0))                           # 0 = background
    to_t = lambda a, d: torch.tensor(np.array(a), dtype=d)
    return to_t(X, torch.float32).unsqueeze(1), to_t(y, torch.long), to_t(M, torch.long)

def block(cin, cout, stride):
    return nn.Sequential(nn.Conv2d(cin, cout, 3, stride, 1, bias=False),
                         nn.BatchNorm2d(cout), nn.ReLU(),
                         nn.Conv2d(cout, cout, 3, 1, 1, bias=False),
                         nn.BatchNorm2d(cout), nn.ReLU())

class Backbone(nn.Module):
    """Stem at stride 2, then four stages C2..C5 at strides 4, 8, 16, 32."""
    channels = {"C2": 16, "C3": 32, "C4": 64, "C5": 128}
    strides = {"C2": 4, "C3": 8, "C4": 16, "C5": 32}
    def __init__(self):
        super().__init__()
        self.stem = nn.Sequential(nn.Conv2d(1, 16, 3, 2, 1, bias=False), nn.BatchNorm2d(16), nn.ReLU())
        self.stages = nn.ModuleDict({"C2": block(16, 16, 2), "C3": block(16, 32, 2),
                                     "C4": block(32, 64, 2), "C5": block(64, 128, 2)})
    def forward(self, x):
        x, feats = self.stem(x), {}
        for name, stage in self.stages.items():
            x = feats[name] = stage(x)
        return feats                                              # every stage's output

class ClassificationHead(nn.Module):
    """Global average pooling of C5, then one linear layer."""
    def __init__(self, n_classes):
        super().__init__()
        self.fc = nn.Linear(128, n_classes)
    def forward(self, feats):
        return self.fc(feats["C5"].mean(dim=(2, 3)))

class DenseHead(nn.Module):
    """Neck: 1x1 projections of the chosen stages, upsampled to stride 4 and summed.
    Head: 3x3 conv and 1x1 per-pixel classifier; scores upsampled to the input size on request."""
    def __init__(self, n_classes, levels, d=32):
        super().__init__()
        self.levels = levels
        self.lateral = nn.ModuleDict({k: nn.Conv2d(Backbone.channels[k], d, 1) for k in levels})
        self.head = nn.Sequential(nn.Conv2d(d, d, 3, padding=1), nn.ReLU(), nn.Conv2d(d, n_classes, 1))
    def forward(self, feats, size=None):
        s4 = feats["C2"].shape[-2:]                               # stride-4 grid
        z = sum(F.interpolate(self.lateral[k](feats[k]), size=s4, mode="bilinear",
                              align_corners=False) for k in self.levels)
        out = self.head(z)                                        # per-pixel scores at stride 4
        return out if size is None else F.interpolate(out, size=size, mode="bilinear",
                                                      align_corners=False)

def receptive_fields(backbone):
    """Receptive field (pixels) of each stage output: r += (k - 1) * jump, jump *= stride."""
    r, jump, out = 1, 1, {}
    for name, part in [("stem", backbone.stem)] + list(backbone.stages.items()):
        for m in part.modules():
            if isinstance(m, nn.Conv2d):
                r, jump = r + (m.kernel_size[0] - 1) * jump, jump * m.stride[0]
        out[name] = r
    return out

def nparams(m):
    return sum(p.numel() for p in m.parameters())

def train(params, loss_fn, n, steps, lr, seed, batch=32):
    g, opt = torch.Generator().manual_seed(seed), torch.optim.Adam(params, lr=lr)
    for _ in range(steps):
        idx = torch.randint(n, (batch,), generator=g)
        loss = loss_fn(idx)
        opt.zero_grad(); loss.backward(); opt.step()

def mean_iou(pred, target, k):
    ious = [((pred == c) & (target == c)).sum().item() / ((pred == c) | (target == c)).sum().item()
            for c in range(k)]
    return float(np.mean(ious))

print("torch", torch.__version__)
torch.manual_seed(0)
backbone = Backbone()

# 1. Train backbone + classification head on 4 shape classes (image-level labels only).
Xtr, ytr, Mtr = make_images(3000, seed=0)
Xte, yte, Mte = make_images(500, seed=1)
cls_head = ClassificationHead(4)
train(list(backbone.parameters()) + list(cls_head.parameters()),
      lambda i: F.cross_entropy(cls_head(backbone(Xtr[i])), ytr[i]), 3000, steps=500, lr=3e-3, seed=0)
backbone.eval().requires_grad_(False)                         # freeze: BN statistics and weights fixed
with torch.no_grad():
    Ftr, Fte = backbone(Xtr), backbone(Xte)                   # features computed once, reused below
    acc = (cls_head(Fte).argmax(1) == yte).float().mean().item()
rf = receptive_fields(backbone)
for name, f in Fte.items():
    print(f"{name}: stride {Backbone.strides[name]:2d}, shape {tuple(f[:1].shape)}, "
          f"receptive field {rf[name]:3d} px")
print(f"backbone: {nparams(backbone):,} parameters")
print(f"classification head on C5: {nparams(cls_head):,} parameters, test accuracy {acc:.3f}")

# 2. Reuse the frozen backbone under dense heads (5 per-pixel classes: background + 4 shapes).
for levels in (["C2"], ["C5"], ["C2", "C3", "C4", "C5"]):
    scores = []
    for seed in range(3):
        torch.manual_seed(seed)
        dense = DenseHead(5, levels)
        train(dense.parameters(),                             # loss on the stride-4 grid
              lambda i: F.cross_entropy(dense({k: v[i] for k, v in Ftr.items()}),
                                        Mtr[i][:, 2::4, 2::4]), 3000, steps=250, lr=3e-3, seed=seed)
        with torch.no_grad():
            pred = dense(Fte, (64, 64)).argmax(1)
        scores.append(mean_iou(pred, Mte, 5))
    share = nparams(dense) / (nparams(dense) + nparams(backbone))
    print(f"dense head on {'+'.join(levels):11s}: {nparams(dense):6,} parameters "
          f"({share:.1%} of backbone + head), test mIoU, seeds 0-2: "
          + ", ".join(f"{s:.3f}" for s in scores))

Output:

torch 2.14.1
C2: stride  4, shape (1, 16, 16, 16), receptive field  15 px
C3: stride  8, shape (1, 32, 8, 8), receptive field  39 px
C4: stride 16, shape (1, 64, 4, 4), receptive field  87 px
C5: stride 32, shape (1, 128, 2, 2), receptive field 183 px
backbone: 296,048 parameters
classification head on C5: 516 parameters, test accuracy 0.988
dense head on C2         :  9,957 parameters (3.3% of backbone + head), test mIoU, seeds 0-2: 0.644, 0.709, 0.685
dense head on C5         : 13,541 parameters (4.4% of backbone + head), test mIoU, seeds 0-2: 0.332, 0.331, 0.331
dense head on C2+C3+C4+C5: 17,221 parameters (5.5% of backbone + head), test mIoU, seeds 0-2: 0.864, 0.861, 0.850

The shared backbone dominates every model, and one set of frozen features serves both tasks. Reading one stage is much worse than reading all four. For $C_5$ alone, in our reading, its $2 \times 2$ grid knows which shape is present but cannot place a boundary finer than 32 pixels. $C_2$ alone is fine enough, but its units see only 15 pixels, less than any shape’s extent (16 to 40 pixels), so local edges must stand in for the whole shape. Combining the stages gives each pixel both.

Common Misconceptions

• “A backbone is a classifier with its last layer cut off.” Often it is modified first: strides replaced by dilation (DeepLab), batch-normalization statistics frozen or replaced by group normalization (He et al., 2019), or a stage moved into the head (Mask R-CNN’s C4 variant).
• “The best ImageNet model makes the best backbone.” The correlation is real but loose, and depends on the detector (Huang et al., 2017; Bochkovskiy et al., 2020).
• “The backbone stays frozen.” In most detection and segmentation training it is fine-tuned with the heads; FPN describes ImageNet pretraining followed by fine-tuning on the detection data as common practice (transfer learning covers the options, including freezing the first layers).
• “Backbone, neck and head are fixed parts.” They are a vocabulary for describing designs, and papers draw the lines differently: Mask R-CNN counts FPN in the backbone, YOLOv4 calls it a neck.

Where It Is Used

Most deep detectors, segmenters, pose estimators and depth networks are built on a backbone. Architectures widely used as backbones, each covered in its own article:

• LeNet (LeCun et al., 1998): convolution and subsampling layers trained from pixels to recognize handwritten characters, the learned feature extractor of the paper’s argument above, far smaller than today’s backbones.
• AlexNet (Krizhevsky et al., 2012): five convolutional and three fully connected layers; the architecture of R-CNN’s original feature extractor.
• VGG (Simonyan & Zisserman, 2015): 16 to 19 weight layers of $3 \times 3$ convolutions; the base network of SSD and the backbone of the strongest Faster R-CNN and FCN models in those papers.
• Inception (Szegedy et al., 2015): GoogLeNet’s 22-layer network of multi-branch modules designed for a fixed computational budget.
• ResNet (He et al., 2016): residual connections and four stages; ResNet-50 and ResNet-101 are the main backbones of the FPN and Mask R-CNN papers.
• MobileNet (Howard et al., 2017): depthwise separable convolutions and two global multipliers that trade accuracy for latency, applied to detection in its paper.
• EfficientNet (Tan & Le, 2019): a searched baseline scaled jointly in depth, width and resolution.

Transformers serve as backbones too. A plain vision transformer produces a feature map at a single stride, so detectors either build a simple pyramid from it, as ViTDet does, or use a hierarchical transformer such as the Swin Transformer, whose stages produce maps at several strides like a CNN’s (both described in the vision transformer article). Backbones are also pretrained without labels, by self-supervised learning such as masked autoencoders, or on image–text pairs, as CLIP’s image encoders are, and then fine-tuned under detection and segmentation heads in the same way (the linked articles give examples).