PSPNet

PSPNet, the Pyramid Scene Parsing Network, is a semantic segmentation model by Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang and Jiaya Jia of the Chinese University of Hong Kong and SenseTime, posted to arXiv in December 2016 and published at CVPR 2017. It is a fully convolutional network of the dilated kind: a ResNet whose last two stages trade striding for dilation, so that features come out at 1/8 of the input resolution. On top of those features it places a pyramid pooling module, which average-pools the map into $1 \times 1$, $2 \times 2$, $3 \times 3$ and $6 \times 6$ bins, compresses each pooled map, upsamples it and concatenates it with the original features, so that each pixel is classified with a summary of the whole image and of the region around it. Training adds an auxiliary loss on the fourth stage. With these parts the authors’ team won the ImageNet Scene Parsing Challenge 2016 on ADE20K, and the paper reported first place on the PASCAL VOC 2012 and Cityscapes benchmarks at the time.

Problem Addressed

Scene parsing, the paper’s name for semantic segmentation of whole scenes with many object and stuff classes, was then dominated by FCN-based models. Inspecting the errors of the FCN baseline released with ADE20K, the authors grouped them into three kinds:

• Mismatched relationship. An FCN labelled a boat on a river as a car: the local appearance fits, but cars are seldom over water.
• Confusion categories. Classes such as field and earth, mountain and hill, or building and skyscraper look alike; the FCN labelled one object as part building and part skyscraper.
• Inconspicuous classes. Small things, such as a streetlight or a pillow on a bed sheet of similar texture, get lost, while very large stuff can exceed the receptive field and receive inconsistent labels.

They related many of these errors, partly or completely, to contextual relationships and global information, naming the lack of a strategy for using global scene clues as the major issue of FCN-based models. A deep ResNet’s theoretical receptive field already exceeds the input, but the paper cites Zhou et al.’s finding that the empirical receptive field of CNN units is much smaller, especially in high layers. Global average pooling, which ParseNet (Liu, Rabinovich and Berg, 2015) had added to an FCN, supplies an image-level vector, but the authors argued that one vector mixes the many objects and stuff of a complex scene and loses their spatial relations. Their answer borrows the idea of the spatial pyramid from classical scene classification and from SPP-net, which pooled feature maps over several grids (see pooling).

Architecture

An input image passes through a dilated ResNet to a 2,048-channel feature map at output stride 8. The pyramid pooling module turns this map into a 4,096-channel map that carries local and pooled context, a $3 \times 3$ convolution and a $1 \times 1$ classifier produce class scores, and the scores are upsampled by 8 to the input size. During training a second classifier reads the fourth stage.

Dilated ResNet backbone

The paper uses an ImageNet-pretrained ResNet “with the dilated network strategy”, citing DeepLab and Yu and Koltun: the last two stages keep their weights but drop their stride, and their $3 \times 3$ convolutions are dilated by 2 and 4, so the final map is 1/8 of the input instead of 1/32. DeepLab explains the operator and its cost. The released models differ from the original ResNet in two details visible in the code. The stem replaces the $7 \times 7$ convolution by three $3 \times 3$ convolutions with 64, 64 and 128 channels (in the deploy prototxts and in semseg’s resnet.py; the semseg README notes that its ImageNet models differ from torchvision’s “in the beginning part”). And every block of a converted stage is dilated, including the first: the ResNet-101 prototxt for VOC has 23 convolutions with dilation 2 and 3 with dilation 4, one per block of stages 4 and 5, and semseg’s pspnet.py sets dilation 2 and 4 on every conv2 of layer3 and layer4. Torchvision’s conversion, described in the DeepLab article, instead keeps the previous rate in a stage’s first block.

The paper reports models on ResNets with 50, 101, 152 and 269 layers. It does not describe the 269-layer network; its acknowledgements thank two colleagues for help training the basic classification models.

Pyramid pooling module

Let $X \in \mathbb{R}^{C \times H \times W}$ be the backbone output, with $C = 2048$. For each bin count $b \in \{1, 2, 3, 6\}$, the module average-pools $X$ to a $b \times b$ grid, applies a $1 \times 1$ convolution that reduces the channels to $C/N$ for $N = 4$ levels (512), and upsamples the result bilinearly back to $H \times W$:

$$P_b = U_{H \times W}\Big(\phi_b\big(\operatorname{AvgPool}_{b \times b}(X)\big)\Big), \qquad Z = \big[\, X,\; P_1,\; P_2,\; P_3,\; P_6 \,\big] \in \mathbb{R}^{2C \times H \times W},$$

where $\phi_b$ is the $1 \times 1$ convolution with batch normalization and ReLU (as in the code), $U$ is bilinear upsampling, and $[\cdot]$ is channel concatenation. The paper states the reduction as “1/N of the original one if the level size of pyramid is N”; the code makes clear that $N$ is the number of levels (int(fea_dim/len(bins)) in semseg). The reduction keeps the four pooled levels together at the same width as $X$; the paper’s stated reason is “to maintain the weight of global feature”, which we read as keeping the pooled context and the local features in balance.

Each level sees the image at a different granularity. The $1 \times 1$ bin is global average pooling; the $2 \times 2$ bins each cover a quarter of the image; the $3 \times 3$ and $6 \times 6$ bins cover ninths and thirty-sixths. After upsampling, every pixel of $Z$ holds its own local features plus compressed averages of the whole image and of the cells around it at three finer grids, bilinearly interpolated between neighbouring cells. The paper notes that the number of levels and the bin sizes can be changed and should suit the size of the input map; the released Caffe models implement the bins as fixed average-pooling windows for one crop size, for example windows of 60, 30, 20 and 10 on the $60 \times 60$ map of a $473 \times 473$ crop. semseg uses PyTorch’s adaptive average pooling instead, which works for any map size.

Head and output

The concatenated map $Z$ goes through a $3 \times 3$ convolution to 512 channels with batch normalization and ReLU, dropout with probability 0.1, and a $1 \times 1$ convolution to the class scores, which are upsampled bilinearly by 8 (the conv5_4, conv5_4/dropout, conv6 and conv6_interp layers of the prototxts; self.cls in semseg). The paper’s text only says that the concatenated features are “followed by a convolution layer”; the widths and the dropout come from the code. The paper also states that PSPNet “does not much increase” computation compared with the dilated FCN.

The short count below follows the prototxts and semseg’s pspnet.py to compare the layers after the backbone with and without the module: the plain head is semseg’s use_ppm=False variant, which applies the same $3 \times 3$ convolution directly to the 2,048 backbone channels. (The paper does not specify its baseline’s head.)

# Parameters and multiply-accumulates (MACs) of the layers after a dilated ResNet's
# 2048-channel output, following the official deploy prototxts and semseg's pspnet.py.
def head_cost(crop, classes, bins=(1, 2, 3, 6), ppm=True, c=2048, mid=512):
    n = (crop - 1) // 8 + 1                       # output stride 8: 473 -> 60, 713 -> 90
    params = macs = 0
    cin = c
    if ppm:
        red = c // len(bins)                      # 1/N of the channels for N levels: 512
        for b in bins:
            params += c * red                     # 1x1 conv on the b x b pooled map
            macs += b * b * c * red
        cin = c + len(bins) * red                 # concatenated with the input: 4096
    params += 9 * cin * mid + mid * classes       # 3x3 conv to 512 channels, 1x1 classifier
    macs += n * n * (9 * cin * mid + mid * classes)
    return n, cin, params, macs

for name, crop, k in (("ADE20K", 473, 150), ("Cityscapes", 713, 19)):
    for ppm in (False, True):
        n, cin, p, m = head_cost(crop, k, ppm=ppm)
        print(f"{name:10s} crop {crop}: map {n}x{n}, {'PPM ' if ppm else 'plain'} head on {cin} channels: "
              f"{p / 1e6:5.1f} M parameters, {m / 1e9:6.1f} GMACs")
for b, crop in ((1, 473), (2, 473), (3, 473), (6, 473), (6, 713)):
    n = (crop - 1) // 8 + 1
    print(f"bin {b}x{b} on a {n}x{n} map: pooling window {n // b}x{n // b} "
          f"({100 * (n // b) / n:.0f}% of the side)")

Output:

ADE20K     crop 473: map 60x60, plain head on 2048 channels:   9.5 M parameters,   34.3 GMACs
ADE20K     crop 473: map 60x60, PPM  head on 4096 channels:  23.1 M parameters,   68.3 GMACs
Cityscapes crop 713: map 90x90, plain head on 2048 channels:   9.4 M parameters,   76.5 GMACs
Cityscapes crop 713: map 90x90, PPM  head on 4096 channels:  23.1 M parameters,  153.0 GMACs
bin 1x1 on a 60x60 map: pooling window 60x60 (100% of the side)
bin 2x2 on a 60x60 map: pooling window 30x30 (50% of the side)
bin 3x3 on a 60x60 map: pooling window 20x20 (33% of the side)
bin 6x6 on a 60x60 map: pooling window 10x10 (17% of the side)
bin 6x6 on a 90x90 map: pooling window 15x15 (17% of the side)

The pooling branches themselves are nearly free: their $1 \times 1$ convolutions run on 50 pooled positions in total, about 0.05 GMACs. The cost of the module is indirect, since doubling the channels that reach the $3 \times 3$ head convolution doubles that layer, from 34.3 to 68.3 GMACs for a $473 \times 473$ crop. Whether that is “not much” depends on the backbone: the DeepLab article counts 102.7 GMACs for a dilated ResNet-50 at output stride 8 on a $513 \times 513$ input, so by our estimate (scaled by map area, and ignoring the different stem) the module adds roughly a quarter to a ResNet-50 model’s total. The last lines show that the bins are fixed fractions of the map, so the pooled regions grow with the input; a crop of 713 has $15 \times 15$ windows at the finest level where a crop of 473 has $10 \times 10$.

Example: global and sub-region context

The paper’s motivation can be reproduced on synthetic scenes in which an object’s class is decided only by context outside any local window. Each $72 \times 72$ image has a gray noise background, small $3 \times 3$ markers whose color sets the scene type (red or green), and $8 \times 8$ textured gray squares labelled “boat” in a red scene and “car” in a green one. The squares look the same in both classes, and every marker is at least 16 pixels from every object center, which keeps all markers outside the $21 \times 21$ window around any object pixel: the receptive field of the plain head’s map cells, before the final bilinear upsampling mixes neighbouring cells 4 pixels apart. A small backbone at output stride 4 ($18 \times 18$ map, receptive field 13 pixels) feeds one of three heads: a plain $3 \times 3$ head; a global-pooling head (“B1” in the paper’s ablation), which concatenates one upsampled $1 \times 1$ pooled branch with 64 channels; and a pyramid pooling module with bins 1, 2, 3 and 6 and the paper’s 1/N reduction (16 channels per level). The global and pyramid heads have the same width and parameter count.

Which head should win depends on how the context is laid out, which the paper does not specify for an experiment like this, so the script runs two layouts. In the global layout, one scene type governs the whole image (one object, six markers). In the halves layout, the left and right halves each have their own random scene type, object and three markers, so a single image-wide summary cannot tell which cue belongs to which object. The script trains each head for 500 steps of 16 images with Adam on 2,000 images and scores 500 test images by per-class IoU, their mean over background, boat and car, and the accuracy on object pixels, where guessing gives about 0.5. It ran with PyTorch 2.14.1 on four CPU threads in about five minutes.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(4)
S, OBJ, MK = 72, 8, 3                      # image size, object side, marker side

def make_scene(rng, layout):
    """Gray noise background; 3x3 markers whose colour gives the scene type (red: 0, green: 1);
    8x8 gray objects labelled boat (1) under scene type 0 and car (2) under type 1."""
    img = 0.5 + 0.08 * rng.standard_normal((3, S, S))
    lab = np.zeros((S, S), dtype=np.int64)
    regions = [(0, S)] if layout == "global" else [(0, S // 2), (S // 2, S)]
    n_mk = 6 if layout == "global" else 3
    scenes, objs = [], []
    for x0, x1 in regions:                  # one object per region
        scene = rng.integers(2)
        scenes.append(scene)
        cy, cx = rng.integers(OBJ // 2, S - OBJ // 2), rng.integers(x0 + OBJ // 2, x1 - OBJ // 2)
        objs.append((cy, cx))
        sl = (slice(cy - OBJ // 2, cy + OBJ // 2), slice(cx - OBJ // 2, cx + OBJ // 2))
        img[(slice(None),) + sl] = 0.5 + 0.25 * rng.standard_normal((3, OBJ, OBJ))
        lab[sl] = 1 + scene
    for (x0, x1), scene in zip(regions, scenes):
        placed = 0
        while placed < n_mk:                # markers at least 16 px from every object centre
            my, mx = rng.integers(1, S - 1), rng.integers(x0 + 1, x1 - 1)
            if all(max(abs(my - cy), abs(mx - cx)) >= 16 for cy, cx in objs):
                colour = np.array([0.9, 0.2, 0.2]) if scene == 0 else np.array([0.2, 0.9, 0.2])
                img[:, my - 1:my + 2, mx - 1:mx + 2] = colour[:, None, None] + 0.05 * rng.standard_normal((3, MK, MK))
                placed += 1
    return img.astype(np.float32), lab

def make_set(n, layout, seed):
    rng = np.random.default_rng(seed)
    xs, ys = zip(*(make_scene(rng, layout) for _ in range(n)))
    return torch.tensor(np.stack(xs)), torch.tensor(np.stack(ys))

def cbr(cin, cout, k=3, s=1):
    return nn.Sequential(nn.Conv2d(cin, cout, k, s, k // 2, bias=False), nn.BatchNorm2d(cout), nn.ReLU())

class Net(nn.Module):
    """Backbone at output stride 4 (18 x 18 map), then a plain head, a global-pooling head (B1)
    or a pyramid pooling module with bins (1, 2, 3, 6) and 1/N channel reduction."""
    def __init__(self, kind, bins=(1, 2, 3, 6), c=64, classes=3):
        super().__init__()
        self.backbone = nn.Sequential(cbr(3, 16), cbr(16, 32, s=2), cbr(32, 32), cbr(32, c, s=2))
        self.kind = kind
        if kind == "b1":
            bins, red = (1,), c                  # one global branch, no reduction
        else:
            red = c // len(bins)
        self.branches = nn.ModuleList(
            nn.Sequential(nn.AdaptiveAvgPool2d(b), cbr(c, red, k=1)) for b in bins) if kind != "plain" else None
        cin = c if kind == "plain" else c + red * len(bins)
        self.head = nn.Sequential(cbr(cin, c), nn.Conv2d(c, classes, 1))

    def forward(self, x):
        f = self.backbone(x)
        if self.branches is not None:
            f = torch.cat([f] + [F.interpolate(b(f), size=f.shape[-2:], mode="bilinear", align_corners=False)
                                 for b in self.branches], dim=1)
        return F.interpolate(self.head(f), size=x.shape[-2:], mode="bilinear", align_corners=False)

def train_and_test(kind, layout, steps=500, batch=16, seed=0):
    Xtr, Ytr = make_set(2000, layout, seed=1)
    Xte, Yte = make_set(500, layout, seed=2)
    torch.manual_seed(seed)
    net = Net(kind)
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    net.train()
    for _ in range(steps):
        idx = torch.randint(0, len(Xtr), (batch,))
        loss = F.cross_entropy(net(Xtr[idx]), Ytr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    net.eval()
    with torch.no_grad():
        pred = torch.cat([net(Xte[i:i + 100]).argmax(1) for i in range(0, len(Xte), 100)])
    conf = torch.bincount(3 * Yte.flatten() + pred.flatten(), minlength=9).reshape(3, 3).double()
    iou = conf.diag() / (conf.sum(0) + conf.sum(1) - conf.diag())
    obj = Yte > 0
    obj_acc = (pred[obj] == Yte[obj]).double().mean().item()
    nparams = sum(p.numel() for p in net.parameters())
    return iou, obj_acc, nparams

for layout in ("global", "halves"):
    for kind in ("plain", "b1", "ppm"):
        iou, obj_acc, nparams = train_and_test(kind, layout)
        print(f"{layout:6s} {kind:5s} params {nparams:6d} | IoU bg/boat/car "
              f"{iou[0]:.3f}/{iou[1]:.3f}/{iou[2]:.3f}  mIoU {iou.mean():.3f} | object-pixel accuracy {obj_acc:.3f}")

Output:

global plain params  70163 | IoU bg/boat/car 0.998/0.465/0.003  mIoU 0.488 | object-pixel accuracy 0.440
global b1    params 111251 | IoU bg/boat/car 0.998/0.845/0.845  mIoU 0.896 | object-pixel accuracy 0.904
global ppm   params 111251 | IoU bg/boat/car 0.998/0.856/0.856  mIoU 0.903 | object-pixel accuracy 0.916
halves plain params  70163 | IoU bg/boat/car 0.997/0.123/0.421  mIoU 0.514 | object-pixel accuracy 0.439
halves b1    params 111251 | IoU bg/boat/car 0.995/0.495/0.556  mIoU 0.682 | object-pixel accuracy 0.653
halves ppm   params 111251 | IoU bg/boat/car 0.996/0.675/0.566  mIoU 0.746 | object-pixel accuracy 0.739

The plain head finds the objects (background IoU near 1) but cannot name them: in the global layout it calls almost every object a boat, so the car IoU is near zero and its object accuracy is at or below chance. This is the paper’s boat-and-car failure in miniature. Both pooled heads recover most objects in the global layout, and there the single global vector does about as well as the pyramid, since one summary of the image is all the task needs. In the halves layout this run gives the order the paper’s argument predicts: plain at chance, global pooling in between, the pyramid ahead by about 9 points of object accuracy. By our reasoning, the pyramid can do better here because its $2 \times 2$ and $6 \times 6$ bins split the image at the boundary between the halves, while the global vector mixes the two cues.

The effect is weaker than the task allows; whether longer training would strengthen it was not tested. By our reasoning, the halves of an image share a scene type about half the time, so a head that knows which cues are present but not where could reach about 75% object accuracy (right when the halves agree, guessing otherwise); the pyramid head’s 0.739 does not clearly exceed that, and both pooled heads stay below their global-layout scores. The example also used one training seed, so the spread across seeds is not measured, and the gaps between the pooled heads (about 1 point in the global layout, about 9 in the halves layout) cannot be separated from run-to-run variation. The run is consistent with the direction the paper predicts; it does not measure the size of the effect.

Training

The objective is the per-pixel softmax cross-entropy of the main classifier plus a weighted cross-entropy of an auxiliary classifier:

$$\mathcal{L} = \mathcal{L}_{\text{main}} + \alpha\, \mathcal{L}_{\text{aux}}, \qquad \alpha = 0.4,$$

with both losses taken over labelled pixels (semseg ignores label 255 and upsamples both score maps to the label size before the loss). The auxiliary classifier sits after the fourth stage, at block res4b22 of ResNet-101; in semseg it is a $3 \times 3$ convolution from 1,024 to 256 channels with batch normalization, ReLU and dropout 0.1, then a $1 \times 1$ classifier. Unlike relay backpropagation, which the paper describes as blocking the backward auxiliary loss to several shallow layers, both losses pass through all earlier layers. The auxiliary branch is discarded at test time.

The authors present this as a decomposition of the optimization: the auxiliary loss produces an initial result at stage 4 and the remaining layers learn the residue. The paper calls this a deeply supervised loss; it cites deeply supervised nets (Lee et al., 2015), which add companion objectives at each hidden layer, for the optimization difficulty of deeper networks. On ADE20K val with a dilated ResNet-50 FCN and no pyramid module, the auxiliary loss raised mean IoU from 35.82% to 37.01%, 37.23%, 37.09% and 36.99% for $\alpha$ = 0.3, 0.4, 0.6 and 0.9.

The procedure, implemented in a modified Caffe:

• Schedule. SGD with momentum 0.9 and weight decay 0.0001; the “poly” learning-rate schedule multiplies the base rate 0.01 by $(1 - t/T)^{0.9}$ at iteration $t$ of $T$, following DeepLab. Training ran 150K iterations on ADE20K, 30K on VOC and 90K on Cityscapes; the authors note that more iterations improved results.
• Augmentation. Random mirroring and random resizing between 0.5 and 2 for all datasets, plus random rotation between −10 and 10 degrees and random Gaussian blur for ADE20K and VOC (data augmentation).
• Batch normalization. The authors found that a large crop and a large batch normalization batch mattered; they used batch size 16 and modified Caffe so that batch statistics were gathered across GPUs. The crop sizes are not given in the paper; the released models use $473 \times 473$ for ADE20K and VOC and $713 \times 713$ for Cityscapes (deploy prototxts; semseg configs).

semseg’s training uses the same values (batch size 16 with synchronized batch normalization, base rate 0.01, power 0.9, auxiliary weight 0.4, scaling 0.5–2.0, rotation ±10°), counts training in epochs (100 for ADE20K, 50 for VOC, 200 for Cityscapes) and gives the newly added layers 10 times the backbone’s learning rate.

Inference

At test time only the main branch runs. The released evaluation scripts resize each image so that its longer side is a base size (512 for ADE20K and VOC, 2,048 for Cityscapes) times a scale, plus one pixel, classify it with crops of the training size in a sliding window with a stride of two-thirds of the crop when it is larger than one crop, average each pixel’s scores over the windows that cover it, and add the scores of the horizontally flipped input. Multi-scale testing averages the probabilities over scales 0.5, 0.75, 1, 1.25, 1.5 and 1.75. The paper’s VOC and Cityscapes test results average several input scales; its ADE20K tables label which rows use multi-scale testing. No CRF or other post-processing is used.

Performance

ADE20K (SceneParse150). The challenge split has about 20K training, 2K validation and 3K test images with 150 classes, scored by pixel accuracy and mean IoU; the challenge score is their mean (see ADE20K for the protocol). The paper’s step-by-step analysis on the validation set, single-scale except the last row:

Model	Mean IoU (%)	Pixel acc. (%)
FCN (reported by the ADE20K authors)	29.39	71.32
DilatedNet (reported by the ADE20K authors)	32.31	73.55
Dilated ResNet-50 FCN baseline	34.28	76.35
+ data augmentation	35.82	77.07
+ auxiliary loss	37.23	78.01
+ pyramid pooling module (PSPNet, ResNet-50)	41.68	80.04
PSPNet, ResNet-269	43.81	80.88
PSPNet, ResNet-269, multi-scale testing	44.94	81.69

The module’s ablation (ResNet-50, val, single-scale) compares one global bin with the four-level pyramid: average pooling beat max pooling in every setting (global bin 40.07% against 39.94%; pyramid 41.07% against 40.18%), the pyramid beat the global bin, and reducing the channels after pooling helped further (41.68% average, 40.87% max). The paper’s “ResNet50-Baseline” in this ablation table scores 37.23/78.01, the row that the analysis table above calls the baseline with augmentation and auxiliary loss; the analysis table’s own “ResNet50-Baseline” is 34.28/76.35. The improvement of 4.45 points that the paper reports for the module is measured from the former. Deeper backbones helped: single-scale mean IoU was 41.68, 41.96, 42.62 and 43.81% for ResNet-50, 101, 152 and 269.

In the challenge, scored on the test set, the team’s ensemble reached 57.21% and its best single model 55.38%; the authors noted that the single-model score was lower than on the validation set, possibly because of a difference between the validation and test distributions. The organizers’ results page lists the winning entry, from team SenseCUSceneParsing, as an ensemble trained on train and val data with score 0.57205, ahead of Adelaide (0.56735) and 360+MCG-ICT-CAS_SP (0.55565), among 75 submissions from 22 teams; the single model at 0.5538 is labelled as trained on the train data.

PASCAL VOC 2012. Using the 10,582-image augmented training set, a ResNet-101 PSPNet without CRF and with multi-scale testing reached 82.6% mean IoU on the test set trained on VOC data only and 85.4% with COCO pretraining, the best on all 20 classes and on 19 of the 20 respectively. The best COCO-pretrained ResNet-based entries in the paper’s table are DeepLab (79.7%), LRR (79.3%) and FCRNs (79.1%). The paper’s text calls these figures “accuracy”; its table reports per-class results and mean IoU.

Cityscapes. On the test set, a ResNet-101 PSPNet trained on the fine annotations scored 78.4% class IoU (56.7% class iIoU, 90.6% category IoU), against 70.4% for DeepLab and 71.6% for the best other entry in the paper’s table; adding the coarse annotations gave 80.2% class IoU. The paper does not say whether the fine-only model was trained on the training set alone or with the validation set. The official repository lists 79.70% mean IoU on the validation set with single-scale testing and 80.91% with multi-scale testing for its released Cityscapes model.

The repository also lists the expected scores of the other two released Caffe models: 41.68/80.04 (single-scale) and 42.78/80.76 (multi-scale) on ADE20K val for PSPNet-50, matching the paper, and 85.41% on VOC 2012 test for PSPNet-101. The later PyTorch reimplementation in semseg reports, for example, 41.89% (ADE20K val, ResNet-50), 79.07% (VOC 2012 val, ResNet-101, trained on the augmented training set) and 78.63% (Cityscapes val, ResNet-101, fine training set), all single-scale.

Strengths

• Context at negligible pooling cost. Region averages at four scales give every pixel an image-level and regional summary; the pooling and its $1 \times 1$ convolutions are a tiny part of the computation.
• Scale with the input. Because the bins are fractions of the map, the pooled regions adapt to the input size, unlike filters whose reach is fixed in pixels.
• A transferable module. The module can be appended to any backbone’s last feature map; UPerNet appended it to a feature pyramid network, as below.
• A documented recipe. The paper and its code give the details that the authors call key to their results: the poly schedule, synchronized batch normalization with 16 images, augmentation and the auxiliary weight.

Limitations

• Cost of the dilated backbone. Output stride 8 for the whole of stages 4 and 5 is expensive, and the module doubles the input of the $3 \times 3$ head convolution. Reproducing the models, the UPerNet authors report 61.1 hours of training for PSPNet with ResNet-50 against 38.7 hours for their FPN-based model with the same module, which reached 41.22/79.98 on ADE20K val against PSPNet’s 41.68/80.04 in their table.
• Coarse boundaries. Scores are predicted at 1/8 resolution and upsampled bilinearly, with no decoder or CRF; the pooled context itself is piecewise constant over bins before upsampling.
• Fixed geometry. Square bins on a fixed grid do not follow object or region shapes; a region boundary that cuts through bins mixes context, as in the halves example. The released Caffe models also tie their pooling windows to one crop size and so rely on sliding-window inference for larger images.
• Dependence on test-time averaging. Several headline results use multi-scale testing with flipping, which multiplies the cost of inference.
• Validation–test gap on ADE20K. The single model’s challenge score was lower than on the validation set, and the authors could only suggest a distribution difference.

Historical Importance

PSPNet was one of the context-aggregation models that defined semantic segmentation in 2016–2017. It established the pyramid pooling module, the auxiliary loss on a dilated ResNet, and a training recipe whose values (base rate 0.01, poly power 0.9, crop 473 or 713, synchronized batch normalization, auxiliary weight 0.4) are carried over unchanged into its first author’s semseg codebase. By our reading, it also illustrates a shift in what the hard part of segmentation was taken to be: FCN and DeepLab v1 worked mainly on resolution and boundaries, while PSPNet framed the main errors on diverse scenes as errors of context.

Its closest relative is DeepLab’s atrous spatial pyramid pooling. Both sit on the same kind of backbone, since PSPNet adopted DeepLab’s dilated strategy, and both gather multi-scale context on the last feature map in parallel branches. They differ in how they sample it. The ASPP of DeepLab v2 (preprint June 2016) applies $3 \times 3$ atrous filters with rates 6 to 24 and sums the branches’ class scores: each output reads nine taps at fixed offsets on the feature map. The pyramid pooling module averages every position in regions that are fixed fractions of the map, so its reach scales with the image and all positions in a bin contribute, but the position of a feature within a bin is lost. DeepLabv3 (2017) rebuilt ASPP with batch normalization, $1 \times 1$ fusion of 256-channel branches, and an image-level branch of global average pooling and a $1 \times 1$ convolution, which corresponds to PSPNet’s $1 \times 1$ bin; the DeepLab article describes that version and its reasons. By our reading, after 2017 the two designs converged more than they competed: both are a dilated backbone, a set of parallel context branches including a global one, concatenation and a classifier.

Variants

The authors’ later work kept the backbone and recipe. PSANet, the point-wise spatial attention network (Zhao et al., ECCV 2018), is released in the same semseg codebase, whose README also links video predictions of ICNet under “high efficiency”. UPerNet (Xiao et al., 2018) appended the pyramid pooling module to the top of a feature pyramid network before its top-down path, citing the gap between theoretical and empirical receptive fields; on ADE20K val with ResNet-50, adding the module to their FPN raised mean IoU from 35.26% to 40.13%. PSPNet has also served as a reference model; HRNet’s comparisons on Cityscapes, for example, include it.

Implementations

• Caffe (official, hszhao/PSPNet). Modified from the Caffe versions of DeepLab v2 and of yjxiong’s fork, with deploy prototxts and MATLAB evaluation scripts for the three released models (PSPNet-50 on ADE20K, PSPNet-101 on VOC 2012 and on Cityscapes). The README links the trained weights; the repository holds no PSPNet training prototxts or solver files, and the last commit dates from April 2022 (as of November 2026).
• PyTorch (hszhao/semseg, by the first author). Training and testing code for PSPNet and PSANet with ResNet-50, 101 and 152, synchronized batch normalization and distributed training, tested with PyTorch 1.4.0 at its last update in May 2020 (MIT licence). Its configurations and reported reimplementation results cover ADE20K, PASCAL VOC 2012 and Cityscapes.
• Other reimplementations. The ADE20K authors’ semantic-segmentation-pytorch code reimplements PSPNet and UPerNet (see ADE20K).