FCN

FCN, the fully convolutional network of Jonathan Long, Evan Shelhamer and Trevor Darrell (CVPR 2015), is a semantic segmentation network trained end to end, pixels to pixels. The authors described it as, to their knowledge, the first work to train fully convolutional networks end to end for pixelwise prediction and from supervised pretraining. It takes an ImageNet-trained CNN, rewrites its fully connected layers as convolutions so that any image maps to a coarse grid of class scores, upsamples the scores inside the network with transposed convolutions, and refines them with skips from finer layers. Its VGG-16 variants FCN-32s, FCN-16s and FCN-8s are named after the stride of their last prediction. A journal version (Shelhamer, Long & Darrell, 2017, IEEE TPAMI) retuned training and reported higher scores.

Problem Addressed

In the authors’ survey, earlier convolutional approaches to dense prediction used small models, patchwise training and post-processing such as superpixel projection or random fields. The previous best system on PASCAL VOC, SDS (Hariharan et al., 2014), classified region proposals with a fine-tuned R-CNN and was not learned end to end. FCN’s premise is that an ImageNet classifier already computes most of what dense labeling needs; the obstacles are that its fully connected layers discard spatial coordinates and that strides and pooling leave its output coarse. The authors call the second a tension between semantics and location.

Architecture

Convolutionalized classifier

Convolution, pooling and pointwise nonlinearities each compute an output from a local input window. Kernel size $k$ and stride $s$ compose as

$$f_{ks} \circ g_{k's'} = (f \circ g)_{k' + (k-1)s',\; ss'},$$

so a network of such layers, which the paper calls a deep filter or fully convolutional network, accepts any input size and returns an output of corresponding, subsampled size. A fully connected layer that sees a whole $7 \times 7 \times 512$ map is just a $7 \times 7$ convolution evaluated at one position. FCN keeps the weights and reinterprets them: in the released FCN-VGG16 models, fc6 is a $7 \times 7$ convolution and fc7 a $1 \times 1$ convolution, each with 4,096 outputs, and the ImageNet classifier is replaced by a $1 \times 1$ convolution with 21 outputs, one per PASCAL class including background. On a larger input the network evaluates the classifier on a grid of windows with the network’s total stride, 32 for VGG-16, sharing the computation of overlapping windows. In the CVPR paper, convolutionalized AlexNet produced a $10 \times 10$ grid from a $500 \times 500$ image in 22 ms, more than five times faster than classifying the crops as separate images, even in efficient batches.

The NumPy example below builds a small classifier on $7 \times 7 \times 3$ feature windows, a miniature of VGG’s pool5, computed from a textured image. One fc6 unit is a matched filter for a centered square. It checks the convolutionalized network against the fully connected one on every crop and computes FCN-VGG16’s fc6 map size with and without 100-pixel padding.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

rng = np.random.default_rng(0)

def features(img, s=4):
    """A stand-in for a CNN's conv layers: intensity and gradient-magnitude
    channels, average-pooled by s (total stride s). Returns h x w x 3."""
    gy, gx = np.gradient(img)
    ch = np.stack([img, np.abs(gx), np.abs(gy)], axis=-1)
    H, W, C = ch.shape
    return ch.reshape(H // s, s, W // s, s, C).mean(axis=(1, 3))

def scene(H, W, squares):
    """Gray textured background with bright 16 x 16 squares at the given corners."""
    yy, xx = np.mgrid[0:H, 0:W]
    img = 0.3 + 0.05 * np.sin(xx / 3.0) + 0.02 * rng.standard_normal((H, W))
    for r, c in squares:
        img[r:r + 16, c:c + 16] = 0.9
    return img

# The "classifier" was trained on 28 x 28 crops -> 7 x 7 x 3 feature windows (like VGG's pool5).
k, C, hidden, K = 7, 3, 16, 2
W6 = rng.standard_normal((hidden, k * k * C)) * 0.1      # fc6: 147 inputs -> 16 units
crop = scene(28, 28, [(6, 6)])                            # a centered square
t = features(crop).ravel()
W6[0] = (t - t.mean()) / np.linalg.norm(t - t.mean())     # unit 0: a matched filter for "square"
b6 = np.zeros(hidden)
W7 = rng.standard_normal((hidden, hidden)) * 0.3; b7 = np.zeros(hidden)   # fc7
W8 = rng.standard_normal((K, hidden)) * 0.3; b8 = np.zeros(K)             # classifier
relu = lambda z: np.maximum(z, 0)

def classify_crop(f):                    # the original net: flatten, then fully connected layers
    h6 = relu(W6 @ f.ravel() + b6)
    h7 = relu(W7 @ h6 + b7)
    return W8 @ h7 + b8, h6

def fully_conv(F):                       # the convolutionalized net
    win = sliding_window_view(F, (k, k), axis=(0, 1))     # h' x w' x C x k x k
    win = win.transpose(0, 1, 3, 4, 2)                    # h' x w' x k x k x C (same order as ravel)
    K6 = W6.reshape(hidden, k, k, C)                      # fc6 as a 7 x 7 convolution
    h6 = relu(np.einsum("ijuvc,ouvc->ijo", win, K6) + b6)
    h7 = relu(h6 @ W7.T + b7)                             # fc7 as a 1 x 1 convolution
    return h7 @ W8.T + b8, h6                             # classifier as a 1 x 1 convolution

# 1. On a single 28 x 28 crop, the two are the same function.
s_fc, _ = classify_crop(features(crop))
s_cv, _ = fully_conv(features(crop))
print("crop: fc scores", s_fc.round(4), "| conv output", s_cv.shape, s_cv[0, 0].round(4))

# 2. On a larger image the convolutionalized net returns a map of scores.
img = scene(96, 128, [(20, 24), (56, 84)])
F = features(img)
S, H6 = fully_conv(F)
print("image", img.shape, "-> features", F.shape, "-> fc6", H6.shape, "-> scores", S.shape)

# Every map cell equals the original net applied to the matching 28 x 28 crop (stride 4).
err = max(np.abs(S[i, j] - classify_crop(F[i:i + k, j:j + k])[0]).max()
          for i in range(S.shape[0]) for j in range(S.shape[1]))
print(f"max |map - per-crop classifier| over {S.shape[0] * S.shape[1]} crops: {err:.1e}")

# The "square" unit of fc6 peaks where a crop is centered on a square.
peaks = np.argwhere(H6[..., 0] > 0.95 * H6[..., 0].max())
print("fc6 unit-0 peaks at map cells", peaks.tolist(),
      "-> crop centers in pixels", (4 * peaks + 14).tolist())

# 3. Output size of FCN-VGG16 (13 'same' 3x3 convs, 5 max pools, fc6 as an unpadded 7x7 conv),
#    with conv1_1 padded by 1 (as in VGG) or by 100 (as in the reference FCN models).
def fc6_size(n, pad):
    n = n + 2 * pad - 2                  # conv1_1 (3 x 3); all other convs keep the size
    for _ in range(5):
        n = -(-n // 2)                   # 2 x 2 max pools, stride 2 (Caffe rounds up)
    return n - 6                         # fc6: 7 x 7, no padding

show = lambda m: f"{m} x {m}" if m > 0 else "none"
for n in (100, 224, 500):
    print(f"input {n} x {n}: fc6 map {show(fc6_size(n, 1))} with pad 1, {show(fc6_size(n, 100))} with pad 100")

Output:

crop: fc scores [-0.6297 -0.8364] | conv output (1, 1, 2) [-0.6297 -0.8364]
image (96, 128) -> features (24, 32, 3) -> fc6 (18, 26, 16) -> scores (18, 26, 2)
max |map - per-crop classifier| over 468 crops: 4.4e-16
fc6 unit-0 peaks at map cells [[3, 4], [3, 5], [4, 4], [4, 5], [12, 19], [12, 20], [13, 19], [13, 20]] -> crop centers in pixels [[26, 30], [26, 34], [30, 30], [30, 34], [62, 90], [62, 94], [66, 90], [66, 94]]
input 100 x 100: fc6 map none with pad 1, 4 x 4 with pad 100
input 224 x 224: fc6 map 1 x 1 with pad 1, 8 x 8 with pad 100
input 500 x 500: fc6 map 10 x 10 with pad 1, 16 x 16 with pad 100

The $96 \times 128$ image yields an $18 \times 26$ score map, one entry per 28-pixel crop at stride 4, each equal to the fully connected network’s output. The square unit peaks on the four cells whose crops are centered within 2 pixels of each square’s center, (28, 32) and (64, 92); a stride-4 grid cannot do better, which is the coarseness the rest of FCN addresses.

Padding. The reference Caffe models pad the first convolution by 100 pixels. By our arithmetic, with VGG’s padding of 1 any input of 192 pixels or less leaves nothing for the unpadded fc6 (Caffe rounds pooled sizes up). The repository’s FAQ says the padding guarantees that the output can be aligned to the input for any input size in the datasets used, with crop layers removing the excess. The papers do not mention it.

Dense output: shift-and-stitch or upsampling

Shift-and-stitch makes a stride-$f$ output dense by running the network on $f^2$ shifted inputs and interlacing the outputs. The CVPR paper shows the equivalent network change, removing a layer’s subsampling and enlarging later filters with zeros, relates it to the à trous algorithm and calls it filter rarefaction; the journal version calls it filter dilation. The filters keep their receptive fields but cannot see finer detail. The authors preferred learned upsampling.

Upsampling as backward convolution. The CVPR paper upsamples by an integer factor $f$ with a backwards convolution of output stride $f$, “sometimes called deconvolution”, today’s transposed convolution, whose filter may be learned. In both papers the final upsampling of each net is fixed to bilinear interpolation, and intermediate $2\times$ layers start as bilinear and are learned. The repository’s FAQ adds that its models fix all interpolation kernels, with no significant difference in accuracy.

Skip architecture: FCN-32s, FCN-16s, FCN-8s

FCN-32s, the convolutionalized VGG-16, upsamples its stride-32 scores in one step, and the authors found its output “dissatisfyingly coarse”. FCN-16s adds a $1 \times 1$ scoring layer on pool4 (stride 16) and sums its scores with a $2\times$ upsampling of the conv7 scores; FCN-8s repeats this with pool3. With $U_2$ a learned $2\times$ upsampling and $W_l$ the scoring layer on layer $l$,

$$s_{16} = U_2(s_{32}) + W_{4} \ast \text{pool4}, \qquad s_{8} = U_2(s_{16}) + W_{3} \ast \text{pool3}, \qquad \hat{y} = U_8(s_{8}),$$

each sum preceded by a crop that aligns the maps. Max fusion made learning difficult because of gradient switching. The journal version notes that summing scores equals concatenating the layers and applying one $1 \times 1$ classifier, both being linear. New scoring layers start at zero, so an upgraded net begins with its predecessor’s predictions.

The PyTorch example (run with PyTorch 2.14.1) scales the design down to synthetic $64 \times 64$ scenes: one to three red or blue squares and disks on noisy stripes, labeled by color. Three conv–pool stages reach stride 8; a $3 \times 3$ and a $1 \times 1$ convolution play fc6 and fc7. The FCN-32s-like net upsamples its stride-8 scores in one fixed bilinear step; the FCN-8s-like net adds zero-initialized scores from the stride-4 and stride-2 features, summed after learnable, bilinear-initialized $2\times$ transposed convolutions. Both train all at once with the same seed, data order and budget (250 Adam steps of 16 images; Adam replaces the paper’s SGD for speed). Boundary accuracy is pixel accuracy within 2 pixels of a true label change.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)
K = 3                                   # classes: 0 background, 1 red object, 2 blue object

def make_scenes(n, g):
    """64 x 64 RGB scenes: noisy stripes of random frequency and angle, with 1-3 red or
    blue squares and disks (the shape does not depend on the class)."""
    yy, xx = torch.meshgrid(torch.arange(64.), torch.arange(64.), indexing="ij")
    imgs, masks = torch.empty(n, 3, 64, 64), torch.zeros(n, 64, 64, dtype=torch.long)
    for i in range(n):
        f, th = 0.3 + 0.5 * torch.rand(1, generator=g), 3.1416 * torch.rand(1, generator=g)
        stripes = torch.sin(f * (xx * torch.cos(th) + yy * torch.sin(th)))
        imgs[i] = 0.5 + 0.25 * stripes * torch.rand(3, 1, 1, generator=g) \
            + 0.1 * torch.randn(3, 64, 64, generator=g)
        for _ in range(int(torch.randint(1, 4, (1,), generator=g))):
            cls = int(torch.randint(1, 3, (1,), generator=g))
            r = float(torch.randint(5, 13, (1,), generator=g))         # half-width or radius
            cy, cx = (r + torch.rand(2, generator=g) * (64 - 2 * r)).tolist()
            if torch.rand(1, generator=g) < 0.5:
                m = ((yy - cy).abs() <= r) & ((xx - cx).abs() <= r)
            else:
                m = (yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2
            base = torch.tensor([0.75, 0.3, 0.25]) if cls == 1 else torch.tensor([0.25, 0.35, 0.75])
            color = base.view(3, 1, 1) + 0.15 * torch.rand(3, 1, 1, generator=g)
            imgs[i] = torch.where(m, color + 0.1 * torch.randn(3, 64, 64, generator=g), imgs[i])
            masks[i][m] = cls
    return imgs, masks

def upsample(factor, learn):
    """Transposed convolution (kernel 2f, stride f) initialized to bilinear interpolation,
    one filter per class as in FCN; `learn` decides whether training may change it."""
    k = 2 * factor
    w1 = 1 - (torch.arange(k) - (k - 1) / 2).abs() / factor
    layer = nn.ConvTranspose2d(K, K, k, stride=factor, padding=factor // 2, bias=False)
    with torch.no_grad():
        layer.weight.zero_()
        layer.weight[range(K), range(K)] = torch.outer(w1, w1)
    layer.weight.requires_grad_(learn)
    return layer

def conv_relu(cin, cout, k=3):
    conv = nn.Conv2d(cin, cout, k, padding=k // 2)
    nn.init.kaiming_normal_(conv.weight, nonlinearity="relu"); nn.init.zeros_(conv.bias)
    return [conv, nn.ReLU()]

class TinyFCN(nn.Module):
    def __init__(self, skips):
        super().__init__()
        self.skips = skips
        self.stage2 = nn.Sequential(*conv_relu(3, 8), nn.MaxPool2d(2))                  # stride 2
        self.stage4 = nn.Sequential(*conv_relu(8, 16), nn.MaxPool2d(2))                 # stride 4
        self.stage8 = nn.Sequential(*conv_relu(16, 32), *conv_relu(32, 32), nn.MaxPool2d(2))
        self.head = nn.Sequential(*conv_relu(32, 64), *conv_relu(64, 64, k=1))          # "fc6", "fc7"
        self.score8 = nn.Conv2d(64, K, 1)                       # 1 x 1 classifier at stride 8
        if skips:
            self.score4, self.score2 = nn.Conv2d(16, K, 1), nn.Conv2d(8, K, 1)
            for s in (self.score4, self.score2):                # zero-initialized, as in FCN
                nn.init.zeros_(s.weight); nn.init.zeros_(s.bias)
            self.up8to4, self.up4to2 = upsample(2, learn=True), upsample(2, learn=True)
            self.up2to1 = upsample(2, learn=False)
        else:
            self.up8to1 = upsample(8, learn=False)              # one 8x step

    def forward(self, x):
        f2 = self.stage2(x); f4 = self.stage4(f2)
        s = self.score8(self.head(self.stage8(f4)))
        if not self.skips:
            return self.up8to1(s)
        s = self.up8to4(s) + self.score4(f4)                    # fuse by summation at stride 4
        s = self.up4to2(s) + self.score2(f2)                    # and at stride 2
        return self.up2to1(s)

def train(net, steps, lr, seed):
    torch.manual_seed(seed)
    opt = torch.optim.Adam([p for p in net.parameters() if p.requires_grad], lr=lr)
    for _ in range(steps):
        idx = torch.randint(0, len(Xtr), (16,))
        loss = F.cross_entropy(net(Xtr[idx]), Ytr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    return net

def evaluate(net, imgs, masks):
    """Mean IoU over the 3 classes, and pixel accuracy within 2 px of a true boundary."""
    with torch.no_grad():
        pred = net(imgs).argmax(1)
    n = torch.bincount(K * masks.flatten() + pred.flatten(), minlength=K * K).reshape(K, K).double()
    iou = n.diag() / (n.sum(0) + n.sum(1) - n.diag())
    m = masks[:, None].float()
    edge = (F.max_pool2d(m, 3, 1, 1) != -F.max_pool2d(-m, 3, 1, 1))  # a label change in 3 x 3
    band = F.max_pool2d(edge.float(), 3, 1, 1)[:, 0].bool()           # widened to +-2 px
    return iou.mean().item(), (pred == masks)[band].float().mean().item()

g = torch.Generator().manual_seed(1234)
Xtr, Ytr = make_scenes(600, g)
Xte, Yte = make_scenes(200, g)
print("train", tuple(Xtr.shape), "test", tuple(Xte.shape), "| pixel share per class:",
      [round(v, 2) for v in (torch.bincount(Ytr.flatten()) / Ytr.numel()).tolist()])

res = {False: [], True: []}
for seed in range(4):
    for skips in (False, True):                       # same seed, data order and budget
        torch.manual_seed(seed)
        net = train(TinyFCN(skips), 250, 3e-3, seed)
        res[skips].append(evaluate(net, Xte, Yte))
for skips, name in ((False, "stride 8, no skips"), (True, "skips from strides 4 and 2")):
    r = torch.tensor(res[skips])
    print(f"{name:<27} mIoU {r[:, 0].min():.3f}-{r[:, 0].max():.3f}   "
          f"boundary accuracy {r[:, 1].min():.3f}-{r[:, 1].max():.3f}")
d = torch.tensor(res[True]) - torch.tensor(res[False])
print(f"gain from skips, per seed: mIoU {d[:, 0].min():+.3f} to {d[:, 0].max():+.3f}, "
      f"boundary accuracy {d[:, 1].min():+.3f} to {d[:, 1].max():+.3f}")
print("output of the last net:", tuple(net(Xte[:1]).shape))

Output:

train (600, 3, 64, 64) test (200, 3, 64, 64) | pixel share per class: [0.88, 0.06, 0.06]
stride 8, no skips          mIoU 0.850-0.870   boundary accuracy 0.774-0.799
skips from strides 4 and 2  mIoU 0.880-0.895   boundary accuracy 0.813-0.835
gain from skips, per seed: mIoU +0.023 to +0.030, boundary accuracy +0.036 to +0.043
output of the last net: (1, 3, 64, 64)

The skips raised mIoU by 0.023–0.030 and boundary accuracy by 0.036–0.043 over seeds 0–3; on seeds 4–9 every seed gained too (0.015–0.041 and 0.022–0.057), so these runs are typical. The larger gain at boundaries fits, in our reading, the CVPR paper’s remark that mean IU emphasizes large-scale correctness while skips refine fine structure. The script takes about 30 s on one CPU thread; results shift slightly with the thread count.

Training

All layers are fine-tuned from the classifier by backpropagation. The loss is the per-pixel multinomial logistic (cross-entropy) loss summed over the output grid, ignoring pixels marked void:

$$\ell(\theta) = -\sum_{(i,j)\,:\,y_{ij} \neq \text{void}} \log \frac{\exp s_{y_{ij}}(i,j;\theta)}{\sum_{c=1}^{K} \exp s_{c}(i,j;\theta)} .$$

Since the loss sums over positions, whole-image training equals patchwise training on every last-layer receptive field, computed far more cheaply. Sampling the loss spatially gave no faster convergence and cost more time, so the authors trained on whole images; class balancing proved unnecessary although about three quarters of the labels are background, and mirroring and jittering gave no noticeable improvement.

CVPR recipe. SGD with momentum 0.9 on minibatches of 20 images, fixed learning rate $10^{-4}$ for FCN-VGG16, weight decay, a zero-initialized scoring layer, and the classifier’s dropout. Training is staged: FCN-32s (three days on one GPU), then FCN-16s from it with the learning rate divided by 100, then FCN-8s (about a day each). Training on the 8,498 images labeled by Hariharan et al. (2011) instead of the 1,112 of the VOC 2011 training set raised FCN-VGG16 by 3.4 points to 59.4 mean IU. Because those images overlap VOC 2011 val, validation used the 736 non-overlapping images. Fine-tuning only the classifier reached about 70% of full fine-tuning.

Journal recipe. The TPAMI version leaves the loss unnormalized, so every pixel weighs the same regardless of batch and image size, and trains online with batch size 1 and momentum 0.99, “heavy” learning. Its rationale: momentum $p$ at batch size $k$ weights past examples roughly like $p'$ at $k'$ when $p^{1/k} = p'^{1/k'}$, and $0.9^{1/20} \approx 0.99$. On a fixed sequence of 100,000 images, FCN-VGG16 scored 51.9 mean IU with 20-image accumulation, 60.7 online at momentum 0.9 and 63.6 with heavy learning; the released FCN-8s solver uses a learning rate of $10^{-14}$. The journal version also trains FCN-8s all at once from VGG-16, scaling each skip stream by a fixed constant to roughly equalize feature norms, since naive joint training tends to diverge (0.01 for pool4 and 0.0001 for pool3 in the released net). This scored 65.4 against 65.5 for staged training in half the time.

Inference

Inference is one forward pass at native size, then a per-pixel argmax. FCN-8s took about 175 ms per image in the CVPR paper (Tesla K40c), against about 50 s for SDS, and about 100 ms in the journal version (Titan X).

Performance

Backbones (CVPR Table 1; VOC 2011 val, single stream, without the extra labels):

Net	Mean IU	Forward time	Conv. layers	Parameters	Receptive field	Max stride
FCN-AlexNet	39.8	50 ms	8	57M	355	32
FCN-VGG16	56.0	210 ms	16	134M	404	32
FCN-GoogLeNet	42.5	59 ms	22	6M	907	32

GoogLeNet is the authors’ reimplementation; it did not match VGG-16 despite similar classification accuracy.

Skips (736-image VOC 2011 val subset, trained with the extra labels):

Net	CVPR mean IU	CVPR pixel acc.	TPAMI mean IU	TPAMI pixel acc.
FCN-32s, classifier only fine-tuned	45.4	83.0	46.6	82.9
FCN-32s	59.4	89.1	63.6	90.5
FCN-16s	62.4	90.0	65.0	91.0
FCN-8s	62.7	90.3	65.5 (staged), 65.4 (at once)	91.2, 91.1

As controls, scoring from pool4 alone performed poorly (22.4 mean IU in the journal version), and lowering the learning rate without a skip gave an insignificant gain.

Benchmarks:

Dataset	Version	Net	Result
PASCAL VOC 2011 / 2012 test	CVPR	FCN-8s	62.7 / 62.2 mean IU (SDS: 52.6 / 51.6)
PASCAL VOC 2011 / 2012 test	TPAMI	FCN-8s	67.5 / 67.2 mean IU
NYUDv2 test, 40 classes	CVPR	FCN-16s, RGB and HHA late fusion	34.0 mean IU, 65.4 pixel acc. (Gupta et al.: 28.6, 60.3)
SIFT Flow test	CVPR	FCN-16s, two heads	39.5 mean IU, 85.2 pixel acc.; 94.3 geometric acc.
SIFT Flow test	TPAMI	FCN-8s, two heads	41.2 mean IU, 85.9 pixel acc.; 94.6 geometric acc.
PASCAL-Context val, 59 classes	arXiv v2	FCN-8s	35.1 mean IU (CFM: 31.5)
PASCAL-Context val, 59 classes	TPAMI	FCN-8s	39.1 mean IU (CFM: 34.4)

The VOC 2012 result was a 20% relative improvement over SDS (30% in the journal version). HHA encodes depth as horizontal disparity, height above ground and angle with gravity; feeding depth as a fourth input channel helped little. PASCAL-Context is from Mottaghi et al. (2014); the CVPR paper has no results on it.

Strengths

• Simplicity. One network, one loss, one forward pass over an image of any size.
• Transfer. Any classifier can be convolutionalized, so segmentation inherits progress in classification.
• Generality. The same recipe handled RGB-D input, multi-task output and scene parsing.

Limitations

• Coarse output. Even FCN-8s predicts at stride 8, so thin structures and boundaries suffer, which mean IU hides: arXiv v2 and the journal version score ground truth downsampled by 8 and upsampled again at 96.4 mean IU on the validation subset (86.1 at factor 32) and conclude that mean IU does not measure fine-scale accuracy well.
• Limited context. Predictions rest on local receptive fields. PSPNet’s authors (Zhao et al., 2017) name the lack of global scene clues as the major issue of FCN-based models.
• Shallow decoder. Fusion is a linear sum of class scores, so in our reading coarse and fine information are never combined nonlinearly.
• Awkward reference models. The 100-pixel padding, crop layers and a $10^{-14}$ learning rate complicate porting.
• Failure cases. The papers show lifejackets in a boat labeled as people and (journal version) hair confused with a dog.

Historical Importance

FCN set the template many later segmentation networks follow: a pretrained backbone used fully convolutionally, a per-pixel loss on whole images, and in-network upsampling. Its successors attack its limits. DeepLab (Chen et al., 2018) uses atrous (dilated) convolution to reduce the downsampling from 32 to 8 and refines with a dense CRF; Yu and Koltun (2016) build a context module from dilated convolutions; PSPNet pools context at several scales; U-Net (Ronneberger et al., 2015) and other encoder–decoders replace score fusion with a feature-level decoder. Mask R-CNN predicts each object’s mask with a small FCN on its region of interest (He et al., 2017), and the feature pyramid network applies FCN-style top-down summation to features instead of scores.

Variants

FCN-32s, FCN-16s and FCN-8s, with the journal version’s FCN-8s at-once, are the main variants; FCN-AlexNet (48.0 mean IU with the extra labels) and FCN-GoogLeNet were backbone comparisons.

Implementations

• Reference Caffe implementation (fcn.berkeleyvision.org): net definitions, solvers and models for PASCAL VOC, NYUDv2, SIFT Flow and PASCAL-Context. The README’s VOC scores are the journal version’s (65.5 mean IU for FCN-8s on the validation subset, 67.2 on VOC 2012 test).
• torchvision fcn_resnet50 and fcn_resnet101 differ from the paper’s nets: a ResNet backbone with dilation instead of stride in its last two stages (output stride 8), a $3 \times 3$ convolution–batch norm–ReLU–dropout head with a $1 \times 1$ classifier, bilinear upsampling, no skips, and an optional auxiliary head. Their weights were trained on a COCO subset with the 20 VOC categories; the documentation reports 60.5 mean IoU for fcn_resnet50 on COCO val2017 with VOC labels.