Fully Convolutional Networks for Semantic Segmentation

“Fully Convolutional Networks for Semantic Segmentation” by Jonathan Long, Evan Shelhamer and Trevor Darrell of UC Berkeley was posted to arXiv in November 2014 and presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2015; Long and Shelhamer are marked as equal contributors. It showed that an ImageNet classification network, rewritten so that every layer is convolutional and fine-tuned on whole images with a per-pixel loss, beat the best semantic segmentation systems of the time without proposals, superpixels or random fields. A journal version, with Shelhamer as first author, appeared in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2017. The paper received a Best Paper Honorable Mention at CVPR 2015 and the Longuet-Higgins Prize in 2025. The network itself, with its layers, training recipe and full results tables, is covered in the FCN article; this article covers the publication.

Problem

By 2014 convolutional networks had moved from whole-image classification to detection, keypoints and correspondence, and the authors present per-pixel labeling as the natural next step from coarse to fine inference. Earlier convolutional approaches to what the paper calls dense prediction, such as the scene labeling of Farabet et al. (2013), shared a set of habits the authors list: small models with limited capacity and receptive fields, training on sampled patches, post-processing by superpixel projection or random fields, shifting inputs and interlacing outputs to get dense predictions, multi-scale pyramids and ensembles, all without supervised pretraining.

The strongest results on PASCAL VOC came from a different direction. SDS (Hariharan et al., 2014) fine-tuned an R-CNN system (Girshick et al., 2014) on region proposals and classified them, and Gupta et al. did the same for RGB-D scenes. These hybrid proposal–classifier pipelines were not learned end to end, and SDS took about 50 seconds per image.

Contribution

The authors claim, to their knowledge, the first work to train fully convolutional networks end to end for pixelwise prediction and from supervised pretraining. The paper contributes:

• A framing. A network built only from local, translation-invariant layers, which the paper calls a deep filter or fully convolutional network, accepts any input size; with a loss summed over output positions, whole-image training equals training on all its receptive fields as one minibatch.
• Adapting classifiers. Fully connected layers are reinterpreted as convolutions whose kernels cover their whole input, so AlexNet, VGG-16 and GoogLeNet become networks that output coarse class-score maps and can be fine-tuned from their ImageNet weights.
• In-network upsampling. Upsampling by an integer factor is written as backwards strided convolution, which the paper notes is sometimes called deconvolution and today is called transposed convolution; it can be fixed to bilinear interpolation or learned.
• The skip architecture. Class scores from deep, coarse layers are fused with scores from shallower, finer layers. The paper describes the tension between “what” and “where” and calls the multi-layer feature hierarchy a deep jet, by analogy with the jet of Koenderink and van Doorn.

The related work places earlier systems inside this framework. Convolutional networks had been run over arbitrary-sized inputs since Matan et al. (1991), and OverFeat and others used fully convolutional inference, but fully convolutional training was rare. Shift-and-stitch becomes a network modification the paper calls filter rarefaction, linked to the à trous algorithm, and patchwise training becomes loss sampling.

Method

Model selection used PASCAL VOC 2011; the sections below follow the paper’s order.

Adapting three classifiers. Each network was “decapitated”, its fully connected layers converted, and a $1 \times 1$ scoring layer with 21 outputs and a bilinear upsampling layer appended. On VOC 2011 validation the convolutionalized VGG-16 reached 56.0 mean IU, against 39.8 for AlexNet and 42.5 for the authors’ own GoogLeNet reimplementation (Table 1), so VGG-16 became the base network.

Skips. FCN-32s upsamples stride-32 scores in one step and is “dissatisfyingly coarse” in the authors’ words. Adding a skip from pool4 (FCN-16s) raised validation mean IU from 59.4 to 62.4, and a further skip from pool3 (FCN-8s) to 62.7, after which the authors judged fusion to have reached diminishing returns. Controls showed that predicting from pool4 alone performed poorly and that lowering the learning rate without a skip gave an insignificant gain. These numbers use the extra training labels of Hariharan et al. (2011) and a 736-image validation subset that excludes images in that set.

Other refinements. In limited experiments shift-and-stitch had a worse cost-to-improvement ratio than skip fusion. Setting the pool5 stride to 1 would have required a $14 \times 14$ fc6 to keep VGG-16’s receptive field, which they found costly and hard to learn.

Patch sampling. To test the common belief that randomly sampled patches speed convergence, they dropped each output cell from the loss with some probability and enlarged the batch to compensate. Sampling did not change convergence per iteration noticeably and cost more wall-clock time, so the other experiments train on whole images. Class balancing and augmentation also proved unnecessary, and fine-tuning only the classifier gave about 70% of full fine-tuning.

Results

• PASCAL VOC. FCN-8s scored 62.7 mean IU on VOC 2011 test and 62.2 on VOC 2012 test, against 52.6 and 51.6 for SDS, a relative improvement of 20%. Inference took about 175 ms, which the paper puts at 114 times faster than SDS for the convolutional network alone and 286 times faster overall.
• NYUDv2. On 40-class RGB-D segmentation, adding depth as a fourth input channel helped little; summing the predictions of an RGB stream and a stream on Gupta et al.’s HHA depth encoding, upgraded to FCN-16s, reached 34.0 mean IU against Gupta et al.’s 28.6.
• SIFT Flow. One FCN-16s with two output heads predicted 33 semantic and 3 geometric classes as well as two separate models, at 39.5 mean IU and 94.3% geometric accuracy.

arXiv revisions. The arXiv record has two versions, of 14 November 2014 and 8 March 2015. The changelog of v2 lists corrections to the PASCAL validation numbers (some validation images had been in the training set), to the SIFT Flow mean IU (which had used an inappropriately strict metric) and to the frequency-weighted IU formula, and timings updated for an improved implementation. v2 also added two appendices: upper bounds on mean IU, computed by downsampling and re-upsampling the ground truth (86.1 at factor 32, 96.4 at factor 8), from which the authors conclude that mean IU is not a good measure of fine-scale accuracy; and results on PASCAL-Context, where FCN-8s scored 35.1 mean IU on the 59-class task. The CVPR text’s tables match v2; the appendices are arXiv-only.

Journal version. The TPAMI paper, posted to arXiv in May 2016 as its journal edition, retuned training: batch size one with momentum 0.99, which the authors call heavy learning and justify by an approximate trade-off between momentum and batch size, and an all-at-once FCN-8s whose skip streams are scaled by fixed constants instead of being trained in stages. It reports 67.2 mean IU on VOC 2012 test (a 30% relative improvement), about 100 ms inference, PASCAL-Context results, and masking experiments on the roles of context and shape.

Impact

The CVPR paper released its Caffe models and code at fcn.berkeleyvision.org. The released models differ from both papers in details: the PASCAL models follow the journal’s heavy-learning recipe, which the README credits with about 5 points of mean IU over the original models; all interpolation kernels are fixed to bilinear, unlike the papers’ learned intermediate layers; and the networks pad the input by 100 pixels, which neither paper mentions (see FCN).

The paper became the template for later segmentation networks. The first DeepLab paper (Chen et al., ICLR 2015) worked directly on the pixel representation “similarly to Long et al.” but kept stride 8 with atrous convolution and added a CRF; see DeepLab. The U-Net paper (Ronneberger et al., 2015) describes its network as building on the fully convolutional network and extending it to work with very few training images; SegNet (Badrinarayanan et al., 2017) is built around a comparison of its decoder with FCN’s; and PSPNet (Zhao et al., 2017) opens by noting that state-of-the-art scene parsing frameworks were mostly FCN-based. These lines of work are surveyed in semantic segmentation and encoder–decoder networks.

The PAMI Technical Committee lists the paper among the CVPR 2015 Best Paper Honorable Mentions and as a 2025 winner of the Longuet-Higgins Prize, which recognizes CVPR papers from ten years earlier with significant impact on computer vision research.

Limitations

The paper names some of its limits itself: fusion stops at stride 8 because gains diminished, mean IU rewards large-scale correctness more than detail, and one failure case shows lifejackets in a boat labeled as people. Its successors stated others.

• Coarse, upsampled output. The first DeepLab paper argued that FCN’s stride-32 scores forced learned upsampling layers, adding complexity and training time; they reported about 10 hours of fine-tuning against FCN’s several days.
• Stage-wise training and memory. SegNet’s authors described FCN’s large encoder (134M parameters) and small decoder (0.5M) as hard to train end to end, hence the staged schedule, and noted that its decoder must store encoder feature maps at inference.
• Missing global context. PSPNet’s authors identified the lack of a strategy for using global scene clues as the major issue of FCN-based models, illustrated by an FCN labeling a boat as a car on ADE20K.

What Came After

Atrous convolution and CRFs (DeepLab), feature-level decoders (U-Net, SegNet) and pyramid pooling (PSPNet) each addressed one of these limits while keeping the paper’s core recipe: a pretrained backbone run fully convolutionally and trained on whole images with a per-pixel loss. The same recipe reappears inside instance segmentation, where Mask R-CNN (He et al., 2017) predicts each region’s mask with a small FCN.