Deep Learning for Vision

Deep Residual Learning for Image Recognition

The 2016 CVPR paper by He, Zhang, Ren and Sun of Microsoft Research that introduced residual networks (ResNets), trained them to 152 layers on ImageNet, and won the ILSVRC 2015 classification task.

intermediate

“Deep Residual Learning for Image Recognition” by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun of Microsoft Research appeared as a technical report on arXiv on 10 December 2015 and was presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Las Vegas in June 2016. It reformulated groups of layers in a convolutional neural network to learn a correction to their input, added back through an identity shortcut, and with these residual networks (ResNets) trained models of 152 layers on ImageNet and over 1,000 layers on CIFAR-10. A ResNet ensemble won the ILSVRC 2015 classification task, and the paper received the CVPR 2016 Best Paper award. The idea as understood today, including the shortcut variants and the analysis of gradient flow, is covered in the residual connection article.

The CVPR version (9 pages) refers to an appendix for its detection implementation and competition entries; that appendix appears only in the arXiv report.

Problem

By 2015, depth looked like the main lever for accuracy. He et al. point to VGG (Simonyan & Zisserman, 2015) and GoogLeNet (Szegedy et al., 2015), and note that the leading ImageNet results all used “very deep” models of sixteen to thirty layers. They asked whether better networks could be had simply by stacking more layers.

The old obstacle, vanishing and exploding gradients, had in their view been largely addressed by normalized initialization, including their own (He et al., 2015), and by intermediate normalization layers, that is batch normalization (Ioffe & Szegedy, 2015). Networks with tens of layers now started to converge under stochastic gradient descent. What remained was the degradation problem: as depth grew, accuracy saturated and then fell, and the deeper network had higher training error, so overfitting could not be the cause. The authors argued from a construction: copy a trained shallow network and append identity layers, and the deeper model has the same training error. A deeper model should therefore do no worse, yet the solvers did not find such solutions in feasible time.

Contribution

  • Residual learning. Instead of asking a few stacked layers to fit a desired mapping H(x)\mathcal{H}(\mathbf{x}), the paper lets them fit F(x)=H(x)−x\mathcal{F}(\mathbf{x}) = \mathcal{H}(\mathbf{x}) - \mathbf{x} and recovers H\mathcal{H} as F(x)+x\mathcal{F}(\mathbf{x}) + \mathbf{x}. The authors hypothesized that if the identity is near optimal, pushing a residual toward zero is easier than fitting the identity with nonlinear layers. They related this to residual encodings such as VLAD and Fisher vectors, and to multigrid and hierarchical-basis solvers, which converge faster because they work with residuals between scales.
  • Parameter-free identity shortcuts. Because the shortcut adds neither parameters nor computation (apart from an elementwise sum), plain and residual networks could be compared at equal depth, width, parameter count and cost. The authors contrasted this with concurrent highway networks, whose gated shortcuts have parameters and can close.
  • Deep, cheap architectures for ImageNet, including a bottleneck design that reached 152 layers at lower cost than VGG.

Method

Plain baseline. The plain networks follow VGG’s design rules: mostly 3×33 \times 3 convolutions, the same number of filters for the same feature-map size, and twice the filters whenever stride-2 convolutions halve the map. A 7×77 \times 7 stride-2 convolution and a max pool start the network; global average pooling and a 1000-way fully connected layer end it. The 34-layer plain network costs 3.6 billion FLOPs (multiply-adds), 18% of VGG-19’s 19.6 billion. The residual version adds a shortcut around every pair of 3×33 \times 3 layers.

Bottleneck networks. For depths of 50, 101 and 152 the authors replaced each two-layer block by a 1×11 \times 1, 3×33 \times 3, 1×11 \times 1 stack that reduces and then restores the channel count, citing training time they could afford. The four stages run at 56×5656 \times 56 to 7×77 \times 7 resolution with 3, 8, 36 and 3 blocks in ResNet-152. The paper lists 1.8, 3.6, 3.8, 7.6 and 11.3 billion FLOPs for the 18-, 34-, 50-, 101- and 152-layer networks, so the deepest still cost less than VGG-16 or VGG-19 (15.3 and 19.6 billion).

Training. Batch normalization follows each convolution, before the activation, and weights are initialized as in He et al. (2015). Training uses SGD with mini-batches of 256, momentum 0.9 and weight decay 0.0001, a learning rate of 0.1 divided by 10 when the error plateaus, and up to 60×10460 \times 10^4 iterations, without dropout. Augmentation follows AlexNet and VGG: the shorter side is resized to a random length in [256,480][256, 480], then a random 224×224224 \times 224 crop or its mirror is taken. Comparisons use 10-crop testing; the best results average fully convolutional scores over five scales.

CIFAR-10. To study depth itself, the authors used deliberately simple networks of 6n+26n + 2 layers with 16, 32 and 64 filters on 32×3232 \times 32, 16×1616 \times 16 and 8×88 \times 8 maps, and zero-padding identity shortcuts throughout, so residual and plain versions had exactly the same parameters. For 110 layers, the initial learning rate of 0.1 was slightly too large to start converging, so they warmed up at 0.01 for about 400 iterations.

Results

ImageNet. The plain 34-layer network had higher top-1 validation error than the plain 18-layer one (28.54% against 27.94%, 10-crop), and higher training error throughout training. With shortcuts the order reversed: 25.03% for 34 layers against 27.88% for 18. At 18 layers the two kinds of network were about equally accurate, but the ResNet converged faster. The authors checked that the plain networks’ gradients had healthy norms and that tripling the iterations did not remove the gap. Deeper ResNets kept improving, with 10-crop top-1/top-5 errors of 22.85/6.71% for ResNet-50, 21.75/6.05% for ResNet-101 and 21.43/5.71% for ResNet-152, against 28.07/9.33% for VGG-16 in their test. A single ResNet-152 reached 4.49% top-5 validation error, which the authors note beat every earlier ensemble result. An ensemble of six models of different depths, only two of them 152-layer at submission time, scored 3.57% top-5 error on the test server and won ILSVRC 2015 classification.

CIFAR-10. Test error fell from 8.75% at 20 layers to 6.97% at 56 and 6.43% at 110 layers (best of five runs; mean 6.61 ± 0.16%), with 1.7 million parameters, fewer than the 19-layer FitNet (2.5 million) and highway network (2.3 million) it was compared with. Plain-110 had more than 60% error. A 1202-layer ResNet (19.4 million parameters) trained to below 0.1% training error but tested at 7.93%, which the authors attributed to overfitting on so small a dataset. Measuring the standard deviation of each 3×33 \times 3 layer’s output (Fig. 7), they found smaller responses in ResNets than in plain networks, and smaller still in deeper ResNets, which they read as support for residual functions being close to zero.

Detection and localization. Replacing VGG-16 with ResNet-101 in Faster R-CNN (Ren et al., 2015), with the detection pipeline otherwise unchanged, raised mAP on PASCAL VOC 2007 test from 73.2% to 76.4% and on VOC 2012 test from 70.4% to 73.8%. On the COCO validation set, mAP@[.5, .95] rose from 21.2% to 27.2%, an increase of 6.0 points that the authors describe as a 28% relative improvement. The arXiv appendix explains how ResNet, which has no hidden fully connected layers, fits Faster R-CNN: layers up to conv4_x (91 convolutional layers in ResNet-101) compute shared feature maps at stride 16, and conv5_x acts as the per-region head after RoI pooling, with batch-normalization statistics frozen during fine-tuning. In the same appendix, with box refinement, global context and multi-scale testing, a three-model ensemble reached 37.4% mAP@[.5, .95] on COCO test-dev. The paper reports first places in ImageNet detection, ImageNet localization, COCO detection and COCO segmentation at the ILSVRC and COCO 2015 competitions; per the appendix, the ImageNet detection ensemble scored 62.1% mAP, 8.5 points ahead of second place, and the localization ensemble reached 9.0% top-5 localization error on the test set.

Impact

The PAMI Technical Committee lists the paper as the CVPR 2016 Best Paper. ResNets were widely adopted as backbones for other vision systems. DETR (Carion et al., 2020), for example, takes ImageNet-pretrained ResNet-50 and ResNet-101 feature maps as the input to its transformer, and the Vision Transformer paper (Dosovitskiy et al., 2021) still described ResNet-like architectures as the state of the art in large-scale image recognition and compared its models against large ResNets.

The residual design also spread beyond convolutional networks. The transformer of Vaswani et al. (2017) places a residual connection, citing He et al., around every attention and feed-forward sub-layer, and the Vision Transformer keeps it.

Limitations

  • The cause was left open. The authors conjectured that deep plain networks have exponentially low convergence rates and deferred the reason for their difficulty to future work. That a residual is easier to learn remained a hypothesis supported by experiments, not a proof.
  • Overfitting at extreme depth. The 1202-layer network did worse than the 110-layer one. The paper used no maxout or dropout and suggested that stronger regularization might help.
  • Practical choices. The bottleneck design was adopted for training cost; the authors note that deeper non-bottleneck ResNets also gain accuracy but are less economical.
  • Block design. The ReLU after each addition means the shortcut is not a pure identity; the authors revised this in their follow-up paper (He et al., 2016b), described in the residual connection article.
  • Competition systems. The headline detection and localization results combine the backbone with box refinement, context, multi-scale testing and ensembles; only the baseline comparison isolates the network’s contribution.

What Came After

The authors’ own follow-up, “Identity Mappings in Deep Residual Networks” (He et al., 2016b), moved batch normalization and ReLU before each convolution and trained a 1001-layer network on CIFAR-10. Zagoruyko and Komodakis (2016) argued that very deep, thin ResNets suffer from diminishing feature reuse and trained wide residual networks with fewer layers; they reported that a 16-layer wide ResNet outperformed all previous deep residual networks, including thousand-layer ones. ResNeXt (Xie et al., 2017), with He as an author, split the bottleneck into many parallel paths of the same shape, equivalent to grouped convolutions, and reported that increasing this “cardinality” improved ImageNet accuracy more than going deeper or wider. DenseNet (Huang et al., 2017) connects each layer to all later layers and combines feature maps by concatenation instead of summation.

Related

  • Residual Connection

    A shortcut that adds a block's input to its output, y = x + F(x), so that the layers learn only a correction to the identity; it lets very deep convolutional networks and transformers train, because signals and gradients always have a path that bypasses the weights.

  • Convolutional Neural Network

    A neural network built from convolutional layers, small learned filters applied at every image position, interleaved with nonlinearities and downsampling, so that features grow from edges to object parts while the parameter count stays independent of image size.

  • Batch Normalization

    A layer that normalizes each channel of a feature map with the mean and variance of the current mini-batch, then rescales it with learned parameters; it allows higher learning rates in convolutional networks but depends on the batch size and behaves differently in training and inference.

  • Pooling

    A fixed, parameter-free layer that summarizes each small window of a feature map by its maximum or average, shrinking the map, widening what later layers see, and tolerating small shifts at the cost of resolution and aliasing.

  • Transformer

    A neural network architecture that stacks identical blocks of multi-head self-attention and a position-wise MLP, each wrapped in a residual connection with layer normalization; introduced for machine translation and now used for image classification, detection and segmentation on sequences of image patches.

References

  1. He, K., Zhang, X., Ren, S. & Sun, J. (2016). Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.
  2. He, K., Zhang, X., Ren, S. & Sun, J. (2015). Deep Residual Learning for Image Recognition. Technical report, arXiv:1512.03385.
  3. Simonyan, K. & Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference on Learning Representations (ICLR). arXiv:1409.1556.
  4. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. & Rabinovich, A. (2015). Going Deeper with Convolutions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–9.
  5. Ioffe, S. & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on Machine Learning (ICML), PMLR 37, 448–456.
  6. He, K., Zhang, X., Ren, S. & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. IEEE International Conference on Computer Vision (ICCV), 1026–1034.
  7. Ren, S., He, K., Girshick, R. & Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28 (NIPS).
  8. He, K., Zhang, X., Ren, S. & Sun, J. (2016). Identity Mappings in Deep Residual Networks. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 630–645.
  9. Zagoruyko, S. & Komodakis, N. (2016). Wide Residual Networks. Proceedings of the British Machine Vision Conference (BMVC), 87.1–87.12.
  10. Xie, S., Girshick, R., Dollár, P., Tu, Z. & He, K. (2017). Aggregated Residual Transformations for Deep Neural Networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5987–5995.
  11. Huang, G., Liu, Z., van der Maaten, L. & Weinberger, K. Q. (2017). Densely Connected Convolutional Networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2261–2269.
  12. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS).
  13. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A. & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 213–229.
  14. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J. & Houlsby, N. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations (ICLR). arXiv:2010.11929.