Very Deep Convolutional Networks for Large-Scale Image Recognition

“Very Deep Convolutional Networks for Large-Scale Image Recognition” is a paper by Karen Simonyan and Andrew Zisserman of the Visual Geometry Group at the University of Oxford. It was first posted to arXiv on 4 September 2014, revised five times, and published at the International Conference on Learning Representations (ICLR 2015). The paper asked one question, how depth affects the accuracy of a convolutional network on ImageNet, and answered it with a controlled series of plain networks of 11 to 19 weight layers, all built from $3 \times 3$ convolutions. The same networks carried the “VGG” team’s entry to the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2014, which won the localization task and placed second in classification, and the two models the authors released, now known as VGG-16 and VGG-19, became standard pretrained feature extractors. This article covers the publication; the architecture, its training and evaluation and its later uses are in the VGG article.

Problem

By 2014 convolutional networks were, in the authors’ words, becoming a commodity in computer vision, and the entries that followed AlexNet improved it in two ways. The best ILSVRC-2013 submissions, by Zeiler and Fergus and by Sermanet et al. (OverFeat), used a smaller receptive window and stride in the first convolutional layer ($7 \times 7$ at stride 2 instead of $11 \times 11$ at stride 4). Another line of work, OverFeat and Howard (2014), trained and tested networks densely over the whole image and at several scales. In the paper’s comparison table, the best earlier classification result without outside training data was 11.7% top-5 test error, by the ILSVRC-2013 winner Clarifai, against 14.8% for Zeiler and Fergus’s six networks and 13.6% for OverFeat’s seven.

Depth had not been varied on its own. Simonyan and Zisserman note that Cireşan et al. (2011) used small filters in much shallower networks not tested on ILSVRC, and that Goodfellow et al. (2014) found that depth helped street-number recognition with 11 weight layers. The paper fixed every other design choice and added convolutional layers, which the authors say was feasible because every filter is $3 \times 3$. GoogLeNet, entered in the same challenge, was developed independently, the authors write; it was also very deep (22 weight layers) with small filters, but had a more complex topology and reduced resolution more aggressively in its first layers.

Contribution

The abstract of the published version lists:

• A thorough evaluation of networks of increasing depth with very small ($3 \times 3$) filters, showing that pushing depth to 16–19 weight layers improves significantly on prior configurations. The authors call this their main contribution.
• The ILSVRC 2014 results built on these findings: first place in localization and second in classification.
• Generalization: the representations reach state-of-the-art results on other datasets.
• Public release of the two best-performing models.

The conclusion adds the reading that state-of-the-art ImageNet accuracy needed no departure from the conventional ConvNet of LeCun et al. (1989) and Krizhevsky et al. (2012), only substantially more depth.

Method

Six configurations, A to E, share one template: $3 \times 3$ convolutions with stride 1 and padding 1, five $2 \times 2$ max-pooling layers, widths doubling from 64 to 512 channels, and three fully connected layers; they differ only in how many convolutions each stage holds, and have 133 to 144 million parameters (Table 2). The design argument is that a stack of three $3 \times 3$ layers has the receptive field of one $7 \times 7$ layer with three nonlinearities instead of one and $27C^2$ instead of $49C^2$ weights for $C$ channels in and out. Training follows Krizhevsky et al. (2012), with the deeper networks partly initialized from the trained 11-layer network A and with training images rescaled to a random scale (scale jittering). At test time the fully connected layers are converted to convolutions and the network runs densely over the whole image at one or several scales, with multi-crop evaluation as a comparison. Training one network took two to three weeks on four NVIDIA Titan Black GPUs in a modified Caffe. The VGG article covers all of this in detail.

Results

All classification numbers are top-5 error on ILSVRC-2012 data (see accuracy), which the challenge used from 2012 to 2014; most experiments used the validation set as the test set, and some runs went to the official test server.

The ILSVRC 2014 Submission

The official ILSVRC 2014 results page lists the VGG team (Simonyan and Zisserman, University of Oxford) in Task 2a, classification and localization with provided training data. Ordered by classification error, the first row is GoogLeNet’s, marked as the winner, at 0.06656; the next four rows are VGG ensembles, the best at 0.07325 (an ensemble including a net trained on images of different sizes, with fusion weights learned on the validation set), followed by MSRA Visual Computing at 0.0806. Ordered by localization error, VGG’s entry “a combination of multiple ConvNets (by averaging)” is marked as the winner, with localization error 0.253231 and classification error 0.07405; the best entry from another team, GoogLeNet’s, is at 0.264414. A single VGG network, described as 13 convolutional and 3 fully connected layers (configuration D), scored 0.08434 in classification and 0.267184 in localization. The page states places only by order and highlighting; the words “first” and “second” come from the paper and the project page. The team’s description on the page names three architectures, which correspond to configurations C, D and E.

The paper describes the classification submission as an average of seven networks with 7.3% test error (Table 6). By the deadline only single-scale models had been trained, plus one scale-jittered D obtained by fine-tuning just its fully connected layers.

After the Submission

Two fully trained scale-jittered networks, D and E, beat the seven-network submission. As published, their combination reached 7.0% test error with dense evaluation and 6.8% with dense and multi-crop evaluation together (Table 6), and one network reached 7.0% (Table 7; its validation numbers match configuration E in Table 5). The authors compare this with GoogLeNet’s 6.7% for seven networks and 7.9% for one, and claim the best single-network result. The per-configuration results are tabulated in the VGG article; the authors judged error to saturate at 19 layers. Only the published version contains a direct test of small filters: replacing each pair of $3 \times 3$ layers in configuration B by one $5 \times 5$ layer gave a top-1 error on a centre crop 7% higher, as the paper puts it.

Localization (Appendix A)

The localization network is configuration D with a last layer that regresses a box (centre, width and height) instead of class scores, trained with a Euclidean loss and following OverFeat’s approach with modifications. Per-class regression, with one box per class (a 4,000-dimensional output), beat a single class-agnostic box (34.3% against 36.4% error with the ground-truth class and a centre crop), reversing OverFeat’s finding, and fine-tuning all layers beat fine-tuning only the first two fully connected layers (33.1% against 34.3%) (Table 8, at $S = 384$). For the challenge, two single-scale models ($S = 256$ and $S = 384$) were fused, since time did not allow scale jittering; boxes predicted densely over the image were merged greedily as in OverFeat, without its multiple pooling offsets. The error is top-5, with one box for each of five predicted classes, and a box counts as correct when its intersection over union with the ground-truth box exceeds 0.5. The fused system had 26.9% validation and 25.3% test error, against 26.7% for GoogLeNet and 29.9% for OverFeat on the test set (Tables 9 and 10); the results page gives GoogLeNet’s best localization entry as 0.264414, slightly lower than the paper’s 26.7%.

Generalization (Appendix B)

The appendix tests the released networks D and E as fixed feature extractors. The last fully connected layer is removed, the network is applied densely to the rescaled image, and its 4,096-dimensional penultimate activations are averaged over positions and over the image and its mirror image. The features are computed at several scales $Q$ and either averaged or stacked across scales, then L2-normalized and fed to a linear SVM; there is no fine-tuning. On PASCAL VOC, averaging over $Q \in \{256, 384, 512, 640, 768\}$ worked as well as stacking, while on Caltech, where objects typically fill the image, stacking three scales worked better. Table 11, test results as printed (mean average precision for VOC, mean class recall over three random splits for Caltech):

Method	VOC-2007	VOC-2012	Caltech-101	Caltech-256
Zeiler & Fergus	–	79.0	86.5 ± 0.5	74.2 ± 0.3
Chatfield et al.	82.4	83.2	88.4 ± 0.6	77.6 ± 0.1
He et al. (SPP)	82.4	–	93.4 ± 0.5	–
Wei et al.	81.5 (85.2*)	81.7 (90.3*)	–	–
VGG Net-D (16 layers)	89.3	89.0	91.8 ± 1.0	85.0 ± 0.2
VGG Net-E (19 layers)	89.3	89.0	92.3 ± 0.5	85.1 ± 0.3
VGG Net-D & Net-E	89.7	89.3	92.7 ± 0.5	86.2 ± 0.3

Starred entries used networks pretrained on an extended 2,000-class ILSVRC set. The table does not support a clean sweep: on Caltech-101 the combination is below He et al.’s 93.4%, which the authors call competitive, and on VOC-2012 Wei et al.’s starred result is 1 point higher, which the authors attribute to the extra classes and a detection-assisted pipeline. On Caltech-256 the margin over Chatfield et al. is 8.6 points. The published version adds single-image action classification on VOC-2012 (Table 12): 79.2% mAP from whole-image features and 84.0% when features of the person’s bounding box are stacked with them, against 76.3% for the best method listed.

Versions

The paper went through six arXiv versions, and Appendix C of the final one lists the major changes. The dates are from the arXiv submission history; the observations marked as ours come from comparing the versions’ texts.

Version	Date	Changes
v1	4 Sep 2014	Experiments run before the ILSVRC submission
v2	15 Sep 2014	Post-submission experiments with scale jittering in training
v3	18 Nov 2014	Generalization experiments on PASCAL VOC and Caltech; models publicly available
v4	19 Dec 2014	ICLR 2015 submission format; multi-crop evaluation
v5	23 Dec 2014	Not listed in Appendix C; by our comparison, a reworded account of training scales and of why dense and multi-crop evaluation are complementary
v6	10 Apr 2015	Camera-ready: configuration B against a shallow $5 \times 5$ network; VOC action classification

The headline numbers moved with the versions. In v1 the best single network had 8.0% validation error and the seven-network submission 7.3% test error. v2 brought fully trained scale-jittered D and E to 7.5% validation error each (7.3% test error for E) and their pair to 7.0% test error. v4 added multi-crop evaluation, which lowered the pair to 6.8% and one network to 7.0%. The localization and generalization experiments were Sections 5 and 6 of the main text in v3 and moved to Appendices A and B in v4; v4’s date is the ICLR 2015 submission deadline.

One change is not in Appendix C (our comparison). Versions 1 to 5 suggest that a better random initialization, such as that of Saxe et al. (2014), might make pre-training unnecessary and leave this to future work; v6 replaces the remark with the finding, made after submission, that the procedure of Glorot and Bengio (2010) allows the networks to be initialized without pre-training. The stated reason for staged initialization also changes: many rectification nonlinearities (v1–v3), the non-convexity of the objective (v4–v5), and the instability of gradients in deep networks (v6).

The camera-ready version is the one published at ICLR 2015, held in San Diego on 7–9 May 2015. The conference schedule lists the paper among the oral presentations, on 7 May, and in the poster session of 9 May. The conference’s pages link to the arXiv record, which serves as the paper’s official version.

Impact

Release of the Models

The authors released configurations D and E trained with scale jittering, in Caffe format through the Caffe Model Zoo and in MatConvNet format. Simonyan’s two Model Zoo entries, GitHub gists, were created on 25 September 2014, three weeks after v1; they describe the models as improved versions of those used in the challenge and ask users to subtract a mean pixel value rather than a mean image. As of October 2026 the project page releases the models under a Creative Commons Attribution licence, lists the weight files at 528 MB and 548 MB, and warns that the toolboxes may lack the paper’s dense multi-scale evaluation, so results can differ from the paper’s.

Reception

The released weights were picked up within weeks. The fifth version of the R-CNN technical report (22 October 2014) downloaded the 16-layer model from the Caffe Model Zoo, calling it “O-Net” for OxfordNet, and raised object detection mAP on VOC 2007 test from 58.5% to 66.0%, with a forward pass about seven times slower than with Krizhevsky et al.’s network. From v4 (19 December 2014), Appendix B listed uses since the release in detection, semantic segmentation, image captioning, and texture and material recognition, all with gains over AlexNet’s architecture. The 16-layer network then became the backbone of FCN, Faster R-CNN and SSD, and VGG features the basis of neural style transfer and perceptual losses; the VGG article covers these uses. The weights also made VGG a standard starting point for transfer learning.

The design rules outlasted the networks. He et al. (2016) built the plain baselines of the ResNet paper on what they call the philosophy of VGG nets: mostly $3 \times 3$ filters, equal filter counts at equal map sizes, and doubled filters when the map is halved.

Limitations

• Cost. The paper reports parameters but no operation count. Its only comparison is that the networks have no more weights than OverFeat’s shallower, wider network (144 million). He et al. (2016) later gave VGG-16 15.3 billion FLOPs, which they define as multiply-adds, about ten times the 1.5-billion budget GoogLeNet was designed for (our arithmetic), and most of its parameters sit in the fully connected head; the VGG article gives the numbers.
• Training depended on workarounds. The deeper networks were trained from partly copied weights, an approach whose stated rationale differs between versions, and the paper reports no comparison with training from random initialization (our observation). He et al. (2015) later used configuration B to show why a fixed initial scale stalls deep networks (see the PReLU paper and weight initialization).
• Depth stopped helping at 19 layers. The authors read the saturation as possibly specific to the dataset size, suggesting that deeper models might pay off on larger datasets. The text says error decreases with depth from A to E, but in Table 3, at a fixed training scale of 256, E (9.0%) is slightly worse than D (8.8%).
• Protocols differ between numbers. Results mix single-scale, multi-scale, dense and multi-crop evaluation, single networks and ensembles, validation and test sets; the 6.8% headline uses two networks with 150 crops each plus dense evaluation at three scales.
• The localization system was limited by time. Time constraints ruled out scale jittering for the localization networks, and OverFeat’s multiple pooling offsets were left out.

What Came After

GoogLeNet’s paper (Szegedy et al., 2015), published at CVPR 2015, described the network that won the 2014 classification task within a design budget of 1.5 billion multiply-adds; see the GoogLeNet paper and Inception. Later networks did without staged initialization, using initialization rules such as He et al.‘s (2015) and batch normalization, both from 2015, and depth beyond 19 layers came with residual connections in the ResNet paper, whose 152-layer network, at 11.3 billion FLOPs, cost less than VGG-16; an ensemble of residual networks won ILSVRC 2015 classification. The networks’ later careers as backbones and perceptual feature extractors are followed in the VGG article.