Going Deeper with Convolutions

“Going Deeper with Convolutions” is a paper by Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke and Andrew Rabinovich, most of them at Google; Liu was at the University of North Carolina at Chapel Hill, Reed at the University of Michigan, and Rabinovich is listed at Google in the arXiv version and at Magic Leap in the CVPR version. It was posted to arXiv on 17 September 2014 and presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Boston in June 2015. The paper introduced the Inception architecture, a convolutional network built from modules that run several filter sizes side by side behind cheap $1 \times 1$ reductions, and GoogLeNet, a 22-layer instance of it designed to stay within 1.5 billion multiply-adds at inference. GoogLeNet won the ILSVRC 2014 classification task with a top-5 error of 6.67% and the detection track that allowed outside training data. The PAMI Technical Committee awarded the paper the Longuet-Higgins Prize in 2025. The family that grew from it, from GoogLeNet to Inception-ResNet, is covered in the Inception article; this article covers the publication.

Two versions differ in a few details: the 12-page arXiv preprint (v1, the only arXiv version) and the 9-page CVPR paper. Where they differ on a point discussed here, the version is named.

Problem

By 2014, deep networks had transformed ImageNet classification, and the authors describe the trend among larger models as more layers and larger layers, with dropout against overfitting. They named two costs of simply making a network bigger. More parameters make it more prone to overfitting, which matters because labelled data for fine-grained categories is expensive to collect. And computation grows fast: if two chained convolutional layers are widened uniformly, their cost rises quadratically, and much of it is wasted if many of the added weights end up near zero. They also cited the growth of mobile and embedded computing, where power and memory use matter. For most experiments their models were therefore designed for a budget of 1.5 billion multiply-adds at inference, so that they could be put to real-world use at reasonable cost rather than remain an academic exercise.

The authors’ theoretical starting point was sparsity. Arora et al. had shown that if a dataset’s distribution can be represented by a large, very sparse deep network, the optimal topology can be built layer by layer by clustering units whose outputs are highly correlated. The proof requires strong conditions, but Szegedy et al. argued that its resemblance to the Hebbian principle, that neurons that fire together wire together, suggested the idea would hold more loosely in practice. Sparse computation, however, was inefficient on the hardware and numerical libraries of the time, which were tuned for dense matrix multiplication; they wrote that even with 100 times fewer arithmetic operations, the overhead of lookups and cache misses would dominate. They asked whether an architecture could follow the sparse structure the theory suggested while computing only with dense blocks.

Two earlier designs shaped the answer. Network in Network (Lin et al., 2014) had added $1 \times 1$ convolutions to increase depth and supplied the global-average-pooling head that GoogLeNet adopted; Serre et al. (2007), modelling the primate visual cortex, had used fixed Gabor filters of several sizes to handle multiple scales. Inception would learn all its filters and repeat its modules many times.

Contribution

The abstract and the paper’s opening sections claim the following:

• An architecture that uses computation better. The Inception architecture increases both depth and width while keeping the computational budget constant. The authors present it as a dense approximation of the sparse structure implied by Arora et al., guided by the Hebbian principle and the intuition of multi-scale processing.
• $1 \times 1$ convolutions as dimension reduction. Where Network in Network used $1 \times 1$ convolutions mainly to add depth, here they mainly remove computational bottlenecks before expensive convolutions, which lets the network grow wider as well as deeper.
• GoogLeNet, the 22-layer instance used in the ILSVRC 2014 submission, which the introduction says uses 12 times fewer parameters than Krizhevsky et al.’s winning network of 2012 while being significantly more accurate.
• State-of-the-art results in ILSVRC 2014 classification and detection.

The authors framed the work as a case study: the architecture began as an attempt to approximate the output of a hypothetical topology-construction algorithm, and they expected such algorithms to automate network design in future.

Method

The Inception module. Each module applies $1 \times 1$, $3 \times 3$ and $5 \times 5$ convolutions and a $3 \times 3$ max pool to the same input and concatenates the outputs along the channel axis. Restricting the kernels to these three sizes was, the authors write, a matter of convenience rather than necessity. The naive form is too expensive, because $5 \times 5$ convolutions on wide inputs cost a great deal and the pooling path keeps every input channel, so the width grows from module to module. The paper’s second idea, reducing dimension wherever computation would otherwise grow too much, puts $1 \times 1$ convolutions with ReLUs before the $3 \times 3$ and $5 \times 5$ convolutions and after the pooling. The module, its cost formula and a worked example are in the Inception article.

GoogLeNet. A conventional stem feeds nine Inception modules on $28 \times 28$, $14 \times 14$ and $7 \times 7$ grids, followed by global average pooling, 40% dropout and a linear classifier; the network is 22 layers deep counting only layers with parameters. The paper reports that moving from fully connected layers to average pooling improved top-1 accuracy by about 0.6%, though dropout remained essential. Two auxiliary classifiers, meant to counter vanishing gradients and to regularize, add 0.3 times their losses during training; later control experiments found their effect minor, around 0.5%. Table 1 gives every layer’s widths, parameters and operations; the Inception article walks through the layout and the auxiliary classifiers.

Training. The networks were trained with Google’s DistBelief system (Dean et al., 2012) using modest model and data parallelism, on CPUs only; the authors estimate that a few high-end GPUs could train GoogLeNet to convergence within a week, limited mainly by memory. Training used asynchronous stochastic gradient descent with momentum 0.9 and a fixed schedule that decreased the learning rate by 4% every 8 epochs, and Polyak averaging (Polyak & Juditsky, 1992) produced the final model. The image sampling changed substantially in the months before the competition, sometimes applied to already converged models together with changes to dropout and the learning rate, so the authors decline to give a single recipe. One prescription verified after the competition samples patches covering 8% to 100% of the image area with aspect ratios between 3/4 and 4/3, plus the photometric distortions of Howard (2013). The arXiv version adds that random interpolation methods for resizing (bilinear, area, nearest neighbour and cubic, with equal probability) were introduced late, together with other changes, so their effect could not be isolated. The crop scheme survives as the default of TorchVision’s RandomResizedCrop (version 0.29.0); the Inception and data augmentation articles cover it.

The name. The paper says the codename Inception comes from the Network in Network paper together with the “we need to go deeper” internet meme, which it cites as a reference, and that “deep” has two meanings: a new level of organization, the module, and increased depth. The arXiv version adds that GoogLeNet was the team name for ILSVRC 2014, an homage to Yann LeCun’s LeNet-5 (LeNet); the CVPR version uses the name only for the network.

Table 1 and Its Recount

The paper prints no total parameter count. Its introduction’s “12 times fewer” than AlexNet’s 60 million parameters would put GoogLeNet near 5 million; the Inception-v3 paper (Szegedy et al., 2016) instead gives around 7 million, a 9-fold reduction. Table 1’s “params” column sums to 6.80 million, and recounting the weights from the table’s own layer widths gives 6.99 million. The recount also exposes a puzzle in the first row. Table 1 lists 2.7K parameters and 34 million operations for the $7 \times 7$ stride-2 convolution from 3 to 64 channels, but a dense layer of that shape has 9,408 weights and costs 118 million multiply-adds on a $112 \times 112$ output grid. A depthwise separable $7 \times 7$ layer with depth multiplier 8 (each input channel filtered by eight $7 \times 7$ kernels, giving 24 channels, then a $1 \times 1$ convolution to 64; see depthwise separable convolution) has exactly 2,712 weights and 34.0 million multiply-adds.

That inference is ours; the GoogLeNet paper’s text does not mention a separable layer. It fits two later statements. Chollet (2017) writes that a depthwise separable convolution was used as the first layer of Inception V1 and Inception V2, citing this paper and the batch-normalization paper. The supplementary material of that batch-normalization paper (Ioffe & Szegedy, 2015) states that its Inception variant used separable convolution with depth multiplier 8 on the first layer, in a sentence separate from its list of changes relative to GoogLeNet. With the separable first layer the recount totals 1.50 billion multiply-adds, the paper’s stated budget, against 1.58 billion with a dense one.

The code below applies both versions of the first layer, with random weights, to a synthetic $3 \times 224 \times 224$ image of a disk, a square outline and stripes, confirming that both produce a $64 \times 112 \times 112$ output, and then recounts every row of Table 1 from its widths (weights only, no biases). Run with NumPy.

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

rng = np.random.default_rng(0)

# Synthetic 3x224x224 image (C, H, W): a disk (red), a square outline (green),
# vertical stripes of period 8 (blue).
yy, xx = np.mgrid[0:224, 0:224]
img = np.zeros((3, 224, 224))
img[0] = (yy - 70) ** 2 + (xx - 70) ** 2 < 40 ** 2
img[1] = (np.abs(yy - 160) <= 40) & (np.abs(xx - 160) <= 40) & ~((np.abs(yy - 160) <= 36) & (np.abs(xx - 160) <= 36))
img[2] = xx % 8 < 4

def windows(x, k=7, s=2):
    """7x7 patches with stride 2 and padding 3: shape (C, H_out, W_out, k, k)."""
    x = np.pad(x, ((0, 0), (k // 2, k // 2), (k // 2, k // 2)))
    return sliding_window_view(x, (k, k), axis=(1, 2))[:, ::s, ::s]

P = windows(img)
# Dense first layer: 64 filters of size 3x7x7.
w_dense = rng.standard_normal((64, 3, 7, 7))
y_dense = np.einsum("chwij,ocij->ohw", P, w_dense)
# Depthwise separable first layer, depth multiplier 8: 3x8 depthwise 7x7 filters, then 1x1 from 24 to 64.
w_dw = rng.standard_normal((3, 8, 7, 7))
w_pw = rng.standard_normal((64, 24))
mid = np.einsum("chwij,cmij->cmhw", P, w_dw).reshape(24, 112, 112)
y_sep = np.einsum("om,mhw->ohw", w_pw, mid)
for name, y, w in [("dense 7x7", y_dense, w_dense.size), ("separable 7x7 x8", y_sep, w_dw.size + w_pw.size)]:
    macs = y_dense.shape[1] * y_dense.shape[2] * w        # every weight used once per output position
    print(f"{name:17s}: output {y.shape}, {w:,} weights, {macs / 1e6:.1f}M mult-adds")

# Recount of Table 1 (weights only, no biases). Each module: input channels, grid,
# #1x1, #3x3 reduce, #3x3, #5x5 reduce, #5x5, pool proj; then the printed params (K) and ops (M).
mods = {"3a": (192, 28, 64, 96, 128, 16, 32, 32, 159, 128), "3b": (256, 28, 128, 128, 192, 32, 96, 64, 380, 304),
        "4a": (480, 14, 192, 96, 208, 16, 48, 64, 364, 73), "4b": (512, 14, 160, 112, 224, 24, 64, 64, 437, 88),
        "4c": (512, 14, 128, 128, 256, 24, 64, 64, 463, 100), "4d": (512, 14, 112, 144, 288, 32, 64, 64, 580, 119),
        "4e": (528, 14, 256, 160, 320, 32, 128, 128, 840, 170), "5a": (832, 7, 256, 160, 320, 32, 128, 128, 1072, 54),
        "5b": (832, 7, 384, 192, 384, 48, 128, 128, 1388, 71)}
rows = [("conv 7x7/2, dense", 7 * 7 * 3 * 64, 112, 2.7, 34),
        ("conv 7x7/2, separable x8", 7 * 7 * 3 * 8 + 24 * 64, 112, 2.7, 34),
        ("conv 1x1 + 3x3", 64 * 64 + 9 * 64 * 192, 56, 112, 360)]
for k, (ci, s, a, r3, b, r5, d, p, kp, mo) in mods.items():
    rows.append((f"inception ({k})", ci * (a + r3 + r5 + p) + 9 * r3 * b + 25 * r5 * d, s, kp, mo))
rows.append(("linear 1024 -> 1000", 1024 * 1000, 1, 1000, 1))
print(f"{'layer':25s} {'weights':>10s} {'/1024':>7s} {'printed':>8s} {'mult-adds':>10s} {'printed':>8s}")
for name, w, s, kp, mo in rows:
    print(f"{name:25s} {w:>10,} {w / 1024:7.1f} {kp:>7}K {w * s * s / 1e6:9.1f}M {mo:>7}M")
for first in ("dense", "separable"):
    use = [r for r in rows if not r[0].startswith("conv 7x7") or first in r[0]]
    print(f"recount, {first} first layer: {sum(r[1] for r in use) / 1e6:.2f}M weights, "
          f"{sum(r[1] * r[2] ** 2 for r in use) / 1e9:.3f}B mult-adds")
printed = [r for r in rows if "dense" not in r[0]]
print(f"printed columns summed: {sum(r[3] for r in printed) / 1e3:.2f}M params, {sum(r[4] for r in printed) / 1e3:.3f}B ops")

Output:

dense 7x7        : output (64, 112, 112), 9,408 weights, 118.0M mult-adds
separable 7x7 x8 : output (64, 112, 112), 2,712 weights, 34.0M mult-adds
layer                        weights   /1024  printed  mult-adds  printed
conv 7x7/2, dense              9,408     9.2     2.7K     118.0M      34M
conv 7x7/2, separable x8       2,712     2.6     2.7K      34.0M      34M
conv 1x1 + 3x3               114,688   112.0     112K     359.7M     360M
inception (3a)               163,328   159.5     159K     128.0M     128M
inception (3b)               388,096   379.0     380K     304.3M     304M
inception (4a)               375,552   366.8     364K      73.6M      73M
inception (4b)               448,512   438.0     437K      87.9M      88M
inception (4c)               509,440   497.5     463K      99.9M     100M
inception (4d)               604,672   590.5     580K     118.5M     119M
inception (4e)               867,328   847.0     840K     170.0M     170M
inception (5a)             1,042,432  1018.0    1072K      51.1M      54M
inception (5b)             1,442,816  1409.0    1388K      70.7M      71M
linear 1024 -> 1000        1,024,000  1000.0    1000K       1.0M       1M
recount, dense first layer: 6.99M weights, 1.583B mult-adds
recount, separable first layer: 6.98M weights, 1.499B mult-adds
printed columns summed: 6.80M params, 1.502B ops

The operations column agrees with the recount to within a million multiply-adds except for module (5a), printed as 54 million against 51.1 million recounted. The parameter column fits less well, at between 0.97 and 1.10 times the printed value in decimal thousands (module 4c: 463K printed, 509,440 recounted). Reading “K” as 1,024 makes the $3 \times 3$ stem row and the classifier match exactly, but not the first layer or the modules. The paper does not explain the differences, and we have no further explanation for them.

Results

Classification protocol. The ILSVRC 2014 classification data has about 1.2 million training images, 50,000 validation images and 100,000 test images in 1,000 classes, and the challenge ranks entries by top-5 error (see accuracy). The team used no outside training data. Its submission combined three test-time techniques:

• Seven networks. Seven versions of the same GoogLeNet model, one of them wider, were trained independently with the same initialization (even the same initial weights, by an oversight, the authors note) and learning-rate policy. They differed only in image sampling and in the random order of training images.
• 144 crops. Each image was resized to four scales, with shorter sides of 256, 288, 320 and 352 pixels. From each, the left, centre and right squares were taken (top, centre and bottom for portrait images), and from each square the four corner and centre $224 \times 224$ crops plus the whole square resized to $224 \times 224$, with their mirror images: $4 \times 3 \times 6 \times 2 = 144$ crops. Howard’s 2013 entry had used a similar scheme, which the authors found slightly worse.
• Averaging. Softmax probabilities were averaged over crops and over networks. Max pooling over crops and averaging over classifiers were tried on validation data and did worse.

Classification results. The final submission scored 6.67% top-5 error on both the validation and test data and ranked first. The authors describe this as a 56.5% relative reduction from SuperVision’s 2012 entry (AlexNet) and about 40% from Clarifai’s 2013 winner, both of which had used outside data. Their Table 2 lists VGG (Simonyan & Zisserman, 2015; see VGG) second with 7.32% and MSRA third with 7.35%. Table 3 separates the contributions on the validation set; the single model is the one with the lowest top-1 validation error:

Models	Crops per image	Cost (relative)	Top-5 error (val)
1	1	1	10.07%
1	10	10	9.15%
1	144	144	7.89%
7	1	7	8.09%
7	10	70	7.62%
7	144	1,008	6.67%

The authors note that so many crops may not be needed in real applications, since the benefit of more crops becomes marginal after a reasonable number.

Detection. The ILSVRC 2014 detection task asks for boxes around objects of 200 classes, counted correct at a Jaccard overlap of at least 50% and scored by mean average precision. GoogLeNet’s entry followed R-CNN (Girshick et al., 2014) with an Inception network as the region classifier. For proposals, it doubled the superpixel size of selective search, which halved its proposals, and added 200 proposals from MultiBox (Erhan et al., 2014). The result was about 60% of R-CNN’s proposals with coverage up from 92% to 93%, worth 1% mAP for a single model. The entry used no bounding-box regression, for lack of time, no localization data for pretraining and, per Table 5, no contextual model; the official results page lists pretraining on ILSVRC 2012 classification data as its outside data. An ensemble of six networks reached 43.9% mAP, ahead of CUHK DeepID-Net (40.7%) and Deep Insight (40.5%) in Table 4, and almost double the 22.6% of the 2013 winner. The text describes the ensemble as raising accuracy from 40% to 43.9%, but Table 5 gives the single model 38.02%, against 40.2% for Deep Insight’s single model, which used context and box regression.

The official results. The ILSVRC 2014 results page lists the team as GoogLeNet. It highlights winners in yellow, and marks this entry as the winner of classification (task 2a, provided training data) by classification error, 0.06656, with the description “No localization” and the reported validation error 6.66%. The localization winner on the same task was VGG. In detection, GoogLeNet is marked as the winner of the track with additional training data (task 1b), with mAP 0.439329 and 142 of 200 categories won, and its single-model entry scored 0.380277; the track with provided data only (task 1a) was won by NUS. The page agrees with the paper’s Table 4 on the detection ranking and lists VGG’s best classification entry at 0.07325, but its best MSRA Visual Computing entry is 0.0806, not the 7.35% of the paper’s Table 2.

Impact

In 2025 the IEEE PAMI Technical Committee gave the paper the Longuet-Higgins Prize, which recognizes CVPR papers from ten years earlier with significant impact on computer vision research; the other 2025 winner was the FCN paper.

The network also became a building block for other tasks. The FCN authors (Long et al., 2015) converted GoogLeNet into a segmentation network alongside AlexNet and VGG, using their own reimplementation because, they note, no public version was available. The YOLO detector (Redmon et al., 2016) describes its network as inspired by GoogLeNet, with $1 \times 1$ reductions followed by $3 \times 3$ convolutions in place of Inception modules. The ResNet paper cites GoogLeNet among the “very deep” models, sixteen to thirty layers, behind the leading ImageNet results. Two of the paper’s design choices outlived the architecture (our summary; the Inception and ResNet articles give the details): a global-average-pooling head in place of large fully connected layers, and the scale and aspect-ratio crop augmentation.

Limitations

• The theory was not tested. The authors cautioned that the architecture’s success might not be attributable to the principles that led to it, which would need much more thorough analysis. Their conclusion also concedes that similar results could be expected from much more expensive non-Inception networks of similar depth and width. The paper reports no controlled comparison with a non-Inception network of equal cost (our reading).
• Hand design and missing details. The 3 to 10 times speed-ups over similarly performing non-Inception networks (2 to 3 times in the arXiv version) required careful manual design, by the authors’ account. The paper omits the details of the deeper and wider seventh network, whose quality the CVPR version calls slightly superior to GoogLeNet’s and the arXiv version slightly inferior. The training recipe was a moving target, and the seven networks shared their initial weights by accident.
• Expensive evaluation. The headline 6.67% needs seven networks and 144 crops per image, 1,008 forward passes; a single network on a single crop scored 10.07% on validation.
• Internal inconsistencies. Table 1’s parameter column does not match its own widths, the 12-fold parameter reduction is not reconciled with any total, and the detection text and Table 5 give different single-model accuracies.
• Criticism from the authors’ own follow-ups. Szegedy et al. (2016) wrote that the architecture’s complexity makes it hard to change, since naive scaling loses much of its computational advantage, and that the GoogLeNet paper does not clearly describe the factors behind its design decisions. They also concluded that the paper’s hypothesis about the auxiliary classifiers was most likely misplaced and that the classifiers act as regularizers instead. Szegedy et al. (2017) added that Inception-v3 had inherited baggage from the earlier designs, chiefly constraints from partitioning the model for distributed training with DistBelief.

What Came After

Szegedy and colleagues developed the design in three further papers within eighteen months of the preprint, each keeping the multi-branch module and changing what surrounds it. The batch-normalization paper (Ioffe & Szegedy; arXiv, February 2015; ICML 2015) added batch normalization to a variant that later papers call BN-Inception or Inception-v2, which replaced each $5 \times 5$ convolution with two $3 \times 3$ convolutions and dropped the pooling layers between modules. “Rethinking the Inception Architecture” (arXiv, December 2015; CVPR 2016) introduced Inception-v3, with asymmetric $1 \times n$ and $n \times 1$ factorizations, grid reduction by parallel strided branches, and label smoothing. Inception-v4 and Inception-ResNet (arXiv, February 2016; AAAI 2017) combined the module with residual connections. Chollet’s Xception (2017) took the module to an extreme with depthwise separable convolutions. Later efficient architectures such as MobileNet likewise report their cost in multiply-adds. The details of each version are in the Inception article.