Inception

Inception is a family of convolutional networks for image classification built from Inception modules: blocks that apply $1 \times 1$, $3 \times 3$ and $5 \times 5$ convolutions and a max-pooling path to the same input in parallel and concatenate their outputs along the channel axis. Christian Szegedy and colleagues, most of them at Google, introduced it with GoogLeNet, the network that won the ILSVRC 2014 classification challenge, posted to arXiv in September 2014 and published at CVPR 2015. Refinements followed: BN-Inception, which added batch normalization (2015); Inception-v3, which factorized large convolutions into smaller and asymmetric ones (2016); and Inception-v4 and Inception-ResNet, the latter with residual connections (2017). Xception (2017) took the module to its limit with depthwise separable convolutions. Inception-v3 remains in wide use as the fixed feature extractor behind the Fréchet Inception Distance. This article covers the family as it is understood today; the 2015 publication itself is the subject of the GoogLeNet paper article.

Problem Addressed

By 2014 the straightforward way to a better ImageNet classifier was a bigger one, and Szegedy et al. (2015) named its two costs: more parameters make a network more prone to overfitting, and widening two chained convolutional layers uniformly increases their computation quadratically, much of it wasted if many of the added weights end up near zero. They set out to increase depth and width at a fixed computational budget; for most experiments the models were designed to stay within 1.5 billion multiply-adds at inference, so that they could be used in practice and not only as an academic exercise.

Their argument started from sparsity. A theoretical result of Arora et al., which they cite, suggests building a network layer by layer by clustering units with correlated outputs, which would give sparse connections; but sparse computation was inefficient on the hardware and libraries of the time, which were tuned for dense matrix multiplication. The Inception module was meant to approximate such a sparse structure with dense, readily available components. The authors themselves warned that the network’s success might not be attributable to these guiding principles.

Architecture

All versions share a plan that AlexNet and VGG did not have: a conventional stem of plain convolutions and pooling at high resolution, then a stack of multi-branch modules on progressively smaller grids, then global average pooling and one linear classifier. What changes between versions is the inside of the modules, how grids are reduced, and how the network is trained.

The Inception Module and 1×1 Reductions

In the naive module, the previous layer feeds four branches: a $1 \times 1$ convolution, a $3 \times 3$ convolution, a $5 \times 5$ convolution and a $3 \times 3$ max pool with stride 1. Padding keeps all four outputs at the input’s height and width, so they can be stacked into one tensor along the channel axis. The branches see the same feature map at different receptive-field sizes, which matches the authors’ intuition that visual information should be processed at several scales and then aggregated. Restricting the kernels to $1 \times 1$, $3 \times 3$ and $5 \times 5$ was, in their words, a matter of convenience (it avoids patch-alignment issues), not necessity, and they expected the share of $3 \times 3$ and $5 \times 5$ filters to rise in higher layers, where features are more abstract and less spatially concentrated.

The naive module has two problems. A $5 \times 5$ convolution on a wide input is expensive, and the pooling branch outputs as many channels as it receives, so the module’s output is always wider than its input and the width grows with every module stacked. GoogLeNet therefore puts $1 \times 1$ convolutions in front of the $3 \times 3$ and $5 \times 5$ convolutions (the “#3×3 reduce” and “#5×5 reduce” columns of the paper’s Table 1) and after the pooling (the “pool proj” column), each followed by a ReLU. The paper justifies the reduction by the success of embeddings: a low-dimensional code can hold much of the information in a large image patch. The $1 \times 1$ convolution itself comes from Network in Network (see convolutional neural network), but there it added depth; in GoogLeNet its main job is to remove computational bottlenecks.

For a module with input width $C$ on an $H \times W$ grid, with $n_1$, $n_3$, $n_5$ and $n_p$ outputs from the four branches and $r_3$, $r_5$ channels after the two reductions, the weight count is

$$C\,(n_1 + r_3 + r_5 + n_p) + 9\, r_3 n_3 + 25\, r_5 n_5 ,$$

and with stride 1 each weight is used once per output position, so the multiply-adds are $HW$ times that. The output has $n_1 + n_3 + n_5 + n_p$ channels, chosen freely rather than forced to exceed $C$. For GoogLeNet’s first module, inception (3a), $C = 192$ on a $28 \times 28$ grid, this gives 163,328 weights and 128 million multiply-adds, the “ops” value printed in Table 1; the same branch widths without reductions cost 2.4 times as much and output 416 channels instead of 256 (see the example below).

GoogLeNet (Inception-v1)

GoogLeNet is the configuration of the paper’s Table 1, used for six of the seven networks in the ILSVRC 2014 ensemble. Its input is a $224 \times 224$ RGB crop with the mean subtracted. The stem is a $7 \times 7$ convolution with stride 2 and 64 filters, a $3 \times 3$ max pool with stride 2, a $1 \times 1$ convolution with 64 filters and a $3 \times 3$ convolution with 192 filters, and another stride-2 max pool; the network diagram (Figure 3) also shows local response normalization in the stem. Nine Inception modules follow: (3a) and (3b) on a $28 \times 28$ grid, (4a) to (4e) on $14 \times 14$, and (5a) and (5b) on $7 \times 7$, with stride-2 max pools between the stages. The output of (5b), $7 \times 7 \times 1024$, is averaged over the grid, passed through 40% dropout and a linear layer to 1,000 classes. All convolutions, including the reductions, use ReLUs. Keeping the low layers conventional was, the paper says, a matter of memory efficiency in the authors’ implementation, not a requirement.

The network is 22 layers deep counting only layers with parameters (27 counting pooling), and the authors estimate about 100 independent building blocks. The head follows Network in Network’s global average pooling (see convolutional neural network), with an extra linear layer that the authors added so the network could be adapted to other label sets.

The paper prints no total parameter count; the Inception-v3 paper puts it at around 7 million. How Table 1’s parameter and operation columns compare with a recount from its widths, including a first layer that only a depthwise separable convolution fits, is discussed in the GoogLeNet paper article.

Auxiliary Classifiers

Because GoogLeNet is deep, the authors worried about propagating gradients back through all its layers. Reasoning that shallower networks do well on ImageNet, so mid-level features should already be discriminative, they attached two small classifiers to the outputs of (4a) and (4d): a $5 \times 5$ average pool with stride 3 (giving $4 \times 4 \times 512$ and $4 \times 4 \times 528$), a $1 \times 1$ convolution with 128 filters and a ReLU, a fully connected layer of 1,024 ReLUs, dropout of 70%, and a linear softmax classifier over the same 1,000 classes. Their losses are added to the main loss with weight 0.3 during training, and the side networks are discarded at inference. The paper expected them to counter the vanishing gradient problem and to regularize, but reports that later control experiments found their effect relatively minor, around 0.5%, and that one of them was enough.

Szegedy et al. (2016) revisited the idea for Inception-v3. In their experiments the auxiliary classifiers did not speed up early training: networks with and without them trained almost identically until high accuracy, after which the version with the side head pulled ahead to a slightly higher plateau, and removing GoogLeNet’s lower side head did no harm. They concluded that the original hypothesis, that the side heads help the low-level features evolve, was most likely misplaced, and that auxiliary classifiers act as regularizers; in support, the main classifier did better when the side head was batch-normalized or had dropout. Inception-v3 keeps one side head on top of the last $17 \times 17$ layer, and batch-normalizing it gave a 0.4% absolute gain in top-1 accuracy. BN-Inception likewise has a single side tower, on top of (4c), with weight 0.4 in the objective; this is stated in the supplementary material of the ICML paper, not in the appendix of arXiv v3.

Factorized Convolutions

Inception-v3 rests on four design principles that Szegedy et al. (2016) present as speculative, drawn from large-scale experiments: avoid representational bottlenecks, especially early, so that the representation shrinks gently from input to output; higher-dimensional representations are easier to process locally; spatial aggregation can be done on reduced embeddings with little loss, because neighbouring units are strongly correlated; and balance width against depth. Under these principles, they replaced large kernels with stacks of smaller ones.

$5 \times 5$ as two $3 \times 3$. Each output of a $5 \times 5$ convolution looks like a small fully connected network sliding over $5 \times 5$ tiles. Replacing it by a $3 \times 3$ convolution followed by a second $3 \times 3$ convolution on the resulting $3 \times 3$ grid covers the same $5 \times 5$ window, and sliding this pair over the image shares weights between neighbouring tiles. With equal numbers of input and output channels the cost falls by a factor $(9 + 9)/25$, a 28% saving in both parameters and multiply-adds. Whether to put a ReLU between the two factors was tested: in their control experiments a linear first layer was always worse, for example 76.2% against 77.2% top-1 validation accuracy in one pair of runs (their Figure 2), a gain they attribute to the larger space of variations the network can learn, especially with batch normalization. BN-Inception had already made the same replacement in 2015.

Asymmetric convolutions. A $3 \times 1$ convolution followed by a $1 \times 3$ convolution covers a $3 \times 3$ window and is 33% cheaper than a $3 \times 3$ convolution with equal input and output widths, against 11% for replacing it with two $2 \times 2$ convolutions. In general an $n \times n$ convolution can be replaced by a $1 \times n$ followed by an $n \times 1$, and the saving grows with $n$: for $n = 7$, 14 weights per channel pair instead of 49. The authors found that this did not work well in early layers but gave very good results on medium grids, from $12 \times 12$ to $20 \times 20$, and Inception-v3 uses $1 \times 7$ and $7 \times 1$ convolutions on its $17 \times 17$ grid. On the coarsest $8 \times 8$ grid the $1 \times 3$ and $3 \times 1$ convolutions run in parallel and their outputs are concatenated, which widens the output in line with the second principle.

This is spatial factorization: a $1 \times n$ followed by an $n \times 1$ filter is a separable filter in the image-processing sense (see convolution). For a single channel without a nonlinearity the pair can only represent rank-1 kernels; across many channels, with a ReLU between the factors, the module is no longer limited that way (our reading). It is a different factorization from the depthwise separable convolution of Xception and MobileNet, which separates space from channels (see depthwise separable convolution).

Grid-Size Reduction

Halving the grid with a pooling layer and then widening the result creates a representational bottleneck, which violates the first principle; widening first with a stride-1 module and then pooling avoids the bottleneck but triples the cost, by the authors’ estimate. Inception-v3 instead reduces the grid with a module of two parallel stride-2 branches, a convolution branch and a pooling branch, whose outputs are concatenated: in the paper’s example, $35 \times 35 \times 320$ becomes $17 \times 17 \times 640$, half from each branch. Inception-v4 uses the same pattern in its Reduction-A and Reduction-B blocks (their diagrams are in arXiv v2 of its paper; the AAAI version shows only their place in the network).

Inception-v3 Layout

Inception-v3 takes $299 \times 299$ inputs. Its stem is a stack of $3 \times 3$ convolutions, including the factorization of the traditional $7 \times 7$ first layer into three $3 \times 3$ layers, and one pooling layer, reaching $35 \times 35$. Three modules with two $3 \times 3$ convolutions in place of $5 \times 5$ follow at $35 \times 35$ with 288 filters, then a grid reduction to $17 \times 17 \times 768$, five modules with $1 \times 7$ and $7 \times 1$ factorizations, a reduction to $8 \times 8 \times 1280$, and two modules with the expanded filter banks, whose concatenated output has 2,048 channels per position. An $8 \times 8$ pooling layer, a linear layer and a softmax complete the network. Table 1 assigns the three module types to the three grids in that order; the paper’s text cites the module figures one number off, and the figure captions agree with the table. The network is 42 layers deep.

The abstract gives about 5 billion multiply-adds and fewer than 25 million parameters; Table 3 lists 4.8 billion operations against 1.5 billion for GoogLeNet. The text gives two different comparisons, a cost about 2.5 times that of GoogLeNet in Section 6 and a 2.5-fold increase over BN-Inception in the conclusion; the table supports the second (2.0 to 4.8 billion). Chollet (2017, Table 3) counts 23,626,728 parameters for Inception-v3 in its configuration without fully connected layers or auxiliary tower.

Residual Inception Blocks

Szegedy et al. (2017) combined Inception modules with ResNet’s residual connections. In an Inception-ResNet block, a cheaper Inception module than in the pure networks is followed by a $1 \times 1$ convolution without activation that expands its output back to the input’s width, and the result is added to the input instead of being concatenated with it. With $F$ the Inception branch and $s$ a scaling constant, a block computes

$$\mathbf{y} = \mathrm{ReLU}\big(\mathbf{x} + s\, F(\mathbf{x})\big).$$

The scaling was a fix for instability. When a residual variant had more than 1,000 filters, the network “died” early in training: the last layer before the average pool produced only zeros after a few tens of thousands of iterations, and neither a lower learning rate nor an extra batch-normalization layer prevented it. Scaling the residuals by factors between 0.1 and 0.3 before the addition stabilized training and never seemed to hurt final accuracy; the AAAI version adds that deeper networks need lower constants. He et al.’s warm-up with a low learning rate was not enough when the number of filters was very high. Batch normalization was applied to the ordinary layers but not on top of the summations, to save memory.

The paper presents three networks. Inception-v4 is a pure Inception network with uniform blocks for each grid size, made possible because the move from DistBelief to TensorFlow removed the need to partition each model replica across machines, a constraint that had shaped the earlier, more irregular designs. Its stem maps $299 \times 299 \times 3$ to $35 \times 35 \times 384$, followed by four Inception-A blocks, Reduction-A, seven Inception-B blocks on $17 \times 17$, Reduction-B, three Inception-C blocks on $8 \times 8 \times 1536$, average pooling, and dropout with keep probability 0.8. Inception-ResNet-v1 has roughly the cost of Inception-v3, and Inception-ResNet-v2 matches the raw cost of Inception-v4; their common schema has 5, 10 and 5 residual blocks on the three grids. Inception-v4’s step time was significantly slower in practice than Inception-ResNet-v2’s, probably, the authors suggest, because of its larger number of layers.

Counting Parameters and Multiply-Adds

The PyTorch code below (output from PyTorch 2.14.1, one thread) runs a synthetic $3 \times 224 \times 224$ image of a disk, a square outline and stripes through GoogLeNet’s stem and module (3a) with random weights, prints the branch shapes, measures (3a) with and without the $1 \times 1$ reductions using PyTorch’s FLOP counter, and checks the factorizations. No pretrained weights are used; the shapes and counts do not depend on them.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.flop_counter import FlopCounterMode

torch.manual_seed(0)
torch.set_num_threads(1)

# A 3x224x224 synthetic image, layout (N, C, H, W): a bright disk (red), a square
# outline (green) and vertical stripes of period 8 (blue).
yy, xx = torch.meshgrid(torch.arange(224.), torch.arange(224.), indexing="ij")
img = torch.zeros(1, 3, 224, 224)
img[0, 0] = (((yy - 70) ** 2 + (xx - 70) ** 2) < 40 ** 2).float()
sq = (yy > 120) & (yy < 200) & (xx > 120) & (xx < 200)
img[0, 1] = (sq & ~((yy > 124) & (yy < 196) & (xx > 124) & (xx < 196))).float()
img[0, 2] = (torch.remainder(xx, 8) < 4).float()

def conv(c_in, c_out, k, stride=1):
    return nn.Sequential(nn.Conv2d(c_in, c_out, k, stride, padding=k // 2), nn.ReLU())

class Inception(nn.Module):
    """GoogLeNet module: 1x1 | 1x1->3x3 | 1x1->5x5 | 3x3 maxpool->1x1, concatenated.
    reduce=False gives the 'naive' module with no 1x1 reductions or pool projection."""
    def __init__(self, c_in, n1, r3, n3, r5, n5, pp, reduce=True):
        super().__init__()
        self.b1 = conv(c_in, n1, 1)
        self.b3 = nn.Sequential(conv(c_in, r3, 1), conv(r3, n3, 3)) if reduce else conv(c_in, n3, 3)
        self.b5 = nn.Sequential(conv(c_in, r5, 1), conv(r5, n5, 5)) if reduce else conv(c_in, n5, 5)
        pool = nn.MaxPool2d(3, stride=1, padding=1)
        self.bp = nn.Sequential(pool, conv(c_in, pp, 1)) if reduce else pool
    def branches(self, x):
        return [self.b1(x), self.b3(x), self.b5(x), self.bp(x)]
    def forward(self, x):
        return torch.cat(self.branches(x), dim=1)          # concatenate along channels

def n_params(m):
    return sum(p.numel() for p in m.parameters())

def mult_adds(m, x):
    with FlopCounterMode(display=False) as fc:
        m(x)
    return fc.get_total_flops() // 2                         # 1 multiply-add = 2 FLOPs

# 1. GoogLeNet stem (Table 1, with a dense first layer), then inception (3a)
stem = nn.Sequential(conv(3, 64, 7, 2), nn.MaxPool2d(3, 2, padding=1),
                     conv(64, 64, 1), conv(64, 192, 3), nn.MaxPool2d(3, 2, padding=1))
widths_3a = (192, 64, 96, 128, 16, 32, 32)
m3a = Inception(*widths_3a)
with torch.no_grad():
    x = stem(img)
    print("stem output:", tuple(x.shape))
    for name, b in zip(["1x1", "3x3", "5x5", "pool"], m3a.branches(x)):
        print(f"  branch {name:4s}: {tuple(b.shape)}")
    print("concatenated:", tuple(m3a(x).shape))

# 2. Cost of (3a) with and without the 1x1 reductions
naive = Inception(*widths_3a, reduce=False)
for name, m in [("with 1x1 reductions", m3a), ("naive (no reductions)", naive)]:
    print(f"{name:22s}: {n_params(m):>9,} params {mult_adds(m, x) / 1e6:7.1f}M mult-adds, "
          f"{m(x).shape[1]} output channels")
c = 192                                                   # naive modules stacked: the pool
for i in range(3):                                        # path passes every channel through
    c_next = 64 + 128 + 32 + c
    m = Inception(c, 64, 0, 128, 0, 32, 0, reduce=False)
    print(f"  naive module {i + 1}: {c} -> {c_next} channels, "
          f"{mult_adds(m, torch.zeros(1, c, 28, 28)) / 1e6:6.1f}M mult-adds")
    c = c_next

# 3. Factorizations at 64 -> 64 channels (weights; mult-adds scale the same per position)
C = 64
pairs = [("5x5", 25, "two 3x3", 9 + 9), ("3x3", 9, "1x3 + 3x1", 3 + 3), ("7x7", 49, "1x7 + 7x1", 7 + 7)]
for big, kb, small, ks in pairs:
    print(f"{big:3s}: {kb * C * C:>7,} weights  {small:9s}: {ks * C * C:>7,}  saving {1 - ks / kb:.0%}")

# Exactness on the image when the kernel factorizes (single channel, linear):
red = img[:, :1]
b3 = torch.tensor([1., 2., 1.]) / 4
b5 = torch.outer(torch.tensor([1., 4., 6., 4., 1.]), torch.tensor([1., 4., 6., 4., 1.])) / 256
k3 = torch.outer(b3, b3)[None, None]
two_3x3 = F.conv2d(F.conv2d(red, k3, padding=1), k3, padding=1)
one_5x5 = F.conv2d(red, b5[None, None], padding=2)
inner = (slice(None), slice(None), slice(4, -4), slice(4, -4))   # zero padding differs at the border
print(f"binomial 5x5 vs two binomial 3x3: max diff {(two_3x3 - one_5x5)[inner].abs().max():.1e}")
col = torch.tensor([[1.], [2.], [1.]])                      # 3x1 smoothing
row = torch.tensor([[-1., 0., 1.]])                          # 1x3 derivative
sobel = col @ row                                            # the 3x3 Sobel kernel
asym = F.conv2d(F.conv2d(red, col[None, None], padding=(1, 0)), row[None, None], padding=(0, 1))
full = F.conv2d(red, sobel[None, None], padding=1)
print(f"Sobel 3x3 vs 3x1 then 1x3: max diff {(asym - full).abs().max():.1e}")
print("singular values, Sobel:", [round(v, 3) for v in torch.linalg.svdvals(sobel).tolist()])
print("singular values, random 3x3:", [round(v, 3) for v in torch.linalg.svdvals(torch.randn(3, 3)).tolist()])

Output:

stem output: (1, 192, 28, 28)
  branch 1x1 : (1, 64, 28, 28)
  branch 3x3 : (1, 128, 28, 28)
  branch 5x5 : (1, 32, 28, 28)
  branch pool: (1, 32, 28, 28)
concatenated: (1, 256, 28, 28)
with 1x1 reductions   :   163,696 params   128.0M mult-adds, 256 output channels
naive (no reductions) :   387,296 params   303.5M mult-adds, 416 output channels
  naive module 1: 192 -> 416 channels,  303.5M mult-adds
  naive module 2: 416 -> 640 channels,  657.5M mult-adds
  naive module 3: 640 -> 864 channels, 1011.5M mult-adds
5x5: 102,400 weights  two 3x3  :  73,728  saving 28%
3x3:  36,864 weights  1x3 + 3x1:  24,576  saving 33%
7x7: 200,704 weights  1x7 + 7x1:  57,344  saving 71%
binomial 5x5 vs two binomial 3x3: max diff 0.0e+00
Sobel 3x3 vs 3x1 then 1x3: max diff 0.0e+00
singular values, Sobel: [3.464, 0.0, 0.0]
singular values, random 3x3: [1.895, 1.716, 0.081]

The module’s parameter counts include biases: 163,696 for (3a), against the 163,328 weights given by the cost formula of the module section. Stacked naive modules widen by 224 channels each, because the pooling branch passes every input channel through, and their cost grows in step: the computational blow-up the paper describes.

A binomial $5 \times 5$ blur is exactly two binomial $3 \times 3$ blurs, because $[1, 4, 6, 4, 1]$ is $[1, 2, 1]$ convolved with itself, and the Sobel kernel is exactly a $3 \times 1$ smoothing followed by a $1 \times 3$ derivative, because it has rank 1; a random $3 \times 3$ kernel has three nonzero singular values and has no such factorization. A single-channel linear pair of $3 \times 3$ layers has 18 weights, fewer than the 25 of a $5 \times 5$ kernel, so it cannot represent every $5 \times 5$ filter either (our reasoning); the multi-channel modules with ReLUs between the factors are what Szegedy et al. compared experimentally.

Training

GoogLeNet was trained on CPUs with Google’s DistBelief system, using asynchronous stochastic gradient descent with momentum 0.9, a learning rate decreased by 4% every 8 epochs, and Polyak averaging for the final model. With weights $0.3$ on the two auxiliary classifiers, the training objective is

$$\mathcal{L} = \mathcal{L}_\text{main} + 0.3\,\big(\mathcal{L}_\text{aux1} + \mathcal{L}_\text{aux2}\big),$$

each term a softmax cross-entropy. The image sampling changed over the months before the competition, so the paper gives no single recipe; one verified after the competition samples patches covering 8% to 100% of the image area with aspect ratios between 3/4 and 4/3, plus Andrew Howard’s photometric distortions. That crop scheme survives as the defaults of TorchVision’s RandomResizedCrop (scale 0.08 to 1, ratio 3/4 to 4/3), whose documentation says it is popularly used to train Inception networks.

BN-Inception added batch normalization before every nonlinearity and changed the training recipe with it, including higher learning rates and no dropout (see the batch normalization paper).

Inception-v3 was trained with TensorFlow on 50 replicas, each on an NVIDIA Kepler GPU, with batches of 32 for 100 epochs. The best models used RMSProp with decay 0.9 and $\epsilon = 1.0$, a learning rate of 0.045 decayed every two epochs by a factor of 0.94, gradient clipping at threshold 2.0, and evaluation with a running average of the parameters. The paper also introduced label smoothing, which mixes the one-hot target with a uniform distribution over the $K = 1000$ classes, with weight $\varepsilon = 0.1$, to keep the largest logit from growing far beyond the others; the cross-entropy loss article gives the formula and its effect on the loss. The authors report a consistent improvement of about 0.2% absolute in both top-1 and top-5 error; in their Table 3 the change from RMSProp alone to RMSProp with label smoothing is 23.1% to 22.8% top-1 and 6.3% to 6.1% top-5. Inception-v4 and Inception-ResNet were trained with the same optimizer and learning-rate schedule on 20 replicas.

Inference

At inference the auxiliary classifiers are dropped and the network is a single path from image to softmax. GoogLeNet and BN-Inception take $224 \times 224$ crops; Inception-v3, Inception-v4 and Inception-ResNet take $299 \times 299$.

The headline results use heavy multi-crop testing: for ILSVRC 2014, Szegedy et al. averaged softmax probabilities over 144 crops per image, taken at four scales, and over the models of the ensemble, and BN-Inception and Inception-v3 report 144-crop results too. The protocol and how much each part of it contributed are in the GoogLeNet paper article.

Szegedy et al. (2016) also asked how much input resolution matters at a fixed cost. Networks with $79 \times 79$, $151 \times 151$ and $299 \times 299$ inputs, with strides and pooling adjusted to keep the computation almost identical, reached 75.2%, 76.4% and 76.6% single-crop top-1 accuracy on the ILSVRC 2012 validation set. The lower-resolution networks took longer to train but ended close to the full-resolution one, which suggests to the authors that dedicated high-cost networks for low-resolution inputs could help with small objects in detection.

Performance

Each version’s ImageNet results, from the paper that introduced it, as top-1 and top-5 error (see accuracy). “Val” is the validation set of the ILSVRC classification challenge (ILSVRC 2014 in the GoogLeNet paper, ILSVRC 2012 in the later ones); the Inception-v3 paper evaluates on 48,238 validation images without the blacklisted entries, which it reports to be about 0.2% (top-1) and 0.1% (top-5) better than all 50,000, and the Inception-v4 paper uses that subset for its single-crop results, about 0.3% and 0.15% more optimistic by its estimate, and the full 50,000 for multi-crop results.

Version (paper)	Protocol	Top-1 error	Top-5 error
GoogLeNet (2015)	1 model, 1 crop, val	–	10.07%
GoogLeNet (2015)	7 models, 144 crops, test	–	6.67%
BN-Inception (2015)	1 model, 1 crop, val	25.2%	7.82%
BN-Inception (2015)	1 model, 144 crops, val	21.99%	5.82%
BN-Inception (2015)	6 models, 144 crops, test	–	4.82%
Inception-v3 (2016)	1 model, 1 crop, val subset	21.2%	5.6%
Inception-v3 (2016)	1 model, 144 crops, val subset	18.77%	4.2%
Inception-v3 (2016)	4 models, 144 crops	17.2% (val)	3.58% (test)
Inception-ResNet-v1 (2017)	1 model, 1 crop, val subset	21.3%	5.5%
Inception-v4 (2017)	1 model, 1 crop, val subset	20.0%	5.0%
Inception-ResNet-v2 (2017)	1 model, 1 crop, val subset	19.9%	4.9%
Inception-v4 (2017)	1 model, 144 crops, val	17.7%	3.8%
Inception-ResNet-v2 (2017)	1 model, 144 crops, val	17.8%	3.7%
Inception-v4 + 3 × Inception-ResNet-v2 (2017)	4 models, 144 crops	16.4% (val)	3.1% (val), 3.08% (test)

GoogLeNet’s 6.67% won ILSVRC 2014 classification; the competition entries, including the detection entry that also won, are described in the GoogLeNet paper article. For Inception-v3’s ensemble, Table 5 gives 3.58% top-5 on the test set and 3.46% on validation, while the abstract rounds to 3.5% (validation) and 3.6% (test) and gives 17.3% top-1 against the table’s 17.2%. The ensemble’s 16.4% top-1 is from the AAAI version; the arXiv version prints 16.5%. Szegedy et al. (2017) found that gains in single-frame accuracy did not carry over into similarly large gains for ensembles.

The Inception-v3 paper’s Table 3 adds its changes one at a time, all single crop on the validation subset, with costs in billions of operations:

Network	Top-1 error	Top-5 error	Cost
GoogLeNet	29%	9.2%	1.5
BN-GoogLeNet	26.8%	–	1.5
BN-Inception	25.2%	7.8%	2.0
Inception-v3-basic	23.4%	–	3.8
+ RMSProp	23.1%	6.3%	3.8
+ label smoothing	22.8%	6.1%	3.8
+ factorized $7 \times 7$	21.6%	5.8%	4.8
+ batch-normalized auxiliary classifier (Inception-v3)	21.2%	5.6%	4.8

The paper does not explain why its GoogLeNet row (9.2% top-5) is lower than GoogLeNet’s own single-crop 10.07%; by its own estimate the validation subset accounts for only about 0.1% of top-5 error. Residual connections mainly changed training speed: Inception-ResNet-v1 trained much faster than Inception-v3 but ended slightly worse in top-1 error and, in the arXiv version’s top-5 plot, slightly better in top-5, and Inception-ResNet-v2 trained faster than Inception-v4 and ended slightly better. The authors conclude that final accuracy seems to depend mainly on model size, and that their results do not support He et al.’s view that residual connections are necessary for training very deep networks, at least for image recognition.

Strengths

• Accuracy per operation. GoogLeNet beat the 2014 field within a budget of 1.5 billion multiply-adds, and Szegedy et al. (2016) report that Inception-v3 cut the top-5 error of He et al.’s PReLU network by 25% (relative) while being about six times cheaper to compute, by their estimate. The authors report that, with careful manual design, Inception networks were 3 to 10 times faster than similarly performing networks without the Inception structure.
• Multi-scale features in one layer. Each module sees its input at several receptive-field sizes at once, and the $1 \times 1$ reductions make the mix affordable.
• Training ideas that spread. Label smoothing and the scale-and-aspect-ratio crop augmentation came out of this line of work and are used well beyond it; the ResNet article describes ResNet training recipes that adopted both.

Limitations

• Hand-tuned complexity. Szegedy et al. (2016) note that the architecture’s complexity makes it hard to change: scaling it up naively loses much of its computational advantage, and the GoogLeNet paper does not explain the factors behind its design decisions. Doubling every filter bank, for example, would quadruple both cost and parameters. The Inception-v4 paper adds that the earlier versions carried baggage from partitioning models for DistBelief.
• Fragmentation. Ma et al. (2018) cite the multi-path blocks of the GoogLeNet series as an example of network fragmentation: many small operators instead of a few large ones, which they found reduces parallelism on GPUs and adds kernel-launch and synchronization overheads. Szegedy et al. (2017) themselves found Inception-v4 significantly slower per step than Inception-ResNet-v2 at the same raw cost.
• Heavy test-time protocols. The best published numbers use 144 crops per image and ensembles, which are far more expensive than the single-crop cost of the network suggests.

Historical Importance

GoogLeNet won ILSVRC 2014 classification with 12 times fewer parameters than AlexNet, by its paper’s count, and set patterns that later networks kept (our summary): cheap $1 \times 1$ convolutions to control width, a global-average-pooling head in place of large fully connected layers, and computation budgets as an explicit design constraint. The family’s later papers carried other ideas into general use. BN-Inception was the network on which batch normalization was demonstrated, Inception-v3 introduced label smoothing and the grid-reduction pattern, and Inception-ResNet was an early hybrid of multi-branch and residual designs.

The family’s most visible use today is as a fixed feature network for evaluating generative models. Salimans et al. (2016) defined the Inception Score with a pretrained Inception model distributed by TensorFlow (inception-2015-12-05), citing the Inception-v3 paper, and Heusel et al. (2017) built the Fréchet Inception Distance on features of the same Inception-v3 model. Video recognition also built on the family: I3D inflated a batch-normalized Inception-v1 into a 3D network.

Variants

• GoogLeNet / Inception-v1 (2014; CVPR 2015). Nine Inception modules with $1 \times 1$ reductions, two auxiliary classifiers, $224 \times 224$ input, described above. The paper also used one deeper and wider Inception network in its ensemble but omitted its details.
• BN-Inception (2015). The network of Ioffe and Szegedy’s batch-normalization paper, which they called simply “Inception”: each $5 \times 5$ convolution replaced by two $3 \times 3$ convolutions with up to 128 filters, 13.6 million parameters and no fully connected layer other than the softmax. The ICML supplementary material lists the other changes from GoogLeNet: the replacement adds 9 weight layers to the maximum depth, 25% to the parameters and about 30% to the computation; the $28 \times 28$ stage has three modules instead of two; modules use average or max pooling; there are no pooling layers between modules, with stride-2 convolution and pooling inside modules (3c) and (4e) instead. In a separate sentence, not in that list, it states that the first convolutional layer used separable convolution with depth multiplier 8, which reduced computation but increased memory consumption during training.
• Inception-v3 (2015; CVPR 2016). Factorized and asymmetric convolutions, efficient grid reduction, $299 \times 299$ input, RMSProp, label smoothing and a batch-normalized auxiliary classifier. The name Inception-v2 is used for two different networks. In arXiv v3 of the paper, Inception-v2 is the factorized network of its Section 6 and the rows of Table 3 that add changes to it, and Inception-v3 is the final row with every change. The CVPR version relabels the section and the table rows “Inception-v3” but keeps “Inception-v2” in the text of its Section 10 and in the conclusion’s description of the 21.2% model. The Inception-v4 paper and Xception, on the other hand, call BN-Inception “Inception-v2”. When a source says Inception-v2, check which network it means.
• Inception-v4, Inception-ResNet-v1 and Inception-ResNet-v2 (2016; AAAI 2017). Uniform pure Inception blocks, and residual Inception blocks with scaled residuals, described above.
• Xception (2016; CVPR 2017). Chollet (2017) reinterpreted the Inception module as a step between a regular convolution and a depthwise separable one, and built a network entirely from depthwise separable convolutions with residual connections, with about as many parameters as Inception-v3. The depthwise separable convolution article covers the operation and Xception’s results.

Implementations

• TorchVision (version 0.29.0): torchvision.models.googlenet and torchvision.models.inception_v3, with ImageNet weights that the documentation says are ported from the original papers. Both models put batch normalization (with $\epsilon = 0.001$) after every convolution, so the GoogLeNet model is a batch-normalized variant rather than the 2014 network; a comment in its source notes that the “5×5” branch uses a $3 \times 3$ kernel, a known bug. The weight metadata give 6,624,904 parameters and 69.778% / 89.530% top-1 / top-5 ImageNet-1K accuracy for GoogLeNet, and 27,161,264 parameters (including the auxiliary classifier) and 77.294% / 93.450% for Inception-v3, evaluated on $299 \times 299$ crops. Both return auxiliary logits in training mode when aux_logits=True.
• Keras (version 3.12.4): InceptionV3, InceptionResNetV2 and Xception in keras.applications, all with $299 \times 299$ default inputs. The Inception-ResNet-v2 implementation scales its residuals by 0.17, 0.1 and 0.2 on the three grids, inside the paper’s range (the last $8 \times 8$ block uses 1.0 and no activation), but stacks 10, 20 and 10 residual blocks, twice the 5, 10 and 5 of the paper’s schema.