EfficientNet

EfficientNet is a family of convolutional networks for image classification introduced by Mingxing Tan and Quoc Le of Google Brain at ICML 2019. Its contribution is a rule for making a network bigger. Instead of adding layers, as the ResNets do, or channels, as MobileNet’s width multiplier does, compound scaling increases depth, width and input resolution together, by fixed ratios tied to one coefficient $\varphi$. The authors applied the rule to a small network found by neural architecture search, EfficientNet-B0, whose main block is MobileNetV2’s inverted residual with squeeze-and-excitation, and obtained B1 to B7. EfficientNetV2 (Tan & Le, ICML 2021) changed the block mix and the training procedure to train faster. EfficientNets also served as backbones, for example in the EfficientDet detectors, and an enlarged version, EfficientNet-L2, was the student network of Noisy Student self-training.

The paper was revised on arXiv after ICML: v3 (November 2019) already changed the ImageNet numbers, and v4 and v5 (September 2020) changed them again and added training details. This article cites the ICML version and v5 and says which version each number comes from.

Problem Addressed

By 2019, ImageNet accuracy was improving mostly by making networks bigger, and Tan and Le observed that it was usually done along one dimension at a time: more layers (ResNet-18 to ResNet-200), more channels (Wide ResNet, MobileNet’s width multiplier) or larger input images (GPipe, at 480 pixels, reached 84.3% top-1 with 557M parameters). Scaling two or three dimensions at once was possible, but in their words it needed tedious manual tuning. Their question was whether there is a principled way to scale a network up for a larger budget.

They framed it as an optimization (§3.1). A network is a sequence of stages $i = 1, \dots, s$, each repeating a layer $\hat{F}_i$ $\hat{L}_i$ times on inputs of shape $\hat{H}_i \times \hat{W}_i \times \hat{C}_i$ (their Eq. 1). Scaling keeps every $\hat{F}_i$ fixed and multiplies all depths by $d$, all widths by $w$ and the resolution by $r$:

$$\max_{d, w, r} \; \mathrm{Accuracy}\big(\mathcal{N}(d, w, r)\big) \quad \text{s.t.} \quad \mathcal{N}(d, w, r) = \bigodot_{i=1}^{s} \hat{F}_i^{\,d \cdot \hat{L}_i}\big(X_{\langle r \hat{H}_i,\, r \hat{W}_i,\, w \hat{C}_i \rangle}\big),$$

with memory and FLOPS below their targets (Eq. 2). Restricting all layers to the same ratios shrinks the search from per-layer choices to three numbers.

Their study of the three dimensions (§3.2, Figure 3) scaled B0 along one at a time. Width coefficients from 1.0 to 5.0, depth coefficients from 1.0 to 8.0 and resolutions from 224 to 560 pixels ($r = 1.0$ to $2.5$) all raised accuracy, but each curve flattened after about 80% top-1. This is their Observation 1: scaling any one dimension improves accuracy, with diminishing returns for bigger models. Figure 4 then scaled width for four baselines: B0 as is, deeper ($d = 2$), at higher resolution ($r = 1.3$) and both. At the same FLOPS, the deeper, higher-resolution baselines reached better accuracy, which gives Observation 2: better accuracy and efficiency require balancing all three. (The text describes the last baseline as $r = 2.0$; the figure legend and caption give $r = 1.3$, 299 pixels.) Their intuition for the coupling is that a larger image needs more layers to enlarge the receptive field and more channels to capture finer patterns over the extra pixels.

Architecture

Compound Scaling

Compound scaling ties the three coefficients to one user-chosen exponent $\varphi$ (Eq. 3):

$$d = \alpha^{\varphi}, \qquad w = \beta^{\varphi}, \qquad r = \gamma^{\varphi}, \qquad \text{s.t. } \alpha \cdot \beta^2 \cdot \gamma^2 \approx 2, \quad \alpha, \beta, \gamma \ge 1 .$$

The squares come from the cost of a convolution. A layer with $k \times k$ kernels, $C_\text{in}$ inputs and $C_\text{out}$ outputs on an $H \times W$ map costs $k^2 C_\text{in} C_\text{out} H W$ multiply-adds, so multiplying every width by $w$ multiplies it by $w^2$, multiplying the resolution by $r$ multiplies $HW$ by $r^2$, and multiplying the number of layers by $d$ multiplies the total by $d$: the cost scales as $d \cdot w^2 \cdot r^2$. With the constraint, each unit increase of $\varphi$ roughly doubles the FLOPS, $(\alpha \beta^2 \gamma^2)^{\varphi} \approx 2^{\varphi}$; a footnote warns that rounding makes actual FLOPS differ from this value.

The constants were found in two steps (§4). With $\varphi = 1$, a small grid search on B0 gave $\alpha = 1.2$, $\beta = 1.1$ and $\gamma = 1.15$; these were then fixed and B0 was scaled with larger $\varphi$ to obtain B1 to B7. Searching $\alpha, \beta, \gamma$ around a large model might work better, the authors note, but would cost far more; the point of the procedure is to search once, on the small network. With the published constants, $\alpha \beta^2 \gamma^2 = 1.92$, slightly below 2 (our arithmetic, in the example below).

EfficientNet-B0

Because scaling never changes the layers themselves, the result can only be as good as the baseline, and Tan and Le searched for one. They used the search space of MnasNet (Tan et al., 2019), the platform-aware search that MobileNetV3 later built on (see MobileNet), with a multi-objective reward

$$\mathrm{ACC}(m) \times \left[ \frac{\mathrm{FLOPS}(m)}{T} \right]^{w}, \qquad T = 400\text{M}, \quad w = -0.07,$$

where $T$ is the target FLOPS and $w$ trades accuracy against cost (this $w$ is the reward exponent, not the width coefficient). Unlike MnasNet, the search optimized FLOPS rather than measured latency, because no particular hardware was targeted. The result resembles MnasNet but is slightly larger, because of the larger FLOPS target (§4).

B0 is the following network (Table 1; each row is a stage, with the input resolution of its first layer):

Stage	Operator	Input resolution	Output channels	Layers
1	Conv $3 \times 3$	$224 \times 224$	32	1
2	MBConv1, $k3 \times 3$	$112 \times 112$	16	1
3	MBConv6, $k3 \times 3$	$112 \times 112$	24	2
4	MBConv6, $k5 \times 5$	$56 \times 56$	40	2
5	MBConv6, $k3 \times 3$	$28 \times 28$	80	3
6	MBConv6, $k5 \times 5$	$28 \times 28$	112	3
7	MBConv6, $k5 \times 5$	$14 \times 14$	192	4
8	MBConv6, $k3 \times 3$	$7 \times 7$	320	1
9	Conv $1 \times 1$ & pooling & FC	$7 \times 7$	1280	1

MBConv is the mobile inverted bottleneck of MobileNetV2 and MnasNet, to which the authors add squeeze-and-excitation. MBConv6 expands the channels sixfold with a $1 \times 1$ convolution, filters them with a $k \times k$ depthwise convolution (the first layer of a stage carries the stride), and projects them back with a linear $1 \times 1$ convolution; MBConv1 has no expansion layer. The block’s design, its linear bottleneck and the shortcut between narrow tensors are explained in the MobileNet article, and the factorization behind it in depthwise separable convolution.

The paper’s table does not give the remaining details; the official TensorFlow code does (TPU repository, efficientnet_builder.py and efficientnet_model.py):

• Squeeze-and-excitation sits after the depthwise convolution, on the expanded tensor, as in MobileNetV3. Its inner width is a quarter of the block’s input channels (se0.25 in the block strings), not of the expanded channels, so in an MBConv6 block it reduces $6C$ channels to $C/4$, a reduction ratio of 24 by our arithmetic, compared with 16 in SE-ResNet. Its inner activation is swish and its gate a sigmoid. How the gate works is covered in the squeeze-and-excitation article.
• Activation: swish, $x \cdot \sigma(x)$, after every convolution except the projection (see activation function, where it appears as SiLU).
• Normalization: batch normalization after every convolution, with momentum 0.99 in TensorFlow’s convention (0.01 in PyTorch’s) and $\epsilon = 10^{-3}$.
• Shortcuts join input and output when a block keeps the resolution and the channel count; during training, the residual branch of such a block is dropped at random for each example (stochastic depth, see Training).
• Head: a $1 \times 1$ convolution to 1280 channels, global average pooling, dropout and a 1000-way fully connected layer.

Built this way (example below), B0 has 16 MBConv layers and matches the 5.3M parameters and 0.39B FLOPS of the paper’s Table 2. Figure 4’s caption describes the baseline as 18 convolutional layers, which matches the 16 blocks plus the stem and head convolutions (our reading).

FLOPS Means Multiply-Adds

The paper never defines FLOPS. Three things show that its numbers count multiply-adds, each multiply and add pair as one operation. Our counts of multiply-adds for B0 to B7 match Table 2 to within 5% (example below); the comparison rows list ResNet-50 at 4.1B, close to the 3.8 billion that the ResNet paper reports as multiply-adds; and the EfficientDet paper, by an overlapping set of authors, states that its FLOPs denote multiply-adds, as in EfficientNet. Other papers count differently: ResNet-RS (Bello et al., 2021, Table 2) lists B0 at 0.8B FLOPs and B7 at 74B, twice the EfficientNet figures. Comparisons of “FLOPs” across papers can therefore be off by a factor of two.

B1 to B7

The paper gives the constants $\alpha, \beta, \gamma$ but not the value of $\varphi$ used for each model. The official code lists the coefficients directly (efficientnet_params, as width, depth, training resolution and dropout rate):

Model	Width $w$	Depth $d$	Resolution	Dropout	Parameters	FLOPS (Table 2)
B0	1.0	1.0	224	0.2	5.3M	0.39B
B1	1.0	1.1	240	0.2	7.8M	0.70B
B2	1.1	1.2	260	0.3	9.2M	1.0B
B3	1.2	1.4	300	0.3	12M	1.8B
B4	1.4	1.8	380	0.4	19M	4.2B
B5	1.6	2.2	456	0.4	30M	9.9B
B6	1.8	2.6	528	0.5	43M	19B
B7	2.0	3.1	600	0.5	66M	37B

These are not powers of 1.1, 1.2 and 1.15 with one integer exponent. Solving $w = 1.1^{\varphi}$, $d = 1.2^{\varphi}$ and $r = 1.15^{\varphi}$ separately gives, for B7, $\varphi \approx 7.3$, 6.2 and 7.0, and for B1, 0, 0.5 and 0.5 (our arithmetic, in the example below). The released models therefore follow the spirit of Eq. 3, all three dimensions growing together at roughly the ratios found on B0, rather than the formula with $\varphi = 1, \dots, 7$. The code also turns real coefficients into layer counts and channel counts: it rounds each stage’s number of layers up ($\lceil d \hat{L}_i \rceil$), and rounds each width $w \hat{C}_i$ to the nearest multiple of 8, adding 8 if that would lose more than 10%. Width scaling applies to the stem and the 1280-channel head as well; depth scaling only to the MBConv stages.

Training

The ICML paper trains on ImageNet with settings similar to MnasNet’s (§5.2): RMSProp with decay 0.9 and momentum 0.9, batch normalization momentum 0.99, weight decay $10^{-5}$, and an initial learning rate of 0.256 decayed by a factor of 0.97 every 2.4 epochs. It uses swish, a fixed AutoAugment policy (see data augmentation), stochastic depth “with drop connect ratio 0.3”, and dropout before the classifier increased linearly from 0.2 for B0 to 0.5 for B7. The loss is the cross-entropy of a softmax over the 1000 classes. The paper gives neither the number of epochs nor the batch size.

The later sources differ in details:

• The arXiv revisions give stochastic depth as a survival probability of 0.8 instead of a drop ratio of 0.3 (from v3 on); v4 and v5 also name the activation SiLU (Swish-1) and add the evaluation protocol: 25K images randomly held out from the training set serve as a minival set for early stopping, and the early-stopped checkpoint is evaluated on the original validation set.
• The official code (commit a2d5e42) uses stepwise dropout (0.2, 0.2, 0.3, 0.3, 0.4, 0.4, 0.5, 0.5 for B0 to B7) rather than a linear ramp, and stochastic depth whose drop rate grows linearly with the block index, from 0 for the first block towards $1 - 0.8 = 0.2$ for the last, in the manner described in the dropout article. Its defaults also include label smoothing of 0.1 and an exponential moving average of the weights with decay 0.9999, which the ICML text does not mention, and about 350 epochs at a batch size of 2,048 with a base learning rate of 0.016 per 256 images, which scales to the paper’s 0.256 at a batch of 4,096 (our arithmetic). AutoAugment or RandAugment is applied only when selected by a flag.

The training recipe matters for the reported numbers. The official README reports that the released checkpoints trained with “baseline ResNet preprocessing” reach similar results to the ICML paper, while those trained with AutoAugment reach higher accuracy, 77.1% for B0 and 84.3% for B7 (as of the pinned commit). These are the numbers of arXiv v5’s Table 2.

Inference

An EfficientNet classifies one image per forward pass at the resolution it was trained for, from 224 pixels for B0 to 600 for B7. The paper’s ImageNet numbers are single-crop, single-model (Figure 1 caption). Its latency comparison (Table 4) ran with batch size 1 on one core of an Intel Xeon E5-2690 CPU, averaged over 20 runs: B1 took 0.098 s against 0.554 s for ResNet-152 at similar accuracy (78.8% against 77.8%), and B7 3.1 s against 19.0 s for GPipe, speed-ups of 5.7 and 6.1 times. On accelerators the picture differs; see Limitations.

Performance

ImageNet. Top-1 and top-5 accuracy on ImageNet, single crop and single model, in the two versions of the paper’s Table 2. Parameters and FLOPS (multiply-adds) are the same in both. arXiv v5 states that the numbers are on the validation set after early stopping on a held-out minival set; the ICML version does not state the split.

Model	ICML 2019: top-1 / top-5	arXiv v5: top-1 / top-5	Parameters	FLOPS
EfficientNet-B0	76.3% / 93.2%	77.1% / 93.3%	5.3M	0.39B
EfficientNet-B1	78.8% / 94.4%	79.1% / 94.4%	7.8M	0.70B
EfficientNet-B2	79.8% / 94.9%	80.1% / 94.9%	9.2M	1.0B
EfficientNet-B3	81.1% / 95.5%	81.6% / 95.7%	12M	1.8B
EfficientNet-B4	82.6% / 96.3%	82.9% / 96.4%	19M	4.2B
EfficientNet-B5	83.3% / 96.7%	83.6% / 96.7%	30M	9.9B
EfficientNet-B6	84.0% / 96.9%	84.0% / 96.8%	43M	19B
EfficientNet-B7	84.4% / 97.1%	84.3% / 97.0%	66M	37B

The comparison rows of the same table set each EfficientNet against earlier networks of similar accuracy: ResNet-50 at 76.0% with 26M parameters and 4.1B FLOPS; ResNet-152 at 77.8% with 60M and 11B; Inception-v3 at 78.8% with 24M and 5.7B; ResNeXt-101 at 80.9% with 84M and 32B; SENet at 82.7% with 146M and 42B; AmoebaNet-C at 83.5% with 155M and 41B; and GPipe at 84.3% with 557M parameters. On this basis the authors claimed an order of magnitude fewer parameters and FLOPS at equal accuracy, and that B7 was 8.4 times smaller than GPipe while more accurate in the ICML version (84.4% against 84.3%); in v5 the two are equal at 84.3%, although v5’s Table 4 still gives B7 as 84.4%.

Scaling other networks. To show that compound scaling is not specific to their baseline, the authors scaled existing networks to similar budgets in different ways (Table 3, ImageNet top-1):

Network	Baseline	Depth only	Width only	Resolution only	Compound
MobileNetV1	70.6% (0.6B)	—	74.2% ($w = 2$, 2.2B)	72.7% ($r = 2$, 2.2B)	75.6% ($d = 1.4, w = 1.2, r = 1.3$; 2.3B)
MobileNetV2	72.0% (0.3B)	76.8% ($d = 4$, 1.2B)	76.4% ($w = 2$, 1.1B)	74.8% ($r = 2$, 1.2B)	77.4% (1.3B)
ResNet-50	76.0% (4.1B)	78.1% ($d = 4$, 16.2B)	77.7% ($w = 2$, 14.7B)	77.5% ($r = 2$, 16.4B)	78.8% (16.7B)

Compound scaling was the most accurate in each row, by 0.6 to 1.4 points over the best single dimension, at a slightly higher cost in each case.

Scaling B0 itself. In Figure 8, compound scaling of B0 beat width, depth and resolution scaling by up to 2.5 points at equal FLOPS. Figure 7 shows class activation maps of models scaled to about four times B0’s FLOPS in each way; in the authors’ reading, the compound-scaled model attends to more relevant regions with more object detail. The arXiv versions from v3 on add the numbers of those models (Table 7 in v5): depth $d = 4$ reached 79.0% at 1.8B FLOPS, width $w = 2$ 78.9% at 1.8B, resolution $r = 2$ 79.1% at 1.9B, and compound scaling with $d = 1.4$, $w = 1.2$, $r = 1.3$ 81.1% at 1.8B, from a baseline of 77.3% (v3’s figure for B0, kept in v5 although v5’s Table 2 gives 77.1%).

Transfer. Fine-tuned from ImageNet checkpoints with the settings of Kornblith et al. (2019) and GPipe, EfficientNets were compared on eight datasets (Table 5; see transfer learning). Against publicly available models they were more accurate on six of the eight datasets, tied on Flowers and slightly lower on FGVC Aircraft, with 4.7 times fewer parameters on average; for example B0 reached 88.1% on CIFAR-100 with 4M parameters against 87.5% for NASNet-A with 85M. Against the best reported results, B7 reached 91.7% on CIFAR-100 (GPipe 91.3%), 98.8% on Flowers and 84.3% on Birdsnap, and tied on FGVC Aircraft (92.9%) and Food-101 (93.0%); these five are the authors’ “5 out of 8”, while CIFAR-10, Stanford Cars and Oxford-IIIT Pets stayed slightly below the previous best.

Strengths

• One rule for many budgets. A single baseline and three constants give a graded series from 0.39B to 37B multiply-adds, without a new search for each size.
• Parameter and multiply-add efficiency. At the time, each EfficientNet matched the ImageNet accuracy of networks with several times more parameters and FLOPS (Table 2), and V2’s Table 1 still shows B6 with fewer parameters and FLOPs than ResNet-RS-420 and NFNet-F1 at similar accuracy.
• A general observation. Balanced scaling helped MobileNets and ResNet-50 too (Table 3), and EfficientDet scaled detectors the same way.
• Transferable features. The ImageNet-trained models transferred well to fine-grained and small datasets (Table 5) and served as backbones for detection and segmentation.

Limitations

Few multiply-adds, slow on accelerators. The efficiency is measured in parameters and multiply-adds, and the network spends both sparingly by using depthwise convolutions on large, expanded tensors and very large input images. Both are expensive in memory traffic. Bello et al. (2021) point out that inverted bottlenecks with depthwise convolutions on large activations have a low ratio of computation to memory access compared with ResNet bottlenecks of dense convolutions on smaller activations; the depthwise separable convolution article estimates this ratio. In their comparison (Table 2, with training setups closely matched, step times on TPUv3 and V100 GPUs), ResNet-RS models matched EfficientNets’ accuracy while being 1.7 to 2.7 times faster on TPUs and 2.1 to 3.3 times on GPUs. One ResNet-RS model with 1.8 times the FLOPs of B6 was 2.7 times faster on a TPUv3, and one with 3.8 times B6’s parameters used 2.3 times less memory, because training memory is dominated by activations rather than parameters. They add that such measurements depend on the software and hardware stack.

Large images during training. The same paper that introduced EfficientNetV2 lists EfficientNet’s training bottlenecks (Tan & Le, 2021, §3.2): very large images use much memory, forcing small batches and slow training; depthwise convolutions are slow in early layers; and scaling every stage equally is sub-optimal. Training B6 at 512 pixels on TPUv3 gave 84.3% at 42 images per second per core with a batch of 32, and ran out of memory at 128; at 380 pixels it gave 84.6% at 76 images per second, and 93 with a batch of 128 (their Table 2). The example below counts how activations grow with the scaled models.

A scaling rule from one small model. The constants were found by a grid search at $\varphi = 1$ on B0, whose resolution was fixed by its own search. Bello et al. argue that rules extrapolated from small models can fail at large scale, that scaling resolution gives diminishing returns, and that resolution should grow more slowly than the compound rule prescribes: re-scaled EfficientNets with lower resolutions and unchanged width and depth (EfficientNet-RS) improved the speed-accuracy trade-off (their Figure 5, appendix).

Recipe-dependent numbers. The two versions of the paper differ by up to 0.8 points for the same architectures (B0: 76.3% and 77.1%), and later papers quote different EfficientNet accuracies again (B7 at 84.7% in V2’s Table 7, the value that RandAugment reported for B7 and the official README gives for its RandAugment checkpoint; V2 does not say which training it used), so comparisons with EfficientNet depend on which training was used.

Historical Importance

EfficientNet appeared in May 2019 and made model scaling a design question of its own: a fixed baseline and a rule instead of an ad hoc deeper, wider or higher-resolution variant. It shares authors and a search space with MnasNet, and carried mobile-style blocks (the inverted residual with squeeze-and-excitation, swish) into networks of tens of billions of multiply-adds. Its scaling idea was taken into detection with EfficientDet, and in the following two years papers on new architectures, among them ResNet-RS, NFNets and EfficientNetV2, measured themselves against EfficientNets. Its weak point, speed on accelerators, became the motivation of that line of work: ResNet-RS and EfficientNetV2 reported large training speed-ups over EfficientNets, in part by scaling resolution more slowly, and the ResNet-RS authors note that LambdaResNets and NFNets use similar scaling strategies. The family also shows a change in what efficiency means, from FLOPS (EfficientNet, 2019) to training time on accelerators (EfficientNetV2, 2021), the change that MobileNet made from multiply-adds to measured latency.

Variants

EfficientNetV2 (2021)

EfficientNetV2 (Tan & Le, ICML 2021) keeps EfficientNet’s training setup and reuses its searched channel sizes, but redesigns the network for training speed and parameter efficiency.

Fused-MBConv. It replaces the $1 \times 1$ expansion and the $3 \times 3$ depthwise convolution of MBConv with one regular $3 \times 3$ convolution, followed by squeeze-and-excitation and the $1 \times 1$ projection (the block comes from Gupta & Tan, 2019). It has more parameters and FLOPs but uses accelerators better. Replacing MBConv by Fused-MBConv in stages 2 to 4 of B4 raised training throughput from 262 to 362 images per second per TPU core and accuracy from 82.8% to 83.1%, at 20.0M instead of 19.3M parameters and 7.5B instead of 4.5B FLOPs; replacing it everywhere gave 132M parameters, 34.4B FLOPs and 81.7% (their Table 3). The text describes this last variant as slower to train; in the table its TPU throughput is lower than the original’s (254 images per second per core), but its V100 throughput is higher (206 against 155 images per second).

Training-aware search. The search space chooses per stage between MBConv and Fused-MBConv, the number of layers, kernels of $3 \times 3$ or $5 \times 5$ and expansion ratios of 1, 4 or 6, keeping EfficientNet’s channel sizes. Up to 1,000 models of about B4’s size were sampled and each trained for about 10 epochs at reduced resolution, with a reward $A \cdot S^{w} \cdot P^{v}$ combining accuracy $A$, normalized training step time $S$ and parameter count $P$, with $w = -0.07$ and $v = -0.05$. The resulting EfficientNetV2-S (their Table 4) uses Fused-MBConv in its first three block stages and MBConv with squeeze-and-excitation (ratio 0.25) in the last three, smaller expansion ratios (1 or 4 in the early stages), only $3 \times 3$ kernels with more layers to compensate, and no final stride-1 stage. It was scaled to M and L with compound scaling modified in two ways: the maximum inference resolution is capped at 480, and extra layers go to the later stages.

Progressive learning with adaptive regularization. Training starts with small images and grows them, as in earlier progressive resizing, but also strengthens regularization with the image size, on the argument that a smaller image gives the same network less effective capacity and so needs less regularization. In their test (Table 5, mean of three runs), RandAugment magnitude 5 was best at 128 pixels (78.3%) and magnitude 15 at 300 pixels (83.2%). Training is divided into four stages of about 87 epochs; image size, RandAugment magnitude, mixup strength and dropout are interpolated linearly from a minimum to a maximum (for V2-S, 128 to 300 pixels, magnitude 5 to 15, dropout 0.1 to 0.3; Table 6), and the final training size is about 20% below the inference size without fine-tuning afterwards. Compared with progressive or random resizing at fixed regularization, the adaptive version raised accuracy from 84.3% to 85.1% and from 83.5% to 84.2% (Table 13, +0.8 and +0.7; the text gives an improvement of 0.7%). The rest of the recipe follows EfficientNet, with 350 epochs at a batch of 4,096, learning-rate warmup, a weight moving average and stochastic depth.

Results (Table 7, ImageNet top-1; training time normalized to 32 TPU cores, inference on a V100 GPU in FP16 with batch 16 using the timm codebase):

Model	Pre-training	Top-1	Parameters	FLOPs	Inference	Training
EfficientNet-B7 (as listed in V2)	ImageNet	84.7%	66M	38B	170 ms	139 h
EfficientNetV2-S	ImageNet	83.9%	22M	8.8B	24 ms	7.1 h
EfficientNetV2-M	ImageNet	85.1%	54M	24B	57 ms	13 h
EfficientNetV2-L	ImageNet	85.7%	120M	53B	98 ms	24 h
EfficientNetV2-L	ImageNet-21k	86.8%	120M	53B	98 ms	26 h
EfficientNetV2-XL	ImageNet-21k	87.3%	208M	94B	—	45 h
ViT-L/16	ImageNet-21k	85.3%	304M	192B	195 ms	172 h

V2-M matched B7’s accuracy while training about 11 times faster on the same hardware, and V2-L (21k) beat ViT-L/16 (21k) by 1.5 points with 2.5 times fewer parameters. ImageNet-21k pre-training (about 13M images, 21,841 classes) took 60 or 30 epochs with cosine decay, followed by 15 epochs of fine-tuning, and finished within two days on 32 TPUv3 cores. To separate architecture from recipe, Table 10 trains B7 with the same progressive learning: it improved to 85.0% in 54 hours, and V2-M still reached 85.1% with 17% fewer parameters, 37% fewer FLOPs, 4.1 times faster training and 3.1 times faster inference. The paper lists V2-S at 22M parameters in Table 7 but 24M in Figure 1 and Tables 8 and 11; TorchVision’s implementation has 21.5M.

EfficientNet-L2 and Noisy Student

Xie et al. (CVPR 2020) used EfficientNets for self-training at scale. Of about 300M unlabeled JFT images, those to which an ImageNet-trained EfficientNet-B0 assigns a label with confidence above 0.3 are kept, up to 130K per class (with duplication for rarer classes), giving 130M images of which 81M are unique. A teacher, at first an EfficientNet-B7 trained on ImageNet, labels them. A larger student is trained on labeled and pseudo-labeled images together while being noised with RandAugment (two operations at magnitude 27), dropout 0.5 and stochastic depth (survival 0.8 at the last layer), whereas the teacher labels the images without noise. The student then becomes the teacher, three times over. For this they scaled B7 further to EfficientNet-L2, wider and deeper but at a lower resolution, with 480M parameters and about five times B7’s training time. The official code lists width 4.3 and depth 5.3; its resolution entry, 800, matches the 800 × 800 input that the paper mentions for its adversarial-robustness tests, and is presumably the size after the final fine-tuning, since the paper trains L2 at a lower resolution than B7 (our inference). Training took six days on a 2,048-core TPUv3 pod, with a short FixRes-style fine-tuning at higher resolution. NoisyStudent with EfficientNet-L2 reached 88.4% top-1 and 98.7% top-5 on the ImageNet validation set, against 85.5% for L2 trained on labels alone and 86.4% for the previous best, a ResNeXt-101 pre-trained on 3.5 billion Instagram images (their Table 2); the authors attribute 0.5 points of the 3.4-point gain over the best previously reported EfficientNet accuracy (B7 at 85.0% in their Table 2) to the larger model and 2.9 to NoisyStudent. On robustness test sets it raised ImageNet-A top-1 accuracy to 83.7% and cut the ImageNet-C mean corruption error to 28.3, against previous best results of 61.0% and 45.7.

EfficientDet (2020)

EfficientDet (Tan, Pang & Le, CVPR 2020) applies the idea to one-stage object detection. It uses EfficientNet backbones with a weighted bi-directional feature pyramid, BiFPN, and scales all parts with one coefficient $\varphi$ by heuristic rules rather than a grid search, which the authors judged too expensive for a detector’s many dimensions: B0 to B6 as backbones, BiFPN width $64 \cdot 1.35^{\varphi}$ and depth $3 + \varphi$, prediction-head depth $3 + \lfloor \varphi / 3 \rfloor$ and input size $512 + 128\varphi$ (their Eqs. 1–3; the configuration table, Table 1, departs from them at the top: input sizes of 1280 for D5 and 1536 for D7, BiFPN depths of 7, 8 and 8 for D5 to D7, and D7 is D6 with a higher resolution). EfficientDet-D7 reached 52.2 AP on COCO test-dev with 52M parameters and 325B FLOPs, which the paper defines as multiply-adds.

Other variants

The official repository (at the pinned commit of December 2020) also released EfficientNet-B8 (width 2.2, depth 3.6, resolution 672), checkpoints trained with RandAugment, AdvProp and NoisyStudent, and EfficientNet-EdgeTPU, EfficientNet-CondConv and EfficientNet-Lite variants for particular hardware.

Implementations

The official implementation is the TensorFlow code in the TPU repository, which defines the block strings, the coefficient table and the rounding rules used above; EfficientNetV2’s code is in the google/automl repository, according to the V2 paper. TorchVision 0.29.0 provides efficientnet_b0 to efficientnet_b7, whose weights are ported from the original checkpoints by way of two third-party PyTorch ports (timm for B0 to B4, EfficientNet-PyTorch for B5 to B7), and efficientnet_v2_s, efficientnet_v2_m and efficientnet_v2_l, the first two trained with TorchVision’s own recipe and the last ported from the official V2 code.

Building B0 and scaling it. The example (PyTorch 2.14.1, one thread) implements MBConv with squeeze-and-excitation as in the official code, builds B0 from Table 1 and counts parameters and multiply-adds at $224 \times 224$ with PyTorch’s FLOP counter, which counts convolutions and matrix products (including the small ones of squeeze-and-excitation) as two FLOPs per multiply-add and ignores element-wise operations. It then scales B0 in two ways, both using the official code’s rounding (layers rounded up, channels to a multiple of 8): by Eq. 3 with integer $\varphi = 1$ to 7, and with the coefficients and resolutions of the official code, next to Table 2. The last column counts the values output by the batch-normalization and activation layers for one image, a rough proxy for the activation memory of training. The input is a synthetic image of a bright disk; the counts do not depend on its content.

import math
import torch
import torch.nn as nn
from torch.utils.flop_counter import FlopCounterMode

torch.manual_seed(0)
torch.set_num_threads(1)

def conv_bn(c_in, c_out, k, stride=1, groups=1, act=True):
    layers = [nn.Conv2d(c_in, c_out, k, stride, k // 2, groups=groups, bias=False),
              nn.BatchNorm2d(c_out)]
    return nn.Sequential(*layers, nn.SiLU()) if act else nn.Sequential(*layers)

class SE(nn.Module):
    """Squeeze-and-excitation on the expanded tensor; inner width from the block's input."""
    def __init__(self, c, c_inner):
        super().__init__()
        self.reduce = nn.Conv2d(c, c_inner, 1)       # with bias, as in the official code
        self.expand = nn.Conv2d(c_inner, c, 1)
    def forward(self, x):
        s = x.mean(dim=(2, 3), keepdim=True)          # squeeze: global average pool
        return x * torch.sigmoid(self.expand(nn.functional.silu(self.reduce(s))))

class MBConv(nn.Module):
    """MBConv-e, k x k: 1x1 expand (skipped if e = 1), k x k depthwise, SE, linear 1x1 project."""
    def __init__(self, c_in, c_out, k, stride, e, se_ratio=0.25):
        super().__init__()
        mid = c_in * e
        layers = [conv_bn(c_in, mid, 1)] if e != 1 else []
        layers += [conv_bn(mid, mid, k, stride, groups=mid),
                   SE(mid, max(1, int(c_in * se_ratio))),
                   conv_bn(mid, c_out, 1, act=False)]       # no activation after projection
        self.body = nn.Sequential(*layers)
        self.skip = stride == 1 and c_in == c_out
    def forward(self, x):                                   # (stochastic depth omitted: eval only)
        return x + self.body(x) if self.skip else self.body(x)

# Table 1, stages 2-8: (expansion e, kernel k, stride of first layer, output channels, layers)
B0 = [(1, 3, 1, 16, 1), (6, 3, 2, 24, 2), (6, 5, 2, 40, 2), (6, 3, 2, 80, 3),
      (6, 5, 1, 112, 3), (6, 5, 2, 192, 4), (6, 3, 1, 320, 1)]

def round_filters(c, w, divisor=8):        # official code: nearest multiple of 8, at most 10% down
    if w == 1.0:
        return c
    new = max(divisor, int(c * w + divisor / 2) // divisor * divisor)
    return new + divisor if new < 0.9 * c * w else new

def round_repeats(n, d):                   # official code: round the layer count up
    return int(math.ceil(n * d))

def efficientnet(w=1.0, d=1.0, classes=1000):
    c = round_filters(32, w)
    layers = [conv_bn(3, c, 3, 2)]                          # stage 1: Conv3x3, stride 2
    for e, k, s, c_out, n in B0:
        c_out = round_filters(c_out, w)
        for i in range(round_repeats(n, d)):
            layers.append(MBConv(c, c_out, k, s if i == 0 else 1, e))
            c = c_out
    head = round_filters(1280, w)                           # stage 9: Conv1x1 & Pooling & FC
    layers += [conv_bn(c, head, 1), nn.AdaptiveAvgPool2d(1), nn.Flatten(),
               nn.Dropout(0.2), nn.Linear(head, classes)]
    return nn.Sequential(*layers).eval()

def image(size):                           # synthetic RGB test image: a bright disk
    yy, xx = torch.meshgrid(torch.arange(size), torch.arange(size), indexing="ij")
    disk = ((xx - size / 2) ** 2 + (yy - size / 2) ** 2 < (size / 4) ** 2).float()
    return disk.expand(1, 3, size, size)

def count(model, size):
    params = sum(p.numel() for p in model.parameters())
    acts = []                              # values stored after every BN/activation, per image
    hooks = [m.register_forward_hook(lambda m, i, o: acts.append(o.numel()))
             for m in model.modules() if isinstance(m, (nn.BatchNorm2d, nn.SiLU))]
    with FlopCounterMode(display=False) as fc, torch.no_grad():
        model(image(size))
    for h in hooks:
        h.remove()
    return params, fc.get_total_flops() / 2, sum(acts)   # PyTorch counts a multiply-add as 2 FLOPs

p0, m0, a0 = count(efficientnet(), 224)
print(f"B0 at 224: {p0:,} parameters, {m0 / 1e9:.3f} G multiply-adds "
      f"(Table 2: 5.3M, 0.39B FLOPS)")
blocks = efficientnet()[1:-5]
print("blocks per stage:", [n for *_, n in B0], "=", len(blocks), "MBConv layers")

alpha, beta, gamma = 1.2, 1.1, 1.15
growth = alpha * beta ** 2 * gamma ** 2
print(f"alpha * beta^2 * gamma^2 = {growth:.4f}")
table2 = {1: ("7.8M", "0.70B"), 2: ("9.2M", "1.0B"), 3: ("12M", "1.8B"), 4: ("19M", "4.2B"),
          5: ("30M", "9.9B"), 6: ("43M", "19B"), 7: ("66M", "37B")}
official = {1: (1.0, 1.1, 240), 2: (1.1, 1.2, 260), 3: (1.2, 1.4, 300), 4: (1.4, 1.8, 380),
            5: (1.6, 2.2, 456), 6: (1.8, 2.6, 528), 7: (2.0, 3.1, 600)}   # (w, d, resolution)

print("\nEquation 3 with integer phi (d = 1.2^phi, w = 1.1^phi, r = 224 * 1.15^phi):")
print("phi     d     w    res  params   mult-adds  ratio to B0  1.92^phi  2^phi")
for phi in range(1, 8):
    d, w, r = alpha ** phi, beta ** phi, round(224 * gamma ** phi)
    p, m, _ = count(efficientnet(w, d), r)
    print(f"{phi:3d}  {d:4.2f}  {w:4.2f}  {r:4d}  {p / 1e6:5.1f}M  {m / 1e9:6.2f}G  "
          f"{m / m0:9.1f}  {growth ** phi:9.1f}  {2 ** phi:5d}")
print(f"mean growth per step of phi: {(m / m0) ** (1 / 7):.2f}x")

print("\nCoefficients of the official code (efficientnet_params) vs Table 2:")
print("model   w    d   res  phi implied by w, d, r  parameters (Table 2)  mult-adds (Table 2)"
      "  ratio  activations")
print(f"B0     1.0  1.0  224    0.0   0.0   0.0     {p0:>10,} (5.3M)  "
      f"{m0 / 1e9:6.3f}G (0.39B)    1.0  {a0 / 1e6:6.1f}M")
for b, (w, d, r) in official.items():
    p, m, a = count(efficientnet(w, d), r)
    tp, tf = table2[b]
    implied = [math.log(v) / math.log(c) for v, c in ((w, beta), (d, alpha), (r / 224, gamma))]
    print(f"B{b}     {w:.1f}  {d:.1f}  {r}   " + "  ".join(f"{x:4.1f}" for x in implied) +
          f"     {p:>10,} ({tp})  {m / 1e9:6.3f}G ({tf})  {m / m0:5.1f}  {a / 1e6:6.1f}M")

Output:

B0 at 224: 5,288,548 parameters, 0.386 G multiply-adds (Table 2: 5.3M, 0.39B FLOPS)
blocks per stage: [1, 2, 2, 3, 3, 4, 1] = 16 MBConv layers
alpha * beta^2 * gamma^2 = 1.9203

Equation 3 with integer phi (d = 1.2^phi, w = 1.1^phi, r = 224 * 1.15^phi):
phi     d     w    res  params   mult-adds  ratio to B0  1.92^phi  2^phi
  1  1.20  1.10   258    9.1M    0.99G        2.6        1.9      2
  2  1.44  1.21   296   12.3M    1.81G        4.7        3.7      4
  3  1.73  1.33   341   16.4M    3.25G        8.4        7.1      8
  4  2.07  1.46   392   25.9M    6.32G       16.4       13.6     16
  5  2.49  1.61   451   33.7M   11.37G       29.5       26.1     32
  6  2.99  1.77   518   45.5M   20.09G       52.1       50.1     64
  7  3.58  1.95   596   68.7M   39.66G      102.8       96.3    128
mean growth per step of phi: 1.94x

Coefficients of the official code (efficientnet_params) vs Table 2:
model   w    d   res  phi implied by w, d, r  parameters (Table 2)  mult-adds (Table 2)  ratio  activations
B0     1.0  1.0  224    0.0   0.0   0.0      5,288,548 (5.3M)   0.386G (0.39B)    1.0    12.9M
B1     1.0  1.1  240    0.0   0.5   0.5      7,794,184 (7.8M)   0.687G (0.70B)    1.8    20.6M
B2     1.1  1.2  260    1.0   1.0   1.1      9,109,994 (9.2M)   0.994G (1.0B)    2.6    26.2M
B3     1.2  1.4  300    1.9   1.8   2.1     12,233,232 (12M)   1.827G (1.8B)    4.7    45.2M
B4     1.4  1.8  380    3.5   3.2   3.8     19,341,616 (19M)   4.394G (4.2B)   11.4    94.1M
B5     1.6  2.2  456    4.9   4.3   5.1     30,389,784 (30M)  10.266G (9.9B)   26.6   187.5M
B6     1.8  2.6  528    6.2   5.2   6.1     43,040,704 (43M)  19.068G (19B)   49.4   317.3M
B7     2.0  3.1  600    7.3   6.2   7.0     66,347,960 (66M)  37.746G (37B)   97.8   549.2M

With the official coefficients, the parameter counts round to Table 2’s values except B2 (V2’s Table 11 lists B2 at 9.1M), and they equal, to the parameter, the counts that TorchVision 0.29.0 records for its ported weights. Its multiply-add counts (the _ops field) agree too, except for B2, which TorchVision evaluates at 288 pixels instead of 260 (the model above gives the same 1.088G at 288). B4 and B5 come out about 4% above Table 2, a gap the paper does not explain beyond its rounding footnote. Integer $\varphi$ in Eq. 3 comes close to the doubling the constraint aims at and ends near B7 at $\varphi = 7$. In our reading, the ratio starts above $2^{\varphi}$ because rounding layer counts up adds a whole layer to every short stage, and falls below it at large $\varphi$, where the product of the constants, 1.92 rather than 2, compounds. From B0 to B7 the activations grow about 43 times and the parameters 12.5 times: the cost of high resolution that Bello et al. and EfficientNetV2 point to falls on memory more than on the parameter count.