MobileNet

MobileNet is a family of convolutional networks designed at Google to run on phones and other devices with little compute, memory and power. Four generations share the name. MobileNet v1 (Howard et al., 2017) builds almost every layer from depthwise separable convolutions and adds two global knobs, a width multiplier and a resolution multiplier, that trade accuracy for cost. MobileNetV2 (Sandler et al., 2018) wraps the depthwise layer in an inverted residual block with a linear bottleneck. MobileNetV3 (Howard et al., 2019) searches the block arrangement for measured phone latency and adds squeeze-and-excitation and a cheap “hard” swish activation. MobileNetV4 (Qin et al., 2024) generalizes the block again and adds an attention layer for mobile accelerators. All four are trained for ImageNet classification and used mainly as backbones for object detection and semantic segmentation on the device. The 2017 publication itself, with its experiments and the misprints in its tables, is the subject of the MobileNets paper article; this article covers the family as it is understood and used today.

Problem Addressed

By 2017, accurate ImageNet classifiers such as VGG-16 and the ResNets cost billions of multiply-adds per image and tens of millions of parameters (138 million for VGG-16), while applications such as robotics and augmented reality, as Howard et al. (2017) put it, need recognition in a timely fashion on computationally limited platforms. Howard et al. (2017) group earlier responses into two kinds: compressing a pretrained large network (product quantization, hashing, pruning, factorizations) and training a small network directly, a group in which they also place distillation. MobileNet is of the second kind, and its stated aim differs from that of many small-network papers, which in the authors’ words focus only on size: MobileNets optimize primarily for latency and also yield small networks. They also wanted one architecture that application developers could shrink to their own budget without designing a new network, which is the role of the two multipliers.

Architecture

All four generations are stacks of stages that halve the resolution and widen the channels, from a $3 \times 224 \times 224$ image (PyTorch’s $(N, C, H, W)$ layout) to $7 \times 7$ maps, global average pooling and a classifier. What changes is the block. Costs below are those each paper prints: MobileNets v1 to V3 count multiply-adds (Mult-Adds or MAdds), and MobileNetV4 counts MACs, multiply-accumulate operations, which is the same unit.

MobileNet v1 (2017)

The network in Table 1 of Howard et al. (2017) is a standard $3 \times 3$ convolution with 32 filters and stride 2, followed by 13 depthwise separable blocks, an average pool and a 1000-way fully connected layer. Counting depthwise and pointwise layers separately, it has 28 layers. Each block is a $3 \times 3$ depthwise convolution and a $1 \times 1$ pointwise convolution, each followed by batch normalization and ReLU. Downsampling is done by stride-2 depthwise convolutions (and the first layer); there is no pooling inside the network. The widths double at each downsampling, 32, 64, 128, 256, 512 and 1024, and the five identical 512-channel blocks at $14 \times 14$ account for 46% of the multiply-adds (5 × 52.3M of 569M, from the paper’s Tables 3 and 6; our arithmetic).

The network as built, in the official TF-Slim code and in the paper’s own cost counts, has stride 1 in the last depthwise layer, where Table 1 prints 2, and uses ReLU6, $\min(\max(x, 0), 6)$, where the text says ReLU; the paper article discusses both.

The factorization, its cost ratio of $1/C_\text{out} + 1/k^2$ and the paper’s single-layer example are explained in the depthwise separable convolution article. That article also gives the paper’s Table 2, which puts about 95% of the network’s multiply-adds in the $1 \times 1$ convolutions (one entry of that table is misprinted; see the paper article).

Width and Resolution Multipliers

The width multiplier $\alpha \in (0, 1]$ thins every layer: a layer with $C_\text{in}$ inputs and $C_\text{out}$ outputs gets $\alpha C_\text{in}$ and $\alpha C_\text{out}$. The resolution multiplier $\rho$ scales the input image, and through it every feature map; in practice it is set implicitly by choosing an input of 224, 192, 160 or 128 pixels. For a separable layer with a $k \times k$ kernel and an $H \times W$ output, the paper’s Eq. (7), in this article’s notation, is

$$k^2 \cdot \alpha C_\text{in} \cdot \rho H \cdot \rho W \;+\; \alpha C_\text{in} \cdot \alpha C_\text{out} \cdot \rho H \cdot \rho W .$$

The pointwise term, which dominates, scales with $\alpha^2$; the depthwise term only with $\alpha$, so the total falls “roughly” by $\alpha^2$, as the authors put it, and a little less in practice: at $\alpha = 0.75$ their Table 6 gives 325 million against 569 million Mult-Adds, a ratio of 0.57 against $\alpha^2 = 0.5625$ (our arithmetic). Both terms scale with $\rho^2$. The parameters scale with $\alpha^2$ too but do not depend on $\rho$. A model with a different $\alpha$ is a new architecture and is trained from scratch. The full model, 1.0 MobileNet-224, reached 70.6% ImageNet accuracy with 569M multiply-adds and 4.2M parameters (Howard et al., Table 6; evaluation protocol not stated); the paper’s tables for the other widths and resolutions, and its comparison of thinner against shallower networks, are in the paper article. MobileNetV2 and V3 kept both multipliers.

MobileNetV2 (2018): Inverted Residuals and Linear Bottlenecks

MobileNetV2’s block, which Sandler et al. call a bottleneck residual block and which is also known as the inverted residual or inverted bottleneck block (MBConv in EfficientNet), has three layers. A $1 \times 1$ convolution expands $d'$ input channels by an expansion factor $t$; a $3 \times 3$ depthwise convolution filters the $t d'$ expanded channels, with stride 1 or 2; and a $1 \times 1$ convolution projects them to $d''$ output channels. Batch normalization follows every convolution; the first two are followed by ReLU6, and the projection by no nonlinearity: it is a linear bottleneck. When the stride is 1 and $d' = d''$, a shortcut adds the block’s input to its output. For an $h \times w$ input the block costs

$$h \cdot w \cdot d' \cdot t \,(d' + k^2 + d'')$$

multiply-adds, one more $1 \times 1$ term than a v1 block, but the inputs and outputs are narrow. The authors use $t = 6$ throughout (except in the first block, which has $t = 1$), having found that values from 5 to 10 give nearly identical results.

The block inverts a ResNet bottleneck. A ResNet bottleneck reduces the channels, convolves, and expands again, and its shortcut joins the wide tensors; MobileNetV2 expands, convolves, and reduces, and its shortcut joins the narrow ones. The motivation for the shortcut is the classical one, gradient flow across layers (residual connection); in the authors’ ablation, shortcuts between bottlenecks trained to higher accuracy than shortcuts between the expanded layers or no shortcut (their Figure 6b).

Why the bottleneck is linear: the authors’ argument. Sandler et al. treat the activations of a layer, over all pixels of all real images, as a “manifold of interest” and assume it lies in a low-dimensional subspace of the activation space. They then make two observations, which they present as informal, with a more formal statement in their supplementary material. First, where the output of $\mathrm{ReLU}(Bx)$ keeps non-zero volume, it is only a linear function of the input, so the nonlinearity contributes nothing there. Second, ReLU can preserve all the information about the input manifold only if that manifold lies in a low-dimensional subspace of the space in which ReLU is applied; applied in a space barely larger than the manifold, it collapses points together. Their Figure 1 shows this: a 2-D spiral embedded by a random matrix into $n$ dimensions, passed through ReLU and mapped back, collapses partly onto itself for $n = 2$ or 3, while for $n = 15$ to 30 the caption describes the transformation as highly non-convex, with no such collapse. The conclusion they draw is to apply ReLU only in the wide expanded layer and keep the narrow bottleneck linear. They support it empirically: with ReLU6 in the bottleneck, ImageNet top-1 accuracy was lower throughout training (their Figure 6a). The text says non-linear bottlenecks hurt “by several percent”; in the figure the two curves end less than one point apart (our reading of the plot). They also note that linear-bottleneck models are strictly less expressive in principle, since a nonlinear model can always operate in its linear regime, so the result is evidence that the nonlinearity destroys information in low-dimensional spaces rather than a proof. The second example below repeats the Figure 1 experiment on features of a synthetic image.

Architecture and memory. The network (Sandler et al., Table 2) is a $3 \times 3$ convolution with 32 filters, 17 bottleneck blocks in seven groups with outputs of 16, 24, 32, 64, 96, 160 and 320 channels, a $1 \times 1$ convolution to 1280 channels, average pooling and the classifier. The text says 19 bottleneck layers; the table’s repeat counts (1, 2, 3, 4, 3, 3, 1) sum to 17, which is also what TorchVision builds. The primary network has 300M multiply-adds and 3.4M parameters, about half of v1’s computation. For widths below 1 the width multiplier is not applied to the last convolution. Because only the narrow tensors cross block boundaries, the largest activation that must be stored at once is much smaller than in v1: 200K against 800K (in Kb, 16-bit activations; their Table 3). The expanded tensor inside a block need not exist all at once either, because the expansion, the per-channel depthwise layer and the projection can be computed in a few slices of channels whose results are summed. The multiply-adds do not change, but replacing one matrix multiplication by several smaller ones hurt runtime through cache misses, and the authors found the trick most useful with 2 to 5 slices.

MobileNetV3 (2019): Search, Squeeze-and-Excitation and Hard Swish

MobileNetV3 keeps the inverted residual block and changes how the network is chosen and what goes inside it (Howard et al., 2019).

Search for latency. The block-level structure comes from platform-aware neural architecture search, which maximizes a reward $\mathrm{ACC}(m) \times [\mathrm{LAT}(m)/\mathrm{TAR}]^w$ that balances accuracy against measured latency relative to a target. For the larger model, with a target latency around 80 ms, the authors found results similar to the earlier MnasNet search and simply reused MnasNet-A1; for the smaller model they searched again with $w = -0.15$ instead of $-0.07$, because accuracy changes much faster with latency in small models. A second method, NetAdapt, then tunes the number of filters layer by layer: at each step it proposes changes that cut latency by at least 1% of the seed model’s, fine-tunes each proposal for 10,000 steps, and keeps the one with the largest ratio of accuracy change to latency change, until the latency target is met. Latency was measured on a single large core of Google Pixel phones with TFLite.

Squeeze-and-excitation. Some blocks contain a squeeze-and-excitation module (Hu et al., 2018), which pools each channel to one number, passes the vector through two small fully connected layers, and rescales the channels by the result. Following MnasNet, it sits after the depthwise convolution, on the expanded tensor, rather than at the end of the block as in SE-ResNet; MobileNetV3 fixes its inner width at a quarter of the expansion channels, which raised accuracy at a modest cost in parameters and no visible latency cost. The final gate is a hard sigmoid.

Hard swish. Swish, $x \cdot \sigma(x)$ (see activation function), improved accuracy as a replacement for ReLU, but the sigmoid is expensive on phones and hard to keep accurate in fixed-point arithmetic. MobileNetV3 replaces it by piecewise-linear versions built from ReLU6:

$$\text{h-sigmoid}(x) = \frac{\mathrm{ReLU6}(x + 3)}{6}, \qquad \text{h-swish}(x) = x \cdot \frac{\mathrm{ReLU6}(x + 3)}{6}.$$

The authors found no discernible accuracy difference from the smooth versions, and list deployment advantages: ReLU6 is implemented everywhere, quantized models avoid differences between sigmoid approximations, and h-swish can be fused into a piecewise function with fewer memory accesses. Because activations shrink as resolution falls, a nonlinearity costs less deeper in the network, and most of swish’s benefit came from the deeper layers, so h-swish is used only in the second half; the earlier blocks use ReLU (the first convolution also uses h-swish, see below). In their ablation on the large model (Table 5), ReLU everywhere gave 74.5% at 50.5 ms, the final placement (h-swish from the 80-channel layers on) 75.2% at 51.4 ms, and an unoptimized h-swish implementation 57.5 ms.

Cheaper first and last layers. Search left the layers at both ends expensive. The final $1 \times 1$ expansion to a high-dimensional feature vector was moved after the average pooling, so it runs on a $1 \times 1$ map instead of $7 \times 7$, which made the projection and depthwise layers of the preceding block unnecessary; together this saved 7 ms (11% of the running time) and 30M multiply-adds with almost no loss of accuracy. The first convolution was cut from 32 to 16 filters, with h-swish, at the same accuracy, saving another 2 ms and 10M multiply-adds.

Large and Small. The result is two models: MobileNetV3-Large, with 15 bottleneck blocks of $3 \times 3$ and $5 \times 5$ depthwise kernels, expansion widths chosen per block by the search (from 16 to 960) and output channels from 16 to 160; and MobileNetV3-Small, with 11 blocks and output channels up to 96, for low-resource uses (Tables 1 and 2). Both end in a 960- or 576-channel $1 \times 1$ convolution, pooling, a 1280-wide $1 \times 1$ layer without batch normalization, and the classifier.

MobileNetV4 (2024): Universal Inverted Bottleneck and Mobile MQA

MobileNetV4 (Qin et al., ECCV 2024) targets a wider range of hardware: mobile CPUs, DSPs, GPUs, the Apple Neural Engine and Google’s EdgeTPU, measured with INT8 models on CPUs, DSPs and EdgeTPUs and 16-bit floats on GPUs and the Apple Neural Engine (supplementary Appendix B).

Universal inverted bottleneck (UIB). The block adds an optional depthwise convolution before the expansion and makes the one between expansion and projection optional. The four combinations give four known or new blocks: the MobileNetV2 inverted bottleneck (middle depthwise only), a ConvNeXt-like block (depthwise before the expansion, cheaper for large kernels), the feed-forward block of a vision transformer (no depthwise, two $1 \times 1$ convolutions), and a new ExtraDW block with both, which the authors describe as a cheap way to add depth and receptive field. The search chooses per block which depthwise layers to include and with which kernel ($3 \times 3$ or $5 \times 5$). The stems use fused inverted bottlenecks, a $k \times k$ convolution followed by a $1 \times 1$ convolution. In the authors’ comparison (Table 1), a search over full UIB blocks reached 83.3% top-1, against 82.3% for a search restricted to inverted bottlenecks at similar cost. In the searched models, ExtraDW blocks tend to start each stage and feed-forward and ConvNeXt-like blocks to end the network (supplementary Appendix D).

Mobile MQA. The hybrid models add attention in the last stages. Multi-query attention shares one key head and one value head across all query heads, which reduces the memory traffic per operation when, as in the low-resolution stages of a vision model at batch size 1, the tokens are few compared to the feature dimension (see self-attention). Mobile MQA also downsamples keys and values with a stride-2 $3 \times 3$ depthwise convolution while keeping full-resolution queries. In their Table 2, three multi-query attention blocks added to MNv4-Conv-L had 39% lower attention latency than three multi-head attention blocks on a Pixel 8 EdgeTPU, at 0.03 points lower accuracy; in Table 3, the key-value downsampling made a hybrid model 23% faster on a phone CPU and 25% on its GPU for 0.06 points of accuracy.

Design rules and search. From correlation studies across devices, the authors favour components that every platform supports well: depthwise and pointwise convolutions, ReLU, batch normalization and simple attention. They report that grouped and multi-path convolutions can be slow despite few MACs, that squeeze-and-excitation, GELU and layer normalization are poorly supported on DSPs, and that squeeze-and-excitation is slow on accelerators, and the published MobileNetV4 specifications (supplementary Tables 11 to 15) contain no squeeze-and-excitation layers. The models (Conv-S, Conv-M, Conv-L, Hybrid-M, Hybrid-L) come from a two-stage search, first over widths with a fixed block and then over the UIB configuration, run separately for each size rather than by scaling one model, with an offline distillation dataset to make the search’s accuracy estimates less noisy.

Training

All MobileNets are trained as ImageNet classifiers with a softmax output and the cross-entropy loss, $\mathcal{L} = -\log p_y$ for the true class $y$; MobileNetV4 adds label smoothing of 0.1. The recipes changed more than the loss:

• v1 was trained in TensorFlow with RMSprop and asynchronous gradient descent, like Inception V3 but with less regularization and augmentation (details in the paper article).
• V2: RMSprop with decay and momentum 0.9, weight decay 0.00004, initial learning rate 0.045 decayed by 0.98 per epoch, 16 asynchronous GPU workers and a batch size of 96.
• V3: synchronous training on a 4×4 TPU Pod, RMSprop with momentum 0.9, learning rate 0.1 with batch size 4096, weight decay $10^{-5}$, an exponential moving average of the weights with decay 0.9999, and “dropout of 0.8” as printed (in TensorFlow code of the time such a value was often a keep probability, so this may mean a drop rate of 0.2; our reading).
• V4 (supplementary Table 10): AdamW with cosine decay, 500 epochs for most models and 9,600 for Conv-S, RandAugment for all models, Mixup and CutMix for Conv-L and Hybrid-L, dropout and stochastic depth.

The recipe matters for comparisons. Qin et al. retrained MobileNets v1 to V3 with modern recipes and obtained 74.0% for v1 (+3.4 points over the original 70.6%), 73.4% for V2 and 75.5% for V3, and used these numbers in their comparisons. They also distilled MobileNetV4 from larger teachers, mixing ImageNet copies with different augmentations and class-balanced JFT-300M images, which raised MNv4-Conv-L from 82.9% to 85.9%; MNv4-Hybrid-L reached 87.0% with this distillation and JFT pretraining.

Inference

At test time a MobileNet is a single forward pass at the chosen resolution, with batch normalization using its stored statistics. The later papers report quantized deployments: MobileNetV3’s Table 4 gives V3-Large at 73.8% top-1 and 44 ms on a Pixel 1 when quantized, against 75.2% and 51 ms in floating point, and MobileNetV4 benchmarks INT8 models on CPUs, DSPs and EdgeTPUs.

Why Multiply-Adds Do Not Predict Latency

Every MobileNet paper makes the point that the count of multiply-adds, the cost unit of v1 and V2, is a poor proxy for speed, and each later version changed what it optimized because of it.

• v1 already argues that operations must also be efficiently implementable; the depthwise separable convolution article covers its argument and why depthwise layers do little arithmetic per byte moved.
• V2 adds memory: the authors note that the inverted design is considerably more memory-efficient, and their Table 4 shows models of similar cost running at very different speeds on one phone core: v1 at 575M multiply-adds in 113 ms, NASNet-A at 564M in 183 ms, and MobileNetV2 1.4 at 585M in 143 ms.
• ShuffleNet V2 (Ma et al., 2018), which defines FLOPs as multiply-adds, names two factors that FLOPs miss, memory access cost and the degree of parallelism, and derives four guidelines: equal input and output widths minimize memory access for a $1 \times 1$ convolution; excessive grouping increases memory access; fragmented multi-branch blocks reduce parallelism; and element-wise operations are not negligible, a class in which they include depthwise convolution because of its high ratio of memory access to FLOPs. By their analysis, MobileNetV2 violates the first and fourth, with its expanded blocks and its depthwise layers and ReLUs on thick feature maps, and on an ARM processor they measured it to be much slower than ShuffleNet v1, Xception and ShuffleNet V2 of comparable FLOPs, especially at small FLOPs.
• V3 therefore searched on measured latency, and its own tables show the gap: MnasNet-small and V3-Small 1.0 both need about 65M multiply-adds but run in 20.3 and 15.8 ms on a Pixel 1 (Table 3), and Table 5 shows the cost of an unoptimized h-swish implementation.
• V4 uses a roofline model: each layer’s time is the larger of its compute time and its memory time,

$$\text{ModelTime} = \sum_i \max\!\left( \frac{\text{MACs}_i}{\text{PeakMACs}},\; \frac{\text{WeightBytes}_i + \text{ActivationBytes}_i}{\text{PeakMemBW}} \right),$$

and hardware is summarized by its ridge point, the ratio of peak MACs to memory bandwidth. Counting MACs alone is the special case of a ridge point of 0. On low-ridge-point hardware such as CPUs, large standard convolutions are expensive and depthwise and fully connected layers cheap; on high-ridge-point accelerators the reverse holds, because those layers are limited by memory bandwidth. MobileNetV4 puts its MAC-heavy layers at the start and its memory-heavy fully connected layers at the end, so that a given device pays for only one of the two (Qin et al., §3).

Performance

ImageNet top-1 accuracy as printed, each model from the paper that introduced it. The v1 and V2 papers do not state the evaluation protocol; V3 does not either; V4 trains on ImageNet-1k and reports validation top-1. Latencies come from different phones, libraries and precisions and should only be compared within a paper.

Model (paper)	Top-1	Cost	Parameters	Latency as printed
1.0 MobileNet-224 (v1, Table 6)	70.6%	569M Mult-Adds	4.2M	—
MobileNetV2 1.0 (V2, Table 4)	72.0%	300M MAdds	3.4M	75 ms, Pixel 1, one large core, TF-Lite
MobileNetV2 1.4 (V2, Table 4)	74.7%	585M MAdds	6.9M	143 ms, same
MobileNetV3-Large 1.0 (V3, Table 3)	75.2%	219M MAdds	5.4M	51 ms, Pixel 1, one large core, TFLite
MobileNetV3-Small 1.0 (V3, Table 3)	67.4%	66M MAdds	2.9M	15.8 ms, same
MNv4-Conv-S (V4, Table 6)	73.8%	0.2G MACs	3.8M	2.4 ms, Pixel 6 CPU, INT8
MNv4-Conv-M (V4, Table 6)	79.9%	1.0G MACs	9.2M	11.4 ms, same
MNv4-Conv-L (V4, Table 6)	82.9%	5.9G MACs	31M	59.9 ms, same
MNv4-Hybrid-L (V4, Table 6)	83.4%	7.2G MACs	35.9M	87.6 ms, same; 3.8 ms on a Pixel 8 EdgeTPU

The same models measure differently across papers: MobileNetV2 1.0 takes 75 ms in the V2 paper and 64 ms on the same phone model in the V3 paper. MobileNetV4’s search targeted 224-pixel inputs for Conv-S, 256 for Conv-M and 384 for Conv-L (supplementary Appendix A).

Detection. On COCO, all with single-stage detectors unless noted:

• v1 was evaluated in SSD and Faster R-CNN, with lower COCO average precision than VGG backbones at a fraction of their cost (Howard et al., 2017, Table 13; see the paper article).
• V2 introduced SSDLite, SSD with every regular convolution in the prediction layers replaced by a depthwise separable one, which cut the head from 14.8M parameters and 1.25B multiply-adds to 2.1M and 0.35B. MobileNetV2 + SSDLite at $320 \times 320$ reached 22.1 mAP on test-dev with 4.3M parameters, 0.8B multiply-adds and 200 ms on a Pixel 1, against 22.2 for v1 + SSDLite (5.1M, 1.3B, 270 ms) and 21.6 for YOLOv2 (50.7M, 17.5B) (Sandler et al., Table 6).
• V3-Large in SSDLite nearly matched V2 (22.0 against 22.1 mAP on the COCO test set) in 137 ms against 162 ms, and 119 ms with the channels between its stride-16 and stride-32 features halved, which the authors justify by the smaller number of COCO classes (Table 6).
• MNv4-Conv-M in RetinaNet with an FPN at 384 pixels reached 32.6 AP on COCO 2017 val in 51.3 ms on a Pixel 6 CPU, about as accurate as MobileNetV2 scaled by 2.0 (32.9 AP) and 23% faster; MNv4-Hybrid-M reached 34.0 AP (Qin et al., Table 7).

Segmentation. Sandler et al. built a mobile DeepLabv3 on the second-to-last feature map of MobileNetV2 (320 instead of 1280 channels). Without the ASPP module, at output stride 16 and single-scale input, it reached 75.32% mIOU on PASCAL VOC 2012 val with 2.11M parameters and 2.75B multiply-adds, against 82.70% for ResNet-101 with ASPP at output stride 8 with multi-scale and left-right flipped test inputs, at 4,870.6B (Table 7; all models pretrained on COCO). MobileNetV3 proposed LR-ASPP (Lite Reduced ASPP), a head that gates the stride-16 features with a large-kernel pooled, sigmoid-activated branch, in the manner of squeeze-and-excitation, applies atrous convolution in the last block, and adds a skip connection from higher-resolution, lower-level features (at stride 8 in their Figure 10). Trained from scratch on Cityscapes fine annotations, MobileNetV3-Large with LR-ASPP reached 72.6% mIOU on the test set at 9.74B multiply-adds for full-resolution $1024 \times 2048$ input, and MobileNetV3-Small 69.4% at 2.90B (Table 8).

Strengths

• One design, many budgets. The width and resolution multipliers (and V3’s Large and Small, V4’s S, M and L) give a graded set of models from tens of millions to several billion multiply-adds.
• Standard operations. Depthwise and $1 \times 1$ convolutions, batch normalization and ReLU-type activations exist in every framework and inference engine; the V2 and V4 papers make support across hardware an explicit goal.
• Quantization-friendly activations (ReLU6, h-swish, h-sigmoid) and low activation memory in the inverted residual design.
• Useful backbones. Every version was evaluated for detection and segmentation in its own paper, and SSDLite and LR-ASPP were designed with them.

Limitations

• Multiply-adds mislead. The memory-bound depthwise and element-wise layers make v1 and V2 slower than their cost suggests, more so on accelerators (see Inference).
• Tuned for one target. MobileNetV3 was searched for Pixel CPU latency; Qin et al. found it fast on CPUs but not on EdgeTPUs, DSPs and GPUs, and report that squeeze-and-excitation, which V3 uses, is poorly supported on DSPs and slow on accelerators.
• Steep losses at the smallest sizes. In v1, accuracy falls slowly with width until $\alpha = 0.25$, where it drops sharply (Howard et al., Table 6; see the paper article).
• Search cost and opacity. V3 and V4 depend on architecture searches whose resulting layer widths have no simple rule; reproducing the search requires the latency measurements and compute the authors had (our assessment; the papers do not give the search cost).
• Recipe-dependent comparisons. The 3.4-point gain from retraining v1 with a modern recipe is larger than many architectural differences in the comparisons above (our observation).

Historical Importance

MobileNet v1 appeared on arXiv in April 2017, between Xception (October 2016) and ShuffleNet (July 2017), which also built networks from depthwise or grouped convolutions. In our reading, its contribution was less the layer, which goes back to Sifre’s 2014 thesis, than a complete, scalable recipe for small networks, evaluated on a range of recognition tasks beyond ImageNet (listed in the paper article).

MobileNetV2’s inverted residual block became the base of later efficient designs. MnasNet added squeeze-and-excitation to it (as described in the V3 paper), MobileNetV3 took both, and EfficientNet’s main block is the same mobile inverted bottleneck (MBConv) with squeeze-and-excitation (Tan & Le, 2019). By 2024, Qin et al. could describe the inverted bottleneck as the standard building block for efficient networks, and they built MobileNetV4’s UIB as its generalization. The family also shows the field’s change of target, from counting operations (v1, 2017), to measured latency on one phone (V3, 2019), to efficiency across heterogeneous hardware (V4, 2024).

Variants

• Sizes. The first three versions were published at several widths, v1 at 0.25 to 1.0, V2 at 0.35 to 1.4 and V3 at 0.35 to 1.25, and later papers use wider ones as baselines (MobileNetV2 2.0 in V4’s Table 6); V3 comes as Large and Small, V4 as Conv-S, Conv-M, Conv-L, Hybrid-M and Hybrid-L.
• Task heads. SSDLite (V2) for detection, Mobile DeepLabv3 (V2) and LR-ASPP (V3) for segmentation, described under Performance.
• Descendants. EfficientNet scales a network of MobileNetV2-style blocks with a compound rule for depth, width and resolution.

Implementations

The official implementations are in TensorFlow: the TF-Slim model library for v1 and V2 and, for V4, the TensorFlow Model Garden (official/vision/modeling/backbones/mobilenet.py, per the V4 paper). TorchVision 0.29.0 has no MobileNet v1; it provides mobilenet_v2, mobilenet_v3_large and mobilenet_v3_small, quantizable versions of mobilenet_v2 and mobilenet_v3_large, ssdlite320_mobilenet_v3_large for detection, and lraspp_mobilenet_v3_large and deeplabv3_mobilenet_v3_large for segmentation. The V4 paper points to the timm library for PyTorch MobileNetV4 weights.

Comparing the blocks. The first example (PyTorch 2.14.1, one thread) builds one block of each of the first two versions and the whole of MobileNetV2 from its Table 2, and counts weights and multiply-adds with PyTorch’s FLOP counter (the whole network on a synthetic image of a bright square). (Reproducing MobileNet v1’s cost tables from its Table 1 is the subject of the paper article’s example.)

import torch
import torch.nn as nn
from torch.utils.flop_counter import FlopCounterMode

torch.manual_seed(0)
torch.set_num_threads(1)

def conv_bn(c_in, c_out, k, stride=1, groups=1, act=nn.ReLU):
    layers = [nn.Conv2d(c_in, c_out, k, stride, k // 2, groups=groups, bias=False),
              nn.BatchNorm2d(c_out)]
    return nn.Sequential(*layers, act()) if act else nn.Sequential(*layers)

def separable_block(c_in, c_out, stride):
    """MobileNet v1: 3x3 depthwise, BN, ReLU; then 1x1 pointwise, BN, ReLU."""
    return nn.Sequential(conv_bn(c_in, c_in, 3, stride, groups=c_in),
                         conv_bn(c_in, c_out, 1))

class InvertedResidual(nn.Module):
    """MobileNetV2: 1x1 expansion + ReLU6, 3x3 depthwise + ReLU6, linear 1x1 projection."""
    def __init__(self, c_in, c_out, stride, t):
        super().__init__()
        hidden = c_in * t
        layers = [conv_bn(c_in, hidden, 1, act=nn.ReLU6)] if t != 1 else []
        layers += [conv_bn(hidden, hidden, 3, stride, groups=hidden, act=nn.ReLU6),
                   conv_bn(hidden, c_out, 1, act=None)]       # no nonlinearity here
        self.body = nn.Sequential(*layers)
        self.residual = stride == 1 and c_in == c_out
    def forward(self, x):
        return x + self.body(x) if self.residual else self.body(x)

# MobileNetV2 (Table 2): expansion t, output channels c, repeats n, first stride s
V2 = [(1, 16, 1, 1), (6, 24, 2, 2), (6, 32, 3, 2), (6, 64, 4, 2),
      (6, 96, 3, 1), (6, 160, 3, 2), (6, 320, 1, 1)]

def mobilenet_v2(classes=1000):
    layers, c = [conv_bn(3, 32, 3, 2, act=nn.ReLU6)], 32
    for t, c_out, n, s in V2:
        for i in range(n):
            layers.append(InvertedResidual(c, c_out, s if i == 0 else 1, t))
            c = c_out
    layers += [conv_bn(c, 1280, 1, act=nn.ReLU6), nn.AdaptiveAvgPool2d(1),
               nn.Flatten(), nn.Linear(1280, classes)]
    return nn.Sequential(*layers)

def n_params(m):   # convolution and FC weights plus FC bias; BatchNorm excluded
    return sum(p.numel() for mod in m.modules() if isinstance(mod, (nn.Conv2d, nn.Linear))
               for p in mod.parameters())

def mult_adds(m, x):
    with FlopCounterMode(display=False) as fc, torch.no_grad():
        m.eval()(x)
    return fc.get_total_flops() // 2      # PyTorch counts a multiply-add as 2 FLOPs

def image(size):                          # synthetic RGB image: a bright square
    x = torch.zeros(1, 3, size, size)
    x[:, :, size // 4: 3 * size // 4, size // 4: 3 * size // 4] = 1.0
    return x

print("One block at 14x14, each family at its own width there (v1: 512, V2: 96):")
blocks = [("v1 separable 512->512", separable_block(512, 512, 1), 512),
          ("V2 inverted residual 96->96, t=6", InvertedResidual(96, 96, 1, 6), 96)]
for name, blk, c in blocks:
    x = torch.rand(1, c, 14, 14)
    print(f"  {name:33s} {n_params(blk):7,} weights {mult_adds(blk, x):11,} mult-adds"
          f"  activations in/out {x.numel():,}")
h, d1, t, k, d2 = 14, 96, 6, 3, 96
print(f"  V2 formula h*w*d'*t*(d'+k^2+d'') = {h * h * d1 * t * (d1 + k * k + d2):,}")

v2 = mobilenet_v2()
print(f"MobileNetV2, 224x224: {n_params(v2) / 1e6:.2f} M params, "
      f"{mult_adds(v2, image(224)) / 1e6:.1f} M mult-adds (paper: 3.4 M, 300 M)")

Output:

One block at 14x14, each family at its own width there (v1: 512, V2: 96):
  v1 separable 512->512             266,752 weights  52,283,392 mult-adds  activations in/out 100,352
  V2 inverted residual 96->96, t=6  115,776 weights  22,692,096 mult-adds  activations in/out 18,816
  V2 formula h*w*d'*t*(d'+k^2+d'') = 22,692,096
MobileNetV2, 224x224: 3.47 M params, 300.8 M mult-adds (paper: 3.4 M, 300 M)

For MobileNetV2 the count of 3.47M weights is above the printed 3.4M (the MobileNetV4 paper lists 3.5M); the multiply-adds match the printed 300M. The V2 block costs less than half as much as the v1 block at the same resolution, and its input and output are a fifth the size. Its expanded inner tensor ($96 \cdot 6 \cdot 14 \cdot 14 = 112{,}896$ values), however, is larger than the block’s input and output and than a v1 block’s, which is why V2’s memory argument relies on not storing that tensor whole.

ReLU in a low-dimensional space. The second example repeats the experiment of MobileNetV2’s Figure 1 on image features. A synthetic $64 \times 64$ image of a dark disk gives, at each of its 572 edge pixels, a two-channel Sobel gradient; these vectors point outward in every direction and form a rough closed loop around the origin: a one-dimensional set in a two-dimensional feature space. A random $n \times 2$ matrix $T$ embeds them in $n$ channels, as an expansion layer would. Without a nonlinearity, the pseudo-inverse of $T$ recovers every feature. With ReLU, a pixel’s feature $x$ can be recovered exactly from $\mathrm{ReLU}(Tx)$ when the rows of $T$ that are positive at $x$ still have rank 2; otherwise a segment of nearby inputs maps to the same output. The script counts both cases over ten random matrices.

import torch
import torch.nn.functional as F

torch.set_num_threads(1)
torch.set_default_dtype(torch.float64)

# Synthetic 64x64 image: a dark disk on a bright background.
yy, xx = torch.meshgrid(torch.arange(64.), torch.arange(64.), indexing="ij")
img = (((xx - 31.5) ** 2 + (yy - 31.5) ** 2) > 20 ** 2).double()[None, None]

# 2-channel feature per pixel: the Sobel gradient (gx, gy). On the disk's edge
# these vectors point outward in every direction: a rough loop, 1-D in 2-D.
sobel = torch.tensor([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]])
kernels = torch.stack([sobel, sobel.T])[:, None]               # (2, 1, 3, 3)
g = F.conv2d(img, kernels, padding=1)[0]                        # (2, 64, 64)
X = g.reshape(2, -1).T
X = X[X.norm(dim=1) > 1e-9]                                     # edge pixels only
print(f"{len(X)} edge pixels, feature dimension {X.shape[1]}")

# As in MobileNetV2's Figure 1: embed the features in n dimensions with a random
# n x 2 matrix T, then apply ReLU (or nothing) and ask what can be recovered.
print("   n   linear: max error   ReLU: pixels sent to 0   ReLU: pixels recoverable")
for n in (2, 3, 5, 15, 30):
    err_lin, zero, ok = [], [], []
    for seed in range(10):
        torch.manual_seed(seed)
        T = torch.randn(n, 2)
        Z = X @ T.T                                    # embedded features, linear
        X_back = Z @ torch.linalg.pinv(T).T            # undo the linear embedding
        err_lin.append(((X_back - X).norm() / X.norm()).item())
        R = torch.relu(Z)
        zero.append((R.sum(1) == 0).double().mean().item())
        # x can be recovered from ReLU(Tx) exactly when the rows of T that are
        # positive at x still have rank 2 (solve on those rows alone).
        ranks = [torch.linalg.matrix_rank(T[a]).item() if a.any() else 0 for a in Z > 0]
        ok.append(sum(r == 2 for r in ranks) / len(ranks))
    span = lambda v: f"{min(v):.2f} to {max(v):.2f}"
    print(f"  {n:2d}   {max(err_lin):.0e}               {span(zero)}             {span(ok)}")

Output:

572 edge pixels, feature dimension 2
   n   linear: max error   ReLU: pixels sent to 0   ReLU: pixels recoverable
   2   7e-15               0.00 to 0.41             0.00 to 0.41
   3   4e-16               0.00 to 0.27             0.50 to 0.50
   5   5e-16               0.00 to 0.23             0.58 to 0.99
  15   5e-16               0.00 to 0.00             1.00 to 1.00
  30   6e-16               0.00 to 0.00             1.00 to 1.00

With ReLU, two, three and five channels lose information for every one of the ten random matrices (seeds 0 to 9), and from 15 channels on every pixel is recoverable in every draw. The exact half at $n = 3$ has a simple cause (our derivation): the disk’s gradients come in opposite pairs $x$ and $-x$, a row positive at one is negative at the other, so with three rows exactly one of each pair has at least two positive rows. The pattern is the one Sandler et al. show for their spiral and the reason for their design: ReLU belongs in the wide expanded layer, not in the narrow bottleneck. The example illustrates their argument; it does not show the accuracy effect, which only training shows.